It will be hard work. It’s always hard work, and hard work from everybody within the team—technical director, mechanics, drivers, engineers—everyone in the team.
—David Leslie, racing driver
There are times when the hardest thing a security professional can do is work with the IT department. It’s strange since many security professionals have come from an IT background. Speaking for myself, I moved into security from network engineering after a large firewall project. Of course, IT professionals work on security tasks their whole career, from cleaning up malware to managing user accounts and passwords. The IT team is on the front line of security every day. If your organization is hacked, they’re going to be just as busy as you are, if not busier. When the auditor delivers a failing report, it’s going to reflect on IT’s efforts. Even worse, high-powered hacking groups and spy agencies single out IT personnel as high-value targets . They know sysadmins have the best access privileges in the company. If they can take over their accounts, the whole network is their buffet table.
So why do IT and security not get along sometimes? One of the biggest reasons is that IT’s entire job revolves around making sure things don’t break and their workload is manageable and building out whatever new thing that the business requested. Sometimes new security controls or processes impact one of these things. Sometimes that alteration is drastic, like when introducing change control or formalized user account management for the first time. This can really burden the IT team with unplanned (and often in their mind, unnecessary) work that could set them back significantly on their other projects. Security controls can also introduce new failure points or roadblocks in IT systems.
There are also sometimes clashes of technical expertise . Many people in IT have specialties and may understand the technical nuances of threats, vulnerabilities, impacts, and controls better than the security team. Therefore, there can easily be disagreements on risk assessment and treatment decisions. As there are no universal standards for what is acceptable security yet, members of the IT team may favor a different approach than the ISMS committee or IT security team. Many IT experts have favorite solutions or vendors, which they may feel are superior to the chosen solution.
You need to overcome these barriers because IT is critical to a successful security program. They will make or break the implementation of the technical controls. They are the eyes, ears, and hands of the IT security group. In the end, both groups are part of the security team and serve the business together.
IT Security vs. IT
So far, I’ve been speaking as if IT security is in a different department than the IT. That may not always be the case in every organization. However, you should be aware the job focus is different. I am distinguishing between the security team and the IT around the implementation and management of technical security controls. The CSO leads the security team, as well as being responsible for assessing and managing risks. The IT team has direct hands on the technology, managing the day-to-day business of keeping the systems and controls running properly.
In the big picture, their goals are the same: keep the tech up and support the business. However, some security requirements may run counter to availability and operations goals of general IT. First, it’s a question of responsibility and leadership. Security is not a technology add-on that you apply or install into existing IT systems. Many IT people are tech-heads, which is why they’re so good at their jobs. They may think they’ve fixed all the security problems by implementing controls. As we’ve seen so far, security is a lot more than just implementing controls. Security begins with goals, risks, and decisions, not some device or software package. It is about deep analysis of risk and requirements. Sometimes the solutions aren’t technological. Sometimes the technology itself is the risk. Understanding technology does not mean someone understands security.
Second, there is a divergence of goals. The security team’s highest priority is managing risk to the level acceptable to management. The IT team’s priorities are satisfying user demands, implementing projects, and dealing with outages. In general, the IT department’s mission revolves around granting access to information and resources, not restricting access. All things being equal, IT prioritizes availability and integrity over confidentiality. While security cares more about confidentiality than the other two. When push comes to shove, I’ve seen IT departments opt to let something be insecure to maintain uptime and keep users happy, despite the elevated risk. There are also cases where security has to reverse a decision that IT has made like this. The security team and ISMS committee own decisions around risk, not the IT department.
A key part of the security team’s job is managing the risk of insiders. Some of the most damaging types of insiders are in the IT department. I’ve even seen cases where a malicious insider was leading the IT department. As the most knowledgeable and powerful tech users in the organization, there are times when an IT staff member can develop a sense of entitlement around technology. These individuals are more prone to feeling that rules are for the users, not them. By accident or on purpose, this can lead to security vulnerabilities or insider malfeasance. This is why you should separate the security group from IT, and give security the mandate to review the IT team’s work. Sometimes security people can go bad or be careless too. So give IT full system privileges and give read-only access to the security team. Security is more of an oversight and management role, while IT does the actual work.
There can be some overlap, but in general, Table 9-1 breaks it down.
Table 9-1. Security Tasks vs. IT tasks
Security Tasks | IT Tasks |
|---|---|
Approves and reviews firewall rules. | Implements and runs firewalls. |
Review and responds to IDS alerts. | Implements and runs IDS system. |
Runs vulnerability scans. | Fixes found vulnerabilities. |
Monitors encryption settings. | Manages VPNs and storage encryption tools. |
Writes and reviews security policies and standards. | Provides input and follows polices and standards. Writes procedures. |
Reviews admin logs on privileged systems. | Works on privileged systems. |
Techie Traps
I’ve mentioned the problems and misunderstandings that security and IT can run into, but not at a detailed level. Let’s look at some specific problems and how to deal with them.
The Infinitely Long IT Work Queue
IT is busy. So busy they don’t have time for your security project. They won’t say no. In fact, they will probably cheerily accept the work—and then dump your project into a to-do list hundreds of tasks deep. Why should a security project be treated any different from all the other critical business requests that are pouring into the IT queue every day? Okay, before we start complaining, let’s back up and look at how IT works.
The IT department is probably understaffed (just you ask them if you aren't sure) and they have a full plate of things to do. It may seem like all they need to do is read a few pages of documentation, run a few commands, and the work is done. If you’ve ever worked in IT or even with a computer, you know that is rarely the case. The simple freedom to choose which projects to work on might be a difficult goal for them to achieve. IT staff often start their day putting out fires and dealing with angry users. This is on top of whatever routine maintenance they have to do and the projects they already had scheduled. This doesn’t include the typical IT system, which is a big haystack of legacy architecture, often poorly implemented and full of bugs. Since IT serves the entire organization, they are often also handed new high priority tasks on a daily basis (or even several times per day). Security tasks must compete with other business divisions for prioritization in the queue.
Some folks perceive IT as having time for just one more thing. Well, if they do, it means something else isn't getting something done. Sometimes the things that aren’t being done can directly affect security like ensuring antivirus clients are working or removing accounts for terminated users. Throwing new security work into the IT work queue can be like squeezing a balloon. Reduce risk with one control could mean increasing risk on others by taking away attention to critical work or creating new unforeseen systems interactions.
To manage this, you need to work with the IT leadership to ensure that security risks are being addressed by priority. Be realistic about what you can ask IT to complete in a timely manner. You may need to prioritize and help them chose what can be actually finished. This is why you’ve done an extensive risk analysis and know what’s important.
Poor IT Hygiene
Related to this problem is the IT team that has neglected or hasn’t had time to clean up after themselves. Here are some examples of security hygiene that is overlooked in the bustle of everyday business:
Expired HTTPS certificates still in use
Terminated local user accounts still enabled
Open but unused firewall ports
Out-of-date access control lists
Partially implemented projects leaving critical files around
Overly permissive access permissions (“we did that just to get things working until we get a chance to come back and fix them… someday”)
Critical system configurations not backed up
Audit logging not set up properly
Out-of-date antivirus signatures or clients
Missing patches on local systems (Usually, the OS is patched, but the browser? Critical apps? Utilities? Java? Acrobat? Flash?)
Obsolete encryption algorithms still enabled
Important passwords not rotated in a timely manner
Missing or out-of-date documentation on critical systems (impacting business continuity)
Old data or configuration files remaining uselessly in directories
Retired hardware and media not scrubbed for disposal
Entire desktops or servers gone off the radar for patching and updates
Out-of-service operating systems still in active usage
All of these kinds of things need to be kept up to maintain the same expected level of risk control. This means the security team needs to schedule a regular audit for these things and assign cleanup tasks to IT. There’s been more than one occasion of a serious breach happening because IT thought a machine was patched with current antivirus when it wasn’t.
The Black Hole
Another similar problem related to IT workloads is “the one firewall engineer” or the one security-savvy developer . All the security projects and maintenance tasks get bottlenecked because there is one person in IT who can do it. No one else in the department has the skills (or the willingness) to do that particular complicated job.
There are several strategies to deal with this. First, if that security work is not a recognized part of that person’s job, it should be. Often by explicitly assigning that task to that individual, they can be freed up from other tasks and concentrate on opening up the bottleneck. You can also have them document and train other staff to perform these tasks, so that the workload can be spread out. Sometimes this means slowing down the pace of work so that they can document and train, and thus everyone can move faster later.
Perpetual Design
Like many other IT projects, security projects can become stuck in a perpetual design phase. IT may have opinions and insights into security matters. There could be long hours of discussion and analysis, only followed by the realization that more discussion needs to be done. Sometimes you’re cursed with the opposite, with no workable ideas or feasible solutions. Maybe there are problems or ideas, but no one is willing to bring them up for fear of being assigned new work. The whole design process can get difficult when both IT and security are bringing different perspective to the table. A Socratic method with multiple perspectives is desirable as inputs, but not at the expense of a stalled project. Security’s role is to hear them and let them know that they’ve been heard, and then ensure them that the design work is completed.
There are several different approaches to help fix this. The first is to make use of the segregation between security and IT. The security team and the ISMS committee should decide upon the type of control and define the requirements for the control. IT can choose the specific technology and vendor, as long as it meets the requirements. When teams are stuck on designs, then this is a good time to bring in outside help. A consultant or value-added reseller may be useful. If there is still discord, it is helpful to remind the design team that they are all working to improve customer service and reduce risk to the company.
Sometimes design paralysis is caused by mistaken goal that the design needs to be free of flaws before going into operations. No design is going to be perfect. In fact, very few designs remain the same once implementation begins. The whole point of PDCA is continuous improvement. You unlikely to get something exactly right the first time. Try something, measure it, and make changes. Gradual improvement in the face of the real world will move you closer to perfection much faster than trying to think of every possible scenario before beginning.
Dragging Projects
Once a project is underway, it could stall or be perpetually delayed. Sometimes the causes are clear, like lack of funding or the necessary skill sets. Other times, it’s not as clear. Like any other IT project of consequence, you need to establish clear milestones and deliverables to track the project’s progression. The ISMS committee needs to clearly assign project responsibility. There should be no doubt which individual is responsible for making sure the project is completed per the requirements. Never assign a project to a department or a team, instead assign it to the manager of that team. That manager can delegate the work within the team. If necessary, you can create a RACI diagram to show who is responsible for what parts of the project.
Sometimes projects can be delayed because there are disagreements to the solution. This can include individuals who did not voice complaints or had their concerns ignored, and work silently behind the scenes to delay things. This isn’t usually outright sabotage, but simply the de-prioritization of key project tasks but always with a plausible excuse. The security team needs to engage complaints head-on from the beginning. Even if they don’t agree with what people say, they do need to make sure that everyone has been heard and knows that they’ve been heard. Remember the listening skills laid out in the last chapter. It is also helpful to get explicit buy-in from department heads before assigning projects. This means asking them if someone on their team can complete the project and getting their assent. If things really get ugly, you can always blame the auditors for forcing the organization to do this.
Other Tools
You can use few other tools when working with IT. Don’t forget simple things like appealing to their professionalism. Everyone is on the same team and things need to be secure. It seems obvious but sometimes reminding people of that can get them on-board.
Another tool to getting their buy-in to a new process is the IKEA effect. This cognitive phenomenon describes how people place a higher value on products that they assemble themselves. Instead of forcing a new control or policy onto the IT department, make the team part of the process. Have them submit the first draft, or allow them to critique the procedure that you’re proposing. As I mentioned before, let IT choose the specific solution, with their favored vendor. As long as the end product is the same, who cares how it gets there? Make them a part of the security process in whatever way you can. Give them advanced warning when you think something big is coming, like a big patch or a major audit finding.
Remember that IT is a department with their own needs and wants. When rolling out something new, consider what’s in it for them. Can this new security process help them serve their users better? Can it improve efficiency for them? Can it save them money? Can they use it to push off unwanted work? Can they use this project as a learning opportunity or a chance for advancement?
Speaking of learning opportunities, a useful thing for the security department to do is to enlist security championsthroughout the organization. Like everyone else, the security team is probably understaffed and can’t be everywhere at once. Security champions are your technical leaders in other departments. Both IT and the development departments are excellent places to cultivate and support champions. Once you get permission from their boss (or ask them to nominate a champion), you give these folks extra training and tools. You can invite them to ISMS committee meetings as observers. They become the departmental experts in a particular control or security process. Good controls to find champions for include change control, system hardening, vulnerability scanning, secure development, and e-mail malware filtering.
Don’t Over-Argue
When all else fails, you may find yourself in an intractable disagreement with someone. If so, don’t prolong it such that you make a permanent enemy of them or anyone else. Your task is to state your case as clearly as you can and leave it at that. You rarely can change someone’s mind by repeating your argument over and over again in exhausting detail. You probably have a better chance of swaying any bystanders with maintaining the high ground by not losing your cool and becoming annoying. Once you’ve made yourself clear, agree to disagree and seek another strategy to move forward. Maybe you need to appeal to authority or the ISMS committee. Maybe you need to concede and make up for it some other way. However, don’t remain in a pointless argument. No one wants to sit around and watch two techies go at it for hours or multiple meetings.
Working with Other Security Pros
Sometimes collaborating with other IT security professionals can be a challenge. There are a wide variety of security specializations and backgrounds. Remember that there a lot of different standards and best practices in the security field. Not all of them line up with each other in the details of philosophy or execution. In addition, auditors are security professionals as well, and you may occasionally disagree with some of their conclusions. If you’re a security professional reading this, you may already disagree with me.
In addition to the techniques that I’ve mentioned so far, as well as your good listening skills, the next best thing to do is to understand what different security professionals do and how they got there. There are many paths to management and mitigation of risk. Disagreement among security professionals is often about which path is better, even though the goal is the same.
IT Security Roles
There are many different kinds of jobs and roles within IT security. For simplicity’s sake, I’m going to break them down into three functions: builder, tester, and responder.
Builder
This book is mostly for builders. Builders design, implement, operate, and maintain security systems and technical controls. Many of these security professionals have come from traditional IT careers. The role is about building IT systems and processes that can withstand attack. Sometimes the focus is small, on a single type of technology, and sometimes it covers an entire organization. Whatever it is, it’s all about defense. The following are the kinds of job titles that security builders have:
Chief Security Officer
Director of Security
Security architect
Network security engineer
Security software developer
Security systems administrator
Information assurance analyst
Technical director
Security analyst
Tester
Testers get to break things. It’s sometimes considered the most fun job in security. They act like hackers and find the holes before the bad guys do. They use all kinds of tools and techniques, sometimes getting very hands-on in trying to subvert the technology. Auditors also test things, although they are more upfront as they review procedures and documentation. The goal again is defense, but this role is more about making sure things were built to last. These are some of the job titles:
System, network, and/or web penetration tester
Application penetration tester
Source code auditor
Vulnerability researcher
Exploit developer
Ethical or white hat hacker
Security research engineer
Security auditor
Responde r
When all else fails, you call in the responders. These are the folks who clean up the mess or tell you what when wrong and who’s to blame. They can wade into the aftermath while maintaining a cool head. Some of them are involved in legal work, gathering evidence and building a case. Others focus solely on the cleanup efforts. These are the job titles:
IT forensics expert
Security operations center analyst
Forensic analyst
Incident responder
Malware analyst
Computer crime investigator
Prosecutor specializing in information security crime
Intrusion analyst
Disaster recovery analyst
Business continuity manager
Hiring for Security
You may want to expand your security team by hiring or bringing in contractors or consultants. Given the variety of skills and experiences associated with security roles, the following are some questions that you can use when examining candidates.
How Do They Do Risk Analysis ?
Given how this is a bedrock skill for what we need to do, you should definitely ask the candidate to describe how they have performed risk analysis in the past. You could even ask them to walk through an analysis with you by giving them a hypothetical problem. See what they focus on and what methods they use. Do they use quantitative or qualitative analysis? Do they know the difference? How do they gather evidence? Do they focus solely on technology or best practices? Do they dismiss new and possible unproven technologies out of hand? Do they mention or understand the legal implications of risk? How about compliance and audit issues? Do they have pet risks that they fear? Can they prioritize risk? Are they focused (or even aware) of potential business drivers behind the risk? Do they use a published risk model? Or do they do it ad hoc?
How Good Are They with Compliance?
Does the candidate understand the compliance requirements that you have? Have they acted as an assessor or auditor on them? Have they implemented to pass audit? Do they understand what passes a control and what fails? Do they understand scoping and scope barriers? Can they audit a third party for that compliance regime? How do they keep up with changes to the audit requirements over the years? A competent individual in a compliance requirement understands the intent behind each control objective. An expert knows how to subvert the rule.
What Controls Are They Familiar With?
Which controls has the candidate worked with in the past? Yes, you’ll hear a lot about specific vendors and technologies, but has the candidate integrated them into a running organization? Maintained them? Audited them? Replaced them? Beyond technical controls, what about working on processes and policies? What about physical controls? Are they overly focused on one kind of control and weak in the others? Do they understand that they have this weakness? Do they understand the applicability and efficacy of control types? For example, what risks does transmission encryption mitigate? What risks does it not mitigate?
How Do They Meet the Challenge ?
How does the candidate keep up with security? It’s a constantly changing field. Do they read, attend conferences, meet with peers, or do online training? See if they push themselves, not just to maintain but also to learn new things. Do they pursue new certifications and skills? Do they have a mentor? Do they mentor others? I’m a big fan of individuals with a “constant learning” attitude. Not having this outlook can be an indicator of burnout.