Protecting Virtual Environments from Hypervisor Sabotage
“Virtualization is not inherently insecure. However, most virtualized workloads are being deployed insecurely.”
—Neil MacDonald, VP and Gartner Fellow, Gartner Inc.
Organizations moving their physical server infrastructure onto virtual platforms for cost savings are finding their virtual hosts and guests are now open to new security and non-compliance risks. Workloads shifted to virtualized platforms to realize operational cost efficiencies are done so at potentially high security costs if proper security policies and tools are not established prior to implementation. As if we didn't need to state the obvious, virtualization doesn't make it any less likely that good people will do bad things.
Administrative access to the server hypervisor/VMM layer and the administrative tools used to access these layers must be tightly controlled to maintain a strong security posture. When multiple resources with many different levels of privileged access are consolidated onto a single physical server without sufficient workflow protocol, separation of duties for network and security controls could be compromised and security policies circumvented.
A least privilege solution provides granular privilege identity management (PIM) across guest operating systems as well as hypervisor hosts through a single centralized management console. Privileged access security risks are mitigated, compliance requirements met, and organizations can adopt virtualization with confidence.
A least privilege solution also provides a cost-effective dedicated solution to centrally address risks from unmanaged administrative privileges in virtualized datacenter environments. In a secure and compliant environment, users privileged access to virtual resources are managed to give them access to only what they need to do their job.
Protecting virtual environments is a difficult and tedious task. On one hand, privileges in this setting must be granularly managed to ensure complete security. On the other hand, it takes less time and energy to allow users to operate with unmanaged privileges in virtualized datacenter environments. Fortunately, there is an answer to this double-edged question, and it allows for the risks in said environment to be mitigated.
Virtual Theft
A key factor to consider when approaching virtualization security is that the hypervisor is always going to be a high-value target due to its control over the entire virtual environment. The misuse of privilege at the hypervisor level can present the following risks:
A hypervisor breach would allow “authorized access” to all hosted/virtualized workloads. Essentially finding the “key to the hypervisor” gives you the “keys to the kingdom.” While root password access in a server environment gives you the keys to the vault inside the bank, the hypervisor gives you access to multiple vaults inside the bank. Unmanaged administrative access to the hypervisor also facilitates two other potentially damaging problems:
- Incorrect or unauthorized configurations magnify risks
- “Virtual sprawl” due to lack of policy enforcement.
Hypervisors must be considered mission-critical and secured appropriately, just like operating systems, and require security because of the risks to the system and applications. Typically, users are simply given administrator rights to perform configuration or policy changes to the virtual environment.
Even if mounting disks is to be allowed, there are built-in workflows to capture policy requests and approvals for that command. Additionally, workflows can be set up to alert a manager if the mount command is used. Finally, a least privilege solution creates an indelible searchable keystroke trail, which would capture all activity that the administrator conducts during the mounting.
Committing virtual theft (Figure 6-1) isn't a difficult thing to accomplish without least privilege management of the hypervisor.
Figure 6-1. Committing virtual theft
Step 1) Administrator with root credentials on hypervisor has access to “Virtual Machine 1” with sensitive data.
Step 2) Administrator copies virtual machine1 to get an exact replica “copy of Virtual Machine 1,” containing sensitive data.
Step 3) Administrator kills “Virtual Machine 1” copy, and mounts the disk image.
Step 4) Administrator is able to navigate to and gain full access to sensitive data on the mounted disk image, unnoticed, while “Virtual Machine 1” is still running fine.
A least privilege solution can prevent this scenario by disallowing the administrator to use the mount command, while allowing her to use other commands required to carry out her job effectively.
The federal government has even recently taken note of such threats, as shown in an October 2010 DarkReading article titled Pentagon's Insider Threat Push Offers Lessons for Enterprises. The research arm of the Pentagon, known as the Defense Advanced Research Projects Agency (DARPA), put out a call last week for better methods of detecting employees just before or after they go rogue and misuse their privileges. “Unfortunately, virtual insider threats have been largely identified due only to incompetence on the part of the perpetrator or by accident,” says Peiter “Mudge” Zatko, the manager in charge of the program at DARPA.
If you search on “virtual guest vulnerability” in your favorite search engine and filter to video results only, you will find a step-by-step guide demonstrating how to do this as well as how to protect against this form of virtual information theft.
Desktop Virtualization
Virtualization has become a hot buzzword over the past several years, and for good reason. With the introduction of Virtual PC several years ago, and now with Windows XP Mode, Microsoft Enterprise Desktop Virtualization (Med- V) & Application Virtualization (App-V), Microsoft is no stranger to virtualization.
With all these technologies, it is easy to understand that there is significant confusion in the market about what virtualization means for privilege management, specifically the ability for virtualization to help with the removal of administrative rights from users. While virtualization can add enormous value in many areas, many organizations will rely on virtualization to help specifically with application compatibility problems.
For example, if an organization cannot get an application to run on Windows 7, even after trying to shim the application with the Application Compatibility Toolkit, the ability to virtualize the application with one of the technologies listed earlier is available.
Unfortunately, virtualization does not help with the elimination of administrative privileges; it simply shifts the problem from a physical world to a virtual world. Some organizations may be comfortable with loosened security for their virtual environments, but most will want the same level of security in the virtual environment that they have in the physical environments, which means enforcing least privilege in the virtual world as well as the physical world.
Removing administrator privileges from accounts on virtual machines is still a critical part of an organizations security posture. If organizations wish to virtualize applications or desktops, and the users still need to perform administrative tasks or run applications that require administrative rights in the virtual environment, then the user will need to be logging in as an administrator. This means that the virtual environment is still the subject of the same security issues as when they are logged in to a physical machine.
Virtualized Desktop Infrastructure (VDI) environments, such as Citrix, require users to run as administrators to access applications and install associated DLLs. Malware and hackers can exploit these administrative privileges. Additionally, they allow users to:
- Change standard desktop configuration settings
- Install unlicensed software and disable security settings
Similarly, in Microsoft or VMware environments, users with administrative access often inadvertently delete printers for other users, causing disruption in business and creating unnecessary helpdesk overhead.
According to Gartner, VDI adoption is growing at a rapid rate. By 2013, it is expected that more than 40%of the worldwide professional PC market and 70%of organizations will have adopted PC application virtualization. Because of privilege requirements associated with application delivery, customers are forced to run their users as administrators in VDI environments. Since the desktop application support cost reduction, Forrester estimates of 80% are compelling; customers accept the relaxed security posture, opening themselves up to risks from accidental, intentional, and indirect misuse of privilege.
Attaining least privilege user posture in virtualized desktop environments is challenging and customers are consistently forced to make compromises on security in favor of cost-savings.
Desktop Registry and File System Virtualization
In Windows Vista, Microsoft introduced Registry and File System Virtualization to solve some of the problems with application compatibility. Some applications require full access to certain areas of the operating system that are off limits to standard users. These applications might try to write data to the “Program Files” directory or the “HKEY_LOCAL_MACHINE” hive of the registry, for example. Standard users do not have permission to write to these areas of the file system and registry, so when a user launches an application on Windows XP that needs access to these locations, they would eventually see an error when the application tries to access data stored in these locations.
In Windows Vista and Windows 7, Microsoft has redirected the access to these locations to a virtual store in an area of the operating system that the user has access to. This attempt to solve the problem of application compatibility for applications that need rights to areas of the file system or registry that are off limits to a standard user introduces several problems. One example is that applications may not be compatible with each other.
For example, if an application has written data to a virtual store, another application that needs access to the data in the virtual store will not be able to access it. A similar problem occurs when an application stores data in a virtual store and multiple users of the same machine need access to it. A simplified example of this would be a game that stores its high score file in the “Program Files” directory. With file system virtualization, the high-score file would be stored in the user's profile, instead of Program Files, and thus any subsequent player would store a copy of the high score in their profile. This means that every user of the machine would have the high score! Imagine how this might impact line-of-business applications that multiple people use on the same machine.
Another issue with registry and file system virtualization is the fact that it can cause significant confusion for end users. If an end user has traditionally stored files in a directory that will be virtualized in Windows 7, the user will not know where to go to get the files if they need to copy, view, or e-mail them because the files will no longer be where the end user intended on storing them; they will actually be in the virtual store in the user's profile.
Reads and writes to the following location:
C:Program Files (x86)My Application A
Would be redirected to the virtual store:
C:Users\%username%AppDataLocalVirtualStoreProgram Files
(x86)My Application A
All subsequent access for that specific application would be redirected to this location as well; however, other applications that need access to this data will not know where to go to get it because it has been virtualized.
The Virtual Shell Game
Ever have a magician or street performer get you to bet on the shell game? You know the game, the one where something is hidden under one of three shells and then they move the shells quickly while you try to keep track of where you think the object is. You can never win this game because the object is never where you think it is courtesy of some incredible sleight of hand. Turns out this is the game that can happen when administrators of virtualized environments don't always set the permissions and/or policies correctly. As stated previously, you may think that everyone can see (or has permission to view/edit) some particular file or application, when in fact only the admin or a subset of users can. This challenge gets blown into huge proportions courtesy of the ease at which new virtual environments can be launched by the administrator with hypervisor access.
Commonly called “virtual sprawl,” this can become the electronic version of the shell game when good people either intentionally want to hide something or accidently misplace key data or applications. Virtual administrators are like the physical server administrators described in the last chapter, but with the ability to do things without necessarily needing more hardware, greatly reducing the typical lags that may result in better planning. It seems that every organization we talk to nowadays is undergoing huge shifts to virtual or cloud-based computing all in the name of green computing, reducing IT capital and operational expenses. Unfortunately, this is also giving rise to a different type of potential insider threat.
Controlling Virtual Sprawl with Least Privilege
Virtual sprawl is the new plague of IT (Figure 6-2). You know a problem has achieved mainstream status when such headlines as “Virtual Sprawl Hits Wall Street” appear.
New applications have made it so easy to create virtualized environments on existing platforms that it seems one crops up every time a line of business manager requests a new resource. If you thought server sprawl was bad, just look at what can happen in a virtualized environment.
It usually doesn't matter what the corporate policy, governance, or regulatory requirements implications are to the administrator creating that new environment; however, the implications to the CSO or anyone looking to satisfy audits are potentially huge.
Without proper controls, administrators with full access to the hypervisor can intentionally, accidentally, or indirectly misuse privilege and cause harm.
So how can an organization protect itself from virtual sprawl?
The easy answer is don't allow it. But that is a lot harder to enforce than to say. According to David Lynch in the Virtualization Journal, “Virtual Sprawl is not the problem. The real solution is to attack the cause as well as the symptom.” A key “cause” is the unregulated administration of the hypervisor. Without policy-based controls, an administrator of your virtual environment has carte blanche to do as they please. Implementing a PIM solution will ensure a least privilege environment whereby the admin will only be able to do what policy dictates and the embedded logging will facilitate any audit and remediation as required.
In addition to virtual sprawl, a PIM solution correctly implemented can also eliminate other potential harms caused by misuse of privilege in virtualized environments.
Figure 6-2. Architecture of the cloud
Top Ten Reasons to Implement Least Privilege for Virtualized Servers
Taking a more tongue-in-cheek approach to highlighting the types of privilege misuse that occurs daily on virtual servers inside most organizations, we thought that a top-ten list approach might appeal to you as well. How many of these have you seen throughout your organization?
#10—Sam, the CSO, can now sleep nights knowing that excess privileges in virtualized environments will no longer be responsible for failing a SOX, HIPAA, PCI, DSS, GLBA, or FDCC and FISMA audit (even though he isn't required to even deal with the last two).
#9—Andy the Auditor can get a full report of who has what entitlements, even across virtualized environments, to instantly satisfy compliance successfully instead of taking weeks of manual effort.
#8—Ted in Tech Support won't be able to reset file and directory permissions on any virtual server he has admin rights to so liberally that anyone with a login can access confidential data just because it makes his job easier.
#7—Sid in Development won't be able to download Apache applications or any other unauthorized open source “tools” and potentially inject malware into our corporate network because he was able to commandeer his own virtual server admin credentials.
#6—Fiona and Felix, our new VMware administrators, won't keep making the same View Composer enablement mistake.
#5—Vito, the ever-industrious programmer and closet gamer, will not be able to run Runescape bots on our corporate virtual servers.
#4—Alice, our outsourced VM support engineer, will no longer be responsible for DNS misconfiguration errors as her role won't facilitate this level of admin privilege.
#3—Fred in IT won't be able to steal company data from our virtual servers like that Goldman programmer and sell it to a competitor for $1M.
#2—Sarah, the CIO, will no longer have to fear that virtual sprawl will be the next topic for a board review because policies weren't enforced at the hypervisor admin level.
#1—Tony, the Palo Alto VMware administrator, will no longer be able clone virtual servers with sensitive data and then delete and remount them to bypass security and steal company secrets.
Role-Based Access Control
One of the most challenging problems when managing large networks, virtual or physical, comes from the high costs and complexities of IT security administration. To help counteract these threats, the conception of Role-Based Access Control (RBAC) began focusing around multi-user and multi-application online systems that were pioneered in the 1970s. The essential notion of RBAC is establishing permissions based on the functional roles within the enterprise, and then applying a spotlight on aptly assigning users to a role or set of roles.
RBAC has evolved into a dominant blueprint for advanced access control mainly because the end goal is to reduce the complexity and cost of security administration for large or small networked applications. From theoretical inception to the present practical interpretation, most IT vendors have incorporated RBAC, in some form or fashion, and technology is finding applications in a broad spectrum of industries, ranging from healthcare to defense, in addition to the mainstream commerce systems for which it was designed.
Vendors have incorporated RBAC features into their database, system management, and operating system products. Ironically, while there has been a growing global market for a product solution to infuse an RBAC concept into reality, every vendor's products have been independently developed without any attempt at standardizing salient RBAC features.
As a result, RBAC is seen as an amorphous concept interpreted in different ways by various researchers, organizations, and system developers ranging from simple to elaborate and sophisticated RBAC models. Without standardized models that can be applied and understood between organizations, it becomes more apparent that these products may never provide the true reduction in cost and complexity that RBAC was originally intended to solve.
After acknowledging the problematic circumstances that could be achieved due to no RBAC regularity, the National Institute of Standards and Technology (NIST) produced the NIST RBAC model to provide a blueprint of standardization over a core set of RBAC features.
The benefits of this undertaking include a common set of benchmarks for vendors already developing RBAC mechanisms to implement into their enterprise operations. It will also give IT consumers, the principal beneficiaries of RBAC technology, a basis for creating uniform acquisition specifications.
Each RBAC implementation varies in its capabilities and method of management. In a multi-platform environment, these differences introduce higher administration hours and costs because the various RBAC models are not consistent in administration and operation methodology. The differences among these implementations also increase the potential for misconfiguration and related security issues. Lastly, most vendors' RBAC solutions are to some extent “host-centric.” Meaning maintenance operations may have to be performed on each host.
RBAC Is Not the Same as ACLs
RBAC differs from access control lists (ACLs) used in traditional discretionary access control systems in that it assigns permissions to specific operations with meaning in the organization rather than to low-level data objects. For example, an ACL could be used to grant or deny “write access” to a particular system file, but it would not dictate how that file could be changed.
In an RBAC-based model, an operation might be to create a “credit account” transaction in a financial application or to populate a “blood-sugar level test” record in a medical application. The assignment of specific permissions to perform a particular operation is beneficial, because the operations are granular in meaning within the application.
RBAC has been shown to be particularly well-suited to separation-of-duties requirements, which ensure that two or more people must be involved in authorizing critical operations. Necessary and sufficient conditions for safety of separation of duties in RBAC have been analyzed.
An underlying principle of separation of duties is that no individual should be able to affect a breach of security through dual privilege. By extension, no person may hold a role that exercises audit, control, or review authority over another concurrently held role.
Too Much Trust?
In a Computerworld article from November 2010, exploring the “scary side of virtualization,” the reporter took some time out in a sidebar to offer some sage staffing advice.
His riposte, “Beware the All-Powerful Admin,” made clear the risk of giving server admins the “keys to the kingdom”—not a good thing, so consultants and IT execs unanimously agree.
They might, for example, create virtual FTP servers “or they may inadvertently use a virtual-machine migration tool to move a server onto different hardware for maintenance reasons, without realizing that the new host is on an untrusted network segment.”
His sage advice is to establish a clear separation of duties in virtual infrastructures, and develop a strong change-management process that includes issuing change-management tickets.
We naturally would agree, but with one caveat. Businesses don't rely on trust alone. We also don't invite businesses to put their faith in some kind of metaphysical state that transcends our human frailties, we simply invite you to recognize that people can and do make mistakes, and when they are people with the “keys to the kingdom,” these mistakes can be costly.
Better to trust your people and take out an insurance policy against human frailties, whether those be fat-fingered mistakes, or willful misuse of responsibility. In any environment, especially the deployment of virtualized environments, strong identity management practices, and specifically control around privileged access, must be put in place. As the number of virtual hosts increases, there is a natural tendency to create islands of identity that are difficult to manage. As individual virtual servers are created that serve the needs of departmental applications, there will typically be a push from the departments for them to own the access to the server, specifically the privileged access, in the name of departmental efficiency. As the number of identity sources increases, the prospect for orphaned or inappropriate privileged access increases. Without a well-orchestrated management scheme for identity management and privileged access, the company will soon lose control, compromising security and audit compliance.
Least Privilege Architecturally Defined for Virtualized Environments
It is not enough to implement a least privilege solution to managing the physical server that houses a virtualized environment; you will also need to ensure each virtual server is individually protected as well.
Virtualized Least Privilege Value
A least privilege solution put in place specifically for virtualized environments (Figure 6-3) enables organizations that move to virtualized platforms to control administrative access to the hypervisor/VMM layer while still realizing all virtualization cost efficiencies. Features include:
- Granular delegation of administrative privileges
- Detailed and flexible reporting, including keystroke logging of admin activities
- Two-click entitlement reports
- Programmable role-constrain mechanisms for segregation of duties
- Secure virtual guest and host hypervisors
- Heterogeneous virtualization platforms such as VMware ESX, Solaris Zones, AIX WPAR, and IBM z/VM
Weighing-In
Virtualization has been, and will continue to be, not just a buzzword but a very real and viable approach for lowering costs in the IT sector. As technology advances, more and more companies will move their assets into this space. We've discussed how important it is to protect assets in the virtual world, and we've brought to light the reasons this protection is necessary. The bottom line is this—least privilege is absolutely necessary for all machines, whether they are physical or virtual.
Let's hear thoughts from our Insider Heroes:
Secure Sam:
Gone are the days where privileges don't need to be managed. We live in a world where data is being stolen on a daily basis, and it is every company's responsibility to secure their mission-critical information. The last thing I want is for a virtual environment within my organization to be out of control. A lot of things can make virtual assets difficult to manage—poorly delegated hypervisor rights, virtual sprawl as a result of bad planning, and too many contributors to guest and host systems. What it comes down to, for my company at least, is this—policies. Policies need to be enforced both for the hypervisor and at the hypervisor level. Just as with physical resources, the virtual ones need to be protected in the same ways. Least privilege is necessary for both security and compliance, and it is especially critical for those assets housed in the clouds and being accessed on virtual platforms.
Least Privilege Lucy:
Managing a network is a challenging task, and one that comes with both high risks and significant costs. This becomes especially true in a virtual environment, where privileges must be administered at the most granular level. In this chapter, we discussed RBAC, which sets permissions based on job descriptions within any given company. This concept is the basis of the least privilege model, and is paramount in the protection of any resources that may have been made virtual. Any desktops, applications, or servers that are taken to the cloud versus being physical still need the exact same protections regular machines do. The most important thing in my mind is to have a very distinct and clear separation of duties in virtual infrastructures. It's so important to have these types of policies in place—it makes security a reality in organizations.
Compliance Carl:
One of the most interesting statements I hear from various companies I audit is that “we went virtual, so we're compliant.” I think there are a lot of misconceptions around virtualization, but one that needs to be corrected immediately is that going virtual takes away non-compliance threats. It absolutely does not. If anything, it adds more risks. As was talked about earlier, granular management of privileged users is necessary for the security of physical machines, but it's even more important to the protection of virtualized resources. When you think about it, a virtual asset is susceptible to far more threats because so many administrative privileges are unmanaged in that environment. A least privilege solution is vital in this case—it provides granular identity management on both guest operating systems as well as to hypervisor hosts. Without confining users to only that data they need for job function, the virtualized environment will not be secure. Granular management of privileges is the only way for companies' sensitive information to remain compliant in the cloud.