Security is a fundamental part of the OpenStack architecture and needs to be maintained in order to protect the various security zones of the stack. OpenStack is a complex platform comprised of many different parts that are actively and continually being developed by multiple different parties. On the surface, this can seem fundamentally insecure, however, not only is OpenStack being developed by thousands of individuals, it is also being tested and scrutinized by thousands of users and developers. These users and developers create a useful feedback loop to other developers and testers. This provides almost constant vigilance against sloppy and insecure code. However, as in commercial software, security issues have still been discovered. This is why the OpenStack Foundation has created an OpenStack Security Team that publishes advisories about identified security issues, descriptions, and links to patches.
These patches are for vulnerabilities in the code for OpenStack. The current security advisory list is maintained at https://security.openstack.org/ossalist.html, where patches are also provided. These patches are mostly upgraded Python files that can simply replace the existing file on the OpenStack control or compute servers. An example of a patch that includes replacement files for a bug discovered in the neutron can be seen at https://review.openstack.org/#/c/299025/. The OpenStack Vulnerability Management Team is responsible for the bug and fix process for security and OpenStack at https://wiki.openstack.org/wiki/Vulnerability_Management.
A much larger part of the platform is the underlying operating system and/or tools. Since OpenStack can support many different hypervisors and run on multiple platforms, we will concentrate our discussion on Linux. The Linux distributions that are most commonly seen running OpenStack are represented in the following chart, taken at the most recent OpenStack Summit (https://www.openstack.org/assets/survey/April-2016-User-Survey-Report.pdf):
As the chart clearly shows, Ubuntu, CentOS, and Red Hat Enterprise Linux are the top three operating systems powering OpenStack today. Not surprisingly, two of these three distributions, Red Hat and Canonical (Ubuntu) are heavily involved with OpenStack development community and the third distribution is not a commercial release, rather it is the Community Development Platform for the Red Hat family of Linux distributions (https://wiki.centos.org/FAQ/General). Therefore, CentOS is included with these, developing OpenStack to run on Red Hat distributions. CentOS typically releases bug fixes for OS issues within 72 hours of Red Hat delivering a new package.
Using any distribution of Linux, it is vital to update any affected software in a timely manner to limit security risks. If the software is part of a package within Red Hat, the vulnerability should be patched as soon as it is released by Red Hat. Often, Red Hat includes patches with their security announcements and then releases it as a security erratum update. Again, it is vital to apply patches and updates as soon as they are released to help eliminate the risk of an attacker using the bug against your OpenStack infrastructure. This can be accomplished using the Yum Package Manager to download from only trusted sources. These trusted sources could be the Red Hat Network (RHN) or a local repository that is under your organization's control. All downloaded packages from outside your organization should be verified for authenticity using the GNU Privacy Guard or GNUPG, a free package used for ensuring the authenticity of package files. If the verification fails, either the package is corrupt or it has been compromised.
The Ubuntu security and patching ecosystem is very similar to Red Hat. Security updates are released by Canonical’s Ubuntu developers whenever they discover and patch vulnerabilities. Security notices can be delivered via e-mail or by subscribing via RSS. Ubuntu allows you to have security updates automatically installed - once configured you don't need to run security updates manually again. Ubuntu allows users to configure automatic security updates via unattended-upgrades using cron-apt. Similar behavior can be applied to Red Hat hosts using yum-cron with update_cmd = minimal-security-severity:Important.
Software repository management is vital in order to provide secure updates to an OpenStack infrastructure. Red Hat has developed a management application called Satellite and Ubuntu has Landscape. Both of these are systems management applications that allow administrators to deploy, patch, manage, and monitor their respective systems. These applications are an on-premise way to download and manage content from remote distribution portals. They reduce the amount of traffic over a corporate WAN during system updates, since all packages are only transferred to the local repository once. These applications also manage the update files locally to limit security risks of outside malicious content.
Satellite can even be configured as a single system or as a series of remote systems that act like proxies and get authorization and subscription information from a central server. These commercial applications also give users a single pane of glass for package management and allow profiles of updates to be created for separate classes of servers in the enterprise.
One alternative is to create and maintain a package repository manually using open source tools such as Puppet, Chef, and Ansible by creating scripts that do replication and validation. However, a more complete option is to use a community project such as Pulp (http://www.pulpproject.org/). Pulp is a free and open-source platform for managing software repositories. It is configurable to support RPM package types (rpm, srpm, errata, and so on), Puppet modules, Docker images, Atomic Trees, Python packages, and more. There is also a way to support Debian packages through a plugin. With Pulp you can do the following:
- Pull in content from distribution repositories to the Pulp server manually on either a one-time-only or recurring basis
- Upload your own content to the Pulp server (OpenStack security patches)
- Publish content as a web-based repository, a series of ISOs, or various other methods
Software patching and repository management get incrementally more difficult as the number of hosts to be patched increases. Starting an OpenStack deployment with a proper patching strategy and the tools to manage and enable patching is critical for scaling an OpenStack cloud. By applying and testing security patches to the OpenStack infrastructure, first in lower environments and then in production, organizations will ensure that both the development and production environments are secure and tested for operations.