Chapter 1: Introduction to Metasploit and Supporting Tools

Before we take a deep dive into various aspects of the Metasploit Framework, let's first lay a solid foundation of some of the absolute basics. In this chapter, we'll conceptually understand what penetration testing is all about and where the Metasploit Framework fits in exactly. We'll also browse through some of the additional tools that enhance the Metasploit Framework's capabilities.

In this chapter, we will cover the following topics:

• The importance of penetration testing
• Understanding the difference between vulnerability assessments and penetration testing
• The need for a penetration testing framework
• Introduction to Metasploit
• Introduction to new features in Metasploit 5.0
• When to use Metasploit
• Making Metasploit effective and powerful using supplementary tools

Technical requirements

The following software is required:

• Kali Linux
• The Metasploit Framework
• Nessus
• NMAP
• w3af
• Armitage

The importance of penetration testing

For over a decade or so, the use of technology has been rising exponentially. Almost all businesses are partially or completely dependent on the use of technology. From Bitcoins to the cloud to the Internet of Things (IoT), new technologies are popping up each day. While these technologies completely change the way we do things, they also bring along threats with them. Attackers discover new and innovative ways to manipulate these technologies for fun and profit! This is a matter of concern for thousands of organizations and businesses around the world.

Organizations worldwide are deeply concerned about keeping their data safe. Protecting data is certainly important. However, testing whether adequate protection mechanisms have been put to work is also equally important. Protection mechanisms can fail, hence, testing them before someone exploits them for real is a challenging task. Having said this, vulnerability assessments and penetration testing have gained high importance and are now trivially included in all compliance programs. If the vulnerability assessment and penetration testing is done correctly, it significantly helps organizations gain confidence in the security controls that they have put in place and that they are functioning as expected!

We will now move on to understanding the difference between vulnerability assessments and penetration testing.

Understanding the difference between vulnerability assessments and penetration testing

Vulnerability assessments and penetration testing are two of the most common phrases that are often used interchangeably. However, it is important to understand the difference between the two. To understand the exact difference, let's consider a real-world scenario.

A thief intends to rob a house. To proceed with his robbery plan, he decides to recon his robbery target. He visits the house (that he intends to rob) casually and tries to gauge what security measures are in place. He notices that there is a window at the back of the house that is often open and so it's easy to break in. In our terms, the thief just performed a vulnerability assessment. Now, after a few days, the thief actually goes to the house again and enters through the back window that he had discovered earlier during his recon phase. In this case, the thief performed an actual penetration into his target house with the intent of robbery.

This is exactly what we can relate to in the case of computing systems and networks. You can first perform a vulnerability assessment of the target in order to assess the overall weaknesses in the system and then later perform a planned penetration test to practically check whether the target is vulnerable or not. Without performing a vulnerability assessment, it would be difficult to plan and execute the actual penetration.

While most vulnerability assessments are non-invasive by nature, the penetration test could cause damage to the target if not done in a controlled manner. Depending on the specific compliance needs, some organizations choose to perform only a vulnerability assessment, while others go ahead and perform a penetration test as well.

Now that we have understood the difference between vulnerability assessments and penetration testing, let's move on to understand the need for a penetration testing framework.

The need for a penetration testing framework

Penetration testing is not just about running a set of a few automated tools against your target. It's a complete process that involves multiple stages and each stage is equally important for the success of the project. Now, for performing all the tasks throughout every stage of penetration testing, we would need to use various tools and might need to perform some tasks manually. Then, at the end, we would need to combine the results from all the different tools together to produce a single meaningful report. This is certainly a daunting task. It would be really easy and timesaving if one single tool could help us perform all the required tasks for penetration testing. This exact need is satisfied by a framework such as Metasploit.

Now let's move on to learning more about the Metasploit Framework.

Introduction to Metasploit

The birth of Metasploit dates back to 16 years ago, when H. D. Moore, in 2003, wrote a portable network tool using Perl. By 2007, it was rewritten in Ruby. The Metasploit project received a major commercial boost when Rapid7 acquired the project in 2009. Metasploit is essentially a robust and versatile penetration testing framework. It can literally perform all the tasks that are involved in a penetration testing life cycle. With the use of Metasploit, you don't really need to reinvent the wheel! You just need to focus on the core objectives, the supporting actions will all be performed through various components and modules of the framework. Also, since it's a complete framework and not just an application, it can be customized and extended as per our requirements.

Metasploit is, no doubt, a very powerful tool for penetration testing. However, it's certainly not a magic wand that can help you hack into any given target system. It's important to understand the capabilities of Metasploit so that it can be leveraged optimally during penetration testing.

IMPORTANT NOTE:

Did you know? The Metasploit Framework has more than 3,000 different modules available for exploiting various applications, products, and platforms, and this number is growing on a regular basis.

While the initial Metasploit project was open source, after the acquisition by Rapid7, commercial-grade versions of Metasploit also came into existence. For the scope of this book, we'll be using the Metasploit Framework edition.

Introduction to new features in Metasploit 5.0

Ever since the Metasploit Framework was born 16 years ago, it has been through significant changes and improvements. In early 2019, Metasploit 5.0 was released, which is considered its first major release since 2011. While the Metasploit is commercially supported and developed by Rapid7, it also has rich community support, which enables its growth.

The latest Metasploit 5.0 version brings in a lot more features and improvements:

• Database and automation API's: The latest Metasploit 5.0 now allow users to run the database as a RESTful service. It also introduces the new JSON-RPC API, which would be of significant help to users who wish to integrate Metasploit with other tools. The API interface can be extremely handy in several automation and orchestration scenarios. It thus makes the framework even more agile and powerful.
• Evasion modules and libraries: In 2018, a new evasion module was introduced that allowed users to develop their own evasions. Metasploit 5.0 includes a special Windows evasion module that helps users create undetectable payloads and bypass security software. We'll learn more about using the new evasion module in Chapter 8, Anti-Virus Evasion and Anti-Forensics.
• Usability improvements and exploitation at scale: While the Metasploit Framework has evolved and matured over time, with the inclusion of the latest exploits, payloads, and so on, it is important to focus on the usability features as well. The ease of use significantly improves the user experience and convenience. Until the time that Metasploit 5.0 was released, all the exploit modules were permitted to execute against a single target host. There could be so many situations wherein it's absolutely required to execute the same exploit against multiple targets. This would then require writing a script. But now, the Metasploit 5.0 provides an out-of-the-box feature to execute an exploit against multiple targets at a time. We can specify the range of IP addresses against which we wish to launch the exploit. This feature can certainly boost the productivity and efficiency in assignments that have a large number of hosts to be tested. We'll be learning more about this feature in Chapter 3, Metasploit Components and Environment Configuration. The latest Metasploit 5.0 framework also has several improvements to the search feature. Searching for modules is now faster out of the box.

We'll now move on to learning when to use the Metasploit Framework in the penetration testing life cycle.

When to use Metasploit

There are literally tons of tools available for performing various tasks related to penetration testing. However, most of the tools serve only one unique purpose. Unlike these tools, Metasploit can perform multiple tasks throughout the penetration testing life cycle. Before we check the exact use of Metasploit in penetration testing, let's have a brief overview of the various phases of penetration testing.

The following diagram shows the typical phases of the penetration testing life cycle:

Figure 1.1 – Phases of the penetration testing life cycle

Now let's move on to understanding the phases in detail:

• Information gathering: Though the information gathering phase may look very trivial, it is one of the most important phases for the success of a penetration testing project. The more you know about your target, the higher the chances are that you will find the right vulnerabilities and exploits to work for you. Hence, it's worth investing substantial time and effort in gathering as much information as possible about the target under the scope.

Information gathering can be of two types, as follows:

Passive information gathering: Passive information gathering involves collecting information about the target through publicly available sources, such as social media and search engines. No direct contact with the target is made.

Active information gathering: Active information gathering involves the use of specialized tools, such as port scanners, to gain information about the target system. It involves making direct contact with the target system, hence there could be a possibility of the information gathering attempt being noticed by the firewall, Intrusion detection systems (IDS), or Intrusion prevention systems (IPS) in the target network.

• Enumeration: Through using active and/or passive information gathering techniques, you can get a preliminary overview of the target system/network. Moving on, enumeration allows us to know what the exact services running on the target system (including types and versions) are, and other information, such as users, shares, and DNS entries. Enumeration prepares a clearer blueprint of the target we are trying to penetrate.
• Gaining access: Based on the target blueprint that we obtained from the information gathering and enumeration phase, it's now time to exploit the vulnerabilities in the target system and gain access. Gaining access to this target system involves exploiting one or more of the vulnerabilities found during the earlier stages and possibly bypassing the security controls deployed in the target system (such as antivirus, firewall, IDS, and IPS).
• Privilege escalation: Quite often, exploiting a vulnerability on the target gives limited access to the system. However, we would want to gain complete root/administrator-level access into the target in order to gain the most out of our exercise. This can be achieved using various techniques to escalate the privileges of the existing user. Once successful, we can have full control over the system with the privileges and can possibly infiltrate deeper into the target.
• Maintaining access: So far, it has taken a lot of effort to gain root/administrator level access into our target system. Now, what if the administrator of the target system restarts the system? All of our hard work will have been in vain. To avoid this, we need to make a provision for persistent access into the target system so that any restarts of the target system won't affect our access.
• Covering tracks: While we have worked really hard to exploit vulnerabilities, escalate privileges, and make our access persistent, it's quite possible that our activities could have triggered an alarm on the security systems of the target system. The incident response team may already be in action, tracing all the evidence that may lead back to us. Based on the agreed penetration testing contract terms, we need to clear all the tools, exploits, and backdoors that we uploaded on the target during the compromise.

Interestingly enough, Metasploit helps us in all the penetration testing stages listed previously.

The following table lists various Metasploit components and modules that can be used across all stages of penetration testing:

Figure 1.2 – Metasploit components and modules

We'll gradually cover all the previous components and modules as we progress through the book. Now we move on to learn how we can make use of supplementary tools to make Metasploit even more effective.

Making Metasploit effective and powerful using supplementary tools

So far, we have seen that Metasploit is a really powerful framework for penetration testing. However, it can be made even more useful if integrated with some other tools. This section covers a few tools that complement Metasploit's capability to perform more precise penetration on the target system. We'll start with the Nessus tool.

Nessus

Nessus is a product from Tenable Network Security and is one of the most popular vulnerability assessment tools. It belongs to the vulnerability scanner category. It is quite easy to use, and it quickly identifies infrastructure-level vulnerabilities in the target system. Once Nessus tells us what vulnerabilities exist on the target system, we can then feed those vulnerabilities to Metasploit to see whether they can be exploited for real.

Its official website is https://www.tenable.com/.

The following screenshot shows the Nessus homepage:

Figure 1.3 – Nessus homepage

Next, we will be discussing different OS-based installation steps for Nessus.

Installation on Windows:

Please follow the following steps to install Nessus on Windows:

1. Navigate to the URL https://www.tenable.com/products/nessus/select-your-operating-system.
2. Under the Microsoft Windows category, select the appropriate version (32-bit/64-bit).
3. Download and install the msi file.
4. Open a browser and navigate to the URL https://localhost:8834/.
5. Set a new username and password to access the Nessus console.
6. For registration, click on the registering this scanner option.
7. Upon visiting http://www.tenable.com/products/nessus/nessus-plugins/obtain-an-activation-code, select Nessus Home and enter your details for registration.
8. Enter the registration code that you receive by email.

Installation on Linux (Debian-based)

Please follow the following steps to install Nessus on Linux:

1. Navigate to the URL https://www.tenable.com/products/nessus/select-your-operating-system.
2. Under the Linux category, Debian 6,7,8 / Kali Linux 1, select the appropriate version (32-bit/AMD64) and download the file.
3. Open a Terminal and browse to the folder where you downloaded the installer (.deb) file.
4. Type the following command:

   dpkg -i <name_of_installer>.deb.

5. Open a browser and navigate to the URL https://localhost:8834/.
6. Set a new username and password to access the Nessus console. For registration, click on the registering this scanner option.
7. Upon visiting http://www.tenable.com/products/nessus/nessus-plugins/obtain-an-activation-code, select Nessus Home and enter your details for registration.
8. Enter the registration code that you receive by email.

Now we move on to understanding the next tool: Network Mapper (NMAP).

NMAP

NMAP is a de-facto tool for network information gathering. It belongs to the information gathering and enumeration category. At a glance, it may appear to be quite a small and simple tool. However, it is so comprehensive that a complete book could be dedicated to how to tune and configure NMAP as per our requirements. NMAP can give us a quick overview of what ports are open and what services are running in our target network. This feed can be given to Metasploit for further action. While a detailed discussion of NMAP is out of the scope of this book, we'll certainly cover all the important aspects of NMAP in the later chapters.

Its official website is https://nmap.org/.

The following screenshot shows a sample NMAP scan:

Figure 1.4 – A sample NMAP scan using command-line interface

While the most common way of accessing NMAP is through the command line, NMAP also has a graphical interface known as Zenmap, which is a simplified interface on the NMAP engine, as follows:

Figure 1.5 – The Zenmap Graphical User Interface (GUI) for NMAP

Next, we will be discussing different OS-based installation steps for NMAP.

Installation on Windows

Please follow the following steps to install NMAP on Windows:

1. Navigate to the site https://nmap.org/download.html.
2. Under the Microsoft Windows binaries section, select the latest version of the .exe file.
3. Install the downloaded file along with WinPCAP (if not already installed).

   Important Note:

   WinPCAP is a program that is required in order to run tools such as NMAP, Nessus, and Wireshark. It contains a set of libraries that allow other applications to capture and transmit network packets.

Please follow the following steps to install NMAP on Linux.

Installation on Linux (Debian-based)

NMAP is, by default, installed on Kali Linux. However, if it is not installed, you can use the following command to install it:

root@kali:~#apt-get install nmap

Now we move on to understand the next tool: w3af

w3af

w3af is an open-source web application security scanning tool. It belongs to the web application security scanner category. It can quickly scan the target web application for common web application vulnerabilities, including the OWASP Top 10. w3af can also be effectively integrated with Metasploit to make it even more powerful.

Its official website is http://w3af.org/:

Figure 1.6 – The w3af console for scanning web application vulnerabilities

We will now discuss the various OS-based installation steps for w3af.

w3af is not available for the Windows platform.

Installation on Linux (Debian-based)

w3af is, by default, installed on Kali Linux. However, if it is not installed, you can use the following command to install it:

root@kali:~# apt-get install w3af

Now we move on to understanding the next tool: Armitage.

Armitage

Armitage is an exploit automation framework that uses Metasploit at the backend. It belongs to the exploit automation category. It offers an easy-to-use user interface for finding hosts in the network, scanning, enumeration, finding vulnerabilities, and exploiting them using Metasploit exploits and payloads. We'll look at an overview of Armitage in Chapter 9, Cyber Attack Management Using Armitage.

Its official website is http://www.fastandeasyhacking.com/index.html.

We can see the console for exploit automation in the following screenshot:

Figure 1.7 – Armitage console for exploit automation

The following are the various OS-based installation steps for Armitage:

• Installation on Windows: Armitage is not supported on Windows.
• Installation on Linux (Debian-based): Armitage is, by default, installed on Kali Linux. However, if it is not installed, you can use the following command to install it:

  root@kali:~# apt-get install armitage

PostgreSQL, Metasploit, and Java are required to set up and run Armitage. However, these are already installed on the Kali Linux system.

Summary

We started this chapter with understanding the relevance of penetration testing and then glanced at the practical difference between vulnerability assessment and penetration testing. We then tried to understand the exact need of a penetration testing framework and got introduced to the Metasploit Framework. We also covered the new features introduced as part of latest Metasploit 5.x Framework.

We also got an overview on when to use the Metasploit Framework in the penetration testing life cycle along with some other useful tools like Nessus, NMAP, and so on.

Now that we have got a high-level overview of what Metasploit is all about and the new features in the latest Metasploit 5.0 version, its applicability in penetration testing, and supporting tools, we'll browse through the installation and environment setup for Metasploit in the next chapter.

Exercise

You can try the following exercises:

• Visit Metasploit's official website and try to learn about the differences in various editions of Metasploit.
• Try to explore more on how Nessus and NMAP can help us during a penetration test.
• Install Nessus and w3af on your Kali Linux system.

Further reading

More information on the Metasploit Framework along with various versions can be found at https://metasploit.help.rapid7.com/docs.