Python has several libraries that are very useful for executing web application assessments, but there are limitations. Python is best used for small automation components of web applications that cannot be simulated manually through a transparent proxy, such as Burp. What this means is that specific work streams that you find in applications may be generated on the fly and cannot be replicated easily through a transparent proxy. This is especially true if there are timing concerns. So, if you need to interact with the backend server using multiple request and response mechanisms, then Python may fit the bill.
There are mainly five libraries that you are going to use while working with web applications. Historically, I have used the urllib2 library the most, and this is because of the great features and easy means to prototype code, but the library is old. You will find that it is missing some major capabilities and more advanced methods of interacting with new age web applications are considered broken, this is in comparison to newer libraries as described following. The httplib2 Python library provides robust capabilities when you are interacting with websites, but it is significantly more difficult to work with than urllib2, mechanize, request, and twill. That said, if you are dealing with tricky detection capabilities related to proxies, this may be your best option as the header data sent can be completely manipulated to perfectly simulate standard browser traffic. This should be fully tested in simulated environments before it is used against real applications. Often, the library provides erroneous responses simply because of the way the client requests were crafted.
If you come from the Perl world, you might instantly gravitate to mechanize as your go-to library, but it does not work well with dynamic websites and, in some situations, it cannot work with them at all. So what is today's answer? The request library. It is very clean and provides the necessary capabilities to quickly meet today's challenges of complex web engagements. To highlight the differences between the two and the prototype code, I have created application credential attack scripts using httplib2 and request. The aim of these scripts is to identify live credential sets and capture the relevant cookie. Once this is done, additional features can be added to either script. Additionally, these two scripts highlight the differences between the library sets.
The first example is the httplib2 version, as shown here:
The second is the request library version, which can be seen in the following screenshot:
Note
The request-based script can be found at https://raw.githubusercontent.com/funkandwagnalls/pythonpentest/master/request_brute.py, and the httplib2 script can be found at https://raw.githubusercontent.com/funkandwagnalls/pythonpentest/master/httplib2_brute.py.
As you can see, they are nearly identical in length, but the crafting of the statements in the request makes the simulation of web traffic simpler.
The benefit of using scripts like these or Burp would be to analyze parameters that could be manipulated, injected, and or brute-forced. Specifically, you are able to interact with code features that are not readily apparent through a web browser at a speed beyond human interaction. Examples of this include the building of exploitation lists for common SQLi or XSS attacks. Build lists of common SQLi attacks or XSS attacks. Then load them into the relevant parameters on the websites to identify the vulnerabilities. You will have to modify the aforementioned scripts to hit the target parameter, but this will significantly speed up the process of identifying potential vulnerabilities.
Note
Some of the best SQLi lists for common injection types for each database instance can be found at http://pentestmonkey.net/category/cheat-sheet/sql-injection. Equally good XSS lists are available at https://www.owasp.org/index.php/XSS_Filter_Evasion_Cheat_Sheet. Some of these details are also built into Burp Suite, as highlighted at https://support.portswigger.net/customer/portal/articles/1783128-Intruder_Common%20Uses.html.
Today, we have to contend with Web Application Firewalls (WAFs) and protection tools that can be bypassed, but you need to know how these protections are set up and what character encoding can bypass them. Remember if there are white or black lists they are keyed on specific character sets and/or encoding, which may block your exploitation attempts. By automating the testing, we can identify the items that key on captures that prevent the exploitation the web applications, and from that we can tailor our injections to bypass the protections put in place.
Tip
Character encoding for web application assessments is completely different from generating payloads. So, you should understand that these statements are not contradictory. The majority of WAFs do not smartly detect and decode data prior to comparing it with their white lists and/or black lists. So, you can bypass these protection mechanisms by changing the character format into something that an application can understand but the WAF cannot.
This is important for tools such as sqlmap, which is fantastic for verifying SQLi, but it should have its request tailored. It should be used only after you have confirmed that there is a plausible injection vulnerability. Then it should be used to build a proof of concept, extract data, or compromise systems. Loading up sqlmap to hit every parameter just to look for SQLi is a very time-consuming process. It can provide potential false positives and break systems.
Tip
Remember that if you do not customize your parameters and the request passed to sqlmap, it will likely turn non-blind injection attacks into blind injection attacks, which will significantly impact the time it takes to finish its task. The tool is probably the best in the market for what it does, but without a smart user, it will sometimes get lost.