The multiprocessing module is an alternative to using the threading module. It is a module that's similar to the threading module, which offers a very similar interface, but at a low level. It works with processes instead of threads. In this case, we will take a similar approach to concurrent.futures. We are establishing a multiprocessing pool and presentation of tasks by assigning a function to the address list.

You can find the following code in the demo_multiprocessing.py file:

#!/usr/bin/python3

import time
import multiprocessing
import logging
import requests

from utils import check_website
from utils import WEBSITE_LIST
NUM_WORKERS = 3

if __name__ == '__main__':
    start_time = time.time()
    with multiprocessing.Pool(processes=NUM_WORKERS) as pool:
        results = pool.map_async(check_website, WEBSITE_LIST)
        results.wait()
        print(results)
    end_time = time.time()
    print("Time for multiprocessing: %s secs" % (end_time - start_time))

This script uses check_website(), which is available in the utils.py file of the same directory:

#!/usr/bin/python3

import requests

WEBSITE_LIST = ['http://www.foxnews.com/',
 'http://www.cnn.com/',
 'http://www.bbc.co.uk/',
 'http://some-other-domain.com/']

class WebsiteException(Exception):
 pass

def ping_website(address, timeout=6000):
    try:
        response = requests.get(address)
        print("Website %s returned status_code=%s" % (address, response.status_code))
        if response.status_code >= 400:
            print("Website %s returned status_code=%s" % (address, response.status_code))
            raise WebsiteException()
    except requests.exceptions.RequestException:
        print("Timeout expired for website %s" % address)
        raise WebsiteException()

def check_website(address):
    try:
        ping_website(address)
    except WebsiteException:
        pass

The following is the output of the execution of demo_multiprocessing.py. For each URL defined in WEBSITE_LIST, check the status code of the domain and show information about it:

Website http://www.bbc.co.uk/ returned status_code=200
 Website http://www.foxnews.com/ returned status_code=200
 Timeout expired for website http://some-other-domain.com/
 Website http://www.cnn.com/ returned status_code=200
 <multiprocessing.pool.MapResult object at 0x00000204C59A8B70>
 Time for multiprocessing: 2.0654103755950928 sec

In this section, you have learned about the concurrent.futures package for processing tasks in an asynchronous  way with the ThreadPoolExecutor class. We also reviewed the multiprocessing package as an alternative to the threading module for creating a pool of processes for assigning tasks.