Chapter 7

Networked Python

THE WORLD TODAY is more connected than it's ever been, and almost everything that you do on computers has some form of online component. The Raspberry Pi is no different. As long as you have a Model B, or a wireless USB dongle, getting your Pi connected to the Internet is trivial. There's the Midori browser you can use to surf the web, and mail clients are available. These are good for consuming content—getting information off the web and using services that other people have created. The power of the Raspberry Pi, however, lies in creating things. With a few lines of Python, you can grab information off the web or use your Raspberry Pi to serve up content and services to the world. Read on to find out how.

Understanding Hosts, Ports, and Sockets

To communicate with another computer, you need to know where to send data to. It might be that you're just sending information to another computer in the same room, or you might be sending it halfway round the world. Regardless, you need to specify an address. The standard way of locating computers is by Internet Protocol (IP) address. There are two types of IP address, version 4 and version 6. At the time of writing, version 4 (IPv4) is almost universally used, so you'll read only about it. IPv6 addresses work in the same basic way, so you shouldn't have any difficulty using these should they become mainstream any time soon.

Locating Computers with IP Addresses

IPv4 addresses contain four numbers separated by dots. To determine your Raspberry Pi's IP addresses (it has more than one), open a terminal (not a Python session, but an LXTerminal session). And run ifconfig. This should output something like the following:

eth0  Link encap:Ethernet HWaddr b8:27:eb:f3:d5:23
      inet addr:192.168.0.13 Bcast:192.168.0.255
           Mask:255.255.255.0
      UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
      RX packets:87523 errors:0 dropped:0 overruns:0 frame:0
      TX packets:59811 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000
      RX bytes:97997131 (93.4 MiB) TX bytes:12573160 (11.9 MiB)

lo    Link encap:Local Loopback
      inet addr:127.0.0.1 Mask:255.0.0.0
      UP LOOPBACK RUNNING MTU:16436 Metric:1
      RX packets:0 errors:0 dropped:0 overruns:0 frame:0
      TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

This shows that there are two network interfaces: eth0 and lo. eth0 is the wired network connection, and lo is the loopback connection that just loops back to the same machine. In the previous example, the Ethernet connection has an IP address of 192.168.0.13, while the loop back is 127.0.0.1 (this is always the same).

As well as by IP addresses, we can also locate computers by hostname (such as www.google.com, or localhost). These are a convenient shorthand for IP addresses. When you use one of these, you computer connects to a name server and asks what IP address that hostname corresponds to, then connects to the returned IP address. Localhost is a slightly unusual case as it always corresponds to 127.0.0.1 and is therefore the local machine.

If you think of the IP addresses (or hostnames) as similar to building addresses in the real world, then within each building, you still have to address it to the right person. In a computer network, this is done with ports. If you have a piece of software that serves information, it will listen on a particular port. Clients can then connect to a particular port, and get that information. Some ports are used for particular services. For example, web servers generally listen on port 80, whereas SSH connections go over port 22.

In Python, sockets are objects that connect to a particular port on a particular host, or listen for a connection coming into a particular port. Once they're connected, you can send and receive data through the socket. That sounds a bit complicated, but it's actually quite simple in practice. The easiest way to understand it is with an example.

Building a Chat Server

Unlike most of the programs in this book, this program has two parts that need to be run at the same time: a client and a server. The server just sits and waits for a connection to come in, while the client establishes a connection. Here's the code for the server:

import socket

comms_socket = socket.socket()
comms_socket.bind(('localhost', 50000))
comms_socket.listen(10)
connection, address = comms_socket.accept()

while True:
    print(connection.recv(4096).decode("UTF-8"))
    send_data = input("Reply: ")
    connection.send(bytes(send_data, "UTF-8"))

You can see that this gets most of its functionality from the socket module. With a server socket, you first need to bind it to a port on the host. (Any of the ports above 50,000 should be free for temporary use. Actually, you can use pretty much any one that's not currently in use, but it's best to avoid the ones below 100 unless you're sure they're not being used.) listen() then sets it to wait for a connection. When a connection comes in, it moves onto the following line:

connection, address = comms_socket.accept()

This sets up a new socket (stored in the variable connection), which is connected to the client. You can now send and receive data over this new connection using the send() and recv() methods. These take streams of bytes not strings, so you have to convert back and forward between the bytes and UTF-8 (universal character set Transformation Format; it's 8-bit) encoding that you use to display the information.

Here's the code for the client:

import socket

comms_socket = socket.socket()
comms_socket.connect(('localhost', 50000))

while True:
    send_data = input("message: ")
    comms_socket.send(bytes(send_data, "UTF-8"))
    print(comms_socket.recv(4096).decode("UTF-8"))

You can see that this time, instead of binding the socket to a host and port, the code connects to them. This time, the code doesn't create a new socket, but sends and receives on the original one.

Other than this, the two programs are very similar. To run a chat, you'll need two Python sessions, and the easiest way to open two is with two LXTerminal windows. In the first window, type python3 server.py (if you've called the server server.py), and in the second, type python3 client.py.

You'll be able to pass messages back and forth between the two programs. Of course, these aren't very networked. In fact, they're running on the same machine. There are places where it's useful to run networking between two programs on the same machine, but generally with networking, you want to send data between two computers.

This code has all the basics needed to communicate between machines, but it just needs a bit of a menu to help users connect to the place they want to go. It'd also be easier if there was a single program that could handle both the client and server sides. An improved version of the chat program with these properties is as follows (you can find it on the website as chapter7-chat.py):

import socket

def server():
    global port
    host = "localhost"

    comms_socket = socket.socket()
    comms_socket.bind((host, port))

    print("Waiting for a chat at ", host, " on port ", port)

    comms_socket.listen(10)
    send_data = ""

    while True:
        connection, address = comms_socket.accept()
        print("opening chat with ", address)
        while send_data != "EXIT":
            print(connection.recv(4096).decode("UTF-8"))
            send_data = input("Reply: ")
            connection.send(bytes(send_data, "UTF-8"))
        send_data = ""
        connection.close()

def client():
    global port
    host = input("Enter the host you want to communicate" +
                 " with(leave blank for localhost) ")
    if host == "":
        host = "localhost"

    comms_socket = socket.socket()

    print("Starting a chat with ", host, " on port ", port)
    comms_socket.connect((host, port))
    while True:
        send_data = input("message: ")
        comms_socket.send(bytes(send_data, "UTF-8"))
        print(comms_socket.recv(4096).decode("UTF-8"))

port = int(input("Enter the port you want to communicate on" +
                 " (0 for default)"))
if port == 0:
    port = 50000
while True:
    print("Your options are:")
    print("1 - wait for a chat")
    print("2 - initiate a chat")
    print("3 - exit")

    option = int(input("option :"))

    if option == 1:
        server()
    elif option == 2:
        client()
    elif option == 3:
        break
    else:
        print("I don't recognise that option")

You should recognise the networking code in this, and the rest you should be fairly familiar with. In order to communicate between two computers, you need to agree on a port, set one to listen for a connection, then connect from the other.

This approach does have a few problems. Firstly, the two chatters have to alternate messages, and secondly you can only communicate with people on your local network. Actually, this second problem depends on how your local network is set up. Remember that we said IP addresses were a little like building addresses? In many ways they are, and local area networks (LANs) are like towns. In the same way you can have two different buildings with the address “1 The High Street,” as long as they are in different towns, you can have two different computers with the IP address 192.168.1.2 as long as they are on different LANs. There are three IP address blocks reserved just for local networks:

• 10.0.0.0 – 10.255.255.255
• 172.16.0.0 – 172.31.255.255
• 192.168.0.0 – 192.168.255.255

Any IPv4 address that falls within these is a local-only address (this means you can only communicate with other computers on your local network), whereas ones that fall outside of it are public (which means any computer on the Internet can send data to them).

It is sometimes possible to connect to a local IP address from outside on the Internet. Whether this is possible depends on your Internet provider and router (look for the port-forwarding settings). However, there are too many different setups for us to be able to provide much guidance here.

Tweeting to the World

Instead of doing battle with your ISP and router, there are a number of other ways to get around these two problems. By far, the easiest is to use another service to handle the messages for you. Twitter is just such a service; it handles text-message passing between one computer and the world.

Each Twitter user can send tweets of up to 140 characters long. If any of these tweets mention another Twitter user (all usernames start with an @), then the tweet will show up in their connections page. If you don't already have a Twitter account, you'll need one for this section. Even if you do have one, it's probably a good idea to get a new one so that you can try things out without sending test messages to all your followers. It's free and quite simple to get an account. Just head to www.twitter.com and follow the instructions. You'll also get a tour of the site, so it should all make a bit more sense after setting up an account.

Twitter is normally used by people sending messages via the website, but that's not the only way of doing it. They also have an Application Programming Interface (API) that allows you to send and receive messages from programs rather than from the web interface.

There's a Twitter module that makes accessing the Twitter API simple. It's not included in Raspbian by default, so you'll need to download it from https://github.com/sixohsix/twitter/tree/master (use the Download Zip button on the bottom right). Once you have it, open an LXTerminal session, then unzip and install the module with:

unzip twitter-master.zip
cd twitter-master
python3 setup.py build
sudo python3 setup.py install

You're now almost ready to go, but as well as needing a Twitter username, you also need to register your application with Twitter to get the appropriate credentials for your application.

Once you've logged in to Twitter, go to http://dev.twitter.com, and select My applications from the user menu in the top-right corner. In the new screen, press Create An Application. On this form, you'll have to enter details of your program.

• The name has to be unique on Twitter, so test-app won't work. Be a little creative, or just mash a few keys to come up with something that hasn't been done before.
• The description can be anything as long as it's over 10 characters.
• The website doesn't have to be a website at all, it just has to look like one. We entered http://none.example.com.
• The callback URL can be left blank.

Other than that, you just have to agree to the Rules of the Road and enter the captcha.

The applications page has a series of tabs. You need to change one of the default settings, so go to the Settings tab, and switch Application Type from Read Only to Read and Write. This will allow you to post statuses as well as read information. Once this is done, press Update This Twitter Application's Settings.

Now you need to create an access token, so switch back to the Details tab, and click Create My Access Token. Once this is done, it'll take Twitter a few moments to update itself, and you may have to refresh the page for the section Your Access Token to appear. Once it refreshes, you're all set up and ready to go. You just need to get four pieces of information from the details page on this website: Access Token, Access Token Secret, Consumer Key, and Consumer Secret. These are all random strings that you'll need to copy and paste into the following example (see chapter7-twitter.py on the website):

import twitter

def print_tweets(tweets):
    for tweet in tweets:
        print('text: ', tweet['text'])
        print('from: ', tweet['user']['screen_name'])

twitter_user = twitter.Twitter(
    auth=twitter.OAuth("ACCESS-TOKEN","ACCESS-TOKEN-SECRET",
                       "CONSUMER-KEY","CONSUMER-SECRET"))

status = twitter_user.statuses

home = status.home_timeline()
print("home")
print_tweets(home)

mentions = status.mentions_timeline()
print("mentions")
print_tweets(mentions)

search_string = input("Enter text to search for, " +
                      "or press enter to skip: ")
if search_string != "":
    search = twitter_user.search.tweets(q=search_string)
    print("search")
    print_tweets(search['statuses'])

tweet = input("Enter tweet, or press enter to exit: ")

if tweet != "":
    twitter_user.statuses.update(status=tweet)

This little Twitter client isn't the most user-friendly one available. In fact, it's hard to imagine that anyone would use it over the website. However, it does have many of the different ways of interacting with Twitter that you may want to include in any applications you create. It should be fairly easy to adapt this code to your needs.

Weather Forecasts with JSON

In the previous example, the Twitter module provided all the basic functionality, but there won't always be modules you can use. Sometimes you'll have to write your own code to interact with web services. Fortunately there is a standard format for sending data back and forth than makes it easy to incorporate web services into your projects. JavaScript Object Notation, more commonly called JSON, is that standard. Originally, it was designed to work with JavaScript, which is a programming language mainly used on web pages, but it also works well with Python.

OpenWeatherMap.org is a website that provides free access to weather forecasts that you can include in your software. It also happens to use JSON. To get a feel for what a JSON document looks like, point your web browser to http://api.openweathermap.org/data/2.5/forecast/daily?cnt=7&units=meteric&mode=json&q=London. This is will return a seven-day forecast for London. It's not particularly easy to read, but you should notice that it looks like a Python dictionary that contains (amongst other things) a list of more dictionaries. Python can pull that information from the Internet using the urllib.request module in the following code:

import urllib.request

url = http://api.openweathermap.org/data/2.5/forecast/" +
      "daily?cnt=7&units=meteric&mode=json&q=London"
req = urllib.request.Request(url)
print(urllib.request.urlopen(req).read())

This will grab the information from OpenWeatherMap.org and print it on the screen. However, the data is in a string. You can't simply access various parts of it as though they are dictionaries and lists even though they look like them. You could build a function to read through the string and split it up, but fortunately you don't have to. The json module can load it and return a dictionary that contains the various parts of it. For example:

import urllib.request, json

url = http://api.openweathermap.org/data/2.5/forecast/
      "daily?cnt=7&units=meteric&mode=json&q=London"
req = urllib.request.Request(url)
forecast_string = urllib.request.urlopen(req).read()
forecast_dict = json.loads(forecast_string.decode("UTF-8"))

print(forecast_dict)

You can now get any information you want out of the forecast_dict data structure. In the following example, we've built a simple weather forecast program that prints out a seven-day forecast for a given city (see chapter7-weather.py on the website):

import urllib.request,json

city = input("Enter City: ")

def getForecast(city) :
    url = http://api.openweathermap.org/data/2.5/forecast/ +
          "daily?cnt=7&units=meteric&mode=json&q="
    url = url + city
    req = urllib.request.Request(url)
    response=urllib.request.urlopen(req)
    return json.loads(response.read().decode("UTF-8"))

forecast = getForecast(city)

print("Forecast for ", city, forecast['city']['country'])

day_num=1
for day in forecast['list']:
    print("Day : ", day_num)
    print(day['weather'][0]['description'])
    print("Cloud Cover : ", day['clouds'])
    print("Temp Min : ", round(day['temp']['min']-273.15, 1),
          "degrees C")
    print("Temp Max : ", round(day['temp']['max']-273.15, 1),
          "degrees C")
    print("Humidity : ", day['humidity'], "%")
    print("Wind Speed : ", day['speed'], "m/s")
    print()
    day_num = day_num+1

Note that the metric unit for temperature is Kelvin. To convert Kelvin to Celsius, simply subtract 273.15. This example uses only some of the data that the API returned. Take a look at the forecast data structure to see what else is in there that might be useful to print out.

Using this same basic method, you should be able to work with any web APIs that support JSON. There's a list of popular services at www.programmableweb.com/apis/directory/1?sort=mashups, where you should find a way to get almost any information your applications need off the web.

Testing Your Knowledge

So far, you've seen how to query an API to get information out of it. Now it's time to test whether you've fully understood what's been going on. Have a go at the exercise that follows, and then refer back to the previous examples for anything you're unsure of.

Exercise 1

You can get the current weather for a city (such as London) using the URL http://api.openweathermap.org/data/2.5/weather?q=London. Use this URL to create a program that tweets the current weather for a location. See the end of the chapter for an example solution to this exercise.

Getting On the Web

So far you've seen how to pass data back and forwards between two computers, and how to send and receive data to and from an online API. The obvious omission in all this is websites. These are, after all, the most popular way of viewing information online. In this section, you'll learn how to use your Raspberry Pi to host a web page.

There are two parts to the web: HTTP and HTML. The former is Hypertext Transfer Protocol (the method that web browsers and websites use to communicate), while the latter is Hypertext Markup Language (the language that web pages are written in). Hypertext is just a fancy name for any text with links embedded in it. There are modules that'll handle HTTP, but you will need to learn a little HTML for this to work.

Modern HTML is a complex language that can be used to create powerful applications with all sorts of animations and interactions. However, the basics of the language are quite simple. Every web page is a separate HTML file, and every HTML file has two parts, the head and the body. The head contains various pieces of information about the page, while the body contains what's displayed on the screen. HTML uses tags to describe the different parts of the document. Almost all tags come in pairs with an opening tag (such as <h1>, which denotes a main heading) and a closing tag (such as </h1>). Tags are always enclosed in triangular brackets, and closing brackets start with a forward slash. The following example uses most of the basic tags (see chapter7-htmleg1.html on the website):

<!DOCTYPE html>
<head>
An example HTML file</title>
</head>
<body>
<h1>An h1 heading</h1>
<h2>An h2 heading</h2>
<p>
A paragraph of text with a <a href=http://www.raspberrypi.org";link <to the Raspberry Pi websites</a>
</p>
<p>
Another paragraph
</p>
</body>

There is much more to HTML than this, but since this is a book about Python rather than HTML, we won't go into more detail. This is more or less what you need for this chapter. If you're interested in learning more, there are loads of great resources for taking things further. http://w3schools.org is a good place to start.

Once you've saved that file, you can just open it on your computer, and it should open in a browser. This is fine for checking your pages, but it's no good for sharing your creation with the world. To do that you need the aforementioned HTTP. In Python, it's really easy to whip up a quick HTTP server. For example:

import http.server, os

os.chdir("/home/pi")
httpd = http.server.HTTPServer(('127.0.0.1', 8000),
        http.server.SimpleHTTPRequestHandler)
httpd.serve_forever()

This will start a web server running on your Pi with the root in your home directory. It'll run on port 8000. If you were paying close attention earlier, you'll remember that 80 is the default port for web servers. We're using 8000 here just in case you're running something else with a web server, but you can change it to 80 if you prefer.

Once you have the code running, you can point your Pi's web browser to http://localhost:8000/. Localhost always points back to the machine it's running on (it's linked to the IP address 127.0.0.1), and : 8000 means use port 8000 rather than the default port 80. You should see a rather underwhelming list of files here.

HTTP works with directories and files in exactly the same way your computer does. It starts from a root directory. In the case of this example, that root directory is /home/pi. In your web browser, you can either specify a directory (which ends with a /) or a filename. If you enter a directory, the web server first checks to see if there is a file called index.html. If there is, it displays that instead of the directory contents. Change the name of your HTML file to index.html, then point your browser to http://localhost:8000 again. This time you should see the file displayed.

Making Your Website Dynamic

The http.server module is great for quickly whipping up a server to share information, but it's not very good at serving up information that changes. It does have some ways of adding a more interactive experience, but there's a better alternative. The tornado module is designed to serve up content that's created on the fly rather than content that's stored in files. This makes it far more suitable for projects that need this extra versatility.

Unlike the previous example, which worked with the operating system's file structure, Tornado creates its own virtual structure, and instead of HTML files, you create classes that output the appropriate HTML. First you need to install it by opening the LXTerminal and entering the following:

sudo apt-get install python3-tornado

Take a look at the following simple example. It just creates a website with "Hello World!" on it.

import tornado.ioloop
import tornado.web

class MainHandler(tornado.web.RequestHandler):
    def get(self):
        self.write("<!DOCTYPE html><head><title>" +
                   "Hello world</title></head>" +
                   "<body>Hello World</body>")

if _  _name_  _ == "_  _main_  _":
    application = tornado.web.Application([
        (r"/", MainHandler),
    ],)
    application.listen(8888)
    tornado.ioloop.IOLoop.instance().start()

There are two parts to this example. In the first part, we define the class that creates the web page, while the second part (which starts with the line if _  _name_  _ == "_  _main_  _":) defines and starts the web server. The main part of this is:

application = tornado.web.Application([
        (r"/", MainHandler),
    ],)

This creates a new Tornado web application and sets the options for it. The main options are a list of tuples that define which class handles which pages. In this example, there's only one page, the root, or / and it's handled by the class MainHandler. There are other options that can be added here, which will be used in future examples. The final two lines just tell it to listen on port 8888 (so you can run it at the same time as the http.server server on port 8000), and set it running. With this running, you can point your web browser to http://localhost:8888 to see it work.

All the handler classes have to extend tornado.web.RequestHandler, which provides all the basic functionality. All you need to add is a get method that calls self.write() (or as you'll see later, self.render()); the superclass handles everything else.

It might seem a little strange to define a class, but not create any objects from it. This is because Tornado uses the class and handles the object creation for you.

You can use Tornado like this to serve up static content, but it's not very good. For starters, all the HTML is inside the Python code, so it's a bit messy. The real advantage of Tornado is when you start creating dynamic pages that can change. The next example will generate a specific greeting for the users. You can use the same file as the previous example, but create a new class with the following code:

class HelloHandler(tornado.web.RequestHandler):
    def get(self, name):
        self.write("<!DOCTYPE html><head>
                   <title>Hello world</title></head>" +
                   "<body>Hello " + name + "</body>")

Then alter the bottom section of the code to bring it in (changes are shown in bold):

if _  _name_  _ == "_  _main_  _":
    application = tornado.web.Application([
        (r"/", MainHandler),
        (r"/hello/(.*)", HelloHandler),
    ],)
    application.listen(8888)
    tornado.ioloop.IOLoop.instance().start()

With these changes made and saved, run the code again. Point your browser to http://localhost:8888/hello/Ben or http://localhost:8888/hello/Alex, or use your own name.

Using Templates

Being able to modify the HTML code like this obviously lets you create far more powerful websites than you could before. However, this still means including the HTML in the Python, which isn't pleasant. The solution to this is to use templates. These are HTML documents with bits of Python in them that Tornado uses to build the final page. A template for the HelloHandler would be

<!DOCTYPE html>
<head>
<title>Hello</title>
</head>
<body>
Hello {{ name }}
</body>

As you can see, the Python variable to print goes inside double curly braces. Save this as hello-template.html in your home directory (/home/pi), and then update the HelloHandler class to be

class HelloHandler(tornado.web.RequestHandler):
    def get(self, name_in):
        self.name=name_in
        self.render("/home/ben/hello-template.html",
              name=self.name)

Then rerun the program. You should get the same response, but you should also be able to see that this code is far cleaner and easier to maintain.

Sending Data Back with Forms

You've probably noticed, though, that not many websites get you to enter information through web addresses. This method is often done for page IDs (as you saw with the weather API).

HTTP has a method for sending back to the server: POST requests. So far, Tornado has only been serving GET requests (hence the method name). POST allows you to send more complex information back using HTML forms. Create a new HTML file with the following code:

<!DOCTYPE html>
<head>
<title>User Information</title>
</head>
<body>
<h1>User Information</h1>
<form action="/hello/" method="post">
Enter your name: <input type="text" name="name">
<input type="submit" value="Sign in">'
</form>
</body>

Save it in your home directory as user-info.html. Now change the HelloHandler class to the following (changes are shown in bold):

class HelloHandler(tornado.web.RequestHandler):
     def get(self):
         self.render("/home/ben/user-info.html")

     def post(self):
         self.render("hello-template.html", name = self.get_
                     argument("name"))

The final code block also needs to be updated to reflect the new changes:

if _  _name_  _ == "_  _main_  _":
    application = tornado.web.Application([
        (r"/", MainHandler),
        (r"/hello/", HelloHandler),
    ],)
    application.listen(8888)
    tornado.ioloop.IOLoop.instance().start()

Now run this, and again point your browser to http://localhost:8888/hello/. Roughly the same thing happens as before, but now you enter your name in the textbox.

The reason this works is the first time you go to http://localhost:8888/hello/, your browser sends a GET request, so Tornado renders user-template.html. When you click Sign In, your browser sends a POST request because the form has the action /hello/ (if an address doesn't have a server name, the browser sends it to the same server) and the method POST. This time Tornado renders hello-template.html. self.get_argument()then grabs the data the user entered. name corresponds to the name you gave the text input in user-info.html.

As well as variables, you can also put certain pieces of Python code inside a template. This code is enclosed between {% and %}. The most common pieces are

{% set my_var = 0 %}
{% for x in y %} ... {% end %}

As you can see, this is a little different from regular Python. You need the word set before a variable assignment, and rather than using indenting (which doesn't work with HTML), code blocks are finished with {% end %}.

As an example, you could change hello-template.html to the following:

<!DOCTYPE html>
<head>
<title>Hello</title>
</head>
<body>
{% set char_number = 0 %}
{% for letter in name %}
{{ letter }} <br>
{% set char_number = char_number + 1 %}
{% end %}
{{ name }} has {{char_number}} characters
</body>

This not hugely useful code, but it does a bit more than the previous template. It displays each letter of the name on a different line (<br> is the HTML tag for a line break), and it also counts the number of letters.

Exercise 2

Create a new web app into which the users can enter a city, and then get a web page displaying a seven-day weather forecast (with data taken from openweathermap.org). An example solution is at the end of the chapter.

Keeping Things Secure

As you develop more and more powerful web applications with Tornado, sooner or later security becomes important. Even if you're only running it on a local network, you may have things that you don't want unauthorised people messing with. The final example in this chapter is a web app that provides some information about the Raspberry Pi.

This web app uses cookies for the login system. Cookies are bits of information that the web app stores in the browser. They can be used for a wide variety of reasons, but here we'll use them to track which sessions are logged in. If the users successfully log in, we set a secure cookie with their usernames. When they log out, we clear it. When they try to view the web app, the program checks if they have the cookie set, and if they don't, it redirects them to the login page. All the files for this are on the web page as chapter7-tornado.zip.

import tornado.ioloop
import tornado.web

users = {"ben": "mypassword"}
        
class SysStatusHandler(tornado.web.RequestHandler):
  def get(self):
    if not self.get_secure_cookie("user"):
      self.redirect("/login")
      return
    if self.get_argument("type") == "processes":
      com = [["pstree"],["top", "-bn1"]]
    elif self.get_argument("type") == "system":
      com = [["uname", "-a"],["uptime"]]
    elif self.get_argument("type") == "syslog":
      com = [["tail", "-n100", "/var/log/syslog"]]
    elif self.get_argument("type") == "storage":
      com = [["df", "-h"], ["free"]]
    elif self.get_argument("type") == "network":
      com = [["ifconfig"]]
    else:
      com = [["df", "-h"], ["free"], ["uname", "-a"], ["who"],
             ["uptime"], ["tail', "/var/log/syslog"],["pstree"],
             ["top", "-bn1"]]
    self.render("sysstatus-template.html", commands=com)
        
class LoginHandler(tornado.web.RequestHandler):
  def get(self):
    self.render("login-template.html")

  def post(self):
    print(self.get_argument("name"))
    print(self.get_argument("password"))
    if self.get_argument("name") in users.keys() and users[self.get_argument("name")] == self.get_argument("password"):
      self.set_secure_cookie("user", self.get_argument("name"))
      self.redirect("/sysstatus?type=system")
    else:
      self.render("login-fail.html")

class LogoutHandler(tornado.web.RequestHandler):
  def get(self):
    self.clear_cookie("user")
    self.render("logout-template.html")

if _  _name_  _ == "_  _main_  _":
  application = tornado.web.Application([
  (r"/login", LoginHandler),
  (r"/sysstatus", SysStatusHandler),
  (r"/logout", LogoutHandler),
  ], cookie_secret="put your own random text here')
  application.listen(8888)
  tornado.ioloop.IOLoop.instance().start()

As you can see, the handler that does most of the work (SysStatusHandler) takes a page argument called type, which can take the following values—processes, system, syslog, storage, network, or all. It then sends a list of lists that corresponds to various Linux system commands in the template.

In HTTP GET requests, arguments are sent after a question mark in the URL, so http://localhost:8888/sysstatus?type=network sends the argument type with the value network to the sysstatus page.

The template that does the hard work is sysstatus-template.html:

<!DOCTYPE html>
<head><title>Raspberry Pi System Status Checker</title></head>
<body><h1>Raspberry Pi System Status Checker</h1>
<a href="sysstatus?type=processes">Processes</a>
<a href="sysstatus?type=system">System</a>
<a href="sysstatus?type=syslog">System Log</a>
<a href="sysstatus?type=storage">Storage</a>
<a href="sysstatus?type=network">Network</a>
<a href="sysstatus?type=all">Display all</a>
<a href="logout">Logout</a>

{% import subprocess %}
{% for command in commands %}

<h2> Command: {{command[0]}} </h2>
<h2> Output: </h2>
<pre>
{% for line in subprocess.check_output(command).splitlines() %}
{{line}}
{% end %}
</pre>
{% end %}

</body>

subprocess is a Python module that lets you run commands on the underlying operating system. In the case of the Raspberry Pi, that's Linux. It takes a list as its input with the first item in the list being the command to run. Any subsequent items are the arguments passed to it. So, the list ["df", '-h"] runs the command df -h, which outputs the current disks on the system and information about where they're mounted and how much free space they have. You can try it out in LXTerminal. In fact, you can try out all of the commands used here in LXTerminal.

The <pre> tags are for pre-formatted text. That is, it makes HTML respect the tabs and newlines in the text rather than just garbling it all together.

The other templates are straightforward. Here's login-fail.html:

<!DOCTYPE html>
<html>
<head>
<title>Login Fail</title>
</head>
<body>
Your login has failed. Please click <a href="login">here</a>
to try again.
</body>

Here's login-template.html:

<!DOCTYPE html>
<head>
<title>Raspberry Pi System Status Login Form</title>
</head>
<body>
<p>
Please enter your login information</p>
<form action="/login" method="post">
Username: <input type="text" name="name">
<br>
Password: <input type="password" name="password">
<br>
<input type="submit" value="Sign in">
</form>
</body>

Finally, here's logout-template.html:

<!DOCTYPE html>
<html>
<head>
<title>Login Fail</title>
</head>
<body>
You have logged out. Please click <a href="login">here</a>
if you wish to log in again.
</body>

The login method we've created here does provide some security, but it is far from completely secure. If you create a web app that handles sensitive information, or allows the users some control over the Raspberry Pi, and it's on a network where you don't trust all the users, you need to properly investigate web security. It's far too big a subject for us to cover here, but a good place to start is the Open Web Application Security Project (see www.owasp.org).

Summary

After reading this chapter, you should understand the following a bit better:

• Data can be sent back and forwards between two computers using the host and port number.
• Hosts can be defined by either IP addresses or hostnames.
• To connect to another computer, you need to use sockets, which allow you to send data.
• APIs are interfaces programmers can use to access web servers and their data.
• The Twitter API has a module that lets you easily retrieve and manipulate information from twitter.com.
• APIs use a universal data format called JSON to make it easier to parse and share data between programs.
• Web pages are written in HTML and served using HTTP.
• http.server allows you to create a really simple web server.
• You can use Tornado to create more complex web applications.
• Tornado templates make it easy to render complex, dynamic information in HTML.
• The server can receive information from users using HTML forms and POST requests.
• Cookies allow you to store information, such as usernames, on the client's browser.

Solutions to Exercises

Exercise 1

Here is an example program that tweets the weather to a particular location.

import twitter, urllib.request, json
twitter_user = twitter.Twitter(
    auth=twitter.OAuth("1824182228-
  
lHOzvEpNA31LTiUCHkLSiqqW5Pbe7BvJKvKT2H6",

"AwYIcpfRFUt6F4hMoGfqMGINDiUrW49R1mjVqY6Bts",
                       "xStMTHb0HQZaEFLi2bsWA",

"YgFhiCpvlqLLe5Is5dRAZWNzlT84KnyZCMKfXIwN8"))

city = input("Which city? ")
url = "http://api.openweathermap.org/data/2.5/weather?q<="
url = url+city
req = urllib.request.Request(url)
forecast_string = urllib.request.urlopen(req).read()
forecast_dict = json.loads(forecast_string.decode("UTF-8"))

tweet_text = city + " weather is ' +
  
forecast_dict['weather'][0]['description']

twitter_user.statuses.update(status=tweet_text)

Exercise 2

There are many ways of doing this, but we used three files. Firstly, a template called city-info.html contains the following:

<!DOCTYPE html>
<head>
<title>Weather</title>
</head>
<body>
<h1>Which City?</h1>
<form action="/weather/" method="post">
Enter a city: <input type="text" name="city">
<input type="submit" value="Submit">
</form>
</body>

Another template called weather-template contains:

<!DOCTYPE html>
<head>
<title>Weather</title>
</head>
<body>
{% set day_num = 0 %}
{% for day in forecast %}
<h2>Day : {{str(day_num)}} </h2>
<h3>{{day['weather'][0]['description']}} </h3>
Cloud Cover : {{str(day['clouds'])}} <br>
Temp Min : {{str(round(day['temp']['min']-273.15, 1))}}
degrees C<br>
Temp Max : {{ str(round(day['temp']['max']-273.15, 1))}}
degrees C<br>
Humidity : {{str(day['humidity'])}} %<br>
Wind Speed : {{str(day['speed'])}}m/s<br>
{% set day_num = day_num + 1 %}
{% end %}
</body>

Finally, the Python file contains:

import tornado.ioloop
import tornado.web
import subprocess
import urllib.request,json
       
class WeatherHandler(tornado.web.RequestHandler):
    def get(self):
        self.render("/home/ben/city-info.html")

    def post(self):
        url = "http://api.openweathermap.org/data/2.5/forecast/"
               daily?cnt=7&units=meteric&mode=json&q="
               + self.get_argument("city")
        req = urllib.request.Request(url)
        response = urllib.request.urlopen(req)
        self.forecast = json.loads(response.read().decode("UTF-8"))
        self.render("weather-template.html",
                     forecast = self.forecast['list'])

if _  _name_  _ == "_  _main_  _":
    application = tornado.web.Application([
        (r"/weather/", WeatherHandler),
    ],)

    application.listen(8888)
    tornado.ioloop.IOLoop.instance().start()