For your project, you will be acquiring data from the EROS archives by using the USGS EarthExplorer site (downloads require a registered user account, which is free). You will use their map to locate scenes—specific geographic areas of satellite imagery that can be downloaded using the EarthExplorer Bulk Download Application (BDA). Your initial focus will be data from scoresbysund, the largest and longest fjord system in the world. Your first set of data will come from the LC82260102014232LGN00 scene ID, a capture that was taken in August 2014 as shown in the following screenshot:

Once you have marked the area in EarthExplorer, click the Data Sets view, expand Landsat Archive, and then select L8 OLI/TIRS. After clicking on Results, you will be presented with a list of scenes, each with a preview. You can click on the thumbnail image to see the preview, to check whether you have the right scene. Once you have located the scene, click on the tiny little package icon (it will be light brown/tan in color). If you haven't logged in, you will be prompted to. Add it to your cart, and then go to your cart to complete the free order.

Next, you will need to open the BDA and download your order from there (when the BDA opens, it will show the pending orders and present you with the option of downloading them). BDA will download a tarball (tar archive) to the given directory. From that directory, you can create a scene directory and unpack the files.

Before creating a Cloud workflow, we need to step through the process manually to identify all the steps and indicate those that may be automated. We will be using a data set from a specific point in time, but what we will define here should be usable by any Landsat 8 data, and some of it will be usable by the older satellite remote sensing data.

We will start by organizing the data. The BDA will save its downloads to a specific location (different for each operating system). Let's move all the data that you've taken with the EarthExplorer BDA to a location that we can easily reference in our IPython Notebook—/EROSData/L8_OLI_TIRS. Ensure that your scene data is in the LC82260102014232LGN00 directory.

Next, let's perform the necessary imports and the variable definitions, as follows:

In [1]: import matplotlib
        matplotlib.use('nbagg')
        %matplotlib inline
In [2]: from IPython.display import Image 
        import sys
        sys.path.append("../lib")
        import eros

With the last line of code, we bring in the custom code created for this chapter and task (based on the demonstration code by Milos Miljkovic, which he delivered as a part of a talk at the PyData 2014 conference). Here are the variables that we will be using:

In [3]: path = "/EROSData/L8_OLI_TIRS"
        scene_id = "LC82260102014232LGN00"

With this in place, we're ready to read some Landsat data and write them to the files in the following way:

In [4]: rgb_image = eros.extract_rgb(path, scene_id)

If you examine the source for the last two function calls, you will see that identifying the files that are associated with the Landsat band data, extracting them from the source data, and then creating a data structure to represent the red, blue, and green channels needed for digital color images. The Landsat 8 bands are as follows:

In your task, you will use bands 1 through 4 for water and RGB, 5 for vegetation, and 7 to pick out the rocks. Let's take a look at the true-color RGB image for the bands that we just extracted by using the following code:

In [5]: eros.show_image(
    rgb_image, "RGB image, data set " + scene_id,
    figsize=(20, 20))

The following image is the result of the preceding code:

The preceding image isn't very clear. The colors are all quite muted. We can gain some insight into this by looking at a histogram of the data files for each color channel by using the following code:

In [6]: eros.show_color_hist(
    rgb_image, xlim=[5000,20000], ylim=[0,10000],
    figsize=(20, 7))

The following histogram is the result of the preceding code:

As you can see, a major part of the color information is concentrated in a narrower band, while the other data is still included. Let's create a new image by using ranges based on a visual assessment of the preceding histogram. We'll limit the red channel to the range of 5900-11000, green to 6200-11000, and blue to 7600-11000:

In [7]: rgb_image_he = eros.update_image(
    rgb_image,
    (5900, 11000), (6200, 11000), (7600, 11000))
        eros.show_image(
    rgb_image_he, "RGB image, histogram equalized",
    figsize=(20, 20))

The following image is the result of the preceding code:

With the preceding changes, the colors really pop out of the satellite data. Next, you need to create your false-color image.

You will use the Landsat 8 band 1 (coastal aerosol) as blue, band 5 (NIR) as green, and band 7 (SWIR) as red to gain an insight into the presence of water on land, ice coverage, levels of healthy vegetation, and exposed rock or open ground. These will be used to generate a false color image. You will use the same method as with the previous image—generating a histogram, analyzing it for the best points of the color spectrum, and then displaying the image. This can be done with the help of the following code:

In [8]: swir2nirg_image = eros.extract_swir2nirg(path, scene_id)
In [9]: eros.show_color_hist(
    swir2nirg_image, xlim=[4000,30000], ylim=[0,10000],
    figsize=(20, 7))

The following histogram is the result of the preceding code:

Let's create the image for the histogram using the following code:

In[10]: swir2nirg_image_he = eros.update_image(
    swir2nirg_image,
    (5900, 15000), (6200, 15000), (7600, 15000))
        eros.show_image(swir2nirg_image_he, "",
    figsize=(20, 20))

The following is the resultant image:

On a 2009 iMac (Intel Core i7, 8 GB RAM), the processing of the Landsat 8 data and the generation of the associated images took about 7 minutes with the RAM usage peaking at around 9 GB. For multiple runs, the IPython kernel needs to be restarted just to free up the RAM quickly enough. It's quite clear that performing these tasks on a moderately equipped workstation would be a logistically and economically unfeasible proposition for thousands (or even hundreds) of scenes.

So, you will instead accomplish these tasks with the help of utility computing. The following are the necessary steps that are used to carry out the tasks:

These need to be migrated to the desired Cloud platform and augmented according to the needs of the associated tools. This brings us to the important question: Which technology should we use?

There is a dizzying array of choices when it comes to selecting a combination of a vendor, an operating system, vendor service options, a configuration management solution, and deployment options for Cloud environments. You can select one of the several OpenStack providers, such as Google Cloud Platform, Amazon AWS, Heroku, and Docker's dotCloud. Linux or BSD is probably the best choice for the host and guest OS, but even that leaves open many possibilities. Some vendors offer RESTful web services, SOAP, or dedicated client libraries that either wrap one of these or provide direct access.

In your case, you've done some testing on the speed needed to transfer considerably large files that are approximately 150 MB in size for each Landsat 8 band from a storage service to a running instance. Combining the speed requirements with usability, you found out that at the current time, Amazon's AWS came up as the winner in a close race against its contending Cloud service platforms. Since we will be using the recent versions of Python (3.4.2) and matplotlib (1.4.2) and we need a distribution that provides these pre-built, we have opted for Ubuntu 15.04. You will spin up the guest OS instances to run each image processing job, but now you need to decide how to configure these and determine the level of automation that is needed.