Maps are a 2D representation of the relevant features of places. Which features are relevant will depend on the purpose of the map; a map for a zoo will show the entrances, thematic areas, gift store, and where each animal is. In this case, there is no need for the sizes and distances to be precise. In a geologic map, we will need accurate distances and representations of the rock units and geologic strata.
Positions on the surface of the earth are described by two coordinates, longitude and latitude. The longitude of a point is the angle between the point and the Greenwich meridian, and the latitude is the angle between the point and the equator. The latitude and longitude are the angles measured with respect to the equator and the Greenwich meridian, as shown in the following diagram:
The longitude can take values between -180 and 180, while the latitude can have values between 90 (the North Pole) and -90 (the South Pole).
Shapes on the earth can be described by listing the coordinates of the points in their boundary in order. We can, for instance, describe an island by listing the coordinates of points on the coastline separated by one kilometer in a clockwise order. This representation won't be perfectly accurate, because it will not represent the irregularities in the coastline that are smaller than one kilometer in size, but this will be useful if we only need to have an idea of the shape of the island.
To create a map of a feature, we need to translate the coordinates that describe the feature to points in a 2D surface. The functions that perform this translation are called projections. As projections intend to represent a 3D surface in a 2D medium, distortions will be introduced as follows:
The Mercator projection severely distorts the areas near the poles
Each projection has been created in order to minimize distortions of some kind. Some projections will represent relative directions accurately but not the area of certain regions; others will do the exact opposite. When using maps, it's important to know what kinds of distortions are acceptable in each case.
In this chapter, we will learn how to create map-based charts with D3. We will learn how to obtain, transform, and use geographic data in GeoJSON and TopoJSON formats to create SVG-based maps. We will create a map to visualize the distortions introduced by the Mercator projection, coloring each country by its area. Maps in which regions are colored by a characteristic of the regions (population, income, and so on) are called choropleths. In this section, we will implement a choropleth map using D3 and GeoJSON. We will also learn how to use TopoJSON to create maps with more compact geographic data files, and how to use TopoJSON to display topologic information, such as the connection between features and boundaries. Lastly, we will learn how to integrate D3 with Mapbox, an excellent map provider.
To create a map, we will need files that describe the coordinates of the features that we intend to include in our map. One of the most reliable sources for medium-scale geographic data is Natural Earth (http://www.naturalearthdata.com), a collaborative effort to curate and organize geographic datasets.
The geographic datasets available at Natural Earth are in the public domain and are available at 1:10,000, 1:50,000, and 1:110,0000 scales. There are vector and raster datasets, and the map files are classified in three categories:
- Cultural: This contains countries, administrative divisions, states and provinces, populated places, roads, urban areas, and parks
- Physical: This describes coastlines, land, islands, oceans, rivers, lakes, and glaciated areas among others
- Raster: This contains images, depicting the relief as shades and with colors based on climate
The files are in the ESRI shapefile format, the de facto standard for geographic data. ESRIshape files represent the geometry of features as sets of points, lines, and polygons. The files might also contain additional attributes about the features, such as the name of the place, population in the last census, or the average income of the population living in that area. A shapefile is a set of several files, which must include the following three files:
The shapefile set can also contain optional files, such as the .prj file, which contains information of the coordinate system and projection in which the vector data is stored.
Shapefiles are widely used in geographic information systems; however, the format is not suited for its use in web platforms. We will transform the shapefiles to JSON-based formats and use D3 to create maps from them.
The most widely used formats to create maps with D3 are the GeoJSON and TopoJSON formats. In this section, we will describe both the formats briefly and learn how to transform ESRIshapefiles to GeoJSON or TopoJSON.
The GeoJSON format encodes geometries, features, or collections of features using the JSON format. We will describe the central aspects of the GeoJSON format; the complete specification is available at http://geojson.org/.
A GeoJSON file always contains one top-level object. The GeoJSON object must have a type attribute; for geometries, the type can be Point, MultiPoint, LineString, MultiLineString, Polygon, or MultiPolygon. For a collection of geometries, features, and collections of features, the type will be GeometryCollection, Feature, or FeatureCollection, respectively. One GeoJSON file can contain a small island or a collection of countries nested in the top-level object.
The GeoJSON objects that represent geometries must have a coordinates attribute, whose contents will depend on the type of geometry. For a Point attribute, the coordinates attribute will be an array of two elements, representing a position:
{"type": "Point", "coordinates": [10.0, 10.0]}A LineString object represents a line, and its coordinates array contains a pair of locations (the longitude and latitude of a place):
{
"type": "LineString",
"coordinates": [[10.0, 10.0], [10.0, 0.0]]
}A Polygon object is more complex. It's intended to represent a polygon that can have holes on it. Polygons are represented by first describing the exterior boundary and then the boundaries of the holes on it. A Polygon object with one hole could be described with the following GeoJSON object:
{
"type": "Polygon",
"coordinates": [
[[0, 0], [0, 10], [10, 10], [0, 10], [10, 0]],
[[2, 2], [8, 2], [8, 8], [8, 2], [2, 2]]
]
}Here, the exterior ring describes a square of size 10, and the hole on it is a centered square of size 6. Also, there are geometries to describe the collections of the mentioned geometries; a MultiPoint object will contain an array of points, a MultiLineString object will contain an array with pairs of points, and a MultiPolygon object will contain an array of polygons.
A Feature object must have a geometry attribute that will contain a geometry object, usually Polygon or MultiPolygon. It may have a properties member that can be used to store nongeographic data about the features, such as the name of the place, population, or average income. For instance, the following feature object describes the country of Aruba; it contains about a dozen properties, and the polygon contains 26 points:
{
"type": "Feature",
"properties": {
...
"type": "Country",
"admin": "Aruba",
"adm0_a3": "ABW",
...
},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[-69.899121, 12.452001],
[-69.895703, 12.422998],
...
]
]
}
}A FeatureCollection object will contain a features array, which will contain feature objects.
As mentioned earlier, GeoJSON files describe features in terms of their geometry, and in some cases, they are highly redundant. If two features share a boundary, the common boundary coordinates will appear twice, once for each feature. Moreover, the description of the boundary might not match exactly, generating artificial gaps between the shapes. These shortcomings are addressed by the TopoJSON format created by Mike Bostock, the creator of D3.
TopoJSON objects encode topologies instead of geometries, that is, describing the relationship between points, lines, and shapes. In a TopoJSON object, shapes are described as sequences of arcs, which are essentially boundary segments. Each arc is defined once, but it can be referenced several times by the shapes that share that arc. This removes redundancy, making TopoJSON lighter than their GeoJSON counterparts. The coordinates in TopoJSON files are encoded in a more efficient way, making TopoJSON even more compact.
The following TopoJSON object represents the country of Aruba. The object contains an array of arcs, a transform attribute with information on how to decode the coordinates of the arcs to longitude and latitude, and the objects member, which contains the description of the features:
{
"type": "Topology",
"objects": {
"aruba": {
"type": "GeometryCollection",
"geometries": [
{
"type": "Polygon",
"arcs": [[0]]
}
]
}
},
"arcs": [
[
[9798, 1517],
[ 201, -1517],
[-2728, 812],
...
]
],
"transform": {
"scale": [
0.000017042719896989698,
0.00001911323944894461
],
"translate": [
-70.06611328125,
12.422998046874994
]
}
}The objects member can contain one or more objects describing geometry, but instead of listing the coordinates of the geometry object, the arcs array contains a list of references to the arcs defined at the top level of the TopoJSON object. If several features share a boundary, they will reference the same arc.
Besides the TopoJSON format, Mike Bostock created utilities to manipulate TopoJSON and GeoJSON files. The TopoJSON program has two components: the command-line program and the client-side library. The command-line program allows you to convert shapefiles, CSV or GeoJSON formats, to TopoJSON. It has options to simplify the features, combine several files in one output, and add or remove properties from the original features. The JavaScript library allows you to parse the TopoJSON files and construct Feature objects. The complete specification of the format and the programs is available at https://github.com/mbostock/topojson.
We might need to manipulate the geographic data files in several ways. We might want to reduce the level of detail of our features to have smaller files and simpler features, include additional metadata not present in the original files, or even filter some features.
To convert the files from one format to another, we will need to install the Geospatial Data Abstraction Library (GDAL). GDAL provides command-line tools to manipulate and convert geographic data between different formats, shapefiles and GeoJSON among them. There are binaries available for Windows, Mac, and Linux systems on their site, http://www.gdal.org/.
Depending on how we want our files, our workflow will include several steps that we might need to redo later. It's a good idea to automate this process in a way that allows us to understand why and how we transformed the files. One way to do this is to use the make program or a similar system. We will use make to download and transform the geographic datasets that we need for this chapter.
For the first chart, we will need to download the cultural vectors from Natural Earth. In a world map, we don't need the most detailed level; we will download medium-scale data. Then, we will uncompress the shapefiles and use the ogr2ogr program to transform the shapefiles to GeoJSON.
We will implement these transformations using make. The Makefile is in the chapter10/data directory of the code bundle. Each step can be done individually in a terminal provided that it is performed in the correct order. In a Makefile, we describe each step in the transformation process as a target, which can have zero or more dependencies. To generate the target file from the dependencies, we need to perform one or more commands. For instance, to generate the ne_50m_admin_0_countries.shp target file, we need to uncompress the ne_50m_admin_0_countries.zip file. This file is a dependency of the target file, because the target can't be generated if the ZIP file doesn't exist. The command to generate the shapefile is unzip ne_50m_admin_0_countries.zip. The following Makefile will generate a GeoJSON file that contains all the countries:
# Variables
ADMIN0_URL = http://.../ne_50m_admin_0_countries.zip
# Targets
# Download the Compressed Shapefiles
ne_50m_admin_0_countries.zip:
curl -LO $(ADMIN0_URL)
# Uncompress the Shapefiles
ne_50m_admin_0_countries.shp: ne_50m_admin_0_countries.zip
unzip ne_50m_admin_0_countries.zip
touch ne_50m_admin_0_countries.shp
# Convert the shapefiles to GeoJSON
countries.geojson: ne_50m_admin_0_countries.shp
ogr2ogr -f GeoJSONcountries.geojso nne_50m_admin_0_countries.shpThe Makefiles manage the dependencies between targets; building the countries.geojson file will check the dependency chain and run the target commands in the correct order. To generate the GeoJSON file in one step, run the following command:
$ make countries.geojson
It will download, uncompress, and transform the shapefiles to GeoJSON. It will only perform the commands to generate the files that are not present in the current directory; it won't download the ZIP file again if the file is already present.
We can use the GeoJSON file to create a TopoJSON version of the same file. By default, topojson will strip all the properties of the original file. We can preserve the properties by using the -p name option:
$ topojson -o countries.topojson -p admin –p continent countries.geojson
Note that the file size of the GeoJSON file is about 4.4 M; the TopoJSON file is only 580 K. We will begin by using the countries.geojson file to create our first maps, and then learn how to use the TopoJSON files to create maps. We will also include this command in the Makefile to be able to replicate this conversion easily.