As Scala can import and invoke Java classes as well as extend them, many of the Scala libraries available as part of the Scala ecosystem are only a thin layer on top of robust and mature Java libraries, to either provide additional features or simplify their usage by adding some syntactic sugar.
One such example is the Scala dispatch library (available at http://dispatch.databinder.net/Dispatch.html), a useful library to achieve HTTP interaction based on Apache's robust HttpClient. Let's run a little dispatch session in the REPL.
As dispatch is an external library; we first need to import it into our SBT project to be able to use it from the REPL console. Add the dispatch dependency to the build.sbt
file of the SampleProject
so that it looks like the following code snippet (make sure to have a blank line between statements in build.sbt
):
name := "SampleProject" … libraryDependencies += "net.databinder.dispatch" %% "dispatch-core" % "0.11.0"
Restart the REPL to make the libraries available, and import them into the session as follows:
scala> import dispatch._, Defaults._ import dispatch._ import Defaults._
Let's make a basic request to an online geolocation service, where the REST API is a simple GET
request to the freegeoip.net/{format}/{ip_or_hostname}
URL as follows:
scala> val request = url("http://freegeoip.net/xml/www.google.com") request: dispatch.Req = Req(<function1>)
Now, we will send the GET
request through HTTP and take the response as a string (wrapping XML as this is what we ask as response format from the service):
scala> val result = Http( request OK as.String) result: dispatch.Future[String] = scala.concurrent.impl.Promise$DefaultPromise@22aeb07c
Notice the result type of dispatch.Future[String]
returned by the interpreter. The previous versions of dispatch were synchronous (and still available under the library name, dispatch-classic
) but the latest versions such as the one we are using cope with modern development practices, namely asynchrony. We will study the asynchronous Scala code later in Chapter 8, Essential Properties of Modern Applications – Asynchrony and Concurrency, but similar to Java, Future
acts as a placeholder for a computation that does not block. This means that we can continue the flow of the program without waiting for the variable to be populated, which is convenient when invoking potentially long-running method calls (such as a REST service). Note, however, that here dispatch.Future
is a different implementation than java.util.concurrent.Future
, which is found in the standard Java library.
To read and display the result of our HTTP request, we can just type the following command lines:
scala> val resultAsString = result() resultAsString: String = "<?xml version="1.0" encoding="UTF-8"?> <Response> <Ip>74.125.225.114</Ip> <CountryCode>US</CountryCode> <CountryName>United States</CountryName> <RegionCode>CA</RegionCode> <RegionName>California</RegionName> <City>Mountain View</City> <ZipCode>94043</ZipCode> <Latitude>37.4192</Latitude> <Longitude>-122.0574</Longitude> <MetroCode>807</MetroCode> <AreaCode>650</AreaCode> </Response> "
Calling result()
here is the syntactic sugar for actually calling the result.apply()
method, a convenient way to make code look elegant in many situations.
Dispatch provides a lot of ways to handle both the request, such as adding headers and parameters, and the processing of the response such as handling the response as XML or JSON, splitting into two different handlers or dealing with streams. To exhibit these behaviors, we are going to call another online service as an example, the Groupon service. Groupon is a service that offers discount coupons when you buy a product or service such as holidays, beauty products, and so on in a variety of categories. The Groupon API can be queried to gather offerings within a geographic location determined by either city or coordinates (latitude and longitude).
To be able to experiment with the API, upon registration to the http://www.groupon.com/pages/api URL, you should obtain a unique client_id
key that authenticates you and that you have to pass along whenever you call the API. Let's illustrate this in the REPL:
scala> val grouponCitiesURL = url("http://api.groupon.com/v2/divisions.xml?client_id=<your own client_key>") grouponCitiesURL: dispatch.Req = Req(<function1>) scala> val citiesAsText = Http(grouponCitiesURL OK as.String) citiesAsText: dispatch.Future[String] = scala.concurrent.impl.Promise$DefaultPromise@4ad28057 scala> citiesAsText() res0: String = <response><divisions><division><id>abbotsford</id><name>Abbotsford</name><country>Canada</country><timezone>Pacific Time (US & Canada)</timezone>...
The REPL limits the amount of output for better readability. Instead of getting the response as a string, let's handle it as XML:
scala> val citiesAsXML = Http(grouponCitiesURL OK as.xml.Elem) citiesAsXML: dispatch.Future[scala.xml.Elem] = scala.concurrent.impl.Promise$DefaultPromise@27ac41a3 scala> citiesAsXML() res1: scala.xml.Elem = <response><divisions><division><id>abbotsford</id><name>Abbotsford</name><country>Canada</country><timezone>Pacific Time (US & Canada)</timezone>...
This time our result is more structured as it is represented as an XML tree. We can print it in a better format by applying a PrettyPrinter
object that will make the output fit within a width of 90 characters with an indentation of 2:
scala> def printer = new scala.xml.PrettyPrinter(90, 2) printer: scala.xml.PrettyPrinter scala> for (xml <- citiesAsXML) println(printer.format(xml)) scala> <response> <divisions> <division> <id>abbotsford</id> <name>Abbotsford</name> <country>Canada</country> <timezone>Pacific Time (US & Canada)</timezone> <timezoneOffsetInSeconds>-25200</timezoneOffsetInSeconds> <timezoneIdentifier>America/Los_Angeles</timezoneIdentifier> <lat>49.0568</lat> <lng>-122.285</lng> ... </division> <division> <id>abilene</id> <name>Abilene, TX</name> <country>USA</country> <timezone>Central Time (US & Canada)</timezone>...
Extracting partial information from our XML structure can be achieved by applying the map
transformations including XPath expressions. XPath expressions are useful to navigate through XML elements to retain only the relevant parts. We can progressively extract pieces of XML and return them as collections such as Lists
or Seqs
(sequences), as shown in the following code snippet:
scala> val cityDivisions = citiesAsXML() map ( city => city \ "division") cityDivisions: scala.collection.immutable.Seq[scala.xml.NodeSeq] = List(NodeSeq(<division><id>abbotsford</id><name>Abbotsford</name><country>Canada</country>... scala> val cityNames = cityDivisions map ( div => (div "name").text) cityNames: scala.collection.immutable.Seq[String] = List(AbbotsfordAbilene, TXAkron / CantonAlbany / Capital RegionAlbuquerqueAllentown...
Here, we got back a sequence of city names for which there are coupons available.
Instead of applying successive map
transformations to extract XML, in Scala, we can use a powerful construct that represents the silver bullet of iterations called for comprehension
or for expression
. Unlike the for
loops found in Java and used for iterating, for comprehension
returns a result. They are specified as follows:
for (sequence) yield expression
In the preceding code, sequence
can contain the following components:
element <- collection
As for Java loops, element
represents a local variable bound to the current element of the iteration whereas collection
represents the data to be iterated. Moreover, the first generator (there needs to be at least one) determines the type of the result. For example, if the input collection is a List
or a Vector
, the for comprehension
will yield a List
or a Vector
, respectively.
if expression
The preceding expression must evaluate to a Boolean value. Filters can be defined either on the same line as generators or separately.
variable = expression
They are intermediate values that can contribute to compute the result.
A for comprehension
construct is much easier to visualize with a few concrete examples:
scala> for { elem <- List(1,2,3,4,5) } yield "T" + elem res3: List[String] = List(T1, T2, T3, T4, T5)
We have transformed List[Int]
into List[String]
using only one generator. Using two generators is illustrated in the following code:
scala> for { word <- List("Hello","Scala") char <- word } yield char.isLower res4: List[Boolean] = List(false, true, true, true, true, false, true, true, true, true)
We can add a filter on any generator. For instance, if we want to retain only the uppercase characters of every word, we can write as follows:
scala> for { word <- List("Hello","Scala") char <- word if char.isUpper } yield char res5: List[Char] = List(H, S)
In the following example, we illustrate how to add a local variable definition:
scala> for { word <- List("Hello","Scala") char <- word lowerChar = char.toLower } yield lowerChar res6: List[Char] = List(h, e, l, l, o, s, c, a, l, a)
Going back to our HTTP Groupon service, we can now extract names of cities using for comprehension
as follows:
scala> def extractCityNames(xml: scala.xml.Elem) = for { elem <- xml \ "division" name <- elem "name" } yield name.text extractCityNames: (xml: scala.xml.Elem)scala.collection.immutable.Seq[String] scala> val cityNames = extractCityNames(citiesAsXML()) cityNames: scala.collection.immutable.Seq[String] = List(Abbotsford, Abilene, TX, Akron / Canton, Albany / Capital Region, Albuquerque, Allentown / Reading, Amarillo, Anchorage...
To be able to query the second part of the API to retrieve special discount deals for a specific area, we also need the latitude and longitude information from the queried cities. Let's do that by returning a tuple including three elements, the first one being the name, the second being the latitude, and the third being the longitude:
scala> def extractCityLocations(xml: scala.xml.Elem) = for { elem<- xml \ "division" name <- elem "name" latitude <- elem "lat" longitude <- elem "lng" } yield (name.text,latitude.text,longitude.text) extractCityLocations: (xml: scala.xml.Elem)scala.collection.immutable.Seq[(String, String, String)] scala> val cityLocations = extractCityLocations(citiesAsXML()) cityLocations: scala.collection.immutable.Seq[(String, String, String)] = List((Abbotsford,49.0568,-122.285), (Abilene, TX,32.4487,-99.7331), (Akron / Canton,41.0814,-81.519), (Albany / Capital Region,42.6526,-73.7562)...
Out of the list of returned cities, we might be interested in just one for now. Let's retrieve only the location for Honolulu using the following command:
scala> val (honolulu,lat,lng) = cityLocations find (_._1 == "Honolulu") getOrElse("Honolulu","21","-157") honolulu: String = Honolulu lat: String = 21.3069 lng: String = -157.858
The find
method in the preceding code takes a predicate as a parameter. As its return type is an Option
value, we can retrieve its content by invoking getOrElse
where we can write a default value in case the find
method does not return any match.
An alternative representation could be done using pattern matching, briefly described in Chapter 1, Programming Interactively within Your Project, as follows:
scala> val honolulu = cityLocations find { case( city, _, _ ) => city == "Honolulu" } honolulu: Option[(String, String, String)] = Some((Honolulu,21.3069,-157.858))
The regular syntax of pattern matching normally uses the match
keyword before all the case
alternatives, so here it is a simplified notation where the match
keyword is implicit. The underscore (_
) as well as the city
variable given in case
are wildcards in the pattern matching. We could have given these underscores variable names but it is not necessary as we are not using them in the predicate (that is, city == "Honolulu"
).
Let's now create a request to query for all the deals that match a particular geographic area:
scala> val dealsByGeoArea = url("http://api.groupon.com/v2/deals.xml?client_id=<your client_id>") dealsByGeoArea: dispatch.Req = Req(<function1>)
An alternative to handle data as tuples is to define case classes to encapsulate elements in a convenient and reusable way. We can, therefore, define a Deal
class and rewrite our previous for comprehension
statement returning the Deal
instances instead of tuples:
scala> case class Deal(title:String = "",dealUrl:String = "", tag:String = "") defined class Deal scala> def extractDeals(xml: scala.xml.Elem) = for { deal <- xml \ "deal" title = (deal \ "title").text dealUrl = (deal \ "dealUrl").text tag = (deal \ "tag" "name").text } yield Deal(title, dealUrl, tag) extractDeals: (xml: scala.xml.Elem)scala.collection.immutable.Seq[Deal]
As we did previously for retrieving cities, we can now retrieve deals via HTTP GET and parse XML this time for the particular city of Honolulu, knowing its latitude and longitude, as follows:
scala> val dealsInHonolulu = Http(dealsByGeoArea <<? Map("lat"->lat,"lng"->lng) OK as.xml.Elem) dealsInHonolulu: dispatch.Future[scala.xml.Elem] = scala.concurrent.impl.Promise$DefaultPromise@a1f0cb1
The <<?
operator means that we attach input parameters of a GET
method to the dealsByGeoArea
request. The Map
object contains the parameters. It is equivalent to the normal representation of HTTP GET where we put the input parameters as key/value pairs in the URL (that is, request_url?param1=value1;param2=value2
). This is in contrast with the <<
operator, which would have specified a POST
request. Creating a structured sequence of Deal
instances out of the raw XML produced by the dealsInHonolulu()
service call can be written as follows:
scala> val deals = extractDeals(dealsInHonolulu()) deals: scala.collection.immutable.Seq[Deal] = List(Deal(Laundry Folding StylesExam with Posture Analysis and One or Three Adjustments at Cassandra Peterson Chiropractic (Up to 85% Off)One initial consultation, one exam, one posture analysis, and one adjustmentOne initial consultation, one exam, one posture analysis, and three adjustments,http://www.groupon.com/deals/cassandra-peterson-chiropractic,Beauty & Spas), Deal(Laundry Folding Styles1.5-Hour Whale-Watching Sunset Tour for an Adult or Child from Island Water Sports Hawaii (50% Off) A 1.5-hour whale watching sunset tour for one childA 1.5-hour whale watching sunset tour for one adult,http://www.groupon.com/deals/island-water-sports-hawaii-18,Arts and EntertainmentOutdoor Pursuits), Deal(Dog or Horse?$25 for Take-Home Teeth-Whit...
Sorting the list of deals by their category is only a matter of applying a groupBy
method on the collection as follows:
scala> val sortedDeals = deals groupBy(_.tag) sortedDeals: scala.collection.immutable.Map[String,scala.collection.immutable.Seq[Deal]] = Map("" -> List(Deal(SkeleCopSix Bottles of 3 Wine Men 2009 Merlot with Shipping Included6 Bottles of Premium Red Wine,http://www.groupon.com/deals/gg-3-wine-men-2009-merlot-package,), Deal(Famous...
Notice how the groupBy
method is a very convenient way of applying the Map
part of a MapReduce job operating on a collection, in our case creating a Map
object where keys are the tags or categories of the Groupon deals and values are a list of the deals that belong to the specific category. A possible tiny Reduce
operation on the Map
object can, for example, consist of counting the number of deals for each category, using the mapValues
method that transforms the values of this (key,value) store:
scala> val nbOfDealsPerTag = sortedDeals mapValues(_.size) nbOfDealsPerTag: scala.collection.immutable.Map[String,Int] = Map("" -> 2, Arts and EntertainmentOutdoor Pursuits -> 1, Beauty & Spas ->3, Food & DrinkCandy Stores -> 1, ShoppingGifts & Giving -> 1, ShoppingFraming -> 1, EducationSpecialty Schools -> 1, Tickets -> 1, Services -> 1, TravelTravel AgenciesEurope, Asia, Africa, & Oceania -> 1)
The example we went through only explores the surface of what we can do with HTTP tools such as dispatch and much more is described in their documentation. The direct interaction with the REPL greatly enhances the learning curve of such APIs.
There are several excellent alternatives of lightweight frameworks for dealing with HTTP interaction, and in the case of dispatch, we have only looked at the client side of things. Lightweight REST APIs can, therefore, be constructed by frameworks such as Unfiltered, Finagle, Scalatra, or Spray to name a few. Spray is currently being architected again to become the HTTP layer of the Play framework (on top of Akka); technologies we are going to cover later on in this book.