What’s In This Chapter
The amount of data available for applications is incredible—and unfortunately, it’s locked behind corporate walls. If you can navigate the multitude of access methods to get to the data, that data is invariably presented in various formats. Accessing different data sources from different vendors—or even the same vendor—and then corralling and controlling that data can be a nightmare. But you can simplify data publishing and even consumption using a cloud-based solution that provides access to disparate data from multiple sources via a common format and access. Enter Azure Datamarket.
Windows Azure Datamarket is a subset of the Windows Azure Marketplace wherein a single Marketplace account can purchase applications and data. Datamarket was the first area in the Azure Marketplace. Microsoft has since introduced Applications published to Datamarket, which is not limited to the .NET and Windows world. Currently located at https://datamarket.azure.com/, the Marketplace gives users access to a store of applications and datasets for purchase. Figure 12-1 shows the Marketplace homepage.
Introduced as code-name Dallas, and based on common nonplatform-specific web technologies such as HTTP and AtomPub, Datamarket provides a single location where content owners can publish and sell content to the masses using a standardized publish/subscribe model, and which users can access from a wide variety of platforms and applications.
The benefits of Marketplace are as follows:
In summary, Marketplace provides not only data but also the common account, subscription, and billing management for applications. Datamarket provides the foundation for publishing and consuming datasets via OData, basic authentication, and OAuth. This is data in the cloud, which means it’s easy to publish, subscribe to, and work with premium content. This chapter focuses on how to locate, subscribe to, and consume data from Azure Datamarket.
As mentioned before, you can consume data from Datamarket in many different ways—from the Service Explorer, which you learn about later in this section, to Microsoft Excel to custom applications. Datamarket allows anyone to browse a list of published datasets. Before you can actually subscribe to and use a dataset , you must register with Marketplace and create a Marketplace account. The following sections walk you through how to do this.
To start using Azure Data Market you must log in using a Windows Live account. If you don’t have a Windows Live account, Marketplace has an option to create a Windows Live account during the sign-in process. If you don’t have a Marketplace account associated with your Windows Live account, you are prompted for registration information. To sign into Marketplace, follow these steps:
After you complete the registration process and have a Marketplace account, you can sign in and out as needed. You can use the My Account pages discussed in the Managing Your Account section to manage your account.
Marketplace enables you browse a listing of available published datasets by clicking the Data tab located at the top of the page. The Datamarket homepage enables you to browse, filter, or search for datasets that you may be interested in subscribing to.
Datamarket categorizes datasets by cost, category, and publisher:
Each published dataset provides an information page that includes details. Clicking on the title of the published content or the associated icon before the title in the search will refer you to the information page.
Figure 12-6 shows an example of an information page and shows information for the Stock Sonar-Sentiment Service of US Stocks dataset.
The information page includes the following:
A subscription to the dataset is required before you can access or view the dataset. This includes both paid and free datasets. To subscribe to the dataset, follow these steps:
Datamarket provides the Service Explorer interface to access subscribed data without creating a custom application. End users can use the Service Explorer tool to query and view data in many different formats including tabular, chart, or raw XML. This interface also enables export from Datamarket to various formats and import into existing applications.
You can access the Service Explorer by clicking the Explore This Dataset link (see Figure 12-10) located on the dataset’s information page. This link is only available when you are signed into Marketplace and viewing the information page of a dataset you have subscribed to. Clicking the Explore This Dataset link opens the Service Explorer page.
The Service Explorer provides a quick view into the data. As an example, Figure 12-11 shows the Service Explorer querying the MortgageInformationAPIs for the average mortgage rate for Illinois. This dataset queries and finds average mortgage rates and monthly mortgage payments. Users can select the query and any query parameters, and view the results in either tabular, chart, or XML format. Users can export results to file as XML or CSV, or format and open the results in Excel PowerPivot or Tableau. The Service Explorer is a useful tool that enables end users to quickly search and export data from subscribed datasets.
End users are not the only audience for Service Explorer. Developers commonly use Service Explorer as a quick way to explore and understand the available queries and data schema while creating an application or creating, or modifying a query.
The process behind Service Explorer involves the following steps:
The query displays above the results windows Applications that have a specific fixed query requirement; the Service Explorer can be used to create the URL query which can then be copied into the application code. You can use simple string manipulation such as String.Format to modify the URL parameters. The query created to display the average mortgage rate for Illinois is:
The Service Explorer can also display the result set as XML. This is particularly useful for non-.NET developers who may not have a proxy to simplify development. Viewing the results of the query as XML allows the developer to see the schema of the results and aids in parsing the XML. Listing 12-1 displays the result for the preceding URL query in XML. This particular query is formatted as AtomPub.
Listing 12-1: Results of a Query Viewed as XML
<feed xmlns:base="https://api.datamarket.azure.com/Data.ashx/Zillow/
MortgageInformationAPIs/GetRateSummary"
xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices"
xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"
xmlns="http://www.w3.org/2005/Atom">
<title type="text" />
<subtitle type="text">
The GetRateSummary API returns the current rates per loan type from Zillow
Mortgage Marketplace. Current supported loan types are 30-year fixed, 15-year
fixed, and 5/1 ARM. Rates are computed from real quotes borrowers receive from
lenders just seconds before the rate data is returned. The GetRateSummary API
returns rates for a specific state if the optional state parameter is used
</subtitle>
<id>https://api.datamarket.azure.com/Data.ashx/Zillow/MortgageInformationAPIs/
GetRateSummary?State='IL'&$top=100</id>
<rights type="text"> Zillow, Inc., 2008</rights>
<updated>2012-03-01T12:53:17Z</updated>
<link rel="self" href="https://api.datamarket.azure.com/Data.ashx/
Zillow/MortgageInformationAPIs/
GetRateSummary?State='IL'&$top=100" />
<entry>
<id>https://api.datamarket.azure.com/Data.ashx/Zillow/MortgageInformationAPIs/
GetRateSummary?State='IL'&$skip=0&$top=1</id>
<title type="text">RateSummaryEntity</title>
<updated>2012-03-01T12:53:17Z</updated>
<link rel="self" href="https://api.datamarket.azure.com/Data.ashx/Zillow/Mortgage
InformationAPIs/
GetRateSummary?State='IL'&$skip=0&$top=1" />
<content type="application/xml">
<m:properties>
<d:ThirtyYearFixedToday m:type="Edm.Double">3.8</d:ThirtyYearFixedToday>
<d:ThirtyYearFixedCountToday m:type="Edm.Double">
94
</d:ThirtyYearFixedCountToday>
<d:FifteenYearFixedToday m:type="Edm.Double">
3.04</d:FifteenYearFixedToday>
<d:FifteenYearFixedCountToday m:type="Edm.Double">
87
</d:FifteenYearFixedCountToday>
<d:FiveOneARMToday m:type="Edm.Double">2.84</d:FiveOneARMToday>
<d:FiveOneARMCountToday m:type="Edm.Double">28</d:FiveOneARMCountToday>
<d:ThirtyYearFixedLastWeek m:type="Edm.Double">
3.75
</d:ThirtyYearFixedLastWeek>
<d:ThirtyYearFixedCountLastWeek m:type="Edm.Double">
5695
</d:ThirtyYearFixedCountLastWeek>
<d:FifteenYearFixedLastWeek m:type="Edm.Double">
3.0
</d:FifteenYearFixedLastWeek>
<d:FifteenYearFixedCountLastWeek m:type="Edm.Double">
4950
</d:FifteenYearFixedCountLastWeek>
<d:FiveOneARMLastWeek m:type="Edm.Double">2.59</d:FiveOneARMLastWeek>
<d:FiveOneARMCountLastWeek m:type="Edm.Double">
2928
</d:FiveOneARMCountLastWeek>
</m:properties>
</content>
</entry>
</feed>
Now that you have an account and have subscribed to one or more datasets, you’ll need to know how to manage your account settings, which includes managing your account keys and datasets. You can manage your account from the My Account pages, which you access from the My Account link located in the top menu bar on the Windows Azure Marketplace interface. This page has links on the left side:
Datamarket data can be exported and used by many existing applications including Excel, PowerPivot, and Tablau (an end user business intelligence tool). Exporting data as XML or as a comma-separated value formatted file (.csv) opens up the number of applications that can use the data. All these tools work well to browse, filter, and explore the data to find some trend or data point.
What about consuming Datamarket data in a mashup or custom application? Datamarket is easily consumed by applications of all types, including .NET or other platforms. Datamarket data can also be consumed by forms-based, web-based, console-based, and mobile applications to meet the requirements of your business.
Datamarket helps to standardize how you can access various data potentially residing in different data stores in the cloud. Datamarket achieves this by using common well-known schemas and protocols such as OAuth, OData, AtomPub, JSON, and HTTPS. Using these common schemas and protocols allows standardized access to data by .NET applications and non-.NET applications alike.
This section covers core concepts that you will need to know to programmatically work with Datamarket datasets.
OData stands for Open Data Protocol, which defines a common access method to data using standard web technologies. This access method adds, edits, deletes, and queries data abstracted from the storage mechanism. Building on common web-based technologies such as AtomPub, JSON, HTTP, and XML, data stores supporting OData can be accessed from any application that can create and process a simple web request. For more information on OData, visit www.odata.org/.
Datamarket data is served up using WCF Data Services, which is a .NET framework supporting the creation and consumption of OData. For the .NET developer, this means that you have the main plumbing required to generate a query to Datamarket and make the results easily available as class entities. There is no need to hand-code web requests, create OData queries, or parse XML results into entity objects. WCF Data Services provides this functionality. For more information on WCF Data Services, go to http://msdn.microsoft.com/en-us/data/bb931106.
Datamarket supports two authentication scenarios at this time: Basic Authentication and OAuth. This chapter focuses only on Basic Authentication but understanding the difference between the two scenarios is important for a developer consuming Datamarket datasets.
Basic Authentication is simply passing a user’s credentials along with the web request. As the name implies, it is one the most basic methods to authenticate a user. In the .NET world it is simple to create a NetworkCredential and attach it to the request. Although simplicity is the key with Basic Authentication, it does have its detractors. Basic Authentication uses a single account to control access. There is no management of users; it is all or nothing because everyone uses the single key for access. All transactions incurred with Basic Authentication are assigned to the owner of the key.
OAuth is a token-based authentication system that is increasingly in use because of the network of available social applications. Datamarket can use OAuth to authenticate users, but compared to Basic Authentication, it’s a bit more challenging to implement. Using OAuth in your application to authenticate to Datamarket requires you to register the application with Marketplace. For more information on OAuth and Datamarket, go to http://msdn.microsoft.com/en-us/library/windowsazure/gg193416.aspx.
Accessing data involves creating and submitting queries to Datamarket. Content providers determine the queries that are allowed for the dataset. The Details tab located on the dataset’s information page (refer to Figure 12-7) includes the supported types of queries. Figure 12-15 shows the supported queries for the Stock Sonar - Sentiment Service of US Stocks dataset.
Datasets in Datamarket supports two types of queries: Fixed and Flexible. Each dataset may support both types of queries:
This section covers how simple it is to create an application that uses a Fixed query to retrieve data from a Datamarket dataset. This application queries and retrieves data from the Zillow MortgageInformationAPIs available from Datamarket. The MortgageInformationAPIs dataset provides two fixed queries to retrieve data from the dataset. This application shown in Figure 12-16 uses the GetRateSummary query to retrieve average mortgage rates. To create the application, follow these steps:
Using Zillow;
Listing 12-2: The Completed MainWindow Constructor
private ObservableCollection<RateSummaryEntity> _rates; public ObservableCollection<RateSummaryEntity> Rates { get { return _rates; } } public MainWindow() { InitializeComponent(); //TODO: Initial _rates collection _rates = new ObservableCollection<RateSummaryEntity>(); this.DataContext = this; LoadStates(); cboState.SelectedIndex = 0; }
Listing 12-3: The GetStateMortgageAvg Method
private void GetStateMortgageAvg(string stateAbbreviation) { Rates.Clear(); MortgageInformationApiContainer mortgageClient = new MortgageInformationApiContainer( new Uri( "https://api.datamarket.azure.com/Zillow/MortgageInformationAPIs")); NetworkCredential credentials = new NetworkCredential(); credentials.UserName = "[Your Email Account]"; credentials.Password = "[Your Marketplace Key]"; mortgageClient.Credentials = credentials; //Catch Average selection string stateAbr = stateAbbreviation == "AV" ? "" : stateAbbreviation; var results = mortgageClient.GetRateSummary(stateAbr); foreach (RateSummaryEntity rates in results) { Rates.Add(rates); } }
Listing 12-4: The Completed cboState_SelectionChanged Event Handler
private void cboState_SelectionChanged(object sender, SelectionChangedEventArgs e) { ComboBox s = sender as ComboBox; State selectedState = s.SelectedItem as State; //TODO: Call GetStateMortgageAvg Function GetStateMortgageAvg(selectedState.Abbreviation); }
The Fixed Query application is now complete. You can build and run the application by pressing F5 in Visual Studio. The application initially calls into GetStateMortgageAvg with the AV abbreviation. This queries the dataset for mortgage rates based on an average of all states. It is in this method that the proxy is used to connect to Datamarket and query the dataset. Notice that a network credential is associated with the request in the GetStateMortgageAgg method. See Figure 12-20.
When you select a state from the drop-down control, the application calls GetStateMortgageAvg passing in the associated state abbreviation. Figure 12-21 displays the current averages for the State of Illinois.
This section shows how simple it is to create an application that uses a Flexible query to retrieve data from a Datamarket dataset. This application queries and retrieves data from the WeatherBug Historical Observations available from Datamarket. This dataset provides a limited number of monthly transactions for free and supports Flexible queries. The Historical Observations supports Flexible queries to query WeatherBug stations and historical weather observations. The Flexible query application, as shown in Figure 12-22, uses a Flexible query to return a list of weather stations located within a specific ZIP code. Selecting a specific weather station generates another query to retrieve a set of historical weather observations generated from the selected station.
Listing 12-5: StationList Property Code
public ObservableCollection<Statlist> StationList { get; private set; }
Listing 12-6: StationObservations Property Code
public ObservableCollection<HistHilo> StationObservations { get; private set; }
Listing 12-7: Initializing Properties
StationList = new ObservableCollection<Statlist>(); StationObservations = new ObservableCollection<HistHilo>();
Listing 12-8: Retrieving Weather Stations
private void GetWeatherStations() { StationList.Clear(); WeatherBugSvc.WeatherBugHistoricalObservationsContainer svc = new WeatherBugSvc.WeatherBugHistoricalObservationsContainer( new Uri(_svcRootUrl)); svc.Credentials = new NetworkCredential(_userId, _pwd); //Query for stations that have the value of _zipCode. //Order by station name IEnumerable<Statlist> stations = svc.Statlist .Where((station) => station.Zip_Code == _zipCode.ToString()) .OrderBy((station) => station.Station_Name); //execute the query and add each result station to the //StationList collection stations.ToList().ForEach((station) => StationList.Add(station)); }
Listing 12-9 shows the complete MainWindow constructor with the GetWeatherStations method call.
Listing 12-9: Completed MainWindows Constructor
public MainWindow() { InitializeComponent(); this.DataContext = this; StationList = new ObservableCollection<Statlist>(); StationObservations = new ObservableCollection<HistHilo>(); GetWeatherStations(); }
Listing 12-10 includes the complete GetStationHiLos method. Use Listing 12-10 to complete the GetStationHiLos method.
Listing 12-10: The Completed GetStationHiLos Method
private void GetStationHiLos(string stationId) { StationObservations.Clear(); WeatherBugSvc.WeatherBugHistoricalObservationsContainer svc = new WeatherBugSvc.WeatherBugHistoricalObservationsContainer( new Uri(_svcRootUrl)); svc.Credentials = new NetworkCredential(_userId, _pwd); IEnumerable<HistHilo> observations = svc.HistHilo .Where((hilo) => hilo.Station_ID == stationId) .OrderByDescending((hilo) => hilo.Observation_Date); observations.ToList().ForEach((hilo) => StationObservations.Add(hilo)); }
The GetStationHiLos method is similar to the GetWeatherStation method. This method filters the HistHilo collection for items that have a matching Station_ID value and places those items in the StationObservations property, which is databound to the user interface. The query returns a collection of HistHilo entity objects. The HistHilo entity class was created when the service reference was added to the project.
The GetStationHiLos method should be called when the selected station changes. This requires a transaction for each change of station. To keep the user-interface grid in sync with the currently selected station, the ComboBox’s SelectionChanged event is used. The completed cboStation_SelectionChanged method is shown in Listing 12-11. Two method calls were added. The first SetStationDetails is called to populate the selected station name and location in the user interface. Then the GetStationHiLos is called to retrieve the stations’ observations.
Listing 12-11: The Completed SelectionChanged Event
private void cboStation_SelectionChanged(object sender, SelectionChangedEventArgs e) { ClearStationDetails(); ComboBox c = sender as ComboBox; Statlist stationItem = c.SelectedItem as Statlist; SetStationDetails(stationItem); GetStationHiLos(stationItem.Station_ID); }
The Flexible query application is now complete. Build and run the application by pressing F5 in Visual Studio. The application initially retrieves the stations for the ZIP code. Selecting a specific station from the ComboBox generates a query to Datamarket for the observable collections for the station. The complete code for example shows the simplicity of accessing data from Datamarket using a Flexible query as most of the common plumbing has been created by the addition of the service reference.
The amount of data available to the user and developer is massive. The various access methods, authentication requirements, account management, and billing processes make it difficult for an organization to efficiently consume the available data. Simply finding the right data source can be problematic.
Datamarket simplifies the overhead of working with many different data sources. It does so by creating a consistent experience when you work with cloud data regardless of who owns the content or how the content is stored. Using open web technologies allows a reach beyond the Microsoft .NET tools. For those using .NET languages, integrating the data into an application is even easier with the support provided by service references and proxies.
Organizations that have premium content should look to Datamarket in order to reduce the amount of overhead of maintain the account management billing as well as a storefront. Datamarket enables content publishers to focus on content and not plumbing.