Splunk has a rich SDK that allows you to create custom dashboards with extended functionality. The aim of this chapter is to show you how to create a custom dashboard using data stored in Hadoop. We are going to visualize data containing aggregated customer data records on Google Maps. The goal is to show a customer activity heatmap.
Splunk has various SDKs for different languages and platforms. We are going to talk about them shortly and won't cover deep application development. Our goal is to get the result as rapidly as possible and using few application development skills. We can always turn to custom development, and our goal is to get quick insights as soon as possible.
There is a nice portal—https://splunkbase.splunk.com—where you can find hundreds of published applications. Extending Splunk with a SDK is not something special; it's a recommended approach to get extended functionality. This book has a chapter describing integration with MongoDB. The MongoDB integration application is taken from the splunkbase portal. We encourage you to look through published apps before starting to develop your own.
You can use several languages to create custom applications for Splunk:
Great, we have a set of popular languages; it shouldn't be a problem to find guys who knows a least one of the listed languages.
Splunk SDK could be used for these purposes:
Splunk provides a REST API for its services. Generally, it means that you can interact with Splunk components using curl/wget
or any other tool or language library that can send a correct HTTP request. Splunk provides so-called endpoints for services. You can call Splunk using a GET request:
search/jobs/export
To stream search results. Any Splunk component has its own endpoint. The Splunk SDK for the languages listed earlier simplifies interaction with the service endpoints. Each language SDK provides bindings for API endpoints and a level of abstraction over HTTP calls. This approach is common; for example, Cloudera manager API provides a Java and Python SDK. These SDKs are just wrappers for REST services of Cloudera manager. The good point is that you can integrate with Splunk and you don't have to be tightly coupled with Ruby or C#, for example. You can even call services using C++.
We are going to cover these topics in order to develop a custom dashboard with heatmaps based on aggregated customer data records collected in Milano, Italy:
Application development using Splunk SDK would make a separate book, which is why we only touch on it briefly here. Let's summarize the good points:
curl/wget
or any other tool that sends HTTP requests.Now it's time to move on to dashboard visualization using the Splunk JS stack. This should help us to reach our goal: getting results using minimal application development skills.