12.3.1.1 Relational Database Management System

Relational database management system has been a popular data storage type for a long time, which was proposed in 1970 in [1]. This model protects users from the details about data organization in machines, and only provides a high level accessing-query language to operate data. However, as the development of Web 2.0 and cloud computing, RDBMS has its shortage. With static schema [2], no linear query execution time and unstable query plan, RDBMS is poor in scalability. For faster and more efficient operations for big data, the authors of [3] provided Cache Augmented Database Management System (CADBMS), improving speed of queries that read and write a certain part of data by caching. CADBMS is very useful for social network applications and others systems with high read-write ratio.

Traditional database queries follow a simple policy that defined constraints must satisfied by each tuple in the query result. This policy is computationally efficient, as the database system can evaluate the query conditions on each tuple individually. However, many practical real-world problems require a collection of result tuples to satisfy constraints collectively, rather than individually. In [4], a new query model named package queries is presented to extend traditional database queries to handle complex constraints. They design PaQL, a SQL-based query language that supports the declarative specification of package queries.

12.3.1.2 NoSQL Database

NoSQL database is also called non-relational database. Data in NoSQL database has no explicit types or patterns, they are in different buckets and related data are linked to each other. In fact, NoSQL database is a general designation, people usually divide them into three main categories: key-value stores, document-based and column-oriented. Data are stored as key-value pairs in key-value database like Amazon's SimpleDB, which supports both structured and unstructured data storage. Document-based databases such as MongoDB and Apache CouchDB store data as a collection of documents, usually JSON-based. Any fields of any length can be added, any type of data can be easily stored. As for column-oriented databases, fairly related data would be linked as an extensible column, which is different to the strictly structured table in RDBMS.

In [5], the authors said that NoSQL database systems nowadays need to make trade-off among consistency, availability and partition tolerance to optimize for their applications. While a hybrid database system can use various kinds of database softwares and take advantage of their features for individual applications and workloads, how to make the database software work together to achieve the highest performance is still a challenging problem. They also provide an extensible database interface for integrating NoSQL databases and adding database operations.

NoSQL databases are mostly non-relational, distributed, open-source and horizontally scalable. The main characteristics of these databases are schema-free, no join, non-relational, easy replication support, simple API and eventually consistent.

12.3.1.3 HDFS-Based Database

Hadoop is now one of the most popular MapReduce data storage solution. However, the programming model of Hadoop is very low level, which makes developers unable to reuse and hard to maintain these programs. Then HDFS [6] comes up. This distributed file system can run on daily-used devices with low cost and high tolerant. The high throughput makes applications with big data set more available and efficient.

Hive [7] is another open-source big data storage solution on the basis of Hadoop. What makes Hive different is that it provides HiveQL, a SQL-like declarative language. Hive compiles HiveSQL into MapReduce jobs executed using Hadoop. The language includes type system and allows user-defined script and custom type. Hive also provides schemas and statistics functions, which make it more useful in query optimization, query compilation and data exploration.

Still, the result of [8] experiments in throughput over the numbers of files, and shows that Hadoop performs poorer when the number grows larger. The bottlenecks are the size of files used, the number of data nodes available and the number of reducers used.

12.3.1.4 In-Memory Database

In-memory database management systems (IMDBMS) are designed for analysis usage like On-Line Transaction Processing (OLTP) and On-Line Analysis Processing (OLAP). MonetDB and Vectorwise are traditional OLAP engines. Nowadays more modern engines occur, including Microsoft Hekaton, HStore/VoltDB, Shore-MT etc. Recently, following the trend of executing OLTP and OLAP within same system on same database state, SAP's Hana and HyPer are developed [9].

In-memory database offers great performance for data with high update rates, thus it can be used in many daily services. One usage scenario of IMDBMS is Location-Based Service (LBS) which plays an important role in different area of WoT applications. In [10], the authors combined a series of techniques to implement in-memory storage for LBS with high inserting efficiency.

Another scenario is managing vector spatial data, which is similar to the former one. The authors of [11] concentrated on reducing the cost of I/O and improving algorithm efficiency by designing and realizing a spatial data access system on the basis of in-memory database.

In-memory database can process large in-memory datasets entirely due to the growth of main memory space currently. However, the speed of main memory operations can not be as fast as CPU's for now. Therefore, the bottleneck of main-memory techniques lies in the process of moving data from memory to CPU caches. The mainstream research aspect turns to near memory computation capabilities by making good use of hardware advantages. A near data processing accelerator named JAFAR [12], is presented for pushing “select” queries down to memory instead of pulling data into caches. By this mean, select operations in column-based data system can enjoy an improvement up to nine times as before.

12.3.1.5 BigTable

BigTable is a distributed storage system which is designed by Google, just as its name, it is proposed to deal with data in large scale. Different from another popular system HDFS, BigTable only supports structured data. Thanks to the distributed features of BigTable, developers and researchers can easily get a cloud storage solution for large-scale data task with no need to build clusters by themselves. However, the public cloud service also becomes a concern for users. How to ensure the integrity of data in cloud becomes a big issue in BigTable.

BigTable serves quantities of projects at Google [13]. Data sizes of these projects can be several petabytes in different data centre and different server. BigTable has been always satisfying the demands on large data scale and low latency.

Aiming to enhance BigTable by providing an integrity solution, the authors of [14] present iBigTable. iBigtable consists of a series of security protocols based on designed data structure and BigTable. These protocols efficiently assure that the data returned by BigTable are integral. Moreover, iBigtable inherits great features of BigTable and has a great compatibility, which allows existing BigTable applications transferring to iBigTable with little change of code.

BigTable provides a flexible, high-performance solution for various products. It is implemented by three significant components: many tablet servers, a master server and a client-based library. Tablet servers manage a set of tablet, including dealing with reading and writing operations on loaded tablets and splitting super large tablets into small ones. These servers are added or removed dynamically from a cluster to accommodate changes in workloads. The master server assigns each tablet to a tablet server in cluster, detects the change in tablet servers, balances load of tablet-servers, collects garbage of files in Google file system and handles schema changes such as creating a table or a column family.

12.3.1.6 Graph Database

Graph database utilizes features of graph to provide a scalable data storage. The queries are based on nodes, properties and edges that represent or store data. Recently, more focus is put in graph for the usability in complicated structure modelling.

In [15], some experiments on graph database Neo4j and relational database MySQL are carried out. The result shows that graph database have great advantage over relational database on structured type queries and full-text searches.

Graph databases have no schema, which is very suitable for XML document storage and biological or chemical data storage. Compare to storage in graph, retrieving data efficiently from large graph database via indices is more difficult and desirable.

Aiming to realize graph mining, a novel solution of indexing graph called gIndex [16] is proposed. Distinguished from existing methods based on path, gIndex utilizes frequent substructure as the basic indices. Frequent substructures have high stability during updates and show the intrinsic features of data, which make it ideal for graph indices. However, the size of indices will grow large in a large data warehouse, so two techniques are proposed to reduce the size, size-increasing support constraint and discriminative fragments. Besides the elegant solving in graph indexing, gIndex also illustrates that data mining can do great help to indexing and query processing, especially frequent pattern mining.

To query graph database is a big issue. Query navigation is the most important part and is heavily used in graph databases. For now, using reachability patterns with regular constraints is widely adopted for query. XPath-like languages is an example [17]. XPath is widely used in XML navigation for its ability to express queries of interest, easy query evaluation for fragments and close connection to yardstick database query languages.

Inspired to use graph to represent genome data, the authors of [18] carry out an investigation in graph-based database and are inspired to use graph to represent genome data. Researchers may build a database based on the adapted graph model storing genome data, which makes genome data storage and retrieval efficient and stable.

12.3.1.7 Comparison Between Different Data Storage Types

Based on the above references analysis, a comparison is given as Table 12.1.

From the table, we could find that WoT data storage is similar to other application data. As a traditional data storage type, RDBMS have many complex restrictions which ensure the reliability and consistency, but also make it lack scalability. However, SQL query language still makes great effect in both SQL and NoSQL database. NoSQL database has no tabular relations like traditional RDBMS does, but it owns a unique mechanism for data storage and retrieval. For the NoSQL database, HDFS-based database and BigTable perform great in distributed storage system. In big data era, they will be significant components. In-memory DB improves the performance on frequently updated data, and will show its power in geographical information systems or location-based applications. Graph database is useful for graph storage and retrieval, which makes a significant effort to social networking and semantic web applications [19].

In general, aiming to adapt to the high heterogeneity of WoT data from distributed data sources, it is a popular way to combine different data types such as RDBMS integrated with HDFS, so as to construct a scalable data storage for WoT applications.

12.3.2.1 Simple Data Model

The authors of [20] demonstrated the most significant factors in WoT: physical entity, resource and service, which can be concluded as physical entities and relationships between them. To describe these key concepts more accurately, the authors built an interlinked metadata model using micro-format such as RDF and micro-data to break the limitation of HTML format and enhance the surface representation metadata.

The authors of [21] explored a well-defined model with good extensibility for WoT information representation and organization. Based on the proposed three mainstream data type as object-cored organizing data, event-based explaining data and knowledge-based using data, the authors presented a model framework. It using two data types as different layers: the object layer and the event layer, to make an improvement over using single type only. The object layer using object-based organizing data represents all objects and relations between them. The event layer contains event-based explaining data which is extracted from raw detailed data processed by the object layer. The event layer regulates events and relations between them based on event semantic link network model with a given reasoning rule set.

The authors of [22] focused on the extraction of event information from heterogeneous and massive raw data. They proposed an approach that can effectively extract events and internal links between them from large dataset based on existing event types in a particular domain. The conceptions of event, event type, link type, and event schema are introduced and a three-layered model which consists of the data-collecting-layer, the event-extracting-layer and the presenting-layer is raised to compress the redundant data.

Aiming to describe dynamic entities in WoT applications, the authors of [23] proposed a specification model to specify the entity services. The model extends OWL-S with service status ontology to illustrate information involved in the services. The extension issues entity status in real-time and releases the information as dynamic services to requesters. With this method, the model constructs and executes transactions intelligently.

12.3.2.2 Integrated Data Model

The authors of [24] proposed an approach of creating ontological models to describe connected objects to implement support of WoT and finally to achieve unified communication between objects. Moreover, a framework is presented to allow seamless integration of semantic models and objects of web applications.

Thing Broker [25] integrates WoT objects with different characteristics, based on different protocols, providing different interfaces and constraints, meanwhile kept simple and flexible to meet the requirement from different applications. Thing Broker provides a uniform Twitter-like RESTful interface to different WoT objects. By giving one abstraction containing configurable attributes to represent each WoT object, Thing Broker manages to involve all kinds of objects in WoT, from physical entities to high-level services.

The authors of [26] delivered a formal model that provides a formal ontology representation of relations between geographic events and observations. The model exploits SEGO, a mechanism based on rules, to reason information about events via in-situ observations, and it illustrates the scenario that ontological vocabularies can be well utilized by a reasoning and querying approach to retrieve events data and sensing information.

12.3.2.3 Semantic Data Model

The authors of [27] proposed an ontology-based WoT data model called the continuum model to reflect entities evolving as space and time change. This model is important in studying the history and predicting future trends and it can track spatial entities evolving through the time and space, which play an important role in capturing semantics of modelled phenomena. This model well combines the spatial functions and temporal capabilities.

The authors of [28] proposed a general methodology that develops consumable semantic data models for smart cities. It transfers large city data of different sources into a uniformed and integrated semantic data model (RDF/OWL) by using different engineering approaches, and it enables semantic interoperability at the concept level and support application developers to design advanced city services and applications.

12.3.2.4 Comparison Between Different Representative Data Models

Based on the above references from the complex level, a comparison is given as Table 12.2.

In short, data representation is used for further WoT data disposing. Simple models such as event, RDF combined with REST API provide a common format for WoT applications. Aiming to support intelligent interaction for WoT in a contextual level, the data representation should focus on integrated model by the integration of multiple simple model such as sensors, event, RDF and other format. Considering not only the data content, but also data relationships, the semantic model based on ontology and linked open data is a promising new model for web-based applications especially with social network.

12.3.3.1 Metadata-Based Data Indexing

Metadata is a kind of special data defined for data management. It can make data easily organized and understood by users without being involved with everything concerning the accessing solution.

In [29], an efficient distributed metadata management scheme was proposed for data management in cloud platform. It uses the technique of metadata distribution based on parent directory path, hierarchical directory structure, cooperative double layer cache mechanism to access the distributed data with reduced latency.

Mobile Metadata [30] were proposed to build a mobile code agent supporting image retrieval based on client in web-based computing environments. Both the data model and mapping functions as a mashup and moves them to the client side. The model provides clear object model and view construction based on client, quick query response time and better exploit of network resources, and it is flexible in expansion.

By means of embedding metadata to represent smart things, a system was developed to control and monitor the state of WoT application environment [31]. The system produces a machine-readable state description of application environment. A Web request will be generated when a smart device reach this state, then an application will implement related operations to reconfigure the user's smart environment automatically. Therefore, intelligent application is implemented by means of metadata even huge quantity requests are involved.

12.3.3.2 Semantic-Based Model Annotation

The authors of [32] presented a framework for semantic and location-based services exploiting enriched maps. In particular, the framework contains mainly an approach for semantically annotating crowd-sourced cartographic data, and an innovative ontology-based function for semantic-based searching and disposing capabilities in current navigation systems.

To extract and link related concepts from raw sensor data and represent them as a topical ontology, a clustering approach extended k-means was used on the basis of rules extracted from external sources. The authors of [33] introduced a knowledge acquisition technique for real-world data processing aiming at topical ontologies creation and evolution. Then, these concepts are marked to make them understandable for user for the purpose of data analysis and data reasoning, and a related system is proposed for software support.

On handing unstructured models, the authors of [34] provided a semi-automatic semantic annotation in visualization scene based on three-layer ontology. The three-layer ontology including general ontology, domain ontology and scene ontology is constructed to form a comprehensive knowledge representation for semantic annotation. It is effective for large scale model management.

In short, metadata and simple index are good for structured data management. However, since WoT data are always unstructured or semi-structured in a dynamic and contextual environment, semantic-based techniques are widely studied, designed and applied to overcome these challenges in the past few years. With the rapid increase in the amount of data and their correlation, automatic metadata generation, ontology generation and evolution, and efficient, low cost and dynamic-updating semantic notation for WoT data indexing and model annotation have attracted great attention.

12.3.4.1 Data Collection

A data collection protocol named EDAL [35] is modelled similarly to the open vehicle routing problem which is proved to be NP-hard. The EDAL is efficient in energy using, aware of delay and balancing in lifetime and used to collect data in wireless sensor network (WSN) domain.

The authors of [36] constructed a universal mobile data collection framework for WoT services. They addressed four basic requirements that are task specification, task managing, status sensing and data managing. Architecture for a general-purpose mobile data collection is also proposed which separates the whole system into two parts: one is the back-end operating on server sides and the other is font-end on mobile devices.

Aiming to carry out data stream analytic, OpenWoT approach [37] was proposed. It designs an event and clustering analytic server to collect sensor data from mobile devices and serves as an interface for data stream analysis. In detail, it uses intelligent servers and edge servers for real-time data collection, annotation and processing and adds some extensions for WoT data streams.

A sensor data stream delivery system was proposed with different delivery cycles for WoT environments [38]. When connected to servers and delivering sensor data stream, the system could provide a dynamic computational and communication performance according to the different requirement of clients.

The authors of [39] presented a hybrid system based on RFID and WSN called HRW, which integrates the traditional RFID systems and WSN systems for efficient data collection. HRW has a set of smart nodes which have both RFID and WSN functions and take place of RFID readers to gather data. Moreover, an enhanced data transmission algorithm which avoids the data redundancy and unnecessary overhead in transmission and security mechanisms which avoid data manipulation and data selective forwarding are proposed.

12.3.4.2 Data pre-Disposing

Considering that redundant information will be generated among nodes close to each other, a spatial correlation model [40] was proposed to minimize total consumption in process of data preservation. The problem of data preservation with data correlation can be transformed to the minimum cost flow problem, thus a more efficient and optimal solution than the greedy algorithm is proposed.

To solve the low efficiency and data redundancy of communications in integration, a data-cleaning algorithm called cross-redundant algorithm was proposed to archive higher performance. The authors of [41] proposed a five-layer system architecture for the integration of WSN and RFID, and chose Bluetooth and ZigBee as the communication protocols.

The authors of [42] proposed a classification method for the data streams based on some supervised classification techniques like SVM (Support Vector Machine) and reduces the volume of data by simple aggregation and approximation density. The former classification and labelling steps are the fundamentals of knowledge discovery in data stream.

A framework processing raw RFID data was proposed to reduce the uncertainty of data [43]. This framework is composed of two parts: a model tracking global objects and a model cleaning local RFID data. The former one is implemented with a Markov-based model and the latter one is implemented with a particle filter based approach.

For the purpose of communication and data collection among devices in heterogeneous network interfaces, a middleware that consists of a Multiple Protocol Transport Network (MPTN) gateway and a coordinated model was proposed [44]. Messaging and data alignment among multiple networks are implemented for concurrent data stream collections.

The authors of [45] proposed a technique preserving privacy while collecting data that can be used in healthcare applications with sensor and RFID. It assures secrecy of data via a data privacy protection mechanism and has been tested to pass various attacks. Moreover, it can be adapted for different scales of network.

12.3.4.3 Information Fusion

In [46], the authors applied OLAP techniques on sensor data to integrate data from different sources and to gather the correlative information for analysis and decision-making. An on-the-fly generating solution is proposed with metadata using the W3C semantic sensor network ontology and W3C RDF data cube vocabulary for generating multidimensional data cubes.

An automatic segmentation methodology [47] was proposed for real-time high-level activity prediction. The end of the predicated activity can be automatically marked and the training dataset can be divided into segments according the previous tagging.

In [48], the authors presented an online sensor data segmentation methodology for real-time activity recognition. A two-layer strategy composed of sensor correlation and time correlation manipulation is introduced to facilitate dynamic segmentation.

A data stream clustering algorithm [49] was proposed, which makes use of sliding window and micro-clusters merging in order to batch the similar quality farm products in the agricultural WoT platform.

The authors of [50] proposed an approach to implement some operations such as inquiries for heterogeneous sensor data that is in the format of RDF. The approach can process multiple data resources at the same time on the basis of ontology and can also integrate heterogeneous sensor data. It can construct SPARQL query statements automatically and query sensor data semantically according to the requirements of users.

12.3.4.4 Distributed Data Processing

In WoT data environment, data are changing on types, states and analysis purposes. Other than centralized master-server implementations, a parallel and particle data processing framework is need to enable the execution of MapReduce pattern in dynamic information infrastructures.

MapReduce is not perfect for every large-scale analytical task, and the cost of high communication and redundant processing makes a big challenge on WoT application. An approach which uses the MapReduce framework for large-scale graph data processing was given in [51]. The approach relies on a density-based partitioning to build balanced partitions of a graph database over a set of machines. The experiments show that the performance and scalability are satisfying for large scale data processing. However, in [52], a technical framework for improving MapReduce was given.

In [53], a parallel distributed processing system was proposed for data analysis. The system manages dependent relations between data and data, as well as data and analytic programs. The system aims to illustrate dependency and uses Hadoop Streaming for distributed parallel processing requirement. There are certain repeated executions in a program and may be executed with distinguished data each time. The specification filters these executions and check dependencies separately at each execution.

In [54], a storage system with high security and scalability on the basis of revised secret sharing scheme was proposed. The system is composed of two scalable, flexible and reliable layer: data layer and system layer. Using secret sharing scheme can avoid the complicated key management when using traditional cryptographic algorithms. Moreover, multiple storage servers for WoT data work together to achieve large storage capacity. However individual servers can still join or leave flexibly at system layer.

In [55], the authors proposed vRead, a programmable framework connecting HDFS I/O flows to the application data directly. vRead supports VMs to ‘read’ data node in disk images, which improves I/O flow without the overhead of virtualization.

12.3.4.5 Comparison Between Data Operations Inside the Platform

Data operations plat a fundamental role for inner platform support. Based on disposing steps, a comparison is given as Table 12.3.

(1) Data collection is always the first step in WoT applications. Considering WoT Data is often large-scale, dynamic, and at high sampling rates, some researches focus on how to organize the data collection tasks. And also a big part of researches focus on data transmission [35,38,49]. The authors of [36] built a general mobile data collection framework and talked about four basic requirements and some open issues in current mobile data collection framework. The authors of [37] made use of the OpenWoT middleware and designed an intelligent server for real time acquisition. Moreover, some researches paid attention to some open issues in data collection, such as privacy [45].

(2) Data pre-disposing is carried out for further data operation. With different disposing purposes, data pre-disposing methods can be divided into data preservation [40], data cleaning [41,43], data alignment [44] and so on. The authors of [42] proposed a classification method for the data streams based on some supervised classification approaches like Support Vector Machine (SVM) and reduced the volume of data.

(3) After collecting data from sensors, how to dispose the semi-structured, streaming data to extract information is another problem. The authors of [22] proposed an approach to extract event information from heterogeneous and massive raw data. The authors of [47,48] proposed some methods of data segmentation in order to recognize and predict human activities. And the authors of [49] designed a new data stream clustering algorithm in the agricultural WoT platform. Some of these researches take use of sliding windows to implement real-time processing.

(4) Despite its evident merits such as scalability, fault-tolerance, and flexibility, MapReduce [51] has limitation in interactive or real-time processing on handling distributed WoT data disposing. It is not perfect for every large-scale analytical task [54], and the high communication cost and redundant processing make a big challenge for IoT application. Therefore, some optimization work on WoT data are still needed for a large-scale processing purpose.

To sum up, the researches for WoT data operation are mainly concentrated on the following respects.

Firstly, researchers will pay more attention to disposing of different characteristics of WoT data, such as the elimination of redundant data, the alignment and merging of heterogeneous data, and online analysis for dynamic data. Secondly, researchers will take the integrations with existing technologies into consideration. On grounds that data in WoT applications are always at large-scale and with high sampling rate, the data disposing in WoT will be combined with the technologies of distributed computing, and streaming computing, such as Hadoop and Storm [56]. Thirdly, researchers will focus on more usage scenarios, such as activity recognition, complex cooperation, etc.

12.3.5.1 Data Interoperability

Although data interoperability development keeps innovative, some challenges exist. The researchers proposed a hub-centric framework [57] on interoperability and validated this framework in a large-scale WoT environment.

A novel Semantic WoT framework was proposed based on the Constrained Application Protocol (CoAP) [58]. The framework supports annotated resources retrieval and logical ranking on the basis of semantic matchmaking services with non-standard interface. To detect high-level events and specify them using machine-readable metadata, the framework also includes some approaches on data mining to deal with raw data gathered.

The authors of [59] proposed SAMPLES to classify network traffic generated by mobile applications. SAMPLES is composed of an offline part and an online part. The offline part is a training system in charge of rule generation and the online part is an engine for application identification and traffic classification. For each input flow, a subset of conjunctive rules is applied to the flow on the basis of pre-filtering conditions. Conjunctive rules are decided by context of lexica and a unique identifier of application in HTTP header.

12.3.5.2 Data-Centric Service Composition

Mashup tools [60] are used for the development of WoT applications, which can connect the dataflow between applications and devices in a graphical way. RESTful interfaces are generated from the WoT data models, which represent a set of sensors and actuators. Generic components extend existing mashup concepts and employ concepts to polymorphic functions as in many programming languages.

By means of the composition of web services with data streams from WoT devices [61], WoT devices are connected with web service in an efficient and extensible way. Thus real-time communication, device integration and data stream mashups are elaborated.

DiscoWoT [62] provided a semantic discovery service that supports semantic discovery of the functionality of smart things by human and machines. On the basis of multiple discovery strategies, the service provided by DiscoWoT supports all strategies using RESTful interface created or updated by users at runtime.

12.3.5.3 Data Analysis

To make WoT smarter [63], data mining was introduced into applications. A system architecture for WoT and big data mining system was proposed, in which lots of WoT devices are integrated into this system to perceive the world and generate data continuously. The system focuses on the integration with devices and data mining technologies, where data mining functions will be provided as service.

Condor [64] was proposed to handle data-parallel style execution of analysis algorithms in WoT system. The analytic processes are naturally data-parallel but the executions are not. Therefore, how to execute these processes in fixed time simultaneously becomes an important challenge. The architecture of the framework allows to synchronously execute any algorithms considering them as black boxes.

In [65], an overview of related issues and challenges in aspect of big data provenance research was presented, like Accessing Big Data, Minimum Computational Overhead Requirement and so on.

To sum up, RESTful service is the main form for external applications support. Application integration across heterogeneous and distributed environments is implemented by means of RESTful service. Thus, a flexible application construction and execution environment is provided for application interaction. However, how to combine REST APIs with inner distributed disposing environment for massive data analysis is not an easy task.