Cooperation and knowledge sharing are at the core of The IoTCrawler project, and we are happy to announce that we are ready to go public for the benefit for other users.
We have now released a new version of the Search Engine for the Internet of Things Framework as open-source on Github:
https://github.com/IoTCrawler/Framework
The Framework includes following major components:
– The MetaData Repository, Scorpio Broker
The core component of the architecture of IoTCrawler is what we call the MetaData Repository, which can be seen as a Context Broker with the features of distributing information following both query-response and publication-subscription patterns among the other components of the IoTCrawler architecture. The Context Broker must be capable of representing semantic, linked data and property graphs. These features have been included in NGSI-LD, a new standard which has been conducted under the ETSI ISG CIM initiative. For this reason, we have selected this technology for the instantiation of our MetaData Repository.
Scorpio is an NGSI-LD compliant context broker developed by NEC Laboratories Europe and NEC Technologies India. This project is part of FIWARE. For more information check the FIWARE Catalogue entry for Core Context. We have selected this broker for the instantiation of our MetaData Repository because of it’s features related to the representation of distribution and federation scenarios.
– The Indexing Component
The Indexing component provides a means for clients to search for IoT entities efficiently. It focuses on IoT streams and sensors, where queries can be based on sensor type and absolute or relative location.
To initiate the process of indexing, a platform manager needs to register a metadata broker (MDR) with the Indexing component. In turn this will trigger the subscription to sensors and streams at the registered MDR. As metadata descriptions are updated at the MDR, the Indexing component will be notified, and will then index the sensors and streams based on location. For scalability, the Indexing component can be configured so that the persistence it relies on (MongoDB) can be sharded. With respect to other components in the framework, the Ranking component relies on Indexing for generating the ranking of search results.
– The Ranking component
This facilitates the ranking mechanism for IoT resources. Ranking and resource selection rely on the registry built (and constantly updated) by crawling and indexing methods. The purpose of Ranking is to aid users and applications to not only find a set of resources relevant to their needs, but also to select the best or most appropriate one(s) from that set. There are multiple criterias for ranking IoT resources such as data type, proximity, latency, availability etc. The Ranking component supports application-dependent, multi-criteria ranking.
– Orchestrator
The orchestrator component is responsible for interactions with client IoT applications. It allows applications to subscribe to streams without having a public endpoint as well as tracks subscription requests. In case of stream failure, orchestrator is able to notify application and provide a list of alternative streams for subscription.
– Search Enabler
The GraphQL-based search enabler component is considered as a main search-component of IoTCrawler. It employs a query language (GraphQL) and a query processor, which works on top of NGSI-LD-compliant component (the ranking Component or MDR). The component eliminates the lack of expressivity and functional capabilities which prevent NGSI-LD from being the main search interface the large-scale IoT metadata deployments gathered in the IoTCrawler platform. The search enablerb fills the gap between low-level sensors and high-level domain semantics about sensors data and deals with the context-dependent entities by maintaining the context in the IoTCrawler platform.
– Authorization Enabler
There are two purposes motivating our choice of enabler: A secure communication between the IoTCrawler components, and controlling the access of registered users to the resources stored in the IoTCrawler platform. It instantiates the Distributed Capability-Based Access Control (DCapBAC) technology by using both an XACML framework, and a Capability Manager. This latter is responsible for issuing authorisations tokens which must be present in each of the requests aimed at the MetaData Repository.
– Semantic Enrichment
The Semantic Enrichment (SE) component is responsible for annotating data streams with Quality of Information (QoI). To calculate the QoI the SE subscribes to the MDR for changes in IoTStreams. When receiveing notifications for a stream it takes the related metadata of the stream and generates the QoI annotation, which is stored in the MDR afterwards to be accessible by other components of the framework.
Explore the value of the IoTCrawler Framework
Do you see your organization as a potential early adopter? If you need inspiration to envision your uptake, take a look at the Scenarios and use-cases created by the project partners. For a further explanation of the IoTCrawler Framework, we recommend that you take a look at the video tutorial: “A General Introduction”.
Give us your feed-back
We are interested in knowing more about new means of exploitation.
How do you plan on using this? Send us an email, use #IoTCrawler on Twitter or leave a comment.
We hope with this important step to further the free co-development and exchange of ideas.
