Storage of Web service data

Each invocation of a Web service results in an entry being made to an invocation index, which contains metadata about the query.

This index stores information such as the timestamp, the inputs, the person who initiated the query, and the ID of the returned results.

Each Web service has its own invocation index that contains this data. The name of the index is generated by combining the Web service name and the name of the group that it belongs to:

web-service-<group>-<service>-invocations

The data that Web services return is also stored in Elasticsearch. This data may be in more than one index - a Web service can return multiple types of information from a single query. These result indices are distinguished by its suffix, which may indicate the type of data that it contains:

web-service-<group>-<service>-results-<suffix>

For example, Webhose provides a suite of cybersecurity APIs that provide different types of useful data.

One of these APIs provides news articles and images. With a Web service driver that can fetch data from this API, a single search query can result in both articles and images being store in the following indices:

web-service-webhose-cyber-results-news
web-service-webhose-cyber-results-images

The invocation index named web-service-webhose-cyber-invocations contains the information for the following call:

{
 "invocation_uuid" : "bee16b10-233e-11ea-bb71-05717fd70b75",
 "timestamp" : 1576856298561,
 "user" : "jane",
 "input" : {
   "query" : "botnet"
 },
 "output" : {
   "news" : [
     "31477c0b7d7a09294de9c7918d4ef43d3f1667f4",
     "e145b2e7afb639e684b33f275d3bd000c3ec7b1c"
   ],
   "images" : [
     "cdd60d5d6023d39b59e8176046d2635173ee42f7",
     "5c426a6b97e63cdfa9cec32004ecbfa67fdbaf92"
   ]
 }
}

The news IDs are the IDs of the documents in the web-service-webhose-cyber-results-news index:

[
 {
   "id" : "31477c0b7d7a09294de9c7918d4ef43d3f1667f4",
   "title" : "FBI warns against using free WiFi networks while traveling",
   "published" : "2019-12-18T18:44:00.000Z",
   "author" : "kalitarios",
   "url" : "https://www.zdnet.com/article/fbi-warns-against-using-free-wifi-networks-while-traveling/"
 },
 {
   "id" : "e145b2e7afb639e684b33f275d3bd000c3ec7b1c",
   "title" : "Hacking back: The dangers of offensive cyber security",
   "published" : "2019-12-18T14:19:00.000Z",
   "author" : "softoasis",
   "url" : "http://www.terrorismwatch.org/2019/12/hacking-back-dangers-of-offensive-cyber.html"
 }
]

The fact that data is stored per entity, in a separate index, each with its own field structure, makes it possible to use the Data Model.

You can link content that is returned by Web services - for example, articles - with other indices or entities. This allows you to use the results coherently in dashboard navigation and in the Graph Browser.