Introduction

Siren Investigate is an open source data intelligence platform built upon Kibana 10.0.0-beta-1. Siren Investigate 10.0.0-beta-1 supports Elasticsearch 5.6.7 and the Federate plugin version 5.6.7-10.0.0-beta-1. The Siren Federate plugin replaces the Siren Join plugin for distributions based on Elasticsearch 5.x.

Siren Investigate allows you to perform complex analytics on large volumes of data by providing customizable visualizations (charts, maps, metrics and tables) on Elasticsearch searches. Visualizations can be organized into multiple dashboards.

Search results can be filtered interactively through a variety of techniques (date ranges, full text queries, field value matching). By setting up relations between indices, it is possible to filter search results matching documents in a different dashboard, e.g., by displaying only companies that received investments in a particular year.

In addition, search results can be filtered and augmented by queries on multiple external datasources such as SQL databases and REST APIs; queries on external datasources can also be used as aggregations in visualizations.

In addition to visualizations provided by Kibana, Siren Investigate provides:

  • The Relational Filter visualization, which allows you to configure relations between fields in different indices and to apply cross-dashboard filters (pivoting).

  • The Siren Investigate Timeline visualization, which displays a timeline with multiple groups of data coming from different indices.

  • The Radar Chart visualization, which is a graphical method for displaying multivariate data with multiple groups of data coming from different indices.

  • The Bubble Diagram visualization, which displays series of data grouped into packed circles.

  • The Scatter Plot visualization, which displays a scatter plot chart in different modes.

  • The Box Plot visualization, which displays a box plot chart from the data.

  • The Kibi Horizontal Bar Chart visualization, which displays a horizontal bar chart.

  • The Multi Chart visualization, which displays different types of charts for the same data and allow to save and to select multiple aggregation configurations.

  • The Enhanced Search Results visualization, which displays query results in a table.

  • The Siren Investigate Query Viewer, which enables the visualization of queries on external datasource through Jade or Handlebars templates.

  • The Siren Investigate Graph Browser, which displays the currently selected Elasticsearch documents as a node of a graph and allows the user to visually explore the connection between vertices.

The Relational Filter visualization requires the Siren Federate plugin 5.6.7-10.0.0-beta-1 for Elasticsearch.

How does Siren Investigate compare to Kibana?

Siren Investigate is currently developed as a fork of Kibana 10.0.0-beta-1. Although configuration objects are mostly the same, it is recommended to keep Siren Investigate and Kibana in separate indices.

What’s new in Siren Investigate v10.0.0-beta-1

To see all the changes, please check the full release notes.

Set Up Siren Investigate

Installing Siren Investigate

Siren Investigate is provided in the following package formats:

tar.gz/zip

The tar.gz packages are provided for installation on Linux and Darwin and are the easiest choice for getting started with Siren Investigate.

The zip package is the only supported package for Windows.

docker

Siren Investigate Docker images are available at https://hub.docker.com/u/sirensolutions/.

If your Elasticsearch installation is protected by X-Pack Security see Using Kibana with X-Pack Security for additional setup instructions.

Running Siren Investigate on Docker

Docker images for Siren Investigate are available from the sirensolutions organization on Dockerhub.

Pulling the Image

Obtaining Siren Investigate for Docker is as simple as issuing a docker pull command.

The Docker image for Siren Investigate 10.0.0-beta-1 can be retrieved with the following command:

docker pull sirensolutions/siren-platform:latest
docker run -d -p 5606:5606 -p 9220:9220 sirensolutions/siren-platform:latest

or for specific version, e.g., 10.0.0-beta-1:

docker pull sirensolutions/siren-platform:10.0.0-beta-1
docker run -d -p 5606:5606 -p 9220:9220 sirensolutions/siren-platform:10.0.0-beta-1

For an image pre-populated with demonstration data:

docker pull sirensolutions/siren-platform-demo-data:latest
docker run -d -p 5606:5606 -p 9220:9220 sirensolutions/siren-platform-demo-data:latest

Environment variable configuration

Under Docker, Siren Investigate can be configured via environment variables. When the container starts, a helper process checks the environment for variables that can be mapped to Siren Investigate command-line arguments.

For compatibility with container orchestration systems, these environment variables are written in all capitals, with underscores as word separators. The helper translates these names to valid Siren Investigate setting names.

Some example translations are shown here:

Example Docker Environment Variables
Environment Variable

Siren Investigate Setting

SERVER_NAME

server.name

KIBANA_DEFAULTAPPID

kibana.defaultAppId

XPACK_MONITORING_ENABLED

xpack.monitoring.enabled

In general, any setting listed in Configuring Siren Investigate or X-Pack Settings can be configured with this technique.

These variables can be set with docker-compose like this:

services:
  kibi:
    image: docker.elastic.co/kibana/kibana:{version}
    environment:
      SERVER_NAME: kibi.example.org
      ELASTICSEARCH_URL: http://elasticsearch.example.org

Since environment variables are translated to CLI arguments, they take precedence over settings configured in siren.yml.

Docker defaults

The default settings when using the siren-kibi image (standalone Siren Investigate image) are:

elasticsearch.url

http://localhost:9220

server.basepath

""

kibi.index

.siren

Install Siren Investigate with .tar.gz

Siren Investigate is provided for Linux and Darwin as a .tar.gz package. These packages are the easiest formats to use when trying out Kibana.

The latest stable version of Kibana can be found on the Download Kibana page. Other versions can be found on the Past Releases page.

Download and install the Linux 64-bit package

The 64-bit Linux archive for Kibana v10.0.0-beta-1 can be downloaded and installed as follows:

wget https://artifacts.elastic.co/downloads/kibana/kibana-10.0.0-beta-1-linux-x86_64.tar.gz
sha1sum kibana-10.0.0-beta-1-linux-x86_64.tar.gz (1)
tar -xzf kibana-10.0.0-beta-1-linux-x86_64.tar.gz
cd kibana-10.0.0-beta-1-linux-x86_64/ (2)
1 Compare the SHA produced by sha1sum or shasum with the published SHA.
2 This directory is known as $KIBANA_HOME.

Download and install the Linux 32-bit package

The 32-bit Linux archive for Kibana v10.0.0-beta-1 can be downloaded and installed as follows:

wget https://artifacts.elastic.co/downloads/kibana/kibana-10.0.0-beta-1-linux-x86.tar.gz
sha1sum kibana-10.0.0-beta-1-linux-x86.tar.gz (1)
tar -xzf kibana-10.0.0-beta-1-linux-x86.tar.gz
cd kibana/ (2)
1 Compare the SHA produced by sha1sum or shasum with the published SHA.
2 This directory is known as $KIBANA_HOME.

Download and install the Darwin package

The Darwin archive for Kibana v10.0.0-beta-1 can be downloaded and installed as follows:

curl -O https://artifacts.elastic.co/downloads/kibana/kibana-10.0.0-beta-1-darwin-x86_64.tar.gz
shasum kibana-10.0.0-beta-1-darwin-x86_64.tar.gz (1)
tar -xzf kibana-10.0.0-beta-1-darwin-x86_64.tar.gz
cd kibana-10.0.0-beta-1-darwin-x86_64/ (2)
1 Compare the SHA produced by sha1sum or shasum with the published SHA.
2 This directory is known as $KIBANA_HOME.

tar.gz packages can be downloaded from the download page. The demo versions contain a pre-configured Elasticsearch cluster in addition to Siren Investigate.

Running Siren Investigate from the Command Line

Siren Investigate can be started from the command line as follows:

./bin/investigate

By default, Siren Investigate runs in the foreground, prints its logs to the standard output (stdout), and can be stopped by pressing Ctrl-C.

Configuring Siren Investigate via Config File

Siren Investigate loads its configuration from the $KIBI_HOME/config/siren.yml file by default. The format of this config file is explained in Configuring Siren Investigate.

Directory Layout of .tar.gz Archives

The .tar.gz packages are entirely self-contained.

This is very convenient because you do not have to create any directories to start using Siren Investigate, and uninstalling is as easy as removing the directory. However, it is advisable to change the default locations of the config and data directories so that you do not delete important data later on.

Type Description Default Location Setting

home

Siren Investigate home directory or $KIBI_HOME

Directory created by unpacking the archive; in demo distributions, the directory is kibi.

bin

Binary scripts including kibi to start the Siren Investigate server and kibi-plugin to install plugins

$KIBI_HOME\bin

config

Configuration files including siren.yml

$KIBI\config

data

The location of the data files written to disk by Siren Investigate and its plugins

$KIBI_HOME\data

optimize

Transpiled source code. Certain administrative actions, e.g., plugin install, result in the source code being retranspiled on the fly.

$KIBI\optimize

plugins

The location of the plugin files. Each plugin will be contained in a subdirectory.

$KIBI\plugins

Install Siren Investigate on Windows

Siren Investigate can be installed on Windows using the .zip package; zip packages can be downloaded from the download page. The demo versions contain a pre-configured Elasticsearch cluster in addition to Siren Investigate.

Running Siren Investigate from the Command Line

Siren Investigate can be started from the command line as follows:

.\bin\kibi

By default, Siren Investigate runs in the foreground, prints its logs to STDOUT, and can be stopped by pressing Ctrl-C.

Configuring Siren Investigate via Config File

Siren Investigate loads its configuration from the $KIBI_HOME/config/siren.yml file by default. The format of this config file is explained in Configuring Siren Investigate.

Directory Layout of .zip Archive

The .zip package is entirely self-contained.

This is very convenient because you do not have to create any directories to start using Siren Investigate, and uninstalling Siren Investigate is as easy as removing the directory. However, it is advisable to change the default locations of the config and data directories so that you do not delete important data later on.

Type Description Default Location Setting

home

Siren Investigate home directory or %KIBI_HOME%

Directory created by unpacking the archive; in demo distributions, the directory is kibi.

bin

Binary scripts including kibi to start the Siren Investigate server and kibi-plugin to install plugins

%KIBI_HOME%\bin

config

Configuration files including siren.yml

%KIBI_HOME%\config

data

The location of the data files written to disk by Siren Investigate and its plugins

%KIBI_HOME%\data

optimize

Transpiled source code. Certain administrative actions, e.g., plugin install, result in the source code being retranspiled on the fly.

%KIBI_HOME%\optimize

plugins

The location of the plugin files. Each plugin will be contained in a subdirectory.

%KIBI_HOME%\plugins

Configuring Siren Investigate

The Siren Investigate server reads properties from the siren.yml file on startup. The default settings configure Siren Investigate to run on localhost:5606. To change the host or port number, or connect to Elasticsearch running on a different machine, you’ll need to update your siren.yml file. You can also enable SSL and set a variety of other options.

External datasource configuration is documented in the JDBC Datasources and External Datasources chapters, while access control configuration is documented in the Search Guard Integration and Siren Investigate access control chapter.

Environment Variable Placeholders

It is possible to use environment variable placeholders in configuration settings. The syntax of placeholders is ${ENV_VARIABLE_NAME}.

For example, to set elasticsearch.url to the value of the environment variable ES_URL, edit config_kibi.yml as follows:

elasticsearch.url: ${ES_URL}
Configuration Settings
server.port:

Default: 5606 - Siren Investigate is served by a back end server. This setting specifies the port to use.

server.host:

Default: "localhost" - This setting specifies the host of the back end server.

server.basePath:

Enables you to specify a path to mount Siren Investigate as if you are running behind a proxy. This only affects the URLs generated by Siren Investigate, your proxy is expected to remove the basePath value before forwarding requests to Siren Investigate. This setting cannot end in a slash (/).

server.maxPayloadBytes:

Default: 1048576 - The maximum payload size in bytes for incoming server requests.

server.name:

Default: "your-hostname" - A human-readable display name that identifies this Siren Investigate instance.

server.defaultRoute:

Default: "/app/kibana" - This setting specifies the default route when opening Siren Investigate. You can use this setting to modify the landing page when opening Siren Investigate.

elasticsearch.url:

Default: "http://localhost:9220" - The URL of the Elasticsearch instance to use for all your queries.

elasticsearch.preserveHost:

Default: true - When this setting’s value is true Siren Investigate uses the hostname specified in the server.host setting. When the value of this setting is false, Siren Investigate uses the hostname of the host that connects to this Siren Investigate instance.

kibana.index:

Default: ".siren" - Siren Investigate uses an index in Elasticsearch to store saved searches, visualizations and dashboards. Siren Investigate creates a new index if the index does not already exist.

kibana.defaultAppId:

Default: "discover" - The default application to load.

tilemap.url:

The URL to the tile service that Siren Investigate uses to display map tiles in tilemap visualizations. By default, Siren Investigate reads this url from an external metadata service, but users can still override this parameter to use their own Tile Map Service. For example: https://tiles.elastic.co/v2/default/{z}/{x}/{y}.png?elastic_tile_service_tos=agree&my_app_name=kibana

tilemap.options.minZoom:

Default: 1 - The minimum zoom level.

tilemap.options.maxZoom:

Default: 10 - The maximum zoom level.

tilemap.options.attribution:

Default: "© [OpenStreetMap]("http://www.openstreetmap.org/copyright")" - The map attribution string.

tilemap.options.subdomains:

An array of subdomains used by the tile service. Specify the position of the subdomain the URL with the token {s}.

regionmap

Specifies additional vector layers for use in Region Map visualizations. Each layer object points to an external vector file that contains a geojson FeatureCollection. The file must use the [WGS84 coordinate reference system](https://en.wikipedia.org/wiki/World_Geodetic_System) and only include polygons. If the file is hosted on a separate domain from Kibana, the server needs to be CORS-enabled so Kibana can download the file. The following example shows a valid regionmap configuration.

regionmap:
  layers:
     - name: "Departments of France"
       url: "http://my.cors.enabled.server.org/france_departements.geojson"
       attribution: "INRAP"
       fields:
          - name: "department"
            description: "Full department name"
          - name: "INSEE"
            description: "INSEE numeric identifier"
name:

Mandatory. A description of the map being provided.

url:

Mandatory. The location of the geojson file as provided by a webserver.

attribution:

Optional. References the originating source of the geojson file.

fields:

Mandatory. Each layer can contain multiple fields to indicate what properties from the geojson features you wish to expose. The example above shows how to define multiple properties.

fields.name:

Mandatory. This value is used to do an inner-join between the document stored in Elasticsearch and the geojson file. e.g. if the field in the geojson is called Location and has city names, there must be a field in Elasticsearch that holds the same values that Kibana can then use to lookup for the geoshape data.

fields.description:

Mandatory. The human readable text that is shown under the Options tab when building the Region Map visualization.

elasticsearch.username: and elasticsearch.password:

If your Elasticsearch is protected with basic authentication, these settings provide the username and password that the Siren Investigate server uses to perform maintenance on the Siren Investigate index at startup. Your Siren Investigate users still need to authenticate with Elasticsearch, which is proxied through the Siren Investigate server.

server.ssl.enabled

Default: "false" - Enables SSL for requests sent by the browser / reverse proxy to the Siren Investigate backend. When set to true, server.ssl.certificateAuthorities and server.ssl.key are required.

server.ssl.cert: and server.ssl.key:

Paths to the PEM-format SSL certificate and SSL key files, respectively.

server.ssl.keyPassphrase

The passphrase that will be used to decrypt the private key. This value is optional as the key may not be encrypted.

server.ssl.certificateAuthorities

Array of paths to PEM encoded certificate files that should be trusted.

server.ssl.supportedProtocols

Default: TLSv1, TLSv1.1, TLSv1.2 - Supported protocols with versions. Valid protocols: TLSv1, TLSv1.1, TLSv1.2

server.ssl.cipherSuites

Default: ECDHE-RSA-AES128-GCM-SHA256, ECDHE-ECDSA-AES128-GCM-SHA256, ECDHE-RSA-AES256-GCM-SHA384, ECDHE-ECDSA-AES256-GCM-SHA384, DHE-RSA-AES128-GCM-SHA256, ECDHE-RSA-AES128-SHA256, DHE-RSA-AES128-SHA256, ECDHE-RSA-AES256-SHA384, DHE-RSA-AES256-SHA384, ECDHE-RSA-AES256-SHA256, DHE-RSA-AES256-SHA256, HIGH,!aNULL, !eNULL, !EXPORT, !DES, !RC4, !MD5, !PSK, !SRP, !CAMELLIA - Details on the format, and the valid options, are available via the [OpenSSL cipher list format documentation](https://www.openssl.org/docs/man1.0.2/apps/ciphers.html#CIPHER-LIST-FORMAT)

elasticsearch.ssl.certificate: and elasticsearch.ssl.key:

Optional settings that provide the paths to the PEM-format SSL certificate and key files. These files validate that your Elasticsearch backend uses the same key files.

elasticsearch.ssl.keyPassphrase

The passphrase that will be used to decrypt the private key. This value is optional as the key may not be encrypted.

elasticsearch.ssl.certificateAuthorities:

Optional setting that enables you to specify a list of paths to the PEM file for the certificate authority for your Elasticsearch instance.

elasticsearch.ssl.verificationMode:

Default: full - Controls the verification of certificates. Valid values are none, certificate, and full. full performs hostname verification, and certificate does not.

elasticsearch.pingTimeout:

Default: the value of the elasticsearch.requestTimeout setting - Time in milliseconds to wait for Elasticsearch to respond to pings.

elasticsearch.requestTimeout:

Default: 30000 - Time in milliseconds to wait for responses from the back end or Elasticsearch. This value must be a positive integer.

elasticsearch.requestHeadersWhitelist:

Default: [ 'authorization' ] - List of Siren Investigate client-side headers to send to Elasticsearch. To send no client-side headers, set this value to [] (an empty list).

elasticsearch.customHeaders:

Default: {} - Header names and values to send to Elasticsearch. Any custom headers cannot be overwritten by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.

elasticsearch.shardTimeout:

Default: 0 - Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.

elasticsearch.startupTimeout:

Default: 5000 - Time in milliseconds to wait for Elasticsearch at Siren Investigate startup before retrying.

pid.file:

Specifies the path where Siren Investigate creates the process ID file.

path.data

Default: ./data The path where Kibana stores persistent data not saved in Elasticsearch

logging.dest:

Default: stdout - Enables you specify a file where Siren Investigate stores log output.

logging.silent:

Default: false - Set the value of this setting to true to suppress all logging output.

logging.quiet:

Default: false - Set the value of this setting to true to suppress all logging output other than error messages.

logging.verbose

Default: false - Set the value of this setting to true to log all events, including system usage information and all requests.

ops.interval

Default: 5000 - Set the interval in milliseconds to sample system and process performance metrics. The minimum value is 100.

status.allowAnonymous

Default: false - If authentication is enabled, setting this to true allows unauthenticated users to access the Siren Investigate server status API and status page.

console.enabled

Default: true Set to false to disable Console. Toggling this will cause the server to regenerate assets on the next startup, which may cause a delay before pages start being served.

elasticsearch.tribe.url:

Optional URL of the Elasticsearch tribe instance to use for all your queries.

elasticsearch.tribe.username: and elasticsearch.tribe.password:

If your Elasticsearch is protected with basic authentication, these settings provide the username and password that the Kibana server uses to perform maintenance on the Kibana index at startup. Your Kibana users still need to authenticate with Elasticsearch, which is proxied through the Kibana server.

elasticsearch.tribe.ssl.cert: and elasticsearch.tribe.ssl.key:

Optional settings that provide the paths to the PEM-format SSL certificate and key files. These files validate that your Elasticsearch backend uses the same key files.

elasticsearch.tribe.ssl.keyPassphrase

The passphrase that will be used to decrypt the private key. This value is optional as the key may not be encrypted.

elasticsearch.tribe.ssl.certificateAuthorities:

Optional setting that enables you to specify a path to the PEM file for the certificate authority for your tribe Elasticsearch instance.

elasticsearch.tribe.ssl.verificationMode:

Default: full - Controls the verification of certificates. Valid values are none, certificate, and full. full performs hostname verification, and certificate does not.

elasticsearch.tribe.pingTimeout:

Default: the value of the elasticsearch.tribe.requestTimeout setting - Time in milliseconds to wait for Elasticsearch to respond to pings.

elasticsearch.tribe.requestTimeout:

Default: 30000 - Time in milliseconds to wait for responses from the back end or Elasticsearch. This value must be a positive integer.

elasticsearch.tribe.requestHeadersWhitelist:

Default: [ 'authorization' ] - List of Siren Investigate client-side headers to send to Elasticsearch. To send no client-side headers, set this value to [] (an empty list).

elasticsearch.tribe.customHeaders:

Default: {} - Header names and values to send to Elasticsearch. Any custom headers cannot be overwritten by client-side headers, regardless of the elasticsearch.tribe.requestHeadersWhitelist configuration.

kibi_core.default_dashboard_title

Default: not set - The dashboard that is displayed when clicking on the Dashboard tab for the first time. This property is deprecated starting from Siren Investigate 4.6.4-4, it was moved to advanced_settings (Setting Advanced Options)

Accessing Siren Investigate

Siren Investigate is a web application that you access through port 5606. All you need to do is point your web browser at the machine where Siren Investigate is running and specify the port number. For example, http://localhost:5606 or http://YOURDOMAIN.com:5606.

When you access Siren Investigate, the Discover page loads by default with the default index pattern selected. The time filter is set to the last 15 minutes and the search query is set to match-all (*).

If you do not see any documents, try setting the time filter to a wider time range. If you still do not see any results, it is possible that you do not have any documents.

Checking Siren Investigate Status

You can reach the Siren Investigate server’s status page by navigating to http://localhost:5606/status. The status page displays information about the server’s resource usage and lists the installed plugins.

kibi status page

Collecting Elasticsearch Diagnostics

The Elasticsearch diagnostics button generates a single file by collecting different metrics about your elasticsearch cluster. All collected information are saved to a local file and never transferred over a network. You can see a full list of elasticsearch API calls by clicking the more info icon next to the button.

kibi status page diagnostics help

Connect Siren Investigate to Elasticsearch

Before you can start using Siren Investigate, you need to tell it which Elasticsearch indices you want to explore. The first time you access Siren Investigate, you are prompted to define an index pattern that matches the name of one or more of your indices. That’s it. That’s all you need to configure to start using Siren Investigate. You can add index patterns at any time from the Management tab.

By default, Siren Investigate connects to the Elasticsearch instance running on localhost. To connect to a different Elasticsearch instance, modify the Elasticsearch URL in the siren.yml configuration file and restart Siren Investigate. For information about using Siren Investigate with your production nodes, see Using Siren Investigate in a Production Environment.

To configure the Elasticsearch indices you want to access with Siren Investigate:

  1. Point your browser at port 5606 to access the Siren Investigate UI. For example, localhost:5606 or http://YOURDOMAIN.com:5606.

    Siren Investigate start page

  2. Specify an index pattern that matches the name of one or more of your Elasticsearch indices. You may have to access the index patterns management in the Management tab. By default, Siren Investigate guesses that you’re working with data being fed into Elasticsearch by Logstash. If that’s the case, you can use the default logstash-* as your index pattern. The asterisk (*) matches zero or more characters in an index’s name. If your Elasticsearch indices follow some other naming convention, enter an appropriate pattern. The "pattern" can also simply be the name of a single index.

  3. Select the index field that contains the timestamp that you want to use to perform time-based comparisons. Siren Investigate reads the index mapping to list all of the fields that contain a timestamp. If your index doesn’t have time-based data, disable the Index contains time-based events option.

  4. Click Create to add the index pattern. This first pattern is automatically configured as the default. When you have more than one index pattern, you can designate which one to use as the default by clicking on the star icon above the index pattern title from Management > Index Patterns.

All done! Siren Investigate is now connected to your Elasticsearch data. Siren Investigate displays a read-only list of fields configured for the matching index.

Siren Investigate relies on dynamic mapping to use fields in visualizations and manage the .siren index. If you have disabled dynamic mapping, you need to manually provide mappings for the fields that Siren Investigate uses to create visualizations. For more information, see Siren Investigate and Elasticsearch Dynamic Mapping.

Start Exploring your Data!

You’re ready to dive in to your data:

  • Search and browse your data interactively from the Discover page.

  • Chart and map your data from the Visualize page.

  • Create and view custom dashboards from the Dashboard page.

For a step-by-step introduction to these core Siren Investigate concepts, see the Getting Started tutorial.

Siren Investigate and Elasticsearch Dynamic Mapping

By default, Elasticsearch enables dynamic mapping for fields. Siren Investigate needs dynamic mapping to use fields in visualizations correctly, as well as to manage the .siren index where saved searches, visualizations, and dashboards are stored.

If your Elasticsearch use case requires you to disable dynamic mapping, you need to manually provide mappings for fields that Siren Investigate uses to create visualizations. You also need to manually enable dynamic mapping for the .siren index.

The following procedure assumes that the .siren index does not already exist in Elasticsearch and that the index.mapper.dynamic setting in elasticsearch.yml is set to false:

  1. Start Elasticsearch.

  2. Create the .siren index with dynamic mapping enabled just for that index:

    PUT .siren
    {
      "index.mapper.dynamic": true
    }
  3. Start Siren Investigate and navigate to the web UI and verify that there are no error messages related to dynamic mapping.

Using Siren Investigate with Tribe nodes

While tribe nodes have been deprecated in Elasticsearch in favor of Cross Cluster Search, you can still use Siren Investigate with tribe nodes until elasticsearch version 7.0. Unlike tribe nodes, using cross cluster search in Siren Investigate requires no server-side configurations and doesn’t disable functionality like Console.

Siren Investigate can be configured to connect to a tribe node for data retrieval. Because tribe nodes can’t create indices, Siren Investigate additionally requires a separate connection to a node to maintain state. When configured, searches and visualizations will retrieve data using the tribe node and administrative actions (such as saving a dashboard) will be sent to non-tribe node.

Configuring Siren Investigate for tribe nodes

Tribe nodes take all of the same configuration options used when configuring Elasticsearch in siren.yml. Tribe options are prefixed with elasticsearch.tribe and at a minimum requires a url:

elasticsearch.url: "<your_administration_node>"
elasticsearch.tribe.url: "<your_tribe_node>"

When configured to use a tribe node, actions that modify Siren Investigate’s state will be sent to the node at elasticsearch.url. Searches and visualizations will retrieve data from the node at elasticsearch.tribe.url. It’s acceptable to use a node for elasticsearch.url that is part of one of the clusters that a tribe node is pointing to.

The full list of configurations can be found at Configuring Kibana.

Limitations

Due to the ambiguity of which cluster is being used, certain features are disabled in Siren Investigate:

  • Console

  • Managing users and roles with the x-pack plugin

Using Siren Investigate in a Production Environment

How you deploy Siren Investigate largely depends on your use case. If you are the only user, you can run it on your local machine and configure it to point to whatever Elasticsearch instance you want to interact with. Conversely, if you have a large number of heavy users, you might need to load balance across multiple instances that are all connected to the same Elasticsearch cluster.

While Siren Investigate isn’t terribly resource intensive, we still recommend running Siren Investigate separate from your Elasticsearch data or master nodes.

Make sure to set the configuration file as readable only to the user running the Siren Investigate process as it contains encryption keys to protect configuration settings stored in Elasticsearch; if you are connecting Siren Investigate to external datasources, we also recommend to use credentials with no write privileges as they are not required by the system.

Access control

Siren Investigate is compatible with Elastic x-pack and Search Guard to enable index and document level access control.

For more information about the Enterprise Edition access control features please see the Access Control section.

Load Balancing Across Multiple Elasticsearch Nodes

If you have multiple nodes in your Elasticsearch cluster, the easiest way to distribute Siren Investigate requests across the nodes is to run an Elasticsearch client node on the same machine as Siren Investigate. Elasticsearch client nodes are essentially smart load balancers that are part of the cluster. They process incoming HTTP requests, redirect operations to the other nodes in the cluster as needed, and gather and return the results. For more information, see {ref}/modules-node.html[Node] in the Elasticsearch reference.

To use a local client node to load balance Siren Investigate requests:

  1. Install Elasticsearch on the same machine as Siren Investigate.

  2. Configure the node as a client node. In elasticsearch.yml, set both node.data and node.master to false:

    # 3. You want this node to be neither master nor data node, but
    #    to act as a "search load balancer" (fetching data from nodes,
    #    aggregating results, etc.)
    #
    node.master: false
    node.data: false
    node.ingest: false
  3. Configure the client node to join your Elasticsearch cluster. In elasticsearch.yml, set the cluster.name to the name of your cluster.

    cluster.name: "my_cluster"
  4. Make sure Siren Investigate is configured to point to your local client node. In siren.yml, the elasticsearch.url should be set to localhost:9220.

    # The Elasticsearch instance to use for all your queries.
    elasticsearch.url: "http://localhost:9220"

Upgrading from a Previous Version

Before upgrading make sure to check the Breaking changes section.

An existing Siren Investigate installation can be upgraded as follows:

  • backup the .siren index.

  • backup the Siren Investigate configuration file (config/siren.yml)

  • backup the .sirenaccess if ACL (Access Control Layer) is enabled.

  • upgrade Elasticsearch;

  • before restarting each Elasticsearch node, make sure to install a compatible version of the Siren Federate plugin and access control plugins if required.

  • download and extract the new Siren Investigate version.

  • copy the previous configuration file to the config directory of the new installation.

  • copy the files from the data directory in your old installation of the new installation.

  • check for breaking changes to the configuration described below.

  • install the compatible versions of third party Siren Investigate/Kibana plugins that you might need in addition to the bundled ones.

  • execute the upgrade command.

Backing up and Restoring the Siren Investigate Indices.

Before upgrading it is strongly recommended to have a backup of the .siren index; the recommended way to perform regular backups of Elasticsearch indexes is through the snapshot and restore modules.

Siren Investigate ships with a command line interface for creating dumps of the .siren index and, in case the ACL is enabled, the .sirenaccess index as well. An index dump is composed of two parts: its mappings and its data.

Backup

The backup command requires a running Elasticsearch instance and the path to a folder where the dumps will be written to.

You can know more about its options by executing the following:

$ ./bin/investigate backup --help

For example, the following line will dump in <MY_FOLDER> the .siren index and the .sirenaccess index if the option kibi_access_control.acl.enabled is true in siren.yml:

$ ./bin/investigate backup --backup-dir <MY_FOLDER>

Restore

The restore command requires a running Elasticsearch instance and the path to a folder where the dumps were written to by the previous backup command.

You can know more about its options by executing the following:

$ ./bin/investigate restore --help

For example, you can restore the previously saved indices by executing the command and pointing to the dump folder, with .sirenaccess as well if the option kibi_access_control.acl.enabled is true in siren.yml:

$ ./bin/investigate restore --backup-dir <MY_FOLDER>

In addition, an upgrade of the Siren Investigate indices is also executed after a successful restore.

Upgrading the .siren index.

To upgrade the objects in the .siren index (dashboards, visualizations, etc.), move to the directory in which Siren Investigate is installed and execute the following command:

bin/investigate upgrade

The command will look for out of date objects and upgrade them, e.g.:

$ bin/investigate upgrade
  log   [17:58:33.494] [info][status][plugin:elasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [17:58:36.127] [info][migrations] Executing migration "Upgrade scripts from version 1 to version 2"
  log   [17:58:36.141] [info][migrations] Executed migration "Upgrade scripts from version 1 to version 2"
  log   [17:58:36.142] [info][migrations] Executing migration "Upgrade kibi graph browser visualization to version 2."
  log   [17:58:36.157] [info][migrations] Executed migration "Upgrade kibi graph browser visualization to version 2."
  log   [17:58:36.158] [info][migrations] Executing migration "Upgrade saved queries from version 1 to version 2"
  log   [17:58:36.242] [info][migrations] Executed migration "Upgrade saved queries from version 1 to version 2"
  log   [17:58:36.242] [info][migrations] Executing migration "Upgrade saved templates from version 1 to version 2"
  log   [17:58:36.303] [info][migrations] Executed migration "Upgrade saved templates from version 1 to version 2"
  log   [17:58:36.303] [info][migrations] Executing migration "Upgrade saved queries definitions in external query terms aggregation, enhanced search results and query viewer."
  log   [17:58:36.400] [info][migrations] Executed migration "Upgrade saved queries definitions in external query terms aggregation, enhanced search results and query viewer."
Upgraded 20 objects.

It is possible to run the command multiple times, however running the command at the same time from multiple machines is not supported.

Breaking changes

This section discusses the changes that you need to be aware of when migrating your application from one version of Siren Investigate to another.

Migrating from Kibi 4.x

Upgrading Elasticsearch

Kibi is compatible with Elasticsearch 5.6.7; this section provides a summary of the upgrade procedure described at Upgrading Elasticsearch with Kibi and Search Guard specific notes.

It is also recommended to read the Breaking Changes in 5.0 section of the Elasticearch documentation.

Pre-requisites

It is recommended to have a backup of the indices in the cluster; if running a production cluster, you will need to request an updated license for Siren Federate.

Search Guard configuration backup

If your cluster is secured by Search Guard, you will need to retrieve the current cluster configuration through the sgadmin tool; after the upgrade, you’ll need to change these files with new permission identifiers and reload it to the cluster.

sgadmin.sh is available in the plugins/search-guard-5/tools directory on each Elasticsearch instance in which Search Guard has been installed; a standalone version (sgadmin-standalone.zip) can be downloaded from this page.

The current configuration can be retrieved by creating a backup directory and running the tool as follows:

mkdir configbak
bash /path/to/sgadmin.sh -r -ks CN\=sgadmin-keystore.jks -cd configbak -kspass password -ts truststore.jks -tspass password -icl -nhnv -h elasticsearch.local -p 9300

Arguments:

  • r: retrieve configuration.

  • cd: the directory in which the current configuration will be saved.

  • ks: the path to the administrative keystore

  • kspass: the password of the administrative keystore.

  • ts: the path to the truststore.

  • tspass: the password of the truststore.

  • icl: ignore cluster name.

  • nhvv: do not verify cluster hostname.

  • h: Elasticsearch hostname.

  • p: Elasticsearch transport port.

If execution is successful, you will find the five Search Guard configuration files in the configbak directory suffixed by the current date.

Before proceeding, ensure that the files are not empty and remove the date suffix; you should end up with the following files which will be required later.

  • sg_action_groups.yml

  • sg_config.yml

  • sg_internal_users.yml

  • sg_roles_mapping.yml

  • sg_roles.yml

Upgrade preparation

To upgrade from Elasticsearch 2.4.x, you will need to perform a full cluster restart.

Before proceeding, disable shard allocation to avoid unnecessary traffic during the shutdown of each node; this can be done by sending the following request with curl:

curl -XPUT http://elasticsearch.local:9200/_cluster/settings -d '{
  "persistent": {
    "cluster.routing.allocation.enable": "none"
  }
}'

If your cluster is secured by Search Guard, remember to include your credentials in each request, e.g.:

curl -uadmin:password --cacert ca.pem -XPUT https://elasticsearch.local:9200/_cluster/settings -d '{
  "persistent": {
    "cluster.routing.allocation.enable": "none"
  }
}'

For a faster recovery, you can optionally stop any indexing activity and issue a synced flush request as follows:

curl -XPOST http://elasticsearch.local:9200/_flush/synced
Node upgrade

Once the previous steps have been carried out, you will need to stop all the nodes in the cluster and upgrade Elasticsearch and plugins on each node as explained in the following sections.

Do not restart nodes until the software has been upgraded on the whole cluster.

Elasticsearch upgrade

If using Elastic packages, you can upgrade Elasticsearch by installing the compatible rpm or deb package through yum or apt; the packages should preserve the existing configuration files and data directories.

If Elasticsearch was installed from a zip or tar distribution, you will need to:

  • download a compatible version in zip or tar format from https://www.elastic.co/downloads/elasticsearch .

  • extract the package to a new location; make sure to not overwrite the existing installation when extracting the archive.

  • copy all files from the config directory of the previous installation to the config directory of the new installation.

  • check the value of path.data in config/elasticsearch.yml; if not set, you will need to copy the data directory from the old installation to the new one, otherwise Elasticsearch will keep using the data directory specified in path.data.

  • ensure that the new location has the correct filesystem permissions for your environment (usually it is owned by the user running Elasticsearch).

Configuration changes

Any index level setting (settings starting with index) declared in elasticsearch.yml must be removed from the file; these settings have now to be declared in Index templates.

The following settings must be removed if present:

  • index.queries.cache.everything

  • siren.filterjoin.cache.size

Startup scripts

If you were starting Elasticsearch using custom systemd/init.d/supervisor scripts, make sure to check the packaging breaking changes section of the Elasticsearch documentation as several environment variables have now to be set in the jvm.options file.

It is also recommended to verify your custom scripts and system properties against the updated Important System Configuration documentation.

Plugins upgrade

If you installed Elasticsearch from system packages, you will need to remove the old versions of your plugins /usr/share/elasticsearch/plugins; old plugins can be removed by using the elasticsearch-plugin command, e.g.:

/usr/share/elasticsearch/bin/elasticsearch-plugin remove siren-join

If installed, the following plugins must be removed:

  • siren-join

  • license-siren

  • kopf

  • search-guard-2

  • search-guard-ssl

Siren Federate

The Siren Federate plugin for Elasticsearch replaces the Siren Join and Siren License plugins; to install the plugin, you need to download the zip package of the plugin from our customer portal and install it through elasticsearch-plugin, e.g.:

./usr/share/elasticsearch/bin/elasticsearch-plugin install file:///path/to/siren-federate-<version>-plugin.zip
Search Guard 5

If your cluster is protected by Search Guard, you will need to replace the Search Guard 2 and Search Guard SSL plugins with the Search Guard 5 plugin; to find out the version of Search Guard 5 compatible with your Elasticsearch installation, please check the Search Guard version matrix.

The plugin can be installed from Maven Central using elasticsearch-plugin:

./usr/share/elasticsearch/bin/elasticsearch-plugin install -b com.floragunn:search-guard-5:<version>

In addition, the compatible version of any commercial Search Guard add-on will have to be downloaded and copied to the plugins/search-guard-5 directory; the following list provides links to the download page of all the add-ons commonly used in Kibi instances:

Cluster restart without Search Guard

If your cluster is protected by Search Guard, skip to the next section for specific instructions.

Once Elasticsearch and plugins are up to date on all nodes, you can begin restarting nodes; it is recommended to restart master eligible nodes first and wait for the cluster to elect a master before proceeding with other nodes.

master eligibile nodes are nodes having node.master set to true or not set in elasticsearch.yml.

Wait for cluster to reach the Yellow status; if not using Search Guard, you can check the status of the cluster by issuing a requests with curl to _cat/health, e.g.:

$ curl http://elasticsearch.local:9200/_cat/health?v
epoch      timestamp cluster status node.total node.data shards ...
1498542620 06:50:20  541     yellow          5         3    106 ...

Once the cluster is in the Yellow status, you can re-enable shard allocation:

$ curl -XPUT http://elasticsearch.local:9200/_cluster/settings -d '
{
  "persistent": {
    "cluster.routing.allocation.enable": "all"
  }
}
'

Then you can issue additional requests to _cat/health to monitor the cluster until it reaches the Green status; you can also check the recovery status of each shard by issuing requests to _cat/recovery, e.g.:

$ curl http://elasticsearch.local:9200/_cat/recovery
Cluster restart with Search Guard

Once Elasticsearch and plugins are up to date on all nodes, you can begin restarting nodes; it is recommended to restart master eligible nodes first and wait for the cluster to elect a master before proceeding with other nodes.

master eligibile nodes are nodes having node.master set to true or not set in elasticsearch.yml.

You won’t be able to issue HTTP requests until the Search Guard configuration has been upgraded, so you will need to use sgadmin to re-enable shard allocation after the cluster reaches the Yellow status.

sgadmin.sh is available in the plugins/search-guard-5/tools directory on each Elasticsearch instance in which Search Guard has been installed; a standalone version (sgadmin-standalone.zip) can be downloaded from this page.

To re-enable shard allocation with sgadmin execute it as follows:

$ bash /path/to/sgadmin.sh -esa -ks CN=sgadmin-keystore.jks -kspass password -ts config/truststore.jks -tspass password -icl -nhnv -h elasticsearch.local -p 9300
Search Guard Admin v5
Will connect to elasticsearch.local:9330 ... done
Persistent and transient shard allocation enabled

Arguments:

  • esa: enable shard allocation

  • ks: the path to the administrative keystore

  • kspass: the password of the administrative keystore.

  • ts: the path to the truststore.

  • tspass: the password of the truststore.

  • icl: ignore cluster name.

  • nhvv: do not verify cluster hostname.

  • h: Elasticsearch hostname.

  • p: Elasticsearch transport port.

Then you need to reload the Search Guard configuration retrieved before upgrading the cluster by running sgadmin as follows:

$ bash /path/to/sgadmin.sh -ks CN\=sgadmin-keystore.jks -cd configbak -kspass password -ts truststore.jks -tspass password -icl -nhnv -h elasticsearch.local -p 9300
Search Guard Admin v5
Will connect to elasticsearch.local:9330 ... done
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
Clustername: escluster
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
searchguard index already exists, so we do not need to create one.
Will update 'config' with configbak/sg_config.yml
   SUCC: Configuration for 'config' created or updated
Will update 'roles' with configbak/sg_roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update 'rolesmapping' with configbak/sg_roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update 'internalusers' with configbak/sg_internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update 'actiongroups' with configbak/sg_action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Done with success

Arguments:

  • cd: the directory containing old Search Guard configuration backup.

  • ks: the path to the administrative keystore

  • kspass: the password of the administrative keystore.

  • ts: the path to the truststore.

  • tspass: the password of the truststore.

  • icl: ignore cluster name.

  • nhvv: do not verify cluster hostname.

  • h: Elasticsearch hostname.

  • p: Elasticsearch transport port.

You should now be able issue authenticated requests to _cat/health to monitor the cluster until it reaches the Green status; you can also check the recovery status of each shard by issuing requests to _cat/recovery, e.g.:

$ curl -uadmin:password --cacert ca.pem https://elasticsearch.local:9200/_cat/health
$ curl -uadmin:password --cacert ca.pem https://elasticsearch.local:9200/_cat/recovery

Once the cluster has recovered, you will need to modify the Search Guard configuration to adjust Kibi specific permissions as described in the next section.

Search Guard configuration changes

This section describes all the Kibi specific changes required to the Search Guard configuration files; it is advised to backup the files in the retrieved configuration directory before making changes.

Once the files have been modified, the updated configuration can be loaded to the cluster using sgadmin.

sg_action_groups.yml

Add the following action groups:

UNLIMITED:
  - "*"

INDICES_ALL:
  - "indices:*"

Set the ALL action group as follows:

ALL:
  - INDICES_ALL

Modify the CREATE_INDEX action group as follows:

CREATE_INDEX:
  - "indices:admin/create"
  - "indices:admin/mapping/put"

Add the INDICES_MONITOR action group:

MONITOR:
  - INDICES_MONITOR

INDICES_MONITOR:
  - "indices:monitor/*"

Update the DATA_ACCESS, READ, WRITE, DELETE, CRUD and SEARCH groups as follows:

DATA_ACCESS:
  - "indices:data/*"
  - CRUD

WRITE:
  - "indices:data/write*"
  - "indices:admin/mapping/put"

READ:
  - "indices:data/read*"
  - "indices:admin/mappings/fields/get*"

DELETE:
  - "indices:data/write/delete*"

CRUD:
  - READ
  - WRITE

SEARCH:
  - "indices:data/read/search*"
  - "indices:data/read/msearch*"
  - "indices:siren/plan*"
  - "indices:siren/mplan*"
  - SUGGEST

Update the INDEX group as follows:

INDEX:
  - "indices:data/write/index*"
  - "indices:data/write/update*"
  - "indices:admin/mapping/put"
  - "indices:data/write/bulk*"

Add the CLUSTER_COMPOSITE_OPS_RO and CLUSTER_COMPOSITE_OPS roles:

CLUSTER_COMPOSITE_OPS_RO:
  - "indices:data/read/mget"
  - "indices:data/read/msearch"
  - "indices:siren/mplan"
  - "indices:data/read/mtv"
  - "indices:admin/aliases/exists*"
  - "indices:admin/aliases/get*"

CLUSTER_COMPOSITE_OPS:
  - "indices:data/write/bulk*"
  - "indices:admin/aliases*"
  - CLUSTER_COMPOSITE_OPS_RO

Remove the KIBI_MSEARCH role.

Add the KIBI_COMPOSITE role:

KIBI_COMPOSITE:
  - "indices:siren/mplan*"

Replace KIBI_CLUSTER, KIBI_READONLY and KIBI_READWRITE with the following definitions:

KIBI_CLUSTER:
  - "indices:data/read/scroll"
  - "cluster:internal/data/transfer/*"
  - "indices:data/read/msearch*"
  - "indices:siren/mplan*"
  - "cluster:admin/plugin/siren/license/get"
  - CLUSTER_COMPOSITE_OPS_RO

KIBI_READONLY:
  - "indices:data/read/field_stats*"
  - "indices:data/read/field_caps*"
  - "indices:data/read/get*"
  - "indices:data/read/mget*"
  - "indices:data/read/search*"
  - "indices:siren/mplan"
  - "indices:siren/plan"
  - "indices:siren/task/search"
  - "indices:admin/mappings/get*"
  - "indices:admin/mappings/fields/get*"
  - "indices:admin/validate/query*"
  - "indices:admin/get*"
  - "indices:admin/version/get*"
  - KIBI_COMPOSITE

KIBI_READWRITE:
  - "indices:admin/exists*"
  - "indices:admin/mapping/put*"
  - "indices:admin/refresh*"
  - "indices:data/write/delete*"
  - "indices:data/write/index*"
  - "indices:data/write/update*"
  - "indices:data/write/bulk*"
  - KIBI_READONLY

If you defined additional action groups, you will need to verify that the permissions work as expected in Search Guard 5.

sg_roles.yml

kibiserver

Replace the kibiserver role with the following; make sure to set the correct names for the main kibi (.kibi by default) and access control (.kibiaccess by default) indices if using a custom configuration.

# Permissions for the Kibi server process.
kibiserver:
  cluster:
      - cluster:monitor/nodes/info
      - cluster:monitor/health
      - cluster:monitor/main
      - cluster:monitor/state
      - cluster:monitor/nodes/stats
      - KIBI_CLUSTER
      - CLUSTER_COMPOSITE_OPS
  indices:
    '*':
      '*':
        - indices:admin/get
    '?kibi':
      '*':
        - ALL
    '?kibiaccess':
      '*':
        - ALL
sentinl

Sentinl is automatically enabled in Kibi 5; if you do not need alerting it is possible to disable the plugin by putting the following in kibi.yml:

sentinl:
  enabled:
    false

If you were previously using Sentinl, replace the sentinl role with the following; make sure to set the correct names for watcher and watcher_alarms indices if using a custom configuration.

Replace the sentinl role with the following; make sure to set the correct names for watcher and watcher_alarms indices if using a custom configuration.

# Permissions for a Sentinl user.
sentinl:
  cluster:
    - "indices:data/write/bulk*"
    - "indices:admin/template/*"
    - KIBI_CLUSTER
  indices:
    'watcher_alarms*':
      '*':
        - KIBI_READWRITE
        - CREATE_INDEX
    '/(watcher|watcher_alarms)/':
      '*':
        - KIBI_READWRITE
        - CREATE_INDEX

You will also need to grant read permissions to each data index that Sentinl should be able to access when executing watches; for example, to grant access to the index readings and all indices starting with logs, the role would become:

# Permissions for a Sentinl user.
sentinl:
  cluster:
    - "indices:data/write/bulk*"
    - "indices:admin/template/*"
    - KIBI_CLUSTER
  indices:
    'watcher_alarms*':
      '*':
        - KIBI_READWRITE
        - CREATE_INDEX
    '/(watcher|watcher_alarms)/':
      '*':
        - KIBI_READWRITE
        - CREATE_INDEX
    'readings':
      '*':
        - KIBI_READONLY
    'logs*':
      '*':
        - KIBI_READONLY
Kibi user roles

In each Kibi user role:

  • replace KIBI_MSEARCH with KIBI_COMPOSITE.

  • remove the following permission on the Kibi index (.kibi by default) from each user role that has it:

    ?kibi:
      null:
        - "indices:data/read/search"
        - "indices:data/read/coordinate-search"

If you are using the Kibi access control plugin and want to enable ACLs on Kibi saved objects (which restricts access to saved objects), ensure that Kibi users do not have access to the Kibi index (.kibi by default), otherwise they might be able to peek at saved objects by issuing custom Elasticsearch queries.

Access to the .kibi index was usually granted by the following lines in previous Kibi releases; these can be removed when using ACLs as the .kibi index will be accessed only by the kibiserver role:

'?kibi':
  '*':
    - KIBI_READWRITE
'?kibi':
  '*':
    - KIBI_READONLY

If there are users with read access to all indices (*), they will be able to see the Kibi index as well; if you need to hide it, you should replace the wildcard with a list of rules for each index.

Monitoring

To grant a user the permission to use monitoring plugins and retrieve diagnostics information from the UI, you will need to add the CLUSTER_MONITOR and INDICES_MONITOR permissions to its role, e.g.:

# Permissions for a Kibi administrator (read-write access to the .kibi index).
kibiadmin:
  cluster:
    - KIBI_CLUSTER
    - cluster:admin/plugin/siren/license/put
    - CLUSTER_MONITOR
  indices:
    '*':
      '*':
        - KIBI_READONLY
        - INDICES_MONITOR
    '?kibi':
      '*':
        - KIBI_READWRITE
    'watcher':
      '*':
        - KIBI_READWRITE
marvel / X-Pack monitoring

If you were using Marvel and are migrating to X-Pack monitoring, the following role can be used in place of the previous sample marvel role:

# Permissions for an X-Pack monitoring agent.
monitoring:
  cluster:
    - CLUSTER_MONITOR
    - 'indices:admin/aliases'
    - 'indices:admin/template/get'
    - 'indices:admin/template/put'
    - 'cluster:admin/ingest/pipeline/get'
    - 'cluster:admin/ingest/pipeline/put'
    - 'indices:data/write/bulk'
  indices:
    '?marvel*':
      '*':
        - ALL
    '?monitoring*':
      '*':
        - ALL

Loading the modified configuration

To load the modified configuration run sgadmin as follows:

$ bash /path/to/sgadmin.sh -ks CN\=sgadmin-keystore.jks -cd confignew -kspass password -ts truststore.jks -tspass password -icl -nhnv -h elasticsearch.local -p 9300
Search Guard Admin v5
Will connect to elasticsearch.local:9330 ... done
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
Clustername: escluster
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
searchguard index already exists, so we do not need to create one.
Will update 'config' with confignew/sg_config.yml
   SUCC: Configuration for 'config' created or updated
Will update 'roles' with confignew/sg_roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update 'rolesmapping' with confignew/sg_roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update 'internalusers' with confignew/sg_internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update 'actiongroups' with confignew/sg_action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Done with success

Arguments:

  • cd: the directory containing the modified Search Guard configuration.

  • ks: the path to the administrative keystore

  • kspass: the password of the administrative keystore.

  • ts: the path to the truststore.

  • tspass: the password of the truststore.

  • icl: ignore cluster name.

  • nhvv: do not verify cluster hostname.

  • h: Elasticsearch hostname.

  • p: Elasticsearch transport port.

License

You will need to upload your license once the cluster is up and running.

In Kibi

To upload the license in Kibi, navigate to the management section.

management no license

This page will show the license has not been installed.

Click on License and select Upload Licence

license page before install

Once you have selected your license and it has been uploaded, the page will show the license details.

license page after install

Via CLI

To upload the license you can use curl as follows:

curl http://localhost:9220/_siren/license -XPUT -T license.sig

If your cluster is protected by Search Guard, remember to specify credentials:

curl -uadmin:password --cacert=ca.pem https://localhost:9220/_siren/license -XPUT -T license.sig

Kibi

Before upgrading Kibi, make sure to perform the following steps:

  • backup the .kibi index.

  • backup the Kibi configuration file (config/kibi.yml)

Then perform the following steps:

  • extract the new Kibi version to a new directory.

  • copy the previous configuration file to the config directory of the new installation.

  • copy the files from the data directory in your old installation of the new installation.

  • copy the files from the pki directory in your old installation of the new installation.

  • update the configuration as described in the next sections.

  • reinstall third party plugins after checking the notes in the next sections.

  • execute bin/kibi upgrade.

  • start the new installation.

Configuration changes

Server SSL support

If SSL support was enabled in the previous Kibi installation, you will need to update kibi.yml as follows:

  • add server.ssl.enabled: true

  • replace server.ssl.cert with server.ssl.certificate

  • replace ssl.verify with ssl.verificationMode; ssl.verificationMode can be set to full, to check both certificate and hostname, certificate to check only the certificate, none to disable verification. If not set, the verification mode defaults to full.

Search Guard client CA

If your cluster is protected by Search Guard, you will need to rename elasticsearch.ssl.ca to elasticsearch.ssl.certificateAuthorities.

Kibi access control ACL

If you want to enable Kibi Access Control ACL, you will need to:

  • add acl.enabled: true to the kibi_access_control section;

  • specify the Search Guard role that will have administrative access to the ACL configuration in admin_role (.kibiadmin by default);

  • specify the name of the index where access control rules will be stored (.kibiaccess by default).

E.g.:

kibi_access_control:
  acl:
    enabled: true
    index: kibiaccess
  admin_role: kibiadmin

Once ACLs are enabled, you will need to login with a user having the role specified in admin_role and visit Access Control → ACL to create Kibi roles with the desired permissions.

For more informations about Kibi Access Control ACL, please check the Search Guard Integration and Siren Investigate access control chapter.

Sense

sense settings must now be declared in the console section, e.g.:

console.proxyConfig:
  - match:
      protocol: "https"

    ssl:
      ca: "pki/searchguard/ca.pem"

Third party plugins

The following plugins have been discontinued:

  • kibi_wordcloud: Kibi now supports the Tag Cloud visualization introduced in Kibana 5. Existing Kibi Word Cloud visualizations will be migrated to Tag Cloud visualizations by the upgrade procedure.

  • heatmap: the heatmap plugin for Kibana 4.x has been integrated into Kibana 5 and thus is available in Kibi. Existing Heatmap visualizations will be upgraded by a Kibi migration; the color palette will be set to Greens as most of the palettes in the original plugin were not ported to Kibana 5.

Handling of text fields in upgraded indices

Fields which were declared in mappings with type set to string and index to analyzed (which was the default before Elasticsearch 5.0) will be handled as fields of type text.

In order to be usable for aggregation and sorting operations, fields of type text need a in-memory structure called fielddata, which is disabled by default to reduce memory usage.

If you have visualizations using text fields in aggregations (for example tag clouds), you will need to enable fielddata in their mapping; for example, the following command sets fielddata to true on the snippet field of an article type in an index called article:

curl -XPUT http://elasticsearch:9200/article/_mapping/article - d'
{
  "properties": {
    "snippet": {
      "type": "text",
      "fielddata": true
    }
  }
}
'

If the existing mapping defines custom parameters (e.g. analyzer) you will get a conflicting mapping error; in that case, you need to include the custom parameters in the updated mapping definition, e.g.:

curl -XPUT http://elasticsearch.local:9200/article/_mapping/article - d'
{
  "properties": {
    "snippet": {
      "type": "text",
      "analyzer": "english",
      "fielddata": true
    }
  }
}
'

After enabling fielddata on an existing text field, you’ll have to refresh all the index patterns containing the field by going to Management/Index Patterns in Kibi, opening each pattern and clicking on the refresh button; once the index pattern is refreshed, the field should appear as aggregatable.

Refreshing an index pattern
Additional documentation

For more information please refer to the following links:

Breaking changes in Kibi and Kibana 5.4.0

This section list the changes in Kibi and Kibana that you need to be aware of when migrating to Kibi 5.4.0.

Kibana binds to localhost by default

{repo}pull/8013[Pull Request 8013]

Details: Kibana (like Elasticsearch) now binds to localhost for security purposes instead of 0.0.0.0 (all addresses). Previous binding to 0.0.0.0 also caused issues for Windows users.

Impact: If you are running Kibana inside a container/environment that does not allow localhost binding, this will cause Kibana not to start up unless server.host is configured in the kibana.yml to a valid IP address/host, etc..

Markdown headers

{repo}pull/7855[Pull Request 7855]

Details: As part of addressing the security issue ESA-2016-03 (CVE-2016-1000220) in the Kibana product, the markdown version has been bumped.

Impact: As a result of the fix to ESA-2016-03, there is a slight change in the markdown format for headers.

Previously, headers are defined using # followed by the title:

###Packetbeat:
   [Dashboard](/#/dashboard/Packetbeat-Dashboard)
   [Web transactions](/#/dashboard/HTTP)

It should now be defined as follows (with a space between # and the title):

### Packetbeat:
    [Dashboard](/#/dashboard/Packetbeat-Dashboard)
    [Web transactions](/#/dashboard/HTTP)

Only whitelisted client headers are sent to Elasticsearch

{repo}pull/6896[Pull Request 6896]

Details: The only headers that are proxied from the browser client to Elasticsearch are the ones set via the elasticsearch.requestHeadersWhitelist server configuration.

Impact: If you’re relying on client headers in Elasticsearch, you will need to whitelist the specific headers in your kibana.yml.

server.defaultRoute is now always prefixed by server.basePath

{repo}pull/6953[Pull Request 6953]

Details: The base path configuration now precedes the default route configuration when accessing the default route.

Impact: If you were relying on both defaultRoute and basePath configurations, you will need to remove the hardcoded basePath from your defaultRoute.

Directory listings of static assets are no longer rendered

{repo}pull/6764[Pull Request 6764]

Details: The server no longer renders a list of static files if you try to access a directory.

Impact: If you were relying on this behavior before, you will need to expose underlying directory listings via a reverse proxy instead.

Console logs display date/time in UTC

{repo}pull/8534[Pull Request 8534]

Details: All server logs now render in UTC rather than the server’s local time.

Impact: If you are parsing the timestamps of Kibana server logs in an automated way, make sure to update your automation to accomodate UTC values.

A column for Average no longer renders along with Standard Deviation

{repo}pull/7827[Pull Request 7827]

Details: From the early days of Kibana, adding a standard deviation metric to a data table also resulted in an average column being added to that data table. This is no longer the case.

Impact: If you want to have both standard deviation and average in the same data table, then add both columns just as you would any other metric.

Minimum size on terms aggregations has been changed from 0 to 1

{repo}pull/8339[Pull Request 8339]

Details: Elasticsearch has removed the ability to specify a size of 0 for terms aggregations, so Kibana’s minimum value has been adjusted to follow suit.

Impact: Any saved visualization that relies on size=0 will need to be updated.

Saved objects with previously deprecated Elasticsearch features

Details: Since Kibana 4.3, users have been able to arbitrarily modify filters via a generic JSON editor. If users took advantage of any deprecated Elasticsearch features in this way, then they will cause errors in Kibana since they’re removed from Elasticsearch 5.0. Check the Elasticsearch breaking changes documentation for more details.

Impact: Discover, Visualize, and Dashboard will error for any saved objects that are relying on removed Elasticsearch functionality. Users will need to update the JSON of any affected filters.

Preparing Kibana index for 6.0

The following will make transformations to your Kibana index. It is recommended you backup your Kibana index prior to running any of the following commands.

In order to migrate to Kibana 6, your Kibana index needs to be reindexed. This will make transformations to the data model, removing types which are being {ref}/removal-of-types.html[removed in Elasticsearch].

A UI is available in X-Pack 5.6 to assist with the migration. The following is provided should you be required to run the migration manually. The commands assume you’re using the default .kibana index, without an alias. If kibana.index is set to something different in your kibana.yml, or you’re using an alias, you will need to modify the commands.

To perform this migration without downtime, you will need to be running Kibana 5.6. This is the only version which supports both of the index types.
  1. Set .kibana index to read-only:

    PUT .kibana/_settings
    {
      "index.blocks.write": true
    }
  2. Create .kibana-6 index

When running versions other than 5.6, you will need to remove index.format and index.mapping.single_type from the settings in the following step.
PUT .kibana-6
{
  "settings" : {
    "number_of_shards" : 1,
    "index.mapper.dynamic": false,
    "index.format": 6,
    "index.mapping.single_type": true
  },
  "mappings" : {
    "doc": {
      "properties": {
        "type": {
          "type": "keyword"
        },
        "updated_at": {
          "type": "date"
        },
        "config": {
          "properties": {
            "buildNum": {
              "type": "keyword"
            }
          }
        },
        "index-pattern": {
          "properties": {
            "fieldFormatMap": {
              "type": "text"
            },
            "fields": {
              "type": "text"
            },
            "intervalName": {
              "type": "keyword"
            },
            "notExpandable": {
              "type": "boolean"
            },
            "sourceFilters": {
              "type": "text"
            },
            "timeFieldName": {
              "type": "keyword"
            },
            "title": {
              "type": "text"
            }
          }
        },
        "visualization": {
          "properties": {
            "description": {
              "type": "text"
            },
            "kibanaSavedObjectMeta": {
              "properties": {
                "searchSourceJSON": {
                  "type": "text"
                }
              }
            },
            "savedSearchId": {
              "type": "keyword"
            },
            "title": {
              "type": "text"
            },
            "uiStateJSON": {
              "type": "text"
            },
            "version": {
              "type": "integer"
            },
            "visState": {
              "type": "text"
            }
          }
        },
        "search": {
          "properties": {
            "columns": {
              "type": "keyword"
            },
            "description": {
              "type": "text"
            },
            "hits": {
              "type": "integer"
            },
            "kibanaSavedObjectMeta": {
              "properties": {
                "searchSourceJSON": {
                  "type": "text"
                }
              }
            },
            "sort": {
              "type": "keyword"
            },
            "title": {
              "type": "text"
            },
            "version": {
              "type": "integer"
            }
          }
        },
        "dashboard": {
          "properties": {
            "description": {
              "type": "text"
            },
            "hits": {
              "type": "integer"
            },
            "kibanaSavedObjectMeta": {
              "properties": {
                "searchSourceJSON": {
                  "type": "text"
                }
              }
            },
            "optionsJSON": {
              "type": "text"
            },
            "panelsJSON": {
              "type": "text"
            },
            "refreshInterval": {
              "properties": {
                "display": {
                  "type": "keyword"
                },
                "pause": {
                  "type": "boolean"
                },
                "section": {
                  "type": "integer"
                },
                "value": {
                  "type": "integer"
                }
              }
            },
            "timeFrom": {
              "type": "keyword"
            },
            "timeRestore": {
              "type": "boolean"
            },
            "timeTo": {
              "type": "keyword"
            },
            "title": {
              "type": "text"
            },
            "uiStateJSON": {
              "type": "text"
            },
            "version": {
              "type": "integer"
            }
          }
        },
        "url": {
          "properties": {
            "accessCount": {
              "type": "long"
            },
            "accessDate": {
              "type": "date"
            },
            "createDate": {
              "type": "date"
            },
            "url": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 2048
                }
              }
            }
          }
        },
        "server": {
          "properties": {
            "uuid": {
              "type": "keyword"
            }
          }
        },
        "timelion-sheet": {
          "properties": {
            "description": {
              "type": "text"
            },
            "hits": {
              "type": "integer"
            },
            "kibanaSavedObjectMeta": {
              "properties": {
                "searchSourceJSON": {
                  "type": "text"
                }
              }
            },
            "timelion_chart_height": {
              "type": "integer"
            },
            "timelion_columns": {
              "type": "integer"
            },
            "timelion_interval": {
              "type": "keyword"
            },
            "timelion_other_interval": {
              "type": "keyword"
            },
            "timelion_rows": {
              "type": "integer"
            },
            "timelion_sheet": {
              "type": "text"
            },
            "title": {
              "type": "text"
            },
            "version": {
              "type": "integer"
            }
          }
        },
        "graph-workspace": {
          "properties": {
            "description": {
              "type": "text"
            },
            "kibanaSavedObjectMeta": {
              "properties": {
                "searchSourceJSON": {
                  "type": "text"
                }
              }
            },
            "numLinks": {
              "type": "integer"
            },
            "numVertices": {
              "type": "integer"
            },
            "title": {
              "type": "text"
            },
            "version": {
              "type": "integer"
            },
            "wsState": {
              "type": "text"
            }
          }
        }
      }
    }
  }
}
  1. Reindex .kibana into .kibana-6:

    POST _reindex
    {
      "source": {
        "index": ".kibana"
      },
      "dest": {
        "index": ".kibana-6"
      },
      "script": {
        "inline": "ctx._source = [ ctx._type : ctx._source ]; ctx._source.type = ctx._type; ctx._id = ctx._type + \":\" + ctx._id; ctx._type = \"doc\"; ",
        "lang": "painless"
      }
    }
  2. Alias .kibana-6 to .kibana and remove legacy .kibana index:

    POST /_aliases
    {
      "actions" : [
        { "add":  { "index": ".kibana-6", "alias": ".kibana" } },
        { "remove_index": { "index": ".kibana" } }
      ]
    }

Getting Started

The demo distribution

The quickest way to get started with Siren Investigate is to download the Siren Platform demo distribution from the downloads page.

The Siren Platform demo distribution includes a ready to use Elasticsearch cluster in the elasticsearch directory; the cluster contains five indices:

company

a collection of companies

article

a collection of articles about companies

investment

a collection of investments in companies

investor

a collection of investors

.siren

Siren Investigate configuration

The indices have been populated through Logstash from the SQLite database in siren-investigate/crunchbase.db, as described in the Loading Data into Elasticsearch chapter.

The demo dataset has been built from a sample of articles gathered from tech blogs in 2013 and from data about companies, investments and investors in the CrunchBase 2013 Snapshot, which is copyright © 2013 AOL Inc.

Users

You can access the demo as three different users:

  • sirenadmin: has read access to all data indices and can modify the Siren Investigate configuration.

  • sirenuser: has read access to all data indices but cannot modify the Siren Investigate configuration.

  • sirennoinvestor: has read access to all data indices except investor; the user has no access to the Investors dashboard.

The password for all users is password.

After starting Elasticsearch and Siren Investigate, as described in the Setup chapter, start your web browser and navigate to http://localhost:5606.

By default Siren Investigate displays the Articles dashboard, dashboards can be configured to display multiple visualizations on the documents stored in a specific index or returned by a saved search on an index pattern.

Each dashboard is represented by a tab containing the dashboard title and the number of documents available to visualizations.

The articles dashboard

You can quickly search specific articles through the search input in the navigation bar. For example, let’s find all the articles written about wireless or wifi:

The dashboard search bar

We can immediately see that there are 10370 articles about those topics and all the visualizations are refreshed to aggregate data for this subset of articles.

Besides simple text, queries in the search bar can be written using the Lucene query syntax, or Elasticsearch Query DSL.

Filters

Visualizations can be used to create filters. For example, you can see all the articles about wireless or wifi published by TechCrunch by clicking on the TechCrunch slice inside the Articles by Source pie chart visualization:

Clicking on a pie slice
The dashboard with the filter on the slice applied
To disable a filter, just move the mouse over it and click on the checkbox icon; you can disable all the filters applied to the dashboard by clicking on Actions and then Disable; for more information about filtering, please read the filters chapter.

Relational Filters

The Relational Filter visualization allows to create cross-dashboard filters. For example, by looking below at the Companies button in the dashboard, you can see that there are 342 companies mentioned in the TechCrunch articles about wireless or wifi.

Relational filter from Articles to Companies

The relational filter created by clicking on the button is displayed in the filter bar, and can be disabled or deleted just like any other filter. Moving the mouse over the filter will display the list of joined indices and their filters:

Relational filter in the filter bar

Relational filter can be accumulated. For example, if you click on the Investment rounds -→ button, you will see data about the 298 investment rounds related to a subset of 342 companies mentioned in the TechCrunch articles about wireless or wifi.

Now Click on Companies to go back to the Companies dashboard. To understand how to define a relational filter, first click on Edit on the top right menu. Then click on the pencil icon inside the Relational widget visualization heading - this will open the configuration editor:

Relational filter configuration

As you can see, it is possible to set two different values for the label displayed on the button and for the label displayed in the filter; it is also possible to use a single configuration for all the dashboards, as the visualization will display only buttons relevant to the currently displayed dashboard.

Click on the Dashboard tab to go back to the Companies dashboard.

for more informations about relational filters, please read the Relational filter chapter.

Query Based Aggregations

It is possible to get additional information about companies by using the results of queries on SQL databases (or any of the datasources supported by Siren Investigate) as aggregations on Elasticsearch documents.

For example, in the Query on Companies visualization you can see that 41 of the 96 companies have competitors and 8 of them are in the top 500 companies by number of employees:

SQL based aggregations

Companies "With competitors" and Top 500 companies (HR count) are queries on the SQLite database. The records returned by the queries are used to filter Elasticsearch documents, which can be then aggregated in a metric.

To better understand this feature, let’s have a look at the Top 500 companies (HR count) query. To see the query, click on Saved Objects in the Management menu.

The query editor

The query returns the id, label and number_of_employees columns from the company table for the top 500 companies by number of employees:

select id, label, number_of_employees
from company
where number_of_employees>0
order by number_of_employees desc
limit 500

Click on Dashboard, then Edit, and then on the pencil icon in the heading of the Query on Companies visualization to customize its configuration. The metrics section defines the aggregations on Elasticsearch documents, displayed as columns in the table. The buckets section defines the groups of Elasticsearch documents aggregated by metrics, displayed as row headers in the table.

Query on Companies configuration

By expanding the Split Rows section inside buckets you can see how the queries are used to define groups of Elasticsearch documents. Scroll down to see the configuration of the fourth filter:

Configuration of an external query terms filter

The filter is configured to execute the query Top 500 companies (HR count) on the SQLite database and return the group of Elasticsearch documents from the current search whose id is equal to one of the id’s in the query results. The documents are then processed by the Count metric.

Let’s add a new aggregation to show the average number of employees. Click on Add metrics inside the metrics section, then select Metric as the metric type; select Average as the aggregation and number_of_employees as the field, then click on the apply changes button apply-changes-button.

Save the visualization by clicking on the Save button, then click on the Dashboard tab to see the updated visualization in the Companies dashboard:

Average aggregation

Click Add sub-buckets at the bottom, then select Split Rows as the bucket type. Choose the Terms aggregation and the countrycode field from the drop-downs. Click the apply changes button apply-changes-button to add an external ring with the new results.

Countrycode aggregation
read the Create A Visualization chapter for an in-depth explanation of aggregations.

In addition to defining groups to aggregate, queries can be used as filters. To see this click on Dashboard, then in the 'Query on Companies' dashboard tile, hover the mouse over the row for Top-500-companies-(HR-count) and click the + icon which appears.

Filter dashboard using a SQL query

Then you will see only the companies mentioned in the articles which are also in the top 500 by number of employees:

Filter dashboard using a SQL query result

Datasource Entity Selection

It is possible to select a company entity (record) in the SQLite database ( and entities in external datasources in general) by clicking on its label in the Companies Table.

The selected entity can be used as a parameter in queries; for example, click on Baidu in Companies Table:

Entity selection

Selecting an entity enables additional queries on external datasources. For example, in the Query on Companies visualization you can see that, amongst the top 500 companies by number of employees mentioned in articles about wireless or wifi, Baidu has one competitor and there are five companies in the same domain. All widgets affected by the selected entity are marked by a purple header.

For the Y-axis metrics aggregation, select Unique Count, with speaker as the field. For Shakespeare plays, it might be useful to know which plays have the lowest number of distinct speaking parts, if your theater company is short on actors. For the X-Axis buckets, select the Terms aggregation with the play_name field. For the Order, select Ascending, leaving the Size at 5.

Leave the other elements at their default values and click the Apply Changes button apply-changes-button. Your chart should now look like this:

Selecting an entity also enables the display of additional data in the Company Info visualization; by clicking on the (show) links you can toggle the list of companies in the same domain and competitors. The data in the tables is fetched from queries on the SQLite database, using the selected company ID as a parameter. The queries are rendered using customizable templates, which will be introduced later.

The selected entity appears as a purple box on the right of the filter bar; to deselect an entity, click on the bin icon displayed when moving the mouse over the purple box.

for additional documentation about entity selection, please read the Datasource entity selection section in the External Datasources chapter.

Enhanced Search Results

The Enhanced search results visualization displays the current set of Elasticsearch documents as a table. For example, Companies Table is configured to display the following fields:

  • Time (foundation date)

  • label (the company name)

  • description

  • category_code

  • founded_year

  • countrycode

  • Why Relevant? (a relational column)

Companies table

By selecting Edit and then clicking on the pencil icon, you are brought to a view where you can choose which fields to display and customize the order of the columns. If the index is time based, the Time column will be always displayed.

Expand the first row by clicking on the right arrow, then scroll down to the homepage_url field and click on the Toggle column icon:

Column positioning

You can click on the arrows to move the column to the desired position:

Column positioning

Click Handlers

You can define click handlers on cells to perform several actions. Let’s add a click handler to open the company homepage when clicking on the cell displaying the URL.

The table is pre-configured with a click handler on label that is used to select an entity in the SQLite database.

To add a new click handler, go into edit mode, scroll down view options and click on Add click handler; select homepage_url in the Column dropdown, then Follow the URL in the On click I want to dropdown. Select homepage_url as the URL field, then click on the click the Apply Changes button apply-changes-button to apply changes.

You can test the click handler immediately by clicking on a cell displaying an homepage URL in the preview displayed on the right:

URL click handler

Relational Column

You can enable the relational column to be displayed when an Elasticsearch document is matched by a query on the SQLite database. The relational column reports on the relationship, based on the queries configured.

In the example below, in the Companies Table, you can see that Big Fish is listed here because it has competitors.

Relational column example
Relational column configuration

Saving the Visualization

Click on the save button in the top right to save the visualization, then click on Dashboard to go back to the Companies dashboard.

for additional documentation about this visualization, please read the Enhanced search results chapter.

Query Templates

Company Info which is an instance of a Siren Investigate query viewer visualization, displays the results of three SQL queries by rendering their results through templates; the queries take the selected entity ID as an input, thus the associated templates will be displayed only when an entity is selected.

Siren Investigate query viewer example

The association between the query and templates can be set in the visualization configuration:

Siren Investigate query viewer configuration

Query templates can be managed by clicking on the Management icon, then select Advanced Settings followed by Templates.

you can find the documentation about templates in the External Datasources chapter; the visualization is documented in the Siren Investigate query viewer chapter.

The Relational Filter visualization allows you to create cross-dashboard filters. In the example above you can see that there are 438 companies mentioned in the TechCrunch articles about wireless or wifi. By clicking on the button, you can switch to the Companies dashboard and visualize the data about these companies: Relational filters can be accumulated. If you click on the Investment rounds -→ button, you will see the data about those investment rounds related to the companies mentioned in the TechCrunch articles

Discover

You can interactively explore your data from the Discover page. You have access to every document in every index that matches the selected index pattern. You can submit search queries, filter the search results, and view document data. You can also see the number of documents that match the search query and get field value statistics. If a time field is configured for the selected index pattern, the distribution of documents over time is displayed in a histogram at the top of the page.

Discover

Setting the Time Filter

The time filter restricts the search results to a specific time period. You can set a time filter if your index contains time-based events and a time-field is configured for the selected index pattern.

By default the time filter is set to the last 15 minutes. You can use the Time Picker to change the time filter or select a specific time interval or time range in the histogram at the top of the page.

To set a time filter with the Time Picker:

  1. Click Time Picker time-picker in the Siren Investigate toolbar.

  2. To set a quick filter, click one of the shortcut links.

    Time filter shortcuts

    Click Select All in the Apply to Dashboards section to apply time filter to all dashboards or select individual dashboards. The time filter is applied to the current dashboard by default.

  3. To specify a time filter relative to the current time, click Relative and specify the start time as a number of seconds, minutes, hours, days, months, or years. You can also specify the end time relative to the current time. Relative times can be in the past or future.

    Relative time filter
  4. To specify both the start and end times for the time filter, click Absolute and select a start and end date. You can adjust the time by editing the To and From fields.

    Absolute time filter
  5. Click the caret in the bottom right corner to close the Time Picker.

To set a time filter from the histogram, do one of the following:

  • Click the bar that represents the time interval you want to zoom in on.

  • Click and drag to view a specific timespan. You must start the selection with the cursor over the background of the chart—​the cursor changes to a plus sign when you hover over a valid start point.

To move forward/backward in time, click the arrows to the left or right of the Time Picker:

Move backwards in time

You can use the browser Back button to undo your changes.

The displayed time range and interval are shown on the histogram. By default, the interval is set automatically based on the time range. To use a different interval, click the link and select an interval.

You can search the indices that match the current index pattern by entering your search criteria in the Query bar. You can perform a simple text search, use the Lucene query syntax, or use the full JSON-based {ref}/query-dsl.html[Elasticsearch Query DSL].

When you submit a search request, the histogram, Documents table, and Fields list are updated to reflect the search results. The total number of hits (matching documents) is shown in the toolbar. The Documents table shows the first five hundred hits. By default, the hits are listed in reverse chronological order, with the newest documents shown first. You can reverse the sort order by clicking the Time column header. You can also sort the table by the values in any indexed field. For more information, see Sorting the Documents Table.

To search your data, enter your search criteria in the Query bar and press Enter or click Search search-button to submit the request to Elasticsearch.

  • To perform a free text search, simply enter a text string. For example, if you’re searching web server logs, you could enter safari to search all fields for the term safari.

  • To search for a value in a specific field, prefix the value with the name of the field. For example, you could enter status:200 to find all of the entries that contain the value 200 in the status field.

  • To search for a range of values, you can use the bracketed range syntax, [START_VALUE TO END_VALUE]. For example, to find entries that have 4xx status codes, you could enter status:[400 TO 499].

  • To specify more complex search criteria, you can use the Boolean operators AND, OR, and NOT. For example, to find entries that have 4xx status codes and have an extension of php or html, you could enter status:[400 TO 499] AND (extension:php OR extension:html).

These examples use the Lucene query syntax. You can also submit queries using the Elasticsearch Query DSL. For examples, see {ref}/query-dsl-query-string-query.html#query-string-syntax[query string syntax] in the Elasticsearch Reference.

Saving searches enables you to reload them into Discover and use them as the basis for visualizations. Saving a search saves both the search query string and the currently selected index pattern.

To save the current search:

  1. Click Save in the Siren Investigate toolbar.

  2. Enter a name for the search and click Save.

You can import, export and delete saved searches from management/Siren/Saved Objects.

To load a saved search into Discover:

  1. Click Open in the Siren Investigate toolbar.

  2. Select the search you want to open.

If the saved search is associated with a different index pattern than is currently selected, opening the saved search also changes the selected index pattern.

Changing Which Indices You’re Searching

When you submit a search request, the indices that match the currently-selected index pattern are searched. The current index pattern is shown below the toolbar. To change which indices you are searching, click the index pattern and select a different index pattern.

For more information about index patterns, see Creating an Index Pattern.

Refreshing the Search Results

As more documents are added to the indices you’re searching, the search results shown in Discover and used to display visualizations get stale. You can configure a refresh interval to periodically resubmit your searches to retrieve the latest results.

To enable auto refresh:

  1. Click the Time Picker Time Picker in the Siren Investigate toolbar.

  2. Click Auto refresh.

  3. Choose a refresh interval from the list.

    autorefresh intervals

When auto refresh is enabled, the refresh interval is displayed next to the Time Picker, along with a Pause button. To temporarily disable auto refresh, click Pause.

If auto refresh is not enabled, you can manually refresh visualizations by clicking Refresh.

Filtering by Field

You can filter the search results to display only those documents that contain a particular value in a field. You can also create negative filters that exclude documents that contain the specified field value.

You add field filters from the Fields list, the Documents table, or by manually adding a filter. In addition to creating positive and negative filters, the Documents table enables you to filter on whether or not a field is present. The applied filters are shown below the Query bar. Negative filters are shown in red.

To add a filter from the Fields list:

  1. Click the name of the field you want to filter on. This displays the top five values for that field.

    filter field
  2. To add a positive filter, click the Positive Filter button Positive Filter. This includes only those documents that contain that value in the field.

  3. To add a negative filter, click the Negative Filter button Negative Filter. This excludes documents that contain that value in the field.

To add a filter from the Documents table:

  1. Expand a document in the Documents table by clicking the Expand button Expand Button to the left of the document’s table entry.

    Expanded Document
  2. To add a positive filter, click the Positive Filter button Positive Filter Button to the right of the field name. This includes only those documents that contain that value in the field.

  3. To add a negative filter, click the Negative Filter button Negative Filter Button to the right of the field name. This excludes documents that contain that value in the field.

  4. To filter on whether or not documents contain the field, click the Exists button Exists Button to the right of the field name. This includes only those documents that contain the field.

To manually add a filter:

  1. Click Add Filter. A popup will be displayed for you to create the filter.

    add filter
  2. Choose a field to filter by. This list of fields will include fields from the index pattern you are currently querying against.

    add filter field
  3. Choose an operation for your filter.

    add filter operator

    The following operators can be selected:

    is

    Filter where the value for the field matches the given value.

    is not

    Filter where the value for the field does not match the given value.

    is one of

    Filter where the value for the field matches one of the specified values.

    is not one of

    Filter where the value for the field does not match any of the specified values.

    is between

    Filter where the value for the field is in the given range.

    is not between

    Filter where the value for the field is not in the given range.

    exists

    Filter where any value is present for the field.

    does not exist

    Filter where no value is present for the field.

  4. Choose the value(s) for your filter.

    add filter value
  5. (Optional) Specify a label for the filter. If you specify a label, it will be displayed below the query bar instead of the filter definition.

  6. Click Save. The filter will be applied to your search and be displayed below the query bar.

To make the filter editor more user-friendly, you can enable the filterEditor:suggestValues advanced setting. Enabling this will cause the editor to suggest values from your indices if you are filtering against an aggregatable field. However, this is not recommended for extremely large datasets, as it can result in long queries.

Managing Filters

To modify a filter, hover over it and click one of the action buttons.

filter allbuttons 5

 

filter-enable Enable Filter

Disable the filter without removing it. Click again to reenable the filter. Diagonal stripes indicate that a filter is disabled.

filter-pin Pin Filter

Pin the filter. Pinned filters persist when you switch contexts in Siren Investigate. For example, you can pin a filter in Discover and it remains in place when you switch to Visualize. Note that a filter is based on a particular index field—​if the indices being searched don’t contain the field in a pinned filter, it has no effect.

filter-toggle Invert Filter

Switch from a positive filter to a negative filter and vice-versa.

filter-delete Remove Filter

Remove the filter.

filter-custom Edit Filter

Edit the filter definition. Enables you to manually update the filter and specify a label for the filter.

To apply a filter action to all of the applied filters, click Actions and select the action.

Editing a Filter

You can edit a filter by changing the field, operator, or value associated with the filter (see the Add Filter section above), or by directly modifying the filter query that is performed to filter your search results. This enables you to create more complex filters that are based on multiple fields.

  1. To edit the filter query, first click the edit button for the filter, then click Edit Query DSL.

    edit filter query
  2. You can then edit the query for the filter.

    edit filter query json

For example, you could use a {ref}/query-dsl-bool-query.html[bool query] to create a filter for the sample log data that displays the hits that originated from Canada or China that resulted in a 404 error:

{
  "bool": {
    "should": [
      {
        "term": {
          "geoip.country_name.raw": "Canada"
        }
      },
      {
        "term": {
          "geoip.country_name.raw": "China"
        }
      }
    ],
    "must": [
      {
        "term": {
          "response": "404"
        }
      }
    ]
  }
}

Viewing Document Data

When you submit a search query, the 500 most recent documents that match the query are listed in the Documents table. You can configure the number of documents shown in the table by setting the discover:sampleSize property in Advanced Settings. By default, the table shows the localized version of the time field configured for the selected index pattern and the document _source. You can add fields to the Documents table from the Fields list. You can sort the listed documents by any indexed field that’s included in the table.

To view a document’s field data, click the Expand button Expand Button to the left of the document’s table entry.

Expanded Document

To view the original JSON document (pretty-printed), click the JSON tab.

To view the document data as a separate page, click the View single document link. You can bookmark and share this link to provide direct access to a particular document.

To display or hide a field’s column in the Documents table, click the Add Column Toggle column in table button.

To collapse the document details, click the Collapse button Collapse Button.

Sorting the Document List

You can sort the documents in the Documents table by the values in any indexed field. If a time field is configured for the current index pattern, the documents are sorted in reverse chronological order by default.

To change the sort order, hover over the name of the field you want to sort by and click the sort button. Click again to reverse the sort order.

Adding Field Columns to the Documents Table

By default, the Documents table shows the localized version of the time field that’s configured for the selected index pattern and the document _source. You can add fields to the table from the Fields list or from a document’s field data.

To add a field column from the Fields list, hover over the field and click its add button.

To add a field column from a document’s field data, expand the document and click the field’s Add Column Toggle column in table button.

Added field columns replace the _source column in the Documents table. The added fields are also added to the Selected Fields list.

To rearrange the field columns, hover over the header of the column you want to move and click the Move left or Move right button.

Move Column

Removing Field Columns from the Documents Table

To remove a field column from the Documents table, hover over the header of the column you want to remove and click the Remove button Remove Field Button.

Viewing Document Context

For certain applications it can be useful to inspect a window of documents surrounding a specific event. The context view enables you to do just that for index patterns that are configured to contain time-based events.

To show the context surrounding an anchor document, click the Expand button Expand Button to the left of the document’s table entry and then click the View surrounding documents link.

Expanded Document

 

The context view displays a number of documents before and after the anchor document. The anchor document itself is highlighted in blue. The view is sorted by the time field specified in the index pattern configuration and uses the same set of columns as the Discover view the context was opened from. If there are multiple documents with the same time field value, the internal document order is used as a secondary sorting criterion by default.

The field used for tiebreaking in case of equal time field values can be configured using the advanced setting context:tieBreakerFields in Management > Advanced Settings, which defaults to the _doc field. The value of this setting can be a comma-separated list of field names, which will be checked in sequence for suitability when a context is about to be displayed. The first suitable field is then used as the tiebreaking field. A field is suitable if the field exists and is sortable in the index pattern the context is based on.

While not required, it is recommended to only use fields which have {ref}/doc-values.html[doc values] enabled to achieve good performance and avoid unnecessary {ref}/modules-fielddata.html[field data] usage. Common examples for suitable fields include log line numbers, monotonically increasing counters and high-precision timestamps.

Context View
The number of documents displayed by default can be configured via the context:defaultSize setting in Management > Advanced Settings.

Changing the Context Size

You can change the number documents displayed before and after the anchor document independently.

To increase the number of displayed documents that are newer than the anchor document, click the Load 5 more button above the document list or enter the desired number into the input box right of the button.

Discover ContextView SizePicker Newer

 

To increase the number of displayed documents that are older than the anchor document, click the Load 5 more button below the document list or enter the desired number into the input box right of the button.

Discover ContextView SizePicker Older

 

The default number of documents loaded with each button click can be configured via the context:step setting in Management > Advanced Settings.

Filtering the Context

Depending on how the documents are partitioned into index patterns, the context view might contain a large number of documents not related to the event under investigation. In order to adapt the focus of the context view to the task at hand, you can use filters to restrict the documents considered by Kibana for display in the context view.

When switching from the discover view to the context view, the previously applied filters are carried over. Pinned filters remain active while normal filters are copied in a disabled state. You can selectively re-enabled them to refine your context view.

New filters can be added via the Add a filter link in the filter bar, by clicking the filter icons appearing when hovering a field, or by expanding documents and clicking the filter icons in the table.

Discover ContextView FilterMontage

Viewing Field Data Statistics

From the Fields list, you can see how many of the documents in the Documents table contain a particular field, what the top 5 values are, and what percentage of documents contain each value.

To view field data statistics, click the name of a field in the Fields list.

Field Statistics

Visualize

Visualize enables you to create visualizations of the data in your Elasticsearch indices. You can then build dashboards that display related visualizations.

Kibana visualizations are based on Elasticsearch queries. By using a series of Elasticsearch {ref}/search-aggregations.html[aggregations] to extract and process your data, you can create charts that show you the trends, spikes, and dips you need to know about.

You can create visualizations from a search saved from Discover or start with a new search query.

Creating a Visualization

To create a visualization:

  1. Click on Visualize in the side navigation.

  2. Click the Create new visualization button or the + button.

  3. Choose the visualization type:

    • Basic charts

      Line, Area and Bar charts

      Compare different series in X/Y charts.

      Heat maps

      Shade cells within a matrix.

      Pie chart

      Display each source’s contribution to a total.

    • Data

      Data table

      Display the raw data of a composed aggregation.

      Metric

      Display a single number.

      Goal and Gauge

      Display a gauge.

    • Maps

      Coordinate map

      Associate the results of an aggregation with geographic locations.

      Region map

      Thematic maps where a shape’s color intensity corresponds to a metric’s value. locations.

    • Time Series

      Timelion

      Compute and combine data from multiple time series data sets.

      Time Series Visual Builder

      Visualize time series data using pipeline aggregations.

    • Other

      Tag cloud

      Display words as a cloud in which the size of the word correspond to its importance

      Markdown widget

      Display free-form information or instructions.

  4. Specify a search query to retrieve the data for your visualization:

    • To enter new search criteria, select the index pattern for the indices that contain the data you want to visualize. This opens the visualization builder with a wildcard query that matches all of the documents in the selected indices.

    • To build a visualization from a saved search, click the name of the saved search you want to use. This opens the visualization builder and loads the selected query.

      When you build a visualization from a saved search, any subsequent modifications to the saved search are automatically reflected in the visualization. To disable automatic updates, you can disconnect a visualization from the saved search.
  5. In the visualization builder, choose the metric aggregation for the visualization’s Y axis:

    • Metric Aggregations:

    • {ref}/search-aggregations-metrics-valuecount-aggregation.html[count]

    • {ref}/search-aggregations-metrics-avg-aggregation.html[average]

    • {ref}/search-aggregations-metrics-sum-aggregation.html[sum]

    • {ref}/search-aggregations-metrics-min-aggregation.html[min]

    • {ref}/search-aggregations-metrics-max-aggregation.html[max]

    • {ref}/search-aggregations-metrics-stats-aggregation.html[standard deviation]

    • {ref}/search-aggregations-metrics-cardinality-aggregation.html[unique count]

    • {ref}/search-aggregations-metrics-percentile-aggregation.html[median] (50th percentile)

    • {ref}/search-aggregations-metrics-percentile-aggregation.html[percentiles]

    • {ref}/search-aggregations-metrics-percentile-rank-aggregation.html[percentile ranks]

    • {ref}/search-aggregations-metrics-top-hits-aggregation.html[top hit]

    • {ref}/search-aggregations-metrics-geocentroid-aggregation.html[geo centroid]

    • Parent Pipeline Aggregations:

    • {ref}/search-aggregations-pipeline-derivative-aggregation.html[derivative]

    • {ref}/search-aggregations-pipeline-cumulative-sum-aggregation.html[cumulative sum]

    • {ref}/search-aggregations-pipeline-movavg-aggregation.html[moving average]

    • {ref}/search-aggregations-pipeline-serialdiff-aggregation.html[serial diff]

    • Sibling Pipeline Aggregations:

    • {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[average bucket]

    • {ref}/search-aggregations-pipeline-sum-bucket-aggregation.html[sum bucket]

    • {ref}/search-aggregations-pipeline-min-bucket-aggregation.html[min bucket]

    • {ref}/search-aggregations-pipeline-max-bucket-aggregation.html[max bucket]

  6. For the visualizations X axis, select a bucket aggregation:

    • {ref}/search-aggregations-bucket-datehistogram-aggregation.html[date histogram]

    • {ref}/search-aggregations-bucket-range-aggregation.html[range]

    • {ref}/search-aggregations-bucket-terms-aggregation.html[terms]

    • {ref}/search-aggregations-bucket-filters-aggregation.html[filters]

    • {ref}/search-aggregations-bucket-significantterms-aggregation.html[significant terms]

For example, if you’re indexing Apache server logs, you could build a horizontal bar chart that shows the distribution of incoming requests by geographic location by specifying a terms aggregation on the geo.src field:

bar terms agg 5

The y-axis shows the number of requests received from each country, and the countries are displayed across the x-axis.

Bar, line, or area chart visualizations use metrics for the y-axis and buckets for the x-axis. Buckets are analogous to SQL GROUP BY statements. Pie charts, use the metric for the slice size and the bucket for the number of slices.

You can further break down the data by specifying sub aggregations. The first aggregation determines the data set for any subsequent aggregations. Sub aggregations are applied in order—​you can drag the aggregations to change the order in which they’re applied.

For example, you could add a terms sub aggregation on the geo.dest field to a vertical bar chart to see the locations those requests were targeting.

bar terms subagg 5

For more information about working with sub aggregations, see Kibana, Aggregation Execution Order, and You.

Enhanced search results

Enhanced search results is a visualisation that shows the documents matched by a query on an Elasticsearch index, similar to the stock Discover table.

In addition to column configuration, the visualization provides the following features:

  • it is possible to enable a column that indicates whether or not a search result is matched by a query on an external datasource. This is described in Relational Column.

  • it is possible to define click handlers on the cells in a column, e.g. to open the URL displayed in a cell. This is described in Click handlers.

Configuration view of the Enhanced search results table

Relational Column

The relational column can be used to display if a search result is matched by a query on an external datasource.

To enable the relational column, check the Enable Relational Column checkbox.

The screenshot below shows the configuration of a relational column named Why Relevant? where the value of a cell depends on the query Top 50 companies (HR count): if the value of the label index field of a document matches the value of the label variable in at least one record returned by the query, the name of the query will be displayed inside the cell.

Relational column configuration
Relational column example

In order to configure the relational column, you must set the following parameters:

  • Column name: the column name that will be displayed in the table header.

  • Source Field: the name of the index field that will be compared to a variable in the query results.

  • Target query: the name of the query to execute.

  • Target query variable name: the name of the query variable that will be compared to the index field specified in Source field.

Click handlers

It is possible to define two different actions when clicking on a cell;

  • Open an URL defined in the corresponding index field.

  • Select an entity in an external datasource matching the corresponding index field.

Follow URL

Select the Follow URL action to open a URL stored in an index field in a new window.

For example, the following configuration defines an handler that opens the URL stored in the field homepage_url when clicking on the cell displaying the label field.

Follow URL on click

To configure a click handler, you must set the following parameters:

  • Column — the name of the column to which the handler will be bound.

  • On click I want to — the action to perform on click, select Follow the URL here.

  • URL field — the name of the field containing the URL.

  • URL format — a custom format string to compose the URL, where @URL@ is replaced with the value of the field set in URL field.

URL format can be used to create dynamic URL; the screenshot below shows a configuration in which the value of the id field is used to define the path of an URL on example.org.

With this configuration, if the id field is set to 11 the resulting URL will be http://example.org/11 .

Follow URL with a custom format on click

Select an entity

Select the Select an entity action if you want to select an entity stored in an external datasource matching the selected Elasticsearch document; for more information about entity selection, please read the Datasource entity selection section.

To configure an entity selection action you must set the following parameters:

  • Column — the name of the column to which the handler will be bound.

  • On click I want to — the action to perform on click, select Select the document here.

  • Redirect to dashboard — if set, clicking on the cell will select the entity and display the specified dashboard.

Configuration of an entity selection handler

Siren Investigate query viewer

This visualization displays the results from multiple queries on external data sources using query templates.

To add a query to the visualization, click on the Add query button, then set the following parameters:

  • Label: the caption for the table, in case of a table template like kibi-table-jade. This sets the variable label to the given value.

  • Source query: the query used by the template.

  • Template: the template used to render results returned by Source query.

If one of the source queries requires an entity to be selected, you can set an entity URI for testing in the input field above the preview.

If a source query is not activated, the corresponding template will not be rendered.

The screenshots below show the configuration and output of a Templated query viewer visualization for a selected company:

Configuration of a Siren Investigate query viewer visualization

Advanced options

By clicking on the Advanced link, you can set additional rendering options.

It is possible to set additional template variables by writing them as JSON object properties in the Template variables textarea.

For example, to customize the heading of the generic table template (this is done automatically by the Label input field above), which is set by default to the id of the source query, you can customize the label variable as follows:

{
    "label": "Info"
}

By default, template contents are hidden and can be displayed by clicking on the show link in the heading; to make template contents visible by default, check Render opened box.

Advanced options

Siren Investigate Multi Chart

This visualization displays a multiple types of chart according to the current selection of multiple configurations.

Multi Chart

multi chart

Multi Chart is not a type of chart by itself, it can contains a set of other charts (such as a Pie chart) and allows you to switch to other type of chart with the same aggregations.

Multi configurations

multi configurations

Visualize settings

Selection 1

New configuration

After changing the aggregation settings and setting the desired type of chart, you can press the "Add this configuration" button to save the configuration as a separate one.

new configuration

Multi Chart has the following options

  • Show type selector - Allows to show/hide the button bar for the chart type selection.

  • Show dropdown menu - Allows to show/hide the dropdown menu for the aggregation configuration selection.

  • Show menu navigation buttons - Allow show/hide the navigation buttons around the dropdown menu.

Data Table

Metric Aggregations:

Count

The {ref}/search-aggregations-metrics-valuecount-aggregation.html[count] aggregation returns a raw count of the elements in the selected index pattern.

Average

This aggregation returns the {ref}/search-aggregations-metrics-avg-aggregation.html[average] of a numeric field. Select a field from the drop-down.

Sum

The {ref}/search-aggregations-metrics-sum-aggregation.html[sum] aggregation returns the total sum of a numeric field. Select a field from the drop-down.

Min

The {ref}/search-aggregations-metrics-min-aggregation.html[min] aggregation returns the minimum value of a numeric field. Select a field from the drop-down.

Max

The {ref}/search-aggregations-metrics-max-aggregation.html[max] aggregation returns the maximum value of a numeric field. Select a field from the drop-down.

Unique Count

The {ref}/search-aggregations-metrics-cardinality-aggregation.html[cardinality] aggregation returns the number of unique values in a field. Select a field from the drop-down.

Standard Deviation

The {ref}/search-aggregations-metrics-extendedstats-aggregation.html[extended stats] aggregation returns the standard deviation of data in a numeric field. Select a field from the drop-down.

Percentiles

The {ref}/search-aggregations-metrics-percentile-aggregation.html[percentile] aggregation divides the values in a numeric field into percentile bands that you specify. Select a field from the drop-down, then specify one or more ranges in the Percentiles fields. Click the X to remove a percentile field. Click + Add to add a percentile field.

Percentile Rank

The {ref}/search-aggregations-metrics-percentile-rank-aggregation.html[percentile ranks] aggregation returns the percentile rankings for the values in the numeric field you specify. Select a numeric field from the drop-down, then specify one or more percentile rank values in the Values fields. Click the X to remove a values field. Click +Add to add a values field.

Parent Pipeline Aggregations:

For each of the parent pipeline aggregations you have to define the metric for which the aggregation is calculated. That could be one of your existing metrics or a new one. You can also nest this aggregations (for example to produce 3rd derivative)

Derivative

The {ref}/search-aggregations-pipeline-derivative-aggregation.html[derivative] aggregation calculates the derivative of specific metrics.

Cumulative Sum

The {ref}/search-aggregations-pipeline-cumulative-sum-aggregation.html[cumulative sum] aggregation calculates the cumulative sum of a specified metric in a parent histogram

Moving Average

The {ref}/search-aggregations-pipeline-movavg-aggregation.html[moving average] aggregation will slide a window across the data and emit the average value of that window

Serial Diff

The {ref}/search-aggregations-pipeline-serialdiff-aggregation.html[serial differencing] is a technique where values in a time series are subtracted from itself at different time lags or period

Sibling Pipeline Aggregations:

Just like with parent pipeline aggregations you need to provide a metric for which to calculate the sibling aggregation. On top of that you also need to provide a bucket aggregation which will define the buckets on which the sibling aggregation will run

Average Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[avg bucket] calculates the (mean) average value of a specified metric in a sibling aggregation

Sum Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[sum bucket] calculates the sum of values of a specified metric in a sibling aggregation

Min Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[min bucket] calculates the minimum value of a specified metric in a sibling aggregation

Max Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[max bucket] calculates the maximum value of a specified metric in a sibling aggregation

You can add an aggregation by clicking the + Add Metrics button.

Enter a string in the Custom Label field to change the display label.

The rows of the data table are called buckets. You can define buckets to split the table into rows or to split the table into additional tables.

Each bucket type supports the following aggregations:

Date Histogram

A {ref}/search-aggregations-bucket-datehistogram-aggregation.html[date histogram] is built from a numeric field and organized by date. You can specify a time frame for the intervals in seconds, minutes, hours, days, weeks, months, or years. You can also specify a custom interval frame by selecting Custom as the interval and specifying a number and a time unit in the text field. Custom interval time units are s for seconds, m for minutes, h for hours, d for days, w for weeks, and y for years. Different units support different levels of precision, down to one second. Intervals are labeled at the start of the interval, using the date-key returned by Elasticsearch. For example, the tooltip for a monthly interval will show the first day of the month.

Histogram

A standard {ref}/search-aggregations-bucket-histogram-aggregation.html[histogram] is built from a numeric field. Specify an integer interval for this field. Select the Show empty buckets checkbox to include empty intervals in the histogram.

Range

With a {ref}/search-aggregations-bucket-range-aggregation.html[range] aggregation, you can specify ranges of values for a numeric field. Click Add Range to add a set of range endpoints. Click the red (x) symbol to remove a range.

Date Range

A {ref}/search-aggregations-bucket-daterange-aggregation.html[date range] aggregation reports values that are within a range of dates that you specify. You can specify the ranges for the dates using {ref}/common-options.html#date-math[date math] expressions. Click Add Range to add a set of range endpoints. Click the red (/) symbol to remove a range.

IPv4 Range

The {ref}/search-aggregations-bucket-iprange-aggregation.html[IPv4 range] aggregation enables you to specify ranges of IPv4 addresses. Click Add Range to add a set of range endpoints. Click the red (/) symbol to remove a range.

Terms

A {ref}/search-aggregations-bucket-terms-aggregation.html[terms] aggregation enables you to specify the top or bottom n elements of a given field to display, ordered by count or a custom metric.

Filters

You can specify a set of {ref}/search-aggregations-bucket-filters-aggregation.html[filters] for the data. You can specify a filter as a query string or in JSON format, just as in the Discover search bar. Click Add Filter to add another filter. Click the labelbutton label button to open the label field, where you can type in a name to display on the visualization.

Significant Terms

Displays the results of the experimental {ref}/search-aggregations-bucket-significantterms-aggregation.html[significant terms] aggregation. The value of the Size parameter defines the number of entries this aggregation returns.

Geohash

The {ref}/search-aggregations-bucket-geohashgrid-aggregation.html[geohash] aggregation displays points based on the geohash coordinates.

Once you’ve specified a bucket type aggregation, you can define sub-buckets to refine the visualization. Click + Add sub-buckets to define a sub-bucket, then choose Split Rows or Split Table, then select an aggregation from the list of types.

You can use the up or down arrows to the right of the aggregation’s type to change the aggregation’s priority.

Enter a string in the Custom Label field to change the display label.

You can click the Advanced link to display more customization options for your metrics or bucket aggregation:

Exclude Pattern

Specify a pattern in this field to exclude from the results.

Include Pattern

Specify a pattern in this field to include in the results.

JSON Input

A text field where you can add specific JSON-formatted properties to merge with the aggregation definition, as in the following example:

{ "script" : "doc['grade'].value * 1.2" }
In Elasticsearch releases 1.4.3 and later, this functionality requires you to enable {ref}/modules-scripting.html[dynamic Groovy scripting].

The availability of these options varies depending on the aggregation you choose.

Select the Options tab to change the following aspects of the table:

Per Page

This field controls the pagination of the table. The default value is ten rows per page.

Checkboxes are available to enable and disable the following behaviors:

Show metrics for every bucket/level

Check this box to display the intermediate results for each bucket aggregation.

Show partial rows

Check this box to display a row even when there is no result.

Enabling these behaviors may have a substantial effect on performance.

Viewing Detailed Information

Visualization Spy

To display the raw data behind the visualization, click the spy-open-button button in the bottom left corner of the container. The visualization spy panel will open. Use the select input to view detailed information about the raw data.

spy-panel

Table

A representation of the underlying data, presented as a paginated data grid. You can sort the items in the table by clicking on the table headers at the top of each column.

Request

The raw request used to query the server, presented in JSON format.

Response

The raw response from the server, presented in JSON format.

Statistics

A summary of the statistics related to the request and the response, presented as a data grid. The data grid includes the query duration, the request duration, the total number of records found on the server, and the index pattern used to make the query.

Debug

The visualization saved state presented in JSON format.

To export the raw data behind the visualization as a comma-separated-values (CSV) file, click on either the Raw or Formatted links at the bottom of any of the detailed information tabs. A raw export contains the data as it is stored in Elasticsearch. A formatted export contains the results of any applicable field formatters.

Markdown Widget

The Markdown widget is a text entry field that accepts GitHub-flavored Markdown text. Siren Investigate renders the text you enter in this field and displays the results on the dashboard. You can click the Help link to go to the help page for GitHub flavored Markdown. Click Apply to display the rendered text in the Preview pane or Discard to revert to a previous version.

Metric

A metric visualization displays a single number for each aggregation you select:

Metric Aggregations:

Count

The {ref}/search-aggregations-metrics-valuecount-aggregation.html[count] aggregation returns a raw count of the elements in the selected index pattern.

Average

This aggregation returns the {ref}/search-aggregations-metrics-avg-aggregation.html[average] of a numeric field. Select a field from the drop-down.

Sum

The {ref}/search-aggregations-metrics-sum-aggregation.html[sum] aggregation returns the total sum of a numeric field. Select a field from the drop-down.

Min

The {ref}/search-aggregations-metrics-min-aggregation.html[min] aggregation returns the minimum value of a numeric field. Select a field from the drop-down.

Max

The {ref}/search-aggregations-metrics-max-aggregation.html[max] aggregation returns the maximum value of a numeric field. Select a field from the drop-down.

Unique Count

The {ref}/search-aggregations-metrics-cardinality-aggregation.html[cardinality] aggregation returns the number of unique values in a field. Select a field from the drop-down.

Standard Deviation

The {ref}/search-aggregations-metrics-extendedstats-aggregation.html[extended stats] aggregation returns the standard deviation of data in a numeric field. Select a field from the drop-down.

Percentiles

The {ref}/search-aggregations-metrics-percentile-aggregation.html[percentile] aggregation divides the values in a numeric field into percentile bands that you specify. Select a field from the drop-down, then specify one or more ranges in the Percentiles fields. Click the X to remove a percentile field. Click + Add to add a percentile field.

Percentile Rank

The {ref}/search-aggregations-metrics-percentile-rank-aggregation.html[percentile ranks] aggregation returns the percentile rankings for the values in the numeric field you specify. Select a numeric field from the drop-down, then specify one or more percentile rank values in the Values fields. Click the X to remove a values field. Click +Add to add a values field.

Parent Pipeline Aggregations:

For each of the parent pipeline aggregations you have to define the metric for which the aggregation is calculated. That could be one of your existing metrics or a new one. You can also nest this aggregations (for example to produce 3rd derivative)

Derivative

The {ref}/search-aggregations-pipeline-derivative-aggregation.html[derivative] aggregation calculates the derivative of specific metrics.

Cumulative Sum

The {ref}/search-aggregations-pipeline-cumulative-sum-aggregation.html[cumulative sum] aggregation calculates the cumulative sum of a specified metric in a parent histogram

Moving Average

The {ref}/search-aggregations-pipeline-movavg-aggregation.html[moving average] aggregation will slide a window across the data and emit the average value of that window

Serial Diff

The {ref}/search-aggregations-pipeline-serialdiff-aggregation.html[serial differencing] is a technique where values in a time series are subtracted from itself at different time lags or period

Sibling Pipeline Aggregations:

Just like with parent pipeline aggregations you need to provide a metric for which to calculate the sibling aggregation. On top of that you also need to provide a bucket aggregation which will define the buckets on which the sibling aggregation will run

Average Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[avg bucket] calculates the (mean) average value of a specified metric in a sibling aggregation

Sum Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[sum bucket] calculates the sum of values of a specified metric in a sibling aggregation

Min Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[min bucket] calculates the minimum value of a specified metric in a sibling aggregation

Max Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[max bucket] calculates the maximum value of a specified metric in a sibling aggregation

You can add an aggregation by clicking the + Add Metrics button.

Enter a string in the Custom Label field to change the display label.

You can click the Advanced link to display more customization options:

JSON Input

A text field where you can add specific JSON-formatted properties to merge with the aggregation definition, as in the following example:

{ "script" : "doc['grade'].value * 1.2" }
In Elasticsearch releases 1.4.3 and later, this functionality requires you to enable {ref}/modules-scripting.html[dynamic Groovy scripting].

The availability of these options varies depending on the aggregation you choose.

Click the Options tab to display the font size slider.

Viewing Detailed Information

Visualization Spy

To display the raw data behind the visualization, click the spy-open-button button in the bottom left corner of the container. The visualization spy panel will open. Use the select input to view detailed information about the raw data.

spy-panel

Table

A representation of the underlying data, presented as a paginated data grid. You can sort the items in the table by clicking on the table headers at the top of each column.

Request

The raw request used to query the server, presented in JSON format.

Response

The raw response from the server, presented in JSON format.

Statistics

A summary of the statistics related to the request and the response, presented as a data grid. The data grid includes the query duration, the request duration, the total number of records found on the server, and the index pattern used to make the query.

Debug

The visualization saved state presented in JSON format.

To export the raw data behind the visualization as a comma-separated-values (CSV) file, click on either the Raw or Formatted links at the bottom of any of the detailed information tabs. A raw export contains the data as it is stored in Elasticsearch. A formatted export contains the results of any applicable field formatters.

Goal and Gauge

A goal visualization displays how your metric progresses toward a fixed goal. A gauge visualization displays in which predefined range falls your metric.

Metric Aggregations:

Count

The {ref}/search-aggregations-metrics-valuecount-aggregation.html[count] aggregation returns a raw count of the elements in the selected index pattern.

Average

This aggregation returns the {ref}/search-aggregations-metrics-avg-aggregation.html[average] of a numeric field. Select a field from the drop-down.

Sum

The {ref}/search-aggregations-metrics-sum-aggregation.html[sum] aggregation returns the total sum of a numeric field. Select a field from the drop-down.

Min

The {ref}/search-aggregations-metrics-min-aggregation.html[min] aggregation returns the minimum value of a numeric field. Select a field from the drop-down.

Max

The {ref}/search-aggregations-metrics-max-aggregation.html[max] aggregation returns the maximum value of a numeric field. Select a field from the drop-down.

Unique Count

The {ref}/search-aggregations-metrics-cardinality-aggregation.html[cardinality] aggregation returns the number of unique values in a field. Select a field from the drop-down.

Standard Deviation

The {ref}/search-aggregations-metrics-extendedstats-aggregation.html[extended stats] aggregation returns the standard deviation of data in a numeric field. Select a field from the drop-down.

Percentiles

The {ref}/search-aggregations-metrics-percentile-aggregation.html[percentile] aggregation divides the values in a numeric field into percentile bands that you specify. Select a field from the drop-down, then specify one or more ranges in the Percentiles fields. Click the X to remove a percentile field. Click + Add to add a percentile field.

Percentile Rank

The {ref}/search-aggregations-metrics-percentile-rank-aggregation.html[percentile ranks] aggregation returns the percentile rankings for the values in the numeric field you specify. Select a numeric field from the drop-down, then specify one or more percentile rank values in the Values fields. Click the X to remove a values field. Click +Add to add a values field.

Parent Pipeline Aggregations:

For each of the parent pipeline aggregations you have to define the metric for which the aggregation is calculated. That could be one of your existing metrics or a new one. You can also nest this aggregations (for example to produce 3rd derivative)

Derivative

The {ref}/search-aggregations-pipeline-derivative-aggregation.html[derivative] aggregation calculates the derivative of specific metrics.

Cumulative Sum

The {ref}/search-aggregations-pipeline-cumulative-sum-aggregation.html[cumulative sum] aggregation calculates the cumulative sum of a specified metric in a parent histogram

Moving Average

The {ref}/search-aggregations-pipeline-movavg-aggregation.html[moving average] aggregation will slide a window across the data and emit the average value of that window

Serial Diff

The {ref}/search-aggregations-pipeline-serialdiff-aggregation.html[serial differencing] is a technique where values in a time series are subtracted from itself at different time lags or period

Sibling Pipeline Aggregations:

Just like with parent pipeline aggregations you need to provide a metric for which to calculate the sibling aggregation. On top of that you also need to provide a bucket aggregation which will define the buckets on which the sibling aggregation will run

Average Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[avg bucket] calculates the (mean) average value of a specified metric in a sibling aggregation

Sum Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[sum bucket] calculates the sum of values of a specified metric in a sibling aggregation

Min Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[min bucket] calculates the minimum value of a specified metric in a sibling aggregation

Max Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[max bucket] calculates the maximum value of a specified metric in a sibling aggregation

You can add an aggregation by clicking the + Add Metrics button.

Enter a string in the Custom Label field to change the display label.

Open the Advanced link to display more customization options:

JSON Input

A text field where you can add specific JSON-formatted properties to merge with the aggregation definition, as in the following example:

{ "script" : "doc['grade'].value * 1.2" }
In Elasticsearch releases 1.4.3 and later, this functionality requires you to enable {ref}/modules-scripting.html[dynamic Groovy scripting].

The availability of these options varies depending on the aggregation you choose.

Click the Options tab to change the following options:

  • Gauge Type select between arc, circle and metric display type.

  • Percentage Mode will show all values as percentages

  • Vertical Split will put the gauges one under another instead of one next to another

  • Show Labels selects whether you want to show or hide the labels

  • Sub Text text for the label that appears below the value

  • Auto Extend Range automatically grows the gauge if value is over its extents.

  • Ranges you can add custom ranges. Each range will get assigned a color. If value falls within that range it will get assigned that color. A chart with a single range is called a goal chart. A chart with multiple ranges is called a gauge chart.

  • Color Options define how to color your ranges (which color schema to use). Color options are only visible if more than one range is defined.

  • Style - Show Scale shows or hides the scale

  • Style - Color Labels whether the labels should have the same color as the range where the value falls in

Pie Charts

The slice size of a pie chart is determined by the metrics aggregation. The following aggregations are available for this axis:

Count

The {ref}/search-aggregations-metrics-valuecount-aggregation.html[count] aggregation returns a raw count of the elements in the selected index pattern.

Sum

The {ref}/search-aggregations-metrics-sum-aggregation.html[sum] aggregation returns the total sum of a numeric field. Select a field from the drop-down.

Unique Count

The {ref}/search-aggregations-metrics-cardinality-aggregation.html[cardinality] aggregation returns the number of unique values in a field. Select a field from the drop-down.

Enter a string in the Custom Label field to change the display label.

The buckets aggregations determine what information is being retrieved from your data set.

Before you choose a buckets aggregation, specify if you are splitting slices within a single chart or splitting into multiple charts. A multiple chart split must run before any other aggregations. When you split a chart, you can change if the splits are displayed in a row or a column by clicking the Rows | Columns selector.

You can specify any of the following bucket aggregations for your pie chart:

Date Histogram

A {ref}/search-aggregations-bucket-datehistogram-aggregation.html[date histogram] is built from a numeric field and organized by date. You can specify a time frame for the intervals in seconds, minutes, hours, days, weeks, months, or years. You can also specify a custom interval frame by selecting Custom as the interval and specifying a number and a time unit in the text field. Custom interval time units are s for seconds, m for minutes, h for hours, d for days, w for weeks, and y for years. Different units support different levels of precision, down to one second. Intervals are labeled at the start of the interval, using the date-key returned by Elasticsearch. For example, the tooltip for a monthly interval will show the first day of the month.

Histogram

A standard {ref}/search-aggregations-bucket-histogram-aggregation.html[histogram] is built from a numeric field. Specify an integer interval for this field. Select the Show empty buckets checkbox to include empty intervals in the histogram.

Range

With a {ref}/search-aggregations-bucket-range-aggregation.html[range] aggregation, you can specify ranges of values for a numeric field. Click Add Range to add a set of range endpoints. Click the red (x) symbol to remove a range.

Date Range

A {ref}/search-aggregations-bucket-daterange-aggregation.html[date range] aggregation reports values that are within a range of dates that you specify. You can specify the ranges for the dates using {ref}/common-options.html#date-math[date math] expressions. Click Add Range to add a set of range endpoints. Click the red (/) symbol to remove a range.

IPv4 Range

The {ref}/search-aggregations-bucket-iprange-aggregation.html[IPv4 range] aggregation enables you to specify ranges of IPv4 addresses. Click Add Range to add a set of range endpoints. Click the red (/) symbol to remove a range.

Terms

A {ref}/search-aggregations-bucket-terms-aggregation.html[terms] aggregation enables you to specify the top or bottom n elements of a given field to display, ordered by count or a custom metric.

Filters

You can specify a set of {ref}/search-aggregations-bucket-filters-aggregation.html[filters] for the data. You can specify a filter as a query string or in JSON format, just as in the Discover search bar. Click Add Filter to add another filter. Click the labelbutton label button to open the label field, where you can type in a name to display on the visualization.

Significant Terms

Displays the results of the experimental {ref}/search-aggregations-bucket-significantterms-aggregation.html[significant terms] aggregation. The value of the Size parameter defines the number of entries this aggregation returns.

After defining an initial bucket aggregation, you can define sub-buckets to refine the visualization. Click + Add sub-buckets to define a sub-aggregation, then choose Split Slices to select a sub-bucket from the list of types.

When multiple aggregations are defined on a chart’s axis, you can use the up or down arrows to the right of the aggregation’s type to change the aggregation’s priority.

You can customize the colors of your visualization by clicking the color dot next to each label to display the color picker.

An array of color dots that users can select

Enter a string in the Custom Label field to change the display label.

You can click the Advanced link to display more customization options for your metrics or bucket aggregation:

Exclude Pattern

Specify a pattern in this field to exclude from the results.

Include Pattern

Specify a pattern in this field to include in the results.

JSON Input

A text field where you can add specific JSON-formatted properties to merge with the aggregation definition, as in the following example:

{ "script" : "doc['grade'].value * 1.2" }
In Elasticsearch releases 1.4.3 and later, this functionality requires you to enable {ref}/modules-scripting.html[dynamic Groovy scripting].

The availability of these options varies depending on the aggregation you choose.

Select the Options tab to change the following aspects of the table:

Donut

Display the chart as a sliced ring instead of a sliced pie.

Show Tooltip

Check this box to enable the display of tooltips.

After changing options, click the Apply changes button to update your visualization, or the grey Discard changes button to keep your visualization in its current state.

Viewing Detailed Information

Visualization Spy

To display the raw data behind the visualization, click the spy-open-button button in the bottom left corner of the container. The visualization spy panel will open. Use the select input to view detailed information about the raw data.

spy-panel

Table

A representation of the underlying data, presented as a paginated data grid. You can sort the items in the table by clicking on the table headers at the top of each column.

Request

The raw request used to query the server, presented in JSON format.

Response

The raw response from the server, presented in JSON format.

Statistics

A summary of the statistics related to the request and the response, presented as a data grid. The data grid includes the query duration, the request duration, the total number of records found on the server, and the index pattern used to make the query.

Debug

The visualization saved state presented in JSON format.

To export the raw data behind the visualization as a comma-separated-values (CSV) file, click on either the Raw or Formatted links at the bottom of any of the detailed information tabs. A raw export contains the data as it is stored in Elasticsearch. A formatted export contains the results of any applicable field formatters.

Coordinate Maps

A coordinate map displays a geographic area overlaid with circles keyed to the data determined by the buckets you specify.

By default, Siren Investigate uses the Open Street Maps service to display map tiles. To use other tile service providers, configure the tilemap settings in siren.yml.

Configuration

Data

Metrics

The default metrics aggregation for a coordinate map is the Count aggregation. You can select any of the following aggregations as the metrics aggregation:

Count

The {ref}/search-aggregations-metrics-valuecount-aggregation.html[count] aggregation returns a raw count of the elements in the selected index pattern.

Average

This aggregation returns the {ref}/search-aggregations-metrics-avg-aggregation.html[average] of a numeric field. Select a field from the drop-down.

Sum

The {ref}/search-aggregations-metrics-sum-aggregation.html[sum] aggregation returns the total sum of a numeric field. Select a field from the drop-down.

Min

The {ref}/search-aggregations-metrics-min-aggregation.html[min] aggregation returns the minimum value of a numeric field. Select a field from the drop-down.

Max

The {ref}/search-aggregations-metrics-max-aggregation.html[max] aggregation returns the maximum value of a numeric field. Select a field from the drop-down.

Unique Count

The {ref}/search-aggregations-metrics-cardinality-aggregation.html[cardinality] aggregation returns the number of unique values in a field. Select a field from the drop-down.

Enter a string in the Custom Label field to change the display label.

Buckets

Coordinate maps use the geohash aggregation. Select a field, typically coordinates, from the drop-down.

  • The_Change precision on map zoom_ box is checked by default. Uncheck the box to disable this behavior. The Precision slider determines the granularity of the results displayed on the map. See the documentation for the {ref}/search-aggregations-bucket-geohashgrid-aggregation.html#_cell_dimensions_at_the_equator[geohash grid] aggregation for details on the area specified by each precision level.

Higher precisions increase memory usage for the browser displaying Siren Investigate as well as for the underlying Elasticsearch cluster.
  • The place markers off grid (use {ref}/search-aggregations-metrics-geocentroid-aggregation.html[geocentroid]) box is checked by default. When this box is checked, the markers are placed in the center of all the documents in that bucket. When unchecked, the markers are placed in the center of the geohash grid cell. Leaving this checked generally results in a more accurate visualization.

Enter a string in the Custom Label field to change the display label.

You can click the Advanced link to display more customization options for your metrics or bucket aggregation:

Exclude Pattern

Specify a pattern in this field to exclude from the results.

Include Pattern

Specify a pattern in this field to include in the results.

JSON Input

A text field where you can add specific JSON-formatted properties to merge with the aggregation definition, as in the following example:

{ "script" : "doc['grade'].value * 1.2" }
In Elasticsearch releases 1.4.3 and later, this functionality requires you to enable {ref}/modules-scripting.html[dynamic Groovy scripting].

The availability of these options varies depending on the aggregation you choose.

Options

Map type

Select one of the following options from the drop-down.

Scaled Circle Markers

Scale the size of the markers based on the metric aggregation’s value.

Shaded Circle Markers

Displays the markers with different shades based on the metric aggregation’s value.

Shaded Geohash Grid

Displays the rectangular cells of the geohash grid instead of circular markers, with different shades based on the metric aggregation’s value.

Heatmap

A heat map applies blurring to the circle markers and applies shading based on the amount of overlap. Heatmaps have the following options:

  • Radius: Sets the size of the individual heatmap dots.

  • Blur: Sets the amount of blurring for the heatmap dots.

  • Maximum zoom: Tilemaps in Siren Investigate support 18 zoom levels. This slider defines the maximum zoom level at which the heatmap dots appear at full intensity.

  • Minimum opacity: Sets the opacity cutoff for the dots.

  • Show Tooltip: Check this box to have a tooltip with the values for a given dot when the cursor is on that dot.

Desaturate map tiles

Desaturate the map’s color in order to make the markers stand out more clearly.

WMS compliant map server

Check this box to enable the use of a third-party mapping service that complies with the Web Map Service (WMS) standard. Specify the following elements:

  • WMS url: The URL for the WMS map service.

  • WMS layers: A comma-separated list of the layers to use in this visualization. Each map server provides its own list of layers.

  • WMS version: The WMS version used by this map service.

  • WMS format: The image format used by this map service. The two most common formats are image/png and image/jpeg.

  • WMS attribution: An optional, user-defined string that identifies the map source. Maps display the attribution string in the lower right corner.

  • WMS styles: A comma-separated list of the styles to use in this visualization. Each map server provides its own styling options.

After changing options, click the Apply changes button to update your visualization, or the grey Discard changes button to keep your visualization in its current state.

Once your tilemap visualization is ready, you can explore the map in several ways:

  • Click and hold anywhere on the map and move the cursor to move the map center. Hold Shift and drag a bounding box across the map to zoom in on the selection.

  • Click the Zoom In/Out viz-zoom buttons to change the zoom level manually.

  • Click the Fit Data Bounds viz-fit-bounds button to automatically crop the map boundaries to the geohash buckets that have at least one result.

  • Click the Latitude/Longitude Filter viz-lat-long-filter button, then drag a bounding box across the map, to create a filter for the box coordinates.

Viewing Detailed Information

Visualization Spy

To display the raw data behind the visualization, click the spy-open-button button in the bottom left corner of the container. The visualization spy panel will open. Use the select input to view detailed information about the raw data.

spy-panel

Table

A representation of the underlying data, presented as a paginated data grid. You can sort the items in the table by clicking on the table headers at the top of each column.

Request

The raw request used to query the server, presented in JSON format.

Response

The raw response from the server, presented in JSON format.

Statistics

A summary of the statistics related to the request and the response, presented as a data grid. The data grid includes the query duration, the request duration, the total number of records found on the server, and the index pattern used to make the query.

Debug

The visualization saved state presented in JSON format.

To export the raw data behind the visualization as a comma-separated-values (CSV) file, click on either the Raw or Formatted links at the bottom of any of the detailed information tabs. A raw export contains the data as it is stored in Elasticsearch. A formatted export contains the results of any applicable field formatters.

Region Maps

Region maps are thematic maps in which boundary vector shapes are colored using a gradient: higher intensity colors indicate larger values, and lower intensity colors indicate smaller values. These are also known as choropleth maps.

regionmap

Configuration

To create a region map, you configure an inner join that joins the result of an Elasticsearch terms aggregation and a reference vector file based on a shared key.

Data

Metrics

Select any of the supported Metric or Sibling Pipeline Aggregations.

Buckets

Configure a Terms aggregation. The term is the key that is used to join the results to the vector data on the map.

Options

Layer Settings
  • Vector map: select from a list of vector maps. This list includes the maps that are hosted by the © Elastic Maps Service, as well as your self-hosted layers that are configured in the config/kibana.yml file. To learn more about how to configure Kibana to make self-hosted layers available, see the regionmap settings documentation.

  • Join field: this is the property from the selected vector map that will be used to join on the terms in your terms-aggregation. When terms cannot be joined to any of the shapes in the vector layer because there is no exact match in the vector layer, Kibana will display a warning. To turn of these warnings, go to Management/Kibana/Advanced Settings and set visualization:regionmap:showWarnings to false.

Style Settings
  • Color Schema: the color range used to color the shapes.

Basic Settings
  • Legend Position: the location on the screen where the legend should be rendered.

  • Show Tooltip: indicates whether a tooltip should be displayed when hovering over a shape..

Time Series Visual Builder

Experimental Feature

Time Series Visual Builder is a time series data visualizer with an emphasis on allowing you to use the full power of Elasticsearch aggregation framework. Time Series Visual Builder allows you to combine an infinite number of aggregations and pipeline aggregations to display complex data in a meaningful way.

Time Series Visual Builder Interface

Time Series Visual Build comes with 5 different visualization types. You can switch between each visualization type using the tabbed picker at the top of the interface.

Time Series

A histogram visualization that supports area, line, bar, and steps along with multiple y-axis. You can fully customize the colors, points, line thickness and fill opacity. This visualization also supports time shifting to compare two time periods. This visualization also supports annotations which can be loaded from a seperate index based on a query.

Time Series Visualization

Metric

A visualization for displaying the latest number in a series. This visualization supports 2 metrics; a primary metric and a secondary metric. The labels and backgrounds can be fully customizable based on a set of rules.

Metric Visualization

Top N

This is a horizontal bar chart where the y-axis is based on a series of metrics and the x-axis is the latest value in those series; sorted in descending order. The color of the bars are fully customizable based on set of rules.

Top N Visualization

Gauge

This is a single value gauge visualization based on the latest value in a series. The face of the gauge can either be a half-circle gauge or full-circle. You can customize the thicknesses of the inner and outer lines to achieve a desired design aesthetic. The color of the gauge and the text are fully customizable based on a set of rules.

Gauge Visualization

Markdown

This visualization allows you to enter Markdown text and embed Mustache template syntax to customize the Markdown with data based on a set of series. This visualization also supports HTML markup along with the ability to define a custom stylesheet.

Markdown Visualization

Interface Overview

The user interface for each visualization is compose of a "Data" tab and "Panel Options". The only exception to that is the Time Series and Markdown visualizations; the Time Series has a third tab for annotations and the Markdown has a third tab for the editor.

Data Tab

The data tab is used for configuring the series for each visualization. This tab allows you to add multiple series, depending on what the visualization supports, with multiple aggregations composed together to create a single metric. Here is a breakdown of the significant components of the data tab UI.

Series Label and Color

Each series supports a label which will be used for legends and titles depending on which visualization type is selected. For series that are grouped by a term, you can specify a mustache variable of {{key}} to substitute the term. For most visualizations you can also choose a color by clicking on the swatch, this will display the color picker.

Label Example

Metrics

Each series supports multiple metrics (aggregations); the last metric (aggregation) is the value that will be displayed for the series, this is indicated with the "eye" icon to the left of the metric. Metrics can be composed using pipeline aggregations. A common use case is to create a metric with a "max" aggregation then create a "derivative" metric and choose the previous "max" metric as the source; this will create a rate.

Derivative Example

Series Options

Each series also supports a set of options which are dependent on the type of visualizations you have selected. Universal across each visualization type you can configure:

  • Data format

  • Time range offset

  • Index pattern, timestamp, and interval override

Default Series Options

For the Time Series visualization you can also configure:

  • Chart type

  • Options for each chart type

  • Legend Visibility

  • Y-Axis options

  • Split color theme

Time Series Series Options

Group By Controls

At the bottom of the metrics there is a set of "Group By" controls that allows you to specify how the series should be grouped or split. There are four choices:

  • Everything

  • Filter (single)

  • Filters (multiple with configurable colors)

  • Terms

By default the series is grouped by everything.

Panel Options Tab

The panel options tab is used for configuring the entire panel; the set of options available is dependent on which visualization you have selected. Below is a list of the options available per visualization:

Time Series

  • Index pattern, timestamp, and Interval

  • Y-Axis min and max

  • Y-Axis position

  • Background color

  • Legend visibility

  • Legend position

  • Panel filter

Metric

  • Index pattern, timestamp, and interval

  • Panel filter

  • Color rules for background and primary value

Top N

  • Index pattern, timestamp, and interval

  • Panel filter

  • Background color

  • Item URL

  • Color rules for bar colors

Gauge

  • Index pattern, timestamp, and interval

  • Panel filter

  • Background color

  • Gauge max

  • Gauge style

  • Inner gauge color

  • Inner gauge width

  • Gauge line width

  • Color rules for gauge line

Markdown

  • Index pattern, timestamp, and interval

  • Panel filter

  • Background color

  • Scroll bar visibility

  • Vertical alignment of content

  • Custom Panel CSS with support for Less syntax

Annotations Tab

The annotations tab is used for adding annotation data sources to the Time Series Visualization. You can configure the following options:

  • Index pattern and time field

  • Annotation color

  • Annotation icon

  • Fields to include in message

  • Format of message

  • Filtering options at the panel and global level

Annotation Tab

Markdown Tab

The markdown tab is used for editing the source for the Markdown visualization. The user interface has an editor on the left side and the available variables from the data tab on the right side. You can click on the variable names to insert the mustache template variable into the markdown at the cursor position. The mustache syntax uses the Handlebar.js processor which is an extended version of the Mustache template language.

Markdown Tab

Tag Clouds

A tag cloud visualization is a visual representation of text data, typically used to visualize free form text. Tags are usually single words, and the importance of each tag is shown with font size or color.

The font size for each word is determined by the metrics aggregation. The following aggregations are available for this chart:

Metric Aggregations:

Count

The {ref}/search-aggregations-metrics-valuecount-aggregation.html[count] aggregation returns a raw count of the elements in the selected index pattern.

Average

This aggregation returns the {ref}/search-aggregations-metrics-avg-aggregation.html[average] of a numeric field. Select a field from the drop-down.

Sum

The {ref}/search-aggregations-metrics-sum-aggregation.html[sum] aggregation returns the total sum of a numeric field. Select a field from the drop-down.

Min

The {ref}/search-aggregations-metrics-min-aggregation.html[min] aggregation returns the minimum value of a numeric field. Select a field from the drop-down.

Max

The {ref}/search-aggregations-metrics-max-aggregation.html[max] aggregation returns the maximum value of a numeric field. Select a field from the drop-down.

Unique Count

The {ref}/search-aggregations-metrics-cardinality-aggregation.html[cardinality] aggregation returns the number of unique values in a field. Select a field from the drop-down.

Standard Deviation

The {ref}/search-aggregations-metrics-extendedstats-aggregation.html[extended stats] aggregation returns the standard deviation of data in a numeric field. Select a field from the drop-down.

Percentiles

The {ref}/search-aggregations-metrics-percentile-aggregation.html[percentile] aggregation divides the values in a numeric field into percentile bands that you specify. Select a field from the drop-down, then specify one or more ranges in the Percentiles fields. Click the X to remove a percentile field. Click + Add to add a percentile field.

Percentile Rank

The {ref}/search-aggregations-metrics-percentile-rank-aggregation.html[percentile ranks] aggregation returns the percentile rankings for the values in the numeric field you specify. Select a numeric field from the drop-down, then specify one or more percentile rank values in the Values fields. Click the X to remove a values field. Click +Add to add a values field.

Parent Pipeline Aggregations:

For each of the parent pipeline aggregations you have to define the metric for which the aggregation is calculated. That could be one of your existing metrics or a new one. You can also nest this aggregations (for example to produce 3rd derivative)

Derivative

The {ref}/search-aggregations-pipeline-derivative-aggregation.html[derivative] aggregation calculates the derivative of specific metrics.

Cumulative Sum

The {ref}/search-aggregations-pipeline-cumulative-sum-aggregation.html[cumulative sum] aggregation calculates the cumulative sum of a specified metric in a parent histogram

Moving Average

The {ref}/search-aggregations-pipeline-movavg-aggregation.html[moving average] aggregation will slide a window across the data and emit the average value of that window

Serial Diff

The {ref}/search-aggregations-pipeline-serialdiff-aggregation.html[serial differencing] is a technique where values in a time series are subtracted from itself at different time lags or period

Sibling Pipeline Aggregations:

Just like with parent pipeline aggregations you need to provide a metric for which to calculate the sibling aggregation. On top of that you also need to provide a bucket aggregation which will define the buckets on which the sibling aggregation will run

Average Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[avg bucket] calculates the (mean) average value of a specified metric in a sibling aggregation

Sum Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[sum bucket] calculates the sum of values of a specified metric in a sibling aggregation

Min Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[min bucket] calculates the minimum value of a specified metric in a sibling aggregation

Max Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[max bucket] calculates the maximum value of a specified metric in a sibling aggregation

You can add an aggregation by clicking the + Add Metrics button.

Enter a string in the Custom Label field to change the display label.

The buckets aggregations determine what information is being retrieved from your data set.

Before you choose a buckets aggregation, select the Split Tags option.

You can specify the following bucket aggregations for tag cloud visualization:

Terms

A {ref}/search-aggregations-bucket-terms-aggregation.html[terms] aggregation enables you to specify the top or bottom n elements of a given field to display, ordered by count or a custom metric.

You can click the Advanced link to display more customization options for your metrics or bucket aggregation:

JSON Input

A text field where you can add specific JSON-formatted properties to merge with the aggregation definition, as in the following example:

{ "script" : "doc['grade'].value * 1.2" }
In Elasticsearch releases 1.4.3 and later, this functionality requires you to enable {ref}/modules-scripting.html[dynamic Groovy scripting].

Select the Options tab to change the following aspects of the chart:

Text Scale

You can select linear, log, or square root scales for the text scale. You can use a log scale to display data that varies exponentially or a square root scale to regularize the display of data sets with variabilities that are themselves highly variable.

Orientation

You can select how to orientate your text in the tag cloud. You can choose one of the following options: Single, right angles and multiple.

Font Size

Allows you to set minimum and maximum font size to use for this visualization.

Visualization Spy

To display the raw data behind the visualization, click the spy-open-button button in the bottom left corner of the container. The visualization spy panel will open. Use the select input to view detailed information about the raw data.

spy-panel

Table

A representation of the underlying data, presented as a paginated data grid. You can sort the items in the table by clicking on the table headers at the top of each column.

Request

The raw request used to query the server, presented in JSON format.

Response

The raw response from the server, presented in JSON format.

Statistics

A summary of the statistics related to the request and the response, presented as a data grid. The data grid includes the query duration, the request duration, the total number of records found on the server, and the index pattern used to make the query.

Debug

The visualization saved state presented in JSON format.

To export the raw data behind the visualization as a comma-separated-values (CSV) file, click on either the Raw or Formatted links at the bottom of any of the detailed information tabs. A raw export contains the data as it is stored in Elasticsearch. A formatted export contains the results of any applicable field formatters.

Heatmap Chart

A heat map is a graphical representation of data where the individual values contained in a matrix are represented as colors. The color for each matrix position is determined by the metrics aggregation. The following aggregations are available for this chart:

Metric Aggregations:

Count

The {ref}/search-aggregations-metrics-valuecount-aggregation.html[count] aggregation returns a raw count of the elements in the selected index pattern.

Average

This aggregation returns the {ref}/search-aggregations-metrics-avg-aggregation.html[average] of a numeric field. Select a field from the drop-down.

Sum

The {ref}/search-aggregations-metrics-sum-aggregation.html[sum] aggregation returns the total sum of a numeric field. Select a field from the drop-down.

Min

The {ref}/search-aggregations-metrics-min-aggregation.html[min] aggregation returns the minimum value of a numeric field. Select a field from the drop-down.

Max

The {ref}/search-aggregations-metrics-max-aggregation.html[max] aggregation returns the maximum value of a numeric field. Select a field from the drop-down.

Unique Count

The {ref}/search-aggregations-metrics-cardinality-aggregation.html[cardinality] aggregation returns the number of unique values in a field. Select a field from the drop-down.

Standard Deviation

The {ref}/search-aggregations-metrics-extendedstats-aggregation.html[extended stats] aggregation returns the standard deviation of data in a numeric field. Select a field from the drop-down.

Percentiles

The {ref}/search-aggregations-metrics-percentile-aggregation.html[percentile] aggregation divides the values in a numeric field into percentile bands that you specify. Select a field from the drop-down, then specify one or more ranges in the Percentiles fields. Click the X to remove a percentile field. Click + Add to add a percentile field.

Percentile Rank

The {ref}/search-aggregations-metrics-percentile-rank-aggregation.html[percentile ranks] aggregation returns the percentile rankings for the values in the numeric field you specify. Select a numeric field from the drop-down, then specify one or more percentile rank values in the Values fields. Click the X to remove a values field. Click +Add to add a values field.

Parent Pipeline Aggregations:

For each of the parent pipeline aggregations you have to define the metric for which the aggregation is calculated. That could be one of your existing metrics or a new one. You can also nest this aggregations (for example to produce 3rd derivative)

Derivative

The {ref}/search-aggregations-pipeline-derivative-aggregation.html[derivative] aggregation calculates the derivative of specific metrics.

Cumulative Sum

The {ref}/search-aggregations-pipeline-cumulative-sum-aggregation.html[cumulative sum] aggregation calculates the cumulative sum of a specified metric in a parent histogram

Moving Average

The {ref}/search-aggregations-pipeline-movavg-aggregation.html[moving average] aggregation will slide a window across the data and emit the average value of that window

Serial Diff

The {ref}/search-aggregations-pipeline-serialdiff-aggregation.html[serial differencing] is a technique where values in a time series are subtracted from itself at different time lags or period

Sibling Pipeline Aggregations:

Just like with parent pipeline aggregations you need to provide a metric for which to calculate the sibling aggregation. On top of that you also need to provide a bucket aggregation which will define the buckets on which the sibling aggregation will run

Average Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[avg bucket] calculates the (mean) average value of a specified metric in a sibling aggregation

Sum Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[sum bucket] calculates the sum of values of a specified metric in a sibling aggregation

Min Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[min bucket] calculates the minimum value of a specified metric in a sibling aggregation

Max Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[max bucket] calculates the maximum value of a specified metric in a sibling aggregation

You can add an aggregation by clicking the + Add Metrics button.

Enter a string in the Custom Label field to change the display label.

The buckets aggregations determine what information is being retrieved from your data set.

Before you choose a buckets aggregation, specify if you are defining buckets for X or Y axis within a single chart or splitting into multiple charts. A multiple chart split must run before any other aggregations. When you split a chart, you can change if the splits are displayed in a row or a column by clicking the Rows | Columns selector.

This chart’s X and Y axis supports the following aggregations. Click the linked name of each aggregation to visit the main Elasticsearch documentation for that aggregation.

Date Histogram

A {ref}/search-aggregations-bucket-datehistogram-aggregation.html[date histogram] is built from a numeric field and organized by date. You can specify a time frame for the intervals in seconds, minutes, hours, days, weeks, months, or years. You can also specify a custom interval frame by selecting Custom as the interval and specifying a number and a time unit in the text field. Custom interval time units are s for seconds, m for minutes, h for hours, d for days, w for weeks, and y for years. Different units support different levels of precision, down to one second. Intervals are labeled at the start of the interval, using the date-key returned by Elasticsearch. For example, the tooltip for a monthly interval will show the first day of the month.

Histogram

A standard {ref}/search-aggregations-bucket-histogram-aggregation.html[histogram] is built from a numeric field. Specify an integer interval for this field. Select the Show empty buckets checkbox to include empty intervals in the histogram.

Range

With a {ref}/search-aggregations-bucket-range-aggregation.html[range] aggregation, you can specify ranges of values for a numeric field. Click Add Range to add a set of range endpoints. Click the red (x) symbol to remove a range.

Date Range

A {ref}/search-aggregations-bucket-daterange-aggregation.html[date range] aggregation reports values that are within a range of dates that you specify. You can specify the ranges for the dates using {ref}/common-options.html#date-math[date math] expressions. Click Add Range to add a set of range endpoints. Click the red (x) symbol to remove a range.

IPv4 Range

The {ref}/search-aggregations-bucket-iprange-aggregation.html[IPv4 range] aggregation enables you to specify ranges of IPv4 addresses. Click Add Range to add a set of range endpoints. Click the red (x) symbol to remove a range.

Terms

A {ref}/search-aggregations-bucket-terms-aggregation.html[terms] aggregation enables you to specify the top or bottom n elements of a given field to display, ordered by count or a custom metric.

Filters

You can specify a set of {ref}/search-aggregations-bucket-filters-aggregation.html[filters] for the data. You can specify a filter as a query string or in JSON format, just as in the Discover search bar. Click Add Filter to add another filter. Click the Label button icon label button to open the label field, where you can type in a name to display on the visualization.

Significant Terms

Displays the results of the experimental {ref}/search-aggregations-bucket-significantterms-aggregation.html[significant terms] aggregation.

Enter a string in the Custom Label field to change the display label.

You can click the Advanced link to display more customization options for your metrics or bucket aggregation:

Exclude Pattern

Specify a pattern in this field to exclude from the results.

Include Pattern

Specify a pattern in this field to include in the results.

JSON Input

A text field where you can add specific JSON-formatted properties to merge with the aggregation definition, as in the following example:

{ "script" : "doc['grade'].value * 1.2" }

The availability of these options varies depending on the aggregation you choose.

Select the Options tab to change the following aspects of the chart:

Show Tooltips

Check this box to enable the display of tooltips.

Highlight

Check this box to enable highlighting of elements with same label

Legend Position

You can select where to display the legend (top, left, right, bottom)

Color Schema

You can select an existing color schema or go for custom and define your own colors in the legend

Reverse Color Schema

Checking this checkbox will reverse the color schema.

Color Scale

You can switch between linear, log and sqrt scales for color scale.

Scale to Data Bounds

The default Y axis bounds are zero and the maximum value returned in the data. Check this box to change both upper and lower bounds to match the values returned in the data.

Number of Colors

Number of color buckets to create. Minimum is 2 and maximum is 10.

Percentage Mode

Enabling this will show legend values as percentages.

Custom Range

You can define custom ranges for your color buckets. For each of the color bucket you need to specify the minimum value (inclusive) and the maximum value (exclusive) of a range.

Show Label

Enables showing labels with cell values in each cell

Rotate

Allows rotating the cell value label by 90 degrees.

Visualization Spy

To display the raw data behind the visualization, click the spy-open-button button in the bottom left corner of the container. The visualization spy panel will open. Use the select input to view detailed information about the raw data.

spy-panel

Table

A representation of the underlying data, presented as a paginated data grid. You can sort the items in the table by clicking on the table headers at the top of each column.

Request

The raw request used to query the server, presented in JSON format.

Response

The raw response from the server, presented in JSON format.

Statistics

A summary of the statistics related to the request and the response, presented as a data grid. The data grid includes the query duration, the request duration, the total number of records found on the server, and the index pattern used to make the query.

Debug

The visualization saved state presented in JSON format.

To export the raw data behind the visualization as a comma-separated-values (CSV) file, click on either the Raw or Formatted links at the bottom of any of the detailed information tabs. A raw export contains the data as it is stored in Elasticsearch. A formatted export contains the results of any applicable field formatters.

Coordinate Maps

A coordinate map displays a geographic area overlaid with circles keyed to the data determined by the buckets you specify.

By default, Siren Investigate uses the Open Street Maps service to display map tiles. To use other tile service providers, configure the tilemap settings in siren.yml.

Configuration

Data

Metrics

The default metrics aggregation for a coordinate map is the Count aggregation. You can select any of the following aggregations as the metrics aggregation:

Count

The {ref}/search-aggregations-metrics-valuecount-aggregation.html[count] aggregation returns a raw count of the elements in the selected index pattern.

Average

This aggregation returns the {ref}/search-aggregations-metrics-avg-aggregation.html[average] of a numeric field. Select a field from the drop-down.

Sum

The {ref}/search-aggregations-metrics-sum-aggregation.html[sum] aggregation returns the total sum of a numeric field. Select a field from the drop-down.

Min

The {ref}/search-aggregations-metrics-min-aggregation.html[min] aggregation returns the minimum value of a numeric field. Select a field from the drop-down.

Max

The {ref}/search-aggregations-metrics-max-aggregation.html[max] aggregation returns the maximum value of a numeric field. Select a field from the drop-down.

Unique Count

The {ref}/search-aggregations-metrics-cardinality-aggregation.html[cardinality] aggregation returns the number of unique values in a field. Select a field from the drop-down.

Enter a string in the Custom Label field to change the display label.

Buckets

Coordinate maps use the geohash aggregation. Select a field, typically coordinates, from the drop-down.

  • The_Change precision on map zoom_ box is checked by default. Uncheck the box to disable this behavior. The Precision slider determines the granularity of the results displayed on the map. See the documentation for the {ref}/search-aggregations-bucket-geohashgrid-aggregation.html#_cell_dimensions_at_the_equator[geohash grid] aggregation for details on the area specified by each precision level.

Higher precisions increase memory usage for the browser displaying Siren Investigate as well as for the underlying Elasticsearch cluster.
  • The place markers off grid (use {ref}/search-aggregations-metrics-geocentroid-aggregation.html[geocentroid]) box is checked by default. When this box is checked, the markers are placed in the center of all the documents in that bucket. When unchecked, the markers are placed in the center of the geohash grid cell. Leaving this checked generally results in a more accurate visualization.

Enter a string in the Custom Label field to change the display label.

You can click the Advanced link to display more customization options for your metrics or bucket aggregation:

Exclude Pattern

Specify a pattern in this field to exclude from the results.

Include Pattern

Specify a pattern in this field to include in the results.

JSON Input

A text field where you can add specific JSON-formatted properties to merge with the aggregation definition, as in the following example:

{ "script" : "doc['grade'].value * 1.2" }
In Elasticsearch releases 1.4.3 and later, this functionality requires you to enable {ref}/modules-scripting.html[dynamic Groovy scripting].

The availability of these options varies depending on the aggregation you choose.

Options

Map type

Select one of the following options from the drop-down.

Scaled Circle Markers

Scale the size of the markers based on the metric aggregation’s value.

Shaded Circle Markers

Displays the markers with different shades based on the metric aggregation’s value.

Shaded Geohash Grid

Displays the rectangular cells of the geohash grid instead of circular markers, with different shades based on the metric aggregation’s value.

Heatmap

A heat map applies blurring to the circle markers and applies shading based on the amount of overlap. Heatmaps have the following options:

  • Radius: Sets the size of the individual heatmap dots.

  • Blur: Sets the amount of blurring for the heatmap dots.

  • Maximum zoom: Tilemaps in Siren Investigate support 18 zoom levels. This slider defines the maximum zoom level at which the heatmap dots appear at full intensity.

  • Minimum opacity: Sets the opacity cutoff for the dots.

  • Show Tooltip: Check this box to have a tooltip with the values for a given dot when the cursor is on that dot.

Desaturate map tiles

Desaturate the map’s color in order to make the markers stand out more clearly.

WMS compliant map server

Check this box to enable the use of a third-party mapping service that complies with the Web Map Service (WMS) standard. Specify the following elements:

  • WMS url: The URL for the WMS map service.

  • WMS layers: A comma-separated list of the layers to use in this visualization. Each map server provides its own list of layers.

  • WMS version: The WMS version used by this map service.

  • WMS format: The image format used by this map service. The two most common formats are image/png and image/jpeg.

  • WMS attribution: An optional, user-defined string that identifies the map source. Maps display the attribution string in the lower right corner.

  • WMS styles: A comma-separated list of the styles to use in this visualization. Each map server provides its own styling options.

After changing options, click the Apply changes button to update your visualization, or the grey Discard changes button to keep your visualization in its current state.

Once your tilemap visualization is ready, you can explore the map in several ways:

  • Click and hold anywhere on the map and move the cursor to move the map center. Hold Shift and drag a bounding box across the map to zoom in on the selection.

  • Click the Zoom In/Out viz-zoom buttons to change the zoom level manually.

  • Click the Fit Data Bounds viz-fit-bounds button to automatically crop the map boundaries to the geohash buckets that have at least one result.

  • Click the Latitude/Longitude Filter viz-lat-long-filter button, then drag a bounding box across the map, to create a filter for the box coordinates.

Viewing Detailed Information

Visualization Spy

To display the raw data behind the visualization, click the spy-open-button button in the bottom left corner of the container. The visualization spy panel will open. Use the select input to view detailed information about the raw data.

spy-panel

Table

A representation of the underlying data, presented as a paginated data grid. You can sort the items in the table by clicking on the table headers at the top of each column.

Request

The raw request used to query the server, presented in JSON format.

Response

The raw response from the server, presented in JSON format.

Statistics

A summary of the statistics related to the request and the response, presented as a data grid. The data grid includes the query duration, the request duration, the total number of records found on the server, and the index pattern used to make the query.

Debug

The visualization saved state presented in JSON format.

To export the raw data behind the visualization as a comma-separated-values (CSV) file, click on either the Raw or Formatted links at the bottom of any of the detailed information tabs. A raw export contains the data as it is stored in Elasticsearch. A formatted export contains the results of any applicable field formatters.

Line, Area, and Bar charts

Line, Area, and Bar charts allow you to plot your data on X/Y axis.

First you need to select your metrics which define Value axis.

Metric Aggregations:

Count

The {ref}/search-aggregations-metrics-valuecount-aggregation.html[count] aggregation returns a raw count of the elements in the selected index pattern.

Average

This aggregation returns the {ref}/search-aggregations-metrics-avg-aggregation.html[average] of a numeric field. Select a field from the drop-down.

Sum

The {ref}/search-aggregations-metrics-sum-aggregation.html[sum] aggregation returns the total sum of a numeric field. Select a field from the drop-down.

Min

The {ref}/search-aggregations-metrics-min-aggregation.html[min] aggregation returns the minimum value of a numeric field. Select a field from the drop-down.

Max

The {ref}/search-aggregations-metrics-max-aggregation.html[max] aggregation returns the maximum value of a numeric field. Select a field from the drop-down.

Unique Count

The {ref}/search-aggregations-metrics-cardinality-aggregation.html[cardinality] aggregation returns the number of unique values in a field. Select a field from the drop-down.

Standard Deviation

The {ref}/search-aggregations-metrics-extendedstats-aggregation.html[extended stats] aggregation returns the standard deviation of data in a numeric field. Select a field from the drop-down.

Percentiles

The {ref}/search-aggregations-metrics-percentile-aggregation.html[percentile] aggregation divides the values in a numeric field into percentile bands that you specify. Select a field from the drop-down, then specify one or more ranges in the Percentiles fields. Click the X to remove a percentile field. Click + Add to add a percentile field.

Percentile Rank

The {ref}/search-aggregations-metrics-percentile-rank-aggregation.html[percentile ranks] aggregation returns the percentile rankings for the values in the numeric field you specify. Select a numeric field from the drop-down, then specify one or more percentile rank values in the Values fields. Click the X to remove a values field. Click +Add to add a values field.

Parent Pipeline Aggregations:

For each of the parent pipeline aggregations you have to define the metric for which the aggregation is calculated. That could be one of your existing metrics or a new one. You can also nest this aggregations (for example to produce 3rd derivative)

Derivative

The {ref}/search-aggregations-pipeline-derivative-aggregation.html[derivative] aggregation calculates the derivative of specific metrics.

Cumulative Sum

The {ref}/search-aggregations-pipeline-cumulative-sum-aggregation.html[cumulative sum] aggregation calculates the cumulative sum of a specified metric in a parent histogram

Moving Average

The {ref}/search-aggregations-pipeline-movavg-aggregation.html[moving average] aggregation will slide a window across the data and emit the average value of that window

Serial Diff

The {ref}/search-aggregations-pipeline-serialdiff-aggregation.html[serial differencing] is a technique where values in a time series are subtracted from itself at different time lags or period

Sibling Pipeline Aggregations:

Just like with parent pipeline aggregations you need to provide a metric for which to calculate the sibling aggregation. On top of that you also need to provide a bucket aggregation which will define the buckets on which the sibling aggregation will run

Average Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[avg bucket] calculates the (mean) average value of a specified metric in a sibling aggregation

Sum Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[sum bucket] calculates the sum of values of a specified metric in a sibling aggregation

Min Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[min bucket] calculates the minimum value of a specified metric in a sibling aggregation

Max Bucket

The {ref}/search-aggregations-pipeline-avg-bucket-aggregation.html[max bucket] calculates the maximum value of a specified metric in a sibling aggregation

You can add an aggregation by clicking the + Add Metrics button.

Enter a string in the Custom Label field to change the display label.

The buckets aggregations determine what information is being retrieved from your data set.

Before you choose a buckets aggregation, specify if you are splitting slices within a single chart or splitting into multiple charts. A multiple chart split must run before any other aggregations. When you split a chart, you can change if the splits are displayed in a row or a column by clicking the Rows | Columns selector.

The X axis of this chart is the buckets axis. You can define buckets for the X axis, for a split area on the chart, or for split charts.

This chart’s X axis supports the following aggregations. Click the linked name of each aggregation to visit the main Elasticsearch documentation for that aggregation.

Date Histogram

A {ref}/search-aggregations-bucket-datehistogram-aggregation.html[date histogram] is built from a numeric field and organized by date. You can specify a time frame for the intervals in seconds, minutes, hours, days, weeks, months, or years. You can also specify a custom interval frame by selecting Custom as the interval and specifying a number and a time unit in the text field. Custom interval time units are s for seconds, m for minutes, h for hours, d for days, w for weeks, and y for years. Different units support different levels of precision, down to one second. Intervals are labeled at the start of the interval, using the date-key returned by Elasticsearch. For example, the tooltip for a monthly interval will show the first day of the month.

Histogram

A standard {ref}/search-aggregations-bucket-histogram-aggregation.html[histogram] is built from a numeric field. Specify an integer interval for this field. Select the Show empty buckets checkbox to include empty intervals in the histogram.

Range

With a {ref}/search-aggregations-bucket-range-aggregation.html[range] aggregation, you can specify ranges of values for a numeric field. Click Add Range to add a set of range endpoints. Click the red (x) symbol to remove a range.

Date Range

A {ref}/search-aggregations-bucket-daterange-aggregation.html[date range] aggregation reports values that are within a range of dates that you specify. You can specify the ranges for the dates using {ref}/common-options.html#date-math[date math] expressions. Click Add Range to add a set of range endpoints. Click the red (x) symbol to remove a range.

IPv4 Range

The {ref}/search-aggregations-bucket-iprange-aggregation.html[IPv4 range] aggregation enables you to specify ranges of IPv4 addresses. Click Add Range to add a set of range endpoints. Click the red (x) symbol to remove a range.

Terms

A {ref}/search-aggregations-bucket-terms-aggregation.html[terms] aggregation enables you to specify the top or bottom n elements of a given field to display, ordered by count or a custom metric.

Filters

You can specify a set of {ref}/search-aggregations-bucket-filters-aggregation.html[filters] for the data. You can specify a filter as a query string or in JSON format, just as in the Discover search bar. Click Add Filter to add another filter. Click the Label button icon label button to open the label field, where you can type in a name to display on the visualization.

Significant Terms

Displays the results of the experimental {ref}/search-aggregations-bucket-significantterms-aggregation.html[significant terms] aggregation.

External query terms filter

a Siren Investigate aggregator where one can define one or more buckets based on some record value (typically a primary key) matching the results of an external query. Multiple such buckets, corresponding to multiple queries, can be defined. For more information see the query menu in the configuration. This displays the results of the external query terms filter aggregation.

Once you’ve specified an X axis aggregation, you can define sub-aggregations to refine the visualization. Click + Add Sub Aggregation to define a sub-aggregation, then choose Split Area or Split Chart, then select a sub-aggregation from the list of types.

When multiple aggregations are defined on a chart’s axis, you can use the up or down arrows to the right of the aggregation’s type to change the aggregation’s priority.

Enter a string in the Custom Label field to change the display label.

You can customize the colors of your visualization by clicking the color dot next to each label to display the color picker.

An array of color dots that users can select

Enter a string in the Custom Label field to change the display label.

You can click the Advanced link to display more customization options for your metrics or bucket aggregation:

Exclude Pattern

Specify a pattern in this field to exclude from the results.

Include Pattern

Specify a pattern in this field to include in the results.

JSON Input

A text field where you can add specific JSON-formatted properties to merge with the aggregation definition, as in the following example:

{ "script" : "doc['grade'].value * 1.2" }
In Elasticsearch releases 1.4.3 and later, this functionality requires you to enable {ref}/modules-scripting.html[dynamic Groovy scripting].

The availability of these options varies depending on the aggregation you choose.

Metrics & Axes

Select the Metrics & Axes tab to change the way each individual metric is shown on the chart. The data series are styled in the Metrics section, while the axes are styled in the X and Y axis sections.

Metrics

Modify how each metric from the Data panel is visualized on the chart.

Chart type

Choose between Area, Line, and Bar types.

Mode

stack the different metrics, or plot them next to each other

Value Axis

choose the axis you want to plot this data too (the properties of each are configured under Y-axes).

Line mode

should the outline of lines or bars appear smooth, straight, or stepped.

Y-axis

Style all the Y-axes of the chart.

Position

position of the Y-axis (left or right for vertical charts, and top or bottom for horizontal charts).

Scale type

scaling of the values (linear, log, or square root)

Advanced Options
Labels - Show Labels

Allows you to hide axis labels

Labels - Filter Labels

If filter labels is enabled some labels will be hidden in case there is not enough space to display them

Labels - Rotate

You can enter the number in degrees for how much you want to rotate labels

Labels - Truncate

You can enter the size in pixels to which the label is truncated

Scale to Data Bounds

The default Y-axis bounds are zero and the maximum value returned in the data. Check this box to change both upper and lower bounds to match the values returned in the data.

Custom Extents

You can define custom minimum and maximum for each axis

X-Axis

Position

position of the X-Axis (left or right for horizontal charts, and top or bottom for vertical charts).

Advanced Options
Labels - Show Labels

Allows you to hide axis labels

Labels - Filter Labels

If filter labels is enabled some labels will be hidden in case there is not enough spave to display them

Labels - Rotate

You can enter the number in degrees for how much you want to rotate labels

Labels - Truncate

You can enter the size in pixels to which the label is truncated

Panel Settings

These are options that apply to the entire chart and not just the individual data series.

Common options

Legend Position

Move your legend to the left, right, top or bottom

Show Tooltip

Enables or disables the display of tooltip on hovering over chart objects

Current Time Marker

Show a line indicating the current time

Grid options

You can enable grid on the chart. By default grid is displayed on the category axis only.

X-axis

You can disable the display of grid lines on category axis

Y-axis

You can choose on which (if any) of the value axes you want to display grid lines

Viewing Detailed Information

Visualization Spy

To display the raw data behind the visualization, click the spy-open-button button in the bottom left corner of the container. The visualization spy panel will open. Use the select input to view detailed information about the raw data.

spy-panel

Table

A representation of the underlying data, presented as a paginated data grid. You can sort the items in the table by clicking on the table headers at the top of each column.

Request

The raw request used to query the server, presented in JSON format.

Response

The raw response from the server, presented in JSON format.

Statistics

A summary of the statistics related to the request and the response, presented as a data grid. The data grid includes the query duration, the request duration, the total number of records found on the server, and the index pattern used to make the query.

Debug

The visualization saved state presented in JSON format.

To export the raw data behind the visualization as a comma-separated-values (CSV) file, click on either the Raw or Formatted links at the bottom of any of the detailed information tabs. A raw export contains the data as it is stored in Elasticsearch. A formatted export contains the results of any applicable field formatters.

Siren Investigate Timeline

The Siren Investigate Timeline visualization displays series of data coming from different saved searches on a single timeline component. Events are color coded to distinguish between different groups.

Each event on a timeline become a clickable term filter which allow to quickly filter the related data based on what is shown on the timeline.

Timeline

Configuration

To configure the visualization, add a new Group and select:

  • Saved search id - date for this group will be taken from corresponding index.

  • Group label - a label for the group.

  • Event label field - field value will be used as individual event label.

  • Event start date - date from this field will be used to position start of the event.

  • Event end date - (optional) date from this field will be used to position end of the event.

  • Events number limit - (optional) limit number of events in this group.

Timeline configuration

Advanced option

By default events from multiple groups are rendered all mixed together. It is possible to show different groups on different levels by enabling the advanced option

  • Groups rendered on separate levels

Timeline advanced configuration

Below timeline where each group is rendered on separate level

Timeline

Siren Investigate Scatter Plot

This visualization displays a scatter plot chart in four different modes: Straight, Significant terms, Any aggregator, Filtered aggregator.

Straight

Straight

This mode doesn’t use aggregates, it pulls the data directly from Elasticsearch using the Random scoring method to get a random sample of records.

  • X values - The value can be String, Date or Numeric. Select a field from the drop-down.

  • Y values - The field value can be String, Date or Numeric. Select a field from the drop-down.

  • X axis label - A label for the X axis.

  • Y axis label - A label for the Y axis.

  • X axis scale - You can select linear, log, or square root scales for the chart’s X axis. You can use a log scale to display data that varies exponentially, such as a compounding interest chart, or a square root scale to regularize the display of data sets with variabilities that are themselves highly variable. This kind of data, where the variability is itself variable over the domain being examined, is known as heteroscedastic data. For example, if a data set of height versus weight has a relatively narrow range of variability at the short end of height, but a wider range at the taller end, the data set is heteroscedastic.

  • Y axis scale - You can select linear, log, or square root scales for the chart’s Y axis.

  • Jitter field - Deterministic jitter to add pseudo random data distribution in the X axis data interval. Jitter is useful for distributing the values across X axis. Doing so allows to show the data distributed across the bucket, in that way the dot is more visible.

  • Jitter scale - You can select linear, log, or square root scales for the Jitter.

  • Label - A label for the dot.

    • Display label - Check this box to enable the display of a label next to the dot.

    • Label hover effect - Check this box to enable the tooltip label.

  • Color - A color for the dot.

  • Color field - The field used as an input to generate the dot colors. Only number field types are allowed.

  • Dot size - A size for the dot.

  • Dot size field - The field used as an input for the dot size. Only number field types are allowed.

  • Dot size scale - You can select linear, log, or square root scales for the dot size.

  • Size - Number of random records to fetch from Eleasticsearch query.

  • Shape opacity - Value from 0 to 1 which defines the dot transparency.

Significant terms

Significant term

In this mode the chart is built from a Significant terms aggregation query result. The X values are taken from the bg_count field and the Y values from doc_count field.

  • Field - the field which will provide terms to be aggregated.

  • Size - the number of significant terms to be aggregated.

  • X axis label - A label for the X axis.

  • Y axis label - A label for the Y axis.

  • Color - A color for the dot.

  • Shape opacity - Value from 0 to 1 which defines the dot transparency.

Any aggregator

Any aggregator

The chart is built from a Date Histogram, Histogram, Terms or Significant terms aggregation query result.

  • Aggregation - Select an aggregation from the drop-down list.

  • X Metric - X axis values. Select a metric from the drop-down list.

  • Y Metric - Y axis values. Select a metric from the drop-down list.

  • Color - A color for the dot.

  • Dot size - A size for the dot.

  • Shape opacity - Value from 0 to 1 which defines the dot transparency.

Filtered aggregator

Filtered aggregator

The chart is built from a Date Histogram, Histogram, Terms or Significant terms aggregation query result. The X and Y values are taken from Filters aggregation results.

  • Aggregation - Select an aggregation from the drop-down list.

  • Filter X - A filter string for the X axis.

  • Filter Y - A filter string for the Y axis.

  • Metric - Metric to be calculated for each filter aggregation. Select a metric from the drop-down list.

  • Color - A color for the dot.

  • Dot size - A size for the dot.

  • Shape opacity - Value from 0 to 1 which defines the dot transparency.

After changing options, click the green Apply changes button to update your visualization, or the grey Discard changes button to return your visualization to its previous state.

Radar Chart

A radar chart introduced in kibi-0.3.0 is a graphical method of displaying multivariate data in the form of a two-dimensional chart of three or more quantitative variables represented on axes starting from the same point. The relative position and angle of the axes is typically uninformative.

Radar chart visualization
Radar chart settings

The radar chart is also known as web chart, spider chart, star chart. It is developed as a standalone plugin suitable to install in Kibi 0.3+, Siren Investigate 10+ and Kibana 4.3+.

Siren Investigate Graph Browser

The Siren Investigate Graph Browser displays Elasticsearch documents as nodes and Siren Investigate relations as links of a graph.

Graph Browser Example
Figure 1. Graph Browser Example

Configuration

To configure the visualization select a compatible datasource (typically the provided Kibi Gremlin Server datasource)

Source configuration
Figure 2. Source configuration

Tooltip

By ticking the Tooltip checkbox checkbox, you can enable node tooltips.

Tooltip
Figure 3. Tooltip

Relations

By default the Graph Browser uses all the configured relations. In case you need to use only a subset of the relations, you can define it here.

Relations
Figure 4. Relations

Scripts

The Graph Browser supports three types of scripts:

  • Expansion - These scripts are used to customize the expansion policy. The provided one (Default Expansion Policy) will retrieve the first level connected elements to the expanded nodes

  • Contextual - These scripts show up in the contextual menu (shown with a RIGHT CLICK) and allow you to perform operations on the graph.
    Provided contextual scripts:

    • Expand by relation - It opens a popup which allows you to choose one or more of the available relations and it expands the selected elements using only the selected ones. This does not override the graph browser configuration, you will see only the configured relations (if available for the selected nodes)

    • Expand by top comention - To be used with company nodes from our demo. This script expands the selected nodes using an elasticsearch aggregation to get the top comentioned company nodes

    • Replace investment with edge - To be used with our demo. This script replaces the investment nodes with a direct link between the company nodes and the investor nodes

    • Select - All - Select all the elements. Same as CTRL + A

    • Select - By edge count - It select nodes based on their link count. You can specify the count through the popup that appears

    • Select - By type - It select nodes based on their type. You can specify the type through the popup that appears

    • Select - Extend - Extends the current selection to the sibling elements

    • Selecte - Invert - Inverts the current selection

    • Shortest Path - Calculates the shortest path between two selected nodes by fetching the connected elements.

    • Show nodes count by type - Shows a popup with information about how many nodes per type are currently displayed

  • On Update - These scripts are executed every time you add new elements to the graph.
    Provided on update scripts:

    • Add time fields - Adds the time field used by the timebar mode

    • Add geo-locations for map visualization - Adds the geographic field used by the map mode

    • Replace investment with edge - Same as the contextual script Replace investment with edge, but executed automatically after every expansion.

    • Signal dead companies - Changes to black all the company nodes that have a deadpooled_date

To create a new script go to ManagementScripts

Scripts Management
Figure 5. Scripts Management

Here you can configure new scripts or modify the saved ones.

Vertices UI Configuration

To configure a vertex, press the Add Vertex:

  • Index Pattern - the index pattern containing the type of documents you want to configure

  • Index Pattern Type - the index pattern type of the documents you want to configure

  • Icon Type - the type of icon you want to use

    • Font awesome - the icon will be selected using the vector icons from Font Awesome

    • Parameterized Relative Path - the icon will be selected from the configured relative path. The relative path points to <kibi_root>/installedPlugins/kibi_graph_browser_vis/public/icons/. You can create subfolders to better organize your custom icons.
      Examples:

      • If the icon filename is stored inside your document, you can use a parameterized relative path. This label: /set1/@[_doc][iconType]@.png will be evaluated using the iconType field of the document.

  • Label type - the type of label you want to use

    • Document Field - the label will be a document field. You can select one of the available fields from the dropdown

    • Parameterized Field - the label will be the evaluation of this parameterized field.
      Examples:

      • Simple label, with the same output as for the document field one: Company: @doc[_source][companyName]@

      • You can compose the label using multiple fields: @[_doc][company_name]@ from @[_doc][country]@.

      • To have a multi-line label just add a new line character (\n): @[_doc][companyName]@ \n @[_doc][country]@

Vertex Configuration
Figure 6. Vertex Configuration

Navigating the Graph

Once your Siren Investigate Graph Browser visualization is ready, you can start your investigations.

Toolbar

You have several operations available:

Toolbar
Figure 7. Toolbar
  1. Undo - By default the graph browser saves the last 5 states. With this function you can go back one step at a time, until there are no more available. You can configure the steps number in kibi advanced settings.

  2. Redo - With the redo you can restore an undoed state. Be careful: if you undo and perform any operation, the redo state will be lost.

  3. Filter - This will add a filter to the current dashboard synced with the graph selection. This lets you:

    • Do your investigation on the graph, select the vertices you’re interested into, activate the filter, pin it and go back to the related dashboard to get more detailed information about those vertices.

    • If you have other visualizations in the same dashboard it lets you have more information on the selected nodes. For example, if you have the current dashboard associated to a companies saved search, you can do your investigation in the graph, activate the filter, select some vertices and get the visualizations to show information on the selected vertices.

  4. Crop - This will delete every element that is not selected

  5. Remove - This will remove all the selected elements. Right next to the Remove button there is a dropdown that shows the Remove All button. This will clean the whole graph, regardless of selected elements or not.

Remove All
Figure 8. Remove All
  1. Expand - This will expand the currently selected nodes. Right next to the expand button there is a dropdown that shows advanced options for the expansion. The advanced options let you configure if you want to use the dashboard filters with the graph expansions.
    Eg: You have a filter on the Companies dashboard that filters out every non-US company. By using that filter within the graph browser you will expand only companies from within the US.

Expand with filters
Figure 9. Expand with filters
  1. Highlight mode - This toggle enables and disables the Highlight mode. The Highlight mode moves to the background everything that is not selected and/or connected to a selected node/link.

Highlightning On
Figure 10. Highlightning On
Highlightning Off
Figure 11. Highlightning Off
  1. Layouts - This button lets you change the current graph’s layout. There are 2 available layouts:

    • Standard - This one is the standard layout used by the graph. Pressing it will force the graph to relayout. Note: selected nodes will preserve their relative position.

    • Hierarchy - This layout lays out nodes top down according to their connections. Note: It needs at least one selected node to work; selected nodes will be moved at the top of the hierarchy.

Standard Layout
Figure 12. Standard Layout
Hierarchy Layout
Figure 13. Hierarchy Layout
  1. Add - The Add button opens a popup with the following options:

    • Add selected document - This will add the currently selected document. You can see your selected document in the upper right purple selection box.Standard Layout

    • Add from saved graph - This will open a popup showing the available saved graphs. By using this feature you will add a set of nodes and links, but you won’t preserve the layout you had when you saved the graph.

    • Add from another dashboard - This will add nodes using the filtered (optionally) dashboard you select.

Add from saved graph
Figure 14. Add from saved graph
  1. Map Mode - This toggle enables or disables the Map mode. The Map mode will move the nodes geographically on an interactive map. You will need to set up a script to configure the geographic properties of the nodes (See Scripts).

Map mode
Figure 15. Map mode
  1. Timebar Mode - This toggle enables or disables the Timebar mode. The Timebar mode will display a timebar at the bottom of the Graph Browser that allows time based filtering of nodes. Once you enable this mode you will be able to add/remove node types to the timebar through the new menu: Timebar Filter
    You will need to set up a script to configure the time property of the nodes (See Scripts).

Timebar mode
Figure 16. Timebar mode
  1. Save Graph - This buttons opens a popup that lets you save the current graph.

Save Graph
Figure 17. Save Graph
  1. Open Graph - This button opens a popup that lets you open a saved graph. Note: unlike the add from saved graph this feature preserves the saved graph layout.

Open Graph
Figure 18. Open Graph

Shortcuts

The Graph Browser supports some shortcuts:

  • CTRL + A: select every element in the graph

  • DEL: delete the selected elements (same as the remove button)

  • CTRL + CLICK: allows you to add elements to the current selection

  • DOUBLE CLICK: expands the selected nodes (same as the expand button)

  • ARROWS: move the selected elements in the input direction

  • Mouse Wheel: changes the zoom level of the graph

Sidebar

Sidebar
Figure 19. Sidebar

The sidebar allows you to:

  1. Move the graph view in the clicked direction

  2. Switch between:

    • Arrow - allows you to select elements

    • Hand - allows you to move the graph regardless of selected elements

  3. Allows you to change the zoom level

Siren Investigate Box Plot

This visualization displays a box plot chart from the data in the current set of Elasticsearch documents.

Usage

Box plot

Please make sure that you have:

  • One Percentiles metric, with three Percentiles defined:

    • Bottom Percentile (Usually around 25%)

    • Mean (Usually around 50%)

    • Top Percentile (Usually around 75%)

  • One Max metric

  • One Min metric

  • One Aggregation (Optional)

Options

Box plot options
  • Y Axis Text - A label for the X axis.

  • X Axis Text - A label for the Y axis.

  • Show values - Check this box to enable the display the value next to its box.

  • Restrict Y axis MAX - Restricts the domain of the Y axis to a maximum value.

    • Global Max Y Value - Y axis domain maximum value.

  • Restrict Y axis MIN - Restricts the domain of the Y axis to a minimum value.

    • Global Min Y Value - Y axis domain minimum value.

After changing options, click the green Apply changes button to update your visualization, or the grey Discard changes button to return your visualization to its previous state.

Bubble Diagram

The Bubble Diagram visualization displays series of data grouped into packed circles.

First

Bubble size

The radius of circles depends on the type of metric aggregations.

Count

The count aggregation returns a raw count of the elements in the selected index pattern.

Average

This aggregation returns the average of a numeric field. Select a field from the drop-down.

Sum

The sum aggregation returns the total sum of a numeric field. Select a field from the drop-down.

Min

The min aggregation returns the minimum value of a numeric field. Select a field from the drop-down.

Max

The max aggregation returns the maximum value of a numeric field. Select a field from the drop-down.

Unique Count

The cardinality aggregation returns the number of unique values in a field. Select a field from the drop-down.

Buckets Aggregations

The buckets aggregations determine what information will come out in the diagram.

You can do a maximum of two agregations at one time. The first aggregation will create the parent circles, while the second aggregation will create the children circles.

Parent circles look slightly different than the children ones. Parent circles have a thicker border and the label written in bold.

The parents bubble are divided by colors. If you do a subaggregation (children) you will see a bubbles divided by family. Children are located near the parent and all have the same color. Families are united. If you drag a bubble, all members of the family will drag along.

Aggregation configuration

Options

In the diagram there are three options

Options configuration
Show Parents

If checked parent bubbles are visible when doing the subaggregation.

Hidden Label

If checked the labels are hidden.

Enable Zoom

Enables zoom on the page. To use the zoom you have to use the mouse wheel.

Circles movements

All circles gravitate towards the center of the visualisation.

When you drag a circle, its family follows it.

bubbles Movement

When moving the mouse over a circle a detailed information is shown in a tooltip.

Detailed information on hover

Filters

You can create filters by double-clicking on the bubbles.

When you double-click a child, you will see the bubble itself and its parent.

Filter Child

When you double-click a parent, you will see the bubble itself and its family.

Filter Parent

Siren Investigate Horizontal Bar Chart

This visualization displays a horizontal bar chart from the data in the current set of Elasticsearch documents.

Usage

Horizontal Bar Chart

Dashboard

A Siren Investigate dashboard displays a set of saved visualizations in a customizable grid layout. You can save a dashboard to share or reload at a later time.

In Siren Investigate, dashboards are displayed in the left-hand panel and can be organized as dashboard groups.

Getting Started

You need at least one saved visualization to use a dashboard.

Building a New Dashboard

Dashboards can be accessed via the home icon or the Siren logo. When you click Dashboard, Siren Investigate displays the first available dashboard or, if no dashboards have been defined, the dashboard creation screen.

New Dashboard screen

You can create a new dashboard by clicking on the icon in the dashboard panel:

Create New Dashboard]

Creating a New Dashboard

Build your dashboard by adding visualizations. By default, Siren Investigate dashboards use a light color theme. To use a dark color theme instead, click the Options button (which you can find on the top horizonal menu or by right clicking on the dashboard name) and check the Use dark theme box.

Dark Theme Example

You can change the default theme in the Advanced section of the Settings tab.

Saving Dashboards

To save the dashboard, click the Save button:

Saving a dashboard

The name of the dashboard can be set in the Save As field.

If Store time with dashboard is checked, the time filter Time Filter currently set will be restored when the dashboard is opened.

To display the number of Elasticsearch documents displayed by the dashboard in the corresponding tab, select a Saved Search:

Dashboard settings

Sharing Dashboards

You can share dashboards with other users by sending a link or by embedding them into HTML pages; make sure that your Siren Investigate installation is properly secured when sharing a dashboard on a public facing server.

To view shared dashboards users must be able to access Siren Investigate; keep this in mind if your Siren Investigate instance is protected by an authentication proxy.

To share a dashboard, click the Share button to display the Sharing panel.

sharing-panel

Click the Copy to Clipboard button share-link to copy the native URL or embed HTML to the clipboard. Click the Generate short URL button share-short-link to create a shortened URL for sharing or embedding.

Embedding Dashboards

To embed a dashboard, copy the embed code from the Share display into your external web application.

Adding Visualizations to a Dashboard

To add a visualization to the dashboard, click the Add button in the toolbar panel, then select a previously created visualization from the list:

Adding a visualization to the dashboard

You can filter the list of visualizations by typing a filter string into the Visualization Filter field.

The visualization you select appears in a container on your dashboard.

If you see a message about the container’s height or width being too small, resize the container.

Reset all dashboards to their default state

remove-all-filters_5 One can save with dashboard some specific filters, a custom query or a certain time range. If you click on the Reset button in the toolbar panel, the temporary filters/queries/time set on all dashboards would be removed, reverted to a dashboard’s default state with the saved filters/query/time.

Customizing Dashboard Elements

The visualizations in your dashboard are stored in resizable containers that you can arrange on the dashboard. This section discusses customizing these containers.

Moving Containers

Click and hold a container’s header to move the container around the dashboard. Other containers will shift as needed to make room for the moving container. Release the mouse button to confirm the container’s new location.

Resizing Containers

Move the cursor to the bottom right corner of the container until the cursor changes to point at the corner. After the cursor changes, click and drag the corner of the container to change the container’s size. Release the mouse button to confirm the new container size.

Removing Containers

Click the x icon at the top right corner of a container to remove that container from the dashboard. Removing a container from a dashboard does not delete the saved visualization in that container.

Viewing Detailed Information

To display the raw data behind the visualization, click the bar at the bottom of the container. Tabs with detailed information about the raw data replace the visualization, as in this example:

Table

A representation of the underlying data, presented as a paginated data grid. You can sort the items in the table by clicking on the table headers at the top of each column.

NYCTA-Table

Request

The raw request used to query the server, presented in JSON format.

NYCTA-Request

Response

The raw response from the server, presented in JSON format.

NYCTA-Response

Statistics

A summary of the statistics related to the request and the response, presented as a data grid. The data grid includes the query duration, the request duration, the total number of records found on the server, and the index pattern used to make the query.

NYCTA-Statistics

To export the raw data behind the visualization as a comma-separated-values (CSV) file, click on either the Raw or Formatted links at the bottom of any of the detailed information tabs. A raw export contains the data as it is stored in Elasticsearch. A formatted export contains the results of any applicable Siren Investigate [field formatters].

Changing the Visualization

Click the Edit button Pencil button at the top right of a container to open the visualization in the Visualize page.

Working with Filters

When you create a filter anywhere in Siren Investigate, the filter conditions display in an oval under the search text entry box:

filter sample 5

Hovering on the filter oval displays the following icons:

filter allbuttons 5
Enable Filter filter-enable

Click this icon to disable the filter without removing it. You can enable the filter again later by clicking the icon again. Disabled filters are displayed with a striped shaded color.

Pin Filter filter-pin

Click this icon to pin a filter. Pinned filters persist across Siren Investigate tabs. You can pin filters from the Visualize tab, click on the Discover or Dashboard tabs, and those filters remain in place.

If you have a pinned filter and you are not seeing any query results, check that your current tab’s index pattern is one that the filter applies to. E.g. a filter name:giovanni will results in 0 results if pinned and therefore "dragged along" to a dashboard whose underlying index does not have a name field, let alone a giovanni value. For this reason a good pattern in Siren Investigate is to use Dashboard Groups to group together dashboard which are based on the same underlying index. In this case the user can safely pin and "drag along" a filter across dashboards in the same group.
Toggle Filter filter-toggle

Click this icon to toggle a filter. By default, filters are inclusion filters, and are displayed in green. Only elements that match the filter are displayed. To change this to an exclusion filter, displaying only elements that don’t match, toggle the filter. Exclusion filters are displayed in red.

Remove Filter filter-delete

Click this icon to remove a filter entirely.

Custom Filter filter-custom

Click this icon to display a text field where you can customize the JSON representation of the filter and specify an alias to use for the filter name:

filter custom json

You can use a JSON filter representation to implement predicate logic, with should for OR, must for AND, and must_not for NOT:

Example 1. OR Example
{
  "bool": {
    "should": [
      {
        "term": {
          "geoip.country_name.raw": "Canada"
        }
      },
      {
        "term": {
          "geoip.country_name.raw": "China"
        }
      }
    ]
  }
}
Example 2. AND Example
{
  "bool": {
    "must": [
      {
        "term": {
          "geoip.country_name.raw": "United States"
        }
      },
      {
        "term": {
          "geoip.city_name.raw": "New York"
        }
      }
    ]
  }
}
Example 3. NOT Example
{
  "bool": {
    "must_not": [
      {
        "term": {
          "geoip.country_name.raw": "United States"
        }
      },
      {
        "term": {
          "geoip.country_name.raw": "Canada"
        }
      }
    ]
  }
}

Click the Done button to update the filter with your changes.

To apply any of the filter actions to all the filters currently in place, click the filter-actions Global Filter Actions button and select an action.

Dashboard Groups

Dashboards can be organized in dashboard groups.

Dashboard Groups Panel

If the dashboard is associated with a saved search, the count of documents on the dashboard is displayed next to the dashboard name. Two additional indicators that might be displayed are:

  • Filters/Queries indicator - the filter icon is displayed if there are any filter or query currently applied on the dashboard

  • Pruned joins indicator - a star symbol is displayed if any of the join operations was pruned.

Edit Dashboard Groups

Dashboard Groups Panel

In the left-hand dashboard panel, you can change the order of the dashboard groups and move dashboards between groups by dragging and dropping. Dashboard groups can be managed by clicking on the Create new group icon in the dashboard panel, and by right clicking on the dashboard group name to get Edit and Delete options.

Dashboard Group Edit Panel

In edit, you can change the title of an existing group, set the icon to a custom image by inserting a URL or use a Font Awesome icon.

Refreshing the Search Results

You can configure a refresh interval to automatically refresh the page with the latest index data. This periodically resubmits the search query.

When a refresh interval is set, it is displayed to the left of the Time Filter in the menu bar.

To set the refresh interval:

  1. Click the Time Filter Time Filter.

  2. Click the Refresh Interval tab.

  3. Choose a refresh interval from the list.

To automatically refresh the data, click the autorefresh Auto-refresh button when the time picker is open and select an autorefresh interval:

autorefresh intervals

When auto-refresh is enabled, Siren Investigate’s top bar displays a pause button and the auto-refresh interval: autorefresh-pause. Click the Pause button to pause auto-refresh.

Timelion

Timelion is a time series data visualizer that enables you to combine totally independent data sources within a single visualization. It’s driven by a simple expression language you use to retrieve time series data, perform calculations to tease out the answers to complex questions, and visualize the results.

For example, Timelion enables you to easily get the answers to questions like:

  • How many pages does each unique user view over time?

  • What’s the difference in traffic volume between this Friday and last Friday?

  • What percent of Japan’s population came to my site today?

  • What’s the 10-day moving average of the S&P 500?

  • What’s the cumulative sum of all search requests received in the last 2 years?  

You might also be interested in these tutorial videos:

Getting Started

Ready to experience all that is Timelion? This getting started tutorial shows you how to:

Creating time series visualizations

This tutorial will be using the time series data from Metricbeat to walk you through a number of functions that Timelion offers. To get started, download Metricbeat and follow the instructions here to start ingesting the data locally.

The first visualization you’ll create will compare the real-time percentage of CPU time spent in user space to the results offset by one hour. In order to create this visualization, we’ll need to create two Timelion expressions. One with the real-time average of system.cpu.user.pct and another with the average offset by one hour.

To start, you will need to define an index, timefield and metric in the first expression. Go ahead and enter the below expression into the Timelion query bar.

.es(index=metricbeat-*, timefield='@timestamp', metric='avg:system.cpu.user.pct')
timelion create01

 

Now you need to add another series with data from the previous hour for comparison. To do so, you’ll have to add an offset arguement to the .es() function. offset will offset the series retrieval by a date expression. For this example, you’ll want to offset the data back one hour and will be using the date expression -1h. Using a comma to separate the two series, enter the following expression into the Timelion query bar:

.es(index=metricbeat-*, timefield='@timestamp', metric='avg:system.cpu.user.pct'), .es(offset=-1h,index=metricbeat-*, timefield='@timestamp', metric='avg:system.cpu.user.pct')
timelion create02

 

It’s a bit hard to differentiate the two series. Customize the labels in order to easily distinguish them. You can always append the .label() function to any expression to add a custom label. Enter the below expression into the Timelion query bar to customize your labels:

.es(offset=-1h,index=metricbeat-*, timefield='@timestamp', metric='avg:system.cpu.user.pct').label('last hour'), .es(index=metricbeat-*, timefield='@timestamp', metric='avg:system.cpu.user.pct').label('current hour')
timelion create03

 

Save the entire Timelion sheet as Metricbeat Example. As a best practice, you should be saving any significant changes made to this sheet as you progress through this tutorial.

Customize and format visualizations

Timelion has plenty of options for customization. You can personalize nearly every aspect of a chart with the functions available. For this tutorial, you will perform the following modifications.

  • Add a title

  • Change a series type

  • Change the color and opacity of a series

  • Modify the legend

In the previous section, you created a Timelion chart with two series. Let’s continue to customize this visualization.

Before making any other modifications, append the title() function to the end of an expression to add a title with a meaningful name. This will make it much easier for unfamiliar users to understand the visualizations purpose. For this example, add title('CPU usage over time') to the original series. Use the following expression in your Timelion querybar:

.es(offset=-1h,index=metricbeat-*, timefield='@timestamp', metric='avg:system.cpu.user.pct').label('last hour'), .es(index=metricbeat-*, timefield='@timestamp', metric='avg:system.cpu.user.pct').label('current hour').title('CPU usage over time')
timelion customize01

 

To differentiate the last hour series a bit more, you are going to change the chart type to an area chart. In order do so, you’ll need to use the .lines() function to customize the line chart. You’ll be setting the fill and width arguements to set the fill of the line chart and line width respectively. In this example, you will set the fill level to 1 and the width of the border to 0.5 by appending .lines(fill=1,width=0.5). Use the following expression in the Timelion query bar:

.es(offset=-1h,index=metricbeat-*, timefield='@timestamp', metric='avg:system.cpu.user.pct').label('last hour').lines(fill=1,width=0.5), .es(index=metricbeat-*, timefield='@timestamp', metric='avg:system.cpu.user.pct').label('current hour').title('CPU usage over time')
timelion customize02

 

Let’s color these series so that the current hour series pops a bit more than the last hour series. The color() function can be used to change the color of any series and accepts standard color names, hexadecimal values or a color schema for grouped series. For this example you will use .color(gray) for the last hour and .color(#1E90FF) for the current hour. Enter the following expression into the Timelion query bar to make the adjustments:

.es(offset=-1h,index=metricbeat-*, timefield='@timestamp', metric='avg:system.cpu.user.pct').label('last hour').lines(fill=1,width=0.5).color(gray), .es(index=metricbeat-*, timefield='@timestamp', metric='avg:system.cpu.user.pct').label('current hour').title('CPU usage over time').color(#1E90FF)
timelion customize03

 

Last but not least, adjust the legend so that it takes up as little space as possible. You can utilize the .legend() function to set the position and style of the legend. For this example, place the legend in the north west position of the visualization with two columns by appending .legend(columns=2, position=nw) to the original series. Use the following expression to make the adjustments:

.es(offset=-1h,index=metricbeat-*, timefield='@timestamp', metric='avg:system.cpu.user.pct').label('last hour').lines(fill=1,width=0.5).color(gray), .es(index=metricbeat-*, timefield='@timestamp', metric='avg:system.cpu.user.pct').label('current hour').title('CPU usage over time').color(#1E90FF).legend(columns=2, position=nw)
timelion customize04

 

Save your changes and continue on to the next section to learn about mathematical functions.

Using mathematical functions

You’ve learned how to create and style a Timelion visualization in the previous two sections. This section will explore the mathematical functions Timelion offers. You will continue to use the Metricbeat data to create a new Timelion visualization for inbound and outbound network traffic. To start, you’ll need to add a new Timelion visualization to the sheet.

In the top menu, click Add to add a second visualization. When added to the sheet, you’ll notice that the query bar has been replaced with the default .es(*) expression. This is because the query is associated with the visualization on the Timelion sheet you have selected.

timelion math01

 

To start tracking the inbound / outbound network traffic, your first expression will calculate the maximum value of system.network.in.bytes. Enter the expression below into your Timelion query bar:

.es(index=metricbeat*, timefield=@timestamp, metric=max:system.network.in.bytes)
timelion math02

 

Monitoring network traffic is much more valuable when plotting the rate of change. The derivative() function is used do just that - plot the change in values over time. This can be easily done by appending the .derivative() to the end of an expression. Use the following expression to update your visualization:

.es(index=metricbeat*, timefield=@timestamp, metric=max:system.network.in.bytes).derivative()
timelion math03

 

Now for the outbound traffic. You’ll need to add a similar calculation for system.network.out.bytes. Since outbound traffic is leaving your machine, it makes sense to represent this metric as a negative number. The .multiply() function will multiply the series by a number, the result of a series or a list of series. For this example, you will use .multiply(-1) to convert the outbound network traffic to a negative value. Use the following expression to update your visualization:

.es(index=metricbeat*, timefield=@timestamp, metric=max:system.network.in.bytes).derivative(), .es(index=metricbeat*, timefield=@timestamp, metric=max:system.network.out.bytes).derivative().multiply(-1)
timelion math04

 

To make this visualization a bit easier to consume, convert the series from bytes to megabytes. Timelion has a .divide() function that can be used. .divide() accepts the same input as .multiply() and will divide the series by the divisor defined. Use the following expression to update your visualization:

.es(index=metricbeat*, timefield=@timestamp, metric=max:system.network.in.bytes).derivative().divide(1048576), .es(index=metricbeat*, timefield=@timestamp, metric=max:system.network.out.bytes).derivative().multiply(-1).divide(1048576)
timelion math05

 

Utilizing the formatting functions .title(), .label(), .color(), .lines() and .legend() learned in the last section, let’s clean up the visualization a bit. Use the following expression to update your visualization:

.es(index=metricbeat*, timefield=@timestamp, metric=max:system.network.in.bytes).derivative().divide(1048576).lines(fill=2, width=1).color(green).label("Inbound traffic").title("Network traffic (MB/s)"), .es(index=metricbeat*, timefield=@timestamp, metric=max:system.network.out.bytes).derivative().multiply(-1).divide(1048576).lines(fill=2, width=1).color(blue).label("Outbound traffic").legend(columns=2, position=nw)
timelion math06

 

Save your changes and continue on to the next section to learn about conditional logic and tracking trends.

Using conditional logic and tracking trends

In this section you will learn how to modify time series data with conditional logic and create a trend with a moving average. This is helpful to easily detect outliers and patterns over time.

For the purposes of this tutorial, you will continue to use Metricbeat data to add another visualization that monitors memory consumption. To start, use the following expression to chart the maximum value of system.memory.actual.used.bytes.

.es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes')
timelion conditional01

 

Let’s create two thresholds to keep an eye on the amount of used memory. For the purposes of this tutorial, your warning threshold will be 12.5GB and your severe threshold will be 15GB. When the maximum amount of used memory exceeds either of these thresholds, the series will be colored accordingly.

If the threshold values are too high or low for your machine, please adjust accordingly.

To configure these two threshold values, you can utilize Timelion’s conditional logic. In this tutorial you will use if() to compare each point to a number, adjust the styling if the condition evaluates to true and use the default styling if the condition evaluates to false. Timelion offers the following six operator values for comparison.

eq

equal

ne

not equal

lt

less than

lte

less than or equal to

gt

greater than

gte

greater than or equal to

Since there are two thresholds, it makes sense to style them differently. Use the gt operator to color the warning threshold yellow with .color('#FFCC11') and the severe threshold red with .color('red'). Enter the following expression into the Timelion query bar to apply the conditional logic and threshold styling:

.es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes'), .es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes').if(gt,12500000000,.es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes'),null).label('warning').color('#FFCC11'), .es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes').if(gt,15000000000,.es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes'),null).label('severe').color('red')
timelion conditional02

 

For additional information on Timelions conditional capabilities, check out the blog post I have but one .condition().

Now that you have thresholds defined to easily identify outliers, let’s create a new series to determine what the trend really is. Timelion’s mvavg() function allows you to calculate the moving average over a given window. This is especially helpful for noisey time series. For this tutorial, you will use .mvavg(10) to create a moving average with a window of 10 data points. Use the following expression to create a moving average of the maximum memory usage:

.es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes'), .es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes').if(gt,12500000000,.es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes'),null).label('warning').color('#FFCC11'), .es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes').if(gt,15000000000,.es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes'),null).label('severe').color('red'), .es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes').mvavg(10)
timelion conditional03

 

Now that you have thresholds and a moving average, let’s format the visualization so it is a bit easier to consume. As with the last section, use the .color(), .line(), .title() and .legend() functions to update your visualization accordingly:

.es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes').label('max memory').title('Memory consumption over time'), .es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes').if(gt,12500000000,.es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes'),null).label('warning').color('#FFCC11').lines(width=5), .es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes').if(gt,15000000000,.es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes'),null).label('severe').color('red').lines(width=5), .es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes').mvavg(10).label('mvavg').lines(width=2).color(#5E5E5E).legend(columns=4, position=nw)
timelion conditional04

 

Save your Timelion sheet and continue on to the next section to add these new visualizations to your dashboard.

Add to dashboard

You’ve officially harnessed the power of Timelion to create time series visualizations. The final step of this tutorial is to add your new visualizations to a dashboard. Below, this section will show you how to save a visualization from your Timelion sheet and add it to an existing dashboard.

To save a Timelion visualization as a dashboard panel, follow the steps below.

  1. Select the visualization you’d like to add to one (or multiple) dashboards

  2. Click the Save option in the top menu

  3. Select Save current expression as Kibana dashboard panel

  4. Name your panel and click Save to save as a dashboard visualization

timelion save01

 

Now you can add this dashboard panel to any dashboard you’d like. This visualization will now be listed in the Visualize list. Go ahead and follow the same process for the rest of the visualizations you created.

Create a new dashboard or open an existing one to add the Timelion visualizations as you would any other visualization.

timelion save02

 

You can also create time series visualizations right from the Visualize app—​just select the Timeseries visualization type and enter a Timelion expression in the expression field.

Inline Help and Documentation

Can’t remember a function or searching for a new one? You can always reference the inline help and documentation in Timelion.

Documentation for the Timelion expression language is built-in. Click Docs in the top menu to view the available functions and access the inline reference. As you start to enter functions in the query bar, Timelion will display the relevant arguments in real time.

Timelion inline help

Dev Tools

The Dev Tools page contains development tools that you can use to interact with your data in Kibana.

Console

The Console plugin provides a UI to interact with the REST API of Elasticsearch. Console has two main areas: the editor, where you compose requests to Elasticsearch, and the response pane, which displays the responses to the request.

Screenshot
Figure 20. The Console UI

Console understands commands in a cURL-like syntax. For example the following Console command

GET /_search
{
  "query": {
    "match_all": {}
  }
}

is a simple GET request to Elasticsearch’s _search API. Here is the equivalent command in cURL.

curl -XGET "http://localhost:9200/_search" -d'
{
  "query": {
    "match_all": {}
  }
}'

In fact, you can paste the above command into Console and it will automatically be converted into the Console syntax.

When typing a command, Console will make context sensitive suggestions. These suggestions can help you explore parameters for each API, or to just speed up typing. Console will suggest APIs, indexes and field names.

Suggestions
Figure 21. API suggestions

Once you have typed a command in to the left pane, you can submit it to Elasticsearch by clicking the little green triangle that appears next to the URL line of the request. Notice that as you move the cursor around, the little triangle and wrench icons follow you around. We call this the Action Menu. You can also select multiple requests and submit them all at once.

The Action Menu
Figure 22. The Action Menu

When the response come back, you should see it in the left hand panel:

Screenshot
Figure 23. The Output Pane

The Console UI

In this section you will find a more detailed description of UI of Console. The basic aspects of the UI are explained in the Console section.

Multiple Requests Support

The Console editor allows writing multiple requests below each other. As shown in the Console section, you can submit a request to Elasticsearch by positioning the cursor and using the Action Menu. Similarly you can select multiple requests in one go:

Multiple Requests
Figure 24. Selecting Multiple Requests

Console will send the request one by one to Elasticsearch and show the output on the right pane as Elasticsearch responds. This is very handy when debugging an issue or trying query combinations in multiple scenarios.

Selecting multiple requests also allows you to auto format and copy them as cURL in one go.

Auto Formatting

Console allows you to auto format messy requests. To do so, position the cursor on the request you would like to format and select Auto Indent from the action menu:

Auto format before
Figure 25. Auto Indent a request

Console will adjust the JSON body of the request and it will now look like this:

Auto format after
Figure 26. A formatted request

If you select Auto Indent on a request that is already perfectly formatted, Console will collapse the request body to a single line per document. This is very handy when working with Elasticsearch’s bulk APIs:

Auto format bulk
Figure 27. One doc per line

Keyboard shortcuts

Console comes with a set of nifty keyboard shortcuts making working with it even more efficient. Here is an overview:

General editing

Ctrl/Cmd + I

Auto indent current request.

Ctrl + Space

Open Auto complete (even if not typing).

Ctrl/Cmd + Enter

Submit request.

Ctrl/Cmd + Up/Down

Jump to the previous/next request start or end.

Ctrl/Cmd + Alt + L

Collapse/expand current scope.

Ctrl/Cmd + Option + 0

Collapse all scopes but the current one. Expand by adding a shift.

When auto-complete is visible

Down arrow

Switch focus to auto-complete menu. Use arrows to further select a term.

Enter/Tab

Select the currently selected or the top most term in auto-complete menu.

Esc

Close auto-complete menu.

History

Console maintains a list of the last 500 requests that were successfully executed by Elasticsearch. The history is available by clicking the clock icon on the top right side of the window. The icons opens the history panel where you can see the old requests. You can also select a request here and it will be added to the editor at the current cursor position.

History Panel
Figure 28. History Panel

Settings

Console has multiple settings you can set. All of them are available in the Settings panel. To open the panel click on the cog icon on the top right.

Setting Panel
Figure 29. Settings Panel

Configuring Console

You can add the following options in the config/siren.yml file:

console.enabled

Default: true Set to false to disable Console. Toggling this will cause the server to regenerate assets on the next startup, which may cause a delay before pages start being served.

Management

The Management application is where you perform your runtime configuration of Siren Investigate, including both the initial setup and ongoing configuration of index patterns, advanced settings that tweak the behaviors of Siren Investigate itself, and the various "objects" that you can save throughout Siren Investigate such as searches, visualizations, and dashboards.

This section is pluginable, so in addition to the out of the box capabitilies, packs such as Siren Investigate Access Control and X-Pack can add additional management capabilities.

Index Patterns

To use Siren Investigate, you have to tell it about the Elasticsearch indices that you want to explore by configuring one or more index patterns. You can also:

  • Create scripted fields that are computed on the fly from your data. You can browse and visualize scripted fields, but you cannot search them.

  • Set advanced options such as the number of rows to show in a table and how many of the most popular fields to show. Use caution when modifying advanced options, as it’s possible to set values that are incompatible with one another.

  • Configure Siren Investigate for a production environment

Creating an Index Pattern to Connect to Elasticsearch

An index pattern identifies one or more Elasticsearch indices that you want to explore with Siren Investigate. Siren Investigate looks for index names that match the specified pattern. An asterisk () in the pattern matches zero or more characters. For example, the pattern myindex- matches all indices whose names start with myindex-, such as myindex-1 and myindex-2.

An index pattern can also simply be the name of a single index.

To create an index pattern to connect to Elasticsearch:

  1. Go to the Settings > Indices tab.

  2. Specify an index pattern that matches the name of one or more of your Elasticsearch indices. By default, Siren Investigate guesses that you’re working with log data being fed into Elasticsearch by Logstash.

    When you switch between top-level tabs, Siren Investigate remembers where you were. For example, if you view a particular index pattern from the Settings tab, switch to the Discover tab, and then go back to the Settings tab, Siren Investigate displays the index pattern you last looked at. To get to the create pattern form, click the Add button in the Index Patterns list.
  3. If your index contains a timestamp field that you want to use to perform time-based comparisons, select the Index contains time-based events option and select the index field that contains the timestamp. Siren Investigate reads the index mapping to list all of the fields that contain a timestamp.

  4. By default, Siren Investigate restricts wildcard expansion of time-based index patterns to indices with data within the currently selected time range. Click Do not expand index pattern when search to disable this behavior.

  5. Click Create to add the index pattern.

  6. To designate the new pattern as the default pattern to load when you view the Discover tab, click the favorite button.

When you define an index pattern, indices that match that pattern must exist in Elasticsearch. Those indices must contain data.

To use an event time in an index name, enclose the static text in the pattern and specify the date format using the tokens described in the following table.

For example, [logstash-]YYYY.MM.DD matches all indices whose names have a timestamp of the form YYYY.MM.DD appended to the prefix logstash-, such as logstash-2015.01.31 and logstash-2015-02-01.

Date Format Tokens
M

Month - cardinal: 1 2 3 …​ 12

Mo

Month - ordinal: 1st 2nd 3rd …​ 12th

MM

Month - two digit: 01 02 03 …​ 12

MMM

Month - abbreviation: Jan Feb Mar …​ Dec

MMMM

Month - full: January February March …​ December

Q

Quarter: 1 2 3 4

D

Day of Month - cardinal: 1 2 3 …​ 31

Do

Day of Month - ordinal: 1st 2nd 3rd …​ 31st

DD

Day of Month - two digit: 01 02 03 …​ 31

DDD

Day of Year - cardinal: 1 2 3 …​ 365

DDDo

Day of Year - ordinal: 1st 2nd 3rd …​ 365th

DDDD

Day of Year - three digit: 001 002 …​ 364 365

d

Day of Week - cardinal: 0 1 3 …​ 6

do

Day of Week - ordinal: 0th 1st 2nd …​ 6th

dd

Day of Week - 2-letter abbreviation: Su Mo Tu …​ Sa

ddd

Day of Week - 3-letter abbreviation: Sun Mon Tue …​ Sat

dddd

Day of Week - full: Sunday Monday Tuesday …​ Saturday

e

Day of Week (locale): 0 1 2 …​ 6

E

Day of Week (ISO): 1 2 3 …​ 7

w

Week of Year - cardinal (locale): 1 2 3 …​ 53

wo

Week of Year - ordinal (locale): 1st 2nd 3rd …​ 53rd

ww

Week of Year - 2-digit (locale): 01 02 03 …​ 53

W

Week of Year - cardinal (ISO): 1 2 3 …​ 53

Wo

Week of Year - ordinal (ISO): 1st 2nd 3rd …​ 53rd

WW

Week of Year - two-digit (ISO): 01 02 03 …​ 53

YY

Year - two digit: 70 71 72 …​ 30

YYYY

Year - four digit: 1970 1971 1972 …​ 2030

gg

Week Year - two digit (locale): 70 71 72 …​ 30

gggg

Week Year - four digit (locale): 1970 1971 1972 …​ 2030

GG

Week Year - two digit (ISO): 70 71 72 …​ 30

GGGG

Week Year - four digit (ISO): 1970 1971 1972 …​ 2030

A

AM/PM: AM PM

a

am/pm: am pm

H

Hour: 0 1 2 …​ 23

HH

Hour - two digit: 00 01 02 …​ 23

h

Hour - 12-hour clock: 1 2 3 …​ 12

hh

Hour - 12-hour clock, 2 digit: 01 02 03 …​ 12

m

Minute: 0 1 2 …​ 59

mm

Minute - two-digit: 00 01 02 …​ 59

s

Second: 0 1 2 …​ 59

ss

Second - two-digit: 00 01 02 …​ 59

S

Fractional Second - 10ths: 0 1 2 …​ 9

SS

Fractional Second - 100ths: 0 1 …​ 98 99

SSS

Fractional Seconds - 1000ths: 0 1 …​ 998 999

Z

Timezone - zero UTC offset (hh:mm format): -07:00 -06:00 -05:00 .. +07:00

ZZ

Timezone - zero UTC offset (hhmm format): -0700 -0600 -0500 …​ +0700

X

Unix Timestamp: 1360013296

x

Unix Millisecond Timestamp: 1360013296123

Setting the Default Index Pattern

The default index pattern is loaded automatically when you view the Discover tab. Siren Investigate displays a star to the left of the name of the default pattern in the Index Patterns list on the Settings > Indices tab. The first pattern you create is automatically designated as the default pattern.

To set a different pattern as the default index pattern:

  1. Go to the Settings > Indices tab.

  2. Select the pattern you want to set as the default in the Index Patterns list.

  3. Click the pattern’s Favorite button.

You can also manually set the default index pattern in Advanced > Settings.

Reloading the Index Fields List

When you add an index mapping, Siren Investigate automatically scans the indices that match the pattern to display a list of the index fields. You can reload the index fields list to pick up any newly-added fields.

Reloading the index fields list also resets Siren Investigate’s popularity counters for the fields. The popularity counters keep track of the fields you’ve used most often within Siren Investigate and are used to sort fields within lists.

To reload the index fields list:

  1. Go to the Settings > Indices tab.

  2. Select an index pattern from the Index Patterns list.

  3. Click the pattern’s Reload button.

Deleting an Index Pattern

To delete an index pattern:

  1. Go to the Settings > Indices tab.

  2. Select the pattern you want to remove in the Index Patterns list.

  3. Click the pattern’s Delete button.

  4. Confirm that you want to remove the index pattern.

beta[]

Elasticsearch supports the ability to run search and aggregation requests across multiple clusters using a module called cross cluster search.

In order to take advantage of cross cluster search, you must configure your Elasticsearch clusters accordingly. Review the corresponding Elasticsearch {ref}/modules-cross-cluster-search.html[documentation] before attempting to use cross cluster search in Kibana.

Once your Elasticsearch clusters are configured for cross cluster search, you can create specific index patterns in Kibana to search across the clusters of your choosing. Using the same syntax that you’d use in a raw cross cluster search request in Elasticsearch, create your index pattern in Kibana with the convention <cluster-names>:<pattern>.

For example, if you want to query logstash indices across two of the Elasticsearch clusters that you set up for cross cluster search, which were named cluster_one and cluster_two, you would use cluster_one,cluster_two:logstash-* as your index pattern in Kibana.

Just like in raw search requests in Elasticsearch, you can use wildcards in your cluster names to match any number of clusters, so if you wanted to search logstash indices across any clusters named cluster_foo, cluster_bar, and so on, you would use cluster_*:logstash-* as your index pattern in Kibana.

If you want to query across all Elasticsearch clusters that have been configured for cross cluster search, then use a standalone wildcard for your cluster name in your Kibana index pattern: :logstash-.

Once an index pattern is configured using the cross cluster search syntax, all searches and aggregations using that index pattern in Kibana take advantage of cross cluster search.

Relations

In this panel, you can define relationships between index patterns. These relationships ultimately form a graph of index patterns. This graph is used

in conjunction with the Siren Federate Plugin For Elasticsearch, allowing to perform join operations between dashboards, i.e., filtering a dashboard’s documents with regards to another.

Graph of Index Patterns

A relationship is defined as a join operation between two indices with the following fields:

  • Left Index Pattern: the left index of the join;

  • Left Type: the type of the left index of the join;

  • Left Field: the field of the left index to join on;

  • Right Index Pattern: the right index of the join;

  • Right Type: the type of the right index of the join;

  • Right Field: the field of the right index to join with; and

  • Label: the label of the relation.

The image below displays a graph of four index patterns, where three relationships have been defined. You can add a new relationship by clicking on the "Add relation" button.

Graph of Index Patterns

Join Limit

Clicking the Edit pencil opens the advanced setting for each relation where you can set the maximum number of unique values returned from the related index pattern for that relation.

As a semi-join, these documents will be filtered based on the presence of a non-empty value for the join field in the other index pattern in the relation.

The limit is a total upper limit on the number of unique values of the join field on the other index pattern for which the join field exists. This limit is dynamically allocated over the shards where the index is located.

The index pattern in question is then filtered by the values returned.

These are used for displaying counts on relational buttons, for the content displayed on the current dashboard, etc.

Setting the limit here to -1 here sets the limit to the default kibi:joinLimit set in the Siren Investigate Advanced Settings and setting the limit to 0 here disables the limit entirely.

Example
  • Index Pattern A has 100 documents and Index Pattern B has 50 documents

  • The documents of A are filtered by adding a relation with B

  • The limit is set to 10 for that relation

  • In this case, the limit is set on B, so 10 unique values are returned from B

    • e.g. If the field joined had only true/false values, only 2 values would be returned, true and false

  • These 10 values are then used to filter the documents in index A

  • Potentially, this could return many more (or many less) than the limit set as there could be a large number of documents in A that match the 10 values returned from B

  • The final number of documents returned is always limited by the Elasticsearch size parameter

Join Type

Siren Federate provides three types of join algorithms; the plugin will try to pick the best algorithm for a given join automatically, however you can force the selection by choosing one of the available options:

  • MERGE_JOIN - Distributed join using merge join algorithm

  • HASH_JOIN - Distributed join using hash join algorithm

  • BROADCAST_JOIN - Broadcast join

A detailed description of each algorithm can be found in the Siren Federate plugin documentation.

Index Patterns Management Icon

A relation is also indicated on the Index Patterns Management application thanks to the Index relation icon icon. If you hover the mouse over it, a tooltip message is displayed indicating the index pattern and field with which that field is joined.

For example, in the image below, that icon is displayed next to the field "id" of the "investor" index, which is joined with the field "investorid" of the "investment" index.

Investor Index

Datasources

For an overview of datasources, please read the JDBC Datasources and External Datasources chapters.

Queries

For an overview of queries, please read the External Datasources chapter.

Templates

You can define templates to format the results of a query on an external datasource and the results of an Elasticsearch query in a Enhanced search results visualization.

Siren Investigate supports three template engines:

There are four pre-defined templates:

  • kibi-json-jade: this template presents the query results as a pretty-printed JSON object using the jade engine. This is useful to test queries while writing them.

  • kibi-table-jade: this template displays the query results in a table, using the jade engine.

  • kibi-table-handlebars: like kibi-table-jade, using the handlebars engine instead.

  • kibi-html-angular: this template for each document displays a panel

  • populated with all property values (Currently supported only in Enhanced

  • search results visualisation)

You can define your own custom template by clicking on the Settings / Templates tab.

Then, pick the engine you prefer and write the template; to see a preview, click on the save button and select a query from the list; depending on the query you selected, the EntityURI may need to be set.

Query template editor

Managing Fields

The fields for the index pattern are listed in a table. Click a column header to sort the table by that column. Click the Controls button in the rightmost column for a given field to edit the field’s properties. You can manually set the field’s format from the Format drop-down. Format options vary based on the field’s type.

You can also set the field’s popularity value in the Popularity text entry box to any desired value. Click the Update Field button to confirm your changes or Cancel to return to the list of fields.

Siren Investigate has field formatters for the following field types:

String Field Formatters

String fields support the String and Url formatters.

The String field formatter can apply the following transformations to the field’s contents:

  • Convert to lowercase

  • Convert to uppercase

  • Convert to title case

  • Apply the short dots transformation, which replaces the content before a . character with the first character of that content, as in the following example:

Original

Becomes

com.organizations.project.ClassName

c.o.p.ClassName

The Url field formatter can take on the following types:

  • The Link type turn the contents of the field into an URL.

  • The Image type can be used to specify an image directory where a specified image is located.

You can customize either type of URL field formats with templates. A URL template enables you to add specific values to a partial URL. Use the string {{value}} to add the contents of the field to a fixed URL.

For example, when:

The resulting URL replaces {{value}} with the user ID from the field.

The {{value}} template string URL-encodes the contents of the field. When a field encoded into a URL contains non-ASCII characters, these characters are replaced with a % character and the appropriate hexadecimal code. For example, field contents users/admin result in the URL template adding users%2Fadmin.

When the formatter type is set to Image, the {{value}} template string specifies the name of an image at the specified URI.

In order to pass unescaped values directly to the URL, use the {{rawValue}} string.

A Label Template enables you to specify a text string that displays instead of the raw URL. You can use the {{value}} template string normally in label templates. You can also use the {{url}} template string to display the formatted URL.

Date Field Formatters

Date fields support the Date, Url, and String formatters.

The Date formatter enables you to choose the display format of date stamps using the moment.js standard format definitions.

The String field formatter can apply the following transformations to the field’s contents:

  • Convert to lowercase

  • Convert to uppercase

  • Convert to title case

  • Apply the short dots transformation, which replaces the content before a . character with the first character of that content, as in the following example:

Original

Becomes

com.organizations.project.ClassName

c.o.p.ClassName

The Url field formatter can take on the following types:

  • The Link type turn the contents of the field into an URL.

  • The Image type can be used to specify an image directory where a specified image is located.

You can customize either type of URL field formats with templates. A URL template enables you to add specific values to a partial URL. Use the string {{value}} to add the contents of the field to a fixed URL.

For example, when:

The resulting URL replaces {{value}} with the user ID from the field.

The {{value}} template string URL-encodes the contents of the field. When a field encoded into a URL contains non-ASCII characters, these characters are replaced with a % character and the appropriate hexadecimal code. For example, field contents users/admin result in the URL template adding users%2Fadmin.

When the formatter type is set to Image, the {{value}} template string specifies the name of an image at the specified URI.

In order to pass unescaped values directly to the URL, use the {{rawValue}} string.

A Label Template enables you to specify a text string that displays instead of the raw URL. You can use the {{value}} template string normally in label templates. You can also use the {{url}} template string to display the formatted URL.

Geographic Point Field Formatters

Geographic point fields support the String formatter.

The String field formatter can apply the following transformations to the field’s contents:

  • Convert to lowercase

  • Convert to uppercase

  • Convert to title case

  • Apply the short dots transformation, which replaces the content before a . character with the first character of that content, as in the following example:

Original

Becomes

com.organizations.project.ClassName

c.o.p.ClassName

Numeric Field Formatters

Numeric fields support the Url, Bytes, Duration, Number, Percentage, String, and Color formatters.

The Url field formatter can take on the following types:

  • The Link type turn the contents of the field into an URL.

  • The Image type can be used to specify an image directory where a specified image is located.

You can customize either type of URL field formats with templates. A URL template enables you to add specific values to a partial URL. Use the string {{value}} to add the contents of the field to a fixed URL.

For example, when:

The resulting URL replaces {{value}} with the user ID from the field.

The {{value}} template string URL-encodes the contents of the field. When a field encoded into a URL contains non-ASCII characters, these characters are replaced with a % character and the appropriate hexadecimal code. For example, field contents users/admin result in the URL template adding users%2Fadmin.

When the formatter type is set to Image, the {{value}} template string specifies the name of an image at the specified URI.

In order to pass unescaped values directly to the URL, use the {{rawValue}} string.

A Label Template enables you to specify a text string that displays instead of the raw URL. You can use the {{value}} template string normally in label templates. You can also use the {{url}} template string to display the formatted URL.

The String field formatter can apply the following transformations to the field’s contents:

  • Convert to lowercase

  • Convert to uppercase

  • Convert to title case

  • Apply the short dots transformation, which replaces the content before a . character with the first character of that content, as in the following example:

Original

Becomes

com.organizations.project.ClassName

c.o.p.ClassName

The Duration field formatter can display the numeric value of a field in the following increments:

  • Picoseconds

  • Nanoseconds

  • Microseconds

  • Milliseconds

  • Seconds

  • Minutes

  • Hours

  • Days

  • Weeks

  • Months

  • Years

You can specify these increments with up to 20 decimal places for both input and output formats.

The Color field formatter enables you to specify colors with specific ranges of values for a numeric field.

When you select the Color field formatter, Siren Investigate displays the Range, Font Color, Background Color, and Example fields.

Click the Add Color button to add a range of values to associate with a particular color. You can click in the Font Color and Background Color fields to display a color picker. You can also enter a specific hex code value in the field. The effect of your current color choices are displayed in the Example field.

colorformatter

The Bytes, Number, and Percentage formatters enable you to choose the display formats of numbers in this field using the numeral.js standard format definitions.

Scripted Fields

Scripted fields compute data on the fly from the data in your Elasticsearch indices. Scripted field data is shown on the Discover tab as part of the document data, and you can use scripted fields in your visualizations. Scripted field values are computed at query time so they aren’t indexed and cannot be searched.

Siren Investigate cannot query scripted fields.
Computing data on the fly with scripted fields can be very resource intensive and can have a direct impact on Siren Investigate’s performance. Keep in mind that there’s no built-in validation of a scripted field. If your scripts are buggy, you’ll get exceptions whenever you try to view the dynamically generated data.

When you define a scripted field in Siren Investigate, you have a choice of scripting languages. Starting with 5.0, the default options are {ref}/modules-scripting-expression.html[Lucene expressions] and {ref}/modules-scripting-painless.html[Painless]. While you can use other scripting languages if you enable dynamic scripting for them in Elasticsearch, this is not recommended because they cannot be sufficiently {ref}/modules-scripting-security.html[sandboxed].

Use of Groovy, Javascript, and Python scripting is deprecated starting in Elasticsearch 5.0, and support for those scripting languages will be removed in the future.

You can reference any single value numeric field in your expressions, for example:

doc['field_name'].value

For more background on scripted fields and additional examples, refer to this blog: Using Painless in Kibana scripted fields

Creating a Scripted Field

To create a scripted field:

  1. Go to Settings > Indices

  2. Select the index pattern you want to add a scripted field to.

  3. Go to the pattern’s Scripted Fields tab.

  4. Click Add Scripted Field.

  5. Enter a name for the scripted field.

  6. Enter the expression that you want to use to compute a value on the fly from your index data.

  7. Click Save Scripted Field.

For more information about scripted fields in Elasticsearch, see {ref}/modules-scripting.html[Scripting].

Updating a Scripted Field

To modify a scripted field:

  1. Go to Settings > Indices

  2. Click the Edit button for the scripted field you want to change.

  3. Make your changes and then click Save Scripted Field to update the field.

Keep in mind that there’s no built-in validation of a scripted field. If your scripts are buggy, you’ll get exceptions whenever you try to view the dynamically generated data.

Deleting a Scripted Field

To delete a scripted field:

  1. Go to Settings > Indices

  2. Click the Delete button for the scripted field you want to remove.

  3. Confirm that you really want to delete the field.

Setting Advanced Options

The Advanced Settings page enables you to directly edit settings that control the behavior of the Siren Investigate application. For example, you can change the format used to display dates, specify the default index pattern, and set the precision for displayed decimal values.

To set advanced options:

  1. Go to Settings > Advanced.

  2. Click the Edit button for the option you want to modify.

  3. Enter a new value for the option.

  4. Click the Save button.

Modifying the following settings can significantly affect Siren Investigate’s performance and cause problems that are difficult to diagnose. Setting a property’s value to a blank field will revert to the default behavior, which may not be compatible with other configuration settings. Deleting a custom setting removes it from Siren Investigate permanently.
Common Settings Reference
query:queryString:options

Options for the Lucene query string parser.

sort:options

Options for the Elasticsearch {ref}/search-request-sort.html[sort] parameter.

dateFormat

The format to use for displaying pretty-formatted dates.

dateFormat:tz

The timezone that Siren Investigate uses. The default value of Browser uses the timezone detected by the browser.

dateFormat:scaled

These values define the format used to render ordered time-based data. Formatted timestamps must adapt to the interval between measurements. Keys are ISO8601 intervals.

dateFormat:dow

This property defines what day weeks should start on.

defaultIndex

Default is null. This property specifies the default index.

defaultColumns

Default is _source. Defines the columns that appear by default on the Discover page.

metaFields

An array of fields outside of _source. Siren Investigate merges these fields into the document when displaying the document.

discover:sampleSize

The number of rows to show in the Discover table.

discover:aggs:terms:size

Determines how many terms will be visualized when clicking the "visualize" button, in the field drop downs, in the discover sidebar. The default value is 20.

doc_table:highlight

Highlight results in Discover and Saved Searches Dashboard. Highlighting makes request slow when working on big documents. Set this property to false to disable highlighting.

doc_table:highlight:all_fields

Improves highlighting by using a separate highlight_query that uses all_fields mode on query_string queries. Set to false if you are using a default_field in your index.

courier:maxSegmentCount

Siren Investigate splits requests in the Discover app into segments to limit the size of requests sent to the Elasticsearch cluster. This setting constrains the length of the segment list. Long segment lists can significantly increase request processing time.

courier:ignoreFilterIfFieldNotInIndex

Set this property to true to skip filters that apply to fields that don’t exist in a visualization’s index. Useful when dashboards consist of visualizations from multiple index patterns.

fields:popularLimit

This setting governs how many of the top most popular fields are shown.

histogram:barTarget

When date histograms use the auto interval, Siren Investigate attempts to generate this number of bars.

histogram:maxBars

Date histograms are not generated with more bars than the value of this property, scaling values when necessary.

visualization:tileMap:maxPrecision

The maximum geoHash precision displayed on tile maps: 7 is high, 10 is very high, 12 is the maximum. {ref}/search-aggregations-bucket-geohashgrid-aggregation.html#_cell_dimensions_at_the_equator[Explanation of cell dimensions].

visualization:tileMap:WMSdefaults

Default properties for the WMS map server support in the coordinate map.

visualization:colorMapping

Maps values to specified colors within visualizations.

visualization:loadingDelay

Time to wait before dimming visualizations during query.

visualization:dimmingOpacity

When part of a visualization is highlighted, by hovering over it for example, ths is the opacity applied to the other elements. A higher number means other elements will be less opaque.

visualization:regionmap:showWarnings

Whether the region map show a warning when terms cannot be joined to a shape on the map.

csv:separator

A string that serves as the separator for exported values.

csv:quoteValues

Set this property to true to quote exported values.

history:limit

In fields that have history, such as query inputs, the value of this property limits how many recent values are shown.

shortDots:enable

Set this property to true to shorten long field names in visualizations. For example, instead of foo.bar.baz, show f.b.baz.

truncate:maxHeight

This property specifies the maximum height that a cell occupies in a table. A value of 0 disables truncation.

indexPattern:fieldMapping:lookBack

The value of this property sets the number of recent matching patterns to query the field mapping for index patterns with names that contain timestamps.

indexPattern:placeholder

The default placeholder value used when adding a new index pattern to Kibana.

format:defaultTypeMap

A map of the default format name for each field type. Field types that are not explicitly mentioned use "default".

format:number:defaultPattern

Default numeral format for the "number" format.

format:bytes:defaultPattern

Default numeral format for the "bytes" format.

format:percent:defaultPattern

Default numeral format for the "percent" format.

format:currency:defaultPattern

Default numeral format for the "currency" format.

savedObjects:perPage

The number of objects shown on each page of the list of saved objects. The default value is 5.

timepicker:timeDefaults

The default time filter selection.

timepicker:refreshIntervalDefaults

The time filter’s default refresh interval.

dashboard:defaultDarkTheme

Set this property to true to make new dashboards use the dark theme by default.

filters:pinnedByDefault

Set this property to true to make filters have a global state by default.

filterEditor:suggestValues

Set this property to true to have the filter editor suggest values for fields, instead of just providing a text input. This may result in heavy queries to Elasticsearch.

notifications:banner

You can specify a custom banner to display temporary notices to all users. This field supports Markdown.

notifications:lifetime:banner

Specifies the duration in milliseconds for banner notification displays. The default value is 3000000. Set this field to Infinity to disable banner notifications.

notifications:lifetime:error

Specifies the duration in milliseconds for error notification displays. The default value is 300000. Set this field to Infinity to disable error notifications.

notifications:lifetime:warning

Specifies the duration in milliseconds for warning notification displays. The default value is 10000. Set this field to Infinity to disable warning notifications.

notifications:lifetime:info

Specifies the duration in milliseconds for information notification displays. The default value is 5000. Set this field to Infinity to disable information notifications.

metrics:max_buckets

The maximum numbers of buckets that cannot be exceeded. For example, this can arise when the user selects a short interval like (e.g. 1s) for a long time period (e.g. 1 year)

timelion:showTutorial

Set this property to true to show the Timelion tutorial to users when they first open Timelion.

timelion:es.timefield

Default field containing a timestamp when using the .es() query.

timelion:es.default_index

Default index when using the .es() query.

timelion:target_buckets

Used for calculating automatic intervals in visualizations, this is the number of buckets to try to represent.

timelion:max_buckets

Used for calculating automatic intervals in visualizations, this is the maximum number of buckets to represent.

timelion:default_columns

The default number of columns to use on a timelion sheet.

timelion:default_rows

The default number of rows to use on a timelion sheet.

timelion:graphite.url

[experimental] Used with graphite queries, this it the URL of your host

timelion:quandl.key

[experimental] Used with quandl queries, this is your API key from www.quandl.com

state:storeInSessionStorage

[experimental] Kibana tracks UI state in the URL, which can lead to problems when there is a lot of information there and the URL gets very long. Enabling this will store parts of the state in your browser session instead, to keep the URL shorter.

context:defaultSize

Specifies the initial number of surrounding entries to display in the context view. The default value is 5.

context:step

Specifies the number to increment or decrement the context size by when using the buttons in the context view. The default value is 5.

context:tieBreakerFields

A comma-separated list of fields to use for tiebreaking between documents that have the same timestamp value. From this list the first field that is present and sortable in the current index pattern is used.

Siren Investigate Specific Settings Reference
kibi:splitTabs

Set to true to split dashboard tabs on two lines.

kibi:timePrecision

Set to generate time filters with certain precision. Possible values are: s, m, h, d, w, M, y.

kibi:relations

Relations between index patterns and dashboards.

kibi:joinLimit

Maximum number of unique source values in a relation returned to filter the target documents

kibi:session_cookie_expire

Set duration of cookie session (in seconds).

kibi:enableAllDashboardsCounts

Enable counts on all dashboards.

kibi:enableAllRelBtnCounts

Enable counts on all relational buttons.

kibi:defaultDashboardTitle

The dashboard that is displayed when clicking on the Dashboard tab for the first time.

kibi:graphUseWebGl

Set to false to disable WebGL rendering

kibi:graphExpansionLimit

Limit the number of elements to retrieve during the graph expansion.

kibi:graphMaxConcurrentCalls

Limit the number of concurrent calls done by the Graph Browser.

kibi:graphRelationFetchLimit

Limit the number of relations to retrieve after the graph expansion.

kibi:shieldAuthorizationWarning

Set to true to show all authorization warnings.

Managing Saved Searches, Visualizations, and Dashboards

You can view, edit, and delete saved searches, visualizations, and dashboards from Settings > Objects. You can also export or import sets of searches, visualizations, and dashboards.

Viewing a saved object displays the selected item in the Discover, Visualize, or Dashboard page. To view a saved object:

  1. Go to Settings > Objects.

  2. Select the object you want to view.

  3. Click the View button.

Editing a saved object enables you to directly modify the object definition. You can change the name of the object, add a description, and modify the JSON that defines the object’s properties.

If you attempt to access an object whose index has been deleted, Siren Investigate displays its Edit Object page. You can:

  • Recreate the index so you can continue using the object.

  • Delete the object and recreate it using a different index.

  • Change the index name referenced in the object’s kibanaSavedObjectMeta.searchSourceJSON to point to an existing index pattern. This is useful if the index you were working with has been renamed.

No validation is performed for object properties. Submitting invalid changes will render the object unusable. Generally, you should use the Discover, Visualize, or Dashboard pages to create new objects instead of directly editing existing ones.

To edit a saved object:

  1. Go to Settings > Objects.

  2. Select the object you want to edit.

  3. Click the Edit button.

  4. Make your changes to the object definition.

  5. Click the Save Object button.

To delete a saved object:

  1. Go to Settings > Objects.

  2. Select the object you want to delete.

  3. Click the Delete button.

  4. Confirm that you really want to delete the object.

To export a set of objects:

  1. Go to Settings > Objects.

  2. Select the type of object you want to export. You can export a set of dashboards, searches, or visualizations.

  3. Click the selection box for the objects you want to export, or click the Select All box.

  4. Click Export to select a location to write the exported JSON.

Exported dashboards do not include their associated index patterns. Re-create the index patterns manually before importing saved dashboards to a Siren Investigate instance running on another Elasticsearch cluster.

To import a set of objects:

  1. Go to Settings > Objects.

  2. Click Import to navigate to the JSON file representing the set of objects to import.

  3. Click Open after selecting the JSON file.

  4. If any objects in the set would overwrite objects already present in Siren Investigate, confirm the overwrite. = Authentication and Access Control

Siren Investigate can be integrated with Elasticsearch clusters protected by either Search Guard or Elastic x-pack.

In this scenario, both Siren Investigate and Gremlin Server (the backend component used by the graph browser visualization) must be configured to serve requests over HTTPS.

Enabling HTTPS in Siren Investigate

It is strongly recommended that you protect your Siren Investigate installation by using a reverse proxy. Some example configurations are given below, but other reverse proxies may also be used.

NginX Reverse Proxy with HTTPS (Linux)

To use NginX as a reverse proxy, add the following virtual server to your configuration: Here we assume letsencrypt has been used to provide the certificate.

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name kibi.example.com;

    root /var/www/html;
    index index.html index.htm;

    ssl_certificate /etc/letsencrypt/live/kibi.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/kibi.example.com/privkey.pem;

    access_log /var/log/nginx/kibi-ssl.access.log;
    error_log /var/log/nginx/kibi-ssl.error.log error;

    include snippets/ssl-params.conf;

    location / {
        proxy_pass http://127.0.0.1:5606;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

In /etc/nginx/snippets/ssl-params.conf configure:

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_ecdh_curve secp384r1;
ssl_session_cache shared:SSL:10m;
#ssl_session_tickets off;
ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;
# Disable preloading HSTS for now.  You can use the commented out header line that includes
# the "preload" directive if you understand the implications.
# Also don't include subdomains by default
#add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
add_header Strict-Transport-Security "max-age=63072000";
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
ssl_dhparam /etc/ssl/certs/dhparam.pem;

The SSL configuration in ssl-params.conf can be shared among multiple virtual servers.

Now generate a unique set of Diffie-Helman parameters (this mitigates the LOGJAM vulnerability):

openssl dhparam 2048 -out /etc/ssl/certs/dhparam.pem

Note that the above constitutes a MINIMUM RECOMMENDED LEVEL of security. Your installation’s requirements may be more stringent.

Apache Reverse Proxy with HTTPS (Linux)

To use Apache HTTPD as a reverse proxy, add the following virtual host to your configuration. Here we assume letsencrypt has been used to provide the certificate.

<VirtualHost *:443>
    ServerName kibi.example.com
    DocumentRoot /var/www/html
    DirectoryIndex index.html index.htm

    CustomLog /var/log/apache2/kibi-ssl.access.log combined
    ErrorLog /var/log/apache2/kibi-ssl.error.log

    SSLEngine on
    SSLStrictSNIVHostCheck off
    SSLCertificateFile /etc/letsencrypt/live/kibi.example.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/kibi.example.com/privkey.pem
    SSLCACertificateFile /etc/letsencrypt/live/kibi.example.com/chain.pem

    <location / >
        ProxyPass http://127.0.0.1:5606
        ProxyPassReverse http://127.0.0.1:5606
    </Location>
</VirtualHost>

Now configure /etc/apache2/conf.d/security.conf:

Header unset X-Powered-By
Header set X-Frame-Options: "sameorigin"
Header set X-Content-Type-Options: "nosniff"
TraceEnable Off
ServerTokens Prod
ServerSignature Off

And /etc/apache2/mods-available/ssl.conf:

<IfModule mod_ssl.c>

SSLRandomSeed startup builtin
SSLRandomSeed startup file:/dev/urandom 512
SSLRandomSeed connect builtin
SSLRandomSeed connect file:/dev/urandom 512

AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl	.crl

SSLPassPhraseDialog  exec:/usr/share/apache2/ask-for-passphrase
SSLSessionCache		shmcb:${APACHE_RUN_DIR}/ssl_scache(512000)
SSLSessionCacheTimeout  300

SSLProtocol all -SSLv2 -SSLv3
SSLHonorCipherOrder on
SSLCipherSuite \
  "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 \
  EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 \
  EECDH EDH+aRSA !3DES \
  !aNULL !eNULL !LOW !MD5 !EXP !PSK !SRP !KRB5 !DSS !RC4 !DES"
SSLCompression off

## Strict Transport Security
Header set Strict-Transport-Security "max-age=15768000"

## Apache 2.4 only
SSLUseStapling on
SSLStaplingResponderTimeout 5
SSLStaplingReturnResponderErrors off
SSLStaplingCache shmcb:/var/run/ocsp(128000)

## Apache >=2.4.8 + OpenSSL >=1.0.2 only
SSLOpenSSLConfCmd DHParameters /etc/ssl/certs/dhparam.pem

</IfModule>

You must enable mod_headers for the SSL security settings above to take effect.

Now generate a unique set of Diffie-Helman parameters (this mitigates the LOGJAM vulnerability):

openssl dhparam 2048 -out /etc/ssl/certs/dhparam.pem

Note that the above constitutes a MINIMUM RECOMMENDED LEVEL of security. Your installation’s requirements may be more stringent.

Native SSL Support

While you should always run Siren Investigate behind an SSL reverse proxy, it is sometimes necessary to also enable SSL support on the Siren Investigate server itself - for example, when the reverse proxy is an appliance, or is installed on a separate server.

Native SSL support can be enabled by copying the certificate and key files to a location readable by the Siren Investigate process and setting the following parameters in config/siren.yml:

  • server.ssl.enabled: set to true to enable SSL.

  • server.ssl.certificate: path to a certificate.

  • server.ssl.key: path to the certificate key.

  • server.ssl.keyPassphrase: the passphrase of the certificate key; if the key is not encrypted the parameter can be omitted.

The certificate and key files must be PEM encoded.

E.g.:

server.ssl.enabled: true
server.ssl.certificate: "pki/server.crt"
server.ssl.key: "pki/server.key"

The Siren Investigate demo distribution includes a sample certificate and key in the pki directory.

For additional SSL settings please refer to the settings chapter.

Enabling HTTPS in Gremlin Server

HTTPS must be enabled in Gremlin Server to secure requests from Siren Investigate, even if Siren Investigate is configured behind a reverse SSL proxy.

To enable HTTPS in the Gremlin Server, set the following parameters in the kibi_core.gremlin_server section of the config/siren.yml file:

  • url: the URL of the Gremlin Server endpoint; make sure that the protocol is set to https.

  • ssl.key_store: the path to the Gremlin Server certificate in Java KeyStore format.

  • ssl.key_store_password: the password of the Gremlin Server certificate keystore.

  • ssl.ca: the path of the certification authority chain bundle that can be used to validate requests from Siren Investigate to the Gremlin API; you can omit this parameter if the certificates for the Siren Investigate HTTPS interface have been issued and signed by a public authority.

E.g.:

kibi_core:
  gremlin_server:
    url: https://127.0.0.1:8061
    ssl:
      key_store: "pki/gremlin.jks"
      key_store_password: "password"
      ca: "pki/cacert.pem"

After restarting Siren Investigate, click on Settings, then click on Datasources, and make sure that the URL of the Siren Investigate Gremlin Server datasource is equal to the url set in siren.yml.

The Siren Investigate demo distribution includes a sample keystore and CA bundle in the pki directory.

Search Guard Integration and Siren Investigate access control

This section offers an overview of how to integrate Search Guard with Siren Investigate; for further reference and detailed options please consult the Search Guard documentation.

Before proceeding, make sure that:

  • Siren Investigate is either running with HTTPS enabled or behind a reverse SSL proxy.

  • The Gremlin Server is running with HTTPS enabled.

Please refer to the Authentication and access control section for instructions on how on to enable HTTPS in both components.

SSL Certificates

All the Elasticsearch nodes in a cluster secured by Search Guard are required to use SSL to encrypt all network traffic.

In addition, changing the Search Guard configuration requires the use of a client SSL certificate to perform administrative actions.

To setup a Search Guard cluster, you will need to generate the following files:

  • A truststore file, common to all nodes, containing the CA certificate chain.

  • A keystore file, for each node, containing the certificate bundle for the node.

  • A keystore file, for each administrative user, containing a certificate bundle that identifies the user.

  • A keystore file containing an SSL certificate for the Elasticsearch HTTP REST API.

These files can be either Java KeyStore files or PKCS12 bundles.

Sample certificates

The Siren Investigate demo distribution includes the following sample certificates in the elasticsearch/config directory:

  • truststore.jks: a sample CA certificate chain.

  • CN=localhost-keystore.jks: a certificate for the bundled Elasticsearch node, used for both transport and REST API encryption.

The password of all Java keystores is password.

In addition, the following certificates are included in the siren-investigate/pki/searchguard directory:

  • CN=sgadmin.crtfull.pem: a certificate bundle with administrative privileges over the Search Guard Management REST API.

  • CN=sgadmin.key.pem: the key of the administrative certificate.

  • ca.pem: the cluster CA certificate chain in PEM format.

Issuing certificates in an existing PKI infrastructure

If your organization has a PKI infrastructure in place, you can generate Java KeyStore files from a PEM bundle by using the keytool command from the Java runtime, e.g.:

$ keytool  \
  -importcert \
  -file ca.pem  \
  -keystore truststore.jks

The command will store the contents of the PEM bundle ca.pem into a file named truststore.jks in the current directory.

The same command can be used to convert certificates signed by your CA for nodes, administrative users and the REST API.

Node certificates must include oid:1.2.3.4.5.5 as a Subject Alternative Name entry to work correctly with Search Guard; if you want to enable hostname verification, make sure that at least one Subject Alternative Name is equal to the DNS name of the node.

Client certificates for administrative users must contain a unique Distinguished Name to identify the user, e.g.:

CN=admin,DC=siren,DC=solutions

Certificates for the REST API can be shared across multiple nodes by setting multiple hostnames in the Subject Alternative Name attribute or by using a wildcard certificate.

Issuing certificates using the TLS certificate generator

Floragunn GmbH provides a TLS certificate generation service which can be used to create a bundle of certificates for evaluation purposes.

To try the certificates in a single node setup, it is possible to just specify localhost as the first hostname and submit the form.

The bundle has the following contents:

  • README.txt: provides an overview of the bundle and the passwords for all the keystores.

  • truststore.jks: the CA certificate chain in Keystore format.

  • node-certificates: the transport certificates for the nodes in several formats; these certificates can also be used for the Elasticsearch REST API.

  • client-certificates: client certificates.

  • root-ca: the root CA bundle in PEM format.

  • signing-ca: the signing CA bundle in PEM format.

Search Guard installation

Install the search-guard-5 plugin on each node in the Elasticsearch cluster by changing to the node directory and running the commands below; to find the most recent version of the plugins for your Elasticsearch version please consult the Search Guard version matrix.

$ bin/elasticsearch-plugin install -b com.floragunn:search-guard-5:<version>

Then, copy the following files to the config directory of each node:

  • The truststore file (e.g. truststore.jks).

  • The keystore file containing the node certificate (e.g. CN=localhost-keystore.jks)

  • The keystore file containing the certificate for the Elasticsearch REST API if different than the node certificate.

Open the config/elasticsearch.yml file and set the following Search Guard options:

Node to node transport options:

  • searchguard.ssl.transport.enabled: needs to be set to true for Search Guard to work.

  • searchguard.ssl.transport.keystore_filepath: the filename of the keystore file that contains the node certificate.

  • searchguard.ssl.transport.keystore_password: the password of the keystore file that contains the node certificate.

  • searchguard.ssl.transport.truststore: the filename of the truststore file that contains the root certificate chain.

  • searchguard.ssl.transport.truststore_password: the password of the truststore file that contains the root certificate chain.

  • searchguard.ssl.transport.enforce_hostname_verification: set to true to enable hostname verification, false otherwise.

REST API options:

  • searchguard.ssl.http.enabled: set to true to enable SSL on the HTTP interface.

  • searchguard.ssl.http.keystore_filepath: the filename of the keystore file that contains the certificate for the HTTP interface.

  • searchguard.ssl.http.keystore_password: the password of the keystore file that contains the certificate for the HTTP interface.

  • searchguard.ssl.http.truststore: the filename of the truststore file that contains the root certificate chain for the HTTP certificate.

  • searchguard.ssl.http.truststore_password: the password of the truststore file that contains the root certificate chain for the HTTP certificate.

Administrative user options:

  • searchguard.authcz.admin_dn: a list of Distinguished Names in SSL client certificates which are authorized to submit administrative requests.

Client certificate authentication options:

  • searchguard.ssl.http.clientauth_mode: set to OPTIONAL to enable optional client certificate authentication on the REST endpoint.

E.g.:

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.truststore_password: password
searchguard.ssl.transport.keystore_filepath: CN=localhost-keystore.jks
searchguard.ssl.transport.keystore_password: password
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: CN=localhost-keystore.jks
searchguard.ssl.http.keystore_password: password
searchguard.ssl.http.truststore_filepath: truststore.jks
searchguard.ssl.http.truststore_password: password
searchguard.authcz.admin_dn:
  - CN=sgadmin
searchguard.ssl.http.clientauth_mode: OPTIONAL
Make sure that all the files in the configuration directory and the certificate files are readable only by the user running Elasticsearch.

Start Elasticsearch:

$ bin/elasticsearch

If either a certificate or a password is incorrect, Elasticsearch will not start.

Access control configuration

Access control configuration (users, roles and privileges) is stored in an Elasticsearch index which can be modified through the sgadmin.sh script.

The script reads the configuration from a local directory containing YAML files and uploads it to the index; the request is authenticated through a client SSL certificate.

Once the configuration has been uploaded, it will be available to all the nodes in the cluster, so it is not necessary to copy the Search Guard configuration directory to all the Elasticsearch nodes, just on the node from where sgadmin is run.

sgadmin.sh is available in the plugins/search-guard-5/tools directory in each Elasticsearch instance in which Search Guard has been installed; a standalone version (sgadmin-standalone.zip) can be downloaded from this page.

Once a Search Guard enabled cluster has been initialized, sgadmin can be used to upload new configurations.

Search Guard configuration

A Search Guard configuration directory contains the following files:

  • sg_config.yml: contains the general configuration.

  • sg_action_groups.yml: contains named groups of permissions.

  • sg_roles.yml: contains the definition of roles.

  • sg_internal_users.yml: the Search Guard internal users database.

  • sg_roles_mapping.yml: contains the mapping between users and roles.

A sample configuration is available in the config/sgconfig directory in the Elasticsearch instance included in the demo distribution; the contents of the files are explained in the next sections and can be used as a general guideline.

For additional configuration options please refer to the official Search Guard documentation.

General configuration (sg_config.yml)

searchguard:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
    authc:
      transport_auth_domain:
        enabled: true
        order: 2
        http_authenticator:
          type: basic
        authentication_backend:
          type: internal
      basic_internal_auth_domain:
        enabled: true
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern

The sg_config.yml file contains the configuration of the authentication mechanisms and backends; the above configuration:

  • Disables the anonymous role (anonymous_auth_enabled: false)

  • Disables support for external proxies (xff.enabled: false)

  • Enables HTTP basic authentication on the internal Search Guard user database.

Action groups (sg_action_groups.yml)

UNLIMITED:
  - '*'

###### INDEX LEVEL ######

INDICES_ALL:
  - 'indices:*'

# for backward compatibility
ALL:
  - INDICES_ALL

MANAGE:
  - 'indices:monitor/*'
  - 'indices:admin/*'

CREATE_INDEX:
  - 'indices:admin/create'
  - 'indices:admin/mapping/put'

MANAGE_ALIASES:
  - 'indices:admin/aliases*'

# for backward compatibility
MONITOR:
  - INDICES_MONITOR

INDICES_MONITOR:
  - 'indices:monitor/*'

DATA_ACCESS:
  - 'indices:data/*'
  - CRUD

WRITE:
  - 'indices:data/write*'
  - 'indices:admin/mapping/put'

READ:
  - 'indices:data/read*'
  - 'indices:admin/mappings/fields/get*'

DELETE:
  - 'indices:data/write/delete*'

CRUD:
  - READ
  - WRITE

SEARCH:
  - 'indices:data/read/search*'
  - 'indices:data/read/msearch*'
  - 'indices:siren/plan*'
  - 'indices:siren/mplan*'
  - SUGGEST

SUGGEST:
  - 'indices:data/read/suggest*'

INDEX:
  - 'indices:data/write/index*'
  - 'indices:data/write/update*'
  - 'indices:admin/mapping/put'
  - 'indices:data/write/bulk*'

GET:
  - 'indices:data/read/get*'
  - 'indices:data/read/mget*'

###### CLUSTER LEVEL ######

CLUSTER_ALL:
  - 'cluster:*'

CLUSTER_MONITOR:
  - 'cluster:monitor/*'

CLUSTER_COMPOSITE_OPS_RO:
  - 'indices:data/read/mget'
  - 'indices:data/read/msearch'
  - 'indices:siren/mplan'
  - 'indices:data/read/mtv'
  - 'indices:admin/aliases/exists*'
  - 'indices:admin/aliases/get*'

CLUSTER_COMPOSITE_OPS:
  - 'indices:data/write/bulk'
  - 'indices:admin/aliases*'
  - CLUSTER_COMPOSITE_OPS_RO

##### SIREN #####

SIREN_CLUSTER:
  - 'indices:data/read/scroll'
  - 'indices:data/read/scroll/clear'
  - 'cluster:internal/data/transfer/*'
  - 'indices:data/read/msearch*'
  - CLUSTER_COMPOSITE_OPS_RO

SIREN_COMPOSITE:
  - 'indices:siren/mplan*'

SIREN_READONLY:
  - 'indices:data/read/field_stats*'
  - 'indices:data/read/field_caps*'
  - 'indices:data/read/get*'
  - 'indices:data/read/mget*'
  - 'indices:data/read/search*'
  - 'indices:siren/plan'
  - 'indices:siren/task/search'
  - 'indices:admin/mappings/get*'
  - 'indices:admin/mappings/fields/get*'
  - 'indices:admin/validate/query*'
  - 'indices:admin/get*'
  - 'indices:admin/version/get*'
  - SIREN_COMPOSITE

SIREN_READWRITE:
  - 'indices:admin/exists*'
  - 'indices:admin/mapping/put*'
  - 'indices:admin/refresh*'
  - 'indices:data/write/delete*'
  - 'indices:data/write/index*'
  - 'indices:data/write/update*'
  - SIREN_READONLY

This file contains named groups of permissions which can be used in the roles configuration file; the above configuration includes Search Guard default groups plus three Siren Investigate specific groups:

  • SIREN_READWRITE: groups all the permissions needed to search and update the main Siren Investigate index (.siren); the group has to be assigned on the main index to all roles that can modify the Siren Investigate configuration.

  • SIREN_READONLY: groups all the permissions needed to search any Elasticsearch index from Siren Investigate. The group has to be assigned on all indices that a role has access to.

  • SIREN_CLUSTER: sets the permission to read results from scrolling searches and send composite requests.

  • SIREN_COMPOSITE: groups all the permissions to execute composite requests not recognized by Search Guard; the group has to be granted on all indices to roles that have access only to a subset of indices (e.g. sirennoinvestor).

Roles (sg_roles.yml)

# Allows any action on the cluster.
sg_all_access:
  cluster:
    - '*'
  indices:
    '*':
      '*':
        - '*'

# Allows reading data from all indices.
sg_readall:
  indices:
    '*':
      '*':
        - READ

# Permissions for a Logstash client.
logstash:
  cluster:
    - 'indices:data/write/bulk*'
    - 'indices:admin/template/*'
    - CLUSTER_MONITOR
    - SIREN_CLUSTER
  indices:
    'logstash-*':
      '*':
        - CRUD
        - CREATE_INDEX
    '*beat*':
      '*':
        - CRUD
        - CREATE_INDEX

# Permissions for an X-Pack monitoring agent.
monitoring:
  cluster:
    - CLUSTER_MONITOR
    - 'indices:admin/aliases'
    - 'indices:admin/template/get'
    - 'indices:admin/template/put'
    - 'cluster:admin/ingest/pipeline/get'
    - 'cluster:admin/ingest/pipeline/put'
    - 'indices:data/write/bulk'
  indices:
    '?marvel*':
      '*':
        - ALL
    '?monitoring*':
      '*':
        - ALL

# Permissions for a Siren Alert user.
sirenalert:
  cluster:
    - SIREN_CLUSTER
    - 'indices:data/write/bulk*'
    - 'indices:admin/template/*'
  indices:
    '*':
      '*':
        - SIREN_READONLY
    'watcher_alarms*':
      '*':
        - SIREN_READWRITE
        - CREATE_INDEX
    '/(watcher|watcher_alarms)/':
      '*':
        - SIREN_READWRITE
        - CREATE_INDEX

# Permissions for the Siren Investigate server process.
sirenserver:
  cluster:
    - cluster:admin/xpack/monitoring/bulk
    - cluster:monitor/nodes/info
    - cluster:monitor/xpack/info
    - cluster:monitor/health
    - cluster:monitor/main
    - cluster:monitor/state
    - cluster:monitor/nodes/stats
    - SIREN_CLUSTER
    - CLUSTER_COMPOSITE_OPS
  indices:
    '*':
      '*':
        - indices:admin/get
    '?siren':
      '*':
        - ALL
    '?sirenaccess':
      '*':
        - ALL

# Permissions for a Siren Investigate administrator (read-write access to the .siren index).
sirenadmin:
  cluster:
    - SIREN_CLUSTER
    - cluster:admin/plugin/siren/license/put
  indices:
    '*':
      '*':
        - SIREN_READONLY
    '?siren':
      '*':
        - SIREN_READWRITE
    'watcher':
      '*':
        - SIREN_READWRITE

# Permissions for a Siren Investigate user (read only access to the .siren index).
sirenuser:
  cluster:
    - SIREN_CLUSTER
  indices:
    '?siren':
      '*':
        - SIREN_READONLY
    'watcher':
      '*':
        - SIREN_READONLY
    'watcher_alarms*':
      '*':
        - SIREN_READONLY
    'article':
      '*':
        - SIREN_READONLY
    'investment':
      '*':
        - SIREN_READONLY
    'company':
      '*':
        - SIREN_READONLY
    'investor':
      '*':
        - SIREN_READONLY
    '*':
      '*':
        - SIREN_COMPOSITE

# Permissions for a Siren Investigate user (read only), with no access to the investor index.
sirennoinvestor:
  cluster:
    - SIREN_CLUSTER
  indices:
    '?siren':
      '*':
        - SIREN_READONLY
    'article':
      '*':
        - SIREN_READONLY
    'company':
      '*':
        - SIREN_READONLY
    'investment':
      '*':
        - SIREN_READONLY
    '*':
      '*':
        - SIREN_COMPOSITE

The file defines the following roles:

  • sg_all_access: allows every action on the cluster.

  • sg_readall: allows to search data on all the indices in the cluster.

  • logstash: defines the permission for a Logstash client with all write and creation privileges enabled on Logstash and Elastic Beats templates and indices.

  • sirenalert: defines the permission for a Siren Alert user; the role is not required if the Siren Alert plugin is not installed.

  • monitoring: defines the permissions for an X-Pack monitoring agent.

  • sirenserver: defines the permissions for the Siren Investigate server process.

  • sirenadmin: defines the permissions for a Siren Investigate user with read/write access to the .siren index.

  • sirenuser: defines the permissions for a Siren Investigate user with readonly access to all indices.

  • sirennoinvestor: defines the permissions for a Siren Investigate user with readonly access to all the indices excluding investor.

A permission is defined by the following syntax:

<username>:
  <indices or cluster>:
    '<index name or regular expression>':
      '<type name or regular expression>':
        - <list of permissions or action group names>

The index name can contain the simple expansion characters * and ? to match any sequence of character/any single character; for further information about defining permissions, please refer to the Search Guard configuration documentation.

Users (sg_internal_users.yml)

# Internal user database
# The hash value is a bcrypt hash and can be generated with plugins/search-guard-5/tools/hash.sh
admin:
  hash: $2a$12$zMeFc6Xi.pcgDVHsvtCV9ePNteVwTE5uGxcKdf7XQcKB9.VkD8iOy
sirenserver:
  hash: $2a$12$zMeFc6Xi.pcgDVHsvtCV9ePNteVwTE5uGxcKdf7XQcKB9.VkD8iOy
sirenadmin:
  hash: $2a$12$zMeFc6Xi.pcgDVHsvtCV9ePNteVwTE5uGxcKdf7XQcKB9.VkD8iOy
sirenuser:
  hash: $2a$12$zMeFc6Xi.pcgDVHsvtCV9ePNteVwTE5uGxcKdf7XQcKB9.VkD8iOy
sirennoinvestor:
  hash: $2a$12$zMeFc6Xi.pcgDVHsvtCV9ePNteVwTE5uGxcKdf7XQcKB9.VkD8iOy
logstash:
  hash: $2a$12$zMeFc6Xi.pcgDVHsvtCV9ePNteVwTE5uGxcKdf7XQcKB9.VkD8iOy
CN=demouser:
  hash: $2a$12$zMeFc6Xi.pcgDVHsvtCV9ePNteVwTE5uGxcKdf7XQcKB9.VkD8iOy
sirenalert:
  hash: $2a$12$zMeFc6Xi.pcgDVHsvtCV9ePNteVwTE5uGxcKdf7XQcKB9.VkD8iOy
monitoring:
  hash: $2a$12$zMeFc6Xi.pcgDVHsvtCV9ePNteVwTE5uGxcKdf7XQcKB9.VkD8iOy

The file defines the credentials for Search Guard internal users; passwords are stored as hashes in the hash attribute beneath each username.

The password for all the accounts above is password.

To change the password of a user, you will need to generate the corresponding hash; this can be done by executing the plugins/search-guard-5/tools/hash.sh script as follows:

$ bash plugins/search-guard-5/tools/hash.sh -p password

The script will output the hash for the password specified after the -p switch.

Role mappings (sg_roles_mapping.yml)

sg_all_access:
  users:
    - admin

sirenserver:
  users:
    - sirenserver

sirenadmin:
  users:
    - sirenadmin

sirenuser:
  users:
    - sirenuser

sirennoinvestor:
  users:
    - sirennoinvestor

logstash:
  users:
    - logstash

sirenalert:
  users:
    - sirenalert

monitoring:
  users:
    - monitoring

The file defines the assignment of roles to users; users authenticating through a client certificate are identified by the Distinguished Name in the certificate.

Uploading the configuration to the cluster

To upload the configuration defined in the previous steps, go to the Elasticsearch directory and execute the plugins/search-guard-5/tools/sgadmin.sh script as follows:

$ bash plugins/search-guard-5/tools/sgadmin.sh \
  -cd config/sgconfig \
  -cn siren-distribution \
  -ts config/truststore.jks \
  -tspass password \
  -ks ../siren-investigate/pki/searchguard/CN\=sgadmin-keystore.jks \
  -kspass password \
  -h localhost \
  -p 9330 \
  -nhnv

To reload the configuration you have to use the same same command with the -rl flag instead of -cd, e.g.:

$ bash plugins/search-guard-5/tools/sgadmin.sh \
  -rl
  -cn siren-distribution \
  -ts config/truststore.jks \
  -tspass password \
  -ks ../siren-investigate/pki/searchguard/CN\=sgadmin-keystore.jks \
  -kspass password \
  -h localhost \
  -p 9330 \
  -nhnv

You will need to specify the following arguments based on your environment configuration:

  • -cd: the path to the directory containing the Search Guard access control configuration.

  • -cn: the name of the Elasticsearch cluster.

  • -ts: the path to the truststore file.

  • -tspass: the password of the truststore file.

  • -ks: the path to the administrative client certificate keystore.

  • -kspass: the password of the client certificate keystore file.

  • -h: the hostname of a node in the cluster.

  • -p: the transport port of the node specified in the -h option.

  • -nhnv: disables host name verification; remove this option if you installed node certificates with the correct hostname (recommended in production).

  • -rl: reloads the configuration and flushes the internal cache.

By default the number of replicas for the searchguard index will be set at creation time to the number of data nodes - 1.

For additional information on how to set replication settings and sgadmin in general please refer to the sgadmin documentation.

If the command executes successfully it will print a summary of the actions executed, e.g.:

Clustername: elasticsearch
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
searchguard index does not exists, attempt to create it ... done
Populate config from /elasticsearch/sg_config
Will update 'config' with sg_config/sg_config.yml
   SUCC: Configuration for 'config' created or updated
Will update 'roles' with sg_config/sg_roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update 'rolesmapping' with sg_config/sg_roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update 'internalusers' with sg_config/sg_internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update 'actiongroups' with sg_config/sg_action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Done with success

You can then verify that SSL and authentication are enabled by making an authenticated request with wget, e.g.:

$ wget --ca-certificate=../siren-investigate/pki/searchguard/ca.pem --http-user=sirenserver --http-password=password -qO - https://localhost:9220

To display information about the certificate as seen by a client you can execute the following command:

$ echo | openssl s_client -servername localhost -connect localhost:9220 -showcerts | openssl x509 -text -inform pem -text -noout

Siren Investigate configuration

Edit config/investigate.yml and specify the credentials of the sirenserver user, e.g.:

elasticsearch.username: 'sirenserver'
elasticsearch.password: 'password'

If HTTPS is enabled for the Elasticsearch REST API, make sure that the elasticsearch.url setting contains a URL starting with https, e.g.:

elasticsearch.url: 'https://localhost:9220'

If the certificate is not signed by a public authority, you will also need to set the elasticsearch.ssl.ca to the path of the CA chain bundle in PEM format, e.g.:

elasticsearch.ssl.ca: 'pki/searchguard/ca.pem'

If you’re using the certificates generated by the TLS generator service, the PEM file containing the certification bundles is available in root-ca/root-ca.pem.

To enable certificate verification, set elasticsearch.ssl.verify to true, e.g.:

elasticsearch.ssl.verify: true

Set the investigate_core.elasticsearch.auth_plugin option to searchguard:

investigate_core:
  elasticsearch:
    auth_plugin: 'searchguard'

E.g.:

investigate_core:
  elasticsearch:
    auth_plugin: 'searchguard'

To enable the Siren Investigate access control plugin, specify the following configuration values in the investigate_access_control section:

  • enabled: set to true to enable the Siren Investigate access control plugin. Defaults to false.

  • backend: backend type of authentication. Currently available backends are searchguard and xpack. Defaults to searchguard.

  • session.ttl: the lifetime of the session in milliseconds. If not set, the session will last as long as the session cookie is valid. Defaults to 3600000 (1 hour).

  • session.keepAlive: if set to true, every time a request is received within the session lifetime, the session lifetime will be extended by session.ttl. Defaults to true.

  • cookie.password: a 32 characters long password used to derive the key used to encrypt and sign cookies.

  • cookie.secure: if set to true, the cookie will be transmitted only if the request is being served over HTTPS. It is possible to set this to false if Siren Investigate is behind an SSL proxy. Defaults to true.

  • cookie.ttl: the lifetime of the session cookie in milliseconds. If not set, the cookie will expire when the browser is closed, which is the recommended setting. Please note that browsers might not remove session cookies when a tab is closed or even across restarts, so it is recommended to set session.ttl for additional protection. Defaults to null.

  • cookie.name: the name of the session cookie. Defaults to kac.

  • admin_role: the Search Guard role authorized to use the Siren Investigate Access Control application. Defaults to sirenadmin.

E.g.:

investigate_access_control:
  enabled: true
  cookie:
    password: '12345678123456781234567812345678'

If Siren Investigate is running behind a reverse SSL proxy like Nginx, remember to set cookie.secure to false otherwise the cookie will not be sent, e.g.:

investigate_access_control:
  enabled: true
  cookie:
    password: '12345678123456781234567812345678'
    secure: false

If you are using the Siren Alert plugin, you can specify the Siren Alert user credentials in the investigate_access_control.sirenalert section, e.g.:

investigate_access_control:
  sirenalert:
    elasticsearch:
      username: sirenalert
      password: password

Restart Siren Investigate after changing the configuration file; if the configuration is correct, you should see an authentication dialog when browsing to Siren Investigate.

Authentication dialog
Figure 30. Authentication dialog

Search Guard management UI

Siren Investigate includes an optional user interface for the Search Guard REST Management API add-on ; in order to use it, the Siren Investigate backend has to connect to the Elasticsearch cluster using a PEM client certificate with administrative privileges.

It is strongly suggested to setup a dedicated Siren Investigate instance to use the Search Guard management UI and allow access to it only to authorized users.

Add-on installation

To install the Search Guard REST Management API add-on it is required to download the correct jar for your Elasticsearch / Search Guard version from this page and copy it to the plugins/search-guard-5 directory of each node in the cluster.

To access the API it is required to use a client certificate with administrative privileges; to enable optional client certificate authentication on the REST interface, ensure that the following option is present in elasticsearch.yml:

searchguard.ssl.http.clientauth_mode: OPTIONAL

Once the plugin has been copied and the configuration updated, the nodes must be restarted; a rolling restart is enough to install the add-on.

When using this add-on, make sure that the sgadmin configuration directory contains only the sg_config.yml file, otherwise sgadmin will replace users, roles, action groups and mappings that might have been modified through the API.

Siren Investigate configuration

Copy the client certificate and its key to a directory readable by Siren Investigate (e.g. pki); then add the following parameters to the investigate_access_control configuration section:

  • admin_role: the Search Guard role that has access to the Search Guard management UI (sirenadmin by default).

  • backends.searchguard.admin.ssl.cert: the path to the administrative client certificate bundle in PEM format.

  • backends.searchguard.admin.ssl.key: the path to the administrative client certificate key in PEM format.

E.g.:

investigate_access_control:
  admin_role: sirenadmin
  backends:
    searchguard:
      admin.ssl.cert: pki/searchguard/CN=sgadmin.crtfull.pem
      admin.ssl.key: pki/searchguard/CN=sgadmin.key.pem

Please note that the administrative client certificate bundle must contain both the full CA chain and the client certificate; if using certificates generated by the TLS generation service, the file name will be CN=sgadmin.crtfull.pem, otherwise it is possible to generate the bundle manually by using cat, e.g.:

$ cat user.crt.pem ca-chain.pem > user.crtfull.pem

Once the certificate is setup, restart Siren Investigate, login with a user having an administrative role, click on the apps button, then click on Access control and finally on Authentication.

The Access control app
Figure 31. The Access control app

If you get an error upon opening the Authentication app, most probably the client certificate does not contain the full CA chain or the add-on has not been installed correctly; please check Elasticsearch and Siren Investigate logs for related errors.

The Authentication section allows to browse, edit and create the following Search Guard resources:

  • Internal users

  • Roles

  • Role mappings

  • Action groups

To verify that the application is working correctly, click on Roles then click on the Open button; you should see the list of roles defined during the initial Search Guard setup or an authorization error if the certificate is incorrect:

Browsing Search Guard roles
Figure 32. Browsing Search Guard roles

Saved objects access control

Siren Investigate features an access control system on saved objects that allows to filter dashboards and visualizations visible to end users.

Setup

To enable this feature, set the following parameters in the investigate_access_control configuration section:

  • admin_role: the Search Guard role that can use the saved objects access control management UI (sirenadmin by default).

  • acl.enabled: set to true to enable access control on saved objects.

  • acl.index: the Elasticsearch index in which access control rules and saved objects metadata will be stored (.sirenaccess by default).

E.g.:

investigate_access_control:
  acl:
    enabled: true

Before restarting Siren Investigate, it is required to allow the backend user (.sirenserver by default) all permissions on the index set in investigate_access_control.acl.index; for example, the following snippet from sg_roles.yml grants all privileges to the sirenserver user on the .sirenaccess index.

# Permissions for the Siren Investigate server process.
sirenserver:
  cluster:
      - cluster:monitor/nodes/info
      - cluster:monitor/health
      - cluster:monitor/state
      - cluster:monitor/nodes/stats
      - SIREN_CLUSTER
  indices:
    '*':
      '*':
        - indices:admin/get
    '?siren*':
      '*':
        - ALL
    '?sirenaccess':
      '*':
        - ALL

In addition, it is recommended to block access on the Siren Investigate index (.siren by default) to users by adding the following permissions on the null type in each user role:

  • indices:data/read/search

E.g.:

sirenuser:
  cluster:
    - SIREN_CLUSTER
  indices:
    /(article|investment|company|investor)/:
      '*':
        - SIREN_READONLY
    watcher:
      '*':
        - SIREN_READONLY
    '*':
      '*':
        - SIREN_COMPOSITE
    watcher_alarms*:
      '*':
        - SIREN_READONLY
    '?siren':
      'null':
        - 'indices:data/read/search'
        - 'indices:data/read/coordinate-search'

In this way, users will be able to include the Siren Investigate index in msearch requests (which is a performance requirement to avoid querying all indices when time based index patterns are configured) but won’t be able to read saved objects from it.

Roles can be updated either by modifying sg_roles.yml and uploading it through sgadmin or by using the Search Guard management application; make sure to set these rules on all Search Guard roles assigned to Siren Investigate users.

Once roles are configured, restart Siren Investigate; if permissions are configured correctly, you will be see an ACL section in the Access control application.

The ACL section
Figure 33. The ACL section

Siren Investigate roles and rules

The ACL Roles panel in the ACL section allows to define Siren Investigate roles, which are collections of permissions on saved objects and UI elements. The main purpose of this system is to hide and block access to:

  • UI elements - applications, e.g.: Timelion, Access control, Siren Alert

  • UI elements - specific functionalities e.g.: export CSV feature

  • UI elements - Siren Investigate sections, e.g.: discover, management

  • Saved objects on unauthorized indices, e.g.: dashboards, searches

to end users and avoid unauthorized changes to configuration objects or use of certain parts of the system.

There are two kinds of rules:

  • rules - to set permissions for saved objects

  • ui rules - to set permissions to view different UI elements

The everyone role defines permissions for all the users in the system, and is mapped by default to any user logged in Siren Investigate; by default it grants all users read only access to the Siren Investigate configuration (Advanced settings), saved searches and index patterns as well as permission to view all applications and UI elements.

The everyone role
Figure 34. The everyone role

Denying access to certain saved objects like saved search using the first sets of rules is usually transparent to the user which means that he will simply not see the objects anywhere in Siren Investigate.

Usually it is not required to create explicit UI rules for the dashboard application as access to specific dashboards can be restricted through saved object rules.

Denying access to an application like Timelion or a Siren Investigate section like management will hide the navigation menu elements, block access at the route level and display an error.

Blocked Timelion application and Siren Investigate management section
Figure 35. Blocked Timelion application and Siren Investigate management section

When the user tries to access app/timelion, the error below is shown.

Blocked timelion error
Figure 36. Blocked Timelion error

When the user tries to access /app/kibana#/management, the error below is shown.

Blocked Siren Investigate management section error
Figure 37. Blocked Siren Investigate management section error

For most setups it makes sense to grant view permissions on visualizations as well, then set specific permissions on dashboards and dashboard groups for each role.

To define a new role, click on the Create role button, then set the following parameters:

  • Role ID: the ID of the role (e.g. sirenuser); must be a lowercase alphanumeric string.

  • Backend roles: a list of Search Guard roles that will be mapped to this Siren Investigate role (e.g. sirenuser)

  • Rules: a list of rules on saved object types.

Each rule is defined by three parameters:

  • Action: allow or deny

  • Permission: the permission to allow or deny

  • Context: the saved object type on which the permission must be enforced.

The Create role button
Figure 38. The Create role button
Saving a role
Figure 39. Saving a role

Object permissions

In addition to role level permissions, it is possible to define permissions on specific objects by visiting Settings > Objects and clicking on the permissions button next to an object:

The object permissions button
Figure 40. The object permissions button

The object permissions form allows to set the owner of the object and custom access rules.

By default the owner is set to the user that created the object; the owner has all permissions on the created object; it is possible to unset the owner of an object by leaving the field blank and clicking on the Save button.

Custom access rules can be used to grant access to an object that would be otherwise hidden; for example, if everyone is not granted to display dashboards but you want to display the Overview dashboard to all users, visit the object permissions form for the Overview dashboard and set the View permission for everyone to Allow.

If everyone can see dashboards but you’d like to hide the IT dashboard to users, set the View permission for everyone to Deny.

The object permissions form
Figure 41. The object permissions form

Notes

Although users are not allowed to view or edit the following types unless they have permission to do so, they will be retrieved and executed by the backend if used by a visualization:

  • Query

  • Query templates

  • Data source

Logstash configuration

To enable authentication in Logstash, set the following parameters in the output.elasticsearch section:

  • user: the username of the user having the logstash role.

  • password: the password of the user having the logstash role.

  • ssl: set to true to enable SSL.

  • truststore: the path to the CA truststore file.

  • truststore_password: the password of the CA truststore file.

E.g.:

output {
    elasticsearch {
       hosts => ['localhost:9220']
       user => logstash
       password => password
       ssl => true
       truststore => '/etc/pki/logstash/truststore.jks'
       truststore_password => password
    }
}

The truststore file must be copied on all nodes running Logstash.

Beats configuration

To enable authentication in a beat which connects directly to Elasticsearch, set the following parameters in the output.elasticsearch section:

  • protocol: set to https.

  • username: the username of the user having the logstash role.

  • password: the password of the user having the logstash role.

  • tls.certificate_authorities: an array containing the path to the CA truststore file in PEM format.

E.g.:

output:

  elasticsearch:
    hosts: ['localhost:9220']

    protocol: 'https'
    username: 'logstash'
    password: 'password'

    tls:
      certificate_authorities: ['/etc/pki/filebeat/ca.pem']

The root certification authority in PEM format must be copied to all nodes running one or more beats.

Console configuration

In order to successfully submit queries from console to a cluster secured by Search Guard set the following parameters in config/siren.yml:

console.proxyConfig:
  - match:
      protocol: 'https'

    ssl:
      ca: 'pki/searchguard/ca.pem'

console.proxyConfig.ssl.ca must point to the CA certificate bundle, so it can be set to the same value as the elasticsearch.ssl.ca parameter.

X-Pack monitoring configuration

In order to store monitoring data in a cluster secured by Search Guard it is required to configure agent exporters to submit data over an authenticated HTTPS connection.

The exporter configuration in elasticsearch.yml must include the following parameters:

  • type: http.

  • host: an array of URLs that will be contacted by the exporter.

  • auth.username: the username of the Marvel agent user.

  • auth.password: the password of the Marvel agent user.

  • ssl.truststore.path: the path to the CA certificate truststore (this will usually be the same as the one specified in the Search Guard configuration).

  • ssl.truststore.password: the password of the CA certificate truststore.

For example, the following configuration defines an exporters which sends data to the cluster at https://localhost:9220, authenticating as the monitoring user:

xpack.monitoring.exporters:
  id1:
    type: http
    host: ['https://localhost:9220']

    auth:
      username: monitoring
      password: password

    ssl:
      truststore.path: truststore.jks
      truststore.password: password

Handling of authorization errors

Siren Investigate parses generic authorization errors from Elasticsearch to report them in a more understandable way.

While using Siren Investigate on a secured cluster, you might see the following errors:

  • "Siren Investigate Relational Filter: Could not load filter Relational visualization: one or more join relations refers to unauthorized data": displayed when a search query contains relations between unauthorized indices.

  • "One or more visualizations Refers to unauthorized data": displayed when a dashboard contains one or more visualizations loading data from unauthorized indices.

  • "Enhanced search results: Refers to unauthorized data": displayed when an "Enhanced search results" visualization tries to load data from unauthorized indices.

  • "Siren Investigate Relational Filter: there are relations with unauthorized data": displayed when there are relationships between unauthorized indices.

  • "One or more saved search refers to unauthorized data": displayed when a saved search is executed on an unauthorized index.

These errors appear for five seconds as a yellow toaster at the top of the screen.

Authorization errors can be hidden by changing the value of the kibi:shieldAuthorizationWarning setting: click on the Settings tab, then on Advanced tab, and uncheck the setting to hide authorization errors messages. Regardless of this setting, all authorization errors will still be reported as warnings in the logs.

Two warning messages example
Figure 42. Two warning messages example

If a dashboard is configured on an index on which you have no permission, all the visualizations loading data from the index will be empty and the custom warning message will be displayed at the top of the screen; in addition, the document count for the dashboard will display the message Unauthorized.

If a dashboard is configured on an index on which you have permission to see only a subset of documents or fields, the visualizations will only process and display the data you’re authorized to view.

A visualization that loads unauthorized data.
Figure 43. A visualizations that loads unauthorized data.
A visualization that refers to an unauthorized index.
Figure 44. A visualizations that refers to an unauthorized index.

X-Pack integration and Siren Investigate access control

If you have an existing Elasticsearch instance with X-Pack installed and wish to integrate with Siren Investigate, there are a few short steps to take:

First, set the backend parameter of the investigate_access_control section of the siren.yml to xpack:

investigate_access_control:
  enabled: true
  backend: xpack

Next, replace the sirenserver role with the following settings; If using a custom configuration, replace the kibi (default: .siren) and access control (default: .sirenaccess) indices with your index names

{
  "cluster": [
    "cluster:admin/plugin/siren/license/get",
    "monitor"
  ],
  "indices" : [
    {
      "names" : [ "*" ],
      "privileges" : [ "indices:admin/get" ]
    },
    {
      "names" : [ ".siren*" ],
      "privileges" : [ "all" ]
    },
    {
      "names" : [ ".sirenaccess" ],
      "privileges" : [ "all" ]
    }
  ]
}

Once you log into Siren Investigate, you can configure roles as needed from Siren Investigate’s Access Control section. You can find an example role below. See the Access Control section for more information.

{
  "cluster": [
    "cluster:admin/plugin/siren/license/get",
    "cluster:internal/data/transfer/start",
    "cluster:internal/data/transfer/delete",
    "cluster:internal/data/transfer/end",
    "cluster:internal/data/transfer/packet"
  ],
  "indices" : [
    {
      "names" : [ "*" ],
      "privileges" : [ "indices:siren/mplan" ]
    },
    {
      "names" : [ "company" ],
      "privileges" : [ "read", "view_index_metadata", "indices:siren", "indices:admin/version/get" ],
      "field_security": {
        "grant" : [ "*" ],
        "except": [ "city" ]
      },
      "query": {
        "match": {
          "category_code": "software"
        }
      }
    },
    {
      "names" : [ "article", "investment" ],
      "privileges" : [ "read", "view_index_metadata", "indices:siren", "indices:admin/version/get" ]
    },
    {
      "names" : [ ".siren*" ],
      "privileges" : [ "read" ]
    }
  ]
}

To enable Kibana monitoring in X-Pack, you’ll need to grant the cluster:admin/xpack/monitoring/bulk permission to the sirenserver role; the permission can be added to the existing permissions in the cluster section of the role.

See the Access control configuration section for more information.

If you do not need Kibana monitoring, you can add the following lines to siren.yml to disable it:

xpack:
  monitoring:
    kibana:
      collection:
        enabled: false

Kerberos/SPNEGO authentication support

This section offers an overview of how to enable Kerberos/SPNEGO authentication in Siren Investigate.

Before enabling Kerberos support you should setup Siren Investigate and Search Guard as described in the Search Guard Integration and Siren Investigate access control chapter.

Limitations

The current implementation requires disabling the Kerberos replay cache in Search Guard, as the Siren Investigate backend needs to make multiple requests to the Elasticsearch cluster on behalf of the user in several places without the ability to generate new service tickets.

As long as all the traffic to Siren Investigate is encrypted and the service ticket lifetime is short (the default in most system is 5 to 10 minutes) this should not pose a significant security risk.

Pre requisites

Service Principal

In order to enable Kerberos authentication, you need to create a service Principal to identify the Elasticsearch REST interface; usually the principal name is HTTP/<public DNS name of the cluster> (e.g. HTTP/es.ad.local).

Active Directory

On an Active Directory domain controller it is possible to use the setspn command to set a Service Principal Name for a domain user; for example, the following command run in an elevated command prompt associates the Service Principal Name HTTP/es.ad.local to a user named elasticsearch:

setspn -A HTTP/es.cluster.local elasticsearch

Please refer to the Active Directory documentation for more details about setspn and Kerberos integration.

Keytab

Once the service Principal is defined, you need to generate a keytab file that will be used by the Kerberos add-on to authenticate with the KDC.

Active Directory

On an Active Directory domain controller you can generate a keytab by running the ktpass command in an elevated command prompt as follows:

ktpass -out es.keytab -princ <principal name>@<domain> /mapuser <principal user> /pass "<principal user password>" /kvno 0

For example, to generate a keytab for the SPN HTTP/es.ad.local, associated to elasticsearch user in the AD.LOCAL domain, you need to run the following command:

ktpass -out es.keytab -princ HTTP/es.ad.local@AD.LOCAL /mapuser elasticsearch /pass "password" /kvno 0

Verification

This verification step is optional but it is useful to ensure that the keytab is correct before configuring Search Guard.

To verify that the keytab works correctly, copy it to a different machine with access to the KDC / Domain controller; the keytab contains the credentials of the service principal user so it should be removed from any intermediate machine used to transfer the file the transfer and from the target machine once the test is complete.

Create a file named krb5.conf in the same directory as the keytab with the contents below; replace AD.LOCAL with your domain name and DC.AD.LOCAL with the name or IP address of your KDC or domain controller, keeping the case of domains as in the example:

[libdefaults]
default_realm = AD.LOCAL
forwardable=true
default_tkt_enctypes = rc4-hmac,aes256-cts-hmac-sha1-96,aes128-cts-hmac-sha1-96
default_tgs_enctypes = rc4-hmac,aes256-cts-hmac-sha1-96,aes128-cts-hmac-sha1-96

[realms]
AD.LOCAL = {
kdc = dc.ad.local:88
default_domain = ad.local
}

[domain_realm]
.ad.local = AD.LOCAL
ad.local = AD.LOCAL
*nix systems

On Linux and MacOS systems, set the KRB5_CONFIG variable temporarily to point to the absolute path of the file created before and run kinit -t <keytab> <principal>, e.g.:

KRB5_CONFIG=./krb5.conf kinit -t es.keytab HTTP/es.ad.local

If the keytab is correct, kinit should exit immediately and not show a password prompt; to verify that the ticket has been issued, execute the klist -v command and check that it outputs the details of the ticket:

klist -v
Credentials cache: API:123
        Principal: HTTP/es.ad.local@ES.AD.LOCAL
    Cache version: 0

Server: krbtgt/AD.LOCAL@AD.LOCAL
Client: HTTP/es.ad.local@AD.LOCAL
Ticket etype: aes256-cts-hmac-sha1-96, kvno 2
Session key: arcfour-hmac-md5
Ticket length: 1194
Auth time:  May 12 19:59:10 2017
End time:   May 13 05:59:10 2017
Ticket flags: enc-pa-rep, pre-authent, initial, forwardable
Addresses: addressless

You can then destroy the ticket by executing the kdestroy command.

Windows systems

If you’re running Elasticsearch nodes on Windows, you can use the Kerberos tools bundled with the Java Runtime Environment to verify the keytab.

If the JRE directory is not in the system path, prepend it to each command.

Execute kinit <principal> -t <keytab> -J-Djava.security.krb5.conf=<path to krb5.conf> to get a ticket, e.g.:

kinit HTTP/es.ad.local -t es.keytab -J-D"java.security.krb5.conf=C:\Users\test\krb5.conf"

If the keytab is correct kinit will print the path to the file where the ticket has been saved, e.g.:

New ticket is stored in cache file C:\Users\test\krb5cc_test

Execute klist to see the details of the ticket; to destroy the ticket you can simply remove the file create by kinit.

Setup and configuration

Search Guard add-on

Kerberos authentication support requires the installation of the commercial Search Guard Kerberos HTTP Authentication add-on; to install it, download the correct jar for your Search Guard version from this page and copy it to the plugins/search-guard-5 directory on each node.

Kerberos configuration file

Create a file named krb5.conf in the config directory of each node with the following contents; replace AD.LOCAL with your domain name and DC.AD.LOCAL with the name or IP address of your KDC/domain controller, keeping the case of domains as in the example:

[libdefaults]
default_realm = AD.LOCAL
forwardable=true
default_tkt_enctypes = rc4-hmac,aes256-cts-hmac-sha1-96,aes128-cts-hmac-sha1-96
default_tgs_enctypes = rc4-hmac,aes256-cts-hmac-sha1-96,aes128-cts-hmac-sha1-96

[realms]
AD.LOCAL = {
kdc = dc.ad.local:88
default_domain = ad.local
}

[domain_realm]
.ad.local = AD.LOCAL
ad.local = AD.LOCAL

Keytab

Copy the keytab file for the service principal to the configuration directory of each Elasticsearch node.

Elasticsearch configuration

Add the following options to the elasticsearch.yml file of each node:

  • searchguard.kerberos.krb5_filepath: the path to the Kerberos configuration file, usually krb5.conf.

  • searchguard.kerberos.acceptor_keytab_filepath: the path to the keytab file relative to the configuration directory of the Elasticsearch node. It is mandatory to store the keytab in this directory.

  • searchguard.kerberos.acceptor_principal: the name of the principal stored in the keytab (e.g. HTTP/es.ad.local).

Example configuration:

searchguard.kerberos.krb5_filepath: 'krb5.conf'
searchguard.kerberos.acceptor_keytab_filepath: 'es.keytab'
searchguard.kerberos.acceptor_principal: 'HTTP/es.ad.local'

To disable the Kerberos replay cache in Search Guard, you’ll need to set the sun.security.krb5.rcache JVM property to none; this can be done by setting the following line in config/jvm.options:

-Dsun.security.krb5.rcache=none

For information on where to set/modify this variable please refer to Running as a service on Linux or Running as a service on Windows.

Cluster restart

Once the previous steps have been completed on all nodes, perform a rolling restart of the cluster.

Search Guard authenticator configuration

To complete the Kerberos configuration you need to modify your sg_config.yml file and upload it to the cluster using sgadmin; if you are using the Search Guard management API make sure to include only the sg_config.yml in the sgadmin configuration directory or you will overwrite internal users, actiongroups, roles and mappings defined through the API.

To enable Kerberos authentication over HTTP, you need to:

  • Add a Kerberos authenticator stanza to searchguard.authc

  • Disable challenge in the existing HTTP Basic authenticator if enabled

Example sg_config.yml:

searchguard:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
    authc:
      kerberos_auth_domain:
        enabled: true
        order: 2
        http_authenticator:
          type: kerberos
          challenge: true
          config:
            krb_debug: false
            strip_realm_from_principal: true
        authentication_backend:
          type: noop
      basic_internal_auth_domain:
        enabled: true
        order: 1
        http_authenticator:
          type: basic
          challenge: false
        authentication_backend:
          type: intern

With the above configuration, if the user is not authenticated Search Guard will reply with a 401 challenge; SPNEGO compatible browsers will then repeat the request automatically with Kerberos credentials if the cluster is in a trusted network or display an authentication popup where the user can enter its domain credentials.

If an HTTP request to the cluster contains an HTTP Basic authorization header, it will still be authenticated by the HTTP authenticator defined in basic_internal_auth_domain; it is necessary to leave this enabled as the Siren Investigate backend uses this method to authenticate with the cluster.

It is possible to enable only a single HTTP challenge; if your browser is configured to automatically send Kerberos credentials in a trusted zone it is possible to disable the challenge attribute by setting kerberos_auth_domain.http_authenticator.challenge to false.

For more details about configuring Search Guard authenticator please refer to the official documentation.

Verification

Once sg_config.yml has been loaded you can verify if the authentication is working by mapping a username in the Active Directory / Kerberos domain to a Search Guard role mapping, e.g.:

sirenuser:
  users:
    - sirenuser
    - domainuser

Once the mapping is loaded to the cluster, logon to a machine in the domain with the domain user and open the cluster URL in a Kerberos enabled browser (e.g. Chrome on Windows).

If everything is setup correctly you should see the default JSON response of Elasticsearch in the browser without having to enter credentials, e.g.:

{
  "name" : "Node",
  "cluster_name" : "cluster",
  "cluster_uuid" : "nimUDAyBQWSskuHoAQG06A",
  "version" : {
    "number" : "5.4.0",
    "build_hash" : "fcbb46dfd45562a9cf00c604b30849a6dec6b017",
    "build_timestamp" : "2017-01-03T11:33:16Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.2"
  },
  "tagline" : "You Know, for Search"
}

If you’re getting an authentication popup, ensure that the Elasticsearch cluster URL is in a trusted zone.

To add a site to the trusted zone on Windows you need to:

  • open Internet Explorer and click on Internet options.

  • click on the Security tab.

  • click on Local Intranet.

  • click on Sites.

  • click on Advanced.

  • add the URL of the cluster to the list (the port can be omitted).

Once the cluster is in the trusted zone try to open the cluster URL again.

Internet Explorer options are also used by Chrome on Windows.

Trusted sites setup
Figure 45. Trusted sites

Troubleshooting

To check why a request is not authenticated you can check the Elasticsearch logs of the client node serving the REST API.

The most common issues are:

  • cluster URL not present in the trusted sites list.

  • a keytab containing an incorrect Service Principal Name and/or a wrong password for the user account associated to the SPN.

  • an incorrect address of the domain controller / KDC in the krb5.conf file.

To get additional debugging information you can set krb_debug to true temporarily in sg_config.yml and upload it to the cluster using sgadmin.

Siren Investigate configuration

To enable SPNEGO support in Siren Investigate, set the kibi_access_control.backends.searchguard.authenticator option to http-negotiate, in siren.yml e.g.:

kibi_access_control:
  #... existing options
  backends:
    searchguard:
      #... existing options
      authenticator: 'http-negotiate'

Then restart Siren Investigate and verify that you can login from a browser in the domain using a user defined in Search Guard.

When SPNEGO support is enabled, cookie based authentication will be disabled; if you need to provide both authentications for different networks, it is possible to start an additional Siren Investigate instance with kibi_access_control.backend.searchguard.authenticator set to http-basic or not set at all.

JWT authentication support

This section offers an overview of how to integrate Siren Investigate with the Search Guard JWT authenticator when Siren Investigate is embedded into an iframe by another application.

Before enabling JWT support you should setup Siren Investigate and Search Guard as described in the Search Guard integration chapter and ensure that it works as expected.

Pre requisites

Search Guard add-on

JWT authentication support require the installation of the commercial Search Guard Kerberos JWT HTTP Authentication add-on; to install it, download the correct jar for your Search Guard version from this page and copy it to the plugins/search-guard-5 directory on each node, then perform a rolling restart of the cluster.

Siren Investigate proxy

It is required that Siren Investigate and the container application are published on the same domain to allow cross frame communication; this can be achieved by implementing a proxy to Siren Investigate in the container application routes or configuring a reverse proxy on a path in the application server configuration.

JWT token issuance

The application that embeds Siren Investigate is responsible for generating JWT tokens; jwt.io provides a good overview of the technology, a browser based debugging tool and a list of libraries for several platforms.

The Search Guard documentation provides an overview of all the claims supported by the add-on and a list of all the configuration options.

The application must specify an expiration date claim (exp) to avoid creating tokens with unlimited duration.

Configuration

Once the add-on has been installed in the cluster, you need to modify sg_config.yml file and upload it to the cluster using sgadmin; if you are using the Search Guard management API make sure to include only the sg_config.yml in the sgadmin configuration directory or you will overwrite internal users, actiongroups, roles and mappings defined through the API

To enable JWT authentication over HTTP, you need to add a JWT authenticator stanza to searchguard.authc; an example sg_config.yml follows:

searchguard:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
    authc:
      jwt_auth_domain:
        enabled: true
        order: 1
        http_authenticator:
          type: jwt
          challenge: false
          config:
            signing_key: "cGFzc3dvcmQ="
            jwt_header: "Authorization"
        authentication_backend:
          type: noop
     basic_internal_auth_domain:
        enabled: true
        order: 2
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: internal

With the above configuration, Search Guard will check if the Authorization header contains a JWT token signed with the signing key specified in http_authenticator.signing_key.

The signing key must be encoded using the base64 algorithm; in the example above the decoded key is password.

If the token is decoded successfully, Search Guard will validate the following claims:

  • iat - Issued At: the date when the token was issued (optional).

  • exp - Expiration Time: the date after which the token should expired; this claim is optional but it is recommended to set it, otherwise tokens will have unlimited duration.

  • nbf - Not Before: the date before which the token should be rejected (optional).

All dates are expressed as seconds since the Epoch in UTC.

If time claims are validated, Search Guard will get the username from the Subject claim (sub), assign role mappings and evaluate role permissions.

If an HTTP request to the cluster contains an HTTP Basic authorization header it will be authenticated by the HTTP authenticator defined in basic_internal_auth_domain; it is necessary to leave this enabled as the Siren Investigate backend uses this method to authenticate with the cluster.

Roles

It is possible to specify user roles in a token claim by setting the roles_key attribute in the authenticator configuration to the desired claim name, e.g.:

#...
      jwt_auth_domain:
        enabled: true
        order: 1
        http_authenticator:
          type: jwt
          challenge: false
          config:
            roles_key: "roles"
            signing_key: "cGFzc3dvcmQ="
            jwt_header: "Authorization"
#...

Once the attribute is set and the configuration is updated, it is possible to assign backend roles to the user by setting the claim defined in http_authenticator.config.roles_key in the token payload, e.g. :

{
  "sub": "sirenuser",
  "exp": 1495711765,
  "roles": "sales,marketing"
}

Please note that in order to map roles set in the JWT token to Search Guard roles you will need to define a role mapping such as the following:

JWT role mapping
Figure 46. JWT role mapping

Verification

To verify that Search Guard JWT authentication is working correctly you can generate a JWT token from your application and pass it to Elasticsearch using curl’s -H option, e.g.:

curl -k -H "Authorization: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJraWJpdXNlciJ9.tqCYxJsORvro59Q01J9HUeFpQtauc81CcTlS5bVl93Y" https://localhost:9200/_searchguard/authinfo

To test if it is working correctly before the application is ready, you can use the jwt.io debugger to generate tokens using the signing key defined in sg_config.yml.

Siren Investigate configuration

To enable JWT support in Siren Investigate, set the kibi_access_control.backends.searchguard.authenticator option to http-jwt, in siren.yml e.g.:

kibi_access_control:
  #... existing options
  backends:
    searchguard:
      #... existing options
      authenticator: 'http-jwt'

Then restart Siren Investigate and open it in a browser; you should get a blank page and the URL should end with login.

To test JWT authentication, open your browser console (CTRL+SHITF+I on Chrome and Firefox) and call setJWTToken of the kibi object, e.g.:

.kibi
.setJWTToken(yourtoken)
.then(function() {
  console.log('JWT token set.');
})
.catch(function(error) {
  console.log('An error occurred setting the token.');
});

Once the token is set, Siren Investigate will store it in an encrypted cookie and send it in every request to the backend; the backend will then forward the JWT token to Search Guard to authenticate the user.

After the token is set, you can switch to the desired Siren Investigate URL by simply changing location.href.

When the user is logged out from the main application, sessionStorage and localStorage should be cleared.

For more information on how to call setJWTToken from the parent frame, please refer to the cross frame communication section.

Indexes and Relations

Indexes and Relations

In Indexes and Relation you can define relationships between your data tables, on the local Siren Elasticsearch nodes or mapped remotely to JDBC databases, and from there you will be able to create dashboards where you can have "relational pivoting" buttons (going from the set of currently selected records to the set of connected records in another table).

Operations that you can do

  • Configure which Elasticsearch Index(es) or Virtual Index(es) you are going to have available inside Siren Investigate. With Elasticsearch indexes, you can also create new "scripted fields".

  • Define relations among these Indexes. This effectively defines a data model, also known as ontology (this effectively makes it so that "Indexes" are now treated as "Classes" and the records can be seen as "Entities") . The ontology also specifies properties of these indexes/classes e.g. icons, labels etc

  • Define "Entity Identifiers": these are Classes of strings or integers you might have here and there in the data representing an entity which are "well understood as such" but you do not (yet?) have a specific index listing them. Typical Entity identifiers are things like IP Address: It’s an entity (and you want to join on it) but you don’t have an "index of all the IPs". Other examples are normalized phone numbers, hashfunctions, userids, name of locations or cities, tags or labels etc.

In summary, from now on with "Classes" we will refer to either Index Patterns or EIDs and for Entities we’ll refer to either the individual records that are in Index Patterns or the individual EID values (e.g. an IP addresses)

Creating an Index Pattern.

Just use the Add Index Pattern button.

Creating relationships

Relationships are defined from a Class to other Classes (But it is not possible to define a relationships between 2 EIDs).

In case of Index Patterns one typically defines A relationship is defined as a join operation between two indices with the following fields:

  • The Left Field: the field of the local index to join on;

  • Right Class: (the EID or Index pattern) to connect to;

  • Right Field (only if the Right Class is an Index Pattern): the field of the right index to join with; and

  • Label: the label of the relation.

Indexes and Relations

New relations are created with the Add Relation button. Relations do NOT need to be created in both originating and target classes as they appear automatically in both edit screens when created.

Pressing the visualize data model as a graph button will show it in a visual representation where the currently selected class is highlighted, e.g. in this case

Relations Graph

How to use Entity Identifiers

Siren 10 introduces the concept of an "Entity Identifier" (EID). Previously, in Siren, to be able to join between two indexes you had to specify that there existed a direct connection between them. e.g. if you had 2 logs which could be connected by the IP value, you would have specified a direct connection, thus creating a relational button between the two.

But what if you have many indexes having IPs (or anything else: MAC Addresses, User IDs, URLs, Port Numbers, Transaction IDs, etc) that are in multiple roles (Source IP, Destination IP) and it might be useful to join from any of these roles and indexes to any other role and index?

Our new relational model allows this. Automatically.

For example, in this configuration, we have defined the IP concept as an EID and tied it in with other indexes where "IPs" show up. For each connection, we specify the name of the relation that describes the role of the IP in that index (Is it the "source" IP in that log or the "blocked" IP?).

Relations Graph

With just this configuration, you can now have buttons that explore the ontology and show you all possible matches across your data. At this point, one click and you’ll be pivoting to the target dashboard, with the right relational filter applied.

For example, to see the records of the Apache logs where the Agent IP matches the Destination IP in the current log, just navigate from "Destination IP" as per the picture:

Automatic relational buttons

EIDs are obviously great for anything that identifies "things" across indexes but does not have an index per se (otherwise, you’d pivot to it). Things like Phone Numbers, but also Tags, Labels from standalone indexes, etc. In practice a single excel spreadsheet can be seen as a "knowledge graph" if you consider labels as identifiers that interconnect records. Here is an example with EIDs (Tissue and Organism) in a Life Science deployment.

Knowledge Graph

Note that the automatic connections between dashboards are seen when using the new relational button. The old one will still require manual inputs on which relation to show where.

Visualise

Again, this is how the new relational button appears in action.

Automatic relational buttons

How to name relations

It is well known that naming is a very hard problem in any domain. In Siren naming entities and relationships wrong will result in hard to navigate dashboards.

When naming things one must put oneselves into the shoes of the user in the moment where the relational navigation is performed. Say that i am looking at "companies", how would i refer to "investments" ?

A possibly natural way is to say a "company" received and "investment". On the other hand if i am thinking of investment, i can say it has been "secured by" a company.

In the UI, look at the directions of the arrows and think of the sentences "X relationship Y and Y relationship X", for example.

How to name relations

In this case we’re using two different verbs, but often the simple solution is to use active/passive e.g. "saw" and "seen by". Sometimes the inverse is the same property is the same e.g. "brother of" or "spouse".

As a rule of thumb , it is always best to keep things quite short. E.g. "source" "is source of" and the like.

The Automatic Relational Button component.

The automatic relational button component typically requires no configuration and can be reused across any dashboard (there is no need to create different ones for different dashboards).

By default, it will show all the possible relationships to any dashboard which has an associated "saved search" which is based on an Index Pattern which is relationally connected to the Index pattern which is associated to the Saved Search underneath the current dashboard.

It is really much much simpler than it sounds. :) just drop this component into a dashboard which is associated with a Saved Search and it will show you all the possible relational connections with other dashboards which have related entities.

Visibility settings

In the component setting one, can change the visibility status of each individual connection at multiple levels.

The 3 state eye component allows to specify "never show", "always show" or "inherit show from the previous".

Relational Browsing

Siren Investigate allows you to filter documents on a dashboard by showing only those that have a relation with documents displayed in a different dashboard, possibly stored in different indices.

Relational filter

The relational filter visualization allows to "pivot" from a dashboard to another by creating a join between multiple indices based on their relations. This allows to interactivelly build the sequence of dashboards to join.

The relational filter visualization is configured based on the relationships between indices defined in the settings tab. For example, let’s take the following indices:

article

an index containing articles; each document in the index has a field called companies which is an array that contains the ID of companies mentioned in the article. This index is displayed on the dashboard entitled Articles.

company

an index containing information about companies; each document in the index has a field called id that contains the ID of the company. This index is displayed on the dashboard entitled Companies.

Both indices are configured so that they are joined on the field companies of article with the field id of company. Then, it is possible to use that configuration in order to create a relational filter that would filter companies based on connected articles (or vice-versa).

In the Articles dashboard, the relational filter visualization is displayed as a button which indicates the number of documents in the Companies dashboard that are mentioned in the articles of the current dashboard.

The screenshot below shows the button for the relation described in the example; there are 18011 companies mentioned in the 602,000 articles currently displayed:

Relational filter button on the Articles dashboard

Clicking on the button will switch you to the Companies dashboard and display the 18011 companies; the relational filter is displayed in the filter bar, as displayed below:

Relational filter on the Companies dashboard

The relational filter visualization requires the Siren Federate plugin 5.6.7-10.0.0-beta-1 for Elasticsearch.

Configuration

Click on the Add filter button to create a new filter in the visualization; the filter is defined by the following parameters:

  • Button label: the label of the button that will be displayed inside the visualization, e.g. Companies -→.

  • Custom filter label: the label of the filter that will be displayed in the filter bar, which by default is …​ related to ($COUNT) from $DASHBOARD.. Several variables are available for customizing the label:

    • $COUNT is a number of items on source dashboard,

    • $DASHBOARD is a source dashboard name.

  • Source dashboard: optional parameter that indicates on which dashboard the relational filter should appear in.

  • Target dashboard: the dashboard to join the current dashboard with. The current dashboard is equal to the previous field if set.

  • Relation: the label of the relation between indices to use for this relational filter. This is set in the relations settings tab.

The screenshot below shows the configuration of a relation from the Articles dashboard to the Companies dashboard, using the mentions relation:

Relational filter configuration

It is possible to define multiple relations in a single Siren Investigate relational filter visualization; the visualization will display only buttons applicable to the currently displayed dashboard.

Usage

When clicking on a button in the relational filter visualization, the current state of the source dashboard is added to the relational filter and applied to the target dashboard. Just move the mouse over relational filter to see an explanation of what is being joined.

Walkthrough example

We start on the Articles dashboard, search for pizza and click on the relational filter to switch to the Companies dashboard.

Relational filter explanation

Hovering over the blue filter displays an explanation. It indicates that the relational filter involves only one join, i.e., the one from Articles to Companies with pizza filtering the articles.

Relational filter explanation

Next, we add a regular filter to the Companies dashboard by clicking on the Positive Filter in the USA row of the Companies by Country visualization.

Relational filter explanation

Now, we click on the Investment rounds -→ button which takes us to the Investment rounds dashboard. The explanation on that filter shows that the investment rounds are filtered as follows:

  • the current investments rounds are joined with companies from the USA; and

  • those companies are joined with articles which match the term pizza.

Relational filter explanation
The sequence of the joins in the explanation are shown in reverse, i.e., the last join is on top.

Viewing Detailed Information

To display the raw data behind the visualization, click the Spy Open Button at the bottom left of the container. Tabs with detailed information about the raw data replace the visualization, as in this example:

Spy panel of the relational filter visualization

This panel provides two kinds data: information about the query behind the relational filter in the Multi Search tab, and details about the visualization object in the Debug tab.

This pane presents information about the msearch request executed to perform the joins. A relational filter corresponds to one query of the msearch.

On the top, the time reported in Multi search request duration informs on how long the msearch request took. There is also additional information about each query of the msearch:

  • Query Duration: The time spent for this particular query.

  • Hits: the total number of documents resulting from the query.

  • Index: the index pattern used to execute the query.

  • Type: the type of the indices matched by the index pattern.

For a particular relational filter, you can get additional information about the query that got executed.

Filterjoin

This displays a table that provides several statistics about each join.

Details about the filterjoin query
Raw Request

The filterjoin query as sent by Siren Investigate. This uses the internal API for defining the join.

Translated Request

The filterjoin query as sent to the Elasticsearch cluster, presented in JSON format.

Response

The raw response from the server, presented in JSON format.

Debug

The Debug tab presents the JSON object that Siren Investigate uses for this relational filter.

Debug spy panel of the relational filter visualization

Join Limit

The number of unique values returned from the source of the relation is limited by the kibi:joinLimit Advanced Setting in the management section. These source values are then used to filter the documents on the destination. In general, the destination is the current dashboard.

For more on this and how to set the limit for each relation individually, see the Join Limit section of the Relation Panel documentation.

JDBC Datasources

Setting up Siren to work with JDBC datasources.

Siren can analyze data by directly querying remote datasources via JDBC.

To do this:

At this point the Virtual Index is effectively identical to a regular index i.e. one that is natively in Elasticsearch.

Configuring the Federate Backend with JDBC Driver

To configure the Federate JDBC connector:

  • Stop the elasticsearch cluster

  • Install Siren federate on each elasticsearch node. Instruction are available here:

  • Choose one node in the cluster and edit the elasticsearch.yml adding the parameter: node.attr.connector.jdbc: true

  • In the same elasticsearch node, add the jdbc driver (jar file) in the folder ES_FOLDER/plugins/siren-federate. The supported jdbc drivers are reported in the table below, with the url from where the drivers can be downloaded.

  • Restart elasticsearch

Table 1. List of supported JDBC drivers
Name JDBC class Default port Download Page Link if not included

PostgreSQL

org.postgresql.Driver

5432

https://jdbc.postgresql.org/download.html

MySQL

com.mysql.jdbc.Driver

3306

https://dev.mysql.com/downloads/connector/

Microsoft SQL Server 2017

com.microsoft.sqlserver.jdbc.SQLServerDriver

1433

https://www.microsoft.com/en-us/download/details.aspx?id=55539

Sybase ASE 15.7+

com.sybase.jdbc4.jdbc.SybDriver OR net.sourceforge.jtds.jdbc.Driver

5000

Oracle 12c+

oracle.jdbc.OracleDriver

1521

http://www.oracle.com/technetwork/database/features/jdbc/default-2280470.html

Presto

com.facebook.presto.jdbc.PrestoDriver

8080

https://repo1.maven.org/maven2/com/facebook/presto/presto-jdbc/0.192/presto-jdbc-0.192.jar

Spark SQL 2.2+

com.simba.spark.jdbc4.Driver OR com.simba.spark.jdbc4.DataSource OR com.simba.spark.jdbc41.Driver OR com.simba.spark.jdbc41.DataSource

https://www.simba.com/product/spark-drivers-with-sql-connector/

Dremio

com.dremio.jdbc.Driver

https://download.siren.io/dremio-jdbc-driver-1.4.4-201801230630490666-6d69d32.jar

Impala

com.cloudera.impala.jdbc41.Driver

21050

https://www.cloudera.com/downloads/connectors/impala/jdbc/2-5-42.html

Siren Investigate Datasource Configuration

Using Siren Investigate, go to Management/Datasource

Navigate to Management/Datasource

Select the JDBC choice in the dropdown

Select JDBC option

Fill in the connection parameters, then press Save in the top right corner

Fill in connection parameters

Check the configuration by pressing Test Connection. If the settings are properly configured, this is the result:

Test connection

Press Yes, take me there to map a table from the DB into an index pattern, as reported in the image below:

Virtual Index Configuration

The Virtual Index name must be lowercase, the Resource name is the name of the table in the database. Once the Virtual Index is configured, press Save in the top right corner. The screenshot below shows what the user should see after saving. Press Yes take me there to map the index to this virtual index.

Virtual Index Configuration Success

Press Add Index Pattern and fill in the name with the same name used for the Virtual Index, in this example indexfromdb, and press Create.

Index Pattern Configuration

From this point, indexfromdb is a normal index pattern and it can be used in Discovery, Visualize etc.

Known Limitations

  • Cross Backend join currently supports only integer keys

  • Cross Backend support has very different scalability according to the direction of the Join, a join which involves sending IDs to a remote system will be possibly hundreds of times less scalable (e.g. thousands vs millions) to one where the IDs are fetched from a remote system.

External Datasources

Siren Investigate provides visualizations and aggregations to integrate data from external datasources. This section explains how to setup datasources and configure queries and query templates.

Configuration

To create a new external datasource navigate to "Settings/Datasources" First fill the datasource title and name then pick one of the types Siren Investigate supports:

  • REST

  • SQLite

  • MySQL

  • PostgreSQL

After selecting the datasource type, additional parameters will appear. Below is a list of additional mandatory parameters for each datasource type:

Common Parameters

  • timeout - connection timeout in milliseconds

  • cache_enabled - enable server side cache for this datasource

  • max_age - the max age of an object in the cache, in milliseconds

To control the maximum number of query results kept in cache, set the kibi_core.datasource_cache_size parameter in siren.yml.

A Siren Investigate restart is required to apply changes to siren.yml configuration file.

REST

  • url - an URL of rest API endpoint

  • response_type - API results format, currently Siren Investigate supports only JSON

  • username (optional) - username with access to the API

  • password (optional) - corresponding password

  • auth_token (optional) - token used in Token Authentication

SQLite

  • db_file_path - path to database file

MySQL, PostgreSQL

  • host - hostname

  • dbname - database name

  • username - username with read access to the database

  • password - corresponding password

Parameters encryption

Sensitive datasource parameters like passwords are encrypted before being stored in the .siren index.

Before creating datasources containing sensitive parameters, make sure to set a custom encryption key by running the replace_encryption_key command:

bin/investigate replace_encryption_key [options] <current_key> <new_key> <new_cipher>
  • <current_key> a base64 encoded string containing the current encryption key.

  • <new_key> a base64 encoded string containing the new encryption key.

  • <new_cipher> the cipher algorithm to use (currently only AES-GCM is supported).

The current encryption key can be read from the siren.yml file in the datasource_encryption_key parameter.

Keys can have a length of 16, 24 or 32 bytes; a quick way to encode a plaintext string to base64 is to use the base64 utility from the coreutils package:

$ echo -n changemechangemechangemechangeme | base64
Y2hhbmdlbWVjaGFuZ2VtZWNoYW5nZW1lY2hhbmdlbWU=
Make sure to set the configuration file as readable only by the user running the Siren Investigate process.

Datasource entity selection

Selected Entities can be used to as source of parameters for queries Each selected entity is uniquely identified by an URI:

  • INDEX/TYPE/ID where INDEX is an index pattern, TYPE is a type of a document, and ID is document ID

As explained in the following sections, queries on external datasources can extract variables from the selected entity URI; in order to allow the user to select an entity, you must add an Enhanced search results visualization to a dashboard and configure at least one click handler to select an entity.

Once the visualization is configured, clicking on the cell will display a purple box in the filter bar, and the variables stored in the entity URI will be available to queries and query templates.

The screenshot below shows the effect of clicking on a cell configured with an entity selection handler; after selecting an entity, the Company Info template viewer shows the information about the company fetched by a query.

Entity selection
Entity selection configuration example

To disable or cancel the selection, click on the icons displayed inside the entity selection widget when the mouse is over it, as displayed in the screenshot below:

Entity selection options

Queries

Queries can be used to provide data to Templates, tag and filter Elasticsearch documents.

To create a new query, click to the "Settings/Queries" tab.

You need then to set the following fields to define a query:

  • Title: the title of the query.

  • Datasource: the name of a configured datasource.

  • Results query: the query declaration.

You may also set a description for the query and one or more tags.

Below is an example configuration of a query on a SQL database called Top 50 companies (HR count) that returns the Top 50 companies by number of employees in a table called company.

Configuration of a SQL endpoint

The preview section will display the results of the query as a table or as a JSON object.

Template rendering is currently a blocking operation, therefore queries returning a large number of results might make the backend unresponsive for an indeterminate amount of time.

Query variables:

One of the most useful features of queries is that it is possible to set some of their parameters before execution by using datasource specific variables, which can be set at runtime by configuring click handlers in the Enhanced search results visualization to select an entity.

Variable values are taken from elasticsearch document selected via selected entity URI.

All properties from selected document can be accessed using the following syntax: @doc[PATH_ELEMENT_1][PATH_ELEMENT_2]…​[PATH_ELEMENT_N]@

  • to get the document id use: @doc[_id]@

  • to get the value of property called action use: @doc[_source][action]@

  • to get the value of nested property called person.age use: @doc[_source][person][age]@

In order to view the results of the query, you have to specify an entity URI manually in the field on the top right;

Below is an example of configuration for a query named Company Info using a variable to get the value of property called id of currently selected entity In the example, @doc[_source][id]@ is replaced with an id taken from selected company. In the Selected Entity box we see that the selected company is from index: company, has a type: Company and has the id AVgfaYQ0Q2VQXwxDgyfY

SQL query with variables

Activation Query

An activation query can be specified to conditionally execute the results query.

For example, if you have a table called Vehicles but some of the queries are only relevant to "Motorcycles" and not to "Cars", the activation query could be used to determine if the results query should be executed when an entity in Vehicles by looking at its type. If the query is not executed, any template or aggregator using the query will be automatically disabled.

On SQL datasources, activation queries will trigger results query execution when returning at least one record.

Example:

SELECT id
FROM Vehicles
WHERE id='@doc[_source][id]@' AND vehicle_type='Motorcycle'

Use cases

Once you’ve configured query templates and queries, you can use them in the following visualizations:

It is also possible to use queries as aggregations as explained below.

External query terms filters aggregation

The query results from an external data source can be used as an aggregation in visualizations.

This allows to compute metrics on Elasticsearch documents joined with query results.

To use a query as an aggregation, select a bucket type and select External Query Terms Filter in the Aggregation dropdown; then, click on the Add an external query terms filter button.

You can then configure how to join the query results with the Elasticsearch documents by setting the following parameters:

  • Source query id: the name of the query on the external datasource.

  • Source query variable: the name of the variable in query results which contains the first value used in the join.

  • Target field: the name of the field in the target index which contains the second value used in the join.

The aggregation will return only documents in the Elasticsearch index whose target field value is equal to the source query variable value in at least one of the results returned by the query; if Negate the query is checked, the aggregation will return only documents in the Elasticsearch index whose target field value is not equal to any of the values of the source query variable in the results returned by the query.

For example, the screenshot below show the configuration of a Data table visualization with three aggregations based on external queries:

  • A query that selects the labels of the competitors of the currently selected company

  • A query that selects the labels of all the companies which have a competitor

  • A query that selects the id’s of the top 500 companies by number of employees

If a query requires a selected entity, and no entity is selected, the computed aggregation will return 0, also the controls to select Selected entity will indicate (red borders arround) that it is necessary to select one.

Configuration of an external query terms filter aggregation on a data table visualization

The screenshot below shows the configuration of two external query terms filter aggregation on a pie chart visualization:

Configuration of an external query terms filter aggregation on a pie chart visualization

Siren Investigate Gremlin Server

The Siren Investigate Gremlin Server component is a backend component required by the Siren Investigate Graph Browser visualisation.

In order to enable the Gremlin Server, make sure that siren.yml contains the following configuration:

kibi_core:
  gremlin_server:
    url: http://127.0.0.1:8061
    path: gremlin_server/gremlin-es2-server.jar

To use Gremlin Server with an authentication enabled cluster, please refer to the Authentication and access control section.

Log4J File Configuration Path:

Log4J configuration file is optional for the Gremlin server. If you want to use your own custom configuration, you can specify the path to your file with the kibi_core.gremlin_server.log_conf_path parameter inside your kibi.yml file. Here is an example of how to configure the log4j.properties file for your Gremlin server:

# For the general syntax of property based configuration files see
# the documentation of org.apache.log4j.PropertyConfigurator.

# The root category uses two appenders: A1 and FILE.
# Both gather all log output starting with the priority INFO.
log4j.rootLogger=INFO, A1, FILE

log4j.appender.A1=org.apache.log4j.ConsoleAppender
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.threshold=INFO
# Print the date in ISO 8601 format
log4j.appender.A1.layout.ConversionPattern=%d [%t] %-5p %c - %m%n

log4j.appender.FILE=org.apache.log4j.FileAppender
log4j.appender.FILE.append=true
log4j.appender.FILE.file=log/gremlin-server.log
log4j.appender.FILE.threshold=INFO
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout
log4j.appender.FILE.layout.ConversionPattern=%-5p %c: %m%n


# Print only messages of level WARN or above in the package org.springframework
log4j.logger.org.springframework=WARN

Siren Alert

Siren Alert provides Investigate with alerting and reporting functionality to monitor, notify and visualize data series changes using standard queries, programmable validators and a variety of configurable actions.

Siren Alert is also designed to simplify the process of creating and managing alerts and reports in Siren Investigate through its App and Spy integration.

Documentation

The full Siren Alert documentation is available here.

Cross Frame Communication

To allow cross frame communication, Siren Investigate exposes an object at window.sireninvestigate; the object can be called only if both Siren Investigate and the container page are in the same domain.

Methods

generateShortUrl(shareAsEmbed, displayNavBar)

Generates a shortened URL containing the current Siren Investigate state and returns a promise fulfilled with the URL.

Parameters:

  • shareAsEmbed: if set to true, the top navigation bar and dashboard tabs will be hidden when opening the shortened URL.

  • displayNavBar: if set to true, the dashboard tabs will not be hidden when sharedAsEmbed is set to true.

Sample usage:

Put the following code in the container page, replacing kibiframe with the ID of the frame in which Siren Investigate is embedded:

document.getElementById('kibiframe')
.contentWindow
.sireninvestigate
.generateShortUrl(true, true)
.then(function(url) {
  console.log("Generated URL: " + url);
})
.catch(function(error) {
  console.log("An error occurred while generating the URL");
});

If possible, it is recommended to purge old documents of type url from the .kibi index periodically; old documents can be identified by looking at the createDate attribute.

setJWTToken(token)

Sets or updates the JWT token for the current session if JWT authentication support is enabled; returns a Promise once the token has been sent to the backend.

Parameters:

  • jwtToken: a base64 encoded JWT token.

Sample usage:

Put the following code in the container page, replacing kibiframe with the ID of the frame in which Siren Investigate is embedded:

document.getElementById('kibiframe')
.contentWindow
.sireninvestigate
.setJWTToken(`eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJraWJpdXNlciJ9.kZhLu15FwxrX4hPE1ciyzw_NufZ_oH7aSGpLZHachPg`)
.then(function() {
  console.log('JWT token set.');
})
.catch(function(error) {
  console.log('An error occurred setting the token.');
});

Once the token is set, you can change the Siren Investigate URL and the user should be authenticated; the application should call the method again with an updated token before the current one expires.

Loading Data into Elasticsearch

This chapter contains basic information on how to load data into Elasticsearch for evaluation purposes.

From a SQL Database using Logstash

The indices in the Siren Platform demo distribution have been populated by running four Logstash configurations over the SQLite database in siren-investigate/crunchbase.db.

The database has the following schema:

SQLite database schema

Index Setup

Before loading data, we need to setup indices and mappings; for example, let’s create an index called company-minimal in the Elasticsearch cluster at http://localhost:9220.

Create the index by running the following command in a terminal window:

curl -X PUT http://localhost:9220/company-minimal

If curl is not available on your system, please download it from http://curl.haxx.se/download.html .

If the index is created correctly, Elasticsearch will return the following response:

{"acknowledged":true}

If you want to destroy the index and start from scratch, execute the following command:

curl -X DELETE http://localhost:9220/company-minimal

Mapping Definition

Mappings allow the user to configure how documents are stored in the index. For example, they allow you to define how fields are matched by the search engine and set their type (string, dates, numbers, locations etc.).

for detailed documentation about indices and mappings we recommend reading the Elasticsearch Reference.

Let’s define a simple mapping to describe a company. The mapping will define the following fields:

  • id: the id of the company in the SQLite database

  • name: the name of the company

  • description: a description of the company

  • homepage: the URL of the company homepage

  • number_of_employees: the number of employees

  • location: the geographical coordinates of the company

Open a text editor and paste the following text:

{
    "CompanyMinimal": {
        "properties": {
            "id": {
                "type": "keyword"
            },
            "number_of_employees": {
                "type": "long"
            },
            "name": {
                "type": "text"
            },
            "description": {
                "type": "text"
            },
            "homepage": {
                "type": "keyword"
            },
            "location": {
                "type": "geo_point"
            }
        }
    }
}

CompanyMinimal is the name of the mapping; properties contains the options for each field.

Save the file to demo/example/CompanyMinimal.mapping inside the directory where you extracted the demo distribution.

To apply the mapping, execute the following command:

curl -X PUT "http://localhost:9220/company-minimal/_mapping/CompanyMinimal" -d "@demo/example/CompanyMinimal.mapping"

If the mapping is created correctly, Elasticsearch will return the following response:

{"acknowledged":true}

SQL Query Definition

To extract the values that will be loaded to the index by Logstash, we need to write a SQL query. Open a text editor and paste the following one:

SELECT id,
  label AS name,
  description,
  homepage_url as homepage,
  number_of_employees,
  CASE WHEN lat IS NULL THEN
    NULL
  ELSE
    lat || ', ' || lng
  END AS location
  FROM company
  LEFT JOIN company_geolocation ON company.id = company_geolocation.companyid

Save the file to demo/example/company-minimal.sql inside the directory where you extracted the demo distribution.

Logstash Configuration

We now need to write a Logstash configuration to process the records returned by the query and populate the company-minimal index.

Support for SQL databases is provided by the Logstash jdbc input plugin; You must download logstash to demo/example directory and install the required plugin

Open a text editor and paste the following:

input {
  jdbc {
    jdbc_driver_library => "sqlitejdbc-v056.jar"
    jdbc_driver_class => "org.sqlite.JDBC"
    jdbc_connection_string => "jdbc:sqlite:crunchbase.db"
    jdbc_user => ""
    jdbc_password => ""
    statement_filepath => "company-minimal.sql"
    jdbc_paging_enabled => true
    jdbc_page_size => 10000
  }
}

filter {
  mutate {
    remove_field => ["@timestamp", "@version"]
  }
}

output {
  elasticsearch {
    hosts => "localhost:9220"
    manage_template => false
    action => "index"
    index => "company-minimal"
    document_type => "CompanyMinimal"
  }
}

The statement_filepath parameter specifies the path to the file containing the SQL query; the jdbc_* parameters set the database connection string and authentication options.

The mutate filter is configured to remove default Logstash fields which are not needed in the destination index.

The output section specifies the destination index; manage_template is set to false as the index mapping has been explicitly defined in the previous steps.

Save the file to demo/example/company-minimal.conf

Copy the SQLite database to demo/example/crunchbase.db, then go to the demo/example directory and run the following command:

cd demo/example
logstash/bin/logstash -f company-minimal.conf

Logstash will execute the query and populate the index.

for more information about Logstash, we recommend reading the Logstash reference and the jdbc input plugin documentation.

Browsing the Index in Siren Investigate

Open http://localhost:5606 in your browser, click on the Management tab then on Index Patterns.

Deselect Index contains time-based events, then write company-minimal in the Index name or pattern field:

Adding the company-minimal index

Click on Create to create the index reference, then click on the Discover tab and select company-minimal in the dark grey dropdown:

Discovering the company-minimal index

Click on the right arrow at the beginning of each row to expand it and see all the loaded fields:

Viewing all the fields in a document

Script to Load the Demo Data

The complete demo data loading process can be repeated by running the demo/sql/bin/index_crunchbase_sqlite.sh script. The script performs the following actions:

  • Creates a copy of the database in the directory containing Logstash configurations

  • Creates the indices article, company, investor and investment

  • Sets the mappings for each index

  • Runs the logstash configuration for each index

The Logstash configurations and Elasticsearch mappings are available in the demo/sql/crunchbase/conf/logstash_sqlite directory.

Acknowledgements

Kibana is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.

Logstash is a trademark of Elasticsearch BV, registered in the U.S. and in other countries.

CrunchBase and CrunchBase Dataset are trademarks of AOL Inc, registered in the U.S. and in other countries.

All the trademarks, registered trademarks and copyrighted terms in the Siren Investigate demo dataset are the property of their respective owners.

Siren Investigate Plugins

Add-on functionality for Siren Investigate/Kibana is implemented with plug-in modules. You can use the bin/investigate-plugin command to manage these modules. You can also install a plugin manually by moving the plugin file to the plugins directory and unpacking the plugin files into a new directory.

Generally Kibana plugins are compatible with Siren Investigate provided that you are using the same baseline version (e.g. Siren Investigate 5.2.2-1 with a plugin designed for Kibana 5.2.2).

Plugin compatibility

The Kibana plugin interfaces are in a state of constant development. We cannot provide backwards compatibility for plugins due to the high rate of change. Kibana enforces that the installed plugins match the version of Kibana itself. Plugin developers will have to release a new version of their plugin for each new Kibana release as a result.

Installing Plugins

Use the following command to install a plugin:

bin/investigate-plugin install <package name or URL>

When you specify a plugin name without a URL, the plugin tool attempts to download an official Elastic plugin, such as:

$ bin/investigate-plugin install x-pack

Installing Plugins from an Arbitrary URL

You can download official Elastic plugins simply by specifying their name. You can alternatively specify a URL to a specific plugin, as in the following example:

$ bin/investigate-plugin install https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-10.0.0-beta-1.zip

You can specify URLs that use the HTTP, HTTPS, or file protocols.

Installing Plugins to an Arbitrary Directory

Use the -d or --plugin-dir option after the install command to specify a directory for plugins, as in the following example:

$ bin/investigate-plugin install file:///some/local/path/x-pack.zip -d path/to/directory
This command creates the specified directory if it does not already exist.

Installing Plugins with Linux packages

The Siren Investigate server needs to be able to write to files in the optimize directory. If you’re installing plugins using sudo or su you’ll want to make sure these commands are ran as the user kibana. This user is already added for you as part of the package installation.

$ sudo -u kibana bin/investigate-plugin install x-pack

If plugins were installed as a different user and the server is not starting, then you will need to change the owner of these files:

$ chown -R kibana:kibana /path/to/kibana/optimize

Updating & Removing Plugins

To update a plugin, remove the current version and reinstall the plugin.

To remove a plugin, use the remove command, as in the following example:

$ bin/investigate-plugin remove

You can also remove a plugin manually by deleting the plugin’s subdirectory under the plugins/ directory.

Removing a plugin will result in an "optimize" run which will delay the next start of Siren Investigate.

Disabling Plugins

Use the following command to disable a plugin:

./bin/investigate --<plugin ID>.enabled=false (1)
Disabling or enabling a plugin will result in an "optimize" run which will delay the start of Siren Investigate.
1 You can find a plugin’s plugin ID as the value of the name property in the plugin’s package.json file.

Configuring the Plugin Manager

By default, the plugin manager provides you with feedback on the status of the activity you’ve asked the plugin manager to perform. You can control the level of feedback with the --quiet and --silent options. Use the --quiet option to suppress all non-error output. Use the --silent option to suppress all output.

By default, plugin manager requests do not time out. Use the --timeout option, followed by a time, to change this behavior, as in the following examples:

Waits for 30 seconds before failing
bin/investigate plugin --install username/sample-plugin --timeout 30s
Waits for 1 minute before failing
bin/investigate plugin --install username/sample-plugin --timeout 1m

Plugins and Custom Siren Investigate Configurations

Use the -c or --config options to specify the path to the configuration file used to start Siren Investigate. By default, Siren Investigate uses the configuration file config/siren.yml. When you change your installed plugins, the bin/investigate plugin command restarts the Siren Investigate server. When you are using a customized configuration file, you must specify the path to that configuration file each time you use the bin/investigate plugin command.

Plugin Manager Exit Codes

0

Success

64

Unknown command or incorrect option parameter

74

I/O error

70

Other error

Contributing

For up-to-date documentation on how to contribute to Kibana and/or develop Kibana plugins (compatible with Siren Investigate) please refer to the Kibana documentation.

Limitations

Kibana currently has the following limitations.

Nested Objects

Kibana cannot perform aggregations across fields that contain nested objects. It also cannot search on nested objects when Lucene Query Syntax is used in the query bar.

Using include_in_parent or copy_to as a workaround is not supported and may stop functioning in future releases.

Release Notes

Siren Investigate 10.0.0

Siren Investigate Changes

Fixed:

Changed:

Plugins

Kibi 5.4.3-4

Kibi Changes

Fixed:

  • Fixed issue where tooltips for pruned joins were missing in some parts of the dashboard

  • Fixed issue in the graph browser where the add menu could not be opened after adding a filter

  • Returned the ability to add columns from fields to the enhanced data table in a dashboard

  • Fixed a bug in timelion that prevented opening the dashboard timepicker

  • Fixed bug where Translated Request and Debug sections of spy panel didn’t show anything

Changed:

  • Update Siren Platform distributions to use Elasticsearch version 5.6.4

  • Update Siren Platform distributions to use Vanguard version 5.6.4

  • Update Siren Platform distributions to use Searchguard version 5.6.4-16

Plugins

Graph Browser

  • Live filters can now be "unlinked" from the graph selection and kept as a normal filter

  • Links are now filterable with the timebar if they have a temporal field and a type configured

Kibi 5.4.3-3

Kibi Changes

Fixed:

  • Fixed issue where doc_table rendering was slow due to large text fields

  • Fixed issue where range filters were not passed by relational filter to other dashboards correctly

Changed:

  • Update Siren Platform distributions to use Elasticsearch version 5.6.3

  • Update Siren Platform distributions to use Vanguard version 5.6.3

  • Update Siren Platform distributions to use Searchguard version 5.6.3-16

Kibi 5.4.3-2

Kibi Changes

Fixed:

  • Fixed issue where Kibi would not start when used with Elasticsearch 5.5.2 due to mapping loading problem

  • Kibi no longer use the elasticsearch.plugins configuration setting to store installed Elasticsearch plugin names

  • Improved X-Pack security compatibility

  • Improved migration from Kibi 4.6.x, where the sourceFiltering property was previously set for any index-pattern

  • Improved migration from Kibi 4.6.x, where the pageSize parameter was previously incorrectly saved as string

  • Various improvements in Sentinl see https://github.com/sirensolutions/sentinl/releases/tag/tag-5.5 for details

Changed:

  • Update Siren Platform distributions to use Elasticsearch version 5.5.2

  • Update Vanguard to version 5.5.2-1

  • Default Access Control plugin policy on conflicting roles changed to DENY_IF_AT_LEAST_ONE_ROLE_IS_DENIED, to revert to previous policy set kibi_access_control.acl.policy property to ALLOW_IF_AT_LEAST_ONE_ROLE_IS_ALLOWED in kibi.yml file

Added:

  • Added new UI element rules in Access Control plugin to controll access to certain parts of the UI e.g: applications (Timelion, Sentinl), Kibi sections (discover, management), specific features (export CSV functionality)

  • Added ability to limit number of collected tuples per shard during the join operations This option can now be set globally via kibi:joinLimit in each relation’s settings in the Relational Panel

  • Added support for dynamic object mapping type to Saved Object API

Kibi 5.4.3-1

Kibi Changes

Changed:

  • Vanguard version bumped from 5.4.3 to 5.4.3-1

Kibi 5.4.3 and Kibana 5.4.3

The lists below cover changes between Kibi 5.4.0 and Kibi 5.4.3

Kibi Changes

New:

  • New dashboard sidebar with drag and drop functionality and other UI improvements

  • Dashboard selection dropdowns now group by dashboard group or saved search

Changed:

  • Saved searches are now shown by their titles rather than IDs

  • pageSize is optional in data table visualization, defaults to discover:sampleSize

  • Improved new dashboard modal layout and text

  • Reduced the size and space usage of many visualizations to increase available dashboard area

  • Removed the CSV export button from the data table visualization

  • Made the top paginator optional on the data table visualization

  • Now allowing the user to set a minimum column width on the data table visualization

  • The defaultDashboard setting in kibi.yml has been deprecated, set in advanced options in the management tab

  • Added generic error for an unknown error in the button count calculation query request

Fixed:

  • Fix incorrect notification when cancelling dashboard save

  • Relational buttons are now disabled until counts are complete

  • Relational filter label tooltip fixed

  • Unnecessary call to FontAwesome CDN removed

  • Time filter change now removes the count in the relational filter

  • Exception on new index pattern creation fixed

  • Share function now returns spaces correctly

  • Visualization mode changes no longer destroys the visualization state

  • When cancelling dashboard edit mode, time is no longer restored

  • Fixed sorting by alias when alias missing on data table visualization

  • Improved the filter tooltip for terms queries

Plugins

Timelion

  • Warning on security exception when accessing restricted indices

Box Plot

  • Log and square root Y-axis scales added

  • Fixed axis labels getting cut off

  • Aligned styles with other visualizations

Graph Browser

  • Kibi Graph Browser is disabled if the Kibi instance is unlicensed, a placeholder is shown if so

  • Geo_fields and time fields are added to the node configuration

  • Can now expand by relation using filters

License

  • License upload/delete buttons in management/license

  • License information now available in management/About page

Multichart

  • Added warning when aggregation type not supported by Kibi Multi Chart

Scatter Plot

  • Colors now persist on selection change

  • Support for .raw and scripted fields added

  • Fixed blank graph when data being rendered is unchanged after e.g. query text removal

  • Fix brush selection rendering results correctly

Kibi 5.4.0 and Kibana 5.4.0

The lists below cover changes between Kibi 4.6.4 and Kibi 5.4.0

Kibi Changes

New:

  • Merged changes from Kibana 5.4.0

  • New sidebar menu for dashboard management and navigation (replaces previous tabs)

  • Uses the new Siren Vanguard (an Elasticsearch plugin which adds relational technology to Elasticsearch, providing the capability to do joins across multiple indices)

  • Enhanced enterprise security - added support for Kerberos/JWT

  • Updated Kibi 4 to 5 migration page

  • Added additional information for shards failures

  • Page size option for the Enhanced Eearch Results visualization to override discover:sampleSize

  • Add build timestamp to info page

  • Handle image error when value formatted as url/image

  • Added migration for the heatmap visualisation

  • Added kibi_wordcloud to tagcloud migration

  • Added a gremlin_server plugin to handle the startup of Gremlin Server

  • Added kibi:graphStatesLimit to the advanced options

  • Added Kibi Category to vis wizard

  • Allow middlewares to set parameters explicitly

  • Kibi config is now a singleton

  • Added server.ssl.ca option to kibi.yml - allows to use a CA file to verify the SSL certificate during development

  • Update default gremlin server datasource during startup

  • Adds timelion sheets to saved object API

  • Ported html-angular view from 4.6.4-1

  • Added backup/restore commands to backup/restore content of .kibi index

  • Added panels border option

  • Added space between panels

Changed:

  • Removed support for Windows 32 bit

  • Move up one dir before running kibi.bat on Windows

  • Removed advanced join settings

  • Use match_none for empty searches

  • Migration of default dashboard title from kibi.yml to advanced settings

  • Updated kibi license file

  • Remove CONTRIBUTING.md

  • Remove tinkerpop3 related queries and filter out tinkerpop3 data sources

  • Hooks before transferVisState, usefull for multichart

  • Check for siren-vanguard instead of siren-platform

  • Removed dashboard groups management section

  • Renamed kibana-plugin binary to kibi-plugin

  • Use where.exe in place of which on windows

  • Use jar for archiving on Windows

  • Added route for getting elasticsearch plugins

  • Removed relational panel settings

  • Removes tinkerpop3 query

Fixed:

  • Bumped kibiutils to version 1.8.0

  • Heatmap layout fixed

  • Improved application logging

  • Improved notifications

  • Improved migration procedure for a rare case where after migration relations definitions were wrong

  • Improvements to tag-cloud visualisation

  • Saved object finder improvements about reserved characters

  • Improved tooltips styling and content

  • Updates counts when global time filter is changed

  • Use column aliases for the sorting menu

  • Add type attribute to up/down buttons so that they don’t trigger a form submission

  • Documentation updated to version 5

  • Fix extra scrollbar in doc table on pagination button click

  • Reset query text on reset button click

  • Ignore Gremlin configuration if not set

  • Do not start the gremlin server if the config is missing

  • Disable the gremlin plugin during the upgrade

  • Ignore plugins/sentinl/phantomjs to avoid braking phantomjs install by Sentinl app

  • Improvements in handling the missing data indices

  • Override the default dark theme button color

  • Filtering on the advanced table triggers the underlying click handler

  • Fixed console when using an https agent

  • Handle sorting on analysed or missing fields

  • Reset pagination on filter change

  • Fixed issue when adding a filter from a SQL based visualisation

  • Fixed missing Vanguard notification

  • Fixed export meta fields as csv

  • X-Pack monitoring instructions and notification fix

  • Timelion fixes

  • Display query title instead of id

  • Increased timeout of esArchiver test

  • Use hashed item store class to handle quota storage exception

  • Fixed legend exception

  • Replace join filters on empty index sets with match_none queries

  • Use the docTable directive from kibana in the enhanced search results visualisation

  • Make management tabs responsive to width

  • Include must_not in getHighlightRequest processing

  • Allows to honor the handleNoResults property

  • Fixed required field highlighted

  • Added missing docker doc

  • Use field_caps instead of field_stats to get field capabilities

  • Clicking on the kibi logo fixed

  • Use requiresTimePicker visualization param

  • Time input fields is made equal

  • Updates README.md

  • Relations menu filter fixed

  • Fixed issue about Proxy class not available on all browsers

  • Selected Documents issue fixed

  • Allow vis to change their es request before serialization proc begin

  • Adds support to save uistate for multiple instance of the same vis type

  • Allows restoring the uiState after click edit visualisation on dashboard

  • Fix for cluster.createClient

  • Do not add join queries to the highlight_query query

  • Take the kacConfiguration from chrome.getInjected

  • Use config.has to check for configs existence

  • Relative time range validation fixed

  • Column alias validation added

  • add hook to access control in the uiSettings API

  • Allows histogram to be interval safe

  • Fixed tabs in management objects

  • Response check added to create_kibi_proxy

  • Alias checking added to search

  • Try every index pattern if the default index is not reachable because of an authorization error

  • Port join wrapping into bool.must

  • Fix handling of missing saved searches

  • Update styles for navbar and filterbar tooltips

  • Check for forward and backslashes in packagePaths.

  • Time sync checklist fixed

  • Fixed plugin install error

  • Removed not used events

  • Set baseURL in eeg

  • Allow to add filter via table details in the visualize page

  • Column rename fixed

  • Use data cluster

  • Do not retrieve the scope of the element

  • Filter label fixed

  • Put Kibi and Kibana version

  • Fixed wrong hint on the rel filter

  • Use urandom in Gremlin Server, documentation updates

  • Border and filter bar color fix with dark theme

  • preserve column layout in the enhanced search results visualisation

  • Ported import export improvements from 4.6.4-1

  • Fixed incorrect mouseup event handling on fontawesome-icon-picker

  • Ported ACL fix in the relational filter visualisation to 5

  • Ported changes for Kibi Enterprise to 5

  • RefreshInterval object check added

  • Use fontawesome-iconpicker 1.2.1 instead of ui-iconpicker

  • Use tag instead of commit hash in package.json to point to kibi-h2o2

  • Build module path correctly on windows

  • Updated native bindings for Darwin

  • Set temp folder based on OS

  • Pagination fixed

  • Proxy the HTTP status code and upstream ttl

  • removed the wrong pointer to kibi.dev.yml

  • Made object actions always visible

  • Do not shorten URLs in shared links UI tests

  • Corrected typeahead for the Discover/Visualize/Dashboard pages

  • Removed old version of the elasticdump dependency

  • Kibana reference renamed

  • Ignore delayed executions that are cancelled

  • Hide the tooltip on destroy

  • Do not submit the form on click of the query history

  • Check that all query_string queries are put into the must clause

  • Keep toaster next to the dashboards bar

  • Validator fixed

  • Ported - introduction of MissingDashboardError

  • Ported documentation about cross frame communication

  • Support slash in entity uri and corrected some bugs about selecting documents

  • Refactor the kibi proxy code to use new functions: onResponse and onBeforeSendRequest

  • Fixed platforms mappings to classifiers

  • Ported how sharing link is generated

  • No results found alignment in Kibi data table fixed

  • Open a new clean (no cache) web browser tab with Kibi if the logo is clicked

  • KibiSequentialJoinVisHelper improved

  • Correctly merge params and headers from datasource and query

  • Moved URL sharing functions to a service

  • Add info box on the relations setting

  • Created method for determining if object is from Kibi

  • Decorate query in kibi state

  • Include basePath in Kibi session redirect, save Kibi state using save method

  • Add info box to the Sequential Join Viz to instruct a user about how to create a relation between 2 types under 1 index.

  • Handle cases where the company or the investor is missing

  • Fix fullscreen mode

  • Added the join icon indicator in the indices management page

Plugins

  • Fix agg config save after apply

  • Fix Scatterplot on Kibi 5

  • Add path to Phantomjs binary in horseman options

  • Patch es client with new Vanguard methods

  • Allow underscore in ACL role ids

  • Fixed and unified plugin versions

  • Update plugin versions to 5.4.0

  • Removed extra wrapping into Kibana folder for non-public plugins

Access Control

  • Ported Access Control plugin to Kibi 5

  • Ported Kerberos / JWT support to 5

  • Changed title into label

  • Removed $cookies dependency

  • Return parameters from middleware methods

  • Fixed wrong config variable name

  • Expose scope in Authentication/ACL editors

  • Fix issue when empty path

Box Plot

  • Box Plot ported to Kibi 5

  • Refactor box_plot visualisation

  • Update Box Plot x axis labelling

  • Show whiskers correctly in Box Plot

Bubble Diagram

  • Bubble diagram ported to Kibi 5

Enterprise Components

  • Port Enterprise Components to Kibi 5

  • Describe unknown xhr errors

  • Object for cross frame communication

Graph Browser

  • Port Graph Browser to Kibi 5

  • Set the default datasource id

  • Change the filter to a bool filter

  • Load scriptSource from file if present

  • Reinit the cached scope

  • Remove scroll API usage in gremlin server

  • Support multiple undo/redo states

  • Added graph script middleware + fix script editor

  • Remove an unneeded Elasticsearch query

  • Keylines upgrade

  • Always add Basic to the auth header

  • Added wrapping Kibana folder

  • Init on getters

  • Set max height to add menu

  • Refactor persistence helper

  • Pass time from the state

  • Add remove all button

  • Reflect deletion of live filter on filter_bar in graph browser filter button

  • Fix graph browser configuration issues

  • Fixed issue where new elements are not tracked by the timebar if filtering was previously enabled

  • Show warning on missing relations on runtime and configuration

  • Fixed an error when switching back to a graph browser dashboard

  • Expand by relation does not retrieve count for retrieved nodes

  • Fix for remove and crop

  • Select on Graph Browser associated with saved search

  • Don’t overwrite existing node upon expansion

  • No such index trying to configure graph

  • Allow graph timebar to work with arcs

  • Remove URL length check on graph selection (port to 5)

  • Self join support

  • Fix highlightning (port to 5)

  • Add graph and script to savedObjectAPITypes

  • Prevent any changes of the gremlin server datasource

  • Better graph tooltip positioning

License

  • License plugin ported to Kibi 5

  • Fixed plugin name

  • Fixed tests on license plugin

Multichart

  • Ported Multichart to Kibi 5

  • Miltichart can now save the smart default configurations

  • Improved save state handling

  • Fix single call on Multichart

  • Multichart SDC, serialisation refactor and unit tests

  • Fix multiple configurations on Multichart

  • Fix property edition on visualize

  • Missing after fetch event handler

  • Focus the graph after script execution

  • Configurable relations

Scatter Plot

  • Scatter plot ported to Kibi 5

Vector Map

  • Vector map ported to Kibi 5

Kibana Changes

Deprecations & Removals

Visualize
  • Remove "Exclude Pattern Flags" and "Include Pattern Flags" from terms and significant terms aggregations {repo}issues/6714[#6714]

  • Deprecate ascending sort for terms aggregations {repo}pull/8167[#8167]

  • Deprecate split chart option for tile map visualization {repo}pull/6001[#6001]

Security fixes

An Open Redirect vulnerability has been fixed with the short URL feature. Previously, a malicious user could use the internal API that powers the short URL feature to create a short URL in kibana that redirected to a different domain.
{security}[ESA-2016-08] ({repo}commit/92ae3ae[92ae3ae])

Kibana 5.0.0 and 5.0.1 were making requests to advanced settings and the short URL service on behalf of the kibana server rather than the current user, which means that being authenticated at all was sufficient to have both read and write access to the advanced settings and short URLs.
Kibana 5.0.2 now authenticates requests for each service on behalf of the current user.
{security}[ESA-2016-10] ({repo}pull/9214[#9214])

When previous versions of Kibana 5 are configured for SSL client access, file descriptors will fail to be cleaned up after certain requests and will accumulate over time until the process crashes. Requests that are canceled before data is sent can also crash the process.
{security}[ESA-2017-02] ({repo}pull/10225[#10225])

Bug Fixes

Core
  • Fix alias support when fetching types {repo}pull/8338[#8338]

  • Report useful error message when sessionStorage is unavailable {repo}pull/8343[#8343]

  • Improved error message when sessionStorage is disabled in the browser {repo}pull/8343[#8343]

  • Trailing slash redirects now include the basepath configuration {repo}pull/8966[#8966]

  • Elasticsearch version checking no longer causes startup error for non-HTTP nodes {repo}pull/9181[#9181]

  • Favicons are now embedded as links rather than as data {repo}pull/8961[#8961]

  • Fix bug where the loading indicator was wider than the screen {repo}pull/8854[#8854]

  • The Kibana logo in the loading screen now shows properly in IE11 {repo}pull/9921[#9921]

  • Browser-specific style overrides are now properly being handled for legacy browsers {repo}pull/9899[#9899]

  • Bump Node.js to version 6.9.5. This was a low severity security release for Node.js, which has minimal impact to Kibana, but is still worth upgrading. {repo}pull/10135[#10135]

Dashboard
  • Prevent dashboard title tooltip from being cut off {repo}pull/6464[#6464]

  • Dashboard no longer set to dirty on load in some situations {repo}pull/9307[#9307]

Discover
  • Only display Visualize button when a field is aggregatable {repo}pull/8694[#8694]

  • Field visualize button no longer loads incorrect URL in some situations {repo}pull/8721[#8721]

  • Sorting on scripted date or boolean fields no longer triggers an error {repo}pull/9261[#9261]

  • Painless scripted fields are now wrapped in a lambda so more complex scripts are possible {repo}pull/9171[#9171]

  • Correctly renders error when scripted field languages fail to load {repo}pull/8639[#8639]

  • Improve spy tab performance on Discover {repo}issues/9464[#9464]

  • Reduce lag experienced when expanding doc table rows {repo}pull/9326[#9326]

  • Prevented a background action that was causing unnecessary CPU cycles {repo}pull/10036[#10036]

Management
  • No longer remove selection when refreshing fields {repo}pull/8312[#8312]

  • Notify user of failures when deleting saved objects {repo}pull/7345[#7345]

  • Add title to visState when the visualization is saved {repo}pull/7185[#7185]

  • Back button now works {repo}pull/5982[#5982]

  • Show no value instead of interpolating 'undefined' with empty values in URL string formatters {repo}pull/6291[#6291]

  • Delete button for color formatters no longer overlaps format dropdown {repo}issues/8864[#8864]

  • Pull Request 10521: Attempting to import a missing type now results in a warning

Filters
  • Use lt instead of lte for safer upper bound in range filter {repo}pull/7129[#7129]

  • Fix date histogram filtering {repo}pull/7126[#7126]

  • Automatic filter pinning option in advanced settings {repo}pull/5730[#5730]

Server
  • Console logs display date/time in UTC {repo}pull/8534[#8534]

Status
  • Plugins without init function no longer show statuses {repo}pull/7953[#7953]

Timepicker
  • Absolute time picker updates when time selection changes {repo}pull/8383[#8383]

  • Prevent relative timepicker values from being negative {repo}pull/6607[#6607]

  • Timepicker now has a collapse button again {repo}issues/9381[#9381]

Visualize
  • Remove average from standard deviation metrics {repo}pull/7827[#7827]

  • Always set output.params.min_doc_count on Histograms {repo}pull/8349[#8349]

  • Set minimum aggregation size to 1, Elasticsearch returns an error for 0 {repo}pull/8339[#8339]

  • Add milliseconds to Date Histogram interval options {repo}pull/6796[#6796]

  • Do not perform unnecessary round-trip to Elasticsearch when there are no changes in request parameters {repo}pull/7960[#7960]

  • Tile map dots no longer shrink to extreme tiny size on some zooms {repo}pull/8000[#8000]

  • Table visualizations display correctly when changing paging options {repo}pull/8422[#8422]

  • Filter non-aggregatable fields from visualization editor {repo}pull/8421[#8421]

  • Prevent charts from unnecessarily rendering twice {repo}pull/8371[#8371]

  • Display custom label for percentile ranks aggregation {repo}pull/7123[#7123]

  • Display custom label for percentile and median metric visualizations {repo}pull/7021[#7021]

  • Back button now works {repo}pull/5986[#5986]

  • Fix extraneous bounds for tilemap {repo}pull/7068[#7068]

  • Median visualization properly shows value rather than ? {repo}pull/7003[#7003]

  • Map zoom is persisted when saving visualization {repo}pull/6835[#6835]

  • Drag aggregations to sort {repo}pull/6566[#6566]

  • Table sort is persisted on save {repo}pull/5953[#5953]

  • Ignore extended bounds when "Show empty buckets" unselected {repo}pull/5960[#5960]

  • Using custom label for standard deviation aggregation {repo}pull/6407[#6407]

  • Tile map bounding boxes no longer create filters with invalid bounds {repo}issues/8946[#8946]

  • Visualizations without spy panels no longer trigger errors in browser console {repo}pull/9115[#9115]

  • Bar graph order is now correct with double split terms {repo}pull/8397[#8397]

  • Proper handling of small slices in pie chart {repo}pull/8986[#8986]

  • Fix label on scripted field date histograms {repo}pull/8638[#8638]

  • UTF-8 charset when exporting aggregate tables {repo}pull/8662[#8662]

  • Fixed various typos in visualization descriptions {repo}pull/8943[#8943]

  • Toggling spy panel no longer throws an error {repo}pull/8877[#8877]

  • Fullscreen spy panel is no longer cut off {repo}pull/8844[#8844]

  • Remove scripted fields from significant terms since they are unsupported {repo}pull/8734[#8734]

  • Using a secondary datetime field no longer triggers an error {repo}issues/9458[#9458]

  • Metric visualizations now show scrollbars when the value overflows the container {repo}pull/9481[#9481]

  • Axis custom extents now support decimal values {repo}pull/9426[#9426]

  • Fixed regression where certain visualizations were being limited to 25 series {repo}issues/10132[#10132]

  • Fixed typo on a tag cloud warning message {repo}pull/10092[#10092]

  • Fixed a bug where data table visualizations would incorrectly appear empty in certain circumstances {repo}issues/9757[#9757]

  • Issue 10153: Fixed regression where include and exclude patterns triggered an error

  • Issue 10295: Fixed regression where grouped bar charts did not properly scale down their y-axis

Sharing
  • Share UI now properly honors the dark theme {repo}issues/8819[#8819]

CLI
  • Spaces are now accepted in plugin URLs and paths during installation {repo}pull/8945[#8945]

  • Plugin install will now fire EPERM errors in Windows less frequently {repo}pull/9260[#9260]

Console
  • Console now autocompletes indexes {repo}pull/8557[#8557]

  • Pull Request 10244: Literal strings in JSON editor are now more clearly identifiable

Timelion
  • The "new" action no longer requires two clicks {repo}pull/8815[#8815]

  • Secondary y-axis no longer removes config on first axis {repo}pull/9197[#9197]

  • Correct padding for Timelion title {repo}pull/8919[#8919]

  • Specifying yaxis() no longer forces a minimum value of 0 {repo}pull/9428[#9428]

  • Improved dark theme support for Timelion axis and legend labels {repo}pull/9422[#9422]

Dev Tools
  • The link to the Dev Tools app is now hidden when no developer tools are enabled {repo}pull/9489[#9489]

Plugins
  • Calling another API route via .inject() no longer fails due to a missing socket {repo}pull/9332[#9332]

Build
  • Issue 9652: Kibana builds now include a NOTICE file

Enhancements

CLI
  • New plugin installer: bin/kibana-plugin {repo}pull/6402[#6402]

  • Ability to specify multiple config files as CLI arguments {repo}pull/6825[#6825]

  • Display plugins versions {repo}pull/7221[#7221]

Core
  • Bind Kibana server to localhost by default {repo}pull/8013[#8013]

  • Only proxy whitelisted request headers to Elasticsearch {repo}pull/6896[#6896]

  • Remove client node filtering in the Elasticsearch version check {repo}pull/6840[#6840]

  • A new design {repo}pull/6239[#6239]

  • Friendly error message when Kibana is already running {repo}pull/6735[#6735]

  • Logging configuration can be reloaded with SIGHUP {repo}pull/6720[#6720]

  • Abortable timeout counter to notifications {repo}pull/6364[#6364]

  • Upgrade Node.js to version 6.9.0 for improved memory use and a segfault fix {repo}pull/8733[#8733]

  • Warn on startup if plugins don’t support the version of Kibana {repo}pull/8283[#8283]

  • Add additional verification to ensure supported Elasticsearch version {repo}pull/8229[#8229]

  • Add unique instance identifier {repo}pull/6378[#6378]

  • Add state:storeInSessionState option enabling shorter URLs and enhancing Internet Explorer support {repo}pull/8022[#8022]

  • Improve user experience when query returns no results {repo}pull/7286[#7286]

  • Display message when "Export All" request fails {repo}pull/6976[#6976]

  • Improved rendering performance and responsiveness across the whole product {repo}pull/7929[#7929]

  • Improved CPU usage when the progress indicator is present {repo}pull/8842[#8842]

  • New loading screen {repo}pull/8970[#8970]

  • Support for searching against tribe nodes {repo}pull/9132[#9132]

  • Automatically select default index pattern if there is only one {repo}pull/9679[#9679]

  • Remove "will be cached for next time" message from loading screen {repo}pull/9383[#9383]

Dashboard
  • Dashboard refresh interval persisted on save {repo}pull/7365[#7365]

Dev Tools
  • Add Dev Tools application, including Console (previously known as Sense) {repo}pull/8171[#8171]

Discover
  • Default columns are configurable {repo}pull/5696[#5696]

  • Render field type in tooltip when mousing over name {repo}pull/6243[#6243]

  • Add field-exists filter button to doc table {repo}pull/6166[#6166]

  • Enable better caching of time-based requests by Elasticsearch {repo}pull/6643[#6643]

  • Improved rendering performance on Discover app with large numbers of fields {repo}pull/9014[#9014]

  • Improved consistency with the sidebar interface {repo}pull/7958[#7958]

Filters
  • Allow more than match queries in custom filters {repo}pull/8614[#8614]

Management
  • Rename Settings to Management {repo}pull/7284[#7284]

  • Add boolean field formatter {repo}pull/7935[#7935]

  • Add painless support for scripted fields {repo}pull/7700[#7700]

  • Custom notification banner configured via advanced settings {repo}pull/6791[#6791]

  • Duration field formatter for numbers {repo}pull/6499[#6499]

  • Title case field formatter for strings {repo}pull/6413[#6413]

  • Ability to exclude specific source fields for an index pattern {repo}pull/7402[#7402]

  • Conflicting field types of an index pattern are now visually flagged in index pattern management {repo}pull/7990[#7990]

  • Color formatter for string fields {repo}pull/8597[#8597]

  • Histogram interval now supports decimal {repo}pull/8566[#8566]

  • Advanced setting for opacity when for point-series charts {repo}pull/8448[#8448]

  • Advanced setting to ignore filters if index does not contain field {repo}pull/8181[#8181]

Plugins
  • Add support for apps to specify their order in the left navigation bar {repo}pull/8767[#8767]

  • Separate plugin version and supported version of Kibana {repo}pull/8222[#8222]

  • Expose the Kibana app base URL, no more hardcoding '/app/kibana' in urls {repo}pull/8072[#8072]

  • Add requireDefaultIndex route option, enabling index pattern independent plugins {repo}pull/7516[#7516]

  • Add plugin preInit extension point {repo}pull/7069[#7069]

  • Plugins can prefix their config values {repo}pull/6554[#6554]

Saved Objects
  • Dashboards, visualizations, and saved searches can now be renamed while saving {repo}pull/9087[#9087]

  • Improved UI when editing saved objects {repo}pull/9543[#9543]

  • Improved UI when viewing saved objects {repo}pull/9535[#9535]

Server
  • Add basePath to server’s defaultRoute {repo}pull/6953[#6953]

  • Do not render directory listings for static assets {repo}pull/6764[#6764]

  • Automatically redirect http traffic to https {repo}pull/5959[#5959]

  • Write process pid file as soon as it is known {repo}pull/4680[#4680]

  • Log most events by default and only errors when in quiet mode {repo}pull/5952[#5952]

Sharing
  • Improve user interface to emphasize difference between Original URLs and Snapshot URLs. {repo}pull/8172[#8172]

Status
  • Emit new state and message, on status change {repo}pull/7513[#7513]

  • Status API now includes the Kibana version and build number {repo}pull/9195[#9195]

Timelion
  • Add Timelion to Kibana core {repo}pull/7994[#7994]

  • Timelion sheets can now be deleted {repo}pull/9191[#9191]

Visualize
  • Add y-axis logarithmic scale for bar charts {repo}pull/7939[#7939]

  • Add option to set legend position {repo}pull/7931[#7931]

  • Add legend tooltips {repo}pull/7890[#7890]

  • Add x-axis title labels {repo}pull/7845[#7845]

  • Tag Cloud visualization {repo}pull/8104[#8104]

  • Brush can now be used to select a subsection of a histogram {repo}pull/9039[#9039]

  • Ability to select legend position for tile map visualizations {repo}pull/8176[#8176]

  • Heatmap visualization {repo}pull/9403[#9403]

  • Line and area charts now support stepped lines {repo}pull/9425[#9425]

  • Tilemap zoom capabilities are now determined automatically when using the default Elastic Tile Service {repo}pull/8630[#8630]

Kibi 4.6.4 and Kibana 4.6.4

Kibi Changes

  • Fixed - cleaned and improved sharing the graph state via URL [Enterprise Edition only]

  • Fixed - static route path for plugins

  • Fixed - use correct scroll_id in migrations

  • Fixed - upgrades between dash releases

  • Fixed - checkIfItIsRelevant in REST datasource returned true when no document was selected but required

  • Fixed - honour timezone settings in Graph Browser timebar [Enterprise Edition only]

  • Improved - upgraded moment-timezone to version 0.5.11

  • Improved - upgraded siren-join to version 2.4.4

  • Improved - upgraded license plugin to version 2.4.4 [Enterprise Edition only]

  • Improved - upgraded Kibi Timeline plugin to version 4.6.4

  • Improved - state is by default saved in session storage state:storeInSessionStorage=true

  • Improved - init explanation only on filter bar filters

  • Improved - added link to sample kibi.yml file into the documentation

  • Improved - tests coverage and quality

  • Improved - removed savedSession objects in favour of sharing through shortened URLs stored in kibi index

  • Improved - field formatter support in Graph Browser tooltips [Enterprise Edition only]

  • Added - merge upstream changes from Kibana 4.6.4.

  • Added - column name aliases to kibi-doc-table plugin

  • Added - priority field for dashboards

  • Added - hashed URL are now enabled by default

  • Added - and option to increase grid resolution

  • Added - kibi:enableAllDashboardsCounts to disable counts on all dashboards.

  • Added - kibi:enableAllRelBtnCounts to disable counts on all relational buttons.

  • Added - search on relations

  • Added - support for ACL rules on saved objects [Enterprise Edition only]

  • Added - new Kibi Multi Chart plugin [Enterprise Edition only]

  • Added - new Kibi Box Plot plugin [Enterprise Edition only]

  • Added - new Kibi Scatter Plot plugin [Enterprise Edition only]

  • Added - new Kibi Bubble Diagram plugin [Enterprise Edition only]

  • Added - new Kibi Vector Map plugin [Enterprise Edition only]

  • Added - new Kibi Horizontal Bar Chart plugin [Enterprise Edition only]

Kibana Changes

Enhancements

  • Pull Request 8733: Upgrade Node.js to version 6.9.0 for improved memory use and a segfault fix.

  • Pull Request 8022: Add state:storeInSessionState option enabling shorter URLs and enhancing Internet Explorer support.

  • Pull Request 8313: Upgrade Hapi server to 14.2.0 to support the new Node.js version.

  • Pull Request 9014: Improved rendering performance on Discover app with large numbers of fields.

Bug Fixes

Kibi 4.6.3-1

Kibi Changes

  • Added - kibi:splitTabs option to split name and counts on tabs

  • Added - kibi:graphUseFiltersFromDashboards option to customize menu when expanding nodes with filters from dashbords

  • Fixed - Not able to scroll smoothly on relations configuration page

  • Fixed - Timepicker does not show on the visualize page for visualizations depending on several searches

  • Fixed - Improved documentation about es compatibility

  • Fixed - If a dashboard contains a kibi_sequential_join_vis, Kibi triggers a search query for the visualization on the default index pattern

  • Fixed - initQtip functions is destroying all qtips

  • Fixed - Sort relations by name and index pattern when choosing relation in relational panel configuration

  • Fixed - Template vars other than "label" ignored in Kibi Query Viewer visualisation

  • Fixed - Currently selected tab not visible when navigating to Settings and back

  • Fixed - Match all Internet Explorer versions in URLOverflowService

  • Fixed - Session persistence bug in Graph Browser [Enterprise Edition only]

  • Fixed - Edge removal issue in Graph Browser [Enterprise Edition only]

  • Improved - Better handling of patterns not matching any index

  • Improved - UI improvements in Kibi Query Viewer visualisation

  • Improved - Simplified option menu when expanding nodes with filters from dashbords

  • Improved - bumped Kibi Timeline visualisation to version 4.6.3-1

  • Improved - bumped Kibi Wordcloud visualisation to version 4.6.3-1

  • Removed - kibi:zoom option - use browser zoom instead

Kibi 4.6.3 and Kibana 4.6.3

Kibi Changes

  • Fixed - upgradeable configurations were not correctly sorted

  • Fixed - outdated filters in the state were not correctly handled

  • Fixed - the relational panel tab was enabled even if relational panel was disabled

  • Fixed - source and target dashboards in relational button configuration were filtered incorrectly

  • Fixed - handling of authorization errors when computing indices in navbar helper and relational sequence buttons

  • Fixed - nested join_sequence creation

  • Fixed - negated joins creation by enclosing the complete query in a must clause

  • Fixed - listen to config events to correctly set the relational filter visible

  • Fixed - do not destroy handlers in cached helper modules to correctly refresh counts on tabs

  • Fixed - fail to access dashboard app if the previous dashboard was deleted

  • Fixed - missing timepicker when creating a new dashboard

  • Fixed - click handlers actions are not cleaned when staging the enhanced search vis

  • Fixed - cases when wrong property name was used in createNotifier

  • Fixed - diff_time now takes into account time precision

  • Fixed - kibi state was not properly propagated which caused an issue where time was not correctly set on a dashboard when done from discovery page

  • Fixed - problems with JSON textarea and label textbox edit

  • Fixed - do not center content if there is data in Kibi Query Viewer

  • Fixed - do not make a relation invalid if type is set to null by kibi-select

  • Fixed - reset count/isPruned property of a dashboard if its count was requested but was not in the metadata object, due to the dashboard not having a savedsearch on save

  • Fixed - now queries for invisible dashboards are not added to count request

  • Fixed - counts not properly updated when using "undo via back button"

  • Fixed - destroy html created by eeg library

  • Improved - check if the state contains outdated filters

  • Improved - relations graph configuration has to be explicitly saved

  • Improved - documented Sense and Marvel integration with Search Guard

  • Improved - support relational filter visualizations which reference no concrete index

  • Improved - made relational button grey when there is no results

  • Improved - store the synced dashboards in the kibistate

  • Improved - various sync-time-to directive improvements

  • Improved - notify the user if the session data has been saved

  • Improved - prevent the Kibi Enhanced Search visualization to make two requests for retrieving the rows

  • Improved - handling of various authorization errors

  • Improved - replaced ui-select with kibi-menu-template

  • Improved - introduced kibi-menu-template for choosing relations in buttons configuration

  • Improved - Display confirmation dialog when leaving a modified relational configuration

  • Improved - check if the relation id is defined at config time for the relational filter visualization

  • Improved - big part of code ported to ES6

  • Improved - use version 4.6.3 of the Kibi Timeline

  • Improved - ported all release notes to master

  • Improved - Searchguard integration documentation

  • Added - merge upstream changes from Kibana 4.6.3

  • Added - Saved objects API

  • Added - support requiresMultiSearch in the visualize editor for refreshing the ES results

  • Added - renamed kibi_core.default_dashboard_id into kibi_core.default_dashboard_title

  • Added - kibi:graphUseWebGl advanced option

  • Added - migration for relational filter button

  • Added - test:coverage grunt task

  • Added - Kibi now ships with Sentinl application [Enterprise Edition only]

  • Added - Map view to Graph Browser [Enterprise Edition only]

  • Added - Timeline view to Graph Browser [Enterprise Edition only]

  • Added - live filter to Graph Browser [Enterprise Edition only]

  • Backported - Kibana fix to ensure dashboard panels appear in the correct order

Kibana Changes

Enhancements

Bug Fixes

  • Pull Request 7457: Plugins now have a home for their data

  • Issue 8090: Fixed blank notifications that were appearing in plugin apps like Sense

  • Issue 8060: In some circumstances, Visualization editor controls would collapse after applying updates. They will now remain expanded

  • Pull Request 7422: Better cleanup on package removal

  • Issue 7590: Fixed logging for package installs using SysV

  • Pull Request 7431: A more accurate description for the Kibana service

Deprecations

  • Issue 6833: Ability to sort a terms aggregation by ascending count will be removed in a future version of Elasticsearch

Kibi 4.5.4-1

Kibi Changes

  • Fixed - upgradeable configurations were not correctly sorted

  • Fixed - outdated filters in the state were not correctly handled

  • Fixed - the relational panel tab was enabled even if relational panel was disabled

  • Fixed - source and target dashboards in relational button configuration were filtered incorrectly

  • Fixed - handling of authorization errors when computing indices in navbar helper and relational sequence buttons

  • Fixed - nested join_sequence creation

  • Fixed - negated joins creation by enclosing the complete query in a must clause

  • Fixed - listen to config events to correctly set the relational filter visible

  • Fixed - do not destroy handlers in cached helper modules to correctly refresh counts on tabs

  • Improved - Searchguard integration documentation

  • Added - migration for relational filter button

  • Added - test:coverage grunt task

Kibi 4.5.4 and Kibana 4.5.4

Kibi Changes

  • Various bug fixes and stability improvements

  • Various documentation improvements

  • Redesigned dashboard groups UI to show indicators for each dashboard in a group: the count, the queries/filters, and whether a join was pruned

  • Improved configuration of relational buttons

  • Added support for configuration upgrades during alpha/beta periods

  • Added methods for coordinated search on Elasticsearch client

  • Added support for time-based indices in join_set

  • Added support for time-based indices in join_sequential

  • Added a button to generate a report for performance investigation

  • Added time synchronization feature to set the same time on multiple dashboards at once

  • Added a spy panel to relational button visualization

  • Added saved objects API for index patterns

  • Added possibility to export Enhanced search results as CSV

  • Added a check for compatible mappings of the joined fields in relational panel configuration

  • Added datasource parameters validation in the datasource editor

  • Kibi demo packages are shipped with heatmap visualization

  • Refactored entity URI selection mechanism - now kept in Kibi state

  • Fixed issue where app state was not updated when switching quickly between dashboards

  • Fixed issue where relational panel was hiding search bar type ahead

  • Fixed issue where "add new index pattern" button is missing

  • Fixed issue where return statement was missing after promise reject

  • Fixed issue where timeout error was shown during join calculation

  • Fixed issue where crypto helper should throw an invalid key length exception

  • Fixed issue where filtering by saved search did not work on the visualize panel

  • Fixed issue where kibistate was initialized from saved dashboard even if it was in the URL

  • Fixed issue where pinned relational filters would crash in some direction

  • Fixed dependency issues on status page and discovery page

  • Stability improvements and fixes for JDBC query component

  • Performance improvement in wordcloud visualization

  • Timeline visualization is respecting dateformat:tz

  • Increased default kibi:graphMaxConcurrentCalls to 15

  • Added support for environment variable expansion in configuration files

  • Make the log configuration of the gremlin server more configurable [Enterprise Edition only]

  • Updated tinkerpop3 query to use the graph batch API [Enterprise Edition only]

  • Added preloaded query example for OnUpdate script [Enterprise Edition only]

  • Added support for Search Guard [Enterprise Edition only]

  • Added access control plugin for Search Guard [Enterprise Edition only]

Kibana Changes

Enhancements

  • Issue 6150: Adds a timeout for the display of error notifications.

  • Issue 3682: Improved handling of empty result sets for the Metric visualization.

  • Issue 4065: Adds custom labels for visualization axes.

  • Issue 6128: Adds configurable length to saved object lists.

Bug Fixes

  • Pull Request 5236: Fixes a heatmap intensity issue in tile maps.

  • Issue 6283: Corrects a date format error in millisecond intervals.

  • Issue 6049: Improvements to autocomplete handling.

  • Issue 6331: Improved handling of long-running queries.

Plugin System Changes

  • Issue 5916: Adds the --list flag to list installed plugins.

Kibi 4.5.3-6

Kibi Changes

  • Siren-join plugin version upgraded to 2.3.4-1

  • Backport a fix for issue where no indices have been resolved for a query such query should be send to kibi index

  • Backport a fix to enable the relational panel before removing the join_set filter

  • Backport kibana PR 6651 - Fix courier "request already started" bug

Kibi 4.5.3-5

Kibi Changes

  • Fixed issue where advanced relations settings were not taken after the time-based indices fix

  • Fixed issue where certain count queries were sent twice

  • Fixed typeahead position issue

  • Optimized ng-animate usage

  • Extended elasticsearch.js to support _coordinate_search and _coordinate_msearch queries

Kibi 4.5.3-4

Kibi Changes

  • Support time-based indices for relational filters and tab counts

Kibi 4.5.3-3

Kibi Changes

  • Added csv export for kibi data table

  • Fixed issues with time syncing across dashboards

Kibi 4.5.3-2

Kibi Changes

  • Fixed autoload issue when loading plugins

  • Upgraded timeline plugin to 4.5.3-2 - support dateFormat:tz property

Kibi 4.5.3-1

Kibi Changes

  • Fixed Do not allow to define relations for relational buttons on scripted fields

  • Fixed problem with jdbc datasource

  • Added a visual hint to identify dashboards which state differ from the default one

  • Added new ability to quickly set time for more than one dashboard

  • Disabled session automatic synching to ES

Kibi 4.5.3 and Kibana 4.5.3

Kibi Changes

  • Various bug fixes and stability improvements

  • Various documentation improvements

  • Refactoring of the Kibi state

  • Support for object migration from Kibi 4.4.2

  • Upgraded ElasticSearch version to 2.3.4

  • Improved Export and Import of all objects

  • JDBC warning message should only be displayed for JDBC datasources

  • Fixed error while switching (fast) between tabs

  • Sort alphabetically saved objects in kibi_select

  • Fixed Huge URL, apparently with too much state

  • Updated Boom version

  • Removed the clone cancel …​ buttons

  • Added headers and params to a REST datasource configuration

  • Improved Kibi tests on Windows

  • Improved server logging

  • Added configurable label for datasource params

  • Relational buttons refreshing twice

  • Empty widgets when the database file is missing

  • Removed the cache checkbox in Tinkerpop3 datasource [Enterprise Edition only]

  • Added advanced option: kibi:graphMaxConcurrentCalls [Enterprise Edition only]

  • Added more graph scripts to Graph Browser [Enterprise Edition only]

  • Scalability improvements in Graph Browser [Enterprise Edition only]

Kibana Changes

Enhancements

  • Issue 6150: Adds a timeout for the display of error notifications.

  • Issue 3682: Improved handling of empty result sets for the Metric visualization.

  • Issue 4065: Adds custom labels for visualization axes.

  • Issue 6128: Adds configurable length to saved object lists.

Bug Fixes

  • Pull Request 5236: Fixes a heatmap intensity issue in tile maps.

  • Issue 6283: Corrects a date format error in millisecond intervals.

  • Issue 6049: Improvements to autocomplete handling.

  • Issue 6331: Improved handling of long-running queries.

Plugin System Changes

  • Issue 5916: Adds the --list flag to list installed plugins.

Kibi 4.4.2-2

Kibi Changes

  • Backport fix for Truncate document label if too long

  • Backport fix for Error when label is used for a purple filter is an array

  • Backport fix for Export/Import takes care about index patterns too

  • Bumped timeline version to 4.4.2-2

Kibi 4.4.2-1

Kibi Changes

  • Backport fix for broken tiles in map widget

  • Backport fix for broken import export of objects

  • Backport fix for openjdk version check [Enterprise Edition only]

  • Backport fix support for nested fields label (Graph Browser) [Enterprise Edition only]

Kibi 4.4.2 and Kibana 4.4.2

Kibi Changes

  • Various bug fixes and stability improvements

  • Various documentation improvements

  • Relational filter was not correctly refreshed

  • Fixed issue with wrong default value for siren.filterjoin.cache.size

  • Default join settings were incorrect

  • Improved datasourcetype conditions

  • Improves Kibi session management

  • No error shown when the key has wrong length

  • Label on Entity Clipboard missing

  • Correct permission on executable files

  • Check that siren-join is installed on all data nodes

  • Update of node-jdbc to jdbc@0.3.1 and sqlite to sqlite3@3.1.4

  • ES client nodes discovery makes the gremlin server not working [Enterprise Edition only]

  • null pointer exception in gremlin server [Enterprise Edition only]

  • id instead of label on the graph [Enterprise Edition only]

  • 500 - [Object object] error [Enterprise Edition only]

  • Incorectly parsed remote_address [Enterprise Edition only]

  • Add more graph scripts [Enterprise Edition only]

  • Saving termsEncoding bloom [Enterprise Edition only]

Kibana Changes

  • Issue 6420: Bump node to v4.3.2

  • Issue 6353: Add basePath to short URLs

  • Issue 6228: HTML unsafe characters in field names no longer break Kibana

  • Issue 6083: Plugin installer support for .tgz file types

  • Issue 5971: Fix active search source hover background issue

  • Issue 5942: Fix for save button disabled when opening spy panel

  • Issue 6133 and 6103: Distro packages now have a description, vendor, maintainer, url, license and priority metadata

Kibi 4.4.1-1

Kibi Changes

  • Fixed a bug - in community distribution the default join settings were incorrect

  • Fixed a bug - relational filter was not correctly refreshed

Kibi 4.4.1

Kibi Changes

  • Various bug fixes and stability improvements.

  • Various error handling improvements

  • Caching can now be enabled/disabled per datasource from UI configuration

  • New version of Kibi Timeline plugin 0.1.4

  • Improved kibi.bat file for Windows

  • Improved documentation

  • Improved quality of the demo dataset

  • Default terms encoding for Siren Join changed to long

  • New Advanced Join Settings plugin [Enterprise Edition only]

  • Full Shield integration [Enterprise Edition only]

  • New Graph Browser visualization [Enterprise Edition only]

  • New Kibi Thinkerpop3 datasource [Enterprise Edition only]

  • New Kibi Gremlin Server component [Enterprise Edition only]

  • New Ansible/Vagrant deployment scripts for GCE and AWS [Enterprise Edition only]

Kibi 0.3.2

Kibi Changes

  • Various bug fixes and stability improvements.

  • New version of kibi_timeline_vis 0.1.2

  • Siren join plugin version upgraded to 2.2.0-1

Kibi 0.3.1 and Kibana 4.4.1

Kibi Changes

  • Various bug fixes and stability improvements.

  • Siren join plugin version upgraded to 2.2.0.

Kibana Changes

  • Bump node.js to 0.12.10 from 0.12.9

  • Issue 6185: Fixes a bug where the active HTTP spinner in the chrome bar is gone

Kibi 0.3 and Kibana 4.4

Kibi Changes

  • Using event times to create index names is no longer supported as of this release. Current versions of Elasticsearch include sophisticated date parsing APIs that Kibana uses to determine date information, removing the need to specify dates in the index pattern name.

Kibana Changes

Enhancements

  • Issue 1362: Color palette selector added.

  • Issue 1553: Kibana can shorten URLs for shared or embedded items.

  • Issue 5733: Time-based index pattern expansion can be set at index pattern creation time.

  • Issue 5775: Adds a configuration option to change the maximum payload size sent to the server.

  • Issue 4966: Logo is now in SVG format.

  • Issue 3625: Downloaded visualizations now use the visualization name as the filename.

  • Issue 5279: Large strings are truncated with an ellipsis (…​).

  • Issue 5241: Truncated visualization names are displayed in full as tooltips.