Setting up Siren Investigate

This section includes information on how to set up Siren Investigate and get it running, including:

  • Supported platforms and version compatibility

  • Installing

  • Configuring

  • Upgrading

The instructions in this section are relevant for production deployments only. For a quick trial of Siren, download the Easy Start version and follow the instructions in the README file.

Supported platforms

Packages of Siren Investigate are provided for and tested with Windows, Linux, and macOS operating systems.

Because Siren Investigate runs on Node.js, the required Node.js binaries for these platforms are included. Running Siren Investigate against a separately maintained version of Node.js is not supported.

Siren Platform is supported on the following Web browsers:

  • Google Chrome

  • Mozilla Firefox

Siren Platform version compatibility

Each Siren Investigate version supports different Elasticsearch/Siren Federate versions in the following way:

  • Minimum supported version: Includes the set of features that are compatible with the Elasticsearch/Siren Federate version that is used.

  • Feature complete version: Includes all of the current Siren Investigate features.

  • Target version: Includes all of the current Siren Investigate features and, being the most up-to-date version, is recommended in most cases.

Version compatibility matrix

APIs are at risk of breaking if your Elasticsearch or Siren Federate back-end system is a more recent release than the Siren Investigate version. To reduce this risk, ensure that the difference is limited to only a minor release from the target version; that is, the second number in the sequence <major_release>.<minor_release>.<patch_release>.

Table 1. Version compatibility matrix
Siren Investigate Siren Federate Elasticsearch

13.5

Target: 8.8.2-32.0
Feature complete: 7.10.2-22.5
Minimum: 7.10.2-22.1

8.8.2
8.7.1
8.5.3
8.4.3
7.17.15
7.17.12
7.17.10
7.17.9
7.16.3
7.13.4
7.10.2

13.4

Target: 8.8.2-32.0
Feature complete: 7.10.2-22.5
Minimum: 7.10.2-22.1

8.8.2
8.7.1
8.5.3
8.4.3
7.17.9
7.16.3
7.13.4
7.10.2

13.3

Target: 8.8.2-32.0
Feature complete: 7.10.2-22.5
Minimum: 7.10.2-22.1

8.8.2
8.7.1
8.5.3
8.4.3
7.17.9
7.16.3
7.13.4
7.10.2

13.2

Target: 8.7.1-31.1
Feature complete: 7.10.2-22.5
Minimum: 7.10.2-22.1

8.7.1
8.5.3
8.4.3
7.17.9
7.16.3
7.13.4
7.10.2

12.1

Target: 7.17.7-29.1
Feature complete: 6.8.0-10.3.0
Minimum: 6.8.0-10.3.0

7.17.7
7.10.2
7.9.2
7.9.1
7.8.0
7.7.1
7.6.2
7.3.2
6.8.12
6.8.6
6.8.2
6.8.0

When using Elasticsearch 8 or later, you must use xpack security to secure your installation. Searchguard security is supported only in Elasticsearch version 7.x or earlier.

Installing Siren Investigate

Siren Investigate is provided in the following package formats:

zip

The zip packages are provided for installation on Windows, Linux and macOS operating systems.

docker

Siren Investigate Docker images are available at the sirensolutions Dockerhub page.

Prerequisites

Ensure that you have obtained a license for Siren Platform on the Siren Trial Page.

On Windows, you must install a modern ZIP manager such as 7-zip to extract the platform archives: Windows' built-in ZIP support cannot process the large number of files in the package.

Installing Siren Investigate with the ZIP package

Siren Investigate is provided for Windows, Linux, and macOS as a .zip package. These packages are the easiest formats to use when you are trying Siren Investigate for the first time.

The latest version of Siren Investigate can be found on the Siren Download Page. Descriptions of the separate demonstration packages are also available.

Installing on Linux or macOS

You can download the Linux or macOS (Darwin) .zip packages from the Siren Download Page.

Save the compressed file and extract it to a local directory.

If you are running macOS Catalina or a later version, you must remove an extended file attribute (xattr) before you extract the downloaded ZIP file. Run the following command:

xattr -d com.apple.quarantine siren-platform-*-darwin-x86_64.zip
Running Siren Investigate from the Terminal

Start Siren Investigate by opening a Terminal window and running the following command:

./bin/investigate

By default, Siren Investigate runs in the foreground, prints its logs to STDOUT, and can be stopped by pressing Ctrl + C. The installation is complete when you see the message “[plugin:graph_browser_vis] Status changed from yellow to green - Initialized” in the log window.

After the installation is complete, open a Web browser and navigate to http://localhost:5606.

Folder layout of the ZIP archive

The .zip packages are entirely self-contained. This is very convenient because you do not have to create any directories to start using Siren Investigate, and uninstalling is as easy as removing the folder.

However, it is advisable to change the default locations of the configuration and data folders so that you do not remove important data later on.

Type Description Default Location Setting

home

Siren Investigate home folder or $INVESTIGATE_HOME

Folder created by unpacking the archive; in demonstration distributions, the folder is siren-investigate.

bin

Binary scripts including kibi to start the Siren Investigate server and kibi-plugin to install plugins

$INVESTIGATE_HOME\bin

config

Configuration files including investigate.yml

$INVESTIGATE_HOME\config

data

The location of the data files written to disk by Siren Investigate and its plugins

$INVESTIGATE_HOME\data

optimize

Transpiled source code. Certain administrative actions, for example plugin install, result in the source code being retranspiled on the fly.

$INVESTIGATE_HOME\optimize

plugins

The location of the plugin files. Each plugin will be contained in a subfolder.

$INVESTIGATE_HOME\plugins

Installing Siren Investigate on Windows

Download the Windows .zip package from the Siren Download Page.

Save the compressed file and extract it to a local directory.

Running Siren Investigate from the command prompt

Start Siren Investigate by opening a command prompt and running the following command:

.\bin\start.bat

You can also run the installation by simply double-clicking on the start.bat file.

By default, Siren Investigate runs in the foreground, prints its logs to STDOUT, and can be stopped by pressing Ctrl + C. The installation is complete when you see the message “[plugin:graph_browser_vis] Status changed from yellow to green - Initialized” in the log window.

After the installation is complete, open a Web browser and navigate to http://localhost:5606.

Folder layout of Windows ZIP archive

The .zip package is entirely self-contained.

This is very convenient because you do not have to create any directories to start using Siren Investigate, and uninstalling Siren Investigate is as easy as removing the folder. However, it is advisable to change the default locations of the configuration and data folders so that you do not remove important data later on.

Type Description Default Location Setting

home

Siren Investigate home folder or %INVESTIGATE_HOME%

Folder created by unpacking the archive; in demonstration distributions, the folder is kibi.

bin

Binary scripts including kibi to start the Siren Investigate server and kibi-plugin to install plugins

%INVESTIGATE_HOME%\bin

config

Configuration files including investigate.yml

%INVESTIGATE_HOME%\config

data

The location of the data files written to disk by Siren Investigate and its plugins

%INVESTIGATE_HOME%\data

optimize

Transpiled source code. Certain administrative actions, for example plugin install, result in the source code being retranspiled on the fly.

%INVESTIGATE_HOME%\optimize

plugins

The location of the plugin files. Each plugin will be contained in a subfolder.

%INVESTIGATE_HOME%\plugins

Running Siren Investigate on Docker

Docker images for Siren Investigate are available from the sirensolutions organization on Dockerhub.

Pulling the Image

Obtaining Siren Investigate for Docker is as simple as issuing a docker pull command.

Option 1: To retrieve the Docker image for the latest version of Siren Investigate, run the following command:

docker pull sirensolutions/siren-platform:latest
docker run -d -p 5606:5606 -p 9220:9220 sirensolutions/siren-platform:latest

Option 2: To retrieve a specific version of Siren Investigate, such as version 10.0.0, run the following command:

docker pull sirensolutions/siren-platform:10.0.0
docker run -d -p 5606:5606 -p 9220:9220 sirensolutions/siren-platform:10.0.0

Option 3: To retrieve an image that is pre-populated with demonstration data, run the following command:

docker pull sirensolutions/siren-platform-demo-data:latest
docker run -d -p 5606:5606 -p 9220:9220 sirensolutions/siren-platform-demo-data:latest

Environment variable configuration

Under Docker, Siren Investigate can be configured by using environment variables. When the container starts, a helper process checks the environment for variables that can be mapped to Siren Investigate command-line arguments.

For compatibility with container orchestration systems, these environment variables are written in all capitals, with underscores as word separators. The helper translates these names to valid Siren Investigate setting names.

Some example translations are provided as follows:

Table 2. Example Docker Environment Variables

Environment Variable

Siren Investigate Setting

SERVER_NAME

server.name

KIBANA_DEFAULTAPPID

kibana.defaultAppId

XPACK_MONITORING_ENABLED

xpack.monitoring.enabled

In general, any setting listed in Configuring Siren Investigate can be configured with this technique.

These variables can be set by using docker-compose as follows:

services:
  investigate:
    image: sirensolutions/siren-investigate:latest
    environment:
      SERVER_NAME: siren.example.org
      ELASTICSEARCH_URL: http://elasticsearch.example.org

Because environment variables are translated to CLI arguments, they take precedence over settings that are configured in the investigate.yml file.

Docker defaults

The default settings when you are using the stand-alone siren-investigate image are as follows:

elasticsearch.url

http://localhost:9220

server.basepath

""

kibana.index

.siren

Upgrading Siren Investigate

Before you upgrade Siren Investigate, read the breaking changes section of the release notes to learn about the potential effects of upgrading.

To upgrade Siren Investigate, follow these steps:

  1. Save a copy of the Siren Investigate configuration file: config/investigate.yml.

  2. Save a copy of any certificate or key that is referenced by config/investigate.yml; by convention, these files are stored in a directory named pki.

  3. Create a back-up of the saved objects index or mappings (for example, .siren) and the security index or mappings (for example, .sirenaccess) by running the following command:

    bin/investigate backup --backup-dir=DIR

  4. In the new version of Siren Investigate, overwrite the investigate.yml file in the config folder with the one that you saved in step 1.

  5. Restore the files or directories that you saved in step 2 by copying them to the same relative locations in the new installation directory.

  6. In the /plugins folder, install the compatible versions of any third-party Siren Investigate plug-ins that you need.

  7. Run the following upgrade command:

    bin/investigate upgrade

To see the options that you can use with the backup script, enter the command bin/investigate backup --help.

Fixing the missing indices warning when upgrading from 12.0.x or earlier

Upgrades from 12.0.x or earlier can sometimes cause the missing indices warning; missing-pattern-added-by-migration-manual-intervention-required. From 12.1 the indexPattern object is merged into the search object. During migration, if an indexPattern for the search is missing, a mock indexPattern object is added to the search with the Elasticsearch index missing-pattern-added-by-migration-manual-intervention-required.

To fix this warning, follow these steps:

  1. Go to search saved object edit page, Management > Saved Objects > Entity tables and searches.

  2. Update or delete each search with the pattern missing-pattern-added-by-migration-manual-intervention-required.

    • To update: go to each search, update its indexPattern properties and save the search. For example, update indexPattern.pattern property with an existing valid index.

    • To delete: go to each search and click Delete Entity Table and Search.

  3. When you have corrected the Elasticsearch index for all Entity Tables, open the data model. If the newly assigned indexPattern.pattern points to an Elasticsearch index with a fields mapping, it automatically assigns the indexPattern.fields property.

If there are more searches in the installation, you can use the following query from dev tools to find the searches.

GET .siren/_search
{
  "query": {
    "match": {
      "search.indexPattern.pattern": "missing-pattern-added-by-migration-manual-intervention-required"
    }
  }
}

Restoring saved object and security indices

It is possible to restore the indices from a previous backup. See below for a description of the structure and legacy backup types.

To see the options that you can use with the restore script, enter the command bin/investigate restore --help.

Structure type backups

If the backup was created after version 12.0.0, it will be a structure type which has the following prefixes:

Prefix

Opening

config-data-<saved-object-index>json

Saved object data file

config-mapping-<saved-object-index>.json

Saved object mapping file

acl-data-<security-index>.json

Security index mapping file

acl-mapping-<security-index>.json

Security index data file

The name of the saved object index or the security index in the file is not used. Instead, the structure type restores to the saved object index and security index that are specified in config/investigate.yml.

If the indices are not explicitly specified in the investigate.yml file, .siren and .sirenaccess will be the default saved object index and security index.

Legacy type backups

If the backup was created before version 12.0.0, the legacy type backup restores to the saved object index and security index that are specified in the backup files. Ensure that the files are named as follows:

Prefix

Opening

data-<saved-object-index>json

Saved object data file

mapping-<saved-object-index>.json

Saved object mapping file

data-<security-index>.json

Security index mapping file

mapping-<security-index>.json

Security index data file

saved-object-index: The value of kibana.index in the config/investigate.yml file when the backup was created. Defaults to .siren. security-index: The value of investigate_access_control.acl.index when the backup was created. Defaults to .sirenaccess.

Configuring Siren Investigate

The Siren Investigate server reads properties from the investigate.yml file on startup. The default settings configure Siren Investigate to run on localhost:5606. To change the host or port number, or connect to Elasticsearch running on a different machine, you must update your investigate.yml file. You can also enable SSL and set a variety of other options.

External datasource configuration is documented in Working with JDBC datasources and access control configuration is documented in Search Guard Classic integration and Siren Investigate access control.

Environment Variable Placeholders

It is possible to use environment variable placeholders in configuration settings. The syntax of placeholders is ${ENV_VARIABLE_NAME}.

For example, to set elasticsearch.url to the value of the environment variable ES_URL, edit config/investigate.yml as follows:

elasticsearch.url: ${ES_URL}
Table 3. Configuration Settings
Setting Description

server.port:

Default: 5606. Siren Investigate is served by a back end server. This setting specifies the port to use.

server.host:

Default: localhost. This setting specifies the host of the back end server.

server.basePath:

Enables you to specify a path to mount Siren Investigate as if you are running behind a proxy. This only affects the URLs generated by Siren Investigate, your proxy is expected to remove the basePath value before forwarding requests to Siren Investigate. This setting cannot end in a slash (/).

server.maxPayloadBytes:

Default: 1048576. The maximum payload size in bytes for incoming server requests.

server.name:

Default: your-hostname. A human-readable display name that identifies this Siren Investigate instance.

server.defaultRoute:

Default: /app/kibana. This setting specifies the default route when opening Siren Investigate. You can use this setting to modify the landing page when opening Siren Investigate.

elasticsearch.url:

Default: http://localhost:9220. The URL of the Elasticsearch instance to use for all your queries.

elasticsearch.preserveHost:

Default: true. When this setting’s value is true Siren Investigate uses the hostname specified in the server.host setting. When the value of this setting is false, Siren Investigate uses the hostname of the host that connects to this Siren Investigate instance.

kibana.index:

Default: .siren. Siren Investigate uses an index in Elasticsearch to store saved searches, visualizations and dashboards. Siren Investigate creates a new index if the index does not already exist.

kibana.defaultAppId:

Default: discover. The default application to load.

tilemap

The tilemap is the base layer the map visualizations. To increase max zoom level for mapping visualizations, copy the following object in the `investigate.yml`file, this will increase the zoom level to 19 from the default of 15.

tilemap:
  url: "https://\{s}.tile.openstreetmap.org/\{z}/\{x}/\{y}.png"
  options:
    attribution: '&copy;
[OpenStreetMap("http://www.openstreetmap.org/copyright")'
    subdomains: ['a']
    minZoom: 0
    maxZoom: 19

tilemap.url:

The URL to the tile service that Siren Investigate uses to display map tiles in tilemap visualizations. By default, Siren Investigate reads this URL from an external metadata service, but users can still override this parameter to use their own Tile Map Service.

tilemap.options.minZoom:

Default: 1. The minimum zoom level.

tilemap.options.maxZoom:

Default: 15. The maximum zoom level.

tilemap.options.attribution:

Default: © [OpenStreetMap]("http://www.openstreetmap.org/copyright"). The map attribution string.

tilemap.options.subdomains:

An array of subdomains used by the tile service. Specify the position of the subdomain in the URL with the token {s}.

regionmap

Specifies additional vector layers for use in Region Map visualizations. Each layer object points to an external vector file that contains a GeoJSON FeatureCollection. The file must use the WGS84 coordinate reference system and only include polygons. If the file is hosted on a separate domain from Siren Investigate, the server needs to be CORS (Cross Origin Request Security) enabled so that Siren Investigate can download the file. The following example shows a valid regionmap configuration.

regionmap:
  layers:
    - name: "Departments of France"
      url: "http://my.cors.enabled.server.org/france_departements.geojson"
      attribution: "INRAP"
      fields:
        - name: "department"
          description: "Full department name"
        - name: "INSEE"
          description: "INSEE numeric identifier"

It is possible to serve files required by regionmap visualizations from a local folder. To serve files on a local URL, use the private_assets_dir configuration parameter in investigate.yml. For more information, see Serving private assets.

regionmap.layers.name:

Mandatory. A description of the map being provided.

regionmap.layers.url:

Mandatory. The location of the GeoJSON file as provided by a web server.

regionmap.layers.attribution:

Optional. References the originating source of the GeoJSON file.

regionmap.layers.fields:

Mandatory. Each layer can contain multiple fields to indicate what properties from the geojson features you want to expose. This example shows how to define multiple properties.

regionmap.layers.fields.name:

Mandatory. This value is used to do an inner-join between the document stored in Elasticsearch and the GeoJSON file. For example, if the field in the GeoJSON is called Location and has city names, there must be a field in Elasticsearch that holds the same values that Siren Investigate can then use to lookup for the GeoShape data.

regionmap.layers.fields.description:

Mandatory. The human readable text that is shown under the Options tab when building the Region Map visualization.

elasticsearch.username: and elasticsearch.password:

If your Elasticsearch is protected with basic authentication, these settings provide the username and password that the Siren Investigate server uses to perform maintenance on the Siren Investigate index at startup. Your Siren Investigate users still need to authenticate with Elasticsearch, which is proxied through the Siren Investigate server.

server.customResponseHeaders

Default: {}. Allows you to specify custom response headers that will be added to HTTP responses from Siren Investigate.

For example, a Content-Security-Policy header can be set by adding the following snippet to the investigate.yml configuration file:

server.customResponseHeaders:
  "Content-Security-Policy": "frame-ancestors 'self'"

server.ssl.enabled

Default: false. Enables SSL for incoming requests from the browser to Siren Investigate. When set to true, server.ssl.certificate and server.ssl.key are required

server.ssl.certificate: and server.ssl.key:

Paths to the PEM-format SSL certificate and SSL key files, respectively.

server.ssl.keyPassphrase:

The passphrase that will be used to decrypt the private key. This value is optional as the key may not be encrypted.

server.ssl.certificateAuthorities:

List of paths to PEM encoded certificate files that should be trusted.

server.ssl.supportedProtocols:

Default: TLSv1.2, TLSv1.3. Supported protocols with versions. Valid protocols: TLSv1.2, TLSv1.3

server.ssl.cipherSuites:

Default: 'TLS_AES_256_GCM_SHA384','TLS_CHACHA20_POLY1305_SHA256','TLS_AES_128_GCM_SHA256',ECDHE-RSA-AES128-GCM-SHA256, ECDHE-ECDSA-AES128-GCM-SHA256, ECDHE-RSA-AES256-GCM-SHA384, ECDHE-ECDSA-AES256-GCM-SHA384, DHE-RSA-AES128-GCM-SHA256, ECDHE-RSA-AES128-SHA256, DHE-RSA-AES128-SHA256, ECDHE-RSA-AES256-SHA384, DHE-RSA-AES256-SHA384, ECDHE-RSA-AES256-SHA256, DHE-RSA-AES256-SHA256, HIGH,!aNULL, !eNULL, !EXPORT, !DES, !RC4, !MD5, !PSK, !SRP, !CAMELLIA.

For more information on the format, and valid options, refer to the OpenSSL cipher list format documentation (https://www.openssl.org/docs/man1.0.2/apps/ciphers.html#CIPHER-LIST-FORMAT).

elasticsearch.ssl.certificate: and elasticsearch.ssl.key:

Optional settings that provide the paths to the PEM-format SSL certificate and key files. These files are used to verify the identity of Siren Investigate to Elasticsearch and are required when xpack.ssl.verification_mode in Elasticsearch is set to either certificate or full.

elasticsearch.ssl.keyPassphrase:

The passphrase that will be used to decrypt the private key. This value is optional as the key may not be encrypted.

elasticsearch.ssl.certificateAuthorities:

Optional setting that enables you to specify a list of paths to the PEM file for the certificate authority for your Elasticsearch instance.

elasticsearch.ssl.verificationMode:

Default: full. Controls the verification of certificates presented by Elasticsearch. Valid values are none, certificate, and full. full performs hostname verification, and certificate does not.

elasticsearch.pingTimeout:

Default: the value of the elasticsearch.requestTimeout setting. Time in milliseconds to wait for Elasticsearch to respond to pings.

elasticsearch.requestTimeout:

Default: 30000. Time in milliseconds to wait for responses from the back end or Elasticsearch. This value must be a positive integer.

elasticsearch.requestHeadersWhitelist:

Default: [ 'authorization' ]. List of Siren Investigate client-side headers to send to Elasticsearch. To send no client-side headers, set this value to [] (an empty list).

elasticsearch.customHeaders:

Default: {}. Header names and values to send to Elasticsearch. Any custom headers cannot be overwritten by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.

elasticsearch.shardTimeout:

Default: 0. Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to switch off.

elasticsearch.startupTimeout:

Default: 5000. Time in milliseconds to wait for Elasticsearch at Siren Investigate startup before retrying.

pid.file:

Specifies the path where Siren Investigate creates the process ID file.

path.data:

Default: ./data. The path where Siren Investigate stores persistent data not saved in Elasticsearch

logging.dest:

Default: stdout. Enables you specify a file where Siren Investigate stores log output.

logging.silent:

Default: false. Set the value of this setting to true to suppress all logging output.

logging.quiet:

Default: false. Set the value of this setting to true to suppress all logging output other than error messages.

logging.verbose:

Default: false. Set the value of this setting to true to log all events, including system usage information and all requests.

ops.interval:

Default: 5000. Set the interval in milliseconds to sample system and process performance metrics. The minimum value is 100.

status.allowAnonymous:

Default: false. If authentication is enabled, setting this to true allows unauthenticated users to access the Siren Investigate server status API and status page.

cpu.cgroup.path.override:

Override for cgroup cpu path when mounted in manner that is inconsistent with /proc/self/cgroup

cpuacct.cgroup.path.override:

Override for cgroup cpuacct path when mounted in manner that is inconsistent with /proc/self/cgroup

console.enabled

Default: true. Set to false to disable Console. Toggling this will cause the server to regenerate assets on the next startup, which may cause a delay before pages start being served.

ingest.job_refresh_interval

Default: 5000. The interval at which the Data Import application will refresh the list of running jobs.

investigate_core.private_assets_dir

An absolute or relative path to a folder. All files from this folder are served at HOST/ui/private_assets URL. For more information, see Serving private assets.

investigate_core.search.max_buckets

Default: 1000. Limits the maximum number of buckets returned to the frontend from an aggregation request. When the number of buckets is bigger than the limit an error is returned. This is to prevent the browser from freezing if the aggregation is misconfigured and is producing too many buckets. This usually happens on time based aggregations where the time interval is too small.

investigate_core.client_side_cache_size

Default: 5000. Limits the maximum number of objects in client side cache. Maximum allowed value 10000.

Accessing Siren Investigate

Siren Investigate is a web application that you access through port 5606. All you need to do is point your web browser at the machine where Siren Investigate is running and specify the port number. For example, http://localhost:5606 or http://example.com:5606.

When you access Siren Investigate, the Discover page loads by default with the default index pattern selected. The time filter is set to the last 15 minutes and the search query is set to match-all (*).

If you do not see any documents, try setting the time filter to a wider time range. If you still do not see any results, it is possible that you do not have any documents.

Checking Siren Investigate status

You can reach the Siren Investigate server’s status page by navigating to http://localhost:5606/status. The status page displays information about the server’s resource usage and lists the installed plugins.

Siren Investigate status page.

Collecting Elasticsearch diagnostics

Click Elasticsearch diagnostics to generate a single file by collecting different metrics about your Elasticsearch cluster. All collected information are saved to a local file and never transferred over a network. You can see a full list of Elasticsearch API calls by clicking the more info (i) icon.

Siren Investigate status page diagnostics help.

For JSON formatted server status details, use the API endpoint at http://localhost:5606/api/status.

Connecting Siren Investigate to backend datasources

Siren can visualize data that is accessible by the Siren backend, which is an Elasticsearch cluster enhanced by the Federate plugin.

The following diagram represents this concept. On the frontend, a user looks at data, for example data in a dashboard or on the Graph Link Analysis system. This data comes from Siren "searches", which reflect queries that are sent to indices that are on the Elasticsearch backend.

It is useful to distinguish between two types of searches:

  • Entity tables: Root definitions that describe which Elasticsearch indices will be used. They can be as simple as the name of the index itself or can be “patterns” e.g. new*_ to indicate “all the indices that begin with news_”. For more information and examples, see Entity tables.

  • Other searches: These are defined by filtering an Index Pattern Search, for example starting from a “News” entity table, one might create a “News that contain the word Brexit” search.

Before you can start using Siren Investigate, you need to tell it which Elasticsearch indices you want to explore.

The first time you access Siren Investigate, you are prompted to define an entity_table that matches the name of one or more of your indices. That is all you need to configure to start using Siren Investigate. You can create entity tables at any time from the Management tab.

By default, Siren Investigate connects to the Elasticsearch instance running on localhost. To connect to a different Elasticsearch instance, modify the Elasticsearch URL in the investigate.yml configuration file and restart Siren Investigate. For information about using Siren Investigate with your production nodes, see Using Siren Investigate in a production environment
  1. Point your browser at port 5606 to access the Siren Investigate user interface. For example, http://localhost:5606 or \http://example.com:5606.

  2. Specify an pattern that matches the name of one or more of your Elasticsearch indices. You may have to access the data model in the Data Model section. By default, Siren Investigate guesses that you are working with data being fed into Elasticsearch by Logstash. If that’s the case, you can use the default logstash-* as your entity table. The asterisk (*) matches zero or more characters in an index’s name. If your Elasticsearch indices follow some other naming convention, enter an appropriate pattern. The "pattern" can also be the name of a single index.

    image

  3. Select the index field that contains the timestamp that you want to use to perform time-based comparisons. Siren Investigate reads the index mapping to list all the fields that contain a timestamp. If your index does not have time-based data, switch off the  Index contains time-based events option.

  4. Click Create to add the entity table.

All done. Siren Investigate is now connected to your Elasticsearch data. Siren Investigate displays a read-only list of fields configured for the matching index.

Siren Investigate relies on dynamic mapping to use fields in visualizations and manage the .siren index. If you have switched off dynamic mapping, you need to manually provide mappings for the fields that Siren Investigate uses to create visualizations. For more information, see Siren Investigate and Elasticsearch dynamic mapping

Start exploring your data

You are ready to dive in to your data:

  • Search and browse your data interactively from the Discover page.

  • Chart and map your data from the Visualizations page.

  • Create and view custom dashboards from the Dashboard page.

For a step-by-step introduction to these core concepts, see the Easy Start tutorial.

Siren Investigate and Elasticsearch dynamic mapping

By default, Elasticsearch enables dynamic mapping for fields. Siren Investigate needs dynamic mapping to use fields in visualizations correctly, as well as to manage the .siren index where saved searches, visualizations, and dashboards are stored.

If your Elasticsearch use case requires you to switch off dynamic mapping, you need to manually provide mappings for fields that Siren Investigate uses to create visualizations. You also need to manually enable dynamic mapping for the .siren index.

The following procedure assumes that the .siren index does not already exist in Elasticsearch and that the index.mapper.dynamic setting in elasticsearch.yml is set to false:

  1. Start Elasticsearch.

  2. Create the .siren index with dynamic mapping enabled just for that index:

    PUT .siren
    {
      "index.mapper.dynamic": true
    }
  3. Start Siren Investigate and navigate to the web UI and verify that there are no error messages related to dynamic mapping.

Using Siren Investigate in a production environment

How you deploy Siren Investigate largely depends on your use case. If you are the only user, you can run it on your local machine and configure it to point to whatever Elasticsearch instance you want to interact with. Conversely, if you have a large number of heavy users, you may need to load balance across multiple instances that are all connected to the same Elasticsearch cluster.

While Siren Investigate is not terribly resource intensive, we still recommend running Siren Investigate separate from your Elasticsearch data or master nodes.

Running Siren Investigate in a dedicated node will require at least 5GB of available memory to allow code optimization on initial startup.

Make sure to set the configuration file as readable only to the user running the Siren Investigate process as it contains encryption keys to protect configuration settings stored in Elasticsearch; if you are connecting Siren Investigate to external datasources, we also recommend to use credentials with no write privileges as they are not required by the system.

Access control

Siren Investigate is compatible with Elastic Stack Security and Search Guard Classic to enable index and document level access control.

For more information about access control features, see the Authentication and access control section.

Configuring a reverse proxy

To protect your system from protocol attacks, it is recommended that you run Siren Investigate behind a reverse proxy.

HTTP servers, such as Apache and NGINX, offer the reverse proxy functionality. See the Enabling HTTPS section for instructions.

Alternatively, you can use a proprietary solution, such as a web application firewall (WAF). Please consult the vendor documentation for instructions.

Load balancing across multiple Elasticsearch nodes

If you have multiple nodes in your Elasticsearch cluster, the easiest way to distribute Siren Investigate requests across the nodes is to run an Elasticsearch client node on the same machine as Siren Investigate. Elasticsearch client nodes are essentially smart load balancers that are part of the cluster. They process incoming HTTP requests, redirect operations to the other nodes in the cluster as needed, and gather and return the results. For more information, see Node in the Elasticsearch reference.

To use a local client node to load balance Siren Investigate requests:

  1. Install Elasticsearch on the same machine as Siren Investigate.

  2. Configure the node as a client node. In elasticsearch.yml, set both node.data and node.master to false:

    #    You want this node to be neither master nor data node, but
    #    to act as a "search load balancer" (fetching data from nodes,
    #    aggregating results, and so on)
    #
    node.master: false
    node.data: false
    node.ingest: false
  3. Configure the client node to join your Elasticsearch cluster. In elasticsearch.yml, set the cluster.name to the name of your cluster.

    cluster.name: "my_cluster"
  4. Make sure Siren Investigate is configured to point to your local client node. In investigate.yml, the elasticsearch.url should be set to http://localhost:9220.

    # The Elasticsearch instance to use for all your queries.
    elasticsearch.url: "http://localhost:9220"