Siren Platform User Guide

Connecting Siren Investigate to backend datasources

Siren can visualize data that is accessible by the Siren backend, which is an Elasticsearch cluster enhanced by the Federate plugin.

The following diagram represents this concept. On the frontend, a user looks at data, for example data in a dashboard or on the Graph Link Analysis system. This data comes from Siren "searches", which reflect queries that are sent to indices that are on the Elasticsearch backend.

It is useful to distinguish between two types of searches:

  • Index Pattern Searches: Root definitions that describe which Elasticsearch indices will be used. They can be as simple as the name of the index itself or can be “patterns” e.g. new_* to indicate “all the indices that begin with news_”. For more information and examples, see Index Pattern Searches.

  • Other searches: These are defined by filtering an Index Pattern Search, for example starting from a “News” Index Pattern Search, one might create a “News that contain the word Brexit” search.

Inside the main Elasticsearch cluster (the one that the NodeJS Siren application is connected to), indices can be of two types:

  • Physical indices: These are regular Elasticsearch indices, typically created via external ETL (e.g. Logstash) or via the Siren reflection process (an ETL that keeps external tables in sync). For more information, see Data Reflection.

  • Virtual indices: These will simulate Elasticsearch indices but will not copy data; instead, they will send queries to the remote backends and will translate the query results.

Both physical and virtual indices can be used to create index patterns (although in Siren 10.3, you cannot create wildcard patterns (*) including virtual indices).

searches_indices.png

Before you can start using Siren Investigate, you need to tell it which Elasticsearch indices you want to explore. The first time you access Siren Investigate, you are prompted to define an index pattern search that matches the name of one or more of your indices. That is it. That is all you need to configure to start using Siren Investigate. You can create index pattern searches at any time from the Management tab.

Tip

By default, Siren Investigate connects to the Elasticsearch instance running on localhost. To connect to a different Elasticsearch instance, modify the Elasticsearch URL in the investigate.yml configuration file and restart Siren Investigate. For information about using Siren Investigate with your production nodes, see Using Siren Investigate in a production environment.

  1. Point your browser at port 5606 to access the Siren Investigate user interface. For example, http://localhost:5606 or http://YOURDOMAIN.com:5606.

  2. Specify an index pattern search that matches the name of one or more of your Elasticsearch indices. You may have to access the index pattern search management in the Management tab. By default, Siren Investigate guesses that you are working with data being fed into Elasticsearch by Logstash. If that’s the case, you can use the default logstash-* as your index pattern search. The asterisk (*) matches zero or more characters in an index’s name. If your Elasticsearch indices follow some other naming convention, enter an appropriate pattern. The "pattern" can also be the name of a single index.

    Start-Page.png
  3. Select the index field that contains the timestamp that you want to use to perform time-based comparisons. Siren Investigate reads the index mapping to list all the fields that contain a timestamp. If your index does not have time-based data, switch off the  Index contains time-based events option.

  4. Click Create to add the index pattern search. This first pattern is automatically configured as the default. When you have more than one index pattern search, you can designate which one to use as the default by clicking on the star icon above the index pattern search title from ManagementData Model.

All done. Siren Investigate is now connected to your Elasticsearch data. Siren Investigate displays a read-only list of fields configured for the matching index.

Note

Siren Investigate relies on dynamic mapping to use fields in visualizations and manage the .siren index. If you have switched off dynamic mapping, you need to manually provide mappings for the fields that Siren Investigate uses to create visualizations. For more information, see Siren Investigate and Elasticsearch Dynamic Mapping.

Start exploring your data

You are ready to dive in to your data:

  • Search and browse your data interactively from the Discover page.
  • Chart and map your data from the Visualizations page.
  • Create and view custom dashboards from the Dashboard page.

For a step-by-step introduction to these core Siren Investigate concepts, see the Getting Started tutorial.

Siren Investigate and Elasticsearch dynamic mapping

By default, Elasticsearch enables dynamic mapping for fields. Siren Investigate needs dynamic mapping to use fields in visualizations correctly, as well as to manage the .siren index where saved searches, visualizations, and dashboards are stored.

If your Elasticsearch use case requires you to switch off dynamic mapping, you need to manually provide mappings for fields that Siren Investigate uses to create visualizations. You also need to manually enable dynamic mapping for the .siren index.

The following procedure assumes that the .siren index does not already exist in Elasticsearch and that the index.mapper.dynamic setting in elasticsearch.yml is set to false:

  1. Start Elasticsearch.
  2. Create the .siren index with dynamic mapping enabled just for that index:

    PUT .siren
    {
      "index.mapper.dynamic": true
    }
  3. Start Siren Investigate and navigate to the web UI and verify that there are no error messages related to dynamic mapping.