Siren Platform User Guide

Index pattern searches

To use Siren Investigate, you have to tell it about the Elasticsearch indices that you want to explore by configuring one or more index pattern searches. You can also:

  • Create scripted fields that are computed on the fly from your data. You can browse and visualize scripted fields, but you cannot search them.
  • Set advanced options such as the number of rows to show in a table and how many of the most popular fields to show. Use caution when modifying advanced options, as it is possible to set values that are incompatible with one another.
  • Configure Siren Investigate for a production environment

Creating an index pattern search to connect to Elasticsearch

An index pattern search identifies one or more Elasticsearch indices that you want to explore with Siren Investigate. Siren Investigate looks for index names that match the specified pattern. An asterisk (*) in the pattern matches zero or more characters. For example, the pattern myindex-* matches all indices whose names start with myindex-, such as myindex-1 and myindex-2.

An index pattern search can also be the name of a single index.

To create an index pattern search to connect to Elasticsearch:

  1. Go to ManagementData Model.
  2. Click Create Index Pattern Search.
  3. Specify an index pattern search that matches the name of one or more of your Elasticsearch indices. By default, Siren Investigate guesses that you are working with log data being fed into Elasticsearch by Logstash.

    Note

    When you switch between top-level tabs, Siren Investigate remembers where you were. For example, if you view a particular index pattern search from the Settings tab, switch to the Discover tab, and then go back to the Settings tab, Siren Investigate displays the index pattern search you last looked at. To get to the create pattern form, click Add in the Index Pattern Search list.

  4. If your index contains a timestamp field that you want to use to perform time-based comparisons, select the Index contains time-based events option and select the index field that contains the timestamp. Siren Investigate reads the index mapping to list all the fields that contain a timestamp.
  5. By default, Siren Investigate restricts wildcard expansion of time-based index patterns to indices with data within the currently selected time range. Click Do not expand index pattern when search to switch off this behavior.
  6. Click Create to add the index pattern.
  7. To designate the new pattern as the default pattern to load when you view the Discover tab, click Favorite.

Note

When you define an index pattern search, indices that match that pattern must exist in Elasticsearch. Those indices must contain data.

Note that the colon ':' has been deprecated in index names, and should not be used.

To use an event time in an index name, enclose the static text in the pattern and specify the date format using the tokens described in the following table.

For example, [logstash-]YYYY.MM.DD matches all indices whose names have a timestamp of the form YYYY.MM.DD appended to the prefix logstash-, such as logstash-2015.01.31 and logstash-2015-02-01.

Table 13. Date Format Tokens

M

Month - cardinal: 1 2 3 … 12

Mo

Month - ordinal: 1st 2nd 3rd … 12th

MM

Month - two digit: 01 02 03 … 12

MMM

Month - abbreviation: Jan Feb Mar … Dec

MMMM

Month - full: January February March … December

Q

Quarter: 1 2 3 4

D

Day of Month - cardinal: 1 2 3 … 31

Do

Day of Month - ordinal: 1st 2nd 3rd … 31st

DD

Day of Month - two digit: 01 02 03 … 31

DDD

Day of Year - cardinal: 1 2 3 … 365

DDDo

Day of Year - ordinal: 1st 2nd 3rd … 365th

DDDD

Day of Year - three digit: 001 002 … 364 365

d

Day of Week - cardinal: 0 1 3 … 6

do

Day of Week - ordinal: 0th 1st 2nd … 6th

dd

Day of Week - 2-letter abbreviation: Su Mo Tu … Sa

ddd

Day of Week - 3-letter abbreviation: Sun Mon Tue … Sat

dddd

Day of Week - full: Sunday Monday Tuesday … Saturday

e

Day of Week (locale): 0 1 2 … 6

E

Day of Week (ISO): 1 2 3 … 7

w

Week of Year - cardinal (locale): 1 2 3 … 53

wo

Week of Year - ordinal (locale): 1st 2nd 3rd … 53rd

ww

Week of Year - 2-digit (locale): 01 02 03 … 53

W

Week of Year - cardinal (ISO): 1 2 3 … 53

Wo

Week of Year - ordinal (ISO): 1st 2nd 3rd … 53rd

WW

Week of Year - two-digit (ISO): 01 02 03 … 53

YY

Year - two digit: 70 71 72 … 30

YYYY

Year - four digit: 1970 1971 1972 … 2030

gg

Week Year - two digit (locale): 70 71 72 … 30

gggg

Week Year - four digit (locale): 1970 1971 1972 … 2030

GG

Week Year - two digit (ISO): 70 71 72 … 30

GGGG

Week Year - four digit (ISO): 1970 1971 1972 … 2030

A

AM/PM: AM PM

a

am/pm: am pm

H

Hour: 0 1 2 … 23

HH

Hour - two digit: 00 01 02 … 23

h

Hour - 12-hour clock: 1 2 3 … 12

hh

Hour - 12-hour clock, 2 digit: 01 02 03 … 12

m

Minute: 0 1 2 … 59

mm

Minute - two-digit: 00 01 02 … 59

s

Second: 0 1 2 … 59

ss

Second - two-digit: 00 01 02 … 59

S

Fractional Second - 10ths: 0 1 2 … 9

SS

Fractional Second - 100ths: 0 1 … 98 99

SSS

Fractional Seconds - 1000ths: 0 1 … 998 999

Z

Timezone - zero UTC offset (hh:mm format): -07:00 -06:00 -05:00 .. +07:00

ZZ

Timezone - zero UTC offset (hhmm format): -0700 -0600 -0500 … +0700

X

Unix Timestamp: 1360013296

x

Unix Millisecond Timestamp: 1360013296123



Setting the default index pattern search

The default index pattern search is loaded automatically when you view the Discover tab. Siren Investigate displays a star to the left of the name of the default pattern in the Index Pattern Search list on the ManagementData Model tab. The first pattern you create is automatically designated as the default pattern.

To set a different default index pattern search:

  1. Go to ManagementData Model.
  2. Select the index pattern search you want to set as the default from the list.
  3. Click Favorite.

Note

You can also manually set the default index pattern search in ManagementAdvanced Settings.

Reloading the index fields list

When you add an index mapping, Siren Investigate automatically scans the indices that match the pattern to display a list of the index fields. You can reload the index fields list to pick up any newly-added fields.

Reloading the index fields list also resets Siren Investigate’s popularity counters for the fields. The popularity counters keep track of the fields you have used most often within Siren Investigate and are used to sort fields within lists.

To reload the index fields list:

  1. Go to  ManagementData Model.
  2. Select an index pattern search from the list.
  3. Click Reload.

Removing an index pattern search

  1. Go to  ManagementData Model.
  2. Select the index pattern search you want to remove in the list.
  3. Click Delete.
  4. Confirm that you want to remove the index pattern search.
Cross cluster search

Elasticsearch supports the ability to run search and aggregation requests across multiple clusters using a module called cross cluster search.

Note

Siren Federate does not currently support cross cluster search.

To take advantage of cross cluster search, you must configure your Elasticsearch clusters accordingly. Refer to the corresponding Elasticsearch documentation before attempting to use cross cluster search in Siren Investigate.

After your Elasticsearch clusters are configured for cross cluster search, you can create specific index patterns in Siren Investigate to search across the clusters of your choosing. Using the same syntax that you would use in a raw cross cluster search request in Elasticsearch, create your index pattern in Siren Investigate with the convention <cluster-names>:<pattern>.

For example, if you want to query logstash indices across two of the Elasticsearch clusters that you set up for cross cluster search, which were named cluster_one and cluster_two, you would use cluster_one:logstash-*,cluster_two:logstash-* as your index pattern in Siren Investigate.

Just like in raw search requests in Elasticsearch, you can use wildcards in your cluster names to match any number of clusters, so if you wanted to search logstash indices across any clusters named cluster_foo, cluster_bar, and so on, you would use cluster_*:logstash-* as your index pattern in Siren Investigate.

If you want to query across all Elasticsearch clusters that have been configured for cross cluster search, then use a standalone wildcard for your cluster name in your Siren Investigate index pattern: *:logstash-*.

After an index pattern is configured using the cross cluster search syntax, all searches and aggregations using that index pattern in Siren Investigate take advantage of cross cluster search.