Management

Elasticsearch supports the ability to run search and aggregation requests across multiple clusters using a module called cross-cluster search.

To take advantage of cross-cluster search, you must configure your Elasticsearch clusters accordingly. Refer to the corresponding Elasticsearch documentation before attempting to use cross-cluster search in Siren Investigate.

After your Elasticsearch clusters are configured for cross-cluster search, you can create specific index patterns in Siren Investigate to search across the clusters of your choosing. Using the same syntax that you would use in a raw cross-cluster search request in Elasticsearch, create your index pattern in Siren Investigate with the convention <cluster-names>:<pattern>.

For example, if you want to query logstash indices across two of the Elasticsearch clusters that you set up for cross-cluster search, which were named cluster_one and cluster_two, you would use cluster_one:logstash-*,cluster_two:logstash-* as your index pattern in Siren Investigate.

Just like in raw search requests in Elasticsearch, you can use wildcards in your cluster names to match any number of clusters, so if you wanted to search logstash indices across any clusters named cluster_foo, cluster_bar, and so on, you would use cluster_*:logstash-* as your index pattern in Siren Investigate.

If you want to query across all Elasticsearch clusters that have been configured for cross cluster search, then use a standalone wildcard for your cluster name in your Siren Investigate index pattern: *:logstash-*.

After an index pattern is configured using the cross-cluster search syntax, all searches and aggregations using that index pattern in Siren Investigate take advantage of cross-cluster search.

You can set a maximum time spent by each join task for a relation in milliseconds. When the timeout has expired, the task passes the documents accumulated at that point on to the next task. This can be overwritten by individual relations.

This is a per-task time limit. As each join contains several tasks, the overall response to the request can be a number of multiples of the set timeout. As a semi-join, these documents are filtered based on the presence of a non-empty value for the join field in the other index pattern in the relation. The index pattern in question is then filtered by the values returned.

To set a global timeout:

You can set the entity resolution types used by a relation to group a set of relations to represent the same conceptual entity. Any documents discovered using these relations will only be considered equivalent if they are ER types.

For more details on setting ER types, see Configuring advanced settings for relations.

To set global entity resolution types:

In Siren Investigate Complex relations leverage complex joins to connect one or more fields between source and target entity tables. Complex relations can be constructed using either a single condition with a comparison operator, i.e. gt gte lt lte contains intersects or within. Or alternatively, they can be created by combining multiple conditions with one or more eq equality operators as well as any of the comparison operators.

Complex joins are enabled since Siren Federate v31 which uses extended join condition syntax to retrieve documents from connected indices.

For information about creating and editing relations, see Creating relations.

For time based indices, the maximum time range can be defined. Setting a longer time range will be prevented by the system and a warning will be displayed.

The maximum number of documents that can be displayed on a dashboard can be set. If the filters are changed to show a larger number of documents, the system will not allow the operation and show a warning.

If you attempt to make a join between two dashboards by using the relational navigator and the number of documents on one or both of the dashboards exceeds the limit, the navigation is prevented and a warning message is displayed.

String fields support the String and URL formatters.

The String field formatter can apply the following transformations to the field’s contents:

The URL field formatter can take on the following types:

You can customize either type of URL field formats with templates. A URL template enables you to add specific values to a partial URL. Use the string {{value}} to add the contents of the field to a fixed URL.

For example, when:

The resulting URL replaces {{value}} with the user ID from the field.

The {{value}} template string URL-encodes the contents of the field. When a field encoded into a URL contains non-ASCII characters, these characters are replaced with a % character and the appropriate hexadecimal code. For example, field contents users/admin result in the URL template adding users%2Fadmin.

When the formatter type is set to Image, the {{value}} template string specifies the name of an image at the specified URI.

To pass unescaped values directly to the URL, use the {{rawValue}} string.

A Label Template enables you to specify a text string that displays instead of the raw URL. You can use the {{value}} template string normally in label templates. You can also use the {{url}} template string to display the formatted URL.

Date fields support the Date, Url, and String formatters.

The Date formatter enables you to choose the display format of date stamps using the moment.js standard format definitions.

The String field formatter can apply the following transformations to the field’s contents:

The URL field formatter can take on the following types:

You can customize either type of URL field formats with templates. A URL template enables you to add specific values to a partial URL. Use the string {{value}} to add the contents of the field to a fixed URL.

For example, when:

The resulting URL replaces {{value}} with the user ID from the field.

The {{value}} template string URL-encodes the contents of the field. When a field encoded into a URL contains non-ASCII characters, these characters are replaced with a % character and the appropriate hexadecimal code. For example, field contents users/admin result in the URL template adding users%2Fadmin.

When the formatter type is set to Image, the {{value}} template string specifies the name of an image at the specified URI.

To pass unescaped values directly to the URL, use the {{rawValue}} string.

A Label Template enables you to specify a text string that displays instead of the raw URL. You can use the {{value}} template string normally in label templates. You can also use the {{url}} template string to display the formatted URL.

Geographic point fields support the String formatter.

The String field formatter can apply the following transformations to the field’s contents:

Numeric fields support the URL, Bytes, Duration, Number, Percentage, String, and Color formatters.

The URL field formatter can take on the following types:

You can customize either type of URL field formats with templates. A URL template enables you to add specific values to a partial URL. Use the string {{value}} to add the contents of the field to a fixed URL.

For example, when:

The resulting URL replaces {{value}} with the user ID from the field.

The {{value}} template string URL-encodes the contents of the field. When a field encoded into a URL contains non-ASCII characters, these characters are replaced with a % character and the appropriate hexadecimal code. For example, field contents users/admin result in the URL template adding users%2Fadmin.

When the formatter type is set to Image, the {{value}} template string specifies the name of an image at the specified URI.

To pass unescaped values directly to the URL, use the {{rawValue}} string.

A Label Template enables you to specify a text string that displays instead of the raw URL. You can use the {{value}} template string normally in label templates. You can also use the {{url}} template string to display the formatted URL.

The String field formatter can apply the following transformations to the field’s contents:

The Duration field formatter can display the numeric value of a field in the following increments:

You can specify these increments with up to 20 decimal places for both input and output formats. The default number of decimals for the Number format is 3, i.e. 0,0.[000]. If there are values smaller than this, but larger than 1e-7, they will be rounded to 0. The fix is to change the Numeral.js format pattern to: 0,0.[0000000]

The Color field formatter enables you to specify colors with specific ranges of values for a numeric field.

When you select the Color field formatter, Siren Investigate displays the Range, Font Color, Background Color, and Example fields.

Click Add Color to add a range of values to associate with a particular color. You can click in the Font Color and Background Color fields to display a color picker. You can also enter a specific hex code value in the field. The effect of your current color choices are displayed in the Example field.

The Bytes, Number, and Percentage formatters enable you to choose the display formats of numbers in this field using the numeral.js standard format definitions.

Range fields support their default formatters, which are date_range, ip_range, and numeric_range. These fields are represeneted in JSON format.

An example of an integer_range field appears as follows:

ip_range fields in CIDR subnet mask notation are displayed as simple strings. For example:

Scripted fields compute data on the fly from the data in your Elasticsearch indices. Scripted field data is shown on the Discover tab as part of the document data, and you can use scripted fields in your visualizations. Scripted field values are computed at query time so they are not indexed and cannot be searched. Note that Siren Investigate cannot query scripted fields.

When you define a scripted field in Siren Investigate, you have a choice of scripting languages. Starting with 5.0, the default options are Lucene expressions and Painless. While you can use other scripting languages if you enable dynamic scripting for them in Elasticsearch, this is not recommended because they cannot be sufficiently sandboxed.

You can reference any single value numeric field in your expressions, for example:

For more background on scripted fields and additional examples, refer to Using Painless in Kibana scripted fields.

Creating a scripted field

For more information about scripted fields in Elasticsearch, see Scripting.

Modifying a scripted field

Deleting a scripted field