Siren Platform User Guide

Migrating to Siren Investigate 10.x

Before starting the upgrade procedure, ensure that Siren Investigate/Kibi is at version 5.4.3-5 or later.

Note

To upgrade to Siren Investigate 10.x, you must already have upgraded to at least version 10.0.

Prerequisites

Elasticsearch 5.6.9 or later is required. If the cluster is secured by Search Guard, it will be necessary to alter the security configuration. Before doing this, make a backup as described in the next section.

For each node in the cluster, you must:

  • Remove the siren-vanguard plugin if installed by running elasticsearch-plugin remove siren-vanguard.
  • Install the version of the siren-federate plugin compatible with your Elasticsearch version.

If you are upgrading Elasticsearch and using Search Guard, ensure you upgrade the plugin as well; to find out the version of Search Guard compatible with your Elasticsearch installation, refer to the Search Guard version matrix.

Search Guard can be installed from Maven Central using elasticsearch-plugin, for example:

./usr/share/elasticsearch/bin/elasticsearch-plugin install -b com.floragunn:search-guard-5:<version>

In addition, the compatible version of any commercial Search Guard add-on must be downloaded and copied to the plugins/search-guard-<version> folder; the following list provides links to the download page of all the add-ons commonly used in Siren Investigate instances:

Search Guard configuration backup

If the cluster is secured by Search Guard, you will need to retrieve the current security configuration through the sgadmin tool, modify it as instructed and reload it.

sgadmin.sh is available in the plugins/search-guard-<version>/tools folder on each Elasticsearch instance in which Search Guard has been installed; a standalone version (sgadmin-standalone.zip) can be downloaded from https://github.com/floragunncom/search-guard/wiki#search-guard-admin-standalone.

The current configuration can be retrieved by creating a backup folder and running the tool as follows:

mkdir configbak
bash /path/to/sgadmin.sh -r -ks CN\=sgadmin-keystore.jks -cd configbak -kspass password -ts truststore.jks -tspass password -icl -nhnv -h elasticsearch.local -p 9300

Arguments:

  • r: Retrieve configuration.
  • cd: The folder in which the current configuration will be saved.
  • ks: The path to the administrative keystore
  • kspass: The password of the administrative keystore.
  • ts: The path to the truststore.
  • tspass: The password of the truststore.
  • icl: Ignore cluster name.
  • nhvv: Do not verify cluster hostname.
  • h: Elasticsearch hostname.
  • p: Elasticsearch transport port.

If execution is successful, you will find the five Search Guard configuration files in the configbak folder suffixed by the current date.

Before proceeding, ensure that the files are not empty and remove the date suffix; you should end up with the following listing:

  • sg_action_groups.yml
  • sg_config.yml
  • sg_internal_users.yml
  • sg_roles_mapping.yml
  • sg_roles.yml

Search Guard configuration changes

This section describes all the Siren Investigate specific changes required to the Search Guard configuration files that were retrieved in the previous section.

After the files have been updated, the updated configuration can be loaded to the cluster using sgadmin as explained at the end of the section.

sg_action_groups.yml

Add the following action groups if missing:

UNLIMITED:
  - "*"

INDICES_ALL:
  - "indices:*"

Set the ALL action group as follows:

ALL:
  - INDICES_ALL

Modify the CREATE_INDEX action group as follows:

CREATE_INDEX:
  - "indices:admin/create"
  - "indices:admin/mapping/put"

Add the INDICES_MONITOR action group:

MONITOR:
  - INDICES_MONITOR

INDICES_MONITOR:
  - "indices:monitor/*"

Update the DATA_ACCESS, READ, WRITE, DELETE, CRUD and SEARCH groups as follows:

DATA_ACCESS:
  - "indices:data/*"
  - CRUD

WRITE:
  - "indices:data/write*"
  - "indices:admin/mapping/put"

READ:
  - "indices:data/read*"
  - "indices:admin/mappings/fields/get*"

DELETE:
  - "indices:data/write/delete*"

CRUD:
  - READ
  - WRITE

SEARCH:
  - "indices:data/read/search*"
  - "indices:data/read/msearch*"
  - "indices:siren/plan*"
  - "indices:siren/mplan*"
  - SUGGEST

Update the INDEX group as follows:

INDEX:
  - "indices:data/write/index*"
  - "indices:data/write/update*"
  - "indices:admin/mapping/put"
  - "indices:data/write/bulk*"

Add or update the CLUSTER_COMPOSITE_OPS_RO and CLUSTER_COMPOSITE_OPS roles as follows:

CLUSTER_COMPOSITE_OPS_RO:
  - "indices:data/read/mget"
  - "indices:data/read/msearch"
  - "indices:siren/mplan"
  - "indices:data/read/mtv"
  - "indices:admin/aliases/exists*"
  - "indices:admin/aliases/get*"

CLUSTER_COMPOSITE_OPS:
  - "indices:data/write/bulk*"
  - "indices:admin/aliases*"
  - CLUSTER_COMPOSITE_OPS_RO

Remove the KIBI_MSEARCH role if present.

Add/update the SIREN_COMPOSITE role with the following definition:

SIREN_COMPOSITE:
  - "indices:siren/mplan*"

If present, replace the KIBI_COMPOSITE role with the following definition for backwards compatibility.

KIBI_COMPOSITE:
  - SIREN_COMPOSITE

Add the SIREN_CLUSTER, SIREN_READONLY and SIREN_READWRITE roles with the following definitions:

SIREN_CLUSTER:
  - "cluster:data/read/lock/create"
  - "cluster:siren/internal*"
  - "cluster:admin/plugin/siren/license/get"
  - "indices:data/read/scroll*"
  - "indices:data/read/msearch*"
  - "indices:siren/mplan*"
  - CLUSTER_COMPOSITE_OPS_RO

SIREN_READONLY:
  - "indices:data/read/field_stats*"
  - "indices:data/read/field_caps*"
  - "indices:data/read/get*"
  - "indices:data/read/mget*"
  - "indices:data/read/search*"
  - "indices:siren/mplan*"
  - "indices:siren/plan*"
  - "indices:admin/get*"
  - "indices:admin/mappings/get*"
  - "indices:admin/mappings/fields/get*"
  - "indices:admin/validate/query*"
  - "indices:admin/version/get*"
  - "indices:data/siren/connector/*"
  - SIREN_COMPOSITE

SIREN_READWRITE:
  - "indices:admin/exists*"
  - "indices:admin/mapping/put*"
  - "indices:admin/refresh*"
  - "indices:data/write/delete*"
  - "indices:data/write/index*"
  - "indices:data/write/update*"
  - "indices:data/write/bulk*"
  - SIREN_READONLY

If present, replace KIBI_CLUSTER, KIBI_READONLY and KIBI_READWRITE with the following definitions for backwards compatibility.

KIBI_CLUSTER:
  - SIREN_CLUSTER

KIBI_READONLY:
  - SIREN_READONLY

KIBI_READWRITE:
  - SIREN_READWRITE
sg_roles.yml
kibiserver

Replace the kibiserver role with the following; make sure you write the correct names for your configuration indices in place of ?siren and ?sirenaccess .

kibiserver:
  cluster:
      - cluster:monitor/nodes/info
      - cluster:monitor/health
      - cluster:monitor/main
      - cluster:monitor/state
      - cluster:monitor/nodes/stats
      - SIREN_CLUSTER
      - CLUSTER_COMPOSITE_OPS
  indices:
    '*':
      '*':
        - indices:admin/get
    '?siren':
      '*':
        - ALL
    '?sirenaccess':
      '*':
        - ALL
Siren Investigate user roles

In each existing user role with access to Siren Investigate:

  • Replace KIBI_MSEARCH with SIREN_COMPOSITE.
  • Replace KIBI_CLUSTER with SIREN_CLUSTER.
  • Replace KIBI_COMPOSITE with SIREN_COMPOSITE.
  • Remove the following permission on the main Siren Investigate index (which was named .kibi in previous versions) from each user role that has it:
    ?kibi:
      null:
        - "indices:data/read/search"
        - "indices:data/read/coordinate-search"

If you are using the Siren Investigate access control plugin and want to enable access control rules on saved objects, ensure that Siren Investigate users do not have access to the Siren Investigate index (which was set to .kibi in previous releases), otherwise they would be able to peek at saved objects by issuing custom Elasticsearch queries.

Access to the Siren Investigate index was usually granted by the following lines in previous releases; these must be removed to use access control rules effectively as the configuration index will be managed exclusively by the kibiserver role:

'?kibi':
  '*':
    - KIBI_READWRITE
'?kibi':
  '*':
    - KIBI_READONLY

If there are users with read access to all indices (*), they will be able to search the Siren Investigate index as well; if you want to inhibit access to it, you should replace the wildcard with an explicit list of indices or a more restrictive index pattern.

License management

To grant a role the permission to upload a Siren Investigate license from the UI, ensure that the user has the cluster:admin/plugin/siren/license/* permission, for example:

sirenadmin:
  cluster:
    - SIREN_CLUSTER
    - 'cluster:admin/siren/connector/*'
    - 'cluster:admin/plugin/siren/license/*'
  ...

Monitoring

To grant a user the permission to use monitoring plugins and retrieve diagnostics information from the UI, you will need to add the CLUSTER_MONITOR and INDICES_MONITOR permissions to its role, for example:

# Permissions for an administrator
sirenadmin:
  cluster:
    ...
  indices:
    '*':
      '*':
        - KIBI_READONLY
        - INDICES_MONITOR
    '?kibi':
      '*':
        - KIBI_READWRITE
    'watcher':
      '*':
        - KIBI_READWRITE

JDBC datasources management

For information on how to set up Search Guard for JDBC datasources support see the 3.11. JDBC datasources section after the upgrade.3.11. JDBC datasources

Marvel and X-Pack monitoring

If you were using Marvel and are migrating to X-Pack monitoring, the following role can be used in place of the previous sample marvel role:

# Permissions for an X-Pack monitoring agent.
monitoring:
  cluster:
    - CLUSTER_MONITOR
    - 'indices:admin/aliases'
    - 'indices:admin/template/get'
    - 'indices:admin/template/put'
    - 'cluster:admin/ingest/pipeline/get'
    - 'cluster:admin/ingest/pipeline/put'
    - 'indices:data/write/bulk'
  indices:
    '?marvel*':
      '*':
        - ALL
    '?monitoring*':
      '*':
        - ALL
Loading the updated configuration

To load the modified configuration run sgadmin as follows:

$ bash /path/to/sgadmin.sh -ks CN\=sgadmin-keystore.jks -cd confignew -kspass password -ts truststore.jks -tspass password -icl -nhnv -h elasticsearch.local -p 9300
Search Guard Admin v5
Will connect to elasticsearch.local:9330 ... done
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
Clustername: escluster
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
searchguard index already exists, so we do not need to create one.
Will update 'config' with confignew/sg_config.yml
   SUCC: Configuration for 'config' created or updated
Will update 'roles' with confignew/sg_roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update 'rolesmapping' with confignew/sg_roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update 'internalusers' with confignew/sg_internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update 'actiongroups' with confignew/sg_action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Done with success

Arguments:

  • cd: The folder containing the modified Search Guard configuration.
  • ks: The path to the administrative keystore
  • kspass: The password of the administrative keystore.
  • ts: The path to the truststore.
  • tspass: The password of the truststore.
  • icl: Ignore cluster name.
  • nhvv: Do not verify cluster hostname.
  • h: Elasticsearch hostname.
  • p: Elasticsearch transport port.

Upgrading Siren Investigate

  1. Extract Siren Investigate 10.x to a new folder.
  2. Copy the config/kibi.yml file from the old installation to the config folder of the new installation.
  3. Copy the data folder from your old installation to the new installation.
  4. Copy the pki folder from your old installation to the new installation.
  5. Update the Search Guard configuration as described in the next section.
  6. Install the upgraded versions of third party plugins if applicable.
  7. Run the upgrade scripts:

    1. bin/investigate upgrade-config;
    2. bin/investigate upgrade.
  8. Start the new installation and ensure it is working as expected.
Using certificates and keystores

Ensure that any certificate files or keystores are accessible and correctly referenced from the new installation, particularly if relative paths are used.

Uploading a license

You must upload your license after the cluster is up and running.

Uploading a license from Siren Investigate

To upload the license from Siren Investigate, navigate to the management section.

management no license

This page will show the license has not been installed.

Click License and select Upload license

License page before install.

After you have selected your license and it has been uploaded, the page will show the license details.

License page after install.

Uploading a license from the command prompt

curl http://localhost:9220/_siren/license -XPUT -T license.sig -H "Content-Type: application/json"

If your cluster is protected by Search Guard, remember to specify credentials:

curl -uadmin:password --cacert=ca.pem https://localhost:9220/_siren/license -XPUT -T license.sig -H "Content-Type: application/json"