Siren Platform User Guide

Upgrading from Investigate 10.1.x or 10.2.x

You can migrate directly from Investigate 10.1.3 and higher to 10.3.0. If upgrading from an earlier version of Investigate, it is recommended that you first upgrade to 10.1.3 (which also supports Elasticsearch 5.x). You can then upgrade from 10.1.3 to 10.3.0.

Note

When upgrading both Investigate and Elasticsearch, you should upgrade the Investigate objects first, and only then upgrade Elasticsearch.

The recommended version of Elasticsearch is 6.8.0 (version 6.5.4 is also supported). Please see the Search Guard Integration and Siren Investigate access control to configure Search Guard security settings correctly for Elasticsearch 6.8.0.

Important

Before you upgrade Siren Investigate, you must take account of the security implications of the new features installed. Details of the necessary security procedures can be found in Security standardization.

An existing Siren Investigate installation can be upgraded as follows:

  1. Backup the 10.1.x .siren index.

  2. Backup the Siren Investigate configuration file (config/investigate.yml).

  3. Backup the .sirenaccess if ACL (Access Control Layer) is enabled

  4. If you are running Elasticsearch 6.3.2, you can optionally upgrade to version 6.8.0 (https://www.elastic.co/guide/en/elasticsearch/reference/6.5/setup-upgrade.html).

  5. Before restarting each Elasticsearch node, ensure you install a compatible version of the Siren Federate plugin and access control plugins (Searchguard or X-pack security) if requiredIntroduction

  6. Download and extract the new Siren Investigate version.

  7. Copy the previous configuration file to the config folder of the new installation.

  8. Check for breaking changes to the configuration at Search Guard Integration and Siren Investigate access control.

  9. Install the compatible versions of third party Siren Investigate/Kibana plugins that you may need into the /plugins folder.

  10. Execute the upgrade command.

Note

Elasticsearch requires a matching version of Search Guard. For example, Elasticsearch 6 requires Search Guard 6. For information on upgrading Search Guard, see https://docs.search-guard.com/latest/upgrading-560.

Backing up and restoring the Siren Investigate indices

Before upgrading, you should have a backup of the .siren index; the recommended way to perform regular backups of Elasticsearch indexes is through the snapshot and restore modules.

Siren Investigate ships with a command line interface for creating dumps of the .siren index and, in case the ACL is enabled, the .sirenaccess index as well. An index dump is composed of two parts: its mappings and its data.

Backup

The backup command requires a running Elasticsearch instance and the path to a folder where the dumps will be written to.

You can find out more about its options by executing the following:

$ ./bin/investigate backup --help

For example, the following line will dump in <MY_FOLDER> the .siren index and the .sirenaccess index if the option investigate_access_control.acl.enabled is true in investigate.yml:

$ ./bin/investigate backup --backup-dir <MY_FOLDER>

Restore

The restore command requires a running Elasticsearch instance and the path to a folder where the dumps were written to by the previous backup command.

You can find out more about its options by executing the following:

$ ./bin/investigate restore --help

For example, you can restore the previously saved indices by executing the command and pointing to the dump folder, with .sirenaccess as well if the option investigate_access_control.acl.enabled is true in investigate.yml:

$ ./bin/investigate restore --backup-dir <MY_FOLDER>

Upgrading the Siren index

To upgrade the objects in the .siren index (dashboards, visualizations, and so on), move to the folder in which Siren Investigate is installed and execute the following command:

bin/investigate upgrade

The command will look for out of date objects and upgrade them, for example:

$ bin/investigate upgrade
  log   [17:58:33.494] [info][status][plugin:elasticsearch] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [17:58:36.127] [info][migrations] Executing migration "Upgrade scripts from version 1 to version 2"
  log   [17:58:36.141] [info][migrations] Executed migration "Upgrade scripts from version 1 to version 2"
  log   [17:58:36.142] [info][migrations] Executing migration "Upgrade graph browser visualization to version 2."
  log   [17:58:36.157] [info][migrations] Executed migration "Upgrade graph browser visualization to version 2."
  log   [17:58:36.158] [info][migrations] Executing migration "Upgrade saved queries from version 1 to version 2"
  log   [17:58:36.242] [info][migrations] Executed migration "Upgrade saved queries from version 1 to version 2"
  log   [17:58:36.242] [info][migrations] Executing migration "Upgrade saved templates from version 1 to version 2"
  log   [17:58:36.303] [info][migrations] Executed migration "Upgrade saved templates from version 1 to version 2"
  log   [17:58:36.303] [info][migrations] Executing migration "Upgrade saved queries definitions in external query terms aggregation, enhanced search results and query viewer."
  log   [17:58:36.400] [info][migrations] Executed migration "Upgrade saved queries definitions in external query terms aggregation, enhanced search results and query viewer."
Upgraded 20 objects.

It is possible to run the command multiple times, however running the command at the same time from multiple machines is not supported.

The upgrade command runs an automatic backup of the siren indices (.siren, .sirenaccess) and restores them (after deleting the existing index) in the event of a problem in the upgrade process - ensuring the system is not left in an unusable state. In the event of a successful upgrade, the backup is removed but if there is an issue, the backed up indexes are stored in the backup folder (defaults to the /data folder).

You can specify various flags to control the backup/restore process.

  • --backup-dir <path>: Custom backup folder path to store the index backup.
  • --config <path>: Path to the configuration file.
  • --delete-backup: Deletes the backup of the indexes after the upgrade process completes.
  • --dont-backup: Runs the upgrade process without creating a backup of the indexes.
  • -y: Accepts all of the options, for example, backup the indexes and remove the indexes before restoring.