Integrating Search Guard Classic

This section offers an overview of how to install and configure Search Guard Classic on an Elasticsearch cluster; if you need to upgrade an Elasticsearch cluster that had been previously configured to work with Siren Investigate, see Upgrading Search Guard Classic.

Before you begin

Ensure that Siren Investigate is running with HTTPS enabled. For more information, see Enabling HTTPS in Siren Investigate.
Ensure that you have prepared all the required security certificates. For more information, see Setting up security certificates.
Download a compatible release of the sgadmin tool for your Elasticsearch release at https://docs.search-guard.com/latest/search-guard-versions and extract it to a directory on a machine that can communicate with the Elasticsearch cluster.

Installing Search Guard Classic

The first installation of Search Guard Classic requires a full cluster restart as all the nodes will need to have TLS enabled to communicate with each other.

Disabling shard allocation

If the cluster contains multiple nodes, it is recommended to disable shard allocation before proceeding to reduce the time required to restart the nodes; this can be done by changing the cluster settings as follows (replace http://localhost:9200 with the URL of your cluster):

curl -s -XPUT 'http://localhost:9200/_cluster/settings?pretty' -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.enable": "none"
  }
}
'

A successful response to the request above will look as follows:

{
  "acknowledged" : true,
  "persistent" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "enable" : "none"
        }
      }
    }
  },
  "transient" : { }
}

Installing the plugin

Install the Search Guard Classic plugin on each node in the Elasticsearch cluster by changing to the directory where Elasticsearch is installed and running the following command:

bin/elasticsearch-plugin install -b <link to a compatible Search Guard Classic release>

If Elasticsearch has been installed from a package, the elasticsearch-plugin command will usually be located at /usr/share/elasticsearch/bin/elasticsearch-plugin.

Search Guard Classic releases for your Elasticsearch version can be found at https://docs.search-guard.com/latest/search-guard-versions .

Search Guard FLX is currently not supported by Siren Investigate, please make sure to use Search Guard Classic.

As an example, if you want to install Search Guard Classic 53.4.0 on an Elasticsearch 7.17.6 node, the command to run will be:

bin/elasticsearch-plugin install -b https://maven.search-guard.com/search-guard-suite-release/com/floragunn/search-guard-suite-plugin/7.17.6-53.4.0/search-guard-suite-plugin-7.17.6-53.4.0.zip

Copy the following files to the config directory of the node:

the file containing the full CA bundle (e.g. chain-ca.pem);
the file containing the node certificate (e.g. CN=localhost.crtfull.pem);
the file containing the private key of the node certificate (e.g. CN=localhost.key.pem).

Next, edit the config/elasticsearch.yml file and set the following Search Guard parameters:

searchguard.ssl.transport.pemcert_filepath: the path of the server certificate file relative to the config directory, e.g. CN=localhost.crtfull.pem.
searchguard.ssl.transport.pemkey_filepath: the name of the file containing the private key for the certificate above, e.g. CN=localhost.key.pem.
searchguard.ssl.transport.pemkey_password: the password of the private key;
searchguard.ssl.transport.pemtrustedcas_filepath: the name of the file containing the CA chain bundle, e.g. chain-ca.pem.
searchguard.ssl.transport.enforce_hostname_verification: set to true to enable hostname verification, set to false to disable it. Node certificates must contain valid SAN entries with their IP addresses and hostnames when this option to be enabled.
searchguard.ssl.http.enabled: set to true.
searchguard.ssl.http.pemcert_filepath: the path of the server certificate file relative to the config directory, e.g. CN=localhost.crtfull.pem.
searchguard.ssl.http.pemkey_filepath: the name of the file containing the private key for the certificate above, e.g. CN=localhost.key.pem.
searchguard.ssl.http.pemkey_password: the password of the private key;
searchguard.ssl.http.pemtrustedcas_filepath: the name of the file containing the CA chain bundle, e.g. chain-ca.pem.
searchguard.authcz.admin_dn: a list containing the common name of the administrative certificate (e.g. ["CN=sgadmin"]).
searchguard.authcz.admin_dn: a list containing the common name of the administrative certificate (e.g. ["CN=sgadmin"]).
searchguard.restapi.roles_enabled: a list of roles which are allowed to access the Search Guard Classic REST configuration API (e.g. ["investigate_admin"]). Not required if you want to manage the Search Guard Classic configuration exclusively through sgadmin.
searchguard.allow_custom_headers: Set to "siren.*" to prevent Search Guard Classic from filtering out custom headers that are set by the Siren Federate plugin. This setting is not required for Elasticsearch versions previous to 7.6.2.

Next, disable Elastic Stack security:

xpack.security.enabled: If you are using Elasticsearch version 7.11 or later, set to false to allow Search Guard Classic to start.

Full example:

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: "CN=localhost.crtfull.pem"
searchguard.ssl.transport.pemkey_filepath: "CN=localhost.key.pem"
searchguard.ssl.transport.pemkey_password: "password"
searchguard.ssl.transport.pemtrustedcas_filepath: "chain-ca.pem"
searchguard.ssl.transport.enforce_hostname_verification: true
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: "CN=localhost.crtfull.pem"
searchguard.ssl.http.pemkey_filepath: "CN=localhost.key.pem"
searchguard.ssl.http.pemkey_password: "password"
searchguard.ssl.http.pemtrustedcas_filepath: "chain-ca.pem"
searchguard.authcz.admin_dn:
- "CN=sgadmin"
searchguard.restapi.roles_enabled:
- "investigate_admin"
searchguard.allow_custom_headers: "_siren_.*"

Ensure that all the files in the configuration folder and the certificate files are readable only by the user Elasticsearch is running as.

Finally restart the node and ensure that there are no configuration errors reported; the node process should be up and report the following in the logs:

[INFO ][c.f.s.c.IndexBaseConfigurationRepository] [QHOKpbG] searchguard index does not exist yet, use either sgadmin to initialize cluster or wait until cluster is fully formed and up

Re-enabling shard allocation

If shard allocation was disabled, after all nodes have restarted it has to be re-enabled in order to allow the allocation of the Search Guard Classic index that will store the security settings for the cluster.

In order to re-enable shard allocation, you will need use the sgadmin tool with the administrative certificate as follows:

tools/sgadmin.sh -nhnv -icl -h localhost -ff -prompt --enable-shard-allocation -cert "CN=sgadmin.crtfull.pem" -key "CN=sgadmin.key.pem" -cacert "chain-ca.pem"

Use the following arguments in the command:

-nhnv: Skips hostname verification, can be omitted if certificates are setup to allow hostname verification.
-ff: Fail fast if there are issues with the configuration without retrying.
-icl: Accepts any cluster name. Can be changed with -cn <cluster name> if you want the tool to check the name of the cluster before uploading the configuration.
-h: The hostname at which Elasticsearch is reachable.
-prompt: Tells sgadmin to prompt for the private key password.
--enable-shard-allocation: Enables shard allocation after a cluster restart.
-cert: The path to the administrative certificate file.
-key: The path to the file containing the private key for the administrative certificate.
-cacert: The path to the file containing the CA bundle.

If the command is successful you should see this message in the output:

Persistent and transient shard allocation enabled

Search Guard Classic configuration

Access control configuration (authentication mechanisms, users, roles and privileges) is defined in a list of yaml files in a directory; these files can be uploaded to a special index in the Elasticsearch cluster using the sgadmin tool with an administrative certificate.

The files in a configuration directory are:

sg_config.yml: general settings and configuration of authentication mechanisms;
sg_action_groups.yml: named collections of permissions;
sg_roles.yml: named roles associated to permissions and action groups;
sg_internal_users.yml: a list of internal users;
sg_roles_mapping.yml: mappings between roles defined in sg_roles.yml and either internal users or roles defined in external systems (e.g. LDAP groups).
sg_tenants.yml: a list of tenant configurations. Tenants are not currently supported by Siren Investigate but the file must exist in the configuration directory.

Sample configurations are available at the following links:

sample Search Guard Classic configuration for Elasticsearch 7.x.

The sample configurations define the following roles:

investigate_system: defines the permissions for the Siren Investigate server process, with read/write access to the internal Siren Investigate indices.
investigate_admin: defines the permissions for a Siren Investigate user that has the administrator permissions. This user has permissions to upload the Siren Investigate license, get monitoring information from the cluster and manage JDBC datasources.
investigate_user: Defines the permissions for a Siren Investigate user with read only access to all indices whose name starts with data-.

These roles are mapped to three users, all having their password set to password:

sirenserver: the user that Siren Investigate uses to manage saved objects stored in Elasticsearch;
sirenadmin: an administrative user that has been given the investigate_admin role;
sirenuser: a standard user that has been given the investigate_user role.

These examples are provided only as a general guideline; for the full list of available security options and configurations please refer to the Search Guard documentation.

sg_action_groups.yml

The sg_action_groups.yml file contains named groups of permissions that can be referred to in the definition of roles.

Action groups are logically divided into index level and cluster level permissions.

Cluster level action groups:

CLUSTER_ALL: all cluster-level actions.
CLUSTER_MONITOR: monitoring actions.
CLUSTER_COMPOSITE_OPS: Groups all the permissions to execute composite requests not recognized by Search Guard Classic; the group has to be granted on all indices to roles that have access only to a subset of indices .
CLUSTER_INTERNAL_FEDERATE: actions specific to the internal workflow of Federate.
CLUSTER_MANAGE: in addition to CLUSTER_INTERNAL_FEDERATE this includes actions to manage its own internal indices.

Index level action groups:

INDICES_ALL: all indices-level actions.
MONITOR: all actions regarding index monitoring, e.g., recovery, segments info, index stats and status.
MANAGE: all monitor and index administration actions.
WRITE: actions to modify an index.
READ: actions to read and search an index.
VIEW_INDEX_METADATA: any action to retrieve metadata about an index, e.g., list of fields, get a setting.

sg_roles.yml

The sg_roles.yml file contains a list of user roles; each role contains a set of permissions at the cluster level and for individual indices.

Example role in Elasticsearch 7:

# Permissions for a standard Investigate user.
investigate_user:
  cluster_permissions:
  - CLUSTER_COMPOSITE_OPS_RO
  - CLUSTER_INTERNAL_FEDERATE
  index_permissions:
  - index_patterns:
    - 'investor'
    allowed_actions:
    - READ
    - VIEW_INDEX_METADATA

Example role in Elasticsearch 6:

# Permissions for a standard Investigate user.
investigate_user:
  cluster:
  - CLUSTER_COMPOSITE_OPS_RO
  - CLUSTER_INTERNAL_FEDERATE
  indices:
    'data-*':
      '*':
      - READ
      - VIEW_INDEX_METADATA

The index name can contain the simple expansion characters * and ? to match any sequence of character or any single character.

Read access to an index can be given by granting the action groups READ and VIEW_INDEX_METADATA; full access by granting INDICES_ALL.

In user has multiple roles that define different permissions for the same index, Search Guard Classic will combine them.

sg_internal_users.yml

The file defines the credentials for Search Guard Classic internal users; passwords are stored in hashed form in the hash attribute beneath each username.

The password for all the accounts in the sample configurations is password.

To change the password of a user in this configuration file, you must generate the corresponding hash; this can be done by executing the tools/hash.sh script inside the sgadmin directory as follows:

tools/hash.sh -p password

The script will output the hash for the password specified after the -p switch.

sg_roles_mapping.yml

This file defines the mappings between users / backend roles and roles defined in sg_roles.yml. Back-end roles are usually roles that are returned by an external authentication system, such as an LDAP group.

In Elasticsearch 7, a mapping has the following format:

<role name>:
  users:
  - <username>
  - <username>
  backend_roles:
  - <backend role>

Uploading the configuration to the cluster

You can use the tools/sgadmin.sh script in the sgadmin distribution to upload the configuration as follows:

tools/sgadmin.sh -nhnv -icl -h localhost -ff -prompt -cert "CN=sgadmin.crtfull.pem" -key "CN=sgadmin.key.pem" -cacert "chain-ca.pem" -cd "sgconfig"

Use the following arguments in the command:

-nhnv: Skips hostname verification, can be omitted if certificates are setup to allow hostname verification.
-ff: Fail fast if there are issues with the configuration without retrying.
-icl: Accepts any cluster name. Can be changed with -cn <cluster name> if you want the tool to check the name of the cluster before uploading the configuration.
-h: The hostname at which Elasticsearch is reachable.
-prompt: Tells sgadmin to prompt for the private key password.
-cert: The path to the administrative certificate file.
-key: The path to the file containing the private key for the administrative certificate.
-cacert: The path to the file containing the CA bundle.
-cd: The path to the configuration directory.

If the command is successful, a list of the actions executed and their outcome will be printed on screen, for example:

Clustername: elasticsearch
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
searchguard index does not exists, attempt to create it ... done
Populate config from sgconfig
Will update 'config' with sgconfig/sg_config.yml
   SUCC: Configuration for 'config' created or updated
...
Done with success

You can then verify that authentication is working as expected by sending an authenticated request with curl, for example:

curl --cacert chain-ca.pem -uadmin "https://localhost:9200/_searchguard/authinfo?pretty"

If authentication is successful, you should get a response containing the user information, e.g.:

{
  "user" : "User [name=admin, roles=[], requestedTenant=null]",
  "user_name" : "admin",
  "backend_roles" : [ ],
  "sg_roles" : [
    "sg_all_access"
  ]
}

Next steps

Configuring Investigate to work with Search Guard Classic