Siren Platform User Guide

All the Elasticsearch nodes in a cluster secured by Search Guard are required to use SSL to encrypt all network traffic.

In addition, changing the Search Guard configuration requires the use of a client SSL certificate to perform administrative actions.

To set up a Search Guard cluster, you must generate the following files:

These files can be either Java KeyStore files or PKCS12 bundles.

$ keytool  \
  -importcert \
  -file ca.pem  \
  -keystore truststore.jks

CN=admin,DC=siren,DC=solutions

Install the Search Guard plugin on each node in the Elasticsearch cluster by changing to the node folder and running the following commands (with the appropriate version number):

$ bin/elasticsearch-plugin install -b com.floragunn:search-guard-6:<version>

To find the most recent version of the plugins for your Elasticsearch version, consult the Search Guard version matrix.

Then, copy the following files to the config folder of each node:

Open the config/elasticsearch.yml file and set the following Search Guard options:

Node to node transport options

REST API options:

Administrative user options:

Client certificate authentication options:

For example:

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.truststore_password: <password>
searchguard.ssl.transport.keystore_filepath: CN=localhost-keystore.jks
searchguard.ssl.transport.keystore_password: <password>
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: CN=localhost-keystore.jks
searchguard.ssl.http.keystore_password: <password>
searchguard.ssl.http.truststore_filepath: truststore.jks
searchguard.ssl.http.truststore_password: <password>
searchguard.authcz.admin_dn:
  - CN=sgadmin
searchguard.ssl.http.clientauth_mode: OPTIONAL

Start Elasticsearch:

$ bin/elasticsearch

If either a certificate or a password is incorrect, Elasticsearch will not start.

Access control configuration (users, roles and privileges) is stored in an Elasticsearch index which can be modified through the sgadmin.sh script.

The script reads the configuration from a local folder containing YAML files and uploads it to the index; the request is authenticated through a client SSL certificate.

After the configuration has been uploaded, it will be available to all the nodes in the cluster, so it is not necessary to copy the Search Guard configuration folder to all the Elasticsearch nodes, just on the node from where sgadmin is run.

sgadmin.sh is available in the plugins/search-guard-<version>/tools folder in each Elasticsearch instance in which Search Guard has been installed; a standalone version (sgadmin-standalone.zip) can be downloaded from https://github.com/floragunncom/search-guard/wiki#search-guard-admin-standalone.

After a Search Guard enabled cluster has been initialized, sgadmin can be used to upload new configurations.

Here we give an example of where to store client certificates and keystores on Siren Investigate. Note: These are examples for a fresh install using the TLS certificate generator.

In siren-investigate/pki (which was created earlier for https support) a new folder searchguard was created with the following:

The password of all Java keystores can be found in the README.txt from top level folder of TLS certificate generator bundle.

searchguard:
  dynamic:
    respect_request_indices_options: true
    kibana:
      do_not_fail_on_forbidden: true
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: false
    authc:
      transport_auth_domain:
        enabled: true
        order: 2
        http_authenticator:
          type: basic
        authentication_backend:
          type: internal
      basic_internal_auth_domain:
        enabled: true
        http_authenticator:
          type: basic
          challenge: true
        authentication_backend:
          type: intern

UNLIMITED:
  - "*"

###### INDEX LEVEL ######

INDICES_ALL:
  - "indices:*"

# for backward compatibility
ALL:
  - INDICES_ALL

MANAGE:
  - "indices:monitor/*"
  - "indices:admin/*"

# for backward compatibility
MONITOR:
  - MANAGE

WRITE:
  - "indices:data/write*"
  - "indices:admin/mapping/put"

READ:
  - "indices:data/read*"

VIEW_INDEX_METADATA:
  - "indices:admin/aliases/get"
  - "indices:admin/aliases/exists"
  - "indices:admin/get"
  - "indices:admin/exists"
  - "indices:admin/mappings/fields/get*"
  - "indices:admin/mappings/get*"
  - "indices:admin/mappings/federate/connector/get*"
  - "indices:admin/mappings/federate/connector/fields/get*"
  - "indices:admin/types/exists"
  - "indices:admin/validate/query"
  - "indices:monitor/settings/get"

###### CLUSTER LEVEL ######

CLUSTER_ALL:
  - "cluster:*"

CLUSTER_MONITOR:
  - "cluster:monitor/*"

CLUSTER_COMPOSITE_OPS:
  -  CLUSTER_COMPOSITE_OPS_RO
  - "indices:data/write/bulk"

CLUSTER_COMPOSITE_OPS_RO:
  - "indices:data/read/mget"
  - "indices:data/read/msearch"
  - "indices:data/read/mtv"
  - "indices:data/read/scroll*"

CLUSTER_MANAGE:
  - CLUSTER_INTERNAL_FEDERATE
  - "cluster:admin/federate/*"
  - "indices:admin/aliases*"

CLUSTER_INTERNAL_FEDERATE:
  - "cluster:internal/federate/*"

# Permissions for the investigate system user
investigate_system:
  cluster:
    - CLUSTER_COMPOSITE_OPS
    - CLUSTER_MANAGE
    - CLUSTER_MONITOR
  indices:
    '?siren*':
      '*':
        - INDICES_ALL
    '*':
      '*':
        - READ
        - VIEW_INDEX_METADATA

# Permissions for a Investigate administrator
investigate_admin:
  cluster:
    - CLUSTER_COMPOSITE_OPS_RO
    - CLUSTER_MANAGE
    - CLUSTER_MONITOR
  indices:
    '*':
      '*':
        - MANAGE
        - READ
        - VIEW_INDEX_METADATA

# Permissions for a Investigate regular user.
investigate_user:
  cluster:
    - CLUSTER_COMPOSITE_OPS_RO
    - CLUSTER_INTERNAL_FEDERATE
  indices:
    'data-*':
      '*':
        - READ
        - VIEW_INDEX_METADATA

<username>:
  <indices or cluster>:
    '<index name or regular expression>':
      '<type name or regular expression>':
        - <list of permissions or action group names>

# Internal user database
# The hash value is a bcrypt hash and can be generated with plugin/tools/hash.sh
sirenserver:
  hash: $2a$12$zMeFc6Xi.pcgDVHsvtCV9ePNteVwTE5uGxcKdf7XQcKB9.VkD8iOy
sirenadmin:
  hash: $2a$12$zMeFc6Xi.pcgDVHsvtCV9ePNteVwTE5uGxcKdf7XQcKB9.VkD8iOy
sirenuser:
  hash: $2a$12$zMeFc6Xi.pcgDVHsvtCV9ePNteVwTE5uGxcKdf7XQcKB9.VkD8iOy

$ bash plugins/search-guard-6/tools/hash.sh -p password

investigate_system:
  users:
    - sirenserver

investigate_admin:
  users:
    - sirenadmin

investigate_user:
  users:
    - sirenuser

<role name>:
  users:
    - <username>
    - <username>

$ bash plugins/search-guard-6/tools/sgadmin.sh \
  -cd config/sgconfig \
  -cn siren-distribution \
  -ts config/truststore.jks \
  -tspass password \
  -ks ../siren-investigate/pki/searchguard/CN\=sgadmin-keystore.jks \
  -kspass password \
  -h localhost \
  -p 9330 \
  -nhnv

$ bash plugins/search-guard-6/tools/sgadmin.sh \
  -rl
  -cn siren-distribution \
  -ts config/truststore.jks \
  -tspass password \
  -ks ../siren-investigate/pki/searchguard/CN\=sgadmin-keystore.jks \
  -kspass password \
  -h localhost \
  -p 9330 \
  -nhnv

Clustername: elasticsearch
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
searchguard index does not exists, attempt to create it ... done
Populate config from /elasticsearch/sg_config
Will update 'config' with sg_config/sg_config.yml
   SUCC: Configuration for 'config' created or updated
Will update 'roles' with sg_config/sg_roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update 'rolesmapping' with sg_config/sg_roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update 'internalusers' with sg_config/sg_internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update 'actiongroups' with sg_config/sg_action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Done with success

$ wget --ca-certificate=../siren-investigate/pki/searchguard/ca.pem --http-user=sirenserver --http-password=password -qO - https://localhost:9220

$ echo | openssl s_client -servername localhost -connect localhost:9220 -showcerts | openssl x509 -text -inform pem -text -noout

Edit config/investigate.yml and specify the credentials of the sirenserver user, for example:

elasticsearch.username: 'sirenserver'
elasticsearch.password: 'password'

If HTTPS is enabled for the Elasticsearch REST API, ensure that the elasticsearch.url setting contains a URL starting with https, for example:

elasticsearch.url: 'https://localhost:9220'

If the certificate is not signed by a public authority, you will also need to set the elasticsearch.ssl.certificateAuthorities to the path of the CA chain bundle in PEM format, for example:

elasticsearch.ssl.certificateAuthorities: 'pki/searchguard/ca.pem'

If you are using the certificates generated by the TLS generator service, the PEM file containing the certification bundles is available in root-ca/root-ca.pem.

To enable certificate verification, set elasticsearch.ssl.verificationMode to full, for example:

elasticsearch.ssl.verificationMode: full

If you want to validate the certificate but not the hostname, set elasticsearch.ssl.verificationMode to certificate, for example:

elasticsearch.ssl.verificationMode: certificate

Set the investigate_core.elasticsearch.auth_plugin option to searchguard:

investigate_core:
  elasticsearch:
    auth_plugin: searchguard

To enable the Siren Investigate access control plugin, specify the following configuration values in the investigate_access_control section:

Example minimal configuration:

investigate_access_control:
  enabled: true
  acl:
    enabled: true
  cookie:
    secure: true
    password: '12345678123456781234567812345678'

Make sure to personalize the session cookie password.

Additional configuration options:

If Siren Investigate is running behind a reverse SSL proxy like Nginx, remember to set cookie.secure to false otherwise the cookie will not be sent, for example:

investigate_access_control:
  enabled: true
  acl:
    enabled: true
  cookie:
    password: '12345678123456781234567812345678'
    secure: false

If you want to use the Siren Alert plugin, you must specify the Siren Alert user credentials in the investigate_access_control.sirenalert section, For example:

investigate_access_control:
  enabled: true
  acl:
    enabled: true
  cookie:
    password: '12345678123456781234567812345678'
    secure: false
  sirenalert:
    elasticsearch:
      username: sirenalert
      password: password

If Siren Alert credentials are not specified, Siren Alert will use the backend credentials to execute the watchers.

Restart Siren Investigate after changing the configuration file; if the configuration is correct, you should see an authentication dialog when browsing to Siren Investigate.

Siren Investigate features an access control system on saved objects that enables you to filter dashboards and visualizations visible to end users; this was enabled in the previous section by setting investigate_access_control.acl.enabled to true.

When the system is enabled, normal users should not have access to the .siren and .sirenaccess indices, as these will be managed by the backend user (.sirenserver).

Siren Investigate includes an optional user interface for the Search Guard REST Management API add-on ; to use it, the Siren Investigate back end has to connect to the Elasticsearch cluster using a PEM client certificate with administrative privileges.

searchguard.ssl.http.clientauth_mode: OPTIONAL

investigate_access_control:
  enabled: true
  acl:
    enabled: true
  admin_role: investigate_admin
  cookie:
    password: '12345678123456781234567812345678'
    secure: false
  backends:
    searchguard:
      admin.ssl.cert: pki/searchguard/CN=sgadmin.crtfull.pem
      admin.ssl.key: pki/searchguard/CN=sgadmin.key.pem
      admin.ssl.keyPassphrase: password

$ cat user.crt.pem ca-chain.pem > user.crtfull.pem

Access control: authentication

After the certificate is set up, restart Siren Investigate, login with a user having an administrative role, click the apps button, then click Access control and finally Authentication.

The Authentication section enables you to browse, edit and create the following Search Guard resources:

• Internal users
• Roles
• Role mappings
• Action groups

To verify that the application is working correctly, click Roles then click Open; you should see the list of roles defined during the initial Search Guard setup or an authorization error if the certificate is incorrect:

If you get an error upon opening the Authentication app, most probably the client certificate does not contain the full CA chain or the add-on has not been installed correctly, check Elasticsearch and Siren Investigate logs for related errors.

If you experience a Siren Investigate crash when opening the application, ensure that the option investigate_access_control.backends.searchguard.admin.ssl.keyPassphrase is set to the correct password.

Access Control: ACL

The ACL Roles panel in the ACL section enables you to define Siren Investigate roles, which are collections of permissions on saved objects and UI elements. The main purpose of this system is to hide and block access to end users and avoid unauthorized changes to configuration objects or use of certain parts of the system:

• UI elements - applications, for example: Timelion, Access control, Siren Alert
• UI elements - specific functionalities, for example: export CSV feature.
• UI elements - Siren Investigate sections, for example: discover, management.
• Saved objects on unauthorized indices, for example: dashboards, searches.

There are two kinds of rules:

• Rules: Set permissions for saved objects.
• UI rules: Set permissions to view different user interface elements.

The everyone role defines permissions for all the users in the system, and is mapped by default to any user logged in Siren Investigate; by default it grants all users read only access to the Siren Investigate configuration (Advanced settings), saved searches and index patterns as well as permission to view all applications and UI elements.

Denying access to certain saved objects like saved search using the first sets of rules is usually transparent to the user which means that for them, those objects are not visible anywhere in Siren Investigate.

Usually it is not required to create explicit UI rules for the dashboard application as access to specific dashboards can be restricted through saved object rules.

Denying access to an application like Timelion or a Siren Investigate section like management will hide the navigation menu elements, block access at the root level and display an error.

When the user tries to access app/timelion, the following error is shown.

When the user tries to access /app/kibana#/management, the following error is shown.

For most setups it makes sense to grant view permissions on visualizations as well, then set specific permissions on dashboards and dashboard groups for each role.

To define a new role, click Create role, then set the following parameters:

• Role ID: The ID of the role (for example sirenuser); must be a lowercase alphanumeric string.
• Backend roles: A list of Search Guard roles that will be mapped to this Siren Investigate role (for example sirenuser).
• Rules: A list of rules on saved object types.

Each rule is defined by three parameters:

• Action: Allow or deny.
• Permission: The permission to allow or deny.
• Context: The saved object type on which the permission must be enforced.

Object permissions

In addition to role level permissions, it is possible to define permissions on specific objects by going to Management → Save Objects and clicking the permissions button next to an object:

The object permissions form enables you to set the owner of the object and custom access rules.

By default, the owner is set to the user that created the object; the owner has all permissions on the created object; it is possible to unset the owner of an object by leaving the field blank and clicking the Save button.

Custom access rules can be used to grant access to an object that would be otherwise hidden; for example, if everyone is not granted to display dashboards but you want to display the Overview dashboard to all users, visit the object permissions form for the Overview dashboard and set the View permission for everyone to Allow.

If everyone can see dashboards but you would like to hide the IT dashboard to users, set the View permission for everyone to Deny.

To enable authentication in Logstash, set the following parameters in the output.elasticsearch section:

For example:

output {
    elasticsearch {
       hosts => ['localhost:9220']
       user => logstash
       password => password
       ssl => true
       truststore => '/etc/pki/logstash/truststore.jks'
       truststore_password => password
    }
}

The truststore file must be copied on all nodes running Logstash.

To enable authentication in a beat which connects directly to Elasticsearch, set the following parameters in the output.elasticsearch section:

For example:

output:

  elasticsearch:
    hosts: ['localhost:9220']

    protocol: 'https'
    username: 'logstash'
    password: 'password'

    tls:
      certificate_authorities: ['/etc/pki/filebeat/ca.pem']

The root certification authority in PEM format must be copied to all nodes running one or more beats.

To successfully submit queries from console to a cluster secured by Search Guard set the following parameters in config/investigate.yml:

console.proxyConfig:
  - match:
      protocol: 'https'

    ssl:
      ca: 'pki/searchguard/ca.pem'

console.proxyConfig.ssl.ca must point to the CA certificate bundle, so it can be set to the same value as the elasticsearch.ssl.ca parameter.

To store monitoring data in a cluster secured by Search Guard it is required to configure agent exporters to submit data over an authenticated HTTPS connection.

The exporter configuration in elasticsearch.yml must include the following parameters:

For example, the following configuration defines an exporter which sends data to the cluster at https://localhost:9220, authenticating as the monitoring user:

xpack.monitoring.exporters:
  id1:
    type: http
    host: ['https://localhost:9220']

    auth:
      username: monitoring
      password: password

    ssl:
      truststore.path: truststore.jks
      truststore.password: password

Create a investigate_system role with the following definition and map it to a sirenserver user:

{
  "investigate_system": {
    "cluster": [
      "cluster:internal/federate/*",
      "cluster:admin/federate/*",
      "cluster:monitor/*"
    ],
    "indices": [
      {
        "names": [
          "/\\.siren.*/"
        ],
        "privileges": [
          "all"
        ]
      },
      {
        "names": [
          "*"
        ],
        "privileges": [
          "indices:data/read*",
          "indices:admin/aliases/get",
          "indices:admin/aliases/exists",
          "indices:admin/get",
          "indices:admin/exists",
          "indices:admin/mappings/fields/get*",
          "indices:admin/mappings/get*",
          "indices:admin/mappings/federate/connector/get*",
          "indices:admin/mappings/federate/connector/fields/get*",
          "indices:admin/types/exists",
          "indices:admin/validate/query",
          "indices:monitor/settings/get"
        ]
      }
    ]
  }
}

If using a custom configuration, replace the configuration index name (.siren by default) and access control index name (.sirenaccess by default) with the correct names.

Set elasticsearch.username and elasticsearch.password to the credentials of the sirenserver user, for example:

elasticsearch.username: sirenserver
elasticsearch.password: password

If HTTPS is enabled for the Elasticsearch REST API, ensure that the elasticsearch.url setting contains a URL starting with https, for example:

elasticsearch.url: 'https://localhost:9220'

If the certificate is not signed by a public authority, you will also need to set the elasticsearch.ssl.certificateAuthorities to the path of the CA chain bundle in PEM format, for example:

elasticsearch.ssl.certificateAuthorities: 'pki/searchguard/ca.pem'

To enable certificate verification, set elasticsearch.ssl.verificationMode to full, for example:

elasticsearch.ssl.verificationMode: full

Set the investigate_core.elasticsearch.auth_plugin option to xpack:

investigate_core:
  elasticsearch:
    auth_plugin: xpack

Then, set the backend parameter of the investigate_access_control section of the investigate.yml to xpack:

investigate_access_control:
  enabled: true
  backend: xpack
  acl:
    enabled: true
  cookie:
    secure: true
    password: '12345678123456781234567812345678'

For a complete description of the options, see .

All users with access to Siren Investigate should have the following role definition:

Example standard user role with access to all indices starting with data- and to all virtual indices starting with db-: 

{
  "investigate_user": {
    "cluster": [
      "cluster:internal/federate/*"
    ],
    "indices": [
      {
        "names": [
          "data-*", "db-*"
        ],
        "privileges": [
          "indices:data/read*",
          "indices:admin/aliases/get",
          "indices:admin/aliases/exists",
          "indices:admin/get",
          "indices:admin/exists",
          "indices:admin/mappings/fields/get*",
          "indices:admin/mappings/get*",
          "indices:admin/mappings/federate/connector/get*",
          "indices:admin/mappings/federate/connector/fields/get*",
          "indices:admin/types/exists",
          "indices:admin/validate/query",
          "indices:monitor/settings/get",
          "indices:admin/template/get"
        ]
      }
    ]
  }
}

For administrative user, ensure you have admin_role configured in the investigate_access_control section in investigate.yml. e.g

investigate_access_control:
  admin_role: investigate_admin

Example administrative user with access to all indices starting with data-, to all virtual indices starting with db-, license management, and permission to manage external datasources and virtual indices starting with db-: 

{
  "investigate_admin": {
    "cluster": [
      "cluster:internal/federate/*",
      "cluster:admin/federate/*",
      "cluster:monitor/*",
      "cluster:admin/xpack/security/*"
    ],
    "indices": [
      {
        "names": [
          "data-*", "db-*"
        ],
        "privileges": [
          "indices:monitor/*",
          "indices:admin/*",
          "indices:data/read*"
        ]
      }
    ]
  }
}

For additional information on datasources configuration, check the section.