Auditing user sessions
If you want to perform internal audits, you can enable the session audit feature. This feature allows you to log the actions that are performed by the logged-in user.
Configuring session logs
To enable the session audit feature, in the
investigate.yml file, set the
enabled parameter to
true in the following code:
siren_audit: enabled: true # The value is false by default. temporary_logs_folder: './optimize' # The folder where the system can store the unsaved entries during shutdown or restart. # The default value is the './optimize' folder. logged_types: # The entry types that are logged. - ui # User actions, such as opening a dashboard and exporting a graph - search # Elasticsearch search or msearch data requests. - count # Elasticsearch search requests used for fetching various counts where no documents are retrieved size= 0. - savedObject # Operations on Investigate configuration objects, such as dashboards and visualizations. - auth # Log in and log out operations. - security # Security related evens, for example, change of object ownership. - dataExport # JSON and CSV data exports operations. - dev # Elasticsearch requests issued from the Dev Tools console. - systemSearch # Elasticsearch search requests internal to Investigate, for example, ping to check Elasticsearch health. logged_saved_object_types: # By default, enable logging on all saved object types. - graph - relational-graph - visualization - ontology-model - config - dashboard - dashboardgroup - eid - relation - template - datasource - script - search - url - doc - fingerprint - mixed-state-docs - sirenapiscript - sidebaroptions logged_saved_object_body_types:  # Types of saved objects to log the body, before and after operation. By default, the value is empty. # You can use any of the types that are listed under `logged_saved_object_types`. # Note: Be aware that the 'graph' type can contain sensitive information in the body. # The body of a graph can be very large, depending on the number of nodes it contains. # For example, 100 nodes is considered a small object, 5k nodes is a large object, and 20k nodes is a very large object. log_response_body: false # If the value is set to 'true', the system logs the response body for Elasticsearch search requests and saved objects requests (excluding responses that contain the body of the object). log_header_names:  # Array of header names to be logged in addition to all headers with "siren-" prefix which are always logged when audit is enabled log_remote_ip_from_header_name: 'my-header' # By default Investigate will check for x-forwarded-for header and extract the first ip address from it # Sometimes a non standard header is used to store the client ip address # In such case this property allows to specify such header name wait_for_outputs_initialization: false # When set to true, Investigate will not start until all configured outputs are initialized disable_when_outputs_down: false # When set to true, Investigate will redirect to status page if any of configured outputs are down # If you do not define the outputs, the system uses the main Elasticsearch cluster to store audit entries. # It is possible to configure more than one remote Elasticsearch output. outputs: - elasticsearch: manage_template: true # default true max_retries: 3 # default 3 retry_delay: 1000 # default 1000 ms queue_size: 1000 # default 1000 flush_interval: 10000 # default 10000 ms flush_size: 50 # default 50 index_prefix: 'siren-audit-' # default siren-audit- index_interval: daily|weekly|monthly # default monthly number_of_shards: 1 # default 1 number_of_replicas: 1 # default 1 url: https://host:port username: username password: password ssl: verificationMode: none | certificate | full # default full certificateAuthorities: [ "/path/to/your/CA.pem" ] certificate: /path/to/your/client.crt key: /path/to/your/client.key keyPassphrase: changeme healthCheck: delay: 5000 timeout: 60000 # When set, it is written to the instanceName field of audit entries # Useful when multiple Investigate instances are configured to log audit entries to the same Elasticsearch cluster investigate_core.instance_name: "my-instance-01"
Session log indices
After the feature is enabled, the system automatically creates an index that stores the audit entries.
If you use the default value of
elasticsearch as the
output type, the system creates the index by using the output configuration credentials.
If no output is defined, the system creates the index on the main Elasticsearch cluster by using the Siren Investigate System credentials.
Make sure that either the remote cluster user or the Siren Investigate Administrator user has the following security permissions:
Can store an index template.
Can create indices with a name starting with the configured prefix, for example, the default prefix is
Can write to those indices.
Session log entries
The system can log the following types of session data:
ui: Actions that are performed by the user as they are interacting with the application. For example, a log entry is created when a user opens a dashboard.
savedObject: Access to system configuration objects. For example, a log entry is created when a user requests a configuration object for particular dashboard.
search: All data-related search queries. For example, a log entry is created when a visualization on a dashboard is fetching the data to render.
count: All queries that obtain the document count. For example, a log entry is created when a query is used to fetch the count of documents visible on a dashboard.
dataExport: A log entry is created each time a user attempts to export data as JSON or CSV.
dev: A log entry is created each time a user send request from Dev tools console.
systemSearch: All internal data-related search queries. For example, an internal log entry is created when an investigate system periodically issue a request to fetch Investigate license information.
log_response_body option is set to
true, you can customize the source fields logged from Elasticsearch search response hits by setting the
log_response_filters option to a list of filters.
A filter is defined by the following attributes:
type: if set, the filter will only apply to that type of session data. It can be set to either
data-export. Defaults to
configuration.includes: a list of field names to include in the audit log entry.
configuration.excludes: a list of field names to exclude from the audit log entry. If a field is excluded it will not be logged even if specified in the
configuration.pattern: if set, the filter is only applied to hits from a specific Elasticsearch index pattern. Defaults to
configuration.excludes support the wildcard matching using the
Excluding or including a field will automatically exclude and include any field nested within that field.
If no filter is defined for an index, all of the fields in hits from that index will be logged.
Excluding the fields
owner from any index
siren_audit: enabled: true log_response_body: true log_response_filters: - configuration: excludes: - amount - owner
Including only the fields
label in hits from the
siren_audit: enabled: true log_response_body: true log_response_filters: - configuration: pattern: company includes: - owner - label
metadata.tags field and all of its children in any hit
siren_audit: enabled: true log_response_body: true log_response_filters: - configuration: excludes: - metadata.tags
Exclude the children of
metadata.tags in any hit
siren_audit: enabled: true log_response_body: true log_response_filters: - configuration: excludes: - metadata.tags.*
Avoid logging fields for any hit coming from the
siren_audit: enabled: true log_response_body: true log_response_filters: - configuration: pattern: 'transactions' excludes: - '*'
Including only the
title field from the article index in a data export
siren_audit: enabled: true log_response_body: true log_response_filters: - type: 'data-export' configuration: pattern: article includes: - title
Including only the fields from the article index in a data export that end with the substring
siren_audit: enabled: true log_response_body: true log_response_filters: - type: 'data-export' configuration: pattern: article includes: - '*label'
Note: If your expression starts with
*, you must wrap it in single quotes.
If you have configured
log_response_body is set to
true, session data from a Data export will be logged to the audit instance in batches of
100 records. This is configurable under the following parameter
siren_audit: enabled: true log_response_body: true log_export_batch_size: 200 # The value is 100 by default. The range is 1 - 2000.
In the event that the output remote cluster is down, the system can accumulate the log entries in a back-end, in-memory queue.
If the queue reaches its maximum size, the system starts to lose the oldest audit entries.
To avoid data loss, system administrators must actively monitor the health of remote audit clusters to keep them online.
You can disable access to Siren Investigate when one of the audit outputs is unavailable