Release Notes

12.0.6

Deprecations

Discover

  • The Discover application has been deprecated and will be removed in a future release. The buttons to create and save child searches from the Discover application have been removed.

Bug fixes

Graph Browser

  • Fixed an issue that prevented sorting graph nodes by count when one or more edges were part of the selection.

  • Fixed an issue that prevented creating edges automatically between numerical fields stored as strings.

Data Model

  • Fixed a crash when opening the Data Model page when a saved search exists without a corresponding index pattern

  • Fixed an issue that prevented editing documents in "Use same index mode" on entity tables pointing to existing Elasticsearch indices.

Record Table

  • The Record Table now refreshes when dashboard is configured to automatically refresh.

  • Fixed an issue that prevented sorting data by a scripted date field.

Dashboard

  • Fixed an issue that prevented nodes from being added to a Graph Browser visualization when dragging a dashboard with a modified state.

  • Fixed an issue that caused a wrong search request to be generated after creating a negated filter having multiple values by holding CTRL/CMD.

  • Fixed an issue that prevented exporting to PDF dashboards containing visualizations with UTF-8 characters.

Record View

  • Fixed an issue that prevented the deletion of a NLP annotation in the record view.

  • Fixed an issue that prevented changes to NLP annotations to be displayed before clicking on the Save button in the record view.

Improvements

Graph Browser

  • Adjusted the formula that determines the size of nodes at high zoom levels to minimize overlapping.

Relational Navigator

  • Modified the relation links to include documents from both the target and the source entity when self relations have the same label in both directions.

Auditing

  • Added a configuration option that allows customization of the Elasticsearch fields to be put in data export audit log entries. Learn more.

Miscellaneous

  • Improved the Elasticsearch periodic health check to wait for the ACL index to be ready in addition to the main Siren Investigate index.

  • Added the investigate_core.search.max_buckets configuration setting. This setting prevents aggregations with a large number of buckets from being processed by visualizations and freezing the browser. The default value of the setting is 1000.

12.0.5

Bug fixes

Data Model

  • Fixed an issue in the data import flow that caused leading zeroes in fields to be automatically stripped.

Auditing

  • Fixed an issue that caused data requests initiated by the Graph Browser to be classified as count requests.

  • Fixed an issue that caused dataExport requests to be assigned to the HOME dataspace instead of the current one.

Graph browser

  • Fixed an issue that could cause a fatal error when switching between two dashboards with a Graph Browser visualization in map mode.

  • Fixed an issue that would cause all the nodes to be expanded when no checkbox was selected in the expansion dialog.

Scripting

  • Fixed an issue that could cause a fatal error when editing a script containing JSX fragments.

Dashboard

  • Fixed an issue that caused a wrong confirmation dialog to appear after deleting a dashboard whilst in edit mode.

Improvements

Data Model

  • Enabled granular editing of relations in the Relations tab. When saving or deleting a relation the changes will now be applied immediately.

  • Improved the performance of the relations tab with a high number of relations.

  • Added a button to aggregate multiple relations between two entities into a single link showing the number of relations. In this mode, the individual relation names are displayed in a tooltip.

Graph browser

  • Links that represent self relations on the same field are now automatically hidden when the source and the target are the same node.

Visualizations

  • Added a configuration option to the Relational Navigator visualization that allows you to group relations having the same label into a single button.

  • Modified the Relational Navigator visualization links to include documents from both the target and the source entity when self relations have the same label in both directions.

  • Removed the automatic capitalization of links in the Relational Navigator visualization.

12.0.4

Security fixes

  • Upgraded Node.js to version 14.19.1. For the full list of fixes, see the 14.19.1 changelog.

Bug fixes

Data Model

  • Fixed an issue that prevented range filters and phrase filters from working correctly when used against scripted fields of type date.

  • Fixed an issue that caused a Mapping Parser exception while trying to add properties to fields of type object in existing entity tables.

  • Fixed an issue in the record view that caused invalid numbers for pagination to be displayed when creating a new record.

  • Fixed issue with dropdown scrollbars having transparent background in the relations configuration tab.

  • Fixed an issue that caused an invalid error to be shown in the editing tab after renaming a field.

  • Restored the functionality of the Visualize button in the fields sidebar.

Dashboard

  • Fixed an issue that caused elements in the record table to overlap when used on a display with a 16:9 aspect ratio.

  • Fixed an issue that caused the annotation modal to be abnormally large when selecting a long text.

  • Fixed links in the record view to open in a new browser tab.

  • Fixed an issue that caused a fatal error when opening a 360 Dashboard with a large number of visualizations.

Visualizations

  • Fixed an issue that caused uneven gaps between the bars in histograms with monthly intervals.

  • Fixed the x-axis tick labels in the Box Plot visualization to have the same format as other visualizations.

Graph Browser

  • Fixed an issue that caused the map mode to be turned off when switching between dashboards.

Miscellaneous

  • Fixed an issue that prevented scrolling in the dataspace selector with a large number of dataspaces.

  • Fixed error reporting in the visual builder default configuration.

Improvements

Auditing

  • Added a configuration option that blocks access to Siren Investigate when an auditing output is unavailable. Learn more.

  • Added a configuration option that allows customization of the Elasticsearch fields to be put in response audit log entries. Learn more.

Access Control

  • Added the ability to set the display name of a user from OpenID Connect claims when using Elastic Stack Security.

Siren Alert

  • When creating an alert, the payload used to preview templates is built using data from the Elasticsearch index rather than sample data.

Scripting

  • When a Siren API script fails to execute a function that is not white listed, an error toast will be displayed at the top of the screen.

12.0.3

Bug fixes

Miscellaneous

  • Fixed an issue that could cause Siren Investigate to get stuck during the startup process if the Elasticsearch cluster crashed.

  • Fixed an issue that would prevent the creation of a dataspace shared with role names containing a - character.

Data Model

  • Fixed an error in the automatic dashboard generation procedure occurring in the presence of fields of type ip.

  • Fixed an issue with the appearance of button tooltips.

Dashboard

  • Fixed an issue that caused the count of documents in a dashboard to be wrong when a filter associated to a different search was enabled.

  • Fixed an issue that caused funnel icon to appear in the sidebar when no filters were applied.

  • Prevented scrolling the dashboard sidebar from changing the scroll position of the active dashboard.

  • Fixed an issue that caused a revision filter to be incorrectly injected when cloning a 360 Dashboard.

  • Fixed an issue that prevented a newly created dashboard from appearing in the sidebar after being created in environments with slow network performance.

Visualizations

  • Fixed an issue that caused the count of documents in a nested slice to be incorrect when enabling the "Other" bucket.

  • Fixed an issue that caused the aggregation legend to appear in Enhanced Tilemap visualizations after disabling the aggregation layer.

  • Fixed an issue that caused a fatal error when clicking twice on the Export button of Record Table visualizations in environments with slow network performance.

  • Fixed an issue that caused the labels on the X axis to appear as undefined when using a Range aggregation.

  • Fixed an issue that caused an error when exporting a Record Table visualization in a dataspace where the advanced setting metaFields was set to null.

  • Fixed an issue that caused an error when exporting a Record Table visualization sorted by a missing field.

  • Fixed an issue that caused an incorrect value to be displayed in the legend when setting a custom label for an aggregation.

  • Fixed an issue that caused the associated search to be hidden in the Record Table configuration panel when a click handler was configured.

  • Fixed an issue that caused persistent error messages during a user session after failure of data retrieval in Controls visualization.

Graph Browser

  • Fixed an issue that could cause a tooltip to remain visible after deleting the corresponding node.

Jira integration

  • Fixed the reporting of authorization errors in the issue selector dialog.

12.0.2

Security fixes

  • Fixed an issue that prevented the access control index from being migrated when upgrading from version 10.5.0 to version 12.0.0 or 12.0.1.

  • Upgraded Node.js to version 14.18.3. For the full list of fixes, see the 14.18.3 changelog.

  • Fixed the Monaco editor configuration to avoid loading parts of the codebase from the jsdelivr CDN.

Bug fixes

Migrations

  • Fixed an issue that prevented upgrading from version 11.0.0 to version 12.0.1.

  • Fixed an issue that prevented upgrading an empty Siren Investigate installation with web services enabled.

  • Fixed an issue that caused filters saved to shared dashboard URLs to not be restored correctly.

Data Model

  • Made the ordering of fields in the target table consistent with the ordering of fields in the source table.

  • Fixed an issue that prevented the save button from being enabled after changing the primary time field or enabling revisions on an entity table.

  • Fixed an issue that caused an error to be displayed when generating a dashboard immediately after creating an entity table.

  • Fixed an issue that prevented field filtering from working in the Search tab field list.

Dashboard

  • Fixed an issue that caused the count of documents to be incorrect in the presence of a filter created from a visualization linked to a search different than the one associated to the dashboard.

  • Fixed an issue that prevented the number of surrounding documents from being displayed.

  • Fixed an issue that prevented a confirmation dialog from being shown when closing a record editing view with unsaved changes.

Visualizations

  • Fixed an issue that prevented horizontal scrollbars from being displayed on the record table.

  • Fixed the Record Table visualization to allow selecting nested fields when exporting data.

Jira Plugin

  • Fixed an issue that prevented results from being displayed when clearing the filter bar.

Internationalization

  • Fixed i18n scripts to be runnable in release builds.

Setup

  • Added the permissions to manage pipelines to the investigate_system role in the Elasticsearch Security initialization script.

  • Added permissions to manage ingest pipelines and templates plus permission to write to data indices to the federate_system role in the Elasticsearch Security initialization script.

Management

  • Fixed an issue that prevented the title of scripts and templates from being displayed in the breadcrumbs of the Management section.

Improvements

Dashboard

  • Improved the Dashboard 360 loading time by optimising the computation of search requests.

  • Dashboard 360 data model pop-ups are now disabled by default to improve performance. To re-enable them you can set the Advanced Setting siren:dashboards360:showDashboardDataModelPopup to true.

  • Added support for ip fields in quick dashboard generation.

Graph Browser

  • Context menus can be closed by pressing the ESC key.

Global Search

  • When entity tables with an external revision index are configured to be searchable, search requests will be executed using the dfs_query_then_fetch mode to ensure that the ranking of edited documents is correct.

Data Model

  • When adding data only writeable tables are displayed.

Dependencies

  • Upgraded React to version 16.14.0.

12.0.1

Bug fixes

Migrations

  • Fixed an issue that caused a migration to fail when the web_service_manager plugin was enabled.

12.0.0

New features

Global search interface

A new Global Search interface is available, which provides instant access to all of the data within Siren Platform’s reach in a single, unified interface. After you find a record of interest, you can view it in more detail. Learn more.

Adding and editing data with revision tracking

You can add or edit records in an existing entity table by using the Record View, which is available in the Global Search interface, in the Graph Browser, in the Record Table visualization, and in the Data model app. Learn more.

Adding links in the Graph Browser

A new contextual function is available in the Graph Browser, which allows you to add or edit links between nodes in the graph.

This allows you to curate the graph for investigations where you are aware of connections between entities that are not displayed. Learn more.

You can also limit the visual clutter in a graph by transforming interconnecting nodes into links (edges) with a newly-developed lens. Learn more.

Importing data in the Data model app

The Data model app has a new interface that provides extensive data import capabilities. It is now a comprehensive location for working with your data; from importing to editing, to adding relations between entities.

You can import data either from a spreadsheet or from a datasource and apply a transform operation by using simple drag and drop functions. The transform can be saved and reused later on new data files that have a similar structure. Learn more.

New features in the Record View

Now, when you expand a record to view its details, the Record View includes the ability to view linked records and to filter records to a dashboard. Learn more.

New linking options for visualizations

If you need to switch the data set that is linked to a visualization, you can select a different entity table or search on the configuration screen.

You can also allow a visualization to be reused on multiple dashboards. Learn more.

Taxonomy browser

In use cases where precision drill-downs are required - for example, when examining technical terms for patents, technical documentation, or scientific literature - Siren Investigate now provides a taxonomy browser.

Along with improved taxonomy annotation capabilities in our built-in natural language processing (NLP), analysts can explore advanced technical and domain-specific datasets and can choose taxonomical categories within a tree visualization.

To add a taxonomy browser to a dashboard, configure and add a Controls visualization with an options tree. Learn more.

Updated terminology in Siren Investigate

'Searches' or 'index pattern searches' are now referred to as entity tables. And entity tables can have subordinates, now called searches. For more information, see the data model documentation and the Glossary.

Compatibility with Avatica for datasource configuration

You can import data into entity tables from Avatica instances that are configured in Siren Federate. Avatica is now the only JDBC framework option that is supported for connecting to datasources and replaces all previous options. Learn more.

Security fixes

Incorrect access control in the the Dev Tools REST API for logged-in users (CVE-2021-43263)

In Siren Investigate versions previous to 11.1.7, the Dev Tools proxy REST API was not checking the permission to view the Dev Tools application. This allowed logged-in users to send requests even without access to the Dev Tools section of the UI. The requests were executed with user privileges. This issue has been fixed in Siren Investigate 11.1.7 and 12.0.0.

Example sic_user and investigate_user roles permissions

The example security configuration for the sic_user and investigate_user roles has been modified to allow read-only access to the siren-import- prefix instead of siren-. This measure prevents access to siren-audit- indices when auditing is enabled and when the indices are configured to store data on the same cluster. It is recommended that an administrator alters the sic_user and investigate_user roles configuration to grant access only to the siren-import- prefix, even if you are not using the auditing feature.

Node.js upgrade

Upgraded Node.js to version 14.18.1. For the full list of fixes, see the 14.8.1 changelog.

Breaking Changes

Internationalization

  • i18n keys prefixed with sirenPlugins.accessControl now must be prefixed with kbn.accessControl instead.

  • i18n keys prefixed with sirenPlugins.ingest now must be prefixed with kbn.ingest instead.

Data import

  • The support for import templates that created Siren Investigate scoped indices has been removed. Entity tables that point to existing scoped indices will work as before but, in order to add new data, you will need to grant write permissions to these indices in the Elasticsearch security configuration.

Graph Browser

  • The signature of the expandByRelation sirenAPI function is now updated. The node ids and relation ids must be passed as parameters of the options object as expandByRelation({ nodeIds, relationIds}).

  • The index-pattern saved object type is now deprecated. If you created lenses to process relations obtained through the f.getKibiRelations method and you need to retrieve the Elasticsearch index pattern that is associated with a search, you can call relation.range._objects.savedSearch.getIndexPattern().

Visualizations

  • It is no longer possible to create new Coordinate Map visualizations. Existing visualizations remain functional, however, it is recommended that you recreate them as Enhanced Coordinate Map visualizations to avoid technical issues.

Deprecations

Search Guard admin certificate

Support for using a local TLS admin client certificate to send requests from the Search Guard configuration UI has been deprecated and will be removed in Siren Investigate version 13.0.

To prepare for this change and also to improve the security of your existing installation, you will first need to declare the roles that are allowed to access the Search Guard management REST API in the elasticsearch.yml file of all the nodes in your cluster. For example, add the following line:

searchguard.restapi.roles_enabled: ["investigate_admin"]

Next, remove the following options from the investigate.yml configuration file and the files that are defined by them from the computer where Siren Investigate is installed:

  • investigate_access_control.backends.searchguard.admin.ssl.cert

  • investigate_access_control.backends.searchguard.admin.ssl.key

  • investigate_access_control.backends.searchguard.admin.ssl.keyPassphrase

To validate the change, restart both the Elasticsearch cluster and Siren Investigate, go to the Authentication tab of the Access control app, and ensure that the list of users or roles is displayed.

Index pattern service deprecation

The indexPatterns service is now deprecated and should not be used. Instead of fetching indexPattern saved objects, the savedSearches service should be used to fetch savedSearch saved objects.

Each 'savedSearch' object contains all of the methods that used to be available in an indexPattern object. To learn more about all of the available methods, see the following file: src/core_plugins/kibana/public/discover/saved_searches/_saved_search.js.

The deprecated service will be removed in a future release. For support with migrating your plugin, go to the Siren Support Portal.

REST API

  • The /investigate-access-control/api/v1/acl/object-permissions/objects route has been deprecated and will be be removed in the next major release.

  • Experimental support for Kable and Timelion in Siren Alert has been deprecated and will be removed in a future release.

Custom watchers

  • The second argument for custom watcher conditions, which contains the raw mappings of the Elasticsearch index that the current dashboard is pointing to has been deprecated and will be removed in a future release.

Script types: contextual and onGraphUpdate

  • The contextual and onGraphUpdate script types are deprecated and are no longer supported as of the next major release. All of the features that were provided by the built-in contextual and onGraphUpdate scripts are now part of the core Graph Browser codebase. You can now write custom scripts by using the facilities that are provided by the Scripting API.

Improvements

Backup and restore

  • Backups of system indices taken in version 12.0.0 and later will be restored to the indices specified in investigate.yml to simplify the creation of new Siren Investigate instances. Backups taken in previous releases will be still restored to the index they were created from.

Data Model

  • Reduced the number of buckets in aggregations used by the dashboard generation wizard from 100 to 20 to avoid performance issues on large indices.

Visualizations

  • Removed the ability to sort columns on fields of type text that do not have a keyword subfield in Records Table visualizations.

  • Added the ability to sort columns on fields using a tag formatter.

Graph Browser

  • Nodes are shown as images automatically for entity tables that have a Default image set in the Data model app.

  • Added missing tooltips to toolbar buttons.

  • Fixed some keyboard shortcuts that were not working properly (spacebar and numeric pad keys).

  • Improved the sizing algorithm for nodes in Map mode.

  • Improved the rendering performance of Time mode.

Scripting

  • The scripts editor is now based on the Monaco open source library.

  • Added axios to the list of libraries that can be enabled through the siren_scripting.librariesWhitelist configuration setting.

Saved objects

  • When you import saved objects from a previous version of Siren Investigate, a confirmation dialog is displayed that reminds the user to execute migrations.

  • Importing objects from future Siren Investigate releases is now forbidden.

  • It is now forbidden to delete index pattern saved objects from the Saved Objects in the Management app.

  • The breadcrumbs now show the saved object title instead of its UUID.

Bug fixes

Dashboard

  • Fixed an issue that would cause an extra count query in certain cases.

  • Fixed an issue that prevented the Time filter widget from appearing automatically on dashboards after changing the underlying entity table to be time-based.

Enhanced Coordinate Map

  • Fixed an issue that prevented the removal of layers from the map when zooming to a location without points of interest.

Graph Browser

  • Fixed an issue that caused time properties that were set by a lens not to be restored when deactivating the lens.

Data Model

  • Fixed an issue that could cause the deletion of an entity table when canceling the automatic generation of a dashboard.

Jira integration

  • Fixed the pixel ratio for images attached to issues on hi-dpi screens.