Release Notes

11.0.1

Breaking changes in Investigate 11.0.1

There are no breaking changes in this release. For a list of breaking changes in Siren Investigate version 11.0.0, see this section.

New features and improvements

Data Model

  • Improved the responsiveness of the icon picker in the Data Model management section.

Security

  • Added an ACL context to hide the expensive query section in the configuration form of index pattern searches.

Visualizations

  • The Controls visualization dropdown menu is no longer restricted by the visualization bounds and it is possible to lay out widgets horizontally.

  • Improved the responsiveness of field selectors in filters and visualization configuration panels.

  • It is now possible to set the bounds of sliders in the Controls visualization to be inclusive or exclusive.

Dashboard

  • Improved the loading performance of the dashboard by fetching only displayed visualizations.

  • Improved the readability of the sidebar dashboard tooltips.

Bug fixes

Data Model

  • It is no longer possible to accidentally save pinned filters to index pattern searches.

  • Fixed an issue that could cause a death screen when displaying the relational model graph.

  • Fixed an issue that could cause a death screen on Firefox when updating field fingerprints.

  • Fixed an issue to check that the back-end user has access to the index as soon as the flag to enable revisions on documents in the original index is enabled.

  • Fixed the positioning of other elements on the page when collapsing the Data Model sidebar.

Dashboard

  • Ensured that a dashboard entry is visible while being dragged across the page.

  • Fixed an issue that could cause pinned filters to disappear when disabled.

  • Fixed an issue that would cause the record details modal to close whenever a column was added to the underlying table.

  • Fixed an issue in the sidebar so that it now gives feedback about pruned joins and failures in count requests.

  • Fixed an issue in Dashboard 360 that would prevent the dashboard time range from being applied.

  • Fixed an issue to ensure that groups on the sidebar remain closed by default right after cloning a dataspace.

Enhanced Coordinate Map

  • Fixed an issue where the leaflet library was not loaded correctly with certain configurations.

Visualizations

  • Fixed a death screen that would occur when editing a visualization that requires a time field if the visualization is associated to an index pattern search without a time field set.

  • Fixed an issue that incorrectly allowed a user to be able to save the Control visualization without specifying all the required configuration parameters.

  • Fixed an issue that could cause the Controls visualization to create an empty filter when clearing selections in dropdowns.

  • Fixed an issue in the Controls visualization to ensure that other controls are brought back upon clearing the form and clicking Apply changes.

Scripting

  • Fixed an issue that prevented scripted panel events from being triggered when changing the time range of the dashboard.

Graph Browser

  • Fixed an issue that would prevent the Graph Browser from working correctly when a user had no access to a dashboard.

  • Addressed situations in which the tooltips on Graph Browser nodes might persist after moving the mouse cursor away.

  • Fixed an issue that could cause a death screen when dragging a dashboard on the Graph Browser.

  • Ensured that saved search filters are applied when finding the shortest path between nodes on the graph.

  • Restored the ability to use bitmap images from external URLs in lenses.

Miscellaneous

  • When switching to another dashboard from the relational navigator, the group containing the dashboard is now expanded automatically.

  • Restored the preview widget in the Templates management section.

  • If multiple users try to edit the order of elements in the sidebar at the same time, Siren Investigate will display a conflict error message.

  • Added a stricter validation check to numeric fields in the Advanced Settings management section.

  • Removed the deprecated datasource saved object type.

  • Fixed a delay in the rendering of the Saved Objects management section.

  • Fixed a death screen that could occur when selecting a data source table.

  • Fixed an issue to ensure that the user gets feedback if Siren Investigate gets a 429 HTTP status code from Elasticsearch.

  • Fixed an issue that prevented changes to the notifications lifetime setting to be applied immediately.

  • The backup and restore commands will now report an error correctly when the Elasticsearch credentials are invalid.

  • Fixed an issue that could cause a death screen after importing an extremely small CSV through the Data Import application.

  • Fixed an issue in the documentation of Elastic Stack security to include missing section links.

11.0.0

New features and improvements

New core features

  • Dataspaces: Dataspaces are generic partitions that can be used to create independent Siren environments for different user groups, such as for a multi-tenant environment or to handle different projects. You can upload CSV and Excel files to a specific dataspace in a way that is secure and accessible only to other roles that have access to that dataspace.

  • Jira integration: The new Jira integration allows you to connect Siren Investigate to a Jira cloud or server instance. You can keep track of assigned tickets and export dashboard information as attachments to the tickets, directly from Siren Investigate. Key features include searching for tickets, selecting an active ticket, and exporting dashboard snapshots.

  • IBM® i2® Analyst’s Notebook export: You can export a graph from the Graph Browser with all lenses applied and download it as an ANB graph file. The ANB graph file can be imported into IBM i2 Analyst’s Notebook.

  • Siren NLP plugin for Elasticsearch: The Siren NLP plugin provides an ingestion pipeline with a variety of processors for enriching documents with entity extraction. You can use the plugin to enrich text fields with annotation for named entities - such as an organization, person, or location - and predefined taxonomies.

New sidebar implementation

The sidebar has been reimplemented from the ground up to provide a snappier editing experience and lay the foundation for major features in the next releases.

New security features

  • Support for shared indices: When you define templates in the Data Import application, it is possible to reuse a single Elasticsearch index across multiple dataspaces by partitioning its contents by dataspace code. When this mechanism is enabled, users will only see documents that are imported in the current dataspace.

  • Support for OpenID Connect with Elastic Stack security: You can now integrate Siren Investigate with the OpenID Connect authentication support that is provided by Elastic Stack security.

New features in the Record Table visualization

  • Document editing: You can now enable support for document editing for index pattern searches and edit or add documents by using the Record Table visualization. Document revisions are stored in separate indices and are transparently overlaid on top of the original data.

  • Cell formatters: You can customize the appearance and behavior of column cells by applying cell formatters such as tags, NLP, and click handlers.

New features in the Enhanced Coordinate Map

  • Marker clustering: Marker clustering is a method that allows documents belonging to a point layer to be represented at once on the current map canvas. It is used by Point of Interest layers and Stored Layer sources.

New dashboard features

  • You can define whether the maximum join cardinality limit is applied to either the source dashboard, the target dashboard, or both dashboards that are involved in the join.

Other improvements

  • When a user makes changes to a saved object, other users will get a notification to reload the application.

  • When a user tries to upload a CSV/Excel file that does not match an import template structure, they will get a warning about mismatching columns.

  • Added support for the format parameter in date range filters.

  • Removed a dependency on native canvas libraries from the back-end code.

  • Removed an obsolete prompt about security configuration from the upgrade procedure.

  • Added validation of the maximum zoom level for the Enhanced Coordinate Map set in investigate.yml.

  • The performance of the validation of star patterns at runtime has been significantly improved.

Breaking changes in Investigate 11.0.0

Reduced the number of permissions required by the back-end user

  • Features such as custom icon packs have been fixed to ensure that it is never a requirement to give end users access to the .siren index. If your security configuration has users with access to the .siren index, it is advisable to remove such permission after upgrading to Siren Investigate 11.0.

  • It is no longer required to give the investigate_system role access to all of the data indices, unless you want to enable revision support on them.

Upgraded Angular to 1.8.0 and jQuery to 3.5.1

For more details about the security issue addressed by this jQuery upgrade, please refer to the release notes of jQuery 3.5.0.

In jQuery version 3.5.0 or later, Angular templates with self-closing tags, such as <input/>, are no longer supported. If you have developed any custom plugins that contain templates with self-closing tags, they should be replaced by explicit opening and closing tags, for example, <input></input>.

By default, Siren Investigate 11.0 enables a backwards-compatible mode for jQuery that still allows you to use self closing tags to simplify the transition for plugin developers; when a self closing tag is detected, a warning will be logged to the browser console.

This mode can be disabled by setting the following options in investigate.yml:

optimize.jqueryLegacyPrefilterEnabled: false
optimize.jqueryMigrateEnabled: false

This compatibility mode will be turned off by default in Siren Investigate 11.1.

Scripting

It is now required to explicitly whitelist the browser APIs and third-party libraries that can be used in scripts. This can be done by specifying both lists in the `investigate.yml`file, as shown in the following example:

siren_scripting:
  enabled: true
  browserApiWhitelist:
  - 'console'
  librariesWhitelist:
  - 'lodash'
  - 'moment'

Graph Browser

  • Expansion scripts are removed from the Graph Browser in favor of using a custom sirenAPI event, graph-expansion.

Other breaking changes

  • Removed the bulk_count method from the saved objects API.

  • Removed support for the deprecated external query filter.

  • The option to set the 'Icon URL' in the dashboard group editor is no longer available. To use personalized icons, you can add custom icon packs into Siren Investigate.

  • The mapping of the invocation index contains the exact field types for inputs that are specified in the inputSchema of the Web service driver. The fielddata parameter for these fields is no longer set.

  • FieldSelect component has been deprecated and replaced by FieldSelectResponsive. It will be removed in future releases.*

Security fixes

Bug Fixes

Enhanced Coordinate Map

  • Addressed an issue that could prevent embedding Siren Investigate into an iframe when displaying an Enhanced Tilemap visualization.

  • When dragging an Enhanced Coordinate Map, the time filter in Elasticsearch request stays consistent.

  • Addressed an issue that could cause requests to Elasticsearch to contain an excessive number of buckets.

  • Addressed an issue that could prevent loading tiles from custom tile servers configured in the investigate.yml file.

  • Addressed incorrect rendering of aggregations on the Enhanced Coordinate Map when using certain precision settings.

  • Fixed an issue that would show stale data on aggregation tooltips in certain cases.

  • Prevented situations in which invalid filters were generated by the geo shape polygon tool.

  • Fixed an issue where a time filter was incorrectly applied to overlay layers.

  • Fixed an incorrect item displayed in the legend at specific zoom levels.

  • Fixed the filtering slider inside the legend.

Data Model

  • Fixed an issue that could cause icons from some FontAwesome families to not appear in the Data Model graph when opening the application for the first time.

  • Addressed an issue that would cause relation names to overflow the dropdown in the relational configuration editor.

  • Addressed an issue that caused arrows with an incorrect direction to be displayed in the Data Model graph.

  • Addressed an issue that would cause the Data Model graph layout to be incorrectly reset after editing relations.

  • Prevented saving incomplete relations in the relational configuration.

  • Allow hiding EID relations in the relational navigator visualization configuration.

  • Forbidden text fields from being selected as primary keys.

  • Fixed an issue that prevented creating child searches containing a filter with a range query.

Visualizations

  • Fixed incorrect rendering of the legend labels inside Multichart visualizations when enabling custom legend positions.

  • Addressed an issue that prevented the selection of certain options in the Heatmap visualization configuration.

  • Fixed an issue that would cause unnecessary delays in dashboard rendering when a Scatter Plot visualization was put on it.

  • Fixed issues in the export of CSV data from the Records Table when the title contains UTF-8 only characters.

  • Addressed an issue that prevented exporting a Record Table to CSV format when the underlying index has custom date formats.

  • Fixed an issue in the Timeline visualization when creating range filters on indices with date fields that have custom formats.

  • Made the Timeline Visualization compatible with Dashboard 360.

Dashboard

  • Addressed an issue that was preventing the cardinality limit from working when disabling a filter.

  • Addressed an issue that prevented restoring a shared link after logging in.

  • Addressed an issue that caused the time range selector to disappear when editing Data Model options for a dashboard.

  • Addressed an issue that would trigger an error notification when switching from a dashboard on a time based saved search to a dashboard on a non time based saved search.

  • Fixed an issue that was causing unnecessary extra requests when entering an invalid search query in a dashboard with a Topic Clustering visualization.

  • Addressed an issue that would cause a notification at the top of the page to not automatically resize after closing the dashboard sidebar.

Dashboard 360

  • Fixed an issue where an endless spinner would be displayed instead of the count of documents on 360-enabled dashboards in certain cases.

  • Addressed an issue that would cause some self relations to be hidden in the Dashboard 360 data model.

  • Addressed an issue that could cause persistent error messages when removing a node from the Dashboard 360 data model.

  • Addressed an issue that caused queries that were too restrictive to be sent from 360-enabled dashboards when visualizations using child searches were configured.

  • Fixed some cryptic errors caused by the presence of deleted searches in Dashboard 360 configurations.

  • Fixed an issue that could cause a death screen when applying certain filters.

Graph Browser

  • Addressed memory leaks.

  • Fixed an infinite recursion in Firefox when opening a Graph Browser visualization.

  • Improved the rendering of numbers in histogram cards series.

  • Allow users to select text inside of histogram cards.

  • Addressed an issue that would cause the configuration of the Time/Location lens configuration to not be persisted correctly.

  • Fixed an issue that prevented the Halo option from working correctly.

  • Adjusted the "Select - By Edge count" Graph Browser script to take into account the currently selected nodes.

  • Fixed an issue that prevented adding documents from dashboards to the Graph Browser in presence of broken saved searches.

  • Fixed an issue that would cause dates to be displayed incorrectly in Graph Browser tooltips.

  • Resolved superfluous warnings when expanding nodes on the Graph Browser using a user with limited permissions.

Siren Alert

  • Optimized Siren Alert not to use the Siren Federate endpoint when the query does not contain semijoins.

  • Ensured that Siren Alert does not schedule alerts during upgrades.

  • Added ellipsis to long watcher names in the Siren Alert application.

  • The "New Watcher" button is hidden on dashboards when access to the Siren Alert application is denied.

  • Removed the obsolete setting sentinl.api.type from Siren Alert.

  • Fixed an issue preventing index name suggestions when creating a new watcher

Miscellaneous

  • Fixed an issue that would cause Siren Investigate to not load correctly when scripting was disabled.

  • Fixed an issue with the back-up command that caused credentials to not be properly encoded when they contain special characters.

  • Fixed an issue that could cause a one-minute delay when opening the Discover application for the first time.

  • Addressed an issue that would skip validation of saved searches at runtime in certain cases.

  • Addressed an issue that prevented scrolling the sidebar automatically when dragging a group or a dashboard.

  • Addressed bugs that required giving read access to the .siren index to normal users.

  • Fixed an issue that prevented displaying Index Pattern objects in the Saved Objects management page in certain cases.

  • Addressed an issue that could cause incorrect order of dashboards in the sidebar.

  • Addressed an issue that could cause a query error when sending a search to index with more than 1024 fields and no default field set.

  • Adjusted some inconsistencies in the dark theme.

  • Fixed an issue that would prevent Siren Investigate from restoring connectivity to a red Elasticsearch cluster in certain cases.

  • Disable session keepalive automatically when using OpenID Connect as the authentication mechanism.

  • Prevented the creation of profiles for variable-response Web services.

  • The response status is now displayed when a sample Web service invocation fails.

  • Ensured that the currentVisualization variable is always available in scripting contexts, even if undefined.

  • Added extra validation checks to the restore command to prevent accidental deletion of the .siren index.

  • Fixed an issue that was preventing the automatic dashboard generator from working on newly-created saved searches.

  • Fixed an issue where the Siren API method getTimeFilter was not returning the currently selected time range.