Setting up document editing

Siren Investigate provides the ability to make manual changes in the documents of your index pattern searches and store them inside an additional 'revision index'.

This feature helps in the process of data curation by providing a way to manually correct small errors in some documents and test the fixes immediately. The applied changes are recorded in the revision index, so they can be analyzed later by a data curator, and they can be applied at scale during the data set’s ingestion process.

Enabling document editing

Before you enable document editing, make sure that there is a primary key field that can be used to uniquely associate a document to its revisions. Siren Investigate does not support editing records of indexes that do not have a primary key field.

  1. Go to the Data model app.

  2. Select the Revisions tab.

  3. Select Enable document editing.

Revision Settings

First, set up the revision index to host your edited documents. In the Revision index field, enter a name and click Create Index. If a revision index already exists, you can select it by typing its name in the field.

Create Revision Index

If an index pattern contains only one index (without , or *), you can use the original index as a revision index and make the changes to it directly. To do that, in the Revision index field, enter the exact name of the original index (with matching case) and click Update Index.

Update Original Index

Next, in the Select primary key field, select the primary key of your index pattern search that will be used for tracking documents.

Revisions Primary Key

In the Editable Fields section you can select the document fields that users will be allowed to edit. Select whether you want to Allow or Forbid some fields, then enter the fields themselves in the apposite text area below.

Revisions Editable Fields

Click Save in the top toolbar.

Filtering modes

In the Filtering mode section, you can select a method for tracking which documents were edited. The method that you select affects the revision filter that is responsible for replacing the original documents with the revised ones, and this has both performance and technical implications.

Revisions filtering mode

Filter by primary key: The primary keys of edited documents are stored explicitly in the revision filter, which grows in size with the number of revised documents.

  • Does not affect your original documents

  • Does not scale to many revised documents

Flag original document: An additional metadata field is added to your original index pattern search. Editing a document changes the metadata to indicate that it was replaced by a document in the revision index.

  • Adds a metadata field to your original documents

  • Scales well with many revised documents

You can switch between filtering modes at any stage, however, it is recommended that you start by selecting Filter by primary key and test the revisions with this setting. You can change the mode to Flag original document when the revision index contains approximately one hundred documents.

Configuring security for revision support

Server user configuration

To allow end users to work with revisions, the Siren Investigate back-end user must be granted the following permissions:

  • Full access to the indices that store revisions, for example, article_revised.

  • Permission to view the mappings for all indices that you want to enable revisions for, for example, article.

The following is an example investigate_system role definition for Elastic Stack security that gives the back-end user full access to all index names that end with _revised. It also grants the permission to view the mappings of the article index:

{
  "cluster": [
    "cluster:internal/federate/*",
    "cluster:admin/federate/*",
    "cluster:monitor/*",
    "manage_index_templates"
  ],
  "indices": [
    {
      "names": [
        "/\\.siren.*/",
        "/siren-.*/",
        "/.*_revised/",
        "/watcher.*/",
        "/web-service-.*/"
      ],
      "privileges": [
        "all"
      ]
    },
    {
      "names": [
        "article"
      ],
      "privileges": [
        "view_index_metadata"
      ]
    }
  ]
}

The following is an analogous investigate_system role definition for Search Guard:

investigate_system:
  cluster_permissions:
  - CLUSTER_COMPOSITE_OPS
  - CLUSTER_MANAGE
  - CLUSTER_MONITOR
  index_permissions:
  - index_patterns:
    - 'siren-*'
    # Grant the back-end user full access to all indices for which the name ends with "_revised".
    - '*_revised'
    - '?siren*'
    - '?map__*'
    - 'watcher*'
    - web-service-*
    allowed_actions:
    - INDICES_ALL
  # Grant the back-end user permission to read the mappings of every index pattern for which you want to enable revisions.
  - index_patterns:
    - 'article'
    allowed_actions:
    - VIEW_INDEX_METADATA

End user configuration

To allow end users to view the edited documents, you must grant at least one of the associated roles read access to the indices where the revisions are stored.

The following is an investigate_user role definition for Elastic Stack security that grants read access to both the article index and the article_revised index.

{
  "cluster": [
    "cluster:internal/federate/*",
  ],
  "indices": [
    {
      "names": [
        "article",
        "article_revised",
      ],
      "privileges": [
        "read",
        "view_index_metadata"
      ]
    }
  ]
}

The following is an investigate_user role definition for Search Guard that grants read access to both the article index and the article_revised index.

investigate_user:
  cluster_permissions:
  - CLUSTER_COMPOSITE_OPS_RO
  - CLUSTER_INTERNAL_FEDERATE
  index_permissions:
  - index_patterns:
    - 'article'
    - 'article_revised'
    allowed_actions:
    - READ
    - VIEW_INDEX_METADATA

To allow a user to edit documents, open the ACL permissions page of your saved object search and select the available options from the dropdown lists.

Saved Search ACL Revise Column

Editing documents

After document editing for a index pattern search is set up, all document tables and Record Table visualizations on the index pattern search contain additional controls to edit documents.

For more information, see the Editing documents section of the Record Table visualization topic.