Reconciliation API

This section focuses on the reconciliation API. For information about performing reconciliation, see Reconciliation.

Only administrators can run reconciliation.

Endpoint ⇒ POST /api/collections/reconciliation

Parameters

Parameter Type Required Description

entityTableIds

string[]

false

Reconcile document references from specific entity tables. You can use a wildcard to represent "all dataspaces", for example: *:search:9b412ba0-533c-411e-b431-532db5d05260 will select all Entity Tables with that id across all dataspaces - this is useful when dealing with cloned dataspaces.

indexPatterns

string[]

false

Reconcile document references from specific Elasticsearch index patterns

dataspaces

string[]

false

Reconcile document references for a list of dataspaces

If none of entityTableIds, indexPatterns or dataspaces are provided, all document references will be reconciled.

You cannot use more than one of these parameters together, they are mutually exclusive.

Sample requests

A request to reconcile document references belonging to the 'company' index:

curl -H "Content-Type: application/json" -H 'kbn-xsrf: any' -u sirenadmin:password -L -XPOST 'http://localhost:5606/api/collections/reconciliation' -d '{ "indexPatterns": ["company"] }'

A request to reconcile document references belonging to the entity table with an ID of search:9b412ba0-533c-411e-b431-532db5d05260:

curl -H "Content-Type: application/json" -H 'kbn-xsrf: any' -u sirenadmin:password -L -XPOST 'http://localhost:5606/api/collections/reconciliation' -d '{ "entityTableIds": ["search:9b412ba0-533c-411e-b431-532db5d05260"] }'

The same request as above, but reconciling documents across all cloned dataspaces with that Entity Table:

curl -H "Content-Type: application/json" -H 'kbn-xsrf: any' -u sirenadmin:password -L -XPOST 'http://localhost:5606/api/collections/reconciliation' -d '{ "entityTableIds": ["*:search:9b412ba0-533c-411e-b431-532db5d05260"] }'

A request to reconcile all document references in dataspace test:

curl -H "Content-Type: application/json" -H 'kbn-xsrf: any' -u sirenadmin:password -L -XPOST 'http://localhost:5606/api/collections/reconciliation' -d '{ "dataspaces": ["test"] }'

A request to reconcile all document references:

curl -H "Content-Type: application/json" -H 'kbn-xsrf: any' -u sirenadmin:password -L -XPOST 'http://localhost:5606/api/collections/reconciliation' -d '{}'

Response

Reconciliation is a resource-intensive, long-running operation. Once user credentials are verified, Investigate will immediately respond with HTTP status code 202 (Accepted) and an operationId you can use to track its progress.

You should avoid starting multiple simultaneous reconciliation operations. Use the following operations API to wait for a reconciliation operation to complete before starting a new reconciliation.
Property Type Description

operationId

string

A UUID string that identifies the operation.

Sample response

{
  "operationId": "7e990a3a-9872-4157-9bea-6f7b6afe64d0"
}

Endpoint ⇒ GET /api/operations/{operationId}/stream

You can use this endpoint to receive a stream of notifications about the status of the operation represented by operationId. The stream follows the server-sent events specification.

The stream will emit the following events, representing a status object with data encoded as a JSON string:

Event Data type Description

update

ProgressData

Reports a progress update, including the overall completion percentage and a textual description of current objects being processed.

done

ReconciliationResult

Reports the final result of the reconciliation operation.

error

ErrorData

Reports an error in the procedure.

Sample request

curl -N -H "Content-Type: application/json" -H 'kbn-xsrf: any' -u sirenadmin:password -L 'http://localhost:5606/api/operations/7e990a3a-9872-4157-9bea-6f7b6afe64d0/stream'

Sample response

event: update
data: {"text":"Update graphs 122/326","progressPercent":37}

event: update
data: {"text":"Update graphs 257/326","progressPercent":78}

id: end
event: done
data: {"totalRecordsCount":3000,"failedRecordsCount":0,"totalCollectionsCount":3,"failedCollectionsCount":0,"elapsedSeconds":3,"failedRecordsDetails":[],"failedCollectionsDetails":[]}

Endpoint ⇒ GET /api/operations/{operationId}/status

You can poll at this endpoint to get information about the current status of the long-running operation represented by operationId. The returned status object has the following schema:

Property Type Description

status

'running' | 'done' | 'error'

Current status of the operation.

data

ProgressData | ReconciliationResult | ErrorData

The data object associated with the current status.

Sample request

curl -H "Content-Type: application/json" -H 'kbn-xsrf: any' -u sirenadmin:password -L 'http://localhost:5606/api/operations/7e990a3a-9872-4157-9bea-6f7b6afe64d0/status'

Sample responses

{
  "status": "running",
  "data": {
    "text": "Update graphs 122/326",
    "progressPercent": 37
  }
}
{
  "status": "done",
  "data": {
    "totalRecordsCount": 3000,
    "failedRecordsCount": 0,
    "totalCollectionsCount": 3,
    "failedCollectionsCount": 0,
    "elapsedSeconds": 3,
    "failedRecordsDetails": [],
    "failedCollectionsDetails": []
  }
}
{
  "status": "error",
  "data": {
    "status": 400,
    "message": "Any error detail here"
  }
}

Status objects

ReconciliationResult

Representation of the result of a reconciliation operation.

Property Type Description

totalRecordsCount

number

Count of affected collection records

failedRecordsCount

number

Count of affected collection records that could not be reconciled

totalCollectionsCount

number

Count of collections with affected document references

failedCollectionsCount

number

Count of collections with affected document references that could not be reconciled

elapsedSeconds

number

Seconds taken for the reconciliation operation to run

failedRecordsDetails

{ _id: string; _index: string; error: string; }[]

Details of any records that could not be reconciled (first 100 entries)

failedCollectionsDetails

{ _id: string; title: string; error: string; }[]

Details of any collections containing document references that could not be updated (first 100 entries)

ProgressData

An object representing current progress in the operation.

Property Type Description

text

string

Textual explanation of the current moment in the processing pipeline.

progressPercent

number

Completion percentage of the whole operation.

ErrorData

An object representing an error in the operation.

Property Type Description

status

number

HTTP status code representing the error.

message

string

Textual description of the error.