Refresh Fields API

Use this API to detect and process newly added fields in your Elasticsearch indices. For more details, see Refreshing the index fields list .

Endpoint ⇒ POST /api/entity_tables/refresh_fields

Parameters

Parameter Type Required Description

entityTableIds

string[]

false

Refresh fields for specified entity tables. You can use a wildcard to represent "all dataspaces", for example: *:search:9b412ba0-533c-411e-b431-532db5d05260 selects all entity tables with that id across all dataspaces - this is useful when dealing with cloned dataspaces.

indexPatterns

string[]

false

Refresh fields for entity tables from specific Elasticsearch index patterns.

dataspaces

string[]

false

Refresh fields for all entity tables from a list of dataspaces.

computeExpensiveFieldProperties

boolean

false

Recompute unique value and single value field properties. Defaults to true.

If none of entityTableIds, indexPatterns or dataspaces are provided, all entity tables across all dataspaces will have their fields refreshed.

You cannot use more than one of these parameters together, they are mutually exclusive.

Sample requests

A request to refresh fields for an entity table with an ID of search:9b412ba0-533c-411e-b431-532db5d05260:

curl -H "Content-Type: application/json" -H 'kbn-xsrf: any' -u sirenadmin:password -L -XPOST 'http://localhost:5606/api/entity_tables/refresh_fields' -d '{ "entityTableIds": ["search:9b412ba0-533c-411e-b431-532db5d05260"] }'

The same request as above, but refreshing fields for that entity table across all cloned dataspaces:

curl -H "Content-Type: application/json" -H 'kbn-xsrf: any' -u sirenadmin:password -L -XPOST 'http://localhost:5606/api/entity_tables/refresh_fields' -d '{ "entityTableIds": ["*:search:9b412ba0-533c-411e-b431-532db5d05260"] }'

A request to refresh fields for all entity tables belonging to the 'company' index:

curl -H "Content-Type: application/json" -H 'kbn-xsrf: any' -u sirenadmin:password -L -XPOST 'http://localhost:5606/api/entity_tables/refresh_fields' -d '{ "indexPatterns": ["company"] }'

A request to refresh fields for all entity tables in the test dataspace:

curl -H "Content-Type: application/json" -H 'kbn-xsrf: any' -u sirenadmin:password -L -XPOST 'http://localhost:5606/api/entity_tables/refresh_fields' -d '{ "dataspaces": ["test"] }'

A request to refresh fields for all entity tables across all dataspaces:

curl -H "Content-Type: application/json" -H 'kbn-xsrf: any' -u sirenadmin:password -L -XPOST 'http://localhost:5606/api/entity_tables/refresh_fields' -d '{}'

A request to refresh fields for all entity tables across all dataspaces without recomputing expensive field properties:

curl -H "Content-Type: application/json" -H 'kbn-xsrf: any' -u sirenadmin:password -L -XPOST 'http://localhost:5606/api/entity_tables/refresh_fields' -d '{ "computeExpensiveFieldProperties": false }'

Response

Refresh Fields is a resource-intensive, long-running operation. Once user credentials are verified, Investigate immediately responds with HTTP status code 202 (Accepted) and an operationId you can use to track its progress.

Avoid starting multiple simultaneous Refresh Fields operations. Use the following operations API to wait for an operation to complete before starting a new one.
Property Type Description

operationId

string

A UUID string that identifies the operation.

Sample response

{
  "operationId": "7e990a3a-9872-4157-9bea-6f7b6afe64d0"
}

Endpoint ⇒ GET /api/operations/{operationId}/stream

You can use this endpoint to receive a stream of notifications about the status of the operation represented by operationId. The stream follows the server-sent events specification.

The stream will emit the following events, representing a status object with data encoded as a JSON string:

Event Data type Description

update

ProgressData

Reports a progress update, including the overall completion percentage and a textual description of current objects being processed.

done

RefreshFieldsResult

Reports the final result of the operation.

error

ErrorData

Reports an error in the procedure.

canceled

{}

Reports that the operation was canceled.

Sample request

curl -N -H "Content-Type: application/json" -H 'kbn-xsrf: any' -u sirenadmin:password -L 'http://localhost:5606/api/operations/7e990a3a-9872-4157-9bea-6f7b6afe64d0/stream'

Sample response

event: update
data: {"text":"Refreshing Entity Table 5/20","progressPercent":25}

event: update
data: {"text":"Refreshing Entity Table 15/20","progressPercent":75}

id: end
event: done
data: {"totalEntityTablesCount":20,"failedEntityTablesCount":0,"failedEntityTablesDetails":[],"elapsedSeconds":3}

Endpoint ⇒ GET /api/operations/{operationId}/status

You can poll at this endpoint to get information about the current status of the long-running operation represented by operationId. The returned status object has the following schema:

Property Type Description

status

'running' | 'done' | 'error' | 'canceled'

Current status of the operation.

data

ProgressData | RefreshFieldsResult | ErrorData

The data object associated with the current status.

Sample request

curl -H "Content-Type: application/json" -H 'kbn-xsrf: any' -u sirenadmin:password -L 'http://localhost:5606/api/operations/7e990a3a-9872-4157-9bea-6f7b6afe64d0/status'

Sample responses

{
  "status": "running",
  "data": {
    "text": "Refreshing Entity Table 15/20",
    "progressPercent": 75
  }
}
{
  "status": "done",
  "data": {
    "totalEntityTablesCount": 20,
    "failedEntityTablesCount": 1,
    "failedEntityTablesDetails": [{
      "id": "search:0790e66b-760c-46c6-a160-a160c9810c34",
      "error": {
        "status": 403,
        "reason": "\"change\" permission denied"
      }
    }],
    "elapsedSeconds": 3
  }
}
{
  "status": "error",
  "data": {
    "status": 400,
    "message": "Any error detail here"
  }
}
{
  "status": "canceled",
  "data": {}
}

Endpoint ⇒ POST /api/operations/{operationId}/cancel

You can use this endpoint to cancel the operation represented by operationId.

Sample request

curl -N -X POST -H 'kbn-xsrf: any' -u sirenadmin:password -L 'http://localhost:5606/api/operations/7e990a3a-9872-4157-9bea-6f7b6afe64d0/cancel'

Status objects

RefreshFieldsResult

Representation of the result of a Refresh Fields operation.

Property Type Description

totalEntityTablesCount

number

Count of entity tables targeted.

failedEntityTablesCount

number

Count of entity tables that could not have their fields refreshed.

failedEntityTablesDetails

SavedObjectBulkErrorDetails

Error details for the first 100 entity tables that could not have their fields refreshed.

elapsedSeconds

number

Seconds taken for the operation to run.

ProgressData

An object representing current progress in the operation.

Property Type Description

text

string

Textual explanation of the current moment in the processing pipeline.

progressPercent

number

Completion percentage of the whole operation.

ErrorData

An object representing an error in the operation.

Property Type Description

status

number

HTTP status code representing the error.

message

string

Textual description of the error.

SavedObjectBulkErrorDetails

An object representing an error associated to a specific saved object.

Property Type Description

id

string

The id of the saved object.

title

string | undefined

The title of the saved object, if it was retrieved.

error.status

number

HTTP status code representing the error.

error.reason

string

Textual description of the error.