Siren Platform User Guide

Datasource reflection jobs

Each datasource reflection job is defined by a configuration. Configurations can be managed on the datasource reflection jobs page:

DIJ_1.png

Each job can be scheduled to run periodically using a cron syntax or run manually through the jobs page. Many actions can be performed on the jobs page, for example:

  • Add a new job.

  • Edit an existing one.

  • Manually run a job.

  • Delete a job configuration.

  • Clone a job configuration.

  • Enable/Disable a scheduled job.

Defining a new datasource reflection job configuration

There are 5 steps in defining a new datasource reflection job:

Step 1 - Data input
Ingest_Step1.png

In Step 1, give a name and a description (optional) to the job, then select a datasource and write the query.

If selecting a virtual index as a datasource, then start with the default query for that virtual index.

Writing a query

The query defined in the reflection configuration is written in the datasource language.

The query can be written using mustache and the following variables are provided, if applicable, when converting the query to a string:

max_primary_key: the maximum value of the primary key in Elasticsearch, available if the primary key is numeric.

last_record_timestamp: the UTC timestamp at which the last record was successfully processed by an reflection job.

last_record: an array with the scalar values in the last record that was successfully processed by the reflection job.

Example:

SELECT * FROM "Player" {{#max_primary_key}}
WHERE "NAME">'{{max_primary_key}}'{{/max_primary_key}}
Step 2 - Transform pipeline (optional)
Ingest_Step2_Pipeline.png

This is an optional step where a pipeline may be defined to enrich the reflected documents.

Step 3 - Mapping
Ingest_Step3_Mapping.png

The mapping for the fields received from the datasource is auto-detected, however the mapping can be changed to other valid field types and new fields may be added to accommodate fields added by a transformation pipeline that are not auto-detected.

Step 4 - Target Index
Ingest_Step4_Target-Index.png

In step 4, the name of the Elasticsearch index to be used for reflection is specified and configured to particular preferences e.g primary key, indexing options, custom credentials.

Step 5 - Schedule
Ingest_Step5_Schedule.png

The final step allow the job to be scheduled. The scheduler accepts cron syntax to schedule jobs.