Configuring a remote Elasticsearch connector

Siren Federate provides the capability to query data from a remote Elasticsearch cluster through the remote clusters module and the Siren Federate connector APIs.

Unlike the JDBC connector, the remote Elasticsearch connector supports wildcard index patterns.

Compatibility with security systems

The Siren Federate roles are used to secure the coordinator and the remote clusters. In order to execute joins spanning several clusters, those roles must be attributed to a same user on every clusters.

The remote Elasticsearch connector is compatible with the following security systems: - Search Guard - Elastic Stack Security

Before you begin

  1. Ensure that the remote clusters are configured as described in Configuring remote clusters.

  2. Install the Siren Federate plugin on the remote clusters.

  3. Set up the remote Elasticsearch clusters. For example, use the following settings:

    curl -X PUT http://localhost:9200/_cluster/settings -H 'Content-type: application/json' -d '
{
    "persistent": {
        "cluster": {
            "remote": {
                "remotefederate": {
                    "seeds": [
                        "127.0.0.1:9330"
                    ]
                }
            }
        }
    }
}
'

Procedure

In this procedure, we are using the example of a remote cluster called remotefederate, which contains indices called logs-2019.01, logs-2019.02, …​, logs-2019.12, and so on.

  1. Define the datasource as an alias to the remote cluster, by using the Siren Federate datasource API as follows:

    curl -X PUT http://localhost:9200/_siren/connector/datasource/remotefederateds -H 'Content-type: application/json' -d '
  {
    "elastic": {
      "alias": "remotefederate"
    }
  }
  '

  2. Define a virtual index on the coordinator cluster that matches the wildcard index pattern logs-*, by using the Siren Federate virtual index API as follows:

    curl -X PUT http://localhost:9200/_siren/connector/index/logsvi -H 'Content-type: application/json' -d '
{
  "datasource": "remotefederateds",
  "resource": "logs-*",
  "key": "_id"
}
'

  3. Execute a join query. For example, the coordinator cluster contains an index called machines, which contains information about IP addresses on machines of interest. To find out about the logs that are associated to these machines, execute the following Federate join query:

    curl -X GET http://localhost:9200/siren/logsvi/_search -H 'Content-Type: application/json' -d '
{
    "query": {
        "join": {
            "indices": [
                "machines"
            ],
            "on": [
                "logs_ip_hash",
                "machines_ip_hash"
            ],
            "request": {
                "query": {
                    "match_all": {

                    }
                }
            }
        }
    }
}
'

    logs_ip_hash is the IP field in the index logsvi and machines_ip_hash is the IP field in the index machines.

    The API returns the following response:

    {
  "took": 150,
  "timed_out": false,
  "hits": {
    "total" : {
        "value": 1,
        "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "logs-2019-11-12",
        "_id": "0",
        "_score": 2,
        "_source": {
          "date": "2019-11-12T12:12:12",
          "message": "trying out Siren"
        }
      }
    ]
  }
}

Known limitations for the Elasticsearch connector

To use Siren Federate with a remote cluster, a coordinator Federate cluster must run version 7.7.1-20.0 or later, and the remote cluster must run Siren Federate version 7.7.1-20.0 or later.