Loading stored layers

Loading stored layers into Elasticsearch

There are two methods for loading the GeoJSON files into Elasticsearch. These are the Folder Structure and the Spatial Path methods. Both will accept files of type json or geojson. The methods differ in how their spatial_path is determined.

Folder structure method

This method requires all GeoJSONs to be contained in a folder structure, that will determine scope/layer of each GeoJSON file.

Folder structure

Spatial path method

This method requires all GeoJSON features to have a spatial_path property in their properties object. This will determine its scope/layer. For more information, see the example.

Spatial path

File validation

Each file needs to meet the following criteria:

  • Cannot be empty

  • Must only contain one of the following type categories:

    • polygon - including Polygon or MultiPolygon

    • line - including LineString or MultiLineString

    • point - Point

  • If using the spatial path loading method, each GeoJSON feature must contain the spatial_path property

If these requirements are not met, the validation process will fail, only files to that point will be loaded.

Spatial path

Spatial path is the scope/layer of a feature. Documents that have the same spatial path, will be part of the same layer when used in Investigate.

Folder structure

When using the folder structure loading method, the spatial path will be constructed based on the folder structure the GeoJSON file is in.

Here is an example of a spatial path for Continents/Europe/Poland/counties.geojson ⇒ continents/europe/poland

The file …​/Poland/counties.geojson contains 34 counties that make up Poland while …​/counties 2.json contains the other 8 Polish counties. The spatial path of both GeoJSONs when ingested is specified below. They will be loaded as one layer on Enhanced Coordinate Map Visualization, i.e. one 42 county layer: continents/europe/poland

Spatial Path

The spatial path loading method requires each feature to contain a spatial_path property in the properties object of each GeoJSON feature. This is checked for during initial validation.

{
  "type": "Feature",
  "properties": {
    "county": "Galway",
    "population": 86000,
    "spatial_path": "continents/europe/ireland"
  },
  "geometry": {
    "type": "Polygon",
    "coordinates": []
  }
}

Usage

To load files into Elasticsearch go to the siren-investigate folder and run the ./bin/load_map_reference_indices script with the appropriate arguments.

Make sure a user that has Elasticsearch write permissions is used for authorization.

Mappings

The load script will automatically differentiate between shape and point geoJSON files and use appropriate mapping file from /src/map_indices_loader to create an elasticsearch index, a custom mapping can be passed in with arguments.

Arguments

  • -p/--path/--inputdir specifies a path to the folder containing files

  • -y/--yml allows for using a custom investigate.yml. If omitted, the default configuration from /config/investigate.yml will be used

  • --username <username> --password <password> allows for specification of Elasticsearch authorization credentials. If <username> and <password> are left blank, environmental variables LOAD_LAYERS_ES_USERNAME, LOAD_LAYERS_ES_PASSWORD will be used. If not used at all, credentials specified in /config/investigate.yml will be used instead

  • --ms allows for use of a custom mappings JSON file for geo_shape objects (Polygons and Lines) instead of the default /src/map_indices_loader/mappings_geoshape.json

  • --mp allows for use of a custom mappings JSON file for geo_point objects (Points) instead of the default /src/map_indices_loader/mappings_geopoint.json

  • -s/--settings allows for use of a custom index settings JSON file instead of the default /src/map_indices_loader/index_settings.json

  • --dryrun will run the validation stage of the script, but will not connect to Elasticsearch

  • --overwrite if an index already exists it will be overwritten with a new one

  • --structure/--spatialpath determines the loading method

  • -n specifies the maximum number of documents per single request - higher number will result in heavier requests (default 500)

  • -r/--simrequests specifies the number of simultaneous requests made to Elasticsearch (default 40)

  • --debug outputs additional debug information, including Elasticsearch authorization credentials

Examples

./bin/load_map_reference_indices.sh -p "/home/siren/geoJsonFolder" --structure

./bin/load_map_reference_indices.sh -p /home/siren/geoJsonFolder -y /home/siren/config.yml --username admin --password password --structure

./bin/load_map_reference_indices.sh -p /home/siren/geoJsonFolder --username --password --spatialpath

The console output below is a successful file load (with --debug) to Elasticsearch.

Console output

Loading stored layers from Elasticsearch into the Enhanced Coordinate Map

Before you begin, you will need to have layers stored in Elasticsearch in indices that have a .map__ prefix. For more information, see Loading Stored Layers into Elasticsearch.

In the top-right of an Enhanced Coordinate Map visualization, click Layer Control. When the Add Layers button is clicked, a modal (below) will appear displaying Stored Layers (e.g. Irish Counties) and the group they are in (i.e. World Countries).

Add Layers Modal

It is possible to add a layer type corresponding to a group and we will explain with the aid of an example below, using the US states layer:

  • If the US states group checkbox is toggled, all layers and groups in that group will also be toggled

    • If the US States layer checkbox is toggled, only the layer will be affected

    • If the California nested group is toggled, only the checkboxes within that group will be toggled

  • If either a nested group or a layer box is unchecked, the group will become indeterminate indicating that some of the boxes in that group are unchecked

  • Finally, if no items in a group are checked, the group checkbox will be unchecked

To add the selected layers to the map but keep them hidden for now, click Add. To add the layers to the map and also make them visible, click Add and Enable.

You can configure the stored layers when your visualization is in edit mode. For more information, see Configuring stored layers.