Siren Platform User Guide

Siren Investigate Graph Browser

The graph browser displays Elasticsearch documents as nodes and Siren Investigate relations as links of a graph.

Figure 1. Graph Browser Example
Graph Browser Example


Configuration

Big nodes threshold

If a node would expand into more than this configured number of nodes it will be considered a big node and the user will be given a choice to proceed or to select a sample.

Figure 2. Big Nodes Handling
Big Nodes Handling


Relations

You can configure the ontology relations you want to use in this visualization. If no relation gets set, they will all be used.

Scripts

The Graph Browser supports three types of scripts:

  • Expansion: Used to customize the expansion policy. The provided one (Default Expansion Policy) will retrieve the first level connected elements to the expanded nodes
  • Contextual: Displayed in the contextual menu (shown with a RIGHT CLICK). Enable you to perform operations on the graph. Provided contextual scripts:

    • Expand by relation: Opens a popup that enables you to choose one or more of the available relations and expands the selected elements using only the selected ones. This does not override the graph browser configuration. You will see only the configured relations (if available for the selected nodes).
    • Expand by top comention: For use with company nodes from the Siren demonstration data. This script expands the selected nodes using an Elasticsearch aggregation to get the top comentioned company nodes.
    • Replace investment with edge: For use with the Siren demonstration data. This script replaces the investment nodes with a direct link between the company nodes and the investor nodes.
    • Select - All: Select all the elements (equivalent to Ctrl+A).
    • Select - By edge count: Select nodes based on their link count. You can specify the count through the popup that appears.
    • Select - By type: Select nodes based on their type. You can specify the type through the popup that appears.
    • Select - Extend: Extends the current selection to the sibling elements.
    • Select - Invert: Inverts the current selection.
    • Shortest Path: Calculates the shortest path between two selected nodes by fetching the connected elements.
    • Show nodes count by type: Shows a popup with information about how many nodes per type are currently displayed.
  • Lenses:Lenses mutate the visual appearance of graph nodes and edges, can be cascaded as well as switched on and off at will during investigation. Provided lens scripts:

    • Size lens: Set the size for all nodes using an expression.
    • Color lens: Define color for all nodes using a field.
    • Conditional lens: Set node properties using expressions.
    • Label lens: Set the label for all nodes using an expression.
    • Associate records based on ontology lens: Replaces a node with associated records based on ontology.
    • Time and location lens: Set time and location properties.
  • On Update: Modify the graph when new nodes are inserted. They can be cascaded. Provided on update scripts:

    • Add time fields: Adds the time field used by the timebar mode.
    • Add geo-locations for map visualization: Adds the geographic field used by the map mode.
    • Replace investment with edge: Similar to the contextual script Replace investment with edge, but executed automatically after every expansion.
    • Signal dead companies: Colors all the company nodes that have a deadpooled_date black.

To create a new script go to ManagementScripts

Figure 3. Scripts Management
Scripts Management


Here you can configure new scripts or modify the saved ones.

Fields to exclude

You can configure a set of fields for each entity that you do not want to retrieve. Typically, you will exclude large fields that do not contribute to the link analysis (for example large textual blobs, technical metadata)for extra performance.

Navigating the Graph

After your graph browser visualization is ready, you can start your investigations.

Toolbar

You have several operations available:

Figure 4. Toolbar
Toolbar


  1. Undo: By default, the graph browser saves the last five states. With this function you can go back one step at a time, until there are no more available. You can configure the steps number in ManagementAdvanced Settings.
  2. Redo: Revert an undo. Note, if you undo and then perform any operation, the redo state will be lost.
  3. Filter: Add a filter to the current dashboard synchronized with the graph selection. This enables you to:

    • Do your investigation on the graph, select the vertices you are interested into, activate the filter, pin it and go back to the related dashboard to get more detailed information about those vertices.
    • If you have other visualizations in the same dashboard, it will let you have more information on the selected nodes. For example, if you have the current dashboard associated with a companies saved search, you can do your investigation in the graph, activate the filter, select some vertices and get the visualizations to show information on the selected vertices.
  4. Crop: Removes every element that is not selected
  5. Remove: Removes all the selected elements. Next to the Remove button there is a box that shows the Remove All button. This will remove the entire graph, regardless of selected elements.

    Remove all
  6. Expand: Expands the currently selected nodes. Next to the expand button, there is a box that shows advanced options for the expansion.
  7. Highlight mode: This toggles the Highlight mode. The Highlight mode moves everything to the background that is not selected or connected to a selected node or link.

    Figure 5. Highlighting on
    Highlighting on


    Figure 6. Highlighting off
    Highlighting off


  8. Layouts: Enables you to change the current graph’s layout or redraw the current layout:

    • Standard (default): Selected nodes preserve their relative position.
    • Hierarchy: Nodes are displayed top down according to their connections. Requires at least one selected node to work. Selected nodes will be moved at the top of the hierarchy.
    Figure 7. Standard layout
    Standard layout


    Figure 8. Hierarchy layout
    Hierarchy layout


  9. Add: Opens a dialog with the following options:

    • Add selected document: Add the currently selected document. You can see your selected document in the upper right purple selection box.
    • Add from saved graph: Opens a dialog showing the available saved graphs. This feature adds a set of nodes and links, but does not preserve the layout you had when you saved the graph.
    • Add from another dashboard: Adds nodes using the dashboard (optionally filtered) that you select.
    Figure 9. Add from saved graph
    Add from saved graph


  10. Map Mode: This toggles the Map mode. The Map mode will move the nodes geographically on an interactive map. You must set up a script to configure the geographic properties of the nodes (See 3.5.19 Siren Investigate Graph Browser).3.5.19 Siren Investigate Graph Browser

    Map mode
  11. Timebar Mode: This toggles the Timebar mode. This mode displays a time bar at the bottom of the graph browser that enables time based filtering of nodes. After you enable this mode you can add or remove node types to the time bar:

    UUID-a79c586d-6807-618f-2877-5b1144383451.png

    Timebar filter

    You must set up a script to configure the time property of the nodes (See 3.5.19 Siren Investigate Graph Browser).3.5.19 Siren Investigate Graph Browser

    Timebar mode
  12. Save graph: Save the current graph.

    Save graph
  13. Open graph: Open a saved graph. Unlike add from saved graph, this feature preserves the saved graph layout.

    Open Graph

Shortcuts

The graph browser supports shortcuts:

  • Ctrl+A: Select every element in the graph.
  • Del: Remove the selected elements (equivalent to the remove button).
  • Ctrl+click: Enables you to add elements to the current selection. Can also be used to create an OR filter from a selection.
  • Double-click: Expands the selected nodes (equivalent to the expand button)
  • Arrows: Move the selected elements in the input direction.
  • Mouse wheel: Changes the zoom level of the graph.

Navigation bar

Navigation bar

The navigation bar enables you to:

  1. Move the graph view in the clicked direction.
  2. Switch between:

    • Arrow: Enables you to select elements.
    • Hand: Enables you to move the graph regardless of selected elements.
  3. Enables you to change the zoom level.

Side bar

Side bar

The side bar enables you to:

  • Show, search, filter, sort, group and change node/links data.
  • Change the current selection.
  • Change node/links attributes (i.e: Color, label, tool tip, and so on).
Lenses tab
Side bar lenses tab

The lenses tab enables you to make alterations on the displayed nodes/links:

  • Color: Enables you to select a field which is then used to color the nodes using a coloring schema.
  • Conditional: Enables you to change a node property value using configurable expressions.
  • Label: Enables you to set the node label using an expression.
  • Size: Use a log scale to adjust the node’s size according to an expression.
  • Spatio-Temporal: Enables you to set the node time and/or geographic location from field values.
  • Associate records based on ontology: Enables you to replace a node with a relation between two of its children.
  • Graph metric: Enables you to apply metrics to the graph including:

    • Betweenness

    • Closeness

    • Connectiveness

    • Degrees

    • Eigenvector

    • Pagerank

See 3.5.19 Siren Investigate Graph Browser for more information on lens expressions.3.5.19 Siren Investigate Graph Browser

Lens parameters
Lens parameters

Each lens has specific parameters which will be used for every graph node.

Conditional lens
Conditional lens

A conditional lens can change a property for all the nodes that satisfy the condition:

  • Color
  • Node font icon
  • Node glyphs
  • Hidden
  • Label
  • Location
  • Node image
  • Size
  • Time
  • Tooltip
Associate records based on ontology lens
Side bar - associate records based on ontology lens

The associate records based on ontology lens can use the node’s underlying model, as in the following example, to replace a node with the relation between two of its children.

Investment model graph view
vestment model graph view

After you configure the lens, two nodes and its relationship will be displayed. For example, apply this lens an investment node:

Investment node

You could obtain the associate records based on ontology as a result:

Associate records based on ontology result
Expansion tab

The expansion tab controls how nodes expand when you double-click them or select a group of nodes and click Expand.

  • Dashboard filters: Restrict nodes produced by an expansion to those present in the selected dashboard.

  • Relations - simple: Restrict nodes produced by an expansion to the selected relations.

  • Relations - aggregated: Quickly show aggregates on graph edges that summarize groups of intermediate nodes.

Selection tab
Selection tab

The selection tab enables you to show, search, filter, sort, group and change node/links data. When this tab is opened, it reacts with your current node selection and loads the data in rows and columns.

The main component is the data grid, every grid’s row represents a node in the graph and every column a field data related information.

Document type selection

The Main selection combo box enables selection between the different document types in the selected nodes.

Selection change

The second column in the grid enables multiple row selection, once selected it will reflect on the graph turning each node bigger and changing the node’s border to red.

After you complete the selection, you can click the Make main selection button floating over the grid to remove the non-selected nodes.

Global filter

Typing in the Filter input enables you to search/filter in all rows and columns.

Local filter

Typing inside of one column’s input enables you to search/filter in all rows of that column.

Grid menu
Grid menu

This menu enables you to show or hide columns and clear all local filters.

Column menu
Column menu

The menu options enable you to:

  • Change the sort order -Multiple column order is supported by keeping shift key pressed on column selection-.
  • Hide the column.
  • Group the data.
  • Add aggregated function, the result of which will be displayed at the bottom.
  • Pin the column to the left or right side of the grid.
Lens Expressions

Siren Investigate’s lens expression parser is based on Jexl.

The expression created within the lens is applied to each node of the selection. Each node contains an object named payload which contains the node’s data returned from Elasticsearch.

Jexl operators

There are a number of operators which can be applied to the payload data for transformation, comparison, and so on.

Here are a selection, there are further details at the Jexl GitHub page.

Operators. Use these operators to perform mathematical operations on values

Operation

Symbol

Example

Negate

!

!truefalse

Add/Concat

+

3 + 47

Subtract

-

4 - 31

Multiply

*

3 * 824

Divide

/

15 / 43.75

Divide and Floor

//

15 // 43

Modulus

%

23 % 21

Power of

^

2^38

Logical AND

&&

true && truetrue

Logical OR

||

`true

Comparisons. Use these expressions to compare two values, the Boolean results can be used for, for example filtering.

Operation

Symbol

Example

Equal

==

1 == 2false

Not Equal

!=

1 != 2true

Greater Than

>

2 > 3false

Greater Than or Equal

>=

3 >= 3true

Less Than

<

2 < 3true

Less Than or Equal

<=

2 ⇐ 4true

Element in array or string

in

"cat" in ["cat", "dog", "mouse"]true

Conditional Operators. Conditional operators return the second or third expression based on the result of the first expression. If the first expression ("Bob" in ["Bob", "Mary"] below) return true, "Yes" is returned. If it returns false, "No" is returned.

Example

Result

"Bob" in ["Bob", "Mary"] ? "Yes" : "No"

"Yes"

Identifiers. Access variables in the payload with dot notation or by using brackets, for example:

{
  name: {
    first: 'John'
    last: 'Smith'
  },
  age: 55,
  colleagues: [
    'Mary',
    'Bob',
    'Ted'
  ],
  teammate: 2
}

Example

Result

name.first

"John"

colleagues[teammate]

"Ted"

name['la' + 'st']

"Smith"

Collection filtering. Arrays of objects (Collections) can be filtered by including a filter expression in brackets. Properties of each collection can be referenced by prefixing them with a leading dot. The result is an array of objects for which the filter returns a truthy value.

{
  users: [
    { first: 'John', last: 'Smith', age: 20},
    { first: 'Mary', last: 'Jones', age: 46},
    { first: 'Ted', last: 'Cotter', age: 16},
    { first: 'Bob', last: 'White', age: 66}
  ],
  adult: 21
}

Example

Result

users[.last == 'Jones']

[{ first: 'Mary', last: 'Jones', age: 46}]

users[.age < adult]

[{ first: 'John', last: 'Smith', age: 20}, first: 'Ted', last: 'Cotter', age: 16}]

users[first == 'John'].last

"Smith"

Lens Expression Functions

In addition to the general Jexl parsing functionality, Siren Investigate also exposes a number of JavaScript-like functions for use in Lens Expressions. Payload values (or the results from earlier parsing) are piped into the function using the | character. These values become the val parameter for the functions below - meaning the val does not need to be added in the () after the function name. In some cases, this value is all that is needed by the function and some functions require extra parameters.

Some functions require string inputs and some require integer or floating-point inputs

Table 11. String Lens Expressions

Function

Example

Explanation

split(val, delimiter[, limit])

payload.IP | split('.', 3)

Splits an IP address by the '.' and returns the first 3 entries as an array

endsWith(val, substring[, length])

payload.name | endsWith('smith', 10)

Returns true if val ends with substring, if length is added, that number of characters from the beginning of val is checked.

startsWith(val, substring[, position])

payload.name | startsWith('smith', 10)

Returns true if val begins with substring, if position is added, the substring from that position to the end of val is checked.

indexOf(val, substring[, length])

payload.name | indexOf('smith', 10)

Returns the position of the first character of substring if val contains substring, if length is added, val is checked from that position.

upper(val)

payload.name | upper

Returns val in upper case.

lower(val)

payload.name | lower

Returns val in lower case.

indexOf(val, start, end)

payload.name | substring(5, 10)

Returns the string within val found between start and end.

replace(val, substring, newSubString)

payload.name | replace('smith', 'jones')

Replaces substring with newSubString in val.



Table 12. Number lens Expressions

Function

Example

Explanation

round(val)

payload.range | round

Returns val rounded to the nearest integer.

trunc(val)

payload.range | trunc

Returns the integer part of val.

sqrt(val)

payload.range | sqrt

Returns √val.

sign(val)

payload.range | sign

Returns 1 if val is positive, -1 if val is negative or 0 if val equals 0.

ceil(val)

payload.price | ceil

Returns the nearest integer greater than val

floor(val)

payload.price | floor

Returns the nearest integer less than val

abs(val)

payload.temperature_change | abs

Returns the absolute value for a Number or 0 if the number is null

exp(val)

payload.difference | exp

Returns val

log(val)

payload.difference | log

Returns the natural logarithm of val, for example ln(val)

random(val)

payload.range | random

Returns val multiplied by a floating-point, pseudo-random number between 0 (inclusive) and 1 (exclusive).



Link analysis

Siren enables you to group nodes manually or automatically based on shared properties, for example:

  • All records located in France or Germany.

  • All IPs in server room A.

  • All patients from the placebo clinical trial arm.

This can reduce graph clutter and make it easier to discover patterns and drill down into clusters during analysis.

For example, the following image shows companies clustered by US state in which their headquarters are located.

Grouping on nodes
Security setup for the graph browser

If you are using Elasticsearch version 5.6.16, you must add the following permission to the sg_roles.yml file to work with JDBC documents in the graph browser:

sirenserver:
 indices:
    '*':
      '*':
        - indices:data/siren/connector/mappings/get