Watchers

A Siren Alert watcher is created using the following structure:

➔ Trigger Schedule

When and How to run the Watcher

➔ Input Query

What Query or Join Query to Execute

➔ Condition

How to conditionally Analyze Response

➔ Transform

How to Adapt or Post-Process data

➔ Actions

How to Notify users about this event

Trigger schedule

The schedule defines a set of constraints that must be met to execute a saved watcher. Any number of constraints can be added to a single schedule, and multiple rules can be combined to achieve complex intervals, programmed using simple text expressions using the Node.JS later module.

watcher anatomy

Interval exceptions can also be defined as follows:

every 2 hours except after 20th hour

Input query

The input parameter is the key element of a watcher, and defines a dynamic range index query feeding the circuit. The input field accepts any standard Elasticsearch query including server side scripts in supported languages and fully supports the Siren Join capabilities out of the box.

"input": {
  "search": {
    "request": {
      "index": [
        "<mos-{now/d}>",
        "<mos-{now/d-1d}>"
      ],
      "body": {}
    }
  }
}

Condition

The condition block is the "entry gate" into the processing pipeline of a Watcher and determines its triggered status.

  • On true condition, the pipeline will proceed further.

  • On false condition, the pipeline will stop (no action will be executed) until its next invocation.

Never condition

Use the never condition to set the condition to false. This means the watch actions are never executed when the watch is triggered. Nevertheless, the watch input is executed. This condition is used for testing. There are no attributes to specify for the never condition.

condition: {
  "never" : {}
}

Compare condition

Use the compare condition to perform a simple comparison against a value in the watch payload.

condition: {
  "compare" : {
    "payload.hits.total" : {
      "gte" : 5
    }
}

Comparison operators (apply to numeric, string and date).

eq Returns true when the resolved value equals the given one

not_eq

Returns true when the resolved value does not equal the given one

lt

Returns true when the resolved value is less than the given one

lte

Returns true when the resolved value is less/equal than/to the given one

gt

Returns true when the resolved value is greater than the given one

gte

Returns true when the resolved value is greater/equal than/to the given one

Array compare condition

Use array_compare to compare an array of values. For example, the following array_compare condition returns true if there is at least one bucket in the aggregation that has a doc_count greater than or equal to 25:

"condition": {
  "array_compare": {
    "payload.aggregations.top_amounts.buckets" : {
      "path": "doc_count" ,
      "gte": {
        "value": 25,
      }
    }
  }
}

Options

array.path

The path to the array in the execution context, specified in dot notation.

array.path.path

The path to the field in each array element that you want to evaluate.

array.path.operator.quantifier

How many matches are required for the comparison to evaluate to true: some or all. Defaults to some, there must be at least one match. If the array is empty, the comparison evaluates to false.

array.path.operator.value

The value to compare against.

Script condition

A condition that evaluates a script. The scripting language is JavaScript. It can be as simple as a function expecting a Boolean condition or counter.

condition: {
  "script": {
    "script": "payload.hits.total > 100"
  }
}

Or it can be as complex as an aggregation parser to filter buckets.

condition: {
  "script": {
    "script": "payload.newlist=[];var match=false;var threshold=10;var start_level=2;var finish_level=3;var first=payload.aggregations[start_level.toString()];function loop_on_buckets(element,start,finish,upper_key){element.filter(function(obj){return obj.key;}).forEach( function ( bucket ) { if (start == finish - 1) { if (bucket.doc_count >= threshold) { match=true;payload.newlist.push({line: upper_key + bucket.key + ' ' + bucket.doc_count}); } } else { loop_on_buckets(bucket[start + 1].buckets, start + 1, finish, upper_key + ' ' + bucket.key); } }); } var upper_key = ''; loop_on_buckets(first.buckets, start_level, finish_level, upper_key);match;"
  }
}

Anomaly detection

Simple anomaly finder based on the three-sigma rule.

  1. Dynamic detection of outliers/peaks/drops:

    {
  "script": {
    "script": "payload.hits.total > 0"
  },
  "anomaly": {
    "field_to_check": "fieldName"
  }
}

  2. Static detection for known ranges/interrupts:

    {
  "script": {
    "script": "payload.hits.total > 0"
  },
  "anomaly": {
    "field_to_check": "fieldName",
    "normal_values": [
      5,
      10,
      15,
      20,
      25,
      30
    ]
  }
}

Range filtering

Use for getting documents which have a value in between some values. For example, get only the documents which have values from 45 to 155 inside Amount field.

{
  "script": {
    "script": "payload.hits.total > 0"
  },
  "range": {
    "field_to_check": "Amount",
    "min": 50,
    "max": 150,
    "tolerance": 5
  }
}

Transform

A transform processes and changes the payload in the watch execution context to prepare it for the watch actions. No actions are executed in the case that the payload is empty after transform processing.

Search transform

A transform that executes a search on the cluster and replaces the current payload in the watch execution context with the returned search response.

"transform": {
  "search": {
    "request": {
      "index": [
        "credit_card"
      ],
      "body": {
        "size": 300,
        "query": {
          "bool": {
            "must": [
              {
                "match": {
                  "Class": 1
                }
              }
            ]
          }
        }
      }
    }
  }
}

Script transform

A transform that executes a script (JavaScript) on the current payload and replaces it with a newly generated one. Its uses include:

  • Converting format types.

  • Generating new payload keys.

  • Interpolating data

Create new payload property:

"transform": {
  "script": {
    "script": "payload.outliers = payload.aggregations.response_time_outlier.values['95.0']"
  }
}

Filter aggregation buckets:

"transform": {
  "script": {
    "script": "payload.newlist=[]; payload.payload.aggregations['2'].buckets.filter(function( obj ) { return obj.key; }).forEach(function(bucket){ console.log(bucket.key); if (doc_count.length > 1){ payload.newlist.push({name: bucket.key }); }});"
  }
}

Chain transform

A transform that executes an ordered list of configured transforms in a chain, where the output of one transform serves as the input of the next transform in the chain.

"transform": {
  "chain": [
    {
      "search": {
        "request": {
          "index": [
            "credit_card"
          ],
          "body": {
            "size": 300,
            "query": {
              "bool": {
                "must": [
                  {
                    "match": {
                      "Class": 1
                    }
                  }
                ]
              }
            }
          }
        }
      }
    },
    {
      script: {
        script: "payload.hits.total > 100"
      }
    }
  ]
}

Actions

Actions are used to deliver any results obtained by a Watcher to users, APIs or new documents in the cluster. Multiple Actions and Groups can be defined for each.

Actions use the {{ mustache }} logic-less template syntax, and work by iterating arrays and expanding tags in a template using values provided in the response payload.

For more details, see Supported actions.

Full watcher example

{
  "_index": "watcher",
  "_type": "watch",
  "_id": "new",
  "_source": {
    "trigger": {
      "schedule": {
        "later": "every 5 minutes"
      }
    },
    "input": {
      "search": {
        "request": {
          "index": [
            "<mos-{now/d}>",
            "<mos-{now/d-1d}>"
          ],
          "body": {}
        }
      }
    },
    "condition": {
      "script": {
        "script": "payload.hits.total > 100"
      }
    },
    "transform": {
      "script": {
        "script": "payload.hits.total += 100"
      }
    },
    "actions": {
      "email_admin": {
        "throttle_period": "15m",
        "email": {
          "to": "alarm@localhost",
          "subject": "Siren Alert Alarm",
          "priority": "high",
          "body": "Found {{payload.hits.total}} Events"
        }
      },
      "slack_admin": {
        "throttle_period": "15m",
        "slack": {
          "channel": "#kibi",
          "message": "Siren Alert Alert! Found {{payload.hits.total}} Events"
        }
      }
    }
  }
}

Supported actions

Following are the currently supported "actions" for Siren Alert watchers.

Note that the actions which have active: true in their config are not enabled unless the active property is set to true.

Email

Send Query results and message using Email/SMTP.

Requires action settings in Siren Investigate configuration.

"email" : {
  "active": true,
  "to" : "root@localhost",
  "from" : "sirenalert@localhost",
  "subject" : "Alarm Title",
  "priority" : "high",
  "body" : "Series Alarm {{ payload._id}}: {{payload.hits.total}}",
  "stateless" : false
}

Email HTML

Send Query results and message using Email/SMTP using HTML body.

Requires action settings in Siren Investigate configuration.

"email_html" : {
  "to" : "root@localhost",
  "from" : "sirenalert@localhost",
  "subject" : "Alarm Title",
  "priority" : "high",
  "body" : "Series Alarm {{ payload._id}}: {{payload.hits.total}}",
  "html" : "<p>Series Alarm {{ payload._id}}: {{payload.hits.total}}</p>",
  "stateless" : false
}

webHook

Deliver a POST request to a remote web API

"webhook" : {
  "active": true,
  "method" : "POST",
  "host" : "remote.server",
  "port" : 9200,
  "path": ":/{{payload.watcher_id}}",
  "body" : "{{payload.watcher_id}}:{{payload.hits.total}}",
  "create_alert" : true
  "cacert" : "ca-certificate.pem",
  "cert" : "certificate.pem",
  "key" : "key.pem"
}

Deliver a GET request to a remote web API

"webhook" : {
  "active": true,
  "method" : "GET",
  "host" : "remote.server",
  "port" : 9200,
  "path" : "/trigger",
  "params" : {
    "watcher": "{{watcher.title}}",
    "query_count": "{{payload.hits.total}}"
  },
  "cacert" : "ca-certificate.pem",
  "cert" : "certificate.pem",
  "key" : "key.pem"
}

webHook using Proxy

Deliver message to remote API using Proxy - Telegram example:

"webhook": {
  "active": true,
  "method": "POST",
  "host": "remote.proxy",
  "port": "3128",
  "path": "https://api.telegram.org/bot{botId}/sendMessage",
  "body": "chat_id={chatId}&text=Count+total+hits:%20{{payload.hits.total}}",
  "headers": {
    "Content-Type": "application/x-www-form-urlencoded"
  },
  "create_alert" : true
}

Slack

Delivery Message to #Slack channel.

Requires action settings in Siren Investigate configuration.

"slack" : {
  "active": true,
  "channel": "#channel",
  "message" : "Series Alarm {{ payload._id}}: {{payload.hits.total}}",
  "stateless" : false
}

Report

Take a website snapshot using PhantomJS and send it using Email/SMTP.

  • Requires action settings in Siren Investigate configuration.

  • Requires Pageres/PhantomJS: npm install -g pageres.

  • Requires Chromium browser to be installed. If not, then Siren downloads chromium provided the user has the permission to do so.

"report" : {
  "active": true,
  "to" : "root@localhost",
  "from" : "kaae@localhost",
  "subject" : "Report Title",
  "priority" : "high",
  "body" : "Series Report {{ payload._id}}: {{payload.hits.total}}",
  "snapshot" : {
    "res" : "1280,900",
    "url" : "http://127.0.0.1/app/kibana#/dashboard/Alerts",
    "path" : "/tmp/",
    "params" : {
      "username" : "username",
      "password" : "password",
      "delay" : 5000,
      "crop" : false
    }
  },
  "stateless" : false
}

Console

Output Query results and message to Console.

"console" : {
  "priority" : "DEBUG",
  "message" : "Average {{payload.aggregations.avg.value}}"
}

Watcher controllers

The following controls are presented when listing existing Watchers:

Watcher controllers

  1. Expand or edit a Watcher.

  2. Manually execute a Watcher.

  3. Remove a Watcher.

  4. Toggle a Watcher on or off.

Watcher visibility

When security is enabled, users can define the visibility of a watcher, alarm or report for different groups of users.

As a watcher is a saved object its visibilty can be configured in the saved objects configuration page (see here).

The visibility of alarms and reports can be restricted in the Watcher editor using the privacy section. They can be set as follows:

watcher privacy settings

To restrict the permissions of alarms and reports, you must use document level security (DLS) by adding the following settings to the sg_roles.yml file of your searchguard security and running the sgadmin.sh script:

alert_viewer_role:
  indices:
    'watcher_alarms-*':
      '*':
        - READ
        - VIEW_INDEX_METADATA
      _dls_: '{
                "bool": {
                  "should": [
                    {
                      "term": {
                        "is_private": false
                      }
                    },
                    {
                      "bool": {
                        "must_not": [
                          {
                            "exists": {
                              "field": "is_private"
                            }
                          }
                        ]
                      }
                    },
                    {
                      "terms": {
                        "roles": [${user.roles}]
                      }
                    },
                    {
                      "term": {
                        "created_by": "${user.name}"
                      }
                    }
                  ]
                }
              }'

Examples

Watchers can be as simple or complex as the query and aggregations they use. Here are some examples to get started with.

Hit watcher

{
  "_index": "watcher",
  "_type": "watch",
  "_id": "new",
  "_source": {
    "trigger": {
      "schedule": {
        "later": "every 5 minutes"
      }
    },
    "input": {
      "search": {
        "request": {
          "index": [
            "<mos-{now/d}>",
            "<mos-{now/d-1d}>"
          ],
          "body": {}
        }
      }
    },
    "condition": {
      "script": {
        "script": "payload.hits.total > 100"
      }
    },
    "transform": {},
    "actions": {
      "email_admin": {
        "throttle_period": "15m",
        "email": {
          "to": "alarm@localhost",
          "from": "sirenalert@localhost",
          "subject": "Siren Alert Alarm",
          "priority": "high",
          "body": "Found {{payload.hits.total}} Events"
        }
      },
      "slack_admin": {
        "throttle_period": "15m",
        "slack": {
          "channel": "#kibi",
          "message": "Siren Alert Alert! Found {{payload.hits.total}} Events"
        }
      }
    }
  }
}

Transform (Elasticsearch 2.x)

{
  "_index": "watcher",
  "_type": "watch",
  "_id": "95th",
  "_score": 1,
  "_source": {
    "trigger": {
      "schedule": {
        "later": "every 5 minutes"
      }
    },
    "input": {
      "search": {
        "request": {
          "index": [
            "<access-{now/d}>",
            "<access-{now/d-1d}>"
          ],
          "body": {
            "size": 0,
            "query": {
              "filtered": {
                "query": {
                  "query_string": {
                    "analyze_wildcard": true,
                    "query": "*"
                  }
                },
                "filter": {
                  "range": {
                    "@timestamp": {
                      "from": "now-5m"
                    }
                  }
                }
              }
            },
            "aggs": {
              "response_time_outlier": {
                "percentiles": {
                  "field": "response_time",
                  "percents": [
                    95
                  ]
                }
              }
            }
          }
        }
      }
    },
    "condition": {
      "script": {
        "script": "payload.aggregations.response_time_outlier.values['95.0'] > 200"
      }
    },
    "transform": {
      "script": {
        "script": "payload.myvar = payload.aggregations.response_time_outlier.values['95.0']"
      }
    },
    "actions": {
      "email_admin": {
        "throttle_period": "15m",
        "email": {
          "to": "username@mycompany.com",
          "from": "sirenalert@mycompany.com",
          "subject": "Siren Alert ALARM {{ payload._id }}",
          "priority": "high",
          "body": "Series Alarm {{ payload._id}}: {{ payload.myvar }}"
        }
      }
    }
  }
}

Siren Alert: insert back to Elasticsearch bulk (using nginx or direct).

{
 "_index": "watcher",
 "_type": "watch",
 "_id": "surprise",
 "_score": 1,
 "_source": {
   "trigger": {
     "schedule": {
       "later": "every 50 seconds"
     }
   },
   "input": {
     "search": {
       "request": {
         "index": "my-requests-*",
         "body": {
           "query": {
             "filtered": {
               "query": {
                 "query_string": {
                   "query": "*",
                   "analyze_wildcard": true
                 }
               },
               "filter": {
                 "range": {
                   "@timestamp": {
                     "from": "now-5m"
                   }
                 }
               }
             }
           },
           "size": 0,
           "aggs": {
             "metrics": {
               "terms": {
                 "field": "first_url_part"
               }
             }
           }
         }
       }
     }
   },
   "condition": {
     "script": {
       "script": "payload.hits.total > 1"
     }
   },
   "transform": {},
   "actions": {
     "ES_bulk_request": {
       "throttle_period": "1m",
       "webhook": {
         "method": "POST",
         "host": "elasticsearch.foo.bar",
         "port": 80,
         "path": ":/_bulk",
         "body": "{{#payload.aggregations.metrics.buckets}}{\"index\":{\"_index\":\"aggregated_requests\", \"_type\":\"data\"}}\n{\"url\":\"{{key}}\", \"count\":\"{{doc_count}}\", \"execution_time\":\"tbd\"}\n{{/payload.aggregations.metrics.buckets}}",
         "headers": {
           "Content-Type": "text/plain; charset=ISO-8859-1"
         },
         "create_alert": true
       }
     }
   }
 }
}

Wizard

Siren Alert provides a wizard to help create watchers using a step-by-step sequence.

Give the watcher a name and choose an execution frequency. For example, run every day.

image

Specify the input query parameters and the condition to trigger on (based on a date histogram aggregation. For example, trigger an alert when there are more than two articles in an hour during the day.

image

To send an alert, set up a variety of actions to happen when your condition is met. For example, send a HTML-formatted email injected with data from the watcher and query response using the mustache templating language.

image

Custom watchers

Together with standard watchers, Siren Alert supports an additional list of use case-specific watcher types that are created from the dashboard. These types of watchers are called custom watchers.

Dashboard button

Custom watchers created from a dashboard inherit its search criteria, including the search pattern, search query, and filters. Each custom watcher type applies specific processing and trigger conditions.

image

Custom watcher types

  • New results: Alerts when there are new results for the given search.

  • Geo fence: Alerts when there are new results within a geographical area.

  • Proximity: Alerts when two entities are closer or further than a specified distance. This type must be enabled manually as it is not applicable to all data sets. Siren provides support for enabling this.

Creating custom watchers

Custom watchers are managed in the Script tab of the Management section. Creating a custom watcher requires writing a script that provides an object with the attributes described below. Siren provides support for creating these scripts.

Attribute Type Description

dashboard.order

number

Used to place watcher type in dashboard box.

dashboard.show

function(dashboard, indexMappings)

Tests whether to show the watcher type in the dashboard box.

params

object<string>

The collection of parameters and default values required by the watcher.

search

function(client, searchParams, params, watcher) ⇒ searchResponse

Executed in the backend that searches Elasticsearch using the client object. searchParams provides search criteria (index, filters, queries, time, originalTimeField). params provides direct access to collection of parameters on the watcher. watcher provides access to the watcher object. Example for the watcher object can be found here.

condition

function(searchResponse) ⇒ boolean

Evaluates the response from the search function and determines if the watcher should perform the alert actions.

template

string

Deprecated: Use attributes below. The HTML template presenting inputs for the watcher parameters. The params attribute is injected (for example ng-model="params.param1").

template.configuration

string

The HTML template presenting the configuration panel.

template.parameters

string

The HTML template presenting inputs for the watcher parameters. The params attribute is injected (for example ng-model="params.param1").

convertTime

function(start, end) ⇒ [moment | string, moment | string]

Converts the time that will be searched. Moment.js objects are passed in to facilitate relative date manipulation (e.g. (start, end) => ([start.startOf('day'), end.subtract(1, 'minute')])). Strings may also be returned for absolute times (e.g. (start, end) => (['2020-01-01T00:00:00.000', '2021-01-01T00:00:00.000'])).

alternativeTimeField.timeField

string

Changes the time field used in the search query. If this is set, dashboardTimeRange must also be set so the dashboard link applies the appropriate global time range. The link will also add a normal filter that selects the time range searched with the alternative time field.

alternativeTimeField.dashboardTimeRange

[string, string]

The global time range set on a dashboard by a watcher’s dashboard link. This is a mandatory field if alternativeTimeField.timeField is provided. This supports both absolute and relative times (e.g. ['1900-01-01T00:00:00', 'now-1y'])).

hideSettings

boolean

Default: false. Hides the settings on the default template for the configuration and actions panels.

addActionsList

array of objects

Defines the array of allowed actions for the watcher and the settings for these default actions. If any action type defined in actions is not found in addActionsList then that action will be disabled for the user. Schema is the same as schema for actions. For example, the following code allows the Elastic action to show up in the Add Actions dropdown on the watcher configuration page.

[{
  name: 'Elasticsearch alarm',
  typeLabel: 'Elastic',
  throttle_period: '1s'
}]

If this is an empty array, the "Add action" button will be hidden.

defaults

object or function

Allows passing of default values via custom watcher script. Currently only title and schedule are supported.

({
  ...,
  defaults: {
    title: 'my watcher',
    schedule: {
      frequency: 'every 5 minutes'
    }
  }
});

or using data provided from the dashboard

({
  ...,
  defaults: (dash) => ({
    title: dash.title,
    schedule: {
      frequency: dash.title === 'CDR' ? 'every 1 minutes' : 'every 1 hours'
    }
  })
});

paramsValidator

function(params) => void

Enables users to create bespoke validation rules for a watcher’s parameters. Throw an error if a parameter value is invalid, for example:

({
  ...,
  params: {
    property1: '<an-invalid-value>'
  },
  paramsValidator: (params) => ({
    if (params.property1 !== 'valid value!') {
      throw new Error('property1 is invalid');
    }
  })
});

Specific custom watcher information

Additional watcher information is provided in the watcher variable and can be accessed on watcher actions.

Property Type Description

dashboard_link

string

Dashboard link with the 'filtered by watcher' results

custom.dashboard_id

string

ID for associated dashboard

custom.index_pattern_id

string

ID for associated index pattern