Documentation

ATLAS Alarm & Alert Service provides a simple way to store generated alarms, handles user subscriptions and generates alerts.

Overview

Components

Subscriptions & alerts

On the Subscriptions page, tick the events you want to hear about. Each subscription can be narrowed down with tags: usually network-site names. The suggestions follow what the event's latest alarms are tagged with: perfSONAR netsites for most network alarms, also networks (ASNs such as AS513) for path alarms such as suspected path element, and perfSONAR host names only for the few alarms about hosts, such as unresolvable host. Hosts can't be added to the other events: their alarms never carry one, so it would never match. Events that have never raised an alarm suggest sites. The suggestions are the netsites, hosts and networks seen in perfSONAR traceroutes over the past month, plus the tags the event's alarms carried in the past week. A site or host that is missing has had no measurements for a month, so its perfSONAR toolkit is probably not working properly; the page points this out. You are alerted only about alarms that carry at least one of your tags; leave the tags empty to receive all of them.

A tag can contain *, which stands for any characters: *MWT2* matches MWT2_UC, MWT2_IU and any other tag containing MWT2, and *-LHCOPNE every tag ending in -LHCOPNE. Tags and patterns ignore upper and lower case.

Events whose alarms report a severity also let you pick the levels you want; by default you get all of them.

When a site has a known, ongoing problem, Acknowledge it in the Ongoing issues panel to stop emails about that site for that event. The mute lasts a week (or 1 or 3 days if you choose), applies only to you, and ends on its own; you can unmute at any time. For events whose alarms report a severity you can also mute only one level, e.g. Mute only minor: alarms of the other levels are still emailed.

Alerts are bundled into a single email, sent every few hours according to the mail interval in your preferences. Turn on vacation mode there to pause them.

Publishing alarms

A new alarm can be created in three simple steps: register it using the REST API, add the alarm-generating code to your program, and run your program periodically.

Registering an alarm

Before registering a new kind of alarm, look at the classification tree of the current alarms and decide whether there already is a category and subcategory where the new alarm would naturally fit. Then describe it in a small JSON document:

{
  "category": "Analytics",
  "subcategory": "Frontier",
  "event": "Failed queries",
  "description": "Code running every 1h at UC k8s cluster. Checks all servers for: Rejected queries - server is busy and doesn't respond to the query; DB disconnections - the query was processed by the Frontier server but the Oracle DB terminated the connection. The code can be found here: https://github.com/ATLAS-Analytics/AlarmAndAlertService/blob/master/frontier-failed-q.py",
  "template": "Servers with failed queries:\n%{servers}\n. Affected tasks: \n%{tasks}\n\tConsult the following link to get a table with the most relevant taskids https://atlas-kibana.mwt2.org:5601/s/frontier/goto/c72d263c3e2b86f394ab99211c99b613\n"
}

Write the description in as much detail as possible: it tells people receiving the alarm how it was generated.

The template defines the text that appears in the alert email. Keep it brief, but give a reader enough to judge how serious the issue is, whether to forward it, and either act on it or start digging further. Every %{variable} is replaced with the value of variable from the alarm’s source field. Only \n and \t are available for formatting.

Once the document is ready, post it to the register endpoint.

Creating alarms

Once the alarm is registered, you can start creating alarms. They are simple JSON documents too:

{
  "category": "network",
  "subcategory": "perfsonar",
  "event": "high packet loss",
  "tags": ["UKI-SOUTHGRID-BRIS-HEP"],
  "body": "ps_1 needs 10000ms",
  "source": {
    "src": "137.222.79.1",
    "avg_value": 1.0,
    "host_src": "dice-io-37-00.acrc.bris.ac.uk",
    "site_src": "UKI-SOUTHGRID-BRIS-HEP"
  }
}

In addition to the classification, an alarm contains:

• tags: a single keyword or a short array that users can subscribe to. It lets them limit the number of alerts they receive. Tags must not contain spaces.
• body: a very brief description of the event.
• source: an optional map whose attributes are placed into the alert template.

Send the document to the create endpoint.

Periodic execution

Alarm-generating code is typically run on a schedule, for example as a cron job, so that new conditions are detected and posted as they happen.

Heartbeats

Heartbeats catch services that silently stop working. Register a heartbeat with an interval (seconds) and min_expected count, then have your service post heartbeats regularly. If fewer than min_expected heartbeats arrive within the interval, an alarm is raised.

{
  "category": "SLATE",
  "subcategory": "Squid",
  "event": "server down",
  "template": "Squid instance: %{instance}, running at %{site}, failed to send required number of heartbeats.",
  "description": "This alarm gets generated if a squid instance does not send at least 4 heartbeats in last 60 seconds. Heartbeats are sent in 10 second intervals.",
  "interval": 60,
  "min_expected": 4
}

REST API

The API has three parts: alarms, users and heartbeats. All request and response bodies are JSON.

Alarms

{
  "sites": ["CERN-PROD-LHCOPNE", "..."],
  "hosts": ["perfsonar.example.org", "..."],
  "asns": ["AS513", "..."],
  "labels": { "AS513": "CERN, CH" },
  "events": [
    { "category": "", "subcategory": "", "event": "", "kinds": ["site", "asn"], "tags": ["", ""] },
    ...
  ]
}

[
  { "category": "", "subcategory": "", "event": "", "levels": ["", ""] },
  ...
]

{
  "category": "",
  "subcategory": "",
  "event": "",
  "description": "",
  "template": ""
}

{
  "category": "",
  "subcategory": "",
  "event": "",
  "description": "",
  "template": ""
}

{
  "category": "",
  "subcategory": "",
  "event": "",
  "tags": ["", ""],
  "body": "",
  "source": { "xxx": "", "yyy": "", ... }
}

{
  "category": "",
  "subcategory": "",
  "event": "",
  "period": 6
}

{
  "category": "",
  "subcategory": "",
  "event": ""
}

Users

{
  "vacation": true
}

[
  {
    "category": "",
    "subcategory": "",
    "event": "",
    "tags": "MWT2 AGLT2",
    "levels": "*"
  },
  ...
]

[
  {
    "category": "",
    "subcategory": "",
    "event": "",
    "tag": "MWT2",
    "until": 1762790400000,
    "levels": ["minor"]
  },
  ...
]

Heartbeats

{
  "category": "",
  "subcategory": "",
  "event": "",
  "description": "",
  "template": "",
  "interval": 60,
  "min_expected": 4
}

{
  "category": "",
  "subcategory": "",
  "event": "",
  "description": "",
  "template": "",
  "interval": 60,
  "min_expected": 4
}

{
  "category": "",
  "subcategory": "",
  "event": "",
  "source": { "site": "MWT2", "instance": "MWT2_Slate_01" }
}

{
  "category": "",
  "subcategory": "",
  "event": "",
  "period": 6
}

{
  "category": "",
  "subcategory": "",
  "event": ""
}