Notifications Reference¶

ZMON provides several means of notification in case of alerts. Notifications will be triggered when alert status change. Please refer to Notification options for different worker configuration options.

Google Hangouts Chat¶

Notify Google Hangouts Chat room with alert status.

send_google_hangouts_chat(webhook_link=None, message=None, color='red')¶

Send Google Hangouts Chat notification.

Parameters:

webhook_link (str) – Webhook Link in Google Hangouts Chat Room. Create a Google Hangouts Chat Webhook and copy the link here.
multiline (bool) – Should the Text in the notification span multiple lines or not? Default is True.
message (str) – Message to be sent. If None, then a message constructed from the alert will be sent.
color (str) – Message color. Default is red if alert is raised.

Note

Message color will be determined based on alert status. If alert has ended, then color will be green, otherwise color argument will be used.

Hipchat¶

Notify Hipchat room with alert status.

send_hipchat(room=None, message=None, token=None, message_format='html', notify=False, color='red', link=False, link_text='go to alert')¶

Send Hipchat notification to specified room.

Parameters:

room (str) – Room to be notified.
message (str) – Message to be sent. If None, then a message constructed from the alert will be sent.
token (str) – Hipchat API token.
message_format (str) – message format - html (default) or text (which correctly treats @mentions).
notify (bool) – Hipchat notify flag. Default is False.
color (str) – Message color. Default is red if alert is raised.
link (bool) – Add link to Hipchat message. Default is False.
link_text (str) – if link param is True, this will be displayed as a link in the hipchat message. Default is go to alert.

Note

Message color will be determined based on alert status. If alert has ended, then color will be green, otherwise color argument will be used.

Example message - using html format (default):

{
    "message": "NEW ALERT: Requests failing with status 500 on host-production-1-entity",
    "color": "red",
    "notify": true
}

Example message - using text format with @mention:

{
    "message": "@here NEW ALERT: Requests failing with status 500 on host-production-1-entity",
    "color": "red",
    "notify": true,
    "message_format": "text"
}

HTTP¶

Provides notification by invoking HTTP call to certain endpoint. HTTP notification uses POST method when invoking the call.

notify_http(url=None, body=None, params=None, headers=None, timeout=5, oauth2=False, include_alert=True)¶

Send HTTP notification to specified endpoint.

Parameters:

url (str) – HTTP endpoint URL. If not passed, then default URL will be used in worker configuration.
body (dict) – Request body.
params (dict) – Request URL params.
headers (dict) – HTTP headers.
timeout (int) – Request timeout. Default is 5 seconds.
oauth2 (bool) – Add OAUTH2 authentication headers. Default is False.
include_alert (bool) – Include alert data in request body. Default is True.

Example:

notify_http('https://some-notification-service/alert', body={'zmon': True}, headers={'X-TOKEN': 1234})

Note

If include_alert is True, then request body will include alert data. This is usually useful, since it provides valuable info like is_alert and changed which can indicate whether the alert has started or ended.

{
    "body": null,
    "alert": {
        "is_alert": true,
        "changed": true,
        "duration": 2.33,
        "captures": {},
        "entity": {"type": "GLOBAL", "id": "GLOBAL"},
        "worker": "plocal.zmon",
        "value": {"td": 0.00037, "worker": "plocal.zmon", "ts": 1472032348.665247, "value": 51.67797677979191},
        "alert_def": {
            "name": "Random Example Alert", "parameters": null, "check_id": 4, "entities_map": [], "responsible_team": "ZMON", "period": "", "priority": 1,
            "notifications": ["notify_http()"], "team": "ZMON", "id": 3, "condition": ">40"
        }
    }
}

Hubot¶

Send Hubot notification.

notify_hubot(queue, hubot_url, message=None)¶

Send Hubot notification.

Parameters:	queue (str) – Hubot queue. hubot_url (str) – Hubot url. message (str) – Notification message.

Mail¶

Send email notifications.

send_mail(subject=None, cc=None, html=False, hide_recipients=True, include_value=True, include_definition=True, include_captures=True, include_entity=True, per_entity=True)¶

Send email notification.

Parameters:

subject (str or unicode or None) – Email subject. You must use a unicode string (e.g. u’äöüß’) if you have non-ASCII characters in there. If None, the alert name will be used.
cc (list) – List of CC recipients.
html (bool) – HTML email.
hide_recipients (bool) – Hide recipients. Will be sent as BCC.
include_value (bool) – Include alert value in notification message.
include_definition (bool) – Include alert definition details in notification message.
include_captures (bool) – Include alert captures in message.
include_entity (bool) – Include affected entities in notification message.
per_entity (bool) – Send new email notification per entity. Default is True.

Note

send_email is an alias for this notification function.

Opsgenie¶

Notify Opsgenie of a new alert status. If alert is active, then a new opsgenie alert will be created. If alert is inactive then the alert will be closed.

notify_opsgenie(message='', teams=None, per_entity=False, priority=None, include_alert=True, description='', **kwargs)¶

Send notifications to Opsgenie.

Parameters:

message (str) – Alert message. If empty, then a message will be generated from the alert data.
teams (str | list) – Opsgenie teams to be notified. Value can be a single team or a list of teams.
per_entity (bool) – Send new alert per entity. This affects the alias value and impacts how de-duplication is handled in Opsgenie. Default is False.
priority (str) – Set Opsgenie priority for this notification. Valid values are P1, P2, P3, P4 or P5.
include_alert (bool) – Include alert data in alert body details. Default is True.
include_captures (bool) – Include captures data in alert body details. Default is False.
description (str) – An optional description. If present, this is inserted into the opsgenie alert description field.

Example:

notify_opsgenie(teams=['zmon', 'ops'], message='Number of failed requests is too high!', include_alert=True)

Note

If priority is not set, then ZMON will set the priority according to the alert priority.

Pagerduty¶

Notify Pagerduty of a new alert status. If alert is active, then a new pagerduty incident with type trigger will be sent. If alert is inactive then incident type will be updated to resolve.

Note

Pagerduty notification plugin uses API v2.

notify_pagerduty(message='', per_entity=False, include_alert=True, routing_key=None, alert_class=None, alert_group=None, **kwargs)¶

Send notifications to Pagerduty.

Parameters:

message (str) – Incident message. If empty, then a message will be generated from the alert data.
per_entity (bool) – Send new alert per entity. This affects the dedup_key value and impacts how de-duplication is handled in Pagerduty. Default is False.
include_alert (bool) – Include alert data in incident payload custom_details. Default is True.
routing_key (str) – Pagerduty service routing_key. If not specified, then the service key configured for the worker will be used.
alert_class (str) – Set the Pagerduty incident class.
alert_group (str) – Set the Pagerduty incident group.

Example:

notify_pagerduty(message='Number of failed requests is too high!', include_alert=True, alert_class='API health', alert_group='production')

Push¶

Send push notification via ZMON notification service.

send_push(url=None, key=None, message=None)¶

Send Push notification to mobile devices.

Parameters:	url (str) – Notification service base URL. key (str) – Notification service API key. message (str) – Message to be sent in notification.

Note

If Message is None then it will be generated from alert status.

Slack¶

Notify Slack channel with alert status. A webhook is required for notifications.

notify_slack(webhook=None, channel='#general', message=None)¶

Send Slack notification to specified channel.

Parameters:	webhook (str) – Slack webhook. If not set, then webhook set in configuration will be used. channel (str) – Channel to be notified. Default is `#general`. message (str) – Message to be sent. If `None`, then a message constructed from the alert will be sent.

Twilio¶

Use Twilio to receive phone calls if alerts pop up. This includes basic ACK and escalation. Requires account at Twilio and the notifiction service deployed. Low investment to get going though. WORK IN PROGRESS.

notifiy_twilio(numbers=[], message="ZMON Alert Up: Some Alert")¶

Make phone call to supplied numbers. First number will be called immediately. After two minutes, another call is made to that number if no ACK. Other numbers follow at 5min interval without ACK.

Parameters:	message (str) – Message to be sent. If `None`, then a message constructed from the alert will be sent. numbers – Numbers to call

Note

Remember to configure your worker for this.

NOTIFICATION_SERVICE_URL
NOTIFICATION_SERVICE_KEY