Create alerts and notifications in Sysdig Monitor
Last updated: May 31, 2023
You can create alerts based on monitoring dashboards in Sysdig Monitor, that notify your team when something needs attention.
Here are some steps on how to setup the Sysdig alerts with Rocket.Chat.
On this page
- Create alerts and notifications in Sysdig Monitor
- Create a Rocket.Chat chat channel and webhook for alert messages
- Create a Sysdig team notification channel
- Creating an Alert
- Related pages
Create a Rocket.Chat chat channel and webhook for alert messages
Rocket.Chat requires an incoming webhook to parse the data from Sysdig.
-
Create a RC chat channel for the alert messages to arrive if there isn't one
-
Create an incoming webhook to the chat channel:
-
Click on your avatar -> Administration -> Integrations -> New (from Incoming tab)
-
Name the webhook in the format of
sysdig-alert-webhook-<APP_TEAM_NAME>
This is how the webhook should look:
Create a Sysdig team notification channel
Create a Sysdig team notification channel:
-
In Sysdig Monitor, go to your user account and click
Settings
-
Click
Notification Channels
andAdd Notification Channel
. ChooseCustom Webhook
as the type -
Use the webhook URL generated from Rocket.Chat and configure the notification channel. Name the channel in the format of
Rocketchat-alert-channel-<APP_TEAM_NAME>
-
Attach the following script into Editor, NOTE This is just an example, please update accordingly based on your use case:
{
"text": "Sysdig Notification",
"attachments": [{
"title": " {{@event_title}}",
"title_link": "{{@alert_url}}",
"pretext": " {{@alert_description}}",
"color": "#f93b10",
"fields": [
{
"title": "State",
"short": true,
"value": "{{@event_state}}"
},
{
"title": "Alert level",
"short": true,
{{#if_severity_high}}
"value": "incident"
{{#else}}
"value": "warning"
{{/if}}
},
{
"title": "Condition",
"short": false,
"value": "{{@alert_warning_condition}}"
},
{
"title": "Link to the Alert",
"short": false,
"value": "{{@alert_url}}"
}
]
}]
}
-
To test this notification, you have to pick one of the Select Alert Types before you can Send Test Notification
-
After tested, Click
Save
and now you should be able to see this notification option available in alert
Creating an Alert
It's recommended to create alerts from an application monitoring metrics, which helps to define good alerting thresholds.
-
Navigate to the
Alerts
section on the left hand navigation bar, you can see all the alerts within this Sysdig team scope -
To create an alert from metrics, head over to the dashboard. Pick the metric panel and click on
Create Alert
from kebab menu icon.
-
The alert contains:
-
Metric
: make sure if has the correct aggregation. Note: For PromQL based alerts, triggering threshold is defined as part of the metric query. See picture below! -
Scope
: the alert scope by default is set toeverywhere
, which means all namespaces from the cluster. Make sure you set the scope to your own namespaces if not specified. For example, you can usekube_namespace_name
andkube_cluster_name
. -
Trigger
: the triggering threshold should be based on the statistic from past monitoring. You can also refer to the data fromPreview
section on the right. -
Notify
: pick the Sysdig notification channel to send alert messages to. -
Others: feel free to explore other features available for alerting!
-
Related pages:
Related resources:
- Sysdig Monitor
- Sysdig Monitor Dashboards
- Sysdig Alerts
- Sysdig Alerts with Kubernetes and PromQL
- Sysdig Teams Blog
- Sysdig User Management Docs
- Sysdig User Roles