Set up advanced metrics in Sysdig Monitor
On this page
- Creating PromQL metrics
- Create a PromQL based alert
- Use Service Discovery to import application metrics endpoints
Creating PromQL metrics
Sysdig scrapes Prometheus metrics, you can create custom queries using PromQL. Here is a great way to start exploring:
Create a PromQL based alert
PromQL can be used in Alerts as well. The following example shows an alert for the Persistent Volume Utilization when hitting 80% full.
-
If you'd like to get PVC-specific metrics, for example, get the max percentage of storage usage:
max(kubelet_volume_stats_used_bytes{agent_tag_cluster="gold",persistentvolumeclaim="<PVC_name>"}) / max(kubelet_volume_stats_capacity_bytes{agent_tag_cluster="gold",persistentvolumeclaim="<PVC_name>"}) * 100
-
Sample PromQL Query:
((avg(kubelet_volume_stats_used_bytes/kubelet_volume_stats_capacity_bytes) by (persistentvolumeclaim)) * 100) >= 80
Use Service Discovery to import application metrics endpoints
Sysdig has a lightweight Prometheus server (Promscrape) that can import your application metrics endpoint into Sysdig metrics.
To enable Promscrape to find your application metrics, do the following:
- Make sure the application metrics endpoint is returning Prometheus metrics. To test this, you can expose the service and curl on the URL.
- Add the following annotations to the application pods:
prometheus.io/scrape: true
prometheus.io/port: <metrics_port>
prometheus.io/path: <metrics_path>
# the path is usually at /metrics
Don't add the annotations to the pods directly. This should be part of the infrastructure code and added in the templates. For example, if the application is using an OpenShift deployment, the annotation should be added at deployment.spec.template.metadata.annotations
.
- Once the annotation is in place, Sysdig can scrape them. On the Sysdig Explore tab, look for the sysdig metrics there (Sysdig does relabeling of the metrics, so they will appear as native sysdig metrics now instead of coming from promQL Query)
Related links:
- Set up a team in Sysdig Monitor
- Create alert channels in Sysdig Monitor
- devops-sysdig RocketChat channel
- Migrate Using Default Configuration
Related resources:
- Sysdig Monitor
- Sysdig Monitor Dashboards
- Sysdig Alerts
- Sysdig Alerts with Kubernetes and PromQL
- Sysdig Teams Blog
- Sysdig Teams Docs
- Sysdig User Management Docs
- Sysdig User Roles