Skip to main content

Set up advanced metrics in Sysdig Monitor

Last updated: January 2, 2024

This page is a comprehensive guide to setting up advanced metrics in Sysdig Monitor. It covers the creation of PromQL metrics, using PromQL in alerts, and employing Service Discovery to import application metrics endpoints into Sysdig. The guide includes practical examples and step-by-step instructions, serving as a valuable resource for users looking to enhance their monitoring capabilities with Sysdig.

On this page

Creating PromQL metrics

Sysdig scrapes Prometheus metrics, and you can leverage PromQL to create custom queries. PromQL is particularly beneficial for advanced metric exploration and offers flexibility in obtaining specific insights from your data. To get started, consider the following example:

Sysdig exploring

Create a PromQL based alert

PromQL can be used in Alerts as well. The following example shows an alert for the Persistent Volume Utilization when hitting 80% full.

  • If you'd like to get PVC-specific metrics, for example, get the max percentage of storage usage:

    max(kubelet_volume_stats_used_bytes{agent_tag_cluster="gold",persistentvolumeclaim="<PVC_name>"}) / max(kubelet_volume_stats_capacity_bytes{agent_tag_cluster="gold",persistentvolumeclaim="<PVC_name>"}) * 100

  • Sample PromQL Query:

    ((avg(kubelet_volume_stats_used_bytes/kubelet_volume_stats_capacity_bytes) by (persistentvolumeclaim)) * 100) >= 80

Configure PromQL alert

Use Service Discovery to import application metrics endpoints

Sysdig has a lightweight Prometheus server (Promscrape) that can import your application metrics endpoint into Sysdig metrics.

To enable Promscrape to find your application metrics, do the following:

  1. Make sure the application metrics endpoint is returning Prometheus metrics. To test this, you can expose the service and curl on the URL.
  2. Add the following annotations to the application pods:
prometheus.io/scrape: true
prometheus.io/port: <metrics_port>
prometheus.io/path: <metrics_path>
# the path is usually at /metrics

Don't add the annotations to the pods directly. This should be part of the infrastructure code and added in the templates. For example, if the application is using an OpenShift deployment, the annotation should be added at deployment.spec.template.metadata.annotations.

  1. Once the annotation is in place, Sysdig can scrape them. On the Sysdig Explore tab, look for the sysdig metrics there (Sysdig does relabeling of the metrics, so they will appear as native sysdig metrics now instead of coming from promQL Query)