Set up advanced metrics in Sysdig Monitor

Creating PromQL metrics

Sysdig scrapes Prometheus metrics, you can create custom queries using PromQL. Here is a great way to start exploring:

Create a PromQL based alert

PromQL can be used in Alerts as well. The following example shows an alert for the Persistent Volume Utilization when hitting 80% full.

If you'd like to get PVC-specific metrics, for example, get the max percentage of storage usage:

max(kubelet_volume_stats_used_bytes{agent_tag_cluster="gold",persistentvolumeclaim="<PVC_name>"}) / max(kubelet_volume_stats_capacity_bytes{agent_tag_cluster="gold",persistentvolumeclaim="<PVC_name>"}) * 100
Sample PromQL Query:

((avg(kubelet_volume_stats_used_bytes/kubelet_volume_stats_capacity_bytes) by (persistentvolumeclaim)) * 100) >= 80

Use Service Discovery to import application metrics endpoints

Sysdig has a lightweight Prometheus server (Promscrape) that can import your application metrics endpoint into Sysdig metrics.

To enable Promscrape to find your application metrics, do the following:

Make sure the application metrics endpoint is returning Prometheus metrics. To test this, you can expose the service and curl on the URL.
Add the following annotations to the application pods:

prometheus.io/scrape: true
prometheus.io/port: <metrics_port>
prometheus.io/path: <metrics_path>
# the path is usually at /metrics

Don't add the annotations to the pods directly. This should be part of the infrastructure code and added in the templates. For example, if the application is using an OpenShift deployment, the annotation should be added at deployment.spec.template.metadata.annotations.

Once the annotation is in place, Sysdig can scrape them. On the Sysdig Explore tab, look for the sysdig metrics there (Sysdig does relabeling of the metrics, so they will appear as native sysdig metrics now instead of coming from promQL Query)

Set up advanced metrics in Sysdig Monitor

On this page

Creating PromQL metrics

Create a PromQL based alert

Use Service Discovery to import application metrics endpoints