Monitoring a PVC in Kubernetes with Prometheus

Something that I believe is a quite normal use case is that you’d want to monitor a Persistent Volume Claim (PVC) in kubernetes, to make sure that you get notified once you start running out of disk space. As a concept it sounds like something that should be supplied out of the box from Kubernetes, alas no.

Fortunately though it’s not too difficult, and there’s a really nice project that makes it easy to do.

The principle is easy, what you want to do is that you want to start a sidecar container in your Pod that periodically will simply run a df command on your volume to check how much disk space linux thinks your volume has.

What you need to do is add the sidecar in the template for your deployment in your kubernetes manifest:

        - name: volume-exporter
          image:  mnadeem/volume_exporter:v0.1.0
          args:
            - --volume-dir=prometheus:/var/lib/prometheus
          ports:
          - name: metrics-volume
            containerPort: 9888
          volumeMounts:
            - name: data
              mountPath: /var/lib/prometheus/
              readOnly: true

Further more you need to configure a scraper in prometheus that will scrape the metrics. Here it’s a bit tricky because it might be that you want prometheus to scrape both your application pod, and the sidecar that checks the PVC. This can be accomplished by adding some arbitrary custom annotation that you make your PVC scraper use:

  template:
    metadata:
      annotations:
        prometheus.io/pvc.path: /metrics
        prometheus.io/pvc.port: '9888'
        prometheus.io/pvc.scrape: 'true'

In your scraper config you configure it to read the configuration:

      - job_name: 'pvc-sidecar-scraper'
        kubernetes_sd_configs:
          - role: pod

        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_pvc_scrape]
            action: keep
            regex: "true"

          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_pvc_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_pvc_port]
            action: replace
            regex: ([^:]+)(?::\d+)?;(\d+)
            replacement: $1:$2
            target_label: __address__

And that’s it, now you’ll get your metrics nicely recorded.