Monitoring a PVC in Kubernetes with Prometheus
Something that I believe is a quite normal use case is that you’d want to monitor a Persistent Volume Claim (PVC) in kubernetes, to make sure that you get notified once you start running out of disk space. As a concept it sounds like something that should be supplied out of the box from Kubernetes, alas no.
Fortunately though it’s not too difficult, and there’s a really nice project that makes it easy to do.
The principle is easy, what you want to do is that you want to start a sidecar container in your
Pod that periodically will simply run a
df command on your volume to check how much disk space linux thinks your volume has.
What you need to do is add the sidecar in the template for your deployment in your kubernetes manifest:
- name: volume-exporter image: mnadeem/volume_exporter:v0.1.0 args: - --volume-dir=prometheus:/var/lib/prometheus ports: - name: metrics-volume containerPort: 9888 volumeMounts: - name: data mountPath: /var/lib/prometheus/ readOnly: true
Further more you need to configure a scraper in prometheus that will scrape the metrics. Here it’s a bit tricky because it might be that you want prometheus to scrape both your application pod, and the sidecar that checks the PVC. This can be accomplished by adding some arbitrary custom annotation that you make your
PVC scraper use:
template: metadata: annotations: prometheus.io/pvc.path: /metrics prometheus.io/pvc.port: '9888' prometheus.io/pvc.scrape: 'true'
In your scraper config you configure it to read the configuration:
- job_name: 'pvc-sidecar-scraper' kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_pvc_scrape] action: keep regex: "true" - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_pvc_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_pvc_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target_label: __address__
And that’s it, now you’ll get your metrics nicely recorded.