I can’t get Prometheus to pick up on my cAdvisor metrics on DOKS. It shows as a successful scrape on the Prometheus dashboard, but I can’t get any container_* metrics to show up in the query. Any ideas?
Here’s my job config for cadvisor:
- job_name: kubernetes-cadvisor
honor_timestamps: true
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
scheme: https
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
kubernetes_sd_configs:
- role: node
relabel_configs:
- separator: ;
regex: __meta_kubernetes_node_label_(.+)
replacement: $1
action: labelmap
- source_labels: [__address__]
separator: ;
regex: ([^:]+)(?::\d+)?
target_label: __address__
replacement: $1:10250
action: replace
- separator: ;
regex: (.*)
target_label: __metrics_path__
replacement: /metrics/cadvisor
action: replace
metric_relabel_configs:
- source_labels: [namespace]
separator: ;
regex: ^$
replacement: $1
action: drop
- source_labels: [pod_name]
separator: ;
regex: ^$
replacement: $1
action: drop
I’ve also tried the metric path as /metrics/cadvisor like some issues of Github have suggested.
This config was taken from the tutorial: https://www.digitalocean.com/community/tutorials/how-to-set-up-a-kubernetes-monitoring-stack-with-prometheus-grafana-and-alertmanager-on-digitalocean#step-6-—-configuring-the-monitoring-stack-optional
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
pod_name
andcontainer_name
were removed from cadvisor on 1.16. Themetric_relabel_configs
onpod_name
is dropping the metrics. Try replace it with justpod
.See https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.16.md#removed-metrics
try using /metrics.It might work.
You can debug the target by going to the UI > Status > Targets. See if the target address is correct.