What can be a reliable way of clearing stale data from PushGateway for an app that has retired (e.g. instance is shut down on unusual circumstances) w/o using manual involvement?
Add a comment
|
2 Answers
you can clear / remove app's metric data/ job by using below url
job name: sample_job
curl --location --request DELETE 'http://localhost:9091/metrics/job/sample_job'
success code - 202
if you are running multiple instances of same app, you can delete individual instance's metric also,
job name: sample_job
and
InstnaceId = 20148
curl --location --request DELETE 'http://localhost:9091/metrics/job/sample_job/instance/20148'
success code - 202
but the only problem could be, If the app delete its own metric from the pushGateway upon its exit, prometheus may lose it. Yes, if the app exits between the prometheus scraping the pushgateway.
Comments
The Pushgateway is for service-level batch jobs, which are independent are individual serving instances that can go down. What you probably want to do is scrape this instance directly using Prometheus, rather than trying to convert Prometheus to push.
I'd recommend reading https://prometheus.io/docs/practices/pushing/
2 Comments
Divs
Thanks! Problem is, this is a work-around as the Prometheus sits outside the allowable network and can't access my app directly. So pushing the metrics to PushGateway by the apps and Prometheus scrapping the PushGateway seems to be the only way around for now.
brian-brazil
It's recommended to run Prometheus directly beside what it is scraping, which avoids that issue.