Metrics not showing on Grafana dashboard, at some points

Question

I have set up a Grafana dashboard, to monitor Kubernetes. It's working as expected but there's an issue of losing metrics/history at some point of time. See the attached screenshot. In the screenshot, no metrics are shown between 21:00 PM to 23:00 PM.

https://i.sstatic.net/BcMHT.jpg

From the above image, is that behavior normal? Thanks.

I'm using Grafana as an addon on Istio.

Part of the Istio helm chart configuration:

grafana:
  enabled: true
  persist: true  
  accessMode: ReadWriteMany

My expectation is that no history/metrics should be lost at any specific time.

Did you add persistentVolume for your prometheus?

FL3SH
– FL3SH

2019-08-13 07:51:35 +00:00
Commented Aug 13, 2019 at 7:51 — FL3SH
– FL3SH, Commented Aug 13, 2019 at 7:51

Marcin Ginszt · Accepted Answer · 2019-08-13 08:19:46Z

2

Istio uses Prometheus to collect the metrics. What you want to do, is to change a Prometheus data retention.

For example, if you are installing it with Helm, just change this parameter in the chart

--set prometheus.retention=2d

source: https://github.com/istio/istio/issues/12567

answered Aug 13, 2019 at 8:19

Marcin Ginszt

1648 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

learnc Over a year ago

This was very useful. Marked it an answer.

Collectives™ on Stack Overflow

Metrics not showing on Grafana dashboard, at some points

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related