#StackBounty: #google-cloud-platform #kubernetes #google-kubernetes-engine #prometheus Correcting clock skew in a GKE cluster

Bounty: 50

I have the following alert configured in prometheus:

alert: ClockSkewDetected
expr: abs(node_timex_offset_seconds{job="node-exporter"})
  > 0.03
for: 2m
  severity: warning
  message: Clock skew detected on node-exporter {{ $labels.namespace }}/{{ $labels.pod }}. Ensure NTP is configured correctly on this host.

This alert is part of the default kube-prometheus stack which I am using.

I find this alert fires for around 10 mins every day or two.

I’d like to know how to deal with this problem (the alert firing!). It’s suggested in this answer that I shouldn’t need to run NTP (via a daemonset I guess) myself on GKE.

I’m also keen to use the kube-prometheus defaults where possible – so I’m unsure about increasing the 0.03 value.

Get this bounty!!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.