Manually Interacting with KubeDNS Records

Reading time ~1 minute

A co-worker of mine was having some issues with KubeDNS in his GKE environment. He was then asking how to see if records had actually been added to DNS and I kind of shrugged (via Slack). But this got me a bit curious. How in the heck do you look and see? I thought the answer was at least worth writing down and remembering.

It’s Just etcd

The KubeDNS pod consists of four containers: etcd, kube2sky, exechealthz, and skydns. It’s kind of self-explanatory what each do, but etcd is a k/v store that holds the DNS records, kube2sky takes Kubernetes services and pods and updates etcd, and skydns is, guess what, a DNS server that uses etcd as its backend. So it looks like all roads point to etcd as far as where our records live.

Checking It Out

Here’s how to look at the records in the etcd container:

  • Find the full name of the pod for kube-dns with kubectl get po --all-namespaces. It should look like kube-dns-v11-xxxxx

  • Describe the pod to list the containers with kubectl describe po kube-dns-v11-xxxxx --namespace=kube-system. We already know what’s there, but it’s helpful anyways.

  • We will now exec into the etcd container and use it’s built-in tools to get the data we want. kubectl exec -ti --namespace=kube-system kube-dns-v11-xxxxx -c etcd -- /bin/sh

  • Once inside the container, let’s list all of the services in the default namespace (I’ve only got one):

# etcdctl ls skydns/local/cluster/svc/default

/skydns/local/cluster/svc/default/kubernetes
  • Now, find the key for that service by calling ls again:
# etcdctl ls skydns/local/cluster/svc/default/kubernetes

/skydns/local/cluster/svc/default/kubernetes/8618524b
  • Finally, we can return the data associated with that key by using the get command!
# etcdctl get skydns/local/cluster/svc/default/kubernetes/8618524b

{"host":"10.55.240.1","priority":10,"weight":10,"ttl":30,"targetstrip":0}

Other Notes

If you want to also test that things are working as expected inside the cluster, follow the great “How Do I Test If It’s Working?” section in the DNS addon repo here

A Noob's Guide to Custom Prometheus Exporters (Revamped!)

Someone was kind enough to send me an email thanking me for the [previous post](https://rsmitty.github.io/Prometheus-Exporters/) I create...… Continue reading

Building a Poor Man's Lightboard

Published on October 25, 2018

A Noob's Guide to Custom Prometheus Exporters

Published on February 28, 2018