Published: Tue 13 September 2022
By Jan Almeroth
In blog .
tags: kubernetes etcd protobuf
When Ubuntu 22.04 LTS was released, I wanted to re-image my Kubernetes cluster nodes so they are in a clean state. Unfortunately, this comes with loosing the „age“ (the duration of cluster membership) for each node.
So I set out to find where this value comes from and if it's possible to tamper with it.
TL;DR Yes, it is. 🤓
Kubernetes
Lets see how far we can get with kubectl itself.
kubectl
# print nodes and their age as a table
kubectl get nodes
Editing a node (kubectl edit nodes <nodeName>) shows the field we are looking for is under the path .metadata.creationTimestamp, but changing it has no effect.
# print a node's creationTimestamp
kubectl get nodes <nodeName> -o json | jq ".metadata.creationTimestamp"
Lets describe our node.metadata.creationTimestamp field:
$ kubectl explain node.metadata.creationTimestamp
KIND: Node
VERSION: v1
FIELD: creationTimestamp <string>
DESCRIPTION:
CreationTimestamp is a timestamp representing the server time when this
object was created. It is not guaranteed to be set in happens-before order
across separate operations. Clients may not set this value. It is
represented in RFC3339 form and is in UTC.
Populated by the system. Read-only. Null for lists. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
Time is a wrapper around time.Time which supports correct marshaling to
YAML and JSON. Wrappers are provided for many of the factory methods that
the time package offers.
Important detail to remember from here:
Populated by the system. Read-only.
API documentation
Since kubectl explain told us to look at the Node v1 API object, we can see it includes ObjectMeta which should creationTimestamp. Alright!
etcd
What is etcd?
As the primary datastore of Kubernetes, etcd stores and replicates all Kubernetes cluster states.
I knew etcd holds all relevant information. So I started to familiarize with it's utilities:
etcdctl
# lets use etcd API version 3
export ETCDCTL_API = 3
# pass certs/keys and endpoints to talk to
export ETCD_ARGS = "\
--cacert /etc/kubernetes/pki/etcd/ca.crt \
--cert /etc/kubernetes/pki/etcd/peer.crt \
--key /etc/kubernetes/pki/etcd/peer.key \
--endpoints https://localhost:2379"
# list all keys stored in etcd
etcdctl $ETCD_ARGS get / --prefix --keys-only
After being successfully connected the question is: what are we looking for?
🍌 Minions 🍌
Listing all keys for a control node gave this:
$ etcdctl $ETCD_ARGS get / --prefix --keys-only | grep <nodeName>
/registry/csinodes/<nodeName>
/registry/leases/kube-node-lease/<nodeName>
/registry/minions/<nodeName>
/registry/pods/kube-system/etcd-<nodeName>
/registry/pods/kube-system/kube-apiserver-<nodeName>
/registry/pods/kube-system/kube-controller-manager-<nodeName>
/registry/pods/kube-system/kube-scheduler-<nodeName>
Listing all keys for a worker node gave that:
$ etcdctl $ETCD_ARGS get / --prefix --keys-only | grep <nodeName>
/registry/csinodes/<nodeName>
/registry/leases/kube-node-lease/<nodeName>
/registry/minions/<nodeName>
Looking at each record it was clear the node parameters are stored under /registry/minions/<nodeName>.
etcdctl $ETCD_ARGS get "/registry/minions/<nodeName>"
The output of the above command includes a lot of garbled (binary) data, lets apply some json magic:
# print json representation of the etcd record
etcdctl $ETCD_ARGS get "/registry/minions/<nodeName>" -w json
# print json representation of the etcd record and decodes the actual value
etcdctl $ETCD_ARGS get "/registry/minions/<nodeName>" -w json | jq -r ".kvs[0].value" | base64 -d
Meh, that's still binary data and no timestamp in sight. Lets see, how kube-etcd-helper might help us here.
kube-etcd-helper
export ETCD_HELPER_ARGS = "\
--cacert /etc/kubernetes/pki/etcd/ca.crt \
--cert /etc/kubernetes/pki/etcd/peer.crt \
--key /etc/kubernetes/pki/etcd/peer.key \
--endpoint https://localhost:2379"
# get value as json
kube-etcd-helper $ETCD_HELPER_ARGS get "/registry/minions/<nodeName>"
# get value as json and parse it with jq
kube-etcd-helper $ETCD_HELPER_ARGS get "/registry/minions/<nodeName>" | tail -n +2 | jq
{
"metadata" : {
"annotations" : {
"kubeadm.alpha.kubernetes.io/cri-socket" : "unix:///var/run/containerd/containerd.sock" ,
"node.alpha.kubernetes.io/ttl" : "0" ,
"volumes.kubernetes.io/controller-managed-attach-detach" : "true"
} ,
"creationTimestamp" : "2021-03-12T21:48:27Z" ,
"labels" : {
"beta.kubernetes.io/arch" : "arm64" ,
"beta.kubernetes.io/os" : "linux" ,
"kubernetes.io/arch" : "arm64" ,
"kubernetes.io/hostname" : "k8s-control-01" ,
"kubernetes.io/os" : "linux" ,
"node-role.kubernetes.io/control-plane" : "" ,
"node.kubernetes.io/exclude-from-external-load-balancers" : ""
} ,
[ ...]
}
}
Yes! Finally we found a data representation that is somewhat readable. So kube-etcd-helper knows how to decode that binary data.
But unfortunately, it does not have the ability to put things back into etcd.
python-etcd3
With python-etcd3 you can easily retrieve a record, do a search/replace and put it back into etcd.
#!/usr/bin/env python3
"""Demo python-etcd3."""
import etcd3
CA_CERT = "/etc/kubernetes/pki/etcd/ca.crt"
CERT_CERT = "/etc/kubernetes/pki/etcd/peer.crt"
CERT_KEY = "/etc/kubernetes/pki/etcd/peer.key"
def main ():
"""Provide main routine."""
k = "/registry/minions/<nodeName>"
c = etcd3 . client ( ca_cert = CA_CERT , cert_cert = CERT_CERT , cert_key = CERT_KEY )
v , _ = c . get ( k ) # retrieve a record
old = b " \x08\xf3\xfe\x81\x99\x06\x10\x00 " # 2022-09-13 13:02:43+00:00
new = b " \x08\xab\xc3\xaf\x82\x06\x10\x00 " # 2021-03-12 21:48:27+00:00
print ( v )
v . replace ( old , new , 1 ) # search/replace
print ( v )
c . put ( k , v ) # put it back
if __name__ == "__main__" :
main ()
Awesome! But what's this gibberish about?
Poking around
I have created some disposable objects with similar content and compared their resulting etcd records. Soon I could spot where to look for the creationTimestamp.
kubectl create configmap my-config1 --from-literal= key1 = config1 && \
sleep 1 && \
kubectl create configmap my-config2 --from-literal= key2 = config2
Then I tried to use one value for the other configmap and soon I made to actually replace it successfully. But I could not yet calculate those values yet... Googling around brought my attention to these files, where message Node and message ObjectMeta were defined:
- https://github.com/kubernetes/api/blob/master/core/v1/generated.proto
- https://github.com/kubernetes/apimachinery/blob/master/pkg/apis/meta/v1/generated.proto
Entering the world of Protobuf...
Protobuf
I have never used Protobuf before so I started with reading the documentation . Soon I played with my own first .proto files and some Python . I copied the Time definition from the Kubernetes project :
syntax = "proto2" ;
package k8s . io.apimachinery.pkg.apis.meta.v1 ;
message Time {
// Represents seconds of UTC time since Unix epoch
// 1970-01-01T00:00:00Z. Must be from 0001-01-01T00:00:00Z to
// 9999-12-31T23:59:59Z inclusive.
optional int64 seconds = 1 ;
// Non-negative fractions of a second at nanosecond resolution. Negative
// second values with fractions must still have non-negative nanos values
// that count forward in time. Must be from 0 to 999,999,999
// inclusive. This field may be limited in precision depending on context.
optional int32 nanos = 2 ;
}
After compiling this definition I could finally create my own creationTimestamp using Python:
protoc --python_out= ./ objectmeta.proto
#!/usr/bin/env python3
"""Demo protobuf."""
import objectmeta_pb2
def main ():
"""Provide main routine."""
fake = objectmeta_pb2 . Time ()
fake . seconds = 1615585707 # 2021-03-12 21:48:27+00:00
fake . nanos = 0
fake_data = fake . SerializeToString ()
print ( fake_data )
if __name__ == "__main__" :
main ()
The output of this script will be:
b'\x08\xab\xc3\xaf\x82\x06\x10\x00'
Now we can use this script to generate timestamps as we like and put these values into etcd using python-etcd3.