Refreshing the Rancher cluster registration token

There may be a time when you'll want to refresh the "clusterregistrationtoken" or CRT for short. You can't do this in Rancher, as far as I know.

First, let's see where this token is saved.

marco@cp1:~$ kubectl get clusterregistrationtoken.management.cattle.io -A
NAMESPACE   NAME            AGE
local       default-token   6d16h

CRT's without extra cluster

Only the local cluster token is available right now.

apiVersion: management.cattle.io/v3
kind: ClusterRegistrationToken
metadata:
  creationTimestamp: "2022-01-15T20:33:33Z"
  generation: 3
  name: default-token
  namespace: local
  resourceVersion: "6746"
  uid: 6b444468-6fd1-44a0-b862-331056a88c4d
spec:
  clusterName: local
status:
  command: kubectl apply -f https://rancher.fe.ax/v3/import/xxxxx_local.yaml
  insecureCommand: curl --insecure -sfL https://rancher.fe.ax/v3/import/xxxxx_local.yaml
    | kubectl apply -f -
  insecureNodeCommand: ""
  insecureWindowsNodeCommand: ""
  manifestUrl: https://rancher.fe.ax/v3/import/xxxxx_local.yaml
  nodeCommand: sudo docker run -d --privileged --restart=unless-stopped --net=host
    -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run  rancher/rancher-agent:576d32a06
    --server https://rancher.fe.ax --token xxxxxxx
    --ca-checksum xxxxxxx
  token: xxxxxxx
  windowsNodeCommand: PowerShell -NoLogo -NonInteractive -Command "& {docker run -v
    c:\:c:\host rancher/rancher-agent:576d32a06 bootstrap --server https://rancher.fe.ax
    --token xxxxxxx --ca-checksum xxxxxxx
    | iex}"

CRT of local

Let's add another cluster. Once we complete the custom cluster creation wizard in Rancher's cluster management, we can see a new namespace created with a unique cluster-ID.

marco@cp1:~$ kubectl get ns
NAME                                     STATUS   AGE
local                                    Active   6d16h
p-rghcj                                  Active   6d16h
cattle-global-data                       Active   6d16h
p-k5j5m                                  Active   6d16h
kube-node-lease                          Active   6d17h
fleet-default                            Active   6d16h
default                                  Active   6d17h
kube-public                              Active   6d17h
cattle-impersonation-system              Active   6d16h
cattle-system                            Active   6d16h
cert-manager                             Active   6d16h
kube-system                              Active   6d17h
cattle-global-nt                         Active   6d16h
cattle-fleet-system                      Active   6d16h
cattle-fleet-clusters-system             Active   6d16h
fleet-local                              Active   6d16h
cluster-fleet-local-local-1a3d67d0a899   Active   6d16h
cattle-fleet-local-system                Active   6d16h
user-zd4f7                               Active   6d16h
c-75snr                                  Active   43m
p-skphs                                  Active   43m
p-phkbr                                  Active   43m

List of namespaces

Here we're checking out the CRT's after cluster creation.

marco@cp1:~$ kubectl get clusterregistrationtoken.management.cattle.io -A
NAMESPACE   NAME            AGE
local       default-token   6d16h
c-75snr     default-token   46m
c-75snr     crt-zztbw       46m

CRT list after cluster creation

We can get the join command from crt-zztbw. Why it's creating a separate token is unknown to me.

apiVersion: management.cattle.io/v3
kind: ClusterRegistrationToken
metadata:
  annotations:
    field.cattle.io/creatorId: user-zd4f7
  creationTimestamp: "2022-01-22T12:32:57Z"
  generateName: crt-
  generation: 2
  labels:
    cattle.io/creator: norman
  name: crt-zztbw
  namespace: c-75snr
  resourceVersion: "1753365"
  uid: d8bee26d-4f62-4743-90e2-4cce25f745f2
spec:
  clusterName: c-75snr
status:
  command: kubectl apply -f https://rancher.fe.ax/v3/import/jml8xtf7pwp8k2njknl7ctchz2r4glkn79bqbqdmvgt428ts7s2pb2_c-75snr.yaml
  insecureCommand: curl --insecure -sfL https://rancher.fe.ax/v3/import/jml8xtf7pwp8k2njknl7ctchz2r4glkn79bqbqdmvgt428ts7s2pb2_c-75snr.yaml
    | kubectl apply -f -
  insecureNodeCommand: ""
  insecureWindowsNodeCommand: ""
  manifestUrl: https://rancher.fe.ax/v3/import/jml8xtf7pwp8k2njknl7ctchz2r4glkn79bqbqdmvgt428ts7s2pb2_c-75snr.yaml
  nodeCommand: sudo docker run -d --privileged --restart=unless-stopped --net=host
    -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run  rancher/rancher-agent:576d32a06
    --server https://rancher.fe.ax --token jml8xtf7pwp8k2njknl7ctchz2r4glkn79bqbqdmvgt428ts7s2pb2
    --ca-checksum f37e412eaa6ed8f643af2cddeef25790eee501a6ec6b8578309059dd07f3ca37
  token: jml8xtf7pwp8k2njknl7ctchz2r4glkn79bqbqdmvgt428ts7s2pb2
  windowsNodeCommand: PowerShell -NoLogo -NonInteractive -Command "& {docker run -v
    c:\:c:\host rancher/rancher-agent:576d32a06 bootstrap --server https://rancher.fe.ax
    --token jml8xtf7pwp8k2njknl7ctchz2r4glkn79bqbqdmvgt428ts7s2pb2 --ca-checksum f37e412eaa6ed8f643af2cddeef25790eee501a6ec6b8578309059dd07f3ca37
    | iex}"

The new cluster registration token

Let Rancher provision the cluster for a while.

Cluster provisioning

After the cluster is provisioned, you can check the secrets for cattle credentials in the new cluster.

apiVersion: v1
data:
  namespace: Yy03NXNucg==
  token: cWpyZHE1OWQ3OGdtYnpscXA1dmt2enRjbnB0cm00ZDc4cjdxc3hycjl3dzVkOGxybHg4eHB3
  url: aHR0cHM6Ly9yYW5jaGVyLmZlLmF4
kind: Secret
metadata:
  annotations:
    field.cattle.io/projectId: c-75snr:p-phkbr
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"namespace":"Yy03NXNucg==","token":"cWpyZHE1OWQ3OGdtYnpscXA1dmt2enRjbnB0cm00ZDc4cjdxc3hycjl3dzVkOGxybHg4eHB3","url":"aHR0cHM6Ly9yYW5jaGVyLmZlLmF4"},"kind":"Secret","metadata":{"annotations":{},"name":"cattle-credentials-f945d4e","namespace":"cattle-system"},"type":"Opaque"}
  creationTimestamp: "2022-01-22T14:07:37Z"
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:data:
        .: {}
        f:namespace: {}
        f:token: {}
        f:url: {}
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
      f:type: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2022-01-22T14:07:37Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:field.cattle.io/projectId: {}
    manager: agent
    operation: Update
    time: "2022-01-22T16:31:53Z"
  name: cattle-credentials-f945d4e
  namespace: cattle-system
  resourceVersion: "13094"
  uid: 88f3edc0-088a-49cc-87eb-bd1ca80f4f55
type: Opaque

Cattle credentials

Now let's look at the part where I posted my credentials online, on a blog, for example, and want to invalidate that token.

I can remove the current CRT, which Fleet will regenerate. In this case, I remove all of them from the local (old) cluster.

marco@cp1:~$ kubectl delete clusterregistrationtoken.management.cattle.io -n c-75snr crt-zztbw
clusterregistrationtoken.management.cattle.io "crt-zztbw" deleted
marco@cp1:~$ kubectl delete clusterregistrationtoken.management.cattle.io -n c-75snr default-token
clusterregistrationtoken.management.cattle.io "default-token" deleted

Deleting CRT's

After which, Fleet adds a new one when we open up the registration page in Rancher.

Cluster registration page

The new token is visible in the local cluster.

marco@cp1:~$ kubectl get clusterregistrationtoken.management.cattle.io -n c-75snr
NAME            AGE
crt-zpbl6       4m5s
default-token   3m5s

CRT's are recreated by Rancher's Fleet

To efficiently show the tokens, we can use custom columns.

kubectl get clusterregistrationtoken.management.cattle.io -n c-75snr -o custom-columns=NAME:.metadata.name,TOKEN:.status.token
NAME            TOKEN
crt-zpbl6       k8zvqgbwdcg5cpbp9jnckb9g6m8z4555vqk79f7vzgpnd94clws2w4
default-token   jntphjz86wl7w7jh624pfchvhnrzgvhdtn26bwnhwfml8rtk4rsnrh

While the old CRT is invalidated, the "testcrt" cluster is still connected. When we reboot the cluster, the following happens.

testcrt cluster is unable to connect due to token mismatch

In the cattle-agent pod, the following logs appear.

time="2022-01-22T17:02:04Z" level=info msg="Connecting to wss://rancher.fe.ax/v3/connect/register with token starting with qjrdq59d78gmbzlqp5vkvztcnpt"
time="2022-01-22T17:02:04Z" level=info msg="Connecting to proxy" url="wss://rancher.fe.ax/v3/connect/register"
time="2022-01-22T17:02:04Z" level=error msg="Failed to connect to proxy. Response status: 400 - 400 Bad Request. Response body: cluster not found" error="websocket: bad handshake"
time="2022-01-22T17:02:04Z" level=error msg="Remotedialer proxy error" error="websocket: bad handshake"

Cattle agent log

We now need to patch the credentials secret. Be sure to encode the token with base64, and the encoded string is not ending in Cg==, which means "newline".

root@cp2:~# kubectl \
  --cluster='testcrt-cp2' \
  -n cattle-system patch secret cattle-credentials-f945d4e \
  --type='json' \
  -p='[{"op" : "replace" ,"path" : "/data/token" ,"value" : "azh6dnFnYndkY2c1Y3BicDlqbmNrYjlnNm04ejQ
1NTV2cWs3OWY3dnpncG5kOTRjbHdzMnc0"}]'
secret/cattle-credentials-f945d4e patched

Patching the new token

After the secret is patched, we need to redeploy the cattle agent to reload the token.

root@cp2:~# kubectl --cluster='testcrt-cp2' rollout restart deployment -n cattle-system cattle-cluster-agent
deployment.apps/cattle-cluster-agent restarted

Once cattle agent is restarted, we'll see it's available in Rancher dashboard again.

testcrt is back online