Refreshing the Rancher cluster registration token
There may be a time when you'll want to refresh the "clusterregistrationtoken" or CRT for short. You can't do this in Rancher, as far as I know.
First, let's see where this token is saved.
marco@cp1:~$ kubectl get clusterregistrationtoken.management.cattle.io -A
NAMESPACE NAME AGE
local default-token 6d16hOnly the local cluster token is available right now.
apiVersion: management.cattle.io/v3
kind: ClusterRegistrationToken
metadata:
creationTimestamp: "2022-01-15T20:33:33Z"
generation: 3
name: default-token
namespace: local
resourceVersion: "6746"
uid: 6b444468-6fd1-44a0-b862-331056a88c4d
spec:
clusterName: local
status:
command: kubectl apply -f https://rancher.fe.ax/v3/import/xxxxx_local.yaml
insecureCommand: curl --insecure -sfL https://rancher.fe.ax/v3/import/xxxxx_local.yaml
| kubectl apply -f -
insecureNodeCommand: ""
insecureWindowsNodeCommand: ""
manifestUrl: https://rancher.fe.ax/v3/import/xxxxx_local.yaml
nodeCommand: sudo docker run -d --privileged --restart=unless-stopped --net=host
-v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:576d32a06
--server https://rancher.fe.ax --token xxxxxxx
--ca-checksum xxxxxxx
token: xxxxxxx
windowsNodeCommand: PowerShell -NoLogo -NonInteractive -Command "& {docker run -v
c:\:c:\host rancher/rancher-agent:576d32a06 bootstrap --server https://rancher.fe.ax
--token xxxxxxx --ca-checksum xxxxxxx
| iex}"
Let's add another cluster. Once we complete the custom cluster creation wizard in Rancher's cluster management, we can see a new namespace created with a unique cluster-ID.
marco@cp1:~$ kubectl get ns
NAME STATUS AGE
local Active 6d16h
p-rghcj Active 6d16h
cattle-global-data Active 6d16h
p-k5j5m Active 6d16h
kube-node-lease Active 6d17h
fleet-default Active 6d16h
default Active 6d17h
kube-public Active 6d17h
cattle-impersonation-system Active 6d16h
cattle-system Active 6d16h
cert-manager Active 6d16h
kube-system Active 6d17h
cattle-global-nt Active 6d16h
cattle-fleet-system Active 6d16h
cattle-fleet-clusters-system Active 6d16h
fleet-local Active 6d16h
cluster-fleet-local-local-1a3d67d0a899 Active 6d16h
cattle-fleet-local-system Active 6d16h
user-zd4f7 Active 6d16h
c-75snr Active 43m
p-skphs Active 43m
p-phkbr Active 43mHere we're checking out the CRT's after cluster creation.
marco@cp1:~$ kubectl get clusterregistrationtoken.management.cattle.io -A
NAMESPACE NAME AGE
local default-token 6d16h
c-75snr default-token 46m
c-75snr crt-zztbw 46m
We can get the join command from crt-zztbw. Why it's creating a separate token is unknown to me.
apiVersion: management.cattle.io/v3
kind: ClusterRegistrationToken
metadata:
annotations:
field.cattle.io/creatorId: user-zd4f7
creationTimestamp: "2022-01-22T12:32:57Z"
generateName: crt-
generation: 2
labels:
cattle.io/creator: norman
name: crt-zztbw
namespace: c-75snr
resourceVersion: "1753365"
uid: d8bee26d-4f62-4743-90e2-4cce25f745f2
spec:
clusterName: c-75snr
status:
command: kubectl apply -f https://rancher.fe.ax/v3/import/jml8xtf7pwp8k2njknl7ctchz2r4glkn79bqbqdmvgt428ts7s2pb2_c-75snr.yaml
insecureCommand: curl --insecure -sfL https://rancher.fe.ax/v3/import/jml8xtf7pwp8k2njknl7ctchz2r4glkn79bqbqdmvgt428ts7s2pb2_c-75snr.yaml
| kubectl apply -f -
insecureNodeCommand: ""
insecureWindowsNodeCommand: ""
manifestUrl: https://rancher.fe.ax/v3/import/jml8xtf7pwp8k2njknl7ctchz2r4glkn79bqbqdmvgt428ts7s2pb2_c-75snr.yaml
nodeCommand: sudo docker run -d --privileged --restart=unless-stopped --net=host
-v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:576d32a06
--server https://rancher.fe.ax --token jml8xtf7pwp8k2njknl7ctchz2r4glkn79bqbqdmvgt428ts7s2pb2
--ca-checksum f37e412eaa6ed8f643af2cddeef25790eee501a6ec6b8578309059dd07f3ca37
token: jml8xtf7pwp8k2njknl7ctchz2r4glkn79bqbqdmvgt428ts7s2pb2
windowsNodeCommand: PowerShell -NoLogo -NonInteractive -Command "& {docker run -v
c:\:c:\host rancher/rancher-agent:576d32a06 bootstrap --server https://rancher.fe.ax
--token jml8xtf7pwp8k2njknl7ctchz2r4glkn79bqbqdmvgt428ts7s2pb2 --ca-checksum f37e412eaa6ed8f643af2cddeef25790eee501a6ec6b8578309059dd07f3ca37
| iex}"Let Rancher provision the cluster for a while.
After the cluster is provisioned, you can check the secrets for cattle credentials in the new cluster.
apiVersion: v1
data:
namespace: Yy03NXNucg==
token: cWpyZHE1OWQ3OGdtYnpscXA1dmt2enRjbnB0cm00ZDc4cjdxc3hycjl3dzVkOGxybHg4eHB3
url: aHR0cHM6Ly9yYW5jaGVyLmZlLmF4
kind: Secret
metadata:
annotations:
field.cattle.io/projectId: c-75snr:p-phkbr
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"namespace":"Yy03NXNucg==","token":"cWpyZHE1OWQ3OGdtYnpscXA1dmt2enRjbnB0cm00ZDc4cjdxc3hycjl3dzVkOGxybHg4eHB3","url":"aHR0cHM6Ly9yYW5jaGVyLmZlLmF4"},"kind":"Secret","metadata":{"annotations":{},"name":"cattle-credentials-f945d4e","namespace":"cattle-system"},"type":"Opaque"}
creationTimestamp: "2022-01-22T14:07:37Z"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:namespace: {}
f:token: {}
f:url: {}
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:type: {}
manager: kubectl-client-side-apply
operation: Update
time: "2022-01-22T14:07:37Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:field.cattle.io/projectId: {}
manager: agent
operation: Update
time: "2022-01-22T16:31:53Z"
name: cattle-credentials-f945d4e
namespace: cattle-system
resourceVersion: "13094"
uid: 88f3edc0-088a-49cc-87eb-bd1ca80f4f55
type: OpaqueNow let's look at the part where I posted my credentials online, on a blog, for example, and want to invalidate that token.
I can remove the current CRT, which Fleet will regenerate. In this case, I remove all of them from the local (old) cluster.
marco@cp1:~$ kubectl delete clusterregistrationtoken.management.cattle.io -n c-75snr crt-zztbw
clusterregistrationtoken.management.cattle.io "crt-zztbw" deleted
marco@cp1:~$ kubectl delete clusterregistrationtoken.management.cattle.io -n c-75snr default-token
clusterregistrationtoken.management.cattle.io "default-token" deleted
After which, Fleet adds a new one when we open up the registration page in Rancher.
The new token is visible in the local cluster.
marco@cp1:~$ kubectl get clusterregistrationtoken.management.cattle.io -n c-75snr
NAME AGE
crt-zpbl6 4m5s
default-token 3m5sTo efficiently show the tokens, we can use custom columns.
kubectl get clusterregistrationtoken.management.cattle.io -n c-75snr -o custom-columns=NAME:.metadata.name,TOKEN:.status.token
NAME TOKEN
crt-zpbl6 k8zvqgbwdcg5cpbp9jnckb9g6m8z4555vqk79f7vzgpnd94clws2w4
default-token jntphjz86wl7w7jh624pfchvhnrzgvhdtn26bwnhwfml8rtk4rsnrhWhile the old CRT is invalidated, the "testcrt" cluster is still connected. When we reboot the cluster, the following happens.
In the cattle-agent pod, the following logs appear.
time="2022-01-22T17:02:04Z" level=info msg="Connecting to wss://rancher.fe.ax/v3/connect/register with token starting with qjrdq59d78gmbzlqp5vkvztcnpt"
time="2022-01-22T17:02:04Z" level=info msg="Connecting to proxy" url="wss://rancher.fe.ax/v3/connect/register"
time="2022-01-22T17:02:04Z" level=error msg="Failed to connect to proxy. Response status: 400 - 400 Bad Request. Response body: cluster not found" error="websocket: bad handshake"
time="2022-01-22T17:02:04Z" level=error msg="Remotedialer proxy error" error="websocket: bad handshake"We now need to patch the credentials secret. Be sure to encode the token with base64, and the encoded string is not ending in Cg==, which means "newline".
root@cp2:~# kubectl \
--cluster='testcrt-cp2' \
-n cattle-system patch secret cattle-credentials-f945d4e \
--type='json' \
-p='[{"op" : "replace" ,"path" : "/data/token" ,"value" : "azh6dnFnYndkY2c1Y3BicDlqbmNrYjlnNm04ejQ
1NTV2cWs3OWY3dnpncG5kOTRjbHdzMnc0"}]'
secret/cattle-credentials-f945d4e patched
After the secret is patched, we need to redeploy the cattle agent to reload the token.
root@cp2:~# kubectl --cluster='testcrt-cp2' rollout restart deployment -n cattle-system cattle-cluster-agent
deployment.apps/cattle-cluster-agent restartedOnce cattle agent is restarted, we'll see it's available in Rancher dashboard again.
