Update API server certSANs

[dungdm93]

What keywords did you search in kubeadm issues before filing this one?

apiserver
sa
certificate
certSANs

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):
v1.13.4

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:30:26Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: VM on GCE
• OS (e.g. from /etc/os-release): Ubuntu 18.04
• Kernel (e.g. uname -a): Linux kube-master 4.18.0-1007-gcp
• Others:

What happened?

After bootstrap k8s cluster using kubeadm, I realize that I forgot public IP/DNS of API server. So I'd like to reconfig k8s cluster by using kubeadm, but it doesn't work.

What you expected to happen?

Cluster update with new config.

How to reproduce it (as minimally and precisely as possible)?

1. Bootstrap cluster

root@kube-master:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc mq state UP group default qlen 1000
    link/ether 42:01:0a:94:00:02 brd ff:ff:ff:ff:ff:ff
    inet 10.148.0.2/32 scope global dynamic ens4
       valid_lft 2525sec preferred_lft 2525sec
    inet6 fe80::4001:aff:fe94:2/64 scope link 
       valid_lft forever preferred_lft forever
root@kube-master:~# kubeadm init
....

# Check PKI Certificates
root@kube-master:~# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout
....
X509v3 Subject Alternative Name: 
    DNS:kube-master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:10.96.0.1, IP Address:10.148.0.2
...

2. Get kubeadm config

kubeadm config view > kubeadm.yaml

3. Update kubeadm.yaml

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
apiServer:
+  certSANs:
+  - 35.198.193.188
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
- controlPlaneEndpoint: ""
+ controlPlaneEndpoint: 10.148.0.2:6443
kubernetesVersion: v1.13.4
...

4. Apply config

root@kube-master:~# kubeadm upgrade apply --config=kubeadm.yaml
root@kube-master:~# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout
# certSANs are the same as above

I also try to review certificates, but the result is the same.

root@kube-master:~# kubeadm alpha certs renew all --config kubeadm.yaml

[neolit123]

root@kube-master:~# kubeadm upgrade apply --config=kubeadm.yaml

try this command instead:
kubeadm config upload from-file --config kubeadm.yaml
https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-config/#cmd-config-from-file

then do kubeadm config view to see if you have made the change.

root@kube-master:~# kubeadm alpha certs renew all --config kubeadm.yaml

you first need to get the previous step right.
do you get some sort of an error here?

you also have the option to just restart kubeadm reset -f and kubeadm init...

[dungdm93]

config already published to k8s configmap with no need of kubeadm config upload. Proof is:

root@kube-master:~# kubectl get cm -n kube-system 
NAME                                 DATA   AGE
coredns                              1      4h29m
extension-apiserver-authentication   6      4h29m
kube-flannel-cfg                     2      4h28m
kube-proxy                           2      4h29m
kubeadm-config                       2      4h29m
kubelet-config-1.13                  1      4h29m

[fabriziopandini]

@dungdm93 I fear that there is no easy answer to your question...
Changing the SANs can be "easily" achieved, but changing the control-plane address complex. Here you can find some community provided solutions #338

[dungdm93]

@fabriziopandini actually, controlPlaneEndpoint in my cluster already is 10.148.0.2:6443. So there is no change control-plane address at all. I just wanna add extra SANs to my apiServer certificates.

[fabriziopandini]

@dungdm93
If it is only to get a certificate re-created, my first try would be

• delete the API server certificate
• recreate it using kubeadm init phase certs apiserver
• force restart the API server

(afaik kubeadm upgrade doesn't replace certificates unless they are near expiration)

[dungdm93]

@fabriziopandini I found a way to workaround but there is something is not true.

1. Before update:

# kubeadm config view
apiServer:
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: ""
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.13.4
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout
...
    X509v3 Subject Alternative Name: 
        DNS:master, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:10.96.0.1, IP Address:10.148.0.39
...

2. Update kubeadm config

# kubeadm upgrade apply --config=kubeadm.yml
# kubeadm config view
apiServer:
  certSANs:
  - 35.197.150.45
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta1
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: ""
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.13.4
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {}

3. Re-generate apiServer certificate

# rm -rf /etc/kubernetes/pki/apiserver.*
# kubeadm init phase certs apiserver
I0326 02:57:52.228780    2975 version.go:237] remote version is much newer: v1.14.0; falling back to: stable-1.13
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.148.0.39]

As you can see from the log, certSANs are still IPs [10.96.0.1 10.148.0.39]. However:

# rm -rf /etc/kubernetes/pki/apiserver.*
# kubeadm init phase certs apiserver --config=kubeadm.yml 
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.148.0.39 35.197.150.45]

This time, kubeadm add extra SAN into the certificate. However, in the third time, if I not specify --config=kubeadm.yml, kubeadm still apply old config version, although, I already update it.

# kubeadm init phase certs apiserver
I0326 02:57:52.228780    2975 version.go:237] remote version is much newer: v1.14.0; falling back to: stable-1.13
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.148.0.39]

[fabriziopandini]

@dungdm93 you have to update the kubeadm-config ConfigMap in the namespace kube-system to make your change persistent (this is what kubeadm uses if you don't provide --config ; same goes for updates)

[dungdm93]

@fabriziopandini In step 2 above, I already update kubeadm-config via

kubeadm upgrade apply --config=kubeadm.yml

Check it:

# kubectl get cm/kubeadm-config -n kube-system -o yaml
apiVersion: v1
data:
  ClusterConfiguration: |
    apiServer:
      certSANs:
      - 35.197.150.45
      extraArgs:
        authorization-mode: Node,RBAC
      timeoutForControlPlane: 4m0s
    apiVersion: kubeadm.k8s.io/v1beta1
    certificatesDir: /etc/kubernetes/pki
    clusterName: kubernetes
    controlPlaneEndpoint: ""
    controllerManager: {}
    dns:
      type: CoreDNS
    etcd:
      local:
        dataDir: /var/lib/etcd
    imageRepository: k8s.gcr.io
    kind: ClusterConfiguration
    kubernetesVersion: v1.13.4
    networking:
      dnsDomain: cluster.local
      podSubnet: 10.244.0.0/16
      serviceSubnet: 10.96.0.0/12
    scheduler: {}
  ClusterStatus: |
    apiEndpoints:
      master:
        advertiseAddress: 10.148.0.39
        bindPort: 6443
    apiVersion: kubeadm.k8s.io/v1beta1
    kind: ClusterStatus
kind: ConfigMap
metadata:
  creationTimestamp: "2019-03-25T14:58:35Z"
  name: kubeadm-config
  namespace: kube-system
  resourceVersion: "13969"
  selfLink: /api/v1/namespaces/kube-system/configmaps/kubeadm-config
  uid: 76d36104-4f0e-11e9-ac8b-42010a940027

[ryangrahamnc]

I get this issue too.

Dump the config to a file and edit:
kubeadm config view > /root/kubeadmconf.yml
vim /root/kubeadmconf.yml to add my new hostname:

apiServer:
  certSANs:
   - "example.com"
...

Apply the change:
kubeadm config upload from-file --config /root/kubeadmconf.yml

kubectl get -n kube-system cm kubeadm-config -o yaml shows my change

Then try to update the keys:

rm /etc/kubernetes/pki/apiserver.*
kubeadm init phase certs apiserver
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout

^ does not show my new hostname

Trying again with --config specified:

rm /etc/kubernetes/pki/apiserver.*
kubeadm init phase certs apiserver --config=/root/kubeadmconf.yml
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout

^ does show my new hostname has been applied

Restarting the api server (via docker ps | grep apiserver and docker restart ${id}) has no effect.