Getting "connection refused" on 6443/api/v1/nodes for a brand new controlplane

[TheSirC]

After installing Talos on a fresh new node and bootstrapping the cluster on that node I get :

talosctl --talosconfig=./talosconfig -n 192.168.1.100 health --wait-timeout 0m59s
discovered nodes: ["192.168.1.102" "192.168.1.104" "192.168.1.103" "192.168.1.105" "192.168.1.100"]
waiting for etcd to be healthy: ...
waiting for etcd to be healthy: OK
waiting for etcd members to be consistent across nodes: ...
waiting for etcd members to be consistent across nodes: OK
waiting for etcd members to be control plane nodes: ...
waiting for etcd members to be control plane nodes: OK
waiting for apid to be ready: ...
waiting for apid to be ready: OK
waiting for all nodes memory sizes: ...
waiting for all nodes memory sizes: OK
waiting for all nodes disk sizes: ...
waiting for all nodes disk sizes: OK
waiting for no diagnostics: ...
waiting for no diagnostics: OK
waiting for kubelet to be healthy: ...
waiting for kubelet to be healthy: OK
waiting for all nodes to finish boot sequence: ...
waiting for all nodes to finish boot sequence: OK
waiting for all k8s nodes to report: ...
waiting for all k8s nodes to report: Get "https://192.168.1.100:6443/api/v1/nodes": dial tcp 192.168.1.100:6443: connect: connection refused

Any idea where that could be coming from ?

Extra info

talosctl version --nodes 192.168.1.100
Client:
        Tag:         v1.10.2
        SHA:         undefined
        Built:
        Go version:  go1.24.3
        OS/Arch:     linux/amd64
Server:
        NODE:        192.168.1.100
        Tag:         v1.10.2
        SHA:         1cf5914b
        Built:
        Go version:  go1.24.3
        OS/Arch:     linux/arm64
        Enabled:     RBAC

controlplane.yaml (also [here](https://gist.github.com/TheSirC/5689f90a11fe54dd06a2f9f1dd23d691))

```yaml

version: v1alpha1 # Indicates the schema used to decode the contents.
debug: false # Enable verbose logging to the console.
persist: true

Provides machine specific configuration options.

machine:
type: controlplane # Defines the role of the machine within the cluster.
token: # The token is used by a machine to join the PKI of the cluster.
# The root certificate authority of the PKI.
ca:
crt: ==
key:
# Extra certificate subject alternative names for the machine's certificate.
certSANs: []
# # Uncomment this to enable SANs.
# - 10.0.0.10
# - 172.16.0.10
# - 192.168.0.10

# Used to provide additional options to the kubelet.
kubelet:
    image: ghcr.io/siderolabs/kubelet:v1.33.0 # The `image` field is an optional reference to an alternative kubelet image.
    defaultRuntimeSeccompProfileEnabled: true # Enable container runtime default Seccomp profile.
    disableManifestsDirectory: true # The `disableManifestsDirectory` field configures the kubelet to get static pod manifests from the /etc/kubernetes/manifests directory.
    
    # # The `ClusterDNS` field is an optional reference to an alternative kubelet clusterDNS ip list.
    # clusterDNS:
    #     - 10.96.0.10
    #     - 169.254.2.53

    # # The `extraArgs` field is used to provide additional flags to the kubelet.
    # extraArgs:
    #     key: value

    # # The `extraMounts` field is used to add additional mounts to the kubelet container.
    # extraMounts:
    #     - destination: /var/lib/example # Destination is the absolute path where the mount will be placed in the container.
    #       type: bind # Type specifies the mount kind.
    #       source: /var/lib/example # Source specifies the source path of the mount.
    #       # Options are fstab style mount options.
    #       options:
    #         - bind
    #         - rshared
    #         - rw

    # # The `extraConfig` field is used to provide kubelet configuration overrides.
    # extraConfig:
    #     serverTLSBootstrap: true

    # # The `KubeletCredentialProviderConfig` field is used to provide kubelet credential configuration.
    # credentialProviderConfig:
    #     apiVersion: kubelet.config.k8s.io/v1
    #     kind: CredentialProviderConfig
    #     providers:
    #         - apiVersion: credentialprovider.kubelet.k8s.io/v1
    #           defaultCacheDuration: 12h
    #           matchImages:
    #             - '*.dkr.ecr.*.amazonaws.com'
    #             - '*.dkr.ecr.*.amazonaws.com.cn'
    #             - '*.dkr.ecr-fips.*.amazonaws.com'
    #             - '*.dkr.ecr.us-iso-east-1.c2s.ic.gov'
    #             - '*.dkr.ecr.us-isob-east-1.sc2s.sgov.gov'
    #           name: ecr-credential-provider

    # # The `nodeIP` field is used to configure `--node-ip` flag for the kubelet.
    # nodeIP:
    #     # The `validSubnets` field configures the networks to pick kubelet node IP from.
    #     validSubnets:
    #         - 10.0.0.0/8
    #         - '!10.0.0.3/32'
    #         - fdc7::/16
# Provides machine specific network configuration options.
network: {}
# # `interfaces` is used to define the network interface configuration.
# interfaces:
#     - interface: enp0s1 # The interface name.
#       # Assigns static IP addresses to the interface.
#       addresses:
#         - 192.168.2.0/24
#       # A list of routes associated with the interface.
#       routes:
#         - network: 0.0.0.0/0 # The route's network (destination).
#           gateway: 192.168.2.1 # The route's gateway (if empty, creates link scope route).
#           metric: 1024 # The optional metric for the route.
#       mtu: 1500 # The interface's MTU.
#       
#       # # Picks a network device using the selector.

#       # # select a device with bus prefix 00:*.
#       # deviceSelector:
#       #     busPath: 00:* # PCI, USB bus prefix, supports matching by wildcard.
#       # # select a device with mac address matching `*:f0:ab` and `virtio` kernel driver.
#       # deviceSelector:
#       #     hardwareAddr: '*:f0:ab' # Device hardware (MAC) address, supports matching by wildcard.
#       #     driver: virtio_net # Kernel driver, supports matching by wildcard.
#       # # select a device with bus prefix 00:*, a device with mac address matching `*:f0:ab` and `virtio` kernel driver.
#       # deviceSelector:
#       #     - busPath: 00:* # PCI, USB bus prefix, supports matching by wildcard.
#       #     - hardwareAddr: '*:f0:ab' # Device hardware (MAC) address, supports matching by wildcard.
#       #       driver: virtio_net # Kernel driver, supports matching by wildcard.

#       # # Bond specific options.
#       # bond:
#       #     # The interfaces that make up the bond.
#       #     interfaces:
#       #         - enp2s0
#       #         - enp2s1
#       #     # Picks a network device using the selector.
#       #     deviceSelectors:
#       #         - busPath: 00:* # PCI, USB bus prefix, supports matching by wildcard.
#       #         - hardwareAddr: '*:f0:ab' # Device hardware (MAC) address, supports matching by wildcard.
#       #           driver: virtio_net # Kernel driver, supports matching by wildcard.
#       #     mode: 802.3ad # A bond option.
#       #     lacpRate: fast # A bond option.

#       # # Bridge specific options.
#       # bridge:
#       #     # The interfaces that make up the bridge.
#       #     interfaces:
#       #         - enxda4042ca9a51
#       #         - enxae2a6774c259
#       #     # Enable STP on this bridge.
#       #     stp:
#       #         enabled: true # Whether Spanning Tree Protocol (STP) is enabled.

#       # # Configure this device as a bridge port.
#       # bridgePort:
#       #     master: br0 # The name of the bridge master interface

#       # # Indicates if DHCP should be used to configure the interface.
#       # dhcp: true

#       # # DHCP specific options.
#       # dhcpOptions:
#       #     routeMetric: 1024 # The priority of all routes received via DHCP.

#       # # Wireguard specific configuration.

#       # # wireguard server example
#       # wireguard:
#       #     privateKey: ABCDEF... # Specifies a private key configuration (base64 encoded).
#       #     listenPort: 51111 # Specifies a device's listening port.
#       #     # Specifies a list of peer configurations to apply to a device.
#       #     peers:
#       #         - publicKey: ABCDEF... # Specifies the public key of this peer.
#       #           endpoint: 192.168.1.3 # Specifies the endpoint of this peer entry.
#       #           # AllowedIPs specifies a list of allowed IP addresses in CIDR notation for this peer.
#       #           allowedIPs:
#       #             - 192.168.1.0/24
#       # # wireguard peer example
#       # wireguard:
#       #     privateKey: ABCDEF... # Specifies a private key configuration (base64 encoded).
#       #     # Specifies a list of peer configurations to apply to a device.
#       #     peers:
#       #         - publicKey: ABCDEF... # Specifies the public key of this peer.
#       #           endpoint: 192.168.1.2:51822 # Specifies the endpoint of this peer entry.
#       #           persistentKeepaliveInterval: 10s # Specifies the persistent keepalive interval for this peer.
#       #           # AllowedIPs specifies a list of allowed IP addresses in CIDR notation for this peer.
#       #           allowedIPs:
#       #             - 192.168.1.0/24

#       # # Virtual (shared) IP address configuration.

#       # # layer2 vip example
#       # vip:
#       #     ip: 172.16.199.55 # Specifies the IP address to be used.

# # Used to statically set the nameservers for the machine.
# nameservers:
#     - 8.8.8.8
#     - 1.1.1.1

# # Used to statically set arbitrary search domains.
# searchDomains:
#     - example.org
#     - example.com

# # Allows for extra entries to be added to the `/etc/hosts` file
# extraHostEntries:
#     - ip: 192.168.1.100 # The IP of the host.
#       # The host alias.
#       aliases:
#         - example
#         - example.domain.tld

# # Configures KubeSpan feature.
# kubespan:
#     enabled: true # Enable the KubeSpan feature.

# Used to provide instructions for installations.
install:
    disk: /dev/vda # The disk used for installations.
    image: ghcr.io/siderolabs/installer:v1.10.2 # Allows for supplying the image used to perform the installation.
    wipe: true # Indicates if the installation disk should be wiped at installation time.
    
    # # Look up disk using disk attributes like model, size, serial and others.
    # diskSelector:
    #     size: 4GB # Disk size.
    #     model: WDC* # Disk model `/sys/block/<dev>/device/model`.
    #     busPath: /pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0 # Disk bus path.

    # # Allows for supplying extra kernel args via the bootloader.
    # extraKernelArgs:
    #     - talos.platform=metal
    #     - reboot=k
# Used to configure the machine's container image registry mirrors.
registries: {}
# # Specifies mirror configuration for each registry host namespace.
# mirrors:
#     ghcr.io:
#         # List of endpoints (URLs) for registry mirrors to use.
#         endpoints:
#             - https://registry.insecure
#             - https://ghcr.io/v2/

# # Specifies TLS & auth configuration for HTTPS image registries.
# config:
#     registry.insecure:
#         # The TLS configuration for the registry.
#         tls:
#             insecureSkipVerify: true # Skip TLS server certificate verification (not recommended).
#             
#             # # Enable mutual TLS authentication with the registry.
#             # clientIdentity:
#             #     crt: LS0tIEVYQU1QTEUgQ0VSVElGSUNBVEUgLS0t
#             #     key: LS0tIEVYQU1QTEUgS0VZIC0tLQ==
#         
#         # # The auth configuration for this registry.
#         # auth:
#         #     username: username # Optional registry authentication.
#         #     password: password # Optional registry authentication.

# Features describe individual Talos features that can be switched on or off.
features:
    rbac: true # Enable role-based access control (RBAC).
    stableHostname: true # Enable stable default hostname.
    apidCheckExtKeyUsage: true # Enable checks for extended key usage of client certificates in apid.
    diskQuotaSupport: true # Enable XFS project quota support for EPHEMERAL partition and user disks.
    # KubePrism - local proxy/load balancer on defined port that will distribute
    kubePrism:
        enabled: true # Enable KubePrism support - will start local load balancing proxy.
        port: 7445 # KubePrism port.
    # Configures host DNS caching resolver.
    hostDNS:
        enabled: true # Enable host DNS caching resolver.
        forwardKubeDNSToHost: true # Use the host DNS resolver as upstream for Kubernetes CoreDNS pods.
    
    # # Configure Talos API access from Kubernetes pods.
    # kubernetesTalosAPIAccess:
    #     enabled: true # Enable Talos API access from Kubernetes pods.
    #     # The list of Talos API roles which can be granted for access from Kubernetes pods.
    #     allowedRoles:
    #         - os:reader
    #     # The list of Kubernetes namespaces Talos API access is available from.
    #     allowedKubernetesNamespaces:
    #         - kube-system
# Configures the node labels for the machine.
nodeLabels:
    node.kubernetes.io/exclude-from-external-load-balancers: ""

# # Provides machine specific control plane configuration options.

# # ControlPlane definition example.
# controlPlane:
#     # Controller manager machine specific configuration options.
#     controllerManager:
#         disabled: false # Disable kube-controller-manager on the node.
#     # Scheduler machine specific configuration options.
#     scheduler:
#         disabled: true # Disable kube-scheduler on the node.

# # Used to provide static pod definitions to be run by the kubelet directly bypassing the kube-apiserver.

# # nginx static pod.
# pods:
#     - apiVersion: v1
#       kind: pod
#       metadata:
#         name: nginx
#       spec:
#         containers:
#             - image: nginx
#               name: nginx

# # Allows the addition of user specified files.

# # MachineFiles usage example.
# files:
#     - content: '...' # The contents of the file.
#       permissions: 0o666 # The file's permissions in octal.
#       path: /tmp/file.txt # The path of the file.
#       op: append # The operation to use

# # The `env` field allows for the addition of environment variables.

# # Environment variables definition examples.
# env:
#     GRPC_GO_LOG_SEVERITY_LEVEL: info
#     GRPC_GO_LOG_VERBOSITY_LEVEL: "99"
#     https_proxy: http://SERVER:PORT/
# env:
#     GRPC_GO_LOG_SEVERITY_LEVEL: error
#     https_proxy: https://USERNAME:PASSWORD@SERVER:PORT/
# env:
#     https_proxy: http://DOMAIN\USERNAME:PASSWORD@SERVER:PORT/

# # Used to configure the machine's time settings.

# # Example configuration for cloudflare ntp server.
# time:
#     disabled: false # Indicates if the time service is disabled for the machine.
#     # description: |
#     servers:
#         - time.cloudflare.com
#     bootTimeout: 2m0s # Specifies the timeout when the node time is considered to be in sync unlocking the boot sequence.

# # Used to configure the machine's sysctls.

# # MachineSysctls usage example.
# sysctls:
#     kernel.domainname: talos.dev
#     net.ipv4.ip_forward: "0"
#     net/ipv6/conf/eth0.100/disable_ipv6: "1"

# # Used to configure the machine's sysfs.

# # MachineSysfs usage example.
# sysfs:
#     devices.system.cpu.cpu0.cpufreq.scaling_governor: performance

# # Machine system disk encryption configuration.
# systemDiskEncryption:
#     # Ephemeral partition encryption.
#     ephemeral:
#         provider: luks2 # Encryption provider to use for the encryption.
#         # Defines the encryption keys generation and storage method.
#         keys:
#             - # Deterministically generated key from the node UUID and PartitionLabel.
#               nodeID: {}
#               slot: 0 # Key slot number for LUKS2 encryption.
#               
#               # # KMS managed encryption key.
#               # kms:
#               #     endpoint: https://192.168.88.21:4443 # KMS endpoint to Seal/Unseal the key.
#         
#         # # Cipher kind to use for the encryption. Depends on the encryption provider.
#         # cipher: aes-xts-plain64

#         # # Defines the encryption sector size.
#         # blockSize: 4096

#         # # Additional --perf parameters for the LUKS2 encryption.
#         # options:
#         #     - no_read_workqueue
#         #     - no_write_workqueue

# # Configures the udev system.
# udev:
#     # List of udev rules to apply to the udev system
#     rules:
#         - SUBSYSTEM=="drm", KERNEL=="renderD*", GROUP="44", MODE="0660"

# # Configures the logging system.
# logging:
#     # Logging destination.
#     destinations:
#         - endpoint: tcp://1.2.3.4:12345 # Where to send logs. Supported protocols are "tcp" and "udp".
#           format: json_lines # Logs format.

# # Configures the kernel.
# kernel:
#     # Kernel modules to load.
#     modules:
#         - name: brtfs # Module name.

# # Configures the seccomp profiles for the machine.
# seccompProfiles:
#     - name: audit.json # The `name` field is used to provide the file name of the seccomp profile.
#       # The `value` field is used to provide the seccomp profile.
#       value:
#         defaultAction: SCMP_ACT_LOG

# # Override (patch) settings in the default OCI runtime spec for CRI containers.

# # override default open file limit
# baseRuntimeSpecOverrides:
#     process:
#         rlimits:
#             - hard: 1024
#               soft: 1024
#               type: RLIMIT_NOFILE

# # Configures the node annotations for the machine.

# # node annotations example.
# nodeAnnotations:
#     customer.io/rack: r13a25

# # Configures the node taints for the machine. Effect is optional.

# # node taints example.
# nodeTaints:
#     exampleTaint: exampleTaintValue:NoSchedule

Provides cluster specific configuration options.

cluster:
id: sXPvJaxULrGVNF_wTs0K_PyeBFhIj0fQ0DhnizJsDAE= # Globally unique identifier for this cluster (base64 encoded random 32 bytes).
secret: # Shared secret of cluster (base64 encoded random 32 bytes).
# Provides control plane specific configuration options.
controlPlane:
endpoint: https://192.168.1.100:6443 # Endpoint is the canonical controlplane endpoint, which can be an IP address or a DNS hostname.
clusterName: lernaca # Configures the cluster's name.
# Provides cluster specific network configuration options.
network:
dnsDomain: cluster.local # The domain used by Kubernetes DNS.
# The pod subnet CIDR.
podSubnets:
- 10.244.0.0/16
# The service subnet CIDR.
serviceSubnets:
- 10.96.0.0/12

    # # The CNI used.
    # cni:
    #     name: custom # Name of CNI to use.
    #     # URLs containing manifests to apply for the CNI.
    #     urls:
    #         - https://docs.projectcalico.org/archive/v3.20/manifests/canal.yaml
token: 1omts1.26f8lym1ow3u0g7b # The [bootstrap token](https://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/) used to join the cluster.
secretboxEncryptionSecret: <REDACTED> # A key used for the [encryption of secret data at rest](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/).
# The base64 encoded root certificate authority used by Kubernetes.
ca:
    crt: <REDACTED>==
    key: <REDACTED>=
# The base64 encoded aggregator certificate authority used by Kubernetes for front-proxy certificate generation.
aggregatorCA:
    crt: <REDACTED>==
    key: <REDACTED>
# API server specific configuration options.
apiServer:
    image: registry.k8s.io/kube-apiserver:v1.33.0 # The container image used in the API server manifest.
    # Extra certificate subject alternative names for the API server's certificate.
    certSANs:
        - 192.168.1.100
    disablePodSecurityPolicy: true # Disable PodSecurityPolicy in the API server and default manifests.
    # Configure the API server admission plugins.
    admissionControl:
        - name: PodSecurity # Name is the name of the admission controller.
          # Configuration is an embedded configuration object to be used as the plugin's
          configuration:
            apiVersion: pod-security.admission.config.k8s.io/v1alpha1
            defaults:
                audit: restricted
                audit-version: latest
                enforce: baseline
                enforce-version: latest
                warn: restricted
                warn-version: latest
            exemptions:
                namespaces:
                    - kube-system
                runtimeClasses: []
                usernames: []
            kind: PodSecurityConfiguration
    # Configure the API server audit policy.
    auditPolicy:
        apiVersion: audit.k8s.io/v1
        kind: Policy
        rules:
            - level: Metadata
    
    # # Configure the API server authorization config. Node and RBAC authorizers are always added irrespective of the configuration.
    # authorizationConfig:
    #     - type: Webhook # Type is the name of the authorizer. Allowed values are `Node`, `RBAC`, and `Webhook`.
    #       name: webhook # Name is used to describe the authorizer.
    #       # webhook is the configuration for the webhook authorizer.
    #       webhook:
    #         connectionInfo:
    #             type: InClusterConfig
    #         failurePolicy: Deny
    #         matchConditionSubjectAccessReviewVersion: v1
    #         matchConditions:
    #             - expression: has(request.resourceAttributes)
    #             - expression: '!(\''system:serviceaccounts:kube-system\'' in request.groups)'
    #         subjectAccessReviewVersion: v1
    #         timeout: 3s
    #     - type: Webhook # Type is the name of the authorizer. Allowed values are `Node`, `RBAC`, and `Webhook`.
    #       name: in-cluster-authorizer # Name is used to describe the authorizer.
    #       # webhook is the configuration for the webhook authorizer.
    #       webhook:
    #         connectionInfo:
    #             type: InClusterConfig
    #         failurePolicy: NoOpinion
    #         matchConditionSubjectAccessReviewVersion: v1
    #         subjectAccessReviewVersion: v1
    #         timeout: 3s
# Controller manager server specific configuration options.
controllerManager:
    image: registry.k8s.io/kube-controller-manager:v1.33.0 # The container image used in the controller manager manifest.
# Kube-proxy server-specific configuration options
proxy:
    image: registry.k8s.io/kube-proxy:v1.33.0 # The container image used in the kube-proxy manifest.
    
    # # Disable kube-proxy deployment on cluster bootstrap.
    # disabled: false
# Scheduler server specific configuration options.
scheduler:
    image: registry.k8s.io/kube-scheduler:v1.33.0 # The container image used in the scheduler manifest.
# Configures cluster member discovery.
discovery:
    enabled: true # Enable the cluster membership discovery feature.
    # Configure registries used for cluster member discovery.
    registries:
        # Kubernetes registry uses Kubernetes API server to discover cluster members and stores additional information
        kubernetes:
            disabled: true # Disable Kubernetes discovery registry.
        # Service registry is using an external service to push and pull information about cluster members.
        service: {}
        # # External service endpoint.
        # endpoint: https://discovery.talos.dev/
# Etcd specific configuration options.
etcd:
    # The `ca` is the root certificate authority of the PKI.
    ca:
        crt: <REDACTED>
        key: <REDACTED>=
    
    # # The container image used to create the etcd service.
    # image: gcr.io/etcd-development/etcd:v3.5.21

    # # The `advertisedSubnets` field configures the networks to pick etcd advertised IP from.
    # advertisedSubnets:
    #     - 10.0.0.0/8
# A list of urls that point to additional manifests.
extraManifests: []
#   - https://www.example.com/manifest1.yaml
#   - https://www.example.com/manifest2.yaml

# A list of inline Kubernetes manifests.
inlineManifests: []
#   - name: namespace-ci # Name of the manifest.
#     contents: |- # Manifest contents as a string.
#       apiVersion: v1
#       kind: Namespace
#       metadata:
#       	name: ci


# # A key used for the [encryption of secret data at rest](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/).

# # Decryption secret example (do not use in production!).
# aescbcEncryptionSecret: z01mye6j16bspJYtTB/5SFX8j7Ph4JXxM2Xuu4vsBPM=

# # Core DNS specific configuration options.
# coreDNS:
#     image: registry.k8s.io/coredns/coredns:v1.12.1 # The `image` field is an override to the default coredns image.

# # External cloud provider configuration.
# externalCloudProvider:
#     enabled: true # Enable external cloud provider.
#     # A list of urls that point to additional manifests for an external cloud provider.
#     manifests:
#         - https://raw.githubusercontent.com/kubernetes/cloud-provider-aws/v1.20.0-alpha.0/manifests/rbac.yaml
#         - https://raw.githubusercontent.com/kubernetes/cloud-provider-aws/v1.20.0-alpha.0/manifests/aws-cloud-controller-manager-daemonset.yaml

# # A map of key value pairs that will be added while fetching the extraManifests.
# extraManifestHeaders:
#     Token: "1234567"
#     X-ExtraInfo: info

# # Settings for admin kubeconfig generation.
# adminKubeconfig:
#     certLifetime: 1h0m0s # Admin kubeconfig certificate lifetime (default is 1 year).

# # Allows running workload on control-plane nodes.
# allowSchedulingOnControlPlanes: true

</details>

[rothgar]

What do the logs from the .1.100 node say?

[TheSirC]

The logs are repeating the following :

`talosctl --nodes=192.168.1.100 logs kubelet`

192.168.1.100: {"ts":1750319219942.5596,"caller":"eviction/eviction_manager.go:292","msg":"Eviction manager: failed to get summary stats","err":"failed to get node info: node \"talos-jnf-of8\" not found"}
192.168.1.100: {"ts":1750319220656.679,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1750319220656.7,"shutdownThreshold":"5m0s"}
192.168.1.100: {"ts":1750319221657.1453,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1750319221657.164,"shutdownThreshold":"5m0s"}
192.168.1.100: {"ts":1750319222656.898,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1750319222656.9185,"shutdownThreshold":"5m0s"}
192.168.1.100: {"ts":1750319223465.575,"caller":"lease/controller.go:145","msg":"Failed to ensure lease exists, will retry","interval":"7s","err":"Get \"https://127.0.0.1:7445/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/talos-jnf-of8?timeout=10s\": context deadline exceeded - error from a previous attempt: read tcp 127.0.0.1:46874->127.0.0.1:7445: read: connection reset by peer"}
192.168.1.100: {"ts":1750319223656.8271,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1750319223656.8464,"shutdownThreshold":"5m0s"}
192.168.1.100: {"ts":1750319224136.7466,"caller":"record/event.go:368","msg":"Unable to write event (may retry after sleeping)","event":"&Event{ObjectMeta:{talos-jnf-of8.184a61c8b71ee2cf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:talos-jnf-of8,UID:talos-jnf-of8,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node talos-jnf-of8 status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:talos-jnf-of8,},FirstTimestamp:2025-06-19 07:41:59.880467151 +0000 UTC m=+5.462871503,LastTimestamp:2025-06-19 07:41:59.880467151 +0000 UTC m=+5.462871503,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:talos-jnf-of8,}","err":"Post \"https://127.0.0.1:7445/api/v1/namespaces/default/events\": EOF"}
192.168.1.100: {"ts":1750319224656.9102,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1750319224656.9292,"shutdownThreshold":"5m0s"}
192.168.1.100: {"ts":1750319225656.7268,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1750319225656.747,"shutdownThreshold":"5m0s"}
192.168.1.100: {"ts":1750319226656.5774,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1750319226656.5967,"shutdownThreshold":"5m0s"}
192.168.1.100: {"ts":1750319226886.9502,"caller":"kubelet/kubelet_node_status.go:75","msg":"Attempting to register node","v":0,"node":{"name":"talos-jnf-of8"}}
192.168.1.100: {"ts":1750319226889.5173,"caller":"kubelet/kubelet_node_status.go:107","msg":"Unable to register node with API server","node":{"name":"talos-jnf-of8"},"err":"Post \"https://127.0.0.1:7445/api/v1/nodes\": read tcp 127.0.0.1:41302->127.0.0.1:7445: read: connection reset by peer"}
192.168.1.100: {"ts":1750319227260.8772,"logger":"UnhandledError","caller":"cache/reflector.go:200","msg":"Failed to watch","reflector":"k8s.io/client-go/informers/factory.go:160","type":"*v1.Service","err":"failed to list *v1.Service: Get \"https://127.0.0.1:7445/api/v1/services?fieldSelector=spec.clusterIP%21%3DNone&limit=500&resourceVersion=0\": EOF - error from a previous attempt: read tcp 127.0.0.1:41260->127.0.0.1:7445: read: connection reset by peer"}
192.168.1.100: {"ts":1750319227533.5693,"logger":"UnhandledError","caller":"cache/reflector.go:200","msg":"Failed to watch","reflector":"k8s.io/client-go/informers/factory.go:160","type":"*v1.Node","err":"failed to list *v1.Node: Get \"https://127.0.0.1:7445/api/v1/nodes?fieldSelector=metadata.name%3Dtalos-jnf-of8&limit=500&resourceVersion=0\": EOF - error from a previous attempt: read tcp 127.0.0.1:41276->127.0.0.1:7445: read: connection reset by peer"}
192.168.1.100: {"ts":1750319227656.8252,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1750319227656.8445,"shutdownThreshold":"5m0s"}
192.168.1.100: {"ts":1750319228656.7988,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1750319228656.8203,"shutdownThreshold":"5m0s"}
192.168.1.100: {"ts":1750319229656.56,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1750319229656.58,"shutdownThreshold":"5m0s"}
192.168.1.100: {"ts":1750319229670.792,"caller":"certificate/transport.go:123","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1750319229670.8076,"shutdownThreshold":"5m0s"}

[rothgar]

The error says the server is not available and may need to be restarted. Did you try that? Do you have local console access to see what the logs say on the server?

[smira]

Please take a look at Troubleshooting documentation.

[TheSirC]

Thank you very much for the pointer! Any specific section in mind ?

I do not see any section related to the problem I described...

[TheSirC]

One error in talosctl --nodes=192.168.1.100 logs controller-runtime stands out in particular for me :

192.168.1.100: 2025-06-30T20:57:50.665Z ERROR controller failed {"component": "controller-runtime", "controller": "secrets.KubernetesDynamicCertsController", "error": "failed to generate aggregator cert: failed to create new certificate: x509: requested SignatureAlgorithm does not match private key type"}

Error which I fail to identify the root cause of. @smira if you have any insights.