Skip to content

NullPointerException in SSABasedGenericKubernetesResourceMatcher #2032

New issue
New issue
Closed
#2042
#2037
Closed
NullPointerException in SSABasedGenericKubernetesResourceMatcher#2032
#2042
#2037
Assignees
csviri
@kosmoz

Description

@kosmoz
kosmoz
opened on Aug 24, 2023

Bug Report

What did you do?

We ran into this issue when using unsupported beta features on an older Kubernetes version:

  • Create a StatefulSet dependent resource with persistentVolumeClaimRetentionPolicy on a cluster running Kubernetes version 1.26.
  • Kubernetes adds persistentVolumeClaimRetentionPolicy to the managedFields but ignores the value
  • When getting the StatefulSet, the SDK expects all managed fields to be present in the resource but in this case, persistentVolumeClaimRetentionPolicy is missing.

What did you expect to see?

At most, a warning should be printed.

What did you see instead? Under which circumstances?

Error during event processing ExecutionScope{ resource id: ResourceID{name='redacted', namespace='redacted'}, version: 8966444451} failed. 
io.javaoperatorsdk.operator.AggregatedOperatorException: Exception(s) during workflow execution. Details:
 - eu.glasskube.operator.apps.vault.dependent.VaultStatefulSet -> java.lang.NullPointerException: Cannot invoke "java.util.Map.get(Object)" because "actualMap" is null
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.SSABasedGenericKubernetesResourceMatcher.keepOnlyManagedFields(SSABasedGenericKubernetesResourceMatcher.java:144)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.SSABasedGenericKubernetesResourceMatcher.fillResultsAndTraverseFurther(SSABasedGenericKubernetesResourceMatcher.java:166)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.SSABasedGenericKubernetesResourceMatcher.keepOnlyManagedFields(SSABasedGenericKubernetesResourceMatcher.java:138)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.SSABasedGenericKubernetesResourceMatcher.fillResultsAndTraverseFurther(SSABasedGenericKubernetesResourceMatcher.java:166)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.SSABasedGenericKubernetesResourceMatcher.keepOnlyManagedFields(SSABasedGenericKubernetesResourceMatcher.java:138)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.SSABasedGenericKubernetesResourceMatcher.matches(SSABasedGenericKubernetesResourceMatcher.java:90)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.KubernetesDependentResource.match(KubernetesDependentResource.java:169)
	at io.javaoperatorsdk.operator.processing.dependent.kubernetes.KubernetesDependentResource.match(KubernetesDependentResource.java:32)
	at io.javaoperatorsdk.operator.processing.dependent.AbstractDependentResource.reconcile(AbstractDependentResource.java:67)
	at io.javaoperatorsdk.operator.processing.dependent.SingleDependentResourceReconciler.reconcile(SingleDependentResourceReconciler.java:19)
	at io.javaoperatorsdk.operator.processing.dependent.AbstractDependentResource.reconcile(AbstractDependentResource.java:52)
	at io.javaoperatorsdk.operator.processing.dependent.workflow.WorkflowReconcileExecutor$NodeReconcileExecutor.doRun(WorkflowReconcileExecutor.java:115)
	at io.javaoperatorsdk.operator.processing.dependent.workflow.NodeExecutor.run(NodeExecutor.java:22)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

	at io.javaoperatorsdk.operator.processing.dependent.workflow.WorkflowResult.throwAggregateExceptionIfErrorsPresent(WorkflowResult.java:41)
	at io.javaoperatorsdk.operator.processing.dependent.workflow.WorkflowReconcileResult.throwAggregateExceptionIfErrorsPresent(WorkflowReconcileResult.java:9)
	at io.javaoperatorsdk.operator.processing.dependent.workflow.DefaultWorkflow.reconcile(DefaultWorkflow.java:95)
	at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:147)
	at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:110)
	at io.javaoperatorsdk.operator.api.monitoring.Metrics.timeControllerExecution(Metrics.java:219)
	at io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:109)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:140)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:121)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:91)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:64)
	at io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:409)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

Environment

Kubernetes cluster type:

vanilla

$ Mention java-operator-sdk version from pom.xml file

4.4.2

$ java -version

openjdk version "17.0.8" 2023-07-18
OpenJDK Runtime Environment (Red_Hat-17.0.8.0.7-1.fc38) (build 17.0.8+7)
OpenJDK 64-Bit Server VM (Red_Hat-17.0.8.0.7-1.fc38) (build 17.0.8+7, mixed mode, sharing)

$ kubectl version

Client Version: v1.28.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.26.4
WARNING: version difference between client (1.28) and server (1.26) exceeds the supported minor version skew of +/-1

Possible Solution

Check if actualMap is null before adding to the result.

Additional context

Related: glasskube/operator#267

Activity

shawkins

shawkins commented on Aug 24, 2023

@shawkins
shawkins
on Aug 24, 2023
Collaborator

Relates to #2028 - it's the same NPE that happens when using a secret with stringData in the desired state.

csviri

csviri commented on Aug 25, 2023

@csviri
csviri
on Aug 25, 2023
Collaborator

Hi @kosmoz , on short notice you might want to try the generic matcher, see:
https://github.com/operator-framework/java-operator-sdk/blob/main/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/dependent/kubernetes/GenericKubernetesResourceMatcher.java#L19-L19

also:
https://javaoperatorsdk.io/docs/v4-4-migration#using-server-side-apply-in-dependent-resources

Would be great if you could provide a simple project to reproduce. Or provide the logs from here: https://github.com/shawkins/java-operator-sdk/blob/c879806ba279c2a285639344a7e34c01118c8c01/operator-framework-core/src/main/java/io/javaoperatorsdk/operator/processing/dependent/kubernetes/SSABasedGenericKubernetesResourceMatcher.java#L86-L86

thank you!

csviri
self-assigned this
on Aug 25, 2023
kosmoz

kosmoz commented on Aug 25, 2023

@kosmoz
kosmoz
on Aug 25, 2023 · edited by kosmoz
Author

Hi! I'm not sure if I will have time to create a minimal reprodicution repo. However the steps would be pretty straight forward:

  1. create an operator with one controller for a CRD
  2. create a managed dependent resource for that CRD. This should be a StatefulSet where persistentVolumeClaimRetentionPolicy is set to something other than null.
  3. On a cluster running a Kubernetes version <1.27 create a resource for that CRD

Maybe I will have time next week, but in the meantime, here are the logs (I pruned them to the lines that I think matter for this error).

2023-08-25T11:01:36.565+02:00 [pool-6-thread-1] TRACE i.j.o.p.d.k.SSABasedGenericKubernetesResourceMatcher - Original actual: 
 StatefulSet(apiVersion=apps/v1, kind=StatefulSet, metadata=ObjectMeta(annotations={}, creationTimestamp=2023-08-25T09:00:45Z, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=1, labels={app.kubernetes.io/instance=test, app.kubernetes.io/managed-by=glasskube-operator, app.kubernetes.io/name=vault, app.kubernetes.io/part-of=vault, app.kubernetes.io/version=1.14.0}, managedFields=[ManagedFieldsEntry(apiVersion=apps/v1, fieldsType=FieldsV1, fieldsV1=FieldsV1(additionalProperties={f:metadata={f:labels={f:app.kubernetes.io/instance={}, f:app.kubernetes.io/managed-by={}, f:app.kubernetes.io/name={}, f:app.kubernetes.io/part-of={}, f:app.kubernetes.io/version={}}, f:ownerReferences={k:{"uid":"2554a7e3-2752-4176-86b8-a45cf1090ad9"}={}}}, f:spec={f:persistentVolumeClaimRetentionPolicy={f:whenDeleted={}, f:whenScaled={}}, f:replicas={}, f:selector={}, f:serviceName={}, f:template={f:metadata={f:labels={f:app.kubernetes.io/instance={}, f:app.kubernetes.io/managed-by={}, f:app.kubernetes.io/name={}, f:app.kubernetes.io/part-of={}, f:app.kubernetes.io/version={}}}, f:spec={f:containers={k:{"name":"vault"}={.={}, f:args={}, f:command={}, f:env={k:{"name":"DB_HOST"}={.={}, f:name={}, f:value={}}, k:{"name":"DB_NAME"}={.={}, f:name={}, f:value={}}, k:{"name":"DB_PASSWORD"}={.={}, f:name={}, f:valueFrom={f:secretKeyRef={}}}, k:{"name":"DB_USERNAME"}={.={}, f:name={}, f:valueFrom={f:secretKeyRef={}}}, k:{"name":"HOSTNAME"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"POD_IP"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"SKIP_SETCAP"}={.={}, f:name={}, f:value={}}, k:{"name":"SUBDOMAIN"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_API_ADDR"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_CLUSTER_ADDR"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_K8S_NAMESPACE"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"VAULT_K8S_POD_NAME"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"VAULT_PG_CONNECTION_URL"}={.={}, f:name={}, f:value={}}}, f:image={}, f:lifecycle={f:preStop={f:exec={f:command={}}}}, f:name={}, f:ports={k:{"containerPort":8200,"protocol":"TCP"}={.={}, f:containerPort={}, f:name={}}, k:{"containerPort":8201,"protocol":"TCP"}={.={}, f:containerPort={}, f:name={}}}, f:resources={f:limits={f:memory={}}, f:requests={f:memory={}}}, f:securityContext={f:allowPrivilegeEscalation={}, f:capabilities={f:drop={}}, f:readOnlyRootFilesystem={}}, f:volumeMounts={k:{"mountPath":"/glasskube/config"}={.={}, f:mountPath={}, f:name={}, f:readOnly={}}, k:{"mountPath":"/glasskube/tls"}={.={}, f:mountPath={}, f:name={}, f:readOnly={}}, k:{"mountPath":"/vault/audit"}={.={}, f:mountPath={}, f:name={}}}}}, f:securityContext={f:fsGroup={}, f:runAsGroup={}, f:runAsNonRoot={}, f:runAsUser={}}, f:serviceAccountName={}, f:terminationGracePeriodSeconds={}, f:volumes={k:{"name":"config"}={.={}, f:configMap={f:name={}}, f:name={}}, k:{"name":"tls"}={.={}, f:name={}, f:secret={f:secretName={}}}}}}, f:volumeClaimTemplates={}}}), manager=vaultreconciler, operation=Apply, subresource=null, time=2023-08-25T09:00:45Z, additionalProperties={}), ManagedFieldsEntry(apiVersion=apps/v1, fieldsType=FieldsV1, fieldsV1=FieldsV1(additionalProperties={f:status={f:collisionCount={}, f:currentReplicas={}, f:currentRevision={}, f:observedGeneration={}, f:replicas={}, f:updateRevision={}, f:updatedReplicas={}}}), manager=kube-controller-manager, operation=Update, subresource=status, time=2023-08-25T09:00:45Z, additionalProperties={})], name=vault-test, namespace=default, ownerReferences=[OwnerReference(apiVersion=glasskube.eu/v1alpha1, blockOwnerDeletion=null, controller=null, kind=Vault, name=test, uid=2554a7e3-2752-4176-86b8-a45cf1090ad9, additionalProperties={})], resourceVersion=2242, selfLink=null, uid=33102e14-0069-4979-8739-64f86f00822e, additionalProperties={}), spec=StatefulSetSpec(minReadySeconds=null, ordinals=null, persistentVolumeClaimRetentionPolicy=null, podManagementPolicy=OrderedReady, replicas=1, revisionHistoryLimit=10, selector=LabelSelector(matchExpressions=[], matchLabels={app.kubernetes.io/instance=test, app.kubernetes.io/name=vault, app.kubernetes.io/part-of=vault}, additionalProperties={}), serviceName=vault-test-headless, template=PodTemplateSpec(metadata=ObjectMeta(annotations={}, creationTimestamp=null, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, labels={app.kubernetes.io/instance=test, app.kubernetes.io/managed-by=glasskube-operator, app.kubernetes.io/name=vault, app.kubernetes.io/part-of=vault, app.kubernetes.io/version=1.14.0}, managedFields=[], name=null, namespace=null, ownerReferences=[], resourceVersion=null, selfLink=null, uid=null, additionalProperties={}), spec=PodSpec(activeDeadlineSeconds=null, affinity=null, automountServiceAccountToken=null, containers=[Container(args=[-c, /usr/local/bin/docker-entrypoint.sh vault server -config=/glasskube/config/config.hcl
], command=[/bin/sh], env=[EnvVar(name=SKIP_SETCAP, value=true, valueFrom=null, additionalProperties={}), EnvVar(name=HOSTNAME, value=null, valueFrom=EnvVarSource(configMapKeyRef=null, fieldRef=ObjectFieldSelector(apiVersion=v1, fieldPath=metadata.name, additionalProperties={}), resourceFieldRef=null, secretKeyRef=null, additionalProperties={}), additionalProperties={}), EnvVar(name=SUBDOMAIN, value=vault-test-headless, valueFrom=null, additionalProperties={}), EnvVar(name=POD_IP, value=null, valueFrom=EnvVarSource(configMapKeyRef=null, fieldRef=ObjectFieldSelector(apiVersion=v1, fieldPath=status.podIP, additionalProperties={}), resourceFieldRef=null, secretKeyRef=null, additionalProperties={}), additionalProperties={}), EnvVar(name=VAULT_API_ADDR, value=https://$(POD_IP):8200, valueFrom=null, additionalProperties={}), EnvVar(name=VAULT_CLUSTER_ADDR, value=https://$(HOSTNAME).$(SUBDOMAIN):8201, valueFrom=null, additionalProperties={}), EnvVar(name=DB_HOST, value=vault-test-db-rw, valueFrom=null, additionalProperties={}), EnvVar(name=DB_NAME, value=vault, valueFrom=null, additionalProperties={}), EnvVar(name=DB_USERNAME, value=null, valueFrom=EnvVarSource(configMapKeyRef=null, fieldRef=null, resourceFieldRef=null, secretKeyRef=SecretKeySelector(key=username, name=vault-test-db-app, optional=null, additionalProperties={}), additionalProperties={}), additionalProperties={}), EnvVar(name=DB_PASSWORD, value=null, valueFrom=EnvVarSource(configMapKeyRef=null, fieldRef=null, resourceFieldRef=null, secretKeyRef=SecretKeySelector(key=password, name=vault-test-db-app, optional=null, additionalProperties={}), additionalProperties={}), additionalProperties={}), EnvVar(name=VAULT_PG_CONNECTION_URL, value=postgres://$(DB_USERNAME):$(DB_PASSWORD)@$(DB_HOST):5432/$(DB_NAME), valueFrom=null, additionalProperties={}), EnvVar(name=VAULT_K8S_POD_NAME, value=null, valueFrom=EnvVarSource(configMapKeyRef=null, fieldRef=ObjectFieldSelector(apiVersion=v1, fieldPath=metadata.name, additionalProperties={}), resourceFieldRef=null, secretKeyRef=null, additionalProperties={}), additionalProperties={}), EnvVar(name=VAULT_K8S_NAMESPACE, value=null, valueFrom=EnvVarSource(configMapKeyRef=null, fieldRef=ObjectFieldSelector(apiVersion=v1, fieldPath=metadata.namespace, additionalProperties={}), resourceFieldRef=null, secretKeyRef=null, additionalProperties={}), additionalProperties={})], envFrom=[], image=hashicorp/vault:1.14.0, imagePullPolicy=IfNotPresent, lifecycle=Lifecycle(postStart=null, preStop=LifecycleHandler(exec=ExecAction(command=[killall, vault], additionalProperties={}), httpGet=null, tcpSocket=null, additionalProperties={}), additionalProperties={}), livenessProbe=null, name=vault, ports=[ContainerPort(containerPort=8200, hostIP=null, hostPort=null, name=https, protocol=TCP, additionalProperties={}), ContainerPort(containerPort=8201, hostIP=null, hostPort=null, name=https-internal, protocol=TCP, additionalProperties={})], readinessProbe=null, resources=ResourceRequirements(claims=[], limits={memory=100Mi}, requests={memory=30Mi}, additionalProperties={}), securityContext=SecurityContext(allowPrivilegeEscalation=false, capabilities=Capabilities(add=[], drop=[ALL], additionalProperties={}), privileged=null, procMount=null, readOnlyRootFilesystem=true, runAsGroup=null, runAsNonRoot=null, runAsUser=null, seLinuxOptions=null, seccompProfile=null, windowsOptions=null, additionalProperties={}), startupProbe=null, stdin=null, stdinOnce=null, terminationMessagePath=/dev/termination-log, terminationMessagePolicy=File, tty=null, volumeDevices=[], volumeMounts=[VolumeMount(mountPath=/glasskube/config, mountPropagation=null, name=config, readOnly=true, subPath=null, subPathExpr=null, additionalProperties={}), VolumeMount(mountPath=/glasskube/tls, mountPropagation=null, name=tls, readOnly=true, subPath=null, subPathExpr=null, additionalProperties={}), VolumeMount(mountPath=/vault/audit, mountPropagation=null, name=audit, readOnly=null, subPath=null, subPathExpr=null, additionalProperties={})], workingDir=null, additionalProperties={})], dnsConfig=null, dnsPolicy=ClusterFirst, enableServiceLinks=null, ephemeralContainers=[], hostAliases=[], hostIPC=null, hostNetwork=null, hostPID=null, hostUsers=null, hostname=null, imagePullSecrets=[], initContainers=[], nodeName=null, nodeSelector={}, os=null, overhead={}, preemptionPolicy=null, priority=null, priorityClassName=null, readinessGates=[], resourceClaims=[], restartPolicy=Always, runtimeClassName=null, schedulerName=default-scheduler, schedulingGates=[], securityContext=PodSecurityContext(fsGroup=1000, fsGroupChangePolicy=null, runAsGroup=1000, runAsNonRoot=true, runAsUser=1000, seLinuxOptions=null, seccompProfile=null, supplementalGroups=[], sysctls=[], windowsOptions=null, additionalProperties={}), serviceAccount=vault-test, serviceAccountName=vault-test, setHostnameAsFQDN=null, shareProcessNamespace=null, subdomain=null, terminationGracePeriodSeconds=10, tolerations=[], topologySpreadConstraints=[], volumes=[Volume(awsElasticBlockStore=null, azureDisk=null, azureFile=null, cephfs=null, cinder=null, configMap=ConfigMapVolumeSource(defaultMode=420, items=[], name=vault-test, optional=null, additionalProperties={}), csi=null, downwardAPI=null, emptyDir=null, ephemeral=null, fc=null, flexVolume=null, flocker=null, gcePersistentDisk=null, gitRepo=null, glusterfs=null, hostPath=null, iscsi=null, name=config, nfs=null, persistentVolumeClaim=null, photonPersistentDisk=null, portworxVolume=null, projected=null, quobyte=null, rbd=null, scaleIO=null, secret=null, storageos=null, vsphereVolume=null, additionalProperties={}), Volume(awsElasticBlockStore=null, azureDisk=null, azureFile=null, cephfs=null, cinder=null, configMap=null, csi=null, downwardAPI=null, emptyDir=null, ephemeral=null, fc=null, flexVolume=null, flocker=null, gcePersistentDisk=null, gitRepo=null, glusterfs=null, hostPath=null, iscsi=null, name=tls, nfs=null, persistentVolumeClaim=null, photonPersistentDisk=null, portworxVolume=null, projected=null, quobyte=null, rbd=null, scaleIO=null, secret=SecretVolumeSource(defaultMode=420, items=[], optional=null, secretName=vault-test-tls, additionalProperties={}), storageos=null, vsphereVolume=null, additionalProperties={})], additionalProperties={}), additionalProperties={}), updateStrategy=StatefulSetUpdateStrategy(rollingUpdate=RollingUpdateStatefulSetStrategy(maxUnavailable=null, partition=0, additionalProperties={}), type=RollingUpdate, additionalProperties={}), volumeClaimTemplates=[PersistentVolumeClaim(apiVersion=v1, kind=PersistentVolumeClaim, metadata=ObjectMeta(annotations={}, creationTimestamp=null, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, labels={}, managedFields=[], name=audit, namespace=null, ownerReferences=[], resourceVersion=null, selfLink=null, uid=null, additionalProperties={}), spec=PersistentVolumeClaimSpec(accessModes=[ReadWriteOnce], dataSource=null, dataSourceRef=null, resources=ResourceRequirements(claims=[], limits={}, requests={storage=1Gi}, additionalProperties={}), selector=null, storageClassName=null, volumeMode=Filesystem, volumeName=null, additionalProperties={}), status=PersistentVolumeClaimStatus(accessModes=[], allocatedResources={}, capacity={}, conditions=[], phase=Pending, resizeStatus=null, additionalProperties={}), additionalProperties={})], additionalProperties={}), status=StatefulSetStatus(availableReplicas=0, collisionCount=0, conditions=[], currentReplicas=1, currentRevision=vault-test-5bd5455664, observedGeneration=1, readyReplicas=null, replicas=1, updateRevision=vault-test-5bd5455664, updatedReplicas=1, additionalProperties={}), additionalProperties={}) 
 original desired: 
 {apiVersion=apps/v1, kind=StatefulSet, metadata={labels={app.kubernetes.io/managed-by=glasskube-operator, app.kubernetes.io/name=vault, app.kubernetes.io/instance=test, app.kubernetes.io/version=1.14.0, app.kubernetes.io/part-of=vault}, name=vault-test, namespace=default, ownerReferences=[{apiVersion=glasskube.eu/v1alpha1, kind=Vault, name=test, uid=2554a7e3-2752-4176-86b8-a45cf1090ad9}]}, spec={persistentVolumeClaimRetentionPolicy={whenDeleted=Delete, whenScaled=Retain}, replicas=1, selector={matchLabels={app.kubernetes.io/name=vault, app.kubernetes.io/instance=test, app.kubernetes.io/part-of=vault}}, serviceName=vault-test-headless, template={metadata={labels={app.kubernetes.io/managed-by=glasskube-operator, app.kubernetes.io/name=vault, app.kubernetes.io/instance=test, app.kubernetes.io/version=1.14.0, app.kubernetes.io/part-of=vault}}, spec={containers=[{args=[-c, /usr/local/bin/docker-entrypoint.sh vault server -config=/glasskube/config/config.hcl
], command=[/bin/sh], env=[{name=SKIP_SETCAP, value=true}, {name=HOSTNAME, valueFrom={fieldRef={apiVersion=v1, fieldPath=metadata.name}}}, {name=SUBDOMAIN, value=vault-test-headless}, {name=POD_IP, valueFrom={fieldRef={apiVersion=v1, fieldPath=status.podIP}}}, {name=VAULT_API_ADDR, value=https://$(POD_IP):8200}, {name=VAULT_CLUSTER_ADDR, value=https://$(HOSTNAME).$(SUBDOMAIN):8201}, {name=DB_HOST, value=vault-test-db-rw}, {name=DB_NAME, value=vault}, {name=DB_USERNAME, valueFrom={secretKeyRef={key=username, name=vault-test-db-app}}}, {name=DB_PASSWORD, valueFrom={secretKeyRef={key=password, name=vault-test-db-app}}}, {name=VAULT_PG_CONNECTION_URL, value=postgres://$(DB_USERNAME):$(DB_PASSWORD)@$(DB_HOST):5432/$(DB_NAME)}, {name=VAULT_K8S_POD_NAME, valueFrom={fieldRef={apiVersion=v1, fieldPath=metadata.name}}}, {name=VAULT_K8S_NAMESPACE, valueFrom={fieldRef={apiVersion=v1, fieldPath=metadata.namespace}}}], image=hashicorp/vault:1.14.0, lifecycle={preStop={exec={command=[killall, vault]}}}, name=vault, ports=[{containerPort=8200, name=https}, {containerPort=8201, name=https-internal}], resources={limits={memory=100Mi}, requests={memory=30Mi}}, securityContext={allowPrivilegeEscalation=false, capabilities={drop=[ALL]}, readOnlyRootFilesystem=true}, volumeMounts=[{mountPath=/glasskube/config, name=config, readOnly=true}, {mountPath=/glasskube/tls, name=tls, readOnly=true}, {mountPath=/vault/audit, name=audit}]}], securityContext={fsGroup=1000, runAsGroup=1000, runAsNonRoot=true, runAsUser=1000}, serviceAccountName=vault-test, terminationGracePeriodSeconds=10, volumes=[{configMap={name=vault-test}, name=config}, {name=tls, secret={secretName=vault-test-tls}}]}}, volumeClaimTemplates=[{apiVersion=v1, kind=PersistentVolumeClaim, metadata={name=audit}, spec={accessModes=[ReadWriteOnce], resources={requests={storage=1Gi}}}}]}}  
2023-08-25T11:01:36.565+02:00 [pool-6-thread-1] DEBUG i.j.o.p.d.k.SSABasedGenericKubernetesResourceMatcher - key: metadata actual map value: {creationTimestamp=2023-08-25T09:00:45Z, generation=1, labels={app.kubernetes.io/instance=test, app.kubernetes.io/managed-by=glasskube-operator, app.kubernetes.io/name=vault, app.kubernetes.io/part-of=vault, app.kubernetes.io/version=1.14.0}, managedFields=[{apiVersion=apps/v1, fieldsType=FieldsV1, fieldsV1={f:metadata={f:labels={f:app.kubernetes.io/instance={}, f:app.kubernetes.io/managed-by={}, f:app.kubernetes.io/name={}, f:app.kubernetes.io/part-of={}, f:app.kubernetes.io/version={}}, f:ownerReferences={k:{"uid":"2554a7e3-2752-4176-86b8-a45cf1090ad9"}={}}}, f:spec={f:persistentVolumeClaimRetentionPolicy={f:whenDeleted={}, f:whenScaled={}}, f:replicas={}, f:selector={}, f:serviceName={}, f:template={f:metadata={f:labels={f:app.kubernetes.io/instance={}, f:app.kubernetes.io/managed-by={}, f:app.kubernetes.io/name={}, f:app.kubernetes.io/part-of={}, f:app.kubernetes.io/version={}}}, f:spec={f:containers={k:{"name":"vault"}={.={}, f:args={}, f:command={}, f:env={k:{"name":"DB_HOST"}={.={}, f:name={}, f:value={}}, k:{"name":"DB_NAME"}={.={}, f:name={}, f:value={}}, k:{"name":"DB_PASSWORD"}={.={}, f:name={}, f:valueFrom={f:secretKeyRef={}}}, k:{"name":"DB_USERNAME"}={.={}, f:name={}, f:valueFrom={f:secretKeyRef={}}}, k:{"name":"HOSTNAME"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"POD_IP"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"SKIP_SETCAP"}={.={}, f:name={}, f:value={}}, k:{"name":"SUBDOMAIN"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_API_ADDR"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_CLUSTER_ADDR"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_K8S_NAMESPACE"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"VAULT_K8S_POD_NAME"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"VAULT_PG_CONNECTION_URL"}={.={}, f:name={}, f:value={}}}, f:image={}, f:lifecycle={f:preStop={f:exec={f:command={}}}}, f:name={}, f:ports={k:{"containerPort":8200,"protocol":"TCP"}={.={}, f:containerPort={}, f:name={}}, k:{"containerPort":8201,"protocol":"TCP"}={.={}, f:containerPort={}, f:name={}}}, f:resources={f:limits={f:memory={}}, f:requests={f:memory={}}}, f:securityContext={f:allowPrivilegeEscalation={}, f:capabilities={f:drop={}}, f:readOnlyRootFilesystem={}}, f:volumeMounts={k:{"mountPath":"/glasskube/config"}={.={}, f:mountPath={}, f:name={}, f:readOnly={}}, k:{"mountPath":"/glasskube/tls"}={.={}, f:mountPath={}, f:name={}, f:readOnly={}}, k:{"mountPath":"/vault/audit"}={.={}, f:mountPath={}, f:name={}}}}}, f:securityContext={f:fsGroup={}, f:runAsGroup={}, f:runAsNonRoot={}, f:runAsUser={}}, f:serviceAccountName={}, f:terminationGracePeriodSeconds={}, f:volumes={k:{"name":"config"}={.={}, f:configMap={f:name={}}, f:name={}}, k:{"name":"tls"}={.={}, f:name={}, f:secret={f:secretName={}}}}}}, f:volumeClaimTemplates={}}}, manager=vaultreconciler, operation=Apply, time=2023-08-25T09:00:45Z}, {apiVersion=apps/v1, fieldsType=FieldsV1, fieldsV1={f:status={f:collisionCount={}, f:currentReplicas={}, f:currentRevision={}, f:observedGeneration={}, f:replicas={}, f:updateRevision={}, f:updatedReplicas={}}}, manager=kube-controller-manager, operation=Update, subresource=status, time=2023-08-25T09:00:45Z}], name=vault-test, namespace=default, ownerReferences=[{apiVersion=glasskube.eu/v1alpha1, kind=Vault, name=test, uid=2554a7e3-2752-4176-86b8-a45cf1090ad9}], resourceVersion=2242, uid=33102e14-0069-4979-8739-64f86f00822e} managedFieldValue: {f:labels={f:app.kubernetes.io/instance={}, f:app.kubernetes.io/managed-by={}, f:app.kubernetes.io/name={}, f:app.kubernetes.io/part-of={}, f:app.kubernetes.io/version={}}, f:ownerReferences={k:{"uid":"2554a7e3-2752-4176-86b8-a45cf1090ad9"}={}}} 
2023-08-25T11:01:36.565+02:00 [pool-6-thread-1] DEBUG i.j.o.p.d.k.SSABasedGenericKubernetesResourceMatcher - key: labels actual map value: {app.kubernetes.io/instance=test, app.kubernetes.io/managed-by=glasskube-operator, app.kubernetes.io/name=vault, app.kubernetes.io/part-of=vault, app.kubernetes.io/version=1.14.0} managedFieldValue: {f:app.kubernetes.io/instance={}, f:app.kubernetes.io/managed-by={}, f:app.kubernetes.io/name={}, f:app.kubernetes.io/part-of={}, f:app.kubernetes.io/version={}} 
2023-08-25T11:01:36.565+02:00 [pool-6-thread-1] DEBUG i.j.o.p.d.k.SSABasedGenericKubernetesResourceMatcher - key: spec actual map value: {podManagementPolicy=OrderedReady, replicas=1, revisionHistoryLimit=10, selector={matchLabels={app.kubernetes.io/instance=test, app.kubernetes.io/name=vault, app.kubernetes.io/part-of=vault}}, serviceName=vault-test-headless, template={metadata={labels={app.kubernetes.io/instance=test, app.kubernetes.io/managed-by=glasskube-operator, app.kubernetes.io/name=vault, app.kubernetes.io/part-of=vault, app.kubernetes.io/version=1.14.0}}, spec={containers=[{args=[-c, /usr/local/bin/docker-entrypoint.sh vault server -config=/glasskube/config/config.hcl
], command=[/bin/sh], env=[{name=SKIP_SETCAP, value=true}, {name=HOSTNAME, valueFrom={fieldRef={apiVersion=v1, fieldPath=metadata.name}}}, {name=SUBDOMAIN, value=vault-test-headless}, {name=POD_IP, valueFrom={fieldRef={apiVersion=v1, fieldPath=status.podIP}}}, {name=VAULT_API_ADDR, value=https://$(POD_IP):8200}, {name=VAULT_CLUSTER_ADDR, value=https://$(HOSTNAME).$(SUBDOMAIN):8201}, {name=DB_HOST, value=vault-test-db-rw}, {name=DB_NAME, value=vault}, {name=DB_USERNAME, valueFrom={secretKeyRef={key=username, name=vault-test-db-app}}}, {name=DB_PASSWORD, valueFrom={secretKeyRef={key=password, name=vault-test-db-app}}}, {name=VAULT_PG_CONNECTION_URL, value=postgres://$(DB_USERNAME):$(DB_PASSWORD)@$(DB_HOST):5432/$(DB_NAME)}, {name=VAULT_K8S_POD_NAME, valueFrom={fieldRef={apiVersion=v1, fieldPath=metadata.name}}}, {name=VAULT_K8S_NAMESPACE, valueFrom={fieldRef={apiVersion=v1, fieldPath=metadata.namespace}}}], image=hashicorp/vault:1.14.0, imagePullPolicy=IfNotPresent, lifecycle={preStop={exec={command=[killall, vault]}}}, name=vault, ports=[{containerPort=8200, name=https, protocol=TCP}, {containerPort=8201, name=https-internal, protocol=TCP}], resources={limits={memory=100Mi}, requests={memory=30Mi}}, securityContext={allowPrivilegeEscalation=false, capabilities={drop=[ALL]}, readOnlyRootFilesystem=true}, terminationMessagePath=/dev/termination-log, terminationMessagePolicy=File, volumeMounts=[{mountPath=/glasskube/config, name=config, readOnly=true}, {mountPath=/glasskube/tls, name=tls, readOnly=true}, {mountPath=/vault/audit, name=audit}]}], dnsPolicy=ClusterFirst, restartPolicy=Always, schedulerName=default-scheduler, securityContext={fsGroup=1000, runAsGroup=1000, runAsNonRoot=true, runAsUser=1000}, serviceAccount=vault-test, serviceAccountName=vault-test, terminationGracePeriodSeconds=10, volumes=[{configMap={defaultMode=420, name=vault-test}, name=config}, {name=tls, secret={defaultMode=420, secretName=vault-test-tls}}]}}, updateStrategy={rollingUpdate={partition=0}, type=RollingUpdate}, volumeClaimTemplates=[{apiVersion=v1, kind=PersistentVolumeClaim, metadata={name=audit}, spec={accessModes=[ReadWriteOnce], resources={requests={storage=1Gi}}, volumeMode=Filesystem}, status={phase=Pending}}]} managedFieldValue: {f:persistentVolumeClaimRetentionPolicy={f:whenDeleted={}, f:whenScaled={}}, f:replicas={}, f:selector={}, f:serviceName={}, f:template={f:metadata={f:labels={f:app.kubernetes.io/instance={}, f:app.kubernetes.io/managed-by={}, f:app.kubernetes.io/name={}, f:app.kubernetes.io/part-of={}, f:app.kubernetes.io/version={}}}, f:spec={f:containers={k:{"name":"vault"}={.={}, f:args={}, f:command={}, f:env={k:{"name":"DB_HOST"}={.={}, f:name={}, f:value={}}, k:{"name":"DB_NAME"}={.={}, f:name={}, f:value={}}, k:{"name":"DB_PASSWORD"}={.={}, f:name={}, f:valueFrom={f:secretKeyRef={}}}, k:{"name":"DB_USERNAME"}={.={}, f:name={}, f:valueFrom={f:secretKeyRef={}}}, k:{"name":"HOSTNAME"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"POD_IP"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"SKIP_SETCAP"}={.={}, f:name={}, f:value={}}, k:{"name":"SUBDOMAIN"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_API_ADDR"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_CLUSTER_ADDR"}={.={}, f:name={}, f:value={}}, k:{"name":"VAULT_K8S_NAMESPACE"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"VAULT_K8S_POD_NAME"}={.={}, f:name={}, f:valueFrom={f:fieldRef={}}}, k:{"name":"VAULT_PG_CONNECTION_URL"}={.={}, f:name={}, f:value={}}}, f:image={}, f:lifecycle={f:preStop={f:exec={f:command={}}}}, f:name={}, f:ports={k:{"containerPort":8200,"protocol":"TCP"}={.={}, f:containerPort={}, f:name={}}, k:{"containerPort":8201,"protocol":"TCP"}={.={}, f:containerPort={}, f:name={}}}, f:resources={f:limits={f:memory={}}, f:requests={f:memory={}}}, f:securityContext={f:allowPrivilegeEscalation={}, f:capabilities={f:drop={}}, f:readOnlyRootFilesystem={}}, f:volumeMounts={k:{"mountPath":"/glasskube/config"}={.={}, f:mountPath={}, f:name={}, f:readOnly={}}, k:{"mountPath":"/glasskube/tls"}={.={}, f:mountPath={}, f:name={}, f:readOnly={}}, k:{"mountPath":"/vault/audit"}={.={}, f:mountPath={}, f:name={}}}}}, f:securityContext={f:fsGroup={}, f:runAsGroup={}, f:runAsNonRoot={}, f:runAsUser={}}, f:serviceAccountName={}, f:terminationGracePeriodSeconds={}, f:volumes={k:{"name":"config"}={.={}, f:configMap={f:name={}}, f:name={}}, k:{"name":"tls"}={.={}, f:name={}, f:secret={f:secretName={}}}}}}, f:volumeClaimTemplates={}} 
2023-08-25T11:01:36.566+02:00 [pool-6-thread-1] DEBUG i.j.o.p.d.k.SSABasedGenericKubernetesResourceMatcher - key: persistentVolumeClaimRetentionPolicy actual map value: null managedFieldValue: {f:whenDeleted={}, f:whenScaled={}}

Notice how in the last line, it says actual map value: null. This is not something keepOnlyManagedFields is designed to deal with, so that's where the error happens.

Side note: This can probably happen for other resources with unsupported properties as well. I don't understand why Kubernetes keeps the managed field but discards the property…

Thanks for your suggested workaround, we already removed the offending statement, since we want to support version 1.26 anyways. I just thought that the SDK could handle this edge-case more gracefully 🙂

Edit: For testing I created a cluster like this:

minikube start --profile=legacy --kubernetes-version 1.26
shawkins

shawkins commented on Aug 25, 2023

@shawkins
shawkins
on Aug 25, 2023
Collaborator

Side note: This can probably happen for other resources with unsupported properties as well. I don't understand why Kubernetes keeps the managed field but discards the property…

I believe that case is happening as well here: kubernetes/kubernetes#118519 (comment)

on short notice you might want to try the generic matcher

While you won't get the NPE the matching won't work for pruned fields - the match logic will see that desired state has something that is missing in the actual.

csviri
linked a pull request that will close this issuereproduce ssa issue with `persistentVolumeClaimRetentionPolicy` #2037on Aug 29, 2023
csviri
mentioned this on Aug 29, 2023
csviri

csviri commented on Aug 29, 2023

@csviri
csviri
on Aug 29, 2023
Collaborator

see #2038

csviri
linked a pull request that will close this issuefeat: sanitization of resources for matching #2042on Sep 14, 2023
csviri
closed this as completedon Sep 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    Participants

    @shawkins@csviri@kosmoz

    Issue actions
      NullPointerException in SSABasedGenericKubernetesResourceMatcher · Issue #2032 · operator-framework/java-operator-sdk