Deprecate vars

[natasha41575]

The new replacements field is intended to be the replacement for the vars feature. Now that its implementation is complete, we should look into deprecating vars.

Some outstanding migration issues:

#3978
#4012
#4337
#4359

These issues have the same underlying problem that users are adding vars in strings where the var is not delimited, e.g. Host(www-$(NAMESPACE).example.com).

We should also take a look at use cases such as #4080 where the user wants to replace a string in an array without having to rely on the order of items in the array.

Generators for service accounts have been proposed as the solution for the first issue, but does not easily resolve the second. options.prefix and options.suffix have been proposed for the third (more details in the issue).

Additionally, #4120 requests a way to replace a template variable.

[k8s-ci-robot]

@natasha41575: This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[aodinokov]

@natasha41575
JFYI Maybe it will give some inspiration :) .. When I experimented with krm-based replacement transformer last year I came to the solution with
multiref source
This solution was inspired by kpt substitutions. where it was possible to concat several values using some template/pattern.

To make something similar to Host(www-$(NAMESPACE).example.com) it's possible to do the following

- source:
    multiref:
      refs:
      - objref:
          kind: Secret
          name: node1-bmc-secret
        fieldref: stringData.somefield # let's say it has value 'docker-ce . grep 19.03.12 ...' and we want to substitute 19.03.12 with value from another yaml ephemeral-catalogue and field versions.docker-ce.. 
      - objref:
          name: ephemeral-catalogue
        fieldref: versions.docker-ce
      template: |
        {{ regexReplaceAll "docker-ce . grep 19.03.12" (index .Values 0) (printf "docker-ce | grep %s" (index .Values 1)) }}
  target:
    objref:
      kind: Secret
      name: node1-bmc-secret
    fieldrefs: 
    - stringData.somefield

Basically what this solution does - in case muliref object is set all values extracted there are passed to go-template as values. go template engine uses that as .Values and it has to produce resulting string that will go to target. Target in this example is the same field as source[0]. Go-template engine had Sprig-functions included.
this example would work for $(VERSION) substring as well... - it was necessary to replace $(VERSION) instead of docker-ce . grep 19.03.12 and to user index .Values 1' instead of printf`, e.g.:

- source:
    multiref:
      refs:
      - objref:
          kind: Secret
          name: node1-bmc-secret
        fieldref: stringData.somefield # let's say it contains Host(www-$(NAMESPACE).example.com)
      - objref:
          name: ephemeral-catalogue
        fieldref: versions.docker-ce # and this field contains the value to put instead of $(NAMESPACE)
      template: |
        {{ regexReplaceAll "$(NAMESPACE)" (index .Values 0) (index .Values 1) }}
  target:
    objref:
      kind: Secret
      name: node1-bmc-secret
    fieldrefs: 
    - stringData.somefield

This approach in practice demonstrated a lot of flexibility - it's possible to concat string, build string with patterns, substitute part of it and many other things, because go template is basically a language and that allows alot.. I guess it's much more flexible than approach with delimiter.

From implementation standpoint it was really easy to implement it:
https://github.com/aodinokov/noctl-airship-poc/blob/0e2e126c0601bdc820ade673d47ff506b328edf0/kpt-functions/replacement/function.go#L78 - definition of multiref source.
https://github.com/aodinokov/noctl-airship-poc/blob/0e2e126c0601bdc820ade673d47ff506b328edf0/kpt-functions/replacement/function.go#L175 - is function that collects all sources and runs go-template with their values
plus some checks that multiref and other options are mutually exclusive...

[yuwenma]

Per offline discussion, we keep var in kustomize for one year according to the kubernetes one year deprecating policy. When var is detected in kustomization.yaml, kustomize warns users about the deprecation date, and request them to migrate to the replacements field.

[natasha41575]

@KnVerey @yuwenma @monopole Are any of the outstanding issues blockers for vars deprecation? I am tempted to mark them deprecated now, and address these users' concerns in between deprecation and removal.

[KnVerey]

The question that comes to mind for me is how exactly we want to do this deprecation. Since this is a field in the Kustomization API (as opposed to a CLI cmd/flag), if we follow the Kubernetes API backwards compatibility policy, we shouldn't ever remove it from v1beta1 Kustomization; instead, we should cut v1 with vars etc. already removed, and have the CLI support both v1beta1 and v1 for the year deprecation period. This would also mean a single migration guide for all the deprecated features, and support in the kustomize edit fix command for updating the API version (and all associated conversions).

So overall the process would be:

1. Create v1 Kustomization and support both in the CLI
2. Ensure kustomize edit fix can perform upgrade to the extent possible, and docs are fully up to date.
3. Emit deprecation warnings on use of v1beta1 (or unspecified, assumed to be v1beta1)*
4. Wait a year from the above and address any feature parity gaps during that time.
5. Make CLI stop supporting v1beta1 with a major version bump

* A huge downside to that is that it requires all users to take some action, not just the ones affected by a deprecation. Perhaps before emitting deprecation warnings, we could also implement a silent conversion feature to enable v1beta1 support indefinitely if and only if the instance is using fields that exist in both versions or have exact equivalents (e.g. bases->resources but not vars->replacements). In other words, if and only if carrying that support will not incur technical debt / inability to remove unsupported features from the codebase.

WDYT?

If we go with that plan, we can start it any time IMO--the first step is mostly a matter of confidence in the feature set we want v1 to retain, and of figuring out how to do this cleanly in the codebase. Experience adding Composition (which I'm starting now) may be informative for the latter.

Regardless, you're right that the long deprecation period leaves us plenty of time to implement more enhancements to replacements as required. Incidentally, several of the asks seem to be for unstructured edits / variable substitution, which we will definitely not support. For the cases where the problem is essentially problematically structured CRD fields, the plugin graduation story may also play a big role in preparedness for feature removal.

[natasha41575]

The overall process you've proposed LGTM. We can start looking at some of the issues in https://github.com/kubernetes-sigs/kustomize/projects/12 to prepare for a v1 Kustomization.

Perhaps before emitting deprecation warnings, we could also implement a silent conversion feature to enable v1beta1 support indefinitely if and only if the instance is using fields that exist in both versions or have exact equivalents (e.g. bases->resources but not vars->replacements).

SGTM. So long as all the fields are supported, I see no issue with converting v1beta1 to v1 silently.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[KnVerey]

/remove-lifecycle stale

[natasha41575]

/lifecycle frozen