OpenAPI Provider by guicassolato · Pull Request #157 · kubernetes-sigs/ingress2gateway

guicassolato · 2024-05-04T11:38:00Z

What type of PR is this?
/kind feature

What this PR does / why we need it:
Adds new openapi provider.

Which issue(s) this PR fixes:
Closes #147

Does this PR introduce a user-facing change?:

- New OpenAPI Provider

Try this PR out:

make build
./ingress2gateway print --providers openapi3 --input-file=pkg/i2gw/providers/openapi3/fixtures/input/1-petstore3.yaml

k8s-ci-robot · 2024-05-04T11:38:09Z

Welcome @guicassolato!

It looks like this is your first PR to kubernetes-sigs/ingress2gateway 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/ingress2gateway has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2024-05-04T11:38:10Z

Hi @guicassolato. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

guicassolato · 2024-05-04T11:46:39Z

@arkodg @youngnick @LiorLieberman et al

Sharing this as a draft for some initial feedback.

Quite a few TODOs ahead still, including:

Apart from more testing of course.

Please check out the test cases already covered under providers/openapi/fixtures.

LiorLieberman

Thanks @guicassolato, that a very useful contribution! left some comments

LiorLieberman · 2024-05-04T14:52:47Z

+## Known limitations
+
+* Only offline translation supported – i.e. `--input_file` required
+* All API operation [paths](https://swagger.io/specification/v3/#paths-object) treated as `Exact` type – i.e. no support for [path templating](https://swagger.io/specification/v3/#path-templating), therefore no `PathPrefix`, nor `RegularExpression` path types output


Curious to know why no support for path templating? a limitation you encountered ?

I'm afraid there aren't many good options for that. At least, not that I could think of.

Delete the part in each path from the first variable placeholder, all the way to the end of the path, and use PathPrefix type → some nasty stuff could happen with the ordering of the route rules

Use RegularExpression type → not core; what flavour of regex would we use?

Path templates are not like server variables that in some case give us at least a closed set of possible values (enum.)

LiorLieberman · 2024-05-04T15:03:28Z

+	spec, err := loader.LoadFromFile(filename)
+	if err != nil {
+		return nil, fmt.Errorf("failed to load OpenAPI spec: %w", err)
+	}
+
+	if err := spec.Validate(ctx); err != nil {
+		return nil, fmt.Errorf("invalid OpenAPI 3.x spec: %w", err)
+	}
+


@levikobi @mlavacca I think we should think about whether we want to preserve this all supported providers by default functionality.

For example here, if we merge this PR, if someone provides --input-file but doesn't provide provider this will always fail (because a regular yaml/json input file would never be a valid openapi spec.

@guicassolato, regardless to the above, I think running openapi provider should always be mutually exclusive to other providers.

What's the use case for converting to all supported providers anyway?

thats how we started, I think we need to change it, but will have a separate issue for that

LiorLieberman · 2024-05-04T15:05:01Z

+	s.mu.Lock()
+	defer s.mu.Unlock()


why do we need a lock? couldn't find a place where you run addResource or getResource concurrently

It's a good practice in general. Since the dependency is injected, the converter cannot prevent external usage of the storage.

In fact, I now realise my implementation is poor. A concurrent call to ReadResourcesFromFile would cause the entire storage to be invalidated while completely bypassing the lock, when arguably the right thing to do would be to use the context to infer whether reading a resource should be additive to the storage or clear it first. At the very least, do the latter, but not forcibly recreate the storage. I will fix that.

From the perspective of the openapi package, there is no way to tell what a client will do with those two competing processes.

Signed-off-by: Guilherme Cassolato <guicassolato@gmail.com>

…ackage

… converter's unused fields

mlavacca · 2024-05-07T15:50:43Z

/cc @mlavacca

guicassolato · 2024-05-08T09:04:56Z

I want to mark this as ready for review, but not before coming to a decision about how to proceed for backendRefs and gatewayClassName.

It was suggested to use command-line args for those. I would like to know how people feel about that?

A few questions to drive the conversation next should we say this indeed is the desired approach:

Should I just define those as flags of the print command?
Since they don't have meaning for other providers converting from Ingress resources, will providers like istio, ingress-nginx, kong, etc ignore the values when supplied?
Perhaps call those options --default-backend-name and --default-gateway-controller-name?
Or they could be namespaced specifically for this provider – e.g. --openapi3-bakcend-name and --openapi3-gateway-controller-name – tho I can see a possible mess in the future should other providers ask for other similar options. If it is provider-specific, should the options be required when --providers includes openapi3 and forbidden otherwise?
Provider-specific options could also be serialised into a single command-line flag (e.g. --provider-options "backend-name=x,controller-name=y")
Is it starting to get a little bit too much to pass in the call to ToGatewayAPIResources? Should we change the API to accept a more generic PrintOptions type?

cc @youngnick @arkodg @LiorLieberman

LiorLieberman · 2024-05-08T09:22:35Z

/cc @arkodg

mlavacca · 2024-05-08T09:48:27Z

I want to mark this as ready for review, but not before coming to a decision about how to proceed for backendRefs and gatewayClassName.

It was suggested to use command-line args for those. I would like to know how people feel about that?

A few questions to drive the conversation next should we say this indeed is the desired approach:

Should I just define those as flags of the print command?

Since they don't have meaning for other providers converting from Ingress resources, will providers like istio, ingress-nginx, kong, etc ignore the values when supplied?

Perhaps call those options --default-backend-name and --default-gateway-controller-name?

Or they could be namespaced specifically for this provider – e.g. --openapi3-bakcend-name and --openapi3-gateway-controller-name – tho I can see a possible mess in the future should other providers ask for other similar options. If it is provider-specific, should the options be required when --providers includes openapi3 and forbidden otherwise?

+1 on this approach. having access to generic --default-backend-name, --default-gateway-controller-name flags might be confusing for users. I'd prefer having openapi3--prefixed command line flags to avoid such a confusion. openapi is a special case, and I do cannot think of many other such special cases in the future

Provider-specific options could also be serialised into a single command-line flag (e.g. --provider-options "backend-name=x,controller-name=y")

Is it starting to get a little bit too much to pass in the call to ToGatewayAPIResources? Should we change the API to accept a more generic PrintOptions type?

+1 on introducing a PrintOptions struct to pass all the needed parameters.

mlavacca

Thanks for this PR! I left some initial generic comments in the review

mlavacca · 2024-05-08T09:35:24Z

+}
+
+type storage struct {
+	mu sync.RWMutex


Why do we need thread safety here? I haven't seen any concurrency in the code, did I miss something?

I tried to answer it here: #157 (comment)

TL;DR - I think not having it is a road to problems in the future.

Not supposed to sound unnecessarily ominous here... I do believe things change, such as – I'm making stuff up here, ok? – start supporting multiple input files (or an input directory equivalently) and, if so, why not calling ReadResourcesFromFile concurrently?

It's a small price to pay IMO, but if more people believe we're better off without the mutex, I can remove it – even if I don't understand why.

mlavacca · 2024-05-08T09:37:41Z

+	gatewayResources := i2gw.GatewayResources{
+		Gateways:        make(map[types.NamespacedName]gatewayv1.Gateway),
+		HTTPRoutes:      make(map[types.NamespacedName]gatewayv1.HTTPRoute),
+		TLSRoutes:       make(map[types.NamespacedName]gatewayv1alpha2.TLSRoute),
+		TCPRoutes:       make(map[types.NamespacedName]gatewayv1alpha2.TCPRoute),
+		ReferenceGrants: make(map[types.NamespacedName]gatewayv1beta1.ReferenceGrant),
+	}


Do we really need to perform this initialization here?

For TLSRoutes, TCPRoutes and ReferenceGrants probably not.

Nil maps are safe to read and it kinds of make sense to me that the caller will not try to write back on the GatewayResources struct without checking first. Well, if it does, it's the caller's problem anyway.

Besides, returning nil for TLSRoutes, TCPRoutes and ReferenceGrants seems semantically the right thing to do, I reckon. I'll fix it.

Thanks!

… (mapping, filtering), instead of custom implementation

* provider-specific conf renamed as provided-specific flags * mutex to read/write provider-specific flag definitions wrapped within a type along with the definitions themselves * minor string handling enhancements (concatenation, trim prefix) * additional comments explaining logics and reasoning throughout the code (thread-safety, helper funcs and expressions, etc)

…hing provider

LiorLieberman

I think I answered all the open comments.

This looks good to me, going to approve it and I asked @mlavacca to review and leave final lgtm.

Thanks Gui!
/approve

mlavacca

Looks great to me, thanks @guicassolato!

/ok-to-test
/approve
/lgtm

LiorLieberman · 2024-05-29T09:31:29Z

/hold for implementing the requested changes from last review.

mlavacca

/lgtm

I'll let @LiorLieberman take a final look and remove the unhold the PR

k8s-ci-robot · 2024-05-30T14:42:37Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: guicassolato, LiorLieberman, mlavacca

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [LiorLieberman,mlavacca]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

LiorLieberman · 2024-05-30T16:08:27Z

/unhold

LiorLieberman · 2024-05-30T16:09:43Z

@guicassolato can you change the release notes to be slightly more detailed? It will make my life easier when releasing a new version

guicassolato · 2024-05-31T09:07:11Z

@guicassolato can you change the release notes to be slightly more detailed? It will make my life easier when releasing a new version

It was merged before I could address this, @LiorLieberman 😞

* Base openapi provider implementation Signed-off-by: Guilherme Cassolato <guicassolato@gmail.com> * Return without error from unimplemented ReadResourcesFromCluster func * Move slice helper functions Map nd Filter into the providers/common package * openapi package renamed openapi3 * thread-safe storage of specs * refactor: resource reader simplified as part of Provider and removing converter's unused fields * make provider resilient to invalid input openapi specs * Gateway and parentRefs * Declare github.com/getkin/kin-openapi as a direct dependency * Use github.com/samber/lo for handling slices based on common patterns (mapping, filtering), instead of custom implementation * Remove initialization of non converted kinds of resources TLSRoutes, TCPRoutes, ReferenceGrants * init func brought further upwards * code format * Provider-specific options --openapi3-backend and --openapi3-gateway-class-name Defined using a newly introduced system of dynamically registered provider-specific configuration flags. * provider-specific flag: --openapi3-gateway-tls-secret * ReferenceGrants for HTTPRoute to Backends and Gateway to TLS Secrets * fix: provider-specific configs for providers with dashes in the name * name Gateway and HTTPRoutes after the OAS title * fix: missing backend ref argument * update README * Support for backend port numbers * refactor: addressed comments from the pr * provider-specific conf renamed as provided-specific flags * mutex to read/write provider-specific flag definitions wrapped within a type along with the definitions themselves * minor string handling enhancements (concatenation, trim prefix) * additional comments explaining logics and reasoning throughout the code (thread-safety, helper funcs and expressions, etc) * log message in case of provider-specific flag supplied without a matching provider * more comments to explain the flow and decision of the converter * lint: typos, gofmt and false positives * return error in case of invalid OpenAPI 3.x spec --------- Signed-off-by: Guilherme Cassolato <guicassolato@gmail.com>

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/feature Categorizes issue or PR as related to a new feature. labels May 4, 2024

k8s-ci-robot requested review from Xunzhuo and levikobi May 4, 2024 11:38

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label May 4, 2024

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 4, 2024

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label May 4, 2024

LiorLieberman reviewed May 4, 2024

View reviewed changes

guicassolato added 4 commits May 6, 2024 10:56

Base openapi provider implementation

b2efd21

Signed-off-by: Guilherme Cassolato <guicassolato@gmail.com>

Return without error from unimplemented ReadResourcesFromCluster func

d52c628

Move slice helper functions Map nd Filter into the providers/common p…

f23ab1e

…ackage

openapi package renamed openapi3

acc95e9

guicassolato force-pushed the openapi-provider branch from a70bfe9 to acc95e9 Compare May 6, 2024 09:39

guicassolato added 3 commits May 6, 2024 12:41

thread-safe storage of specs

6a22070

refactor: resource reader simplified as part of Provider and removing…

7151f6a

… converter's unused fields

make provider resilient to invalid input openapi specs

f9d8d13

guicassolato force-pushed the openapi-provider branch from 8df08c1 to f9d8d13 Compare May 6, 2024 11:20

guicassolato added 2 commits May 7, 2024 14:52

Gateway and parentRefs

8f48ece

Declare github.com/getkin/kin-openapi as a direct dependency

4c4ebaa

k8s-ci-robot requested a review from mlavacca May 7, 2024 15:50

k8s-ci-robot requested a review from arkodg May 8, 2024 09:22

mlavacca reviewed May 8, 2024

View reviewed changes

Use github.com/samber/lo for handling slices based on common patterns…

98db36e

… (mapping, filtering), instead of custom implementation

guicassolato added 3 commits May 21, 2024 15:41

log message in case of provider-specific flag supplied without a matc…

febc49a

…hing provider

more comments to explain the flow and decision of the converter

d9a40ab

guicassolato requested a review from LiorLieberman May 21, 2024 16:14

LiorLieberman reviewed May 23, 2024

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 23, 2024

mlavacca approved these changes May 29, 2024

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 29, 2024

k8s-ci-robot assigned mlavacca May 29, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 29, 2024

lint: typos, gofmt and false positives

48dc906

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. and removed lgtm "Looks good to me", indicates that a PR is ready to be merged. labels May 29, 2024

guicassolato force-pushed the openapi-provider branch 3 times, most recently from 33f49b0 to b6d703e Compare May 29, 2024 10:27

return error in case of invalid OpenAPI 3.x spec

b174c23

guicassolato force-pushed the openapi-provider branch from b6d703e to b174c23 Compare May 29, 2024 10:30

guicassolato requested a review from LiorLieberman May 29, 2024 10:48

mlavacca approved these changes May 30, 2024

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 30, 2024

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 30, 2024

k8s-ci-robot merged commit 21f9c46 into kubernetes-sigs:main May 30, 2024

Conversation

guicassolato commented May 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented May 4, 2024

Uh oh!

k8s-ci-robot commented May 4, 2024

Uh oh!

guicassolato commented May 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LiorLieberman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guicassolato May 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mlavacca commented May 7, 2024

Uh oh!

guicassolato commented May 8, 2024

Uh oh!

LiorLieberman commented May 8, 2024

Uh oh!

mlavacca commented May 8, 2024

Uh oh!

mlavacca left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LiorLieberman left a comment

Choose a reason for hiding this comment

Uh oh!

mlavacca left a comment

Choose a reason for hiding this comment

Uh oh!

LiorLieberman commented May 29, 2024

Uh oh!

mlavacca left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented May 30, 2024

Uh oh!

LiorLieberman commented May 30, 2024

Uh oh!

LiorLieberman commented May 30, 2024

Uh oh!

guicassolato commented May 31, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

guicassolato commented May 4, 2024 •

edited

Loading

guicassolato commented May 4, 2024 •

edited

Loading

guicassolato May 4, 2024 •

edited

Loading