Kiota Coverage Test Suite

[darrelmiller]

Using the GitHub search mechanism, walk the advertised OpenAPI descriptions, generate clients and compile those clients. We can exclude Microsoft Graph because we already test that elsewhere

Benefits:

• This will give us advanced visibility into OpenAPI descriptions that are index but don't work with Kiota
• This will give us metrics on the number of APIs index
• This will provide confidence about our support for broad coverage of published APIs
• Using the warnings and errors we detect about Kiota's ability to support certain constructs, we can prioritize where to enhance support.

[darrelmiller]

Example of code to walk the Apis.guru list https://github.com/microsoft/OpenAPI.NET/blob/vnext/test/Microsoft.OpenApi.SmokeTests/ApiGurus.cs#L41

[sebastienlevert]

Some thoughts on the APIs we should be testing against. Using this https://www.postman.com/explore/most-popular-apis-this-year to identify the most used. I'm linking either the yaml file if available, or the Postman collection, from which we could export the collection and transform it to OpenAPI.

• Salesforce Platform APIs (https://postman.com/salesforce-developers/workspace/salesforce-developers/collection/12721794-67cb9baa-e0da-4986-957e-88d8734647e2?ctx=documentation)
• Twitter API v2 (https://api.apis.guru/v2/specs/twitter.com/current/2.21/openapi.yaml)
• Notion Public Beta (https://api.apis.guru/v2/specs/notion.com/1.0.0/openapi.yaml)
• WhatsApp Cloud API (https://api.apis.guru/v2/specs/whatsapp.local/1.0/openapi.yaml)
• Microsoft Graph (https://raw.githubusercontent.com/microsoftgraph/msgraph-metadata/master/openapi/v1.0/openapi.yaml)
• Slack Web API (https://api.apis.guru/v2/specs/slack.com/1.7.0/openapi.yaml)
• PayPal APIs (https://postman.com/paypal/collection/19024122-92a85d0e-51e7-47da-9f83-c45dcb1cdf24)
• Stripe (https://api.apis.guru/v2/specs/stripe.com/2020-08-27/openapi.yaml)
• Google Maps Platform (https://raw.githubusercontent.com/googlemaps/openapi-specification/main/dist/google-maps-platform-openapi3.yml)
• Razorpay APIs (https://postman.com/razorpaydev/workspace/razorpay-public-workspace/collection/12492020-952c7295-118c-400f-8f2c-5266ef6f689a?ctx=documentation)
• MongoDB Data APIs (https://postman.com/mongodb-devrel/workspace/mongodb-public/collection/17898583-25682080-e247-4d25-8e5c-1798461c7db4?ctx=documentation)
• Zoho CRM REST APIs (https://postman.com/zohocrmdevelopers/workspace/zoho-crm-developers/collection/8522016-0a15778a-ccb1-4676-98b7-4cf1fe7fc940?ctx=documentation)
• Datadog (https://postman.com/datadog/workspace/datadog-s-public-workspace/collection/7274195-66ef21d8-e159-4d7d-8ded-c511e1abe189?ctx=documentation)
• Meraki Dashboard API (https://api.apis.guru/v2/specs/meraki.com/1.12.0/openapi.yaml)
• Pipedrive API (https://developers.pipedrive.com/docs/api/v1/openapi.yaml)
• Twilio Messaging / SMS (https://api.apis.guru/v2/specs/twilio.com/api/1.38.3/openapi.yaml)
• Amplitude Analytics APIs (https://postman.com/amplitude-dev-docs/workspace/amplitude-developers/collection/20044411-88bd70c1-9e00-483c-aaf1-2b7159f9bb2b?ctx=documentation)
• DocuSign REST API (https://api.apis.guru/v2/specs/docusign.net/v2.1/openapi.yaml)
• BookingAPI (https://postman.com/clientapi/workspace/apitude-by-hotelbeds/collection/295910-c411a5fa-523c-4048-afef-5afd52a294b4?ctx=documentation)

Now, these would be considered as "most popular" but probably not diverse enough to help us identify edge cases. Should we just loop through all of APIs guru catalog and generate every possible clients?

[darrelmiller]

It's what speakeasy did. https://github.com/speakeasy-api/openapi-directory/tree/main/SDKs I have mixed feelings about it.

[sebastienlevert]

I absolutely agree with you. I don't think it's necessarily adding real value... Using the top APIs is certainly a good first step, but I don't feel we will find all the crazy things we can see in an OpenAPI description... @baywet or @RabebOthmani any idea on how we can get a high level of confidence we are generating valid clients with a diverse set of API definitions?

[baywet]

As I've started working on this, I did have a couple of remarks:

• I don't think using msgraph in the smoke tests set brings much value as we're designing with Microsoft Graph in mind and already have extensive processes for quality (raptor etc...)
• I don't think going and getting every random API definition out there is going to provide much ROI. There are duplicates (multiple flavors of Azure, GitHub, etc...) and the triage process is going to take much longer for each new API definition we use in the test set.

Bottom line, starting with 15/20 popular APIs like Seb listed is a good starting place. I do have a couple of remarks on that list:

• Is it worth including the postman collections? If we identify defects of the description, we have nowhere to report them.
• Should we include Twitter, given what's currently going on with their API story?

Early implementations already uncovered a couple of things #2351, #2352, #2353 (and that's without counting the issues I've reported to GitHub directly)

[sebastienlevert]

I agree for Twitter, we might want to hold on it. Though, it's a very popular API so with (or without) the paid APIs, I think it's still relevant from a generation perspective.

For Postman collections, you might be right... It makes 11 OpenAPI descriptions... Any other candidates we want to include?