Description
Is your feature request related to a problem? Please describe
Opening up this feature request for discussion.
Writing Java Rest Tests (that run outside of the cluster) can be verbose and clunky. It can require doing a lot by hand (i.e. creating random vectors of a certain distribution) and can be difficult to connect to the external clusters - I tried running the k-NN gradle tests via the :integTestRemote command against an AWS OpenSearch cluster and it could not connect to it for some reason, even after hours of debugging.
Overall, this limits the speed at which tests can be written and how extensive the tests can be.
Describe the solution you'd like
I want to be able to write integration tests in python that can be run against external clusters. I want to be able to use libraries from python like faker, numpy, etc. I want to really easily be able to configure the tests to run against different OpenSearch cluster types. I dont want to have to interact too heavily with gradle.
Itd also be cool to assign tests version compatibility so we can write tests that we confirm work against old versions.
Related component
Build
Describe alternatives you've considered
That being said, it could just be hacked together in the k-NN plugin, but I believe having this ability will help the OpenSearch ecosystem overall, so itd be nice to have in core.
Additional context
No response