OpenSearch: Automate increment to next development iteration

[dblock]

Is your feature request related to a problem? Please describe

We begin incrementing versions after we decide that we want a release. This takes a day and makes it that nightly distribution builds do not include full bundle until T-14 days to release (#1140). Instead, prepare the project to make the next release after the previous release is done, so that we can save that day and have release candidates always available for the next development iteration.

After automatic tagging has happened, increment the version for the next development iteration. For example:

• when we release 1.2.1, increment version to 1.2.2 on the 1.2 branch
• when we release 1.3.0, increment version to 1.4.0 on 1.x branch and 1.3.1 on 1.3 branch
• when we release 2.0, increment version to 3.0 on main

x 15 plugins

Describe the solution you'd like

Every project should have a workflow that notices new tags, and increments versions accordingly and makes a PR.

• On OpenSearch this change is a bit involved because it needs custom work in bcwVersions.
• In plugins it's mostly build.gradle, but there are plugins with their own weird custom steps

Acceptance Criteria

• POC Concept/Solution to Automate the version increment to next development iteration
  POC: Concept/Solution to Automate the version increment to next development iteration. #2215
• Implement Automated workflow that will increment the version for plugin and take care of all other code changes required to pass the CI.
  Implement Automated workflow that will increment the version for plugin and take care of all other code changes #2223
  Pr: Workflow for version increment automation.  #2291
• PR is merged upon review and ready to use.

Additional Acceptance Criteria

Follow Core Branching Strategy (Ensure 1.x and 2.x branches).
opensearch-project/opensearch-plugins#142

Campaign Issues:

OpenSearch

Solution Proposed
Gradle project: Staging to add gradle tasks (setVersion and versionIncrement) that support version increment automation for supported versions and add/update gradle.properties file to the project.

• opensearch-plugins
  Pr's: Add Version increment automation task opensearch-plugin-template-java#32
• alerting
  Pr's:
  Staging for version increment automation alerting#489
• anomaly-detection
  Staging for version increment automation anomaly-detection#624
  Pr's:
  1.x: Staging for version increment automation anomaly-detection#607
  1.3: Staging for version increment automation anomaly-detection#603
  main: Staging for version increment automation anomaly-detection#608
• asynchronous-search
  Pr's: Staging for version increment automation asynchronous-search#157
  1.x: Staging for version increment automation asynchronous-search#161
• common-utils
  Pr's:
  1.3: Staging for version increment automation. common-utils#192
  1.x: Staging for version increment automation common-utils#199
  main: Staging for version increment automation common-utils#200
• dashboards-reports
  Pr's: Staging for version increment automation reporting#391
• index-management
  Pr's: Staging for version increment automation index-management#409
• job-scheduler
  Pr's:
  1.x: Staging for version increment automation.  job-scheduler#203
  1.3: Staging for version increment automation.  job-scheduler#196
  Main: Staging for version increment automation job-scheduler#204
• k-NN
  Pr's:
  1.3: Staging for version increment automation k-NN#432
  1.x: Staging for version increment automation k-NN#436
  main: Staging for version increment automation k-NN#437
  Staging for version increment automation k-NN#442
  Pr's: Staging for version increment automation k-NN#432
• notifications
  Pr's: Staging for version increment automation notifications#476
• performance-analyzer
  Pr's:
  1.x: Staging Version increment automation performance-analyzer#239
  main: Staging for version increment automation performance-analyzer#238
• performance-analyzer-rca
  Pr's:
  main: Staging for version increment automation performance-analyzer-rca#197
  2.1: Version increment automation: update task name performance-analyzer-rca#213
  2.2: Version increment automation: update task name performance-analyzer-rca#214
• security
  Pr's:
  main: Staging for version increment automation security#1932
  1.x: Staging for version increment automation security#1933
• sql
  Pr's: Staging for version increment automation sql#684
• observability
  Pr's: Staging for version increment automation observability#848
• cross-cluster-replication
  Pr's:
  main: Staging for version increment automation  cross-cluster-replication#449
  backport 2.1: [backport 2.1]Staging for version increment automation  cross-cluster-replication#465
  1.3: Staging for version increment automation cross-cluster-replication#466
• ml-commons
  Pr's:
  1.x: Staging for version increment automation ml-commons#363
  main: Staging for version increment automation ml-commons#362
  1.3: [backport 1.3]Staging for version increment automation ml-commons#375
• geospatial
  Pr's: Staging for version increment automation  geospatial#91

OpenSearch Dashboards

Proposal Pending

Remove the dependency with hardcoded zips for 1.x versions:

• index-management
  Example with hardcoded zips: Version Increment to 1.3.3 OpenSearch release index-management#374
• dashboards-reports
  Example with hardcoded zips: Version Increment to 1.3.3 OpenSearch release reporting#366
• sql
  Example with hardcoded zips: Incremented version to 1.3.2. sql#593

Describe alternatives you've considered

No response

Additional context

It takes a day for 1 person to increment the version everywhere, documented in opensearch-project/opensearch-plugins#119. Issues that would reduce that amount of work aside of what's proposed here.

• [BUG] Checked in versions of job-scheduler-*-SNAPSHOT.zip  anomaly-detection#17
• [BUG] Replace checked in reports-scheduler/src/test/resources/job-scheduler/opensearch-job-scheduler-1.2.3.0-*.zip with a dynamic dependency reporting#263
• [BUG] Project integration tests require same version of plugins to be checked in index-management#123
• Replace maven by gradle security#1533
• [BUG] Use snapshots from distribution/maven for dependencies  cross-cluster-replication#274

[peternied]

@dblock what do you think about updating the component release template with these as steps? Then we could create a campaign to automation this step in each repository, which a base case that hits 80% of them with a copypasta workflow

[ohltyler]

@dblock is there any further blocker for other component teams to set this up themselves? Should this be added as a campaign for 2.0.0?

[dblock]

It didn't work for common-utils: https://github.com/opensearch-project/common-utils/actions/runs/2000270385, needs to be debugged, opensearch-project/common-utils#138.

It didn't work for OpenSearch, https://github.com/opensearch-project/OpenSearch/actions/runs/2000274118, the PR GHA needed to be added to permissions (I did that).

It didn't quite work for job-scheduler, opensearch-project/job-scheduler#148, DCO was one problem.

Before adding more we need to fix these, but I didn't get to it. If someone wants to pick these up, go for it!

[VachaShah]

@dblock I added the secrets for common-utils. I will fix DCO for both job-scheduler and common-utils.