Restrict downgrade

[matthew1001]

PR description

This PR aims to prevent accidental downgrade of Besu, which can potentially cause the DB to be irrevocably corrupted.

The approach I've used is as follows:

• Add a new file VERSION_METADATA.json in the configured data path
  • Contains a single besuVersion field, e.g. {"besuVersion":"23.10.3"}
• Add a new --version-compatibility-protection configuration option
• Update the gradle prereqs to pull in org.apache.maven:maven-artifact to provide access to the maven ComparableVersion class
• Add a function performVersionCompatibilityChecks() to VersionMetadata as the first function to call after configuration options have been validated.
  • performVersionCompatibilityChecks() throws an exception if the version in VERSION_METADATA.json is higher (when compared using the maven ComparableVersion class) than the version as recorded in the BesuCommand class implementation version

Any value after the first - character in the version number is ignored. This limits version comparison to the 3 calver digits, which is all that can be logically compared as higher or lower. An example of a version number that has trailing characters is 23.10.4-dev-c9338660 where the latest commit has been appended to the version. The -dev-c9338660 is ignored in the version comparison.

The validation logic is as follows:

• If the VERSION_METADATA.json file doesn't exist, no further checks are made and the node starts. The VERSION_METADATA.json file is written to allow version checks from this point onwards.
• If the file and besuVersion field are present and the version is lower than the installed/runtime version, Besu updates it to have the latest version in it and continues to start up
• If the file and besuVersion field are present and the version is greater than the installed/runtime version, Besu fails to start if --version-compatibility-protection is set, because a lower version of Besu is being started than the version that most recently updated the file.

Fixed Issue(s)

Fixes #6266

[github-actions]

• I thought about documentation and added the doc-change-required label to this PR if updates are required.
• I thought about the changelog and included a changelog update if required.
• If my PR includes database changes (e.g. KeyValueSegmentIdentifier) I have thought about compatibility and performed forwards and backwards compatibility tests

[non-fungible-nelson]

LGTM functionality wise --> what you are proposing is a change to the existing default behavior (allowing downgrades). Should we consider the inverse (a flag to prevent downgrades or just switching to --prevent-downgrades)? It's really a coin-toss. I think users on Mainnet would be more likely to downgrade/upgrade more often and may benefit from the default remaining the same. I am open to either, however. This does not have to be a blocker.

[matthew1001]

Thanks for the comments @non-fungible-nelson I'll give that some thought and reply properly tomorrow.

In the meantime I wanted to highlight that I've just updated the description and commits somewhat, as I originally had the checks happening in the RocksDB plugin but have decided that it's better that Besu can perform this check regardless of the storage provider being used. The main difference in the latest commits is the introduction of a new VERSION_METADATA.json file, rather than adding besuVersion as a field to the DATABASE_METADATA.json file.

[matthew1001]

Should we consider the inverse (a flag to prevent downgrades or just switching to --prevent-downgrades)? It's really a coin-toss. I think users on Mainnet would be more likely to downgrade/upgrade more often and may benefit from the default remaining the same.

Yeah I agree that it changes behaviour, but the main cost to a user who is expecting to be able to do this is to set --allow-downgrade and leave it turned on. The downside to someone who accidentally downgrades is potentially much worse, so I think I'd argue that it's a good reason for a behaviour/breaking change.

[matthew1001]

Moving out of draft as I think this is ready for review

[matthew1001]

PR refactored to offer --version-compatibility-protection option which defaults to false, instead of --allow-downgrade that defaulted to true. See related discord discussion https://discord.com/channels/905194001349627914/1196772941619273779/1196773376556015666

[matthew1001]

I've manually re-tested with the new --version-compatibility-protection:

• Without it set, the VERSION_METADATA.json file is populated if it doesn't exist, but no version check is made
• With it set, if the version hasn't changed from what is in VERSION_METADATA.json Besu starts as expected
• With it set, if the version is older than the version in VERSION_METADATA.json Besu fails to start with
  • Besu version 24.1.2 is lower than version 24.1.3 that last started. Remove --version-compatibility-protection option to allow Besu to start at the lower version (warning - this may have unrecoverable effects on the database).

[fab-10]

LGTM, just a couple of edits