[META] System Indices access for Extensions

[peternied]

OpenSearch plugins utilize system indexes to store information that persists through the cluster, such as the security configuration for the Security Plugin or .replication-metadata-store indexes for cross cluster replication. These indexes contain data specific to the plugin and should not be altered by any other plugin or application.

This meta issue is tracking what associated work needs to be done

Design

• [Question] How to Support System Indexes for Extensions? #2530
• [Discuss] When should plugins use system indices? OpenSearch#6589
• Pick a solution

Implementation

• [Feature] Update SecurityIndexAccessEvaluator to support new index permission #2553
• Add configuration flag to disable system index permissions #2845
• [FEATURE] Allow super-admin to control which system indices are not permission-able #3038

Questions

• [Question] Should system index permissions check allow access to system indices with .* pattern? #3259

Release

• [DOC] Add documentation for permission-able system indices documentation-website#4736

Future Investment

• Update extensions migration documentation #2554

[peternied]

Option 2b: Extension service account, scoped to specific system index [Recommended] is best aligned based on the current considerations and discussion topic. I'll add associated work to this meta issue.

[peternied]

@opensearch-project/security I've added work items for this META issue, please review the tasks on this item and if we need any more detail.

Small note; I did not create an issue for the documentation because I think we should create the task on the documentation-website by whoever will own that work when its ready.

[cwperks]

I wanted to leave an additional comment here. I'm starting not to love the term system indices in this context. IMO System Indices are indices critical for OpenSearch to run. These indices deserve special protection and are critical for an extension to run, but should we keep utilizing the term system index?

I've started calling these reserved indices as the extension would like to reserve these indices for it to store data its needs to function properly. Any thoughts on the naming?

Its important to note that many plugins utilize system index protection provided by the security plugin without explicitly being a system index by implementing SystemIndexPlugin.

[samuelcostae]

I agree with your point @cwperks , though I lack end user viewpoint, what made me agree with you is that from a code structure point of view, the issue became much clearer to me once i considered these "Extension Indicices" as a different category from the "protected/system Indices" that already exist in the SecurityIndexAccessEvaluator class

[peternied]

'System index' is the name of the feature in OpenSearch so we seem pretty locked in. I'm open to alternative names, when I think of 'extension index' / 'protected index' or something, it would be implemented differently without the relationship to SystemIndexPlugin which is a requirement for backward compatibility with the existing plugins.