[API] Refactor: core cache design and impl

[Xunzhuo]

Pull Request Description

refactor: core cache desgin and impl

---

Contribution Guidelines (Expand for Details)

We appreciate your contribution to aibrix! To ensure a smooth review process and maintain high code quality, please adhere to the following guidelines:

Pull Request Title Format

Your PR title should start with one of these prefixes to indicate the nature of the change:

• [Bug]: Corrections to existing functionality
• [CI]: Changes to build process or CI pipeline
• [Docs]: Updates or additions to documentation
• [API]: Modifications to aibrix's API or interface
• [CLI]: Changes or additions to the Command Line Interface
• [Misc]: For changes not covered above (use sparingly)

Note: For changes spanning multiple categories, use multiple prefixes in order of importance.

Submission Checklist

• PR title includes appropriate prefix(es)
• Changes are clearly explained in the PR description
• New and existing tests pass successfully
• Code adheres to project style and best practices
• Documentation updated to reflect changes (if applicable)
• Thorough testing completed, no regressions introduced

By submitting this PR, you confirm that you've read these guidelines and your changes align with the project's contribution standards.

[varungup90]

@zhangjyr is also working on same implementation.

[Xunzhuo]

@varungup90 is there a PR for it?

[Jeffwan]

@Xunzhuo I think jingyuan is working on heterogeneous routing which includes refactor related logics. Here's his RFC issue #868

[Xunzhuo]

This refactor does not break the previous usage for cache pkg, just improve the code maintainability, which I think is not conflict with the #868.

[Jeffwan]

@Xunzhuo that's even better. I will take a look tomorrow

[Xunzhuo]

CI has passed, plz help review @varungup90 @Jeffwan

[Jeffwan]

@Xunzhuo I reviewed the PR, and the refactor looks straightforward with a few key changes:

1. Extracting a root-level interface: Introduced aCache interface with PodCache, ModelCache, MetricCache and TraceCache interfaces.
2. Refactoring cache.go: Split the lengthy cache.go file into multiple, more manageable files
3. Method renaming: Updated method names for clarity, e.g., GetModelsForPod -> ListModelsByPod etc.

These changes improve maintainability, especially for new contributors. Let me know if there are any specific areas you'd like me to focus on. BTW, Feel free to write key changes in the PR description next time.

Consider it changes a lots of files, I would like to fetch the branch and spend some time to double test the new code structure before merging. I will come back with some results soon

[zhangjyr]

Thanks for the PR. I have reviewed the new interfaces. Decoupling different cache data providers is necessary. However, PodCache and ModelCache should be unified as one interface, as they solely depend on pod information. We should maintain the atomicity of updating pod metadata, model(adapter) metadata, and their relation.

Another thing I need to clarify is that RequestCount series methods store real-time workload statistics, which are distinguished from cached metrics pulled from other data sources. I would suggest either:

1. Using cached metrics as a store and implementing real-time RequestCount updates in extended metrics providers.
2. Wrap RequestCount logic to a new real-time metrics provider.

The cache and routing part will be refactored to improve concurrent access in #884

[Jeffwan]

However, PodCache and ModelCache should be unified as one interface, as they solely depend on pod information. We should maintain the atomicity of updating pod metadata, model(adapter) metadata, and their relation.

@zhangjyr The informer code remains the same, I think it's still pod change driven. PodCache and ModelCache are two logical interfaces that only provides Get/List capabilities. Atomicity should be good.

[Xunzhuo]

@Jeffwan Yep, that's what I originally think.

[Xunzhuo]

@Xunzhuo I reviewed the PR, and the refactor looks straightforward with a few key changes:

1. Extracting a root-level interface: Introduced aCache interface with PodCache, ModelCache, MetricCache and TraceCache interfaces.

2. Refactoring cache.go: Split the lengthy cache.go file into multiple, more manageable files

3. Method renaming: Updated method names for clarity, e.g., GetModelsForPod -> ListModelsByPod etc.

These changes improve maintainability, especially for new contributors. Let me know if there are any specific areas you'd like me to focus on. BTW, Feel free to write key changes in the PR description next time.

Thanks, it's most of the refactor key points.

I will create a clear issue and PR description in later works.

[zhangjyr]

However, PodCache and ModelCache should be unified as one interface, as they solely depend on pod information. We should maintain the atomicity of updating pod metadata, model(adapter) metadata, and their relation.

@zhangjyr The informer code remains the same, I think it's still pod change driven. PodCache and ModelCache are two logical interfaces that only provides Get/List capabilities. Atomicity should be good.

@Jeffwan I see your point. The proposal interfaces only defined the read operations. The data provisioning currently is still intact. I think we can merge the PR after #884 merged.

[Jeffwan]

@zhangjyr I feel this one is straightforward, PR #884 has more feature refactors, let's merge this one first? WDYT?