Giskard v3 Roadmap

[mattbit]

This is the roadmap for Giskard v3. Not familiar with v3 yet? Read the announcement.

This is a living document — priorities may shift based on feedback.

Components overview

Status	Component	Notes
✅ Ready	giskard-core	In use
✅ Ready	relais	In use
✅ Ready	giskard-agents	In use
🔜 Release candidate	giskard-checks	Converging!
🔜 Release candidate	giskard-scan	Integrated
🔜 Release candidate	RAG evals via new "Quality Scan"
🔜 Release candidate	Hub SDK integration
🚧 WIP	Documentation & migration guide	In progress

If something is missing or wrongly prioritized, say so below or open a thread in the discussions. We read everything!

[BILLKISHORE]

The table here has drifted from what's on main. Reading the code today:

• giskard-scan is Beta in the README, not "in progress"
• part of the RAG toolkit already landed inside it (scan/generators/knowledge_base/, scan/utils/knowledge_base.py). docs(readme): correct v2/v3 feature mapping — RAG now in giskard-scan #2610 mentions the move but this table doesn't
• giskard-checks is Beta in the README, Alpha here

Worth a refresh, since the README points people here.

On the RAG toolkit, I compared main against RAGET at v2.19.2 to see what's actually left to build. Where v3 is now: KnowledgeBase builds from a flat list[str] and does embeddings plus nearest-neighbour lookup, five generators (direct, sycophancy, out-of-scope, multi-topic, split-questions), and components tagged component:llm, component:retrieval, component:history.

Things v2 had that I can't find on main:

Topic clustering. v2's KnowledgeBase ran HDBSCAN over reduced embeddings and named each topic with an LLM. The KNOWLEDGE_BASE component score was max minus min correctness across topics, so without topics that score can't be computed at all. It also means no per-topic breakdown in the report.

Five components vs three. QUESTION_ATTRIBUTION in question_types.py covered GENERATOR, RETRIEVER, REWRITER, ROUTING, KNOWLEDGE_BASE. Worth noting "out of scope" attributed to ROUTING and KNOWLEDGE_BASE in v2, where the v3 out-of-scope generator tags llm and retrieval. Different vocabulary, so possibly deliberate, but routing has no v3 equivalent that I can see.

Retrieval metrics. v2 wrapped RAGAS for context precision, context recall, faithfulness and answer relevancy. Nothing equivalent on main, and no recall@k, MRR or nDCG anywhere in libs/.

Ingestion. v2 had KnowledgeBase.from_pandas. v3 is from_texts only.

Three v2 question types also have no v3 counterpart: complex/paraphrased, distracting element, and situational.

The v3 announcement lists "information retrieval benchmarking" as a new RAG capability. Is that still planned, and does it land inside giskard-scan or as the separate giskard-rag package the README mentions?

I'd like to take a piece of this. Retrieval metrics and knowledge-base topic clustering both look self-contained enough not to collide with anything currently open. Happy to write a short design up front if that's easier to review. Which would help most?

(Unrelated to the roadmap, but while reading the repo I hit a CI issue and opened #2686 for it: make check-format is failing on a clean main because ruff is installed unpinned and 0.16 turned on markdown formatting. #2687 is a small tools fix on top.)