Required Kafka topics for KafkaSQL storage and data import when migration from v2.6.x

[AndreaScarselliAx]

During migration, we discovered that importing data from v2 to v3 fails silently without the registry-events Kafka topic. We could not find any information about the required topics in the documentation.

Observed behavior without registry-events topic:

• Import API returns HTTP 204 (success)
• Artifact metadata is created
• BUT artifact versions are NOT imported (empty)
• Logs show: Failed to import artifact version: Not authorized to access topics: [registry-events]

Questions

1. What are ALL the required Kafka topics for KafkaSQL storage in v3?

The https://www.apicur.io/registry/docs/apicurio-registry/3.0.x/getting-started/assembly-migrating-registry-v2-v3.html doesn't explicitly list the required topics. We had to discover through trial and error that we need:
- kafkasql-journal
- kafkasql-snapshots
- registry-events
We didn't have this topic while using Apicurio Registry v2 with KafkaSQL

Are all 4 topics mandatory? Specifically:
- Is registry-events required for KafkaSQL storage or only for import?
- Are kafkasql-journal and kafkasql-snapshots mandatory or optional?
2. What is the correct configuration for each topic?

We used the following, but want to confirm it's correct:

Topic	Cleanup Policy	Partitions	Purpose (our understanding)
Main schemas topic	compact	1	Stores schema/artifact data
kafkasql-journal	delete	1	Write-ahead log for KafkaSQL
kafkasql-snapshots	compact	1	Periodic state snapshots
registry-events	compact	1	Required for import functionality

3. What ACLs are required for each topic?

We granted Read, Write, Describe, DescribeConfigs, and Create for all topics. Is this correct?

4. Should the import API return an error when required topics are missing?

Currently, it returns HTTP 204 (success) even when registry-events is missing, which is misleading.

Reproduction

1. Deploy Apicurio Registry 3.1.2 with KafkaSQL
2. Create only the main schemas topic (or only 3 of the 4 topics)
3. Import data from v2: POST /apis/registry/v3/admin/import
4. Result: Import reports success but artifacts have no versions

Thank you!

Andrea

[jsenko]

Hello, there is some information in the documentation here https://www.apicur.io/registry/docs/apicurio-registry/3.1.x/getting-started/assembly-registry-high-availability.html#_kafkasql_topic_configuration but we definitely need to improve it.

There are 3 topics used by Registry, kafkasql-journal is the "Main schemas topic". The failure during the import is not directly related to registry-events, it's just one of the topics that Registry currently expects. As of 3.1.4, Registry will perform a topic check on startup and fail if a topic is missing or misconfigured.

Read, Write, Describe and DescribeConfigs should be enough if you create the topics manually, and Registry has to be able to Create topics if the topics do not exist and apicurio.kafkasql.topic.auto-create=true.

We'll let you know more details when we discuss how to improve this.

[jsenko]

Note: The fourth topic you mean is probably the journal topic of Registry v2, it is not needed after the migration. It has the same default name as in v3, but the migration guide mentions this.