Handling Consumer Downtime in NATS JetStream Without Losing Messages

[dvl-numeez]

Designing a NATS JetStream System with Multiple Consumers in Go

I am designing a system using NATS JetStream, where a publisher sends messages to a specific subject. This part is straightforward. However, the challenge lies in handling consumers efficiently.

Requirements for Consumers

1. A consumer should be able to subscribe to a specific subject and consume messages published to it.
2. If a consumer goes down while the publisher is still active, the consumer should be able to fetch the messages published during its downtime.
3. When a consumer comes back online, it should not consume all historical messages of the subject—only the ones that were published while it was down.
4. The system should support multiple consumer processes, each subscribing to different or the same subjects.

Challenges Faced

I initially used nats.Durable() to ensure consumers can fetch messages published during downtime. However, since nats.Durable() requires a unique name, each process needs a unique durable consumer name.

The problem arises when:

• I generate a random durable name per process using a function (e.g., with timestamps or UUIDs).
• If a process stops and restarts, the newly generated durable name will be different, meaning it won’t be able to retrieve the messages published while it was down.
• This results in message loss for that consumer, as it cannot pick up where it left off.

Question

Can I achieve this desired behavior in NATS JetStream while using Go?
If so, what would be the best approach to persist the durable name so that when a process restarts, it resumes with the same durable name and retrieves the messages published during its downtime?

Any insights or suggestions would be greatly appreciated.

Thank you in advance!