Skip to content

Added flush/produce termination callbacks #148

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Dec 16, 2018

Conversation

accelerated
Copy link
Contributor

@accelerated accelerated commented Dec 13, 2018

Added BufferedProducer::FlushTerminationCallback.

Reason:

  1. The current FlushFailureCallback can be called multiple times subject to the number of retries set. However, the application has no way of knowing how many retries are left (the retry count is not passed into the FlushFailureCallback) so unless it keeps track via a map or some other logic of each message retry count it cannot detect when a message will be dropped. This makes message accounting tedious and complex.
  2. Second reason is that the application needs to know when a message delivery fails. In this case, the ProduceFailureCallback cannot be called because we cannot build a Message object w/o a valid rdkafka handle. The application may also need to do some cleanup of the opaque data it has passed. Otherwise this data can be leaked. Also the application can keep stats of how many messages fail or perhaps re-publish them into another kafka queue, log them, etc. And for that the FlushFailureCallback is insufficient.
  3. Same logic applies to the ProduceFailureCallback vs ProduceTerminationCallback pair.

@demin80
Copy link
Contributor

demin80 commented Dec 13, 2018

This would be quite useful for our apps! If a message is dropped by cppkafka, the apps need to be notified.

@accelerated accelerated changed the title Added flush termination callback Added flush/produce termination callbacks Dec 13, 2018
@mfontanini
Copy link
Owner

Sorry I didn't reply, I'll check this (and the other(s) PR(s)) over the weekend.

@accelerated
Copy link
Contributor Author

accelerated commented Dec 14, 2018

Sorry I didn't reply, I'll check this (and the other(s) PR(s)) over the weekend.

No rush, I'm off for a few weeks so I won't be able to fix your comments anyway. But I'll check when I get back in Jan. Some other ppl at work are also waiting for these PRs so they will most likely code around them in the meantime. At least it's good to know if you'll approve them or not so they know what to do....

@mfontanini mfontanini merged commit ccc6738 into mfontanini:master Dec 16, 2018
@accelerated accelerated deleted the flush_failure branch April 26, 2019 18:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants