@BackOff does not recognize the __listener placeholder

[Krejtcha]

In what version(s) of Spring for Apache Kafka are you seeing this issue?

Spring Boot 4.0.0, 4.0.1

Describe the bug

Having annotations like this on a listener method:

@RetryableTopic(
    backOff = @BackOff(delayString = "#{__listener.config.backOffDelay.toMillis()}"),
    attempts = "#{__listener.config.attempts}",
    dltTopicSuffix = "#{__listener.config.dltTopicSuffix}")
@KafkaListener(topics = "#{__listener.config.topics}")

The __listener placeholder is recognized for attempts , dltTopicSuffix and topics, but when used also in @BackOff delayString the following error occurs while starting the application:

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'classified': Expression parsing failed
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:610)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:525)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:333)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:371)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:331)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:196)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.instantiateSingleton(DefaultListableBeanFactory.java:1218)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingleton(DefaultListableBeanFactory.java:1184)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:1121)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:993)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:620)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:756)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:445)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:321)
	at org.springframework.boot.test.context.SpringBootContextLoader.lambda$loadContext$2(SpringBootContextLoader.java:156)
	at org.springframework.util.function.ThrowingSupplier.get(ThrowingSupplier.java:58)
	at org.springframework.util.function.ThrowingSupplier.get(ThrowingSupplier.java:46)
	at org.springframework.boot.SpringApplication.withHook(SpringApplication.java:1465)
	at org.springframework.boot.test.context.SpringBootContextLoader$ContextLoaderHook.run(SpringBootContextLoader.java:605)
	at org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:156)
	at org.springframework.boot.test.context.SpringBootContextLoader.loadContext(SpringBootContextLoader.java:115)
	at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContextInternal(DefaultCacheAwareContextLoaderDelegate.java:247)
	at org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.lambda$loadContext$0(DefaultCacheAwareContextLoaderDelegate.java:167)
	... 21 more
Caused by: org.springframework.beans.factory.BeanExpressionException: Expression parsing failed
	at org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:185)
	at org.springframework.beans.factory.config.EmbeddedValueResolver.resolveStringValue(EmbeddedValueResolver.java:55)
	at org.springframework.kafka.annotation.BackOffFactory.resolve(BackOffFactory.java:117)
	at org.springframework.kafka.annotation.BackOffFactory.resolveDuration(BackOffFactory.java:87)
	at org.springframework.kafka.annotation.BackOffFactory.createFromAnnotation(BackOffFactory.java:60)
	at org.springframework.kafka.annotation.RetryableTopicAnnotationProcessor.processAnnotation(RetryableTopicAnnotationProcessor.java:151)
	at org.springframework.kafka.annotation.RetryTopicConfigurationProvider.findRetryConfigurationFor(RetryTopicConfigurationProvider.java:127)
	at org.springframework.kafka.annotation.KafkaListenerAnnotationBeanPostProcessor.processMainAndRetryListeners(KafkaListenerAnnotationBeanPostProcessor.java:530)
	at org.springframework.kafka.annotation.KafkaListenerAnnotationBeanPostProcessor.processMainAndRetryListeners(KafkaListenerAnnotationBeanPostProcessor.java:517)
	at org.springframework.kafka.annotation.KafkaListenerAnnotationBeanPostProcessor.processKafkaListener(KafkaListenerAnnotationBeanPostProcessor.java:502)
	at org.springframework.kafka.annotation.KafkaListenerAnnotationBeanPostProcessor.postProcessAfterInitialization(KafkaListenerAnnotationBeanPostProcessor.java:405)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsAfterInitialization(AbstractAutowireCapableBeanFactory.java:442)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1820)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:603)
	... 43 more
Caused by: org.springframework.expression.spel.SpelEvaluationException: EL1008E: Property or field '__listener' cannot be found on object of type 'org.springframework.beans.factory.config.BeanExpressionContext' - maybe not public or not valid?
	at org.springframework.expression.spel.ast.PropertyOrFieldReference.readProperty(PropertyOrFieldReference.java:272)
	at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:123)
	at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:111)
	at org.springframework.expression.spel.ast.CompoundExpression.getValueRef(CompoundExpression.java:60)
	at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:96)
	at org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl.java:114)
	at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:267)
	at org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:182)
	... 56 more

To Reproduce

Have a kafka listener e.g. like this:

public class ExampleListener {

    private final Config config;

    public ExampleListener () {
        // only for test purposes, in reality a more complex object is used and provided in the constructor,
        // its content is based on hierarchy of application properties and customizers
        // also there are many of these objects so they cannot be simply a bean
        this.config = new Config();
        this.config.getTopics().add("example-topic");
        this.config.setDltTopicSuffix("-d");
        this.config.setAttempts(4);
        this.config.setBackOffDelay(Duration.ofMillis(100));
    }

    @RetryableTopic(
        backOff = @BackOff(delayString = "#{__listener.config.backOffDelay.toMillis()}"),
        attempts = "#{__listener.config.attempts}",
        dltTopicSuffix = "#{__listener.config.dltTopicSuffix}")
    @KafkaListener(topics = "#{__listener.config.topics})
    public void onMessage(final ConsumerRecord<String, String> consumerRecord) {
        ...
    }

    public Config getConfig() {
        return config;
    }

    public static class Config {

        private final Set<String> topics = new HashSet<>();

        private String dltTopicSuffix;

        private Integer attempts;

        private Duration backOffDelay;

        public Set<String> getTopics() {
            return topics;
        }

        public String getDltTopicSuffix() {
            return dltTopicSuffix;
        }

        public void setDltTopicSuffix(final String dltTopicSuffix) {
            this.dltTopicSuffix = dltTopicSuffix;
        }

        public Integer getAttempts() {
            return attempts;
        }

        public void setAttempts(final Integer attempts) {
            this.attempts = attempts;
        }

        public Duration getBackOffDelay() {
            return backOffDelay;
        }

        public void setBackOffDelay(final Duration backOffDelay) {
            this.backOffDelay = backOffDelay;
        }
    }
}

Expected behavior

The __listener placeholder should be recognized also by the @BackOff annotation, the delayString should be evaluated correctly and the application should start.

Note

This worked in Spring Boot 3 with the old @Backoff annotation from org.springframework.retry.annotation.Backoff. The following configuration before migration to Spring Boot 4 worked:

@RetryableTopic(
    backoff = @Backoff(delayExpression = "#{__listener.config.backOffDelay.toMillis()}"),
    attempts = "#{__listener.config.attempts}",
    dltTopicSuffix = "#{__listener.config.dltTopicSuffix}")
@KafkaListener(topics = "#{__listener.config.topics}")

[artembilan]

I treat this as not a bug, but improvement: we have never claimed that such a place holder would work with @BackOff.

Would be nice to have this feature anyway.
If you see how to fix, please, consider to contribute it.

Thanks

[Krejtcha]

Well, I considered it a bug, because if was working with the org.springframework.retry.annotation.Backoff in Spring Boot 3 and it is working also in other Kafka related annotations: @RetryableTopic, @KafkaListener

Regarding the usage of the __listener - it is mentioned here: https://docs.spring.io/spring-kafka/reference/kafka/receiving-messages/listener-annotation.html#annotation-properties, you are right though that it is not mentioned for @RetryableTopic or @BackOff, at least I did not find it.

[artembilan]

OK. Please, confirm that it works with Spring Boot 3, but does not with Spring Boot 4.
Thanks

[Krejtcha]

Having a listener defined like this in Spring Boot 3.5 works:

public class ExampleListener {

    private final Config config;

    public ExampleListener () {
        // only for test purposes, in reality a more complex object is used and provided in the constructor,
        // its content is based on hierarchy of application properties and customizers
        // also there are many of these objects so they cannot be simply a bean
        this.config = new Config();
        this.config.getTopics().add("example-topic");
        this.config.setDltTopicSuffix("-d");
        this.config.setAttempts(4);
        this.config.setBackOffDelay(Duration.ofMillis(100));
    }

    @RetryableTopic(
        // this is the difference: org.springframework.retry.annotation.Backoff is used and delayExpression instead of delayString
        backoff = @Backoff(delayExpression = "#{__listener.config.backOffDelay.toMillis()}"),
        attempts = "#{__listener.config.attempts}",
        dltTopicSuffix = "#{__listener.config.dltTopicSuffix}")
    @KafkaListener(topics = "#{__listener.config.topics})
    public void onMessage(final ConsumerRecord<String, String> consumerRecord) {
        ...
    }

    public Config getConfig() {
        return config;
    }

    public static class Config {

        private final Set<String> topics = new HashSet<>();

        private String dltTopicSuffix;

        private Integer attempts;

        private Duration backOffDelay;

        public Set<String> getTopics() {
            return topics;
        }

        public String getDltTopicSuffix() {
            return dltTopicSuffix;
        }

        public void setDltTopicSuffix(final String dltTopicSuffix) {
            this.dltTopicSuffix = dltTopicSuffix;
        }

        public Integer getAttempts() {
            return attempts;
        }

        public void setAttempts(final Integer attempts) {
            this.attempts = attempts;
        }

        public Duration getBackOffDelay() {
            return backOffDelay;
        }

        public void setBackOffDelay(final Duration backOffDelay) {
            this.backOffDelay = backOffDelay;
        }
    }
}

Is it enough as a confirmation?

[artembilan]

OK. I see now.
So, in Spring Boot 3.5 (or to be precise in Spring Kafka 3.3), the @BackOff annotation is from already deprecated Spring Retry project: org.springframework.retry.annotation.Backoff.
That one really comes with a delayExpression attribute.
After migrating a retry functionality to Spring Core Retry, that @BackOff annotation was copied into this project in some form: https://github.com/spring-projects/spring-kafka/pull/4059/files#diff-ee885c1e99e86f5e53e5a8035db3f451e1bbf55b75170c7318782ea261579677.
And looks like the functionality to resolve SpEL expression is lost after such a migration.

I think we can use this ticket as an improvement to fulfill those missed parts about SpEL expressions resolution.

Thank you!

[Krejtcha]

Thank you.

Just to clarify, the SpEL evaluation is not broken completely e.g. @BackOff(delayString = "${delay.from.some.application.property}") still works, just the @BackOff(delayString = "#{__listener...}") functionality does not work anymore.

[artembilan]

There is difference. The ${} is called properties placeholder. And `#{}’ is for SpEL.
See more in docs:
https://docs.spring.io/spring-framework/reference/core/beans/environment.html#beans-placeholder-resolution-in-statements
https://docs.spring.io/spring-framework/reference/core/expressions.html