Improve performance of DefaultSingletonBeanRegistry.getSingletonNames(..) for high-concurrent-load applications [SPR-7949]

[spring-projects-issues]

mck opened SPR-7949 and commented

Under high-concurrent load situation, in an application using many prototype beans, contention occurs in DefaultSingletonBeanRegistry.getSingletonNames(..)

This method has uses singletonObjects as a mutex for all map operations.
Using a ReadWriteLock this contention could be removed, as in our case 99% of the calls here are read-only.

We see hundreds of thread dumps like in a single jvm:

java.lang.Thread.State: BLOCKED (on object monitor)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:180)

• waiting to lock <0x00007fb8519b11b0> (a java.util.concurrent.ConcurrentHashMap)
  at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:452)
  at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:317)
  at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:296)
  at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:250)
  at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1083)

java.lang.Thread.State: BLOCKED (on object monitor)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingletonNames(DefaultSingletonBeanRegistry.java:277)

• waiting to lock <0x00007fb856ab64d8> (a java.util.concurrent.ConcurrentHashMap)
  at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:352)
  at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:185)
  at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:829)
  at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:744)
  at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:703)
  at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.resolvedCachedArgument(AutowiredAnnotationBeanPostProcessor.java:436)
  at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.access$0(AutowiredAnnotationBeanPostProcessor.java:432)
  at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:469)
  at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:84)
  at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:283)
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1074)
  at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireBean(AbstractAutowireCapableBeanFactory.java:295)

java.lang.Thread.State: BLOCKED (on object monitor)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:180)

• waiting to lock <0x00007fb8519b11b0> (a java.util.concurrent.ConcurrentHashMap)
  at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:452)
  at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:317)
  at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:296)
  at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:250)
  at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1083)

---

Affects: 3.0.4, 3.0.5

Attachments:

• SPR-7949.patch (15.19 kB)

Issue Links:

• Cache by-type lookups in DefaultListableBeanFactory [SPR-6870] #11536 Cache by-type lookups in DefaultListableBeanFactory ("duplicates")
• Threads blocking on getSingleton method on DefaultSingletonBeanRegistry class [SPR-9190] #13828 Threads blocking on getSingleton method on DefaultSingletonBeanRegistry class ("is duplicated by")

17 votes, 16 watchers

[spring-projects-issues]

mck commented

(why can't i edit the description? the first and last thread dumps are identical... (how frustrating...))

[spring-projects-issues]

mck commented

Simple patch that changes the mutex to a ReadWriteLock.

Subclasses in org.springframework.beans have been rewritten to lock() instead of synchronized.

This does change the (internal) API of DefaultSingletonBeanRegistry.getSingletonMutex() so i can understand if this is better suggested for the next major release...

[spring-projects-issues]

mck commented

This patch (applied to 3.0.5.RELEASE) has been running for 3 weeks in production ( http:/finn.no ) and has successfully fixed spring's poor performance around prototype beans for us.

[spring-projects-issues]

mck commented

related to #11536

[spring-projects-issues]

ggn commented

We are also facing the same issue where numerous threads are blocked on the synchronized block in getSingleton method.
We are using spring 3.0.6 on Tomcat 6.0.35, JDK 1.6
I downloaded spring 3.0.7 and 3.1 hoping to see some difference in that code, but i could not find any difference. Please help how i can get around with this issue. This issue is causing our server to hang once in a while.

Please help.
Thanks
ggn

[spring-projects-issues]

ggn commented

We are also facing the same issue where numerous threads are blocked on the synchronized block in getSingleton method.
We are using spring 3.0.6 on Tomcat 6.0.35, JDK 1.6
I downloaded spring 3.0.7 and 3.1 hoping to see some difference in that code, but i could not find any difference. Please help how i can get around with this issue. This issue is causing our server to hang once in a while.

Please help.
Thanks
ggn

[spring-projects-issues]

Vladimir Rozhkov commented

Have same problems in our environment.
Our application heavily use prototype beans - minimum 15 threads, total 40000 calls to getBean("..") and it becomes to very huge performance gap.

Any chance to get this fix in 3.2M2? Also, could we raise priority to Normal?

Thanks.

[spring-projects-issues]

Vladimir Rozhkov commented

Also, why we trying to get singleton instance first, even if bean has prototype scope?

protected <T> T doGetBean(
			final String name, final Class<T> requiredType, final Object[] args, boolean typeCheckOnly)
			throws BeansException {

		final String beanName = transformedBeanName(name);
		Object bean;

		// Eagerly check singleton cache for manually registered singletons.
		Object sharedInstance = getSingleton(beanName);
		if (sharedInstance != null && args == null) {

Is there reasons for it? I guess we should consider bean scope first to prevent execution of unnecessary singleton searching.

[spring-projects-issues]

Vladimir Rozhkov commented

Attached patch doesn't actually fixes the problem. We still have tons of threads locked on writeLock insinde getSingleton.

Also, after moving bean definition resolving to the top of the method, we still have lock in

org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.registerDependentBean(DefaultSingletonBeanRegistry.java:367)
        - waiting to lock <0x00002aaac9754d48> (a java.util.concurrent.ConcurrentHashMap)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:323)
        at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106)
        at org.springframework.beans.factory.support.ConstructorResolver.resolvePreparedArguments(ConstructorResolver.java:757)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:125)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:993)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:885)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:485)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
        at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1075)

So Spring cannot handle accurately requests to getting prototype beans in 15 concurrent threads with 4000 invocations of getBean in each. This is huge gap.

[spring-projects-issues]

Chris Beams commented

This issue duplicates #11536, which is now resolved. Please give it a try (available now in 3.2.0.BUILD-SNAPSHOT).