ActiveMQ Pooling
Over the past weeks, I spend some times load testing ServiceMix and ActiveMQ.
I discovered two things:
* ActiveMQ broker is currently single threaded for a given JMS connection. The main effect is that if you send messages from several threads on the same connection, all threads will be processing sequentially (well, not exactly, but they will all be processed by a single thread on the broker side).
* ActiveMQ Resource Adapter does not pool sessions and producers: this means that sending a single message with Jencks using the JMS best practices in a J2EE environment (create a connection, create a session, create a producer, send, close all) will take three consecutive roundtrips to the JMS broker and is very CPU intensive for the broker (at least, when using JMX).
The first problem is mainly apparent when using persistent messages and there are a few things than can be used to speed up things (see the LogicBlaze tuning guide for ActiveMQ). However these tuning tweaks are not always desirable, and there was an easy solution: as the broker use a single thread for given JMS connection, we just have to use multiple connections. Hence, I enhanced the org.apache.activemq.pool package so that the PooledConnectionFactory can reuse more than one connection.
The second point was more difficult to solve, as I did not had time to rewrite the ActiveMQ Resource Adapter -- and I'm still not sure of the downside of pooling sessions and consumers inside a full J2EE environment (security for example). But my needs were only to be able to have XA support and session pooling at the same time. Once again, the solution was to enhance the existing PooledConnectionFactory to support XA resource enlistment. This was quite easy, as the jencks project already contained some code written by a contributor. Unfortunately, I could not reuse it, as it uses the XA JMS api, and in such a case, ActiveMQ force the use of an XA transaction, and I wanted the same connection to be reused with and without transactions. So I rewrite it.
Last, but not least, I needed this ConnectionPool to be correctly used with the Jencks inbound JCA support. It sounds easy (and I thought it was), but at first, the two JMS session were enlisted in the transaction, but they were not recognized as using the same Resource Manager. It was inefficient, but worse, the transaction did not complete successfully. Back to the drawing board ...
Well, the good news is that it now works !
So jencks now features a new module called jencks-amqpool which defines three different factory beans:
Note that the three connection factories inherit from each other, so that it is completely safe to use the JCA connection factory in all cases.
Code is available at http://fisheye.codehaus.org/browse/jencks/trunk/jencks-amqpool/src/main/java/org/jencks/amqpool/ and will be include in the upcoming 2.0 release of Jencks.
Here are some simple performance tests. The test spawns 10 threads, each sending 100 messages through the same JMS connection factory.
Using a single connection for the factory:
As you can see, the performance boost over the previous pooled connection factory is quite important: 4 times faster ! (unless you use transaction batching).
For the JCA outbound part, the tests are launched with 10 threads, 100 messages per thread.
The pooled connection factory is three times faster and 12 times faster when using 10 connections ! In addition, the tests were using 10% of my CPU with the pooled connection factory, but 100% with JCA managed connection factory.
I discovered two things:
* ActiveMQ broker is currently single threaded for a given JMS connection. The main effect is that if you send messages from several threads on the same connection, all threads will be processing sequentially (well, not exactly, but they will all be processed by a single thread on the broker side).
* ActiveMQ Resource Adapter does not pool sessions and producers: this means that sending a single message with Jencks using the JMS best practices in a J2EE environment (create a connection, create a session, create a producer, send, close all) will take three consecutive roundtrips to the JMS broker and is very CPU intensive for the broker (at least, when using JMX).
The first problem is mainly apparent when using persistent messages and there are a few things than can be used to speed up things (see the LogicBlaze tuning guide for ActiveMQ). However these tuning tweaks are not always desirable, and there was an easy solution: as the broker use a single thread for given JMS connection, we just have to use multiple connections. Hence, I enhanced the org.apache.activemq.pool package so that the PooledConnectionFactory can reuse more than one connection.
The second point was more difficult to solve, as I did not had time to rewrite the ActiveMQ Resource Adapter -- and I'm still not sure of the downside of pooling sessions and consumers inside a full J2EE environment (security for example). But my needs were only to be able to have XA support and session pooling at the same time. Once again, the solution was to enhance the existing PooledConnectionFactory to support XA resource enlistment. This was quite easy, as the jencks project already contained some code written by a contributor. Unfortunately, I could not reuse it, as it uses the XA JMS api, and in such a case, ActiveMQ force the use of an XA transaction, and I wanted the same connection to be reused with and without transactions. So I rewrite it.
Last, but not least, I needed this ConnectionPool to be correctly used with the Jencks inbound JCA support. It sounds easy (and I thought it was), but at first, the two JMS session were enlisted in the transaction, but they were not recognized as using the same Resource Manager. It was inefficient, but worse, the transaction did not complete successfully. Back to the drawing board ...
Well, the good news is that it now works !
So jencks now features a new module called jencks-amqpool which defines three different factory beans:
- PooledConnectionFactory: the standard JMS pooled connection factory
- XaPooledConnectionFactory: this one supports XA transaction auto enlistement
- JcaPooledConnectionFactory: to be used with jencks JCA inbound support
Note that the three connection factories inherit from each other, so that it is completely safe to use the JCA connection factory in all cases.
Code is available at http://fisheye.codehaus.org/browse/jencks/trunk/jencks-amqpool/src/main/java/org/jencks/amqpool/ and will be include in the upcoming 2.0 release of Jencks.
Here are some simple performance tests. The test spawns 10 threads, each sending 100 messages through the same JMS connection factory.
Using a single connection for the factory:
- without transactions: 38 msgs/sec
- with a transaction for each message: 41 msgs/sec
- with a transaction for 10 messages: 283 msgs/sec
- without transactions: 150 msgs/sec
- with a transaction for each message: 173 msgs/sec
- with a transaction for 10 messages: 353 msgs/sec
As you can see, the performance boost over the previous pooled connection factory is quite important: 4 times faster ! (unless you use transaction batching).
For the JCA outbound part, the tests are launched with 10 threads, 100 messages per thread.
- JCA managed connection factory: 11 msgs/sec
- Pooled connection factory: 31 msgs/sec
- Pooled connection factory with 10 connections: 143 msgs/sec
The pooled connection factory is three times faster and 12 times faster when using 10 connections ! In addition, the tests were using 10% of my CPU with the pooled connection factory, but 100% with JCA managed connection factory.
Comments