Tuesday, December 21, 2010

Thoughts about ServiceMix

I’ve been involved in ServiceMix, the Open Source ESB, nearly since its inception. ServiceMix was started by James Strachan and Rob Davies in May 2005 and the 1.0 version was released in August 2005. I joined LogicBlaze in October 2005 (leaving Mule behind, as I was a committer on Mule) to work on the 2.x releases (2.0 was released in November 2005) and then on the 3.x after the move from Codehaus to the Apache Software Foundation (3.0 was released in May 2007) and the 4.x versions based on OSGi (4.0 was released in March 2009). Even though I’m now focusing more on the OSGi side now (being the VP of Karaf), I’ve done my share of work on ServiceMix (I’m still the first committer in terms of number of commits according to http://svnsearch.org/svnsearch/repos/ASF/search?path=/servicemix) and I’ve been the VP of ServiceMix at the ASF for a few years.

ServiceMix started as a very lightweight implementation of JBI spec. The 2.x version brought much more JBI compliance and the 3.x has seen migration from the lightweight component to fully fledge JBI components and full JBI compliance. In doing so, ServiceMix became a container, as required per the JBI spec and over-time lost a bit of its lightweight-ness. That’s exactly at the same time that Camel was created as a routing library, based on the experience with ServiceMix. After the 3.0 was released it became apparent that the JBI spec was way too limited with respect to the container (classloaders and even the packaging are a real pain in JBI), so we started experimenting with OSGi at that time and that led the path to ServiceMix 4.x and the Karaf project, which started as ServiceMix Kernel.

In 2007, the SCA spec came out, backed by IBM and Oracle, and during a few months, there were a lot of heated debate around JBI versus SCA. It eventually settled a bit when people started understanding that those specs were not really competing, as JBI was targeted around components interoperability while SCA target was application development and could be built on top of JBI. However, JBI was not backed by the big vendors (only really SUN), and when the spec lead for JBI 2.0 left SUN with no replacement, it became clear that the JBI spec was dead.

Another point is that over time, Camel grew to become a top level project and be the very successful project we know, and for some time, we did not really know what to do with the overlap between Camel components and ServiceMix components.

Since JBI 2.0 doesn't appear to be going anywhere we realised we should focus on Camel for the EIP support and connectivity and that OSGi was a better long term standard to represent the registry of endpoints, so we've refactored ServiceMix NMR to be more lightweight, based on the lightweight OSGi container and based around the OSGi registry; with JBI support an optional legacy connector. So we now have a simple lightweight approach to providing a middleware agnostic registry of endpoints with full hot-deploy and supporting powerful class loader versioning schemes via the OSGi registry.

Long live ServiceMix, Camel and OSGi!

Monday, September 27, 2010

Two Karaf related Camel components

While thinking about a centralized logging system for Karaf and FUSE ESB, I had this idea that instead of using a built-in JMS appender, such as the one provided by Log4j, we could instead easily use Camel for that. Camel is really the best fit for such a thing, as we'd be able to add advanced things such as redelivery, batching, compression and choose easily the transport we want (JMS or plain TCP).

The OSGi EventAdmin service is also an important point for monitoring events in the OSGi runtime, as most of the OSGi services do publish events to it (Blueprint bundles events, bundle events, etc...). So this was another need for a camel component.

Given Camel 2.5 will be released soon, I did not want to destabilize trunk just before the release so I've committed them to a github fork for now.

Those two components are really easy to use:

<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0">
<camelcontext xmlns="http://camel.apache.org/schema/blueprint">
<from uri="paxlogging:camel"/>
<to uri="stream:out"/>

In your Karaf installation, add the osgi:camel as a root appender, and you'll see events printed to the console, though not in a nice way, as the goal is really to send them over the wire.

For the event admin component, things are really easy too:

<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0">
<camelcontext xmlns="http://camel.apache.org/schema/blueprint">
<from uri="eventadmin:*"/>
<to uri="stream:out"/>

The name of the consumer endpoint identifies which topic the endpoint will consume from. This component can also be used as a producer to actually publish events to the Event Admin.

I think those components still miss a bit of configuration, but they seem to be a great way to distribute log events and OSGi events to the outside world in a very powerful configurable way.

As usual, feedback welcome. I think those two components can have some real use case, so I'd like them to be put back into the Camel trunk for 2.6.

Wednesday, September 22, 2010

Introducing Cade, the Config ADmin Extender

The OSGi Alliance is working on enhancements to the OSGi ConfigAdmin and some of the experimentation have been unveiled by Peter Kriens in a recent blog. The point is I was in need for enhancing the Apache Karaf blueprint configurations to be more dynamic, for example to restart the SSH server if its configuration had changed.

So I decided to start hacking on this idea and create cade, the Config ADmin Extender. The project is really small and allows you to easily access OSGi configurations in a type-safe manner. It seems to work quite well, so I plan to start using it in Karaf for the various parts that need a bit of dynamism for handling configuration changes.

I haven't done any release yet, but I hope to do that really soon. The source code is Apache Licensed, so feel free to have a look and provide feedback.

Monday, September 06, 2010


Over the last months, I’ve been pondering over the use of OBR, the OSGi Bundle Repository. OBR describes resources (mostly OSGi bundles) in terms of capabilities and requirements. The OBR resolver is able to find the list of bundles needed to fulfill a given bundle requirements. This allows deploying a bundle and its required dependencies without caring too much about the exact dependencies being deployed. In addition, it takes into account the already locally installed bundles and thus reduces unneeded duplication of libraries in different versions.

When I started to use the Felix OBR repository earlier this year, I quickly ran into performance problems during both the parsing phase and the resolution phase. It was nearly unusable for repositories more than a few dozens of resources, which mean in real world scenarii. So I spent some time optimizing things a bit. I also enhanced the API to provide more meaningful informations and to allow more control over the resolution. That work has been released as part of the 1.6.x series of Felix BundleRepository.

One of the enhancement I made was the ability to have a more powerful query mechanism for resources. The goal was to be able to actually use OBR to help people find needed OSGi bundles. This is often one of the main problem people have when building OSGi applications in the integration space. The SpringSource guys have done a good job at providing their Enterprise Bundle Repository and Apache ServiceMix also regularly provide a bunch of OSGi bundles for third party libraries

The main problem with those repositories is that they tend to become huge (more than 7 Mb of xml), which means it takes a long time to download, parse and consumes a lot of memory once it has been parsed. If you use it on multiple JVMs, it becomes quickly a problem and a waste.

That’s when I started to think about the idea of a remote OBR repository. So here we are: I’ve created a small project named RemoteOBR at FuseSource Forge. This project decouples the client API from the real repository manager which is remotely queried using a REST protocol.

It’s not complete yet, but it is already usable, so I invite everyone to go the RemoteOBR website, download, try it and provide feedback.

Thursday, June 17, 2010

Apache Karaf

Yesterday, the Apache Software Foundation established Karaf as a Top Level Project.
It will take some time before the infrastructure resources are correctly set up, but you should see something at http://karaf.apache.org soon.

Wednesday, April 14, 2010

New version of Apache Felix BundleRepository released!

Felix BundleRepository 1.6.0 released!

I'm very happy that the Apache Felix Bundlerepository subproject has been released last week. I've been working on it recently and I think it should even have deserved a 2.0.0 release.

This new version provides a new API, very similar to the org.osgi.service.bundlerepository package but which allows more control over the resolution. Performances for both parsing and resolution have been improved by a factory 10, which in my opinion finally move piece of code into a usable state.

At the same time, the Felix Web Console has been released and provides a refactored page for bundle repository browsing (amongst other things). Have a look !

Next steps on this subproject will include the move to the new Felix Framework resolver to be able to support more OSGi constraints such as uses clauses and thus have results that match the OSGi resolver.

I'd also love to have an up to date and correct view of maven central repository using BundleRepository. Combined with the web console which allows querying bundle repository in a powerful way, I think that would help a lot finding OSGi bundles.

Tuesday, March 16, 2010

Spring-DM, Aries Blueprint and custom namespaces

Since Spring 2.0, Spring supports custom namespace handlers. This feature is really useful and can be used to reduce the verbosity of your spring based applications. Spring-DM has brought this feature along, so it's also available in an OSGi environment. Unfortunately, the integration of those namespace handlers leads to a lot of problems, mostly due to the legacy spring way of configuring the handlers which does not map well at all in OSGi.

In a standard environment, Spring lives in a simple classloader. The classloader which is used to create the spring application is not supposed to change at any time, and everything that is needed for the application is supposed to be available from that classloader. So let's see what happens with custom namespace handlers. Spring detects custom namespace handlers through the presence of a file in the following location META-INF/spring.handlers. This file contains the name of the class supporting a given namespace. In a non OSGi environment, spring detects the namespace handlers by iterating through the resources and finding all these files. However, in an OSGi environment, things must be done a bit differently. The direction which has been taken by Spring-DM is to watch all started bundles and check for this META-INF/spring.handlers file. The custom namespace is then registered and available for use. This leads to multiple problems.

The first one comes from the fact that there is now way to express a dependency onto a "started" bundle. The OSGi specification defines the Require-Bundle header and Import-Package, but both headers do not care about the fact that the bundle is started or not, only that it is resolved. This means that when you deploy your bundles, you need to control the start order of all your bundles to make sure that all custom namespace handlers are started before any application that use them. Failing to do so will result in applications not being started and the solution is to manually restart the bundle. This really goes against the OSGi best practices.

The second problem comes is caused by the non-staged approach in the parsing of the spring xml configuration files. The Spring-DM extender parses the xml and each time a non standard element is found, try to locate the appropriate namespace handler in its internal registry and delegate to that namespace. Once again, this does not work well in OSGi because namespace handlers can come and go and you can't be sure that the namespace are available at the time your bundle start.

The next problem is related to classloading. The namespace handler does not come with its own classloader than can contribute to the application (remember spring has been designed to work with a single static classloader). This means that in order to use a custom namespace, the classloader of the bundle needs to be able to access all the classes that the custom namespace handler will refer to. If you have ever tried to use CXF in OSGi, you know what I mean: you need to import dozens of packages for your application to start to allow the namespace handler to find the classes it needs.

Another limitation is that custom namespace can't be versioned with Spring-DM and it may lead to having to define different namespaces to solve class-related compatibility issues. Let's say you have a namespace foo. This namespace allows to to create an interface named bar (version 1.0). Later, you have a revision of this package . This new version of the interface may or may not be compatible with the previous 1.0 version, but this has nothing to do with the namespace or xml schema itself. With Spring-DM, you need to version your schema because you can't have two namespace handlers supporting the same namespace. This may be problematic because you end up with two solutions. The first one is to always change your namespace (by including the version in the namespace), which means all the bundles using your namespace need to change their xml configuration even it it was compatible. The other solution is to not allow two versions to live in the same OSGi environment. This is simply not possible unless you fully control everything and this goes against the OSGi very goal.

I really don't blame the Spring-DM guys: I think their goal was to be fully backward compatible and that's what they achieved. Sadly, the end result does not behave well in an OSGi environment.

Now, the nice part is that the Apache Aries Blueprint implementation which I've worked on for quite some time now does a much better job at handling those custom namespaces. All four problems have been solved by leveraging OSGi. The implementation leverages the service registry for custom namespace handlers, each bundle containing a custom handler registering its own service. When starting an application, the blueprint extender will make a first pass on the xml configuration files and detect which namespace are used, then put the application in a waiting state until all namespace handlers are available. Each namespace handler can load its own classes thus avoiding the need for the blueprint bundle to import classes that it's not really aware of (because it uses a custom xml namespace and not plain beans). Last, by ensuring class-space consistency, a much cleaner versioning can be done on the various namespaces.

So give it a try if you hit one of the mentioned issues.