Dumping OSGi – SpringSource dm server

There are two basic approaches to developing diagnostics for any enterprise Java application server (be it OSGi based or not). The first approach is to create lots of custom code that treats every subsystem, service and component uniquely different in terms of presentation, collection and modeling forcing the creation of extraneous interfaces and/or additions to existing classes to support state extraction. The second approach involves defining a common model and representation of state data (including meta data) that can be used across multiple diagnostics (state) datasources, implemented by a single provider using runtime object state inspection (without any code changes) and supporting exploration within a single management console in a consistent manner and accessible whilst being disconnected.

Unfortunately nearly all application server vendors select the first approach including those newcomers that claim to be anti-bloat ware.

Naturally we opted for the second option building a kind of dynamic state based configuration management database (CMDB) snapshot facility with supporting disconnected offline analysis tooling.

Amazingly some vendors require you to start an actual server instance to view the contents of a dump in a readable format.

With JXInsight all that is required to capture diagnostics state is to simply drop a technology specific observation extension into the class path of an application server. Here I have used our jxinsight-ext-aj-osgi-bundle-observe.jar to collect actual state diagnostics for live OSGi bundles.

Note: A snapshot can contain both dynamic & static, public & private, state. With static state typically representing configuration loaded from a persistent store such as a file system.

The depth to which object (graph) state is collected within a diagnostic snapshot is configurable but typically 6-8 is sufficient for most offline analysis purposes.

By combining both state and metadata into a single diagnostic snapshot we can render, highlight and annotate various items within the model – all in a consistent manner with as much detail as one would required in whatever role we assume during the problem analysis.

Note: We record the time(stamp) the object was constructed or first appeared to the diagnostics runtime as well as the system identity hash code.

If we want to enhance the collection capabilities of the diagnostics system we simply drop in another extension library. Here I have added jxinsight-ext-aj-osgi-service-observe.jar

We can even record much more dynamic state including that which is being referenced directly or indirectly during request processing. Here I have added jxinsight-ext-aj-osgi-service-diagnostics.jar to the classpath to collect at any moment in time within the management console the object graph state related to arguments and target objects currently executing methods defined in the one or more of the OSGi service related interfaces. The method being called is getServiceReferences by thread server-dm-1 with a single string argument on a target object of type BundContext.

One Billion Operations Per Second

The problem with (published) performance benchmarks is that they are there to be beaten (luckily only ever by ourselves) and today well we have shattered all our previous benchmark results, including one last week, which means I am now faced with the daunting task of going over all my previous related postings and redoing the testing or just simply linking to the latest and greatest numbers. I think I will go with second option for now.

JXInsight’s multi-resource metering & monitoring (& profiling) technology is truly the fastest and most scalable Java solution on the market that is designed specifically for production usage in the most demanding of environments. Getting to this level of software quality is never easy. Its been a long and hard slog but Rod and I (co-founders) have managed to consistently out class the competition in terms of open & extensible API design, efficient runtime & tooling, and in the delivery of innovative information visualizations for the vast amount of data that the product can collect today.

Visualizing Sequential & Hierarchical Execution Aspects of Metered Software

Last week I managed to get back working on cool and innovative (hopefully you will agree) visualization work that I had designed (lots of sketches scattered around my office) a long time back but never managed to get the push to actually realize them within the product until today when we released a first in a series of visualization updates I am actively working on that will raise the bar (sky high) once again (well that is the motivational chant used currently).

In this first update I have tried to tackle a common problem with most application performance profiling/tracing/logging/journaling tools that use ordered child nodes within trees to display sequential execution behavior. The problem being that they do not scale in the enterprise context (too many nested branches and large sequences of nodes), are wasteful of precious screen real-estate, and cause repetitive strain injury (RSI) with excessive clicking to expand & contract nodes (seriously).

The solution is to not use a tree. Instead we use an ordered table that overlays the hierarchical execution nature (caller-to-callee) on top of a flattened sequential execution flow and makes it contextual (based on the current selection). And best of all we can display the execution of multiple concurrent threads side-by-side (to be exact its row by row) which is impossible with trees as the basis for any such visualization. Here is an animated gif showing this in action.

By the way the graphical tree overlay also highlights re-entrant calls on the billed probe stack. Can you see the visual pattern indicator?

Jazoon ‘09: The Conference

This was my first time to attend (and speak) at the Jazoon conference and though I was only able to attend for 2 days I really did have a great time attending the conference and talking in person (and drinking) with a number of well known individuals in the Java industry including Bela “a blast” Ban (JBoss), Mike “I’m nice” Keith (Oracle), and Kirk Pepperdine (Kodewerk) who dared discuss and debate application performance management & monitoring approaches with me over lunches and walks to and from our hotel.

Kirk just about to experience my Irish resolve.

I did attend the keynote session given by Adrian Colyer (SpringSource) and I must say I was impressed with his presentation skills even though it was a tad bit too long in terms of the green theme and light on in-depth technical analysis but I suppose there is only so much you can do in such a short time especially when predicting the future which could all change with an acquisition.

Adrian was as ever economical with the facts in his assessment of which VM language will win the war when he claimed that his choice of Groovy was made prior to the acquisition of the company behind Groovy & Grails – hence it was a pure technical decision. He failed to state that the VC that provided financial support to SpringSource at the very outset had also invested in the acquired company and had been promoting the technology to Rod and Co. for some time. Whether it had any bearing on the final decision is a different matter but it should have been disclosed in his argument to the audience questioning his objectivity.

The same cannot be said for the other main session I attended in which Peter “let me put this chastity belt on your classloaders before everything drops out under” Kriens (OSGi) who did a very poor job of presenting a supposedly technical session which in fact turned out to be more like “let me preach & market my 10 year old modular framework (that keeps re-inventing new component models)”. Peter really used every cheap marketing trick in the book when it came to the information (can we even call it [data] that, more like the gospel according to Peter) he presented in his slides. Below is an actual slide from the presentation.

Heaven awaits those who embrace OSGi?

The session might have been somewhat informative if he stopped talking about his 10 years of servitude, left out the gimmicky slides, and limited his bashing of Java and Sun at every opportunity during the talk. It felt like 10 years had passed when the session was over which should not have been the case for this particular technology which will have served an important purpose when Java does indeed deliver a built-in and much improved replacement. His insistence that modularity should be managed entirely outside of the language and oblivious to the Java runtime sounded like someone completely divorced from reality (vital lies?).

When Juergen Hoeller (SpringSource) was pulled over to our table I did question him on whether Rod (SpringSource) should continue blogging or go back to coding factory classes. What is worse? His reply was “No comment” but his body language betrayed him.

During Peters talk he mentioned that we was instructed by the OSGi committee to not talk about the original target environment for the technology – the embedded market.

Not all OSGi members share Peter’s view that the default behavior for OSGi should be to break existing enterprise Java applications which he refers to as “no pain, no gain”. There appears to be two camps. The hard-liners (and groupies/followers whose only response to issues raised is to name call those who dare mention them) and those (mainly app server vendors) who just want to get things to work sometimes only using the technology to segregate container libraries from application deployments.

Peter has responded (partially) and apparently “new” is (only) good when it is in relation to “OSGi (component) programming models” but not when it is in relation to an approach to modularity that involves enhancements to the language and runtime which are outside of Peter’s control (hence his rejection). I cannot believe that Peter & Co. would have designed OSGi as it is today if they had the opportunity to extend the language, runtime and tooling to accommodate their design. I refuse to believe that they did not compromise their design (and resulting implementations) especially in light of this and the original target environment – the embedded space. At the very least they would have looked at ways of reducing they own class loader hacks. I also doubt that Sun would have allowed any proposed design break compatibility in a number of ways that OSGi has done so (and some for no good reason). Even if Sun fails to delivery a deployment centric modularity approach we still might gain from their efforts as a smaller (modular) VM could gives us the ultimate class isolation container – an operating system process (and many of them hosting specific services connected to a grid/cloud). Lets not be fooled in to believing that to achieve modularity we just put a class chastity belt around our private parts. In the enterprise space state is just as important as behavior and it is not so easy to tackle in or outside of a container which is why most systems go occasionally offline (at least internally) to sync changes across different data processing domains (messaging, databases, …). Even if you can isolate classes from each other you cannot do the same for the external side effects (of multiple & different versions). This last item seems foreign to many OSGi disciples who apparently do not understand one of the main reasons why operations stop and restart processes in applying a managed change – to be sure the process can indeed be restarted when something goes terribly amiss in production outside of the container (hardware). It is much easier rolling back at the time the change is applied.

The “chastity belt” reference reflects my view that modularity, which should be managed at many levels in the design and development process and not just in bundling, has more to do with education and support during the early application life-cycle than a cruel troublesome enforcement at the late stage of post-deployment (into the bad real world). We need modularity to be supported in different ways (not just a single realization) and at different stages with each building on top and augmenting that which precedes its. I think currently there is way to much emphasis placed on bundle separation (thinking being that the more modules the greater the modularity). We should look at packaging (bundling) of application artifacts to indicate deployment collocation for reasons other than a “modularity nirvana”. We should also encourage diversity in approaches to how modularity is enforced at runtime (if it need be or not) and not just standardize on a particular implementation. The language and tooling needs to help capture and convey intent not target implementation. Modularity is not achieved by just creating many little bundles and hiding classes which seems to be Peter’s fixation (crucifix). I have seen very modular software designs that never needed runtime policing of visibility (outside of what is provided by the language) maybe they were not so dynamic (at least within a single runtime and in terms of service implementation) but how many organizations really need such capabilities and have the ability and tooling to effectively manage and control such environments in an enterprise context. Very few and even if they had the capability within the organization most try to avoid such scenarios (multiple live bundle/library versions in a single container) because the risks (diagnostics is problematic) far outweigh the benefits (“its cool”).

Peter did not actually address my main criticism points of his presentation. The cheap tricks in how he presented the information to back up his OSGi marketing campaign like the dependency graph of the Java class library with no comparable graph of a similar runtime and SDK built over the same period (Java 1.x-1.6) and offering the same breath and depth of services using OSGi. How much different would it indeed look like? Would any noticeable differences reflect the choice of the component approach or just the talent and longevity of the people involved. Great products are created by great teams. Yes there is a need for a framework (of reference and values) along with a support system but does it need to be policed in the way proposed which in my opinion reflects the constraints on the original design space (or the authors cynical view of the rest of us)? Does it need to be marketed as “the” silver bullet for the software industry that ultimately fixes us to a future designed 10 years ago? I hope not. We need to learn from previous work along with the mistakes made and 10 years gives you plenty of time to do this no matter how hard you try not to as time and the pace of innovation is not on your side.