Today’s web applications have execution paths with huge stack depths  - a trend that is unlikely to be reversed anytime soon. New web application frameworks deliver increased developer productivity at a cost that is to some degree incurred during performance testing/tuning and production problem resolution in the increased complexity involved in the analysis of the software execution being monitored.

Profiling tools based on thread call stack sampling are now pretty much useless due to their excessive overhead and the amount of noise within any derived performance model. Even with standard package filtering options available in most code profilers (call stack sampling or event/execution based) one ends up losing far too much execution context because it is largely an all or nothing choice in terms of each package.

Here is a partial call path for an inbound HTTP request collected from an instance of Apache ESME running in a Tomcat server that has already had standard filtering of web container packages applied to it. More than 30 deep and we have not even seen sight of actual ESME code.

Note: You need to navigate much much deeper  (> 60) to begin to understand what is the underlying domain action(s) performed in the processing of this request.

To address this problem we recently released an update to JXInsight that delivers a highly innovative approach to managing such complexity without losing all of a request’s execution path context as processing moves down through the various web application layers - smart dynamic runtime filtering.

This is achieved by enabling our tierpoint probes provider and defining (via a few system properties) logical tiers mapped to one or more probe groups (typically a package or class though methods are supported). The provider then limits resource metering to execution points at the boundaries of such tiers filtering out all internal method dispatches within a tier.

Here is the same HTTP request as above but this time with our tierpoint provider enabled and tiers defined for Lift, Apache ESME, Apache Derby and Scala. From L.doFilter to E.find we have excluded 60 call frames.

Note: This is performed within the managed runtime which means we can drastically reduce the amount of metering performed whilst still assigning inherent resource metering accordingly across tiers (packages).

Note: The probe names for each element of a track are displayed in shortened form to conserve precious (blog/desktop) real-estate.

I was recently contacted by Solomon Duskis to see whether I would be interested in assisting him with some performance tuning work he was involved with on the JBoss RESTEasy open source project. He was in particular concerned that his findings based on data reported by other profiling tools was inaccurate due to their excessive overhead especially when the entry point operations being tested had sub-millisecond execution times.

Looking at the initial tests results of a prolonged benchmark run, which exhibited relatively (but to be expected) high GC activity and extremely short high frequency method executions, I thought this would be an interesting test for some new features I had been working on in recent software updates to JXInsight specifically adjusted clock time metering and self-optimizing hotspot detection.

Note: Once you start profiling & tuning at the microsecond level you can effectively rule out using any thread cpu based timers as their resolution is very low and their cost of access relatively high.

The first thing I did to understand the software execution behavior of the benchmark was to execute a run with our complete coverage dynamic probes extension, weaving (at load-time) resource metering into all methods within the org.jboss.resteasy.* and org.codehaus.jackson.*. For this run I disabled the default clock.time meter and instead installed our probe.count meter so that I could count the number of metered method executions made directly and indirectly by each package, class and method.

The Count column below tells me the number of probes fired per package – in total approximately 604 million. The Total column tells me how many probes, including those in other packages, were fired and metered during the course of executing the probes listed in the Count column with the last column, Total (I), telling me how many of those probes are actually inherent (directly attributed) to the package.

Note: The Total (I) column can be less than the Count column because probes do not include their own firing within their reported resource metering which also explains why the Total (I) column can be greater than the Count column as in the case or org.codehaus.jackson.* which calls back into org.jboss.resteasy.*. This might be a little bit confusing because we have effectively mapped a resource meter to the actual occurrence (count) of the resource metering but to me that is the beauty of our approach – we can meter anything including the act of metering itself.

The metering data above helped me in very quickly understanding the degree of impact a standard code profiler would have before I even ran such a test. Knowing that most other profilers have an overhead of between 1 and 5 microseconds per method execution I can estimate the total benchmark time from a baseline of 30 seconds. In the best case we are talking about 10 1/2 minutes (634 seconds) though from my own testing the best code profiler took more than 16 minutes (980 seconds). In the worst case more than a hour. With such numbers you are only ever likely to find low hanging fruit along with a large number of red herrings especially when one factors in the amount of GC activity generated.

The above does not really present much of a problem to JXInsight because its metering of any method’s executions is strategy based (In fact we support multiple chained metering strategies). With the default hotspot metering strategy installed the runtime will dynamically disable the instrumentation after a number (default is 1000) of method executions if at that time a specified metering statistic does not exceed a particular threshold. But when a hotspot metering threshold is defined as low as 5 microseconds as in the case of my own benchmarking even the initial number of warmup metered method executions before the hotspot determination is made can have some degree of impact on the actual test run timing and the resulting hotspot determination (especially callers which can be incorrectly labeled hotspots).

Previously the solution was to generate and deploy a new hotspot instrumentation extension after and before each repeat of the benchmark test (see Instrument->Measure->Refine->Repeat). In JXInsight 5.7.41 we offered up another option that can be enabled to defer the generation of extensions until much later in the test cycle by automatically persisting (and reloading) the hotspot metering analysis across consecutive benchmark test runs allowing the runtime to immediately upon initialization disable non-hotspots before their first firing whilst refining the previous hotspot candidate list. This is all done without any intervention by the performance engineer. The performance engineer just keeps re-running the benchmark until the test times and number of hotspots remains constant across runs. The hotspot metering runtime continually builds on-top of what it previously determined – self optimizing.

Here is the output of 13 benchmark test runs I executed in a matter of a few minutes.

Note: For the 9th test run I changed the hotspot metering configuration (whilst still reusing the previous hotspot analysis) to use the inherent min statistic to get rid pass thru elements within a call chain.

Note: For the 13th test run I used a hotspot instrumentation extension generated from within the console based on the metering model collected from the 12th test run removing completely all instrumentation from non-hotspot methods and disabling the metering strategy probes provider extension.

Note: The benchmark duration is measured in seconds and the metering threshold in microseconds.

Update: In our latest internal build, 5.7.42, which includes the ability to enable continuous hotspot performance analysis the benchmark time has now dropped down under 33 seconds.

Update: JXInsight 5.7.42 has been released with continuous hotspot metering analysis following an initial hotspot classification allowing for reclassification and subsequent disablement of associated instrumentation in the event of changes in resource usage.

Near to the end of the testing we were still metering 10 millions calls and incurring only a 5 second slow down compared with a competitors best time of 950 seconds. Here is the breakdown of the resource metering by the main top packages for 12th test run prior to the generation of the hotspot instrumentation extension.

Note: The clock.adj.time meter was used instead of the clock.time meter to discount gc activity in the hotspot metering analysis.

Looking at the inherent clock time averages for hotspot candidates (a partial list is show below) we can see that there was still room for further reduction (in overhead) if we increased the hotspot metering warmup count or instead this was continuously re-assessed during the course of an individual test execution (coming soon).

In a follow-up to this entry I am going to compare the hotspot results from a number of tools showing how surprisingly different they can be beyond the low hanging fruit cases and how widely inaccurate they are no matter what profiling technique is used – execution/event based or call stack sampling. Free Advice != Good Advice.

Whilst I have long argued that execution based profilers (metering agents & activity monitors) can out-perform sample based profilers on many levels (when implemented right) I have always recognized that statistical sampling techniques do occasionally have their place in ones arsenal in tackling the complexity of a real-world application/system performance model collected from a production environment under particular runtime constraints. That is why when we introduced our pioneering strategy based resource metering approach I made sure to include a number of out-of-the-box strategies that have sampling characteristics. This list initially included frequency, interval, sample, and burst. All of which have the following metering behavior – a non-metered gap and a metered run.

The two main differences in the metering strategies relate to the unit of measurement for both gap and run (time or frequency), and the scope of the strategy (global or probe name based).

Note: JXInsight’s Probes technology supports complex chaining of metering strategies allowing you to derive your very own metering strategy from base (built-in) strategies.

Each of the above mentioned metering strategies have their strengths and weaknesses depending on the application (or problem) context but none are truly comparable to how a standard call stack sampling profiler works so in a recent update to our product I approached this problem from a different angle within our runtime model by way of a new meter named sample.count which is basically a low-resolution clock (counter) with a user configurable time interval (sample) for increments (ticks). In doing so I addressed one of the key performance indicators (KPI) of a typical execution based profiler – the number of relatively costly (mostly native) counter access calls (which would otherwise occur in close time proximity in a heavily load system).

So how does this match much closer with the behavior of a call stack sampler? Well first of all the number of samples (meter increments) and the cost of the underlying native system calls is approximately the same (it is also a known quantity). Second of all only fired probes (metered activities) which have a metered execution period spanning at least one sample window increment (tick) will have their metering totals updated which means that the accuracy for high frequency and low cost methods is similar. Were it differs from sampling is in the increased accuracy of the performance model by including frequency (invocation) counts as well as collecting the min, avg and max for sample.count meter readings. In addition we can still utilize our default hotspot metering strategy by simply changing the strategy’s meter and the threshold value used.

Note: Thread call stack samplings in ability to track the complete duration (or ticks) of a method’s invocation restricts its application solely to development environments.

Naturally I was curious to see how a sample.count metering model compared with one based on the clock.time meter. So I metered the startup execution of a JBoss 4.x application server instance with a sample JBoss Seam application deployed. For this initial run I disabled our default hotspot metering strategy. The startup took less than 31 seconds with 776,468 method invocations metered within the org.jboss.* package which. This translates to 1.5 million clock time access calls – 50k / sec.

Note: “Count” is the number of method invocations metered and “Total” is the sum of the ticks during the execution period of the metered method invocation.

Here is the sample.count metering model for a similar startup sequence. The breakdown of the metering “Total %” column looks very similar to that of the clock.time model. The startup time took less than 30 seconds which translates (approximately) to 30,000 clock ticks with a default tick interval of 1ms – 1k / sec.

Of course one would expect the previous models to be very similar. A real comparison test would be with the hotspot metering strategy enabled which attempts to eliminate the metering of low cost instrumented method invocations (reducing the cumulative count). Here is the clock.time metering model for the startup. The number of metered probes fired has now halved though this reduction would be much higher in a long running system as we by default meter the first 1,000 firings per named probe.

With the hotspot metering strategy configured to use the sample.count meter the breakdown of both the “Count” and “Total” columns looks again similar to the clock.time metering model.

Though I still prefer to use our sample based metering strategies (more so because of familiarity) which guarantee accurate metering of those probes sampled the sample.count meter approach allows me to create a filtering like mechanism to the model at the granularity that suits the context of the investigation. With the meter I get the frequency for every probe fired which the strategy approach does not afford me but I only aggregate the metering total for those probes fired and metered that exceed a meter threshold I have defined.

To be in anyway successful at managing & monitoring real-world Java enterprise applications in production (that excludes Spring “PetClinic”) one needs to assess the runtime impact of the various performance data collection techniques and their actual (tool) implementations. Unfortunately most programmers fail to understand this and instead develop/promote/deploy tools that are clearly completely inadequate for the task at hand – in a production context. They incorrectly assume ease of use equates to lightweight when in fact the only thing lightweight is the effort and due diligence performed in understanding the problem domain and managing the associated risks. It is for this reason why most operations teams immediately discount (without any evaluation) the use of monitoring tools a developer might have used in any way during localized tuning.

Recently this was made so evident when its was falsely claimed (without any qualification other than wishful thinking) that a Java sample based profiling tool could out-perform JXInsight – a dynamic strategy based resource metering solution. I will save the publication of a benchmark comparison for a (near) future blog but instead show you a chart (taken from within the “visual” tool itself) depicting the performance impact of such a technique on an application with a realistic number of threads (100) running with call stacks of realistic sizes (100-200).

Note: The similarities of both the 100 ms and 1,000 ms run sections indicates the tool was not able to perform at the lower of the sampling rates.

Even at a 10 second sample interval rate the runtime impact is excessive ignoring the fact that at this resolution the data would be completely meaningless unless the application was a dog in terms of performance which begs the question “How did such a beast get into production in the first place?”.

Yes the overhead of sampling is zero when not being performed but what type of enterprise application is not monitored continuously in production? Instrumentation needs to be a primary source of resource metering & software execution metric performance data. Sampling should be considered (and only after discounting many other sources and techniques) when the current instrumentation coverage is not adequate at the time a problem appears though I am doubtful of its effectiveness with the noise that would be created when enabled based on the charting above.

It is notoriously hard to detect and distinguish busy and hung threads in todays complex enterprise middleware even more so when the middleware is distributed across multiple processing nodes with temporal execution dependency (locking, co-ordination). A large part of the reason for this is the lack of current execution information associated with frames within thread stack dumps. For one we cannot easily determine from two thread dumps listing the same call stack for a thread whether the thread in question is hung waiting on a response from a resource  or whether the thread is performing the same operation again following receipt of another request into a processing queue. We might not even be able to detect the case were the thread is merely busy making calls from the stack listed in the dump to other fast executing methods not listed.

To help with the problem analysis we have recently enhanced our optional stack probes provider to collect additional information such as the number of times a uniquely identifiable stack appears across two mark requests from within our management console and the number of calls (or frames) made during the execution interval of a caller frame.

Here is a Java class I quickly wrote to demonstrate the problem and the solution. I have created two Runnable classes that mimic the behavior of hung and busy threads.

Here is a sample of a partial stack dump during the execution of the above class. As you can see it is pretty hard to actually determine the type of problem that may be present ignoring the obvious clue in the class naming.

Below is a resource metering snapshot taken during a similar execution run with our dynamic probes instrumentation enabled. What we can quickly see from this view is that the run() call executed by Thread-0 has only made one additional direct (or indirect) call during its ongoing execution interval. Whereas the run() call executed by Thread-1 has been busy performing more than 1 million calls (stack push operations).

Now we cannot be sure that Thread-0 is effectively hung because it might have only just began executing prior to the snapshot capture. Likewise we cannot be sure of our initial prognosis for Thread-1 as it might just be finishing up or it has been active for a prolonged period in the runtime waking up occasionally to service queued requests.

So lets make a mark (reset) from within our management console that instructs the remote managed runtime to mark all current resource metering related data including associated stacks. Then we wait a little while (a few seconds) and then capture another snapshot shown below. From the Marks table column we can see that some parts of the stacks present in this new snapshot capture were present prior to our recent mark remote operation – marks > 0. The only new stack frame is the Busy.doit() call which has a thread execution sequence id of 3031907.

In addition we can see from the Count Delta column that only the run() call executed by Thread-1 has performed additional (instrumented) calls during its still unfinished execution.

To be sure that the stacks visible from within the management console have indeed an execution period of longer than a few seconds we can make another mark. Now any element of a stack listed in a snapshot following the last mark operation with a Marks value greater than 1 has at least an execution duration of the time difference between both mark operations.

For those elements with a Marks value of 1 we can only say with confidence that the stack (or partial stack) has existed since the period of the last mark to the point at which a snapshot is captured which can be done repeatedly without marking.

Now if that is not exciting enough we can easily perform the same steps across a complete cluster of nodes from a single view within our management console and this works for all probe related extensions whether the probes represent actual Java methods or contextually mapped Ruby and Python calls.

Following on from our “The Fastest Ruby Profiler is a Java Profiler” I am pleased to announce that our upcoming probes based resource metering extension for Jython is the fastest function call profiling solution for Python applications executed within a Java runtime. With our dynamic hotspot resource metering strategy the overhead of our probes instrumentation can drop down to approximately 8 nanoseconds for non-hotspot function calls.

Here is the Python micro-benchmark I have used in determining this.

Here is a chart showing the average time (secs) reported by multiple executions of the run method without (baseline) and with our probes based instrumentation. JXInsight’s instrumentation overhead is just over 2% but that is 2% on a method, call, that effectively does nothing.

The overhead incurred in the instrumentation and measurement of fine grain method executions, which dominate the execution (frequency) profile of most web applications, is a key performance indicator (KPI) for any enterprise application performance management and monitoring solution.

Below is a screenshot of the resulting resource metering in our application management console. Note the strike-through on the call:12 function indicating the associated instrumentation has been disabled following the execution of 1000 probe firings and the determination by the hotspot metering strategy that the named probe, benchmark.call.12, is not a hotspot.

Note: Line numbers can be excluded from the probe name assigned to a function marking all same named functions within a module as hot or cold spots depending on the combined resource metering behavior for all implementations.

The following chart shows the performance of the same benchmark executed by the native c-based Python engine along with the standard cProfile. The overhead is significant.

Here is a chart comparing the overhead (%) of JXInsight with that of the Python cProfile.

I attempted to run the benchmark with the Python based profile solution but a single run() call took over 200 seconds compared to 2-4 seconds for all other reported test runs. Surprisingly the Jython version of the Python profile solution executed in just under 120 seconds.

If you are interested in seeing more of what will be possible with the upcoming extension then check out the following images showing contextual labeling as well as multi-language metering (profiling).

www.jinspired.com/images/jxinsight.5.7.32.jython.to.python.p.1.gif
www.jinspired.com/images/jxinsight.5.7.32.jython.to.python.p.2.gif

One of the many cool and innovative features we have recently pushed out the door with our regular software updates is an enhancement to tagging that really makes this unique feature extremely useful and versatile in the analysis of software under construction during development & test or under management in production. Now when you set a tag at either the global or thread context level not only do we aggregate resource metering for the tagged metering group but we do this for every other metering group updated during the tagged execution period. We do this by creating an extended @tag metering group that concatenates the name of the metered group updated with the value of tag.

Being able to inject tags remotely from within our management console or command line interface or locally from within the actual managed runtime via our Open API gives us the ability answer two very broad questions:

1. What is the resource usage for a package/class/method during a particular tagged execution period?

2. How does the resource usage for a particular probe compare across tagged and non-tagged execution periods in terms of count, total, min, max and average?

Previously the way to answer such questions at a global level across various profiling and application performance management (APM) tools would be to reset all data collected prior to the execution period, specify a period filter on an event journal table or log, or in our case mark the current totals and then perform delta analysis following the completion of the execution metered. But this leads to the creation of multiple disconnected performance models that cannot be easily compared and reconciled. In addition it is not possible to narrow the analysis to one or more threads or have concurrent tagged execution periods with each thread having its own tag value which is required when the tag values reflect the some aspect of the execution context such as the web application user name, request parameter or session attribute.

Here are some concrete kind use cases where this capability might be much more appropriate during analysis across multiple application life cycle management phases.

1. During the testing of a web application interaction consisting of a series of steps and pages a tag based on the step or page name is set globally allowing the developer to break down the resource metering costs for a particular package/class/method by the tagged step (or page).

2. During unit testing a tag is set at the thread context level based on the name of each @Test method executed providing code coverage like analysis for direct and indirect executed code by test method. This data can also be used in generating dependency analysis reports for such tests.

3. During a system failure (i.e. database bounce) a global tag is injected identifying the failure period allowing service level management to later break-out such costs on a per activity/interaction basis.

4. In diagnosing a user specific problem a thread context level tag is set identifying the user or the particular interaction at the entry point of request processing enabling resource metering to be reported both from an operational perspective (code) and contextual (user marker) perspective side by side.

5. In delivering elastic computing capabilities to an application a tag is injected into a cluster of runtimes at each dynamic change (a capacity profile) in the computing capacity made available to the application in the cloud which can then later be analyzed offline to determine reasons for the increase in capacity not being fully utilized due to possible inherent scalability problems with the applications own code base. Likewise one can easily determine which particular activities are impacted more than others by decreases in computing capacity due to elasticity policies.

Recently we announced that JXInsight can execute 1 billion instrumented Java method calls per second once the multi-resource (hotspot) strategy based metering runtime determines a particular method is a non-hotspot. So I was curious to see how our JRuby-to-Ruby cross language profiling extension faired with such optimizations as our probes technology is not in anyway tied to metering just named Java code execution constructs. The good news is that our technology can drop the overhead so low that it even out-performs no-op instrumentation calls in Ruby itself which means we are faster than any Ruby language based profiling or benchmarking solution (naturally when executed on the Java platform – JRuby VM).

Here is the Ruby micro-benchmark I used to determine the above.

Here is a chart showing the average time reported by multiple executions of the run method without and with our probes based instrumentation. Our instrumentation overhead is just over 15% but that is 15% on a method, call, that does nothing. Effectively our overhead is just under 15 nanoseconds for non-hotspots.

The overhead incurred in the instrumentation and measurement of fine grain method executions, which dominate the execution profile of most web applications, is a key performance indicator (KPI) for any enterprise performance management and monitoring solution.

Here is a partial screenshot of the resulting resource metering in our application management console. Note the strike-through on the call method indicating the associated instrumentation has been disabled following the execution of 1000 probe firings and the determination by the hotspot metering strategy that the named probe, Object.call, is a non-hotspot.

To determine the best (if not impossible) time a Ruby based profiling solution could do I added both a before and after method call to the call method which is typical of how most dynamic instrumentation agents work across various languages and runtimes. Both methods do nothing.

Here is a revised chart with the average execution time for the above manual instrumentation added. JXInsight is 3 times faster than a Ruby solution that does nothing.

Note: Disabling our hotspot metering strategy and comparing the performance with a version having a callout to Benchmark.measure() JXInsight is more than 15 times more efficient. Moving to JRuby has many advantages for production applications both in terms of the raw speed of the runtime itself and the supporting management and monitoring tools.

Hardware
Model Name: MacBook Pro
Model Identifier: MacBookPro5,1
Processor Name Intel Core 2 Duo
Processor Speed: 2.53 GHz
Number Of Processors: 1
Total Number Of Cores: 2
L2 Cache: 6 MB
Memory: 4 GB
Bus Speed: 1.07 GHz

JRuby
jruby 1.3.1 (ruby 1.8.6p287) (2009-06-15 2fd6c3d)

Runtime
java version "1.6.0_15"
Java(TM) SE Runtime Environment (build 1.6.0_15-b03-223)
Java HotSpot(TM) 64-Bit Server VM (build 14.1-b02-92, mixed mode)


There are two basic approaches to developing diagnostics for any enterprise Java application server (be it OSGi based or not). The first approach is to create lots of custom code that treats every subsystem, service and component uniquely different in terms of presentation, collection and modeling forcing the creation of extraneous interfaces and/or additions to existing classes to support state extraction. The second approach involves defining a common model and representation of state data (including meta data) that can be used across multiple diagnostics (state) datasources, implemented by a single provider using runtime object state inspection (without any code changes) and supporting exploration within a single management console in a consistent manner and accessible whilst being disconnected.

Unfortunately nearly all application server vendors select the first approach including those newcomers that claim to be anti-bloat ware.

Naturally we opted for the second option building a kind of dynamic state based configuration management database (CMDB) snapshot facility with supporting disconnected offline analysis tooling.

Amazingly some vendors require you to start an actual server instance to view the contents of a dump in a readable format.

With JXInsight all that is required to capture diagnostics state is to simply drop a technology specific observation extension into the class path of an application server. Here I have used our jxinsight-ext-aj-osgi-bundle-observe.jar to collect actual state diagnostics for live OSGi bundles.

Note: A snapshot can contain both dynamic & static, public & private, state. With static state typically representing configuration loaded from a persistent store such as a file system.

The depth to which object (graph) state is collected within a diagnostic snapshot is configurable but typically 6-8 is sufficient for most offline analysis purposes.

By combining both state and metadata into a single diagnostic snapshot we can render, highlight and annotate various items within the model – all in a consistent manner with as much detail as one would required in whatever role we assume during the problem analysis.

Note: We record the time(stamp) the object was constructed or first appeared to the diagnostics runtime as well as the system identity hash code.

If we want to enhance the collection capabilities of the diagnostics system we simply drop in another extension library. Here I have added jxinsight-ext-aj-osgi-service-observe.jar

We can even record much more dynamic state including that which is being referenced directly or indirectly during request processing. Here I have added jxinsight-ext-aj-osgi-service-diagnostics.jar to the classpath to collect at any moment in time within the management console the object graph state related to arguments and target objects currently executing methods defined in the one or more of the OSGi service related interfaces. The method being called is getServiceReferences by thread server-dm-1 with a single string argument on a target object of type BundContext.

The problem with (published) performance benchmarks is that they are there to be beaten (luckily only ever by ourselves) and today well we have shattered all our previous benchmark results, including one last week, which means I am now faced with the daunting task of going over all my previous related postings and redoing the testing or just simply linking to the latest and greatest numbers. I think I will go with second option for now.

JXInsight’s multi-resource metering & monitoring (& profiling) technology is truly the fastest and most scalable Java solution on the market that is designed specifically for production usage in the most demanding of environments. Getting to this level of software quality is never easy. Its been a long and hard slog but Rod and I (co-founders) have managed to consistently out class the competition in terms of open & extensible API design, efficient runtime & tooling, and in the delivery of innovative information visualizations for the vast amount of data that the product can collect today.

Last week I managed to get back working on cool and innovative (hopefully you will agree) visualization work that I had designed (lots of sketches scattered around my office) a long time back but never managed to get the push to actually realize them within the product until today when we released a first in a series of visualization updates I am actively working on that will raise the bar (sky high) once again (well that is the motivational chant used currently).

In this first update I have tried to tackle a common problem with most application performance profiling/tracing/logging/journaling tools that use ordered child nodes within trees to display sequential execution behavior. The problem being that they do not scale in the enterprise context (too many nested branches and large sequences of nodes), are wasteful of precious screen real-estate, and cause repetitive strain injury (RSI) with excessive clicking to expand & contract nodes (seriously).

The solution is to not use a tree. Instead we use an ordered table that overlays the hierarchical execution nature (caller-to-callee) on top of a flattened sequential execution flow and makes it contextual (based on the current selection). And best of all we can display the execution of multiple concurrent threads side-by-side (to be exact its row by row) which is impossible with trees as the basis for any such visualization. Here is an animated gif showing this in action.

By the way the graphical tree overlay also highlights re-entrant calls on the billed probe stack. Can you see the visual pattern indicator?

This was my first time to attend (and speak) at the Jazoon conference and though I was only able to attend for 2 days I really did have a great time attending the conference and talking in person (and drinking) with a number of well known individuals in the Java industry including Bela “a blast” Ban (JBoss), Mike “I’m nice” Keith (Oracle), and Kirk Pepperdine (Kodewerk) who dared discuss and debate application performance management & monitoring approaches with me over lunches and walks to and from our hotel.

Kirk just about to experience my Irish resolve.

I did attend the keynote session given by Adrian Colyer (SpringSource) and I must say I was impressed with his presentation skills even though it was a tad bit too long in terms of the green theme and light on in-depth technical analysis but I suppose there is only so much you can do in such a short time especially when predicting the future which could all change with an acquisition.

Adrian was as ever economical with the facts in his assessment of which VM language will win the war when he claimed that his choice of Groovy was made prior to the acquisition of the company behind Groovy & Grails – hence it was a pure technical decision. He failed to state that the VC that provided financial support to SpringSource at the very outset had also invested in the acquired company and had been promoting the technology to Rod and Co. for some time. Whether it had any bearing on the final decision is a different matter but it should have been disclosed in his argument to the audience questioning his objectivity.

The same cannot be said for the other main session I attended in which Peter “let me put this chastity belt on your classloaders before everything drops out under” Kriens (OSGi) who did a very poor job of presenting a supposedly technical session which in fact turned out to be more like “let me preach & market my 10 year old modular framework (that keeps re-inventing new component models)”. Peter really used every cheap marketing trick in the book when it came to the information (can we even call it [data] that, more like the gospel according to Peter) he presented in his slides. Below is an actual slide from the presentation.

Heaven awaits those who embrace OSGi?

The session might have been somewhat informative if he stopped talking about his 10 years of servitude, left out the gimmicky slides, and limited his bashing of Java and Sun at every opportunity during the talk. It felt like 10 years had passed when the session was over which should not have been the case for this particular technology which will have served an important purpose when Java does indeed deliver a built-in and much improved replacement. His insistence that modularity should be managed entirely outside of the language and oblivious to the Java runtime sounded like someone completely divorced from reality (vital lies?).

When Juergen Hoeller (SpringSource) was pulled over to our table I did question him on whether Rod (SpringSource) should continue blogging or go back to coding factory classes. What is worse? His reply was “No comment” but his body language betrayed him.

During Peters talk he mentioned that we was instructed by the OSGi committee to not talk about the original target environment for the technology – the embedded market.

Not all OSGi members share Peter’s view that the default behavior for OSGi should be to break existing enterprise Java applications which he refers to as “no pain, no gain”. There appears to be two camps. The hard-liners (and groupies/followers whose only response to issues raised is to name call those who dare mention them) and those (mainly app server vendors) who just want to get things to work sometimes only using the technology to segregate container libraries from application deployments.

Peter has responded (partially) and apparently “new” is (only) good when it is in relation to “OSGi (component) programming models” but not when it is in relation to an approach to modularity that involves enhancements to the language and runtime which are outside of Peter’s control (hence his rejection). I cannot believe that Peter & Co. would have designed OSGi as it is today if they had the opportunity to extend the language, runtime and tooling to accommodate their design. I refuse to believe that they did not compromise their design (and resulting implementations) especially in light of this and the original target environment – the embedded space. At the very least they would have looked at ways of reducing they own class loader hacks. I also doubt that Sun would have allowed any proposed design break compatibility in a number of ways that OSGi has done so (and some for no good reason). Even if Sun fails to delivery a deployment centric modularity approach we still might gain from their efforts as a smaller (modular) VM could gives us the ultimate class isolation container – an operating system process (and many of them hosting specific services connected to a grid/cloud). Lets not be fooled in to believing that to achieve modularity we just put a class chastity belt around our private parts. In the enterprise space state is just as important as behavior and it is not so easy to tackle in or outside of a container which is why most systems go occasionally offline (at least internally) to sync changes across different data processing domains (messaging, databases, …). Even if you can isolate classes from each other you cannot do the same for the external side effects (of multiple & different versions). This last item seems foreign to many OSGi disciples who apparently do not understand one of the main reasons why operations stop and restart processes in applying a managed change – to be sure the process can indeed be restarted when something goes terribly amiss in production outside of the container (hardware). It is much easier rolling back at the time the change is applied.

The “chastity belt” reference reflects my view that modularity, which should be managed at many levels in the design and development process and not just in bundling, has more to do with education and support during the early application life-cycle than a cruel troublesome enforcement at the late stage of post-deployment (into the bad real world). We need modularity to be supported in different ways (not just a single realization) and at different stages with each building on top and augmenting that which precedes its. I think currently there is way to much emphasis placed on bundle separation (thinking being that the more modules the greater the modularity). We should look at packaging (bundling) of application artifacts to indicate deployment collocation for reasons other than a “modularity nirvana”. We should also encourage diversity in approaches to how modularity is enforced at runtime (if it need be or not) and not just standardize on a particular implementation. The language and tooling needs to help capture and convey intent not target implementation. Modularity is not achieved by just creating many little bundles and hiding classes which seems to be Peter’s fixation (crucifix). I have seen very modular software designs that never needed runtime policing of visibility (outside of what is provided by the language) maybe they were not so dynamic (at least within a single runtime and in terms of service implementation) but how many organizations really need such capabilities and have the ability and tooling to effectively manage and control such environments in an enterprise context. Very few and even if they had the capability within the organization most try to avoid such scenarios (multiple live bundle/library versions in a single container) because the risks (diagnostics is problematic) far outweigh the benefits (“its cool”).

Peter did not actually address my main criticism points of his presentation. The cheap tricks in how he presented the information to back up his OSGi marketing campaign like the dependency graph of the Java class library with no comparable graph of a similar runtime and SDK built over the same period (Java 1.x-1.6) and offering the same breath and depth of services using OSGi. How much different would it indeed look like? Would any noticeable differences reflect the choice of the component approach or just the talent and longevity of the people involved. Great products are created by great teams. Yes there is a need for a framework (of reference and values) along with a support system but does it need to be policed in the way proposed which in my opinion reflects the constraints on the original design space (or the authors cynical view of the rest of us)? Does it need to be marketed as “the” silver bullet for the software industry that ultimately fixes us to a future designed 10 years ago? I hope not. We need to learn from previous work along with the mistakes made and 10 years gives you plenty of time to do this no matter how hard you try not to as time and the pace of innovation is not on your side.

Mike over at Canoo has a write-up of the talk I gave today at the Jazoon 09 conference on applying activity based costing in the cloud. I understand some of his concerns but I hope he will see that the model itself has application in many areas not all cost related (at least not directly) including performance and capacity management.

I am delighted he is excited (“it’s certainly an incredibly powerful idea”) about the possibility of our OpenCore Probes API becoming a standard which we can all expect to be available in the runtime and which can be extended by cloud or runtime vendors via the registration of their own custom meters.

I would like to add that metering does not necessarily imply that every activity has to be costed. Metering is a powerful and scalable way of analyzing the resource usage of a software without suffering from the obvious scalability and performance problems of call or path trace tools. Activity analysis can be the sole purpose of the data collected and the model generated. During the demo I showed how you can even create a meter based on the frequency or resource usage of another metered activity. This is what so cool about our design and implementation – you can model the execution of a block code as an activity or meter or both.

Activity based metering (ABM) is an incredibly important source of information on behavioral patterns of users which can in fact increase development budgets in areas that have higher profit margins or are used much more frequently (and hopefully appreciated) by clients. More and more of our daily activities are mirrored in the execution of software in todays web and business applications. Threads are our delegates (agents) acting on our behalf.

It is also extremely important when you are a Web 2.0 startup with a growing user base and no clue of what business model can be applied that will eventually offset operational costs that will likely to continue to grow in the future.

Next time I intend ending the session with a clip from the Matrix showing our life streams (threads/calls) flowing down the panel with a little William (or Smith) assuming control of one or more threads temporarily.

In a previous blog entry, Instrument->Measure->Refine->Repeat , I discussed how to approach the performance analysis of software with request processing execution times in the low microseconds via an iterative hotspot detection process that seeks to refine the instrumentation and measurement with each cycle, reducing overhead and minimizing perturbation in the data collected. In this follow-up I would like to present an alternative and complimentary approach that allows us to largely discount any instrumentation and measurement overhead by focusing on the rate of change in key performance indicators (KPI) that are not time based.

But first allow me to ask the question “Why do we profile the execution of software?“. For many developers the answer is simple – “to tune”. But this is not entirely the only valid reason and in fact this answer is more of a by-product of other activities and tasks within software performance engineering (SPE).

For large complex systems we observe and measure the execution of software in order to understand, model, quantify, and manage its behavior (resources usage & utilization, performance, scalability, runtime system dependencies) across multiple software releases and under different operating & executing contexts. Naturally we need a way to effectively manage all this information. This is achieved by metering (tracking) a relatively small number of indicators that measure operational activities that have a significant impact on performance, correlating the rate of change in such indicators with higher-level activities and various operating parameters.

Note: In fact the ability to detect and assess underlying changes in both the system & software execution models across deployments is of paramount importance in delivering service level management and reducing the time window for problem resolution in production environments.

The choice of a good key performance indicator depends largely on the type of software being monitored and managed. For standard business applications this can be the number and timing for requests serviced by various resource managers (file system, database, messaging middleware, data grid, compute grid). For low latency request processing systems there are a number of useful candidates for key performance indicators that are not entirely time dependent or impacted by instrumentation including: method execution counts, field read/write counts, object allocation counts & sizes (and to some lesser extent cpu time).

Lets apply this approach to the performance analysis of Oracle Coherence used in my previous blog entry which you should have already read at this stage. Before looking at any execution profile data you first need to know what was the activity being tested and how was it parameterized. Here it is:

>cache cache
>#10 bulkput
>#10 bulkput

Executing the above command sequence in the Oracle Coherence interactive shell results in a total of 20,000 SafeNamedCache.put() calls. We can see this below in the metering table with the Count column representing the number of times this particular method was called.

Lets keep this number, 20,000, in mind when looking at all numbers presented below.

The Total column represents the number of methods executed within the com.tangosol.* package during the execution of the SafeNamedCache.put() 20,000 calls by the same calling thread. The Total(I) tells us the number of methods called that were directly (inherently) attributed to the execution of the  SafeNamedCache.put()

So 20,000 SafeNamedCache.put() calls resulted in 13.7 million method calls, 40,000 of which were made directly from within the method itself. But what about the other 38 million (51 less 13) call? Why only 13.7 million? The reason is that these other 38 million calls were made outside of the execution of the cache operation and more importantly by other threads. Expansion of the metering model tree reveals one of the main contributing classes, PacketPublisher, whose method executions resulted in a total of more than 22 million (1.3 million x 17) method calls, 4 million of which were made directly from within the class.

Note: For extremely low latency systems the overhead of excessive method calls can be relatively high even if in-lined to some degree because there still needs to be some housekeeping performed by the runtime (call stack mgmt). When looking for possible candidates to tune start with those probes with high averages.

In analyzing the performance of our own software product we use two low level meters – field.read.count and field.write.count. Below is such an analysis with 37.6 million reads performed on fields residing in the com.tangosol.* package during the execution of 52.1 million method calls also within the com.tangosol.* package.

Note: We ourselves monitor call paths within our probes runtime at the nanosecond level by averaging billions of test executions and by managing the number of field reads and writes.

Comparing the Count column with the Total(I) columns shows that a vast number of calls within the PacketPublisher class are simple field getters – an average of 1 field read per method call. It also reveals that during the execution of methods within this class 18.5 millions fields are read with 1 million within the class’s own methods and the rest in methods executed elsewhere in the com.tangosol.* package.

Another useful meter in analyzing software with a large amount of thread co-ordination is waiting.count which tracks the number of Object.wait() calls made by a thread.

Note: The waiting.count meter can be used in “profiling nothing” as a high number of waits calls is generally indicative of worker threads starved of work items to process.

For low latency systems you really need to look at the rate of object allocation and the resulting byte sizes. This is not only important for performance management but for capacity management as well as most OutOfMemoryErrors are the result of high concurrent request processing placing too much demand on the available memory pool and not some permanent memory leakage (more like a temporal & transient waste).

Note: Execution count profiling allows one to identify and quantify the impact of particular request execution parameters have on key performance indicators – correlating rate of change across multiple variables.

Focusing on the average object allocation count per method execution can sometimes reveal opportunities for tuning which by the way does not necessarily imply that objects should be cached – always try to eliminate work completely before improving the efficiency of the work via reuse.

From the metering table below we can see how many objects are allocated directly from within a method and how many are typically allocated during its execution including from within methods called directly and indirectly.

Note: Object allocation metering can also reveal the cost of initialization – methods only ever executed once.

Key performance indicators can be used to drive change (tune) but they can also be used to effectively manage and track change not necessarily tied to the achievement of an aligned target. They can also be used to identify unplanned changes that would otherwise result in customers notifying you of visible side effects at a much higher granularity and largely disconnected from the problem domain.

In a follow-up to this entry I will show how easy it is to create your own custom key performance indicators with our aspect library extensions.

Kirk Pepperdine recently queried whether JXInsight could indeed profile nothing. Naturally JXInsight can profile the “nothing” in all its many forms (resource contention, coupling & co-ordination) but to keep the first part of this follow-up short & sweet let me show you how to detect latency in browser based operations from a software execution model perspective. In this scenario a single button click on a web page to cancel a hotel booking that results in the user returning to a page displaying the list of current bookings remaining.

And here it is – our timeline analysis of the two resulting HTTP requests processed by the same worker thread (this need not always be the case) within an application server.

The light blue rectangles represent the time the thread was processing within the servlet layer. The yellow and brown the time the thread was processing within the JNDI and EJB layers respectively. The bracket above the brown rectangle in the first HTTP request represents the actual active resource (JDBC) transaction window.

Finally the whitespace (the “nothing”) in the middle of the timeline graphic is the time spent outside of the container sending/receiving/processing the request at the client – 15-20% of the total user experience time.

Next up profiling the “nothing” from a system execution model perspective.