Whilst I have long argued that execution based profilers (metering agents & activity monitors) can out-perform sample based profilers on many levels (when implemented right) I have always recognized that statistical sampling techniques do occasionally have their place in ones arsenal in tackling the complexity of a real-world application/system performance model collected from a production environment under particular runtime constraints. That is why when we introduced our pioneering strategy based resource metering approach I made sure to include a number of out-of-the-box strategies that have sampling characteristics. This list initially included frequency, interval, sample, and burst. All of which have the following metering behavior – a non-metered gap and a metered run.

The two main differences in the metering strategies relate to the unit of measurement for both gap and run (time or frequency), and the scope of the strategy (global or probe name based).

Note: JXInsight’s Probes technology supports complex chaining of metering strategies allowing you to derive your very own metering strategy from base (built-in) strategies.

Each of the above mentioned metering strategies have their strengths and weaknesses depending on the application (or problem) context but none are truly comparable to how a standard call stack sampling profiler works so in a recent update to our product I approached this problem from a different angle within our runtime model by way of a new meter named sample.count which is basically a low-resolution clock (counter) with a user configurable time interval (sample) for increments (ticks). In doing so I addressed one of the key performance indicators (KPI) of a typical execution based profiler – the number of relatively costly (mostly native) counter access calls (which would otherwise occur in close time proximity in a heavily load system).

So how does this match much closer with the behavior of a call stack sampler? Well first of all the number of samples (meter increments) and the cost of the underlying native system calls is approximately the same (it is also a known quantity). Second of all only fired probes (metered activities) which have a metered execution period spanning at least one sample window increment (tick) will have their metering totals updated which means that the accuracy for high frequency and low cost methods is similar. Were it differs from sampling is in the increased accuracy of the performance model by including frequency (invocation) counts as well as collecting the min, avg and max for sample.count meter readings. In addition we can still utilize our default hotspot metering strategy by simply changing the strategy’s meter and the threshold value used.

Note: Thread call stack samplings in ability to track the complete duration (or ticks) of a method’s invocation restricts its application solely to development environments.

Naturally I was curious to see how a sample.count metering model compared with one based on the clock.time meter. So I metered the startup execution of a JBoss 4.x application server instance with a sample JBoss Seam application deployed. For this initial run I disabled our default hotspot metering strategy. The startup took less than 31 seconds with 776,468 method invocations metered within the org.jboss.* package which. This translates to 1.5 million clock time access calls – 50k / sec.

Note: “Count” is the number of method invocations metered and “Total” is the sum of the ticks during the execution period of the metered method invocation.

Here is the sample.count metering model for a similar startup sequence. The breakdown of the metering “Total %” column looks very similar to that of the clock.time model. The startup time took less than 30 seconds which translates (approximately) to 30,000 clock ticks with a default tick interval of 1ms – 1k / sec.

Of course one would expect the previous models to be very similar. A real comparison test would be with the hotspot metering strategy enabled which attempts to eliminate the metering of low cost instrumented method invocations (reducing the cumulative count). Here is the clock.time metering model for the startup. The number of metered probes fired has now halved though this reduction would be much higher in a long running system as we by default meter the first 1,000 firings per named probe.

With the hotspot metering strategy configured to use the sample.count meter the breakdown of both the “Count” and “Total” columns looks again similar to the clock.time metering model.

Though I still prefer to use our sample based metering strategies (more so because of familiarity) which guarantee accurate metering of those probes sampled the sample.count meter approach allows me to create a filtering like mechanism to the model at the granularity that suits the context of the investigation. With the meter I get the frequency for every probe fired which the strategy approach does not afford me but I only aggregate the metering total for those probes fired and metered that exceed a meter threshold I have defined.

To be in anyway successful at managing & monitoring real-world Java enterprise applications in production (that excludes Spring “PetClinic”) one needs to assess the runtime impact of the various performance data collection techniques and their actual (tool) implementations. Unfortunately most programmers fail to understand this and instead develop/promote/deploy tools that are clearly completely inadequate for the task at hand – in a production context. They incorrectly assume ease of use equates to lightweight when in fact the only thing lightweight is the effort and due diligence performed in understanding the problem domain and managing the associated risks. It is for this reason why most operations teams immediately discount (with any evaluation) the use of monitoring tools a developer might have used in any way during localized tuning.

Recently this was made so evident when its was falsely claimed (without any qualification other than wishful thinking) that a Java sample based profiling tool could out-perform JXInsight – a dynamic strategy based resource metering solution. I will save the publication of a benchmark comparison for a (near) future blog but instead show you a chart (taken from within the “visual” tool itself) depicting the performance impact of such a technique on an application with a realistic number of threads (100) running with call stacks of realistic sizes (100-200).

Note: The similarities of both the 100 ms and 1,000 ms run sections indicates the tool was not able to perform at the lower of the sampling rates.

Even at a 10 second sample interval rate the runtime impact is excessive ignoring the fact that at this resolution the data would be completely meaningless unless the application was a dog in terms of performance which begs the question “How did such a beast get into production in the first place?”.

Yes the overhead of sampling is zero when not being performed but what type of enterprise application is not monitored continuously in production? Instrumentation needs to be a primary source of resource metering & software execution metric performance data. Sampling should be considered (and only after discounting many other sources and techniques) when the current instrumentation coverage is not adequate at the time a problem appears though I am doubtful of its effectiveness with the noise that would be created when enabled based on the charting above.

It is notoriously hard to detect and distinguish busy and hung threads in todays complex enterprise middleware even more so when the middleware is distributed across multiple processing nodes with temporal execution dependency (locking, co-ordination). A large part of the reason for this is the lack of current execution information associated with frames within thread stack dumps. For one we cannot easily determine from two thread dumps listing the same call stack for a thread whether the thread in question is hung waiting on a response from a resource  or whether the thread is performing the same operation again following receipt of another request into a processing queue. We might not even be able to detect the case were the thread is merely busy making calls from the stack listed in the dump to other fast executing methods not listed.

To help with the problem analysis we have recently enhanced our optional stack probes provider to collect additional information such as the number of times a uniquely identifiable stack appears across two mark requests from within our management console and the number of calls (or frames) made during the execution interval of a caller frame.

Here is a Java class I quickly wrote to demonstrate the problem and the solution. I have created two Runnable classes that mimic the behavior of hung and busy threads.

Here is a sample of a partial stack dump during the execution of the above class. As you can see it is pretty hard to actually determine the type of problem that may be present ignoring the obvious clue in the class naming.

Below is a resource metering snapshot taken during a similar execution run with our dynamic probes instrumentation enabled. What we can quickly see from this view is that the run() call executed by Thread-0 has only made one additional direct (or indirect) call during its ongoing execution interval. Whereas the run() call executed by Thread-1 has been busy performing more than 1 million calls (stack push operations).

Now we cannot be sure that Thread-0 is effectively hung because it might have only just began executing prior to the snapshot capture. Likewise we cannot be sure of our initial prognosis for Thread-1 as it might just be finishing up or it has been active for a prolonged period in the runtime waking up occasionally to service queued requests.

So lets make a mark (reset) from within our management console that instructs the remote managed runtime to mark all current resource metering related data including associated stacks. Then we wait a little while (a few seconds) and then capture another snapshot shown below. From the Marks table column we can see that some parts of the stacks present in this new snapshot capture were present prior to our recent mark remote operation – marks > 0. The only new stack frame is the Busy.doit() call which has a thread execution sequence id of 3031907.

In addition we can see from the Count Delta column that only the run() call executed by Thread-1 has performed additional (instrumented) calls during its still unfinished execution.

To be sure that the stacks visible from within the management console have indeed an execution period of longer than a few seconds we can make another mark. Now any element of a stack listed in a snapshot following the last mark operation with a Marks value greater than 1 has at least an execution duration of the time difference between both mark operations.

For those elements with a Marks value of 1 we can only say with confidence that the stack (or partial stack) has existed since the period of the last mark to the point at which a snapshot is captured which can be done repeatedly without marking.

Now if that is not exciting enough we can easily perform the same steps across a complete cluster of nodes from a single view within our management console and this works for all probe related extensions whether the probes represent actual Java methods or contextually mapped Ruby and Python calls.

Following on from our “The Fastest Ruby Profiler is a Java Profiler” I am pleased to announce that our upcoming probes based resource metering extension for Jython is the fastest function call profiling solution for Python applications executed within a Java runtime. With our dynamic hotspot resource metering strategy the overhead of our probes instrumentation can drop down to approximately 8 nanoseconds for non-hotspot function calls.

Here is the Python micro-benchmark I have used in determining this.

Here is a chart showing the average time (secs) reported by multiple executions of the run method without (baseline) and with our probes based instrumentation. JXInsight’s instrumentation overhead is just over 2% but that is 2% on a method, call, that effectively does nothing.

The overhead incurred in the instrumentation and measurement of fine grain method executions, which dominate the execution (frequency) profile of most web applications, is a key performance indicator (KPI) for any enterprise application performance management and monitoring solution.

Below is a screenshot of the resulting resource metering in our application management console. Note the strike-through on the call:12 function indicating the associated instrumentation has been disabled following the execution of 1000 probe firings and the determination by the hotspot metering strategy that the named probe, benchmark.call.12, is not a hotspot.

Note: Line numbers can be excluded from the probe name assigned to a function marking all same named functions within a module as hot or cold spots depending on the combined resource metering behavior for all implementations.

The following chart shows the performance of the same benchmark executed by the native c-based Python engine along with the standard cProfile. The overhead is significant.

Here is a chart comparing the overhead (%) of JXInsight with that of the Python cProfile.

I attempted to run the benchmark with the Python based profile solution but a single run() call took over 200 seconds compared to 2-4 seconds for all other reported test runs. Surprisingly the Jython version of the Python profile solution executed in just under 120 seconds.

If you are interested in seeing more of what will be possible with the upcoming extension then check out the following images showing contextual labeling as well as multi-language metering (profiling).

www.jinspired.com/images/jxinsight.5.7.32.jython.to.python.p.1.gif
www.jinspired.com/images/jxinsight.5.7.32.jython.to.python.p.2.gif

One of the many cool and innovative features we have recently pushed out the door with our regular software updates is an enhancement to tagging that really makes this unique feature extremely useful and versatile in the analysis of software under construction during development & test or under management in production. Now when you set a tag at either the global or thread context level not only do we aggregate resource metering for the tagged metering group but we do this for every other metering group updated during the tagged execution period. We do this by creating an extended @tag metering group that concatenates the name of the metered group updated with the value of tag.

Being able to inject tags remotely from within our management console or command line interface or locally from within the actual managed runtime via our Open API gives us the ability answer two very broad questions:

1. What is the resource usage for a package/class/method during a particular tagged execution period?

2. How does the resource usage for a particular probe compare across tagged and non-tagged execution periods in terms of count, total, min, max and average?

Previously the way to answer such questions at a global level across various profiling and application performance management (APM) tools would be to reset all data collected prior to the execution period, specify a period filter on an event journal table or log, or in our case mark the current totals and then perform delta analysis following the completion of the execution metered. But this leads to the creation of multiple disconnected performance models that cannot be easily compared and reconciled. In addition it is not possible to narrow the analysis to one or more threads or have concurrent tagged execution periods with each thread having its own tag value which is required when the tag values reflect the some aspect of the execution context such as the web application user name, request parameter or session attribute.

Here are some concrete kind use cases where this capability might be much more appropriate during analysis across multiple application life cycle management phases.

1. During the testing of a web application interaction consisting of a series of steps and pages a tag based on the step or page name is set globally allowing the developer to break down the resource metering costs for a particular package/class/method by the tagged step (or page).

2. During unit testing a tag is set at the thread context level based on the name of each @Test method executed providing code coverage like analysis for direct and indirect executed code by test method. This data can also be used in generating dependency analysis reports for such tests.

3. During a system failure (i.e. database bounce) a global tag is injected identifying the failure period allowing service level management to later break-out such costs on a per activity/interaction basis.

4. In diagnosing a user specific problem a thread context level tag is set identifying the user or the particular interaction at the entry point of request processing enabling resource metering to be reported both from an operational perspective (code) and contextual (user marker) perspective side by side.

5. In delivering elastic computing capabilities to an application a tag is injected into a cluster of runtimes at each dynamic change (a capacity profile) in the computing capacity made available to the application in the cloud which can then later be analyzed offline to determine reasons for the increase in capacity not being fully utilized due to possible inherent scalability problems with the applications own code base. Likewise one can easily determine which particular activities are impacted more than others by decreases in computing capacity due to elasticity policies.

Recently we announced that JXInsight can execute 1 billion instrumented Java method calls per second once the multi-resource (hotspot) strategy based metering runtime determines a particular method is a non-hotspot. So I was curious to see how our JRuby-to-Ruby cross language profiling extension faired with such optimizations as our probes technology is not in anyway tied to metering just named Java code execution constructs. The good news is that our technology can drop the overhead so low that it even out-performs no-op instrumentation calls in Ruby itself which means we are faster than any Ruby language based profiling or benchmarking solution (naturally when executed on the Java platform – JRuby VM).

Here is the Ruby micro-benchmark I used to determine the above.

Here is a chart showing the average time reported by multiple executions of the run method without and with our probes based instrumentation. Our instrumentation overhead is just over 15% but that is 15% on a method, call, that does nothing. Effectively our overhead is just under 15 nanoseconds for non-hotspots.

The overhead incurred in the instrumentation and measurement of fine grain method executions, which dominate the execution profile of most web applications, is a key performance indicator (KPI) for any enterprise performance management and monitoring solution.

Here is a partial screenshot of the resulting resource metering in our application management console. Note the strike-through on the call method indicating the associated instrumentation has been disabled following the execution of 1000 probe firings and the determination by the hotspot metering strategy that the named probe, Object.call, is a non-hotspot.

To determine the best (if not impossible) time a Ruby based profiling solution could do I added both a before and after method call to the call method which is typical of how most dynamic instrumentation agents work across various languages and runtimes. Both methods do nothing.

Here is a revised chart with the average execution time for the above manual instrumentation added. JXInsight is 3 times faster than a Ruby solution that does nothing.

Note: Disabling our hotspot metering strategy and comparing the performance with a version having a callout to Benchmark.measure() JXInsight is more than 15 times more efficient. Moving to JRuby has many advantages for production applications both in terms of the raw speed of the runtime itself and the supporting management and monitoring tools.

Hardware
Model Name: MacBook Pro
Model Identifier: MacBookPro5,1
Processor Name Intel Core 2 Duo
Processor Speed: 2.53 GHz
Number Of Processors: 1
Total Number Of Cores: 2
L2 Cache: 6 MB
Memory: 4 GB
Bus Speed: 1.07 GHz

JRuby
jruby 1.3.1 (ruby 1.8.6p287) (2009-06-15 2fd6c3d)

Runtime
java version "1.6.0_15"
Java(TM) SE Runtime Environment (build 1.6.0_15-b03-223)
Java HotSpot(TM) 64-Bit Server VM (build 14.1-b02-92, mixed mode)


There are two basic approaches to developing diagnostics for any enterprise Java application server (be it OSGi based or not). The first approach is to create lots of custom code that treats every subsystem, service and component uniquely different in terms of presentation, collection and modeling forcing the creation of extraneous interfaces and/or additions to existing classes to support state extraction. The second approach involves defining a common model and representation of state data (including meta data) that can be used across multiple diagnostics (state) datasources, implemented by a single provider using runtime object state inspection (without any code changes) and supporting exploration within a single management console in a consistent manner and accessible whilst being disconnected.

Unfortunately nearly all application server vendors select the first approach including those newcomers that claim to be anti-bloat ware.

Naturally we opted for the second option building a kind of dynamic state based configuration management database (CMDB) snapshot facility with supporting disconnected offline analysis tooling.

Amazingly some vendors require you to start an actual server instance to view the contents of a dump in a readable format.

With JXInsight all that is required to capture diagnostics state is to simply drop a technology specific observation extension into the class path of an application server. Here I have used our jxinsight-ext-aj-osgi-bundle-observe.jar to collect actual state diagnostics for live OSGi bundles.

Note: A snapshot can contain both dynamic & static, public & private, state. With static state typically representing configuration loaded from a persistent store such as a file system.

The depth to which object (graph) state is collected within a diagnostic snapshot is configurable but typically 6-8 is sufficient for most offline analysis purposes.

By combining both state and metadata into a single diagnostic snapshot we can render, highlight and annotate various items within the model – all in a consistent manner with as much detail as one would required in whatever role we assume during the problem analysis.

Note: We record the time(stamp) the object was constructed or first appeared to the diagnostics runtime as well as the system identity hash code.

If we want to enhance the collection capabilities of the diagnostics system we simply drop in another extension library. Here I have added jxinsight-ext-aj-osgi-service-observe.jar

We can even record much more dynamic state including that which is being referenced directly or indirectly during request processing. Here I have added jxinsight-ext-aj-osgi-service-diagnostics.jar to the classpath to collect at any moment in time within the management console the object graph state related to arguments and target objects currently executing methods defined in the one or more of the OSGi service related interfaces. The method being called is getServiceReferences by thread server-dm-1 with a single string argument on a target object of type BundContext.

The problem with (published) performance benchmarks is that they are there to be beaten (luckily only ever by ourselves) and today well we have shattered all our previous benchmark results, including one last week, which means I am now faced with the daunting task of going over all my previous related postings and redoing the testing or just simply linking to the latest and greatest numbers. I think I will go with second option for now.

JXInsight’s multi-resource metering & monitoring (& profiling) technology is truly the fastest and most scalable Java solution on the market that is designed specifically for production usage in the most demanding of environments. Getting to this level of software quality is never easy. Its been a long and hard slog but Rod and I (co-founders) have managed to consistently out class the competition in terms of open & extensible API design, efficient runtime & tooling, and in the delivery of innovative information visualizations for the vast amount of data that the product can collect today.

Last week I managed to get back working on cool and innovative (hopefully you will agree) visualization work that I had designed (lots of sketches scattered around my office) a long time back but never managed to get the push to actually realize them within the product until today when we released a first in a series of visualization updates I am actively working on that will raise the bar (sky high) once again (well that is the motivational chant used currently).

In this first update I have tried to tackle a common problem with most application performance profiling/tracing/logging/journaling tools that use ordered child nodes within trees to display sequential execution behavior. The problem being that they do not scale in the enterprise context (too many nested branches and large sequences of nodes), are wasteful of precious screen real-estate, and cause repetitive strain injury (RSI) with excessive clicking to expand & contract nodes (seriously).

The solution is to not use a tree. Instead we use an ordered table that overlays the hierarchical execution nature (caller-to-callee) on top of a flattened sequential execution flow and makes it contextual (based on the current selection). And best of all we can display the execution of multiple concurrent threads side-by-side (to be exact its row by row) which is impossible with trees as the basis for any such visualization. Here is an animated gif showing this in action.

By the way the graphical tree overlay also highlights re-entrant calls on the billed probe stack. Can you see the visual pattern indicator?

This was my first time to attend (and speak) at the Jazoon conference and though I was only able to attend for 2 days I really did have a great time attending the conference and talking in person (and drinking) with a number of well known individuals in the Java industry including Bela “a blast” Ban (JBoss), Mike “I’m nice” Keith (Oracle), and Kirk Pepperdine (Kodewerk) who dared discuss and debate application performance management & monitoring approaches with me over lunches and walks to and from our hotel.

Kirk just about to experience my Irish resolve.

I did attend the keynote session given by Adrian Colyer (SpringSource) and I must say I was impressed with his presentation skills even though it was a tad bit too long in terms of the green theme and light on in-depth technical analysis but I suppose there is only so much you can do in such a short time especially when predicting the future which could all change with an acquisition.

Adrian was as ever economical with the facts in his assessment of which VM language will win the war when he claimed that his choice of Groovy was made prior to the acquisition of the company behind Groovy & Grails – hence it was a pure technical decision. He failed to state that the VC that provided financial support to SpringSource at the very outset had also invested in the acquired company and had been promoting the technology to Rod and Co. for some time. Whether it had any bearing on the final decision is a different matter but it should have been disclosed in his argument to the audience questioning his objectivity.

The same cannot be said for the other main session I attended in which Peter “let me put this chastity belt on your classloaders before everything drops out under” Kriens (OSGi) who did a very poor job of presenting a supposedly technical session which in fact turned out to be more like “let me preach & market my 10 year old modular framework (that keeps re-inventing new component models)”. Peter really used every cheap marketing trick in the book when it came to the information (can we even call it [data] that, more like the gospel according to Peter) he presented in his slides. Below is an actual slide from the presentation.

Heaven awaits those who embrace OSGi?

The session might have been somewhat informative if he stopped talking about his 10 years of servitude, left out the gimmicky slides, and limited his bashing of Java and Sun at every opportunity during the talk. It felt like 10 years had passed when the session was over which should not have been the case for this particular technology which will have served an important purpose when Java does indeed deliver a built-in and much improved replacement. His insistence that modularity should be managed entirely outside of the language and oblivious to the Java runtime sounded like someone completely divorced from reality (vital lies?).

When Juergen Hoeller (SpringSource) was pulled over to our table I did question him on whether Rod (SpringSource) should continue blogging or go back to coding factory classes. What is worse? His reply was “No comment” but his body language betrayed him.

During Peters talk he mentioned that we was instructed by the OSGi committee to not talk about the original target environment for the technology – the embedded market.

Not all OSGi members share Peter’s view that the default behavior for OSGi should be to break existing enterprise Java applications which he refers to as “no pain, no gain”. There appears to be two camps. The hard-liners (and groupies/followers whose only response to issues raised is to name call those who dare mention them) and those (mainly app server vendors) who just want to get things to work sometimes only using the technology to segregate container libraries from application deployments.

Peter has responded (partially) and apparently “new” is (only) good when it is in relation to “OSGi (component) programming models” but not when it is in relation to an approach to modularity that involves enhancements to the language and runtime which are outside of Peter’s control (hence his rejection). I cannot believe that Peter & Co. would have designed OSGi as it is today if they had the opportunity to extend the language, runtime and tooling to accommodate their design. I refuse to believe that they did not compromise their design (and resulting implementations) especially in light of this and the original target environment – the embedded space. At the very least they would have looked at ways of reducing they own class loader hacks. I also doubt that Sun would have allowed any proposed design break compatibility in a number of ways that OSGi has done so (and some for no good reason). Even if Sun fails to delivery a deployment centric modularity approach we still might gain from their efforts as a smaller (modular) VM could gives us the ultimate class isolation container – an operating system process (and many of them hosting specific services connected to a grid/cloud). Lets not be fooled in to believing that to achieve modularity we just put a class chastity belt around our private parts. In the enterprise space state is just as important as behavior and it is not so easy to tackle in or outside of a container which is why most systems go occasionally offline (at least internally) to sync changes across different data processing domains (messaging, databases, …). Even if you can isolate classes from each other you cannot do the same for the external side effects (of multiple & different versions). This last item seems foreign to many OSGi disciples who apparently do not understand one of the main reasons why operations stop and restart processes in applying a managed change – to be sure the process can indeed be restarted when something goes terribly amiss in production outside of the container (hardware). It is much easier rolling back at the time the change is applied.

The “chastity belt” reference reflects my view that modularity, which should be managed at many levels in the design and development process and not just in bundling, has more to do with education and support during the early application life-cycle than a cruel troublesome enforcement at the late stage of post-deployment (into the bad real world). We need modularity to be supported in different ways (not just a single realization) and at different stages with each building on top and augmenting that which precedes its. I think currently there is way to much emphasis placed on bundle separation (thinking being that the more modules the greater the modularity). We should look at packaging (bundling) of application artifacts to indicate deployment collocation for reasons other than a “modularity nirvana”. We should also encourage diversity in approaches to how modularity is enforced at runtime (if it need be or not) and not just standardize on a particular implementation. The language and tooling needs to help capture and convey intent not target implementation. Modularity is not achieved by just creating many little bundles and hiding classes which seems to be Peter’s fixation (crucifix). I have seen very modular software designs that never needed runtime policing of visibility (outside of what is provided by the language) maybe they were not so dynamic (at least within a single runtime and in terms of service implementation) but how many organizations really need such capabilities and have the ability and tooling to effectively manage and control such environments in an enterprise context. Very few and even if they had the capability within the organization most try to avoid such scenarios (multiple live bundle/library versions in a single container) because the risks (diagnostics is problematic) far outweigh the benefits (“its cool”).

Peter did not actually address my main criticism points of his presentation. The cheap tricks in how he presented the information to back up his OSGi marketing campaign like the dependency graph of the Java class library with no comparable graph of a similar runtime and SDK built over the same period (Java 1.x-1.6) and offering the same breath and depth of services using OSGi. How much different would it indeed look like? Would any noticeable differences reflect the choice of the component approach or just the talent and longevity of the people involved. Great products are created by great teams. Yes there is a need for a framework (of reference and values) along with a support system but does it need to be policed in the way proposed which in my opinion reflects the constraints on the original design space (or the authors cynical view of the rest of us)? Does it need to be marketed as “the” silver bullet for the software industry that ultimately fixes us to a future designed 10 years ago? I hope not. We need to learn from previous work along with the mistakes made and 10 years gives you plenty of time to do this no matter how hard you try not to as time and the pace of innovation is not on your side.

Mike over at Canoo has a write-up of the talk I gave today at the Jazoon 09 conference on applying activity based costing in the cloud. I understand some of his concerns but I hope he will see that the model itself has application in many areas not all cost related (at least not directly) including performance and capacity management.

I am delighted he is excited (“it’s certainly an incredibly powerful idea”) about the possibility of our OpenCore Probes API becoming a standard which we can all expect to be available in the runtime and which can be extended by cloud or runtime vendors via the registration of their own custom meters.

I would like to add that metering does not necessarily imply that every activity has to be costed. Metering is a powerful and scalable way of analyzing the resource usage of a software without suffering from the obvious scalability and performance problems of call or path trace tools. Activity analysis can be the sole purpose of the data collected and the model generated. During the demo I showed how you can even create a meter based on the frequency or resource usage of another metered activity. This is what so cool about our design and implementation – you can model the execution of a block code as an activity or meter or both.

Activity based metering (ABM) is an incredibly important source of information on behavioral patterns of users which can in fact increase development budgets in areas that have higher profit margins or are used much more frequently (and hopefully appreciated) by clients. More and more of our daily activities are mirrored in the execution of software in todays web and business applications. Threads are our delegates (agents) acting on our behalf.

It is also extremely important when you are a Web 2.0 startup with a growing user base and no clue of what business model can be applied that will eventually offset operational costs that will likely to continue to grow in the future.

Next time I intend ending the session with a clip from the Matrix showing our life streams (threads/calls) flowing down the panel with a little William (or Smith) assuming control of one or more threads temporarily.

In a previous blog entry, Instrument->Measure->Refine->Repeat , I discussed how to approach the performance analysis of software with request processing execution times in the low microseconds via an iterative hotspot detection process that seeks to refine the instrumentation and measurement with each cycle, reducing overhead and minimizing perturbation in the data collected. In this follow-up I would like to present an alternative and complimentary approach that allows us to largely discount any instrumentation and measurement overhead by focusing on the rate of change in key performance indicators (KPI) that are not time based.

But first allow me to ask the question “Why do we profile the execution of software?“. For many developers the answer is simple – “to tune”. But this is not entirely the only valid reason and in fact this answer is more of a by-product of other activities and tasks within software performance engineering (SPE).

For large complex systems we observe and measure the execution of software in order to understand, model, quantify, and manage its behavior (resources usage & utilization, performance, scalability, runtime system dependencies) across multiple software releases and under different operating & executing contexts. Naturally we need a way to effectively manage all this information. This is achieved by metering (tracking) a relatively small number of indicators that measure operational activities that have a significant impact on performance, correlating the rate of change in such indicators with higher-level activities and various operating parameters.

Note: In fact the ability to detect and assess underlying changes in both the system & software execution models across deployments is of paramount importance in delivering service level management and reducing the time window for problem resolution in production environments.

The choice of a good key performance indicator depends largely on the type of software being monitored and managed. For standard business applications this can be the number and timing for requests serviced by various resource managers (file system, database, messaging middleware, data grid, compute grid). For low latency request processing systems there are a number of useful candidates for key performance indicators that are not entirely time dependent or impacted by instrumentation including: method execution counts, field read/write counts, object allocation counts & sizes (and to some lesser extent cpu time).

Lets apply this approach to the performance analysis of Oracle Coherence used in my previous blog entry which you should have already read at this stage. Before looking at any execution profile data you first need to know what was the activity being tested and how was it parameterized. Here it is:

>cache cache
>#10 bulkput
>#10 bulkput

Executing the above command sequence in the Oracle Coherence interactive shell results in a total of 20,000 SafeNamedCache.put() calls. We can see this below in the metering table with the Count column representing the number of times this particular method was called.

Lets keep this number, 20,000, in mind when looking at all numbers presented below.

The Total column represents the number of methods executed within the com.tangosol.* package during the execution of the SafeNamedCache.put() 20,000 calls by the same calling thread. The Total(I) tells us the number of methods called that were directly (inherently) attributed to the execution of the  SafeNamedCache.put()

So 20,000 SafeNamedCache.put() calls resulted in 13.7 million method calls, 40,000 of which were made directly from within the method itself. But what about the other 38 million (51 less 13) call? Why only 13.7 million? The reason is that these other 38 million calls were made outside of the execution of the cache operation and more importantly by other threads. Expansion of the metering model tree reveals one of the main contributing classes, PacketPublisher, whose method executions resulted in a total of more than 22 million (1.3 million x 17) method calls, 4 million of which were made directly from within the class.

Note: For extremely low latency systems the overhead of excessive method calls can be relatively high even if in-lined to some degree because there still needs to be some housekeeping performed by the runtime (call stack mgmt). When looking for possible candidates to tune start with those probes with high averages.

In analyzing the performance of our own software product we use two low level meters – field.read.count and field.write.count. Below is such an analysis with 37.6 million reads performed on fields residing in the com.tangosol.* package during the execution of 52.1 million method calls also within the com.tangosol.* package.

Note: We ourselves monitor call paths within our probes runtime at the nanosecond level by averaging billions of test executions and by managing the number of field reads and writes.

Comparing the Count column with the Total(I) columns shows that a vast number of calls within the PacketPublisher class are simple field getters – an average of 1 field read per method call. It also reveals that during the execution of methods within this class 18.5 millions fields are read with 1 million within the class’s own methods and the rest in methods executed elsewhere in the com.tangosol.* package.

Another useful meter in analyzing software with a large amount of thread co-ordination is waiting.count which tracks the number of Object.wait() calls made by a thread.

Note: The waiting.count meter can be used in “profiling nothing” as a high number of waits calls is generally indicative of worker threads starved of work items to process.

For low latency systems you really need to look at the rate of object allocation and the resulting byte sizes. This is not only important for performance management but for capacity management as well as most OutOfMemoryErrors are the result of high concurrent request processing placing too much demand on the available memory pool and not some permanent memory leakage (more like a temporal & transient waste).

Note: Execution count profiling allows one to identify and quantify the impact of particular request execution parameters have on key performance indicators – correlating rate of change across multiple variables.

Focusing on the average object allocation count per method execution can sometimes reveal opportunities for tuning which by the way does not necessarily imply that objects should be cached – always try to eliminate work completely before improving the efficiency of the work via reuse.

From the metering table below we can see how many objects are allocated directly from within a method and how many are typically allocated during its execution including from within methods called directly and indirectly.

Note: Object allocation metering can also reveal the cost of initialization – methods only ever executed once.

Key performance indicators can be used to drive change (tune) but they can also be used to effectively manage and track change not necessarily tied to the achievement of an aligned target. They can also be used to identify unplanned changes that would otherwise result in customers notifying you of visible side effects at a much higher granularity and largely disconnected from the problem domain.

In a follow-up to this entry I will show how easy it is to create your own custom key performance indicators with our aspect library extensions.

Kirk Pepperdine recently queried whether JXInsight could indeed profile nothing. Naturally JXInsight can profile the “nothing” in all its many forms (resource contention, coupling & co-ordination) but to keep the first part of this follow-up short & sweet let me show you how to detect latency in browser based operations from a software execution model perspective. In this scenario a single button click on a web page to cancel a hotel booking that results in the user returning to a page displaying the list of current bookings remaining.

And here it is – our timeline analysis of the two resulting HTTP requests processed by the same worker thread (this need not always be the case) within an application server.

The light blue rectangles represent the time the thread was processing within the servlet layer. The yellow and brown the time the thread was processing within the JNDI and EJB layers respectively. The bracket above the brown rectangle in the first HTTP request represents the actual active resource (JDBC) transaction window.

Finally the whitespace (the “nothing”) in the middle of the timeline graphic is the time spent outside of the container sending/receiving/processing the request at the client – 15-20% of the total user experience time.

Next up profiling the “nothing” from a system execution model perspective.

I recently came across a customer that was using our probes Open API in a novel way to simulate a real-world business process and composed activities for the purpose of cost analysis. This is quite unusual as the technology is mainly used in metering and costing services and activities that are purely software based. But here the customer has created a software execution engine that models the real-world and uses our software instrumentation & measurement technologies to meter the simulated model in terms of the real-world cost drivers.

Note: Our probes technology is unlike any other performance management solution in that it is not tied in anyway to method execution constructs. Composite named probes can represent any identifiable execution activity and resources can be abstract representing such things as chargeable cloud service interactions. This allows us to meter across languages within a single execution context.

This usage of our technology made me revisit a recent discussion I had with a cloud computing vendor regarding how to apply an activity based costing approach to hosted computing appliances in the cloud. The vendor in question wanted to initially offer very high level metering information to customers before rolling out a complete activity based costing solution that would allow customers to expose activities within their software to our extensible resource metering engine delivering an unprecedented cost and performance management offering. What if I created a software model of the cloud platform itself – a model that would act as a proxy or facade on to the underlying cloud and that would receive events or control actions directly.

The following table shows the mapping from the cloud model to our Open API interfaces and then to the corresponding activity based costing concepts.

Note: This mapping is largely only applicable to the context described here as a resource in activity based costing can be mapped to a Meter, Counter, or Measure within our Open API.

Lets create a mock implementation to demonstrate this approach to using activity based costing to meter at the VM level by starting with a rather simplified VM interface representing a virtual machine within a cloud.

Here is a Mock implementation class of the VM interface. I have added a units field representing the number of processors assigned to the virtual machine. Naturally most of these calls would be delegated to an agent running on the virtual machine itself or possibly some central information system.

Below is the Worker thread class that will run within the runtime for the lifetime of the VM being modeled. In the run() method the worker registers two Resources, vm.cpu.time and vm.up.time with the current thread resource metering context before starting the metered activity.

Note: JXInsight comes with 30+ built in resource meters and offers the ability to register custom resources via the (pushed based) Counter interface or (pulled based) Measure interface.

Finally here is the Cloud class I used to interact with the model from the command line. The class supports 3 commands: start (vm), stop (vm) and tag. The tag command allows the customer to set an alternative metering cost structure for probes (activities) being metered.

Before running up the mock simulation I created a jxinsight.override.config file to setup the meter-to-resource mapping as well as some unit costing. Here is the contents of the file.

jxinsight.server.probes.tag.enabled=true
jxinsight.server.probes.meter.clock.time.include=false
jxinsight.server.probes.meter.resources=vm.cpu.time,vm.up.time
jxinsight.server.probes.meter.unit.cost.include=true
jxinsight.server.probes.meter.resource.vm.cpu.time.unit.cost=2

And here is a small transcript of an interaction with the cloud.

start com.acme.sales.crm.vm1 4
# tv advert airing
tag tvadvert
# vm1 is loaded need more capacity
start com.acme.sales.crm.vm2 4
# resuming normal service
stop com.acme.sales.crm.vm2
tag
stop com.acme.sales.crm.vm1

Connecting our management console to the Java runtime which is acting as a proxy to the cloud infrastructure I captured the following resource metering & costing model.

Note: The metering (cost) group structure shown below is derived from the composite name used in creating a probe (activity) which itself a parsed representation of the virtual machine identifier.

In a follow-up to this blog entry I will show how easy it is to combine this cloud model based approach with an actual standard software activity based costing approach provided to customers and used by cloud service solutions (storage, queues).

One of the hardest challenges for any application performance management tool is in reducing the runtime overhead to levels acceptable for production environments whilst collecting as much measurement data as possible for the purposes of service level management and problem management. For the typical code profiling solution this is an impossible task without significant filtering which incidentally most of the time results in the actual bottleneck being hidden as filtering is usually based on namespaces rather the actual performance profile of the methods themselves.

Even with significant filtering most profiling tools cannot perform at levels required for applications that have incredibly low request processing times whilst executing a significant number of method calls. If a request executes in 250 microseconds (which pretty much rules out call stack sampling) and during its execution it makes 2,500 method calls it is pretty easy to see that achieving an acceptable level of overhead, 2% (5 microseconds), is an insurmountable task especially when a system call to the clock counter costs between 0.2 – 0.3 microseconds and you need to do this twice for each of the 2,500 method calls.

With a dynamic measurement (metering) solution such as the product I work on you can reduce the overhead of an instrumented method call, that is not deemed a hotspot (dynamically at runtime), to just under 0.03 microseconds but multiplying that by 2,500 we still come out above the acceptable levels for this extreme type of request processing.

Can such applications then be profiled? Yes. There are two approaches one which I will discuss in this blog entry and the other in a future entry.

Instrument->Measure->Refine->Repeat
Clearly a solution to the problem described above must involve reducing the cost of both the instrumentation and measurement. But how can this be done in such a way that we do not eliminate expensive calls from the performance model whilst ensuring we do not inadvertently include calls perturbed (directly and indirectly) by the remaining instrumentation. The solution is to iteratively refine the instrumentation based on measurements that identify execution hotspots. With each iteration the hotspot list is reassessed in light of the reduced level of instrumentation.

For this process to be much more effective than the guesswork performed by way of manual namespace filtering the underlying runtime and tooling must at minimum

1. Perform the measurement with as low overhead as possible (which is why our product out-performs the competition by a huge margin)
2. Allow the hotspot determination to be based on the dynamic analysis of resource usage across one or more meters (via composite cost meters)
3. Automate the accurate generation of the refined instrumentation set (it is a one-click operation in our product).

To demonstrate this approach I am going to use a real-world software example – Oracle Coherence. Shipped with the Coherence product is a command line interface that allows one to interactively execute Map.put() and Map.get() operations on a distributed cache. The command line facility also allows you to perform such operations in bulk and to repeat them. Here is the command sequence I will execute across each test presented below.

> cache cache
> #100 bulkput

The above commands will create a distributed cache named “cache” and repeat 100 times a bulkput command which by default executes 1,000 Map.put(Integer,byte[]) commands. In total 100,000 Map.put calls are executed.

Below is a metering model collected following the above command sequence which does include a small number of call counts related to the start-up and excludes constructor calls. Over 2,300 method calls are made per Map.put call. Running the test without any instrumentation the average execution time is 0.265 microseconds per Map.put call. This is our baseline.

Note: This particular use case is a best (if unrealistic) case for Coherence as the command uses a byte[] (easy IO mapping work) as the put value, whereas most applications would have objects that need to be serialized which is very expensive relatively speaking that is.

The above metering was collected without any of our production-oriented features enabled such as our dynamic hotspot metering strategy. By re-enabling such features whilst keeping the same level of instrumentation we get the metering model shown below with a significant reduction (1000x) in the number of firing probes that were actually metered (calls measured). Unfortunately we still take a performance hit in dynamically assessing whether a called method (firing probe) is a hotspot or not (metered or not). The average execution time for each Map.put is now 0.640 microseconds.

Note: The definition of a hotspot is highly configurable within our runtime. For the tests reported in this entry I defined it to be any method with an average inherent wall clock time of 50 or more microseconds after 1000 executions globally.

Fortunately from within the management console we can automatically generate a new probe extension (jar) based on those probes labeled hotspots within the metering model exported from the managed runtime. One click operation!!! Instead of instrumenting over 4,500 methods we now have a small list of methods to be instrumented spread across a total of 5 classes.

Below is the metering model following the deployment of the new probe extension containing the refined instrumentation. We have now halved the metering count but more importantly the instrumentation overhead associated with methods not deemed hotspots has been removed. The average execution time has dropped back down to 0.285 microseconds.

Generating a new revised probe extension based on the hotspots in this new metering model reveals a further reduction in the methods and classes deemed hotspots.

Here is the final metering model captured based on the refined hotspots – a ever so small reduction. The average execution time has now dropped to 0.275 microseconds which equates to 0.010 microseconds of overhead which is largely a result of the underlying system clock calls.

The following bar chart compares the changes across each iteration with the baseline reported above and that of another Java profiler.

Note: Hopefully this explains why today we ship with over 600 technology specific extension libraries that specifically target the main client call entry points (mainly interfaces) into each technology.

In a follow-up to this entry I will revisit the tests again but this time using another technique ideal for CPU analysis in situations were the resolution of the system counter is insufficient – execution count profiling.