Allow me to kill two birds with one benchmark test. Here is the clean version.

Using JXInsight/OpenCore’s agent the code was instrumented dynamically at load time. No code changes needed. I ran with two different configurations. One that simply meters in memory and the second one that along with metering performs binary logging of the begin and end metering events, the meter readings as well as the name of the probe and thread context.

Here is the manual instrumentation needed to have a comparable test with log4j logging approach.

With NewRelic‘s Java agent there is no means to transparently instrument code that falls outside the few frameworks they support. You are forced to use a NewRelic specific @Trace annotation.

Here is a comparison of the average clock time cost reported after executing each test 100 million to 1 billion times.

You might initially think that NewRelic does not fair so bad compared to the log4j approach but bear in mind that NewRelic does not actually do any file IO. Instead it aggregates the data (losing all trace history in our case) and then dispatches it as a small packet to their web site service in 1 minute intervals.

None of our customers could tolerate the overhead incurred by the log4j and NewRelic with transaction latency as low as 200-300 microseconds and with 10 or more measurement points within such.

OpenCore without the recorder probes provider enabled and using the following configuration had an average clock time overhead of 130 ns with more than 50% of that attributed to the two clock time access reads.

j.s.p.strategy.enabled=false
j.s.p.meter.clock.time.include=false
j.s.p.meter.jvm.clock.time.include=true
j.s.p.aggregates.enabled=false
j.s.p.global.enabled=false

With the recorder enabled as follows the overhead increased to 710 ns and thats for two metering event IO writes. That’s 10x better than log4j but considering this is mostly IO on a slow hard disk the efficiency differences are hugely bigger. I can’t imagine what the other 7 microseconds is being spent on by log4j.

j.s.p.recorder.enabled=true

Below is a slice of the CPU usage monitoring during each of the benchmark test runs. OpenCore is not only so much faster it also uses far much less system resources.

There is a reason why cloud platform vendors like Heroku (Salesforce) and CloudBees offer NewRelic subscriptions but don’t actually use it themselves internally to monitor and manage their own critical systems and services. They are not in the business of conserving your resource consumption (or cost).

Relatively speaking log4j and NewRelic are pretty darn expensive and for the value they offer it puzzles me why anyone would ever consider such for Java application monitoring. Granted not everyone or application operates at such speeds and resolutions but bear in mind that today we advise our customers that the recorder be used mainly in development and focused performance testing, yet it is 10 times faster than these solutions promoted/advised as production solutions.

Yes you can find low hanging fruit with whatever “crappy” approach and agent you choose but if you are still using Crappy 2.0 after resolving such issues then you must question whether you are truly trying hard enough to be good if not better (than the rest).

Clearly what I consider “crappy” and what Brain Doll over at NewRelic considers “crappy” are light years apart and in different time/space dimensions.

Apart from these two blips the Velocity EU conference was a resounding success. Lots of great discussions in the corridors and in the speakers room and not forgetting some really thought provoking talks from John Allspaw, Theo Schlossnagle, Jeff Veen, Johannes Mainusch, and the WTF’er of the year Artur Bergman.

Hardware
Model Name: iMac
Model Identifier: iMac11,1
Processor Name: Intel Core i7
Processor Speed: 2.8 GHz
Number Of Processors: 1
Total Number Of Cores: 4
L2 Cache (per core): 256 KB
L3 Cache: 8 MB
Memory: 8 GB
Processor Interconnect Speed: 4.8 GT/s

log4j.properties
log4j.rootLogger=warn, root
log4j.appender.root=org.apache.log4j.FileAppender
log4j.appender.root.File=log4j.log
log4j.appender.root.layout=org.apache.log4j.PatternLayout
log4j.appender.root.layout.ConversionPattern=%p %t %c – %m%n

I hope to see you there. If you have not registered please consider doing so now. It looks like a great line-up and to be an important conference in the European calendar for performance and operations of web and cloud services.

And if you see me wandering around don’t be shy I promise not to bite .

From Mgmt Dashboards & Consoles to Mgmt Code & Control

If you’re not metering you’re not trying hard enough to be the best – Part 3 of 3 (JXInsight/OpenCore, AppDynamics)
If you’re not metering you’re not trying hard enough to be the best – Part 2 of 3 (JXInsight/OpenCore, DTrace, NetBeans Profiler/VisualVM)
If you’re not metering you’re not trying hard enough to be the best – Part 1 of 3

Online & Offline Intelligence in Java Application Performance Measurement (JXInsight/OpenCore, AppDynamics)

JXInsight/OpenCore Competitive Comparison (JXInsight/OpenCore, AppDynamics, dynaTrace, NewRelic)

Which Ruby VM? Consider Monitoring! (JXInsight/OpenCore, NewRelic)

The Java Application Performance Management Vendor Showdown (AppDynamics, dynaTrace, NewRelic)

The Good and B(AD) of Application Performance Management Measurement (JXInsight/OpenCore, AppDynamics)

Don’t spend thousands if not millions licensing such products if at the end of the day they are only suitable for one type of application – an extremely slow performing web app that makes hundreds of remote database and web service calls in servicing a single request like it was emitting neutrinos that defy the laws of physics (latency).

Coming from a CORBA/J2EE background (@ Inprise/Borland) I can very well understand the allure of a container like approach (or glorified process launcher aka CloudFoundry) to architecting and deploying a new cloud application or service. But the problem I have with this is that I’m not convinced that an integrated container is the best option and should be the interaction point for services. The more a service gets integrated the more likely its falls within the control and execution domain of the cloud application or service itself which is probably the weakest link in the service delivery (or service supply chain). Containers can still serve as useful bootstrap constructs for application code but I think we need for them to be as lightweight (bare metal) as possible, to merely act as a telephone booth that connects callers (applications) and indirectly user conversations/interactions to the grid/cloud and its services and data ports/pools. Phone booths can come in various colors and shapes but at the end of the day they should do very little but do it sufficiently good – getting your code connected at the appropriate point on the grid (service provider market). Eventually such containers will merely serve as loader or launchers that pipe the application task/flow/job code and data context back and forth over the grid to remote service interaction points that offer miniature execution worlds (or containers) for interacting with local/collocated data.

I can see Trinity making such a call to and from the Matrix.

One might argue that containers could potentially offer enhancements (smart proxies) to such service exchanges but this argument falls over when you consider the rate at which cloud services are likely to evolve (if we don’t lock in the future to the present with PaaS). Containers and PaaS platforms are unlikely to keep up with the rate and scale of a healthy cloud software service economy until standardization which is not about less choice “whats on the menu” as some vendors would have you believe. Applications should be designed to merely orchestrate interactions with services not manage (scale up and out) them directly. Don’t PaaS, instead Push. If it does not differentiate then delegate it. The more this is done at all layers and service interaction points the more homogenous it becomes at each local delivery point allowing for greater efficiencies, cost reductions and value creation.

Amazon AWS has the right approach. For now we need to focus on the service infrastructure creating possible additional levels of abstractions. We need to encourage the creation of a service economy with multiple suppliers per service type leading to increased competition and improved quality & reliability. We need to encourage the usage of services not just across applications but across service themselves. All services should be encouraged to support delegation of some aspect of their delivery and execution to other services (in particular storage) were economically (for both customer & supplier) and technically feasible.

Amazon is already doing this somewhat today with many of its services using other services in its portfolio in particular CloudWatch. What Amazon does not do today is allow for this delegation to be intercepted and redirected to another service implementation and provider.

We need to encourage some degree of standardization (even if just at the contractual or conceptual level) in the interaction and exchanges between applications and services and across services such as metering, QoS, usage billing, security, service delegation & routing, execution mobility & location, and resource management & reservation. This should be possible at the account (or application) level or on a per interaction (caller or user) basis. Let the service providers optimize and manage the cross service interactions that we direct and drive through our edge point service interactions which will carry a service(s) context within request playloads.

Cloud is not just about pulling down computing resources in a on-demand fashion but pushing execution and cost from out of our code, application, runtimes, containers even computing resources (on-demand or not). The greatest change will always be in the application tier. Lets cleanly separate that from what we need to be much more reliable and that does not mean hiding it behind some container interface but having it run in a completely different execution & management domain that is beyond our reach (out of harms way) because clearly the vast majority of software engineers and operations staff can’t operate at the level we have come to (unrealistically) expect of our infrastructure.

The last thing we need at the moment is a black box container for the edge execution node that restricts the service world to that of a particular PaaS vendor, service provider, technology, runtime or language. Lets create much more compatibility, coordination and robustness in the infrastructure and services making the edge nodes (devices or containers) as unrestricted and diverse as possible until we have gained a greater understanding and appreciation of the cloud and its possibilities which could very well invalidate our current application architecture and deployment approaches (i.e. webapps / containers) today.

Update [2011-08-24 13:00] A number of PaaS marketeers & evangelists (marketeers with a little bit of knowledge) tried to disparage the opinion expressed here due to funnily enough my experience in metering and QoS which one would expect to be a cortex of any PaaS offering. Krishnan Subramanian over at CloudAve did seem to agree with my points in that PaaS eliminates much of the visibility and control that is needed to deliver a cloud class, Amazon like scale & reliability, service. One reason for my discounting of PaaS at present which I touched on above is that I believe companies should ensure they have as much visibility and control into the parts that differentiate their services and/or applications. This differentiation is generally closer to the user (and processing pipeline) and layered onto of services and infrastructure. The very same area that most PaaS solutions target and want to turn into some black box that could potentially be shared. This runs counter to my view that we should control & change what differentiates a service delegating & orchestrating the REST to other services and infrastructure components. I believe so strongly in the separation & control of this differentiation that I expect all services to allow delegation of some aspects of their execution & delivery. I should be able to call to a service and in the service context pass my storage end point (or that of the user consuming my service and the called service indirectly) as well as my metering/charge end points just like a company issuing a purchase order with a shipping and billing address. Eventually with code mobility and more intelligent service/data routing this can be extended to allow a service consumer to pass along a context to a compute pool to be used by the service provider in the fulfillment of a request. The only parts of an application or service that should be auto-scaled is that which distinguishes you from the rest who are probably also using the same service suppliers. Leave the scaling of other services and software that you have not developed to experts that have designed and successfully operated it at a much greater scale than your own particular workload.

But what if the code were the same but the context associated with a particular execution (method invocation) was different and represented the driver for QoS prioritization. For example the value of a financial trade or the expected profit from a transaction or order.

Lets revisit the sample code in my last posting changing the same Worker class to accept a different value parameter in its constructor depending on whether its an odd or even numbered loop.

In the Worker class below a thread specific metering counter, value, is repeatedly incremented with the $ parameter, that was passed into its constructor, before every call to doWork().

Here is a sample output without any dynamic QoS runtime enhancement.

big=59,543,287
small=59,535,999

What needs to happen now is for the QoS system to dynamically prioritize the execution of Worker.doWork() across the 16 threads based on the change in the value counter with every loop execution and within the metering context of the enclosing Worker.run() method. Here are the configuration entries required to achieves this with OpenCore’s metering runtime.

First disable the clock.time meter and create a meter mapped to a counter named value.

jxinsight.server.probes.meter.clock.time.include=false
jxinsight.server.probes.meter.counters=value

Disable the default enabled strategy probes provider as we don’t want the hotspot metering strategy disabling our low latency Worker.doWork() method.

jxinsight.server.probes.strategy.enabled=false

We need metering to be performed but not at a process or aggregated group level.

jxinsight.server.probes.global.enabled=false
jxinsight.server.probes.aggregates.enabled=false

Now enable the optional QoS probes provider extension.

jxinsight.server.probes.qos.enabled=true

Lets define a resource named $ with priority queuing on resource reservations which happens at the time a method begins to execute.

jxinsight.server.probes.qos.resources=$
jxinsight.server.probes.qos.resource.$.queue.priority.enabled=true

The reservation method used in determining the required reserve value is lease which needs for a meter to be associated with the resource.

jxinsight.server.probes.qos.resource.$.reservation=lease
jxinsight.server.probes.qos.resource.$.meter=value

A fast lane is now defined with a lower bound of 100 on each individual reservation matching and passing through it.

jxinsight.server.probes.qos.resource.$.lanes=fast
jxinsight.server.probes.qos.resource.$.lane.fast.lower=100

Reservations passing through the fast lane get a priority level promotion of 1. The default is 0 for all services without an explicit priority level defined.

jxinsight.server.probes.qos.resource.$.lane.fast.priority.level.promotion=1

The QoS policing is now applied to all probes matching one or more metering groups listed against our worker service which by default is automatically associated with our global resource $.

jxinsight.server.probes.qos.services=worker
jxinsight.server.probes.qos.service.worker.name.groups=QoS$Worker

Running the sample code again but this time with the above entries added to a jxinsight.override.config file results in the following output.

big=110,406,039
small=1,100,306

Same code but different execution behavior driven by a value measurement!!!

Note: I used OpenCore 6.2.M19.1 during testing which has a minor fix for the lease reservation method.

One question that came up a number of times was “Could OpenCore’s QoS for Apps technology be used in such environments and if so does metering need always be performed?”. Well with the release of OpenCore 6.2.M19 today I am delighted to be able answer YES and NO to each part respectively.

Note: Not only are we the only viable performance measurement solution provider in low latency Java environments we are also the only active performance management solution provider.

To demonstrate this I have created a small sample application which performs a busy processing loop making calls to two possible Callable implementations – VIP and Guest. In its main thread it spawns a number of threads each calling repeatedly into one of the two Callable implementations.

Here is the implementation of both Callable‘s. Yes they are exactly the same and with an execution cost of approximately 1 microsecond.

Here are the outputs from an instrumented run without any meters included and hence no activity metering.

vips=118,790,243
guests=119,123,397

Fairly evenly distributed.

Now if you were asked to make VIP perform more work than Guest in a heavy loaded environment with some level of CPU resource contention/starvation without changing the processing loop or number of threads apportioned you would probably change the Thread creation code to call Thread.setPriority(int) with different values for VIP and Guest workers. Here are the results setting VIP threads to MAX_PRIORITY and Guest threads to MIN_PRIORITY.

vips=157,162,800
guests=80,044,896

The different thread priority levels did make a difference (2:1) but what if it is not enough of a difference required to justify VIP status?

Here is how to achieve a similar prioritization using OpenCore’s qos probes provider extension by creating a logical resource, cpu, in our QoS model representing execution/processor time with unlimited capacity but with prioritization enabled and classifying each Callable.call implementation with a different QoS service and priority level. More importantly no software metering is performed.

Note: By default if a QoS service does not define a list of QoS resources it consumes then it uses the global QoS resource list.

jxinsight.server.probes.qos.enabled=true
jxinsight.server.probes.qos.forwarding.enabled=false
jxinsight.server.probes.qos.resources=cpu
jxinsight.server.probes.qos.resource.cpu.queue.priority.enabled=true
jxinsight.server.probes.qos.services=vip,guest
jxinsight.server.probes.qos.service.vip.name.groups=QoS$VIP.call
jxinsight.server.probes.qos.service.vip.priority.level=1
jxinsight.server.probes.qos.service.guest.name.groups=QoS$Guest.call
jxinsight.server.probes.qos.service.guest.priority.level=0

Below are the results from running the above code again this time with QoS enabled and no code changes other than dynamic byte code instrumentation.

vips=210,552,741
guests=1,283,421

An impressive 210:1 ratio!! Admittedly a little bit extreme though this is a contrived but very demanding proof test involving single digit microsecond level QoS policing.

The last time I talked with Werner Vogels at the CloudConnect 2011 conference he mentioned that this area of concern accounted for as much as 50% of the code base and was a significant differentiator in the design, development and deployment of their services – distinguishing the men from the boys in the cloud computing space. If you look at many of the upcoming PaaS offerings such as VMware’s CloudFoundry/OpenPaas/vFrabic, CloudBees RUN@cloud and RedHat’s OpenShift, you will be challenged to find any hard evidence indicating a similar level of investment in self-observation and self-regulation. In fact you are more than likely to find nothing (or nothing of worth) other than sugar laced management dashboards appealing to newcomers needing some false sense of security, comfort and control running counter to an important driver of cloud computing adoption which can be summed up with “if we need to look at a dashboard it’s already far too late” or “web scale != human scale”.

Clearly what is needed is to make the cut in the cloud and ensure some degree of longevity, scalability and hopefully financial success is for more attention to be given to self-observation and self-regulation of software within the runtime themselves and at each layer in the execution stack and across applications and services (collocated or not).

Whilst it might be wishful thinking hoping for a standard mechanism/interface/protocol/library to be agreed upon and implemented across all runtimes and services (avoding a reinvention splurge and wasted effort) I do hope that the Java community might look to JXInsight/OpenCore‘s extensible activity based metering engine as the basis for this new era in software resilience engineering and runtime execution adaptation and control.

Note: We have pushed for a standardization around OpenCore’s metering & metric technology with IBM, Sun, and even Oracle though with much less success than we had hoped for but we are still confident this might change in time.

OpenCore would deliver many of the much needed capabilities, and more, to applications without having to invest vast amounts of engineering time in polluting & possibly incorrectly altering existing codebases and application processing logic.

QoS for Apps
http://opencore.jinspired.com/?page_id=3058

CARS
http://opencore.jinspired.com/?page_id=3063

I.AM
http://opencore.jinspired.com/?page_id=3843

OpenCore – A Cloud Cortex for PaaS
http://opencore.jinspired.com/?p=3039

Using OpenCore’s Open API one can very quickly estimate the cost of various options and extensions to metering runtime without having to go through a lot of trial and error in pre-production and production.

Here is a top section of a code listing which I am going to use in estimating the latency cost for two key performance components of the metering runtime – meters and probes.

In the first part of the code listing the list of meters installed are iterated and individually calibrated in the sense that we are determining the degree of instrument error.

In the second part of the code listing above 5 different named probes are calibrated with each subsequent one having an extended name which in our metering runtime translates into a hierarchy of metering groups for metering (cost, latency) assignment.

The third part is pretty much the same as the second part except for a change in the probe name prefix and the usage of a disabled label which will be explained later.

Here is the method used to calibrate a Probes.Meter. Pretty straight forward calling Probes.Measure.getValue() repeatedly.

Note: Any metering of a probe will result in at least two getValue() calls.

To calibrate a Probes.Probe the Probes.Context, representing the calling thread, is used to create an instance returned via the begin() method. The end() method is then immediately called on the instance.

This last calibrate() method is near identical to the last method except for the disablement check that is performed before the begin() and end() methods are called if at all.

Note: Our AOP extension library performs a similar check but in a much more efficient manner that makes it become an no-op once the probe has been disabled.

Now all that needs to be done is to run up the Calibrate class with our OpenCore metering runtime.

java -agentpath:/OpenCore/bin/osx-32/libjxinsight.jnilib=prod -cp /OpenCore/lib/opencore.jar:. Calibrate

Here is a sample output of an execution.

clock.time=44ns

com=18ns
com.acme=18ns
com.acme.app=15ns
com.acme.app.sys=15ns
com.acme.app.sys.comp=15ns

org=0ns
org.acme=0ns
org.acme.app=0ns
org.acme.app.sys=0ns
org.acme.app.sys.comp=0ns

You might be surprised with the probe results considering the latency cost of the wall clock time access (44ns x 2) but what has happened is the hotspot metering strategy enabled by default has disabled the metering of the firing probes because of their cost profile. This also explains why the cost overhead drops to zero (naturally with rounding) in the second probe calibrate() method.

Lets disable the hotspot metering strategy altogether by adding the following to our jxinsight.override.config file.

Note: With this disabled I will not present the output the second probe metering calibrate results.

jxinsight.server.probes.strategy.enabled=false

Here is a sample output of an execution.

clock.time=44ns

com=143ns
com.acme=150ns
com.acme.app=164ns
com.acme.app.sys=174ns
com.acme.app.sys.comp=187ns

We can now experiment with different type of meters. Lets disable the default clock.time meter and used the alternative jvm.clock.time meter which calls to System.nanoTime().

jxinsight.server.probes.strategy.enabled=false
jxinsight.server.probes.meter.clock.time.include=false
jxinsight.server.probes.meter.jvm.clock.time.include=true

Here is a sample output of an execution.

jvm.clock.time=40ns

com=140ns
com.acme=146ns
com.acme.app=156ns
com.acme.app.sys=164ns
com.acme.app.sys.comp=174ns

Lets now turn off aggregate group metering.

jxinsight.server.probes.strategy.enabled=false
jxinsight.server.probes.meter.clock.time.include=false
jxinsight.server.probes.meter.jvm.clock.time.include=true
jxinsight.server.probes.aggregates.enabled=false

Here is a sample output of an execution.

jvm.clock.time=39ns

com=139ns
com.acme=126ns
com.acme.app=123ns
com.acme.app.sys=123ns
com.acme.app.sys.comp=123ns

We can even turn off the global (consolidated) metering if all we care about is thread level metering information.

jxinsight.server.probes.strategy.enabled=false
jxinsight.server.probes.meter.clock.time.include=false
jxinsight.server.probes.meter.jvm.clock.time.include=true
jxinsight.server.probes.aggregates.enabled=false
jxinsight.server.probes.global.enabled=false

Here is a sample output of an execution.

jvm.clock.time=39ns

com=108ns
com.acme=107ns
com.acme.app=107ns
com.acme.app.sys=107ns
com.acme.app.sys.comp=107ns

We can also see the impact in adding optional data collectors to the metering runtime. Here I am enabling the stack probes provider which allows efficient capturing and inspection of the probe stack across all metered threads at any moment in time.

jxinsight.server.probes.strategy.enabled=false
jxinsight.server.probes.meter.clock.time.include=false
jxinsight.server.probes.meter.jvm.clock.time.include=true
jxinsight.server.probes.aggregates.enabled=false
jxinsight.server.probes.global.enabled=false
jxinsight.server.probes.stack.enabled=true

Here is a sample output of an execution.

jvm.clock.time=39ns

com=125ns
com.acme=124ns
com.acme.app=124ns
com.acme.app.sys=123ns
com.acme.app.sys.comp=123ns

We can even eliminate metering entirely but still have the ability to inspect the probe stack at any time from a console or within the runtime itself.

jxinsight.server.probes.strategy.enabled=false
jxinsight.server.probes.meter.clock.time.include=false
jxinsight.server.probes.meter.jvm.clock.time.include=true
jxinsight.server.probes.aggregates.enabled=false
jxinsight.server.probes.global.enabled=false
jxinsight.server.probes.stack.enabled=true
jxinsight.server.probes.stack.forwarding.enabled=false

Here is a sample output of an execution.

jvm.clock.time=39ns

com=33ns
com.acme=33ns
com.acme.app=24ns
com.acme.app.sys=21ns
com.acme.app.sys.comp=21ns

With OpenCore’s metering runtime you have complete cost transparency along with unprecedented control!!!

There are a number of reasons for this but probably the most common is that the vast majority of BTM products are basically APM products re-branded and re-marketed (nearly always overnight). This is especially obvious in products when the definition of a business transaction constitutes of 1. a simple relabeling of code namespaces (packages, classes and methods) to something more (business) user friendly, 2. limiting performance measurement to just entry (and sometimes exit points) across processing tiers and 3. applying expensive state inspection of the call arguments at the entry point of a what is basically a trace path.

Rarely is there any actual business context or measurement such as the value or cost associated with the request/workflow processing. All such solutions are still largely focused on measuring latency. Which granted is critical for most businesses though not complete or comprehensive at least in a business context.

In a previous article, Business Transaction Management to Business Transaction Metering, I showed how to add such business context. In this article I am going to show how the business classification aspect can be solved using OpenCore’s metering engine extensions and exposing such classifications as Behavioral Trait Metering Metrics (BTM 2.0).

To make this a little bit difficult (for other vendors) the behavioral trait classification will only be possible at the point of completion of the request code execution. The following code mocks a business transaction which can successfully complete all required processing steps or fail in only partially executing all required steps (ignoring ordering concerns).

Below is a snapshot of a metering model that was taken during the metered execution of the above code using the following jxinsight.override.config properties file.

jxinsight.server.probes.strategy.enabled=false
jxinsight.server.probes.console.enabled=true

From the metering table above it can be seen that 28,715 “business transactions” were processed. But how many actually completed successfully, which in our behavioral trait definition means executing all 3 steps, can’t be determined because the code above allowed for steps to be executed repeatedly or skipped on a per request basis.

All that can be said is that the maximum possible number of requests processed completely is at most 14,260 – the number of doTearDown metered probe firings. But this number is likely to well overstated because in the code above it was possible to execute doTearDown without ever executing doSetup or doWork.

To turn the metering information into metrics sampled at fixed intervals all that is required is to add the following to the jxinsight.override.config file.

jxinsight.server.probes.metrics.enabled=true
jxinsight.server.metrics.console.enabled=true

With the metrics extension to the metering runtime enabled we get 4 metrics for every process level metering with a metering created for each probe and meter pairing. Considering that OpenCore can very easily create 10,000-100,000 metering objects per process this is likely to be overkill by no stretch of the imagination and still its not possible to answer our previous question.

Lets create two behavioral trait based metrics that are more business aligned (and actionable). One metric that tracks the number of requests that completed all steps. Another metric that tracks the number that partially execute the required steps.

There are a number of ways this can be achieved via the many extension points in the OpenCore metering runtime. Here I elected to use the optional interceptor probes providers and implement both ProbesInterceptorFactory and ProbesInterceptor interfaces.

Here is were the behavioral analysis is done in the Interceptor.end(...) method called by the runtime on completion of a metered probe which in our case translates to a Java method.

When the Traits.process() method is initially invoked a SavePoint is created of the current threads metering data. This SavePoint is then used to generate a ChangeSet when the Traits.process() method completes containing all the metering changes that occurred between the start and end points within the current thread. The ChangeSet is then searched for ChangePoint‘s matching the name of each process step. If all steps are present then we can be sure that the process completed its entire execution flow. If one or more for the step’s can’t be found in the ChangeSet then that particular invocation is deemed to have failed.

Here is the configuration I used to automatically install our behavioral trait interception routine and automatically register counters as same named metrics with the OpenCore metric runtime.

jxinsight.server.probes.strategy.enabled=false
jxinsight.server.probes.interceptor.enabled=true
jxinsight.server.probes.interceptors=traits
jxinsight.server.probes.interceptor.traits.factory.class=\
 com.jinspired.opencore.sample.traits.InterceptorFactory

jxinsight.server.metrics.counters=\
 behavioral.trait.success.count,behavioral.trait.failure.count
jxinsight.server.probes.console.enabled=true
jxinsight.server.metrics.console.enabled=true

Finally a much more simplified sampled metrics table listing.

Note: The final figure was 1,613 successful executions out of a total of 28,715 executions.

The above is only the tip of the iceberg to what’s possible with OpenCore’s metering runtime that puts all this data within hands (local call) reach. There are simply no limits to how software behavior and its resource consumption can be classified unlike other solutions which offer simplistic dialogs in defining business transaction matching criteria.

Check-out these links to see what lies beyond the door once you push ahead with OpenCore’s revolutionary approach to real-time software analytics and management.

Automated Performance Management starts with Software’s Self Observation

OpenCore Open API Samples

OpenCore Probes – Data in Real Time (DIRT)

CARS = Cost Aware Runtimes and Services

Architectural Enforcement with OpenCore Probes

Listening to Software as it Executes

Before I present the visualization in context lets look a the metering models in tabular form generated by the default hotspot strategy and budget probes provider. The first one up is hotspot.

Here is the second metering model generated by the new budget probes provider which operates similar to hotspot but with additional minimum overhead guarantees needed in performance monitoring of extremely low latency applications such as financial trading platforms.

If you look at the tree root you’ll see that the overall count of fired probes that were metered has been reduced slightly within this new option to the metering runtime (which is good but to be explained in another posting). This seems to be the case for the metering totals. So it’s looking good but could it be more beautiful and immediate without resorting to generating a comparison tree table of every inner and outer tree node especially when the expanded tree structure is wide and deep and most totals will deviate to some degree.

Well during my perusal of both snapshots I became very confident in my initial test assessment because of a striking and memorable visual pattern that was depicted repeatedly in the console as I navigated both model hierarchies. Here is the hotspot metering table this time with the tree visualization not cropped in the screenshot.

And here is the budget metering table with its tree visualization. For me nothing could be more immediate and helpful than the repeating of this pleasing and emotive visual pattern.

Here is another nested part of the hotspot metering visualized.

Again without looking at names and numbers I see the very same visual pattern in the budget version confirming the similarity of the models which is great from the point of view of our testing.

Want to see more? Check out the visualization in action. We think its pretty amazing and very original.

But how can the depth of a call stack be modeled as a meter when meters are basically thread specific cumulative counters and the depth of a call stack goes up and down. Once we hit the maximum stack depth the meter would never change and thus no further metering assignment would be made against those activities on the metering stack that did have a stack depth but not one that exceeded a previous maximum recorded for the thread. The trick is not to see the current value of a meter as the stack depth – but the delta from some starting point.

Metering is primarily focused on the change that occurs between the beginning and ending of a metered activity. The meter readings themselves are largely irrelevant – at most secondary. What is required is that the maximum stack depth be tracked within the current execution context of a service entry point call.

Here is how this can be achieved using OpenCore’s optional interceptor probes provider and its Open API extension point interfaces. The code only ever increases the underlying value of the resource measure registered, accessed by the getValue() method, when the stack depth has been exceeded. The current depth in terms of metered activities exceeds the max recorded. Both the depth and max are reset following the completion of the outer most method which is the only method that is metered as indicated by the boolean expression in the return statement of the begin() method. The value field in fact represent the number of changes (increments) in the max field within and across multiple outer most method invocations.

Here are the system properties I added to a jxinsight.override.config file to install the extension point and map the registered resource measure to a meter named "stack.depth".

jxinsight.server.probes.strategy.enabled=false
jxinsight.server.probes.interceptor.enabled=true
jxinsight.server.probes.interceptors=stack.depth
jxinsight.server.probes.interceptor.stack.depth.factory.class=\
com.jinspired.opencore.probes.meter.stack.depth.InterceptorFactory
jxinsight.server.probes.meter.resources=stack.depth

Below is a metering model collected following the completion of a JBoss 6.0 server startup and some simple web request to their noticeably slow responding web based mgmt console. Only the outer most method invocations are metered with the stack depth recorded for those methods not metered but called directly and indirectly from such metered points. From the metering model we can see that greatest stack depth recorded was 120 and this was during the invocation of process() method listed first. The minimum of 48 is smallest maximum stack depth for an invocation of the process() method.

Note: A benefit in modeling the stack depth as a meter is that we get additional statistical data including min, max and avg out of the box and that is before we enable more exotic statistical measurements afforded by OpenCore’s metering runtime which by default is configured with a bare-metal and extremely efficient runtime stack.

Typically one wants to limit the metering until a particular method appears on the stack representing the actual service rather than the underlying request dispatching middleware. This can be achieved using the entrypoint probes provider along with the custom extension point. The following additional properties ensure that metering does not occur until a method is called within the org.jboss.seam package or its sub-packages.

jxinsight.server.probes.entrypoint.enabled=true
jxinsight.server.probes.entrypoint.name.groups=org.jboss.seam

And here is the revised metering model. I did say that anything can be metered!! All that is required is a change in thinking.

I have created the following graphics to better explain the techniques which I will then apply to the original measurement problem that was raised in the discussion thread. Lets first start with the typical micro-benchmark. Its consists of a very simple test harness/driver class that repeatedly executes a code block that needs to be measured. The call stack is extremely small and only a single call-site is defined from which the code to be tested is called. Nothing could be simpler for a dynamic compiler and adaptive optimization execution engine and generally the memory bus is not taxed at all.

But real applications have much more call paths with much greater depths.

An obvious solution would be to try to recreate some of the real-world aspects of this execution in the test harness itself. Unfortunately this is time consuming and the results never as realistic.

Our novel approach is to bring the code over into some arbitrary host application environment. It’s incredibly cost effective and introduces that realism and randomness lacking in our contrived but still very useful micro-benchmarks. We achieve this by using the same engine that is used to meter and monitor the host application in development, test or production – OpenCore. When the metering engine is asked to meter the execution of code specific to the host application our micro test code is executed and measured. Then using the numerous metering strategies and optional dynamic metering probes providers we can control the mechanism and degree of randomness in the execution of the test.

Lets now see how this works in practice and applied to the measurement of Exception throwing overhead which is driven by the stack depth at which the Exception is created and the call stack accessed. Here is a simple micro-benchmark test harness that calls a method named call${n} which forwards the call onto another method named call${n-1}.

At the every end of the call path an Exception is constructed and an expensive call is made to acquire the call stack as an array of StackTraceElement instances.

Running the test harness with different call{n} invoked from the main methods for-loop I get the following average overhead costings. The stack depth is certainly the key performance indicator for the cost profile which is as one would expect considering what is performed when getStackTrace() is called.

50 => 43 μs
100 => 85 μs
200 => 173 μs
300 => 260 μs

But how representative is this of the real-world unit cost? Lets package up the code for injection into some real-world host application environment.

Below I have created custom OpenCore ProbesPluginFactory which registers a new named Probes.Measure instance with each thread specific Probes.Context created by the metering runtime.

Every time an arbitrary block of code within the host application is executed and metered the getValue() method will be called twice – before and after the code’s execution. I only want to measure the cost (wall clock time) once so I use the metering count of another meter, clock.time, to determine whether I should perform the micro-benchmark measurement and then update the thread specific measurement counter.

After packaging up the above class into a jar named, opencore-ext-est-time-meter.jar, and placing it on the classpath alongside opencore.jar I added the following system properties to a jxinsight.override.config file.

First I disabled the default enabled strategy probes provider as I am going to use an alternative dynamic metering probes provider.

jxinsight.server.probes.strategy.enabled=false

I then enabled the plugin probes provider and registered the custom  ProbesPluginFactory class.

jxinsight.server.probes.plugin.enabled=true
jxinsight.server.probes.plugins=est.time
jxinsight.server.probes.plugin.est.time.factory.class=\
org.opencore.meter.est.PluginFactory

Next I mapped a Probes.Meter to the custom Probes.Measure registered in the ProbesPlugin.apply() method with the Probes.Context.

jxinsight.server.probes.meter.resources=est.time

Because I want to control the point at which probe metering occurs in terms of stack depth I enabled our recently released depth probes provider and set the range to be 50(inclusive)-51(non-inclusive).

jxinsight.server.probes.depth.enabled=true
jxinsight.server.probes.depth.lower=50
jxinsight.server.probes.depth.upper=51

I then ran up a JBoss server instance in which the plugin was registered and clicked repeatedly on a few web pages in a Seam application – our host. Here is the est.time metering profile for those methods that were metered at depth 50.

From the table above I can see the average clock time cost is between 151-176 which is more than 4 times that of my micro-benchmark results. Lets try with a depth of 100.

jxinsight.server.probes.depth.lower=100
jxinsight.server.probes.depth.upper=101

Looking at the revised metering table below I can see that the average has increased in line with previous increases in the depth within our micro-benchmark but still the unit cost is more than 4 times the base unit cost recorded.

So what’s the average cost of a stack trace collection for a single thread with a call stack depth of 300. Its between 260 μs and 1,000 μs (1 ms). Now multiply that by a large number of threads and you can see why call stack sampling and exception logging are not suitable in production.

http://williamlouth.wordpress.com/2008/11/18/an-exceptionally-bad-start/
http://williamlouth.wordpress.com/2009/10/21/java-call-sampling-overhead-in-the-wild/

Customer: How can I configure OpenCore’s metering runtime to only meter activities which directly or indirectly cause other activities to be subsequently executed. I need to make this decision, via the metering strategy, at the point of execution of the “causing” activity and before the “resulting” activity is executed. In fact I am interested in applying this metering rule along the entire path of execution leading up to the resulting activity. I can’t know this at the time of the execution of the “causing” activity because of the many layers of indirection, the huge number of possible paths in our code base, and the amount of change that occurs with each release of the application metered…..Here are some metering [snapshots] we have collected….

Support: Looking over the metering models I can make out a typical 3-tier application workflow: [service] entry points -> [business] execution points -> [services, resources] exit points. To make this easier for both of us I’ve created a small simplified test application and made an metering recording (see attachment) of its execution which we can both playback at either end under different configurations of the metering runtime.

Customer: I assume I need the Java classes. Can you send them please?

Support: Using the recording log there’s no real need to share or execute the original test code and we can be sure the meter reading values will always be the same no matter which one of us executes it. Only the [cost] assignment of the meter reading will change with each playback we’ll execute…..Here is the code for 3 classes each representing a tier within the application….Again we are never going to use these classes as I have already recorded the calls made into OpenCore’s Open API by the instrumentation applied by the agent when I ran it up here locally.

Customer: Thanks. The code pattern looks quite similar to what we have. Just needed to some see code….. Still not sure of the “magic” recording & playback bit but I am sure I’m in for an interesting ride

Support: Lets first playback the metering recording with the console probes provider enabled so we can both explore the metering model in the OpenCore metering and metrics console and explore aspects of its execution flow and metering measurement.

Customer: Cool!

Support: To playback the metering recording you need to execute the following command line changing the native library part to the platform on which on the playback is being performed. You can actually record on one platform and playback on another.

java -classpath ${OPENCORE}/lib/opencore.jar -agentpath:${OPENCORE}/bin/[solaris|osx|linux|aix|hpux|zlinux]-[x86|sparc|ppc|parisc|ia64]-[31|32|64]/libjxinsight.[so|sl|jnilib]=prod com.jinspired.jxinsight.server.probes.RecorderPlayback ${recording.log}

or

java -classpath %OPENCORE%\lib\opencore.jar -agentpath:%OPENCORE%\bin\windows-[x86-[32|64]|ia64]\jxinsight.dll=prod com.jinspired.jxinsight.server.probes.RecorderPlayback %LOG%

Support: Before doing that create a jxinsight.override.config in the directory in which you will execute the above adding the following line to enable the console.

jxinsight.server.probes.console.enabled=true

Support: When the playback is completed a file ending with "-probes.ocs" will be exported. Mount the directory in the console and open up the model snapshot. It should look exactly like the following screenshot except for any sorting you might have in your local preferences.jdb file.

Customer: Exactly the same.

Support: Great. Now lets do another playback of the recording this time with the tracking probes provider enabled so you can more easily call out the execution behavior that its much more interesting in production for your reporting purposes. Here are the lines in my jxinsight.override.config file.

jxinsight.server.probes.console.enabled=true
jxinsight.server.probes.tracking.enabled=true

Customer: Looking at the tracking table in the console here I can see the path of execution which I would like to meter exclusively. Its EntryPoint.[main()->doWork()->delegate()]->ExecPoint.[doWork()->delegate()]->ExitPoint.doWork(). Of course this needs to be applied to different paths, classes and methods names but with similar packaging in our codebase.

Support: So the metering strategy needs to be configured to not meter activities which do not call into, directly or indirectly, the ExecPoint.doWork() method?

Customer: Yes and no. We don’t want to meter any activity that does not call into any method in the opencore.outbound.* package.

Support: I see. Lets create a custom meter, exit.count, that is based on the clock.time metering of any metered method in opencore.outbound.* package. We are going to meter the metering itself. Here are the entries in my jxinsight.override.config file

jxinsight.server.probes.meter.probes=exit
jxinsight.server.probes.meter.probe.exit.group=opencore.outbound
jxinsight.server.probes.meter.probe.exit.meter=clock.time
jxinsight.server.probes.meter.probe.exit.name=exit.count
jxinsight.server.probes.meter.probe.exit.statistic=count

Customer: Looking at the metering for the exit.count meter in my model I can now clearly point out which metered activities [methods] do not call into the opencore.outbound.* package. Those with a blank [zero] in their Total column. Yes?

Support: Exactly. Now lets configure the metering strategy to base is decision making on this new custom meter and not the default one of clock.time. We are going to us the past metering behavior to alter the future metering behavior at runtime. No developer guesswork here – based entirely on actual behavior. Please add the following to your existing configuration.

jxinsight.server.probes.strategy.hotspot.meter=exit.count
jxinsight.server.probes.strategy.hotspot.threshold=1
jxinsight.server.probes.strategy.hotspot.statistic=avg

Customer: In my new metering model, generated by the playback, I see that the overall metering count has dropped because the hotspot metering strategy has disabled a number of probes that never made calls into the opencore.openbound package.

Support: Yes. You are basically parameterizing the training of the metering runtime.

Customer: Tracking is looking good!

Customer: And my clock.time metering model looks much better. Fantastic!

Customer: So once this is up and running for a much longer period of time in production only those methods that consistently make outbound (based on mapping to our package structure) calls will continue to be metered and costed?

Support: Correct. By default after every 1,000 metered firings of a probe the hotspot metering strategy will decide whether to continue to meter a named activity or not. You can change this with the following system property if you wish to make this determination much sooner or later in the course of your production monitoring.

jxinsight.server.probes.strategy.hotspot.warmup=100

Customer: Looking over the tracking table again I realized we are not really interested in metering the methods in the opencore.outbound.* that are not invoked directly by external packages. I only really need the ExitPoint.doWork() to be metered. It would be great if we could exclude prepare(), delegate() and cleanup(). Again these methods would not be easily discernible from our large and complex codebase even in just this package – that’s if everything remained constant for just one day ;-(.

Support: No problem. We can use the optional tierpoint probes provider to solve this without you having to attempt to list every single possible surface point in this package. Here is what I added to my local configuration to create a tier which restricts metering at runtime to only the tier’s surface points:

jxinsight.server.probes.tierpoint.enabled=true
jxinsight.server.probes.tierpoint.tiers=exit
jxinsight.server.probes.tierpoint.tier.exit.name.groups=opencore.outbound

Customer: Wow. The prepare(), delegate() and cleanup() methods in the ExitPoint class are not metered any more. Is there anything this thing can’t do?

Support: If it was not able to do something today it would probably be able to do it tomorrow . It is incredibly extensible.

Customer: One of our engineers here was wondering whether this same technique could be applied to only meter those methods that call directly into the opencore.outbound.* package and the methods called themselves. We can’t always determine this because of the use of reflection and some other layers on indirection that we don’t instrument today. In the recording this would be ExecPoint.delegate() and ExitPoint.doWork(). We tried changing the hotspot statistic to to inherent.avg but that did not work. Am I pushing it to its limit?

Support: That is because the metering "Total (I)" column for the exit.count meter is blank [zero] for all other probes except those in the opencore.outbound.* package. Not to worry lets change the definition of our custom meter, exit.count, keeping everything else pretty much the same. The following maps our meter to a metering queue which increments a [thread specific] running count on every non-reentrant call into a metering group which can represent a package, class or method in our model.

jxinsight.server.probes.strategy.hotspot.statistic=inherent.total
jxinsight.server.probes.queue.enabled=true
jxinsight.server.probes.meter.queues=exit.count
jxinsight.server.probes.meter.queue.exit.count.group=opencore.outbound

Customer: Strange. That change seems to have disabled the ExitPoint.doWork() method and re-enabled the ExitPoint.[prepare(),work(),prepare()] methods.

Support: Oops. I forgot to exclude those probes in the opencore.outbound.* package from being enhanced by the strategy probes provider. Since the enqueuing happens before they are metered they will never see a change in the exit.count meter. The reason for some probes being re-enabled is because once the ExitPoint.doWork() probe is disabled those methods previously called by it now become surface points in the tier. This is why they are not disabled and why they have a count of 1000. I did say this is all dynamic. The only thing being pushed to its limit here is both of our mental capacities . To OpenCore its a stroll in the park. Add the following.

jxinsight.server.probes.strategy.filter.enabled=true
jxinsight.server.probes.strategy.filter.exclude.name.groups=opencore.outbound

Customer: I am in awe…..

At the moment there are two major Ruby VM camps. Those VM’s written in C/C++ versus a single one written in Java which itself is executed by a runtime written primarily in C/C++. Each camp has its own performance gains and losses but for the most part its a pretty close race except until it comes to production monitoring in which there is a clear winner.

Below is a Ruby class I am going to use to determine and compare the unit cost of New Relic, which is the most touted and overrated (you’ll soon see why) monitoring solution for C/C++ based Ruby VM’s though its written in Ruby, with JINSPIRED’s OpenCore, which is the highest performing metering/monitoring solution in Java with unique native JRuby-to-Ruby integration.

New Relic offers two means in which to instrument a Ruby class for performance measurement and monitoring purposes. Here is our Ruby class sprinkled with instrumentation calls to the first of these. Both Loop.call() and Loop.run() are instrumented.

The second way of collecting performance data with New Relic is using a transaction tracer. This generally collects much more detailed information related to invocation patterns than the simple metric aggregating method tracer though in our test case it has very little work to do because its only applied to the empty Loop.call() method.

This is the best, best case for such a collector. In fact when it comes to more contextual traces such as SQL queries the overhead is substantially higher but this overhead can be masked if your database is slow performing.

Here are the New Relic “very best case” overhead costs per call invocation running the two instrumented versions of our Loop Ruby class on two different Ruby VM’s after subtracting the cost in executing a non-instrumented version of the Loop.call() method. Clearly your choice of instrumentation technique has a bearing on the winning Ruby VM though I would question whether any application could afford such excessive overhead.

It took less than 1 sec to perform a single Loop.run() call without any instrumentation. With transaction tracing it took 67 minutes!!!

For the OpenCore comparison benchmark run no code changes are required. All that’s needed is the setting of the environment variable, JRUBY_OPTS, to

-J-agentpath:/OpenCore/bin/osx-32/libjxinsight.jnilib=prod -J-javaagent:/OpenCore/bundle/jruby/shell/opencore-ext-aj-javaagent.jar

and VERIFY_JRUBY to true.

I performed two test runs with OpenCore’s metering engine. The first run had our dynamic cost aware hotspot strategy enabled, which is the default setting for production, and the second run with it disabled. The difference between both OpenCore test runs is that after a number of Loop.call() invocations the hotspot strategy would effectively disable all further metering of the instrumented method because of its runtime cost profile. The instrumentation would be still present but short-circuited.

The overhead difference is huge giving JRuby a significant lead over all Ruby VM implementations in being able to perform between 4,000 and 83,000 more OpenCore metered call invocations for every single traced call invocation performed by New Relic.

OpenCore + JRuby offers Ruby developers deploying applications into production with greater code coverage, lower overhead, less risk, as well as savings in computing costs and resource consumption. More importantly it eliminates the guess work in deciding what should be instrumented and measured via the built-in cost awareness and self regulation of the metering runtime.

Here is a screen video showing how to use both the JRuby built-in profiler and OpenCore’s JRuby-to-Ruby metering extension.

Software
JINSPIRED OpenCore 6.2.M7
New Relic RPM 2.13.5.beta4

java version "1.6.0_22"
Java(TM) SE Runtime Environment (build 1.6.0_22-b04-307-10M3261)
Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03-307, mixed mode)


Environment
Model Name: iMac
Model Identifier: iMac11,1
Processor Name: Intel Core i7
Processor Speed: 2.8 GHz
Number Of Processors: 1
Total Number Of Cores: 4
L2 Cache (per core): 256 KB
L3 Cache: 8 MB
Memory: 8 GB
Processor Interconnect Speed: 4.8 GT/s