The StreamBase Event Processing Blog

StreamBase 6.0 Product Analysis and Thoughts on Leadership

June is a big month for CEP product releases, and the past few weeks have been no exception:  Aleri  announced version 3, Apama announced version 4, and StreamBase announced version 6.0.  Unsurprisingly, all these announcements contain proclamations of leadership.  Aleri's headline claimed a better look and feel, and Apama announced integrated sample applications and unsubstantiated performance claims.  By contrast, I think StreamBase announced an important breakthrough:  the first visual debugger, profiler, and unit testing environment in the industry. 

One of the main reasons I chose to join StreamBase is that in my time in the CEP industry, I've heard from customers, prospects, and industry insiders that the technology is simply superb:  it's fast, it's open, it's scalable (multi-threaded), and it has adopted a language metaphor (SQL) that is standards-based and familiar to most developers.  The visual environment uniquely enables business users and IT alike.  

I think release 6.0 extends that lead.  The ability to debug, test, and profile event processing applications is absolutely critical to ensure fast, reliable, and managable deployment in large enterprises, as well as adoption by smaller firms that need to do more with less. 

The release contains the following:  

1. StreamBase Visual Debugger - One of the hidden problems with some CEP platforms is that the visual tools were added as an afterthought, and, as a result, they generate code that's impossible to test, support and debug.  This is not the case with StreamBase, that started with a visual development tool from the very core, which has made it an incredibly powerful, integrated platform on which to debug and analyze applications, because there is no code generation - what you see is what you get and it's what you debug. 
2. Profiling - Profiling provides the ability to test, profile, and provision software and hardware resources associated with a CEP applications.  Once again, with StreamBase profiling, you profile and optimize the actual code you designed, so the result is a better, faster, more reliable application.
3. Unit Testing -  As the CEP market matures, larger and larger development groups are involved in CEP application development, which motivates the need for more sophisticated group development capability to test applications.  StreamBase Unit Testing allows teams to work in parallel, and allows teams to reduce application errors during development.  Unit testing also enables business developers to more fully participate in application development by providing IT the ability to test code supplied by business analysts before putting it into production,

The release of these features further demonstrates that CEP platforms are coming of age and ready for enterprise adoption and large, group development.  Yes these platforms are mature - today.

Event Processing Technical Society (EPTS): Thanks, Opher

The press release finally hit the wire today announcing the EPTS.  The goal is simple:  promote awareness about event processing technologies.  Immediate reaction has ensued, both negative and positive.   To me, having been involved in many of these discussions, I think any step forward is a good step.

This has taken about 2 years to formally announce this group. And, although the spirit of this has been to work as a community, and not single anyone out, there's no doubt that without a determined leader in the group nothing would have happened - from my point of view that's Opher Etzion - so I'm going to simplys link to Opher's blog, and his perspective, on this release as the best source for perspective.  I remember meeting Opher in New York at IBM for the first time about 3 years ago and discussing a vague vision of vendors, analysts, and academics all coming together to help resolve disputes and push a common understanding.

Here's hoping that the EPTS continues to advance this very important field.

Thanks, Opher.

Real-Time P&L Use Case Requirements

Back to our exploration of another use case for CEP:  Real-Time P&L.  Last week, I published an introduction to RTP&L from a paper I did with Bob Giffords.  Tomorrow, Corwin Yu and I will hold a webcast:  The Best Practices of Real-time P&L (register here).   We'll explore the requirements and best practices of real-time P&L systems, and demonstrate the StreamBase Real-Time P&L Framework live.

What are the requirements of real-time P&L?  Today’s trader execution environment involves orchestrating many different broker, market, and in-house platforms, typically with multiple execution management (EMS) or order management systems (OMS) dedicated to different asset classes.

Consolidated and direct feeds from firms like Reuters, Bloomberg, Comstock and the Exchanges and ECN platforms need to be integrated along with specialist broker and in-house feeds and distributed to the appropriate analytical and trading systems. Trades too need to be tracked, not just back to orders but allocated to underlying instruments, portfolios and strategies if they are to inform decisions. OTC transactions may not be fully automated, so data collection for real time P&L will also need to link into the appropriate work-flow processes within the firm.

Some traders are even integrating elementized news feeds as well as reference data feeds like corporate action announcements into their pricing systems. These can be XML feeds from the news wire services to pull out and tag key event data or market sentiment indicators from the articles themselves.

Where needed for complex instruments, pricing models mediate the data streams to value them based on market reference prices. Indeed there are cases where multiple valuation sources are employed and these need to be cleansed of errors or outliers, normalised and harmonised into a consistent set of prices to be used for the P&L. Where appropriate, foreign exchange translations may be needed to bring all values into the firm’s reporting currency.

The P&L system next aggregates all the valuations and compares them to various baselines to work out the relative gains and losses. These can be rolled up to the book, strategy or portfolio level, and the firm may then apply alert or market signal filters.

On the output side these filters might trigger intraday alerts and generate analytics and graphical charts to enhance trader or investment manager control. Heads of desk may want to drill down into the detail and understand why the values are changing. Automated compliance auditing algorithms can be run and compliance officers may use this real-time view of trading activity to monitor that regulatory and prudential controls are enforced to satisfy demands for intra day due diligence.

As traders gain confidence, the outputs from the P&L calculation can be fed back to algorithmic trading engines as signals to generate automated orders using investment strategy models.

Next up:  A Case Study on Real-Time P&L.

What Does it Mean to be Mature?

My rebuttal to Ivy Schmerken's article called Deciphering the Myths Around Complex Event Processing has drawn a surprising amount of controversy.  In Ivy's rebuttal to my rebuttal, she clarified that the point although CEP is used in mission critical systems in the world's largest banks, that it isn't "mature" because:

1. There isn't a CEP language standard
2. There is debate about whether some CEP engines have a TIBCO adapter
3. One CEP user had a problem that the CEP companies all focus on different application areas

These seem to be the basis for broadly concluding that "CEP isn't mature."   Tim Bass also agreed with the view that CEP isn't mature, but didn't cite why.

Wow - talk about glass half empty thinking!  But I think the issue here is we have to define what we're talking about, and examine the state of the market in a clear framework, rather than taking one data point (one panel discussion with one person) and extrapolating a broad conclusion.

First of all, let me start by clarifying:  of course CEP is not as "mature" if you compare it to, for example, J2EE app servers or relational databases.  But let's look a little deeper, and look at a well-known model - the diffusion of innovations model (originally developed in the 50's by Bohlen and Beal, popularized by Everett Rodgers in 1962, then by Moore's Crossing the Chasm in 1991.  The original research actually analyzed adoption of farming practices, and developed the following phases:

1. Innovators - had larger farms, were more educated, more prosperous and more risk-oriented
2. Early adopters - younger, more educated, tended to be community leaders
3. Early majority - more conservative but open to new ideas, active in community and influence to neighbors
4. Late majority - older, less educated, fairly conservative and less socially active
5. Laggards - very conservative, smalls farms and capital, oldest and least educated

I cited customers that have deployed CEP that fit the "early majority" characterization.  That is: "conservative, open to new ideas, active in the community and influential to neighbors."  Interestingly, on this point, there was no debate, yet STILL we leap to the statement that CEP isn't mature.

But let's look at the arguments for immaturity, namely:

• Some vendors don't have all the needed adapters
• There's no CEP language standard languages among vendors
• There are no interoperability standards
• My guess is that Tim would add best practices and reference architectures lacking (I agree)

On the other hand, there are many areas of maturity that enables the early majority to deploy, including:

• Battle-tested CEP server engines (multi-threaded, highly scalable)
• Adapters for interoperability (there are hundreds out there from vendors today and I will argue that CEP standards for interoperability are a red herring on another day)
• There is a vibrant community building around CEP on line
• Robust development tools for business analysts and IT
• The existence of robust debugging and profiling tools
• Language support for Java, C++ and graphical approaches
• Relational database connectivity
• Tick database connectivity
• Visualization / dashboards, connectivity to .NET, web dashboards, and more
• Simulation and testing environments
• Fault tolerance
• Management functionality
• State recovery

Wow that's a lot of stuff!   I'd call a platform that has all of these things mature, and there are definately a number of CEP platforms that have all of these.

Should the community focus on standards and interoperability more?  Yes, of course.  But just because there are different approaches to solving a difficult problem, and that the industry is starting to come together, we shouldn't feel compelled to thwart that growth with a claim that the products are not "mature" when they actually are in a lot of ways.