The StreamBase Event Processing Blog

On CEP Standards Confusion and Progress

(Richard Tibbetts is the Chief Architect for StreamBase, co-author of a recent paper with Oracle entitled Towards a Streaming SQL Standard, and StreamBase's representative to the Event Processing Technology Symposium, or EPTS).

There has been swift and strong response to the paper that Anand Srinivasan (from Oracle) and I presented last month at VLDB in Auckland. The paper, Towards a Streaming SQL Standard (download a PDF of the paper), seems to have touched a nerve in the community. All of a sudden, everyone is very excited about standards, and has an opinion to share. Some, such as Louis Lovas of Apama, perhaps concerned about being left behind, have been predictably negative and discouraging.  Opher Etzion from IBM shared a balanced, constructive point of view.  And then others have used this advancement as yet another excuse to re-hash the tired debate about "what is CEP?"  (we have provided no link to these discussions because the debate is pointless).

I'd like to address some of the mis-perceptions about this paper, because it's easy and deceptive to leap to the conclusion that "Oracle and StreamBase have proposed a new standard."   Instead, there are two key points to bear in mind when reading our work:

#1. This is an academic paper.  The goal of this paper is to advance standardization discussions by raising the processing model as a core element of any CEP standard. With event processing, when the results are produced can be as important as what the results are. That is, the same answer, produced later, is not the same answer. In SQL standardization, which deals with finite result sets, timing has not been a problem. In CEP however, it is a difference between implementations. The paper doesn't take a position on which existing approach is best. It merely states the problem. The paper doesn't propose a standard. It proposes one solution to one problem in standardization, as part of a broader effort.  We think this is an important observation, revealed by hard work and research.  We leave it to the community to decide if this observation is as important as we think it is.

#2.  Standardization is a very slow process. StreamBase is dedicated to standardization.  But standardization is a long process, and we've been working on it for almost 5 years.  And we will continue that commitment.  It will be years before we have a complete complex event processing language standard, with multiple conforming implementations.  There will be open standards bodies, and lots of work between groups of vendors to resolve their differences. In the early stages, it will likely be the smaller groups that make more progress.  Sure, we have a bias:  we think SQL is a great foundation to start with.  But we're happy to debate all approaches, and examine deeply the tradeoffs, just like we did with the Oracle research.

So StreamBase hasn't declared "victory" in standards; we are simply moving the discourse forward. I'm pleased that the community has responded strongly, because reaction means the debate has advanced.  We think thoughtful standards research is the best way to continue to promote standardization, and the rising tide of CEP, because a rising tide lifts all boats.  And we encourage all CEP vendors to do the same.

Getting Back to Use Cases and Patterns for CEP

Giles Nelson from Apama blogged today about how depressed he was to see a useless debate about acronyms, SOR, and CEP.  But unfortunately Giles threw the baby out with the bath water!   My initial post was about why CEP is used by Smart Order Routing applications, and that's a useful - no, critical - discussion.

When I ran Apama, the focus of my blog was use case discussion, patterns, requirements, best practices,  and descriptions of the references customers that illustrated CEP at work.  I think the community has lost this voice, and we need to bring it back.  There are a lot of good things happening at SteamBase, Apama, Aleri, Coral8, and other CEP vendors.  It's our job to educate, although I'm guilty of jumping into the bath water about the difference between Routers and SOR. 

So I'm going to RE-post the baby (the SOR use case discussion), and throw out the bath water (the Router debate).  I hope the blogosphere joins in on other use cases and positive discourse about how CEP is being used.

Towards a CEP Standard

There has been a lot of discussion in community groups, at conferences, and message boards over the years - back and forth about the best way to achieve progress on standards in the complex event processing (CEP) space.   This week StreamBase announced a partnership with Oracle - that the two companies have come together to research and publish on ways to reconcile a few of the approaches that have been developed for CEP.  The paper is being presented by our chief architect, Richard Tibbetts, at the VLDB conference this week.   We'll be blogging about this research in the future.   Here is the press release and a few links to coverage:

• Press Release: StreamBase and Oracle Partner on Complex Event Processing Standard Approach

• Banking Technology News coverage of the announcement

• A-Team news coverage

In the past perhaps StreamBase would have presented this news as further evidence that SQL has dominated the world of CEP as the no-brainer winner;  notice this is not the tenor of the release at all.  Instead, this research has taken years to develop, and the thoughful presentation of ideas in the paper is intended to provoke a discourse that can keep us on a path of convergence as CEP continues to become more and more important in the enterprise IT landscape. 

Stay tuned - more to come on this topic (post-script to this post, here is a post from Richard Tibbetts on the paper)!

StreamBase 6.0 Product Analysis and Thoughts on Leadership

June is a big month for CEP product releases, and the past few weeks have been no exception:  Aleri  announced version 3, Apama announced version 4, and StreamBase announced version 6.0.  Unsurprisingly, all these announcements contain proclamations of leadership.  Aleri's headline claimed a better look and feel, and Apama announced integrated sample applications and unsubstantiated performance claims.  By contrast, I think StreamBase announced an important breakthrough:  the first visual debugger, profiler, and unit testing environment in the industry. 

One of the main reasons I chose to join StreamBase is that in my time in the CEP industry, I've heard from customers, prospects, and industry insiders that the technology is simply superb:  it's fast, it's open, it's scalable (multi-threaded), and it has adopted a language metaphor (SQL) that is standards-based and familiar to most developers.  The visual environment uniquely enables business users and IT alike.  

I think release 6.0 extends that lead.  The ability to debug, test, and profile event processing applications is absolutely critical to ensure fast, reliable, and managable deployment in large enterprises, as well as adoption by smaller firms that need to do more with less. 

The release contains the following:  

1. StreamBase Visual Debugger - One of the hidden problems with some CEP platforms is that the visual tools were added as an afterthought, and, as a result, they generate code that's impossible to test, support and debug.  This is not the case with StreamBase, that started with a visual development tool from the very core, which has made it an incredibly powerful, integrated platform on which to debug and analyze applications, because there is no code generation - what you see is what you get and it's what you debug. 
2. Profiling - Profiling provides the ability to test, profile, and provision software and hardware resources associated with a CEP applications.  Once again, with StreamBase profiling, you profile and optimize the actual code you designed, so the result is a better, faster, more reliable application.
3. Unit Testing -  As the CEP market matures, larger and larger development groups are involved in CEP application development, which motivates the need for more sophisticated group development capability to test applications.  StreamBase Unit Testing allows teams to work in parallel, and allows teams to reduce application errors during development.  Unit testing also enables business developers to more fully participate in application development by providing IT the ability to test code supplied by business analysts before putting it into production,

The release of these features further demonstrates that CEP platforms are coming of age and ready for enterprise adoption and large, group development.  Yes these platforms are mature - today.

Event Processing Technical Society (EPTS): Thanks, Opher

The press release finally hit the wire today announcing the EPTS.  The goal is simple:  promote awareness about event processing technologies.  Immediate reaction has ensued, both negative and positive.   To me, having been involved in many of these discussions, I think any step forward is a good step.

This has taken about 2 years to formally announce this group. And, although the spirit of this has been to work as a community, and not single anyone out, there's no doubt that without a determined leader in the group nothing would have happened - from my point of view that's Opher Etzion - so I'm going to simplys link to Opher's blog, and his perspective, on this release as the best source for perspective.  I remember meeting Opher in New York at IBM for the first time about 3 years ago and discussing a vague vision of vendors, analysts, and academics all coming together to help resolve disputes and push a common understanding.

Here's hoping that the EPTS continues to advance this very important field.

Thanks, Opher.

What Does it Mean to be Mature?

My rebuttal to Ivy Schmerken's article called Deciphering the Myths Around Complex Event Processing has drawn a surprising amount of controversy.  In Ivy's rebuttal to my rebuttal, she clarified that the point although CEP is used in mission critical systems in the world's largest banks, that it isn't "mature" because:

1. There isn't a CEP language standard
2. There is debate about whether some CEP engines have a TIBCO adapter
3. One CEP user had a problem that the CEP companies all focus on different application areas

These seem to be the basis for broadly concluding that "CEP isn't mature."   Tim Bass also agreed with the view that CEP isn't mature, but didn't cite why.

Wow - talk about glass half empty thinking!  But I think the issue here is we have to define what we're talking about, and examine the state of the market in a clear framework, rather than taking one data point (one panel discussion with one person) and extrapolating a broad conclusion.

First of all, let me start by clarifying:  of course CEP is not as "mature" if you compare it to, for example, J2EE app servers or relational databases.  But let's look a little deeper, and look at a well-known model - the diffusion of innovations model (originally developed in the 50's by Bohlen and Beal, popularized by Everett Rodgers in 1962, then by Moore's Crossing the Chasm in 1991.  The original research actually analyzed adoption of farming practices, and developed the following phases:

1. Innovators - had larger farms, were more educated, more prosperous and more risk-oriented
2. Early adopters - younger, more educated, tended to be community leaders
3. Early majority - more conservative but open to new ideas, active in community and influence to neighbors
4. Late majority - older, less educated, fairly conservative and less socially active
5. Laggards - very conservative, smalls farms and capital, oldest and least educated

I cited customers that have deployed CEP that fit the "early majority" characterization.  That is: "conservative, open to new ideas, active in the community and influential to neighbors."  Interestingly, on this point, there was no debate, yet STILL we leap to the statement that CEP isn't mature.

But let's look at the arguments for immaturity, namely:

• Some vendors don't have all the needed adapters
• There's no CEP language standard languages among vendors
• There are no interoperability standards
• My guess is that Tim would add best practices and reference architectures lacking (I agree)

On the other hand, there are many areas of maturity that enables the early majority to deploy, including:

• Battle-tested CEP server engines (multi-threaded, highly scalable)
• Adapters for interoperability (there are hundreds out there from vendors today and I will argue that CEP standards for interoperability are a red herring on another day)
• There is a vibrant community building around CEP on line
• Robust development tools for business analysts and IT
• The existence of robust debugging and profiling tools
• Language support for Java, C++ and graphical approaches
• Relational database connectivity
• Tick database connectivity
• Visualization / dashboards, connectivity to .NET, web dashboards, and more
• Simulation and testing environments
• Fault tolerance
• Management functionality
• State recovery

Wow that's a lot of stuff!   I'd call a platform that has all of these things mature, and there are definately a number of CEP platforms that have all of these.

Should the community focus on standards and interoperability more?  Yes, of course.  But just because there are different approaches to solving a difficult problem, and that the industry is starting to come together, we shouldn't feel compelled to thwart that growth with a claim that the products are not "mature" when they actually are in a lot of ways.

On the Evils of SQL and CEP

OK, now that I've settled in at StreamBase, it's time to start to address the number one question I get - this "Mark and Colin hate SQL" thing.  It's come up immediately and often in my first 5 days here, including, for example, this post from Brenda Michelson where she discussed the recent goings on at StreamBase.

The first thing I had to do is check what Colin and I actually said!  I found the original post in the Apama blog on the topic; ironically enough, Colin jumped in and responded.   So, here we are - right on the record:

On CEP and Standardization 

Re-reading this discourse, I am happy to report that both:

1) I still believe exactly what I wrote a year ago, and

2) There is no contradiction in now being at StreamBase

The point of this post, and my historical (apparently colorful and memorable) objection to StreamBase, was simply that they had run off and assumed that SQL was the only approach to a CEP language. The core of the issue is that there are many relevant ways to build a CEP language, including:

• Graphical point-and-click environments
• CEP rules
• CEP languages
• Java
• SQL-like syntax

There was one more point made in that post that bears repeating, although I think it's fitting to make one small edit:

At Apama [StreamBase], we will continue to avoid polarizing discussions about standards, and participate in the nascent standards discussions that are inclusive of all approaches.  We'll also continue to advocate other important issues like customer use cases, the business value of those use cases, architectural discussion about the relationship between EDA and SOA, continued work on the event processing glossary, and more.

Constructive discourse that unites, not divides, will help the tide rise for event processing, and lift all boats along with it - customers, vendors, and more.

Stay tuned for future posts on the beauty, elegance, and superiority of SQL for CEP ;-)

StreamSQL

As you get older you take for granted lessons learned much earlier in your career. Then when someone asks why such-and-such is the case you do a double take, and have to engage the question with integrity and fairness. If someone suggests that pentagonal wheels are better than round ones it’s worth at least a few moments reflection.

So I admit it. My basic assumption is that standards, be they de facto or de jure, are inevitable in pretty much all areas of technology. Trivially, though not conclusively, I would note that this is simply a pattern I have observed. We have standard protocols at every layer of the network stack. We have standard API’s to operating systems. We have compatibility standards for laptop computers. We have standard connectors for IO buses. There are standards in database systems, in middleware, in document formats, in eCommerce choreographies and in digital audio formats. I am pleased at the local standards in electrical outlets, but not at the state of global inconsistency – my assumption is that this will change. Once you start down the list it is hard to find things for which there are not standards, or in which there are not some emerging standards. These standards may take a year or two to establish themselves (eg J2EE took 12-18 months), but I have assumed an inevitability of standards in the kinds of things that companies like StreamBase do. Including in CEP.

An observed inevitability often reflects an economic reality. The principle is that markets generally move towards the eradication of inefficiencies. If standards-based approaches are more efficient for customers than proprietary ones then standards will win, over some time period. An example might serve us here: In a recent case of a customer switching their solution from another SQL-style CEP product to StreamBase as they moved from prototyping work to their deployment architecture, the replacement took about two or three days. You read that right. With very little retraining of people, minor changes around SQL syntax (and to be fair some differences in detail semantics), replacement of the Stream Processing/CEP server, and not much else the customer had switched vendors. If that sounds like the pattern you’ve seen with SQL databases, J2EE application servers, CORBA systems, or JMS servers, then that’s because the example is identical to what was seen in those markets.

When you go back and ask yourself why standards tend to emerge in technology you have to note that proprietary solutions involve enormously greater costs around vendor lock-in. By contrast standards force price/performance competition, and the customer wins. I believe there may be circumstances in which pentagonal wheels are better (moon landers?), but in the general case circular wheels are an inevitable solution.

For the record: StreamBase is committed to standards. We funded the development of the initial StreamSQL spec by all of the leading academics in the field of Stream Processing, including but not limited to Professor Mike Stonebraker and the four other professors that were founders of StreamBase. We put the StreamSQL spec into the public domain. We are working diligently with competitors, partners and customers to iterate that specification, including of course the commitment to invest to keep StreamBase compliant with whatever is decided. The fruits of this work will become apparent over forthcoming months.

Stay tuned!