The StreamBase Event Processing Blog

StreamBase 6.0 Product Analysis and Thoughts on Leadership

June is a big month for CEP product releases, and the past few weeks have been no exception:  Aleri  announced version 3, Apama announced version 4, and StreamBase announced version 6.0.  Unsurprisingly, all these announcements contain proclamations of leadership.  Aleri's headline claimed a better look and feel, and Apama announced integrated sample applications and unsubstantiated performance claims.  By contrast, I think StreamBase announced an important breakthrough:  the first visual debugger, profiler, and unit testing environment in the industry. 

One of the main reasons I chose to join StreamBase is that in my time in the CEP industry, I've heard from customers, prospects, and industry insiders that the technology is simply superb:  it's fast, it's open, it's scalable (multi-threaded), and it has adopted a language metaphor (SQL) that is standards-based and familiar to most developers.  The visual environment uniquely enables business users and IT alike.  

I think release 6.0 extends that lead.  The ability to debug, test, and profile event processing applications is absolutely critical to ensure fast, reliable, and managable deployment in large enterprises, as well as adoption by smaller firms that need to do more with less. 

The release contains the following:  

1. StreamBase Visual Debugger - One of the hidden problems with some CEP platforms is that the visual tools were added as an afterthought, and, as a result, they generate code that's impossible to test, support and debug.  This is not the case with StreamBase, that started with a visual development tool from the very core, which has made it an incredibly powerful, integrated platform on which to debug and analyze applications, because there is no code generation - what you see is what you get and it's what you debug. 
2. Profiling - Profiling provides the ability to test, profile, and provision software and hardware resources associated with a CEP applications.  Once again, with StreamBase profiling, you profile and optimize the actual code you designed, so the result is a better, faster, more reliable application.
3. Unit Testing -  As the CEP market matures, larger and larger development groups are involved in CEP application development, which motivates the need for more sophisticated group development capability to test applications.  StreamBase Unit Testing allows teams to work in parallel, and allows teams to reduce application errors during development.  Unit testing also enables business developers to more fully participate in application development by providing IT the ability to test code supplied by business analysts before putting it into production,

The release of these features further demonstrates that CEP platforms are coming of age and ready for enterprise adoption and large, group development.  Yes these platforms are mature - today.

Event Processing Technical Society (EPTS): Thanks, Opher

The press release finally hit the wire today announcing the EPTS.  The goal is simple:  promote awareness about event processing technologies.  Immediate reaction has ensued, both negative and positive.   To me, having been involved in many of these discussions, I think any step forward is a good step.

This has taken about 2 years to formally announce this group. And, although the spirit of this has been to work as a community, and not single anyone out, there's no doubt that without a determined leader in the group nothing would have happened - from my point of view that's Opher Etzion - so I'm going to simplys link to Opher's blog, and his perspective, on this release as the best source for perspective.  I remember meeting Opher in New York at IBM for the first time about 3 years ago and discussing a vague vision of vendors, analysts, and academics all coming together to help resolve disputes and push a common understanding.

Here's hoping that the EPTS continues to advance this very important field.

Thanks, Opher.

What Does it Mean to be Mature?

My rebuttal to Ivy Schmerken's article called Deciphering the Myths Around Complex Event Processing has drawn a surprising amount of controversy.  In Ivy's rebuttal to my rebuttal, she clarified that the point although CEP is used in mission critical systems in the world's largest banks, that it isn't "mature" because:

1. There isn't a CEP language standard
2. There is debate about whether some CEP engines have a TIBCO adapter
3. One CEP user had a problem that the CEP companies all focus on different application areas

These seem to be the basis for broadly concluding that "CEP isn't mature."   Tim Bass also agreed with the view that CEP isn't mature, but didn't cite why.

Wow - talk about glass half empty thinking!  But I think the issue here is we have to define what we're talking about, and examine the state of the market in a clear framework, rather than taking one data point (one panel discussion with one person) and extrapolating a broad conclusion.

First of all, let me start by clarifying:  of course CEP is not as "mature" if you compare it to, for example, J2EE app servers or relational databases.  But let's look a little deeper, and look at a well-known model - the diffusion of innovations model (originally developed in the 50's by Bohlen and Beal, popularized by Everett Rodgers in 1962, then by Moore's Crossing the Chasm in 1991.  The original research actually analyzed adoption of farming practices, and developed the following phases:

1. Innovators - had larger farms, were more educated, more prosperous and more risk-oriented
2. Early adopters - younger, more educated, tended to be community leaders
3. Early majority - more conservative but open to new ideas, active in community and influence to neighbors
4. Late majority - older, less educated, fairly conservative and less socially active
5. Laggards - very conservative, smalls farms and capital, oldest and least educated

I cited customers that have deployed CEP that fit the "early majority" characterization.  That is: "conservative, open to new ideas, active in the community and influential to neighbors."  Interestingly, on this point, there was no debate, yet STILL we leap to the statement that CEP isn't mature.

But let's look at the arguments for immaturity, namely:

• Some vendors don't have all the needed adapters
• There's no CEP language standard languages among vendors
• There are no interoperability standards
• My guess is that Tim would add best practices and reference architectures lacking (I agree)

On the other hand, there are many areas of maturity that enables the early majority to deploy, including:

• Battle-tested CEP server engines (multi-threaded, highly scalable)
• Adapters for interoperability (there are hundreds out there from vendors today and I will argue that CEP standards for interoperability are a red herring on another day)
• There is a vibrant community building around CEP on line
• Robust development tools for business analysts and IT
• The existence of robust debugging and profiling tools
• Language support for Java, C++ and graphical approaches
• Relational database connectivity
• Tick database connectivity
• Visualization / dashboards, connectivity to .NET, web dashboards, and more
• Simulation and testing environments
• Fault tolerance
• Management functionality
• State recovery

Wow that's a lot of stuff!   I'd call a platform that has all of these things mature, and there are definately a number of CEP platforms that have all of these.

Should the community focus on standards and interoperability more?  Yes, of course.  But just because there are different approaches to solving a difficult problem, and that the industry is starting to come together, we shouldn't feel compelled to thwart that growth with a claim that the products are not "mature" when they actually are in a lot of ways.

On the Evils of SQL and CEP

OK, now that I've settled in at StreamBase, it's time to start to address the number one question I get - this "Mark and Colin hate SQL" thing.  It's come up immediately and often in my first 5 days here, including, for example, this post from Brenda Michelson where she discussed the recent goings on at StreamBase.

The first thing I had to do is check what Colin and I actually said!  I found the original post in the Apama blog on the topic; ironically enough, Colin jumped in and responded.   So, here we are - right on the record:

On CEP and Standardization 

Re-reading this discourse, I am happy to report that both:

1) I still believe exactly what I wrote a year ago, and

2) There is no contradiction in now being at StreamBase

The point of this post, and my historical (apparently colorful and memorable) objection to StreamBase, was simply that they had run off and assumed that SQL was the only approach to a CEP language. The core of the issue is that there are many relevant ways to build a CEP language, including:

• Graphical point-and-click environments
• CEP rules
• CEP languages
• Java
• SQL-like syntax

There was one more point made in that post that bears repeating, although I think it's fitting to make one small edit:

At Apama [StreamBase], we will continue to avoid polarizing discussions about standards, and participate in the nascent standards discussions that are inclusive of all approaches.  We'll also continue to advocate other important issues like customer use cases, the business value of those use cases, architectural discussion about the relationship between EDA and SOA, continued work on the event processing glossary, and more.

Constructive discourse that unites, not divides, will help the tide rise for event processing, and lift all boats along with it - customers, vendors, and more.

Stay tuned for future posts on the beauty, elegance, and superiority of SQL for CEP ;-)

 

StreamSQL

As you get older you take for granted lessons learned much earlier in your career. Then when someone asks why such-and-such is the case you do a double take, and have to engage the question with integrity and fairness. If someone suggests that pentagonal wheels are better than round ones it’s worth at least a few moments reflection.

So I admit it. My basic assumption is that standards, be they de facto or de jure, are inevitable in pretty much all areas of technology. Trivially, though not conclusively, I would note that this is simply a pattern I have observed. We have standard protocols at every layer of the network stack. We have standard API’s to operating systems. We have compatibility standards for laptop computers. We have standard connectors for IO buses. There are standards in database systems, in middleware, in document formats, in eCommerce choreographies and in digital audio formats. I am pleased at the local standards in electrical outlets, but not at the state of global inconsistency – my assumption is that this will change. Once you start down the list it is hard to find things for which there are not standards, or in which there are not some emerging standards. These standards may take a year or two to establish themselves (eg J2EE took 12-18 months), but I have assumed an inevitability of standards in the kinds of things that companies like StreamBase do. Including in CEP.

An observed inevitability often reflects an economic reality. The principle is that markets generally move towards the eradication of inefficiencies. If standards-based approaches are more efficient for customers than proprietary ones then standards will win, over some time period. An example might serve us here: In a recent case of a customer switching their solution from another SQL-style CEP product to StreamBase as they moved from prototyping work to their deployment architecture, the replacement took about two or three days. You read that right. With very little retraining of people, minor changes around SQL syntax (and to be fair some differences in detail semantics), replacement of the Stream Processing/CEP server, and not much else the customer had switched vendors. If that sounds like the pattern you’ve seen with SQL databases, J2EE application servers, CORBA systems, or JMS servers, then that’s because the example is identical to what was seen in those markets.

When you go back and ask yourself why standards tend to emerge in technology you have to note that proprietary solutions involve enormously greater costs around vendor lock-in. By contrast standards force price/performance competition, and the customer wins. I believe there may be circumstances in which pentagonal wheels are better (moon landers?), but in the general case circular wheels are an inevitable solution.

For the record: StreamBase is committed to standards. We funded the development of the initial StreamSQL spec by all of the leading academics in the field of Stream Processing, including but not limited to Professor Mike Stonebraker and the four other professors that were founders of StreamBase. We put the StreamSQL spec into the public domain. We are working diligently with competitors, partners and customers to iterate that specification, including of course the commitment to invest to keep StreamBase compliant with whatever is decided. The fruits of this work will become apparent over forthcoming months.

Stay tuned!