The StreamBase Event Processing Blog

The Gartner Event Processing Conference

This week Gartner held its Event Processing conference, which was followed by the meeting of the EPTS group.   The first blogosphere impressions I've seen come from Marc Adler - Mark did his own presentation which I heard was very good and attended some others, including mine.  He wrote his thoughts up on his blog here.

I have received a number of requests for my slides from the presentation.  Feel free to contact me at [email protected] and I'll forward them along.

Stay tuned to this blog as more blogosphere comments are posted, I'll link to them in this post.

On CEP Standards Confusion and Progress

(Richard Tibbetts is the Chief Architect for StreamBase, co-author of a recent paper with Oracle entitled Towards a Streaming SQL Standard, and StreamBase's representative to the Event Processing Technology Symposium, or EPTS).

There has been swift and strong response to the paper that Anand Srinivasan (from Oracle) and I presented last month at VLDB in Auckland. The paper, Towards a Streaming SQL Standard (download a PDF of the paper), seems to have touched a nerve in the community. All of a sudden, everyone is very excited about standards, and has an opinion to share. Some, such as Louis Lovas of Apama, perhaps concerned about being left behind, have been predictably negative and discouraging.  Opher Etzion from IBM shared a balanced, constructive point of view.  And then others have used this advancement as yet another excuse to re-hash the tired debate about "what is CEP?"  (we have provided no link to these discussions because the debate is pointless).

I'd like to address some of the mis-perceptions about this paper, because it's easy and deceptive to leap to the conclusion that "Oracle and StreamBase have proposed a new standard."   Instead, there are two key points to bear in mind when reading our work:

#1. This is an academic paper.  The goal of this paper is to advance standardization discussions by raising the processing model as a core element of any CEP standard. With event processing, when the results are produced can be as important as what the results are. That is, the same answer, produced later, is not the same answer. In SQL standardization, which deals with finite result sets, timing has not been a problem. In CEP however, it is a difference between implementations. The paper doesn't take a position on which existing approach is best. It merely states the problem. The paper doesn't propose a standard. It proposes one solution to one problem in standardization, as part of a broader effort.  We think this is an important observation, revealed by hard work and research.  We leave it to the community to decide if this observation is as important as we think it is.

#2.  Standardization is a very slow process. StreamBase is dedicated to standardization.  But standardization is a long process, and we've been working on it for almost 5 years.  And we will continue that commitment.  It will be years before we have a complete complex event processing language standard, with multiple conforming implementations.  There will be open standards bodies, and lots of work between groups of vendors to resolve their differences. In the early stages, it will likely be the smaller groups that make more progress.  Sure, we have a bias:  we think SQL is a great foundation to start with.  But we're happy to debate all approaches, and examine deeply the tradeoffs, just like we did with the Oracle research.

So StreamBase hasn't declared "victory" in standards; we are simply moving the discourse forward. I'm pleased that the community has responded strongly, because reaction means the debate has advanced.  We think thoughtful standards research is the best way to continue to promote standardization, and the rising tide of CEP, because a rising tide lifts all boats.  And we encourage all CEP vendors to do the same.

On the Requirements of Smart Order Routing (SOR) and CEP

The requirements of Smart Order Routing (SOR) applications are a good fit for complex event processing (CEP) because SOR applications need to evaluate a confluence of events, detect complex patterns of market events and manage feedback loops of trade execution state in real-time.  And they have to do all of this in a very low latency environment, with a lot of events.

Let's back up and look at the motivation for SOR.  The reason banks use SOR is because the very structure of trading is changing dramatically, and has been for years.  Regulatory changes in the equities market have spurred the rise of trading venue alternatives - ATSs, ECNs - dark pools of liquidity.  Traditional trading exchanges are shutting down because of these cheaper electronic options.  In the U.S. market there are as many as 30 ECNs today.  And in the unregulated FX market, dozens of banks trade electronically with each other, and there will be 15 electronic FX venues by the end of the year.  Moreover, the rise in interest in dark pools of liquidity add yet another complex set of events that must be detected, aggregated, combined, matched, and manipulated in real time.  It's not uncommon for a single SOR system to connect to 10 or more markets and multiple asset classes.  Not only is this a confluence of events, it's a stunningly complicated environment in which to create a complex, real-time model in which to apply "simple" routing decisions. On this basis alone, SOR needs CEP.

Now let's look at processing complexity.  One example of complex sequences of events that can be managed by CEP is in the area of order and execution state management.  Orders in many trading liquidity pools can be partially filled, and those partial fills are communicated by FIX messages and are returned to the SOR engine.  The FIX messages themselves are events.  SOR operates by analyzing the confluence of events from market data feeds, order flows from OMS systems, and executions, aggregating and analyzing those events in real time, and adjust routing decisions on the fly.  For example, an order for 10,000 shares of IBM might have 10 partial fills, from multiple markets, in a 3 second time window, and make several routing decisions in that time frame.  In other words, even just ONE order can spawn "a confluence" of related messages by itself!  These executions are often reconciled with the original parent order, and the SOR may alter its execution strategy (governed by CEP rules) as the fills are returned and the markets move.

Other SOR requirements that require sophisticated event pattern detection is for liquidity detection and optimization.  Bill Harts and Rob Almgren, former head of quantitative research with Bank of America, published a paper on liquidity detection and estimation that is predicated on CEP as an implementation engine.

Another critical requirement of SOR systems is high performance, scalability, and low latency.  Addressing the event processing challenges described above in the face of multiple streams of event data, the complex interactions created by the order flow process, and you get thousands of events a second against SOR logic, the requirement for millisecond-level latency.  A multi-threaded CEP engine is critical for this kind of application, because it's SOR typically monitor markets in their entirety, and not just for the best bid and offer, but even for level 2 quote data (which shows all the public quotes of market makers wishing to sell or buy stock and recently executed orders).

So the requirements of SOR are a great fit for CEP because it combines events from multiple streams, looks for patterns, and takes action - complex action that spawns more events a deeper confluence of events, and more action.  It does this in a high-performance, low-latency, complex environment. 

Not only is SOR a great example of a CEP application, it's arguable that CEP is mandatory for today's SOR.

Getting Back to Use Cases and Patterns for CEP

Giles Nelson from Apama blogged today about how depressed he was to see a useless debate about acronyms, SOR, and CEP.  But unfortunately Giles threw the baby out with the bath water!   My initial post was about why CEP is used by Smart Order Routing applications, and that's a useful - no, critical - discussion.

When I ran Apama, the focus of my blog was use case discussion, patterns, requirements, best practices,  and descriptions of the references customers that illustrated CEP at work.  I think the community has lost this voice, and we need to bring it back.  There are a lot of good things happening at SteamBase, Apama, Aleri, Coral8, and other CEP vendors.  It's our job to educate, although I'm guilty of jumping into the bath water about the difference between Routers and SOR. 

So I'm going to RE-post the baby (the SOR use case discussion), and throw out the bath water (the Router debate).  I hope the blogosphere joins in on other use cases and positive discourse about how CEP is being used.

Smart Order Routing and CEP - Made for Each Other

Blogs are dangerous, especially when theoretical arguments are used to make an ostensibly valid point. Such is the case in a recent posting that concludes that Smart Order Routing (SOR) is not a CEP application.  This is a misinformed posting, and I'd like to clarify why. 

First a little context:  at StreamBase, we've been applying CEP to a wide variety of SOR applications with dozens of investment banks - bulge bracket firms, tier 2 firms,  exchange / liquidity / ATS providers, and hedge funds:  our team builds SOR applications every day and have years of experience.

Let's examime the basis for the conclusion that SOR is not a CEP application, (excerpts from this CEP blog post):

"Routing an order kicked off by a simple order match against a deep liquidity pool (or other market factor) does not define complex event processing nor detecting a complex event - the core idea behind CEP.   Order routing based on simple rules is BPM, plain and simple."

This is a laymen's view of SOR, and lacks understanding of the complexity that a real system exhibits.  Sure, SOR applications make some simple routing decisions.  But the reality is that "order routing based on simple rules" is not modern SOR at all.  Moreover, it's light years away from a good BPM application.  Sure, "rules" are important (James Taylor's post points this out), but let's look deeper beyond a surface-level understanding of SOR.

Here's the next part of the argument:  "SOR is not a CEP because CEP deals with evaluating a confluence of events and then taking action."  SOR doesn't "evaluate a "confluence of events?"  This argument is now officially absurd, and dead wrong. 

Let's back up and look the motivation for SOR.  The reason banks are adopting SOR is because the very structure of trading is changing dramatically, and has been for years.  Regulatory changes in the equities market have spurred the rise of alternatives - ATSs, ECNs - dark pools of liquidity.  In the U.S. market there are as many as 30 ECNs today.  And in the unregulated FX market, dozens of banks trade electronically with each other, and there will be 15 electronic FX venues by the end of the year.  Moreover, the rise in interest in dark pools of liquidity add yet another complex set of events that must be detected, aggregated, combined, matched, and manipulated in real time.  It's not uncommon for a single SOR system to connect to 10 or more markets and multiple asset classes.  Not only is this a confluence of events, it's a stunningly complicated environment in which to create a complex, real-time model in which to apply "simple" routing decisions. On this basis alone, SOR needs CEP.

Now let's look at processing complexity.  "Dumb" order routing, the kind of routing the blogger describes, would make a route decision and emit a message.  But there is little value in that.  "Real" order routing is much more sophisticated.  For example, orders in many liquidity pools can be partially filled, and those partial fills are communicated by FIX messages and are returned to the SOR engine.  SOR operates by analyzing the confluence of events from market data feeds, order flows from OMS systems, and executions, aggregating and analyzing those events in real time, and adjust routing decisions on the fly.  For example, an order for 10,000 shares of IBM might have 10 partial fills, from multiple markets, in a 3 second time window, and make several routing decisions in that time frame.  In other words, even just ONE order can spawn "a confluence" of related messages by itself!  These executions are often reconciled with the original parent order, and the SOR may alter its execution strategy (governed by CEP rules) as the fills are returned and the markets move.

And there are many more areas of SOR that require even more sophisticated complex event processing -  one example is liquidity detection and optimization.  Bill Harts and Rob Almgren, former head of quantitative research with Bank of America, published a paper on liquidity detection and estimation that is predicated on CEP as an implementation engine.

Another critical factor that is ignored is the requirement for performance, scalability, and latency.  Add all these requirements to process multiple streams of event data, and the complex interactions created by the order flow process, and you get thousands of events a second against SOR logic, the requirement for millisecond-level latency.  A multi-threaded CEP engine becomes even more critical for this kind of application.  I've never heard any SOR practitioner ever utter the letters "BPM" in this context, it's simply a non-starter.

So SOR is a great application for CEP because it combines events from multiple streams, looks for patterns, and takes action - complex action that spawns more events a deeper confluence of events, and more action.  It does this in a high-perforance, low-latency, complex environment. 

Not only is SOR a great example of a CEP application, it's arguable that CEP is mandatory for today's SOR.