The StreamBase Event Processing Blog

Why the CEP Tide Has Turned Toward StreamBase

Last week, CMC Markets announced that they switched from Apama to StreamBase as their enterprise-wide event processing platform. As FXWeek reported, CMC “initially spent three months deploying a similar system from Apama, but dropped the deal in favor of StreamBase.” As a firm that trades over $1.4 trillion a year, CMC’s decisions about their infrastructure are not taken lightly. So their decision gives us pause to ask: why the switch? 

Asif Adatia, Chief Information Officer for CMC, described their decision like this: “the ease of development for both financial engineers and technical engineers, the performance, and scalability of the platform were our key criteria.” Let’s examine each part of that statement.

1. Demonstrably better ease of use. You might think that the efficacy of a development environment would be a subjective call. In some ways, it is - when it comes to the visual esthetics of a tool. On the other hand, usability is often the driver for market-altering innovation - profound shifts in the accessibility of a new tool can suddenly unlock barriers to innovation. The change in the last 10 years in finance has been profound. 10 years ago, algorithmic trading wasn't even named, now 70% of the U.S. equities trade execution happens algorithmically. In their report "The World According to Quants: Enter Alpha Generation Platforms", the AITE group reported that certain trading algorithms have a shelf-life of less than 3 months. In the face of all that rapid change and ephemeral opportunity, the ability to quickly build real-time trading infrastructure (algos, execution, risk, compliance, pricing, TCA, surveillance) is a matter of survival.

Firms like CMC, CME Group, and RBC didn't choose StreamBase for its aesthetics, they chose StreamBase because its graphical development environment gives them a fundamental competitive edge. The tools aren't just easy to use, they allow business users and engineers to work together because the language of StreamBase matches the way they think. The StreamBase dataflow language was designed from the beginning to program event-based systems the way you would draw them on a whiteboard, and the proof lies in what customers say, as Adatia told FX week, “Our financial engineers find it really easy to put the models together.” 

And this ease of use delivers another critical innovation: firms get this usability and time to market advantage and the performance benefits of event processing - at the same time. StreamBase isn't like other CEP tools that make the developer resort to low-level programming to make the system perform and scale. Relieving the user from the burden of thinking about plumbing was described by Adatia to A-Team Insight “[CMC] create their models readily, without resorting to code-level programming.” PhaseCapital reported that they built their trading system with StreamBase in 4 months rather than the 4 years it would take to build it with traditional development tools and they change their strategies every day. These are not subjective opinions about aesthetics, they are qualitative and quantitative testimonials to the innovation that StreamBase's interface can provide.

2. Built-in performance you can see. On Wall Street, milliseconds, and microseconds matter. CMC switched to StreamBase because they wanted the fastest engine they could get. 

StreamBase was designed for speed from the very beginning. Release 1.0 in 2005, was designed to take advantage of modern hardware, from the multi-threaded server to the StreamBase language which produces highly performance applications without burdening the developer. The StreamBase language plans performance like a database pre-processor plans query optimization. In 2007, StreamBase engineers broke down another barrier with a straight-through event processing compiler. Recently, in 2008, direct bytecode generation optimization took the StreamBase performance innovation even further ahead of all other CEP platforms.

When performance is something you can see, it's a remarkable result. CMC said it this way: '[we don't talk about our specific latency numbers publically, but the performance is] good though and you can compare us on FX pricing to a bunch of our competitors.  And you will see our performance is noticeably better at times. Absolutely."  The world's largest electronic futures and options exchange in the world, CME Group chose StreamBase after using performance as a critical selection criteria, because they process over 11 million trades a day.

3. Scalability to grow. You might assume better scalability means StreamBase-based applications can be run on multiple CPU cores, and that’s true - better use of hardware means CMC needs less hardware. CMC described that their StreamBase system easily handles "More than twice the real-life peak rate for price ticks encountered so far.”  

But better scalability isn’t only about increasing volumes - it’s about ease of management, ease of customer support, and the ease with which high performance and scalability can be attained by developers.

And scalability is also about expanding usage. Apama established early on algorithmic trading as an important use case for CEP. But now, as CEP's use explodes into new application areas, firms are looking more deeply at how easy it is to build their second, third, and fourth applications. Just last week we had a call with a customer who guessed they had deployed "about 100 StreamBase applications." 

And now we have come full circle back to the StreamBase development environment. Our mantra is to “deliver high performance and scalable applications without having the developer think about performance and scalability.” Streambase makes performance native, which is one more reason it is unique.  

CMC is not alone. Organizations that are adopting StreamBase CEP are seeing these same benefits in ease of use and performance. CEP is the disruptive technology that frees their engineers to focus on creating business value instead of plumbing. CEP empowers their business analysts to work with the actual software instead of wishlists and design documents. CEP allows the whole organization to be more agile, to deploy changes to their infrastructure faster, and to take advantage of new opportunities ahead of the competition.

StreamBase and Scala

As the ponytail (and the tiny bio below) imply, my posts will focus on technical details such as language semantics, general programming issues, compiler design, or anything else I feel is nifty.

Anyone who follows programming languages will probably have noticed Scala cropping up a lot lately.  Scala is an interesting language on several fronts, but the one that sparked my interest was its strong interoperability with Java.  Often times academics forget that to be useful a language needs to integrate with existing code and libraries.  But Scala has clearly been designed with heavy focus on industrial usability as well as academic power.  When I read about Scala on Android, I guessed it wouldn't be hard to integrate with StreamBase.  It turns out I was mostly right (and we will definitely be fixing the small pain points I hit).  What follows is an guide to embedding Scala into a StreamBase application.

First we have to set up StreamBase Studio to support Scala development.

1. Install StreamBase 6.4
2. Add the Scala 2.7.5 Eclipse Plugin to StreamBase Studio (Help -> Software Updates -> "Add Site...")
3. Manually download the AJDT Weaving Plugin 1.6.4
4. Unzip the weaving plugin into your studio install
   
   • make sure the contents of the zip's "plugins" directory goes into Studio's "plugins" directory (I accidentally got that wrong the first time)
   • Update 8/31/2009:  On Windows unzipping directly into a Studio install fails with an obscure error message.  Instead, you should simply unzip into an empty directory and copy the contents of the created "plugins" and "features" directories into Studio's respective subdirectories of "C:\Program Files\StreamBase Systems\StreamBase.6.4\lib\studio". (You may need to correct that path if your studio install is in a non-standard location.)
5. Restart Studio
6. Enable "JDT Weaving" (Preferences -> JDT Weaving -> "Click to ENABLE")
7. Restart Studio
8. Verify that "JDT Weaving" is enabled (Preferences -> JDT Weaving)

Right there you see 95% of the trouble I had.  Definitely easy to do, once you know what to do, but figuring out that JDT Weaving thing (especially the specific version needed) took some searching.

Now we can get started using Scala in StreamBase.  The impatient can

1. Download the sample
2. Import it (File -> Import ... -> Existing Projects into Workspace -> "Select archive file").

or the dedicated can create a new project from scratch

1. Create a new StreamBase project (Package Explorer -> Right Click -> New -> "StreamBase Project")
2. Add the Scala Nature to it (Right Click on the Project -> Scala -> "Add Scala Nature")

Yeah that is really it.  Turns out that Eclipse does some clever and powerful things by having projects include natures like this.  We now have a project that includes the Java, Scala, and StreamBase natures and they all play nicely together.

Let's create a basic Java Operator so we have something with which to compare (File -> New -> "StreamBase Java Operator").  This auto-generated operator is good enough for your purposes.  I ended up deleting all the comments and variables and what not, so I could strip it down as small as possible for the web.

package com.streambase.sample;

import com.streambase.sb.StreamBaseException;
import com.streambase.sb.Tuple;
import com.streambase.sb.operator.Operator;
import com.streambase.sb.operator.Parameterizable;
import com.streambase.sb.operator.TypecheckException;

public class SampleJavaOperator extends Operator implements Parameterizable {
    public SampleJavaOperator() {
        setPortHints(1, 1);
        setDisplayName(this.getClass().getSimpleName());
        setShortDisplayName(this.getClass().getSimpleName());
    }

    public void typecheck() throws TypecheckException {
        requireInputPortCount(1);
        setOutputSchema(0, getInputSchema(0));
    }

    public void processTuple(int inputPort, Tuple tuple) throws StreamBaseException {
        sendOutput(0, tuple);
    }
}

Let's try a naive translation into Scala.

package com.streambase.sample;

import com.streambase.sb.StreamBaseException;
import com.streambase.sb.Tuple;
import com.streambase.sb.operator.Operator;
import com.streambase.sb.operator.Parameterizable;
import com.streambase.sb.operator.TypecheckException;

class SampleScalaOperator extends Operator with Parameterizable {
    def SampleScalaOperator = { 
        setPortHints(1, 1);
        setDisplayName(this.getClass().getSimpleName());
        setShortDisplayName(this.getClass().getSimpleName());
    }
 
    override def typecheck = {
        requireInputPortCount(1);
        setOutputSchema(0, getInputSchema(0));
    }
 
    override def processTuple(port:Int, tuple:Tuple) = {
        sendOutput(0, tuple);
    }
}

Wow that wasn't too hard, of course it didn't save us any code either.  But it does give us enough to test and figure out if it works at all.  I guess we need to make a simple sbapp to exercise these Operators we have just written.

Yeah, that was easy.  Alright, you caught me, I cheated.  Studio does not (yet) automatically find operators written in Scala, so I had to hand edit the sbapp file.  But everything works after that, this app can be run and everything works as you would expect.

Now for a slightly less trivial example.  I hear that Scala has a really nice concurrency primitive called an actor.  Well, StreamBase has a multi-threaded architecture so let's see if the two can be made to play nicely together.  Once again here is a full project that you can import as above.

class SampleScalaOperator extends Operator with Parameterizable {
    private object coordinator extends Actor {
        def act = {
            loop { 
                receive { 
                    case t: Tuple => sendOutput(0, t)
                    case None => exit()
                }
            }
        }
    }
 
    def SampleScalaOperator = { 
        setPortHints(1, 1)
        val name = getClass.getSimpleName
        setDisplayName(name)
        setShortDisplayName(name)
    }
 
    override def typecheck = {
        requireInputPortCount(1)
        setOutputSchema(0, getInputSchema(0))
    }
 
    override def processTuple(port:Int, t:Tuple) = {
        actor {
            val times = t.getInt("times")
            val delays = t.getLong("delays")
            for (i <- 0 until times) {
                Thread.sleep(delays)
                coordinator ! t 
            }
        }
    }
 
    override def init = { coordinator.start } 
    override def shutdown = { coordinator ! None }
}

Don't forget to edit the input schema for your app to include an int field named "times" and a long field named "delays".  Now we can run the app, enqueue tuples, and watch the delayed tuples arriving out of order.  We could have simply called "sendOutput" from the actor in "processTuple", but I wanted to include actors sending messages to other actors instead of just being glorified threads.

All in all, really solid work on the part of the Scala developers.  I am definitely looking forward to trying Scala on some larger projects.  Hopefully, I will be able share more as I learn it.

(Matt Fowles is a Senior Software Engineer for StreamBase, whose interests include programming language, compiler, and virtual machine design.)

Design for Debugging (by a debugger)

(Matt Fowles is a Senior Software Engineer for StreamBase, whose interests include programming language, compiler, and virtual machine design.)

As the ponytail (and the tiny bio above) imply, my posts will focus on technical details such as language semantics, general programming issues, compiler design, or anything else I feel is nifty.

Bob Hagmann from Aleri recently posted a great piece on designing and debugging CEP apps; unfortunately, he leaves out what that process of debugging might actually look like.  While some languages force you to spelunk through printf output or trace files to track down a bug, I have found that the returns from an integrated debugger are immeasurable (and greatly magnified by good design).  StreamBase ships with a visual debugger built into its development environment, so I will expand on Bob's post with a concrete example using the StreamBase visual debugger.  Consider the following sample application that continuously calculates Bollinger Bands for stock quotes (for those playing along at home this sample ships with StreamBase and can be loaded with "File -> Load StreamBase Sample...")

 

This app actually calculates Bollinger bands in two different ways, one using a simple moving average and the other using an exponential moving average.  For simplicity, the two paths are separate branches of the main app, allowing each path to be debugged separately.  Within each path, the calculation is divided into a first step, where the moving average and standard deviation are calculated, and a second step, where the intermediate results are used to calculate the Bollinger bands.  This allows us to use the StreamBase debugger to inspect intermediate results and inspect their values.

With the app paused at a breakpoint, we can inspect the fields of the current tuple or at any earlier point in the active data flow, allowing easy inspection of transformations and invariants that need to be maintained.  The StreamBase debugger also lets the user modify the value of fields in the current frame before resuming execution, providing a simple way to test downstream components with hand crafted data.

Of course in an application this simple, one doesn't even need a debugger or careful design.  But, when debugging a larger application like a smart order routing system, organizing your code into staged transformations where each stage is a submodule can be invaluable.

This sort of design not only allows you to step into each module and debug its inner workings separately, but it also provides clean abstraction barriers for unit testing.  And once you have unit tests for each stage and a debugger to help you fix regressions, you will be free to refactor, optimize, and extend your application with impunity.

Language Design Lessons 2 - Lists and Custom Functions

(Matt Fowles is a Senior Software Engineer for StreamBase, whose interests include programming language, compiler, and virtual machine design.)

As the ponytail (and the tiny bio above) imply, my posts will focus on technical details such as language semantics, general programming issues, compiler design, or anything else I feel is nifty.

This post is the second in a series exploring how new features in a language often impact unexpected corners of the language and can lead to a better whole.

In StreamBase 6.3, to be announced in early April, we added lists as a parameterized data type to StreamSQL, allowing easy representation of data of unknown length:

CREATE SCHEMA offer_details (
  symbol string,
  num_shares double,
  share_price double,
  user_id string
);

CREATE SCHEMA order_book (
  bids list(offer_details),
  asks list(offer_details),
)

It is worth noting that StreamSQL's parameterized data types are more like C++'s than Java's: a parameterized data type is not complete until it has been given a parameter. For example, list(int) or list(list(string)) are both complete types, but list alone does not specify a type. While Java's generics suffer because of their use of type erasure, StreamBase's parameterized types provide strong, static typechecking.

One of StreamBase's selling points is the ease of extending the StreamBase Expression Language with custom functions written in Java or C++.  When integrating with Java, we automatically map between StreamBase types and their Java equivalent (e.g. StreamBase int to Java int or StreamBase string to Java String) for both arguments to and returns types of custom functions.

Obviously, the natural representation for a list(int) or list(string) in Java is a List<Integer> or List<String> respectively; unfortunately, at the JVM level these types don't exists. Both types become merely List after type erasure. Thus, we cannot automatically infer the type of an argument or return for parameterized data types. We could require users to explicitly declare the types of their custom functions in the configuration file, but that is a bit inelegant (and also prevents users from writing a single Java function that can take several different list types like list(int), list(long), list(double)).

After some discussion, we decided that the users needed a more direct hook into the typechecking process for their functions. Thus, we added the concept of a function resolver. This is probably easiest to describe with an example:

class ExprUtil {
  @CustomFunctionResolver("concatenateResolver")
  public static List<?> concatenate(List<?> v1, List<?> v2) {
    List<?> res = new ArrayList<?>();
    if (v1 != null) { res.addAll(v1); }
    if (v2 != null) { res.addAll(v2); }
    return res;
  }

  public static CompleteDataType concatenateResolver(
        CompleteDataType arg1, CompleteDataType arg2) {
    if (arg1.getElementType() == null) {
      return null; // not a list
    }
    if (arg1.equals(arg2)) { return arg1; }
    return null;
  }
}

The concatenate function above will take 2 lists (provided they are the same type) and return a list of that type. We use the @CustomFunctionResolver to annotate the method with just such a resolver, and everything else is taken care of.  During typecheck time, the resolver will be called as necessary to determine whether a function is a legitimate candidate. Then at runtime, the concatenate function will be called appropriately. One can, of course, do more complicated manipulation of the types involved if inclined:

class ExprUtil {
  @CustomFunctionResolver("flattenResolver")
 public static List<?> flatten(List<List<?>> top) {
    List<?> res = new ArrayList<?>();
    if (top != null) {
      for (List<?> sub : top) {
        if (sub != null) { res.addAll(sub); }
      }
    }
    return res;
  }


  public static CompleteDataType flattenResolver(
        CompleteDataType arg) {
    CompleteDataType inner = arg.getElementType();
    if (inner == null) {
      return null; // arg not a list
    }
    if (inner.getElementType() != null) {
      return null; // arg not a list of lists
    }
    return inner;
  }
}

As an added bonus, while implementing this feature, we realized that hierarchical data is just another parameterized data type.   Prior to adding type resolvers we didn't allow tuples as arguments to custom functions.  But now that we have a simple, clean API for interfacing parameterized types between Java and StreamSQL, we can (and do).

Language Design Lessons 1 - Hierarchical Data and Wildcards

(Matt Fowles is a Senior Software Engineer for StreamBase, whose interests include programming language, compiler, and virtual machine design.)

As the ponytail (and the tiny bio above) imply, my posts will focus on technical details such as language semantics, general programming issues, compiler design, or anything else I feel is nifty.

In language design, the most powerful and elegant features come from having simple, consistent semantics.  Over several posts, I plan to explore and expand on this idea through concrete examples encountered while developing both StreamSQL Text and StreamSQL EventFlow.  In this post, I will explain how adding hierarchical data to our language forced us to simplify and greatly improve our wildcard rules.

In StreamBase 6.0, we added support for hierarchical data in the form of a tuple data type.  This, among other things, allows us to reuse common structures in multiple different schemas:

CREATE SCHEMA offer_details (
  symbol string,
  num_shares double,
  share_price double,
  user_id string
);

CREATE SCHEMA top_of_book (
  best_bid offer_details,
  best_ask offer_details
);

CREATE SCHEMA purchase (
  seller_id string,
  buyer_id string,
  details offer_details
);

This, of course, helps clean up the code and makes later refactoring easier, but data is only useful if it can be easily transformed and manipulated.  Clearly we are going to want to be able to

• create nested tuples
• extract data from nested tuples 

all with a syntax that people will find intuitive.  Fortunately, the debate over syntax for accessing hierarchical data has long been settled in the programming language world; simply use . like most other languages (also we already use . to specify the source of a field when it is ambiguous from context).  We also provide a tuple function to create nested tuples:

CREATE INPUT STREAM input_purchases purchase;
CREATE STREAM account_transfers;

SELECT
  details.num_shares * details.share_price AS amount,
  tuple(
    seller_id AS destination,
    buyer_id AS source
  ) AS accounts
FROM input_purchases INTO account_transfers;

and that satisfies our original requirements.  Of course, it doesn't provide us with a way to quickly manipulate the entirety of a nested tuple, nor does it allow us to easily combine two sub-tuples into a larger one.  I find it is usually better from a design standpoint if I can extend and strengthen an existing concept instead of trying to create a new one.  Luckily, * comes to our rescue.   Since StreamSQL Text already has a simple syntax for accessing multiple fields at once, we will simply strengthen the wildcard expressions that we had before.

Let's start with some simple definitions.  Consider the following statement:

SELECT f(*) AS prefix_*_postfix FROM in INTO out;

A Wildcard Literal is just the asterisk token, *.

A Wildcard Rule is the entire expression, including both expression and target (in this case f(*) AS prefix_*_postfix).

A Wildcard Expression is the portion of the expression before the AS keyword (in this case f(*) ).

A Wildcard Target is the portion of the expression after the AS keyword (in this case prefix_*_postfix). A wildcard target is a * token with an optional string prefix and postfix. For example, foo, *bar, and foo*bar are all valid targets.

In cases where the context is clear, I may drop wildcard from the above names and simply refer to them as rule, expression, and target (I will try to take care as expression is a heavily used term).  With these definitions out of the way, let's return to the discussion of improving wildcard rules.

Prior to StreamBase 6.0, wildcard rules held a fairly restricted place in the StreamSQL Text grammar.   A wildcard expression could be either a wildcard literal (i.e. *) or a function with only a wildcard literal argument (i.e. firstval(*)).  The ability to call functions on wildcards was, in fact, a bit of a hack included for passing all the fields in a tuple to aggregate functions.

SELECT
  firstval(*) AS first_*,
  lastval(*) AS last_*
FROM in[SIZE 4 TUPLES] INTO out

While the feature worked with functions other than firstval and lastval, they were the real reason that it was there.  As always, when extending an existing feature, we should try to simplify and maintain consistency.  Fortunately, the original rules for wildcards (while simple) were not particularly consistent in terms of fitting into the StreamBase Expression Language.  So the first step is to find a better definition for Wildcard Expressions that doesn't include any special cases like "only argument" or "wildcard literal".

A wildcard expression can be any expression in the expression language that uses * as a variable name (possibly in several locations). For example, in.details.*, f(g(*)), and f(*) + g(*) + 2 are all valid wildcard expressions.

Part of the appeal of this definition, is that it reduces wildcards to a case that we already know how to handle, namely variable names.  So there should be relatively little surprise about where it is legal to put a *.

This alone provides us with enough power to flatten nested tuples easily:

CREATE STREAM input_book top_of_book;
CREATE STREAM flattened_book;

SELECT
 best_bid.* AS bid_*,  
 best_ask.* AS ask_*
FROM input_book INTO flattened_book;

I confess, a contrived example, but it is really just for illustrative purposes.  But it doesn't quite allow us to combine two tuples into a bigger one.  Before StreamBase 6.0, wildcard rules were legal only in the target lists for select statements.  Just like earlier, a simpler rule without any special cases solves our problem:

  Wildcard rules are legal anywhere a list of expressions is legal.

Simple, easy to explain; sounds good so far.  What does this buy us that we couldn't do before?  Now, wildcard rules can be used to pass arguments to a function!  For example:

CREATE INPUT STREAM foo (int x, int y, int z);
SELECT min(* AS *) AS min FROM foo INTO bar;

expands to

CREATE INPUT STREAM foo (x int, y int, z int);
SELECT min(x AS x, y AS y, z AS z) AS min FROM foo INTO bar;

Alright, that is a fairly contrived example too, but the real reason we wanted it was to combine two separate sub-tuples into one big one.

SELECT 
  tuple(
    sub1.* AS *,
    sub2.* AS *
  ) AS combined
FROM in INTO out;

As a result of adding hierarchical data to our language, we ended up with the ability to use wildcards in compound expressions and as arguments to functions.  All while simplifying the semantics for wildcard rules from something with several special cases and provisos into something already well understood.  Whenever I can accomplish more with less, I know that I am on the right track.

Analyzing the "New and Revolutionary" Features from Tibco BusinessEvents 3.0

Recently, Tibco announced version 3.0 of their complex event processing (CEP) product, BusinessEvents.  There's a lot of very nice marketing on their website, including an interview with Alan Lundberg - Alan is a very nice guy and a great evangelist for CEP.  But unfortunately part of TIBCO's marketing department has gone too far, because the 3.0 release lays claim to "new and revolutionary" features that are actually quite mature and advanced in many of the leading CEP platforms.  Since I've run two of them, I thought I would comment on TIBCO's claims of newness and revolutionaryness.

Let's examine the features" listed in the BusinessEvents 3.0 press release:

#1: "Decision Manager: New business user Interface provides simpler access for business users to write rules in their domains of expertise."

Business user oriented interfaces have been central to many CEP products for years, so this is neither new, nor revolutionary.  For example, StreamBase Studio is the graphical CEP development platform interface for the StreamBase platform, and was part of the StreamBase product since release 1.0.  Apama's "Event Modeler" graphical environment, which is an optional development tool and generates MonitorScript code, was added to the Apama product a number of years ago as well.  And Agent Logic, a CEP vendor that specializes in the government market, is well known for excellent graphical CEP rule creation.

#2: Events Stream Processing: BusinessEvents query language allows you to continuously query event streams in real-time and take immediate action.

Continuous queries is the heart and soul of all CEP platforms; implementations vary, but, once again, this isn't new.  From a historical context, you can read the academic papers from project Aurora at MIT that yielded StreamBase, from back in 2005.  Coral8 even calls it's entire language the "Continuous computation language," or CCL.

#3: Distributed BusinessEvents: Distributed cache and rules engine that provides practically unlimited scalability letting you monitor twice the magnitude of events previously possible.

Again, the original concept behind early CEP research was rooted in the assumption that event sources are exploding, and that any event-based computing fabric must be designed to support tremendous volumes of events, with low latency.  Most of CEP vendors have supported distributed caching for years, and have moved on to much more advanced scalability techniques, including support for grids, low-latency connectivity options like binary FIX or Infiniband, and more.  The historical context on distributed caching in CEP began long ago.  StreamBase, for example, has been multi-threaded since day one, and supports grids, 64-bit, and a variety of other scalability options.  Coral8 is also reputed to be very fast, and is multi-threaded.  Apama, which has been around the longest, is single threaded, so Tibco has at least beaten them to the scalability punch. 

And in addition, some platforms have very sophisticated functionality that allows application developers to develop CEP logic without needing to worry about the complexity of threading behavior (MUTEXs, critical sections, etc.) - and that's an interesting area of potential "revolution," but Tibco didn't describe what ground they have broken.

Once again, "first" is not necessary "best."  But as more and more technologists become exposed to CEP and the different approaches vendors offer, a discussion can help prospective members of the CEP user community make more informed purchasing decisions.

On CEP Standards Confusion and Progress

(Richard Tibbetts is the Chief Architect for StreamBase, co-author of a recent paper with Oracle entitled Towards a Streaming SQL Standard, and StreamBase's representative to the Event Processing Technology Symposium, or EPTS).

There has been swift and strong response to the paper that Anand Srinivasan (from Oracle) and I presented last month at VLDB in Auckland. The paper, Towards a Streaming SQL Standard (download a PDF of the paper), seems to have touched a nerve in the community. All of a sudden, everyone is very excited about standards, and has an opinion to share. Some, such as Louis Lovas of Apama, perhaps concerned about being left behind, have been predictably negative and discouraging.  Opher Etzion from IBM shared a balanced, constructive point of view.  And then others have used this advancement as yet another excuse to re-hash the tired debate about "what is CEP?"  (we have provided no link to these discussions because the debate is pointless).

I'd like to address some of the mis-perceptions about this paper, because it's easy and deceptive to leap to the conclusion that "Oracle and StreamBase have proposed a new standard."   Instead, there are two key points to bear in mind when reading our work:

#1. This is an academic paper.  The goal of this paper is to advance standardization discussions by raising the processing model as a core element of any CEP standard. With event processing, when the results are produced can be as important as what the results are. That is, the same answer, produced later, is not the same answer. In SQL standardization, which deals with finite result sets, timing has not been a problem. In CEP however, it is a difference between implementations. The paper doesn't take a position on which existing approach is best. It merely states the problem. The paper doesn't propose a standard. It proposes one solution to one problem in standardization, as part of a broader effort.  We think this is an important observation, revealed by hard work and research.  We leave it to the community to decide if this observation is as important as we think it is.

#2.  Standardization is a very slow process. StreamBase is dedicated to standardization.  But standardization is a long process, and we've been working on it for almost 5 years.  And we will continue that commitment.  It will be years before we have a complete complex event processing language standard, with multiple conforming implementations.  There will be open standards bodies, and lots of work between groups of vendors to resolve their differences. In the early stages, it will likely be the smaller groups that make more progress.  Sure, we have a bias:  we think SQL is a great foundation to start with.  But we're happy to debate all approaches, and examine deeply the tradeoffs, just like we did with the Oracle research.

So StreamBase hasn't declared "victory" in standards; we are simply moving the discourse forward. I'm pleased that the community has responded strongly, because reaction means the debate has advanced.  We think thoughtful standards research is the best way to continue to promote standardization, and the rising tide of CEP, because a rising tide lifts all boats.  And we encourage all CEP vendors to do the same.

Towards a CEP Standard

There has been a lot of discussion in community groups, at conferences, and message boards over the years - back and forth about the best way to achieve progress on standards in the complex event processing (CEP) space.   This week StreamBase announced a partnership with Oracle - that the two companies have come together to research and publish on ways to reconcile a few of the approaches that have been developed for CEP.  The paper is being presented by our chief architect, Richard Tibbetts, at the VLDB conference this week.   We'll be blogging about this research in the future.   Here is the press release and a few links to coverage:

• Press Release: StreamBase and Oracle Partner on Complex Event Processing Standard Approach

• Banking Technology News coverage of the announcement

• A-Team news coverage

In the past perhaps StreamBase would have presented this news as further evidence that SQL has dominated the world of CEP as the no-brainer winner;  notice this is not the tenor of the release at all.  Instead, this research has taken years to develop, and the thoughful presentation of ideas in the paper is intended to provoke a discourse that can keep us on a path of convergence as CEP continues to become more and more important in the enterprise IT landscape. 

Stay tuned - more to come on this topic (post-script to this post, here is a post from Richard Tibbetts on the paper)!

StreamBase 6.0 Product Analysis and Thoughts on Leadership

June is a big month for CEP product releases, and the past few weeks have been no exception:  Aleri  announced version 3, Apama announced version 4, and StreamBase announced version 6.0.  Unsurprisingly, all these announcements contain proclamations of leadership.  Aleri's headline claimed a better look and feel, and Apama announced integrated sample applications and unsubstantiated performance claims.  By contrast, I think StreamBase announced an important breakthrough:  the first visual debugger, profiler, and unit testing environment in the industry. 

One of the main reasons I chose to join StreamBase is that in my time in the CEP industry, I've heard from customers, prospects, and industry insiders that the technology is simply superb:  it's fast, it's open, it's scalable (multi-threaded), and it has adopted a language metaphor (SQL) that is standards-based and familiar to most developers.  The visual environment uniquely enables business users and IT alike.  

I think release 6.0 extends that lead.  The ability to debug, test, and profile event processing applications is absolutely critical to ensure fast, reliable, and managable deployment in large enterprises, as well as adoption by smaller firms that need to do more with less. 

The release contains the following:  

1. StreamBase Visual Debugger - One of the hidden problems with some CEP platforms is that the visual tools were added as an afterthought, and, as a result, they generate code that's impossible to test, support and debug.  This is not the case with StreamBase, that started with a visual development tool from the very core, which has made it an incredibly powerful, integrated platform on which to debug and analyze applications, because there is no code generation - what you see is what you get and it's what you debug. 
2. Profiling - Profiling provides the ability to test, profile, and provision software and hardware resources associated with a CEP applications.  Once again, with StreamBase profiling, you profile and optimize the actual code you designed, so the result is a better, faster, more reliable application.
3. Unit Testing -  As the CEP market matures, larger and larger development groups are involved in CEP application development, which motivates the need for more sophisticated group development capability to test applications.  StreamBase Unit Testing allows teams to work in parallel, and allows teams to reduce application errors during development.  Unit testing also enables business developers to more fully participate in application development by providing IT the ability to test code supplied by business analysts before putting it into production,

The release of these features further demonstrates that CEP platforms are coming of age and ready for enterprise adoption and large, group development.  Yes these platforms are mature - today.

Event Processing Technical Society (EPTS): Thanks, Opher

The press release finally hit the wire today announcing the EPTS.  The goal is simple:  promote awareness about event processing technologies.  Immediate reaction has ensued, both negative and positive.   To me, having been involved in many of these discussions, I think any step forward is a good step.

This has taken about 2 years to formally announce this group. And, although the spirit of this has been to work as a community, and not single anyone out, there's no doubt that without a determined leader in the group nothing would have happened - from my point of view that's Opher Etzion - so I'm going to simplys link to Opher's blog, and his perspective, on this release as the best source for perspective.  I remember meeting Opher in New York at IBM for the first time about 3 years ago and discussing a vague vision of vendors, analysts, and academics all coming together to help resolve disputes and push a common understanding.

Here's hoping that the EPTS continues to advance this very important field.

Thanks, Opher.