The StreamBase Event Processing Blog

StreamSQL

As you get older you take for granted lessons learned much earlier in your career. Then when someone asks why such-and-such is the case you do a double take, and have to engage the question with integrity and fairness. If someone suggests that pentagonal wheels are better than round ones it’s worth at least a few moments reflection.

So I admit it. My basic assumption is that standards, be they de facto or de jure, are inevitable in pretty much all areas of technology. Trivially, though not conclusively, I would note that this is simply a pattern I have observed. We have standard protocols at every layer of the network stack. We have standard API’s to operating systems. We have compatibility standards for laptop computers. We have standard connectors for IO buses. There are standards in database systems, in middleware, in document formats, in eCommerce choreographies and in digital audio formats. I am pleased at the local standards in electrical outlets, but not at the state of global inconsistency – my assumption is that this will change. Once you start down the list it is hard to find things for which there are not standards, or in which there are not some emerging standards. These standards may take a year or two to establish themselves (eg J2EE took 12-18 months), but I have assumed an inevitability of standards in the kinds of things that companies like StreamBase do. Including in CEP.

An observed inevitability often reflects an economic reality. The principle is that markets generally move towards the eradication of inefficiencies. If standards-based approaches are more efficient for customers than proprietary ones then standards will win, over some time period. An example might serve us here: In a recent case of a customer switching their solution from another SQL-style CEP product to StreamBase as they moved from prototyping work to their deployment architecture, the replacement took about two or three days. You read that right. With very little retraining of people, minor changes around SQL syntax (and to be fair some differences in detail semantics), replacement of the Stream Processing/CEP server, and not much else the customer had switched vendors. If that sounds like the pattern you’ve seen with SQL databases, J2EE application servers, CORBA systems, or JMS servers, then that’s because the example is identical to what was seen in those markets.

When you go back and ask yourself why standards tend to emerge in technology you have to note that proprietary solutions involve enormously greater costs around vendor lock-in. By contrast standards force price/performance competition, and the customer wins. I believe there may be circumstances in which pentagonal wheels are better (moon landers?), but in the general case circular wheels are an inevitable solution.

For the record: StreamBase is committed to standards. We funded the development of the initial StreamSQL spec by all of the leading academics in the field of Stream Processing, including but not limited to Professor Mike Stonebraker and the four other professors that were founders of StreamBase. We put the StreamSQL spec into the public domain. We are working diligently with competitors, partners and customers to iterate that specification, including of course the commitment to invest to keep StreamBase compliant with whatever is decided. The fruits of this work will become apparent over forthcoming months.

Stay tuned!

Charlie McCreevy gets serious about MiFID

MiFID (the Markets in Financial Instruments Directive) is big news in European Union financial circles. For those familiar with Reg NMS (Big PDF) in the US, MiFID is similar in intent but for EU markets.

The short version of what it’s about goes something like this: Suppose you are an investor and you ask a broker to execute a trade on your behalf. How do you know that they will get you a good deal? With current market structures and regulatory mechanisms you have very little by way of visibility or guarantee. MiFID is designed to change that.

“The finalised MiFID is set to be rolled out in November of this year with the aim of creating a single European securities standard for all financial instruments.

Among the 73 articles and 71 recitals are new pre- and post-trade transparency requirements for equity markets, and more extensive transaction reporting requirements.

Companies must be able to fully reconstitute not just transaction data, but the full progress of any transaction for five years or for the duration of the relationship, which for transactions such as mortgages can be very long indeed”.

                                                - Ian Williams, vunet.com, 25 Apr 2007

(Ian was commenting on a recent report from P J Di Giammarino of the JWG-IT think tank).

Charlie McCreevy, EU Commissioner and former Irish Minister of Finance, is getting serious about ensuring EU countries open up their markets by legislating MiFID:

 "I urge Member States to keep to the timetable that they themselves have agreed. Further delays could well expose Europe's firms and banks to serious competitive disadvantage”.

                                                - Charlie McCreevy, EU Commissioner, 26 Apr 2007

So what’s the hold up? Ian Williams set it out in the above referenced article:

“The cost of MiFID implementation is set to surpass £1bn in the UK alone”.

                                                - Ian Williams, vunet.com, 25 Apr 2007

The first problem is that the pre-trade and post-trade transparency requirements involve real-time surveillance of several dozen markets and MTF’s in various jurisdictions for prices and depth of liquidity assessments, followed by intelligent execution of the trade potentially across multiple liquidity pools and over some optimal time period. The latency requirement is as-close-to-0-milliseconds-as-possible. If the benchmark for “best execution” (something that is not well defined today) is broader than simply price and timing variables then automating it is correspondingly harder to do.

The second problem is that there is a requirement to be able to demonstrate that for a particular order the execution was in fact Good. Given the number of transactions and the amount of data required to put together a report on any one transaction for the purposes of proof, this is a substantial storage, retrieval and analytical challenge.

There are various other pieces of MiFID, including capital sufficiency requirements, client data management and other non-real-time things. Those are naturally of peripheral interest from a CEP point of view.

One way to address this is a “Crawl, Walk, Run” strategy. The StreamBase MiFID offering is about engaging a phased, iterative program that reduces cost, risk and implementation time. It also sets up a very flexible solution that can evolve with the MiFID requirements and the roll out of MiFID across Europe.

I suspect that Charlie McCreevy is right, that some jurisdictions and financial institutions will miss the November 2007 deadline, to their financial detriment. CEP is an important part of the solution.

StreamBase 3.7

It’s not my intent to spend much time on this blog talking about the StreamBase product, but on occasions it’ll make sense to share with you the latest StreamBase product news.

One of the areas of the product I’m most passionate about is the experience for application developers who use our software. Custom-coding applications has remained the most common approach to solving high performance CEP-style problems until recently, so we put a lot of energy into delivering a radically more productive and enjoyable application development experience. It’s important to extend our leadership in this area, and our latest release raises the bar yet further. 

Some high points:

• Enhanced reusability of CEP applications: In the same way that Java or C++ engineers make their code modular so as to enable reuse, StreamBase has the concept of CEP Modules. In StreamBase 3.7 we have made it even easier to reuse, parameterize, and share application code, scripts, and templates. This is particularly helpful for large end-user application development teams, independent software vendors (ISVs) and original equipment manufacturers (OEMs) who embed StreamBase inside of their solutions.
•  Improved Data Stream Visibility and Simulation: Many customers test their applications using stored streams, but others who don’t have sample data files handy want to simulate feeds. We’ve just enhanced our built-in feed simulator so it’s easier to create and customize a simulated feed on-the-fly that’s similar to a live feed. This helps reduce the development and testing cycle. And we’ve also enhanced the ability to graphically monitor applications.
• Enhanced Scalability and Manageability of Run-Time Applications: In custom coded applications it takes a lot of work for developers to partition and manage applications that are distributed across multiple CPU’s. For customers who are building and managing multiple applications, our new CEP “containers” connect and consolidate the applications more readily onto fewer servers, help with sharing system resources, and scale with reduced development effort. Many customers also want the ability to change run-time applications on-the-fly, and users can now readily change running applications when updates to computations or analytics are required. Under the covers we’ve also made it easier to optimize concurrency at a threading level, giving developers much more powerful access to the performance of the leading high performance CEP platform.
• Enhanced connectivity: In addition to the new adapters for financial market feeds, we’ve released new Eclipse-based wizards that make it easier to create custom input and output adapters, and we’re offering more flexibility to independently pause, resume, and control adapters, which are a critical part of all CEP applications.

These reflect just a few of the enhancements in the 3.7 release.  As demand for StreamBase by the developer community continues to grow, you can expect to see more news about what we're doing to extend StreamBase leadership as the development and deployment platform of choice for high performance CEP applications.

More information and downloads are available from our web site.

Events in Algo Trading

The application of high performance complex event processing technology to algorithmic trading is gaining a great deal of attention these days, and I’d like to share a few thoughts on this area. Just last Wednesday, we presented on this topic to an audience of about 40 senior Wall Street executives in New York.

Algorithmic trading is essentially the real time analysis and execution of transaction decisions in the financial markets. Strategies generally involve identifying patterns within historical data to predict future market behavior, and building rules into a model so that order placement timing may be optimized to cause the least impact on a stock's price.

Although the natural application of CEP to algorithmic trading is in the system which executes trading strategies, what really resonated with our audience is that many other components of the end-to-end-system require CEP. As depicted in the figure below these include up-front data cleansing and normalization, risk management and profit & loss, execution strategy and order management, exchange interfaces, and the integration of all of these complex event flows.  CEP allows these to be done at considerably higher performance, much shorter time-to-deployment and at vastly lower lifecycle costs than traditional approaches.

Stay tuned for subsequent posts where I'll speak more about the best practices and capabilities in these areas.

Quick update on MMO Games

I meant to follow up my comments on MMORPG/Virtual World stuff with a pointer to a recent StreamBase solution in this space.  Transatlantic perambulations interrupted that good intention.  So, belatedly, here are the press release and application description (PDF).  And some customer thoughts on the topic:

“After we evaluated StreamBase, we ran the
numbers to determine what it would take to
develop similar technology internally. We
quickly realized that it would be both time and
cost prohibitive and would even lead to missing
critical deadlines. Therefore, partnering with
StreamBase was clearly the right choice to
deliver on budget and on time.”

    — Bill Dalton, Technical Director, BioWare

CitySense - A glimpse of our sensor networks future?

Interesting Sensor Networks article in the Boston Herald today.

The City of Cambridge (Massachusetts) is setting up a wireless sensor network to capture environmental data and communicate it back to interested parties, including the general public.  "1,000 footballs on lampposts" may be the first experience some people have of a real world Sensor Network.  It would be easy to ignore it as an obscure academic project and miss the significace of what it adumbrates. 

Most of the interest in sensors has revolved around RFID tags.  Caution about RFID hype is been urged in many quarters, most recently in InformationWeek, and the cautionary words are welcome in the wake of the wide-eyed RFID enthusiasm. But the essential insight is correct - electronic sensors will be everywhere. And that will include RFID-like solutions.   The following comments from Professor Mike Stonebraker capture the idea better than I can express it:

"As the cost of micro sensors continues to decline over the next decade, we could see a world in which everything of material significance gets sensor-tagged. This will include you, your kids, your car, your (or your company's) valuable possessions, library books, people requiring medical (or criminal justice system) supervision, taxicabs, cell phones, grocery store items, etc. In fact, only imagination will limit the applications of this technology, the most significant of which have yet to be thought of".

Note that the reference is not specifically to RFID, but to sensor networks in general.  The advent and broad deployment of sensor networks seems inevitable.  For people in the software world there is another inevitability: Vast quantities of raw data. I would observe that most of this data is pretty uninteresting.  Furthermore the sheer volume of the data would overwhelm any attempt to store it, much less query it sensibly at some later stage.  So we're expecting an enormous amount of data, some small subset of which is quite valuable, and no way to handle it using store-and-query architectures.  Sounds like a CEP problem to me!

StreamBase ran a project for the US military some months back, in which we had to run CEP queries on battlefield monitoring data.  "Tell me when the center of masss of any platoon is more than 500 yards from where they're supposed to be", "Tell me when more than three enemy vehicles cross over such-and-such a line", and so on. This is a classic sensor network application.  Sensors create raw data that is trivially handled by a high performance CEP engine, and is very hard to handle in any other fashion.

So I'm excited to see Cambridge putting this network together.  I am not sure that the Cambridge CitySense network will need a StreamBase class event processing engine in the immediate future, but we're seeing the beginning of the radical evolution.