The StreamBase Event Processing Blog

Q&A with CMC Markets: Trading $1.4 Trillion on CEP

This week CMC Markets announced that they have deployed complex event processing (CEP) in their global trading infrastructure that processes $1.4 trillion a year on over 5,000 trading instruments, on over 115,000 trades a day. 

You might be thinking - $1.4 trillion in trading? Who is this CMC Markets? CMC is the leading global provider of retail contacts for difference (CFDs) and one of the largest financial spread betting leaders. CMC is a sophisticated, distributed organization - according to Inside Market Data, "the StreamBase platform is being rolled out across CMC's global operations, including offices in Australia, Canada, China, Germany and the UK, and is supporting a wide range of applications including its market-making, risk management, and hedging tools to support trading in FX, indexes, futures, commodities, and equities."

This kind of global, mission critical commitment to CEP is important news, so we spoke with CMC’s Chief Information Officer, Asif Adatia, to find out more about CMC, their needs, why they chose CEP, their business results, and their future plans with CEP.

Q1: The AITE Group report from July, 2007 said your average trade volume was about $100 billion monthly.  Is that correct now?

That’s an old number. Now we’re at about $200 billion a month or $10B a day.  For example, if I look at today’s stats we’ve done $7.5 billion now at trading shops.

Q2: Have your trading volumes increased a lot over the last couple of years?

I think all of our stats have increased.  We’ve got more clients, more trades per day.  Which explains some of the re-architecture work we’re doing.  Yeah, we’re doing very well at the moment.  This business fundamentally is doing very well from a key metrics perspective.

Q3: Right.  Can I ask were your upgrade plans hampered at all this last year by the recession or have you been able to carry on?

Peter Cruddas, our CEO, gave us massive support from a technology perspective.  If anything I’ve got more to do. We put a plan together, the board bought into it and it’s pretty exciting.  For me it’s phenomenal, we’ve been able to do everything we wanted to do.  This re-architecture is an effort to take our business to another level.

Q4: So there hasn’t been anything that had to be put on hold because of the crisis?

No, definitely not from a technology perspective, no.  If anything, the question is “can I do more and get it done faster.”  I’ve certainly not had restrictions.

Q5: This re-architecture is replacing your existing in-house built architecture.  What other technology upgrades are you making?

We’ve got a whole style change from our front end and our transactional engines to our back office.  There’s a program over the coming months where we are upgrading all of our systems.

Today, StreamBase is doing pricing, and it’s far more sophisticated than our legacy market maker platform we previously had.  Ultimately, StreamBase will replace a lot of the intelligence behind the old system.

Q6: Can you give me a couple of examples?  And over what period you will conclude?

So the back office as an example, we started to work on that. We’ve got a batch orientated back office currently, and we’re in the process of creating a real time P&L and cash balances being realized in P&L.  Today you wait until the end of the day, you mark the market and you adjust your P&L’s and you realize your P&L then. We’re moving a real time back office.  The timing is probably end of year. And then again because it’s so different, there will be a coordinated effect with our marketing and our client service guys.  Because it is a big difference from the way we have been working. That’s just one example of many.

Q7: Can you describe the project StreamBase supporting?

StreamBase supports FX for all of our major currencies, exotics, as well as forwards.  It will support indices futures, futures on indices, cash commodities, treasuries and equities, single stock.

Q8: Can you describe how the StreamBase application processes data and pricing for CMC?

We cleanse our prices as they come through. We get multiple feeds from various brokers.  We cleanse our prices and we’ve got a reasonably sophisticated model where we look at groupings - if there are multiple feeds coming in and you want to take the best price, you want to take out the ones that are outliers. So the mathematics behind are sophisticated - it doesn’t just do a standard mean, it takes a look at standard deviation of all of these prices and then looks at the volatility over multi-factor, high frequency scenarios. Then we start to look at which ones are outliers.

As I said StreamBase helps us look at what’s happening and very quickly take algorithms in and out - fix them and put them back in as well.

Q9: What are your FX market data sources?

I can’t give you the names but it’s primarily the usual suspects.  Tier one brokers provide liquidity and pricing to firms like ourselves.  We’ve got quite a few brokers that we deal with and their price feeds. We’ve also got feeds for other asset classes.  And along with the brokers, we have multiple data feeds from vendors, exchanges, alternative ECNs and OTC price providers

Q10: Are you using the StreamBase adapters to connect directly to those price feeds?

Yes, it’s actually is a combination.  StreamBase provides feed handlers.  So very quickly we were able to connect and use the StreamBase feed handlers. And then we’re in the process with some of our brokers, again, StreamBase allows us to connect to our own feed handlers so we do that.  So as I said at the beginning flexibility was one of the key criteria.  StreamBase offers that to us as well.  So we can mix and match.

Q11: Are you live? 

Yes, we’re live now.

Q12: You mentioned scalability and performance - what kind of demands are you putting on the StreamBase system now?  I’m thinking updates, latency, that kind of thing?

So at the moment we’re able to process 17,000 ticks per second on FX.  It doesn’t really hurt the system or the processes that much.  And we’ve got it scaled up to for twice as large as that. Can it handle our peak? Our peak to date has been nowhere close to 17,000 at the moment.  We’ve got what we call a non-functional regression test environment where we just overload it.  We overload our environment and haven’t broken it.  We got so much going on right now that if it can cope with double our peak, we can focus on other things.

Q13: So what kind of a different will users notice?  What’s going to be different for them? 

If you look at FX pricing live right now our spread is pretty much in line and competitive.  We used to have an issue back in before we implemented some of this.  We used to have issues with latency.  Those problems have disappeared now.  We’re able to process a lot more - about 17,000 ticks a second, which is pretty cool. We’ve still got plenty of headroom as well.  And we’re also now taking multiple feeds in.

From a usability and user experience, clients will just get better pricing, no re-quotes as we roll this out across all of our asset classes.  Re-quotes will disappear, type of spreads, better spreads.  We’re able to handle volatile markets - fed announcements; all of that kind of stuff is no longer an issue for us. We’re probably one of the few retail players out there that if you’ll notice during any highly volatility period, CMC markets are usually there providing two way quotes.  So that, I think, is the upside for clients that our infrastructure is solid in times of high volatility.

Q14: Can you describe two way pricing?  It’s one of your differentiators right?

The ability to quote a two way price is not the issue - the ability to quote and allow you to trade at that price is the difficult part and our systems can do that - especially for FX, as we add more asset classes to our new infrastructure we should be able to do the same for all asset classes, hence the urgency to get everything done yesterday.

Q15: So what makes your pricing better?  Is it just the lower latency?

I think it’s a number of things.  Yeah, it’s latency, it’s flow. It’s lower latency, robust feeds, and having multiple feeds coming in.  The result is to offer the retail client almost institutional pricing, which is unique.

Q16: Can you put your latency improvements in numeric terms?

We’re able to handle 17,000 ticks per second from multiple sources.  Previously we looked at probably 3000 ticks per second.  This doesn’t make a big difference during normal trading days,  but when it volumes start to pick up - and this is only one asset class - we’re able to process this number of ticks and still be able to provide a two way price, a clean price you’re able to trade on in milliseconds.  That’s the upside.  Market data is a big issue for all the vendors - even the big players out there.  I won’t mention any names but during summer last year you will notice that there were massive delays in all of the larger players out there with respect to providing prices to clients in general.  And, again, we believe that now those kinds of delays have disappeared from our architecture.  So can I quantify the number of ticks per second?  We can handle the latency that we’ve got, yes, but it’s an improving target.  And one where, because of the StreamBase multi threaded architecture, we can run it across multiple CPU’s now.  If we ever get a bottleneck we just plug in another blade, pull up another service, and we get better latency and better pricing.

Q17: So the scalability is obviously great. Do you have any metrics you want to offer that quantify the latency you’re experiencing?

No, we don’t give them out. Speed is a strategic advantage for us, so we think those numbers are competitive information.

That’s all right, fair enough.

It’s good though and you can compare us on FX pricing to a bunch of our competitors.  And you will see our performance is noticeably better at times. Absolutely.

Q18: Okay.  You mentioned initially in the beginning the broader investment in your architecture.  Is that correct?

Yeah, we are in the process of rethinking a lot of our systems and our core systems at the moment.  This is just part of that.

Q19: What other applications will you use StreamBase for?

Yeah, the flexibility of the platform is the key thing for us.  There are lots of ideas - for example, surveillance.  Again, the concept behind surveillance is that you want to process a large number of trades and constantly be able to look at patterns. From what we know of StreamBase, it could lend itself well to that.  But, again, we’re going through pricing first then risk - the classics.  We’ll do the classics first and then all the other added value things we may want, because we have the infrastructure in place now.

Q20: And when you say when you look into the future you might use it for surveillance purposes, what that be looking at for clients?  Or is that more for looking at potential opportunities that you might have been missing or is it a bit of both?

It’s a bit of both actually.  Surveillance is a broad area. One of the things we do is look for people behaving badly. So you want to be able to protect the firm from people doing financial fraud.  You want to be able to stop fraud before it happens because you want to be able to flag that in real time.  StreamBase will be capable of that as well.  So there’s a number of projects, again, that we would look at after we’ve done the basics.

What makes this possible is that because the StreamBase infrastructure is there, the tools are there, the adapters are there to feed the platform. Then it’s a matter of just gluing it together.

Q21: Right, okay.  Very interesting.  Are you going to then be incorporating this into your market maker platform?  That’s the trading platforms that you guys have.

Yes.

Q22: So this is deployed across your entire enterprise?

Yes, it is global deployment.

Q23: How long did it take you to deploy the system?

6 months.

Q24: What were your requirements when you were looking for a solution and why did you choose StreamBase? 

We wanted two things: a scalable solution, and a flexible solution.  We’ve got a bunch of financial engineers that want to be able to prototype or even test different models and scenarios.  And we wanted that to be as easy as possible, so ease of use was a big part of the criteria. 

Q25: So you selected StreamBase CEP over other CEP products for its simplicity of development, performance, and scalability.  Is it working?

Yes - the flexibility of StreamBase is key to us - so much so that we’re finding new uses for it.  We originally brought it in for pricing and risk, and now that we understand the concept and some of the things it can do, it should facilitate other projects as well.

Q26: Can you talk about performance - was that about latency or capacity management?

With performance it was a combination of things.  A number of ticks, as you are probably well aware of market data, isn’t going down.  So it’s the number of ticks you can process and how you can scale across multiple different environments.  It’s the number of different feeds you can then handle.  That was part of the criteria. 

And then it was the ease of managing all of that performance infrastructure.  Clearly there are definitely products out there that can do some of these things, but it was the combination of speed and ease of use that was critical.  And at the end of the day for us, the overriding factor actually was the financial engineers signing off, because they could participate in the process and work with the StreamBase tools.  They felt we can make this product work for us.  We like this product.

Q27: When you were doing an evaluation of different CEP platforms, what did the evaluation entail?

There was performance, scalability and ease of use and how our financial engineers found the products in general.  So when we did look at the other guys, one of the nice things we liked about StreamBase was the development environment, StreamBase Studio.  The Studio environment is intuitive; it allows the financial engineers put models together. It’s graphical. You can connect it together quickly.  It’s quite novel and we like that. 

So StreamBase had to meet both the technical requirements as well as the financial engineers’ requirements.  And StreamBase met both those needs quite well.

Q28: You mentioned that you initially used at Apama.  Why did you drop Apama? 

As I mentioned, the ease of development for both financial engineers and technical engineers, the performance, and scalability of the platform were our key criteria. We dropped Apama because we liked StreamBase better from all of those criteria.

Innovation by the Numbers at CME Group

(Mark Palmer is the president and CEO of StreamBase Systems.  Follow him at twitter.com/mrkwpalmer)

Recently, I spent a day in Chicago with CME Group, the world’s world’s largest and  most diverse derivatives exchange, to talk about their approach to innovation.  As the CEO of an innovative software company founded by forward-looking MIT techies, I’m lucky enough to work with a lot of innovative people.  But as we talked about the size and scale of CME Group’s business, it struck me that this may be among the most technically challenging environments on planet earth, and the numbers tell a compelling story of challenge and opportunity for a firm that many know of, but few have seen inside. 

CME Group is the modern descendent of the Chicago Mercantile Exchange (CME), the American financial and commodity derivative exchange which was founded in 1898 as the Chicago Butter and Egg Board, and in the past 111 years, grown and transformed into the world’s largest electronic futures and options exchange (Globex).

But I wasn’t visiting CME Group to check out their museum, I was there to dig down on the business and technical motivation behind the news that they have selected a new development paradigm (press release, coverage in Wall Street & Technology), event processing (CEP), on which to build future innovations.  This is a watershed moment for the CEP industry, and an important case study that gives us pause to wonder about the future of automated business, the future of real-time, mission-critical infrastructure, and the role of event processing as the platform for that innovation.

So let’s look at innovation, CEP, and the challenges faced by CME Group: 

...1.2 quadrillion dollars ($1,200,000,000,000,000,000) 

                                           ...The underlying value of the contracts traded by CME Group in 2008

How can a firm stay on the innovation edge while simultaneously supporting the most mission critical operations?

The decision to process 1.2 quadrillion dollars of value was not taken lightly at CME Group. It’s been said that “nobody gets fired for buying IBM” – that is, that big companies are the safe choice when a decision is most critical. 

But when innovation must ride beside reliability, decisions get trickier.  A company like CME Group must balance innovation with mission-critical capability.  And, in the case of CEP, CME Group’s decision illustrates that CEP has come of age.

However, there’s been a rash of recent debate about the entrance of IBM and Microsoft into the CEP market.  Recently, the New York Times published an article, which declared that innovation is moving to big companies from small companies – a conclusion that is not only wrong, it flies in the face of all the most basic tenets of innovation.  So in the innovative, fast paced CEP market, we are wary of, but don’t fear, the entrance of giants like IBM and Microsoft.  Indeed, we welcome it, because it draws more awareness.  The more people look, the more will buy StreamBase.

And from the customer’s point of view   I asked Steve Goldman, CME Group’s Director of Enterprise Architecture, if he had looked at IBM and Microsoft, and his response was casual:  “The decision was very difficult but we felt that based on our environment and needs thatStreamBase was a proven solution.” 

11,000,000...

                   ...The current average daily volume of futures contracts traded every day by CME Group...

“Our data feeds are like fire hoses,” Goldman said.  “Sometimes, we process a few hundred market data changes a second.  But when the markets get volatile, we process tens of thousands of market data changes per second. And, to make it even more challenging, our customers demand the fastest response time – in milliseconds – when volatility is at its peak.” 

Real-time response is becoming more and more important in mainstream business.   Any environment that deals with events (stock market price changes, web site click data, and so on), can benefit from event processing.  And the technological challenge of managing events at high speed, and with low-latency, requires a host of arcane tooling and expertise.  These geeky areas will also repay deeper exploration, including specialized messaging hardware and software, clever use of modern computing models like multi-core CPUs, grids, and clouds, FPGA’s, and efficient management techniques to keep it all running smoothly.

But modern CEP engines are becoming an essential tool for managing enterprise-class streaming IT environments – a better way to manage information that comes through a fire hose.

Hundreds....

                ...The number of products offered by CME Group...

While most businesses might have a few products, or a big business might have a dozen products, CME Group has hundreds of products. But CME Group’s products aren’t made of physical materials; they are built with software development tools.  So the speed and effectiveness of software development at the CME Group dictates the speed and growth of their business.

So the news that CME Group has chosen a new enterprise-wide development paradigm is important news.   CEP, which has been rising to prominence in the past 5 years, flips traditional database-oriented computing on its head.  CEP allows an application developer to design logic that examines what’s happening now instead of what happened in the past.  Options pricing and settlement consumes vast amounts of streaming information, so the CEP model fits perfectly, and allows CME Group to quickly create new products that process those streams.

Any business that has streams of data – from web clickstreams to point-of-sale transactions to financial market data - can apply CEP to more quickly and easily bring new products to market.  When the development methodology matches the form of the data, innovation approaches the speed of human thought.

80...

        ...The number of countries where CME Group's Globex provides regulated exchange access...

Although everyone talks of the inevitability of a global economy,  few firms understand its practical challenges than CME Group. CME Group offers electronic, regulated exchange access in over 80 countries.

The host of technologies that enables a 7x24, global operations are many.  But CEP, by making streams of data from all over the globe more intelligent, can make network event processing smarter and faster, and is becoming an important element in enterprise IT infrastructure.

Which leads to a critical innovation issue:  how to strike the right balance between a new technology and a proven technology?  A full answer to that question is outside the scope of this article, but the key elements for enterprise software products include well-designed threading architectures, bullet proof fault tolerance systems, robust management tools, comprehensive testing and debugging tools, and profiling tools. Consider that CME Group traces its origin to 1848, has NEVER had a default or a loss of customer funds resulting from failure, today announced its enterprise deployment of CEP technology.   This is the clearest signal yet that event processing is coming of age as it continues through its rapid growth as a software market.

A few milliseconds....

                                ...the response time necessary at the most heavily loaded times....

To see the manual forms used at exchange over 50 years ago puts the innovation in the capital markets  in perspective.  It took many seconds to fill out the information about trades back then; it takes 400 milliseconds (1/1000 of a second) for the human eye to blink;  today, CME Group’s trading decisions take place much faster than the blink of an eye – in just a few milliseconds.  How can this kind of latency be achieved in the face of massive volumes and massive global distribution?

We discussed the role of CEP in turning the traditional database-oriented model on its head.  Databases, of course, rely on writing data to disk.  The typical seek time for a disk is about 9 milliseconds.  CME Group’s data can change tens of thousands of times per second;  in order to make decisions in a few milliseconds

Exchanges settling contracts, brokers splitting up and routing client trades, electronic liquidity providers making markets, hedge funds finding and executing arbitrage trades, banks delivering currency rates…all in less than one hundredth of a human blink!  Applications are measured in microseconds, and the tools that do the measuring have resolution in nanoseconds, that’s one billionth of a second.  

StreamBase has been emerging as the top CEP company for some time now due to its proven rapid development environment, standards-based approach, high performance, low latency, and mission critical, fault tolerant architecture.   CME Group’s selection of StreamBase is yet another proof point that demonstrates that CEP is ready for the big time, and that StreamBase is leading the way.

That’s why CME Group made this important decision for their real-time, low latency operations – the need for speed, and the need for CEP, is critical at CME Group.

Massive scale.  Instantaneous response.  Constant change.  Nonstop operations. Taken individually, these are big technical challenges; taken simultaneously, they are an extreme business risk.  But risk for one is opportunity for another, and as CME Group shows us, by keeping an eye on the three key “abilities” – usability, reliability, and scalability –firms can keep innovating as business conditions continue to get more complicated.

 

5 Observations about PhaseCapital's Selection of CEP

PhaseCapital, a Boston-based “quant” investment adviser, announced their selection of StreamBase as their CEP platform.  There has been a lot of interest in this announcement, for many reasons (e.g., here's the front-page coverage in Inside Market Data (warning:  IMD is password protected)). I was involved in some of the interviews with Eric Pritchett, CEO of PhaseCapital, and there were 5 especially interesting things that struck me:

1. Increased use of CEP as infrastructure.  PhaseCapital uses StreamBase at the heart of their high-frequency, low latency trading infrastructure.  It's a better way to build event driven systems, as Eric was quoted in the IMD interview:  "StreamBase technology allows PhaseCapital to abstract the rest of its trading architecture from the feed handlers themselves, enabling the firm to make changes to its data without having to re-engineer other parts of the system."  In the past year, we've seen an increase in firms deploying CEP as event-driven application infrastructure, rather than an embedded component of, for example, and algo trading application. 

2. Expansion of CEP use cases.  PhaseCapital's application of CEP is very broad - it's at the heart of their portfolio management system and risk engine, for U.S equities, futures, and equity options.  They use it for market data management of multiple exchanges to their trading system, and to order state handling and smart order routing. OK, I'll quote myself on this one:  "CEP isn't just for algorithmic trading anymore."  It's becoming a key part of enterprise infrastructure.

3. A comprehensive customer-driven CEP product evaluation.  PhaseCapital performed a comprehensive evaluation - they installed and tested "all the major CEP products," and ran real use cases against the products.  They looked at language, performance, usability, tools, and scalability of the CEP architectures.  Eric summed it up nicely here: "Our team consists of scientists and professional engineers; we are techies at heart, so we appreciate the value of a well-designed system that supports our development and deployment philosophy."

4. The continued rise of the need to automate.  In the article The Five New Technology Priorities, I wrote about the first priority:  automation.  In one interview last week, Dr. Geoff Goodell, who ran the evaluation, described how critical CEP is to their company:  "We're a small team of 15 at PhaseCapital - this CEP-based system IS our trading operation - everything is automated."  I think the need to automate systems is at the heart of StreamBase's recent growth - firms need to do more with less.

5. StreamBase:  the new market leader?   I've spent almost a year watching CEP evaluations play out from StreamBase's side and I'm starting to observe patterns of who's winning business and why.   In just the last six months, there's been a shift - publically, StreamBase has scored wins with BNY ConvergEx, RBC Capital Markets, BlueCrest, and now PhaseCapital.   That's more customer wins than all the other CEP vendors combined in the same period.   What's going on?   It seems to me that the field is separating into "first generation" and "second generation" CEP.  For example, most of Apama's attributes are first generation (single threaded server architecture, unmaintainable code generation for their CEP language, non-SQL oriented language, lack of debugging tools, and an adapter architecture that adds unacceptable latency due to its heritage of having CORBA embedded).  Need proof?  As Dr. Goodell described, Apama didn't even make the first cut in their evaluation.  On the other hand, Aleri was the other final CEP product in PhaseCapital's evaluation, partly because they fared well relative to Apama because of its multi-threaded server architecture. 

The real test of a market and products aren't what the vendors say about their product;  it's what their customers say about their product, and it's exciting to see the CEP landscape and buzz about new applications, new industry trends, and new product category leaders as the market continues to grow.

BlueCrest Deploys StreamBase, and Talking about Event Data Management

Serious discussion of event data management, CEP, and how they relate to event driven applications is a topic that's gone under the radar for a while.  Even though databases are always a central element of a CEP-based system, we haven't pushed the exploration of these use cases with any vigor.  That needs to change, and we've put a lot of effort this year at StreamBase into broadening the discourse about CEP beyond point applications like algorithmic trading.  Market data management is one such area.

This week's announcement about a new StreamBase customer, BlueCrest, helps broaden that discourse. BlueCrest deployed Vertica, and StreamBase as an intelligent, low-latency, aggregated market data management backplane.  Real-time, event driven data management is a great use of CEP, and brings the "event database" center stage in an event driven architecture (EDA).  In this case, the architecture is used to create a real-time, aggregated view of "everything that ticks" in the financial markets for BlueCrest.  The importance of this architecture is nicely put by Justo Ruiz-Ferrer, the head of trading strategy systems and quantitative development at BlueCrest:

“Great data management is as important to us as quantitative trading models.  The combination of StreamBase and Vertica enables us to get a low-latency, aggregated view of everything that ticks in the increasingly complicated, fragmented, high-speed financial markets.” BlueCrest said it wanted to be able to quickly reconfigure data feed connections and plug the information into real-time models.

"Great data management is as important to us as quantitative trading models."  What an extraordinary statement.  As the capital markets go through its massive gyrations, intelligent, low latency data management is becoming as important as their quantitative trading models.  More than ever, I hear electronic trading gurus say stuff like this - that the the quality of trading is only as good as the quality of the data.  With the markets constantly fragmenting, segmenting, compressing, and then breaking up again, it's harder than ever to keep up, and trade in new asset classes.   In the case of BlueCrest, their market data backplane manages "everything from traditional fixed income, equities, FX and related derivatives, to exotic commodity, energy and alternative asset classes."  All in real-time, in an event driven architecture.

Soon we'll release a comprehensive case study on this blog about the BlueCrest system to help share its interesting technical elements with the community.  More to follow on that later, but for now, here's the BlueCrest / Vertica / StreamBase architecture:

More to come on the role of serious event data management in EDA!

On the Requirements of Smart Order Routing (SOR) and CEP

The requirements of Smart Order Routing (SOR) applications are a good fit for complex event processing (CEP) because SOR applications need to evaluate a confluence of events, detect complex patterns of market events and manage feedback loops of trade execution state in real-time.  And they have to do all of this in a very low latency environment, with a lot of events.

Let's back up and look at the motivation for SOR.  The reason banks use SOR is because the very structure of trading is changing dramatically, and has been for years.  Regulatory changes in the equities market have spurred the rise of trading venue alternatives - ATSs, ECNs - dark pools of liquidity.  Traditional trading exchanges are shutting down because of these cheaper electronic options.  In the U.S. market there are as many as 30 ECNs today.  And in the unregulated FX market, dozens of banks trade electronically with each other, and there will be 15 electronic FX venues by the end of the year.  Moreover, the rise in interest in dark pools of liquidity add yet another complex set of events that must be detected, aggregated, combined, matched, and manipulated in real time.  It's not uncommon for a single SOR system to connect to 10 or more markets and multiple asset classes.  Not only is this a confluence of events, it's a stunningly complicated environment in which to create a complex, real-time model in which to apply "simple" routing decisions. On this basis alone, SOR needs CEP.

Now let's look at processing complexity.  One example of complex sequences of events that can be managed by CEP is in the area of order and execution state management.  Orders in many trading liquidity pools can be partially filled, and those partial fills are communicated by FIX messages and are returned to the SOR engine.  The FIX messages themselves are events.  SOR operates by analyzing the confluence of events from market data feeds, order flows from OMS systems, and executions, aggregating and analyzing those events in real time, and adjust routing decisions on the fly.  For example, an order for 10,000 shares of IBM might have 10 partial fills, from multiple markets, in a 3 second time window, and make several routing decisions in that time frame.  In other words, even just ONE order can spawn "a confluence" of related messages by itself!  These executions are often reconciled with the original parent order, and the SOR may alter its execution strategy (governed by CEP rules) as the fills are returned and the markets move.

Other SOR requirements that require sophisticated event pattern detection is for liquidity detection and optimization.  Bill Harts and Rob Almgren, former head of quantitative research with Bank of America, published a paper on liquidity detection and estimation that is predicated on CEP as an implementation engine.

Another critical requirement of SOR systems is high performance, scalability, and low latency.  Addressing the event processing challenges described above in the face of multiple streams of event data, the complex interactions created by the order flow process, and you get thousands of events a second against SOR logic, the requirement for millisecond-level latency.  A multi-threaded CEP engine is critical for this kind of application, because it's SOR typically monitor markets in their entirety, and not just for the best bid and offer, but even for level 2 quote data (which shows all the public quotes of market makers wishing to sell or buy stock and recently executed orders).

So the requirements of SOR are a great fit for CEP because it combines events from multiple streams, looks for patterns, and takes action - complex action that spawns more events a deeper confluence of events, and more action.  It does this in a high-performance, low-latency, complex environment. 

Not only is SOR a great example of a CEP application, it's arguable that CEP is mandatory for today's SOR.

Getting Back to Use Cases and Patterns for CEP

Giles Nelson from Apama blogged today about how depressed he was to see a useless debate about acronyms, SOR, and CEP.  But unfortunately Giles threw the baby out with the bath water!   My initial post was about why CEP is used by Smart Order Routing applications, and that's a useful - no, critical - discussion.

When I ran Apama, the focus of my blog was use case discussion, patterns, requirements, best practices,  and descriptions of the references customers that illustrated CEP at work.  I think the community has lost this voice, and we need to bring it back.  There are a lot of good things happening at SteamBase, Apama, Aleri, Coral8, and other CEP vendors.  It's our job to educate, although I'm guilty of jumping into the bath water about the difference between Routers and SOR. 

So I'm going to RE-post the baby (the SOR use case discussion), and throw out the bath water (the Router debate).  I hope the blogosphere joins in on other use cases and positive discourse about how CEP is being used.

Smart Order Routing and CEP - Made for Each Other

Blogs are dangerous, especially when theoretical arguments are used to make an ostensibly valid point. Such is the case in a recent posting that concludes that Smart Order Routing (SOR) is not a CEP application.  This is a misinformed posting, and I'd like to clarify why. 

First a little context:  at StreamBase, we've been applying CEP to a wide variety of SOR applications with dozens of investment banks - bulge bracket firms, tier 2 firms,  exchange / liquidity / ATS providers, and hedge funds:  our team builds SOR applications every day and have years of experience.

Let's examime the basis for the conclusion that SOR is not a CEP application, (excerpts from this CEP blog post):

"Routing an order kicked off by a simple order match against a deep liquidity pool (or other market factor) does not define complex event processing nor detecting a complex event - the core idea behind CEP.   Order routing based on simple rules is BPM, plain and simple."

This is a laymen's view of SOR, and lacks understanding of the complexity that a real system exhibits.  Sure, SOR applications make some simple routing decisions.  But the reality is that "order routing based on simple rules" is not modern SOR at all.  Moreover, it's light years away from a good BPM application.  Sure, "rules" are important (James Taylor's post points this out), but let's look deeper beyond a surface-level understanding of SOR.

Here's the next part of the argument:  "SOR is not a CEP because CEP deals with evaluating a confluence of events and then taking action."  SOR doesn't "evaluate a "confluence of events?"  This argument is now officially absurd, and dead wrong. 

Let's back up and look the motivation for SOR.  The reason banks are adopting SOR is because the very structure of trading is changing dramatically, and has been for years.  Regulatory changes in the equities market have spurred the rise of alternatives - ATSs, ECNs - dark pools of liquidity.  In the U.S. market there are as many as 30 ECNs today.  And in the unregulated FX market, dozens of banks trade electronically with each other, and there will be 15 electronic FX venues by the end of the year.  Moreover, the rise in interest in dark pools of liquidity add yet another complex set of events that must be detected, aggregated, combined, matched, and manipulated in real time.  It's not uncommon for a single SOR system to connect to 10 or more markets and multiple asset classes.  Not only is this a confluence of events, it's a stunningly complicated environment in which to create a complex, real-time model in which to apply "simple" routing decisions. On this basis alone, SOR needs CEP.

Now let's look at processing complexity.  "Dumb" order routing, the kind of routing the blogger describes, would make a route decision and emit a message.  But there is little value in that.  "Real" order routing is much more sophisticated.  For example, orders in many liquidity pools can be partially filled, and those partial fills are communicated by FIX messages and are returned to the SOR engine.  SOR operates by analyzing the confluence of events from market data feeds, order flows from OMS systems, and executions, aggregating and analyzing those events in real time, and adjust routing decisions on the fly.  For example, an order for 10,000 shares of IBM might have 10 partial fills, from multiple markets, in a 3 second time window, and make several routing decisions in that time frame.  In other words, even just ONE order can spawn "a confluence" of related messages by itself!  These executions are often reconciled with the original parent order, and the SOR may alter its execution strategy (governed by CEP rules) as the fills are returned and the markets move.

And there are many more areas of SOR that require even more sophisticated complex event processing -  one example is liquidity detection and optimization.  Bill Harts and Rob Almgren, former head of quantitative research with Bank of America, published a paper on liquidity detection and estimation that is predicated on CEP as an implementation engine.

Another critical factor that is ignored is the requirement for performance, scalability, and latency.  Add all these requirements to process multiple streams of event data, and the complex interactions created by the order flow process, and you get thousands of events a second against SOR logic, the requirement for millisecond-level latency.  A multi-threaded CEP engine becomes even more critical for this kind of application.  I've never heard any SOR practitioner ever utter the letters "BPM" in this context, it's simply a non-starter.

So SOR is a great application for CEP because it combines events from multiple streams, looks for patterns, and takes action - complex action that spawns more events a deeper confluence of events, and more action.  It does this in a high-perforance, low-latency, complex environment. 

Not only is SOR a great example of a CEP application, it's arguable that CEP is mandatory for today's SOR.