The StreamBase Event Processing Blog

Why the CEP Tide Has Turned Toward StreamBase

Last week, CMC Markets announced that they switched from Apama to StreamBase as their enterprise-wide event processing platform. As FXWeek reported, CMC “initially spent three months deploying a similar system from Apama, but dropped the deal in favor of StreamBase.” As a firm that trades over $1.4 trillion a year, CMC’s decisions about their infrastructure are not taken lightly. So their decision gives us pause to ask: why the switch? 

Asif Adatia, Chief Information Officer for CMC, described their decision like this: “the ease of development for both financial engineers and technical engineers, the performance, and scalability of the platform were our key criteria.” Let’s examine each part of that statement.

1. Demonstrably better ease of use. You might think that the efficacy of a development environment would be a subjective call. In some ways, it is - when it comes to the visual esthetics of a tool. On the other hand, usability is often the driver for market-altering innovation - profound shifts in the accessibility of a new tool can suddenly unlock barriers to innovation. The change in the last 10 years in finance has been profound. 10 years ago, algorithmic trading wasn't even named, now 70% of the U.S. equities trade execution happens algorithmically. In their report "The World According to Quants: Enter Alpha Generation Platforms", the AITE group reported that certain trading algorithms have a shelf-life of less than 3 months. In the face of all that rapid change and ephemeral opportunity, the ability to quickly build real-time trading infrastructure (algos, execution, risk, compliance, pricing, TCA, surveillance) is a matter of survival.

Firms like CMC, CME Group, and RBC didn't choose StreamBase for its aesthetics, they chose StreamBase because its graphical development environment gives them a fundamental competitive edge. The tools aren't just easy to use, they allow business users and engineers to work together because the language of StreamBase matches the way they think. The StreamBase dataflow language was designed from the beginning to program event-based systems the way you would draw them on a whiteboard, and the proof lies in what customers say, as Adatia told FX week, “Our financial engineers find it really easy to put the models together.” 

And this ease of use delivers another critical innovation: firms get this usability and time to market advantage and the performance benefits of event processing - at the same time. StreamBase isn't like other CEP tools that make the developer resort to low-level programming to make the system perform and scale. Relieving the user from the burden of thinking about plumbing was described by Adatia to A-Team Insight “[CMC] create their models readily, without resorting to code-level programming.” PhaseCapital reported that they built their trading system with StreamBase in 4 months rather than the 4 years it would take to build it with traditional development tools and they change their strategies every day. These are not subjective opinions about aesthetics, they are qualitative and quantitative testimonials to the innovation that StreamBase's interface can provide.

2. Built-in performance you can see. On Wall Street, milliseconds, and microseconds matter. CMC switched to StreamBase because they wanted the fastest engine they could get. 

StreamBase was designed for speed from the very beginning. Release 1.0 in 2005, was designed to take advantage of modern hardware, from the multi-threaded server to the StreamBase language which produces highly performance applications without burdening the developer. The StreamBase language plans performance like a database pre-processor plans query optimization. In 2007, StreamBase engineers broke down another barrier with a straight-through event processing compiler. Recently, in 2008, direct bytecode generation optimization took the StreamBase performance innovation even further ahead of all other CEP platforms.

When performance is something you can see, it's a remarkable result. CMC said it this way: '[we don't talk about our specific latency numbers publically, but the performance is] good though and you can compare us on FX pricing to a bunch of our competitors.  And you will see our performance is noticeably better at times. Absolutely."  The world's largest electronic futures and options exchange in the world, CME Group chose StreamBase after using performance as a critical selection criteria, because they process over 11 million trades a day.

3. Scalability to grow. You might assume better scalability means StreamBase-based applications can be run on multiple CPU cores, and that’s true - better use of hardware means CMC needs less hardware. CMC described that their StreamBase system easily handles "More than twice the real-life peak rate for price ticks encountered so far.”  

But better scalability isn’t only about increasing volumes - it’s about ease of management, ease of customer support, and the ease with which high performance and scalability can be attained by developers.

And scalability is also about expanding usage. Apama established early on algorithmic trading as an important use case for CEP. But now, as CEP's use explodes into new application areas, firms are looking more deeply at how easy it is to build their second, third, and fourth applications. Just last week we had a call with a customer who guessed they had deployed "about 100 StreamBase applications." 

And now we have come full circle back to the StreamBase development environment. Our mantra is to “deliver high performance and scalable applications without having the developer think about performance and scalability.” Streambase makes performance native, which is one more reason it is unique.  

CMC is not alone. Organizations that are adopting StreamBase CEP are seeing these same benefits in ease of use and performance. CEP is the disruptive technology that frees their engineers to focus on creating business value instead of plumbing. CEP empowers their business analysts to work with the actual software instead of wishlists and design documents. CEP allows the whole organization to be more agile, to deploy changes to their infrastructure faster, and to take advantage of new opportunities ahead of the competition.

Q&A with CMC Markets: Trading $1.4 Trillion on CEP

This week CMC Markets announced that they have deployed complex event processing (CEP) in their global trading infrastructure that processes $1.4 trillion a year on over 5,000 trading instruments and over 115,000 trades a day. 

You might be thinking - $1.4 trillion in trading? Who is this CMC Markets? CMC is the leading global provider of retail contracts for difference (CFDs) and one of the largest financial spread betting leaders. CMC is a sophisticated, distributed organization - according to Inside Market Data, "the StreamBase platform is being rolled out across CMC's global operations, including offices in Australia, Canada, China, Germany and the UK, and is supporting a wide range of applications including its market-making, risk management, and hedging tools to support trading in FX, indices, futures, commodities, and equities."

This kind of global, mission critical commitment to CEP is important news, so we spoke with CMC’s Chief Information Officer, Asif Adatia, to find out more about CMC, their needs, why they chose CEP, their business results, and their future plans with CEP.

Q1: The AITE Group report from July, 2007 said your average trade volume was about $100 billion monthly.  Is that correct now?

That’s an old number. Now we’re at about $200 billion a month or $10 billion a day.  For example, if I look at today’s stats we’ve done $7.5 billion now at trading shops.

Q2: Have your trading volumes increased a lot over the last couple of years?

I think all of our stats have increased.  We’ve got more clients, more trades per day, which explains some of the re-architecture work we’re doing.  Yeah, we’re doing very well at the moment.  This business fundamentally is doing very well from a key metrics perspective.

Q3: Right.  Can I ask, were your upgrade plans hampered at all this last year by the recession or have you been able to carry on?

Peter Cruddas, our CEO, gave us massive support from a technology perspective.  If anything I’ve got more to do. We put a plan together, the board bought into it and it’s pretty exciting.  For me it’s phenomenal, we’ve been able to do everything we wanted to do.  This re-architecture is an effort to take our business to another level.

Q4: So there hasn’t been anything that had to be put on hold because of the crisis?

No, definitely not from a technology perspective, no.  If anything, the question is “can I do more and get it done faster.”  I’ve certainly not had restrictions.

Q5: This re-architecture is replacing your existing in-house built architecture.  What other technology upgrades are you making?

We’ve got a whole style change from our front end and our transactional engines to our back office.  There’s a program over the coming months where we are upgrading all of our systems.

Today, StreamBase is doing pricing, and it’s far more sophisticated than our legacy market maker platform we previously had.  Ultimately, StreamBase will replace a lot of the intelligence behind the old system.

Q6: Can you give me a couple of examples?  And over what period you will conclude?

So the back office as an example, we started to work on that. We’ve got a batch orientated back office currently, and we’re in the process of creating a real time P&L and cash balances being realized in P&L.  Today you wait until the end of the day, you mark the market and you adjust your P&L’s and you realize your P&L then. We’re moving a real time back office.  The timing is probably end of year. And then again because it’s so different, there will be a coordinated effect with our marketing and our client service guys.  Because it is a big difference from the way we have been working. That’s just one example of many.

Q7: Can you describe the project StreamBase is supporting?

StreamBase supports FX for all of our major currencies, exotics, as well as forwards.  It will support indices futures, futures on indices, cash commodities, treasuries and equities, single stock.

Q8: Can you describe how the StreamBase application processes data and pricing for CMC?

We cleanse our prices as they come through. We get multiple feeds from various brokers.  We cleanse our prices and we’ve got a reasonably sophisticated model where we look at groupings - if there are multiple feeds coming in and you want to take the best price, you want to take out the ones that are outliers. So the mathematics behind it are sophisticated - it doesn’t just do a standard mean, it takes a look at standard deviation of all of these prices and then looks at the volatility over multi-factor, high frequency scenarios. Then we start to look at which ones are outliers.

As I said StreamBase helps us look at what’s happening and very quickly take algorithms in and out - fix them and put them back in as well.

Q9: What are your FX market data sources?

I can’t give you the names but it’s primarily the usual suspects.  Tier one brokers provide liquidity and pricing to firms like ourselves.  We’ve got quite a few brokers that we deal with and their price feeds. We’ve also got feeds for other asset classes.  And along with the brokers, we have multiple data feeds from vendors, exchanges, alternative ECNs and OTC price providers

Q10: Are you using the StreamBase adapters to connect directly to those price feeds?

Yes, it’s actually is a combination.  StreamBase provides feed handlers.  So very quickly we were able to connect and use the StreamBase feed handlers. And then we’re in the process with some of our brokers, again, StreamBase allows us to connect to our own feed handlers so we do that.  So as I said at the beginning flexibility was one of the key criteria.  StreamBase offers that to us as well.  So we can mix and match.

Q11: Are you live? 

Yes, we’re live now.

Q12: You mentioned scalability and performance - what kind of demands are you putting on the StreamBase system now?  I’m thinking updates, latency, that kind of thing?

So at the moment we’re able to process 17,000 ticks per second on FX.  It doesn’t really hurt the system or the processes that much.  And we’ve got it scaled up to for twice as large as that. Can it handle our peak? Our peak to date has been nowhere close to 17,000 at the moment.  We’ve got what we call a non-functional regression test environment where we just overload it.  We overload our environment and haven’t broken it.  We've got so much going on right now that if it can cope with double our peak, we can focus on other things.

Q13: So what kind of a difference will users notice?  What’s going to be different for them? 

If you look at FX pricing live right now our spread is pretty much in line and competitive.  We used to have an issue back in before we implemented some of this.  We used to have issues with latency.  Those problems have disappeared now.  We’re able to process a lot more - about 17,000 ticks a second, which is pretty cool. We’ve still got plenty of headroom as well.  And we’re also now taking multiple feeds in.

From a usability and user experience, clients will just get better pricing, no re-quotes as we roll this out across all of our asset classes.  Re-quotes will disappear, type of spreads, better spreads.  We’re able to handle volatile markets - Fed announcements; all of that kind of stuff is no longer an issue for us. We’re probably one of the few retail players out there that if you’ll notice during any highly volatility period, CMC markets are usually there providing two way quotes.  So that, I think, is the upside for clients that our infrastructure is solid in times of high volatility.

Q14: Can you describe two way pricing?  It’s one of your differentiators right?

The ability to quote a two way price is not the issue - the ability to quote and allow you to trade at that price is the difficult part and our systems can do that - especially for FX, as we add more asset classes to our new infrastructure we should be able to do the same for all asset classes, hence the urgency to get everything done yesterday.

Q15: So what makes your pricing better?  Is it just the lower latency?

I think it’s a number of things.  Yeah, it’s latency, it’s flow. It’s lower latency, robust feeds, and having multiple feeds coming in.  The result is to offer the retail client almost institutional pricing, which is unique.

Q16: Can you put your latency improvements in numeric terms?

We’re able to handle 17,000 ticks per second from multiple sources.  Previously we looked at probably 3000 ticks per second.  This doesn’t make a big difference during normal trading days, but when volumes start to pick up - and this is only one asset class - we’re able to process this number of ticks and still be able to provide a two way price, a clean price you’re able to trade on in milliseconds.  That’s the upside.  Market data is a big issue for all the vendors - even the big players out there.  I won’t mention any names but during summer last year you will notice that there were massive delays in all of the larger players out there with respect to providing prices to clients in general.  And, again, we believe that now those kinds of delays have disappeared from our architecture.  So can I quantify the number of ticks per second?  We can handle the latency that we’ve got, yes, but it’s an improving target.  And one where, because of the StreamBase multi threaded architecture, we can run it across multiple CPU’s now.  If we ever get a bottleneck we just plug in another blade, pull up another service, and we get better latency and better pricing.

Q17: So the scalability is obviously great. Do you have any metrics you want to offer that quantify the latency you’re experiencing?

No, we don’t give them out. Speed is a strategic advantage for us, so we think those numbers are competitive information.

That’s all right, fair enough.

It’s good though and you can compare us on FX pricing to a bunch of our competitors.  And you will see our performance is noticeably better at times. Absolutely.

Q18: Okay.  You mentioned initially in the beginning the broader investment in your architecture.  Is that correct?

Yeah, we are in the process of rethinking a lot of our systems and our core systems at the moment.  This is just part of that.

Q19: What other applications will you use StreamBase for?

Yeah, the flexibility of the platform is the key thing for us.  There are lots of ideas - for example, surveillance.  Again, the concept behind surveillance is that you want to process a large number of trades and constantly be able to look at patterns. From what we know of StreamBase, it could lend itself well to that.  But, again, we’re going through pricing first then risk - the classics.  We’ll do the classics first and then all the other added value things we may want, because we have the infrastructure in place now.

Q20: And when you say when you look into the future you might use it for surveillance purposes, will that be looking at clients?  Or is that more for looking at potential opportunities that you might have been missing, or is it a bit of both?

It’s a bit of both actually.  Surveillance is a broad area. One of the things we do is look for people behaving badly. So you want to be able to protect the firm from people doing financial fraud.  You want to be able to stop fraud before it happens because you want to be able to flag that in real time.  StreamBase will be capable of that as well.  So there’s a number of projects, again, that we would look at after we’ve done the basics.

What makes this possible is that because the StreamBase infrastructure is there, the tools are there, the adapters are there to feed the platform. Then it’s a matter of just gluing it together.

Q21: Right, okay.  Very interesting.  Are you going to then be incorporating this into your market maker platform?  That’s the trading platforms that you guys have.

Yes.

Q22: So this is deployed across your entire enterprise?

Yes, it is global deployment.

Q23: How long did it take you to deploy the system?

6 months.

Q24: What were your requirements when you were looking for a solution and why did you choose StreamBase? 

We wanted two things: a scalable solution, and a flexible solution.  We’ve got a bunch of financial engineers that want to be able to prototype or even test different models and scenarios.  And we wanted that to be as easy as possible, so ease of use was a big part of the criteria. 

Q25: So you selected StreamBase CEP over other CEP products for its simplicity of development, performance, and scalability.  Is it working?

Yes - the flexibility of StreamBase is key to us - so much so that we’re finding new uses for it.  We originally brought it in for pricing and risk, and now that we understand the concept and some of the things it can do, it should facilitate other projects as well.

Q26: Can you talk about performance - was that about latency or capacity management?

With performance it was a combination of things.  The number of ticks, as you are probably well aware, of market data, isn’t going down.  So it’s the number of ticks you can process and how you can scale across multiple different environments.  It’s the number of different feeds you can then handle.  That was part of the criteria. 

And then it was the ease of managing all of that performance infrastructure.  Clearly there are definitely products out there that can do some of these things, but it was the combination of speed and ease of use that was critical.  And at the end of the day for us, the overriding factor actually was the financial engineers signing off, because they could participate in the process and work with the StreamBase tools.  They felt we can make this product work for us.  We like this product.

Q27: When you were doing an evaluation of different CEP platforms, what did the evaluation entail?

There was performance, scalability and ease of use and how our financial engineers found the products in general.  So when we did look at the other guys, one of the nice things we liked about StreamBase was the development environment, StreamBase Studio.  The Studio environment is intuitive; it allows the financial engineers to put models together. It’s graphical. You can connect it together quickly.  It’s quite novel and we like that. 

So StreamBase had to meet both the technical requirements as well as the financial engineers’ requirements.  And StreamBase met both those needs quite well.

Q28: You mentioned that you initially used Apama.  Why did you drop Apama? 

As I mentioned, the ease of development for both financial engineers and technical engineers, the performance, and scalability of the platform were our key criteria. We dropped Apama because we liked StreamBase better from all of those criteria.

Press Releases: Dead or Reborn?

A colleague of mine from a press agency asked for my thoughts about a recent study where 45% of respondents said that press releases are losing their relevancy. As the CEO of a company that cares deeply about how my company communicates, and who believes in social media, I've written that I think that old school PR is dead. So it might not be surprising to some to hear that I think most communications professionals have it backwards - again.

Press releases aren't dead, their role is being reborn. Social media is immensely powerful. But that power enables the transmission of misinformation just as it enables the transmission of the truth. A press release is one of the few remaining versions of the truth. 

In the old days of communications, a press release was for the press. Not anymore. Press releases used to be teasers - a tool written by PR professionals and used to get the attention of a writer. This use of a press release is waning in importance. And it's why, as the survey suggests, as many of 97% of press releases sent to media outlets are never acted on. 

New school communications recognize social media as increasingly important sources of opinion. Blog posts use press releases as the main source of the truth.

New school communications are faster. But tweets are ephemeral; press releases are forever. Good press releases are carefully considered - it can take weeks to edit them because they document major milestones and events.The integrity of a tweet is a function of the integrity of the twitter, and some tweeters are twits. Press releases are the official record.

New school communications have brought the "man on the street" into the discussion, which complements a company's view (via the press release), industry commentators, critics, and investigators.

In the new school era of mass opinions, press releases hold more weight than ever before. For example, I work in the financial services industry, and banks are notoriously secretive. Getting a bank to agree to a press release is a big deal. The very existence of a press release is news in and of itself.

Of course, the public is turning to other sources of information. Of course press releases can become over processed by corporate PR filters. Of course old school PR is dead. But press releases are the official record of business, and in the social media era, press releases set the record straight, because they are the record.