The StreamBase Event Processing Blog

The Subprime Crisis and Real-Time P&L

In the past few weeks I've been in New York and London, visiting investment firms who use complex event processing (CEP) technology in their trading operations - before one visit, I was surprised to be briefed that the technical team we were visiting had built a "real time P&L" system. I was surprised because P&L management always seemed to be a very basic element of every algorithmic trading system I've worked on in the past.

Last year, John Bates and I wrote in the Trade about how the electronification and rise of algorithmic trading and regulation were starting to "push the front and back office closer together." At the time we were thinking specifically about compliance, although TCA, best execution, and risk are clearly inter-related when it comes to electronic trading. P&L was one of the functions we saw then starting to creep into the front office function of trading, often in the form of pre-trade TCA which was being used as part of algorithmic execution strategies.

So to find real time P&L as a common stand alone function was a bit of a surprise to me, but in talking to the firms deploying it, it became clear that there's another force at play: the sub-prime crisis. Billions have been lost inside of a day - it's not longer acceptable to wait to the end of the day to gain insight into the firms' trading position, when it may be too late.

CEP uniquely enables real-time P&L. Because an event processing platform can ingest or output any event stream, a firm can add their own feeds, their currencies, their own price roll-ups or their own netting options. Since CEP languages like StreamSQL are open and extensible, developers can change their P&L valuation algorithms with multiple rule sets for each instrument or book, or change the alert thresholds. Indeed any P&L rule or configuration definition can be amended and maintained by the investment firm. And, event processing rule filters can be cascaded so that output signals can be used in subsequent cycles to change the P&L calculation depending on earlier market dynamics.

From an IT perspective, the ability to change parameters or even to add new rules during real time operations provides for a deeper level of flexibility and dynamism. For example, a news flow item might trigger volatility requiring a different analysis, or a sudden loss of profit may demand a more granular analysis to track down the problem. The ability to alter and deploy rules during the day allows firms to adjust how they formulate their real-time view of the market.

So it's becoming clearer why real-time P&L and event processing are being more strongly linked: the simultaneous need for intraday insight, combined with the continued maturity of event processing platforms that can implement it in an open, flexible, and dynamic way.

CEP: It's Not Just About Algorithmic Trading

Being at Apama for so many years, I was fortunate to learn about the applications being built with Complex Event Processing (CEP) technology. The academic or deep technical issues of CEP, although fascinating, are well covered by others - I was more interested in the business value of CEP. That is, when a business person considers buying paying money for this technology, how do we answer the question: "Yes my techies tell me CEP is neat and new, but - so what? How will it impact my business?" So naturally, now that I'm with StreamBase, my first order of business is to get back on the road and see what's new through the eyes of a very different product and company. I'm looking to answer the "so what" question from a fresh perspective.

Apama is very public and open about its adoption for algorithmic trading. (Disclosure: my comments about Apama are restricted to public information, given my direct involvement in running the company). Indeed, algorithmic trading was so dominant a use of Apama that some wondered if algorithmic trading was the only application of CEP. And the deep customization of Apama as an algorithmic trading solution only emboldened this perception: of the 150+ articles, blog posts, etc. I did as general manager of Apama, roughly 95% of them were about algorithmic trading.

But looking through the eyes of StreamBase customers, for me, has shed a new light on the business question of "so what?" It's already been a fascinating look into the broad application of CEP. Naturally capital markets has adopted StreamBase aggressively; but what's new and different is the wide variety of use cases, far beyond algorithmic trading. Crossing engines, real time P&L, smart order routing, more generic, supporting applications in the trading space like trade signaling, alerting, and monitoring. Market data cleansing and optimization. And, since StreamBase has historically been a horizontal, general purpose platform, the applications outside capital markets in the government space, such as intelligence and defense applications, are fascinating as well. More on these specific use cases in the future.

On the other hand, one thing has remained consistent: the core value proposition, or answer to "so what?" It's never been about "feeds and speeds"; speed and scalability are just table stakes for mission critical applications. The core value of CEP is that it empowers business users. Graphical tools allow a business analyst, or even a pure business person, to quickly express their "secret sauce" - their knowledge of how they might best react and capitalize on changing business conditions. CEP empowers business users to make better decisions, more quickly than ever before. More quickly than their competition. More intelligently than the next guy.

I'm seeing customers quickly imagine, express, test, deploy, and evolve their most innovative ideas, ideas that can help them better understand and react to what's happening now, while the opportunity to seize the advantage still exists. That's an important answer to the question: "so what?" And I'm finding it's an answer any business person can appreciate.

On the Evils of SQL and CEP

OK, now that I've settled in at StreamBase, it's time to start to address the number one question I get - this "Mark and Colin hate SQL" thing.  It's come up immediately and often in my first 5 days here, including, for example, this post from Brenda Michelson where she discussed the recent goings on at StreamBase.

The first thing I had to do is check what Colin and I actually said!  I found the original post in the Apama blog on the topic; ironically enough, Colin jumped in and responded.   So, here we are - right on the record:

On CEP and Standardization 

Re-reading this discourse, I am happy to report that both:

1) I still believe exactly what I wrote a year ago, and

2) There is no contradiction in now being at StreamBase

The point of this post, and my historical (apparently colorful and memorable) objection to StreamBase, was simply that they had run off and assumed that SQL was the only approach to a CEP language. The core of the issue is that there are many relevant ways to build a CEP language, including:

• Graphical point-and-click environments
• CEP rules
• CEP languages
• Java
• SQL-like syntax

There was one more point made in that post that bears repeating, although I think it's fitting to make one small edit:

At Apama [StreamBase], we will continue to avoid polarizing discussions about standards, and participate in the nascent standards discussions that are inclusive of all approaches.  We'll also continue to advocate other important issues like customer use cases, the business value of those use cases, architectural discussion about the relationship between EDA and SOA, continued work on the event processing glossary, and more.

Constructive discourse that unites, not divides, will help the tide rise for event processing, and lift all boats along with it - customers, vendors, and more.

Stay tuned for future posts on the beauty, elegance, and superiority of SQL for CEP ;-)

 

Thank You

Mark Palmer here - formally of Apama, now with StreamBase.

8 months ago, I wrote my last blog entry about complex event processing (CEP) after years of work thinking and writing.  It was a passion - many of us in the industry believe that CEP technology is a fundamentally new computing "physics" that has the opportunity to change the software industry.  The CEP market was growing like crazy and things were going great.

Then, abruptly, I left the blogosphere.   

Blogs are often personal, yet my focus has always been professional (although some at StreamBase have argued this point!).  But today, I feel compelled to get personal.  For just one post.

The primary reason I vanished so abruptly was that my wife died last year after an 18-month battle with cancer.  I have two great kids - Jack, 7, and Ruby, 5, and, naturally, my attention turned exclusively to my family.  While I was gone, I was fortunate to have tremendous support from friends, family, and the cancer community.

But even though I had left work and the CEP scene, I was surprised to receive ongoing support from the software community.  For example, when I returned to work, I checked my "old" blog - I was surprised to find that you had kept reading - over 40,000 people had read my writing on the Apama blog.   I doubt many visitors knew what was going on for me, but I appreciated that others appreciated the work. 

And, when Sue passed away, I decided to run the Boston Marathon (just 11 days away now) for charity - specifically, for Dana Farber.  Sue was treated there, and it is an amazing place.  Frankly, I hope you never find out how amazing it is.  But, again, I felt the support of the "work" community - in memory of Sue, and for the benefit of the Dana Farber Cancer Institute, over 60% of the $29,000+ I raised in her memory came from colleagues who didn't even know her.   And now that I've re-emerged, people from the "work" community continue to support Dana Farber in her memory (here's the link for my Dana Farber Boston Marathon donation page in memory of Sue).  I would never have imagined this kind of support and good will would come from the people I've met over the years at work. 

I've been struggling to figure out how to re-enter writing about CEP, because I couldn't figure out what to "write first."  Today I finally figured out why - there was something I had to say first:  Thank you.

- Mark Palmer