The StreamBase Event Processing Blog

Real-Time P&L Best Practice, Motivation, and More

I’ve had the good fortune of working with Bob Giffords, the former CTO of a leading bond trading platform and a well-known industry spokesperson in the financial markets in EMEA, on a paper on Real Time P&L.  The paper will be published soon, but in advance and as an augmentation, the web is a great resource.  The web has the advantage of hyperlinking to other resources, for a more interactive style of communication, and to solicit feedback, which is always welcome and is constantly offered by posting here even if it was not appreciated! 

The paper begins by discussion the motivation for real-time P&L systems, the requirements for such systems, some architectural best practices for deployments of such an architecture based on CEP, a case study, and a view of moving forward – what lies ahead in this application area.

First up, a discussion of the motivation for real-time P&L

Motivation for Real Time P&L

The financial markets are accelerating: transaction volumes are up, latencies are down, complex cross asset trading up, revenue margins down. Recently markets have seen sudden spikes in volumes, and nervous volatility when the old rules of thumb broke down. Technology and global regulators have both changed those rules by increasing transparency, intensifying competition and multiplying e-commerce relationships exponentially. Reforms such as Reg NMS in the US and MiFID in Europe have further increased the pressure, along with Basel II and the fair value accounting rules of the new international financial reporting standards (IFRS). The IFRS require, for example, firms to mark more of their assets and liabilities to market, while Basel II is much more explicit about risk adjusted capital reserves needed. Now, when markets move, traders need to catch them on-the-fly to cut their losses and go with the flow to ensure compliance with all the rules and customer mandates. The difference between just-in-time and just-too-late has just got a lot bigger.

It was not so long ago when daily profit and loss (P&L) and risk evaluations were state of the art. In a multicurrency world with ever more automated robotraders and instant, electronic global news, they are now painfully slow. This impacts not only the high frequency traders or global market makers trying to balance thousands of real time orders and quotes against a flood of market data with awesome peaks at tens or hundreds of thousands of messages per second. 

Now the credit squeeze of 2007 has renewed the focus on costs and risk. It demonstrated that some firms can lose billions in a single day, algorithmic trading systems can get confused in highly volatile markets, and end of day positions may mask serious intraday exposures. So now even traditional investment managers are keen to optimise their trading performance and track the impact of intraday movements and fluctuating daylight exposures. If risk can change by the second, traders, investment managers and compliance officers too need to track their P&L in real time in order to provide a sensible framework for judging risk as a key factor in making trading decisions during the day. Not to have at least some idea of intraday performance is like flying blind.

Of course, some trading tools and broker algorithms already have real-time analytics. However, these all work differently and firms typically employ multiple solutions and lack ways to integrate them into a cross-platform view of risk.   In addition, the increased adoption of cross-asset trading further complicates the real-time risk picture, and motivates the need to assess P&L across the enterprise. 

However, is such a firm-wide portfolio map, marked to market in real time, really feasible, however desirable? Would it not absorb huge amounts of technology resources and be meaningless anyway, due to insufficient or stale market data? Would not the overhead of software maintenance inhibit innovation and the trading of new asset types like credit default or volatility swaps, so helpful for total return funds.

Agile investment firms are in fact demonstrating that real-time P&L is not only feasible, but highly lucrative as well. It opens up investment opportunities and trading styles, which might otherwise be excluded as too risky or too  demanding in resources.

New technologies like complex event processing (CEP), applied as a “white box” systemfor real-time P&L, have turned tracking real time market movements into a practical proposition for ordinary firms. They can also play a key role in easing the transition to fragmented markets with their notorious dark pools and to the complex array of order, execution and liquidity management systems to support them.

Next section:  The Requirements of Real-Time P&L

CEP Myths: Mature or Not?

Ivy Schmerken recently wrote an article on CEP in Wall Street & Technology called "Deciphering the Myths Around Complex Event Processing."  The myths were:  1) the technology is mature, 2) it's sell-side only, and 3) it's "out of the box and easy to use."   

I thought myths 1 and 3 were well argued, but I couldn't find support for the claim that CEP isn't mature. Quite the opposite, actually.  For example, Ivy cited AITE group's Adam Honore's opinion that CEP's use is expanding and cited a number of the public references like the FSA and Turquoise.  I wrote an entry on this topic a few weeks ago as well based on initial visits with StreamBase customers and their usage.  Public disclosure of initiatives like this argue for maturity, not immaturity.

Marc Adler from Citi said: "CEP is not this golden pill that you take and everything is solved."  True, but this should be a safe statement about any kind of technology - you get out of it what you put into it, and especially in the sophisticated realm of capital markets software systems.  Indeed, the longest pole in any CEP project is integration with the various OMS, EMS, middleware, and database components that often play a big role in any system.  Pragmatic understanding of what it takes to deploy CEP, in my book, is a good thing, and Marc's comments are a helpful reminder of the best practice to apply sound engineering practice, especially when it comes to a relatively new technology like CEP.

I've spoken with Ivy numerous times over the year, and she has always written thoughtful articles, so I'd guess she would respect this rebuttal - frankly, I think the content of this article argues the opposite conclusion - that CEP is robust and maturing and reliable, not "immature."

Perspective on Today's Vhayu / StreamBase Partnership

Today our press release hit the wire describing a partnership with Vhayu.   Here's the link to the release on the Vhayu web site:

Vhayu and StreamBase Partner to Provide Real-Time and Historical Market Data Solutions for Hedge Funds and Investment Banks

There was significant press coverage of this release, I think for two reasons.  First, partnerships like these are often "Barney" relationships - as in, vendors who issue press releases that essentially say:  "I love you, you love me, we're a happy family" (if you don't have kids or don't know Barney, check out his famous song here.)  This isn't a Barney release;  this is a partnership driven by shared customer need.

The second reason this release is significant, and most importantly, is the why it's important to our customers.  Customers in capital markets care because they must "apply better analytics on shorter timescales." (quoting the press release).   Specifically, Vhayu provides the ability to back-test strategies before they are put into the live market, and identify new strategies, and evolve the intelligence of their proprietary trading, compliance, and risk techniques;  StreamBase allows developers to compose, deploy, test, debug, and evolve those strategies on streaming data.   Increasingly, the cycle time for the discovery, development, deployment, and evolution of CEP-based automation is THE basis for competition among firms as they race to be the first, smartest mover in the increasingly electronic markets.  Firms that employ both historical tick stores and CEP get to market  quickly, and stay ahead by continuously "genetically tuning" their techniques.

Partnership activity is a sign of a healthy, growing CEP market, so it's good news for everyone.

Event Processing in the FT Misses the Action

Over the past few years, there has been a noticeable increase in awareness of event processing. This week the Financial Times (FT), one of the most widely read papers in the world, did a nice article on event processing called Event processing: A monitor that can set alarms ringing, quoting StreamBase and our friends at Apama. Algorithmic trading, real-time P&L, and real-time logistics systems done at Dell Computer were all cited as examples of applications that benefit from CEP today.

The article was great, but one flaw jumped out to me, probably due to the nature of the Dell Computer case study cited - the fact that these applications are simply monitoring for changes in event streams. In my first month of travel talking to StreamBase customers, I'm finding the same kind of applications that we had with Apama that do much more than simply monitor streaming data and firing off advisory signals to humans.

It is true that many of the patterns detected by a CEP engine are used as decision-support - to inform a human. Real-time decision support is an important business requirement in most CEP applications. But, at the same time, CEP can automate action that might not even be possible if left to manual intervention. For example, last week I met with a hedge fund who uses CEP monitor a firm's intraday P&L. The system monitors trading activity during the day. When P&L moves too quickly toward its internal limits, the system alerts the head of the trading desk. This is the decision support element of the system. But, at the same time, StreamBase rules can automate action, depending on the nature of the risk. For example, orders above a given size (specified by data that drives the CEP rules) can have their order state automatically altered by the CEP rules so that they can be reviewed by the head of desk. In this way, decisions can be "automated," yet guided by, human decision-making. Small orders, that wouldn't significantly change the position of the firm, and that would be distracting and time consuming to dequeue, are allowed to go through.

And, other "automated actions" include the spawning of new rules based on changing conditions. For example, when risk profiles change suddenly, the system spawns a new rule based on that can evaluate individual order books that are contributing the most to the change. This kind of chained rules - rules that trigger other rules, and maintain complex state, are often the central reason the industry still calls forms of event processing "complex" - it's the ability to identify rich, complex patterns of events that makes the technology so powerful.

I was on a panel this week at the DWT show in London where one of the participants claimed that "robotraders" were making traders obsolete. I have seen no evidence of this. Sure, there are less traders than there were 5 years ago - much of what used to be done manually is now automated. But almost every CEP decision making system is a textured one - some elements of decision making are automated, some are completely left to humans, and some are guided by data and rules - rules that are altered and improved as business conditions change over time.

Applications that do more than just monitor and filter streams unlock the deeper business value in event processing technology: by helping automate intelligent action they can help firms seize opportunity and avert risk before it's too late.