The StreamBase Event Processing Blog

Best Practice #5: Balance Man and Machine

This post is #5 in a series of lessons learned from leading StreamBase customers who have effectively deployed real-time business systems. 

The prospect of automating a business often sparks a debate about what is more effective, man or the machine. It turns out that the debate presents a false choice, and effective automation lies in the proper balance between the two.

Best practice #5, "Balance Man and Machine," is about how successful firms strike a balance between machine-driven and human decision- making.

THE SITUATION: FIRMS MUST AUTOMATE TO SURVIVE

In the capital markets, open outcry exchanges have been closing down for years. Last October, the IntercontinentalExchange closed down open outcry trading of soft commondities in New York, a practice that goes back at least 142 years. When it closed, the 1,000 traders left working on the floor "mostly monitored their market using handheld pricing screens even when they're sitting in the pits." That is, ironically, the manual brokers were literally using the automated tools that were making their jobs obsolete.

And the impact went beyond the survival of some jobs to the survival of some firms. For example, from 2008 to 2012, the once dominant NYSE and NASDAQ exchanges saw their combined market share slip from 67% to 45%, in large part due to upstart rivals BATS and Direct Edge known for their technology-based innovation. 

Obviously, automation has changed Wall Street forever, and it's about to change other industries.

THE PROBLEM: AUTOMATION THREATENS EXISTING CULTURE AND CAN ADD RISK

In the capital markets, the debate has raged on for years about what's more effective - human traders or algorithms.  A quick glance at a time-restricted Google search in 2007 for "Man Versus Machine" reminds us about how massive the debate was! 

Underneath this media coverage was a real issue: culture.

"Manual" traders, and the hierarchy of power over trading systems that was built over years by those who made their mark as manual traders, resisted the ideas that computers could do better than humans. Technologists, eager to make their mark on the P&L of the bank, fought to prove that computers could do a better job. 

At the same time, automation can introduce risk. Events like today's Twitter Hoax that President Obama had been injured at 1:07PM caused the markets to plunge 200 points in about 2 minutes. At 1:10PM, the first denials came out on Twitter, and by 1:13, the Dow had recoverd most of the ground it had lost.  So risk can no longer be done with traditional tools, which evaluate positions at the end of the day. By the end of the delay, it can be too late.

This polarized view of the battle between man and machine was wrong. 10 years on, automated trading is here to stay, and so are the people who guide its use. The truth is that neither man nor machine alone is better at trading; the truth is, a balanced approach is what was created. 

THE SOLUTION: FIVE STEPS TO BALANCE MAN & MACHINE

Automating businesses must change. Here are four observations from the transformation in the capital markets, and the best practices other firms in other industries can learn to facilitate their own change.

1) Business-geek hybrids are common

Traders used to have mostly degrees in business; today, they either additionally have math and science degrees, or have learned more about the implications of algorithms on trading. 

Best Practice: Proactively hire data scientists and put them into leadership positions so that they can lead a different, quantitative way of thinking about business transformation.

2) Humans are more comfortable taking cues from real-time analytics

Yes, lots of trading is automated. But automation also signals humans to make the big decisions. For example, Geneva Trading, one of the largest oil futures traders in the world, uses automation to monitor "millions of trading opportunities" simultaneously. Some opportunities are no-brainers, and the computer acts automatically. But sometimes the computer has no idea how to act, and punts the decision to a human by sending a text message, email, or alerting them on their desktop. 

Best Practice: Couple data scientists with operational staff that are similarly willing to embrace a technologically-driven customer resource management approach. These are customer service personelle who are used to analyzing information supplied by automated systems, and working with clients to solve high-value-add problems. The recent trend in capital markets toward execution consulting services is a good example of this trend. 

3) Real-time risk and surveillance becomes more important

One great myth about algorithmic trading is that it's done without oversight. In fact, all the clients I meet with use real-time analytics to temper, or stop, trading decisions when the data, statistically, does not make sense. One of our customers, PhaseCapital, was the inspiration for the blog post "Fast is Good, But Fast and Dumb is Dangerous" that describes the firms' real-time analytics that helped them trade more safely on the day of the May, 2010 Flash Crash. 

Best Practice: Employ real-time risk, surveilence, and compliance systems. One of the most prevalent applications of real-time platforms in the last 2 years has been real-time risk, compliance, and surveillence applications. 

4) Low-touch and high-touch functions must be integrated

Automation is useful to handle well known, low risk decisions, the ones that require little over site other than a big red panic button to make systems stop. Bigger, high touch decisions are ones handled by humans. By using computers to automate the low touch decisions, humans are empowered to make the tough, big decisions. 

Best Practice: Start with a balanced sense of technical requirements, rather than a world separated by automated and manual workflow. 

5) Infuse events with historical context

In the world of automation, the critical technology stack is event driven; the technology stack for manual processes is historical data driven. These two systems must coexist. Event-driven messaging, event-oriented programming languages (e.g., CEP), event-driven user interfaces, and event-oriented databases (e.g., tick stores) and streaming business intelligence must all sit along side of their historical counterparts. 

Best Practice: Integrate real-time and historical data to give humans oversight and insight into what automated algorithms are doing.  

THE IMPLICATION: AUTOMATION EMPOWERS HUMANS

Balanced well, automation can be a huge competitive weapon for a business. By applying lessons from one of the most deeply automated industries in the business world, firms embarking on automation initiatives  can increase their chance of success and reduce their risk of failure. 

Best Practice #3: Plan for Spikes, Not Aggregate Flow

This post is a part of a series of best practices from leading StreamBase enterprise customers who have effectively deployed "real-time" business systems. 

"Can StreamBase handle 300 million messages a day?" asked a senior IT manager the other day.

"Well, it's common for a single StreamBase server to handle 100,000 messages a second for applications like yours," said our field architect. 

"OK, but can you handle 300 million messages a day?" 

Why this bizarre miscommunication? 100,000 messages per second is 8.7 BILLION messages per day, yet the IT manager was worried StreamBase wouldn't handle his volumes.

Experienced event processing architects think about scalability in a different way. So, for this best practice blog entry, I teamed up with one of our most senior field architects, Hayden Schultz, to describe the art of scalability design and how to think about scalability in real-time systems. 

THE SITUATION: AGGREGATE DATA CAPACITY PLANNING IS COMMON PRACTICE

Many technologists think about capacity planning in terms of aggregate data volumes. That is, they predict how much data they need to process in a day and provision their network, database, and hardware accordingly.

THE PROBLEM: REAL-TIME DEMANDS DIFFERENT METRICS

Aggregate volume isn't the right way to plan real-time systems.  Real-time systems often have a 100-to-one, 1,000-to-one, or even a million-to-one ratio of event-computation-to-action ratio. Technologists need to think in terms of stream burst rates, and what the right behavior is during bursts.  

THE SOLUTION: THINK IN THREE SCALABILITY DIMENSIONS 

There are three dimensions to scalability in real-time systems: 

1)  Bursts behavior. It's not uncommon for bursts to be so narrow that you don't really need to design around them because while performance will degrade during the burst, the burst is fairly brief so the system performance degrades so briefly that it's not noticeable. A stable real-time system is designed so that the maximum data rate that it's capable of processing is greater than the average data rate, so if a burst is reasonably narrow, the latency of the responses increases during the burst while incoming data is buffered; when the burst is over, the backlog of data is quickly processed and the system resumes its normal performance.

2) Consider Conflation. Increasing the response latency during bursts can be the wrong approach for some systems, but the cost of designing for spikes 3-5X higher than the average can be expensive. A technique called conflation helps mitigate this issue. Processing events that are buffered during bursts has the unfortunate tendency to process events that have already been superseded by more recent events, so the system is using its overtaxed resources to compute results that are obsolete while letting the most current results age and perhaps become obsolete in turn. Conflation means that new events overwrite processed buffered events so that only the most current data is processed. The data rate is reduced and obsolete events are not used. Conflation may slightly increase average response time, but can dramatically reduce latency during traffic spikes. 

3) Computation reduction. Some systems have computational loads so that a new result cannot be produced for each input event. These systems typically produce results at a fixed frequency. This is common when the result is sent to a graphical user interface (GUI). A typical GUI can't update nearly as quickly as many CEP applications can produce results, and even if it could the updates would overwhelm the users trying to use the GUI. Sending results at a fixed rate solves this problem. If the data rate of events entering the CEP system should become too large a common strategy is to reduce the computation frequency. When the burst is over, the computation frequency can be increased to its normal rate.  

THE IMPLICATION: CORRECT SYSTEMS AT THE RIGHT COST

Planning system capacity in terms of burst rates, data reduction, and computational reduction helps balance correct system behavior and cost, and turns handling a billion messages a day into child's play.

Best Practice #1: Build Apps So Quickly You Won’t Mind Throwing Some Away

This post is the first in a series of best practices gleaned from leading StreamBase customers who have effectively deployed "real-time" business systems.

During our recent customer tour, I was concerned to hear that one of our customers had recently decommissioned several of the 30 applications they built with StreamBase. It turned out that this was by design and a virtue, not a sign of a lack of success.

What I found leads me to the first in a series of "Business Best Practices" around real-time application development: Build Applications So Quickly You Won’t Mind Throwing Some Away.

.

THE SITUATION: IT IS OVERWHELMED; FAILURE IS NOT AN OPTION

Most technology groups are overwhelmed with requests for new features and capabilities from the business. At the same time, they need to innovate. And worse, some cultures actively discourage failure. These conditions lead IT to play it safe, and a conservative culture can be born that leads to safe technology choices, and an acceptance of long, expensive software development cycles.

THE SOLUTION: THE FOUR TENETS OF RAPID APP DEVELOPMENT WITH CEP

The StreamBase customer I visited changed their thinking about engineering several years ago. They changed IT leadership to one that favors speed and experimentation over control and safety. They adopted StreamBase as a key enterprise development platform because they can build apps in 2-4 weeks, rather than 6-9 months. This rapid speed of development means IT can experiment with new ideas and simply throw away those that don’t work out in the end.

In his words, there are four tenets of StreamBase responsible for this:

1) Visual programming. Event-based applications are expressed in StreamBase EventFlow™, our visual programming language designed to implement event-based systems just as you would describe them on a white board, with event-driven inputs, such as streaming financial market data, and event outputs, like messages published to a message bus. 

2) Polyglot programming. Very few organizations develop applications in just one language; they use several languages depending on the job. For event-based, real-time applications, the programming language of choice is often StreamBase. But we also encourage the use of other programming environments with CEP, like Java, .NET, MATLAB, R, Python, HTML5, JavaScript, and more. StreamBase has integrated support for these environments in Studio, which makes it easier for developers to learn, but also easier to integrate existing code in an event-driven application. Polyglots (a developer who knows and uses several programming languages) can more easily adopt StreamBase and build applications more quickly because they can connect to any code on any platform quickly and easily.

3) Pre-built connectivity. The StreamBase catalog of over 150 connectivity points turns integration - usually a development task that is completed only after months of intricate coding with sockets or other messaging APIs - into a drag-and-drop experience, in some cases cutting integration time down to minutes.

4) Complete event container. Projects that use databases and traditional app servers for event-driven applications have to build custom container logic. Event-based containers require low-level socket programming, delicate threading design, event management, and more. As our customer said “it’s a whole other project just to manage and connect to event streams.” StreamBase, by contrast, comes pre-built with connectivity, monitoring, threading, management, and testing tools. You just start using them, building business logic.

THE IMPLICATION: APPS TAKE WEEKS TO DEPLOY; NOT MONTHS

This head of IT technology at a major bank told me that the combination of these four elements allows him and his team to take on new ideas more aggressively:

"The cost of doing something in StreamBase is so low that even when we envision a project that we aren't sure is going to fly, we're able to take it on because it's not a big investment of time.  It's a few weeks or a month, not 6-9 months on a development roadmap. This radically shorter development cycle helps us build apps more quickly, experiment with new ideas, and throw away ideas that don't work. StreamBase helps us do this because it's visual and takes care of all the event-based plumbing for us, so we can focus on the business logic rather than how to get apps up and running."

THE LESSON: BUILD APPLICATIONS SO QUICKLY YOU WON’T MIND THROWING SOME AWAY

It's refreshing to find organizations that embrace experimentation and failure, and unique tools to make it happen. So the best practice of applying rapid development techniques can help IT departments take on more, experiment more, and innovate more is our first best practice for real-time enterprise applications: Build Applications So Quickly You Won’t Mind Throwing Some Away.

90 Seconds on How to Develop with Java and CEP

Not long ago I sat down with Phil Martin, a senior consultant at Datasports, for a 30-minute online Q&A and demo on “How to Develop with Java and CEP”. Phil’s been developing applications with Java and StreamBase CEP for over 3 years now and we wanted to pick his brain about his experiences developing applications with these two languages.

If you’re a Java developer looking to build real-time event based applications you should definitely check it out in its entirety here. Then again, if you’re a developer you probably don’t have 30 minutes of free time to watch the video...unless of course you're developing with StreamBase!

So instead I'll make it easier for you. Below you’ll find the highlights from this conversation, and the best part is you should be able to digest it all in about 90 seconds.

Q: Phil, what kind of use cases and applications are a good fit for CEP development?

A: It should be a real-time system where performance is an important concern. And it should be unsurprisingly an event or message based type of processing model where those events or messages are clearly defined, where the content and format is known at design time.  And I think that you get increasing benefit from a platform like StreamBase if you have a large number of disparate kinds of systems that you’re looking to integrate with. 

Q: What aspects of my application should be developed in CEP and which should be developed in Java?

A: Anything that implements a low level protocol type of logic probably belongs in the Java domain, for example things that are implementing interactions with vendor provided client libraries or protocol libraries or doing low level network disk or database IO. In general other than that I find that the CEP programming model works the best for almost all business logic.

Q: What are the benefits of developing with CEP?

A: When I’m developing code I want to concentrate on the things the human being does well and have the computer do things that machines do well.  And for me that means I want to be focused as much as possible on business logic.  And what I find is that using a platform like StreamBase lets me stay focused on business logic.  The other benefit that I see is visual programming. I find the day-to-day model for doing development more productive. I also find that the resultant applications tend to be fast and stable.

Q: What have been your experiences around time-to-market gains with CEP?

A: My general rule of thumb is developing with CEP is three to five times faster than using a traditional programming model like Java or .NET development.

Q: How does CEP address some of the biggest frustrations Java developers face?

A: My biggest frustration also relates to the biggest strength in Java, which is that for every level of your application infrastructure there are choices to be made.  There are choices around what version of the JRE, what kind of framework you’re going to use to write your application, the list goes on and on. By having so many of those choices already made for me and made well by the StreamBase platform I can do what I want to do, which is to stay focused on solving business problems.

Q: There are some objections that we sometimes hear from Java developers around having to learn a new language and concerns about visual programming and its effect on performance. Why shouldn’t Java developers be concerned about that?

A: In terms of being able to actually get ramped up, get hands on and start doing productive work I find that curve with StreamBase is actually very fast compared with other environments. I would normally estimate that someone who is never touched StreamBase before can actually be productive in a matter of weeks.

The StreamBase environment is compiled the same as any Java applications, the components and infrastructure are developed with performance in mind.

I found that the visual environment makes the processes of refactoring and reusing code and tuning the threading model much simpler, much clearer, and much, much faster than what it would be in a traditional programming environment.

Thanks to Phil for doing this webinar with us. You can check out his blog at http://blog.datasports.ca/ where he blogs about financial systems engineering, including StreamBase, Java, and the Canadian markets.

5 Best Practices of Execution Consulting Services

Yesterday, StreamBase hosted a webinar entitled “The 5 Best Practices of Execution Consulting Services” - an interview with StreamBase CTO Richard Tibbetts and Stephane Loiseau, Managing Director and Head of Equity Execution at Société Générale. The session described an important trend in the capital markets - the rise of execution consulting services (ECS).

ECS is defined as a sell-side service that helps buy-side clients achieve greater trade performance and efficiency. As electronic trading space gets more competitive, broker-dealers like Société Générale need to differentiate their trading services from their competitors. ECS is where the rubber meets the road, and provides the link between a broker and their clients to help them understand the market, market data, improve their trading performance and achieve their benchmarks.

ECS isn’t just one person’s job - it requires a mix of trading, technology and people. The 5 best practices Stephane presented are:

1. Find the Right People for the Job. Staffing the ECS function is one of the biggest challenges Stephane has at Société Générale. Today buy side has become more sophisticated with the use of electronic trading. They are not only familiar with the self-service tools and algorithms provided by sell-side, also exploring new technology and strategies internally. The expectation of ECS has increased. ECS used to be simply a technology job, because algorithms were new, and the service was new. But over the years, the trader’s role has evolved. An ECS team needs staff with a background in trading, but someone with experience in trade operations is critical to help understand the processing of an order from A to Z. Finally, the client relationship role is essential. Trading isn’t just about playing with technology; you’re interacting with the client to provide a service. Finally, there’s the intersection of these people skills and tools - the right ECS tools enable an ECS team to focus on providing great service. These people and tools are normally found in different organizational silos. So the best practice of ECS is to craft an ECS team with the right skills, team composition, and tools to get the job done.

2. Continually Re-Assess Market Micro Structure. Yes, the market is complex, liquidity is fragmented. These are true statements. But understanding all the aspects of microstructure can lead to better trading quality. If your ECS team understands where liquidity is coming from, what market events are affecting the market in real-time, and how speed affects trading, then they can help clients get better trading performance. For example, if your platform aggressively deals with dark pools in Europe, you can’t build trading strategies around where the liquidity is. The challenge is that market micro-structure knowledge has a shelf life of a few weeks. For example, if a venue changes its pricing structure, that’s a major event - that changes monthly. News changes daily. Market data changes in real-time. So best practice #2 is that ECS staff must continually re-assess market microstructure in order to help clients get the best trading results.

3. Establish a Dynamic, Real-Time System for Monitoring, Reporting, and Action. Historically, traders decided the actions they are going take, then carried on with those actions throughout the day, and results are assessed at the end of the day. This no longer the case - you need to do all of this in real-time. In order to accomplish this, ECS teams need a great monitoring system; it’s just too much data for the human brain to consume. At Société Générale, they analyze the market in real-time, based on microstructure and also macro level news events. But monitoring tools can’t be stale and static, which is why a robust alerting system is so important - you need to have some form of triggers that will help you know when to change your strategies - they could be based on volatility, or any other metric the ECS team decides is important on that day, or moment. So Stephane’s third best practice is to provide killer tools to the ECS team to monitor, assess, and adjust trading strategies in real-time.

4. Find a Balance Between Technology and High Touch Human Interaction. Over the last month, where volumes have been so low, Société Générale’s ECS team has adjusted the trading triggers they consider. ECS is not just about low touch (fully algo trading, all technology based) or just high touch (human trader interaction); these models are converging very quickly. Now ECS teams provide a lot more human interaction to change an algorithm, tweak their parameters, or help a client change their strategy as conditions change. For example, if you have an order for a client, and you start to work the order with an algorithm, and you can see there is an opportunity to cross the order on the floor, you need to call the customer and ask him or her if you want to close the cross. So you are seeing an opportunity, consulting with your clients, and saving them money. Another example is if the algorithm is not performing the way it should be - this is when the ECS consultant would reach out to the customer - should we split up the order into 2 and work them independently? So the fourth best practice of ECS is to strike a new balance between high and low touch service.

5. Understand that TCA is a Process Throughout the Trading Cycle, Not Just a Metric. Stephane feels that the notion of TCA must transform from an end-of-day metric to a trading lifecycle driver - the ECS team must think about the TCA throughout the lifecycle of trading. The ECS team must think about how their actions will affect the TCA grade you want to achieve. Then actions should be made carefully based on your data, which tools you use, which venues to trade, and so on. For example, on a specific order, the ECS team can decide to work a million shares on its own, but also to drop 500,000 in a VWAP algo for 2 hours with a specific metric just to make sure they are participating in the market. And the team must constantly monitor and adjust that strategy during the day with their monitoring tools (best practice 3). Stephane provided another simple example - when the trading day gets really busy, ECS staff can put more flow into the algorithm than manually trading. So you’re going to use data from previous days and from real-time to make your decisions and use as much data as you can throughout the lifecycle of the order. So changing the way the ECS team views TCA is the fifth best practice of ECS.

The session concluded with some Q&A, including how Stephane views ECS developing outside of the U.S. and Europe, how important it is on the trading floor, and how customized algorithms fit into the ECS strategy. For the Q&A, and to hear the full interview, click here.

Tech Q&A with CEP Developers, and StreamBase

Today StreamBase hosted an online roundtable entitled “How to Empower Your Development Team with CEP and Visual Programming” featuring developers from J.P. Morgan, GAIN Capital, and Datasports. Combined, the panelists have 35+ years of development experience, and years of CEP experience, and they were kind enough to share those experiences with us, discussing the adoption of CEP technology, developing applications with CEP, and the best practices they’ve learned along the way.

For developers, development managers, architects, and business stakeholders this roundtable tells an interesting story about how CEP technology can truly transform the role of the developer. Take a look at some of the highlights below, watch the replay, then comment below and tell us about your experiences with CEP. 

Talk about your experience with CEP

Phil Martin, Datasports: I like using StreamBase because I like tools that let the computers do stuff that the computers are good at, and let me do stuff that requires thought. As a developer, you wind up doing some tasks over and over again; the structure of the application, domain specific plumbing, etc. Everytime feels like wasted effort. StreamBase lets me focus on the stuff that requires thought, not the details of implementation, connectivity, processing events, and so on. Another comment is around the aggregate operator. It is analogous to another API to learn. It’s an incredibly rich and flexible operator that I think almost qualifies as its own sub language or sub paradigm.  There’s a lot there and a lot of subtlety in its behavior, learn about and study it!

Mike Grandilone, J.P. Morgan Asset Management: The shift to a message passing structure is sometimes a shift in thinking. It's key to understanding how StreamBase handles the life of a tuple; we gained that through experience.  

Talk about the applications you've built with StreamBase

Joseph Chua, GAIN Capital: After 4 days of training and going through the examples, I built my first StreamBase application in just three hours, and it was fully deployed in a week. It connects to market data, manipulates the data, and stores it in a high-speed database. We've also built market making and real-time surveillence applications, and many more to come.

Phil: I've built several applications with StreamBase. One is an "indication of interest" (IOI) app that monitors trading interest and compares it against a CRM system and sends alerts to individual sales traders because they have a customer who either has traded on the symbol, transacted above a given threshold, or has a real-time order for it. The correlation of these factors in real-time also includes real-time news sources. Also, I've worked on two algorithmic trading systems that host their strategies in StreamBase and perform real-time risk checks on the algos, and an ETF arbitrage system where there was only one strategy, but it was a very involved strategy.

Mike: We brought a smart order routing system up and running in 6 months with 2 developers, which we thought was very fast. We also got the business people involved in the canvas and the development - this is a very important factor in getting applications done quickly.

Talk about the benefits of StreamBase and visual programming

Phil: StreamBase helps facilitate communication. Whenever you want to convey information about a system, you always draw a picture. And they almost always look like StreamBase Event Flow. If you have taken the necessary care to clearly delineate modules, then, depending on the technical sophistication of who you are talking to, you can choose how far to drill down. It's extremely helpful in faciliating technical discourse; with the business users, it saves time.

Mike: Visual development has really improved the speed of our development. Especially when you're doing a proof of concept. It helps you understand the application's logic, and also communicate with the business; in typical programming environments like Java, you're not going to show many people the code, but you can do it with StreamBase.

Joseph: The big value is that all the technical details are abstracted away in terms of connectivity and low-level code; we can then communicate with the business owners, so development cycles are shorter and more efficient. The other value we have found is they we can very quickly build a prototype solution so the business can make decisions more quickly with respect to requirements and decisions to develop the applications.

Phil: One thing that's helpful: with StreamBase, you can tell at a glance that spaghetti code is spaghetti code. You can tell if an application was thoughtfully designed without hours and hours and hours of code review.

What best practices have you learned from your experiences with CEP?

Mike: You need to shift your thinking, to think in terms of events, rather than traditional static applications. And, just like any application, build a testing environment at the same time as your application logic.

Joseph: Identify the connection points quickly, be open minded and willing to unlearn the more traditional ways of development, use StreamBase interfaces, and make sure you continuously refactor the application.

Phil: There are lot of resources for ramping up StreamBase skills - invest the effort to develop your skills. The speed of putting together something quickly is a great asset. The thing to keep in mind is that although the development speed of StreamBase is a benefit in the short term, it can, if you're not careful, work against you in the long term.

Phil: While the principles aren't different, the practice IS different. The first thing that sold me is seeing the ease with which you can change and tune your threading model, take things in and out of the threading part of your system, with multiplicity and dispatch methods. Scalability and performance is about taking more advantage of the cores in a system, and careful management of it becomes more and more important.

What was your experience integrating other code into CEP?

Mike: We have separate routing logic written in Java for routing logic that we integrated with our StreamBase application. We've done a lot of work with custom Java operators in StreamBase - it's very straight forward - no problem in doing it. There's lots of power in developing in Java for code that isn't event driven that allowed us to bring in the power of algorithmic trading as a drop-in to the StreamBase environment. Also, we have a C# front end GUI, which uses the .NET interface from StreamBase - this was easy, although the challenge was how to take a GUI that isn't event-based and make it work with an event-driven application. We're in production with that now.

The StreamBase community

Phil: I've put two components onto the StreamBase Component Exchange - mostly adapters. The publishing process was straightforward. One is an adapter that watches for events on the file system and creates StreamBase events when files are created, renamed, modified, or deleted, and another one listens for calls on a web service interface and creates events for the applciation. I suggest other developers look at the utility classes – see what are other developers are doing, and pick up some tips and techniques and helpful hands-on examples of how they are building their applications. 

StreamBase Resources for Developers

Below are the StreamBase resources mentioned in this roundtable:

• Trial version of StreamBase: www.streambase.com/download
• Product documentation: docs.streambase.com
• Personalized demonstrations: www.streambase.com/forms/request-a-demonstration
• “How To” developer webinars: www.streambase.com/webinars
• StreamBase Component Exchange: www.streambase.com/sbx
• StreamBase University: www.streambase.com/sbu
• StreamBase Training and Certification: www.streambase.com/community/training-and-certification
• Datasports blog for StreamBase CEP developers: http://datasports.ca

StreamBase Loves Erlang

The philosophy of Erlang and StreamBase align quite well. Quoting Mike Williams, one of the three inventors of Erlang:

• Find the right methods—Design by Prototyping
• It is not good enough to have ideas, you must also be able to implement them and know they work
• Make mistakes on a small scale, not in a production project

These philosophies of rapid development also apply to the way we've designed StreamBase; additionally, our customers also demand that much less technical users participate in the rapid implementation of ideas. StreamBase has a visual development approach that allows less technical, more "business" users - with mathematics, analytics, and business domain expertise - use the same tools as professional software developers.

What if we could add Erlang developers to that party? Event-driven, visual development, coupled with the power and rapid development of Erlang? 

Well you can. Darach Ennis (@darachennis) shows us how in his recent presentation at Erlang UG in London. 

StreamBase has a graphical, compiler-oriented programming language, targeted at serious developers, that can also be used by the less technical. Erlang programmers, and the Erjang folks with whom we share a virtual machine, do their thing in Erlang, and that code becomes a graphical "operator" on the StreamBase Studio pallet. In Darach's example he's used Erlang to implement a MACD analytic, and integrate that into a StreamBase-based trading system so it operates in the event-driven CEP application. 

The combination of StreamBase and Erlang combines the best of both worlds: StreamBase for rapid, graphical development to develop low latency applications that process 100's of thousands of events a second. Combining "how," "why," and "under what conditions," real time streams are matched and processed. StreamBase allows business analysts, actuaries, quantitative contributors and developers to share a common language. An event-driven integration whiteboard, if you will, that compiles to efficient optimized native binary code. Add Erlang for development of analytics like MACD and you have a powerful combination. 

Erlang support will be contributed soon to the StreamBase Component Exchange (SBX), our public "app exchange" for CEP integrations, applications, frameworks, and sample code. You'll already find support for Java, C++, R, Matlab, python, .NET. If you want to plug in some cool analytics written in C++, just do it. You want to exploit Basho's Riak? Sure, plug it in.

As a result, StreamBase allows developers control over how they exploit the facilities at their disposal. Use our graphical DSL for flow oriented algorithms and real time low latency high frequency data management. It's high performance from a human productivity and systemic perspective. 

Learn more by watching Darach's presentation here, and follow him on Twitter at @darachennis.

 

The Flash Crash & the Role of Financial Software Engineering

Even though the flash crash happened seven months ago, the debate rages on. And since CEP is a common piece of our modern financial computing fabric many firms have entered the debate with vested interest. Unfortunately, most vendors wrap flash crash observations with veiled sales pitches for CEP-based risk management, or disingenuous calls to "catch the rogues and scoundrels" with CEP-based surveillance systems. Although these applications are important, to suggest they are a magical elixir to ward off future breakdowns is misleading at best. 

But thoughtful discourse does continue, and the recent Securities and Exchange Commission report has reinvigorated debate. Richard Tibbetts, StreamBase CTO, weighed in with a piece called "What Does the Flash Crash Mean For Financial Software Engineering?" on Tabb Forum this week. Richard's view on the flash crash examines the role of financial software engineering and the efficacy of data processing, integrity, and reliability within the fabric of the infrastructure. The role of financial engineering is an essential, overlooked, and misunderstood piece of the puzzle. Richard's thoughts go like this: 

"In all the conversation about the crash, the real historic change is going unmentioned. In the very last paragraph of the Lessons Learned section of the report, the authors note: “another area of focus going forward should be on the integrity and reliability of market centers’ data processes, especially those that involve the publication of trades and quotes to the consolidated market data feeds.”

This is a key insight – late in coming – that will expand over time. Traditionally the SEC has focused on regulating the behavior of market participants, identifying deliberately malicious behavior or patterns of behavior. But now they are beginning to realize that data processing, integrity, and reliability can be just as important. The efficient functioning of markets is in just as much danger from software system failures as it is from malicious traders..."

The role of financial engineering is much broader than CEP, and deserves deeper discourse. To read more, continue reading Richard's piece on Tabb Forum (which requires a free registration) here, and watch for more over the coming months on Tabb Forum, on this blog, and on Richard's blog on software engineering and innovation, Hyperextended Metaphor.

Gain Capital's Selection of CEP and "Whiteboard Programming"

In the late 90's, visual programming was The Next Big Thing. Instead of writing millions and millions of lines of incomprehensible computer instructions, visual programming tools promised a world where you could "paint" applications using high-level objects. Visual programming promised a future where applications would be built more quickly, more easily, with result in an application that would be less expensive to maintain. Unfortunately that era of visual programming did not deliver on many of it's promises. Tools were primitive, workstations weren't up to the task, many potential users didn't know why they would want to write programs, and the languages didn't line up well with the use cases. In the end, the tool with the most staying power was UML, which is a modeling language rather than a programming language, and which used traditional textual programming languages to create the "real" program. UML became fodder for big-project architecture and requirements documents, not a tool for real programmers.

Last week I was reminded that I'm fortunate to be running a company that delivers on the promise of visual programming. This struck me as I was listening to press interviews with Andrew Haines, the CTO of Gain Capital. Andrew discussed the reasons behind their selection of StreamBase. He talked about how they had evaluated all the major CEP vendors and selected StreamBase for its speed, ease of use, and our being a good partner. And he talked about how strategic their use is - Gain processes almost $200 billion in trades a month. All good stuff.

But the comment that struck me was when Andrew said Gain has found StreamBase Event Flow programming language to be 50% faster than traditional programming techniques (citing an experiment they did that measured StreamBase versus .NET). This comment reminded me of the 90's and the promise of visual programming, or what we sometimes call "Whiteboard Programming." Whiteboard programming (watch a short video here) is the right visual metaphor for event processing because developers design event-driven applications naturally on a whiteboard, with data flows, icons for business logic, and arcs that connect logic to next part of their event driven architecture. StreamBase's visual development tools allow developers to "paint" their application. But the visual representation of their whiteboard isn't just a picture of the application, it is the application. The application is tested, profiled, and debugged visually, and the promise of visual programming, for event-based applications, is fulfilled.

StreamBase EventFlow benefits from several improvements in the state of the art since the 90s, as well as our own innovation. First, it's built into a modern IDE based on the Eclipse platform, already a powerful and flexible tool before StreamBase starts with it. StreamBase also takes many of its ideas from the agile programming methodology developed over the last 15 years, including automated refactoring, unit testing, and a focus on quick iterations. More specific to StreamBase, the StreamSQL Event Flow language derives from declarative data processing languages developed at MIT, Brown, Brandeis, and Stanford universities, with rigorous models behind them. The language is designed to not only be visual, but also fundamentally to be directly executable (unlike UML) and to have powerful facilities for extensibility and abstraction (unlike end-user oriented visual languages). The result is a language that is accessible to financial engineers, quantitative analysts, and business analysts, but powerful enough that application and infrastructure developers are comfortable working on the same platform. Despite the ability to add functionality to the platform in many traditional languages (Java, C++, .NET, Python, textual SQL), StreamBase developers universally adopt the visual programming model for it's productivity and teamwork benefits.

Andrew's comment reminded me why StreamBase is on a run of amazing accolades as one of the most innovative technologies in the world, including Time Magazine, the World Economic Forum Technology Pioneer program, and MIT Technology Review. These esteemed technologies realize that, as Clayton Christensen suggests in The Innovator’s Dilemma, simplicity is always a disruptive innovation.  And the proof lies in clients like Gain Capital, who get to market 50% faster, with 50% less cost, with 50% less cost of ownership over time. 

But at the same time, other CEP venders aren't gaining customer success stories like Gain Capital. Why not? Gain evaluated numerous other CEP products from Progress Software, Aleri, and TIBCO as reported by FX Week. Coral8 was acquired by Aleri, then Aleri was acquired by Sybase. After the acquisition, Aleri engineers revealed that although their product was fast, customer implementations took years and lots of consulting to build. Apama, the early pioneer in CEP, tried to develop a visual programming tool after they developed a textual language. The textual language works, but the visual one doesn't, which is why Apama's visual tool is being scrapped as they integrate the CEP language with Savvion's visual BPM development environment.

What Gain Capital's selection of StreamBase reminds us is that simplicity is always disruptive, and that the promise of visual programming is just as relevant today as it was 10 years ago, and being fulfilled by the leading CEP companies.

Q&A with CMC Markets: Trading $1.4 Trillion on CEP

This week CMC Markets announced that they have deployed complex event processing (CEP) in their global trading infrastructure that processes $1.4 trillion a year on over 5,000 trading instruments and over 115,000 trades a day. 

You might be thinking - $1.4 trillion in trading? Who is this CMC Markets? CMC is the leading global provider of retail contracts for difference (CFDs) and one of the largest financial spread betting leaders. CMC is a sophisticated, distributed organization - according to Inside Market Data, "the StreamBase platform is being rolled out across CMC's global operations, including offices in Australia, Canada, China, Germany and the UK, and is supporting a wide range of applications including its market-making, risk management, and hedging tools to support trading in FX, indices, futures, commodities, and equities."

This kind of global, mission critical commitment to CEP is important news, so we spoke with CMC’s Chief Information Officer, Asif Adatia, to find out more about CMC, their needs, why they chose CEP, their business results, and their future plans with CEP.

Q1: The AITE Group report from July, 2007 said your average trade volume was about $100 billion monthly.  Is that correct now?

That’s an old number. Now we’re at about $200 billion a month or $10 billion a day.  For example, if I look at today’s stats we’ve done $7.5 billion now at trading shops.

Q2: Have your trading volumes increased a lot over the last couple of years?

I think all of our stats have increased.  We’ve got more clients, more trades per day, which explains some of the re-architecture work we’re doing.  Yeah, we’re doing very well at the moment.  This business fundamentally is doing very well from a key metrics perspective.

Q3: Right.  Can I ask, were your upgrade plans hampered at all this last year by the recession or have you been able to carry on?

Peter Cruddas, our CEO, gave us massive support from a technology perspective.  If anything I’ve got more to do. We put a plan together, the board bought into it and it’s pretty exciting.  For me it’s phenomenal, we’ve been able to do everything we wanted to do.  This re-architecture is an effort to take our business to another level.

Q4: So there hasn’t been anything that had to be put on hold because of the crisis?

No, definitely not from a technology perspective, no.  If anything, the question is “can I do more and get it done faster.”  I’ve certainly not had restrictions.

Q5: This re-architecture is replacing your existing in-house built architecture.  What other technology upgrades are you making?

We’ve got a whole style change from our front end and our transactional engines to our back office.  There’s a program over the coming months where we are upgrading all of our systems.

Today, StreamBase is doing pricing, and it’s far more sophisticated than our legacy market maker platform we previously had.  Ultimately, StreamBase will replace a lot of the intelligence behind the old system.

Q6: Can you give me a couple of examples?  And over what period you will conclude?

So the back office as an example, we started to work on that. We’ve got a batch orientated back office currently, and we’re in the process of creating a real time P&L and cash balances being realized in P&L.  Today you wait until the end of the day, you mark the market and you adjust your P&L’s and you realize your P&L then. We’re moving a real time back office.  The timing is probably end of year. And then again because it’s so different, there will be a coordinated effect with our marketing and our client service guys.  Because it is a big difference from the way we have been working. That’s just one example of many.

Q7: Can you describe the project StreamBase is supporting?

StreamBase supports FX for all of our major currencies, exotics, as well as forwards.  It will support indices futures, futures on indices, cash commodities, treasuries and equities, single stock.

Q8: Can you describe how the StreamBase application processes data and pricing for CMC?

We cleanse our prices as they come through. We get multiple feeds from various brokers.  We cleanse our prices and we’ve got a reasonably sophisticated model where we look at groupings - if there are multiple feeds coming in and you want to take the best price, you want to take out the ones that are outliers. So the mathematics behind it are sophisticated - it doesn’t just do a standard mean, it takes a look at standard deviation of all of these prices and then looks at the volatility over multi-factor, high frequency scenarios. Then we start to look at which ones are outliers.

As I said StreamBase helps us look at what’s happening and very quickly take algorithms in and out - fix them and put them back in as well.

Q9: What are your FX market data sources?

I can’t give you the names but it’s primarily the usual suspects.  Tier one brokers provide liquidity and pricing to firms like ourselves.  We’ve got quite a few brokers that we deal with and their price feeds. We’ve also got feeds for other asset classes.  And along with the brokers, we have multiple data feeds from vendors, exchanges, alternative ECNs and OTC price providers

Q10: Are you using the StreamBase adapters to connect directly to those price feeds?

Yes, it’s actually is a combination.  StreamBase provides feed handlers.  So very quickly we were able to connect and use the StreamBase feed handlers. And then we’re in the process with some of our brokers, again, StreamBase allows us to connect to our own feed handlers so we do that.  So as I said at the beginning flexibility was one of the key criteria.  StreamBase offers that to us as well.  So we can mix and match.

Q11: Are you live? 

Yes, we’re live now.

Q12: You mentioned scalability and performance - what kind of demands are you putting on the StreamBase system now?  I’m thinking updates, latency, that kind of thing?

So at the moment we’re able to process 17,000 ticks per second on FX.  It doesn’t really hurt the system or the processes that much.  And we’ve got it scaled up to for twice as large as that. Can it handle our peak? Our peak to date has been nowhere close to 17,000 at the moment.  We’ve got what we call a non-functional regression test environment where we just overload it.  We overload our environment and haven’t broken it.  We've got so much going on right now that if it can cope with double our peak, we can focus on other things.

Q13: So what kind of a difference will users notice?  What’s going to be different for them? 

If you look at FX pricing live right now our spread is pretty much in line and competitive.  We used to have an issue back in before we implemented some of this.  We used to have issues with latency.  Those problems have disappeared now.  We’re able to process a lot more - about 17,000 ticks a second, which is pretty cool. We’ve still got plenty of headroom as well.  And we’re also now taking multiple feeds in.

From a usability and user experience, clients will just get better pricing, no re-quotes as we roll this out across all of our asset classes.  Re-quotes will disappear, type of spreads, better spreads.  We’re able to handle volatile markets - Fed announcements; all of that kind of stuff is no longer an issue for us. We’re probably one of the few retail players out there that if you’ll notice during any highly volatility period, CMC markets are usually there providing two way quotes.  So that, I think, is the upside for clients that our infrastructure is solid in times of high volatility.

Q14: Can you describe two way pricing?  It’s one of your differentiators right?

The ability to quote a two way price is not the issue - the ability to quote and allow you to trade at that price is the difficult part and our systems can do that - especially for FX, as we add more asset classes to our new infrastructure we should be able to do the same for all asset classes, hence the urgency to get everything done yesterday.

Q15: So what makes your pricing better?  Is it just the lower latency?

I think it’s a number of things.  Yeah, it’s latency, it’s flow. It’s lower latency, robust feeds, and having multiple feeds coming in.  The result is to offer the retail client almost institutional pricing, which is unique.

Q16: Can you put your latency improvements in numeric terms?

We’re able to handle 17,000 ticks per second from multiple sources.  Previously we looked at probably 3000 ticks per second.  This doesn’t make a big difference during normal trading days, but when volumes start to pick up - and this is only one asset class - we’re able to process this number of ticks and still be able to provide a two way price, a clean price you’re able to trade on in milliseconds.  That’s the upside.  Market data is a big issue for all the vendors - even the big players out there.  I won’t mention any names but during summer last year you will notice that there were massive delays in all of the larger players out there with respect to providing prices to clients in general.  And, again, we believe that now those kinds of delays have disappeared from our architecture.  So can I quantify the number of ticks per second?  We can handle the latency that we’ve got, yes, but it’s an improving target.  And one where, because of the StreamBase multi threaded architecture, we can run it across multiple CPU’s now.  If we ever get a bottleneck we just plug in another blade, pull up another service, and we get better latency and better pricing.

Q17: So the scalability is obviously great. Do you have any metrics you want to offer that quantify the latency you’re experiencing?

No, we don’t give them out. Speed is a strategic advantage for us, so we think those numbers are competitive information.

That’s all right, fair enough.

It’s good though and you can compare us on FX pricing to a bunch of our competitors.  And you will see our performance is noticeably better at times. Absolutely.

Q18: Okay.  You mentioned initially in the beginning the broader investment in your architecture.  Is that correct?

Yeah, we are in the process of rethinking a lot of our systems and our core systems at the moment.  This is just part of that.

Q19: What other applications will you use StreamBase for?

Yeah, the flexibility of the platform is the key thing for us.  There are lots of ideas - for example, surveillance.  Again, the concept behind surveillance is that you want to process a large number of trades and constantly be able to look at patterns. From what we know of StreamBase, it could lend itself well to that.  But, again, we’re going through pricing first then risk - the classics.  We’ll do the classics first and then all the other added value things we may want, because we have the infrastructure in place now.

Q20: And when you say when you look into the future you might use it for surveillance purposes, will that be looking at clients?  Or is that more for looking at potential opportunities that you might have been missing, or is it a bit of both?

It’s a bit of both actually.  Surveillance is a broad area. One of the things we do is look for people behaving badly. So you want to be able to protect the firm from people doing financial fraud.  You want to be able to stop fraud before it happens because you want to be able to flag that in real time.  StreamBase will be capable of that as well.  So there’s a number of projects, again, that we would look at after we’ve done the basics.

What makes this possible is that because the StreamBase infrastructure is there, the tools are there, the adapters are there to feed the platform. Then it’s a matter of just gluing it together.

Q21: Right, okay.  Very interesting.  Are you going to then be incorporating this into your market maker platform?  That’s the trading platforms that you guys have.

Yes.

Q22: So this is deployed across your entire enterprise?

Yes, it is global deployment.

Q23: How long did it take you to deploy the system?

6 months.

Q24: What were your requirements when you were looking for a solution and why did you choose StreamBase? 

We wanted two things: a scalable solution, and a flexible solution.  We’ve got a bunch of financial engineers that want to be able to prototype or even test different models and scenarios.  And we wanted that to be as easy as possible, so ease of use was a big part of the criteria. 

Q25: So you selected StreamBase CEP over other CEP products for its simplicity of development, performance, and scalability.  Is it working?

Yes - the flexibility of StreamBase is key to us - so much so that we’re finding new uses for it.  We originally brought it in for pricing and risk, and now that we understand the concept and some of the things it can do, it should facilitate other projects as well.

Q26: Can you talk about performance - was that about latency or capacity management?

With performance it was a combination of things.  The number of ticks, as you are probably well aware, of market data, isn’t going down.  So it’s the number of ticks you can process and how you can scale across multiple different environments.  It’s the number of different feeds you can then handle.  That was part of the criteria. 

And then it was the ease of managing all of that performance infrastructure.  Clearly there are definitely products out there that can do some of these things, but it was the combination of speed and ease of use that was critical.  And at the end of the day for us, the overriding factor actually was the financial engineers signing off, because they could participate in the process and work with the StreamBase tools.  They felt we can make this product work for us.  We like this product.

Q27: When you were doing an evaluation of different CEP platforms, what did the evaluation entail?

There was performance, scalability and ease of use and how our financial engineers found the products in general.  So when we did look at the other guys, one of the nice things we liked about StreamBase was the development environment, StreamBase Studio.  The Studio environment is intuitive; it allows the financial engineers to put models together. It’s graphical. You can connect it together quickly.  It’s quite novel and we like that. 

So StreamBase had to meet both the technical requirements as well as the financial engineers’ requirements.  And StreamBase met both those needs quite well.

Q28: You mentioned that you initially used Apama.  Why did you drop Apama? 

As I mentioned, the ease of development for both financial engineers and technical engineers, the performance, and scalability of the platform were our key criteria. We dropped Apama because we liked StreamBase better from all of those criteria.