The StreamBase Event Processing Blog

Best Practice #5: Balance Man and Machine

This post is #5 in a series of lessons learned from leading StreamBase customers who have effectively deployed real-time business systems. 

The prospect of automating a business often sparks a debate about what is more effective, man or the machine. It turns out that the debate presents a false choice, and effective automation lies in the proper balance between the two.

Best practice #5, "Balance Man and Machine," is about how successful firms strike a balance between machine-driven and human decision- making.

THE SITUATION: FIRMS MUST AUTOMATE TO SURVIVE

In the capital markets, open outcry exchanges have been closing down for years. Last October, the IntercontinentalExchange closed down open outcry trading of soft commondities in New York, a practice that goes back at least 142 years. When it closed, the 1,000 traders left working on the floor "mostly monitored their market using handheld pricing screens even when they're sitting in the pits." That is, ironically, the manual brokers were literally using the automated tools that were making their jobs obsolete.

And the impact went beyond the survival of some jobs to the survival of some firms. For example, from 2008 to 2012, the once dominant NYSE and NASDAQ exchanges saw their combined market share slip from 67% to 45%, in large part due to upstart rivals BATS and Direct Edge known for their technology-based innovation. 

Obviously, automation has changed Wall Street forever, and it's about to change other industries.

THE PROBLEM: AUTOMATION THREATENS EXISTING CULTURE AND CAN ADD RISK

In the capital markets, the debate has raged on for years about what's more effective - human traders or algorithms.  A quick glance at a time-restricted Google search in 2007 for "Man Versus Machine" reminds us about how massive the debate was! 

Underneath this media coverage was a real issue: culture.

"Manual" traders, and the hierarchy of power over trading systems that was built over years by those who made their mark as manual traders, resisted the ideas that computers could do better than humans. Technologists, eager to make their mark on the P&L of the bank, fought to prove that computers could do a better job. 

At the same time, automation can introduce risk. Events like today's Twitter Hoax that President Obama had been injured at 1:07PM caused the markets to plunge 200 points in about 2 minutes. At 1:10PM, the first denials came out on Twitter, and by 1:13, the Dow had recoverd most of the ground it had lost.  So risk can no longer be done with traditional tools, which evaluate positions at the end of the day. By the end of the delay, it can be too late.

This polarized view of the battle between man and machine was wrong. 10 years on, automated trading is here to stay, and so are the people who guide its use. The truth is that neither man nor machine alone is better at trading; the truth is, a balanced approach is what was created. 

THE SOLUTION: FIVE STEPS TO BALANCE MAN & MACHINE

Automating businesses must change. Here are four observations from the transformation in the capital markets, and the best practices other firms in other industries can learn to facilitate their own change.

1) Business-geek hybrids are common

Traders used to have mostly degrees in business; today, they either additionally have math and science degrees, or have learned more about the implications of algorithms on trading. 

Best Practice: Proactively hire data scientists and put them into leadership positions so that they can lead a different, quantitative way of thinking about business transformation.

2) Humans are more comfortable taking cues from real-time analytics

Yes, lots of trading is automated. But automation also signals humans to make the big decisions. For example, Geneva Trading, one of the largest oil futures traders in the world, uses automation to monitor "millions of trading opportunities" simultaneously. Some opportunities are no-brainers, and the computer acts automatically. But sometimes the computer has no idea how to act, and punts the decision to a human by sending a text message, email, or alerting them on their desktop. 

Best Practice: Couple data scientists with operational staff that are similarly willing to embrace a technologically-driven customer resource management approach. These are customer service personelle who are used to analyzing information supplied by automated systems, and working with clients to solve high-value-add problems. The recent trend in capital markets toward execution consulting services is a good example of this trend. 

3) Real-time risk and surveillance becomes more important

One great myth about algorithmic trading is that it's done without oversight. In fact, all the clients I meet with use real-time analytics to temper, or stop, trading decisions when the data, statistically, does not make sense. One of our customers, PhaseCapital, was the inspiration for the blog post "Fast is Good, But Fast and Dumb is Dangerous" that describes the firms' real-time analytics that helped them trade more safely on the day of the May, 2010 Flash Crash. 

Best Practice: Employ real-time risk, surveilence, and compliance systems. One of the most prevalent applications of real-time platforms in the last 2 years has been real-time risk, compliance, and surveillence applications. 

4) Low-touch and high-touch functions must be integrated

Automation is useful to handle well known, low risk decisions, the ones that require little over site other than a big red panic button to make systems stop. Bigger, high touch decisions are ones handled by humans. By using computers to automate the low touch decisions, humans are empowered to make the tough, big decisions. 

Best Practice: Start with a balanced sense of technical requirements, rather than a world separated by automated and manual workflow. 

5) Infuse events with historical context

In the world of automation, the critical technology stack is event driven; the technology stack for manual processes is historical data driven. These two systems must coexist. Event-driven messaging, event-oriented programming languages (e.g., CEP), event-driven user interfaces, and event-oriented databases (e.g., tick stores) and streaming business intelligence must all sit along side of their historical counterparts. 

Best Practice: Integrate real-time and historical data to give humans oversight and insight into what automated algorithms are doing.  

THE IMPLICATION: AUTOMATION EMPOWERS HUMANS

Balanced well, automation can be a huge competitive weapon for a business. By applying lessons from one of the most deeply automated industries in the business world, firms embarking on automation initiatives  can increase their chance of success and reduce their risk of failure. 

The Best Practices of Real-Time Innovation

Event-driven systems are new to many firms. For many, automating existing manual processing, using event-driven techniques, properly simulating real-time analytics systems is a new experience. 

Fortunately, some industries have been on the forefront of designing for real-time. Namely, banks in the capital markets trading arena have been automating and building event-based applciations for years. 

So on a recent customer tour with our customers, we've been asking them for their best lessons learned and the business impact the systems they've built have made. 

Best Practice #1: Build Apps So Quickly You Won't Mind Throwing Some Away

One IT executive told me "CEP allows me to build event-driven apps in 2-4 weeks, rather than 6-9 months with an app server and database - so I can say "yes" to more requests from their business clients, which helps us innovate more quickly. We keep the great ideas, and discard the ones that don't work."

Best Practice #2: Start Small, Succeed, then Scale

The most consistently successful firms that moved to real-time on Wall Street started with a small, critical application, learned lessons from its development, then applied those lessons learned to other projects and groups. Over time, they developed a center of excellence with firm-wide sucess in real-time application development.

 

Best Practice #3: Plan for Spikes, Not Aggregate Volumes

Systems that use primarily historical have trained many technologists to capacity plan their systems for the aggreate amount of data they manage. Real-time systems are different, and should be planned for spikes in volume, rather than only aggreate volumes. This best practice explores three approaches: burst behavior, the use of conflation, and computational reduction, that helps applications gracefully manage spikes and deliver correct and timely response to users and automated action. 

Keep coming back throughout 2013 as we continue to add to our library of real-time system best practices!

Best Practice #4: Incorporate Statistical Computing with Event Driven Systems

Numerical computing environments are often used for algorithm development, data analysis, visualization, matrix manipulations, and statistical analysis. In finance, quants use it to do time-series analysis, portfolio optimization, asset liability modeling, quantitative risk modeling, and derivative pricing. This best practice describes how to does this computing in real-time, using MATLAB with StreamBase CEP together.

Best Practice #5: Balance Man And Machine

The prospect of automating a business often sparks a debate about what is more effective, man or the machine. It turns out that the debate presents a false choice, and effective automation lies in the proper balance between the two.

This article explores 5 steps to balance man and machine, including the infusion of more data scientists in executive positions, the application of real-time risk management and surveillance, the infusion of historical and real-time analytics in one place, and more.

 

Best Practice #3: Plan for Spikes, Not Aggregate Flow

This post is a part of a series of best practices from leading StreamBase enterprise customers who have effectively deployed "real-time" business systems. 

"Can StreamBase handle 300 million messages a day?" asked a senior IT manager the other day.

"Well, it's common for a single StreamBase server to handle 100,000 messages a second for applications like yours," said our field architect. 

"OK, but can you handle 300 million messages a day?" 

Why this bizarre miscommunication? 100,000 messages per second is 8.7 BILLION messages per day, yet the IT manager was worried StreamBase wouldn't handle his volumes.

Experienced event processing architects think about scalability in a different way. So, for this best practice blog entry, I teamed up with one of our most senior field architects, Hayden Schultz, to describe the art of scalability design and how to think about scalability in real-time systems. 

THE SITUATION: AGGREGATE DATA CAPACITY PLANNING IS COMMON PRACTICE

Many technologists think about capacity planning in terms of aggregate data volumes. That is, they predict how much data they need to process in a day and provision their network, database, and hardware accordingly.

THE PROBLEM: REAL-TIME DEMANDS DIFFERENT METRICS

Aggregate volume isn't the right way to plan real-time systems.  Real-time systems often have a 100-to-one, 1,000-to-one, or even a million-to-one ratio of event-computation-to-action ratio. Technologists need to think in terms of stream burst rates, and what the right behavior is during bursts.  

THE SOLUTION: THINK IN THREE SCALABILITY DIMENSIONS 

There are three dimensions to scalability in real-time systems: 

1)  Bursts behavior. It's not uncommon for bursts to be so narrow that you don't really need to design around them because while performance will degrade during the burst, the burst is fairly brief so the system performance degrades so briefly that it's not noticeable. A stable real-time system is designed so that the maximum data rate that it's capable of processing is greater than the average data rate, so if a burst is reasonably narrow, the latency of the responses increases during the burst while incoming data is buffered; when the burst is over, the backlog of data is quickly processed and the system resumes its normal performance.

2) Consider Conflation. Increasing the response latency during bursts can be the wrong approach for some systems, but the cost of designing for spikes 3-5X higher than the average can be expensive. A technique called conflation helps mitigate this issue. Processing events that are buffered during bursts has the unfortunate tendency to process events that have already been superseded by more recent events, so the system is using its overtaxed resources to compute results that are obsolete while letting the most current results age and perhaps become obsolete in turn. Conflation means that new events overwrite processed buffered events so that only the most current data is processed. The data rate is reduced and obsolete events are not used. Conflation may slightly increase average response time, but can dramatically reduce latency during traffic spikes. 

3) Computation reduction. Some systems have computational loads so that a new result cannot be produced for each input event. These systems typically produce results at a fixed frequency. This is common when the result is sent to a graphical user interface (GUI). A typical GUI can't update nearly as quickly as many CEP applications can produce results, and even if it could the updates would overwhelm the users trying to use the GUI. Sending results at a fixed rate solves this problem. If the data rate of events entering the CEP system should become too large a common strategy is to reduce the computation frequency. When the burst is over, the computation frequency can be increased to its normal rate.  

THE IMPLICATION: CORRECT SYSTEMS AT THE RIGHT COST

Planning system capacity in terms of burst rates, data reduction, and computational reduction helps balance correct system behavior and cost, and turns handling a billion messages a day into child's play.

Best Practice #2: Start Small, Succeed, then Scale

This post is the second in a series of best practices gleaned from leading StreamBase customers who have effectively deployed "real-time" business systems.

Even though CEP and stream processing is being popularized in marketing campaigns such as IBM's "Smarter Planet" and TIBCO CEO Vivek Ranidive's book "The Two Second Advantage," the reality is that, for many organizations, a real-time, sense-and-respond business model is still elusive. 

A consistent best practice employed by our customer base is to "Start Small, Succeed, then Scale."

THE SITUATION: REAL-TIME BECOMES A "MUST DO" AGENDA

On August 1st, 2012, Knight Capital lost $440 million in 45 minutes when their real-time algorithms went out of control. That kind of event caught the interest of most CEOs in the capital markets who all of a sudden put real-time controls, risk management and reporting high on their list of "must do" IT projects. 

That said, many would be surprised to know how little of Wall Street is actually automated with respect to the management function of high-speed trading. Many of the largest firms process 100's of millions of trading events a day, yet still manage their business by looking at historical information.

But moving to real-time isn't just a matter of installing new, faster hardware - it's a shift in thinking, a shift in culture. That can mean implementing new IT infrastructure from scratch, or even hiring new teams to manage old business processes that used to be more manual. 

THE PROBLEM: LEGACY SYSTEMS AND PEOPLE NEED AN OVERHAUL

When it comes to trading, most banks have great real-time technology in place. But when it comes to managing that trading activity, they're still often using rear-view-mirror reporting technology like traditional business intelligence tools. You can see this in software vendor market size by comparing the business intelligence market (around $10B) to CEP and the new "Live Business Intelligence" segment (around $350M, and growing). 

But the problem isn't just about technology - the staff required to manage real-time business processes are more quantitative in mind-set. For example, heads of risk management usually understand mathematics, and the models used to trade, because the algorithms themselves introduce operational and cost risk. 

So, given that people, processes, and technology typically have to be transformed, the most successful clients in our customer base have taken an incremental approach, by deliberately transforming the most critical elements to start with. 

But how?

THE SOLUTION: CHOOSE A CRITICAL "FIRST DOMINO" TO PUSH OVER

The transformation in the capital markets provides a good template on how to transform from rear-view mirror to real-time.

12 years ago, less than 5% of trading was driven by computer algorithms; now, around 60-70% of trading volume in the U.S. market is guided by algorithms. Banks chose their most critical operation - trading - to focus on first. The most mathematically oriented traders survived; pure "relationship" oriented traders have waned. Now, a dozen years later, back-office, clearing, settlement, and risk management are all moving to real-time.

Trading was the first domino to be pushed over. 

How did they do it?  The pattern went something like this:

1. Choose an important function (e.g., trading)

2. Put your best people on it (choose the new model of people required for mathematics-based operations)

3. Build the new system

4. Fail

5. Return to step 3, as necessary until you get it right

6. Capture learning, choose new leadership that "gets" the new model

7. Repeat with adjacent business functions

8. Build a center of excellence

9. Scale (pick the next system to transform based on the lessons learned).

THE IMPLICATION: REDUCED RISK; LONG TERM EFFECTS

The implication of starting small and planning to fail with an important domino is, ironically enough, reduced risk. Risk is reduced because by choosing an important, small function, you can fail faster. You can choose to put your top talent on the project. You can adjust more quickly. And, for long-term sustainability, you create a core center of excellence that the rest of the organization can leverage as you roll systems out over time. 

THE LESSON: ACHIEVE LASTING TRANSFORMATION TO REAL-TIME BY STARTING SMALL

So by starting small, succeeding with that small project, then scaling out the rest of the enterprise, the organization lowers the risk of early failure and starts on the long journey of employing a disruptive shift in thinking that can last decades. 

Best Practice #1: Build Apps So Quickly You Won’t Mind Throwing Some Away

This post is the first in a series of best practices gleaned from leading StreamBase customers who have effectively deployed "real-time" business systems.

During our recent customer tour, I was concerned to hear that one of our customers had recently decommissioned several of the 30 applications they built with StreamBase. It turned out that this was by design and a virtue, not a sign of a lack of success.

What I found leads me to the first in a series of "Business Best Practices" around real-time application development: Build Applications So Quickly You Won’t Mind Throwing Some Away.

.

THE SITUATION: IT IS OVERWHELMED; FAILURE IS NOT AN OPTION

Most technology groups are overwhelmed with requests for new features and capabilities from the business. At the same time, they need to innovate. And worse, some cultures actively discourage failure. These conditions lead IT to play it safe, and a conservative culture can be born that leads to safe technology choices, and an acceptance of long, expensive software development cycles.

THE SOLUTION: THE FOUR TENETS OF RAPID APP DEVELOPMENT WITH CEP

The StreamBase customer I visited changed their thinking about engineering several years ago. They changed IT leadership to one that favors speed and experimentation over control and safety. They adopted StreamBase as a key enterprise development platform because they can build apps in 2-4 weeks, rather than 6-9 months. This rapid speed of development means IT can experiment with new ideas and simply throw away those that don’t work out in the end.

In his words, there are four tenets of StreamBase responsible for this:

1) Visual programming. Event-based applications are expressed in StreamBase EventFlow™, our visual programming language designed to implement event-based systems just as you would describe them on a white board, with event-driven inputs, such as streaming financial market data, and event outputs, like messages published to a message bus. 

2) Polyglot programming. Very few organizations develop applications in just one language; they use several languages depending on the job. For event-based, real-time applications, the programming language of choice is often StreamBase. But we also encourage the use of other programming environments with CEP, like Java, .NET, MATLAB, R, Python, HTML5, JavaScript, and more. StreamBase has integrated support for these environments in Studio, which makes it easier for developers to learn, but also easier to integrate existing code in an event-driven application. Polyglots (a developer who knows and uses several programming languages) can more easily adopt StreamBase and build applications more quickly because they can connect to any code on any platform quickly and easily.

3) Pre-built connectivity. The StreamBase catalog of over 150 connectivity points turns integration - usually a development task that is completed only after months of intricate coding with sockets or other messaging APIs - into a drag-and-drop experience, in some cases cutting integration time down to minutes.

4) Complete event container. Projects that use databases and traditional app servers for event-driven applications have to build custom container logic. Event-based containers require low-level socket programming, delicate threading design, event management, and more. As our customer said “it’s a whole other project just to manage and connect to event streams.” StreamBase, by contrast, comes pre-built with connectivity, monitoring, threading, management, and testing tools. You just start using them, building business logic.

THE IMPLICATION: APPS TAKE WEEKS TO DEPLOY; NOT MONTHS

This head of IT technology at a major bank told me that the combination of these four elements allows him and his team to take on new ideas more aggressively:

"The cost of doing something in StreamBase is so low that even when we envision a project that we aren't sure is going to fly, we're able to take it on because it's not a big investment of time.  It's a few weeks or a month, not 6-9 months on a development roadmap. This radically shorter development cycle helps us build apps more quickly, experiment with new ideas, and throw away ideas that don't work. StreamBase helps us do this because it's visual and takes care of all the event-based plumbing for us, so we can focus on the business logic rather than how to get apps up and running."

THE LESSON: BUILD APPLICATIONS SO QUICKLY YOU WON’T MIND THROWING SOME AWAY

It's refreshing to find organizations that embrace experimentation and failure, and unique tools to make it happen. So the best practice of applying rapid development techniques can help IT departments take on more, experiment more, and innovate more is our first best practice for real-time enterprise applications: Build Applications So Quickly You Won’t Mind Throwing Some Away.