This post is the second in a series of best practices gleaned from leading StreamBase customers who have effectively deployed "real-time" business systems.
Even though CEP and stream processing is being popularized in marketing campaigns such as IBM's "Smarter Planet" and TIBCO CEO Vivek Ranidive's book "The Two Second Advantage," the reality is that, for many organizations, a real-time, sense-and-respond business model is still elusive.
A consistent best practice employed by our customer base is to "Start Small, Succeed, then Scale."
THE SITUATION: REAL-TIME BECOMES A "MUST DO" AGENDA
On August 1st, 2012, Knight Capital lost $440 million in 45 minutes when their real-time algorithms went out of control. That kind of event caught the interest of most CEOs in the capital markets who all of a sudden put real-time controls, risk management and reporting high on their list of "must do" IT projects.
That said, many would be surprised to know how little of Wall Street is actually automated with respect to the management function of high-speed trading. Many of the largest firms process 100's of millions of trading events a day, yet still manage their business by looking at historical information.
But moving to real-time isn't just a matter of installing new, faster hardware - it's a shift in thinking, a shift in culture. That can mean implementing new IT infrastructure from scratch, or even hiring new teams to manage old business processes that used to be more manual.
THE PROBLEM: LEGACY SYSTEMS AND PEOPLE NEED AN OVERHAUL
When it comes to trading, most banks have great real-time technology in place. But when it comes to managing that trading activity, they're still often using rear-view-mirror reporting technology like traditional business intelligence tools. You can see this in software vendor market size by comparing the business intelligence market (around $10B) to CEP and the new "Live Business Intelligence" segment (around $350M, and growing).
But the problem isn't just about technology - the staff required to manage real-time business processes are more quantitative in mind-set. For example, heads of risk management usually understand mathematics, and the models used to trade, because the algorithms themselves introduce operational and cost risk.
So, given that people, processes, and technology typically have to be transformed, the most successful clients in our customer base have taken an incremental approach, by deliberately transforming the most critical elements to start with.
But how?
THE SOLUTION: CHOOSE A CRITICAL "FIRST DOMINO" TO PUSH OVER
The transformation in the capital markets provides a good template on how to transform from rear-view mirror to real-time.
12 years ago, less than 5% of trading was driven by computer algorithms; now, around 60-70% of trading volume in the U.S. market is guided by algorithms. Banks chose their most critical operation - trading - to focus on first. The most mathematically oriented traders survived; pure "relationship" oriented traders have waned. Now, a dozen years later, back-office, clearing, settlement, and risk management are all moving to real-time.
Trading was the first domino to be pushed over.
How did they do it? The pattern went something like this:
1. Choose an important function (e.g., trading)
2. Put your best people on it (choose the new model of people required for mathematics-based operations)
3. Build the new system
4. Fail
5. Return to step 3, as necessary until you get it right
6. Capture learning, choose new leadership that "gets" the new model
7. Repeat with adjacent business functions
8. Build a center of excellence
9. Scale (pick the next system to transform based on the lessons learned).
THE IMPLICATION: REDUCED RISK; LONG TERM EFFECTS
The implication of starting small and planning to fail with an important domino is, ironically enough, reduced risk. Risk is reduced because by choosing an important, small function, you can fail faster. You can choose to put your top talent on the project. You can adjust more quickly. And, for long-term sustainability, you create a core center of excellence that the rest of the organization can leverage as you roll systems out over time.
THE LESSON: ACHIEVE LASTING TRANSFORMATION TO REAL-TIME BY STARTING SMALL
So by starting small, succeeding with that small project, then scaling out the rest of the enterprise, the organization lowers the risk of early failure and starts on the long journey of employing a disruptive shift in thinking that can last decades.