The StreamBase Event Processing Blog

Best Practice #2: Start Small, Succeed, then Scale

This post is the second in a series of best practices gleaned from leading StreamBase customers who have effectively deployed "real-time" business systems.

Even though CEP and stream processing is being popularized in marketing campaigns such as IBM's "Smarter Planet" and TIBCO CEO Vivek Ranidive's book "The Two Second Advantage," the reality is that, for many organizations, a real-time, sense-and-respond business model is still elusive. 

A consistent best practice employed by our customer base is to "Start Small, Succeed, then Scale."

THE SITUATION: REAL-TIME BECOMES A "MUST DO" AGENDA

On August 1st, 2012, Knight Capital lost $440 million in 45 minutes when their real-time algorithms went out of control. That kind of event caught the interest of most CEOs in the capital markets who all of a sudden put real-time controls, risk management and reporting high on their list of "must do" IT projects. 

That said, many would be surprised to know how little of Wall Street is actually automated with respect to the management function of high-speed trading. Many of the largest firms process 100's of millions of trading events a day, yet still manage their business by looking at historical information.

But moving to real-time isn't just a matter of installing new, faster hardware - it's a shift in thinking, a shift in culture. That can mean implementing new IT infrastructure from scratch, or even hiring new teams to manage old business processes that used to be more manual. 

THE PROBLEM: LEGACY SYSTEMS AND PEOPLE NEED AN OVERHAUL

When it comes to trading, most banks have great real-time technology in place. But when it comes to managing that trading activity, they're still often using rear-view-mirror reporting technology like traditional business intelligence tools. You can see this in software vendor market size by comparing the business intelligence market (around $10B) to CEP and the new "Live Business Intelligence" segment (around $350M, and growing). 

But the problem isn't just about technology - the staff required to manage real-time business processes are more quantitative in mind-set. For example, heads of risk management usually understand mathematics, and the models used to trade, because the algorithms themselves introduce operational and cost risk. 

So, given that people, processes, and technology typically have to be transformed, the most successful clients in our customer base have taken an incremental approach, by deliberately transforming the most critical elements to start with. 

But how?

THE SOLUTION: CHOOSE A CRITICAL "FIRST DOMINO" TO PUSH OVER

The transformation in the capital markets provides a good template on how to transform from rear-view mirror to real-time.

12 years ago, less than 5% of trading was driven by computer algorithms; now, around 60-70% of trading volume in the U.S. market is guided by algorithms. Banks chose their most critical operation - trading - to focus on first. The most mathematically oriented traders survived; pure "relationship" oriented traders have waned. Now, a dozen years later, back-office, clearing, settlement, and risk management are all moving to real-time.

Trading was the first domino to be pushed over. 

How did they do it?  The pattern went something like this:

1. Choose an important function (e.g., trading)

2. Put your best people on it (choose the new model of people required for mathematics-based operations)

3. Build the new system

4. Fail

5. Return to step 3, as necessary until you get it right

6. Capture learning, choose new leadership that "gets" the new model

7. Repeat with adjacent business functions

8. Build a center of excellence

9. Scale (pick the next system to transform based on the lessons learned).

THE IMPLICATION: REDUCED RISK; LONG TERM EFFECTS

The implication of starting small and planning to fail with an important domino is, ironically enough, reduced risk. Risk is reduced because by choosing an important, small function, you can fail faster. You can choose to put your top talent on the project. You can adjust more quickly. And, for long-term sustainability, you create a core center of excellence that the rest of the organization can leverage as you roll systems out over time. 

THE LESSON: ACHIEVE LASTING TRANSFORMATION TO REAL-TIME BY STARTING SMALL

So by starting small, succeeding with that small project, then scaling out the rest of the enterprise, the organization lowers the risk of early failure and starts on the long journey of employing a disruptive shift in thinking that can last decades. 

Best Practice #1: Build Apps So Quickly You Won’t Mind Throwing Some Away

This post is the first in a series of best practices gleaned from leading StreamBase customers who have effectively deployed "real-time" business systems.

During our recent customer tour, I was concerned to hear that one of our customers had recently decommissioned several of the 30 applications they built with StreamBase. It turned out that this was by design and a virtue, not a sign of a lack of success.

What I found leads me to the first in a series of "Business Best Practices" around real-time application development: Build Applications So Quickly You Won’t Mind Throwing Some Away.

.

THE SITUATION: IT IS OVERWHELMED; FAILURE IS NOT AN OPTION

Most technology groups are overwhelmed with requests for new features and capabilities from the business. At the same time, they need to innovate. And worse, some cultures actively discourage failure. These conditions lead IT to play it safe, and a conservative culture can be born that leads to safe technology choices, and an acceptance of long, expensive software development cycles.

THE SOLUTION: THE FOUR TENETS OF RAPID APP DEVELOPMENT WITH CEP

The StreamBase customer I visited changed their thinking about engineering several years ago. They changed IT leadership to one that favors speed and experimentation over control and safety. They adopted StreamBase as a key enterprise development platform because they can build apps in 2-4 weeks, rather than 6-9 months. This rapid speed of development means IT can experiment with new ideas and simply throw away those that don’t work out in the end.

In his words, there are four tenets of StreamBase responsible for this:

1) Visual programming. Event-based applications are expressed in StreamBase EventFlow™, our visual programming language designed to implement event-based systems just as you would describe them on a white board, with event-driven inputs, such as streaming financial market data, and event outputs, like messages published to a message bus. 

2) Polyglot programming. Very few organizations develop applications in just one language; they use several languages depending on the job. For event-based, real-time applications, the programming language of choice is often StreamBase. But we also encourage the use of other programming environments with CEP, like Java, .NET, MATLAB, R, Python, HTML5, JavaScript, and more. StreamBase has integrated support for these environments in Studio, which makes it easier for developers to learn, but also easier to integrate existing code in an event-driven application. Polyglots (a developer who knows and uses several programming languages) can more easily adopt StreamBase and build applications more quickly because they can connect to any code on any platform quickly and easily.

3) Pre-built connectivity. The StreamBase catalog of over 150 connectivity points turns integration - usually a development task that is completed only after months of intricate coding with sockets or other messaging APIs - into a drag-and-drop experience, in some cases cutting integration time down to minutes.

4) Complete event container. Projects that use databases and traditional app servers for event-driven applications have to build custom container logic. Event-based containers require low-level socket programming, delicate threading design, event management, and more. As our customer said “it’s a whole other project just to manage and connect to event streams.” StreamBase, by contrast, comes pre-built with connectivity, monitoring, threading, management, and testing tools. You just start using them, building business logic.

THE IMPLICATION: APPS TAKE WEEKS TO DEPLOY; NOT MONTHS

This head of IT technology at a major bank told me that the combination of these four elements allows him and his team to take on new ideas more aggressively:

"The cost of doing something in StreamBase is so low that even when we envision a project that we aren't sure is going to fly, we're able to take it on because it's not a big investment of time.  It's a few weeks or a month, not 6-9 months on a development roadmap. This radically shorter development cycle helps us build apps more quickly, experiment with new ideas, and throw away ideas that don't work. StreamBase helps us do this because it's visual and takes care of all the event-based plumbing for us, so we can focus on the business logic rather than how to get apps up and running."

THE LESSON: BUILD APPLICATIONS SO QUICKLY YOU WON’T MIND THROWING SOME AWAY

It's refreshing to find organizations that embrace experimentation and failure, and unique tools to make it happen. So the best practice of applying rapid development techniques can help IT departments take on more, experiment more, and innovate more is our first best practice for real-time enterprise applications: Build Applications So Quickly You Won’t Mind Throwing Some Away.

Geneva Energy Markets Deploys StreamBase for Oil Futures Trading

We love making customer announcements, so today we're thrilled to announce that Geneva Energy Markets, LLC (GEM), one of the leading oil traders in the world, has deployed StreamBase for their automated trading infrastructure. Moreover, they said that the StreamBase-based system "helps monitor, analyze and act on hundreds of thousnds of trading opportunities eavery day, and has led to one of its most profitable trading periods in history."

The StreamBase-based system helps GEM continuously monitor, analyze and act on trading opportunities by continuously evaluating, for example, heating oil prices in New York, gasoline in Europe, West Texas Intermediate crude oil, and more. Based on GEM’s trading algorithms, the system evaluates hundreds of thousands of combinations in parallel a day, decides how much the opportunity is worth, and acts, in milliseconds. The automated system executes pre-defined strategies electronically, and alerts traders of “high touch” opportunities for manual trading attention.

Mark Vonderheide, Managing Partner at GEM, nicely articulates a pragmatic view on the balance of low touch (i.e., automated) and high touch (i.e., manual) trading in this quote:

"Our StreamBase-based system helps us strike the balance between automated trading and manual trading,” said Mark Vonderheide, Managing Partner, GEM."

Mark also said nice things about our firm, which means a lot to us:

“We started with a vision to automate based on CEP in the fall of 2009, and StreamBase has not only met, but exceeded our expectations on what we wanted to achieve. The development environment has helped us get our applications into market quickly, and, just as important, the company has stood by us all the way as a partner, something that you don’t see with many software companies."

Today’s enterprise data is constantly in motion – the better you process that data, the better your results. Geneva Energy Markets is a great example of a firm that uses StreamBase to continuously correlate real-time conditions, and act in real time. Together with our partner, Trading Technologies, we think the oil swaps market is a prime opportunity for CEP-based innovation.