(Mark Palmer is the president and CEO of StreamBase Systems. Follow him at twitter.com/mrkwpalmer)
Recently, I spent a day in Chicago with CME Group, the
world’s world’s largest and most
diverse derivatives exchange, to talk about their approach to innovation. As the CEO of an innovative software company founded by forward-looking MIT techies, I’m lucky
enough to work with a lot of innovative people. But as we talked about the size and scale of CME Group’s
business, it struck me that this may be among the most technically challenging
environments on planet earth, and the numbers tell a compelling story of
challenge and opportunity for a firm that many know of, but few have seen
inside.
CME Group is the modern descendent of the Chicago Mercantile
Exchange (CME), the American financial and commodity derivative exchange
which was founded in 1898 as the Chicago Butter
and Egg Board, and in the past 111 years, grown and transformed into the
world’s largest electronic futures and options exchange (Globex).
But I wasn’t visiting CME Group to check out their museum, I was there to dig
down on the business and technical motivation behind the news that they have selected a new development paradigm (press release, coverage in Wall Street & Technology),
event processing (CEP), on which to build future innovations. This is a watershed moment for the CEP
industry, and an important case study that gives us pause to wonder about the
future of automated business, the future of real-time, mission-critical
infrastructure, and the role of event processing as the platform for that
innovation.
So let’s look at innovation, CEP, and the challenges faced by CME Group:
...1.2 quadrillion dollars ($1,200,000,000,000,000,000)
...The underlying value of the contracts traded by CME Group in 2008
How can a firm stay on the innovation edge while simultaneously supporting the most mission critical operations?
The decision to process 1.2 quadrillion dollars of value was not taken lightly at CME Group. It’s been said that “nobody gets fired for buying IBM” – that is, that big companies are the safe choice when a decision is most critical.
But when innovation must ride beside reliability, decisions
get trickier. A company like CME
Group must balance innovation with mission-critical capability. And, in the case of CEP, CME Group’s
decision illustrates that CEP has come of age.
However, there’s been a rash of recent debate about the
entrance of IBM and Microsoft into the CEP market. Recently, the
New York Times published an article, which declared that innovation is
moving to big companies from small companies – a conclusion that is not
only wrong, it flies in the face of all the most basic tenets of innovation. So in the innovative, fast paced CEP
market, we
are wary of, but don’t fear, the entrance of giants like IBM and Microsoft. Indeed, we welcome it, because it draws
more awareness. The more people
look, the more will buy StreamBase.
And from the customer’s point of view I asked Steve Goldman, CME
Group’s Director of Enterprise Architecture, if he had looked at IBM and
Microsoft, and his response was casual:
“The decision was very difficult but we felt that based on our
environment and needs thatStreamBase was a proven solution.”
11,000,000...
...The current average daily volume of futures contracts traded every day by CME Group...
“Our data feeds are like fire hoses,” Goldman said. “Sometimes, we process a few hundred market data changes a second. But when the markets get volatile, we process tens of thousands of market data changes per second. And, to make it even more challenging, our customers demand the fastest response time – in milliseconds – when volatility is at its peak.”
Real-time response is becoming more and more important in
mainstream business. Any
environment that deals with events (stock market price changes, web site click
data, and so on), can benefit from event processing. And the technological challenge of managing events at high
speed, and with low-latency, requires a host of arcane tooling and expertise. These geeky areas will also repay
deeper exploration, including specialized messaging hardware and software,
clever use of modern computing models like multi-core CPUs, grids, and clouds,
FPGA’s, and efficient management techniques to keep it all running smoothly.
But modern CEP engines are becoming an essential tool for managing enterprise-class streaming IT environments – a better way to manage information that comes through a fire hose.
Hundreds....
...The number of products offered by CME Group...
While most businesses might have a few products, or a big business might have a dozen products, CME Group has hundreds of products. But CME Group’s products aren’t made of physical materials; they are built with software development tools. So the speed and effectiveness of software development at the CME Group dictates the speed and growth of their business.
So the news that CME Group has chosen a new enterprise-wide
development paradigm is important news. CEP, which has been rising to prominence in the past 5
years, flips traditional database-oriented computing on its head. CEP allows an application developer to
design logic that examines what’s happening now instead of what happened
in the past. Options
pricing and settlement consumes vast amounts of streaming information, so the
CEP model fits perfectly, and allows CME Group to quickly create new products
that process those streams.
Any business that has streams of data – from web
clickstreams to point-of-sale transactions to financial market data - can apply
CEP to more quickly and easily bring new products to market. When the development methodology
matches the form of the data, innovation approaches the speed of human thought.
80...
...The number of countries where CME Group's Globex provides regulated exchange access...
Although everyone talks of the inevitability of a global
economy, few firms understand its
practical challenges than CME Group. CME Group offers electronic, regulated
exchange access in over 80 countries.
The host of technologies that enables a 7x24, global
operations are many. But CEP, by
making streams of data from all over the globe more intelligent, can make
network event processing smarter and faster, and is becoming an important
element in enterprise IT infrastructure.
Which leads to a critical innovation issue: how to strike the right balance between
a new technology and a proven technology?
A full answer to that question is outside the scope of this article, but
the key elements for enterprise software products include well-designed
threading architectures, bullet proof fault tolerance systems, robust
management tools, comprehensive testing and debugging tools, and profiling
tools. Consider that CME Group traces its origin to 1848, has NEVER had a
default or a loss of customer funds resulting from failure, today announced its
enterprise deployment of CEP technology. This is the clearest signal yet that event processing
is coming of age as it continues through its rapid growth as a software market.
A few milliseconds....
...the response time necessary at the most heavily loaded times....
To see the manual forms used at exchange over 50 years ago
puts the innovation in the capital markets in perspective. It took many seconds to fill out the
information about trades back then; it takes 400 milliseconds (1/1000 of a
second) for the human eye to blink;
today, CME Group’s trading decisions take place much faster than the blink of an eye – in just a few
milliseconds. How can this kind of
latency be achieved in the face of massive volumes and massive global
distribution?
We discussed the role of CEP in turning the traditional
database-oriented model on its head.
Databases, of course, rely on writing data to disk. The typical seek time for a disk is
about 9 milliseconds. CME Group’s
data can change tens of thousands of times per second; in order to make decisions in a few
milliseconds
Exchanges settling contracts, brokers splitting up and
routing client trades, electronic liquidity providers making markets, hedge
funds finding and executing arbitrage trades, banks delivering currency
rates…all in less than one hundredth of a human blink! Applications are measured in
microseconds, and the tools that do the measuring have resolution in
nanoseconds, that’s one billionth of a second.
StreamBase has been emerging as the top CEP company for some
time now due to its proven rapid development environment, standards-based
approach, high performance, low latency, and mission critical, fault tolerant
architecture. CME Group’s
selection of StreamBase is yet another proof point that demonstrates that CEP
is ready for the big time, and that StreamBase is leading the way.
That’s why CME Group made this important decision for their
real-time, low latency operations – the need for speed, and the need for CEP,
is critical at CME Group.
Massive scale.
Instantaneous response.
Constant change. Nonstop
operations. Taken individually, these are big technical challenges; taken
simultaneously, they are an extreme business risk. But risk for one is opportunity for another, and as CME
Group shows us, by keeping an eye on the three key “abilities” – usability,
reliability, and scalability –firms can keep innovating as business conditions
continue to get more complicated.
