The StreamBase Event Processing Blog

CEP in Brazil, Breaking New Speed Barriers

(Mark Palmer is the CEO of StreamBase Systems. Follow him at twitter.com/mrkwpalmer)

This week StreamBase cleared another hurdle in the race to take CEP into the mainstream by announcing the first application of its technology to facilitate high-speed trading between assets traded in Brazil and overseas (coverage by Jonathan Wheatley from the Financial Times, São Paulo). Kairos is an excellent case study about three fundamental motivations for CEP technology: speed to get into market, speed to execute event-based systems, and speed of application evolution.

The Brazilian trading market illustrates why time-to-market is so important in the capital markets. As the Financial Times reports, "The BM&FBovesp was formed last year by the merger of the São Paulo futures and stock exchanges - and is currently the fastest-growing exchange in the world according to Mondo Visione, which tracks global exchanges. Its market capitalisation grew by 17.6 per cent in September and by 65.7 per cent over the past year." Kairos believes that algorithmic trading volume on BM&FBovesp will have grown fourfold by the end of the year. That kind of change in market structure presents opportunity to the firms who seize it first.

Kairos' use of event-based technology helped them be the first firm to trade across borders between Brazil and overseas. "With the [event feed] adapters in place, we just connect to the market data and it is easy to implement our strategies in the software - that's one of the main reasons we chose StreamBase." said Gustavo Jacob, CTO at Kairos (from Kairos Bows StreamBase for Algo Data by Max Bowie from Inside Market Data). In an interview I attended Gustavo also spoke of evolving the system every single day as trading opportunities emerged. 

StreamBase was founded on the idea of targeting a 50X improvement in performance over traditional, transactional database technologies. But comparisons between CEP and transactional database technology stopped years ago and now we're more often compared to custom C++ or Java code. Kairos said that their new CEP-based system is 100 milliseconds faster than their previous built-for-purpose application and that it can be changed rapidly—two achievements that are "not possible using standard electronic systems."

Finally, in the A-Team article Kairos Asset Management Taps StreamBase for Algo Trading, Alberto Araújo, the managing director of Kairos, explained their evaluation process and findings:

“Our evaluation process started in mid-2008, when we started looking at the leading CEP companies. Our developers downloaded StreamBase and got to work developing a new algo trading system and were amazed at how easy it was to build applications. After a few months of testing, StreamBase was the clear winner,” says Alberto Araujo, COO at Kairos. As well as evaluating StreamBase, Kairos also considered CEP technology from Progress/Apama and Coral8 (since acquired by Aleri). StreamBase won, says Kairos CIO Gustavo Jacob, because of the ease and speed with which new trading strategies can be created, typically in around two weeks. Specific functionality that gave StreamBase an edge was its visual approach to event workflow and debugging, unit testing and profiling capabilities, and existing connectivity."

We've had an amazing year of competitive public endorsements, including CME Group, PhaseCapital, RBC, ConvergEx, and BlueCrest, and now Kairos. We believe passionately that we have the best CEP product in the industry, and actively encourage anyone to try it to see for themselves.

Performance in three dimensions—speed to get into market, speed to execute event-bases systems, and speed of application evolution—are why CEP entrepreneurs are having fun, growing, and breaking down new business barriers.

CEP: Closing the Loop on Business Intelligence

(Richard Tibbetts is the Chief Technology Officer of StreamBase.  Follow him on twitter as @tibbetts) 

There has been a lot of talk recently that compares complex event processing (CEP) and business intelligence (BI) technology, and what they compare to each other.  Much of it suggests that CEP is a logical extension of BI into real-time, continuous analysis.  For example, Seth Grimes (@sethgrimes) suggested the distinction here and Paul Vincent from TIBCO responded by quoting Tony Baer from Ovum (@TonyBaer) who twitted the following:

"So, in essence, CEP makes BI literally actionable.  IOW (in other words) BI is embedded in execution."

These posts triggered a Twitter discussion about how true Tony’s statement was and the breadth of functionality available from CEP platforms like StreamBase. It culminated in Neil Raden’s (@NeilRaden) question:

Are you suggesting that CEP, on its own, is an adequate technology for adaptive control & integrating models/rules from BI and BRMS? (tweet)

Neil’s question crystallized the way CEP is misunderstood by those who are more familiar with traditional data management technology. They try to assess a tool that’s new by comparing it to another tool that’s built for an entirely different purpose.

The core technology of CEP is real-time continuous data transformation. These transformations can include event pattern detection, data enrichment, data analysis, and data reduction. They generally also include support for business rules and for executing code written in traditional programming languages like Java or C++. All this is bundled into a high performance engine designed to run scalably and reliably, and supported by powerful development environments and administrative and visualization tools.

That’s a lot of capabilities that can be brought to bear on many different problems. No wonder some people aren’t entirely sure where to start when attempting to understand what is possible with CEP.

If you know BI, then you try to fit CEP into the BI context. Yet, in fact, StreamBase has never been compared to BI by a prospective customer, despite the fact that “Operational Business Intelligence” has been a buzzword in that area for about four years. If only we could do BI faster, the theory goes, businesses could take advantage of it in the millions of little decisions that impact the business on a day-to-day basis. CEP and BI have similar sounding mantras, yet practically speaking, customers never intermingle the two.

What those immersed in BI don't realize is that the barrier to operational data analysis is not the speed of the data warehouse. As any trader will tell you, the fastest data analysis in the world won’t help you if every trade requires human approval.

In order to take advantage of modern data analysis, you need to stop depending on people to pull the oars, and start making them captains of their own ship. This requires software that can not only analyze a situation, but also take action, and then control that action over time. Workers ("Automation Managers" in the diagram on the left) monitor these systems and cope with exceptions, while the majority of events and decisions are processed automatically, without incident. The role of the user is in guiding real-time activity, not stopping the flow, making a decision, and starting the flow again.

Consider modern trading systems. Years ago, trading floors had hundreds of traders – half of them were making low-level trading decisions based on real-time market data that they watched; orders were placed by typing them into an order management system (OMS); it took 50-100 milliseconds for each trader, once he made a decision, to press the button to buy or sell. Today algorithms designed by quants buy and sell stocks in microseconds, not milliseconds, and one trader, with a real-time application, can guide the behavior that does the work of 1,000 old-school traders, an order of magnitude faster than just five years ago. That’s change in an industry, motivated by a new way of thinking of real-time data processing – complex event processing.

These software applications become the backbone of an automated business. Change is the norm - as the business climate evolves, agility becomes a critical success factor. If the system can be developed and delivered efficiently enough, a wide variety of decisions become suitable for automation. By making millions of small decisions intelligently, massive efficiency gains are possible. Decision support transforms from being a part of the process, to guiding automated processes – a fundamentally new way of thinking that makes old-school BI thinking inapplicable.

But transforming a business to real-time requires not only new technologies, but a new way of thinking as well. It requires a whole set of business practices that only a few companies are now comfortable with. And it requires analysts and software architects who take a wholly different look at how they approach systems. Neil Raden himself talks about this in his book, Smart Enough Systems.

You have to turn operational decision making into a corporate asset you can measure, control, and improve.

Wall Street reached the tipping point about five years ago, and has been creating not only the software systems but also the expertise necessary to run a truly real time business. CEP systems on Wall Street not only monitor markets and customers for opportunities, they also take actions and continuously monitor those actions for successful implementation.

The answer to Neil’s question is that yes, if a business is ready to adaptively control processes with models, rules, and analytics, then CEP is the ideal technology for integrating analysis with action in real time.  

Hire Great People: 6 "How To" Tips

This week, I attended Accel Partner's CEO Summit at Stanford University, along with 150 CEOs from Accel's portfolio companies. We were joined by some well-known public CEOs and the Accel partners in a series of "fire side chats" and networking sessions to talk about leadership and entrepreneurship. During a morning session, Michael Dell was asked for his best advice for entrepreneurs, and reminded us to: hire great people.

During the day, the theme of hiring great people raised some interesting ideas from the audience and the panelists, and even on Twitter as I tweeted online during the event.

1. Spend HOURS with candidates and insist on a good cultural fit.  Michael Dell went on to share a simple, but powerful idea: toss out the idea that interviews should last an hour, and spend HOURS with new candidates. He told a story about a time where he picked up a candidate and drove to work together for an extended period of time, getting to know each other. Apparently Dell is a very "aggressive" driver. The candidate's reaction under stress told him something about his character.  

2. Go to a U2 concert.  Mark Zukenburg (founder) and Sheryl Sandberg (COO BusinessWeek profile), co-leaders of Facebook, talked about going to a U2 Concert together, and spending nights and evenings talking about the future and culture of Facebook before committing to work together.

3. If there's no cultural fit, fire away.  Lee Scott, recently retired CEO of Wal-Mart, said that there were two things at Wal-Mart that would get you fired immediately, regardless of performance: stealing and not "matching the Wal-Mart culture," which is:

    a) The customer is the boss, 

    b) Treat people with dignity,

    c) Improve every day

4. Hire people better than you.  Lee Scott's story of a key point in his career illustrated this well. He was a logistics manager, and was suddenly promoted to VP and handed one of the most critical new roles in Wal-Mart. The move ignited his rise to the top at Wal-Mart.  The reason he was handed the job?  "Because you were the only one on our list who had hired someone strong enough below him that we knew would do as good or better job than you would, so we could promote you," he was told.

5. Ask candidates: "Describe your sense of humor." In response to my Twitter tweet about this (follow me at @mrkwpalmer), my former colleague and CEO Peter Beard said that his favorite interview question was to ask candidates to describe their sense of humor, or to tell their favorite joke.  Leanne Browne from Progress said she thinks the best way to interview someone is to take them to the pub. I was reminded how one of my most insightful interviews happened on the golf course, where I learned that the candidate was a great golfer (he was very skilled at work too), he was late for the tee time (he was consistently late for meetings), he was lazy and always drove in the cart instead of walking a short distance (he cut corners at work too), and he was very creative in his shot selection (he was creative at work too).

6. Get great investors. These comments came in the context of Accel's CEO Summit. Accel was just named number one on Red Herring's List of the Top 100 Global Venture Capitalists. Enrique Salem, CEO of Symantec, explained how his VC's helped him decide whether to sell BrightMail or go public, eventually deciding on a $370M sale to Symantec (on $25M trailing revenues) that catapulted their business in a way he didn't think they could do on his own.

6 great reminders about one of the most important things we all do: choose the people we surround ourselves with.

(Mark Palmer is the president and CEO of StreamBase Systems.  Follow him at twitter.com/mrkwpalmer)

StreamBase and Scala

(Matt Fowles is a Senior Software Engineer for StreamBase, whose interests include programming language, compiler, and virtual machine design.)

As the ponytail (and the tiny bio above) imply, my posts will focus on technical details such as language semantics, general programming issues, compiler design, or anything else I feel is nifty.

Anyone who follows programming languages will probably have noticed Scala cropping up a lot lately.  Scala is an interesting language on several fronts, but the one that sparked my interest was its strong interoperability with Java.  Often times academics forget that to be useful a language needs to integrate with existing code and libraries.  But Scala has clearly been designed with heavy focus on industrial usability as well as academic power.  When I read about Scala on Android, I guessed it wouldn't be hard to integrate with StreamBase.  It turns out I was mostly right (and we will definitely be fixing the small pain points I hit).  What follows is an guide to embedding Scala into a StreamBase application.

First we have to set up StreamBase Studio to support Scala development.

1. Install StreamBase 6.4
2. Add the Scala 2.7.5 Eclipse Plugin to StreamBase Studio (Help -> Software Updates -> "Add Site...")
3. Manually download the AJDT Weaving Plugin 1.6.4
4. Unzip the weaving plugin into your studio install
   
   • make sure the contents of the zip's "plugins" directory goes into Studio's "plugins" directory (I accidentally got that wrong the first time)
   • Update 8/31/2009:  On Windows unzipping directly into a Studio install fails with an obscure error message.  Instead, you should simply unzip into an empty directory and copy the contents of the created "plugins" and "features" directories into Studio's respective subdirectories of "C:\Program Files\StreamBase Systems\StreamBase.6.4\lib\studio". (You may need to correct that path if your studio install is in a non-standard location.)
5. Restart Studio
6. Enable "JDT Weaving" (Preferences -> JDT Weaving -> "Click to ENABLE")
7. Restart Studio
8. Verify that "JDT Weaving" is enabled (Preferences -> JDT Weaving)

Right there you see 95% of the trouble I had.  Definitely easy to do, once you know what to do, but figuring out that JDT Weaving thing (especially the specific version needed) took some searching.

Now we can get started using Scala in StreamBase.  The impatient can

1. Download the sample
2. Import it (File -> Import ... -> Existing Projects into Workspace -> "Select archive file").

or the dedicated can create a new project from scratch

1. Create a new StreamBase project (Package Explorer -> Right Click -> New -> "StreamBase Project")
2. Add the Scala Nature to it (Right Click on the Project -> Scala -> "Add Scala Nature")

Yeah that is really it.  Turns out that Eclipse does some clever and powerful things by having projects include natures like this.  We now have a project that includes the Java, Scala, and StreamBase natures and they all play nicely together.

Let's create a basic Java Operator so we have something with which to compare (File -> New -> "StreamBase Java Operator").  This auto-generated operator is good enough for your purposes.  I ended up deleting all the comments and variables and what not, so I could strip it down as small as possible for the web.

package com.streambase.sample;

import com.streambase.sb.StreamBaseException;
import com.streambase.sb.Tuple;
import com.streambase.sb.operator.Operator;
import com.streambase.sb.operator.Parameterizable;
import com.streambase.sb.operator.TypecheckException;

public class SampleJavaOperator extends Operator implements Parameterizable {
    public SampleJavaOperator() {
        setPortHints(1, 1);
        setDisplayName(this.getClass().getSimpleName());
        setShortDisplayName(this.getClass().getSimpleName());
    }

    public void typecheck() throws TypecheckException {
        requireInputPortCount(1);
        setOutputSchema(0, getInputSchema(0));
    }

    public void processTuple(int inputPort, Tuple tuple) throws StreamBaseException {
        sendOutput(0, tuple);
    }
}

Let's try a naive translation into Scala.

package com.streambase.sample;

import com.streambase.sb.StreamBaseException;
import com.streambase.sb.Tuple;
import com.streambase.sb.operator.Operator;
import com.streambase.sb.operator.Parameterizable;
import com.streambase.sb.operator.TypecheckException;

class SampleScalaOperator extends Operator with Parameterizable {
    def SampleScalaOperator = { 
        setPortHints(1, 1);
        setDisplayName(this.getClass().getSimpleName());
        setShortDisplayName(this.getClass().getSimpleName());
    }
 
    override def typecheck = {
        requireInputPortCount(1);
        setOutputSchema(0, getInputSchema(0));
    }
 
    override def processTuple(port:Int, tuple:Tuple) = {
        sendOutput(0, tuple);
    }
}

Wow that wasn't too hard, of course it didn't save us any code either.  But it does give us enough to test and figure out if it works at all.  I guess we need to make a simple sbapp to exercise these Operators we have just written.

Yeah, that was easy.  Alright, you caught me, I cheated.  Studio does not (yet) automatically find operators written in Scala, so I had to hand edit the sbapp file.  But everything works after that, this app can be run and everything works as you would expect.

Now for a slightly less trivial example.  I hear that Scala has a really nice concurrency primitive called an actor.  Well, StreamBase has a multi-threaded architecture so let's see if the two can be made to play nicely together.  Once again here is a full project that you can import as above.

class SampleScalaOperator extends Operator with Parameterizable {
    private object coordinator extends Actor {
        def act = {
            loop { 
                receive { 
                    case t: Tuple => sendOutput(0, t)
                    case None => exit()
                }
            }
        }
    }
 
    def SampleScalaOperator = { 
        setPortHints(1, 1)
        val name = getClass.getSimpleName
        setDisplayName(name)
        setShortDisplayName(name)
    }
 
    override def typecheck = {
        requireInputPortCount(1)
        setOutputSchema(0, getInputSchema(0))
    }
 
    override def processTuple(port:Int, t:Tuple) = {
        actor {
            val times = t.getInt("times")
            val delays = t.getLong("delays")
            for (i <- 0 until times) {
                Thread.sleep(delays)
                coordinator ! t 
            }
        }
    }
 
    override def init = { coordinator.start } 
    override def shutdown = { coordinator ! None }
}

Don't forget to edit the input schema for your app to include an int field named "times" and a long field named "delays".  Now we can run the app, enqueue tuples, and watch the delayed tuples arriving out of order.  We could have simply called "sendOutput" from the actor in "processTuple", but I wanted to include actors sending messages to other actors instead of just being glorified threads.

All in all, really solid work on the part of the Scala developers.  I am definitely looking forward to trying Scala on some larger projects.  Hopefully, I will be able share more as I learn it.

Winning Paper Evaluations, Good; Winning Customer Evaluations, Better!

(Mark Palmer is the president and CEO of StreamBase Systems.  Follow him at twitter.com/mrkwpalmer)

Today, Forrester released their evaluation of CEP vendors, and declared StreamBase a leader in the space. Wonderful news - we're very proud. 

That said, analysts typically perform paper evaluations.  Forrester's paper evaluation is a good tool, and helps prospects create short lists for competitive evaluations.  But as a technologist, I'm thankful for the recent public disclosure of findings from some deep-dive customer evaluations from the last six months.  These evaluations were performed by firms that installed the products and ran, in many cases, stringent benchmarks and scalability tests. They evaluated the debugging tools by using them, and used deployment features like high availability.  They looked at how easily threading or partitioning behavior can be designed and controlled by doing it.  

In some cases, these firms released hard numbers.  In others, we have generic performance characterization information that we provide on an as-needed basis.  And in many cases, they put their money where their mouths are - they made one choice and bought the product. In case you missed them, here's the public take on some of these evaluations:

• CME Group (Press release, CME Group by the Numbers article, follow @CMEGroup on Twitter)
• RBC (Press coverage in Inside Market Data)
• BNY ConvergEx (Press coverage in The Trade)
• PhaseCapital (Blog post: 5 Observations about PhaseCapital's selection of CEP, blog post: 8 Lessons for Entrepreneurs - Based on PhaseCapital)
• BlueCrest (BlueCrest case study, interview with Justo Ruiz-Ferrer from BlueCrest)   

Among the more interesting evaluations was the CME Group's work.  They performed the most exhaustive CEP evaluation I've seen in years - perhaps ever - and their findings reflect what we typically see when you dig in deep and stand StreamBase, Aleri, Coral8, IBM, Oracle, TIBCO, and Progress Apama against each other.

Steve Goldman, the director of Technology Architecture from CME Group, said it nicely and even put it on video for us, where he cited usability ("The business users and technology users felt the StreamBase technology was heads above everybody else"), horizontal scalability & performance ("This was very, very important"), and sustainability ("The ability to run the platform reliably and with fault tolerance was key.") as the key reasons why the CME Group chose StreamBase as the leader, and the choice for their enterprise CEP deployment.

Here's the clip of Steve from CME Group, in his own words:

  

This endorsement was so strong and decisive that some of our competitors have started making crazy claims, including that StreamBase gave their product to CME Group for the endorsement.  In some cases the claims are so unprofessional that I enjoy nothing more than personally responding to them (with appropriate confidentiality of course).  In fact, I enjoy doing this so much, I decided to put it right out here on the blog and offer my direct contact.  I can be reached at mpalmer@streambase.com, or directly in my office:  +1 781 761-0812.  Naturally, sales solicitations are not welcome and will not be returned. 

Thomson Reuters Acquires Vhayu: What Does it Mean?

The news today in the field of event processing is that Vhayu Technologies was acquired by Thomson Reuters, sparking a flurry of inquiries from press and analysts asking: "How does this change the complex event processing (CEP) landscape?"  The short answer is: it doesn't.  

Event storage is not synonymous with event processing.  Vhayu's tag lne is:"Store and query years of tick data."  The ability to store and query years of tick (event) data is crucial for back-testing real-time systems, for historical analysis, and for research and development of high-frequncy trading strategies.  Event processing is used to help applications monitor, analyze and act on streaming data in real-time. Usually, customers use both at the same time to process historical, and real-time, streams of events. 

Indeed, this is why Vhayu and Thomson Reuters are both strong partners with StreamBase (Vhayu and StreamBase Partner to Provide Real-Time and Historical Market Data Solutions... and StreamBase and Reuters Extend Next-Generation Market Data Development Environment).  These are not "Barney" partnerships ("I love you, you love me:") - the partnerships are driven by customer use.  For example, RBC uses StreamBase and Vhayu together.  And StreamBase and Reuters is one of the most common and critical link our platform supports, and has supported, for years.  

So what does all of this mean?  It means that the event processing market is continuing to grow in importance - both the database side and the event-processing side.  It means the "IBM of Capital Markets," Thomson Reuters, has endorsed event processing, again.  And when viewed in the context of strategic enterprise endorsements of CEP like CME Group, it means event processing is continuing to gain market and strategic momentum in the capital markets.   

StreamBase & Phase Capital on CNBC: Trading on Twitter

Trading on Twitter: Opportunity, Danger, or Folly?

(Mark Palmer is the president and CEO of StreamBase Systems.  Follow him at twitter.com/mrkwpalmer)

This week my company issued what I thought was an innocuous announcement - our software now allows applications to process Twitter tweets with an application and analyze the contents of the messages in real-time. A firestorm of debate ensued, and the issue gained mainstream media coverage in the Financial Times, the Telegraph, CNBC,and dozens of others. The debate gives us pause to consider the role of the sweeping adoption of Twitter as a real-time, global information source, and how that information might be used beyond the mostly social nature of Twitter today.

One of the examples we cited for this technology is automated trading on Wall Street, where any of the 16 million tweets a day can be analyzed in real time as a means to inform trading decisions.  Wall Street tends to be an innovator with new technologies, so it’s useful to use it as a petri dish to examine some of the opportunities and issues around Twitter in a business context.

Trading on rumor is a time-honored trading technique.   “Buy on rumor; sell on news” is one of the oldest mantras on Wall Street and Main Street alike.  Any tool that improves the speed or quality of transmitting information is of interest to traders.  Twitter does that.   But be careful what you ask for – lots of the tweets on Twitter are garbage – scam artists and manipulators.  On the other hand, there are thousands of credible sources of information, from the SEC (@SEC_Investor_Ed), to the New York Times (@NYTimes), and even exchanges like CMEGroup (@CMEGroup), which has over 500,000 people subscribed to its tweets about the futures and options markets, and more. 

And beyond any individual source, and probably more uniquely, Twitter allows instant access to the sentiment of the masses.

In the Wall Street & Technology article, Todd C. Mirabella, chief investment officer and principal of New York hedge fund QAT, says one of the uses he has found for Twitter is to help look at market volatility and compare it. "So how we get our information, that's the trick. Twitter can give us information on retail " [Tweeters] are the herd," he says.

For example, anything new that Apple does throws Apple’s stock into volatile gyrations.  The recent launch of the iPhone 3GS set off a rash of Twitter tweets about what the herd thinks – taken together, they can reveal sentiment – good and bad. Getting that feedback a few seconds faster, or a little better can make a big difference in trading returns. 

Business leaders, from CEOs to inventors, use Twitter.  For example, Richard Branson (@richardbranson), the CEO of Virgin, tweets regularly.  Now I’m pretty sure the news in the first tweet about Jimmy Fallon isn’t actionable for trading Virgin stock, but the fact that he’s ordering $2.1 billion worth of new planes might be.

Nasir Zubairi, former product manager for algorithmic trading and FX E-commerce, RBS, put it this way in the Wall Street & Technology article:  "[Twitter can be] the fastest source of news, particularly in reference to trading and algorithmic trading," he says. "News feeds are gaining a lot of popularity. Twitter provides the medium in that people, who are there, experiencing things, are able to broadcast their sentiments to their user base via tweets. This provides lots of competitive advantage to firms with trading strategies."

News about world events moves the foreign currency markets as Mr. Mirabella also pointed out.  He’s not yet trading on Twitter but said he sees real use for Twitter in currency trading: "We like to trade in ranges," he explains. "That helps us characterize the extreme edges in any specific currency. We've been trying to follow more individualized Twitter sites to see if we can get extreme views [of Tweeters]. It does give us flavor for what types of ranges we're looking at."

But real use of Twitter has some serious issues.  Technically, it wasn’t built for the kind of quality of service that a robust news source, like Reuters (@Reuters), provides – what happens when Twitter is unavailable?  There are questions about latency of the transmissions – is it fast enough?  Can it be faster?  Authentication and forgery are challenges – there’s a big difference between the real Richard Branson (@richardbranson), and the fake Richard Branson (@sirdickbranson) - who is responsible for the quality and reliability of the information transmitted via Twitter?  Should it be regulated?

And regulatory issues abound, as well.  Trading decisions are moving faster, so regulators have to move more quickly, too.  In December the unfortunate trading run on United Airlines (UAL) (read the story from the Wall Street Journal) stock to drop from $12 to $3 in just 15 minutes only served to remind us how quickly conditions can change in an era of sweeping automation.  The chain reaction was started by an erroneous trading signal from an old story that the Google news crawler picked up from 2002.  How can we guard against these unintended technology-triggered mistakes?

The adoption of Twitter in a social context has been swift, massive, and global.   As technology like event processing makes the content contained in Tweets readily available to business applications, the questions of what to do with this power and responsibility looms large.

Postscript and continued debate (July 1, 2009)

Since this writing, there has been a flurry of response - positive, negative, and vitriolic.  

My favorite coverage was the excellent, balanced story by Melanie Rodier at Wall Street & Technology: Algo Traders Connect to Twitter. She did her homework, including comments by some well-respected people from the trading industry. Read it first.

CNBC's Day-Trading 2.0:  Twitter Takes on Stocks took a day-trader's view on using Twitter to understand market sentiment.  

The Telegraph's coverage, Hedge fund managers betting Twitter will give them an edge in rapid trading also covered the intelligence and military side of the potential use of Twitter.

A spirited message board debate on Server Side was provoked by John Davis (@jtdavis) from OnixS. Like many message board debates, there are rat holes, but there's good stuff there, too.

There was an assertion that StreamBase's announcement of Twitter connectivity is a cheap marketing gimmick. It isn't. This announcement was driven by current and anticipated customer demand, as you can see from our history of connecting to Alpha Trading, Interactive Data, Lime Brokerage, Reuters, other news and data sources of many types, and so on. The article you're reading illustrates the balanced way we're talking about Twitter.  

One blogger, who used to work for one of our competitors, TIBCO, even took the opportunity to predict that StreamBase will be trounced by the "big boys" as they enter our market! To that, I say: "bring it on." Dr. Michael Stonebraker is one of the inventors and thought leaders in event processing, and StreamBase is known as one of the leading businesses in the CEP market, renowned for its performance and ease of use. Are we wary of their deep pockets? Yes. But the big companies have yet to field compelling products - in fact, we rarely compete with them. Furthermore, the "big boys" have not showed signs of innovation and intrapreneurship as I wrote in When Microsoft enters your market:  cause for worry or cause for celebration? And in Innovation: why size doesn't matter, I countered the argument that big companies are the most effective at innovating in a market like CEP. Richard Tibbetts, StreamBase CTO, clarified IBM's recent claims of innovation as misleading in On IBM, unstructured data, and CEP.  Finally, and most importantly, our customer endorsements from CME Group, BNY ConvergEx, BlueCrest, and PhaseCapital and many more say it best - directly from our customer's mouths.

Innovation by the Numbers at CME Group

(Mark Palmer is the president and CEO of StreamBase Systems.  Follow him at twitter.com/mrkwpalmer)

Recently, I spent a day in Chicago with CME Group, the world’s world’s largest and  most diverse derivatives exchange, to talk about their approach to innovation.  As the CEO of an innovative software company founded by forward-looking MIT techies, I’m lucky enough to work with a lot of innovative people.  But as we talked about the size and scale of CME Group’s business, it struck me that this may be among the most technically challenging environments on planet earth, and the numbers tell a compelling story of challenge and opportunity for a firm that many know of, but few have seen inside. 

CME Group is the modern descendent of the Chicago Mercantile Exchange (CME), the American financial and commodity derivative exchange which was founded in 1898 as the Chicago Butter and Egg Board, and in the past 111 years, grown and transformed into the world’s largest electronic futures and options exchange (Globex).

But I wasn’t visiting CME Group to check out their museum, I was there to dig down on the business and technical motivation behind the news that they have selected a new development paradigm (press release, coverage in Wall Street & Technology), event processing (CEP), on which to build future innovations.  This is a watershed moment for the CEP industry, and an important case study that gives us pause to wonder about the future of automated business, the future of real-time, mission-critical infrastructure, and the role of event processing as the platform for that innovation.

So let’s look at innovation, CEP, and the challenges faced by CME Group: 

...1.2 quadrillion dollars ($1,200,000,000,000,000,000) 

                                           ...The underlying value of the contracts traded by CME Group in 2008

How can a firm stay on the innovation edge while simultaneously supporting the most mission critical operations?

The decision to process 1.2 quadrillion dollars of value was not taken lightly at CME Group. It’s been said that “nobody gets fired for buying IBM” – that is, that big companies are the safe choice when a decision is most critical. 

But when innovation must ride beside reliability, decisions get trickier.  A company like CME Group must balance innovation with mission-critical capability.  And, in the case of CEP, CME Group’s decision illustrates that CEP has come of age.

However, there’s been a rash of recent debate about the entrance of IBM and Microsoft into the CEP market.  Recently, the New York Times published an article, which declared that innovation is moving to big companies from small companies – a conclusion that is not only wrong, it flies in the face of all the most basic tenets of innovation.  So in the innovative, fast paced CEP market, we are wary of, but don’t fear, the entrance of giants like IBM and Microsoft.  Indeed, we welcome it, because it draws more awareness.  The more people look, the more will buy StreamBase.

And from the customer’s point of view   I asked Steve Goldman, CME Group’s Director of Enterprise Architecture, if he had looked at IBM and Microsoft, and his response was casual:  “The decision was very difficult but we felt that based on our environment and needs thatStreamBase was a proven solution.” 

11,000,000...

                   ...The current average daily volume of futures contracts traded every day by CME Group...

“Our data feeds are like fire hoses,” Goldman said.  “Sometimes, we process a few hundred market data changes a second.  But when the markets get volatile, we process tens of thousands of market data changes per second. And, to make it even more challenging, our customers demand the fastest response time – in milliseconds – when volatility is at its peak.” 

Real-time response is becoming more and more important in mainstream business.   Any environment that deals with events (stock market price changes, web site click data, and so on), can benefit from event processing.  And the technological challenge of managing events at high speed, and with low-latency, requires a host of arcane tooling and expertise.  These geeky areas will also repay deeper exploration, including specialized messaging hardware and software, clever use of modern computing models like multi-core CPUs, grids, and clouds, FPGA’s, and efficient management techniques to keep it all running smoothly.

But modern CEP engines are becoming an essential tool for managing enterprise-class streaming IT environments – a better way to manage information that comes through a fire hose.

Hundreds....

                ...The number of products offered by CME Group...

While most businesses might have a few products, or a big business might have a dozen products, CME Group has hundreds of products. But CME Group’s products aren’t made of physical materials; they are built with software development tools.  So the speed and effectiveness of software development at the CME Group dictates the speed and growth of their business.

So the news that CME Group has chosen a new enterprise-wide development paradigm is important news.   CEP, which has been rising to prominence in the past 5 years, flips traditional database-oriented computing on its head.  CEP allows an application developer to design logic that examines what’s happening now instead of what happened in the past.  Options pricing and settlement consumes vast amounts of streaming information, so the CEP model fits perfectly, and allows CME Group to quickly create new products that process those streams.

Any business that has streams of data – from web clickstreams to point-of-sale transactions to financial market data - can apply CEP to more quickly and easily bring new products to market.  When the development methodology matches the form of the data, innovation approaches the speed of human thought.

80...

        ...The number of countries where CME Group's Globex provides regulated exchange access...

Although everyone talks of the inevitability of a global economy,  few firms understand its practical challenges than CME Group. CME Group offers electronic, regulated exchange access in over 80 countries.

The host of technologies that enables a 7x24, global operations are many.  But CEP, by making streams of data from all over the globe more intelligent, can make network event processing smarter and faster, and is becoming an important element in enterprise IT infrastructure.

Which leads to a critical innovation issue:  how to strike the right balance between a new technology and a proven technology?  A full answer to that question is outside the scope of this article, but the key elements for enterprise software products include well-designed threading architectures, bullet proof fault tolerance systems, robust management tools, comprehensive testing and debugging tools, and profiling tools. Consider that CME Group traces its origin to 1848, has NEVER had a default or a loss of customer funds resulting from failure, today announced its enterprise deployment of CEP technology.   This is the clearest signal yet that event processing is coming of age as it continues through its rapid growth as a software market.

A few milliseconds....

                                ...the response time necessary at the most heavily loaded times....

To see the manual forms used at exchange over 50 years ago puts the innovation in the capital markets  in perspective.  It took many seconds to fill out the information about trades back then; it takes 400 milliseconds (1/1000 of a second) for the human eye to blink;  today, CME Group’s trading decisions take place much faster than the blink of an eye – in just a few milliseconds.  How can this kind of latency be achieved in the face of massive volumes and massive global distribution?

We discussed the role of CEP in turning the traditional database-oriented model on its head.  Databases, of course, rely on writing data to disk.  The typical seek time for a disk is about 9 milliseconds.  CME Group’s data can change tens of thousands of times per second;  in order to make decisions in a few milliseconds

Exchanges settling contracts, brokers splitting up and routing client trades, electronic liquidity providers making markets, hedge funds finding and executing arbitrage trades, banks delivering currency rates…all in less than one hundredth of a human blink!  Applications are measured in microseconds, and the tools that do the measuring have resolution in nanoseconds, that’s one billionth of a second.  

StreamBase has been emerging as the top CEP company for some time now due to its proven rapid development environment, standards-based approach, high performance, low latency, and mission critical, fault tolerant architecture.   CME Group’s selection of StreamBase is yet another proof point that demonstrates that CEP is ready for the big time, and that StreamBase is leading the way.

That’s why CME Group made this important decision for their real-time, low latency operations – the need for speed, and the need for CEP, is critical at CME Group.

Massive scale.  Instantaneous response.  Constant change.  Nonstop operations. Taken individually, these are big technical challenges; taken simultaneously, they are an extreme business risk.  But risk for one is opportunity for another, and as CME Group shows us, by keeping an eye on the three key “abilities” – usability, reliability, and scalability –firms can keep innovating as business conditions continue to get more complicated.

 

Finding Alpha: Be Fast, Be Smart, or Be Dirty

(Richard Tibbetts is Chief Technology Officer at StreamBase Systems. Follow him on Twitter as @tibbetts)

When it comes to algorithmic trading, everyone is looking for alpha, outsized returns which exceed your risk. In an efficient market, there would never be any alpha, so seeking alpha means seeking inefficiencies in the market. The problem with market inefficiencies is that they don't last. The mere act of using them depletes the available revenue - when others join in, the opportunity can go away very quickly.

So not only do you have to find alpha, you have to find sustainable alpha. Since a lot of customers use StreamBase and complex event processing for algorithmic trading, I see a lot of alpha seekers. Three attributes seem to consistently contribute to sustainable alpha: being smart, being fast, or being dirty. To create a sustainable strategy, mix in at least one, and preferably more of these components.

Smart — The best-known way to create sustainable alpha is to be smarter than the competition. Unfortunately, on Wall Street, genius is a commodity, so trading techniques must be both creative, and evolve. Industry analyst AITE Group found that some algorithms have a shelf life of only a few weeks. And, since market conditions are constantly changing, “smart” is never “static.” Finally, if you can discover something based on easily available public information, the other geniuses won't be far behind. So smart, the best way to achieve alpha, must also be creative, and must also constantly evolve and get smarter as market conditions evolve.

Fast — Another way of securing alpha is to be faster than the competition. This means making decisions quickly, then executing your trades before competitors can execute theirs. Efficiency and speed is important for any system, but if you are betting on being the fastest, ultra-low-latency execution management will be keeping you up at night. The challenge with fast is that it's easy to spend money to get faster. If you are a small company, larger competitors with will always be able to outspend you, so you must take advantage of your size. In 8 Lessons for Entrepreneurs – A Case Study Based on Phase Capital, the lessons from this small, innovative firm included leveraging automation, building incrementally and quickly, and choosing the right partner. On the other hand, large companies can fight off more nimble competitors with their scale, reach, and ability to choose and deploy bleeding edge technology that smaller IT departments cannot support. In either case, you must exploit your natural advantages to stay fast.

Dirty — Finally, you can be dirtier than the competition. I don't mean being ethically questionable. Instead, I mean working with markets, information, and technology that others avoid. Developers and quants both like systems that make sense, clean systems that have been developed by like-minded engineers and are easier to work with. Getting your hands dirty with legacy systems, systems that haven't yet been polished to industry standards, or systems which were never designed for trading exploitation is one way to distinguish yourself. For example, trading in emerging markets often requires getting your hands dirty. But it also lets you trade securities that others aren't following as closely. Trading uncommon or unknown products is one strategy for finding alpha. So is trading based on non-financial information. This has famously been done with weather data and Amazon sales information. Steve Steinberg gave a great presentation on this at O'Reilly's Money:Tech. Working with these data sources may require checking your engineering sensibilities at the door, but the rewards can be substantial.

Lots of trading firms use complex event processing as their platform to create alpha-seeking strategies. But not all CEP platforms are the same. So, when selecting an alpha-generating technology platform, it is important to know what kind of strategy you are building, and what key technical elements you’ll need to meet your requirements.

To build smart strategies, a development platform must have a flexible programming language, good development tools (debugger, profiling, testing), and a good backtesting environment so that you can design, test, deploy, and evolve smart alpha-seeking strategies. There is no one more frustrated than a smart quant who is told by developers that his new strategy can't be built in their system. For example, can your platform handle multi-security strategies? Cross-asset strategies? Plug in analytics written in your quant’s' favorite programming languages? All these features and more may be required to support smart strategies.

To build fast strategies, you will need a platform and deployment environment that supports ultra-low latency. This will generally require collocation at the site of your favorite exchange. If your strategy requires interaction with multiple exchanges, than your deployment architecture must take into account the wide area network latencies involved. Your development platform should also support low-latency data transports like Infiniband, and have a multi-threaded architecture that optimizes the processing of individual events. And for some fast strategies, throughput and scalability will also be important, so multi-core technologies, clustering, and cloud computing techniques may be required.

To build dirty strategies, your platform needs to ingest any kind of data you might need. This could include unstructured data processing for handling news, analyst reports, underlying data that effects, for example, a derivative like weather information, and so on. An effective platform for dirty alpha-seeking strategies requires APIs and development tools which support building connectivity quickly. Pre-built vendor connectivity is nice to have, but for a dirty strategy you are likely to connect to arcane information sources or technologies.

If you are planning to find alpha, know what your goals are. When selecting a platform, make sure it supports the kind of alpha you will generate. If you choose correctly, the platform will help secure your alpha against the competition.