The StreamBase Event Processing Blog

9 Predictions for the Future of Event Processing (CEP)

The information technology word is buzzing about event processing (or complex event processing - CEP) as one of the hottest areas in software innovation. For example, Time Magazine declared event processing to be one of 8 “Innovations that will change your life,” the World Economic Forum at Davos named event processing as one of the 26 top innovations in the world, Gartner analyst Roy Schulte calls event processing the key to “agile companies,” and event processing is all over top trends lists for 2010, including Entrepreneur / VentureBeat, CIO Magazine, and well-respected IT industry analyst Brenda Michelson.

Before getting to predictions for the future, let’s step back and look at what the hype is all about.

What’s the problem? Why event processing? What is event processing?

Let’s start with the problem: recently, and for the foreseeable future, the physics of data has fundamentally changed. The first 50 years of data processing managed data at rest; the next 50 years of data processing will manage information in motion. Sensor data, digitized stock market feeds, click-stream data from online commerce, and events from real-time social media activity - all this information is in motion, streaming through the global high-speed network of computers (watch this 1-minute video tutorial on event processing for an overview).

Not only is today’s data in motion, there’s more of it. In 2 minutes, the data stream from Twitter generates the amount of text found in the entire works of William Shakespeare. A single feed of U.S. stock market data generates the same volume in under 3 seconds.

Some industries, like financial services, have quietly figured out how to harness the power of this streaming data - for example, 70% of the trading done in the U.S. stock market every day depends on streaming data, and much of it uses event processing technology.  The federal government uses event processing broadly, and uses of event processing outside of these markets are growing.

The early impact of event processing has been profound because it presents an opportunity to innovate by applying a new computing physics for moving data. In the late 90‘s the inventors of the relational database (Dr. Michael Stonebraker and others) had a simple idea: to process moving data with a language (SQL) borrowed from the lingua franca of databases, but which implements its processing in a way that eliminates the latency and restrictions of storing data to disk. The objective was to help applications find needles before they go into the haystack. Metaphorically speaking, a business that uses event processing is like the bear in the photo above: the bear can know - in real-time - where the salmon is. If the bear had to wait for a database, he’d be finding out next week where the salmon was.

It’s a simple idea, but not easy to implement. But 10 years on and driven by pioneers in financial services, event processing now processes quadrillions of dollars a year on today’s version of the virtual, electronic Wall Street. The natural gestation period of 10 years is ending, the technology is proven, and it’s fun to see it proclaimed to be the next big thing.

Predictions for the Evolution of Event Processing 

OK, so lots of smart people think event processing is hot and important. But history is littered with dead technologies that were once proclaimed as “the next big thing.” What does the future of event processing hold? Since a streambase sounds as important as a database, will Oracle own this space? What role will open source play? Will the existing IT infrastructure providers be the gorillas or the monkeys of the event processing era? 

As the CEO of what most consider to be the leading pure play event processing company, I'm fortunate to see signs that foreshadow the emergence of a great industry. So here are my predictions for the future of event processing, beginning with the dramatic change in the nature of event processing customer deployments that has occurred in just the past 18 months.

#1: 100+ event processing applications per firm milestone will be passed. Event processing turned an important corner in 2009; it went from an innovative enigma to a key architectural elixir in some of the world’s most innovative companies. Previously, early adopters used event processing in applications like algorithmic trading, fraud detection, location-based services, and homeland security. These applications were an important first steps, and a sign of things to come.

But the turning point in 2009 was news that some of the world's biggest companies went big with event processing. The watershed event was when the CME Group, one of the largest financial exchanges in the world, which trades $1.3 quadrillion (1,300,000,000,000) in assets a year, adopted event processing enterprise-wide. And other big names made big announcements about event processing like RBC, BNY ConvergEx, CMC Markets, and City Index. These firms aren't just building a single application with event processing; they made broad, multi-application, enterprise-wide commitments to a new technological backbone.

Beginning in 2010, watch for the next step - for individual firms to deploy over a hundred event processing applications throughout their new IT foundation - a sign that event processing is coming of age as the backbone for the real-time enterprise.

to be continued....

I'll release a new blog post every day or so with the rest of my predictions, and announce them on Twitter (@mrkwpalmer).

Why the CEP Tide Has Turned Toward StreamBase

Last week, CMC Markets announced that they switched from Apama to StreamBase as their enterprise-wide event processing platform. As FXWeek reported, CMC “initially spent three months deploying a similar system from Apama, but dropped the deal in favor of StreamBase.” As a firm that trades over $1.4 trillion a year, CMC’s decisions about their infrastructure are not taken lightly. So their decision gives us pause to ask: why the switch? 

Asif Adatia, Chief Information Officer for CMC, described their decision like this: “the ease of development for both financial engineers and technical engineers, the performance, and scalability of the platform were our key criteria.” Let’s examine each part of that statement.

1. Demonstrably better ease of use. You might think that the efficacy of a development environment would be a subjective call. In some ways, it is - when it comes to the visual esthetics of a tool. On the other hand, usability is often the driver for market-altering innovation - profound shifts in the accessibility of a new tool can suddenly unlock barriers to innovation. The change in the last 10 years in finance has been profound. 10 years ago, algorithmic trading wasn't even named, now 70% of the U.S. equities trade execution happens algorithmically. In their report "The World According to Quants: Enter Alpha Generation Platforms", the AITE group reported that certain trading algorithms have a shelf-life of less than 3 months. In the face of all that rapid change and ephemeral opportunity, the ability to quickly build real-time trading infrastructure (algos, execution, risk, compliance, pricing, TCA, surveillance) is a matter of survival.

Firms like CMC, CME Group, and RBC didn't choose StreamBase for its aesthetics, they chose StreamBase because its graphical development environment gives them a fundamental competitive edge. The tools aren't just easy to use, they allow business users and engineers to work together because the language of StreamBase matches the way they think. The StreamBase dataflow language was designed from the beginning to program event-based systems the way you would draw them on a whiteboard, and the proof lies in what customers say, as Adatia told FX week, “Our financial engineers find it really easy to put the models together.” 

And this ease of use delivers another critical innovation: firms get this usability and time to market advantage and the performance benefits of event processing - at the same time. StreamBase isn't like other CEP tools that make the developer resort to low-level programming to make the system perform and scale. Relieving the user from the burden of thinking about plumbing was described by Adatia to A-Team Insight “[CMC] create their models readily, without resorting to code-level programming.” PhaseCapital reported that they built their trading system with StreamBase in 4 months rather than the 4 years it would take to build it with traditional development tools and they change their strategies every day. These are not subjective opinions about aesthetics, they are qualitative and quantitative testimonials to the innovation that StreamBase's interface can provide.

2. Built-in performance you can see. On Wall Street, milliseconds, and microseconds matter. CMC switched to StreamBase because they wanted the fastest engine they could get. 

StreamBase was designed for speed from the very beginning. Release 1.0 in 2005, was designed to take advantage of modern hardware, from the multi-threaded server to the StreamBase language which produces highly performance applications without burdening the developer. The StreamBase language plans performance like a database pre-processor plans query optimization. In 2007, StreamBase engineers broke down another barrier with a straight-through event processing compiler. Recently, in 2008, direct bytecode generation optimization took the StreamBase performance innovation even further ahead of all other CEP platforms.

When performance is something you can see, it's a remarkable result. CMC said it this way: '[we don't talk about our specific latency numbers publically, but the performance is] good though and you can compare us on FX pricing to a bunch of our competitors.  And you will see our performance is noticeably better at times. Absolutely."  The world's largest electronic futures and options exchange in the world, CME Group chose StreamBase after using performance as a critical selection criteria, because they process over 11 million trades a day.

3. Scalability to grow. You might assume better scalability means StreamBase-based applications can be run on multiple CPU cores, and that’s true - better use of hardware means CMC needs less hardware. CMC described that their StreamBase system easily handles "More than twice the real-life peak rate for price ticks encountered so far.”  

But better scalability isn’t only about increasing volumes - it’s about ease of management, ease of customer support, and the ease with which high performance and scalability can be attained by developers.

And scalability is also about expanding usage. Apama established early on algorithmic trading as an important use case for CEP. But now, as CEP's use explodes into new application areas, firms are looking more deeply at how easy it is to build their second, third, and fourth applications. Just last week we had a call with a customer who guessed they had deployed "about 100 StreamBase applications." 

And now we have come full circle back to the StreamBase development environment. Our mantra is to “deliver high performance and scalable applications without having the developer think about performance and scalability.” Streambase makes performance native, which is one more reason it is unique.  

CMC is not alone. Organizations that are adopting StreamBase CEP are seeing these same benefits in ease of use and performance. CEP is the disruptive technology that frees their engineers to focus on creating business value instead of plumbing. CEP empowers their business analysts to work with the actual software instead of wishlists and design documents. CEP allows the whole organization to be more agile, to deploy changes to their infrastructure faster, and to take advantage of new opportunities ahead of the competition.

Hire Great People: 6 "How To" Tips

This week, I attended Accel Partner's CEO Summit at Stanford University, along with 150 CEOs from Accel's portfolio companies. We were joined by some well-known public CEOs and the Accel partners in a series of "fire side chats" and networking sessions to talk about leadership and entrepreneurship. During a morning session, Michael Dell was asked for his best advice for entrepreneurs, and reminded us to: hire great people.

During the day, the theme of hiring great people raised some interesting ideas from the audience and the panelists, and even on Twitter as I tweeted online during the event.

1. Spend HOURS with candidates and insist on a good cultural fit.  Michael Dell went on to share a simple, but powerful idea: toss out the idea that interviews should last an hour, and spend HOURS with new candidates. He told a story about a time where he picked up a candidate and drove to work together for an extended period of time, getting to know each other. Apparently Dell is a very "aggressive" driver. The candidate's reaction under stress told him something about his character.  

2. Go to a U2 concert.  Mark Zukenburg (founder) and Sheryl Sandberg (COO BusinessWeek profile), co-leaders of Facebook, talked about going to a U2 Concert together, and spending nights and evenings talking about the future and culture of Facebook before committing to work together.

3. If there's no cultural fit, fire away.  Lee Scott, recently retired CEO of Wal-Mart, said that there were two things at Wal-Mart that would get you fired immediately, regardless of performance: stealing and not "matching the Wal-Mart culture," which is:

    a) The customer is the boss, 

    b) Treat people with dignity,

    c) Improve every day

4. Hire people better than you.  Lee Scott's story of a key point in his career illustrated this well. He was a logistics manager, and was suddenly promoted to VP and handed one of the most critical new roles in Wal-Mart. The move ignited his rise to the top at Wal-Mart.  The reason he was handed the job?  "Because you were the only one on our list who had hired someone strong enough below him that we knew would do as good or better job than you would, so we could promote you," he was told.

5. Ask candidates: "Describe your sense of humor." In response to my Twitter tweet about this (follow me at @mrkwpalmer), my former colleague and CEO Peter Beard said that his favorite interview question was to ask candidates to describe their sense of humor, or to tell their favorite joke.  Leanne Browne from Progress said she thinks the best way to interview someone is to take them to the pub. I was reminded how one of my most insightful interviews happened on the golf course, where I learned that the candidate was a great golfer (he was very skilled at work too), he was late for the tee time (he was consistently late for meetings), he was lazy and always drove in the cart instead of walking a short distance (he cut corners at work too), and he was very creative in his shot selection (he was creative at work too).

6. Get great investors. These comments came in the context of Accel's CEO Summit. Accel was just named number one on Red Herring's List of the Top 100 Global Venture Capitalists. Enrique Salem, CEO of Symantec, explained how his VC's helped him decide whether to sell BrightMail or go public, eventually deciding on a $370M sale to Symantec (on $25M trailing revenues) that catapulted their business in a way he didn't think they could do on his own.

6 great reminders about one of the most important things we all do: choose the people we surround ourselves with.

Trading on Twitter: Opportunity, Danger, or Folly?

This week my company issued what I thought was an innocuous announcement - our software now allows applications to process Twitter tweets with an application and analyze the contents of the messages in real-time. A firestorm of debate ensued, and the issue gained mainstream media coverage in the Financial Times, the Telegraph, CNBC,and dozens of others. The debate gives us pause to consider the role of the sweeping adoption of Twitter as a real-time, global information source, and how that information might be used beyond the mostly social nature of Twitter today.

One of the examples we cited for this technology is automated trading on Wall Street, where any of the 16 million tweets a day can be analyzed in real time as a means to inform trading decisions.  Wall Street tends to be an innovator with new technologies, so it’s useful to use it as a petri dish to examine some of the opportunities and issues around Twitter in a business context.

Trading on rumor is a time-honored trading technique.   “Buy on rumor; sell on news” is one of the oldest mantras on Wall Street and Main Street alike.  Any tool that improves the speed or quality of transmitting information is of interest to traders.  Twitter does that.   But be careful what you ask for – lots of the tweets on Twitter are garbage – scam artists and manipulators.  On the other hand, there are thousands of credible sources of information, from the SEC (@SEC_Investor_Ed), to the New York Times (@NYTimes), and even exchanges like CMEGroup (@CMEGroup), which has over 500,000 people subscribed to its tweets about the futures and options markets, and more. 

And beyond any individual source, and probably more uniquely, Twitter allows instant access to the sentiment of the masses.

In the Wall Street & Technology article, Todd C. Mirabella, chief investment officer and principal of New York hedge fund QAT, says one of the uses he has found for Twitter is to help look at market volatility and compare it. "So how we get our information, that's the trick. Twitter can give us information on retail " [Tweeters] are the herd," he says.

For example, anything new that Apple does throws Apple’s stock into volatile gyrations.  The recent launch of the iPhone 3GS set off a rash of Twitter tweets about what the herd thinks – taken together, they can reveal sentiment – good and bad. Getting that feedback a few seconds faster, or a little better can make a big difference in trading returns. 

Business leaders, from CEOs to inventors, use Twitter.  For example, Richard Branson (@richardbranson), the CEO of Virgin, tweets regularly.  Now I’m pretty sure the news in the first tweet about Jimmy Fallon isn’t actionable for trading Virgin stock, but the fact that he’s ordering $2.1 billion worth of new planes might be.

Nasir Zubairi, former product manager for algorithmic trading and FX E-commerce, RBS, put it this way in the Wall Street & Technology article:  "[Twitter can be] the fastest source of news, particularly in reference to trading and algorithmic trading," he says. "News feeds are gaining a lot of popularity. Twitter provides the medium in that people, who are there, experiencing things, are able to broadcast their sentiments to their user base via tweets. This provides lots of competitive advantage to firms with trading strategies."

News about world events moves the foreign currency markets as Mr. Mirabella also pointed out.  He’s not yet trading on Twitter but said he sees real use for Twitter in currency trading: "We like to trade in ranges," he explains. "That helps us characterize the extreme edges in any specific currency. We've been trying to follow more individualized Twitter sites to see if we can get extreme views [of Tweeters]. It does give us flavor for what types of ranges we're looking at."

But real use of Twitter has some serious issues.  Technically, it wasn’t built for the kind of quality of service that a robust news source, like Reuters (@Reuters), provides – what happens when Twitter is unavailable?  There are questions about latency of the transmissions – is it fast enough?  Can it be faster?  Authentication and forgery are challenges – there’s a big difference between the real Richard Branson (@richardbranson), and the fake Richard Branson (@sirdickbranson) - who is responsible for the quality and reliability of the information transmitted via Twitter?  Should it be regulated?

And regulatory issues abound, as well.  Trading decisions are moving faster, so regulators have to move more quickly, too.  In December the unfortunate trading run on United Airlines (UAL) (read the story from the Wall Street Journal) stock to drop from $12 to $3 in just 15 minutes only served to remind us how quickly conditions can change in an era of sweeping automation.  The chain reaction was started by an erroneous trading signal from an old story that the Google news crawler picked up from 2002.  How can we guard against these unintended technology-triggered mistakes?

The adoption of Twitter in a social context has been swift, massive, and global.   As technology like event processing makes the content contained in Tweets readily available to business applications, the questions of what to do with this power and responsibility looms large.

Postscript and continued debate (July 1, 2009)

Since this writing, there has been a flurry of response - positive, negative, and vitriolic.  

My favorite coverage was the excellent, balanced story by Melanie Rodier at Wall Street & Technology: Algo Traders Connect to Twitter. She did her homework, including comments by some well-respected people from the trading industry. Read it first.

CNBC's Day-Trading 2.0:  Twitter Takes on Stocks took a day-trader's view on using Twitter to understand market sentiment.  

The Telegraph's coverage, Hedge fund managers betting Twitter will give them an edge in rapid trading also covered the intelligence and military side of the potential use of Twitter.

A spirited message board debate on Server Side was provoked by John Davis (@jtdavis) from OnixS. Like many message board debates, there are rat holes, but there's good stuff there, too.

There was an assertion that StreamBase's announcement of Twitter connectivity is a cheap marketing gimmick. It isn't. This announcement was driven by current and anticipated customer demand, as you can see from our history of connecting to Alpha Trading, Interactive Data, Lime Brokerage, Reuters, other news and data sources of many types, and so on. The article you're reading illustrates the balanced way we're talking about Twitter.  

One blogger, who used to work for one of our competitors, TIBCO, even took the opportunity to predict that StreamBase will be trounced by the "big boys" as they enter our market! To that, I say: "bring it on." Dr. Michael Stonebraker is one of the inventors and thought leaders in event processing, and StreamBase is known as one of the leading businesses in the CEP market, renowned for its performance and ease of use. Are we wary of their deep pockets? Yes. But the big companies have yet to field compelling products - in fact, we rarely compete with them. Furthermore, the "big boys" have not showed signs of innovation and intrapreneurship as I wrote in When Microsoft enters your market:  cause for worry or cause for celebration? And in Innovation: why size doesn't matter, I countered the argument that big companies are the most effective at innovating in a market like CEP. Richard Tibbetts, StreamBase CTO, clarified IBM's recent claims of innovation as misleading in On IBM, unstructured data, and CEP.  Finally, and most importantly, our customer endorsements from CME Group, BNY ConvergEx, BlueCrest, and PhaseCapital and many more say it best - directly from our customer's mouths.

Innovation by the Numbers at CME Group

(Mark Palmer is the president and CEO of StreamBase Systems.  Follow him at twitter.com/mrkwpalmer)

Recently, I spent a day in Chicago with CME Group, the world’s world’s largest and  most diverse derivatives exchange, to talk about their approach to innovation.  As the CEO of an innovative software company founded by forward-looking MIT techies, I’m lucky enough to work with a lot of innovative people.  But as we talked about the size and scale of CME Group’s business, it struck me that this may be among the most technically challenging environments on planet earth, and the numbers tell a compelling story of challenge and opportunity for a firm that many know of, but few have seen inside. 

CME Group is the modern descendent of the Chicago Mercantile Exchange (CME), the American financial and commodity derivative exchange which was founded in 1898 as the Chicago Butter and Egg Board, and in the past 111 years, grown and transformed into the world’s largest electronic futures and options exchange (Globex).

But I wasn’t visiting CME Group to check out their museum, I was there to dig down on the business and technical motivation behind the news that they have selected a new development paradigm (press release, coverage in Wall Street & Technology), event processing (CEP), on which to build future innovations.  This is a watershed moment for the CEP industry, and an important case study that gives us pause to wonder about the future of automated business, the future of real-time, mission-critical infrastructure, and the role of event processing as the platform for that innovation.

So let’s look at innovation, CEP, and the challenges faced by CME Group: 

...1.2 quadrillion dollars ($1,200,000,000,000,000,000) 

                                           ...The underlying value of the contracts traded by CME Group in 2008

How can a firm stay on the innovation edge while simultaneously supporting the most mission critical operations?

The decision to process 1.2 quadrillion dollars of value was not taken lightly at CME Group. It’s been said that “nobody gets fired for buying IBM” – that is, that big companies are the safe choice when a decision is most critical. 

But when innovation must ride beside reliability, decisions get trickier.  A company like CME Group must balance innovation with mission-critical capability.  And, in the case of CEP, CME Group’s decision illustrates that CEP has come of age.

However, there’s been a rash of recent debate about the entrance of IBM and Microsoft into the CEP market.  Recently, the New York Times published an article, which declared that innovation is moving to big companies from small companies – a conclusion that is not only wrong, it flies in the face of all the most basic tenets of innovation.  So in the innovative, fast paced CEP market, we are wary of, but don’t fear, the entrance of giants like IBM and Microsoft.  Indeed, we welcome it, because it draws more awareness.  The more people look, the more will buy StreamBase.

And from the customer’s point of view   I asked Steve Goldman, CME Group’s Director of Enterprise Architecture, if he had looked at IBM and Microsoft, and his response was casual:  “The decision was very difficult but we felt that based on our environment and needs thatStreamBase was a proven solution.” 

11,000,000...

                   ...The current average daily volume of futures contracts traded every day by CME Group...

“Our data feeds are like fire hoses,” Goldman said.  “Sometimes, we process a few hundred market data changes a second.  But when the markets get volatile, we process tens of thousands of market data changes per second. And, to make it even more challenging, our customers demand the fastest response time – in milliseconds – when volatility is at its peak.” 

Real-time response is becoming more and more important in mainstream business.   Any environment that deals with events (stock market price changes, web site click data, and so on), can benefit from event processing.  And the technological challenge of managing events at high speed, and with low-latency, requires a host of arcane tooling and expertise.  These geeky areas will also repay deeper exploration, including specialized messaging hardware and software, clever use of modern computing models like multi-core CPUs, grids, and clouds, FPGA’s, and efficient management techniques to keep it all running smoothly.

But modern CEP engines are becoming an essential tool for managing enterprise-class streaming IT environments – a better way to manage information that comes through a fire hose.

Hundreds....

                ...The number of products offered by CME Group...

While most businesses might have a few products, or a big business might have a dozen products, CME Group has hundreds of products. But CME Group’s products aren’t made of physical materials; they are built with software development tools.  So the speed and effectiveness of software development at the CME Group dictates the speed and growth of their business.

So the news that CME Group has chosen a new enterprise-wide development paradigm is important news.   CEP, which has been rising to prominence in the past 5 years, flips traditional database-oriented computing on its head.  CEP allows an application developer to design logic that examines what’s happening now instead of what happened in the past.  Options pricing and settlement consumes vast amounts of streaming information, so the CEP model fits perfectly, and allows CME Group to quickly create new products that process those streams.

Any business that has streams of data – from web clickstreams to point-of-sale transactions to financial market data - can apply CEP to more quickly and easily bring new products to market.  When the development methodology matches the form of the data, innovation approaches the speed of human thought.

80...

        ...The number of countries where CME Group's Globex provides regulated exchange access...

Although everyone talks of the inevitability of a global economy,  few firms understand its practical challenges than CME Group. CME Group offers electronic, regulated exchange access in over 80 countries.

The host of technologies that enables a 7x24, global operations are many.  But CEP, by making streams of data from all over the globe more intelligent, can make network event processing smarter and faster, and is becoming an important element in enterprise IT infrastructure.

Which leads to a critical innovation issue:  how to strike the right balance between a new technology and a proven technology?  A full answer to that question is outside the scope of this article, but the key elements for enterprise software products include well-designed threading architectures, bullet proof fault tolerance systems, robust management tools, comprehensive testing and debugging tools, and profiling tools. Consider that CME Group traces its origin to 1848, has NEVER had a default or a loss of customer funds resulting from failure, today announced its enterprise deployment of CEP technology.   This is the clearest signal yet that event processing is coming of age as it continues through its rapid growth as a software market.

A few milliseconds....

                                ...the response time necessary at the most heavily loaded times....

To see the manual forms used at exchange over 50 years ago puts the innovation in the capital markets  in perspective.  It took many seconds to fill out the information about trades back then; it takes 400 milliseconds (1/1000 of a second) for the human eye to blink;  today, CME Group’s trading decisions take place much faster than the blink of an eye – in just a few milliseconds.  How can this kind of latency be achieved in the face of massive volumes and massive global distribution?

We discussed the role of CEP in turning the traditional database-oriented model on its head.  Databases, of course, rely on writing data to disk.  The typical seek time for a disk is about 9 milliseconds.  CME Group’s data can change tens of thousands of times per second;  in order to make decisions in a few milliseconds

Exchanges settling contracts, brokers splitting up and routing client trades, electronic liquidity providers making markets, hedge funds finding and executing arbitrage trades, banks delivering currency rates…all in less than one hundredth of a human blink!  Applications are measured in microseconds, and the tools that do the measuring have resolution in nanoseconds, that’s one billionth of a second.  

StreamBase has been emerging as the top CEP company for some time now due to its proven rapid development environment, standards-based approach, high performance, low latency, and mission critical, fault tolerant architecture.   CME Group’s selection of StreamBase is yet another proof point that demonstrates that CEP is ready for the big time, and that StreamBase is leading the way.

That’s why CME Group made this important decision for their real-time, low latency operations – the need for speed, and the need for CEP, is critical at CME Group.

Massive scale.  Instantaneous response.  Constant change.  Nonstop operations. Taken individually, these are big technical challenges; taken simultaneously, they are an extreme business risk.  But risk for one is opportunity for another, and as CME Group shows us, by keeping an eye on the three key “abilities” – usability, reliability, and scalability –firms can keep innovating as business conditions continue to get more complicated.

 

27 Twitter Lessons for CEOs

(Mark Palmer is the president and CEO of StreamBase Systems.  Follow him at twitter.com/mrkwpalmer)

I’m a CEO, and I use Twitter.  I started “tweeting,” or using Twitter, as a test a few months ago. When I told one of my board members, he was visibly shaken - all he knew about Twitter he had learned from his teenage daughter, and I could tell he wondered about my focus.  But as I engaged with it, I kept a list of the pros and cons from a business perspective.  At first, Twitter seemed like a big waste of time - but slowly, my list of pros grew, and my list of cons shrunk.  Three months later, I’m convinced that Twitter is an essential business tool.

I follow Twitter Tips (@twitter_tips), but I couldn't find a list of tips for CEOs from another CEO.  So, even though I was loath to post Yet-Another-Article-About-Twitter, I thought my list might be informative, fun, and best of all, already done.  So here's my private list of 26 Twitter lessons and observations:

1. The business case of Twitter is compelling.   Twitter, when combined with other social networking tools, deliver measurable and compelling business value.  Lots of it.

Since my companies started using social networking a few years ago, (blogging, LinkedIn, Twitter, and so on), we've reduced marketing spend by over 50% and roughly tripled marketing effectiveness (measured by lead generation volume, cost per lead, press coverage, and other quantitative metrics) - at the same time.  We showed the chart at the right at my last board meeting.  Marketing spend is the green area chart in back, and leads are in the bar chart. 

But let's not fan the already blazing flames of Twitter hype too much - although these numbers don't lie, they only tell part of the story.  There are essential elements of communication effectiveness that are just as important as the medium you use. For example, the quality of the content (see Lesson #19, Content is still king in the Twitter-sphere) matters more than whether you use Twitter or paper.   And its important to understand all social networking tools and use them properly, and in an integrated way (see Lesson #25, Learn how to Twublish).

Clay Shirky, in his excellent TED talk, makes the case that we are living through the largest increase in expressive capability in human history because it allows groups to communicate natively.  From a business perspective, it simply cannot be dismissed as a fad.

2. Do use Twitter to transmit values, standards, and ethics.  Peter Drucker said an essential job of the CEO is “to set the values, the standards, and the ethics of an organization.” Social networking helps me communicate my values, standards, and ethics to hundreds of people at once.  It’s like having hundreds of water cooler chats in real-time, from wherever I am - while I’m visiting a customer, speaking at a conference, or meeting with my board. 

That is, when I tweet (or blog) about the exciting things I see on the road, you can feel StreamBase’s enthusiasm for our customers.  When our CTO tweets about innovation, other technologists get a unique insider's view into the latest developments.  When I tweet about running the Boston Marathon (video on www.notallceosarejerks.com) for cancer research, I reveal my ethics. When I tweet about competitive announcements, prospects learn why our company is unique.

3. Do use Twitter for press relations – Old school PR is dead.  Journalists track CEOs for interesting stories – social networking helps you feed them.  Traditional PR agencies still have role, and most need to evolve.  I recently evaluated almost 20 press agencies, with a dominant eye to how well they understood social networking.  Only two had social networking deeply engrained in their culture; most were still stuck in the old days of point-to-point, relationship based PR.  PR firms must evolve, or die.

4. The mundane can be funny and constructive. Great tweets can be about mundane moments.  Zappos CEO Tony Hsieh (@zappos) is a master at it - here are three bad tweets I recently received, and 3 tweets from Tony about the same topics:

• BAD TWEET:  “Having coffee”
• GOOD TWEET:  “Had coffee w/ my mom, told her I gave a presentation @ Tony Robbins conference. She was really excited. In her words: "Wow! Robin Williams!"

• BAD TWEET: “Going into a meeting now” 
• GOOD TWEET: “Visited meetup.com office, met w/ CEO. Thankfully the meetup at meetup did not cause a rift in the space-time continuum”  

• BAD TWEET: “My allergies are killing me!”
• GOOD TWEET: “Taking allergy pills is like having Snow White multiple personality disorder. You go from Sneezy/Grumpy to Sleepy/Dopey/Happy.”

6. Do use Twitter for employee communication – Social networking is like a virtual water cooler, creating a steady, informal stream of communication about market events, customer stories, and the day-to-day developments in the company.

7. Do use Twitter to augment investor and analyst relations – Industry and financial analysts like getting a view direct from the CEO; they're busy and track a lot of companies, so the unobtrusive, expedient channel through social networking lets them track you without hours of briefings.

8. Do use Twitter for peer CEO learning – I follow other CEOs to see their observations - the good ones help shine a light on my own day.  My favorite sources are exectweets, searching for CEO at wefollow.com, or search for CEO at twellow.com.

9. Do use Twitter to promote rising stars – We recently promoted one of our founders, Richard Tibbetts (@tibbetts) to CTO.  He’s an amazing technologist, and getting him out and speaking is great for us, and great for the industry. Matt Fowles, one of our top engineers, has written some great technical blog entries. Twitter helps raise their visibility, and educate the market.

10. Define your Twitter goals carefully.   Signing up for Twitter is easy.  Writing tweets is easy.  But figuring out your goals for tweeting is hard. I already blog, write magazine articles, and speak at conferences, so it took me a while to figure out my list of goals for Twitter.  My main goal, of course, was lesson #2 - to transmit StreamBase's values, standards, and ethics.

11. Twitter can make you a better communicator - Strunk and White's rule: “omit meaningless words” is a great one, and Twitter forces you to omit meaningless words by its very structure - the discipline imposed by 140 characters is a good discipline.

12. Don’t use Twitter for traditional marketing. Some companies seem to think Twitter’s just another vehicle for traditional marketing.  H&R Block’s Stacey Gratz, marketing manager, explained: "[On Twitter,] we soon realized that we needed to listen and share, rather than pushing out marketing messages." Read about the H&R Block Twitter case study here.

13. Do use Twitter for thought leadership.  My company, StreamBase, is a leading visionary in an enterprise software market called event processing.  The market is still less than $100M, but it's growing, and controversial.  In a new area such as ours, education is important, and social networking provides another vehicle to educate prospects and customers. 

14. Do use Twitter to connect when you’re in the field – CEOs are always on the road, and sometimes it’s hard to connect.  Twitter is a great way to let prospects, customers, and analysts where you’ll be (if you want them to know).

15. Do use Twitter to transmit your brand's conscience.  Habit #1 of Bruce Philip's 5 Habits of successful executives:  “They are their brand’s conscience.”  Michael Hyatt (@MichaelHyatt) of Thomas Nelson Publishers does this well.  Richard Branson (@sirdickbranson) of Virgin does this well.  CME Group (@CMEGroup) has burst on the scene with over 420,000 followers about futures and options.  These tools allow direct communication and a great vehicle to allow customers and prospects to understand the essence of your brand and culture. 

16. Do use Twitter for product feedback and announcements.  Ford bet the marketing farm on social media for their Fiesta launch, and Twitter and blogging are the perfect vehicles to go more deeply on product features, and create general awareness. 

17. Do use Twitter to respond to rumors quickly.  Recently, we had to fire a well-known employee, and we got calls from false rumors that we were downsizing.  Twitter helped me do that in 2 minutes with 140 characters.

18. Don’t use Twitter to sell.  I’ve seen a lot of overt selling “my new book is available for $19.99! – buy it now!”  Yuk.

19. Don’t inadvertently give too much competitive information away.  I couldn’t tweet for a whole week because if the competition knew I was in Chicago, I would have jeopardized a sales situation.

20. Content is still king in the Twitter-sphere.  Even though Twitter messages are only 140 characters, the quality of those 140 characters is still king.   The link between content quality and business value (lesson #1) is direct.

21. Turn your communications approach upside down. If you haven't already noticed, there are plenty of proclamations that the death of the corporate web site is upon us, that traditional journalism and PR is dead, and that product management can be more effective using Twitter.  Each of these statements is at least partially true.  

When the Cluetrain Manifesto (about the book, short form, the entire ebook,) boldly and simply stated that a “powerful global conversation had begun,” they were right – and the time has come for every business to rethink their communication strategy.

22. Don’t compete with SpongeBob.  My 8-year-old son, Jack, asked what I was doing one day.  I told him I was tweeting, and told him what Twitter is.  He asked:  “How many people follow you?” 

I said:  “about 500.” 

Jack asked: “How many people follow Sponge Bob?”

I looked it up.  “5,000.” 

Jack said:  “Dad - if you tell more jokes, maybe you’ll be more popular.”

I don’t want to compete with SpongeBob.  I’m don’t care how many followers I have.  Neither should you!

23. Don’t tweet too much; don’t tweet too little. You get a lot of benefits from Twitter for just 10 minutes of effort a day.  On the other hand, it’s important not to tweet too much - I recently saw someone who had tweeted 26,000 times – yikes!  My rule of thumb is to tweet between 0 to 3 times a day.  That's about as many truly insightful things as I encounter in 24 hours.

24. Twitter helps keep me informed in real-time, personally, with less effort, and dynamically.  Today one of our engineers asked:  “doesn’t Twitter just give you too much information?”  My response was that it’s the opposite – it helps me cull the information deluge down to the stream I care about.  Currently, I listen to:

• The New York Times (@nytimes)
• NPR (@nprnews)
• My favorite industry analysts (@bmichelson)
• My competitors (you know who you are)
• My customers (CME Group, and many more)
• My cousin (@marissapage) who’s really, really funny. 
• Other CEOs (many to mention, check out my followers)

I’m informed in real-time, as news happens, and my list of information streams is very personal.

It’s less effort with Twitter to add new sources, scan existing ones, and read from my iPhone than traditional mediums, including personalized websites.  And the 140 character Twitter constraint, makes people “put it on a bumper sticker,” so I can scan the headlines.  I’ve never been more informed in such little time.

Finally, the whole thing is dynamic, so I can constantly change what I’m tuned into.

25. Don’t start using Twitter unless you love to communicate, and are willing to stay committed.  It’s hard to get in the habit of tweeting, and CEOs are busy!  Moreover, tweeting well is an art, and takes some thinking.  You have to want to communicate, and be willing to stay committed to creating ideas on your own; you can’t outsource the job, either.

26. Learn how to Twublish.  I used to write byline articles for industry journals. Traditional publishing is static, formal, unidirectional, and controlled by an external editor.  Then I started to blog, connect via LinkedIn, share photos with FlickR, and message with Twitter.  Taken together - and they must be used together - these tools form a new way of publishing - I think of it as "Twublishing."  

To publish with Twitter, or Twublish, involves writing to our blog.  Then I tweet about points in the article on Twitter.  Comments in the blog and direct messages on Twitter help me get feedback on the quality of my thinking.  Often, this leads me to new places, and adjacent topics.  Or I more fully develop the original idea, and more deeply hyperlink the original to new sources I discover.  The feedback deepens the original thought and creates articles that evolve into portals for other information on the same topic.  This post, for example, has links to 30 other information sources about the use of Twitter from a CEO's perspective.  

Twublishing is dynamic, informal, bi-directional, and unconstrained by an editor.  I can mix in video and images.  It's hyperlinked so my thoughts can become portals to other people's ideas that were the building blocks of my own.  And it's temporal - that is, as information changes over time and others respond, I can update or augment the original to refer to the new information.  

Other than the whopping $250 I used to occasionally get for a magazine article, publishing with social networking tools is better in every way.  

27. I’m more aware of why I love my job because of Twitter.  As I was returning from a sales trip, I tweeted:  “Anyone who thinks the capital markets are collapsing should go out and spend a day with the traders – lots of innovation going on out here!”  The act of paying attention to the ironic, funny, shocking, and curious things that happen from day to day makes me more aware of why I love my job.

 

Innovation: Why Size Doesn't Matter

The American spirit is an entrepreneurial spirit. So it’s no surprise that the recent economic upheaval has spurred a rash of discussion about entrepreneurship and innovation. Today, for example, the New York Times reported that we are in the midst of a shift where innovation is moving to big companies, and away from small companies. But the article missed the heart of the entrepreneurial matter - it’s approach, not size, that forms fertile ground for innovation.

What is the approach of a successful, innovative, big company? Although the Times reported based on “new” thinking, I think the landmark old thinking is still best. In 1985, Gifford Pinchot published Intrapreneuring, which describes the elements of innovation success inside big companies (and thus coined the term intrapreneuring, which was added to the American Heritage Dictionary in 1987). He studied the issue because he found that big companies tend to be terrible at innovation: 

You might think that larger organizations would be stodgy in thinking up new ideas, but, because of a wealth of management talent, would be very good at executing them. It turns out, however, that just the reverse is true:  our large organizations are producing large numbers of good ideas but generally are unable to implement them." 

Which begs the question - what are the keys to innovation within big companies? To begin with, they must form small, autonomous, focused groups. These people, and groups, act like small companies, and Pinchot described the outlook of these groups with his "Intrapreneur's 10 Commandments": 

1. Come to work each day willing to be fired.
   
2. Circumvent any orders aimed at stopping your mission.
   
3. Do any job needed to make your project work, regardless of your job description.
   
4. Find people to help you.
   
5. Follow your intuition about the people you choose, and work only with the best.
   
6. Work underground as long as you can – publicity triggers the corporate immune mechanism.
   
7. Never bet on a race unless you are running in it.
   
8. Remember it is easier to ask for forgiveness than for permission.
   
9. Be true to your goal, but be realistic about the ways to achieve them.
   
10. Honor your sponsors.   
    

These commandments will seem simple and obvious to intrapreneurs, and anathema to people that tend to thrive in typical big companies. Worse, most of the big company leaders I've worked with don't recognize these critical innovation success factors. Indeed, most companies actively discourage this behavior.

So if you are a big company that wants to innovate, what kind of people should you empower? If you want to be an Intrapreneur, do you have what it takes? Pinchot tackles that one, too - by summarizing the attributes of traditional managers versus Intrapreneurs:

	Traditional Manager	Intrapreneur
Primary Motives	Wants promotion and other traditional corporate rewards.  Power-motivated.	Wants freedom and access to corporate resources.  Goal-oriented and self-motivated, but also responds to corporate rewards and recognition.
Decisions	Agrees with those in power.  Delays decision until gets a feel of what bosses want.	Adept at getting others to agree to private vision.
Time Orientation	Responds to quotas and budgets, weekly, monthly, quarterly, annual planning horizons, the next promotion or transfer	End goals of 3-15 years, depending on type of venture.  Urgency to meet self-imposed and corporate timetables.
Action	Delegates action.  Supervising and reporting take most of energy.	Has a bias for action. May know how to delegate, but when necessary does what needs to be done.
Courage and Destiny	Sees others in charge of his or her destiny.  Can be forceful and ambitious, but may be fearful of others’ ability to do him or her in.	Self-confident and courageous.  Many Intrapreneurs are cynical about the system, but optimistic about their ability to outwit it.
Attention	Primarily on events inside corporation	Both inside and outside.  Sells insiders on need of venture and marketplace, but also focuses on customers.
Risk	Careful.	Likes moderate risk.  Generally not afraid of being fired so sees little personal risk.
Market Research	Has market studies done to discover needs and guide product conceptualization.	Does own market research and intuitive market evaluation like the entrepreneur.

The Times article misses the point because the key to innovation lies not in the size of the company, but in the acceptance and support of the entrepreneurial condition. And since our economic future depends on getting innovation right, both in big companies and small, we're lucky that we already know how to innovate. We just need more companies and leaders that have the guts to get out there and do it.

When Microsoft Enters Your Market (CEP): Cause for Worry or Celebration?

(Mark Palmer is president and CEO of StreamBase Systems)

This week, Microsoft announced they will enter the technology space inhabited by my company, StreamBase.  Naturally, I got a flurry of questions, ranging from: "Is Microsoft's entrance going to crush your business?" to "Could Microsoft entering your market be validating?"

The question behind these questions is:  "How can a company with less than 100 people compete with Microsoft?"

First let's start with some background:  what is complex event processing, or CEP?  CEP is an enterprise software technology - big companies buy it to do big things.  CEP is kind of like a database, but turned on its head for real-time decision-making.  For example, Orbitz uses CEP to monitor their business in real time - when there's a problem, Orbitz knows about it in milliseconds, which means they can do something about it before its too late.  And they can change the logic for detecting and responding to problems on the fly. Databases, in contrast to CEP, are designed to help a company look at what already happened, not what's happening now.  Clearly the database is an essential tool that can inform future decisions.  But knowing what's happening now helps identify and therefore seize business opportunities that exist for just a short time, like fixing your problem on Orbitz before you leave and go to Travelocity.  CEP is increasingly becoming a critical tool for the enterprise.

The most extreme, and common, use of CEP is in the capital markets - firms like Deutsche Bank, JP Morgan, and RBC use CEP to automate their trading decisions.  The financial markets produce millions of events every second, and opportunities to make money last, in some cases, for minutes, or, in some cases, for milliseconds. The systems that identify and exploit those opportunities are under constant development. In trading, rapid development and instant action aren't luxuries;  they directly impact revenue and competitive differentiation.

So, can Microsoft dominate CEP?  In his groundbreaking book Intrapreneuring, Gifford Pinchot researched the success and failures of innovative efforts in large U.S. corporations.  At a macro level, he found that big companies, statistically, are terrible at innovation: 

You might think that larger organizations would be stodgy in thinking up new ideas, but, because of a wealth of management talent, would be very good at executing them.  It turns out, however, that just the reverse is true:  our large organizations are producing large numbers of good ideas but generally are unable to implement them." 

This, of course, is why the venture capital industry is alive and well - innovation naturally comes from small firms.  However, Pinchot went on to describe that when big companies act like little companies, they can innovate.  But they must create small groups, accept and expect failure, let these groups be autonomous, hire different kinds of people, compensate them differently, manage them differently, and get close to customers.  All these attributes fly in the face of traditional big company culture.  

But Microsoft has pulled off innovation before - are they doing it here according to Pinchot's findings? It's a CEOs job to watch for competitive threats, and I see no evidence to suggest Microsoft is approaching CEP in a way that portends success.  Nimble moves by Microsoft could include:  recruit top visionary CEP engineers (they haven't); create a small, independent group (they didn't - the product is coming from the SQL server group, which has hundreds of engineers); get out and talk to customers directly and co-develop with visionary partners (we see no evidence of this); recruit and empower entrepreneurs and empower them to break down corporate structures (Microsoft didn't do this either).

So the physics that would make an entrepreneur worry just aren't there at Microsoft's CEP effort.

Which leads me to my current answer: the fact that Microsoft is trying to get into CEP is a big, important endorsement for CEP.  And since we think we have the best CEP product available (as many others who have evaluated us have found), we invite the competition.  If Microsoft's endorsement makes corporations more aware of CEP, it's better for StreamBase.  

Welcome, Microsoft!

Eight Lessons for Entrepreneurs – A Case Study Based on PhaseCapital

Recent economic times have gotten my usually optimistic dad down.  He worries that the economy is mortally wounded.  Everywhere you turn there’s bad news.  Don’t worry, says I!  The heart of the economy is entrepreneurship, and as the CEO of an entrepreneurial software company that has entrepreneurial customers, I get to see innovation every day.

Case in point:  recently, a customers of ours (PhaseCapital, a Boston-based quantitative investment advisor firm), was interviewed by Waters magazine about their business.  PhaseCapital’s CEO, Eric Pritchett described their approach to the arcane world of electronic trading.  In his story I saw reminders of good lessons of innovation and entrepreneurship in an industry many outsiders view as troubled and failing.

#1 - Troubled economic times make new business models a source of fresh innovation.   In the capital markets, the old school model of trading is based on prime brokers who provide services to the buy side (hedge funds).  The service those brokers provided, in Eric’s words, were “flying [buy-side trading clients] around to conferences to meet with CEOs and give them access to analyst reports [for stock trading guidance].”  New school buy-side firms are beginning to disintermediate brokers from trading decisions.  PhaseCapital is a case in point - they have their own ideas on what to buy and sell.  As a result, new school execution brokers like Lime Brokerage “use their commission flow to pay for useful technology resources that [we] would otherwise have to build.”  The changing business relationship between the client and execution broker provides business-model innovation: lower-cost, higher value service to buy-side firms who focus on high-frequency trading as a differentiator.

#2 - Reliance on black-box technology can limit innovation.   If everyone uses the same tools, where do you get your advantage?   Pre-built applications, or black boxes are good for business functions where innovation is limited (such as accounting, HR, etc.).  But technology that empowers a firm’s competitive differentiation can benefit from a white box approach to technology.   A white box is like using Lego’s to build a unique system.  Mr. Pritchett cites StreamBase as an example of getting the white box approach right by providing the basic building blocks they needed to build their trading system quickly, but don’t introduce "too much abstraction” from their differentiated ideas.

#3 - Open source that’s ready for prime time can disrupt.  In the capital markets, technology that drives down the cost of trading can be disruptive.  PhaseCapital uses QuickFixJ the open source implementation FIX (a standard for trading information exchange between banks and clients) for its trading operations.    But being for standards is like being for world peace -- getting the right standards at the right time (when they’re ready) is where the rubber meets the road.  Eclipse is ready.  Open source FIX is ready. Open source technologies, picked when they’re ripe, can reduce cost and maintain quality, and be disruptive against firms with older, more expensive infrastructure.

#4 - Decision-making speed can be disruptive.  The need for speed is insatiable in the capital markets.   A long time ago, most opportunities lasted for minutes.  Now, with automated trading, some last for seconds;  some last for milliseconds; some last for microseconds.  CEP helps firms make decisions while they still matter.  Mr. Pritchett described it in technical terms:  [Lime and StreamBase] deliver “a strong deployment platform for low-latency trading operations.”  Traditional database technology is designed for decisions that take hours, days, or weeks.  CEP is designed for decisions that that take seconds, milliseconds, or microseconds.  The faster a business moves, the faster it can seize business opportunity.

#5 - Describe your focus in 23 words.  It’s easy to say that focus is good.  It’s easy to note that lasers (focused light) are more powerful than light bulbs (defused light).  Yet the hardest job I have as a CEO of an entrepreneurial enterprise is to focus.  It’s hard to decide what not to do.  Eric’s description of PhaseCapital’s organizational uniqueness was razor-sharp, and a great example of organizational self-awareness:  “[Our strengths are] signaling, opportunity identification, position sizing, risk process and in realizing the alpha opportunities that we are trying to harvest through high-frequency trading.” Can you be that clearly focused about your strengths in 23 words?

#6 - Bet boldly on promising ideas.  PhaseCapital bet boldly on automation.  And therefore they bet on an entirely automated infrastructure, and new technologies that enable automated trading as opposed to traditional click trading, where traders push the button to buy and sell.  StreamBase and Lime Brokerage were their bets for technology, and they’re building a complete system, including algorithmic trading, smart order routing, market data management, for both equities and foreign exchange.   

#7 - Quickly land -– then expand.  PhaseCapital bet boldly, but they didn’t try to build Rome in a day.  Innovators know that you must set and stick to a vision, but that you have to constantly learn along the way.   Innovation comes interactively -- instead of taking four years to build it all, they set a quick target: trade a subset of the S&P500.  They landed that goal.  Then expanded to the next step:  the entire S&P500.  Then to foreign exchange.  And so on.

#8 - Entrepreneurs need entrepreneurial partners to succeed.  Innovation is a mindset, and entrepreneurs are more successful when they partner with other firms that get it. Our customers often tell me they buy our software because our company thinks like they do.  It’s the little things that make it work -– engineering-to-engineering communication, direct CEO involvement and sponsorship, creative business deals and contracts.  PhaseCapital worked closely with us because that’s the way we like to work with customers.  That shared mindset makes for good partners.

Even though the pulse of the U.S. economy might be a bit shallow, PhaseCapital proves that the heartbeat of innovation and entrepreneurship is strong.