The StreamBase Event Processing Blog

What is Event Processing (CEP)?

The World Economic Forum at Davos has chosen event processing (or CEP) as one of the 26 most innovative technologies to impact the future of business and society. Despite the growing importance of event processing, it's still new to many. So here's a list of the best resources on CEP, organized by perspective:

CEP 101 for business people: Watch this 1 minute intro video on CEP, event processing, and StreamBase.

CEP on Wikipedia: Visit the Wikipedia CEP page.

CEP for IT architects: Brenda Michelson's post on event driven architecture is from 2006, but it's still the best introduction I've seen.

CEP and its use in capital markets: Visit the high frequency trading section of the StreamBase blog.

CEP in depth by Gartner analyst Roy Schulte: Read Event Processing: Designing IT Systems for Agile Companies, by Roy and Mani Chandy.

CEP and social media: Read Trading on Twitter: Opportunity, Danger, or Folly?

CEP case studies:  Real-life stories from the CME Group (Chicago Mercantile Exchange), BlueCrest Capital Management, and PhaseCapital.

CEP on Twitter:  Follow the CEP industry Twitter list at @mrkwpalmer/cep.

Evaluating CEP products. Read evaluation stories from CMC Markets or CME Group, Kairos and PhaseCapital. Read my biased analysis here, or download StreamBase and try it for yourself. 

(Mark Palmer is the CEO of StreamBase Systems.  Follow him at twitter.com/mrkwpalmer)

Prediction #9: Event processing will Yield a Great New Software Powerhouse

The ninth and final prediction for event processing is that the processing of moving information will yield a great independent software company. 

While it's nice that Time Magazine declared event processing is a technology that will "change your life," and the World Economic Forum at Davos named event processing as one of the 26 top innovations in the world, the previous 8 predictions explain why event processing will prove fertile ground for a great software sector:

• The business case - to enable the rapid construction of real-time applications  - has already impacted two of our most critical markets: the capital markets and the intelligence agencies
• The solution requires a fundamentally new computing physics where traditional static computing technologies don't fit
• The technology is proven, with major companies already using CEP enterprise-wide
• The revolutionary simplicity of event processing is, as Clayton Christensen suggests, "always disruptive," and therefore facilitates disruptive innovation
• This innovation has become even more critical in an era where competitive advantage is becoming more quantitative
• The market for data management is huge and the big software stack vendors continue to miss the mark with their technology offerings
• The possibility of an entirely new software stack is upon us - open source is a big part of it, and innovation in the field should continue for the next 10 years

Put all of the conditions together, and it's ripe territory for the emergence of a new software powerhouse that disrupts the old guard. 

Prediction #8: Open Source CEP Won't Impact the Market, But Open Source Will

Although event processing platforms is robust and proven, there is still constant innovation in the space. Although the efforts of open source projects Esper and Cayuga are admirable they both miss the importance of simplicity through graphical event languages and powerful development tools. They also aren't proven for mission critical usage, or optimized for the high performance, low latency demands.

However, open source is helping spur the growth of event processing in important ways:

1. The rise of Eclipse provides instant access to a vast community of development tool plug-ins for event processing development. 
2. Connectivity is commoditizing with developments in QuickFix, AMQP implementations like RabbitMQ, and DDS implementations like OpenSplice. 
3. Events-in-the-cloud is becoming easier with open source cloud computing and web based messaging protocols.

The rise of open source for development, connectivity, and cloud computing is also why software stacks will continue to miss the mark with their proprietary stacks of software.

So as the new event-based software stack begins to emerge, event processing providers that embrace this chance will continue to grow as the enterprise platform of choice.

Prediction #7: Stream-Based Platforms will Continue to Lead a Siege Against the $15B Data-Base Market

It’s flattering that Entrepreneur / VentureBeat chose StreamBase as one of the technologies that will continue to lead a siege against the $15 billlion database market, but the truth is that there are multiple technologies in the real-time computing stack required to make this happen, including high speed connectivity, column-oriented data stores and CEP.

Event-based platforms are about real-time computing; databases, business intelligence, and data warehouses are about looking at what happened in the past. By combining the two to operate on streaming data, you create an optimal platform for the real-time enterprise.

So watch for the seige on this market to continue as data management techniques help corporations find needles before they go into the haystack of corporate databases.

Prediction #6: A New Event-Based Software Stack will Emerge

Just because the traditional stack vendors aren't players in event processing, that doesn't mean there isn't a complete software solution required for event processing. The new event processing stack has elements that the big stack vendors don't want you to know about (because they can't sell them to you).

The elements of any software infrastructure are 1) Connectivity, 2) Business Logic, 3) Presentation Layer (GUI), and 4) Database.  

For event-based systems, connectivity is the most important element. It includes legacy middleware like TIBCO, but more importantly includes new innovative messaging like high-speed solutions Solace and Tervela, Google Chrome (namely websockets), and standards like AMQP and DDS. AMQP is very attractive as a standards based messaging wire protocol. Solace, Tervela, RedHat, Apache (QPID), RabbitMQ and various other messaging providers/vendors are working hard on AMQP interoperability. CEP is great at analytics. AMQP is great at distribution. CEP over AMQP simplifies integration. Industry specific connectivity is also critical. For example, in the capital markets, connectivity to FIX, Reuters, Wombat, FX venues, and so on - this connectivity helps speed integration and bring CEP online.

Business logic is the domain of event processing like StreamBase. 

At the database level, relational databases still play a role to store history, but just as important, if not more so, are column-oriented databases like Vertica. 

Finally, some of the most innovative presentation layer technology for time-series data is coming from the Eclipse open source community. 

What’s the pattern in all this stack innovation?  None of it comes from the big companies - it’s coming from small innovative startups that are breaking down the physics of traditional computing as we form a new stack for the next 50 years of innovation.

...to be continued....

Prediction #5: Software Stacks will Continue to Miss the Mark

Rupert Murdoch said: “The world is changing fast. Big will not beat small anymore. It will be the fast beating the slow.” This is certainly true in the technology space - big companies can innovate, if they stay small.

In the world of enterprise software, Oracle, IBM, Microsoft, and TIBCO are big, and they don't act small. In fact, over the last decade, they have acquired their way to an ever more complex portfolio of products - they act even bigger than they actually are. Moreover, these big product stacks are designed for traditional computing - computing that looks at what happened in the past, not what's happening in real-time. 

Recently, these big vendors became aware of event processing. What’s the natural thing to do? Buy some technology and slap it on the stack! Even Progress Software, who acquired CEP pioneer Apama, lost its focus to chase the big stack companies (for example, this week they acquired a small business process management technology (Savvion) as the centerpiece of yet-another-stack-strategy).

These stacks are to event processing as the record industry is to iTunes - a different industry for a different era. Customers don't want big stacks; they want tools that solve their problems.

The truth lies in the results, as measured by customers and the specific business value they achieve. In the past year, the endorsements have gone to small, focused event processing companies, not the big stack vendors. Consider the last 12 months of customer endorsements:

In the past year, TIBCO (quoting the TIBCO CEO) claims to have won the following customers:  A “western states utility”, a “major Asian bank”, and a “major Indian mobile company.”

By contrast, in the past year, StreamBase customers aren't anonymous; they are serious corporations. They are enterprise-wide endorsements. They manage quadrillions of dollars of business every year. And they furnish technical details. The partial list of public endorsements include: CME Group, RBC, BNY ConvergEx, City Index, CMC Markets (who switched from a stack vendor (Progress Apama)), Curex, PhaseCapital, BlueCrest and Kairos. And these are public endorsements in one of the most secretive industries there is: the capital markets.

Credible customer stories don't lie, and they show that in CEP, small is beating big. Watch for this trend to continue as the nimble, pure-play players continue their astounding growth and Hollywood endorsements, and run circles around their less nimble competitors pushing big stacks of software designed for yesterday's applications.

...to be continued...

Prediction #4: Quantitative Thinking will Trump Traditional Thinking

Earlier this month, the Bureau of Labor Statistics forecasted the 2 of the top 3 biggest growth areas of labor in the U.S. to be tech-driven: a whopping 1.5 million jobs over the next 10 years (coverage from the Huffington Post). Yet the vast majority of Fortune 100 CEOs are still technology slackers (as measured by participation in social media. Hardly a perfect measure, but it's scary).

Modern tech-driven innovation will be led by a new kind of business leader who has a keen sense of technology, and barriers between C-Level executives and IT must fall. Two industries already illustrate the trend that will prevail in all industries over the next 50 years: capital markets and baseball.

Yup, that's right, let's start with baseball. In the past 20 years, the organizational principles of baseball have changed from gut feel and folklore to robust statistical analysis of every single element of the game (then applying gut feel). Bill James began his statistical analysis of what made teams win in 1977, and it was brought to the popular consciousness by the best seller Moneyball in 2003. The Boston Red Sox and their fabled 86-year losing streak was not just broken after they adopted a quantitative approach to baseball, they became the most successful team of the decade. And its no accident that the main owner of the Red Sox is financial guru John Henry, which leads us to the other exemplar for future business: financial services.

Financial services is already 20 years into this transition. In the past 10 years, mathematical, real-time trading has risen from less than 5% to over 70% of trading activity on U.S. stocks alone, and the rest of the industry is becoming electronic. 20 years ago, in my first job out of college, I worked on a trading floor. The traders were MBAs from Harvard and Stanford - brilliant, personable, and competitive. The ultimate knowledge workers. Now, the top traders are mathematicians. They are competitive but in a more cerebral way. 20 years ago, trading floors were loud and boisterous - traders shouted down phones, and at each other; now, they are intense mission control centers. Shouting has been replaced by the clicking of keys on keyboards. Analytical acumen is now just as important as intense physical passion.

Over the next 50 years, other industries will evolve in much the same way: the winners of the modern economy will adopt quantitative strategies as much as traditional business strategies.

Event processing facilitates quantitative thinking with its modern, visual development tools. These tools help facilitate analytical thinking, and also lower the barrier between business concept and implementation (see the related prediction that cognitive physics will be as important as computing physics)

For example, a trading strategist can describe a strategy like: “If the price of IBM goes up more than 2% in any 10 minute time window, but HP doesn’t, then buy HP, but only if the spread holds.” These words, this strategy, this business logic, can be expressed directly with visual tools by business analysts and IT. And the modern business leader will need to communicate fluently in this language - they won't have to code it, but they'll have to understand the math of their business deeply, and in detail.

By lowering the barrier to express quantitative business strategies, innovation cycles accelerate.

But with only 2 of the Fortune 100 CEOs on Twitter, the opportunity to out-think your competition is wide open. So the winners of tomorrow will be led by those who get comfortable with technology today.

...to be continued...

The 10 Commandments of Intrapreneuring

(Mark Palmer is the CEO of StreamBase Systems.  Follow him at twitter.com/mrkwpalmer. This post was originally released in 2009, but rewritten in January of 2010)

The American spirit is an entrepreneurial spirit. So it’s no surprise that the recent
economic upheaval has spurred a rash of discussion about entrepreneurship and innovation. Today, for example, the New York Times reported that we are in the midst of a shift where innovation is moving to big companies, and away from small companies. But the article missed the heart of the entrepreneurial matter - it’s approach, not size, that forms fertile ground for innovation.

Size of a company is a specious way to characterize innovation. But is there a better way? Gifford Pinchot's classic book Intrapreneuring analyzed the leading big innovative companies of his day (the book was published in 1985), and his findings have stood the test of time. 

To begin with, he found that big companies, as a group, aren't good at innovation:

You might think that larger organizations would be stodgy in thinking up new ideas, but, because of a wealth of management talent, would be very good at executing them. It turns out, however, that just the reverse is true:  our large organizations are producing large numbers of good ideas but generally are unable to implement them." 

Unfortunately, I think this finding still stands, 25 years later.

But he went on to describe how big companies can adopt the right spirit, hire the right people, and design the right incentives to innovate. A few central points are helpful to review.

To innovate, big companies must form small, autonomous, focused groups. These groups must be incentivized to act like small companies, by adopting the "Intrapreneur's 10 Commandments": 

1. Come to work each day willing to be fired
   
2. Circumvent any orders aimed at stopping your mission
   
3. Do any job needed to make your project work, regardless of your job description
   
4. Find people to help you
   
5. Follow your intuition about the people you choose, and work only with the best
   
6. Work underground as long as you can – publicity triggers the corporate immune mechanism
   
7. Never bet on a race unless you are running in it
   
8. Remember it is easier to ask for forgiveness than for permission
   
9. Be true to your goal, but be realistic about the ways to achieve them
   
10. Honor your sponsors
    

These commandments will seem obvious to entrepreneurs, but they are both rare and threatening in most big corporations.

So if you're the CEO of a big company and you're serious about innovation, what should you promote? First the CEO must check his or her own personality. Not all CEOs are alike. Some personality types are best suited for the "cash cow" part of a corporate lifecycle, and some are best suited for the early stages.  

What are the characteristics of people who can thrive in innovative firms? Pinchot provided a tool for this too, with his breakdown of 19 characteristics of intrapreneurs versus traditional managers. Here's an exerpt from that table, (or download the entire list here, published with permission from the author).

	Traditional Manager	Intrapreneur
Primary Motives	Wants promotion and other traditional corporate rewards.  Power-motivated.	Wants freedom and access to corporate resources.  Goal-oriented and self-motivated, but also responds to corporate rewards and recognition.
Decisions	Agrees with those in power.  Delays decision until gets a feel of what bosses want.	Adept at getting others to agree to private vision.
Time Orientation	Responds to quotas and budgets, weekly, monthly, quarterly, annual planning horizons, the next promotion or transfer	End goals of 3-15 years, depending on type of venture.  Urgency to meet self-imposed and corporate timetables.
Action	Delegates action.  Supervising and reporting take most of energy.	Has a bias for action. May know how to delegate, but when necessary does what needs to be done.
Courage and Destiny	Sees others in charge of his or her destiny.  Can be forceful and ambitious, but may be fearful of others’ ability to do him or her in.	Self-confident and courageous.  Many Intrapreneurs are cynical about the system, but optimistic about their ability to outwit it.
Attention	Primarily on events inside corporation	Both inside and outside.  Sells insiders on need of venture and marketplace, but also focuses on customers.
Risk	Careful.	Likes moderate risk.  Generally not afraid of being fired so sees little personal risk.
Market Research	Has market studies done to discover needs and guide product conceptualization.	Does own market research and intuitive market evaluation like the entrepreneur.sponsors

These are just a few key key elements a culture that's designed for innovation. So many more are critical, including compensation plans, equity structure, organizational structure, and so on. But I find that for people with the entrepreneurial mindset and some experience, those things become obvious.

Since our economic future depends on getting innovation right, both in big companies and small, we're lucky that we already know how to innovate. We just need more companies and leaders that have the guts to get out there and do it.

Prediction #3: Cognitive Physics Will Be as Important as Computing Physics

As Clayton Christensen suggests in The Innovator’s Dilemma, simplicity is always a disruptive innovation, Today this is more true than ever. The half-life of software is weeks and days, not years and decades. In computing’s first 50 years of technology, frictionless electronic distribution wasn’t the norm; now, with high-speed networks, cloud computing, and software as a service, customers expect new features immediately, and because switching costs are so much lower, they'll leave — or be acquired — more quickly than before.

If you want to do a job faster and better, you need the right tools. For knowledge-based work, the right tool aligns the "cognitive physics" — that is, good tool choice aligns the way of thinking about a problem with the way the solution is built. When cognitive physics matches implementation physics, barriers to implementation fall. Event processing tools can break down these barriers in an organization that has moving data. For example, PhaseCapital, a Boston-based hedge fund, built their entire trading system with event processing in four months - CEO Eric Pritchett estimates it would have taken them four years with traditional programming tools. But more importantly, PhaseCapital changes their system every day. They test and refine their existing ideas. Invent new ones. 

So the future of innovation with event processing will be governed by its ability to reduce the friction between the way you think about a problem and the way you solve a problem. For an event-based business, ideas turn into applications more quickly, improvements are made more quickly, and problems are fixed more quickly. And this kind of speed — the speed to find, implement, and get a new idea to market — is a key element of a truly "agile" company.

...to be continued Monday with prediction #4...

Prediction #2: The "Deluge of Real-Time Data" Will Not Drive Event Processing Adoption

Yes, it’s true that three seconds of stock market data can generate more data than Shakespeare did in his entire life’s work. Yes, the best event processing platforms can process that volume with latency measured in microseconds (a microsecond is 1/millionth of a second; it takes light 5.4 microseconds to travel a mile in a vacuum). Yes, many times, speed is the main selection criteria among CEP products (for example, trading specialist CMC Markets processed 3,000 events with one vendor, switched, and currently processes 17,000 events a second). Yes, StreamBase is known for its blinding speed and we make it faster with every release.

But will speed be the most dominant reason why event processing is adopted? No.

Still, headlines proclaim that the raison d’etre of event processing is to "tame the data deluge," but these stories miss the point. No, most often, event processing will be adopted for different reasons.

....to be continued with prediction #3....