The StreamBase Event Processing Blog

Twitter and Trading: A Ripple in the Big Data Wave

In 2009, StreamBase began talking publicly about using CEP with Twitter. The topic sparked a flurry of mainstream media coverage about Complex Event Processing (CEP) on Wall Street on outlets such as CNBC, the Wall Street Journal, and the Financial Times. Even Popular Mechanics named CEP as one of 10 Tech Concepts to watch, along with Fracking and Flywheel Energy Storage! (watch this short video for an introduction to CEP)

Last week, USA Today published an article about CEP and trading on the front page of its Money section: Wall Street traders mine tweets to gain competitive edge. The report covered developments such as: 

• The opening of Derwent Capital, a firm that specializes in news-driven trading strategies, including Twitter
• A report from Adam Honore from AITE group whose two recent reports (on trading strategies and on unstructured data) revealed that interest in unstructured data has jumped from 2% in 2008 to 35% in 2011
• Rich Brown, who talked about how the Thomson Reuters news sentiment engine is more effective than the "quirky and abbreviated language of Twitter"
• Johan Bollen, a professor of informatics at Indiana University, whose research claims an 87% accuracy rate in using Twitter mood measurements to predict Dow stock prices three to four days later
• MarketPsych, a firm that used to run sentiment-based strategies, switched from being a hedge fund to a software provider

The article shows that Twitter continues to be a controversial topic: some tweets are clearly important - for example, the CME Group tweets news about the commodities market and has over 780,000 followers. On the other hand, computer-measured sentiment is just that - one more barometer of what the masses are thinking. Good investment decisions are powered by experience and knowledge, and sentiment is one input of many.

The noise over use of Twitter on Wall Street drowns out the real story, which is why I was quoted in the article that Twitter is "a ripple, not a wave" on Wall Street (and Main Street as well).

The big wave is big data, and Twitter generates just a few buckets of sand in world of big data today. 

In "New Rules for Big Data," the Economist reported that Google processes a petabyte of data an hour. Here's a way to put Twitter in context*: think of a bucket of sand. If every grain of sand in the bucket was 1 byte of data, then:

• The entire work of Shakespeare fills just one bucket of sand (about 5MB)
• A fast financial market data feed (OPRA) fills a beach of sand in 24 hours (about 5TB) 
• Google processes all the sand in the world every week (about 100PB)
• We generate 60% more sand every year

In these terms, all of Twitter generates only a sand castle of quality data a day. While some of that data is very valuable, such as tweets from CME Group (@CMEGroup), most of it simply tells us about mass sentiment.

On the other hand, the debate about Twitter does reveal the opportunity and challenge of big data. The Economist used a quote from Oscar Wilde in 1894 to put that challenge in context - Wilde said: “It is a very sad thing that nowadays there is so little useless information."

So the big ruckus about Twitter on Wall Street reveals the real big story - today's big data challenge.

* Thanks to Rob Doherty at Cloudswitch for sharing Chad Sakac's (EMC) buckets of sand concept, the Economist New Rules for Big Data for sizing data from Google, and @RJMetrics on big data (TED Philly talk) for conceptual description of big data.

On StreamBase 7, Its Breakthrough Usability, and Performance for CEP

Today we announced SteamBase 7, the next major version of our market-leading Complex Event Processing (CEP) software. In press and analyst pre-briefings last week, we discussed a lot of cool new features. Here’s a quick tour.

A HUGE STEP FORWARD IN CEP USABILITY

Chief among the innovations of StreamBase 7 is the revamped StreamBase Studio. The interface innovations of the iPhone, iPad, and modern operating systems has made great interface design expected. StreamBase has always been the best and easiest to use CEP platform. But with StreamBase 7, our engineering team has taken a huge leap forward in usability.

Drawn from the cutting edge consumer user experience in platforms like Apple iOS and Windows 7, Studio 7 dramatically increases the speed and quality of application innovation. 

To achieve this usability, StreamBase 7 has new code organizational features, integrated annotation features for English descriptions of functions, gorgeous diagram rendering techniques, and facilities to isolate and encapsulate the necessary complexity of a sophisticated application. Literally, StreamBase users can now read StreamBase application code at a glance, and organize it effectively so that the Business and IT can collaborate in a way never possible before with any CEP product. These features that make this all happen include:

1. A beautifully redesigned user experience, including the use of transparency (similar to Windows 7 Aero), overlays to communicate information at a glance (e.g., iOS number-in-red-circle), gradients rather than solid color to ease readability (think in terms of Windows 7 and iOS) and, and complete color customization (a la Windows 7 themes.) 
2. Support for custom modules (groups of application logic) and extension point styles.  Better control over modularity allows IT to manage large applications, and manage how and where business users or other groups can contribute code. For example, a risk domain expert can create risk checks in one module while IT focusses on connectivity, performance and integration in other modules.
3. Advanced graphical code layout, with new group layout features and code rendering controls
4. A new rendering look and feel to ensure code is easier to follow at a glance
5. New mouse and keyboard gestures to increase EventFlow coding productivity, such as on-canvas renaming of components
6. New hotkeys that give the ability to instantly create CEP language operators
7. Layout-alignment shortcuts

The result is that business users and IT can work in one environment, with one set of tools, with one set of code, dramatically improving time to market, quality and ability to evolve systems once they are deployed.

In recent years, StreamBase deployments have become bigger, more global, and more complicated. So StreamBase 7 improves usability throughout the entire software development lifecycle, including:

1. New code refactoring schemes. So developers can quickly identify changes in their application logic and adjust the rest of the application more quickly as modifications are made.
2. 400% faster application execution while debugging. So problems can be identified more quickly
3. The ability to attach Studio to a running StreamBase server. For better diagnostics in deployment.
4. A new XML editor. Allows operations to more effectively configure StreamBase deployments.
5. Dynamic extension points that allow users to configure StreamBase applications to be configured quickly at runtime. When a new algorithm or analytic is produced by a developer or business users, this code can be dynamically inserted into the StreamBase code, tested, and deployed.
6. Support for 64-bit workstations for very large StreamBase application development

PERFORMANCE IMPROVEMENTS IN FOUR DIMENSIONS

From the first release of StreamBase, the focus has been on performance. Previous SteamBase performance innovations include the StreamSQL compiler, the ability to visualize parallelism and scalability, data parallelism and dispatch, and our open architecture for highly available systems all combine to create the fastest, most scalable platform for ultra low latency applications.

Version 7 of StreamBase continues to break down CEP performance barriers in four dimensions: ultra low latency connectivity, predictability of latency, FIX performance, and with high-speed computing technologies.

1: MICROSECOND-LEVEL LATENCY

All CEP platforms have an architecture for absorbing and emitting events. Some run out of the main CEP engine process, which can add significant latency overhead to event processing. StreamBase already features an super-fast in-process adapter architecture, but goes further in StreamBase 7 with an optimized “zero-context-switch adapter architecture” which allows StreamBase applications to process messages in a single thread from the time it enters the engine, through StreamBase logic, to the time it leaves the engine.  This architecture allows messages to pass through the StreamBase engine in microseconds, and has been optimized to speed up CEP application performance.

2: PREDICTABILITY OF LATENCY

The predictability of performance has been improved for Streambase 7 applications, as users now have access to server configuration options that provide the user with fine-grain control of the tradeoffs between ultra-low-latency and throughput. And, overall platform improvements have yielded over 20% improvement in application performance across the board.

3: ULTRA LOW LATENCY FIX CONNECTIVITY

Many StreamBase customers in the capital markets is already known for its low latency connectivity architecture and version 7 comes with over 22 new connectivity options. Many of those connectivity options leverage FIX, and in StreamBase 7 FIX connectivity has been dramatically enhanced. In benchmark testing, the new StreamBase FIX engine has clocked in at 50-75 microseconds send time depending on message size, all while running on Linux and commodity hardware.

4: HIGH-SPEED COMPUTING AND GPUs FOR MANAGING BIG DATA AND NOSQL STORES

The state of the art of ultra-low-latency is rapidly expanding, and many StreamBase customers have been on the leading edge of applying new technologies to deal with big data in innovative ways. 

GPUs are being used to continuously produce metrics, factors, and parametrics, contribute them to memory data structures for ultra low latency streams of data to adapt in flight to compute accelerated algorithms. They are also used to easily integrate accelerated compute and messaging or to perform collaborate distributed event processing in a high level flow oriented language. 

To support this trend for high-speed computing with StreamBase, we have added several connectivity options in StreamBase 7 to make it easier for developers to connect StreamBase to their existing and next generation IT infrastructure, including hardware-based middleware for ultra-low latency operations, including enhanced low latency middleware connectivity such as Wombat, 29West, Solace, and Tervela.

REACTION TO STREAMBASE 7

Analyst Adam Honore’ said at the end of our briefing: “wow, you’ve been busy!” and he’s right - as customers build more sophisticated applications with StreamBase, with bigger demands from the business users, and bigger demands from IT, with the ever-increasing need for speed, StreamBase 7 extends our lead as the best-in-class CEP platform in the industry.

The benefit of bringing the business and IT together is faster time to market, with higher quality. With tight communication, the business and IT make mistakes faster, learn from them faster, and innovate faster. As Thomas Edison said: “I make more mistakes than anyone I know. And sooner or later, I patent them.” 

But my favorite anecdote comes from one of our customers - Chris Lees, vice president of SunGard’s global trading business. He put it this way: "The usability of StreamBase Studio helped us develop a new Valdi product, we used it to describe requirements collaboratively with IT and because we were working with the actual code, we made better architectural decisions early that made for a better first release.”

The Flash Crash & the Role of Financial Software Engineering

Even though the flash crash happened seven months ago, the debate rages on. And since CEP is a common piece of our modern financial computing fabric many firms have entered the debate with vested interest. Unfortunately, most vendors wrap flash crash observations with veiled sales pitches for CEP-based risk management, or disingenuous calls to "catch the rogues and scoundrels" with CEP-based surveillance systems. Although these applications are important, to suggest they are a magical elixir to ward off future breakdowns is misleading at best. 

But thoughtful discourse does continue, and the recent Securities and Exchange Commission report has reinvigorated debate. Richard Tibbetts, StreamBase CTO, weighed in with a piece called "What Does the Flash Crash Mean For Financial Software Engineering?" on Tabb Forum this week. Richard's view on the flash crash examines the role of financial software engineering and the efficacy of data processing, integrity, and reliability within the fabric of the infrastructure. The role of financial engineering is an essential, overlooked, and misunderstood piece of the puzzle. Richard's thoughts go like this: 

"In all the conversation about the crash, the real historic change is going unmentioned. In the very last paragraph of the Lessons Learned section of the report, the authors note: “another area of focus going forward should be on the integrity and reliability of market centers’ data processes, especially those that involve the publication of trades and quotes to the consolidated market data feeds.”

This is a key insight – late in coming – that will expand over time. Traditionally the SEC has focused on regulating the behavior of market participants, identifying deliberately malicious behavior or patterns of behavior. But now they are beginning to realize that data processing, integrity, and reliability can be just as important. The efficient functioning of markets is in just as much danger from software system failures as it is from malicious traders..."

The role of financial engineering is much broader than CEP, and deserves deeper discourse. To read more, continue reading Richard's piece on Tabb Forum (which requires a free registration) here, and watch for more over the coming months on Tabb Forum, on this blog, and on Richard's blog on software engineering and innovation, Hyperextended Metaphor.

CEP News at SIFMA 2010

The SIFMA show in New York is always a big week for capital markets news, and this year is no exception. This year the big news from StreamBase is that there’s so much of it! Last year, we announced a marquee customer, CME Group, and sparked an industry debate about the role of real-time sentiment analysis in CEP when we announced our StreamBase adapter to Twitter.

So what’s going on this year?

New CEP applications: pre-trade risk and more from SunGard. SunGard’s recent selection of StreamBase is the endorsement of the year in the CEP market. Sungard have announce many applications they’re building, from smart order routing to their dark pool. But perhaps the most interesting area from a market trend perspective has been pre-trade risk management. With impending market micro-structure regulation, real-time risk management is becoming an increasingly important area of innovation, a lot of it based on CEP. Watch this space for more news about larger role real-time risk will play in the capital markets.

New innovation with CEP: CME Group. We're happy to announce our customer CME Group won the Inside Market Data award for Best Real-Time Market Data initiative based on their innovative use of StreamBase. Steve Goldman, Director of Enterprise Architecture at CME Group, said it nicely: “We continue to advance our trading environments and capabilities by integrating human intelligence with cutting-edge technologies, such as StreamBase. Through these initiatives we are able to process market data in a reliable and low latency manner while providing transparent and accurate settlement pricing for our customers.”

New cool technology: using GPUs for Low-Latency CEP applications with NVIDIA. NVIDIA GPUs are one of the hottest technologies in computer architecture today, allowing vasts amounts of data to be processed in parallel for low cost and low environmental footprint. However, the current state-of-the-art is to use them to analyze historical data in a research context. StreamBase and NVIDIA have come together to allow real-time data analysis using GPUs for low-latency applications. For organizations using GPUs to research signal processing, options pricing, and risk, CEP-based GPU solutions will enable them to deploy the same powerful analytics models against real-time data in a low latency trading environment.

New technology partner: Solace Systems for hardware accelerated messaging: Solace Systems is a leader in hardware accelerated messaging, delivering ultra low latency and flexibility. Many large sell-side organizations are making Solace the foundation of their next generation realtime data management infrastructure. The integration with StreamBase enables quick and easy deployment of applications written using the StreamBase Studio development environment on the Solace messaging infrastructure to publish and subscribe to all different kinds of data. Using Solace and StreamBase together enables firms to develop scalable low latency applications.

New partner: Redkite brings real-time to surveillance: Our partner Redkite has built their surveillance application on StreamBase (shown at right). Redkite’s solution uniquely leverages the virtues of CEP: extensibility, intraday configurability, case management with audit trails, and simultaneous access to real-time and historical information. 

New, increased activity in FX. The FX market continues to be one the hottest CEP markets out there. We issued our FX market survey last month, and continue to announce a string of FX customer endorsements including CMC Markets, City Index, Gain Capital, and Curex, and many many more to come - stay tuned!

New StreamBase Component Exchange contributions: StreamBase component exchange (SBX) continues to grow as our electronic exchange platform for StreamBase CEP components. Recent contributions include how to connect Python to StreamBase, Bollinger Bands signalling algorithm, linear regression, least squares fitting, Monte Carlo options pricing, and a StreamBase adapter for R. For more, check out SBX here.

New movement among key CEP companies. We were thrilled to announce Fraser Herrick has joined us at StreamBase VP, Head of EMEA Sales. Previously Fraser was a key executive at Progress Apama and we've teamed up with him again. We also wish our good friend Louis Lovas the best of luck as well as he also left Apama for CEP newcomer OneMarketData. 

New developments in Brazil. Business in Brazil is booming, as Kairos Asset Management (KAM) expanded their deployment of StreamBase. KAM added derivatives to their CEP-based trading infrastructure, and we're seeing more derivatives trading with CEP on the global stage as well.

New ideas on entrepreneurship and innovation: StreamBase stepped onto the global innovation stage last year with our attendance at the World Economic Forum at Davos. We continue to participate in the broader conversation about innovation and entrepreneurship, and this week I’ll participate at Venture Summit East a session at Harvard Business School entitled “Going Global.” Stay tuned for blog entries from Venture Summit East.

Stop by the StreamBase booth #1854 (Level 1- just past the Press Booth) at SIFMA to talk to us about all the latest news - our cocktail reception is from 4:00-6:00 on Tuesday and Wednesday. 

And follow our real-time coverage of SIFMA on Twitter: StreamBase (@StreamBase), Richard Tibbetts our CTO (@tibbetts), and me (@mrkwpalmer). See you in New York!

SunGard Selects StreamBase for CEP - A New Way of Working

SunGard is one of the biggest names in the capital markets, so when they announce a corporate software selection, it's important news. 

SunGard's VP of Liquidity Services, Chris Lees told Securities and Industry News: "One of the reasons why we thought about a CEP [is because] we were looking to make real-time decisions on what is effectively streaming data. CEP processing is very adapted to that space." He described why the SunGard selection demonstrates that CEP is going mainstream, from algo trading to dozens of applications, to trade equities, futures, fixed income and options. The platform will be used by SunGard's Assent Liquidity Services, which provides sponsored access, dark pool access and algorithmic trading services, and in conjunction with its Brass Order Management System (OMS). With over $3B in revenue and 20,000 customers in over 50 countries, SunGard is big, and they join RBC, City Index, CMC Markets, CME Group, PhaseCapital, BlueCrest, and others who have made the leap to CEP.

The authoring, testing, and debugging environment — StreamBase Studio — isn't just a graphical modeling tool (e.g., UML), and it's not just a code generation tool (e.g., Apama) — it's a complete, native, unified graphical programming environment. It's not just a modeling tool, it's not a code generation front-end. What you see is what you get.The best way we've been able to describe it is as "whiteboard programming."

Every StreamBase customer uses this pure visual programming, and it's a new way of working because you generate applications more quickly, without sacrificing performance. And because there is no generated code, it's easier to maintain.

SunGard's endorsement of StreamBase confirms our 2010 prediction #3 for CEP, that cognitive physics will become as important as computing physics, where the speed of development is a central source of disruptive technology-driven business advantage.

UPDATE (4/6/2010)

The SunGard / StreamBase partnership was covered broadly, but there were a few angles, as usual, reported that my original post didn't highlight, including:

SunGard to build pre-trade risk tool on StreamBase

The Trade reported that SunGard will focus on developing pre-trade risk controls with StreamBase, a decision that was "prompted in part by the US Securities and Exchange Commission’s proposal to require brokers offering clients direct access to exchanges, including sponsored access, to impose pre-trade risk checks."

They reported that pre-trade risk controls would involve validating incoming orders against market data, for example, to check for errors or anomalies. “You end up with a requirement for completing quite a complex set of calculations in a very short window of time, because you don’t want to delay getting the order to the market while using a lot of streaming data,” Chris Lees, vice president of liquidity services at SunGard, told theTRADEnews.com. “For us, that is the perfect sweet-spot for StreamBase’s event processing platform.”

Read the full article in The Trade here.

SunGard to Buy StreamBase?

Acquisition rumors are a natural part of the software business, and Adam Honore' kicked it off in his blog. I addressed the question here in my blog, and Finextra picked up the whole thread here. The rumor is all over Twitter. 

My remarks in the comments below stand, and I would add that if you compare StreamBase as a platform to the relational database, you see why its a good thing when an industry establishes a defacto platform. The business fuels the innovation and maturity of the leader, and all participants benefit. SunGard, and our many other customers, begin to compete on the efficacy of their use of the CEP platform, not just the fact that they are using it.

What Does Informatica's Acquisition of 29West Mean for the Low Latency / CEP Landscape?

The complex event processing (CEP) landscape continued to evolve last week as Informatica acquired low-latency middleware specialist 29West. The acquisition puts Informatica in competition with IBM’s low latency offering (WebSphere MQ) and Tibco (Rendezvous). But how will this move change the low latency landscape? What are the implications for CEP?

29West is well-respected for low-latency messaging (actually their specialty is ultra-low latency, on order of microseconds). Former Talarian executive Mark Mahowald founded 29West in 2002 after Tibco acquired his old company. 29West's flagship customer, Wombat Financial Software, was acquired by NYSE Technology in 2008. Wombat is optimized for 29West and heavily in use in the capital markets, and by StreamBase customers. Also, last September, Informatica acquired CEP specialist AgentLogic. 

In "9 predictions for the future of event processing (CEP)" from January, our predictions included that software stacks will continue to miss the [CEP] mark, and that a new event processing stack will emerge. Informatica's recent moves might prove both of these predictions to be wrong.

Informatica now owns two key unique pieces of a low-latency "stack": CEP and messaging. So Informatica may “hit the mark” if they combine 29West and AgentLogic (by contrast, Progress Software is steering Apama away from low-latency toward high-latency business process management applications by merging CEP with BPM in order to compete with IBM and Oracle).

So the 29West acquisition is a step toward a new event-based software stack for Informatica. But what comes next?

1. Will Informatica target low-latency messaging or low-latency everything?

29West instantly makes Informatica a formidable low-latency rival to Tibco Rendezvous and IBM WebSphere MQ. But their CEP engine AgentLogic is more of a desktop CEP platform, not low-latency CEP, and is typically used by federal government intelligence analysts.

So will Informatica evolve AgentLogic to become an enterprise-class, low-latency CEP engine? And, if they do, might they combine AgentLogic with 29West into one low-latency event processing stack? 

2. Will Informatica break out of the federal market with CEP?

Informatica’s AgentLogic CEP product has a fine reputation and customer base in the federal market. 29West’s most dominant customer base is in the capital markets (30% of their customers). Will Informatica attempt to cross-sell CEP to 29West customers in the capital markets?

3. Will Informatica’s CEP be middleware-agnostic or 29West-specific? 

General-purpose CEP platforms are middleware agnostic - over 75% of StreamBase customers use more than one type of middleware. For example, foreign exchange (FX) trading applications frequently connect to 2-10 venues simultaneously, and rarely over 29West. Orders are emitted via FIX, rarely via 29West. Will Informatica spend the effort to build the connectivity users outside the federal government demand?

Depending on the answers to these three questions, Informatica’s direction with AgentLogic and 29West could begin to clarify their low-latency event processing strategy, and create a significant entrant into the race to build the next big software powerhouse based on events.

5 Questions Customers Should Ask About the New Progress Software and RPM

This week, Progress Software embarked on its first big strategy shift in 29 years as they announced the Responsive Process Management suite. This announcement is the culmination of a dramatic change in focus that's been coming for months as new CEO Rick Reidy slashed headcount, picked a new target market, fired most of the old guard leadership, and consolidated a previously balkanized organization structure. 

As a former executive at Progress Software (2003-2007, general manager of Apama), and now the CEO of StreamBase, I’ve been asked by Progress prospects, customers and analysts about the "new Progress." So far, analyst coverage of the new Progress has been politely indifferent, so I'll try to help harden the discourse and pose a few simple, but tough questions about the new strategy. But first, let's recap the new plan.

THE ELEMENTS OF PROGRESS SOFTWARE 2.0

Progress Software has a new product strategy, a new sales organization, a new target market, and a new culture. Bold is the only word for these changes, and Rick Reidy and the board of Progress have to get some credit for having the courage to change. In fact, there's so much change I’ll refer to the new Progress as “Progress 2.0." John Rymer from Forrester ably reported on Progress Software’s Coming Out Party on his blog. So rather than report what’s already been said, I’ll analyze the Progress 2.0 business as an investor would view a new business plan, and look at the essential elements of the business as a whole: 1) target customer, 2) product, 3) customer, 4) differentiation 5) selling strategy, and 6) culture and leadership.

The Progress 2.0 target customer. Progress 2.0 CTO John Bates describes their new target customer as a "global 2000 enterprise that needs operational responsiveness.” The first part of this - the G2000 - is a big shift for Progress, which was built on the small and medium sized (SMB) market. As for the subset of firms within the G2000 Progress 2.0 will target, that's unclear. Analyst Sandy Kemsley reported it this way: “[Progress says operational responsive means companies that] want real-time business visibility, the ability to immediately sense and respond, and continuous business process improvement in a cycle of responsive process management." In other words, the target customer for Progress 2.0 is a big company that wants to make better decisions faster.

The Progress 2.0 product: RPM.  Instead of selling 20 best-of-breed products, Progress 2.0 has a single product suite called the Responsive Process Management suite, or RPM. RPM combines 7 of 20 Progress products and “accelerators,” or pre-packaged applications for market surveillance, smart airlines, and telecom revenue assurance. The product elements of the RPM suite are SOA management (Actional), complex event processing (Apama), and business process management (Savvion), messaging (the Sonic Enterprise Service Bus), IBM System z connectivity (DataDirect Shadow), RDBMS mapping technology, and a portal.

The prototypical RPM customer. At their analyst day, Progress used a fictitious "smart airline," as an example of RPM use. The demo showed CEP, BPM and network monitoring working together to manage the operations of an airline. Southwest Airlines, State Street Global Markets and Italian telco 3Italia all presented at their analyst day, but none of them use RPM; in fact, State Street and 3Italia use only one Progress product.

The Progress 2.0 differentiation.  Historically, Progress has won deals by having the best product in a category like BPM, CEP, ESB, CORBA, JDBC, and an embedded database. But by combining these formally best-of-breed products into one suite, Progress 2.0 can't differentiate based on how good the pieces are; they must differentiate on some attribute of the suite. While other vendors like IBM, Oracle and Tibco already have all these products, Progress claims their differentiation is that they are the first vendor to combine them into one suite.

The Progress 2.0 selling strategy. Progress has made sweeping operational changes. They unified their 10+ specialist sales teams into one sales organization that sells RPM. That means that a sales team (rep, pre-sales engineer, and consulting) that used to sell, for example, Apama in the financial services will now sell a suite of seven products. In addition, Progress will sell more consulting - historically, only 5-8% of Progress revenues came from consulting - their new competitors, TIBCO, IBM, and Oracle, sell a lot more consulting - 20%-40% of overall revenues.

The Progress 2.0 culture. Historically, Progress innovated by buying innovative companies. Now, with RPM as the key to their future, Progress will need to innovate from within. Progress didn't announce any cultural changes designed to spark that innovation. 

FIVE FUNDAMENTAL QUESTIONS CUSTOMERS SHOULD ASK ABOUT PROGRESS SOFTWARE 2.0

From a customer's perspective, the Progress 2.0 business plan leaves several big questions unanswered. Here are five of them:

#1: "Who does Progress 2.0 NOT sell to?"

As Bill Cosby once said: "I don’t know the secret to success, but I do know that the secret to failure is trying to please everybody." Cosby is right - focus is the most essential element of business success. Yet on analyst day, when asked about the fate of the individual products that compose RPM, CTO John Bates said that he "believes that [Progress] can continue to compete in their specialty areas such as CEP and BPM, but also as an integrated product suite.” John was either pandering to existing customers or naive to suggest that a company the size of Progress can compete simultaneously with larger foes like IBM and smaller innovative companies like StreamBase. And even if Progress 2.0 focused on their RPM strategy, that focus is blurry. "Big companies that want to make better decisions faster" isn’t a type of customer, it’s a marketing slogan. 

So prospects should probe behind the marketing slogans and understand who Progress 2.0 WON'T try to please.

#2: "Exactly who is the prototypical RPM customer, and does it look like me?"

Every business needs a stalking horse - a prototypical customer that's in their sweet spot. And every VC that funds a new company insists that they win that pilot customer and make them successful before going after a second, third, and fourth customer. 

But at their coming out party, Progress used old customers to justify their new strategy. Three of them use only Apama. The other, Southwest Airlines, uses three Progress products, but with no connection or synergy among the products. The only value Southwest shows from buying three products from Progress is that they  can have a single contract. Then, without a real customer to serve as an example, Progress demoed a fictitious company called “National Airways" which is rumored to be modeled after a deal that Progress lost to new rival TIBCO at Air France.

If you're an existing Progress customer, ask them to describe their prototypical customer for RPM, and then compare yourself to it. If your firm doesn't resemble the stalking horse, think hard about options for the future. 

#3: "How is Progress 2.0 unique?" [In other words, is Progress 2.0 viable?]

Joe Alsop, former Progress CEO, always asked one excellent question when assessing companies to acquire: “What’s your special sauce - what makes you unique?”

What is Progress 2.0‘s defensible advantage that will enable them to beat IBM and Oracle? Will RPM be technologically better? Does Progress have some unique channel IBM and Oracle doesn't have? Is RPM cheaper than Oracle?  Does RPM outperform IBM's offering? Since Progress consultants are vastly outnumbered by their bigger foes, are they smarter? Faster? More vertically aligned? 

We don't know because Progress doesn't say why they're different. If I were a customer or investor, I'd ask directly, because the longevity and success of a company depends on both understanding, then exploiting, your special sauce.

#4: "Specifically, what is the future of Apama (and Sonic, and Savvion, and Actional)? 

In my field of competition (CEP), Progress used to be a leader. Now, Apama is eliminated quickly (other than in geographies where StreamBase has no sales presence), because CEP is an innovative and changing field, and Apama is distracted - Apama sales people have to learn 7 new products to sell and the Apama engineering team is integrating CEP with products that their existing customers don't care about. 

And it will only get worse. In the future, instead of building CEP, Progress will be building RPM. Instead of building solutions for trading on Wall Street, Progress will build solutions for telecommunications firms like 3Italia. Instead of fixing the graphical development environment to make it suitable for finance applications, Apama's tools will be merged with Savvion's BPM tools, and so on. Apama innovation will stop for years.

While Apama stalls, the CEP market is exploding - CEP was named one of the most innovative technologies in the world by the World Economic Forum in Davos, and customers are making enterprise wide decisions for CEP. The time for rapid innovation is now, and customers are demanding better tools, more robust debugging, even lower latency and higher throughput, not a faster BPM tool.

So CEP prospects should ask to see the RPM roadmap and see if matches with your future requirements.

#5: "What innovation changes has Progress made?"

Progress isn't known for innovation - only 1.5 of their 20 products were organically developed. For Progress 2.0, their stated differentiation is that they've put three products together that, as Doug Henschen from InformationWeek reported, IBM, Oracle, and Tibco already have. So their success will depend on their ability to stay ahead of their competition. 

As described in big companies can innovate if they act small, Progress has historically been classically anti-innovation - terrified of failure, internally focused, and conservative. The exceptions were in the divisions that were allowed to develop their own culture; now, in unifying the culture, they'll need to re-think how they manage Progress 2.0.

Customers should ask Progress to be explicit about the management measures, policies, and philosophical changes that will help facilitate innovation.

Summarizing the questions to ask Progress

Progress Software has good people and good products, and has made a big, bold move with RPM. But the Progress 2.0 business plan raises serious questions about their business, and how they will compete. Leaders at prospective customers shouldn't put money on the table without getting good answers to these fundamental questions.

A Man’s Undercover Mission into the World of Women Leaders

Recently, I attended the World Economic Forum in Davos, Switzerland, the gathering of 2,500 CEOs, heads of state, and visionaries from the arts. At Davos, I saw Chinese vice premier Li Keqiang, Bill Clinton, Bill Gates, met Miss Universe, and participated with the “titans of social media” — as the founders of Twitter, Facebook, MySpace and LinkedIn squared off. Heady stuff. But mid week, I went to a session that eclipsed them all. It was innocuously entitled The Power of the Purse.

Davos is an economic forum and I went for economic interests. But I went to this session for different reasons. At work I’m a CEO, but at home I’m a widow. When my wife died of cancer three years ago I took on a new, foreign role — mom, and so I attended this session with my 7-year-old daughter Ruby in mind. I thought this session might give me some insight into the world as she will encounter it.

But as I took my seat, I felt out of place. There I was, one of a handful of men with 60 of the most accomplished women in the world. I felt like I had mistakenly slipped behind enemy lines, then been cordially invited to an intimate power lunch in the general’s tent. The panel included Sheryl Sandberg, COO of Facebook, author Margaret Atwood, Laura Liswood, senior advisor at Goldman Sachs and author of The Loudest Duck, and Laura Tyson, former chair of Bill Clinton’s Council of Economic Advisors.

The two-hour session was like an intellectual popcorn popper; one profound observation shot up after another, and I frantically took notes. The conversation ricocheted between social, sexual, business, and emotional topics. So rather than present them as one thought, I thought I’d list the tastiest kernels:

1. Women, if viewed as an emerging market, are bigger than China and India combined. The economic power of women is rising rapidly. By 2013, the worldwide income of women will rise by $5.1 trillion, which is greater than China's expected growth of $3 trillion over the same time period. And their power is vast: women’s income, worldwide, is greater than the income of China and India combined.

2. 80% of buying decisions in the world are made by women. At first I didn’t believe this statistic. Later, I met the VP of Nissan Motors for Asia. Now I figured of any purchase, that cars would be a male-dominated decision. Wrong. He told me that 70% of the decisions to buy a Nissan are made by women.

3. Men spend money on women and whisky; women spend money on the family. Margaret Atwood observed that this is why 94% of the borrowers from the renowned Grameen Bank are women, each of whom is playing a key role in allowing millions of individuals to work their way out of poverty with increased independence. And historically, this is probably why in Native North American communities the job of dividing up the kill was given to older women — they knew which families needed more.

4. The board of directors gap is still too wide. The CEO sitting next to me told me a story about a woman friend of hers who recently got a call from a recruiter who had been tasked to find a woman for the board of a venerable company that had sold exclusively to women for over 20 years. The board had never included a woman; she would be the first one. Amazed, she declined the interview.

5. Hire women executives, raise profits. When I observed that I had worked in precious few technology companies that had women executives, the woman next to me said: “Well you are leaving money on the table. Studies show that companies with women executives have a healthier top and bottom line.” After this session, I believe her.

6. The more powerful the woman is, the happier the marriage is. I have to admit that I struggled to find research to support the claim that the probability of divorce decreases as a women’s income increases; in fact I found the opposite. But this group felt strongly that professionally balanced marriages are happier marriages and this instinctively made sense to me.

7. A call for balance, not quotas. Surprisingly, this panel of women seemed undisturbed with the natural bias that men — or indeed any group — have to work with people they’re comfortable with. The call to action for male CEOs and boards was for an increased but balanced presence of women leaders, not a coup d’etat.

8. Vocation is just as important as raising kids. The prevailing sentiment among these successful women is that they feel their jobs are as important as their role as a mom. Not more, not less.

9. The way the women interacted was different and in many ways, better. As I was listening, I realized that this Davos session was different than all of the others. Like all the sessions, the speakers were intelligent and passionate. But if the sessions were like a musical performance, the male-dominated ones were like a series of screaming solo acts. By contrast, The Power of the Purse session was more like a skilled jazz improvisation — the speakers were good, built on one another, and the sum of the observations was greater than the individual points. Any business could use more of this type of discourse.

10. A question for CEOs to ask: would you want your daughter to work at your company? Laura Liswood posed this question. She spoke of the framework for understanding the dynamics of diversity in the workplace.

11. Women are just plain better at some leadership roles. I’ve been involved in 15 corporate mergers and acquisitions, and one of the toughest things to do well is post- transaction integration. The women in this session quoted findings that women who lead acquisitions are more successful than their male counterparts. This one resonated on my own logic scale.

12. Are men ready to be displaced? The men my 7-year-old daughter will encounter 20 years from now will be radically different than today’s men. The women at Davos forecast a near-term crisis of self-esteem for men and a re-adjustment of our self- images. That’s an exciting — and scary — thought. What are the implications of a dramatic shift in self-image and power for men? For women? For business? For society? Will men be introspective enough to aid and abet this subtle yet dramatic shift to a more feminine view of power and leadership? This seemed like the deepest of topics that was dealt just a glancing blow, and it’s one that continues to ruminate in my mind.

13. Women work not from choice, but from necessity. Margaret Atwood graciously emphasized this point for me. Many women — worldwide — work not from choice but from necessity. They do it to earn money to feed their kids. It has never been the case that all men worked and all women stayed home. That was a middle-class Victorian ideal, but it was true for only a few women then. The rest were factory workers, farm workers, domestic help, and prostitutes. In fact, huge numbers of “women workers” were prostitutes then, many of them children.

14. Call to action for male CEOs: start an affirmative search for women leaders. When the topic of quotas came up, most women rejected the idea. Laura D'Andrea Tyson advocated the idea of an affirmative search: for men to leave our comfort zone and search for more women leaders, more women as board members, and more women in political office. Profits will go up, decisions will be better, and society will be a better place.

Whenever I leave a thought-provoking event like the World Economic Forum, I ask myself: “With everything I learned, what’s the one thing I’ll do differently?” From the 2010 Davos Forum, the answer was simple — I’ll heed Laura Tyson’s call for an affirmative search to include more women on my leadership teams and encourage other men to do the same. And if you’re still not convinced to do it for your business, do it for your daughters.

(Mark Palmer is the CEO of StreamBase Systems.  Follow him at twitter.com/mrkwpalmer)

 

 

CEP Innovation Goes Global: Announcing the StreamBase Component Exchange (SBX)

As the use of StreamBase continues to expand, our customers have asked us to help them access other StreamBase developers in order to facilitate innovation and tap the power of the growing StreamBase community.

So today StreamBase announced the StreamBase Component Exchange (SBX) - the first community to allow developers to download and share reusable CEP components. SBX is open to individual developers, technology partners, and application partners that wish to share their StreamBase-based applications and components.

SBX will help push the CEP market to the next level by providing a global, open library of technical innovation, where ideas can be exchanged freely. The kinds of components that are available include:

Adapters. Adapters connect in-bound and out-bound event streams to the CEP platform. For example, the Twitter adapter allows StreamBase to consume and produce Twitter “tweet” streams. 

Integrations. Some components implement non-adapter integrations. For example, the Calling Python from StreamBase component allows for the integration of the Python programming language with StreamBase. 

Utilities. Utility components are small operators or functions that extend the CEP platform. For example, the Delay Operator component defers processing an event into the future.

Domain-specific algorithms. Domain-specific components illustrate the use of common industry-specific techniques. For example, the Options Pricing using Monte Carlo component is available to the financial services industry and demonstrates put and call options using a Monte Carlo method.

Third-party frameworks and applications. Third-party developed applications can also be contributed to the SBX to promote CEP-based products, CEP-based connectivity, and CEP application best practices and advanced techniques.

Developers can access SBX directly from StreamBase Studio (download here), or from the SBX website at sbx.streambase.com. 

Welcome to the SBX community!

What Does the Sybase / Aleri Acquisition Mean?

Today, Sybase announced they have acquired the CEP vendor Aleri. This news comes the week after StreamBase CEP was named as one of the 26 most innovative technologies in the world by the World Economic Forum at Davos. 

What does this all mean?  The jury will be out for a while, but there’s a lot we can say now about the move, and what the future is likely to hold for Sybase and their CEP strategy:

Sybase's new focus on events is a clear sign of strength of the independent CEP market. The acquisition of Aleri was a strategic one that shows a clear focus on event processing. For that we applaud Sybase! As I wrote in my recent article on predictions for CEP, a new event-based software stack is emerging. Sybase now has an event-oriented database (Sybase IQ) and an event-oriented computing platform (adding Aleri). We welcome a healthier, bigger competitor and relish the competition.

The real CEP stack is best of breed, not Sybase bred. There has been a lot of discussion in the Twitter-sphere about the idea that Sybase + Aleri is a move toward bringing BI and CEP together. Not so. As our CTO Richard Tibbetts (@tibbetts) recently wrote, BI and CEP are different animals. Customers of CEP have all sorts of data warehousing tools, for different purposes. A great CEP platform must work with all of these. For example, at StreamBase, we work with traditional databases (e.g., Oracle, Microsoft, MySQL, and, yes, Sybase!), specialist databases (e.g., Reuter's Vhayu), data warehouses (e.g., Vertica), and traditional BI (e.g., SAS). Frankly, the most important of these players are Oracle and Microsoft, not Sybase. So, will customers buy ALL those data warehousing tools from one vendor? Unlikely. 

The winning CEP powerhouse will be the one that stays focused. Last month, I predicted that CEP will yield a great new software powerhouse. On their briefing call, Aleri cited the difficulty of selling their software to large firms as a reason they wanted a more economically viable partner in Sybase. This is a poor excuse; confidence is a function of focus, and the quality of the product, people, and finances. StreamBase is private and independent, too, and racked up public blue-chip wins from the CME Group, City Index, RBC, CMC Markets, BNY ConvergEx, and dozens more in 2009.  As John Rymer (@johnrrymer) from Forrester tweeted today, "StreamBase has a very strong product and great financial services savvy." A great product and financial services savvy is why StreamBase is becoming the undisputed leader in CEP, beating bigger competitors based on quality.

Sybase has their innovation physics backwards. At Davos I spoke on the topic of innovation, and one truism kept being repeated among the 2,500 business leaders: that big companies can innovate, if they act small. Yet in Sybase's own press release, they reveal their basic misunderstanding of this principle when they say this: "The addition of Sybase development resources to the Aleri platform will accelerate the pace of innovation adding new features to the CEP engine." Innovation doesn't come with "money and resources," it comes from a deep organizational understanding of the art of entrepreneurship, and an ability to execute, for which Sybase is no longer known.

So StreamBase welcomes Sybase as a focused competitor in the CEP space once they integrate the four CEP products they now own. And, until they do, we welcome customers of Aleri, Coral8, and Sybase RAP as they migrate to StreamBase as their platform of choice.