The StreamBase Event Processing Blog

CEP: Closing the Loop on Business Intelligence

(Richard Tibbetts is the Chief Technology Officer of StreamBase.  Follow him on twitter as @tibbetts) 

There has been a lot of talk recently that compares complex event processing (CEP) and business intelligence (BI) technology, and what they compare to each other.  Much of it suggests that CEP is a logical extension of BI into real-time, continuous analysis.  For example, Seth Grimes (@sethgrimes) suggested the distinction here and Paul Vincent from TIBCO responded by quoting Tony Baer from Ovum (@TonyBaer) who twitted the following:

"So, in essence, CEP makes BI literally actionable.  IOW (in other words) BI is embedded in execution."

These posts triggered a Twitter discussion about how true Tony’s statement was and the breadth of functionality available from CEP platforms like StreamBase. It culminated in Neil Raden’s (@NeilRaden) question:

Are you suggesting that CEP, on its own, is an adequate technology for adaptive control & integrating models/rules from BI and BRMS? (tweet)

Neil’s question crystallized the way CEP is misunderstood by those who are more familiar with traditional data management technology. They try to assess a tool that’s new by comparing it to another tool that’s built for an entirely different purpose.

The core technology of CEP is real-time continuous data transformation. These transformations can include event pattern detection, data enrichment, data analysis, and data reduction. They generally also include support for business rules and for executing code written in traditional programming languages like Java or C++. All this is bundled into a high performance engine designed to run scalably and reliably, and supported by powerful development environments and administrative and visualization tools.

That’s a lot of capabilities that can be brought to bear on many different problems. No wonder some people aren’t entirely sure where to start when attempting to understand what is possible with CEP.

If you know BI, then you try to fit CEP into the BI context. Yet, in fact, StreamBase has never been compared to BI by a prospective customer, despite the fact that “Operational Business Intelligence” has been a buzzword in that area for about four years. If only we could do BI faster, the theory goes, businesses could take advantage of it in the millions of little decisions that impact the business on a day-to-day basis. CEP and BI have similar sounding mantras, yet practically speaking, customers never intermingle the two.

What those immersed in BI don't realize is that the barrier to operational data analysis is not the speed of the data warehouse. As any trader will tell you, the fastest data analysis in the world won’t help you if every trade requires human approval.

In order to take advantage of modern data analysis, you need to stop depending on people to pull the oars, and start making them captains of their own ship. This requires software that can not only analyze a situation, but also take action, and then control that action over time. Workers ("Automation Managers" in the diagram on the left) monitor these systems and cope with exceptions, while the majority of events and decisions are processed automatically, without incident. The role of the user is in guiding real-time activity, not stopping the flow, making a decision, and starting the flow again.

Consider modern trading systems. Years ago, trading floors had hundreds of traders – half of them were making low-level trading decisions based on real-time market data that they watched; orders were placed by typing them into an order management system (OMS); it took 50-100 milliseconds for each trader, once he made a decision, to press the button to buy or sell. Today algorithms designed by quants buy and sell stocks in microseconds, not milliseconds, and one trader, with a real-time application, can guide the behavior that does the work of 1,000 old-school traders, an order of magnitude faster than just five years ago. That’s change in an industry, motivated by a new way of thinking of real-time data processing – complex event processing.

These software applications become the backbone of an automated business. Change is the norm - as the business climate evolves, agility becomes a critical success factor. If the system can be developed and delivered efficiently enough, a wide variety of decisions become suitable for automation. By making millions of small decisions intelligently, massive efficiency gains are possible. Decision support transforms from being a part of the process, to guiding automated processes – a fundamentally new way of thinking that makes old-school BI thinking inapplicable.

But transforming a business to real-time requires not only new technologies, but a new way of thinking as well. It requires a whole set of business practices that only a few companies are now comfortable with. And it requires analysts and software architects who take a wholly different look at how they approach systems. Neil Raden himself talks about this in his book, Smart Enough Systems.

You have to turn operational decision making into a corporate asset you can measure, control, and improve.

Wall Street reached the tipping point about five years ago, and has been creating not only the software systems but also the expertise necessary to run a truly real time business. CEP systems on Wall Street not only monitor markets and customers for opportunities, they also take actions and continuously monitor those actions for successful implementation.

The answer to Neil’s question is that yes, if a business is ready to adaptively control processes with models, rules, and analytics, then CEP is the ideal technology for integrating analysis with action in real time.  

Winning Paper Evaluations, Good; Winning Customer Evaluations, Better!

(Mark Palmer is the president and CEO of StreamBase Systems.  Follow him at twitter.com/mrkwpalmer)

Today, Forrester released their evaluation of CEP vendors, and declared StreamBase a leader in the space. Wonderful news - we're very proud. 

That said, analysts typically perform paper evaluations.  Forrester's paper evaluation is a good tool, and helps prospects create short lists for competitive evaluations.  But as a technologist, I'm thankful for the recent public disclosure of findings from some deep-dive customer evaluations from the last six months.  These evaluations were performed by firms that installed the products and ran, in many cases, stringent benchmarks and scalability tests. They evaluated the debugging tools by using them, and used deployment features like high availability.  They looked at how easily threading or partitioning behavior can be designed and controlled by doing it.  

In some cases, these firms released hard numbers.  In others, we have generic performance characterization information that we provide on an as-needed basis.  And in many cases, they put their money where their mouths are - they made one choice and bought the product. In case you missed them, here's the public take on some of these evaluations:

• CME Group (Press release, CME Group by the Numbers article, follow @CMEGroup on Twitter)
• RBC (Press coverage in Inside Market Data)
• BNY ConvergEx (Press coverage in The Trade)
• PhaseCapital (Blog post: 5 Observations about PhaseCapital's selection of CEP, blog post: 8 Lessons for Entrepreneurs - Based on PhaseCapital)
• BlueCrest (BlueCrest case study, interview with Justo Ruiz-Ferrer from BlueCrest)   

Among the more interesting evaluations was the CME Group's work.  They performed the most exhaustive CEP evaluation I've seen in years - perhaps ever - and their findings reflect what we typically see when you dig in deep and stand StreamBase, Aleri, Coral8, IBM, Oracle, TIBCO, and Progress Apama against each other.

Steve Goldman, the director of Technology Architecture from CME Group, said it nicely and even put it on video for us, where he cited usability ("The business users and technology users felt the StreamBase technology was heads above everybody else"), horizontal scalability & performance ("This was very, very important"), and sustainability ("The ability to run the platform reliably and with fault tolerance was key.") as the key reasons why the CME Group chose StreamBase as the leader, and the choice for their enterprise CEP deployment.

Here's the clip of Steve from CME Group, in his own words:

  

This endorsement was so strong and decisive that some of our competitors have started making crazy claims, including that StreamBase gave their product to CME Group for the endorsement.  In some cases the claims are so unprofessional that I enjoy nothing more than personally responding to them (with appropriate confidentiality of course).  In fact, I enjoy doing this so much, I decided to put it right out here on the blog and offer my direct contact.  I can be reached at mpalmer@streambase.com, or directly in my office:  +1 781 761-0812.  Naturally, sales solicitations are not welcome and will not be returned. 

Innovation by the Numbers at CME Group

(Mark Palmer is the president and CEO of StreamBase Systems.  Follow him at twitter.com/mrkwpalmer)

Recently, I spent a day in Chicago with CME Group, the world’s world’s largest and  most diverse derivatives exchange, to talk about their approach to innovation.  As the CEO of an innovative software company founded by forward-looking MIT techies, I’m lucky enough to work with a lot of innovative people.  But as we talked about the size and scale of CME Group’s business, it struck me that this may be among the most technically challenging environments on planet earth, and the numbers tell a compelling story of challenge and opportunity for a firm that many know of, but few have seen inside. 

CME Group is the modern descendent of the Chicago Mercantile Exchange (CME), the American financial and commodity derivative exchange which was founded in 1898 as the Chicago Butter and Egg Board, and in the past 111 years, grown and transformed into the world’s largest electronic futures and options exchange (Globex).

But I wasn’t visiting CME Group to check out their museum, I was there to dig down on the business and technical motivation behind the news that they have selected a new development paradigm (press release, coverage in Wall Street & Technology), event processing (CEP), on which to build future innovations.  This is a watershed moment for the CEP industry, and an important case study that gives us pause to wonder about the future of automated business, the future of real-time, mission-critical infrastructure, and the role of event processing as the platform for that innovation.

So let’s look at innovation, CEP, and the challenges faced by CME Group: 

...1.2 quadrillion dollars ($1,200,000,000,000,000,000) 

                                           ...The underlying value of the contracts traded by CME Group in 2008

How can a firm stay on the innovation edge while simultaneously supporting the most mission critical operations?

The decision to process 1.2 quadrillion dollars of value was not taken lightly at CME Group. It’s been said that “nobody gets fired for buying IBM” – that is, that big companies are the safe choice when a decision is most critical. 

But when innovation must ride beside reliability, decisions get trickier.  A company like CME Group must balance innovation with mission-critical capability.  And, in the case of CEP, CME Group’s decision illustrates that CEP has come of age.

However, there’s been a rash of recent debate about the entrance of IBM and Microsoft into the CEP market.  Recently, the New York Times published an article, which declared that innovation is moving to big companies from small companies – a conclusion that is not only wrong, it flies in the face of all the most basic tenets of innovation.  So in the innovative, fast paced CEP market, we are wary of, but don’t fear, the entrance of giants like IBM and Microsoft.  Indeed, we welcome it, because it draws more awareness.  The more people look, the more will buy StreamBase.

And from the customer’s point of view   I asked Steve Goldman, CME Group’s Director of Enterprise Architecture, if he had looked at IBM and Microsoft, and his response was casual:  “The decision was very difficult but we felt that based on our environment and needs thatStreamBase was a proven solution.” 

11,000,000...

                   ...The current average daily volume of futures contracts traded every day by CME Group...

“Our data feeds are like fire hoses,” Goldman said.  “Sometimes, we process a few hundred market data changes a second.  But when the markets get volatile, we process tens of thousands of market data changes per second. And, to make it even more challenging, our customers demand the fastest response time – in milliseconds – when volatility is at its peak.” 

Real-time response is becoming more and more important in mainstream business.   Any environment that deals with events (stock market price changes, web site click data, and so on), can benefit from event processing.  And the technological challenge of managing events at high speed, and with low-latency, requires a host of arcane tooling and expertise.  These geeky areas will also repay deeper exploration, including specialized messaging hardware and software, clever use of modern computing models like multi-core CPUs, grids, and clouds, FPGA’s, and efficient management techniques to keep it all running smoothly.

But modern CEP engines are becoming an essential tool for managing enterprise-class streaming IT environments – a better way to manage information that comes through a fire hose.

Hundreds....

                ...The number of products offered by CME Group...

While most businesses might have a few products, or a big business might have a dozen products, CME Group has hundreds of products. But CME Group’s products aren’t made of physical materials; they are built with software development tools.  So the speed and effectiveness of software development at the CME Group dictates the speed and growth of their business.

So the news that CME Group has chosen a new enterprise-wide development paradigm is important news.   CEP, which has been rising to prominence in the past 5 years, flips traditional database-oriented computing on its head.  CEP allows an application developer to design logic that examines what’s happening now instead of what happened in the past.  Options pricing and settlement consumes vast amounts of streaming information, so the CEP model fits perfectly, and allows CME Group to quickly create new products that process those streams.

Any business that has streams of data – from web clickstreams to point-of-sale transactions to financial market data - can apply CEP to more quickly and easily bring new products to market.  When the development methodology matches the form of the data, innovation approaches the speed of human thought.

80...

        ...The number of countries where CME Group's Globex provides regulated exchange access...

Although everyone talks of the inevitability of a global economy,  few firms understand its practical challenges than CME Group. CME Group offers electronic, regulated exchange access in over 80 countries.

The host of technologies that enables a 7x24, global operations are many.  But CEP, by making streams of data from all over the globe more intelligent, can make network event processing smarter and faster, and is becoming an important element in enterprise IT infrastructure.

Which leads to a critical innovation issue:  how to strike the right balance between a new technology and a proven technology?  A full answer to that question is outside the scope of this article, but the key elements for enterprise software products include well-designed threading architectures, bullet proof fault tolerance systems, robust management tools, comprehensive testing and debugging tools, and profiling tools. Consider that CME Group traces its origin to 1848, has NEVER had a default or a loss of customer funds resulting from failure, today announced its enterprise deployment of CEP technology.   This is the clearest signal yet that event processing is coming of age as it continues through its rapid growth as a software market.

A few milliseconds....

                                ...the response time necessary at the most heavily loaded times....

To see the manual forms used at exchange over 50 years ago puts the innovation in the capital markets  in perspective.  It took many seconds to fill out the information about trades back then; it takes 400 milliseconds (1/1000 of a second) for the human eye to blink;  today, CME Group’s trading decisions take place much faster than the blink of an eye – in just a few milliseconds.  How can this kind of latency be achieved in the face of massive volumes and massive global distribution?

We discussed the role of CEP in turning the traditional database-oriented model on its head.  Databases, of course, rely on writing data to disk.  The typical seek time for a disk is about 9 milliseconds.  CME Group’s data can change tens of thousands of times per second;  in order to make decisions in a few milliseconds

Exchanges settling contracts, brokers splitting up and routing client trades, electronic liquidity providers making markets, hedge funds finding and executing arbitrage trades, banks delivering currency rates…all in less than one hundredth of a human blink!  Applications are measured in microseconds, and the tools that do the measuring have resolution in nanoseconds, that’s one billionth of a second.  

StreamBase has been emerging as the top CEP company for some time now due to its proven rapid development environment, standards-based approach, high performance, low latency, and mission critical, fault tolerant architecture.   CME Group’s selection of StreamBase is yet another proof point that demonstrates that CEP is ready for the big time, and that StreamBase is leading the way.

That’s why CME Group made this important decision for their real-time, low latency operations – the need for speed, and the need for CEP, is critical at CME Group.

Massive scale.  Instantaneous response.  Constant change.  Nonstop operations. Taken individually, these are big technical challenges; taken simultaneously, they are an extreme business risk.  But risk for one is opportunity for another, and as CME Group shows us, by keeping an eye on the three key “abilities” – usability, reliability, and scalability –firms can keep innovating as business conditions continue to get more complicated.

 

Case Study: BlueCrest Capital Management

Motivation for Real-Time Market Data Management

BlueCrest Capital Management is a leading European hedge fund, based in London, with over $15 billion in assets-under-management and an award-winning reputation for combining ‘technology, diligence and innovation to produce the most efficient risk-taking possible’.

Since its launch in 2000, BlueCrest has aimed to stay ahead of the curve by anticipating future challenges and investing early in skills and technology. The firm’s traders are highly technical with many trained mathematicians. Complex statistical models form the foundation of some of its trading strategies, and these rely on a rich variety of data-feeds to enable them to trade “anything that ticks”, as their Head of Trading Strategy Systems, Justo Ruiz-Ferrer describes it, adding, “We trade everything from traditional fixed income, equities, FX and related derivatives, to exotic commodity, energy and alternative asset classes.” In such highly competitive markets, BlueCrest soon realized that good data management was as important as their quantitative models. “You may have a Ferrari, but if you fill up your tank at the supermarket, you won’t get the performance,” observes Ruiz-Ferrer.

Scalability and real-time performance were key factors with BlueCrest’s data storage demands exploding at 200 to 400% per year, and exchange based derivative markets hitting new highs of billions of price ticks per day, not to mention the OTC platforms, as ever more robotraders enter the jittery markets.

A New Market Data Management System

In 2007, just as the credit crisis was breaking, BlueCrest set up a team under Justo Ruiz-Ferrer, to develop a state-of-the-art market data management system.  BlueCrest trades 24 hours a day, six days a week, across multiple markets using a wide range of data feeds.  As markets move day to day and week to week, BlueCrest needed to rapidly reconfigure data feed connections and plug the data into real-time models while optimizing management of the necessary data feed licenses. BlueCrest devised a solution that combines the rapid time-to-market event processing capabilities of StreamBase with the instant storage and retrieval functionality of Vertica. It provides a total market data management solution that is able to meet the needs of low-latency trading and the demanding innovation of their quantitative analysts to achieve greater profitability.  As the range of data sources constantly grows with increased cross asset class and exotic derivatives, time to market, low latency and auditable compliance with complex data licensing rules were essential.

Ad hoc engineering of new data feed adapters had proven to be both expensive and risky due to complex software interfaces, few standards, and low reusability of components. The new system therefore aimed to decouple users from the data sources to create a common interface for accessing, conflating, enriching and distributing real-time data with minimal latency and assured compliance. Yet it also had to encourage flexible data sharing across the firm.

BlueCrest’s new market data system provides both current and historical snapshots and on-demand event processing for both simple requests like last price traded to complex scenarios like volatility surfaces or correlation signals with time windows. It also has to guide the user through the rich jungle of data sources and compliance rules, and help resolve queries that may have alternative solutions across the data providers. Each source and derived data-object needs its own access controls, as these cannot always be inferred.

Market Data Mechanics

The system was built in three layers: (1) An event layer that includes the data sources and a core event-processing engine from StreamBase to extract, collate and enrich the source data; (2) The storage layer consists of a real-time, historical tick database from Vertica, backed up on disc, and a last tick store for last known value (LKV) caching; while (3) the control layer includes compliance and management components written in-house in C#.  The control layer checks user access permissions, retrieves data from the storage layer or activates data streams and then dynamically sets up ‘Collectors’ to relay the resulting ‘event-curves’, such as volatility, yield or momentum curves to all subscribed users or their software applications. The whole system is driven by real-time requests from the users or their software models.

Data streams fuel the system. Ticker plant and internal messaging buses feed data to the StreamBase complex event processing engine to clean, normalize, filter and enrich the event-curves. As part of this enrichment, StreamBase can mix real-time events with the historical data in Vertica as needed to evaluate the complex rules, yet still keep pace with the input data rates. StreamBase then forwards the new event curve objects to both Vertica for storage and the Collectors in the control layer, which encrypt them to ensure compliance, cache them in the last tick store, and distribute them to all waiting user processes. Whenever any point changes, StreamBase generates a new event, and therefore provides a stream of event curves rather than just data points. “It’s immensely powerful given the high speed inputs and the complex handling requirements,” says Ruiz-Ferrer.

The system works round the clock six days a week supporting user services across different geographical locations. Further servers with collectors, local caches and control software are also located in each office to minimize latencies for last value type queries, which represent the bulk of the workload.

All the core data feeds are now plumbed into the system including the major data aggregators and many direct exchange feeds. However, much of the data actually comes from internal BlueCrest services over the messaging bus. “We can subscribe to everything,” says Ruiz-Ferrer, “so we load tested to ten times the whole feed, just to be safe.”

Security is tight, hence the encryption, which is the most expensive component in terms of CPU resource, according to Ruiz-Ferrer. Moreover, the system keeps track of who is accessing which service at any time to ensure compliance with the licensing rules. One advantage is that BlueCrest can optimize license fees to maximize sharing when data is not in use. “What a user sees depends on the market data licence. Without the licence, users only see limited price movements, whereas with the license they see everything,” explains Ruiz-Ferrer.

In London all the processes run on a single blade server to eliminate network hops. Each layer of the architecture can be separately scaled across its own set of server blades to meet changing performance and resilience needs. Thus the whole sequence from real-time source to waiting user servers in London takes under 20 milliseconds on a benchmark test of 2 million updates per second with very low server loadings.

For the middle office and research models, the priority was to leverage the full knowledge base of the firm with great flexibility and rich functionality rather than minimizing microseconds. Indeed the core StreamBase component can typically be completed in well under a millisecond, but complex enrichment and security do add some latency.

Rapid Time to Market

The project build started in January and by May it was in production, even allowing for a one-month delay due to other priorities. The team now is working to port other data feeds and analytics to StreamBase, enriching the visualization tools, as well as making the whole system fail-safe with a high speed interconnect to the back-up site.

Prior to the market data management solution it could take 2 months or more to add and integrate a new data feed. Now a new service is connected in just two weeks because everything is standardized. “This makes the whole business much more responsive and confident,” says Ruiz-Ferrer.

Having evaluated many other solutions, the BlueCrest team chose StreamBase because of its real-time performance and flexibility, with many adapters available out-of-the-box for connecting to data feed handlers, messaging and data base systems including Vertica, and trading platforms. They found the visual workflow-like design capabilities of StreamBase particularly helpful for rapid development and maintenance.

“StreamBase and Vertica have allowed us to develop our market data environment rapidly with a sound run time behaviour, excellent extensibility and minimum effort”, concludes Ruiz-Ferrer. “By investing in a high-speed data storage and an ultra low-latency complex event processing solution, BlueCrest has ensured that its vital market models get the highest octane data mix available to stay ahead of the pack and negotiate the market chicanes at full throttle.”   BlueCrest has demonstrated once again how innovation can meet the challenges of the rapidly evolving global capital markets. 

About Bob Giffords
Bob Giffords is an independent banking technology analyst based in the UK, who carries out research on the global evolution of the global capital markets. Following a long career in systems development and consulting to the finance sector, he was CTO for a leading bond trading platform in Europe. He is now a frequent speaker at securities industry conferences and writes for various industry journals including The Trade, Waters, Dealing with Technology, Swift Dialogue and Financial World. He can be reached at bob.giffords@btinternet.com.

About StreamBase
StreamBase Systems, Inc, a leader in high-performance Complex Event Processing (CEP), provides software for rapidly building systems that analyze and act on real-time streaming data for instantaneous decision-making. StreamBase’s Event Processing Platform(tm) combines a rapid application development environment, an ultra low-latency high-throughput event server, and the broadest connectivity to real-time and historical data and leading EMS/ OMS software platforms. Six of the top ten Wall Street investment banks and three of the top five hedge funds use StreamBase to power mission-critical applications to increase revenue, lower costs, and reduce risk. It is also used by government agencies for highly specialized intelligence work. The company is headquartered in Lexington, Massachusetts with European offices in London.  For more information visit www.streambase.com.

About Vertica
Vertica Systems is the market innovator for high-performance analytic database management systems that run on industry-standard hardware. Co-founded by database pioneer Dr. Michael Stonebraker, Vertica has developed grid-based, column-oriented analytic database technology that lets companies of any size store and query very large databases orders of magnitude faster and more affordably than other solutions. The Vertica Analytic Database is available as software only, as an appliance or online as a cloud computing solution. The technology’s unmatched speed, scalability, flexibility and ease of use helps customers like JP Morgan Chase, Verizon, Mozilla, Comcast, Level 3 Communications and Vonage capitalize on business opportunities in real time. Vertica is headquartered in Billerica, Mass. For more information, visit Vertica.com.

BlueCrest Deploys StreamBase, and Talking about Event Data Management

Serious discussion of event data management, CEP, and how they relate to event driven applications is a topic that's gone under the radar for a while.  Even though databases are always a central element of a CEP-based system, we haven't pushed the exploration of these use cases with any vigor.  That needs to change, and we've put a lot of effort this year at StreamBase into broadening the discourse about CEP beyond point applications like algorithmic trading.  Market data management is one such area.

This week's announcement about a new StreamBase customer, BlueCrest, helps broaden that discourse. BlueCrest deployed Vertica, and StreamBase as an intelligent, low-latency, aggregated market data management backplane.  Real-time, event driven data management is a great use of CEP, and brings the "event database" center stage in an event driven architecture (EDA).  In this case, the architecture is used to create a real-time, aggregated view of "everything that ticks" in the financial markets for BlueCrest.  The importance of this architecture is nicely put by Justo Ruiz-Ferrer, the head of trading strategy systems and quantitative development at BlueCrest:

“Great data management is as important to us as quantitative trading models.  The combination of StreamBase and Vertica enables us to get a low-latency, aggregated view of everything that ticks in the increasingly complicated, fragmented, high-speed financial markets.” BlueCrest said it wanted to be able to quickly reconfigure data feed connections and plug the information into real-time models.

"Great data management is as important to us as quantitative trading models."  What an extraordinary statement.  As the capital markets go through its massive gyrations, intelligent, low latency data management is becoming as important as their quantitative trading models.  More than ever, I hear electronic trading gurus say stuff like this - that the the quality of trading is only as good as the quality of the data.  With the markets constantly fragmenting, segmenting, compressing, and then breaking up again, it's harder than ever to keep up, and trade in new asset classes.   In the case of BlueCrest, their market data backplane manages "everything from traditional fixed income, equities, FX and related derivatives, to exotic commodity, energy and alternative asset classes."  All in real-time, in an event driven architecture.

Soon we'll release a comprehensive case study on this blog about the BlueCrest system to help share its interesting technical elements with the community.  More to follow on that later, but for now, here's the BlueCrest / Vertica / StreamBase architecture:

More to come on the role of serious event data management in EDA!

EDA and CEP: Where's the Beef?

Jack van Hoof wrote a post this week called "EDA versus CEP, Again..." and Joe McKendrick echoed his remarks in his post "Why 'Event Driven Architecture' is more than Complex Event Processing."   Jack and Joe both suggest that CEP vendors are incorrectly trying to equate CEP with EDA.   Jack's main point is that EDA is architecture, and CEP is a tool.  My reaction was:  "Well, of course that's true!"   But what confuses me is exactly who Jack and Joe think is getting it wrong.  

I got worried for a split second, because last summer I wrote a bunch of articles about EDA, event processing, and CEP.  I wanted to make sure I hadn't been responsible for the confusion - so I went back and re-read my writing*** and I'm pretty sure Jack and I would agree on terms. Indeed, I often referred to Jack's thoughtful writing on the subject in those articles, and he commented on them.

But I think there's a point that wasn't covered by Joe and Jack - that CEP, in fact, is a really important element of an effective EDA.  It's not a required element, but it sure makes it better.  I believe that CEP will prove just as essential to EDA as a relational database is to client / server applications, or as application servers are to three-tier architecture.  And it's a fair point that because it's becoming more common in mainstream IT, that more best practices, patterns, use cases, and discourse about effective EDA is needed.

Yet, as one of the posters in Jack's blog laments:  "Why aren't more people talking about EDA, because that's what's really important?"  Perhaps EDA isn't new, and CEP is.  My first job out of school was as a developer working on an event driven trading system, in 1988.  Our group built a multicast middleware driven, real-time trading system.  The only other analagous software around was the Teknekron Information Bus, or TIB.  We used Reuter's "new" digital market data feeds, and integrated that with event-driven trading screens and applications.  It was an excellent event driven architecture.   It was new. In 1988.  We didn't use CEP because it didn't exist.  Until recently.

And perhaps that's the point:  maybe the reason why folks don't talk as much about EDA is that the disruptive element of EDA IS CEP?

*** Unfortunately Progress Apama has (I'm assuming mistakenly) attributed my writing to someboday named "Progress Apama," not the correct author: "Mark Palmer."   Regardless, the articles to which I refer (e.g., "The Myths of EDA and SOA," and "The 10 Imperatives of Algo Trading") were written by me, and I stand by them.