The StreamBase Event Processing Blog

Introducing Live BI and StreamBase LiveView

For some time now, StreamBase has been working on a new product concept that's a little bit CEP, a little bit data warehouse, and a little bit business intelligence. Today we've unveiled it as StreamBase LiveView. Our customers have called it a "game changer," and a product that "fills a massive hole in the BI space." Hyperbole like that needs to be backed up, so here we go.

Complex event processing (introduction on Google+) was envisioned by its inventors as a "database for moving data" as firms increasingly deal with data in motion from mobile devices, social media and digitized data of all types, and in all industries. In its first decade of commercial use, CEP has come to power automated trading on Wall Street. A who's who list of firms have been public about their adoption of CEP, so let's examine the live BI problem in the context of Wall Street.

Industry analysts estimate that over 60% of U.S. equities trading is automated. The problem is, with so much automation, it's impossible to see and act on what's going on, because a typical trading environment can generate 10's of millions of events a day. Existing business intelligence and data warehouse tools aren't used in the front office of Wall Street, because they're designed to summarize what's already happened, not what's happening now. 

StreamBase LiveView brings BI and streaming data together, for the first time. Applications that need Live BI are everywhere on Wall Street - trading risk management, position management, P&L management, surveillance, compliance, and trading itself, as Waters Technology magazine describes in the article that StreamBase Seeks 'Radical Transparancy' in Business Intelligence.

And applications in other markets can be found in any environment that has big data in motion: telecom (location aware insight and live network operations), security (live fraud detection, live surveillance), and e-commerce (live social media analytics, ad placement, big streaming analytics for Hadoop and Flume). These apps can be deployed on premise or distributed in the cloud. 

StreamBase LiveView is built with StreamBase CEP, which provides connectivity to over 125 event-based sources of data and the extreme scalability required by live BI. The server-based architecture is designed to process hundreds of millions of events per node, absorbed from a network of event sources such as an enterprise message bus or scalable event absorption layer like Flume or IBM Streams. At the front end, users can slice and dice streams of data with ad-hoc query tools in the StreamBase LiveView Desktop (shown above). But here's the key: once users ask the questions they want answered, StreamBase LiveView keeps answering their questions - as conditions change in real-time; users see an always-live view of streaming data.

For the user, all this technology means just one thing: they see business intelligence that's constantly fresh and relevant to their business operations. It's not a view of 5 days ago, or even 5 minutes ago. The data is live, the insight is live, and live insight can lead to live action.

Facebook, Twitter, Google+ give us a live view of what's happening in our social networks; StreamBase LiveVew gives firms a live view of their enterprise data.

To learn more, watch the StreamBase LiveView presentation, follow new materials, articles, and resources on Google+ (+StreamBase and CEO +Mark Palmer), read industry analyst Curt Monash, DMBS2.0, or follow StreamBase Systems (@streambase), CEO Mark Palmer (@mrkwpalmer ), and CTO Richard Tibbetts (@tibbets) on Twitter.

 

Thomson Reuters Selects StreamBase CEP for Elektron High-Speed Cloud, Quant Platform, Apps, and FX

Today we're excited to report that Thomson Reuters has selected StreamBase as its complex event processing (CEP) platform for a broad set of initiatives. CEP "flips the database on its head" by providing the ability to monitor, analyze, and act on events happening in real-time, rather than looking in the rear-view-mirror of what happened yesterday, last week, or last month. CEP powers real-time, big data applications on Wall Street including high frequency trading, pre-trade risk management, surveillance, market data management, smart order routing, pricing, settlement, real-time P&L, monitoring - applications from front to back office on Wall Street.

Why did Thomson Reuters partner with StreamBase for CEP? Initially, they announced four reasons: 

StreamBase in Elektron Brings CEP to Thomson Reuters' High-Speed Cloud

The StreamBase platform lies at the heart of a new generation of real-time, automated applications on Wall Street. As the use of CEP continues to grow, our customers increasingly ask to deploy some of their applications in the cloud. They want their automated processing to happen as close to the exchange as possible for the lowest possible latency, and they want secure and reliable access to their market data.

Thomson Reuters announced that they will provide StreamBase CEP in Elektron, their open, global, ultra-high speed and resilient network for the financial industry. By adding StreamBase CEP to Elektron, Thomson Reuters is creating the industry’s only platform that includes comprehensive real-time and historical data, high capacity network connectivity, managed hosting services, and CEP for rapid application development and strategy deployment.

The First End-to-End Quant Platform for Back Testing, Algorithmic Development and Deployment

Although StreamBase is all about acting on what's happening now, the knowledge of what happened yesterday is essential for any automated system. History is essential to discover trends and patterns in event streams. History is essential to back test automated strategies. History is essential to gain perspective on real-time actions at execution time. Over 90% of applications that use StreamBase also use some form of historical event data management.

Thomson Reuters will integrate StreamBase with the Enterprise Platform for Velocity Analytics, which integrates all Thomson Reuters content, including tick history, reference data, corporate actions, condition code processing and cancellation and correction messages. It also brings together many disparate third party applications including data management solutions, EMS, OMS, price discovery, back testing, trade surveillance and low latency execution capabilities across the quantitative research and trading workflow.

By integrating Velocity Analytics with StreamBase CEP, Thomson Reuters has created the first end-to-end quant platform for back testing, algorithmic development and deployment. 

Thomson Reuters Applications "Powered By" StreamBase

Thomson Reuters will deliver a suite of applications that are “powered by” StreamBase CEP to address low-latency workflow. CEP inside will provide Thomson Reuters clients the benefit of time to market usually associated with turnkey solutions, while at the same time allowing full customizability of the underlying application. One application Thomson Reuters already provides is the real-time surveillance application from Redkite financial which is based on the StreamBase CEP platform. 

StreamBase and Thomson Reuters FX Solutions

The FX markets have rapidly emerged as an important market for the capabilities of CEP. Thomson Reuters announced that StreamBase CEP will complement the existing Thomson Reuters Dealing Aggregator FX trading solution, available exclusively to Thomson Reuters Dealing customers, by providing CEP capability in tandem with the low latency algo trading platform. 

Why did Thomson Reuters choose StreamBase? Mike Powell, Global Head of Enterprise Real-Time Solutions for Thomson Reuters, said it this way: “The financial market landscape continues to evolve rapidly, meaning that nimbleness and time to market have become essential for market participants. Elektron empowers our customers to innovate and achieve high performance through a seamless infrastructure that brings together global market participants, delivers world-class content and creates a suite of new added-value services. The combination of StreamBase’s market leading CEP engine together with Elektron will provide a platform for us to deliver flexible and sophisticated solutions to our customers faster and more efficiently.”

For StreamBase, we’re excited about the vast resources and capabilities of Thomson Reuters can provide our customers. We’re honored to be selected as Thomson Reuters’ CEP platform of choice - their selection validates our focused approach to CEP, the ever increasing importance of the cloud and real-time big data, the disruption of simplicity, and the continued need for low latency and scalability on Wall Street. 

 

StreamBase Loves Erlang

The philosophy of Erlang and StreamBase align quite well. Quoting Mike Williams, one of the three inventors of Erlang:

• Find the right methods—Design by Prototyping
• It is not good enough to have ideas, you must also be able to implement them and know they work
• Make mistakes on a small scale, not in a production project

These philosophies of rapid development also apply to the way we've designed StreamBase; additionally, our customers also demand that much less technical users participate in the rapid implementation of ideas. StreamBase has a visual development approach that allows less technical, more "business" users - with mathematics, analytics, and business domain expertise - use the same tools as professional software developers.

What if we could add Erlang developers to that party? Event-driven, visual development, coupled with the power and rapid development of Erlang? 

Well you can. Darach Ennis (@darachennis) shows us how in his recent presentation at Erlang UG in London. 

StreamBase has a graphical, compiler-oriented programming language, targeted at serious developers, that can also be used by the less technical. Erlang programmers, and the Erjang folks with whom we share a virtual machine, do their thing in Erlang, and that code becomes a graphical "operator" on the StreamBase Studio pallet. In Darach's example he's used Erlang to implement a MACD analytic, and integrate that into a StreamBase-based trading system so it operates in the event-driven CEP application. 

The combination of StreamBase and Erlang combines the best of both worlds: StreamBase for rapid, graphical development to develop low latency applications that process 100's of thousands of events a second. Combining "how," "why," and "under what conditions," real time streams are matched and processed. StreamBase allows business analysts, actuaries, quantitative contributors and developers to share a common language. An event-driven integration whiteboard, if you will, that compiles to efficient optimized native binary code. Add Erlang for development of analytics like MACD and you have a powerful combination. 

Erlang support will be contributed soon to the StreamBase Component Exchange (SBX), our public "app exchange" for CEP integrations, applications, frameworks, and sample code. You'll already find support for Java, C++, R, Matlab, python, .NET. If you want to plug in some cool analytics written in C++, just do it. You want to exploit Basho's Riak? Sure, plug it in.

As a result, StreamBase allows developers control over how they exploit the facilities at their disposal. Use our graphical DSL for flow oriented algorithms and real time low latency high frequency data management. It's high performance from a human productivity and systemic perspective. 

Learn more by watching Darach's presentation here, and follow him on Twitter at @darachennis.

 

On StreamBase 7, Its Breakthrough Usability, and Performance for CEP

Today we announced SteamBase 7, the next major version of our market-leading Complex Event Processing (CEP) software. In press and analyst pre-briefings last week, we discussed a lot of cool new features. Here’s a quick tour.

A HUGE STEP FORWARD IN CEP USABILITY

Chief among the innovations of StreamBase 7 is the revamped StreamBase Studio. The interface innovations of the iPhone, iPad, and modern operating systems has made great interface design expected. StreamBase has always been the best and easiest to use CEP platform. But with StreamBase 7, our engineering team has taken a huge leap forward in usability.

Drawn from the cutting edge consumer user experience in platforms like Apple iOS and Windows 7, Studio 7 dramatically increases the speed and quality of application innovation. 

To achieve this usability, StreamBase 7 has new code organizational features, integrated annotation features for English descriptions of functions, gorgeous diagram rendering techniques, and facilities to isolate and encapsulate the necessary complexity of a sophisticated application. Literally, StreamBase users can now read StreamBase application code at a glance, and organize it effectively so that the Business and IT can collaborate in a way never possible before with any CEP product. These features that make this all happen include:

1. A beautifully redesigned user experience, including the use of transparency (similar to Windows 7 Aero), overlays to communicate information at a glance (e.g., iOS number-in-red-circle), gradients rather than solid color to ease readability (think in terms of Windows 7 and iOS) and, and complete color customization (a la Windows 7 themes.) 
2. Support for custom modules (groups of application logic) and extension point styles.  Better control over modularity allows IT to manage large applications, and manage how and where business users or other groups can contribute code. For example, a risk domain expert can create risk checks in one module while IT focusses on connectivity, performance and integration in other modules.
3. Advanced graphical code layout, with new group layout features and code rendering controls
4. A new rendering look and feel to ensure code is easier to follow at a glance
5. New mouse and keyboard gestures to increase EventFlow coding productivity, such as on-canvas renaming of components
6. New hotkeys that give the ability to instantly create CEP language operators
7. Layout-alignment shortcuts

The result is that business users and IT can work in one environment, with one set of tools, with one set of code, dramatically improving time to market, quality and ability to evolve systems once they are deployed.

In recent years, StreamBase deployments have become bigger, more global, and more complicated. So StreamBase 7 improves usability throughout the entire software development lifecycle, including:

1. New code refactoring schemes. So developers can quickly identify changes in their application logic and adjust the rest of the application more quickly as modifications are made.
2. 400% faster application execution while debugging. So problems can be identified more quickly
3. The ability to attach Studio to a running StreamBase server. For better diagnostics in deployment.
4. A new XML editor. Allows operations to more effectively configure StreamBase deployments.
5. Dynamic extension points that allow users to configure StreamBase applications to be configured quickly at runtime. When a new algorithm or analytic is produced by a developer or business users, this code can be dynamically inserted into the StreamBase code, tested, and deployed.
6. Support for 64-bit workstations for very large StreamBase application development

PERFORMANCE IMPROVEMENTS IN FOUR DIMENSIONS

From the first release of StreamBase, the focus has been on performance. Previous SteamBase performance innovations include the StreamSQL compiler, the ability to visualize parallelism and scalability, data parallelism and dispatch, and our open architecture for highly available systems all combine to create the fastest, most scalable platform for ultra low latency applications.

Version 7 of StreamBase continues to break down CEP performance barriers in four dimensions: ultra low latency connectivity, predictability of latency, FIX performance, and with high-speed computing technologies.

1: MICROSECOND-LEVEL LATENCY

All CEP platforms have an architecture for absorbing and emitting events. Some run out of the main CEP engine process, which can add significant latency overhead to event processing. StreamBase already features an super-fast in-process adapter architecture, but goes further in StreamBase 7 with an optimized “zero-context-switch adapter architecture” which allows StreamBase applications to process messages in a single thread from the time it enters the engine, through StreamBase logic, to the time it leaves the engine.  This architecture allows messages to pass through the StreamBase engine in microseconds, and has been optimized to speed up CEP application performance.

2: PREDICTABILITY OF LATENCY

The predictability of performance has been improved for Streambase 7 applications, as users now have access to server configuration options that provide the user with fine-grain control of the tradeoffs between ultra-low-latency and throughput. And, overall platform improvements have yielded over 20% improvement in application performance across the board.

3: ULTRA LOW LATENCY FIX CONNECTIVITY

Many StreamBase customers in the capital markets is already known for its low latency connectivity architecture and version 7 comes with over 22 new connectivity options. Many of those connectivity options leverage FIX, and in StreamBase 7 FIX connectivity has been dramatically enhanced. In benchmark testing, the new StreamBase FIX engine has clocked in at 50-75 microseconds send time depending on message size, all while running on Linux and commodity hardware.

4: HIGH-SPEED COMPUTING AND GPUs FOR MANAGING BIG DATA AND NOSQL STORES

The state of the art of ultra-low-latency is rapidly expanding, and many StreamBase customers have been on the leading edge of applying new technologies to deal with big data in innovative ways. 

GPUs are being used to continuously produce metrics, factors, and parametrics, contribute them to memory data structures for ultra low latency streams of data to adapt in flight to compute accelerated algorithms. They are also used to easily integrate accelerated compute and messaging or to perform collaborate distributed event processing in a high level flow oriented language. 

To support this trend for high-speed computing with StreamBase, we have added several connectivity options in StreamBase 7 to make it easier for developers to connect StreamBase to their existing and next generation IT infrastructure, including hardware-based middleware for ultra-low latency operations, including enhanced low latency middleware connectivity such as Wombat, 29West, Solace, and Tervela.

REACTION TO STREAMBASE 7

Analyst Adam Honore’ said at the end of our briefing: “wow, you’ve been busy!” and he’s right - as customers build more sophisticated applications with StreamBase, with bigger demands from the business users, and bigger demands from IT, with the ever-increasing need for speed, StreamBase 7 extends our lead as the best-in-class CEP platform in the industry.

The benefit of bringing the business and IT together is faster time to market, with higher quality. With tight communication, the business and IT make mistakes faster, learn from them faster, and innovate faster. As Thomas Edison said: “I make more mistakes than anyone I know. And sooner or later, I patent them.” 

But my favorite anecdote comes from one of our customers - Chris Lees, vice president of SunGard’s global trading business. He put it this way: "The usability of StreamBase Studio helped us develop a new Valdi product, we used it to describe requirements collaboratively with IT and because we were working with the actual code, we made better architectural decisions early that made for a better first release.”

CEP Innovation Goes Global: Announcing the StreamBase Component Exchange (SBX)

As the use of StreamBase continues to expand, our customers have asked us to help them access other StreamBase developers in order to facilitate innovation and tap the power of the growing StreamBase community.

So today StreamBase announced the StreamBase Component Exchange (SBX) - the first community to allow developers to download and share reusable CEP components. SBX is open to individual developers, technology partners, and application partners that wish to share their StreamBase-based applications and components.

SBX will help push the CEP market to the next level by providing a global, open library of technical innovation, where ideas can be exchanged freely. The kinds of components that are available include:

Adapters. Adapters connect in-bound and out-bound event streams to the CEP platform. For example, the Twitter adapter allows StreamBase to consume and produce Twitter “tweet” streams. 

Integrations. Some components implement non-adapter integrations. For example, the Calling Python from StreamBase component allows for the integration of the Python programming language with StreamBase. 

Utilities. Utility components are small operators or functions that extend the CEP platform. For example, the Delay Operator component defers processing an event into the future.

Domain-specific algorithms. Domain-specific components illustrate the use of common industry-specific techniques. For example, the Options Pricing using Monte Carlo component is available to the financial services industry and demonstrates put and call options using a Monte Carlo method.

Third-party frameworks and applications. Third-party developed applications can also be contributed to the SBX to promote CEP-based products, CEP-based connectivity, and CEP application best practices and advanced techniques.

Developers can access SBX directly from StreamBase Studio (download here), or from the SBX website at sbx.streambase.com. 

Welcome to the SBX community!

On CEP Standards Confusion and Progress

(Richard Tibbetts is the Chief Architect for StreamBase, co-author of a recent paper with Oracle entitled Towards a Streaming SQL Standard, and StreamBase's representative to the Event Processing Technology Symposium, or EPTS).

There has been swift and strong response to the paper that Anand Srinivasan (from Oracle) and I presented last month at VLDB in Auckland. The paper, Towards a Streaming SQL Standard (download a PDF of the paper), seems to have touched a nerve in the community. All of a sudden, everyone is very excited about standards, and has an opinion to share. Some, such as Louis Lovas of Apama, perhaps concerned about being left behind, have been predictably negative and discouraging.  Opher Etzion from IBM shared a balanced, constructive point of view.  And then others have used this advancement as yet another excuse to re-hash the tired debate about "what is CEP?"  (we have provided no link to these discussions because the debate is pointless).

I'd like to address some of the mis-perceptions about this paper, because it's easy and deceptive to leap to the conclusion that "Oracle and StreamBase have proposed a new standard."   Instead, there are two key points to bear in mind when reading our work:

#1. This is an academic paper.  The goal of this paper is to advance standardization discussions by raising the processing model as a core element of any CEP standard. With event processing, when the results are produced can be as important as what the results are. That is, the same answer, produced later, is not the same answer. In SQL standardization, which deals with finite result sets, timing has not been a problem. In CEP however, it is a difference between implementations. The paper doesn't take a position on which existing approach is best. It merely states the problem. The paper doesn't propose a standard. It proposes one solution to one problem in standardization, as part of a broader effort.  We think this is an important observation, revealed by hard work and research.  We leave it to the community to decide if this observation is as important as we think it is.

#2.  Standardization is a very slow process. StreamBase is dedicated to standardization.  But standardization is a long process, and we've been working on it for almost 5 years.  And we will continue that commitment.  It will be years before we have a complete complex event processing language standard, with multiple conforming implementations.  There will be open standards bodies, and lots of work between groups of vendors to resolve their differences. In the early stages, it will likely be the smaller groups that make more progress.  Sure, we have a bias:  we think SQL is a great foundation to start with.  But we're happy to debate all approaches, and examine deeply the tradeoffs, just like we did with the Oracle research.

So StreamBase hasn't declared "victory" in standards; we are simply moving the discourse forward. I'm pleased that the community has responded strongly, because reaction means the debate has advanced.  We think thoughtful standards research is the best way to continue to promote standardization, and the rising tide of CEP, because a rising tide lifts all boats.  And we encourage all CEP vendors to do the same.

Getting Back to Use Cases and Patterns for CEP

Giles Nelson from Apama blogged today about how depressed he was to see a useless debate about acronyms, SOR, and CEP.  But unfortunately Giles threw the baby out with the bath water!   My initial post was about why CEP is used by Smart Order Routing applications, and that's a useful - no, critical - discussion.

When I ran Apama, the focus of my blog was use case discussion, patterns, requirements, best practices,  and descriptions of the references customers that illustrated CEP at work.  I think the community has lost this voice, and we need to bring it back.  There are a lot of good things happening at SteamBase, Apama, Aleri, Coral8, and other CEP vendors.  It's our job to educate, although I'm guilty of jumping into the bath water about the difference between Routers and SOR. 

So I'm going to RE-post the baby (the SOR use case discussion), and throw out the bath water (the Router debate).  I hope the blogosphere joins in on other use cases and positive discourse about how CEP is being used.

Towards a CEP Standard

There has been a lot of discussion in community groups, at conferences, and message boards over the years - back and forth about the best way to achieve progress on standards in the complex event processing (CEP) space.   This week StreamBase announced a partnership with Oracle - that the two companies have come together to research and publish on ways to reconcile a few of the approaches that have been developed for CEP.  The paper is being presented by our chief architect, Richard Tibbetts, at the VLDB conference this week.   We'll be blogging about this research in the future.   Here is the press release and a few links to coverage:

• Press Release: StreamBase and Oracle Partner on Complex Event Processing Standard Approach

• Banking Technology News coverage of the announcement

• A-Team news coverage

In the past perhaps StreamBase would have presented this news as further evidence that SQL has dominated the world of CEP as the no-brainer winner;  notice this is not the tenor of the release at all.  Instead, this research has taken years to develop, and the thoughful presentation of ideas in the paper is intended to provoke a discourse that can keep us on a path of convergence as CEP continues to become more and more important in the enterprise IT landscape. 

Stay tuned - more to come on this topic (post-script to this post, here is a post from Richard Tibbetts on the paper)!

StreamBase 6.0 Product Analysis and Thoughts on Leadership

June is a big month for CEP product releases, and the past few weeks have been no exception:  Aleri  announced version 3, Apama announced version 4, and StreamBase announced version 6.0.  Unsurprisingly, all these announcements contain proclamations of leadership.  Aleri's headline claimed a better look and feel, and Apama announced integrated sample applications and unsubstantiated performance claims.  By contrast, I think StreamBase announced an important breakthrough:  the first visual debugger, profiler, and unit testing environment in the industry. 

One of the main reasons I chose to join StreamBase is that in my time in the CEP industry, I've heard from customers, prospects, and industry insiders that the technology is simply superb:  it's fast, it's open, it's scalable (multi-threaded), and it has adopted a language metaphor (SQL) that is standards-based and familiar to most developers.  The visual environment uniquely enables business users and IT alike.  

I think release 6.0 extends that lead.  The ability to debug, test, and profile event processing applications is absolutely critical to ensure fast, reliable, and managable deployment in large enterprises, as well as adoption by smaller firms that need to do more with less. 

The release contains the following:  

1. StreamBase Visual Debugger - One of the hidden problems with some CEP platforms is that the visual tools were added as an afterthought, and, as a result, they generate code that's impossible to test, support and debug.  This is not the case with StreamBase, that started with a visual development tool from the very core, which has made it an incredibly powerful, integrated platform on which to debug and analyze applications, because there is no code generation - what you see is what you get and it's what you debug. 
2. Profiling - Profiling provides the ability to test, profile, and provision software and hardware resources associated with a CEP applications.  Once again, with StreamBase profiling, you profile and optimize the actual code you designed, so the result is a better, faster, more reliable application.
3. Unit Testing -  As the CEP market matures, larger and larger development groups are involved in CEP application development, which motivates the need for more sophisticated group development capability to test applications.  StreamBase Unit Testing allows teams to work in parallel, and allows teams to reduce application errors during development.  Unit testing also enables business developers to more fully participate in application development by providing IT the ability to test code supplied by business analysts before putting it into production,

The release of these features further demonstrates that CEP platforms are coming of age and ready for enterprise adoption and large, group development.  Yes these platforms are mature - today.

Event Processing Technical Society (EPTS): Thanks, Opher

The press release finally hit the wire today announcing the EPTS.  The goal is simple:  promote awareness about event processing technologies.  Immediate reaction has ensued, both negative and positive.   To me, having been involved in many of these discussions, I think any step forward is a good step.

This has taken about 2 years to formally announce this group. And, although the spirit of this has been to work as a community, and not single anyone out, there's no doubt that without a determined leader in the group nothing would have happened - from my point of view that's Opher Etzion - so I'm going to simplys link to Opher's blog, and his perspective, on this release as the best source for perspective.  I remember meeting Opher in New York at IBM for the first time about 3 years ago and discussing a vague vision of vendors, analysts, and academics all coming together to help resolve disputes and push a common understanding.

Here's hoping that the EPTS continues to advance this very important field.

Thanks, Opher.