The StreamBase Event Processing Blog

Tech Q&A with CEP Developers, and StreamBase

Today StreamBase hosted an online roundtable entitled “How to Empower Your Development Team with CEP and Visual Programming” featuring developers from J.P. Morgan, GAIN Capital, and Datasports. Combined, the panelists have 35+ years of development experience, and years of CEP experience, and they were kind enough to share those experiences with us, discussing the adoption of CEP technology, developing applications with CEP, and the best practices they’ve learned along the way.

For developers, development managers, architects, and business stakeholders this roundtable tells an interesting story about how CEP technology can truly transform the role of the developer. Take a look at some of the highlights below, watch the replay, then comment below and tell us about your experiences with CEP. 

Talk about your experience with CEP

Phil Martin, Datasports: I like using StreamBase because I like tools that let the computers do stuff that the computers are good at, and let me do stuff that requires thought. As a developer, you wind up doing some tasks over and over again; the structure of the application, domain specific plumbing, etc. Everytime feels like wasted effort. StreamBase lets me focus on the stuff that requires thought, not the details of implementation, connectivity, processing events, and so on. Another comment is around the aggregate operator. It is analogous to another API to learn. It’s an incredibly rich and flexible operator that I think almost qualifies as its own sub language or sub paradigm.  There’s a lot there and a lot of subtlety in its behavior, learn about and study it!

Mike Grandilone, J.P. Morgan Asset Management: The shift to a message passing structure is sometimes a shift in thinking. It's key to understanding how StreamBase handles the life of a tuple; we gained that through experience.  

Talk about the applications you've built with StreamBase

Joseph Chua, GAIN Capital: After 4 days of training and going through the examples, I built my first StreamBase application in just three hours, and it was fully deployed in a week. It connects to market data, manipulates the data, and stores it in a high-speed database. We've also built market making and real-time surveillence applications, and many more to come.

Phil: I've built several applications with StreamBase. One is an "indication of interest" (IOI) app that monitors trading interest and compares it against a CRM system and sends alerts to individual sales traders because they have a customer who either has traded on the symbol, transacted above a given threshold, or has a real-time order for it. The correlation of these factors in real-time also includes real-time news sources. Also, I've worked on two algorithmic trading systems that host their strategies in StreamBase and perform real-time risk checks on the algos, and an ETF arbitrage system where there was only one strategy, but it was a very involved strategy.

Mike: We brought a smart order routing system up and running in 6 months with 2 developers, which we thought was very fast. We also got the business people involved in the canvas and the development - this is a very important factor in getting applications done quickly.

Talk about the benefits of StreamBase and visual programming

Phil: StreamBase helps facilitate communication. Whenever you want to convey information about a system, you always draw a picture. And they almost always look like StreamBase Event Flow. If you have taken the necessary care to clearly delineate modules, then, depending on the technical sophistication of who you are talking to, you can choose how far to drill down. It's extremely helpful in faciliating technical discourse; with the business users, it saves time.

Mike: Visual development has really improved the speed of our development. Especially when you're doing a proof of concept. It helps you understand the application's logic, and also communicate with the business; in typical programming environments like Java, you're not going to show many people the code, but you can do it with StreamBase.

Joseph: The big value is that all the technical details are abstracted away in terms of connectivity and low-level code; we can then communicate with the business owners, so development cycles are shorter and more efficient. The other value we have found is they we can very quickly build a prototype solution so the business can make decisions more quickly with respect to requirements and decisions to develop the applications.

Phil: One thing that's helpful: with StreamBase, you can tell at a glance that spaghetti code is spaghetti code. You can tell if an application was thoughtfully designed without hours and hours and hours of code review.

What best practices have you learned from your experiences with CEP?

Mike: You need to shift your thinking, to think in terms of events, rather than traditional static applications. And, just like any application, build a testing environment at the same time as your application logic.

Joseph: Identify the connection points quickly, be open minded and willing to unlearn the more traditional ways of development, use StreamBase interfaces, and make sure you continuously refactor the application.

Phil: There are lot of resources for ramping up StreamBase skills - invest the effort to develop your skills. The speed of putting together something quickly is a great asset. The thing to keep in mind is that although the development speed of StreamBase is a benefit in the short term, it can, if you're not careful, work against you in the long term.

Phil: While the principles aren't different, the practice IS different. The first thing that sold me is seeing the ease with which you can change and tune your threading model, take things in and out of the threading part of your system, with multiplicity and dispatch methods. Scalability and performance is about taking more advantage of the cores in a system, and careful management of it becomes more and more important.

What was your experience integrating other code into CEP?

Mike: We have separate routing logic written in Java for routing logic that we integrated with our StreamBase application. We've done a lot of work with custom Java operators in StreamBase - it's very straight forward - no problem in doing it. There's lots of power in developing in Java for code that isn't event driven that allowed us to bring in the power of algorithmic trading as a drop-in to the StreamBase environment. Also, we have a C# front end GUI, which uses the .NET interface from StreamBase - this was easy, although the challenge was how to take a GUI that isn't event-based and make it work with an event-driven application. We're in production with that now.

The StreamBase community

Phil: I've put two components onto the StreamBase Component Exchange - mostly adapters. The publishing process was straightforward. One is an adapter that watches for events on the file system and creates StreamBase events when files are created, renamed, modified, or deleted, and another one listens for calls on a web service interface and creates events for the applciation. I suggest other developers look at the utility classes – see what are other developers are doing, and pick up some tips and techniques and helpful hands-on examples of how they are building their applications. 

StreamBase Resources for Developers

Below are the StreamBase resources mentioned in this roundtable:

• Trial version of StreamBase: www.streambase.com/download
• Product documentation: docs.streambase.com
• Personalized demonstrations: www.streambase.com/forms/request-a-demonstration
• “How To” developer webinars: www.streambase.com/webinars
• StreamBase Component Exchange: www.streambase.com/sbx
• StreamBase University: www.streambase.com/sbu
• StreamBase Training and Certification: www.streambase.com/community/training-and-certification
• Datasports blog for StreamBase CEP developers: http://datasports.ca

Introducing Live BI and StreamBase LiveView

For some time now, StreamBase has been working on a new product concept that's a little bit CEP, a little bit data warehouse, and a little bit business intelligence. Today we've unveiled it as StreamBase LiveView. Our customers have called it a "game changer," and a product that "fills a massive hole in the BI space." Hyperbole like that needs to be backed up, so here we go.

Complex event processing (introduction on Google+) was envisioned by its inventors as a "database for moving data" as firms increasingly deal with data in motion from mobile devices, social media and digitized data of all types, and in all industries. In its first decade of commercial use, CEP has come to power automated trading on Wall Street. A who's who list of firms have been public about their adoption of CEP, so let's examine the live BI problem in the context of Wall Street.

Industry analysts estimate that over 60% of U.S. equities trading is automated. The problem is, with so much automation, it's impossible to see and act on what's going on, because a typical trading environment can generate 10's of millions of events a day. Existing business intelligence and data warehouse tools aren't used in the front office of Wall Street, because they're designed to summarize what's already happened, not what's happening now. 

StreamBase LiveView brings BI and streaming data together, for the first time. Applications that need Live BI are everywhere on Wall Street - trading risk management, position management, P&L management, surveillance, compliance, and trading itself, as Waters Technology magazine describes in the article that StreamBase Seeks 'Radical Transparancy' in Business Intelligence.

And applications in other markets can be found in any environment that has big data in motion: telecom (location aware insight and live network operations), security (live fraud detection, live surveillance), and e-commerce (live social media analytics, ad placement, big streaming analytics for Hadoop and Flume). These apps can be deployed on premise or distributed in the cloud. 

StreamBase LiveView is built with StreamBase CEP, which provides connectivity to over 125 event-based sources of data and the extreme scalability required by live BI. The server-based architecture is designed to process hundreds of millions of events per node, absorbed from a network of event sources such as an enterprise message bus or scalable event absorption layer like Flume or IBM Streams. At the front end, users can slice and dice streams of data with ad-hoc query tools in the StreamBase LiveView Desktop (shown above). But here's the key: once users ask the questions they want answered, StreamBase LiveView keeps answering their questions - as conditions change in real-time; users see an always-live view of streaming data.

For the user, all this technology means just one thing: they see business intelligence that's constantly fresh and relevant to their business operations. It's not a view of 5 days ago, or even 5 minutes ago. The data is live, the insight is live, and live insight can lead to live action.

Facebook, Twitter, Google+ give us a live view of what's happening in our social networks; StreamBase LiveVew gives firms a live view of their enterprise data.

To learn more, watch the StreamBase LiveView presentation, follow new materials, articles, and resources on Google+ (+StreamBase and CEO +Mark Palmer), read industry analyst Curt Monash, DMBS2.0, or follow StreamBase Systems (@streambase), CEO Mark Palmer (@mrkwpalmer ), and CTO Richard Tibbetts (@tibbets) on Twitter.

 

Can Innovative Surveillance Be a New Source of Competitive Edge?

In sports, great defense can be a catalyst for great offense. Can the same idea hold true for surveillance - the defense - in the capital markets? This was a question raised by today's panel of surveillance experts at today's StreamBase webinar (watch the replay). 

First, TABB Group's Miranda Mizan assessed the current surveillance market. She described how surveillance is morphing from a defensive obligation to an offensive differentiator, because the markets are changing so dramatically. Mizan argues that the quantity of data, speed of trading, and complexity of instruments have made traditional surveillance techniques insufficient. And since regulation (e.g., MIFID II) is changing so quickly, she argues that firms must re-think their surveillance strategies and technology choices.

Justin Amos joined Miranda. Justin is a former global markets sales and trading executive at JPMorgan, and the MD of Redkite Financial Systems, a UK-based firm whose system called Redeye has been quickly becoming the standard for modern, real-time surveillance systems. Justin described the requirements of today's surveillance systems by describing the motivation for Redeye, which he likened to "anti-virus software for trading."

Together, Justin and Miranda touched on 12 "next generation" surveillance requirements, including:

1. Real-time analytics. Surveillance systems must analyze market data at the speed of automated trading.
2. Historical "big data" analytics. On-line access to trade-by-trade history for investigation, trending, and discovery.
3. Data fusion. Mash-ups of increasingly complex data, including unstructured data.
4. Inclusion of new data sources. The ability to integrate live data, static data, and social media data.
5. Multi-asset class support. Modern surveillance must work across asset classes.
6. Pre-built surveillance algorithms. Standard algorithms like front running, spiking, tailgating, and ramping.
7. Adaptive algorithm modification. The ability to adapt algorithms on the fly to reduce false positives.
8. Alert algorithm extensibility. The ability to quickly add new algorithms to the system as market dynamics evolve
9. Customizable surveillance algorithms. The ability for a firm to tweak an algorithm for their own models.
10. Flexible workflow management and case management. Control over the process of alert analysis, resolution, and tracking.
11. Accessibility for business analyst. With regulation changing so quickly, more firms want their own control over surveillance behavior.
12. Cloud-based or on-premise deployment. The cloud allows more rapid rollout, and may ease global scalability challenges

The panel argued that existing surveillance systems can't handle these requirements. Taken together, the challenge makes what was once traditional "defense," or a compliance function, very new and challenging - a source of differentiation.

Miranda went deeper, with examples of how brokers and the buy side are using modernized surveillance systems: agency brokers are beginning to use surveillance tools to monitor their trading algorithms and optimize their client service and order routing for best execution; the buy side are using surveillance to gain greater control to know that their trading is safe, its orders get to the right places, and that dark pools are working correctly in relation to the rest of the market; sell side institutions are using surveillance tools to institute process and control across the organization, globally, and integrated with automated systems.

Richard Tibbetts, CTO of StreamBase, gave a brief introduction of StreamBase CEP, and why complex event processing is increasingly being used to power surveillance applications; indeed, CEP provides the engine to address many of the 12 modern surveillance requirements set out by the panel.

Simon Townsend, CIO of Redkite, gave a "whistle stop tour" around Redkites' application. He hit on most of the 12 requirements above, and even showed didn't just claim that their algorithms are extensible and dynamic (requirements 6, 8 and 9) - he showed it in StreamBase Studio, and showed how business analysts can extend the Redeye surveillance algorithms.

One of the most exciting times is when a market is transformed by new ideas. In the capital markets, perhaps that innovation is upon us in surveillance, where tools that meet modern trading requirements can turn a defensive obligation into a tool for innovation.

StreamBase Loves Erlang

The philosophy of Erlang and StreamBase align quite well. Quoting Mike Williams, one of the three inventors of Erlang:

• Find the right methods—Design by Prototyping
• It is not good enough to have ideas, you must also be able to implement them and know they work
• Make mistakes on a small scale, not in a production project

These philosophies of rapid development also apply to the way we've designed StreamBase; additionally, our customers also demand that much less technical users participate in the rapid implementation of ideas. StreamBase has a visual development approach that allows less technical, more "business" users - with mathematics, analytics, and business domain expertise - use the same tools as professional software developers.

What if we could add Erlang developers to that party? Event-driven, visual development, coupled with the power and rapid development of Erlang? 

Well you can. Darach Ennis (@darachennis) shows us how in his recent presentation at Erlang UG in London. 

StreamBase has a graphical, compiler-oriented programming language, targeted at serious developers, that can also be used by the less technical. Erlang programmers, and the Erjang folks with whom we share a virtual machine, do their thing in Erlang, and that code becomes a graphical "operator" on the StreamBase Studio pallet. In Darach's example he's used Erlang to implement a MACD analytic, and integrate that into a StreamBase-based trading system so it operates in the event-driven CEP application. 

The combination of StreamBase and Erlang combines the best of both worlds: StreamBase for rapid, graphical development to develop low latency applications that process 100's of thousands of events a second. Combining "how," "why," and "under what conditions," real time streams are matched and processed. StreamBase allows business analysts, actuaries, quantitative contributors and developers to share a common language. An event-driven integration whiteboard, if you will, that compiles to efficient optimized native binary code. Add Erlang for development of analytics like MACD and you have a powerful combination. 

Erlang support will be contributed soon to the StreamBase Component Exchange (SBX), our public "app exchange" for CEP integrations, applications, frameworks, and sample code. You'll already find support for Java, C++, R, Matlab, python, .NET. If you want to plug in some cool analytics written in C++, just do it. You want to exploit Basho's Riak? Sure, plug it in.

As a result, StreamBase allows developers control over how they exploit the facilities at their disposal. Use our graphical DSL for flow oriented algorithms and real time low latency high frequency data management. It's high performance from a human productivity and systemic perspective. 

Learn more by watching Darach's presentation here, and follow him on Twitter at @darachennis.

 

On StreamBase 7, Its Breakthrough Usability, and Performance for CEP

Today we announced SteamBase 7, the next major version of our market-leading Complex Event Processing (CEP) software. In press and analyst pre-briefings last week, we discussed a lot of cool new features. Here’s a quick tour.

A HUGE STEP FORWARD IN CEP USABILITY

Chief among the innovations of StreamBase 7 is the revamped StreamBase Studio. The interface innovations of the iPhone, iPad, and modern operating systems has made great interface design expected. StreamBase has always been the best and easiest to use CEP platform. But with StreamBase 7, our engineering team has taken a huge leap forward in usability.

Drawn from the cutting edge consumer user experience in platforms like Apple iOS and Windows 7, Studio 7 dramatically increases the speed and quality of application innovation. 

To achieve this usability, StreamBase 7 has new code organizational features, integrated annotation features for English descriptions of functions, gorgeous diagram rendering techniques, and facilities to isolate and encapsulate the necessary complexity of a sophisticated application. Literally, StreamBase users can now read StreamBase application code at a glance, and organize it effectively so that the Business and IT can collaborate in a way never possible before with any CEP product. These features that make this all happen include:

1. A beautifully redesigned user experience, including the use of transparency (similar to Windows 7 Aero), overlays to communicate information at a glance (e.g., iOS number-in-red-circle), gradients rather than solid color to ease readability (think in terms of Windows 7 and iOS) and, and complete color customization (a la Windows 7 themes.) 
2. Support for custom modules (groups of application logic) and extension point styles.  Better control over modularity allows IT to manage large applications, and manage how and where business users or other groups can contribute code. For example, a risk domain expert can create risk checks in one module while IT focusses on connectivity, performance and integration in other modules.
3. Advanced graphical code layout, with new group layout features and code rendering controls
4. A new rendering look and feel to ensure code is easier to follow at a glance
5. New mouse and keyboard gestures to increase EventFlow coding productivity, such as on-canvas renaming of components
6. New hotkeys that give the ability to instantly create CEP language operators
7. Layout-alignment shortcuts

The result is that business users and IT can work in one environment, with one set of tools, with one set of code, dramatically improving time to market, quality and ability to evolve systems once they are deployed.

In recent years, StreamBase deployments have become bigger, more global, and more complicated. So StreamBase 7 improves usability throughout the entire software development lifecycle, including:

1. New code refactoring schemes. So developers can quickly identify changes in their application logic and adjust the rest of the application more quickly as modifications are made.
2. 400% faster application execution while debugging. So problems can be identified more quickly
3. The ability to attach Studio to a running StreamBase server. For better diagnostics in deployment.
4. A new XML editor. Allows operations to more effectively configure StreamBase deployments.
5. Dynamic extension points that allow users to configure StreamBase applications to be configured quickly at runtime. When a new algorithm or analytic is produced by a developer or business users, this code can be dynamically inserted into the StreamBase code, tested, and deployed.
6. Support for 64-bit workstations for very large StreamBase application development

PERFORMANCE IMPROVEMENTS IN FOUR DIMENSIONS

From the first release of StreamBase, the focus has been on performance. Previous SteamBase performance innovations include the StreamSQL compiler, the ability to visualize parallelism and scalability, data parallelism and dispatch, and our open architecture for highly available systems all combine to create the fastest, most scalable platform for ultra low latency applications.

Version 7 of StreamBase continues to break down CEP performance barriers in four dimensions: ultra low latency connectivity, predictability of latency, FIX performance, and with high-speed computing technologies.

1: MICROSECOND-LEVEL LATENCY

All CEP platforms have an architecture for absorbing and emitting events. Some run out of the main CEP engine process, which can add significant latency overhead to event processing. StreamBase already features an super-fast in-process adapter architecture, but goes further in StreamBase 7 with an optimized “zero-context-switch adapter architecture” which allows StreamBase applications to process messages in a single thread from the time it enters the engine, through StreamBase logic, to the time it leaves the engine.  This architecture allows messages to pass through the StreamBase engine in microseconds, and has been optimized to speed up CEP application performance.

2: PREDICTABILITY OF LATENCY

The predictability of performance has been improved for Streambase 7 applications, as users now have access to server configuration options that provide the user with fine-grain control of the tradeoffs between ultra-low-latency and throughput. And, overall platform improvements have yielded over 20% improvement in application performance across the board.

3: ULTRA LOW LATENCY FIX CONNECTIVITY

Many StreamBase customers in the capital markets is already known for its low latency connectivity architecture and version 7 comes with over 22 new connectivity options. Many of those connectivity options leverage FIX, and in StreamBase 7 FIX connectivity has been dramatically enhanced. In benchmark testing, the new StreamBase FIX engine has clocked in at 50-75 microseconds send time depending on message size, all while running on Linux and commodity hardware.

4: HIGH-SPEED COMPUTING AND GPUs FOR MANAGING BIG DATA AND NOSQL STORES

The state of the art of ultra-low-latency is rapidly expanding, and many StreamBase customers have been on the leading edge of applying new technologies to deal with big data in innovative ways. 

GPUs are being used to continuously produce metrics, factors, and parametrics, contribute them to memory data structures for ultra low latency streams of data to adapt in flight to compute accelerated algorithms. They are also used to easily integrate accelerated compute and messaging or to perform collaborate distributed event processing in a high level flow oriented language. 

To support this trend for high-speed computing with StreamBase, we have added several connectivity options in StreamBase 7 to make it easier for developers to connect StreamBase to their existing and next generation IT infrastructure, including hardware-based middleware for ultra-low latency operations, including enhanced low latency middleware connectivity such as Wombat, 29West, Solace, and Tervela.

REACTION TO STREAMBASE 7

Analyst Adam Honore’ said at the end of our briefing: “wow, you’ve been busy!” and he’s right - as customers build more sophisticated applications with StreamBase, with bigger demands from the business users, and bigger demands from IT, with the ever-increasing need for speed, StreamBase 7 extends our lead as the best-in-class CEP platform in the industry.

The benefit of bringing the business and IT together is faster time to market, with higher quality. With tight communication, the business and IT make mistakes faster, learn from them faster, and innovate faster. As Thomas Edison said: “I make more mistakes than anyone I know. And sooner or later, I patent them.” 

But my favorite anecdote comes from one of our customers - Chris Lees, vice president of SunGard’s global trading business. He put it this way: "The usability of StreamBase Studio helped us develop a new Valdi product, we used it to describe requirements collaboratively with IT and because we were working with the actual code, we made better architectural decisions early that made for a better first release.”

Build a CEP App in 46 Seconds

It's frustrating to constantly tell customers and prospects how great StreamBase is at rapid development with hackneyed phrases like "you can build applications in weeks or days, not months or years," or "a major tenet of what makes StreamBase different is ease of use, and we're the only CEP in the industry that does true visual language for serious application development." 

So over the coming months, you'll be seeing more from StreamBase *showing* the power of visual programing, StreamBase, and CEP. More tutorials. More product features. More customer stories.

For example, tomorrow we're hosting a webinar that includes a 10-minute tutorial on how to build a foreign exchange dealing application with StreamBase. Even if you don't know anything about trading or the capital markets, it's a great introduction to StreamBase, and to CEP. Want a quick preview? Here's the tutorial in 46 seconds. Check it out:

 

To attend the webinar tomorrow, Thursday, March 24 at 10am ET, 4pm UK, register here.

The Flash Crash & the Role of Financial Software Engineering

Even though the flash crash happened seven months ago, the debate rages on. And since CEP is a common piece of our modern financial computing fabric many firms have entered the debate with vested interest. Unfortunately, most vendors wrap flash crash observations with veiled sales pitches for CEP-based risk management, or disingenuous calls to "catch the rogues and scoundrels" with CEP-based surveillance systems. Although these applications are important, to suggest they are a magical elixir to ward off future breakdowns is misleading at best. 

But thoughtful discourse does continue, and the recent Securities and Exchange Commission report has reinvigorated debate. Richard Tibbetts, StreamBase CTO, weighed in with a piece called "What Does the Flash Crash Mean For Financial Software Engineering?" on Tabb Forum this week. Richard's view on the flash crash examines the role of financial software engineering and the efficacy of data processing, integrity, and reliability within the fabric of the infrastructure. The role of financial engineering is an essential, overlooked, and misunderstood piece of the puzzle. Richard's thoughts go like this: 

"In all the conversation about the crash, the real historic change is going unmentioned. In the very last paragraph of the Lessons Learned section of the report, the authors note: “another area of focus going forward should be on the integrity and reliability of market centers’ data processes, especially those that involve the publication of trades and quotes to the consolidated market data feeds.”

This is a key insight – late in coming – that will expand over time. Traditionally the SEC has focused on regulating the behavior of market participants, identifying deliberately malicious behavior or patterns of behavior. But now they are beginning to realize that data processing, integrity, and reliability can be just as important. The efficient functioning of markets is in just as much danger from software system failures as it is from malicious traders..."

The role of financial engineering is much broader than CEP, and deserves deeper discourse. To read more, continue reading Richard's piece on Tabb Forum (which requires a free registration) here, and watch for more over the coming months on Tabb Forum, on this blog, and on Richard's blog on software engineering and innovation, Hyperextended Metaphor.

Gain Capital's Selection of CEP and "Whiteboard Programming"

In the late 90's, visual programming was The Next Big Thing. Instead of writing millions and millions of lines of incomprehensible computer instructions, visual programming tools promised a world where you could "paint" applications using high-level objects. Visual programming promised a future where applications would be built more quickly, more easily, with result in an application that would be less expensive to maintain. Unfortunately that era of visual programming did not deliver on many of it's promises. Tools were primitive, workstations weren't up to the task, many potential users didn't know why they would want to write programs, and the languages didn't line up well with the use cases. In the end, the tool with the most staying power was UML, which is a modeling language rather than a programming language, and which used traditional textual programming languages to create the "real" program. UML became fodder for big-project architecture and requirements documents, not a tool for real programmers.

Last week I was reminded that I'm fortunate to be running a company that delivers on the promise of visual programming. This struck me as I was listening to press interviews with Andrew Haines, the CTO of Gain Capital. Andrew discussed the reasons behind their selection of StreamBase. He talked about how they had evaluated all the major CEP vendors and selected StreamBase for its speed, ease of use, and our being a good partner. And he talked about how strategic their use is - Gain processes almost $200 billion in trades a month. All good stuff.

But the comment that struck me was when Andrew said Gain has found StreamBase Event Flow programming language to be 50% faster than traditional programming techniques (citing an experiment they did that measured StreamBase versus .NET). This comment reminded me of the 90's and the promise of visual programming, or what we sometimes call "Whiteboard Programming." Whiteboard programming (watch a short video here) is the right visual metaphor for event processing because developers design event-driven applications naturally on a whiteboard, with data flows, icons for business logic, and arcs that connect logic to next part of their event driven architecture. StreamBase's visual development tools allow developers to "paint" their application. But the visual representation of their whiteboard isn't just a picture of the application, it is the application. The application is tested, profiled, and debugged visually, and the promise of visual programming, for event-based applications, is fulfilled.

StreamBase EventFlow benefits from several improvements in the state of the art since the 90s, as well as our own innovation. First, it's built into a modern IDE based on the Eclipse platform, already a powerful and flexible tool before StreamBase starts with it. StreamBase also takes many of its ideas from the agile programming methodology developed over the last 15 years, including automated refactoring, unit testing, and a focus on quick iterations. More specific to StreamBase, the StreamSQL Event Flow language derives from declarative data processing languages developed at MIT, Brown, Brandeis, and Stanford universities, with rigorous models behind them. The language is designed to not only be visual, but also fundamentally to be directly executable (unlike UML) and to have powerful facilities for extensibility and abstraction (unlike end-user oriented visual languages). The result is a language that is accessible to financial engineers, quantitative analysts, and business analysts, but powerful enough that application and infrastructure developers are comfortable working on the same platform. Despite the ability to add functionality to the platform in many traditional languages (Java, C++, .NET, Python, textual SQL), StreamBase developers universally adopt the visual programming model for it's productivity and teamwork benefits.

Andrew's comment reminded me why StreamBase is on a run of amazing accolades as one of the most innovative technologies in the world, including Time Magazine, the World Economic Forum Technology Pioneer program, and MIT Technology Review. These esteemed technologies realize that, as Clayton Christensen suggests in The Innovator’s Dilemma, simplicity is always a disruptive innovation.  And the proof lies in clients like Gain Capital, who get to market 50% faster, with 50% less cost, with 50% less cost of ownership over time. 

But at the same time, other CEP venders aren't gaining customer success stories like Gain Capital. Why not? Gain evaluated numerous other CEP products from Progress Software, Aleri, and TIBCO as reported by FX Week. Coral8 was acquired by Aleri, then Aleri was acquired by Sybase. After the acquisition, Aleri engineers revealed that although their product was fast, customer implementations took years and lots of consulting to build. Apama, the early pioneer in CEP, tried to develop a visual programming tool after they developed a textual language. The textual language works, but the visual one doesn't, which is why Apama's visual tool is being scrapped as they integrate the CEP language with Savvion's visual BPM development environment.

What Gain Capital's selection of StreamBase reminds us is that simplicity is always disruptive, and that the promise of visual programming is just as relevant today as it was 10 years ago, and being fulfilled by the leading CEP companies.

Why the CEP Tide Has Turned Toward StreamBase

Last week, CMC Markets announced that they switched from Apama to StreamBase as their enterprise-wide event processing platform. As FXWeek reported, CMC “initially spent three months deploying a similar system from Apama, but dropped the deal in favor of StreamBase.” As a firm that trades over $1.4 trillion a year, CMC’s decisions about their infrastructure are not taken lightly. So their decision gives us pause to ask: why the switch? 

Asif Adatia, Chief Information Officer for CMC, described their decision like this: “the ease of development for both financial engineers and technical engineers, the performance, and scalability of the platform were our key criteria.” Let’s examine each part of that statement.

1. Demonstrably better ease of use. You might think that the efficacy of a development environment would be a subjective call. In some ways, it is - when it comes to the visual esthetics of a tool. On the other hand, usability is often the driver for market-altering innovation - profound shifts in the accessibility of a new tool can suddenly unlock barriers to innovation. The change in the last 10 years in finance has been profound. 10 years ago, algorithmic trading wasn't even named, now 70% of the U.S. equities trade execution happens algorithmically. In their report "The World According to Quants: Enter Alpha Generation Platforms", the AITE group reported that certain trading algorithms have a shelf-life of less than 3 months. In the face of all that rapid change and ephemeral opportunity, the ability to quickly build real-time trading infrastructure (algos, execution, risk, compliance, pricing, TCA, surveillance) is a matter of survival.

Firms like CMC, CME Group, and RBC didn't choose StreamBase for its aesthetics, they chose StreamBase because its graphical development environment gives them a fundamental competitive edge. The tools aren't just easy to use, they allow business users and engineers to work together because the language of StreamBase matches the way they think. The StreamBase dataflow language was designed from the beginning to program event-based systems the way you would draw them on a whiteboard, and the proof lies in what customers say, as Adatia told FX week, “Our financial engineers find it really easy to put the models together.” 

And this ease of use delivers another critical innovation: firms get this usability and time to market advantage and the performance benefits of event processing - at the same time. StreamBase isn't like other CEP tools that make the developer resort to low-level programming to make the system perform and scale. Relieving the user from the burden of thinking about plumbing was described by Adatia to A-Team Insight “[CMC] create their models readily, without resorting to code-level programming.” PhaseCapital reported that they built their trading system with StreamBase in 4 months rather than the 4 years it would take to build it with traditional development tools and they change their strategies every day. These are not subjective opinions about aesthetics, they are qualitative and quantitative testimonials to the innovation that StreamBase's interface can provide.

2. Built-in performance you can see. On Wall Street, milliseconds, and microseconds matter. CMC switched to StreamBase because they wanted the fastest engine they could get. 

StreamBase was designed for speed from the very beginning. Release 1.0 in 2005, was designed to take advantage of modern hardware, from the multi-threaded server to the StreamBase language which produces highly performance applications without burdening the developer. The StreamBase language plans performance like a database pre-processor plans query optimization. In 2007, StreamBase engineers broke down another barrier with a straight-through event processing compiler. Recently, in 2008, direct bytecode generation optimization took the StreamBase performance innovation even further ahead of all other CEP platforms.

When performance is something you can see, it's a remarkable result. CMC said it this way: '[we don't talk about our specific latency numbers publically, but the performance is] good though and you can compare us on FX pricing to a bunch of our competitors.  And you will see our performance is noticeably better at times. Absolutely."  The world's largest electronic futures and options exchange in the world, CME Group chose StreamBase after using performance as a critical selection criteria, because they process over 11 million trades a day.

3. Scalability to grow. You might assume better scalability means StreamBase-based applications can be run on multiple CPU cores, and that’s true - better use of hardware means CMC needs less hardware. CMC described that their StreamBase system easily handles "More than twice the real-life peak rate for price ticks encountered so far.”  

But better scalability isn’t only about increasing volumes - it’s about ease of management, ease of customer support, and the ease with which high performance and scalability can be attained by developers.

And scalability is also about expanding usage. Apama established early on algorithmic trading as an important use case for CEP. But now, as CEP's use explodes into new application areas, firms are looking more deeply at how easy it is to build their second, third, and fourth applications. Just last week we had a call with a customer who guessed they had deployed "about 100 StreamBase applications." 

And now we have come full circle back to the StreamBase development environment. Our mantra is to “deliver high performance and scalable applications without having the developer think about performance and scalability.” Streambase makes performance native, which is one more reason it is unique.  

CMC is not alone. Organizations that are adopting StreamBase CEP are seeing these same benefits in ease of use and performance. CEP is the disruptive technology that frees their engineers to focus on creating business value instead of plumbing. CEP empowers their business analysts to work with the actual software instead of wishlists and design documents. CEP allows the whole organization to be more agile, to deploy changes to their infrastructure faster, and to take advantage of new opportunities ahead of the competition.

StreamBase and Scala

As the ponytail (and the tiny bio below) imply, my posts will focus on technical details such as language semantics, general programming issues, compiler design, or anything else I feel is nifty.

Anyone who follows programming languages will probably have noticed Scala cropping up a lot lately.  Scala is an interesting language on several fronts, but the one that sparked my interest was its strong interoperability with Java.  Often times academics forget that to be useful a language needs to integrate with existing code and libraries.  But Scala has clearly been designed with heavy focus on industrial usability as well as academic power.  When I read about Scala on Android, I guessed it wouldn't be hard to integrate with StreamBase.  It turns out I was mostly right (and we will definitely be fixing the small pain points I hit).  What follows is an guide to embedding Scala into a StreamBase application.

First we have to set up StreamBase Studio to support Scala development.

1. Install StreamBase 6.4
2. Add the Scala 2.7.5 Eclipse Plugin to StreamBase Studio (Help -> Software Updates -> "Add Site...")
3. Manually download the AJDT Weaving Plugin 1.6.4
4. Unzip the weaving plugin into your studio install
   
   • make sure the contents of the zip's "plugins" directory goes into Studio's "plugins" directory (I accidentally got that wrong the first time)
   • Update 8/31/2009:  On Windows unzipping directly into a Studio install fails with an obscure error message.  Instead, you should simply unzip into an empty directory and copy the contents of the created "plugins" and "features" directories into Studio's respective subdirectories of "C:\Program Files\StreamBase Systems\StreamBase.6.4\lib\studio". (You may need to correct that path if your studio install is in a non-standard location.)
5. Restart Studio
6. Enable "JDT Weaving" (Preferences -> JDT Weaving -> "Click to ENABLE")
7. Restart Studio
8. Verify that "JDT Weaving" is enabled (Preferences -> JDT Weaving)

Right there you see 95% of the trouble I had.  Definitely easy to do, once you know what to do, but figuring out that JDT Weaving thing (especially the specific version needed) took some searching.

Now we can get started using Scala in StreamBase.  The impatient can

1. Download the sample
2. Import it (File -> Import ... -> Existing Projects into Workspace -> "Select archive file").

or the dedicated can create a new project from scratch

1. Create a new StreamBase project (Package Explorer -> Right Click -> New -> "StreamBase Project")
2. Add the Scala Nature to it (Right Click on the Project -> Scala -> "Add Scala Nature")

Yeah that is really it.  Turns out that Eclipse does some clever and powerful things by having projects include natures like this.  We now have a project that includes the Java, Scala, and StreamBase natures and they all play nicely together.

Let's create a basic Java Operator so we have something with which to compare (File -> New -> "StreamBase Java Operator").  This auto-generated operator is good enough for your purposes.  I ended up deleting all the comments and variables and what not, so I could strip it down as small as possible for the web.

package com.streambase.sample;

import com.streambase.sb.StreamBaseException;
import com.streambase.sb.Tuple;
import com.streambase.sb.operator.Operator;
import com.streambase.sb.operator.Parameterizable;
import com.streambase.sb.operator.TypecheckException;

public class SampleJavaOperator extends Operator implements Parameterizable {
    public SampleJavaOperator() {
        setPortHints(1, 1);
        setDisplayName(this.getClass().getSimpleName());
        setShortDisplayName(this.getClass().getSimpleName());
    }

    public void typecheck() throws TypecheckException {
        requireInputPortCount(1);
        setOutputSchema(0, getInputSchema(0));
    }

    public void processTuple(int inputPort, Tuple tuple) throws StreamBaseException {
        sendOutput(0, tuple);
    }
}

Let's try a naive translation into Scala.

package com.streambase.sample;

import com.streambase.sb.StreamBaseException;
import com.streambase.sb.Tuple;
import com.streambase.sb.operator.Operator;
import com.streambase.sb.operator.Parameterizable;
import com.streambase.sb.operator.TypecheckException;

class SampleScalaOperator extends Operator with Parameterizable {
    def SampleScalaOperator = { 
        setPortHints(1, 1);
        setDisplayName(this.getClass().getSimpleName());
        setShortDisplayName(this.getClass().getSimpleName());
    }
 
    override def typecheck = {
        requireInputPortCount(1);
        setOutputSchema(0, getInputSchema(0));
    }
 
    override def processTuple(port:Int, tuple:Tuple) = {
        sendOutput(0, tuple);
    }
}

Wow that wasn't too hard, of course it didn't save us any code either.  But it does give us enough to test and figure out if it works at all.  I guess we need to make a simple sbapp to exercise these Operators we have just written.

Yeah, that was easy.  Alright, you caught me, I cheated.  Studio does not (yet) automatically find operators written in Scala, so I had to hand edit the sbapp file.  But everything works after that, this app can be run and everything works as you would expect.

Now for a slightly less trivial example.  I hear that Scala has a really nice concurrency primitive called an actor.  Well, StreamBase has a multi-threaded architecture so let's see if the two can be made to play nicely together.  Once again here is a full project that you can import as above.

class SampleScalaOperator extends Operator with Parameterizable {
    private object coordinator extends Actor {
        def act = {
            loop { 
                receive { 
                    case t: Tuple => sendOutput(0, t)
                    case None => exit()
                }
            }
        }
    }
 
    def SampleScalaOperator = { 
        setPortHints(1, 1)
        val name = getClass.getSimpleName
        setDisplayName(name)
        setShortDisplayName(name)
    }
 
    override def typecheck = {
        requireInputPortCount(1)
        setOutputSchema(0, getInputSchema(0))
    }
 
    override def processTuple(port:Int, t:Tuple) = {
        actor {
            val times = t.getInt("times")
            val delays = t.getLong("delays")
            for (i <- 0 until times) {
                Thread.sleep(delays)
                coordinator ! t 
            }
        }
    }
 
    override def init = { coordinator.start } 
    override def shutdown = { coordinator ! None }
}

Don't forget to edit the input schema for your app to include an int field named "times" and a long field named "delays".  Now we can run the app, enqueue tuples, and watch the delayed tuples arriving out of order.  We could have simply called "sendOutput" from the actor in "processTuple", but I wanted to include actors sending messages to other actors instead of just being glorified threads.

All in all, really solid work on the part of the Scala developers.  I am definitely looking forward to trying Scala on some larger projects.  Hopefully, I will be able share more as I learn it.

(Matt Fowles is a Senior Software Engineer for StreamBase, whose interests include programming language, compiler, and virtual machine design.)