The StreamBase Event Processing Blog

Tech Q&A with CEP Developers, and StreamBase

Today StreamBase hosted an online roundtable entitled “How to Empower Your Development Team with CEP and Visual Programming” featuring developers from J.P. Morgan, GAIN Capital, and Datasports. Combined, the panelists have 35+ years of development experience, and years of CEP experience, and they were kind enough to share those experiences with us, discussing the adoption of CEP technology, developing applications with CEP, and the best practices they’ve learned along the way.

For developers, development managers, architects, and business stakeholders this roundtable tells an interesting story about how CEP technology can truly transform the role of the developer. Take a look at some of the highlights below, watch the replay, then comment below and tell us about your experiences with CEP. 

Talk about your experience with CEP

Phil Martin, Datasports: I like using StreamBase because I like tools that let the computers do stuff that the computers are good at, and let me do stuff that requires thought. As a developer, you wind up doing some tasks over and over again; the structure of the application, domain specific plumbing, etc. Everytime feels like wasted effort. StreamBase lets me focus on the stuff that requires thought, not the details of implementation, connectivity, processing events, and so on. Another comment is around the aggregate operator. It is analogous to another API to learn. It’s an incredibly rich and flexible operator that I think almost qualifies as its own sub language or sub paradigm.  There’s a lot there and a lot of subtlety in its behavior, learn about and study it!

Mike Grandilone, J.P. Morgan Asset Management: The shift to a message passing structure is sometimes a shift in thinking. It's key to understanding how StreamBase handles the life of a tuple; we gained that through experience.  

Talk about the applications you've built with StreamBase

Joseph Chua, GAIN Capital: After 4 days of training and going through the examples, I built my first StreamBase application in just three hours, and it was fully deployed in a week. It connects to market data, manipulates the data, and stores it in a high-speed database. We've also built market making and real-time surveillence applications, and many more to come.

Phil: I've built several applications with StreamBase. One is an "indication of interest" (IOI) app that monitors trading interest and compares it against a CRM system and sends alerts to individual sales traders because they have a customer who either has traded on the symbol, transacted above a given threshold, or has a real-time order for it. The correlation of these factors in real-time also includes real-time news sources. Also, I've worked on two algorithmic trading systems that host their strategies in StreamBase and perform real-time risk checks on the algos, and an ETF arbitrage system where there was only one strategy, but it was a very involved strategy.

Mike: We brought a smart order routing system up and running in 6 months with 2 developers, which we thought was very fast. We also got the business people involved in the canvas and the development - this is a very important factor in getting applications done quickly.

Talk about the benefits of StreamBase and visual programming

Phil: StreamBase helps facilitate communication. Whenever you want to convey information about a system, you always draw a picture. And they almost always look like StreamBase Event Flow. If you have taken the necessary care to clearly delineate modules, then, depending on the technical sophistication of who you are talking to, you can choose how far to drill down. It's extremely helpful in faciliating technical discourse; with the business users, it saves time.

Mike: Visual development has really improved the speed of our development. Especially when you're doing a proof of concept. It helps you understand the application's logic, and also communicate with the business; in typical programming environments like Java, you're not going to show many people the code, but you can do it with StreamBase.

Joseph: The big value is that all the technical details are abstracted away in terms of connectivity and low-level code; we can then communicate with the business owners, so development cycles are shorter and more efficient. The other value we have found is they we can very quickly build a prototype solution so the business can make decisions more quickly with respect to requirements and decisions to develop the applications.

Phil: One thing that's helpful: with StreamBase, you can tell at a glance that spaghetti code is spaghetti code. You can tell if an application was thoughtfully designed without hours and hours and hours of code review.

What best practices have you learned from your experiences with CEP?

Mike: You need to shift your thinking, to think in terms of events, rather than traditional static applications. And, just like any application, build a testing environment at the same time as your application logic.

Joseph: Identify the connection points quickly, be open minded and willing to unlearn the more traditional ways of development, use StreamBase interfaces, and make sure you continuously refactor the application.

Phil: There are lot of resources for ramping up StreamBase skills - invest the effort to develop your skills. The speed of putting together something quickly is a great asset. The thing to keep in mind is that although the development speed of StreamBase is a benefit in the short term, it can, if you're not careful, work against you in the long term.

Phil: While the principles aren't different, the practice IS different. The first thing that sold me is seeing the ease with which you can change and tune your threading model, take things in and out of the threading part of your system, with multiplicity and dispatch methods. Scalability and performance is about taking more advantage of the cores in a system, and careful management of it becomes more and more important.

What was your experience integrating other code into CEP?

Mike: We have separate routing logic written in Java for routing logic that we integrated with our StreamBase application. We've done a lot of work with custom Java operators in StreamBase - it's very straight forward - no problem in doing it. There's lots of power in developing in Java for code that isn't event driven that allowed us to bring in the power of algorithmic trading as a drop-in to the StreamBase environment. Also, we have a C# front end GUI, which uses the .NET interface from StreamBase - this was easy, although the challenge was how to take a GUI that isn't event-based and make it work with an event-driven application. We're in production with that now.

The StreamBase community

Phil: I've put two components onto the StreamBase Component Exchange - mostly adapters. The publishing process was straightforward. One is an adapter that watches for events on the file system and creates StreamBase events when files are created, renamed, modified, or deleted, and another one listens for calls on a web service interface and creates events for the applciation. I suggest other developers look at the utility classes – see what are other developers are doing, and pick up some tips and techniques and helpful hands-on examples of how they are building their applications. 

StreamBase Resources for Developers

Below are the StreamBase resources mentioned in this roundtable:

• Trial version of StreamBase: www.streambase.com/download
• Product documentation: docs.streambase.com
• Personalized demonstrations: www.streambase.com/forms/request-a-demonstration
• “How To” developer webinars: www.streambase.com/webinars
• StreamBase Component Exchange: www.streambase.com/sbx
• StreamBase University: www.streambase.com/sbu
• StreamBase Training and Certification: www.streambase.com/community/training-and-certification
• Datasports blog for StreamBase CEP developers: http://datasports.ca

Introducing Live BI and StreamBase LiveView

For some time now, StreamBase has been working on a new product concept that's a little bit CEP, a little bit data warehouse, and a little bit business intelligence. Today we've unveiled it as StreamBase LiveView. Our customers have called it a "game changer," and a product that "fills a massive hole in the BI space." Hyperbole like that needs to be backed up, so here we go.

Complex event processing (introduction on Google+) was envisioned by its inventors as a "database for moving data" as firms increasingly deal with data in motion from mobile devices, social media and digitized data of all types, and in all industries. In its first decade of commercial use, CEP has come to power automated trading on Wall Street. A who's who list of firms have been public about their adoption of CEP, so let's examine the live BI problem in the context of Wall Street.

Industry analysts estimate that over 60% of U.S. equities trading is automated. The problem is, with so much automation, it's impossible to see and act on what's going on, because a typical trading environment can generate 10's of millions of events a day. Existing business intelligence and data warehouse tools aren't used in the front office of Wall Street, because they're designed to summarize what's already happened, not what's happening now. 

StreamBase LiveView brings BI and streaming data together, for the first time. Applications that need Live BI are everywhere on Wall Street - trading risk management, position management, P&L management, surveillance, compliance, and trading itself, as Waters Technology magazine describes in the article that StreamBase Seeks 'Radical Transparancy' in Business Intelligence.

And applications in other markets can be found in any environment that has big data in motion: telecom (location aware insight and live network operations), security (live fraud detection, live surveillance), and e-commerce (live social media analytics, ad placement, big streaming analytics for Hadoop and Flume). These apps can be deployed on premise or distributed in the cloud. 

StreamBase LiveView is built with StreamBase CEP, which provides connectivity to over 125 event-based sources of data and the extreme scalability required by live BI. The server-based architecture is designed to process hundreds of millions of events per node, absorbed from a network of event sources such as an enterprise message bus or scalable event absorption layer like Flume or IBM Streams. At the front end, users can slice and dice streams of data with ad-hoc query tools in the StreamBase LiveView Desktop (shown above). But here's the key: once users ask the questions they want answered, StreamBase LiveView keeps answering their questions - as conditions change in real-time; users see an always-live view of streaming data.

For the user, all this technology means just one thing: they see business intelligence that's constantly fresh and relevant to their business operations. It's not a view of 5 days ago, or even 5 minutes ago. The data is live, the insight is live, and live insight can lead to live action.

Facebook, Twitter, Google+ give us a live view of what's happening in our social networks; StreamBase LiveVew gives firms a live view of their enterprise data.

To learn more, watch the StreamBase LiveView presentation, follow new materials, articles, and resources on Google+ (+StreamBase and CEO +Mark Palmer), read industry analyst Curt Monash, DMBS2.0, or follow StreamBase Systems (@streambase), CEO Mark Palmer (@mrkwpalmer ), and CTO Richard Tibbetts (@tibbets) on Twitter.

 

Real-Time Big Data and the 11 Principles of Modern Surveillance Systems

(This article is also posted in the TABB Group's TABBForum, and generated a broad discussion in the comments there. To participate, visit the original post here.)

In his TabbFORUM article, Dave Tolladay eloquently argues that real-time surveillance is crucial in today's high-frequency trading landscape, and I agree. Ten years ago, real-time computing began to influence trading; today, HFT is pervasive and existing surveillance techniques, designed to run at the speed of click trading, should be considered to be legacy architectures in need of a radically new, modern approach. 

But managing big data in real-time is a new problem of physics, and those physics come center stage when it comes to surveillance. A recent panel on modern surveillance systems, Miranda Mizen from TABB Group, Justin Amos from Redkite Financial Systems and Richard Tibbetts from StreamBase compared modern surveillance systems to “anti-virus software for trading.” 

In the course of their conversation, Justin, Miranda and Richard touched on 11 key points about the “next generation” architectural requirements for surveillance, including: 

1. Real-time analytics. Surveillance systems must analyze market data at the speed of automated trading. That means real-time computing and complex event processing (CEP) as one of the critical technology enablers of modern surveillance systems. 

2. Historical “big data” analytics. Surveillance systems must put activity in context – suspicious activity can be an aberration or a pattern, based on historical actions. The big data movement has spurred a renaissance in the database industry with firms like Netezza (now IBM), Vertica (now HP), MongoDB (open source) as the flag bearers for the NoSQL (Not Only SQL) database revolution that aims to supplant legacy RDBMS systems to deal with the rising deluge of data described in the Economist piece. In surveillance, “big data analytics” allow on-line access to trade-by-trade history for investigation, trending, and discovery to be mashed up with real-time data to provide a real-time and historical context to behavior. 

3. Inclusion of new types of data sources. Justin suggested that the inclusion of real-time news – and even social media data sources – can be a benefit in correlating, for example, suspicious trading on news. 

4. Data fusion. Real-time, historical and new data sources must be fused for a complete picture of intra-day trading behavior, and a 360 degree view of market data, activity and sentiment. 

5. Multi-asset class support. Since HFT is becoming increasingly cross-asset class, so must modern surveillance systems. 

6. Pre-built algorithms. Todayʼs surveillance systems must support standard algorithms like front running, spiking, tailgating, and ramping that can mistakenly or purposefully occur in computer-based HFT systems. 

7. Adaptive algorithms. The ability to adapt algorithms on the fly to reduce false positives is important, due to the increasingly dynamic capability of modern HFT systems to adapt their behavior. 

8. Algorithm extensibility. The ability to quickly add new algorithms to the system as market dynamics evolve is also a key modern requirement for systems – they must keep pace with the rate of change of modern trading algorithms. 

9. Flexible workflow management and case management. Surveillance systems generally donʼt make automated decisions to change trading behavior or stop trading on their own; instead, they tend to guide a series of workflow steps to alert compliance officers, portfolio managers or traders and manage the process of alert analysis, resolution and tracking. 

10. Accessibility for the business analyst. With regulation changing so quickly, more firms want their own control over surveillance behavior, so some provide tools that allow compliance personnel to discover and implement surveillance algorithms on their own. 

11. Cloud-based or on-premise deployment. Surveillance is not typically the core expertise of trading groups, so the ability to deploy systems either in the cloud or on premise allows for more rapid and global rollout of surveillance tools. 

Real-time big data is a change in computing physics on Wall Street. New requirements motivate new solutions, and the new solutions and innovation coming from the surveillance community continues to prove that innovation never sleeps on Wall Street.

 

StreamBase News at SIFMA

The SIFMA show in New York is always a big week for news for firms that do business in the capital markets. This year, we had so much new news  that we issued a series of announcements over the last several weeks. In case you missed any of them, here's a summary of what we're talking about at SIFMA this year.

1. Thomson Reuters Selects StreamBase CEP. This is a huge endorsement from a leader in the capital markets. Thomson Reuters will utilize StreamBase in their open, global, ultra-high speed and resilient network for the financial industry, Elektron, as an integrated element in their Enterprise Platform for Velocity Analytics, as an embedded CEP engine in Thomson Reuters applications, and in their Thomson Reuters Dealing Aggregator FX Trading Solution. We're very honored to be selected by Thomson Reuters!

2. BMF&BOVESPA Selects StreamBase CEP. The largest exchange operator in South America selected StreamBase as their enterprise-wide deployment platform for event-based applications - yet another huge endorsement and evidence of the increased use of CEP for enterprise-wide deployments. 

3. StreamBase in Asia: Complex Event Processing in Trading with SunGard. Our partner SunGard recently held a online panel discussion about algorithmic trading, market data management, risk management, and other other applications in Hong Kong with Benjamin Becar, algorithmic trading product management for SunGard in Hong Kong, and Richard Tibbetts, StreamBase CTO. You can watch the replay of the session here.

4. New product release: StreamBase 7. Fantastic new usability, and performance improvements in four different dimensions.

5. StreamBase University Program. Our new program to provide StreamBase software and certification resources to universities to help students learn about the growing field of CEP in school. The StreamBase University Program is integrated with our StreamBase Developer Certification Program and StreamBase Component Exchange community for sharing StreamBase integrations, code examples, frameworks, and adapters. 

6. Erjang / Erlang integration and the growing StreamBase Component Exchange Community. The latest contribution from Darach Ennis (@darachennis) has been a big hit in the Erlang community and on SlideShare. Watch it here: Complex Erjang / Erlang Processing with @StreamBase. 

7. CEP in Risk Management and Surveillance. CEP isn't just for algo trading anymore! Broad adoption by firms like Thomson Reuters and BMF&BOVESPA, and increasing use in applications such as risk management and surveillance illustrate that CEP is becoming a strategic platform for firms on Wall Street, and for third party developed applications with "CEP Inside."

8. "Faster than a Flash" - MIT Technology Review on CEP for market data risk management. MIT Technology Review published a story on how proprietary trading firm PhaseCapital used CEP to have ZERO bad trades on the day of the flash crash. While other pundits promote fear of a looming "splash crash" - bigger and worse than the "flash crash," we're simply trying to promote firms that are actually doing something about it with innovative technology approaches to make trading safer.

I'll be at the show in New York this week and addressing questions on Twitter about these developments and others at @mrkwpalmer.

5 Trends Revealed by BM&FBOVESPA's Selection of CEP

Today StreamBase announced that BM&FBOVESPA, the largest exchange operator in Latin America, has selected CEP (for an introduction to complex event processing watch this short video) as their enterprise-wide deployment platform for event-based applications.

Big, public declarations like this one beg the simple question: "what's going on here?" I think BM&FBOVESPA's decision to use CEP points to 5 key trends:

1. Brazil continues their development as an economic power and an innovation power. The Brazilian economy has been surging in past years, and BM&FBOVESPA's use of CEP further reflects the innovation in this increasingly important market. According to analyst AITE group, high frequency trading is up 52% since last November (read their report on the Brazilian market Brazil: The Best of the BRIC?). CEP will be used to build innovative applications that will help the markets operate more efficiently, and more safely. While the U.S. struggles with increased regulation, the conditions in Brazil continue to become more attractive in terms of liquidity and fairness for worldwide investors. 

2. Visual CEP enables "Smart Jobs." NPR and Wired Magazine fascinating study this month on areas where the economy is doing well found that "Smart Jobs" are they key to current and future growth and will continue to grow in importance for years to come. The Smart Jobs on Wall Street are jobs where traditional IT AND business analysts work together to innovate by building smarter trading infrastructure. Marcio Castro, Chief Technology Officer of BM&FBOVESPA, said it this way: “The development of capital markets in Latin America is experiencing the fastest growing phase in decades. Innovation is the key for us to develop a liquid, low-latency and fair marketplace for our worldwide investors. [StreamBase was selected to help enable this innovation].” To learn more about how the StreamBase visual programming environment, StreamBase Studio, enables Smart Jobs, watch "Build a CEP App in 46 Seconds." 

3. CEP is an increasingly strategic piece of the enterprise IT landscape. BM&FBOVESPA's commitment to CEP isn't for an algo trading app or two - that was the first generation of CEP use in Brazil. Bovespa shows that CEP is becoming an essential platform for applications of all types, from front to back office. CEP is growing up and increasingly critical in the most critical IT environments.

4. IT infrastructure continues to become more event-based. Traditional IT architecture utilize a static computing model; they store a bunch of data, and allow humans to query that data to make decisions. Modern IT architecture is event-based: repetitive decisions are automated - decisions don't have to be left to the end of the day, or end of the month. In trading, tasks that used to be performed at the end of the day, like risk management, are now performed during the day - in real-time.  BM&FBovespa's move shows that the static platforms of the past are being augmented by the event-based platforms of the future.

5. StreamBase is taking Brazil by storm. OK, I have to brag just a little! It doesn't get much bigger than an endorsement by a global leader like BM&FBOVESPA. Other CEP companies moved first in Brazil, but when customers look for the leader in CEP, they turn to StreamBase. 

Fascinating things are going on in Brazil - now, with the largest exchange in Latin America featuring CEP to power innovation, new innovation lies ahead - stay tuned!

Thomson Reuters Selects StreamBase CEP for Elektron High-Speed Cloud, Quant Platform, Apps, and FX

Today we're excited to report that Thomson Reuters has selected StreamBase as its complex event processing (CEP) platform for a broad set of initiatives. CEP "flips the database on its head" by providing the ability to monitor, analyze, and act on events happening in real-time, rather than looking in the rear-view-mirror of what happened yesterday, last week, or last month. CEP powers real-time, big data applications on Wall Street including high frequency trading, pre-trade risk management, surveillance, market data management, smart order routing, pricing, settlement, real-time P&L, monitoring - applications from front to back office on Wall Street.

Why did Thomson Reuters partner with StreamBase for CEP? Initially, they announced four reasons: 

StreamBase in Elektron Brings CEP to Thomson Reuters' High-Speed Cloud

The StreamBase platform lies at the heart of a new generation of real-time, automated applications on Wall Street. As the use of CEP continues to grow, our customers increasingly ask to deploy some of their applications in the cloud. They want their automated processing to happen as close to the exchange as possible for the lowest possible latency, and they want secure and reliable access to their market data.

Thomson Reuters announced that they will provide StreamBase CEP in Elektron, their open, global, ultra-high speed and resilient network for the financial industry. By adding StreamBase CEP to Elektron, Thomson Reuters is creating the industry’s only platform that includes comprehensive real-time and historical data, high capacity network connectivity, managed hosting services, and CEP for rapid application development and strategy deployment.

The First End-to-End Quant Platform for Back Testing, Algorithmic Development and Deployment

Although StreamBase is all about acting on what's happening now, the knowledge of what happened yesterday is essential for any automated system. History is essential to discover trends and patterns in event streams. History is essential to back test automated strategies. History is essential to gain perspective on real-time actions at execution time. Over 90% of applications that use StreamBase also use some form of historical event data management.

Thomson Reuters will integrate StreamBase with the Enterprise Platform for Velocity Analytics, which integrates all Thomson Reuters content, including tick history, reference data, corporate actions, condition code processing and cancellation and correction messages. It also brings together many disparate third party applications including data management solutions, EMS, OMS, price discovery, back testing, trade surveillance and low latency execution capabilities across the quantitative research and trading workflow.

By integrating Velocity Analytics with StreamBase CEP, Thomson Reuters has created the first end-to-end quant platform for back testing, algorithmic development and deployment. 

Thomson Reuters Applications "Powered By" StreamBase

Thomson Reuters will deliver a suite of applications that are “powered by” StreamBase CEP to address low-latency workflow. CEP inside will provide Thomson Reuters clients the benefit of time to market usually associated with turnkey solutions, while at the same time allowing full customizability of the underlying application. One application Thomson Reuters already provides is the real-time surveillance application from Redkite financial which is based on the StreamBase CEP platform. 

StreamBase and Thomson Reuters FX Solutions

The FX markets have rapidly emerged as an important market for the capabilities of CEP. Thomson Reuters announced that StreamBase CEP will complement the existing Thomson Reuters Dealing Aggregator FX trading solution, available exclusively to Thomson Reuters Dealing customers, by providing CEP capability in tandem with the low latency algo trading platform. 

Why did Thomson Reuters choose StreamBase? Mike Powell, Global Head of Enterprise Real-Time Solutions for Thomson Reuters, said it this way: “The financial market landscape continues to evolve rapidly, meaning that nimbleness and time to market have become essential for market participants. Elektron empowers our customers to innovate and achieve high performance through a seamless infrastructure that brings together global market participants, delivers world-class content and creates a suite of new added-value services. The combination of StreamBase’s market leading CEP engine together with Elektron will provide a platform for us to deliver flexible and sophisticated solutions to our customers faster and more efficiently.”

For StreamBase, we’re excited about the vast resources and capabilities of Thomson Reuters can provide our customers. We’re honored to be selected as Thomson Reuters’ CEP platform of choice - their selection validates our focused approach to CEP, the ever increasing importance of the cloud and real-time big data, the disruption of simplicity, and the continued need for low latency and scalability on Wall Street. 

 

Can Innovative Surveillance Be a New Source of Competitive Edge?

In sports, great defense can be a catalyst for great offense. Can the same idea hold true for surveillance - the defense - in the capital markets? This was a question raised by today's panel of surveillance experts at today's StreamBase webinar (watch the replay). 

First, TABB Group's Miranda Mizan assessed the current surveillance market. She described how surveillance is morphing from a defensive obligation to an offensive differentiator, because the markets are changing so dramatically. Mizan argues that the quantity of data, speed of trading, and complexity of instruments have made traditional surveillance techniques insufficient. And since regulation (e.g., MIFID II) is changing so quickly, she argues that firms must re-think their surveillance strategies and technology choices.

Justin Amos joined Miranda. Justin is a former global markets sales and trading executive at JPMorgan, and the MD of Redkite Financial Systems, a UK-based firm whose system called Redeye has been quickly becoming the standard for modern, real-time surveillance systems. Justin described the requirements of today's surveillance systems by describing the motivation for Redeye, which he likened to "anti-virus software for trading."

Together, Justin and Miranda touched on 12 "next generation" surveillance requirements, including:

1. Real-time analytics. Surveillance systems must analyze market data at the speed of automated trading.
2. Historical "big data" analytics. On-line access to trade-by-trade history for investigation, trending, and discovery.
3. Data fusion. Mash-ups of increasingly complex data, including unstructured data.
4. Inclusion of new data sources. The ability to integrate live data, static data, and social media data.
5. Multi-asset class support. Modern surveillance must work across asset classes.
6. Pre-built surveillance algorithms. Standard algorithms like front running, spiking, tailgating, and ramping.
7. Adaptive algorithm modification. The ability to adapt algorithms on the fly to reduce false positives.
8. Alert algorithm extensibility. The ability to quickly add new algorithms to the system as market dynamics evolve
9. Customizable surveillance algorithms. The ability for a firm to tweak an algorithm for their own models.
10. Flexible workflow management and case management. Control over the process of alert analysis, resolution, and tracking.
11. Accessibility for business analyst. With regulation changing so quickly, more firms want their own control over surveillance behavior.
12. Cloud-based or on-premise deployment. The cloud allows more rapid rollout, and may ease global scalability challenges

The panel argued that existing surveillance systems can't handle these requirements. Taken together, the challenge makes what was once traditional "defense," or a compliance function, very new and challenging - a source of differentiation.

Miranda went deeper, with examples of how brokers and the buy side are using modernized surveillance systems: agency brokers are beginning to use surveillance tools to monitor their trading algorithms and optimize their client service and order routing for best execution; the buy side are using surveillance to gain greater control to know that their trading is safe, its orders get to the right places, and that dark pools are working correctly in relation to the rest of the market; sell side institutions are using surveillance tools to institute process and control across the organization, globally, and integrated with automated systems.

Richard Tibbetts, CTO of StreamBase, gave a brief introduction of StreamBase CEP, and why complex event processing is increasingly being used to power surveillance applications; indeed, CEP provides the engine to address many of the 12 modern surveillance requirements set out by the panel.

Simon Townsend, CIO of Redkite, gave a "whistle stop tour" around Redkites' application. He hit on most of the 12 requirements above, and even showed didn't just claim that their algorithms are extensible and dynamic (requirements 6, 8 and 9) - he showed it in StreamBase Studio, and showed how business analysts can extend the Redeye surveillance algorithms.

One of the most exciting times is when a market is transformed by new ideas. In the capital markets, perhaps that innovation is upon us in surveillance, where tools that meet modern trading requirements can turn a defensive obligation into a tool for innovation.

On StreamBase 7, Its Breakthrough Usability, and Performance for CEP

Today we announced SteamBase 7, the next major version of our market-leading Complex Event Processing (CEP) software. In press and analyst pre-briefings last week, we discussed a lot of cool new features. Here’s a quick tour.

A HUGE STEP FORWARD IN CEP USABILITY

Chief among the innovations of StreamBase 7 is the revamped StreamBase Studio. The interface innovations of the iPhone, iPad, and modern operating systems has made great interface design expected. StreamBase has always been the best and easiest to use CEP platform. But with StreamBase 7, our engineering team has taken a huge leap forward in usability.

Drawn from the cutting edge consumer user experience in platforms like Apple iOS and Windows 7, Studio 7 dramatically increases the speed and quality of application innovation. 

To achieve this usability, StreamBase 7 has new code organizational features, integrated annotation features for English descriptions of functions, gorgeous diagram rendering techniques, and facilities to isolate and encapsulate the necessary complexity of a sophisticated application. Literally, StreamBase users can now read StreamBase application code at a glance, and organize it effectively so that the Business and IT can collaborate in a way never possible before with any CEP product. These features that make this all happen include:

1. A beautifully redesigned user experience, including the use of transparency (similar to Windows 7 Aero), overlays to communicate information at a glance (e.g., iOS number-in-red-circle), gradients rather than solid color to ease readability (think in terms of Windows 7 and iOS) and, and complete color customization (a la Windows 7 themes.) 
2. Support for custom modules (groups of application logic) and extension point styles.  Better control over modularity allows IT to manage large applications, and manage how and where business users or other groups can contribute code. For example, a risk domain expert can create risk checks in one module while IT focusses on connectivity, performance and integration in other modules.
3. Advanced graphical code layout, with new group layout features and code rendering controls
4. A new rendering look and feel to ensure code is easier to follow at a glance
5. New mouse and keyboard gestures to increase EventFlow coding productivity, such as on-canvas renaming of components
6. New hotkeys that give the ability to instantly create CEP language operators
7. Layout-alignment shortcuts

The result is that business users and IT can work in one environment, with one set of tools, with one set of code, dramatically improving time to market, quality and ability to evolve systems once they are deployed.

In recent years, StreamBase deployments have become bigger, more global, and more complicated. So StreamBase 7 improves usability throughout the entire software development lifecycle, including:

1. New code refactoring schemes. So developers can quickly identify changes in their application logic and adjust the rest of the application more quickly as modifications are made.
2. 400% faster application execution while debugging. So problems can be identified more quickly
3. The ability to attach Studio to a running StreamBase server. For better diagnostics in deployment.
4. A new XML editor. Allows operations to more effectively configure StreamBase deployments.
5. Dynamic extension points that allow users to configure StreamBase applications to be configured quickly at runtime. When a new algorithm or analytic is produced by a developer or business users, this code can be dynamically inserted into the StreamBase code, tested, and deployed.
6. Support for 64-bit workstations for very large StreamBase application development

PERFORMANCE IMPROVEMENTS IN FOUR DIMENSIONS

From the first release of StreamBase, the focus has been on performance. Previous SteamBase performance innovations include the StreamSQL compiler, the ability to visualize parallelism and scalability, data parallelism and dispatch, and our open architecture for highly available systems all combine to create the fastest, most scalable platform for ultra low latency applications.

Version 7 of StreamBase continues to break down CEP performance barriers in four dimensions: ultra low latency connectivity, predictability of latency, FIX performance, and with high-speed computing technologies.

1: MICROSECOND-LEVEL LATENCY

All CEP platforms have an architecture for absorbing and emitting events. Some run out of the main CEP engine process, which can add significant latency overhead to event processing. StreamBase already features an super-fast in-process adapter architecture, but goes further in StreamBase 7 with an optimized “zero-context-switch adapter architecture” which allows StreamBase applications to process messages in a single thread from the time it enters the engine, through StreamBase logic, to the time it leaves the engine.  This architecture allows messages to pass through the StreamBase engine in microseconds, and has been optimized to speed up CEP application performance.

2: PREDICTABILITY OF LATENCY

The predictability of performance has been improved for Streambase 7 applications, as users now have access to server configuration options that provide the user with fine-grain control of the tradeoffs between ultra-low-latency and throughput. And, overall platform improvements have yielded over 20% improvement in application performance across the board.

3: ULTRA LOW LATENCY FIX CONNECTIVITY

Many StreamBase customers in the capital markets is already known for its low latency connectivity architecture and version 7 comes with over 22 new connectivity options. Many of those connectivity options leverage FIX, and in StreamBase 7 FIX connectivity has been dramatically enhanced. In benchmark testing, the new StreamBase FIX engine has clocked in at 50-75 microseconds send time depending on message size, all while running on Linux and commodity hardware.

4: HIGH-SPEED COMPUTING AND GPUs FOR MANAGING BIG DATA AND NOSQL STORES

The state of the art of ultra-low-latency is rapidly expanding, and many StreamBase customers have been on the leading edge of applying new technologies to deal with big data in innovative ways. 

GPUs are being used to continuously produce metrics, factors, and parametrics, contribute them to memory data structures for ultra low latency streams of data to adapt in flight to compute accelerated algorithms. They are also used to easily integrate accelerated compute and messaging or to perform collaborate distributed event processing in a high level flow oriented language. 

To support this trend for high-speed computing with StreamBase, we have added several connectivity options in StreamBase 7 to make it easier for developers to connect StreamBase to their existing and next generation IT infrastructure, including hardware-based middleware for ultra-low latency operations, including enhanced low latency middleware connectivity such as Wombat, 29West, Solace, and Tervela.

REACTION TO STREAMBASE 7

Analyst Adam Honore’ said at the end of our briefing: “wow, you’ve been busy!” and he’s right - as customers build more sophisticated applications with StreamBase, with bigger demands from the business users, and bigger demands from IT, with the ever-increasing need for speed, StreamBase 7 extends our lead as the best-in-class CEP platform in the industry.

The benefit of bringing the business and IT together is faster time to market, with higher quality. With tight communication, the business and IT make mistakes faster, learn from them faster, and innovate faster. As Thomas Edison said: “I make more mistakes than anyone I know. And sooner or later, I patent them.” 

But my favorite anecdote comes from one of our customers - Chris Lees, vice president of SunGard’s global trading business. He put it this way: "The usability of StreamBase Studio helped us develop a new Valdi product, we used it to describe requirements collaboratively with IT and because we were working with the actual code, we made better architectural decisions early that made for a better first release.”

The Flash Crash & the Role of Financial Software Engineering

Even though the flash crash happened seven months ago, the debate rages on. And since CEP is a common piece of our modern financial computing fabric many firms have entered the debate with vested interest. Unfortunately, most vendors wrap flash crash observations with veiled sales pitches for CEP-based risk management, or disingenuous calls to "catch the rogues and scoundrels" with CEP-based surveillance systems. Although these applications are important, to suggest they are a magical elixir to ward off future breakdowns is misleading at best. 

But thoughtful discourse does continue, and the recent Securities and Exchange Commission report has reinvigorated debate. Richard Tibbetts, StreamBase CTO, weighed in with a piece called "What Does the Flash Crash Mean For Financial Software Engineering?" on Tabb Forum this week. Richard's view on the flash crash examines the role of financial software engineering and the efficacy of data processing, integrity, and reliability within the fabric of the infrastructure. The role of financial engineering is an essential, overlooked, and misunderstood piece of the puzzle. Richard's thoughts go like this: 

"In all the conversation about the crash, the real historic change is going unmentioned. In the very last paragraph of the Lessons Learned section of the report, the authors note: “another area of focus going forward should be on the integrity and reliability of market centers’ data processes, especially those that involve the publication of trades and quotes to the consolidated market data feeds.”

This is a key insight – late in coming – that will expand over time. Traditionally the SEC has focused on regulating the behavior of market participants, identifying deliberately malicious behavior or patterns of behavior. But now they are beginning to realize that data processing, integrity, and reliability can be just as important. The efficient functioning of markets is in just as much danger from software system failures as it is from malicious traders..."

The role of financial engineering is much broader than CEP, and deserves deeper discourse. To read more, continue reading Richard's piece on Tabb Forum (which requires a free registration) here, and watch for more over the coming months on Tabb Forum, on this blog, and on Richard's blog on software engineering and innovation, Hyperextended Metaphor.

CEP News at SIFMA 2010

The SIFMA show in New York is always a big week for capital markets news, and this year is no exception. This year the big news from StreamBase is that there’s so much of it! Last year, we announced a marquee customer, CME Group, and sparked an industry debate about the role of real-time sentiment analysis in CEP when we announced our StreamBase adapter to Twitter.

So what’s going on this year?

New CEP applications: pre-trade risk and more from SunGard. SunGard’s recent selection of StreamBase is the endorsement of the year in the CEP market. Sungard have announce many applications they’re building, from smart order routing to their dark pool. But perhaps the most interesting area from a market trend perspective has been pre-trade risk management. With impending market micro-structure regulation, real-time risk management is becoming an increasingly important area of innovation, a lot of it based on CEP. Watch this space for more news about larger role real-time risk will play in the capital markets.

New innovation with CEP: CME Group. We're happy to announce our customer CME Group won the Inside Market Data award for Best Real-Time Market Data initiative based on their innovative use of StreamBase. Steve Goldman, Director of Enterprise Architecture at CME Group, said it nicely: “We continue to advance our trading environments and capabilities by integrating human intelligence with cutting-edge technologies, such as StreamBase. Through these initiatives we are able to process market data in a reliable and low latency manner while providing transparent and accurate settlement pricing for our customers.”

New cool technology: using GPUs for Low-Latency CEP applications with NVIDIA. NVIDIA GPUs are one of the hottest technologies in computer architecture today, allowing vasts amounts of data to be processed in parallel for low cost and low environmental footprint. However, the current state-of-the-art is to use them to analyze historical data in a research context. StreamBase and NVIDIA have come together to allow real-time data analysis using GPUs for low-latency applications. For organizations using GPUs to research signal processing, options pricing, and risk, CEP-based GPU solutions will enable them to deploy the same powerful analytics models against real-time data in a low latency trading environment.

New technology partner: Solace Systems for hardware accelerated messaging: Solace Systems is a leader in hardware accelerated messaging, delivering ultra low latency and flexibility. Many large sell-side organizations are making Solace the foundation of their next generation realtime data management infrastructure. The integration with StreamBase enables quick and easy deployment of applications written using the StreamBase Studio development environment on the Solace messaging infrastructure to publish and subscribe to all different kinds of data. Using Solace and StreamBase together enables firms to develop scalable low latency applications.

New partner: Redkite brings real-time to surveillance: Our partner Redkite has built their surveillance application on StreamBase (shown at right). Redkite’s solution uniquely leverages the virtues of CEP: extensibility, intraday configurability, case management with audit trails, and simultaneous access to real-time and historical information. 

New, increased activity in FX. The FX market continues to be one the hottest CEP markets out there. We issued our FX market survey last month, and continue to announce a string of FX customer endorsements including CMC Markets, City Index, Gain Capital, and Curex, and many many more to come - stay tuned!

New StreamBase Component Exchange contributions: StreamBase component exchange (SBX) continues to grow as our electronic exchange platform for StreamBase CEP components. Recent contributions include how to connect Python to StreamBase, Bollinger Bands signalling algorithm, linear regression, least squares fitting, Monte Carlo options pricing, and a StreamBase adapter for R. For more, check out SBX here.

New movement among key CEP companies. We were thrilled to announce Fraser Herrick has joined us at StreamBase VP, Head of EMEA Sales. Previously Fraser was a key executive at Progress Apama and we've teamed up with him again. We also wish our good friend Louis Lovas the best of luck as well as he also left Apama for CEP newcomer OneMarketData. 

New developments in Brazil. Business in Brazil is booming, as Kairos Asset Management (KAM) expanded their deployment of StreamBase. KAM added derivatives to their CEP-based trading infrastructure, and we're seeing more derivatives trading with CEP on the global stage as well.

New ideas on entrepreneurship and innovation: StreamBase stepped onto the global innovation stage last year with our attendance at the World Economic Forum at Davos. We continue to participate in the broader conversation about innovation and entrepreneurship, and this week I’ll participate at Venture Summit East a session at Harvard Business School entitled “Going Global.” Stay tuned for blog entries from Venture Summit East.

Stop by the StreamBase booth #1854 (Level 1- just past the Press Booth) at SIFMA to talk to us about all the latest news - our cocktail reception is from 4:00-6:00 on Tuesday and Wednesday. 

And follow our real-time coverage of SIFMA on Twitter: StreamBase (@StreamBase), Richard Tibbetts our CTO (@tibbetts), and me (@mrkwpalmer). See you in New York!