The StreamBase Event Processing Blog

How to Analyze Sensor Data in Real-Time with CEP

This is a blog post was contributed by senior field architect Mark Hudson from StreamBase's London office. 

    The Moving Finger writes; and, having writ,

    Moves on; nor all your Piety nor Wit

    Shall lure it back to cancel half a Line,

    Nor all your Tears wash out a Word of it.

    -- The Rubaiyat of Omar Khayyam 

The famous Persian poet was referring, I suppose, to the impossibility of living one's life over again.  Fortunately, in the world of stream processing, things are easier - with the right tools.  It is indeed feasible to watch what the moving finger of real-time data writes.  Better still, yes you can go back - at least to extract valuable insights while there is still time to avoid shedding tears.

Typical StreamBase customers in financial markets are avid consumers of time-based data - "tick data" in the industry jargon - for pricing trades and spotting profit opportunities.

Outside financial markets, there’s increasing interest from every kind of company in the potential of CEP.  Very frequently, we hear the need to mine usable insights from a flood of "sensor data".

SENSORS AND (NOT) SENSIBILITY

What's a sensor for this purpose?  Well, it could be a gadget in a pump or a generator, way out in the wilds (sometimes literally - in the depths of a forest maybe).  It could be an SMS-like message from an aircraft flight computer.  It could be a sensor on a door, part of a burglar alarm system.  One among many, connected to a remote monitoring centre.

The common requirement here is figuring out the signal from the noise.  Sensors generally aren't clever.  Usually, they are cheap, reliable and rather dumb.  They send a lot of repeat messages.  Each sensor usually reports a limited number of measurements or alarm conditions.  The sensors aren't clever enough, or don't have enough context awareness, to report more than bare, isolated facts.

The result?  The central monitoring function is "flooded" by very large amounts, in total, of individually low-quality (high noise) signals.  As often as not, time is a factor, in figuring out the real signal.  Events - signals, alarm conditions, unusually small or large measurements - mean a great deal more when several happen in a short burst.

MAKING SENSE OF SENSOR DATA

It's usually even more significant if a certain set of events happens in a specific order.  When I get a migraine it’s flashing lights then headache then nausea in that order.  A “march of events” as my doctor calls it.

Suppose we have a generator with temperature, voltage and vibration sensors.  A reading of 160C happens typically several times a day.  The sensor reports every minute, so an individual "hot" period can result in 5-10 "high temperature" alerts but this means little in itself.  Similarly, the under-voltage and excessive vibration indicators happen all the time.  Crying wolf, if you like.

A much more significant sequence would be something like the following:

1) Vibration alarms being sent for 10 minutes then

2) Temperature consistently higher than the hourly average for a sustained 20 minutes then

3) Under specified voltage by 10% for 20 minutes

and all these things happen in the same hour in that order

HIDDEN MEANING IN THE SENSORY DELUGE

You can probably see where I'm going with this.  There is a worn bearing that causes a cooling fan to malfunction for a while.  This lets the temperature go up, at which point a feedback loop in the generator controller throttles back the voltage until things cool down.

Normally, say with a database and a language such as Java, it would be very difficult to detect these kinds of multiple, complex, ordered & time-dependent conditions.  But with CEP, we can build a system with just 20 visual elements!

Here's our generator scenario written in StreamBase's visual EventFlow language.  It starts with a flow of sensor messages.  These typically come from an “input adapter” such as the IBM WebSphere MQ example shown in the picture.  (StreamBase has over 150 other adapters too).

Finally – though not shown explicitly here – we would take the “bearing problems” combined signals, and act on these.  Maybe we send an email or SMS to the operations team.  Maybe we send a message directly to the generator to shut down.  Maybe we log the event in a database for inspection every morning.  Any or all of these responses would be equally easy to plug in.

CEP, such as the StreamBase EventFlow here, makes it very natural to express the solution.  You can do some very complicated programming easily.  It looks and feels the way an engineer would describe the requirements.

By the way, here’s the StreamBase configuration panel for that clever little Pattern operator:

 

As you can see, there is a specialized mini-syntax in here for event ordering.  The ‘input1’ is the complex vibration alarms criterion, the ‘input2’ is the complex temperature criterion, and the ‘input3’ is the voltage criterion.  (That isn’t obvious from the top-level EventFlow diagram, but if you were to look into the details, you’d find it so).  So we’re telling the pattern operator to watch for the sequence vibration criterion followed by temperature criterion followed by voltage criterion, as in the original specification.  We could even pick & choose further between the details of our three ordered event criteria, by using a more complex Predicate than just the ‘true’ value here.  But you’d need to see more details of the underlying operator settings to understand this fully.

 

AGILE EVENT HANDLING

Just as important, the specialist engineer can understand the logic, and can work with the developer to achieve the correct pattern detection.  I'm not a generator engineer, so there are probably some flaws in my scenario.  But it doesn't really matter.  The right engineer would be able to spot patterns that are diagnostic of underlying issues.  That would include problems serious enough to justify a service visit - even to that generator, far out in the woods, powering an aircraft warning light.

Just as important: even expert engineers might not know all the important patterns already.  Playing around with the data allows new patterns to be spotted.

The burglar alarm scenarios are even easier to think up.  Three intruder alerts on five successive days at different doors belonging to the same company anyone?  But our recognition of such "insightful" patterns grows all the time.  So, it's important to trap new combinations in an agile way.

In conclusion, I can’t promise “Piety nor Wit”.  That’s too much for any software (even StreamBase!). 

However, with a clutch of Aggregate operators, a Pattern operator, and few others, finding true sense in the data could at least be yours.  You may even get your engineer out into the woods before the moving finger spells disaster.  Perhaps even Omar Khayyam would have been impressed!

Drop me an email if you’d like to see the details of a StreamBase EventFlow solution like this for sensor monitoring. 

 

The Best Practices of Real-Time Innovation

Event-driven systems are new to many firms. For many, automating existing manual processing, using event-driven techniques, properly simulating real-time analytics systems is a new experience. 

Fortunately, some industries have been on the forefront of designing for real-time. Namely, banks in the capital markets trading arena have been automating and building event-based applciations for years. 

So on a recent customer tour with our customers, we've been asking them for their best lessons learned and the business impact the systems they've built have made. 

Best Practice #1: Build Apps So Quickly You Won't Mind Throwing Some Away

One IT executive told me "CEP allows me to build event-driven apps in 2-4 weeks, rather than 6-9 months with an app server and database - so I can say "yes" to more requests from their business clients, which helps us innovate more quickly. We keep the great ideas, and discard the ones that don't work."

Best Practice #2: Start Small, Succeed, then Scale

The most consistently successful firms that moved to real-time on Wall Street started with a small, critical application, learned lessons from its development, then applied those lessons learned to other projects and groups. Over time, they developed a center of excellence with firm-wide sucess in real-time application development.

 

Best Practice #3: Plan for Spikes, Not Aggregate Volumes

Systems that use primarily historical have trained many technologists to capacity plan their systems for the aggreate amount of data they manage. Real-time systems are different, and should be planned for spikes in volume, rather than only aggreate volumes. This best practice explores three approaches: burst behavior, the use of conflation, and computational reduction, that helps applications gracefully manage spikes and deliver correct and timely response to users and automated action. 

Keep coming back throughout 2013 as we continue to add to our library of real-time system best practices!

Best Practice #4: Incorporate Statistical Computing with Event Driven Systems

Numerical computing environments are often used for algorithm development, data analysis, visualization, matrix manipulations, and statistical analysis. In finance, quants use it to do time-series analysis, portfolio optimization, asset liability modeling, quantitative risk modeling, and derivative pricing. This best practice describes how to does this computing in real-time, using MATLAB with StreamBase CEP together.

Best Practice #5: Balance Man And Machine

The prospect of automating a business often sparks a debate about what is more effective, man or the machine. It turns out that the debate presents a false choice, and effective automation lies in the proper balance between the two.

This article explores 5 steps to balance man and machine, including the infusion of more data scientists in executive positions, the application of real-time risk management and surveillance, the infusion of historical and real-time analytics in one place, and more.

 

Best Practice #1: Build Apps So Quickly You Won’t Mind Throwing Some Away

This post is the first in a series of best practices gleaned from leading StreamBase customers who have effectively deployed "real-time" business systems.

During our recent customer tour, I was concerned to hear that one of our customers had recently decommissioned several of the 30 applications they built with StreamBase. It turned out that this was by design and a virtue, not a sign of a lack of success.

What I found leads me to the first in a series of "Business Best Practices" around real-time application development: Build Applications So Quickly You Won’t Mind Throwing Some Away.

.

THE SITUATION: IT IS OVERWHELMED; FAILURE IS NOT AN OPTION

Most technology groups are overwhelmed with requests for new features and capabilities from the business. At the same time, they need to innovate. And worse, some cultures actively discourage failure. These conditions lead IT to play it safe, and a conservative culture can be born that leads to safe technology choices, and an acceptance of long, expensive software development cycles.

THE SOLUTION: THE FOUR TENETS OF RAPID APP DEVELOPMENT WITH CEP

The StreamBase customer I visited changed their thinking about engineering several years ago. They changed IT leadership to one that favors speed and experimentation over control and safety. They adopted StreamBase as a key enterprise development platform because they can build apps in 2-4 weeks, rather than 6-9 months. This rapid speed of development means IT can experiment with new ideas and simply throw away those that don’t work out in the end.

In his words, there are four tenets of StreamBase responsible for this:

1) Visual programming. Event-based applications are expressed in StreamBase EventFlow™, our visual programming language designed to implement event-based systems just as you would describe them on a white board, with event-driven inputs, such as streaming financial market data, and event outputs, like messages published to a message bus. 

2) Polyglot programming. Very few organizations develop applications in just one language; they use several languages depending on the job. For event-based, real-time applications, the programming language of choice is often StreamBase. But we also encourage the use of other programming environments with CEP, like Java, .NET, MATLAB, R, Python, HTML5, JavaScript, and more. StreamBase has integrated support for these environments in Studio, which makes it easier for developers to learn, but also easier to integrate existing code in an event-driven application. Polyglots (a developer who knows and uses several programming languages) can more easily adopt StreamBase and build applications more quickly because they can connect to any code on any platform quickly and easily.

3) Pre-built connectivity. The StreamBase catalog of over 150 connectivity points turns integration - usually a development task that is completed only after months of intricate coding with sockets or other messaging APIs - into a drag-and-drop experience, in some cases cutting integration time down to minutes.

4) Complete event container. Projects that use databases and traditional app servers for event-driven applications have to build custom container logic. Event-based containers require low-level socket programming, delicate threading design, event management, and more. As our customer said “it’s a whole other project just to manage and connect to event streams.” StreamBase, by contrast, comes pre-built with connectivity, monitoring, threading, management, and testing tools. You just start using them, building business logic.

THE IMPLICATION: APPS TAKE WEEKS TO DEPLOY; NOT MONTHS

This head of IT technology at a major bank told me that the combination of these four elements allows him and his team to take on new ideas more aggressively:

"The cost of doing something in StreamBase is so low that even when we envision a project that we aren't sure is going to fly, we're able to take it on because it's not a big investment of time.  It's a few weeks or a month, not 6-9 months on a development roadmap. This radically shorter development cycle helps us build apps more quickly, experiment with new ideas, and throw away ideas that don't work. StreamBase helps us do this because it's visual and takes care of all the event-based plumbing for us, so we can focus on the business logic rather than how to get apps up and running."

THE LESSON: BUILD APPLICATIONS SO QUICKLY YOU WON’T MIND THROWING SOME AWAY

It's refreshing to find organizations that embrace experimentation and failure, and unique tools to make it happen. So the best practice of applying rapid development techniques can help IT departments take on more, experiment more, and innovate more is our first best practice for real-time enterprise applications: Build Applications So Quickly You Won’t Mind Throwing Some Away.

The 3 Principles of Analyzing Data in Motion



 

When I first saw an event processing language 10 years ago, as a C++ developer, it opened my eyes to a new development "physics" that made dealing with information in motion simple. Today, it seems like all modern data is in motion: streaming social media data, market data in financial services, and the increasing adoption of the internet of things and sensor-driven systems. Clearly, the technology needed to manage data in motion is different than that needed to manage data at rest; and that's where StreamBase complex event processing, or CEP comes in - its the ultimate platform to monitor, analyze, and act on data that's in motion.

Complex event processing doesn't mean the technology is complicated to use; it's quite the opposite.... the term refers to the ability to identify complex sequences of conditions - as the data moves through an enterprise.

Three CEP capabilities, aggregation, correlation, and temporal analytics, provide the foundation for dealing with data in motion.

For example, in the gift card market, let's say you wanted to detect a common fraud condition, such as: "Tell me when the number of gift card redemptions at any POS machine is above $2,000 in any 1-hour period." - this situation can be easily detected by the CEP's aggregation facility, which continuously calculates metrics across sliding time-bound windows of moving data which is used to understand real-time trends. Continuous aggregation is hard to do with traditional tools; but with CEP, it's built in - simply drag the "aggregation" operator onto your application template, and you're off.

The second key new principle of CEP is Correlation. Correlation means connecting to multiple streams of data in motion and, over a period of time that may be seconds or days, identify that condition A was followed by B, then C. For example, if we connect to streams of gift card redemptions from 1,000 point of sale terminals, CEP would allow me to continuously identify conditions that compare the POS terminals to each other like: "Alert me if the gift card redemptions in one store are above 150% the average of the other stores."

Finally, CEP is designed for temporal analytics, which is the idea of having time as a first-class computing element. Temporal analytics are critical for processing data where the rate and momentum of change matters. For example, in fraud, sudden surges of activity are common indications of fraudulent activity - so detecting "If the number of gift card sale and card activations within 4 hours is greater than the average number of daily activations of that store in the previous week, stop approving activations." Unlike computing models designed to summarize and roll up historical data, CEP is designed to ask and answer these questions on data as it changes.

These three elements: aggregation, correlation, and temporal analytics, set CEP apart from all other computing platforms by helping technologists look at what's happening now, not just what happened in the past more quickly than ever imagined before. Data in motion, and the use of CEP to analyze it, is the future of computing.

90 Seconds on How to Develop with Java and CEP

Not long ago I sat down with Phil Martin, a senior consultant at Datasports, for a 30-minute online Q&A and demo on “How to Develop with Java and CEP”. Phil’s been developing applications with Java and StreamBase CEP for over 3 years now and we wanted to pick his brain about his experiences developing applications with these two languages.

If you’re a Java developer looking to build real-time event based applications you should definitely check it out in its entirety here. Then again, if you’re a developer you probably don’t have 30 minutes of free time to watch the video...unless of course you're developing with StreamBase!

So instead I'll make it easier for you. Below you’ll find the highlights from this conversation, and the best part is you should be able to digest it all in about 90 seconds.

Q: Phil, what kind of use cases and applications are a good fit for CEP development?

A: It should be a real-time system where performance is an important concern. And it should be unsurprisingly an event or message based type of processing model where those events or messages are clearly defined, where the content and format is known at design time.  And I think that you get increasing benefit from a platform like StreamBase if you have a large number of disparate kinds of systems that you’re looking to integrate with. 

Q: What aspects of my application should be developed in CEP and which should be developed in Java?

A: Anything that implements a low level protocol type of logic probably belongs in the Java domain, for example things that are implementing interactions with vendor provided client libraries or protocol libraries or doing low level network disk or database IO. In general other than that I find that the CEP programming model works the best for almost all business logic.

Q: What are the benefits of developing with CEP?

A: When I’m developing code I want to concentrate on the things the human being does well and have the computer do things that machines do well.  And for me that means I want to be focused as much as possible on business logic.  And what I find is that using a platform like StreamBase lets me stay focused on business logic.  The other benefit that I see is visual programming. I find the day-to-day model for doing development more productive. I also find that the resultant applications tend to be fast and stable.

Q: What have been your experiences around time-to-market gains with CEP?

A: My general rule of thumb is developing with CEP is three to five times faster than using a traditional programming model like Java or .NET development.

Q: How does CEP address some of the biggest frustrations Java developers face?

A: My biggest frustration also relates to the biggest strength in Java, which is that for every level of your application infrastructure there are choices to be made.  There are choices around what version of the JRE, what kind of framework you’re going to use to write your application, the list goes on and on. By having so many of those choices already made for me and made well by the StreamBase platform I can do what I want to do, which is to stay focused on solving business problems.

Q: There are some objections that we sometimes hear from Java developers around having to learn a new language and concerns about visual programming and its effect on performance. Why shouldn’t Java developers be concerned about that?

A: In terms of being able to actually get ramped up, get hands on and start doing productive work I find that curve with StreamBase is actually very fast compared with other environments. I would normally estimate that someone who is never touched StreamBase before can actually be productive in a matter of weeks.

The StreamBase environment is compiled the same as any Java applications, the components and infrastructure are developed with performance in mind.

I found that the visual environment makes the processes of refactoring and reusing code and tuning the threading model much simpler, much clearer, and much, much faster than what it would be in a traditional programming environment.

Thanks to Phil for doing this webinar with us. You can check out his blog at http://blog.datasports.ca/ where he blogs about financial systems engineering, including StreamBase, Java, and the Canadian markets.

Tech Q&A with CEP Developers, and StreamBase

Today StreamBase hosted an online roundtable entitled “How to Empower Your Development Team with CEP and Visual Programming” featuring developers from J.P. Morgan, GAIN Capital, and Datasports. Combined, the panelists have 35+ years of development experience, and years of CEP experience, and they were kind enough to share those experiences with us, discussing the adoption of CEP technology, developing applications with CEP, and the best practices they’ve learned along the way.

For developers, development managers, architects, and business stakeholders this roundtable tells an interesting story about how CEP technology can truly transform the role of the developer. Take a look at some of the highlights below, watch the replay, then comment below and tell us about your experiences with CEP. 

Talk about your experience with CEP

Phil Martin, Datasports: I like using StreamBase because I like tools that let the computers do stuff that the computers are good at, and let me do stuff that requires thought. As a developer, you wind up doing some tasks over and over again; the structure of the application, domain specific plumbing, etc. Everytime feels like wasted effort. StreamBase lets me focus on the stuff that requires thought, not the details of implementation, connectivity, processing events, and so on. Another comment is around the aggregate operator. It is analogous to another API to learn. It’s an incredibly rich and flexible operator that I think almost qualifies as its own sub language or sub paradigm.  There’s a lot there and a lot of subtlety in its behavior, learn about and study it!

Mike Grandilone, J.P. Morgan Asset Management: The shift to a message passing structure is sometimes a shift in thinking. It's key to understanding how StreamBase handles the life of a tuple; we gained that through experience.  

Talk about the applications you've built with StreamBase

Joseph Chua, GAIN Capital: After 4 days of training and going through the examples, I built my first StreamBase application in just three hours, and it was fully deployed in a week. It connects to market data, manipulates the data, and stores it in a high-speed database. We've also built market making and real-time surveillence applications, and many more to come.

Phil: I've built several applications with StreamBase. One is an "indication of interest" (IOI) app that monitors trading interest and compares it against a CRM system and sends alerts to individual sales traders because they have a customer who either has traded on the symbol, transacted above a given threshold, or has a real-time order for it. The correlation of these factors in real-time also includes real-time news sources. Also, I've worked on two algorithmic trading systems that host their strategies in StreamBase and perform real-time risk checks on the algos, and an ETF arbitrage system where there was only one strategy, but it was a very involved strategy.

Mike: We brought a smart order routing system up and running in 6 months with 2 developers, which we thought was very fast. We also got the business people involved in the canvas and the development - this is a very important factor in getting applications done quickly.

Talk about the benefits of StreamBase and visual programming

Phil: StreamBase helps facilitate communication. Whenever you want to convey information about a system, you always draw a picture. And they almost always look like StreamBase Event Flow. If you have taken the necessary care to clearly delineate modules, then, depending on the technical sophistication of who you are talking to, you can choose how far to drill down. It's extremely helpful in faciliating technical discourse; with the business users, it saves time.

Mike: Visual development has really improved the speed of our development. Especially when you're doing a proof of concept. It helps you understand the application's logic, and also communicate with the business; in typical programming environments like Java, you're not going to show many people the code, but you can do it with StreamBase.

Joseph: The big value is that all the technical details are abstracted away in terms of connectivity and low-level code; we can then communicate with the business owners, so development cycles are shorter and more efficient. The other value we have found is they we can very quickly build a prototype solution so the business can make decisions more quickly with respect to requirements and decisions to develop the applications.

Phil: One thing that's helpful: with StreamBase, you can tell at a glance that spaghetti code is spaghetti code. You can tell if an application was thoughtfully designed without hours and hours and hours of code review.

What best practices have you learned from your experiences with CEP?

Mike: You need to shift your thinking, to think in terms of events, rather than traditional static applications. And, just like any application, build a testing environment at the same time as your application logic.

Joseph: Identify the connection points quickly, be open minded and willing to unlearn the more traditional ways of development, use StreamBase interfaces, and make sure you continuously refactor the application.

Phil: There are lot of resources for ramping up StreamBase skills - invest the effort to develop your skills. The speed of putting together something quickly is a great asset. The thing to keep in mind is that although the development speed of StreamBase is a benefit in the short term, it can, if you're not careful, work against you in the long term.

Phil: While the principles aren't different, the practice IS different. The first thing that sold me is seeing the ease with which you can change and tune your threading model, take things in and out of the threading part of your system, with multiplicity and dispatch methods. Scalability and performance is about taking more advantage of the cores in a system, and careful management of it becomes more and more important.

What was your experience integrating other code into CEP?

Mike: We have separate routing logic written in Java for routing logic that we integrated with our StreamBase application. We've done a lot of work with custom Java operators in StreamBase - it's very straight forward - no problem in doing it. There's lots of power in developing in Java for code that isn't event driven that allowed us to bring in the power of algorithmic trading as a drop-in to the StreamBase environment. Also, we have a C# front end GUI, which uses the .NET interface from StreamBase - this was easy, although the challenge was how to take a GUI that isn't event-based and make it work with an event-driven application. We're in production with that now.

The StreamBase community

Phil: I've put two components onto the StreamBase Component Exchange - mostly adapters. The publishing process was straightforward. One is an adapter that watches for events on the file system and creates StreamBase events when files are created, renamed, modified, or deleted, and another one listens for calls on a web service interface and creates events for the applciation. I suggest other developers look at the utility classes – see what are other developers are doing, and pick up some tips and techniques and helpful hands-on examples of how they are building their applications. 

StreamBase Resources for Developers

Below are the StreamBase resources mentioned in this roundtable:

• Trial version of StreamBase: www.streambase.com/download
• Product documentation: docs.streambase.com
• Personalized demonstrations: www.streambase.com/forms/request-a-demonstration
• “How To” developer webinars: www.streambase.com/webinars
• StreamBase Component Exchange: www.streambase.com/sbx
• StreamBase University: www.streambase.com/sbu
• StreamBase Training and Certification: www.streambase.com/community/training-and-certification
• Datasports blog for StreamBase CEP developers: http://datasports.ca

Introducing Live BI and StreamBase LiveView

For some time now, StreamBase has been working on a new product concept that's a little bit CEP, a little bit data warehouse, and a little bit business intelligence. Today we've unveiled it as StreamBase LiveView. Our customers have called it a "game changer," and a product that "fills a massive hole in the BI space." Hyperbole like that needs to be backed up, so here we go.

Complex event processing (introduction on Google+) was envisioned by its inventors as a "database for moving data" as firms increasingly deal with data in motion from mobile devices, social media and digitized data of all types, and in all industries. In its first decade of commercial use, CEP has come to power automated trading on Wall Street. A who's who list of firms have been public about their adoption of CEP, so let's examine the live BI problem in the context of Wall Street.

Industry analysts estimate that over 60% of U.S. equities trading is automated. The problem is, with so much automation, it's impossible to see and act on what's going on, because a typical trading environment can generate 10's of millions of events a day. Existing business intelligence and data warehouse tools aren't used in the front office of Wall Street, because they're designed to summarize what's already happened, not what's happening now. 

StreamBase LiveView brings BI and streaming data together, for the first time. Applications that need Live BI are everywhere on Wall Street - trading risk management, position management, P&L management, surveillance, compliance, and trading itself, as Waters Technology magazine describes in the article that StreamBase Seeks 'Radical Transparancy' in Business Intelligence.

And applications in other markets can be found in any environment that has big data in motion: telecom (location aware insight and live network operations), security (live fraud detection, live surveillance), and e-commerce (live social media analytics, ad placement, big streaming analytics for Hadoop and Flume). These apps can be deployed on premise or distributed in the cloud. 

StreamBase LiveView is built with StreamBase CEP, which provides connectivity to over 125 event-based sources of data and the extreme scalability required by live BI. The server-based architecture is designed to process hundreds of millions of events per node, absorbed from a network of event sources such as an enterprise message bus or scalable event absorption layer like Flume or IBM Streams. At the front end, users can slice and dice streams of data with ad-hoc query tools in the StreamBase LiveView Desktop (shown above). But here's the key: once users ask the questions they want answered, StreamBase LiveView keeps answering their questions - as conditions change in real-time; users see an always-live view of streaming data.

For the user, all this technology means just one thing: they see business intelligence that's constantly fresh and relevant to their business operations. It's not a view of 5 days ago, or even 5 minutes ago. The data is live, the insight is live, and live insight can lead to live action.

Facebook, Twitter, Google+ give us a live view of what's happening in our social networks; StreamBase LiveVew gives firms a live view of their enterprise data.

To learn more, watch the StreamBase LiveView presentation, follow new materials, articles, and resources on Google+ (+StreamBase and CEO +Mark Palmer), read industry analyst Curt Monash, DMBS2.0, or follow StreamBase Systems (@streambase), CEO Mark Palmer (@mrkwpalmer ), and CTO Richard Tibbetts (@tibbets) on Twitter.

 

Can Innovative Surveillance Be a New Source of Competitive Edge?

In sports, great defense can be a catalyst for great offense. Can the same idea hold true for surveillance - the defense - in the capital markets? This was a question raised by today's panel of surveillance experts at today's StreamBase webinar (watch the replay). 

First, TABB Group's Miranda Mizan assessed the current surveillance market. She described how surveillance is morphing from a defensive obligation to an offensive differentiator, because the markets are changing so dramatically. Mizan argues that the quantity of data, speed of trading, and complexity of instruments have made traditional surveillance techniques insufficient. And since regulation (e.g., MIFID II) is changing so quickly, she argues that firms must re-think their surveillance strategies and technology choices.

Justin Amos joined Miranda. Justin is a former global markets sales and trading executive at JPMorgan, and the MD of Redkite Financial Systems, a UK-based firm whose system called Redeye has been quickly becoming the standard for modern, real-time surveillance systems. Justin described the requirements of today's surveillance systems by describing the motivation for Redeye, which he likened to "anti-virus software for trading."

Together, Justin and Miranda touched on 12 "next generation" surveillance requirements, including:

1. Real-time analytics. Surveillance systems must analyze market data at the speed of automated trading.
2. Historical "big data" analytics. On-line access to trade-by-trade history for investigation, trending, and discovery.
3. Data fusion. Mash-ups of increasingly complex data, including unstructured data.
4. Inclusion of new data sources. The ability to integrate live data, static data, and social media data.
5. Multi-asset class support. Modern surveillance must work across asset classes.
6. Pre-built surveillance algorithms. Standard algorithms like front running, spiking, tailgating, and ramping.
7. Adaptive algorithm modification. The ability to adapt algorithms on the fly to reduce false positives.
8. Alert algorithm extensibility. The ability to quickly add new algorithms to the system as market dynamics evolve
9. Customizable surveillance algorithms. The ability for a firm to tweak an algorithm for their own models.
10. Flexible workflow management and case management. Control over the process of alert analysis, resolution, and tracking.
11. Accessibility for business analyst. With regulation changing so quickly, more firms want their own control over surveillance behavior.
12. Cloud-based or on-premise deployment. The cloud allows more rapid rollout, and may ease global scalability challenges

The panel argued that existing surveillance systems can't handle these requirements. Taken together, the challenge makes what was once traditional "defense," or a compliance function, very new and challenging - a source of differentiation.

Miranda went deeper, with examples of how brokers and the buy side are using modernized surveillance systems: agency brokers are beginning to use surveillance tools to monitor their trading algorithms and optimize their client service and order routing for best execution; the buy side are using surveillance to gain greater control to know that their trading is safe, its orders get to the right places, and that dark pools are working correctly in relation to the rest of the market; sell side institutions are using surveillance tools to institute process and control across the organization, globally, and integrated with automated systems.

Richard Tibbetts, CTO of StreamBase, gave a brief introduction of StreamBase CEP, and why complex event processing is increasingly being used to power surveillance applications; indeed, CEP provides the engine to address many of the 12 modern surveillance requirements set out by the panel.

Simon Townsend, CIO of Redkite, gave a "whistle stop tour" around Redkites' application. He hit on most of the 12 requirements above, and even showed didn't just claim that their algorithms are extensible and dynamic (requirements 6, 8 and 9) - he showed it in StreamBase Studio, and showed how business analysts can extend the Redeye surveillance algorithms.

One of the most exciting times is when a market is transformed by new ideas. In the capital markets, perhaps that innovation is upon us in surveillance, where tools that meet modern trading requirements can turn a defensive obligation into a tool for innovation.

StreamBase Loves Erlang

The philosophy of Erlang and StreamBase align quite well. Quoting Mike Williams, one of the three inventors of Erlang:

• Find the right methods—Design by Prototyping
• It is not good enough to have ideas, you must also be able to implement them and know they work
• Make mistakes on a small scale, not in a production project

These philosophies of rapid development also apply to the way we've designed StreamBase; additionally, our customers also demand that much less technical users participate in the rapid implementation of ideas. StreamBase has a visual development approach that allows less technical, more "business" users - with mathematics, analytics, and business domain expertise - use the same tools as professional software developers.

What if we could add Erlang developers to that party? Event-driven, visual development, coupled with the power and rapid development of Erlang? 

Well you can. Darach Ennis (@darachennis) shows us how in his recent presentation at Erlang UG in London. 

StreamBase has a graphical, compiler-oriented programming language, targeted at serious developers, that can also be used by the less technical. Erlang programmers, and the Erjang folks with whom we share a virtual machine, do their thing in Erlang, and that code becomes a graphical "operator" on the StreamBase Studio pallet. In Darach's example he's used Erlang to implement a MACD analytic, and integrate that into a StreamBase-based trading system so it operates in the event-driven CEP application. 

The combination of StreamBase and Erlang combines the best of both worlds: StreamBase for rapid, graphical development to develop low latency applications that process 100's of thousands of events a second. Combining "how," "why," and "under what conditions," real time streams are matched and processed. StreamBase allows business analysts, actuaries, quantitative contributors and developers to share a common language. An event-driven integration whiteboard, if you will, that compiles to efficient optimized native binary code. Add Erlang for development of analytics like MACD and you have a powerful combination. 

Erlang support will be contributed soon to the StreamBase Component Exchange (SBX), our public "app exchange" for CEP integrations, applications, frameworks, and sample code. You'll already find support for Java, C++, R, Matlab, python, .NET. If you want to plug in some cool analytics written in C++, just do it. You want to exploit Basho's Riak? Sure, plug it in.

As a result, StreamBase allows developers control over how they exploit the facilities at their disposal. Use our graphical DSL for flow oriented algorithms and real time low latency high frequency data management. It's high performance from a human productivity and systemic perspective. 

Learn more by watching Darach's presentation here, and follow him on Twitter at @darachennis.

 

On StreamBase 7, Its Breakthrough Usability, and Performance for CEP

Today we announced SteamBase 7, the next major version of our market-leading Complex Event Processing (CEP) software. In press and analyst pre-briefings last week, we discussed a lot of cool new features. Here’s a quick tour.

A HUGE STEP FORWARD IN CEP USABILITY

Chief among the innovations of StreamBase 7 is the revamped StreamBase Studio. The interface innovations of the iPhone, iPad, and modern operating systems has made great interface design expected. StreamBase has always been the best and easiest to use CEP platform. But with StreamBase 7, our engineering team has taken a huge leap forward in usability.

Drawn from the cutting edge consumer user experience in platforms like Apple iOS and Windows 7, Studio 7 dramatically increases the speed and quality of application innovation. 

To achieve this usability, StreamBase 7 has new code organizational features, integrated annotation features for English descriptions of functions, gorgeous diagram rendering techniques, and facilities to isolate and encapsulate the necessary complexity of a sophisticated application. Literally, StreamBase users can now read StreamBase application code at a glance, and organize it effectively so that the Business and IT can collaborate in a way never possible before with any CEP product. These features that make this all happen include:

1. A beautifully redesigned user experience, including the use of transparency (similar to Windows 7 Aero), overlays to communicate information at a glance (e.g., iOS number-in-red-circle), gradients rather than solid color to ease readability (think in terms of Windows 7 and iOS) and, and complete color customization (a la Windows 7 themes.) 
2. Support for custom modules (groups of application logic) and extension point styles.  Better control over modularity allows IT to manage large applications, and manage how and where business users or other groups can contribute code. For example, a risk domain expert can create risk checks in one module while IT focusses on connectivity, performance and integration in other modules.
3. Advanced graphical code layout, with new group layout features and code rendering controls
4. A new rendering look and feel to ensure code is easier to follow at a glance
5. New mouse and keyboard gestures to increase EventFlow coding productivity, such as on-canvas renaming of components
6. New hotkeys that give the ability to instantly create CEP language operators
7. Layout-alignment shortcuts

The result is that business users and IT can work in one environment, with one set of tools, with one set of code, dramatically improving time to market, quality and ability to evolve systems once they are deployed.

In recent years, StreamBase deployments have become bigger, more global, and more complicated. So StreamBase 7 improves usability throughout the entire software development lifecycle, including:

1. New code refactoring schemes. So developers can quickly identify changes in their application logic and adjust the rest of the application more quickly as modifications are made.
2. 400% faster application execution while debugging. So problems can be identified more quickly
3. The ability to attach Studio to a running StreamBase server. For better diagnostics in deployment.
4. A new XML editor. Allows operations to more effectively configure StreamBase deployments.
5. Dynamic extension points that allow users to configure StreamBase applications to be configured quickly at runtime. When a new algorithm or analytic is produced by a developer or business users, this code can be dynamically inserted into the StreamBase code, tested, and deployed.
6. Support for 64-bit workstations for very large StreamBase application development

PERFORMANCE IMPROVEMENTS IN FOUR DIMENSIONS

From the first release of StreamBase, the focus has been on performance. Previous SteamBase performance innovations include the StreamSQL compiler, the ability to visualize parallelism and scalability, data parallelism and dispatch, and our open architecture for highly available systems all combine to create the fastest, most scalable platform for ultra low latency applications.

Version 7 of StreamBase continues to break down CEP performance barriers in four dimensions: ultra low latency connectivity, predictability of latency, FIX performance, and with high-speed computing technologies.

1: MICROSECOND-LEVEL LATENCY

All CEP platforms have an architecture for absorbing and emitting events. Some run out of the main CEP engine process, which can add significant latency overhead to event processing. StreamBase already features an super-fast in-process adapter architecture, but goes further in StreamBase 7 with an optimized “zero-context-switch adapter architecture” which allows StreamBase applications to process messages in a single thread from the time it enters the engine, through StreamBase logic, to the time it leaves the engine.  This architecture allows messages to pass through the StreamBase engine in microseconds, and has been optimized to speed up CEP application performance.

2: PREDICTABILITY OF LATENCY

The predictability of performance has been improved for Streambase 7 applications, as users now have access to server configuration options that provide the user with fine-grain control of the tradeoffs between ultra-low-latency and throughput. And, overall platform improvements have yielded over 20% improvement in application performance across the board.

3: ULTRA LOW LATENCY FIX CONNECTIVITY

Many StreamBase customers in the capital markets is already known for its low latency connectivity architecture and version 7 comes with over 22 new connectivity options. Many of those connectivity options leverage FIX, and in StreamBase 7 FIX connectivity has been dramatically enhanced. In benchmark testing, the new StreamBase FIX engine has clocked in at 50-75 microseconds send time depending on message size, all while running on Linux and commodity hardware.

4: HIGH-SPEED COMPUTING AND GPUs FOR MANAGING BIG DATA AND NOSQL STORES

The state of the art of ultra-low-latency is rapidly expanding, and many StreamBase customers have been on the leading edge of applying new technologies to deal with big data in innovative ways. 

GPUs are being used to continuously produce metrics, factors, and parametrics, contribute them to memory data structures for ultra low latency streams of data to adapt in flight to compute accelerated algorithms. They are also used to easily integrate accelerated compute and messaging or to perform collaborate distributed event processing in a high level flow oriented language. 

To support this trend for high-speed computing with StreamBase, we have added several connectivity options in StreamBase 7 to make it easier for developers to connect StreamBase to their existing and next generation IT infrastructure, including hardware-based middleware for ultra-low latency operations, including enhanced low latency middleware connectivity such as Wombat, 29West, Solace, and Tervela.

REACTION TO STREAMBASE 7

Analyst Adam Honore’ said at the end of our briefing: “wow, you’ve been busy!” and he’s right - as customers build more sophisticated applications with StreamBase, with bigger demands from the business users, and bigger demands from IT, with the ever-increasing need for speed, StreamBase 7 extends our lead as the best-in-class CEP platform in the industry.

The benefit of bringing the business and IT together is faster time to market, with higher quality. With tight communication, the business and IT make mistakes faster, learn from them faster, and innovate faster. As Thomas Edison said: “I make more mistakes than anyone I know. And sooner or later, I patent them.” 

But my favorite anecdote comes from one of our customers - Chris Lees, vice president of SunGard’s global trading business. He put it this way: "The usability of StreamBase Studio helped us develop a new Valdi product, we used it to describe requirements collaboratively with IT and because we were working with the actual code, we made better architectural decisions early that made for a better first release.”