The StreamBase Event Processing Blog

LiveView 1.4 – Historical Data Access

This is the second of three posts about the new LiveView 1.4 release.

StreamBase LiveView is a radical departure from traditional database and business intelligence products. On the surface, it’s like a database – users pose ad-hoc queries, and LiveView provides a data set that matches those queries.

But LiveView is different in two fundamental ways: first, queries are run against live, streaming data – like stock market data, operational analytics, or even Twitter data streams. Second, these queries are run continuously and query results are updated continuously. This continuous processing and re-evaluation of queries is what gives LiveView its name – you get a live view of data in motion.

LiveView 1.4 continues to expand the capabilities of this continuous query processing model, fusing historical data with real-time continuous queries of live data.

Fusing Live and Historical Data for End Users

LiveView provides real-time analytical visibility into data in motion. But once you give a head of trading a live view of his risk and P&L, the first question he will ask is, “How is my risk now different than yesterday’s risk? Than last week’s risk? Than last month’s risk?” Answering these kinds of questions requires the fusion of real-time and historical information. LiveView 1.4 makes integration of live and historical data easier.

LiveView 1.4 adds a new table type - JDBC tables. JDBC table support lets IT administrators connect to a historical database and define queries to be executed against that database. To an end user, LiveView makes JDBC tables look just like live tables, which allows users to access both types of data – live and historical – in the same way, with one user interface.

Historical data is often stored in data warehouses with support for SQL access, and it is with this kind of stored historical data that LiveView 1.4 is designed to integrate.

Fusing Live and Historical Data for Back Testing

Historical contextualization of real time data drives better decisions. For example, an order may not be “large” on an absolute scale, but it might be large for that client. A new algorithmic order that is underperforming the benchmark may represent the first time that client has ever used that algorithm. Looking at historical trading volumes, by client, or by security, can highlight changes in client behavior that may represent opportunities for relationship building.

JDBC table support in LiveView 1.4 helps LiveView customers test these assumptions by overlaying historical data with live data.

Using LiveView to Populate Historical Data Stores

Not all organizations have existing historical data repositories. In some cases new systems are created to store this data, and need to be populated. LiveView can easily populate historical databases based on the real time data it has captured, either with batch end of day loads or parallel capture.

In the batch scenario, a revolving set of data is removed from the live LiveView system (e.g., closed orders) and dumped into the historical store.

In a trickle scenario, the same LiveView publisher that delivers data to LiveView trickle loads the historical store using the JDBC table.

Live and Historical Data Fused to Provide Insight

The purpose of LiveView is to collect, analyze, and stage live data, enabling every decision to be made with up-to-date data. With the new ability to fuse historical data with live, streaming data, IT has new options to slice and dice both types of data, with one platform.

LiveView 1.4 – New Continuous Queries

This is the first of three posts about the new LiveView 1.4 release.

The big innovation of StreamBase LiveView is its Continuous Query (CQ) mechanism. In many ways, a continuous query looks like a standard database query – a predicate is sent to the server, and a result set is returned. But a LiveView query is a query that keeps being answered, continuously, and automatically. The LiveView server registers queries and, after the current snapshot answer is delivered, the server pushes subsequent matches to the query to the client to provide an always-live view of the query.

So LiveView query results are never out of date – always live.

Continuous query evaluation is a new area of technology innovation, and live query processing demands special features that aren’t found in traditional query processors. In LiveView 1.4, we introduce two major enhancements to the LiveView CQ model: Top-N queries and Time-Windowed queries.

Top-N Queries

Click to view full screen version on the StreamBase Facebook page

LiveView is often used to manage customer data - for example, many brokerages on Wall Street use
LiveView to continuously understand the state of orders and trades, and to decide how to optimize trading in real-time. A common requirement is for firms to understand: “Which clients are the most at risk?” or “Who is doing the most trading?” With LiveView, it is easy to create a pre-aggregated, real-time view of hundreds of such trading risk analytics.

The problem is, most brokerages have hundreds, or thousands of clients, and the amount of data can overwhelm some user interfaces used to analyze these metrics, such as Excel, or a web client, so our clients often create complex analytics to store only the most interesting real-time metrics to display to traders.  And because LiveView query results are constantly live and updating, sometimes it’s desirable to have the CQ mechanism pre-sort the results for you.

Now, with LiveView 1.4, a continuous query can narrow the amount of data automatically for clients. The CQ can now be parameterized to only show the top values in the table according to any sort key. For example, an Excel sheet might want to display only the 10 most at-risk counterparties, or the 100 largest open orders. A Top-N querie avoids data overload, reduces data clutter on the client, and helps focus user attention on the real-time data that matters the most.

As with all LiveView queries, Top-N queries are continuous. As data goes in or out of scope, rows are added or removed from the result set displayed automatically to users. If data values change, data is re-sorted, and new results may come into the view due to growing or shrinking of the result set.

The result is a new way to explore real-time data and to focus attention on what matters most when making real-time decisions.

Time-Windowed Queries

 

Click to view full sized image on the StreamBase Facebook page.

One of the unique attributes of event processing systems is that they are designed to understand the temporal attributes of streaming data. LiveView can order a table by the event arrival, and appended at the end of a table (e.g., new client orders as they arrive), or events can replace existing fields place (e.g., applying real-time pricing updates to existing client orders).

 

So streaming data has time dimensions that can be handled in a variety of temporal ways by LiveView, allowing decision makers to evaluate data based on its relationship to time. For example, a trader might want continuously view orders from the last 5 minutes organized by size, or open orders that have not been filled that were submitted over 1 minute ago.

LiveView Time-Windowed Queries allows users to restrict a query by “when” the event happened, based on one of these time dimensions. Only data that satisfies the WHEN condition will be displayed, and, once again, due to the CQ model of LiveView, is continuously updated and live.

For example, a query for open orders older than 1 minute would have data leave the view when the order closed. The display the trader looks at will be automatically updated by LiveView. Each cell in the table would be updated in place, as soon as its value changed.

In our other example - a query for the last 5 minutes of counterparty quotes  - will continuously slide forward in time, so a trader is continuously looking at the latest and greatest prices quoted in the market, without ever re-issuing a query, or hitting the “refresh” button on the user interface.

StreamBase handles data in a variety of ways. For example, an order might carry timestamps indicating when it was sent to the market, when it was acknowledged, when it was filled, or a timestamp applied by a third-party system. All of these values are available for time-windowed querying in LiveView.

Continuous Alerting on Top-N and Time-Windowed Continuous Queries

StreamBase LiveView also has native support for alerting. Administrators and end-users alike can ask to be alerted (via SMS, email, etc.) when a continuous query predicate is matched. In LiveView 1.4, alerting has been extended to work with these two new query types, Top-N and Time-Windowed queries.

LiveView – Making Real-Time Analytics Accessible and Actionable

When working with live data, time is the biggest factor. Traditional database-driven systems aren’t designed for real-time analytics; event processing systems are. By using Time-Windowed and Top-N queries in LiveView 1.4, firms can change the way they think about data and turn real-time information into action.

Algo Trading Saves Mark Cuban Millions During "Twitter Twitch"

Geeks on Wall Street make the market safer, turning a potential "Hack Crash" into a "Twitter Twitch," saving Cuban millons

Mark Palmer is the CEO of StreamBase Systems, a technology provider of real-time analytics technology.

Mark Cuban has provocative views on everything from the NBA draft to the Iraq war, so his latest shot at the financial trading industry comes as no surprise; the problem is, he lashed out at the Wall Street mavericks whose high tech trading systems worked perfectly, and probably saved him a lot of money.

In case you missed it, the Twitter account for the Associated Press was hacked, a hoax tweet was sent out that Barack Obama was injured in a bomb explosion. The Dow Jones fell 145 points in about 120 seconds and then recovered 134 points in the ensuing 3 minutes. So, in just 300 seconds, the DJIA traveled 279 points. 

Cuban told CNBC that "all of this algorithmically driven trading has created a ton of downstream problems," and proposed some "solutions" to the "problems" that include taxing trades that are held for less than one hour.

But here's where Cuban has the whole thing backwards: the technology he wants to shackle saved him, and the man on the street, millions.

On the day of the Hack, I was speaking about high frequency trading at a Bloomberg conference with 200 people in the audience. I asked the audience to raise their hands if they personally lost any money that day. Not one hand went up.

This wasn't a Hack Crash; it was a "Twitter Twitch." In fact, high frequency trading was exactly the antidote that saved all "regular" investors (even Mr. Cuban) money. 

It's true that Wall Street uses computers to make millions of trading decisions a second, and it's probably true that computers that detected the hacked tweet twitched in seconds. 

But those same algorithms that watch millions of data points a second could tell, in real-time, that something was amiss. The Tweet was algorithmically refuted - Obama was fine and healthy - the descent stopped immediately, and the markets returned to being healthy in just a few seconds.

Without real-time technology, the market would not have corrected so quickly; the hack created a twitch, not a crash, and the only ones in the market who lost money were probably the dumbest of algorithms that chased the lie. Most algorithmic trading insiders know that in the most volatile times, it's the human trader who tends to win; algorithms tend to get whip-lashed chasing purely statistical trends that they don't understand. 

Nobody, even a billionaire, can wish away the fact that the world is electronically fueled, and that social interaction, hoaxes, and trading are all done in real-time. To limit the use of real-time technologies on Wall Street would deny the markets the very tools it needs to make the market safer for everyone, and will continue to turn the next potential crash into a twitch.

Best Practice #5: Balance Man and Machine

This post is #5 in a series of lessons learned from leading StreamBase customers who have effectively deployed real-time business systems. 

The prospect of automating a business often sparks a debate about what is more effective, man or the machine. It turns out that the debate presents a false choice, and effective automation lies in the proper balance between the two.

Best practice #5, "Balance Man and Machine," is about how successful firms strike a balance between machine-driven and human decision- making.

THE SITUATION: FIRMS MUST AUTOMATE TO SURVIVE

In the capital markets, open outcry exchanges have been closing down for years. Last October, the IntercontinentalExchange closed down open outcry trading of soft commondities in New York, a practice that goes back at least 142 years. When it closed, the 1,000 traders left working on the floor "mostly monitored their market using handheld pricing screens even when they're sitting in the pits." That is, ironically, the manual brokers were literally using the automated tools that were making their jobs obsolete.

And the impact went beyond the survival of some jobs to the survival of some firms. For example, from 2008 to 2012, the once dominant NYSE and NASDAQ exchanges saw their combined market share slip from 67% to 45%, in large part due to upstart rivals BATS and Direct Edge known for their technology-based innovation. 

Obviously, automation has changed Wall Street forever, and it's about to change other industries.

THE PROBLEM: AUTOMATION THREATENS EXISTING CULTURE AND CAN ADD RISK

In the capital markets, the debate has raged on for years about what's more effective - human traders or algorithms.  A quick glance at a time-restricted Google search in 2007 for "Man Versus Machine" reminds us about how massive the debate was! 

Underneath this media coverage was a real issue: culture.

"Manual" traders, and the hierarchy of power over trading systems that was built over years by those who made their mark as manual traders, resisted the ideas that computers could do better than humans. Technologists, eager to make their mark on the P&L of the bank, fought to prove that computers could do a better job. 

At the same time, automation can introduce risk. Events like today's Twitter Hoax that President Obama had been injured at 1:07PM caused the markets to plunge 200 points in about 2 minutes. At 1:10PM, the first denials came out on Twitter, and by 1:13, the Dow had recoverd most of the ground it had lost.  So risk can no longer be done with traditional tools, which evaluate positions at the end of the day. By the end of the delay, it can be too late.

This polarized view of the battle between man and machine was wrong. 10 years on, automated trading is here to stay, and so are the people who guide its use. The truth is that neither man nor machine alone is better at trading; the truth is, a balanced approach is what was created. 

THE SOLUTION: FIVE STEPS TO BALANCE MAN & MACHINE

Automating businesses must change. Here are four observations from the transformation in the capital markets, and the best practices other firms in other industries can learn to facilitate their own change.

1) Business-geek hybrids are common

Traders used to have mostly degrees in business; today, they either additionally have math and science degrees, or have learned more about the implications of algorithms on trading. 

Best Practice: Proactively hire data scientists and put them into leadership positions so that they can lead a different, quantitative way of thinking about business transformation.

2) Humans are more comfortable taking cues from real-time analytics

Yes, lots of trading is automated. But automation also signals humans to make the big decisions. For example, Geneva Trading, one of the largest oil futures traders in the world, uses automation to monitor "millions of trading opportunities" simultaneously. Some opportunities are no-brainers, and the computer acts automatically. But sometimes the computer has no idea how to act, and punts the decision to a human by sending a text message, email, or alerting them on their desktop. 

Best Practice: Couple data scientists with operational staff that are similarly willing to embrace a technologically-driven customer resource management approach. These are customer service personelle who are used to analyzing information supplied by automated systems, and working with clients to solve high-value-add problems. The recent trend in capital markets toward execution consulting services is a good example of this trend. 

3) Real-time risk and surveillance becomes more important

One great myth about algorithmic trading is that it's done without oversight. In fact, all the clients I meet with use real-time analytics to temper, or stop, trading decisions when the data, statistically, does not make sense. One of our customers, PhaseCapital, was the inspiration for the blog post "Fast is Good, But Fast and Dumb is Dangerous" that describes the firms' real-time analytics that helped them trade more safely on the day of the May, 2010 Flash Crash. 

Best Practice: Employ real-time risk, surveilence, and compliance systems. One of the most prevalent applications of real-time platforms in the last 2 years has been real-time risk, compliance, and surveillence applications. 

4) Low-touch and high-touch functions must be integrated

Automation is useful to handle well known, low risk decisions, the ones that require little over site other than a big red panic button to make systems stop. Bigger, high touch decisions are ones handled by humans. By using computers to automate the low touch decisions, humans are empowered to make the tough, big decisions. 

Best Practice: Start with a balanced sense of technical requirements, rather than a world separated by automated and manual workflow. 

5) Infuse events with historical context

In the world of automation, the critical technology stack is event driven; the technology stack for manual processes is historical data driven. These two systems must coexist. Event-driven messaging, event-oriented programming languages (e.g., CEP), event-driven user interfaces, and event-oriented databases (e.g., tick stores) and streaming business intelligence must all sit along side of their historical counterparts. 

Best Practice: Integrate real-time and historical data to give humans oversight and insight into what automated algorithms are doing.  

THE IMPLICATION: AUTOMATION EMPOWERS HUMANS

Balanced well, automation can be a huge competitive weapon for a business. By applying lessons from one of the most deeply automated industries in the business world, firms embarking on automation initiatives  can increase their chance of success and reduce their risk of failure. 

Best Practice #4: Incorporate Statistical Computing within Event Driven Systems

This is a blog post is a collaboration between StreamBase consultant Adam Diamant and CEO Mark Palmer, from StreamBase. It is best practice #4 in the series “The Best Practices of Real-Time Innovation” featured on the StreamBase event processing blog.

Click to view the full StreamBase / Matlab example on the StreamBase Facebook page

During a recent StreamBase customer tour, a theme developed around customers who incorporate statistical computing with their real-time analytical applications. For example, integrating CEP with Python (NumPy, SciPy), R, MATLAB, LAPACK, and Quantlib helps users perform statistical analysis against information in motion.

So best practice #4 is "Incorporate Statistical Computing with Event Driven Systems."

THE SITUATION: QUANTITATIVE ANALYSIS STUCK LOOKING BACKWARDS

Harvard business review calls data science the “sexiest job in the 21^st century.” Even baseball teams like my own Boston Red Sox hire mathematicians to crunch data.

In most industries, data scientists are still looking at information in the rear-view mirror, simulating outcomes based on historical data. Most seek to predict long-term trends and make decisions on daily, weekly, monthly, or yearly boundaries.

But an increasing amount of data science challenges are being applied in the present, based on data that’s as close to live as possible. For example, in finance, quants use real-time statistical analysis on data that’s sometimes less than a millisecond old to perform real-time portfolio optimization, real-time asset liability modeling, real-time quantitative risk modeling, and real-time derivative pricing.

As mobile computing, sensor technology, and big data technologies continue to advance the volume, velocity, and variety of data accessible to data scientists, they will increasingly be required to analyze information in real-time, as it streams by.

THE SOLUTION: USE CEP AND STATISTICAL COMPUTING PACKAGES TOGETHER AGAINST LIVE DATA

Numerical computing environments (we'll use MATLAB for an example, since it's widely known) are often used for algorithm development, data analysis, visualization, matrix manipulations, and statistical analysis. In finance, quants use it to do time-series analysis, portfolio optimization, asset liability modeling, quantitative risk modeling, and derivative pricing.

These two environments can be integrated to perform continuous, on-the-fly, statistical analysis of data in motion, enabling automated decision making and online evaluation of live business conditions.  The diagram above shows a working system that uses StreamBase to continuously absorb a stream of events from an external source (e.g., stock market data, sensor data, customer transactions) and be fed into MATLAB for real-time analysis of a stream of information.

StreamBase is used to connect, normalize and clean large amounts of streaming data. Next it calls MATLAB for statistical analysis and then StreamBase takes an action based on the results of the calculations.  For example, in the diagram above, a matrix is passed from StreamBase to MATLAB, a QR decomposition is performed, and the result is passed back into StreamBase for use downstream.  All of this is done in real-time by leveraging StreamBase’s ability to handle large amounts of streaming data with MATLAB’s ability to perform complex mathematical calculations.

The 6-part StreamBase tutorial by Adam Diamant demonstrates how to integrate MATLAB and StreamBase. The session covers:

1)   Introduction: why incorporate StreamBase and statistical computing?

2)   Opening and closing the MATLAB connection in StreamBase

3)   Passing data from MATLAB to StreamBase

4)   Evaluating expressions in StreamBase

5)   Evaluating functions in MATLAB

6)   Example integrations

The code used in the tutorial is free and available for download on the StreamBase Component Exchange (SBX) for those with a licensed copy of MATLAB. You can also download StreamBase for free at www.streambase.com.

THE IMPLICATION: FAMILIAR STATISTICAL ANALYTICS, BUT IN REAL-TIME

The resulting system architecture helps bring traditionally backwards-looking analytical tools into the world of real-time computing. Out-of-the-box integration makes it easy to bridge these two worlds – real-time and historical analytics – in just minutes, which opens the door to the data scientist to apply their algorithms to not only observe, but act on business conditions, while the actions still matter. 

Post Script:

Some folks have had problems downloading the full resolution image of this app. Here it is:

 

How to Analyze Sensor Data in Real-Time with CEP

This is a blog post was contributed by senior field architect Mark Hudson from StreamBase's London office. 

    The Moving Finger writes; and, having writ,

    Moves on; nor all your Piety nor Wit

    Shall lure it back to cancel half a Line,

    Nor all your Tears wash out a Word of it.

    -- The Rubaiyat of Omar Khayyam 

The famous Persian poet was referring, I suppose, to the impossibility of living one's life over again.  Fortunately, in the world of stream processing, things are easier - with the right tools.  It is indeed feasible to watch what the moving finger of real-time data writes.  Better still, yes you can go back - at least to extract valuable insights while there is still time to avoid shedding tears.

Typical StreamBase customers in financial markets are avid consumers of time-based data - "tick data" in the industry jargon - for pricing trades and spotting profit opportunities.

Outside financial markets, there’s increasing interest from every kind of company in the potential of CEP.  Very frequently, we hear the need to mine usable insights from a flood of "sensor data".

SENSORS AND (NOT) SENSIBILITY

What's a sensor for this purpose?  Well, it could be a gadget in a pump or a generator, way out in the wilds (sometimes literally - in the depths of a forest maybe).  It could be an SMS-like message from an aircraft flight computer.  It could be a sensor on a door, part of a burglar alarm system.  One among many, connected to a remote monitoring centre.

The common requirement here is figuring out the signal from the noise.  Sensors generally aren't clever.  Usually, they are cheap, reliable and rather dumb.  They send a lot of repeat messages.  Each sensor usually reports a limited number of measurements or alarm conditions.  The sensors aren't clever enough, or don't have enough context awareness, to report more than bare, isolated facts.

The result?  The central monitoring function is "flooded" by very large amounts, in total, of individually low-quality (high noise) signals.  As often as not, time is a factor, in figuring out the real signal.  Events - signals, alarm conditions, unusually small or large measurements - mean a great deal more when several happen in a short burst.

MAKING SENSE OF SENSOR DATA

It's usually even more significant if a certain set of events happens in a specific order.  When I get a migraine it’s flashing lights then headache then nausea in that order.  A “march of events” as my doctor calls it.

Suppose we have a generator with temperature, voltage and vibration sensors.  A reading of 160C happens typically several times a day.  The sensor reports every minute, so an individual "hot" period can result in 5-10 "high temperature" alerts but this means little in itself.  Similarly, the under-voltage and excessive vibration indicators happen all the time.  Crying wolf, if you like.

A much more significant sequence would be something like the following:

1) Vibration alarms being sent for 10 minutes then

2) Temperature consistently higher than the hourly average for a sustained 20 minutes then

3) Under specified voltage by 10% for 20 minutes

and all these things happen in the same hour in that order

HIDDEN MEANING IN THE SENSORY DELUGE

You can probably see where I'm going with this.  There is a worn bearing that causes a cooling fan to malfunction for a while.  This lets the temperature go up, at which point a feedback loop in the generator controller throttles back the voltage until things cool down.

Normally, say with a database and a language such as Java, it would be very difficult to detect these kinds of multiple, complex, ordered & time-dependent conditions.  But with CEP, we can build a system with just 20 visual elements!

Here's our generator scenario written in StreamBase's visual EventFlow language.  It starts with a flow of sensor messages.  These typically come from an “input adapter” such as the IBM WebSphere MQ example shown in the picture.  (StreamBase has over 150 other adapters too).

Finally – though not shown explicitly here – we would take the “bearing problems” combined signals, and act on these.  Maybe we send an email or SMS to the operations team.  Maybe we send a message directly to the generator to shut down.  Maybe we log the event in a database for inspection every morning.  Any or all of these responses would be equally easy to plug in.

CEP, such as the StreamBase EventFlow here, makes it very natural to express the solution.  You can do some very complicated programming easily.  It looks and feels the way an engineer would describe the requirements.

By the way, here’s the StreamBase configuration panel for that clever little Pattern operator:

 

As you can see, there is a specialized mini-syntax in here for event ordering.  The ‘input1’ is the complex vibration alarms criterion, the ‘input2’ is the complex temperature criterion, and the ‘input3’ is the voltage criterion.  (That isn’t obvious from the top-level EventFlow diagram, but if you were to look into the details, you’d find it so).  So we’re telling the pattern operator to watch for the sequence vibration criterion followed by temperature criterion followed by voltage criterion, as in the original specification.  We could even pick & choose further between the details of our three ordered event criteria, by using a more complex Predicate than just the ‘true’ value here.  But you’d need to see more details of the underlying operator settings to understand this fully.

 

AGILE EVENT HANDLING

Just as important, the specialist engineer can understand the logic, and can work with the developer to achieve the correct pattern detection.  I'm not a generator engineer, so there are probably some flaws in my scenario.  But it doesn't really matter.  The right engineer would be able to spot patterns that are diagnostic of underlying issues.  That would include problems serious enough to justify a service visit - even to that generator, far out in the woods, powering an aircraft warning light.

Just as important: even expert engineers might not know all the important patterns already.  Playing around with the data allows new patterns to be spotted.

The burglar alarm scenarios are even easier to think up.  Three intruder alerts on five successive days at different doors belonging to the same company anyone?  But our recognition of such "insightful" patterns grows all the time.  So, it's important to trap new combinations in an agile way.

In conclusion, I can’t promise “Piety nor Wit”.  That’s too much for any software (even StreamBase!). 

However, with a clutch of Aggregate operators, a Pattern operator, and few others, finding true sense in the data could at least be yours.  You may even get your engineer out into the woods before the moving finger spells disaster.  Perhaps even Omar Khayyam would have been impressed!

Drop me an email if you’d like to see the details of a StreamBase EventFlow solution like this for sensor monitoring. 

 

The Best Practices of Real-Time Innovation

Event-driven systems are new to many firms. For many, automating existing manual processing, using event-driven techniques, properly simulating real-time analytics systems is a new experience. 

Fortunately, some industries have been on the forefront of designing for real-time. Namely, banks in the capital markets trading arena have been automating and building event-based applciations for years. 

So on a recent customer tour with our customers, we've been asking them for their best lessons learned and the business impact the systems they've built have made. 

Best Practice #1: Build Apps So Quickly You Won't Mind Throwing Some Away

One IT executive told me "CEP allows me to build event-driven apps in 2-4 weeks, rather than 6-9 months with an app server and database - so I can say "yes" to more requests from their business clients, which helps us innovate more quickly. We keep the great ideas, and discard the ones that don't work."

Best Practice #2: Start Small, Succeed, then Scale

The most consistently successful firms that moved to real-time on Wall Street started with a small, critical application, learned lessons from its development, then applied those lessons learned to other projects and groups. Over time, they developed a center of excellence with firm-wide sucess in real-time application development.

 

Best Practice #3: Plan for Spikes, Not Aggregate Volumes

Systems that use primarily historical have trained many technologists to capacity plan their systems for the aggreate amount of data they manage. Real-time systems are different, and should be planned for spikes in volume, rather than only aggreate volumes. This best practice explores three approaches: burst behavior, the use of conflation, and computational reduction, that helps applications gracefully manage spikes and deliver correct and timely response to users and automated action. 

Keep coming back throughout 2013 as we continue to add to our library of real-time system best practices!

Best Practice #4: Incorporate Statistical Computing with Event Driven Systems

Numerical computing environments are often used for algorithm development, data analysis, visualization, matrix manipulations, and statistical analysis. In finance, quants use it to do time-series analysis, portfolio optimization, asset liability modeling, quantitative risk modeling, and derivative pricing. This best practice describes how to does this computing in real-time, using MATLAB with StreamBase CEP together.

Best Practice #5: Balance Man And Machine

The prospect of automating a business often sparks a debate about what is more effective, man or the machine. It turns out that the debate presents a false choice, and effective automation lies in the proper balance between the two.

This article explores 5 steps to balance man and machine, including the infusion of more data scientists in executive positions, the application of real-time risk management and surveillance, the infusion of historical and real-time analytics in one place, and more.

 

Best Practice #3: Plan for Spikes, Not Aggregate Flow

This post is a part of a series of best practices from leading StreamBase enterprise customers who have effectively deployed "real-time" business systems. 

"Can StreamBase handle 300 million messages a day?" asked a senior IT manager the other day.

"Well, it's common for a single StreamBase server to handle 100,000 messages a second for applications like yours," said our field architect. 

"OK, but can you handle 300 million messages a day?" 

Why this bizarre miscommunication? 100,000 messages per second is 8.7 BILLION messages per day, yet the IT manager was worried StreamBase wouldn't handle his volumes.

Experienced event processing architects think about scalability in a different way. So, for this best practice blog entry, I teamed up with one of our most senior field architects, Hayden Schultz, to describe the art of scalability design and how to think about scalability in real-time systems. 

THE SITUATION: AGGREGATE DATA CAPACITY PLANNING IS COMMON PRACTICE

Many technologists think about capacity planning in terms of aggregate data volumes. That is, they predict how much data they need to process in a day and provision their network, database, and hardware accordingly.

THE PROBLEM: REAL-TIME DEMANDS DIFFERENT METRICS

Aggregate volume isn't the right way to plan real-time systems.  Real-time systems often have a 100-to-one, 1,000-to-one, or even a million-to-one ratio of event-computation-to-action ratio. Technologists need to think in terms of stream burst rates, and what the right behavior is during bursts.  

THE SOLUTION: THINK IN THREE SCALABILITY DIMENSIONS 

There are three dimensions to scalability in real-time systems: 

1)  Bursts behavior. It's not uncommon for bursts to be so narrow that you don't really need to design around them because while performance will degrade during the burst, the burst is fairly brief so the system performance degrades so briefly that it's not noticeable. A stable real-time system is designed so that the maximum data rate that it's capable of processing is greater than the average data rate, so if a burst is reasonably narrow, the latency of the responses increases during the burst while incoming data is buffered; when the burst is over, the backlog of data is quickly processed and the system resumes its normal performance.

2) Consider Conflation. Increasing the response latency during bursts can be the wrong approach for some systems, but the cost of designing for spikes 3-5X higher than the average can be expensive. A technique called conflation helps mitigate this issue. Processing events that are buffered during bursts has the unfortunate tendency to process events that have already been superseded by more recent events, so the system is using its overtaxed resources to compute results that are obsolete while letting the most current results age and perhaps become obsolete in turn. Conflation means that new events overwrite processed buffered events so that only the most current data is processed. The data rate is reduced and obsolete events are not used. Conflation may slightly increase average response time, but can dramatically reduce latency during traffic spikes. 

3) Computation reduction. Some systems have computational loads so that a new result cannot be produced for each input event. These systems typically produce results at a fixed frequency. This is common when the result is sent to a graphical user interface (GUI). A typical GUI can't update nearly as quickly as many CEP applications can produce results, and even if it could the updates would overwhelm the users trying to use the GUI. Sending results at a fixed rate solves this problem. If the data rate of events entering the CEP system should become too large a common strategy is to reduce the computation frequency. When the burst is over, the computation frequency can be increased to its normal rate.  

THE IMPLICATION: CORRECT SYSTEMS AT THE RIGHT COST

Planning system capacity in terms of burst rates, data reduction, and computational reduction helps balance correct system behavior and cost, and turns handling a billion messages a day into child's play.

StreamBase and CEP Usage Expands in China

The trading market in China for event processing continues to grow.

Today, we announced a partnership with Zhejiang Hithink Flush Information Network Co., Ltd. (“Hithink”), who will use StreamBase to build trading products for the Chinese market, and also sell and support CEP-based application development in China. This partnership comes as a result of our StreamBase ISV Acclerator Program and Gunagfa's public adoption StreamBase last year.

Hithink has a history of successfully building market data and trading systems for their retail and institutional clients, with over 1,000 employees, 50 million customers and revenues of over 200M RMB.

Hithink will use the StreamBase platform to improve their existing infrastructure and develop new high performance trading applications such as algorithmic trading, transaction cost analysis, and market data management that make use of key CEP features such as aggregation, correlation and temporal analytics. 

Image Credit: Want China Times

Best Practice #2: Start Small, Succeed, then Scale

This post is the second in a series of best practices gleaned from leading StreamBase customers who have effectively deployed "real-time" business systems.

Even though CEP and stream processing is being popularized in marketing campaigns such as IBM's "Smarter Planet" and TIBCO CEO Vivek Ranidive's book "The Two Second Advantage," the reality is that, for many organizations, a real-time, sense-and-respond business model is still elusive. 

A consistent best practice employed by our customer base is to "Start Small, Succeed, then Scale."

THE SITUATION: REAL-TIME BECOMES A "MUST DO" AGENDA

On August 1st, 2012, Knight Capital lost $440 million in 45 minutes when their real-time algorithms went out of control. That kind of event caught the interest of most CEOs in the capital markets who all of a sudden put real-time controls, risk management and reporting high on their list of "must do" IT projects. 

That said, many would be surprised to know how little of Wall Street is actually automated with respect to the management function of high-speed trading. Many of the largest firms process 100's of millions of trading events a day, yet still manage their business by looking at historical information.

But moving to real-time isn't just a matter of installing new, faster hardware - it's a shift in thinking, a shift in culture. That can mean implementing new IT infrastructure from scratch, or even hiring new teams to manage old business processes that used to be more manual. 

THE PROBLEM: LEGACY SYSTEMS AND PEOPLE NEED AN OVERHAUL

When it comes to trading, most banks have great real-time technology in place. But when it comes to managing that trading activity, they're still often using rear-view-mirror reporting technology like traditional business intelligence tools. You can see this in software vendor market size by comparing the business intelligence market (around $10B) to CEP and the new "Live Business Intelligence" segment (around $350M, and growing). 

But the problem isn't just about technology - the staff required to manage real-time business processes are more quantitative in mind-set. For example, heads of risk management usually understand mathematics, and the models used to trade, because the algorithms themselves introduce operational and cost risk. 

So, given that people, processes, and technology typically have to be transformed, the most successful clients in our customer base have taken an incremental approach, by deliberately transforming the most critical elements to start with. 

But how?

THE SOLUTION: CHOOSE A CRITICAL "FIRST DOMINO" TO PUSH OVER

The transformation in the capital markets provides a good template on how to transform from rear-view mirror to real-time.

12 years ago, less than 5% of trading was driven by computer algorithms; now, around 60-70% of trading volume in the U.S. market is guided by algorithms. Banks chose their most critical operation - trading - to focus on first. The most mathematically oriented traders survived; pure "relationship" oriented traders have waned. Now, a dozen years later, back-office, clearing, settlement, and risk management are all moving to real-time.

Trading was the first domino to be pushed over. 

How did they do it?  The pattern went something like this:

1. Choose an important function (e.g., trading)

2. Put your best people on it (choose the new model of people required for mathematics-based operations)

3. Build the new system

4. Fail

5. Return to step 3, as necessary until you get it right

6. Capture learning, choose new leadership that "gets" the new model

7. Repeat with adjacent business functions

8. Build a center of excellence

9. Scale (pick the next system to transform based on the lessons learned).

THE IMPLICATION: REDUCED RISK; LONG TERM EFFECTS

The implication of starting small and planning to fail with an important domino is, ironically enough, reduced risk. Risk is reduced because by choosing an important, small function, you can fail faster. You can choose to put your top talent on the project. You can adjust more quickly. And, for long-term sustainability, you create a core center of excellence that the rest of the organization can leverage as you roll systems out over time. 

THE LESSON: ACHIEVE LASTING TRANSFORMATION TO REAL-TIME BY STARTING SMALL

So by starting small, succeeding with that small project, then scaling out the rest of the enterprise, the organization lowers the risk of early failure and starts on the long journey of employing a disruptive shift in thinking that can last decades.