The StreamBase Event Processing Blog

LiveView 1.4 – New Continuous Queries

This is the first of three posts about the new LiveView 1.4 release.

The big innovation of StreamBase LiveView is its Continuous Query (CQ) mechanism. In many ways, a continuous query looks like a standard database query – a predicate is sent to the server, and a result set is returned. But a LiveView query is a query that keeps being answered, continuously, and automatically. The LiveView server registers queries and, after the current snapshot answer is delivered, the server pushes subsequent matches to the query to the client to provide an always-live view of the query.

So LiveView query results are never out of date – always live.

Continuous query evaluation is a new area of technology innovation, and live query processing demands special features that aren’t found in traditional query processors. In LiveView 1.4, we introduce two major enhancements to the LiveView CQ model: Top-N queries and Time-Windowed queries.

Top-N Queries

Click to view full screen version on the StreamBase Facebook page

LiveView is often used to manage customer data - for example, many brokerages on Wall Street use
LiveView to continuously understand the state of orders and trades, and to decide how to optimize trading in real-time. A common requirement is for firms to understand: “Which clients are the most at risk?” or “Who is doing the most trading?” With LiveView, it is easy to create a pre-aggregated, real-time view of hundreds of such trading risk analytics.

The problem is, most brokerages have hundreds, or thousands of clients, and the amount of data can overwhelm some user interfaces used to analyze these metrics, such as Excel, or a web client, so our clients often create complex analytics to store only the most interesting real-time metrics to display to traders.  And because LiveView query results are constantly live and updating, sometimes it’s desirable to have the CQ mechanism pre-sort the results for you.

Now, with LiveView 1.4, a continuous query can narrow the amount of data automatically for clients. The CQ can now be parameterized to only show the top values in the table according to any sort key. For example, an Excel sheet might want to display only the 10 most at-risk counterparties, or the 100 largest open orders. A Top-N querie avoids data overload, reduces data clutter on the client, and helps focus user attention on the real-time data that matters the most.

As with all LiveView queries, Top-N queries are continuous. As data goes in or out of scope, rows are added or removed from the result set displayed automatically to users. If data values change, data is re-sorted, and new results may come into the view due to growing or shrinking of the result set.

The result is a new way to explore real-time data and to focus attention on what matters most when making real-time decisions.

Time-Windowed Queries

 

Click to view full sized image on the StreamBase Facebook page.

One of the unique attributes of event processing systems is that they are designed to understand the temporal attributes of streaming data. LiveView can order a table by the event arrival, and appended at the end of a table (e.g., new client orders as they arrive), or events can replace existing fields place (e.g., applying real-time pricing updates to existing client orders).

 

So streaming data has time dimensions that can be handled in a variety of temporal ways by LiveView, allowing decision makers to evaluate data based on its relationship to time. For example, a trader might want continuously view orders from the last 5 minutes organized by size, or open orders that have not been filled that were submitted over 1 minute ago.

LiveView Time-Windowed Queries allows users to restrict a query by “when” the event happened, based on one of these time dimensions. Only data that satisfies the WHEN condition will be displayed, and, once again, due to the CQ model of LiveView, is continuously updated and live.

For example, a query for open orders older than 1 minute would have data leave the view when the order closed. The display the trader looks at will be automatically updated by LiveView. Each cell in the table would be updated in place, as soon as its value changed.

In our other example - a query for the last 5 minutes of counterparty quotes  - will continuously slide forward in time, so a trader is continuously looking at the latest and greatest prices quoted in the market, without ever re-issuing a query, or hitting the “refresh” button on the user interface.

StreamBase handles data in a variety of ways. For example, an order might carry timestamps indicating when it was sent to the market, when it was acknowledged, when it was filled, or a timestamp applied by a third-party system. All of these values are available for time-windowed querying in LiveView.

Continuous Alerting on Top-N and Time-Windowed Continuous Queries

StreamBase LiveView also has native support for alerting. Administrators and end-users alike can ask to be alerted (via SMS, email, etc.) when a continuous query predicate is matched. In LiveView 1.4, alerting has been extended to work with these two new query types, Top-N and Time-Windowed queries.

LiveView – Making Real-Time Analytics Accessible and Actionable

When working with live data, time is the biggest factor. Traditional database-driven systems aren’t designed for real-time analytics; event processing systems are. By using Time-Windowed and Top-N queries in LiveView 1.4, firms can change the way they think about data and turn real-time information into action.

Potamus Selects StreamBase LiveView

Today, Potamus Trading, a wholesale broker-dealer based in Boston, announced they have selected StreamBase LiveView as a key element of their infrastructure and trading operations. StreamBase LiveView will be used to analyze trading performance, operations and financial risk in real-time. 

The 2010 Flash Crash, the more recent $440M loss by Knight Capital, and the inexorable rise in regulation and complexity in the electronic trading infrastructure make insight, control, and risk management capabilities increasingly important. StreamBase LiveView was created to address these rising demands in the capital markets, and today's selection by Potamus Trading is another example of why.

“As a broker-dealer in today’s market, it is important that we are also a partner, a reliable liquidity provider, an electronic trading advisor, and an expert on algorithmic execution,” said Eric Pritchett, CEO of Potamus Trading. StreamBase LiveView allows Potamus to incorporate real-time analytics into their trading infrastructure assessing trade performance during the day, while trading is live. LiveView is like a data warehouse, which provides interactive analysis, but the data is literally live - ad hoc queries are executed against live flow.

This live view of trading activity and LiveView's real-time ad-hoc query capabilities allow Potamus to respond quickly and help their broker-dealer customers optimize their executions.

This visibility, according to Potamus, also contributes to operational and financial risk management.

Potamus Trading's CEO Eric Pritchett discussed his view on the rising need for execution consulting services in this week's roundtable, moderated by Sang Lee, co-founder and managing director of AITE Group and included Bernard Donefer, Associate Director, Subotnick Financial Services Center, Baruch College, and Richard Tibbetts, CTO of StreamBase. 

StreamBase LiveView Demonstration and Animated Short

In the first week since the "worldwide premier" of our two StreamBase LiveView videos at the Securities Industry and Finanical Markets Association shown in NYC, we've had almost 1,000 views on Vimeo and YouTube. This product is bringing the power of CEP to entirely new markets and users, and it's very exciting to see interest in it grow so rapidly.

Here's the StreamBase LiveView demonstration video:

 

And our animated short, produced by Grumo Media (thanks, Pedram!)

 

Execution Consulting Services 2.0 Defined

AITE Group’s most recent survey of Wall Street reveals that the vast majority of broker-dealers feel that the importance of their customer service, or "Execution Consulting Services" is more important than quality of their trading algorithms. 

87% of respondents view the manageability and intelligence regarding their trading systems as more important than the actual algorithms behind the strategies themselves.

The majority of broker-dealers in the AITE Group survey now feel that "low touch service models" are no longer acceptable. Low touch (see figure) means the traditional, reactive customer support model – the customer calls and eventually the organization reacts. According to AITE Group’s survey, broker-dealer respondents reported an average 3-5 day turnaround time to resolve buy-side inquiries.

AITE Group’s report defines a next generation approach to client support that they call "Execution Consulting Services 2.0" which attempts to leap frog the traditional approach and drastically reduce the average 3-5 day response to close to real-time.

AITE Group defines ECS 2.0 as having 3 main elements:

1. High-touch advisory services concerning the optimal use of electronic trading service

2. Analytics tools for trade execution quality, risk, and transaction cost

3. Real-time tools that provide live insight, not just rear-view mirror historical reports

ECS 2.0 though isn't just the job of just one person. AITE Group sees the ECS group consisting of a cross-functional team including relationship managers, market structure experts, traders, IT, quants, and research analysts within the Electronic Trading Group.  In order to provide high-touch ECS 2.0 services there is growing need for support tools that can provide a single, unified view of customer order flow and standing with the firm.

The AITE Group report goes on to identify the key technologies that broker-dealers need to better support their customers:

1. Active alerting (79%)

2. Pre-trade and post-trade trade analysis (79%)

3. Single view into client data (57%)

4. Real-time trade analysis (50%)

5. Real-time network monitoring (43%)

6. Ad-hoc query of real-time data flow (36%) 

We built StreamBase LiveView to address these requirements, even though at the time we hadn't heard the term "Execution Consulting Services" used widely. But this is exactly what our development partner, and our freshman class customers for StreamBase LiveView 1.0 asked for and use StreamBase LiveView for today.

As a result, LiveView is beginning to bring event processing to the masses. Traditionally, CEP has been at the heart of apps that need machine-to-machine automated processing, like algorithmic trading. But LiveView brings that model to apps that require machine-to-human interaction, like ECS. LiveView processes streaming, real-time data and harnesses it in a way that humans can use to manage automated infrastructure like automated trading. These applications require real-time analytics, the aggregation of streaming data that can change millions of times a day, continuous query processing and in-memory data warehousing. ECS2.0 teams require ad-hoc, interactive visualization of that warehouse, the ability to slice and dice data at will, and an alerting system that pushes events to ECS staff.

For more information, read the AITE Group report or sign up for the roundtable on ECS taking place next week, or watch the StreamBase LiveView Demonstration at www.streambase.com/liveview where you will also find technical whitepapers, case studies, and lots more.

 

Gain Capital: A Glimpse into the Future of Analytics?

Imagine trying to drive your car while ONLY looking in the rear-view mirror? If you're like most businesses, you make your decisions by looking at reports, which is like driving while only looking in the rear-view mirror.

First generation analytics and BI tools - Excel, Tableau, Qliktech - provide a map of where you've been. They're so essential that they are a $10 billion software market. But using old data to drive your business only tells you what potholes you already hit and what turns you could have made.

StreamBase LiveView provides a live dashboard of where you should go next. It analyzes live data; not old data. LiveView absorbs data streams with just milliseconds of latency - sensor data, high-speed market data, client orders, streaming news, tweets, FIX messages, network messages, emails, log data. LiveView gives you a live view of data - before everyone else sees the report tomorrow!

Live insight helps you see potholes before you run over them, and decide where to turn next. For a more complete overview of LiveView, watch the 4-minute intro presentation here. 

Today, GAIN Capital went public about their use of LiveView and how they use it to steer their business differently. Rob Perez, VP of Trading and Risk Management Systems at GAIN Capital, shared a terrific example of the transformation live analytics can bring. GAIN's business is trading, and success is linked to the quality of connectivity to the electronic markets. Gain services over 140 countries, with an average monthly transaction volume of over $275 billion. 

Previously, GAIN managed their trading operations with traditional tools - databases, log files and business intelligence. At the end of the day or week, reports are generated, problems are discovered, and addressed.

Now, LiveView helps GAIN understand and optimize their trading operations in real-time. Perez said in EuromoneyFXNews: "We an see if any metrics are moving outside of our comfort zone, and we can immediately reach out to our liquidity providers to understand what's going on."

At any time, GAIN clients can trade on any of 10 or more markets. It's essential to GAIN that problems with prices, connectivity, or quality are elimintated. Now, GAIN can now determine:

• What deals are being rejected now?
• Which liquidity providers are giving good prices now?
• Which liquidity providers are having problems now?

Live insight helps GAIN decide to call a poorly performing partner the moment there's a problem. Finding a problem as it's happening makes it easier to describe and easier to fix. And finding problems in real-time gives GAIN the opportunity to correct it before their clients notice. If the issue is with a partner who can't fix it on the spot, Gain can adjust and route around that market. 

StreamBase LiveView is the first platform of its kind to provide a complete analytics solution for live data and real-time action, which we believe could be a new disruptive physics that allows companies to broaden the decisions they make by looking ahead, not only behind. 

So, what decisions would YOU make differently if you stopped looking only in the rear-view mirror?

 

Three Stories from StreamBase LiveView Customers

A year and a half in the making, we're thrilled to release StreamBase LiveView 1.0 (press release). Since we released the concept video last November (which drew almost 4,000 views on YouTube), our early access program has been a huge success. We'll announce an exciting set of customers in the coming weeks from that program.  

What's all the excitement about? While the technology is awesome (download this great StreamBase LiveView Technical Paper by CTO Richard Tibbets, Steve Yang, Rob MacNeil, and Dave Rydzewski), it's really all about the value our customers get from the stuff we build.

Here were three "aha" moments I had with those visionary adopters that, for me, crystallize why LiveView is so special.

1. Real-Time Service Experience Management. Recently the head of trading at a big bank told me that when a big buy side client like Fidelity calls to ask about the performance of a trade, they:

• Issue a request to IT
• IT searches log files, databases, and reports to gather raw data
• IT analyzes the data with one or more of: a quant, a risk expert, an execution expert, and a market structure expert
• The team writes a report
• The team briefs the sales trader on the account
• Finally, after 3-5 DAYS - they explain the situation to the client
• They decide on a course of action
• Adjustments are made - algorithms or strategies are changed

No more. Now, LiveView helps them make many of these decisions in seconds. LiveView creates a live, 360 degree view of order flow, executions, risk, transaction cost, risk, and operations. LiveView lets sales staff see what's happening before the data is lost in an IT black hole of log files, databases, and reports. 

Now, the moment the client calls, a salesman, risk manager, or trade execution specialist can answer if orders are being received correctly? Is the risk profile in line? Are the trades in line with the cost benchmarks? Are my 5 EMSs up and running? Are connections to the exchanges fast? Does the client want to adjust their strategies? 

The transformation is exciting: a typical reactive support function becomes proactive, opportunities lead to bigger opportunities, and problems become sales opportunities.

2. Real-Time Selling. In January, at a meeting at another bank, a sales trader asked: "Could I ask LiveView in the morning to alert me whenever a big order comes in from an important client using the new algorithm we released last week so I can bird-dog it?"

This was the perfect question, because it strikes at the heart of what LiveView does: it's like setting hundreds or thousands of business probes, and being alerted proactively when one is tripped. The probes can be turned on, turned off, and adjusted on the fly. Any system, internal or external, packaged app or custom built.

Firms that provide a highly automated service to clients need a real-time intelligence system for sales people. The ability to sell more depends on the ability to anticipate that a key client is headed for trouble, or when the time is right to encourage more sales.

And by building real-time intelligence into the system, everyone becomes a sales person, because the conditions that encourage more sales are right in their face.

3. Real-Time Fraud Detection. My last story took 3 months to unfold. Another LiveView early adopter wanted to implement a "fraud detection" analytic with LiveView. The firm needs to monitor machine-driven algorithms, and look for suspicious patterns. Sounds simple, right? Not so simple!

In this case, the patterns of interest required correlation of 5 streams of real-time data. Patterns happen within 15-30 second windows, during which thousands of dollars could be lost. Attacks come in bursts.

The data required to find these patterns was loaded into a data warehouse and reports were checked each day. Decisions to act were made every day.

LiveView now intercepts the data before it hit the warehouse by connecting LiveView to the source of data. It took 3 days to integrate these sources because it took that long to find someone who knew where 3 of the data streams came from!

StreamBase detects fraud patterns in milliseconds. But the really interesting part came next.

Once this firm could see patterns of fraud, they were faced with a new challenge: what to DO about it? How many times did the pattern need to be repeated until active surveillance is started?  Should the action be quarantined for a period, or halted immediately? All these questions were new, and the answers to them keeps changing.

The fact that the answers keep changing highlights the importance of ease of use. Analytics must be changed quicky and be made available to fraud experts - in some cases, in hours - as understanding deepens, and as the bad guys change their tactics.

Better, higher value-add customer service for highly automated industries. Knowledge workers who anticipate sales opportunities. Spotting fraud in high-speed transactions streams and taking action. These stories are the stories of the future for the real-time enterprise. These stories tell the story of why StreamBase LiveView is so special.

Introducing Live BI and StreamBase LiveView

For some time now, StreamBase has been working on a new product concept that's a little bit CEP, a little bit data warehouse, and a little bit business intelligence. Today we've unveiled it as StreamBase LiveView. Our customers have called it a "game changer," and a product that "fills a massive hole in the BI space." Hyperbole like that needs to be backed up, so here we go.

Complex event processing (introduction on Google+) was envisioned by its inventors as a "database for moving data" as firms increasingly deal with data in motion from mobile devices, social media and digitized data of all types, and in all industries. In its first decade of commercial use, CEP has come to power automated trading on Wall Street. A who's who list of firms have been public about their adoption of CEP, so let's examine the live BI problem in the context of Wall Street.

Industry analysts estimate that over 60% of U.S. equities trading is automated. The problem is, with so much automation, it's impossible to see and act on what's going on, because a typical trading environment can generate 10's of millions of events a day. Existing business intelligence and data warehouse tools aren't used in the front office of Wall Street, because they're designed to summarize what's already happened, not what's happening now. 

StreamBase LiveView brings BI and streaming data together, for the first time. Applications that need Live BI are everywhere on Wall Street - trading risk management, position management, P&L management, surveillance, compliance, and trading itself, as Waters Technology magazine describes in the article that StreamBase Seeks 'Radical Transparancy' in Business Intelligence.

And applications in other markets can be found in any environment that has big data in motion: telecom (location aware insight and live network operations), security (live fraud detection, live surveillance), and e-commerce (live social media analytics, ad placement, big streaming analytics for Hadoop and Flume). These apps can be deployed on premise or distributed in the cloud. 

StreamBase LiveView is built with StreamBase CEP, which provides connectivity to over 125 event-based sources of data and the extreme scalability required by live BI. The server-based architecture is designed to process hundreds of millions of events per node, absorbed from a network of event sources such as an enterprise message bus or scalable event absorption layer like Flume or IBM Streams. At the front end, users can slice and dice streams of data with ad-hoc query tools in the StreamBase LiveView Desktop (shown above). But here's the key: once users ask the questions they want answered, StreamBase LiveView keeps answering their questions - as conditions change in real-time; users see an always-live view of streaming data.

For the user, all this technology means just one thing: they see business intelligence that's constantly fresh and relevant to their business operations. It's not a view of 5 days ago, or even 5 minutes ago. The data is live, the insight is live, and live insight can lead to live action.

Facebook, Twitter, Google+ give us a live view of what's happening in our social networks; StreamBase LiveVew gives firms a live view of their enterprise data.

To learn more, watch the StreamBase LiveView presentation, follow new materials, articles, and resources on Google+ (+StreamBase and CEO +Mark Palmer), read industry analyst Curt Monash, DMBS2.0, or follow StreamBase Systems (@streambase), CEO Mark Palmer (@mrkwpalmer ), and CTO Richard Tibbetts (@tibbets) on Twitter.