The StreamBase Event Processing Blog

The Best Practices of Real-Time Innovation

Event-driven systems are new to many firms. For many, automating existing manual processing, using event-driven techniques, properly simulating real-time analytics systems is a new experience. 

Fortunately, some industries have been on the forefront of designing for real-time. Namely, banks in the capital markets trading arena have been automating and building event-based applciations for years. 

So on a recent customer tour with our customers, we've been asking them for their best lessons learned and the business impact the systems they've built have made. 

Best Practice #1: Build Apps So Quickly You Won't Mind Throwing Some Away

One IT executive told me "CEP allows me to build event-driven apps in 2-4 weeks, rather than 6-9 months with an app server and database - so I can say "yes" to more requests from their business clients, which helps us innovate more quickly. We keep the great ideas, and discard the ones that don't work."

Best Practice #2: Start Small, Succeed, then Scale

The most consistently successful firms that moved to real-time on Wall Street started with a small, critical application, learned lessons from its development, then applied those lessons learned to other projects and groups. Over time, they developed a center of excellence with firm-wide sucess in real-time application development.

 

Best Practice #3: Plan for Spikes, Not Aggregate Volumes

Systems that use primarily historical have trained many technologists to capacity plan their systems for the aggreate amount of data they manage. Real-time systems are different, and should be planned for spikes in volume, rather than only aggreate volumes. This best practice explores three approaches: burst behavior, the use of conflation, and computational reduction, that helps applications gracefully manage spikes and deliver correct and timely response to users and automated action. 

Keep coming back throughout 2013 as we continue to add to our library of real-time system best practices!

Best Practice #4: Incorporate Statistical Computing with Event Driven Systems

Numerical computing environments are often used for algorithm development, data analysis, visualization, matrix manipulations, and statistical analysis. In finance, quants use it to do time-series analysis, portfolio optimization, asset liability modeling, quantitative risk modeling, and derivative pricing. This best practice describes how to does this computing in real-time, using MATLAB with StreamBase CEP together.

Best Practice #5: Balance Man And Machine

The prospect of automating a business often sparks a debate about what is more effective, man or the machine. It turns out that the debate presents a false choice, and effective automation lies in the proper balance between the two.

This article explores 5 steps to balance man and machine, including the infusion of more data scientists in executive positions, the application of real-time risk management and surveillance, the infusion of historical and real-time analytics in one place, and more.

 

Best Practice #2: Start Small, Succeed, then Scale

This post is the second in a series of best practices gleaned from leading StreamBase customers who have effectively deployed "real-time" business systems.

Even though CEP and stream processing is being popularized in marketing campaigns such as IBM's "Smarter Planet" and TIBCO CEO Vivek Ranidive's book "The Two Second Advantage," the reality is that, for many organizations, a real-time, sense-and-respond business model is still elusive. 

A consistent best practice employed by our customer base is to "Start Small, Succeed, then Scale."

THE SITUATION: REAL-TIME BECOMES A "MUST DO" AGENDA

On August 1st, 2012, Knight Capital lost $440 million in 45 minutes when their real-time algorithms went out of control. That kind of event caught the interest of most CEOs in the capital markets who all of a sudden put real-time controls, risk management and reporting high on their list of "must do" IT projects. 

That said, many would be surprised to know how little of Wall Street is actually automated with respect to the management function of high-speed trading. Many of the largest firms process 100's of millions of trading events a day, yet still manage their business by looking at historical information.

But moving to real-time isn't just a matter of installing new, faster hardware - it's a shift in thinking, a shift in culture. That can mean implementing new IT infrastructure from scratch, or even hiring new teams to manage old business processes that used to be more manual. 

THE PROBLEM: LEGACY SYSTEMS AND PEOPLE NEED AN OVERHAUL

When it comes to trading, most banks have great real-time technology in place. But when it comes to managing that trading activity, they're still often using rear-view-mirror reporting technology like traditional business intelligence tools. You can see this in software vendor market size by comparing the business intelligence market (around $10B) to CEP and the new "Live Business Intelligence" segment (around $350M, and growing). 

But the problem isn't just about technology - the staff required to manage real-time business processes are more quantitative in mind-set. For example, heads of risk management usually understand mathematics, and the models used to trade, because the algorithms themselves introduce operational and cost risk. 

So, given that people, processes, and technology typically have to be transformed, the most successful clients in our customer base have taken an incremental approach, by deliberately transforming the most critical elements to start with. 

But how?

THE SOLUTION: CHOOSE A CRITICAL "FIRST DOMINO" TO PUSH OVER

The transformation in the capital markets provides a good template on how to transform from rear-view mirror to real-time.

12 years ago, less than 5% of trading was driven by computer algorithms; now, around 60-70% of trading volume in the U.S. market is guided by algorithms. Banks chose their most critical operation - trading - to focus on first. The most mathematically oriented traders survived; pure "relationship" oriented traders have waned. Now, a dozen years later, back-office, clearing, settlement, and risk management are all moving to real-time.

Trading was the first domino to be pushed over. 

How did they do it?  The pattern went something like this:

1. Choose an important function (e.g., trading)

2. Put your best people on it (choose the new model of people required for mathematics-based operations)

3. Build the new system

4. Fail

5. Return to step 3, as necessary until you get it right

6. Capture learning, choose new leadership that "gets" the new model

7. Repeat with adjacent business functions

8. Build a center of excellence

9. Scale (pick the next system to transform based on the lessons learned).

THE IMPLICATION: REDUCED RISK; LONG TERM EFFECTS

The implication of starting small and planning to fail with an important domino is, ironically enough, reduced risk. Risk is reduced because by choosing an important, small function, you can fail faster. You can choose to put your top talent on the project. You can adjust more quickly. And, for long-term sustainability, you create a core center of excellence that the rest of the organization can leverage as you roll systems out over time. 

THE LESSON: ACHIEVE LASTING TRANSFORMATION TO REAL-TIME BY STARTING SMALL

So by starting small, succeeding with that small project, then scaling out the rest of the enterprise, the organization lowers the risk of early failure and starts on the long journey of employing a disruptive shift in thinking that can last decades. 

The 3 Principles of Analyzing Data in Motion



 

When I first saw an event processing language 10 years ago, as a C++ developer, it opened my eyes to a new development "physics" that made dealing with information in motion simple. Today, it seems like all modern data is in motion: streaming social media data, market data in financial services, and the increasing adoption of the internet of things and sensor-driven systems. Clearly, the technology needed to manage data in motion is different than that needed to manage data at rest; and that's where StreamBase complex event processing, or CEP comes in - its the ultimate platform to monitor, analyze, and act on data that's in motion.

Complex event processing doesn't mean the technology is complicated to use; it's quite the opposite.... the term refers to the ability to identify complex sequences of conditions - as the data moves through an enterprise.

Three CEP capabilities, aggregation, correlation, and temporal analytics, provide the foundation for dealing with data in motion.

For example, in the gift card market, let's say you wanted to detect a common fraud condition, such as: "Tell me when the number of gift card redemptions at any POS machine is above $2,000 in any 1-hour period." - this situation can be easily detected by the CEP's aggregation facility, which continuously calculates metrics across sliding time-bound windows of moving data which is used to understand real-time trends. Continuous aggregation is hard to do with traditional tools; but with CEP, it's built in - simply drag the "aggregation" operator onto your application template, and you're off.

The second key new principle of CEP is Correlation. Correlation means connecting to multiple streams of data in motion and, over a period of time that may be seconds or days, identify that condition A was followed by B, then C. For example, if we connect to streams of gift card redemptions from 1,000 point of sale terminals, CEP would allow me to continuously identify conditions that compare the POS terminals to each other like: "Alert me if the gift card redemptions in one store are above 150% the average of the other stores."

Finally, CEP is designed for temporal analytics, which is the idea of having time as a first-class computing element. Temporal analytics are critical for processing data where the rate and momentum of change matters. For example, in fraud, sudden surges of activity are common indications of fraudulent activity - so detecting "If the number of gift card sale and card activations within 4 hours is greater than the average number of daily activations of that store in the previous week, stop approving activations." Unlike computing models designed to summarize and roll up historical data, CEP is designed to ask and answer these questions on data as it changes.

These three elements: aggregation, correlation, and temporal analytics, set CEP apart from all other computing platforms by helping technologists look at what's happening now, not just what happened in the past more quickly than ever imagined before. Data in motion, and the use of CEP to analyze it, is the future of computing.

Betfair Adopts StreamBase for Real-Time Analytics

(By Richard Tibbetts, StreamBase CTO)

Today we announced that Betfair has selected StreamBase for Real Time Analytics. I’m excited about this news as it demonstrates the importance of real-time data and analytics for driving the best possible consumer user experience for streaming, online services, and the role of StreamBase as the ideal platform for event-driven environments like Betfair.

American readers of this blog may be unfamiliar with Betfair.  Betfair is the world’s largest betting community and one of the world’s leading online betting and gaming operators. At the heart of Betfair is its pioneering Betting Exchange, where customers come together to bet on odds set by Betfair or offered by other customers, instead of with a traditional bookmaker. The Betting Exchange provides better pricing, flexibility, and a level playing field in a massive market. In April of this year, Betfair had over 4 million registered customers worldwide and processed, on average, more than 7 million transactions per day on the Betting Exchange. 

Betfair provides this service via Internet, mobile, and tablet devices that maximize the playing experience, and the Betfair infrastructure implements a service that implements its sophisticated betting market, real-time data processing, and real time clearing of bets.

Betfair is well known for its technically ambitious support for betting during a match. Players can bet while watching a live match, at the last possible moment. When I first met Alex Kozlenkov (Head of Research at Betfair) he described his challenge of executing 100,000 transactions per second against a single product, each transaction (or event) potentially impacts the price of that product, and must be processed in order.

And it’s not just the transactions that are streaming in the Betfair environment, they are also a leader in using real-time data to make the customer experience more engaging, fun, and unique. We’re in the midst of a renaissance of innovation in data management due to “Big Data” – innovative tools and infrastructure designed to manage increasing volume, velocity, and variety of information.  The betting world is no different - new real-time data feeds provide streaming play-by-play data from cricket matches, and real-time social media data brings an opportunity to crowd-source context about the events from fans watching live (in related news, StreamBase recently announced integration with GNIP and connectivity to Hadoop via Flume that enables connectivity to Big Data)

By investing in real-time analytics, Betfair can tap into streaming information and from the betting ecosystem and leverage it to provide a better experience for users live on the site, on mobile devices, and on tablets, providing innovative context and experience to the Betfair experience.

Consumers are now conditioned by social media to expect information and data to be real–time, whether it be pictures of a night out, stock twits, or a location update. Betfair represents a step in the march of all businesses to become more social media aware, real-time, and web-scale. Customers expect more than just a scrolling Twitter feed in the sidebar – they expect real-time context and relevance. The winning firms will integrate a variety of data sources - structured and unstructured, historical and real-time - and distill them down to insight that matters, when it matters. The real-time business of the future requires all of this, and requires real-time analytics and event processing as the new IT computing fabric.

StreamBase LiveView Demonstration and Animated Short

In the first week since the "worldwide premier" of our two StreamBase LiveView videos at the Securities Industry and Finanical Markets Association shown in NYC, we've had almost 1,000 views on Vimeo and YouTube. This product is bringing the power of CEP to entirely new markets and users, and it's very exciting to see interest in it grow so rapidly.

Here's the StreamBase LiveView demonstration video:

 

And our animated short, produced by Grumo Media (thanks, Pedram!)

 

Introducing Live BI and StreamBase LiveView

For some time now, StreamBase has been working on a new product concept that's a little bit CEP, a little bit data warehouse, and a little bit business intelligence. Today we've unveiled it as StreamBase LiveView. Our customers have called it a "game changer," and a product that "fills a massive hole in the BI space." Hyperbole like that needs to be backed up, so here we go.

Complex event processing (introduction on Google+) was envisioned by its inventors as a "database for moving data" as firms increasingly deal with data in motion from mobile devices, social media and digitized data of all types, and in all industries. In its first decade of commercial use, CEP has come to power automated trading on Wall Street. A who's who list of firms have been public about their adoption of CEP, so let's examine the live BI problem in the context of Wall Street.

Industry analysts estimate that over 60% of U.S. equities trading is automated. The problem is, with so much automation, it's impossible to see and act on what's going on, because a typical trading environment can generate 10's of millions of events a day. Existing business intelligence and data warehouse tools aren't used in the front office of Wall Street, because they're designed to summarize what's already happened, not what's happening now. 

StreamBase LiveView brings BI and streaming data together, for the first time. Applications that need Live BI are everywhere on Wall Street - trading risk management, position management, P&L management, surveillance, compliance, and trading itself, as Waters Technology magazine describes in the article that StreamBase Seeks 'Radical Transparancy' in Business Intelligence.

And applications in other markets can be found in any environment that has big data in motion: telecom (location aware insight and live network operations), security (live fraud detection, live surveillance), and e-commerce (live social media analytics, ad placement, big streaming analytics for Hadoop and Flume). These apps can be deployed on premise or distributed in the cloud. 

StreamBase LiveView is built with StreamBase CEP, which provides connectivity to over 125 event-based sources of data and the extreme scalability required by live BI. The server-based architecture is designed to process hundreds of millions of events per node, absorbed from a network of event sources such as an enterprise message bus or scalable event absorption layer like Flume or IBM Streams. At the front end, users can slice and dice streams of data with ad-hoc query tools in the StreamBase LiveView Desktop (shown above). But here's the key: once users ask the questions they want answered, StreamBase LiveView keeps answering their questions - as conditions change in real-time; users see an always-live view of streaming data.

For the user, all this technology means just one thing: they see business intelligence that's constantly fresh and relevant to their business operations. It's not a view of 5 days ago, or even 5 minutes ago. The data is live, the insight is live, and live insight can lead to live action.

Facebook, Twitter, Google+ give us a live view of what's happening in our social networks; StreamBase LiveVew gives firms a live view of their enterprise data.

To learn more, watch the StreamBase LiveView presentation, follow new materials, articles, and resources on Google+ (+StreamBase and CEO +Mark Palmer), read industry analyst Curt Monash, DMBS2.0, or follow StreamBase Systems (@streambase), CEO Mark Palmer (@mrkwpalmer ), and CTO Richard Tibbetts (@tibbets) on Twitter.

 

Real-Time Big Data and the 11 Principles of Modern Surveillance Systems

(This article is also posted in the TABB Group's TABBForum, and generated a broad discussion in the comments there. To participate, visit the original post here.)

In his TabbFORUM article, Dave Tolladay eloquently argues that real-time surveillance is crucial in today's high-frequency trading landscape, and I agree. Ten years ago, real-time computing began to influence trading; today, HFT is pervasive and existing surveillance techniques, designed to run at the speed of click trading, should be considered to be legacy architectures in need of a radically new, modern approach. 

But managing big data in real-time is a new problem of physics, and those physics come center stage when it comes to surveillance. A recent panel on modern surveillance systems, Miranda Mizen from TABB Group, Justin Amos from Redkite Financial Systems and Richard Tibbetts from StreamBase compared modern surveillance systems to “anti-virus software for trading.” 

In the course of their conversation, Justin, Miranda and Richard touched on 11 key points about the “next generation” architectural requirements for surveillance, including: 

1. Real-time analytics. Surveillance systems must analyze market data at the speed of automated trading. That means real-time computing and complex event processing (CEP) as one of the critical technology enablers of modern surveillance systems. 

2. Historical “big data” analytics. Surveillance systems must put activity in context – suspicious activity can be an aberration or a pattern, based on historical actions. The big data movement has spurred a renaissance in the database industry with firms like Netezza (now IBM), Vertica (now HP), MongoDB (open source) as the flag bearers for the NoSQL (Not Only SQL) database revolution that aims to supplant legacy RDBMS systems to deal with the rising deluge of data described in the Economist piece. In surveillance, “big data analytics” allow on-line access to trade-by-trade history for investigation, trending, and discovery to be mashed up with real-time data to provide a real-time and historical context to behavior. 

3. Inclusion of new types of data sources. Justin suggested that the inclusion of real-time news – and even social media data sources – can be a benefit in correlating, for example, suspicious trading on news. 

4. Data fusion. Real-time, historical and new data sources must be fused for a complete picture of intra-day trading behavior, and a 360 degree view of market data, activity and sentiment. 

5. Multi-asset class support. Since HFT is becoming increasingly cross-asset class, so must modern surveillance systems. 

6. Pre-built algorithms. Todayʼs surveillance systems must support standard algorithms like front running, spiking, tailgating, and ramping that can mistakenly or purposefully occur in computer-based HFT systems. 

7. Adaptive algorithms. The ability to adapt algorithms on the fly to reduce false positives is important, due to the increasingly dynamic capability of modern HFT systems to adapt their behavior. 

8. Algorithm extensibility. The ability to quickly add new algorithms to the system as market dynamics evolve is also a key modern requirement for systems – they must keep pace with the rate of change of modern trading algorithms. 

9. Flexible workflow management and case management. Surveillance systems generally donʼt make automated decisions to change trading behavior or stop trading on their own; instead, they tend to guide a series of workflow steps to alert compliance officers, portfolio managers or traders and manage the process of alert analysis, resolution and tracking. 

10. Accessibility for the business analyst. With regulation changing so quickly, more firms want their own control over surveillance behavior, so some provide tools that allow compliance personnel to discover and implement surveillance algorithms on their own. 

11. Cloud-based or on-premise deployment. Surveillance is not typically the core expertise of trading groups, so the ability to deploy systems either in the cloud or on premise allows for more rapid and global rollout of surveillance tools. 

Real-time big data is a change in computing physics on Wall Street. New requirements motivate new solutions, and the new solutions and innovation coming from the surveillance community continues to prove that innovation never sleeps on Wall Street.

 

Fast is Good, But Fast and Dumb is Dangerous

Last week I was on a panel at the Bloomberg Enterprise Technology Summit (@BBGlink) with Steve Goldman from CME Group (@CMEGroup), the CTO of NYSE Technologies, Bloomberg TV Markets Reporter Dominic Chu (@The_Domino) and we talked about how speed is important, but that the real innovation going on today is about making applications smarter, not just faster. The conversation was constructive and productive and was re-tweeted all over the internet. 

Then, this weekend, Barron's article "Next Danger: 'Splash Crash'" featured an interview with Progress Software's CTO Dr. John Bates. Barron's cites his view that high frequency trading poses an "extreme risk" of creating a "splash crash," his own term, which he describes as a worse than the flash crash: "trading systems could clog up, limited bandwidth could choke orders, exchanges could freeze up -- splashing across all of the affected asset classes. Pandemonium."

Bates "frets that, absent more oversight, terrorists wielding smart machines could attack the markets in an attempt to cripple our economy,"  and has indeed been fretting publicly for a while now in interviews blog posts, such as:

• "The trouble with algorithms: wild children or reckless parents?"
• "Beware the splash crash"
• "Red flags in morning, firms take warning"
• "When does a rogue trader become a scoundrel?"

It's unhelpful to focus on the fear; we should focus on solutions. Every automated application - from kiosk-based airport check-in to security in a retail store - needs surveillance. Trading is no different. This is not new. What's new is that trading occurs at speeds never before seen, and when things go wrong, it's easy to blame the bankers of nefarious activity.

So today's article in MIT Technology Review, "Faster than a Flash," was a refreshing break from provocative predictions of doom and gloom. MIT features Boston-based proprietary trading firm PhaseCapital (disclosure: PhaseCapital is a StreamBase customer). CEO Eric Pritchett described to MIT reporter Erica Naone that his algorithmically-driven trading firm made NO erroneous trades on the day of the flash crash. 

How did Phase Capital use algorithms and provide a safe trading harbor to their investors on the day of the flash crash? They use the same technology to analyze their market data in real-time that they use to trade in real-time: complex event processing, or CEP (intro here).

Corwin Yu, director of electronic trading for PhaseCapital, explained their innovation this way:

Typically, PhaseCapital processes 30,000 to 40,000 'ticks,' or pieces of market data, per second. During the flash crash, that number jumped to more than 289,000 ticks per second— much of it representing stocks swinging wildly. "A lot of scrubs kicked in and realized that this data didn't make sense," Yu says. For example, some large publicly held companies, such as Accenture, traded for less than $1 during the flash crash. PhaseCapital continued trading, but filtered data that appeared suspicious. Because it didn't act on that data, it was spared erroneous trades.

Yu concluded that speed is essential in the world of trading today, but that the trick is to be "fast and smart." Read about PhaseCapital's innovation in MIT Technology Review.

The solution for a safer market isn't heavy-handed regulation of "wild, reckless algorithms" in order to catch the "scoundrels" of Wall Street. Fast does good on Wall Street; but fast and dumb is dangerous. So we should tune out the fear mongers and tune in to the innovators that can help make Wall Street be fast - and smart.

Thomson Reuters Selects StreamBase CEP for Elektron High-Speed Cloud, Quant Platform, Apps, and FX

Today we're excited to report that Thomson Reuters has selected StreamBase as its complex event processing (CEP) platform for a broad set of initiatives. CEP "flips the database on its head" by providing the ability to monitor, analyze, and act on events happening in real-time, rather than looking in the rear-view-mirror of what happened yesterday, last week, or last month. CEP powers real-time, big data applications on Wall Street including high frequency trading, pre-trade risk management, surveillance, market data management, smart order routing, pricing, settlement, real-time P&L, monitoring - applications from front to back office on Wall Street.

Why did Thomson Reuters partner with StreamBase for CEP? Initially, they announced four reasons: 

StreamBase in Elektron Brings CEP to Thomson Reuters' High-Speed Cloud

The StreamBase platform lies at the heart of a new generation of real-time, automated applications on Wall Street. As the use of CEP continues to grow, our customers increasingly ask to deploy some of their applications in the cloud. They want their automated processing to happen as close to the exchange as possible for the lowest possible latency, and they want secure and reliable access to their market data.

Thomson Reuters announced that they will provide StreamBase CEP in Elektron, their open, global, ultra-high speed and resilient network for the financial industry. By adding StreamBase CEP to Elektron, Thomson Reuters is creating the industry’s only platform that includes comprehensive real-time and historical data, high capacity network connectivity, managed hosting services, and CEP for rapid application development and strategy deployment.

The First End-to-End Quant Platform for Back Testing, Algorithmic Development and Deployment

Although StreamBase is all about acting on what's happening now, the knowledge of what happened yesterday is essential for any automated system. History is essential to discover trends and patterns in event streams. History is essential to back test automated strategies. History is essential to gain perspective on real-time actions at execution time. Over 90% of applications that use StreamBase also use some form of historical event data management.

Thomson Reuters will integrate StreamBase with the Enterprise Platform for Velocity Analytics, which integrates all Thomson Reuters content, including tick history, reference data, corporate actions, condition code processing and cancellation and correction messages. It also brings together many disparate third party applications including data management solutions, EMS, OMS, price discovery, back testing, trade surveillance and low latency execution capabilities across the quantitative research and trading workflow.

By integrating Velocity Analytics with StreamBase CEP, Thomson Reuters has created the first end-to-end quant platform for back testing, algorithmic development and deployment. 

Thomson Reuters Applications "Powered By" StreamBase

Thomson Reuters will deliver a suite of applications that are “powered by” StreamBase CEP to address low-latency workflow. CEP inside will provide Thomson Reuters clients the benefit of time to market usually associated with turnkey solutions, while at the same time allowing full customizability of the underlying application. One application Thomson Reuters already provides is the real-time surveillance application from Redkite financial which is based on the StreamBase CEP platform. 

StreamBase and Thomson Reuters FX Solutions

The FX markets have rapidly emerged as an important market for the capabilities of CEP. Thomson Reuters announced that StreamBase CEP will complement the existing Thomson Reuters Dealing Aggregator FX trading solution, available exclusively to Thomson Reuters Dealing customers, by providing CEP capability in tandem with the low latency algo trading platform. 

Why did Thomson Reuters choose StreamBase? Mike Powell, Global Head of Enterprise Real-Time Solutions for Thomson Reuters, said it this way: “The financial market landscape continues to evolve rapidly, meaning that nimbleness and time to market have become essential for market participants. Elektron empowers our customers to innovate and achieve high performance through a seamless infrastructure that brings together global market participants, delivers world-class content and creates a suite of new added-value services. The combination of StreamBase’s market leading CEP engine together with Elektron will provide a platform for us to deliver flexible and sophisticated solutions to our customers faster and more efficiently.”

For StreamBase, we’re excited about the vast resources and capabilities of Thomson Reuters can provide our customers. We’re honored to be selected as Thomson Reuters’ CEP platform of choice - their selection validates our focused approach to CEP, the ever increasing importance of the cloud and real-time big data, the disruption of simplicity, and the continued need for low latency and scalability on Wall Street. 

 

Can Innovative Surveillance Be a New Source of Competitive Edge?

In sports, great defense can be a catalyst for great offense. Can the same idea hold true for surveillance - the defense - in the capital markets? This was a question raised by today's panel of surveillance experts at today's StreamBase webinar (watch the replay). 

First, TABB Group's Miranda Mizan assessed the current surveillance market. She described how surveillance is morphing from a defensive obligation to an offensive differentiator, because the markets are changing so dramatically. Mizan argues that the quantity of data, speed of trading, and complexity of instruments have made traditional surveillance techniques insufficient. And since regulation (e.g., MIFID II) is changing so quickly, she argues that firms must re-think their surveillance strategies and technology choices.

Justin Amos joined Miranda. Justin is a former global markets sales and trading executive at JPMorgan, and the MD of Redkite Financial Systems, a UK-based firm whose system called Redeye has been quickly becoming the standard for modern, real-time surveillance systems. Justin described the requirements of today's surveillance systems by describing the motivation for Redeye, which he likened to "anti-virus software for trading."

Together, Justin and Miranda touched on 12 "next generation" surveillance requirements, including:

1. Real-time analytics. Surveillance systems must analyze market data at the speed of automated trading.
2. Historical "big data" analytics. On-line access to trade-by-trade history for investigation, trending, and discovery.
3. Data fusion. Mash-ups of increasingly complex data, including unstructured data.
4. Inclusion of new data sources. The ability to integrate live data, static data, and social media data.
5. Multi-asset class support. Modern surveillance must work across asset classes.
6. Pre-built surveillance algorithms. Standard algorithms like front running, spiking, tailgating, and ramping.
7. Adaptive algorithm modification. The ability to adapt algorithms on the fly to reduce false positives.
8. Alert algorithm extensibility. The ability to quickly add new algorithms to the system as market dynamics evolve
9. Customizable surveillance algorithms. The ability for a firm to tweak an algorithm for their own models.
10. Flexible workflow management and case management. Control over the process of alert analysis, resolution, and tracking.
11. Accessibility for business analyst. With regulation changing so quickly, more firms want their own control over surveillance behavior.
12. Cloud-based or on-premise deployment. The cloud allows more rapid rollout, and may ease global scalability challenges

The panel argued that existing surveillance systems can't handle these requirements. Taken together, the challenge makes what was once traditional "defense," or a compliance function, very new and challenging - a source of differentiation.

Miranda went deeper, with examples of how brokers and the buy side are using modernized surveillance systems: agency brokers are beginning to use surveillance tools to monitor their trading algorithms and optimize their client service and order routing for best execution; the buy side are using surveillance to gain greater control to know that their trading is safe, its orders get to the right places, and that dark pools are working correctly in relation to the rest of the market; sell side institutions are using surveillance tools to institute process and control across the organization, globally, and integrated with automated systems.

Richard Tibbetts, CTO of StreamBase, gave a brief introduction of StreamBase CEP, and why complex event processing is increasingly being used to power surveillance applications; indeed, CEP provides the engine to address many of the 12 modern surveillance requirements set out by the panel.

Simon Townsend, CIO of Redkite, gave a "whistle stop tour" around Redkites' application. He hit on most of the 12 requirements above, and even showed didn't just claim that their algorithms are extensible and dynamic (requirements 6, 8 and 9) - he showed it in StreamBase Studio, and showed how business analysts can extend the Redeye surveillance algorithms.

One of the most exciting times is when a market is transformed by new ideas. In the capital markets, perhaps that innovation is upon us in surveillance, where tools that meet modern trading requirements can turn a defensive obligation into a tool for innovation.