(This article is also posted in the TABB Group's TABBForum, and generated a broad discussion in the comments there. To participate, visit the original post here.)
In his TabbFORUM article, Dave Tolladay eloquently argues that real-time surveillance is crucial in today's high-frequency trading landscape, and I agree. Ten years ago, real-time computing began to influence trading; today, HFT is pervasive and existing surveillance techniques, designed to run at the speed of click trading, should be considered to be legacy architectures in need of a radically new, modern approach.
But managing big data in real-time is a new problem of physics, and those physics come center stage when it comes to surveillance. A recent panel on modern surveillance systems, Miranda Mizen from TABB Group, Justin Amos from Redkite Financial Systems and Richard Tibbetts from StreamBase compared modern surveillance systems to “anti-virus software for trading.”
In the course of their conversation, Justin, Miranda and Richard touched on 11 key points about the “next generation” architectural requirements for surveillance, including:
1. Real-time analytics. Surveillance systems must analyze market data at the speed of automated trading. That means real-time computing and complex event processing (CEP) as one of the critical technology enablers of modern surveillance systems.
2. Historical “big data” analytics. Surveillance systems must put activity in context – suspicious activity can be an aberration or a pattern, based on historical actions. The big data movement has spurred a renaissance in the database industry with firms like Netezza (now IBM), Vertica (now HP), MongoDB (open source) as the flag bearers for the NoSQL (Not Only SQL) database revolution that aims to supplant legacy RDBMS systems to deal with the rising deluge of data described in the Economist piece. In surveillance, “big data analytics” allow on-line access to trade-by-trade history for investigation, trending, and discovery to be mashed up with real-time data to provide a real-time and historical context to behavior.
3. Inclusion of new types of data sources. Justin suggested that the inclusion of real-time news – and even social media data sources – can be a benefit in correlating, for example, suspicious trading on news.
4. Data fusion. Real-time, historical and new data sources must be fused for a complete picture of intra-day trading behavior, and a 360 degree view of market data, activity and sentiment.
5. Multi-asset class support. Since HFT is becoming increasingly cross-asset class, so must modern surveillance systems.
6. Pre-built algorithms. Todayʼs surveillance systems must support standard algorithms like front running, spiking, tailgating, and ramping that can mistakenly or purposefully occur in computer-based HFT systems.
7. Adaptive algorithms. The ability to adapt algorithms on the fly to reduce false positives is important, due to the increasingly dynamic capability of modern HFT systems to adapt their behavior.
8. Algorithm extensibility. The ability to quickly add new algorithms to the system as market dynamics evolve is also a key modern requirement for systems – they must keep pace with the rate of change of modern trading algorithms.
9. Flexible workflow management and case management. Surveillance systems generally donʼt make automated decisions to change trading behavior or stop trading on their own; instead, they tend to guide a series of workflow steps to alert compliance officers, portfolio managers or traders and manage the process of alert analysis, resolution and tracking.
10. Accessibility for the business analyst. With regulation changing so quickly, more firms want their own control over surveillance behavior, so some provide tools that allow compliance personnel to discover and implement surveillance algorithms on their own.
11. Cloud-based or on-premise deployment. Surveillance is not typically the core expertise of trading groups, so the ability to deploy systems either in the cloud or on premise allows for more rapid and global rollout of surveillance tools.
Real-time big data is a change in computing physics on Wall Street. New requirements motivate new solutions, and the new solutions and innovation coming from the surveillance community continues to prove that innovation never sleeps on Wall Street.