The StreamBase Event Processing Blog

Algo Trading Saves Mark Cuban Millions During "Twitter Twitch"

Geeks on Wall Street make the market safer, turning a potential "Hack Crash" into a "Twitter Twitch," saving Cuban millons

Mark Palmer is the CEO of StreamBase Systems, a technology provider of real-time analytics technology.

Mark Cuban has provocative views on everything from the NBA draft to the Iraq war, so his latest shot at the financial trading industry comes as no surprise; the problem is, he lashed out at the Wall Street mavericks whose high tech trading systems worked perfectly, and probably saved him a lot of money.

In case you missed it, the Twitter account for the Associated Press was hacked, a hoax tweet was sent out that Barack Obama was injured in a bomb explosion. The Dow Jones fell 145 points in about 120 seconds and then recovered 134 points in the ensuing 3 minutes. So, in just 300 seconds, the DJIA traveled 279 points. 

Cuban told CNBC that "all of this algorithmically driven trading has created a ton of downstream problems," and proposed some "solutions" to the "problems" that include taxing trades that are held for less than one hour.

But here's where Cuban has the whole thing backwards: the technology he wants to shackle saved him, and the man on the street, millions.

On the day of the Hack, I was speaking about high frequency trading at a Bloomberg conference with 200 people in the audience. I asked the audience to raise their hands if they personally lost any money that day. Not one hand went up.

This wasn't a Hack Crash; it was a "Twitter Twitch." In fact, high frequency trading was exactly the antidote that saved all "regular" investors (even Mr. Cuban) money. 

It's true that Wall Street uses computers to make millions of trading decisions a second, and it's probably true that computers that detected the hacked tweet twitched in seconds. 

But those same algorithms that watch millions of data points a second could tell, in real-time, that something was amiss. The Tweet was algorithmically refuted - Obama was fine and healthy - the descent stopped immediately, and the markets returned to being healthy in just a few seconds.

Without real-time technology, the market would not have corrected so quickly; the hack created a twitch, not a crash, and the only ones in the market who lost money were probably the dumbest of algorithms that chased the lie. Most algorithmic trading insiders know that in the most volatile times, it's the human trader who tends to win; algorithms tend to get whip-lashed chasing purely statistical trends that they don't understand. 

Nobody, even a billionaire, can wish away the fact that the world is electronically fueled, and that social interaction, hoaxes, and trading are all done in real-time. To limit the use of real-time technologies on Wall Street would deny the markets the very tools it needs to make the market safer for everyone, and will continue to turn the next potential crash into a twitch.

Best Practice #5: Balance Man and Machine

This post is #5 in a series of lessons learned from leading StreamBase customers who have effectively deployed real-time business systems. 

The prospect of automating a business often sparks a debate about what is more effective, man or the machine. It turns out that the debate presents a false choice, and effective automation lies in the proper balance between the two.

Best practice #5, "Balance Man and Machine," is about how successful firms strike a balance between machine-driven and human decision- making.

THE SITUATION: FIRMS MUST AUTOMATE TO SURVIVE

In the capital markets, open outcry exchanges have been closing down for years. Last October, the IntercontinentalExchange closed down open outcry trading of soft commondities in New York, a practice that goes back at least 142 years. When it closed, the 1,000 traders left working on the floor "mostly monitored their market using handheld pricing screens even when they're sitting in the pits." That is, ironically, the manual brokers were literally using the automated tools that were making their jobs obsolete.

And the impact went beyond the survival of some jobs to the survival of some firms. For example, from 2008 to 2012, the once dominant NYSE and NASDAQ exchanges saw their combined market share slip from 67% to 45%, in large part due to upstart rivals BATS and Direct Edge known for their technology-based innovation. 

Obviously, automation has changed Wall Street forever, and it's about to change other industries.

THE PROBLEM: AUTOMATION THREATENS EXISTING CULTURE AND CAN ADD RISK

In the capital markets, the debate has raged on for years about what's more effective - human traders or algorithms.  A quick glance at a time-restricted Google search in 2007 for "Man Versus Machine" reminds us about how massive the debate was! 

Underneath this media coverage was a real issue: culture.

"Manual" traders, and the hierarchy of power over trading systems that was built over years by those who made their mark as manual traders, resisted the ideas that computers could do better than humans. Technologists, eager to make their mark on the P&L of the bank, fought to prove that computers could do a better job. 

At the same time, automation can introduce risk. Events like today's Twitter Hoax that President Obama had been injured at 1:07PM caused the markets to plunge 200 points in about 2 minutes. At 1:10PM, the first denials came out on Twitter, and by 1:13, the Dow had recoverd most of the ground it had lost.  So risk can no longer be done with traditional tools, which evaluate positions at the end of the day. By the end of the delay, it can be too late.

This polarized view of the battle between man and machine was wrong. 10 years on, automated trading is here to stay, and so are the people who guide its use. The truth is that neither man nor machine alone is better at trading; the truth is, a balanced approach is what was created. 

THE SOLUTION: FIVE STEPS TO BALANCE MAN & MACHINE

Automating businesses must change. Here are four observations from the transformation in the capital markets, and the best practices other firms in other industries can learn to facilitate their own change.

1) Business-geek hybrids are common

Traders used to have mostly degrees in business; today, they either additionally have math and science degrees, or have learned more about the implications of algorithms on trading. 

Best Practice: Proactively hire data scientists and put them into leadership positions so that they can lead a different, quantitative way of thinking about business transformation.

2) Humans are more comfortable taking cues from real-time analytics

Yes, lots of trading is automated. But automation also signals humans to make the big decisions. For example, Geneva Trading, one of the largest oil futures traders in the world, uses automation to monitor "millions of trading opportunities" simultaneously. Some opportunities are no-brainers, and the computer acts automatically. But sometimes the computer has no idea how to act, and punts the decision to a human by sending a text message, email, or alerting them on their desktop. 

Best Practice: Couple data scientists with operational staff that are similarly willing to embrace a technologically-driven customer resource management approach. These are customer service personelle who are used to analyzing information supplied by automated systems, and working with clients to solve high-value-add problems. The recent trend in capital markets toward execution consulting services is a good example of this trend. 

3) Real-time risk and surveillance becomes more important

One great myth about algorithmic trading is that it's done without oversight. In fact, all the clients I meet with use real-time analytics to temper, or stop, trading decisions when the data, statistically, does not make sense. One of our customers, PhaseCapital, was the inspiration for the blog post "Fast is Good, But Fast and Dumb is Dangerous" that describes the firms' real-time analytics that helped them trade more safely on the day of the May, 2010 Flash Crash. 

Best Practice: Employ real-time risk, surveilence, and compliance systems. One of the most prevalent applications of real-time platforms in the last 2 years has been real-time risk, compliance, and surveillence applications. 

4) Low-touch and high-touch functions must be integrated

Automation is useful to handle well known, low risk decisions, the ones that require little over site other than a big red panic button to make systems stop. Bigger, high touch decisions are ones handled by humans. By using computers to automate the low touch decisions, humans are empowered to make the tough, big decisions. 

Best Practice: Start with a balanced sense of technical requirements, rather than a world separated by automated and manual workflow. 

5) Infuse events with historical context

In the world of automation, the critical technology stack is event driven; the technology stack for manual processes is historical data driven. These two systems must coexist. Event-driven messaging, event-oriented programming languages (e.g., CEP), event-driven user interfaces, and event-oriented databases (e.g., tick stores) and streaming business intelligence must all sit along side of their historical counterparts. 

Best Practice: Integrate real-time and historical data to give humans oversight and insight into what automated algorithms are doing.  

THE IMPLICATION: AUTOMATION EMPOWERS HUMANS

Balanced well, automation can be a huge competitive weapon for a business. By applying lessons from one of the most deeply automated industries in the business world, firms embarking on automation initiatives  can increase their chance of success and reduce their risk of failure. 

The Best Practices of Real-Time Innovation

Event-driven systems are new to many firms. For many, automating existing manual processing, using event-driven techniques, properly simulating real-time analytics systems is a new experience. 

Fortunately, some industries have been on the forefront of designing for real-time. Namely, banks in the capital markets trading arena have been automating and building event-based applciations for years. 

So on a recent customer tour with our customers, we've been asking them for their best lessons learned and the business impact the systems they've built have made. 

Best Practice #1: Build Apps So Quickly You Won't Mind Throwing Some Away

One IT executive told me "CEP allows me to build event-driven apps in 2-4 weeks, rather than 6-9 months with an app server and database - so I can say "yes" to more requests from their business clients, which helps us innovate more quickly. We keep the great ideas, and discard the ones that don't work."

Best Practice #2: Start Small, Succeed, then Scale

The most consistently successful firms that moved to real-time on Wall Street started with a small, critical application, learned lessons from its development, then applied those lessons learned to other projects and groups. Over time, they developed a center of excellence with firm-wide sucess in real-time application development.

 

Best Practice #3: Plan for Spikes, Not Aggregate Volumes

Systems that use primarily historical have trained many technologists to capacity plan their systems for the aggreate amount of data they manage. Real-time systems are different, and should be planned for spikes in volume, rather than only aggreate volumes. This best practice explores three approaches: burst behavior, the use of conflation, and computational reduction, that helps applications gracefully manage spikes and deliver correct and timely response to users and automated action. 

Keep coming back throughout 2013 as we continue to add to our library of real-time system best practices!

Best Practice #4: Incorporate Statistical Computing with Event Driven Systems

Numerical computing environments are often used for algorithm development, data analysis, visualization, matrix manipulations, and statistical analysis. In finance, quants use it to do time-series analysis, portfolio optimization, asset liability modeling, quantitative risk modeling, and derivative pricing. This best practice describes how to does this computing in real-time, using MATLAB with StreamBase CEP together.

Best Practice #5: Balance Man And Machine

The prospect of automating a business often sparks a debate about what is more effective, man or the machine. It turns out that the debate presents a false choice, and effective automation lies in the proper balance between the two.

This article explores 5 steps to balance man and machine, including the infusion of more data scientists in executive positions, the application of real-time risk management and surveillance, the infusion of historical and real-time analytics in one place, and more.

 

Geneva Energy Markets Deploys StreamBase for Oil Futures Trading

We love making customer announcements, so today we're thrilled to announce that Geneva Energy Markets, LLC (GEM), one of the leading oil traders in the world, has deployed StreamBase for their automated trading infrastructure. Moreover, they said that the StreamBase-based system "helps monitor, analyze and act on hundreds of thousnds of trading opportunities eavery day, and has led to one of its most profitable trading periods in history."

The StreamBase-based system helps GEM continuously monitor, analyze and act on trading opportunities by continuously evaluating, for example, heating oil prices in New York, gasoline in Europe, West Texas Intermediate crude oil, and more. Based on GEM’s trading algorithms, the system evaluates hundreds of thousands of combinations in parallel a day, decides how much the opportunity is worth, and acts, in milliseconds. The automated system executes pre-defined strategies electronically, and alerts traders of “high touch” opportunities for manual trading attention.

Mark Vonderheide, Managing Partner at GEM, nicely articulates a pragmatic view on the balance of low touch (i.e., automated) and high touch (i.e., manual) trading in this quote:

"Our StreamBase-based system helps us strike the balance between automated trading and manual trading,” said Mark Vonderheide, Managing Partner, GEM."

Mark also said nice things about our firm, which means a lot to us:

“We started with a vision to automate based on CEP in the fall of 2009, and StreamBase has not only met, but exceeded our expectations on what we wanted to achieve. The development environment has helped us get our applications into market quickly, and, just as important, the company has stood by us all the way as a partner, something that you don’t see with many software companies."

Today’s enterprise data is constantly in motion – the better you process that data, the better your results. Geneva Energy Markets is a great example of a firm that uses StreamBase to continuously correlate real-time conditions, and act in real time. Together with our partner, Trading Technologies, we think the oil swaps market is a prime opportunity for CEP-based innovation.

Potamus Selects StreamBase LiveView

Today, Potamus Trading, a wholesale broker-dealer based in Boston, announced they have selected StreamBase LiveView as a key element of their infrastructure and trading operations. StreamBase LiveView will be used to analyze trading performance, operations and financial risk in real-time. 

The 2010 Flash Crash, the more recent $440M loss by Knight Capital, and the inexorable rise in regulation and complexity in the electronic trading infrastructure make insight, control, and risk management capabilities increasingly important. StreamBase LiveView was created to address these rising demands in the capital markets, and today's selection by Potamus Trading is another example of why.

“As a broker-dealer in today’s market, it is important that we are also a partner, a reliable liquidity provider, an electronic trading advisor, and an expert on algorithmic execution,” said Eric Pritchett, CEO of Potamus Trading. StreamBase LiveView allows Potamus to incorporate real-time analytics into their trading infrastructure assessing trade performance during the day, while trading is live. LiveView is like a data warehouse, which provides interactive analysis, but the data is literally live - ad hoc queries are executed against live flow.

This live view of trading activity and LiveView's real-time ad-hoc query capabilities allow Potamus to respond quickly and help their broker-dealer customers optimize their executions.

This visibility, according to Potamus, also contributes to operational and financial risk management.

Potamus Trading's CEO Eric Pritchett discussed his view on the rising need for execution consulting services in this week's roundtable, moderated by Sang Lee, co-founder and managing director of AITE Group and included Bernard Donefer, Associate Director, Subotnick Financial Services Center, Baruch College, and Richard Tibbetts, CTO of StreamBase. 

Guangfa Building The Fastest Trading System in China with StreamBase CEP and Thomson Reuters

Guangfa (GF) Securities, one of the largest securities brokers in China, announced this week that they are building out a StreamBase CEP-based trading system, which will be the fastest in China, and includes innovative rapid pairs trading and order execution algorithms.

The GF system will be an end-to-end trading system, but what's remarkable about their system is that it's not being built with legions of programmers and consultants.  GF has a small specialist team that have re-used open-source components from the StreamBase Component Exchange and extended them with their own "secret sauce" and knowledge of the Chinese micro-market structure. 

GF's rapid time to market is made possible due to the real differences in how the leading products are built. Esper is a simple, "roll your own" toolkit, and lacks enterprise features and connectivity. Apama is a low-level scripting language with lots of pre-built applications for FX and surveillance, but doesn't have the robust development tools or visual programming support offered by StreamBase CEP.

StreamBase CEP is built for the professional developer and quantitative analyst. It's a comprehensive, event-based system platform, featuring a visual CEP development platform, seamless integration to event data, Java, Matlab, and over 150 environments typically found on the trading floor. This makes it possible for a small team of professional staff to build a complete system in months, not years.

StreamBase CEP is designed for speed. Speed of development - when measured, usually 40%-75% faster than other approaches. Speed of deployment - out of the box, applications that have sub-millisecond latency. Speed to work with Java - with the ability to quickly snap in existing Javacode, or extend event-based logic with Java. And finally, speed of integration, with an ecosystem of over 150 external integration points including our partners in China, Thomson Reuters.

Learning about this difference is repaid to developers, since time to market has a tangible impact on the business bottom line. We recently produced a 20 minute CEP programming tutorial by StreamBase Solutions Architect Mark Hudson that gives developers a good introduction to the power of visual programming with StreamBase Studio. Watch it and learn how you can get systems to market more quickly and with higher quality, leveraging what you have and the development practices you already use.

Do more with less and do it quickly.

 

Guest Blog by Peter Rossi on NETIQ, Aegis and CEP integration

In the increasingly complicated world of IT – Big Data, Cloud, Super Datacentres, DevOps and Agile Development legacy IT support teams are overwhelmed with the new world that they live in.

Traditional server architectures on a 3 or 5 year refresh cycle are being replaced with cloud solutions which may only exist for a matter of hours, but are equally important to the success of the organisations they underpin.

In most organisations IT monitoring systems are designed in from the ground up, as most CIO’s and CTO’s have been caught out by an unexplained outage at some point. However, many IT departments still live blissfully unaware of what is happening in their infrastructure…. Until it’s too late!

Regardless of the actual monitoring technology you will generally see a team of IT professionals attempting to understand this raft of new data. They have to assess multiple streams of monitoring data, understand its context, point the finger at the potential problem and then perform some form of remediation to resolve it. 

This has led to the rise of Complex Event Processing (CEP) engines coupled with IT work flow automation tools, such as NetIQ Aegis which utilises StreamBase as its CEP engine.

NetIQ Aegis allows you to listen to multiple streams of data from your IT monitoring systems simultaneously, for example:

• Server Alerts
• Storage Systems
• Network Switches
• Network Intrusion Detection Systems
• Firewalls
• Audit Logs

You can then create “triggers” using the StreamBase CEP to perform data correlation in real time; this is referred to as a temporal query. In the old world of querying static data, imagine this to be a static query that the data washes over.

This kind of “data in flight” analysis has changed the financial markets – you can now analyse events as they are happening – this type of processing allows you to watch for events across all available data streams in real time – e.g.

• If you see a missing ping alert within any 5 minute window:
  • AND the URL www.myurl.com stops responding
  • AND there is an alert from the storage subsystem
  • THEN Trigger

Every time the CEP engine sees a group of matching events it raises a new alert or trigger…. This is where it gets interesting….

NetIQ Aegis can use event triggers to kick off workflow processes. These workflow’s can contain any number of steps to remediate known faults or raise service desk tickets or make any number of changes to your systems to resolve the problem, automatically, with no human input.

This isn’t to say that we don’t need skilled IT professionals anymore, far from it, but why have a highly skilled worker repeating very standard and similar fixes all day long, when they could be put to use making your business more successful? Complex Event Processing can help you achieve this.

About the author

Peter Rossi is an IT automation expert in the UK currently working on exciting new highly automated cloud project Skyscape Cloud Services (www.skyscapecloud.com). Independently of this, Peter is the author of a well known Microsoft scripting blog www.poshpete.com which focusses on the use of PowerShell in automation. 

Follow Peter on Twitter 

 

Betfair Adopts StreamBase for Real-Time Analytics

(By Richard Tibbetts, StreamBase CTO)

Today we announced that Betfair has selected StreamBase for Real Time Analytics. I’m excited about this news as it demonstrates the importance of real-time data and analytics for driving the best possible consumer user experience for streaming, online services, and the role of StreamBase as the ideal platform for event-driven environments like Betfair.

American readers of this blog may be unfamiliar with Betfair.  Betfair is the world’s largest betting community and one of the world’s leading online betting and gaming operators. At the heart of Betfair is its pioneering Betting Exchange, where customers come together to bet on odds set by Betfair or offered by other customers, instead of with a traditional bookmaker. The Betting Exchange provides better pricing, flexibility, and a level playing field in a massive market. In April of this year, Betfair had over 4 million registered customers worldwide and processed, on average, more than 7 million transactions per day on the Betting Exchange. 

Betfair provides this service via Internet, mobile, and tablet devices that maximize the playing experience, and the Betfair infrastructure implements a service that implements its sophisticated betting market, real-time data processing, and real time clearing of bets.

Betfair is well known for its technically ambitious support for betting during a match. Players can bet while watching a live match, at the last possible moment. When I first met Alex Kozlenkov (Head of Research at Betfair) he described his challenge of executing 100,000 transactions per second against a single product, each transaction (or event) potentially impacts the price of that product, and must be processed in order.

And it’s not just the transactions that are streaming in the Betfair environment, they are also a leader in using real-time data to make the customer experience more engaging, fun, and unique. We’re in the midst of a renaissance of innovation in data management due to “Big Data” – innovative tools and infrastructure designed to manage increasing volume, velocity, and variety of information.  The betting world is no different - new real-time data feeds provide streaming play-by-play data from cricket matches, and real-time social media data brings an opportunity to crowd-source context about the events from fans watching live (in related news, StreamBase recently announced integration with GNIP and connectivity to Hadoop via Flume that enables connectivity to Big Data)

By investing in real-time analytics, Betfair can tap into streaming information and from the betting ecosystem and leverage it to provide a better experience for users live on the site, on mobile devices, and on tablets, providing innovative context and experience to the Betfair experience.

Consumers are now conditioned by social media to expect information and data to be real–time, whether it be pictures of a night out, stock twits, or a location update. Betfair represents a step in the march of all businesses to become more social media aware, real-time, and web-scale. Customers expect more than just a scrolling Twitter feed in the sidebar – they expect real-time context and relevance. The winning firms will integrate a variety of data sources - structured and unstructured, historical and real-time - and distill them down to insight that matters, when it matters. The real-time business of the future requires all of this, and requires real-time analytics and event processing as the new IT computing fabric.

Gain Capital: A Glimpse into the Future of Analytics?

Imagine trying to drive your car while ONLY looking in the rear-view mirror? If you're like most businesses, you make your decisions by looking at reports, which is like driving while only looking in the rear-view mirror.

First generation analytics and BI tools - Excel, Tableau, Qliktech - provide a map of where you've been. They're so essential that they are a $10 billion software market. But using old data to drive your business only tells you what potholes you already hit and what turns you could have made.

StreamBase LiveView provides a live dashboard of where you should go next. It analyzes live data; not old data. LiveView absorbs data streams with just milliseconds of latency - sensor data, high-speed market data, client orders, streaming news, tweets, FIX messages, network messages, emails, log data. LiveView gives you a live view of data - before everyone else sees the report tomorrow!

Live insight helps you see potholes before you run over them, and decide where to turn next. For a more complete overview of LiveView, watch the 4-minute intro presentation here. 

Today, GAIN Capital went public about their use of LiveView and how they use it to steer their business differently. Rob Perez, VP of Trading and Risk Management Systems at GAIN Capital, shared a terrific example of the transformation live analytics can bring. GAIN's business is trading, and success is linked to the quality of connectivity to the electronic markets. Gain services over 140 countries, with an average monthly transaction volume of over $275 billion. 

Previously, GAIN managed their trading operations with traditional tools - databases, log files and business intelligence. At the end of the day or week, reports are generated, problems are discovered, and addressed.

Now, LiveView helps GAIN understand and optimize their trading operations in real-time. Perez said in EuromoneyFXNews: "We an see if any metrics are moving outside of our comfort zone, and we can immediately reach out to our liquidity providers to understand what's going on."

At any time, GAIN clients can trade on any of 10 or more markets. It's essential to GAIN that problems with prices, connectivity, or quality are elimintated. Now, GAIN can now determine:

• What deals are being rejected now?
• Which liquidity providers are giving good prices now?
• Which liquidity providers are having problems now?

Live insight helps GAIN decide to call a poorly performing partner the moment there's a problem. Finding a problem as it's happening makes it easier to describe and easier to fix. And finding problems in real-time gives GAIN the opportunity to correct it before their clients notice. If the issue is with a partner who can't fix it on the spot, Gain can adjust and route around that market. 

StreamBase LiveView is the first platform of its kind to provide a complete analytics solution for live data and real-time action, which we believe could be a new disruptive physics that allows companies to broaden the decisions they make by looking ahead, not only behind. 

So, what decisions would YOU make differently if you stopped looking only in the rear-view mirror?

 

b-next Selects StreamBase: Regulation Goes Real-Time

Today's announcement by StreamBase partner b-next is part of steady shift toward real-time in the capital markets. b-next, a pioneering provider of multi-venue, multi- asset class capital markets surveillance and compliance solutions, announced their selection of StreamBase for their CMC:Suite solution set. By incorporating event processing, b-next's customers, such as regulators, exchanges and banks, will be able to rapidly implement near real-time market abuse surveillance solutions. This will enable relevant market participants to comply with recent ESMA guidelines and future proof themselves against additional regulation expected in the coming years.

Although event processing has often been associated with sub millisecond decision making and algorithmic trading, trading was indeed, just the tip of the iceberg. Pre-trade risk management followed, and now the entire infrastructure of the capital markets applying automation technologies like event processing. b-next’s systems is further evidence of that shift, as the CMC:Suite solution helps reduce compliance officers' response period by helping them immediately identify suspicious trading patterns and generating alerts. All in real-time.

"We're one flash crash away from being required to understand our risk in real-time," the head of electronic trading at a top-tier firm told me recently. His firm has deployed event processing globally to manage their trade flow, risk, and positions. After years of being deployed in the heart of electronic trading systems, recently the technology has been increasingly used to manage the same automated infrastructure its often used to power. 

We welcome our newest partner b-next to the StreamBase family!

About b-next

b-next is a provider of multi-venue, multi-asset class, Capital Markets Surveillance and Compliance software solutions to manage risk, meet regulatory mandates and drive trading efficiencies. The b-next Capital Markets Compliance (CMC) solution offers  a single integrated compliance platform for Market Abuse, Insider Trading, Conflict of Interest, Derivatives/OTC Monitoring, Best Execution, Fraud Detection and Prevention, AML and more. With over 150 customers, b-next has a 20 year history of delivering solutions to market participants, exchanges and capital markets regulators.