This post is #5 in a series of lessons learned from leading StreamBase customers who have effectively deployed real-time business systems.
The prospect of automating a business often sparks a debate about what is more effective, man or the machine. It turns out that the debate presents a false choice, and effective automation lies in the proper balance between the two.
Best practice #5, "Balance Man and Machine," is about how successful firms strike a balance between machine-driven and human decision- making.
THE SITUATION: FIRMS MUST AUTOMATE TO SURVIVE
In the capital markets, open outcry exchanges have been closing down for years. Last October, the IntercontinentalExchange closed down open outcry trading of soft commondities in New York, a practice that goes back at least 142 years. When it closed, the 1,000 traders left working on the floor "mostly monitored their market using handheld pricing screens even when they're sitting in the pits." That is, ironically, the manual brokers were literally using the automated tools that were making their jobs obsolete.
And the impact went beyond the survival of some jobs to the survival of some firms. For example, from 2008 to 2012, the once dominant NYSE and NASDAQ exchanges saw their combined market share slip from 67% to 45%, in large part due to upstart rivals BATS and Direct Edge known for their technology-based innovation.
Obviously, automation has changed Wall Street forever, and it's about to change other industries.
THE PROBLEM: AUTOMATION THREATENS EXISTING CULTURE AND CAN ADD RISK
In the capital markets, the debate has raged on for years about what's more effective - human traders or algorithms. A quick glance at a time-restricted Google search in 2007 for "Man Versus Machine" reminds us about how massive the debate was!
Underneath this media coverage was a real issue: culture.
"Manual" traders, and the hierarchy of power over trading systems that was built over years by those who made their mark as manual traders, resisted the ideas that computers could do better than humans. Technologists, eager to make their mark on the P&L of the bank, fought to prove that computers could do a better job.
At the same time, automation can introduce risk. Events like today's Twitter Hoax that President Obama had been injured at 1:07PM caused the markets to plunge 200 points in about 2 minutes. At 1:10PM, the first denials came out on Twitter, and by 1:13, the Dow had recoverd most of the ground it had lost. So risk can no longer be done with traditional tools, which evaluate positions at the end of the day. By the end of the delay, it can be too late.
This polarized view of the battle between man and machine was wrong. 10 years on, automated trading is here to stay, and so are the people who guide its use. The truth is that neither man nor machine alone is better at trading; the truth is, a balanced approach is what was created.
THE SOLUTION: FIVE STEPS TO BALANCE MAN & MACHINE
Automating businesses must change. Here are four observations from the transformation in the capital markets, and the best practices other firms in other industries can learn to facilitate their own change.
1) Business-geek hybrids are common
Traders used to have mostly degrees in business; today, they either additionally have math and science degrees, or have learned more about the implications of algorithms on trading.
Best Practice: Proactively hire data scientists and put them into leadership positions so that they can lead a different, quantitative way of thinking about business transformation.
2) Humans are more comfortable taking cues from real-time analytics
Yes, lots of trading is automated. But automation also signals humans to make the big decisions. For example, Geneva Trading, one of the largest oil futures traders in the world, uses automation to monitor "millions of trading opportunities" simultaneously. Some opportunities are no-brainers, and the computer acts automatically. But sometimes the computer has no idea how to act, and punts the decision to a human by sending a text message, email, or alerting them on their desktop.
Best Practice: Couple data scientists with operational staff that are similarly willing to embrace a technologically-driven customer resource management approach. These are customer service personelle who are used to analyzing information supplied by automated systems, and working with clients to solve high-value-add problems. The recent trend in capital markets toward execution consulting services is a good example of this trend.
3) Real-time risk and surveillance becomes more important
One great myth about algorithmic trading is that it's done without oversight. In fact, all the clients I meet with use real-time analytics to temper, or stop, trading decisions when the data, statistically, does not make sense. One of our customers, PhaseCapital, was the inspiration for the blog post "Fast is Good, But Fast and Dumb is Dangerous" that describes the firms' real-time analytics that helped them trade more safely on the day of the May, 2010 Flash Crash.
Best Practice: Employ real-time risk, surveilence, and compliance systems. One of the most prevalent applications of real-time platforms in the last 2 years has been real-time risk, compliance, and surveillence applications.
4) Low-touch and high-touch functions must be integrated
Automation is useful to handle well known, low risk decisions, the ones that require little over site other than a big red panic button to make systems stop. Bigger, high touch decisions are ones handled by humans. By using computers to automate the low touch decisions, humans are empowered to make the tough, big decisions.
Best Practice: Start with a balanced sense of technical requirements, rather than a world separated by automated and manual workflow.
5) Infuse events with historical context
In the world of automation, the critical technology stack is event driven; the technology stack for manual processes is historical data driven. These two systems must coexist. Event-driven messaging, event-oriented programming languages (e.g., CEP), event-driven user interfaces, and event-oriented databases (e.g., tick stores) and streaming business intelligence must all sit along side of their historical counterparts.
Best Practice: Integrate real-time and historical data to give humans oversight and insight into what automated algorithms are doing.
THE IMPLICATION: AUTOMATION EMPOWERS HUMANS
Balanced well, automation can be a huge competitive weapon for a business. By applying lessons from one of the most deeply automated industries in the business world, firms embarking on automation initiatives can increase their chance of success and reduce their risk of failure.