The high-profile trading failure at BATS this week sparked a rash of negative publicity, gracing the front page of media outlets from the Wall Street Journal to Forbes. In the software industry, more software companies have proclaimed their techology would have sheltered BATS from their storm than cheap umbrella sidewalk vendors on a rainy day in New York City.
An especially troubling rash of "thought leadership" came from my own industry - real-time analytics and complex event processing. Forbes, for example, published an article by SAP (who sells a CEP product) BATS hits the rocks, which may have lifted its sales that proclaims that the BATS incident portends that "CEP technology is about to move from the backwaters of data management technology to center stage."
In Wall Street and Technology, John Bates, the CTO of Progress Software was quoted in the artice Flash Crash at BATS renews Market Concerns to say firms "clearly lacked adequate risk controls such as real-time surveillance and monitoring. Hopefully this incident serves as a wake up call." What kind of software do you think Bates sells? Drum roll please... real-time surveillence and monitoring!"
My company sells these kinds of solutions too. And I know BATS well (disclosure: BATS is not a StreamBase client), and they are as technologically advanced as any firm in the world. So this kind of ambulance chasing is not only unbecoming, it's also dead wrong. It's like suggesting Formula One racing will be safer if the cars can be made to go faster (CEP) and we do a better job of filming them as they go careening into a wall (monitoring). Indeed, BATS quickly reported that they had a bug in their system, so comments about risk management and surveillance are irrelevent.
That said, there's an issue here for the capital markets - but the issue is cultural, not technological.
What's happening in the capital markets is that we are in the midst of real-time automation renaissance. 10 years ago, less than 5% of stock trading in the U.S. markets was done algorithmically; today, as much as 70% is done on computer-implemented, and human designed, algorithms.
Larry Tabb, the founder and chief executive of the Tabb Group, said in the Times that “The markets basically gutted their high-cost, nonstop infrastructures for very fast, low-cost infrastructures." Now we are talking - Larry is on the right path!
In software, mistakes happen frequently. Professor Neil Johnson at the University of Miami analyzed 18,520 "ultrafast Black Swan [trading] events" from 2005-2011. In Stock Market Flaws Not So Rare, the New York Times that just one of the 13 U.S. stock exchanges - NASDAQ - declared 139 breakdowns at other exchanges last year alone, up from 69 the year before.
Software breaks. We have a word for it: bugs. But that's not new. What is new is that the errors can do damage so much more quickly, and, because the world is now so hyperconnected, everyone knows about it more quickly than ever before.
So what is the magic elixir to making the capital markets fast, fair, and safe?
There are still voices out there that seem to think that the answer is to discontinue the use of computers for automation. Ha! The whining about the evils of capitalism and sophisticated technologies continues to rain down from dissenters posting their admonations from inside their pup tents via shiny new iPad 3's. These dissenters need to Get Over It - technology is here to stay, and its use is expanding quickly from Wall Street to Main Street.
Making automation safe and reliable requires complex, cultural change:
1. Continue to refine reporting requirements. You can't manage what you can't measure. In the United States, exchanges didn't report outages on other exchanges until 2007. Nasdaq reports that there have been 110 outages in the last year like the one that triggered the BATS mishap. Continued measurement and transparency will lead the industry toward greater reliability. With more analysis and reporting, more accountability is possible, and regulators could focus on correlating events among exchanges and brokerages to look for systemic flaws.
2. Refine and lighten regulation; don't just pile it on. Dodd Frank and MIFID 2 have put regulation on the must-do agenda of all firms. But it's still largely up to the reader to separate what's essential from what's fluff, and that's not effective. Exchanges like BATS are extrodinarily regulated - so more, heavier regulation is not the answer.
3. More systemic attention to software quality. Moore's Law - the idea that the power of computing doubles approximately every two years, does not apply to software quality. The BATS crash shines a light on the issue that is inbred in innovation; the rising importance of software testing, quality, and simulation. Most firms have ad-hoc processes for quality and should have continuous testing, regression testing, and real world simulation. CEP can make systems go faster, and surveillence helps spot problems more quicky, but when more attention is paid to testing and simulation, less errors occur.
These changes are complex and cultural, not technological. Tools will help, but not solve, the challenge of ensuring a safe, reliable, and fair market.
