The StreamBase Event Processing Blog

Ease of Use: The Next Frontier for Event Processing

We're back! The acquisition of StreamBase by TIBCO was a big change, but as we near our "First 100 day" milestone, we're starting to come back to our regular conversation on the blog, and in our various social media outlets, including, of course, our new home @TIBCO. 

We kick it off with an article I wrote for IT Briefcase called Ease of Use: The Next Frontier for Event Processing: 

The future of computing will be event-driven, and recently, we’ve reached a new frontier in its emergence.

In June, the two leading “pure play” event-processing vendors, StreamBase and Apama, were acquired in the same week. These acquisitions prompted industry analysts to ponder who will be the winners in general-purpose analytic platforms for high velocity data. Philip Howard from Bloor Research wondered about the future of StreamBase, about IBM, SAS, and open-source projects like Cassandra, Impala, and Twitter Storm. Each has vastly different capabilities, focus, and ecosystems. Indeed, it’s still very early days for real-time technologies, but there is broad agreement that the next revolution in enterprise technology stacks will be driven by the need for real time.

But before picking a winner, shouldn’t we examine a more basic question: What do Main Street firms need to become more real time?

Continued at IT Briefcase...

Come Meet TIBCO StreamBase at SIFMA 2013

The Securities Industry and Financial Markets Association (SIMFA) Technology Expo this week (June 18th and 19th) is steeped in tradition. Although in-person shows are on the decline in favor of our voluminous social media communication, there’s still nothing like vendors looking their customers in the eye and talking about business.

Our stand will feature product demonstrations of StreamBase Event Processing and StreamBase LiveView. We’ve added a ton of new features in the past year, and we’re excited to show them to you, and get your feedback.

We’ll feature new solutions that include end-to-end business monitoring and pairs trading. And we’ll be giving you a sneak preview of the continuing work in the ‘Big Data” space we’ve done with Twitter Storm which is so new (it’s yet to be released) that you’ll have to come to the booth to check it out.

And of course the biggest news this year is that we have become part of TIBCO. I’m looking forward to talking to our customers and partners to discuss the “StreamBase 2.0” – the next step in our evolution, this time with the enviable array of event-driven products at our disposal! TIBCO possess the most prolific set of real-time computing technologies in the industry today. I’d like to hear your thoughts about how we can best unlock the value in integrating StreamBase with the TIBCO portfolio, and, as always, to hear ideas about how we can keep making StreamBase CEP and LiveView the best event processing platform in the industry.

TIBCO StreamBase’s stand is number 1854 in the Rhinelander Complex at the Hilton Hotel, Avenue of the Americas. We’ll have a happy hour reception on Tuesday, June 18th, from 3-5PM. See you there this week!

Algo Trading Saves Mark Cuban Millions During "Twitter Twitch"

Geeks on Wall Street make the market safer, turning a potential "Hack Crash" into a "Twitter Twitch," saving Cuban millons

Mark Palmer is the CEO of StreamBase Systems, a technology provider of real-time analytics technology.

Mark Cuban has provocative views on everything from the NBA draft to the Iraq war, so his latest shot at the financial trading industry comes as no surprise; the problem is, he lashed out at the Wall Street mavericks whose high tech trading systems worked perfectly, and probably saved him a lot of money.

In case you missed it, the Twitter account for the Associated Press was hacked, a hoax tweet was sent out that Barack Obama was injured in a bomb explosion. The Dow Jones fell 145 points in about 120 seconds and then recovered 134 points in the ensuing 3 minutes. So, in just 300 seconds, the DJIA traveled 279 points. 

Cuban told CNBC that "all of this algorithmically driven trading has created a ton of downstream problems," and proposed some "solutions" to the "problems" that include taxing trades that are held for less than one hour.

But here's where Cuban has the whole thing backwards: the technology he wants to shackle saved him, and the man on the street, millions.

On the day of the Hack, I was speaking about high frequency trading at a Bloomberg conference with 200 people in the audience. I asked the audience to raise their hands if they personally lost any money that day. Not one hand went up.

This wasn't a Hack Crash; it was a "Twitter Twitch." In fact, high frequency trading was exactly the antidote that saved all "regular" investors (even Mr. Cuban) money. 

It's true that Wall Street uses computers to make millions of trading decisions a second, and it's probably true that computers that detected the hacked tweet twitched in seconds. 

But those same algorithms that watch millions of data points a second could tell, in real-time, that something was amiss. The Tweet was algorithmically refuted - Obama was fine and healthy - the descent stopped immediately, and the markets returned to being healthy in just a few seconds.

Without real-time technology, the market would not have corrected so quickly; the hack created a twitch, not a crash, and the only ones in the market who lost money were probably the dumbest of algorithms that chased the lie. Most algorithmic trading insiders know that in the most volatile times, it's the human trader who tends to win; algorithms tend to get whip-lashed chasing purely statistical trends that they don't understand. 

Nobody, even a billionaire, can wish away the fact that the world is electronically fueled, and that social interaction, hoaxes, and trading are all done in real-time. To limit the use of real-time technologies on Wall Street would deny the markets the very tools it needs to make the market safer for everyone, and will continue to turn the next potential crash into a twitch.

Best Practice #5: Balance Man and Machine

This post is #5 in a series of lessons learned from leading StreamBase customers who have effectively deployed real-time business systems. 

The prospect of automating a business often sparks a debate about what is more effective, man or the machine. It turns out that the debate presents a false choice, and effective automation lies in the proper balance between the two.

Best practice #5, "Balance Man and Machine," is about how successful firms strike a balance between machine-driven and human decision- making.

THE SITUATION: FIRMS MUST AUTOMATE TO SURVIVE

In the capital markets, open outcry exchanges have been closing down for years. Last October, the IntercontinentalExchange closed down open outcry trading of soft commondities in New York, a practice that goes back at least 142 years. When it closed, the 1,000 traders left working on the floor "mostly monitored their market using handheld pricing screens even when they're sitting in the pits." That is, ironically, the manual brokers were literally using the automated tools that were making their jobs obsolete.

And the impact went beyond the survival of some jobs to the survival of some firms. For example, from 2008 to 2012, the once dominant NYSE and NASDAQ exchanges saw their combined market share slip from 67% to 45%, in large part due to upstart rivals BATS and Direct Edge known for their technology-based innovation. 

Obviously, automation has changed Wall Street forever, and it's about to change other industries.

THE PROBLEM: AUTOMATION THREATENS EXISTING CULTURE AND CAN ADD RISK

In the capital markets, the debate has raged on for years about what's more effective - human traders or algorithms.  A quick glance at a time-restricted Google search in 2007 for "Man Versus Machine" reminds us about how massive the debate was! 

Underneath this media coverage was a real issue: culture.

"Manual" traders, and the hierarchy of power over trading systems that was built over years by those who made their mark as manual traders, resisted the ideas that computers could do better than humans. Technologists, eager to make their mark on the P&L of the bank, fought to prove that computers could do a better job. 

At the same time, automation can introduce risk. Events like today's Twitter Hoax that President Obama had been injured at 1:07PM caused the markets to plunge 200 points in about 2 minutes. At 1:10PM, the first denials came out on Twitter, and by 1:13, the Dow had recoverd most of the ground it had lost.  So risk can no longer be done with traditional tools, which evaluate positions at the end of the day. By the end of the delay, it can be too late.

This polarized view of the battle between man and machine was wrong. 10 years on, automated trading is here to stay, and so are the people who guide its use. The truth is that neither man nor machine alone is better at trading; the truth is, a balanced approach is what was created. 

THE SOLUTION: FIVE STEPS TO BALANCE MAN & MACHINE

Automating businesses must change. Here are four observations from the transformation in the capital markets, and the best practices other firms in other industries can learn to facilitate their own change.

1) Business-geek hybrids are common

Traders used to have mostly degrees in business; today, they either additionally have math and science degrees, or have learned more about the implications of algorithms on trading. 

Best Practice: Proactively hire data scientists and put them into leadership positions so that they can lead a different, quantitative way of thinking about business transformation.

2) Humans are more comfortable taking cues from real-time analytics

Yes, lots of trading is automated. But automation also signals humans to make the big decisions. For example, Geneva Trading, one of the largest oil futures traders in the world, uses automation to monitor "millions of trading opportunities" simultaneously. Some opportunities are no-brainers, and the computer acts automatically. But sometimes the computer has no idea how to act, and punts the decision to a human by sending a text message, email, or alerting them on their desktop. 

Best Practice: Couple data scientists with operational staff that are similarly willing to embrace a technologically-driven customer resource management approach. These are customer service personelle who are used to analyzing information supplied by automated systems, and working with clients to solve high-value-add problems. The recent trend in capital markets toward execution consulting services is a good example of this trend. 

3) Real-time risk and surveillance becomes more important

One great myth about algorithmic trading is that it's done without oversight. In fact, all the clients I meet with use real-time analytics to temper, or stop, trading decisions when the data, statistically, does not make sense. One of our customers, PhaseCapital, was the inspiration for the blog post "Fast is Good, But Fast and Dumb is Dangerous" that describes the firms' real-time analytics that helped them trade more safely on the day of the May, 2010 Flash Crash. 

Best Practice: Employ real-time risk, surveilence, and compliance systems. One of the most prevalent applications of real-time platforms in the last 2 years has been real-time risk, compliance, and surveillence applications. 

4) Low-touch and high-touch functions must be integrated

Automation is useful to handle well known, low risk decisions, the ones that require little over site other than a big red panic button to make systems stop. Bigger, high touch decisions are ones handled by humans. By using computers to automate the low touch decisions, humans are empowered to make the tough, big decisions. 

Best Practice: Start with a balanced sense of technical requirements, rather than a world separated by automated and manual workflow. 

5) Infuse events with historical context

In the world of automation, the critical technology stack is event driven; the technology stack for manual processes is historical data driven. These two systems must coexist. Event-driven messaging, event-oriented programming languages (e.g., CEP), event-driven user interfaces, and event-oriented databases (e.g., tick stores) and streaming business intelligence must all sit along side of their historical counterparts. 

Best Practice: Integrate real-time and historical data to give humans oversight and insight into what automated algorithms are doing.  

THE IMPLICATION: AUTOMATION EMPOWERS HUMANS

Balanced well, automation can be a huge competitive weapon for a business. By applying lessons from one of the most deeply automated industries in the business world, firms embarking on automation initiatives  can increase their chance of success and reduce their risk of failure. 

StreamBase and CEP Usage Expands in China

The trading market in China for event processing continues to grow.

Today, we announced a partnership with Zhejiang Hithink Flush Information Network Co., Ltd. (“Hithink”), who will use StreamBase to build trading products for the Chinese market, and also sell and support CEP-based application development in China. This partnership comes as a result of our StreamBase ISV Acclerator Program and Gunagfa's public adoption StreamBase last year.

Hithink has a history of successfully building market data and trading systems for their retail and institutional clients, with over 1,000 employees, 50 million customers and revenues of over 200M RMB.

Hithink will use the StreamBase platform to improve their existing infrastructure and develop new high performance trading applications such as algorithmic trading, transaction cost analysis, and market data management that make use of key CEP features such as aggregation, correlation and temporal analytics. 

Image Credit: Want China Times

Potamus Selects StreamBase LiveView

Today, Potamus Trading, a wholesale broker-dealer based in Boston, announced they have selected StreamBase LiveView as a key element of their infrastructure and trading operations. StreamBase LiveView will be used to analyze trading performance, operations and financial risk in real-time. 

The 2010 Flash Crash, the more recent $440M loss by Knight Capital, and the inexorable rise in regulation and complexity in the electronic trading infrastructure make insight, control, and risk management capabilities increasingly important. StreamBase LiveView was created to address these rising demands in the capital markets, and today's selection by Potamus Trading is another example of why.

“As a broker-dealer in today’s market, it is important that we are also a partner, a reliable liquidity provider, an electronic trading advisor, and an expert on algorithmic execution,” said Eric Pritchett, CEO of Potamus Trading. StreamBase LiveView allows Potamus to incorporate real-time analytics into their trading infrastructure assessing trade performance during the day, while trading is live. LiveView is like a data warehouse, which provides interactive analysis, but the data is literally live - ad hoc queries are executed against live flow.

This live view of trading activity and LiveView's real-time ad-hoc query capabilities allow Potamus to respond quickly and help their broker-dealer customers optimize their executions.

This visibility, according to Potamus, also contributes to operational and financial risk management.

Potamus Trading's CEO Eric Pritchett discussed his view on the rising need for execution consulting services in this week's roundtable, moderated by Sang Lee, co-founder and managing director of AITE Group and included Bernard Donefer, Associate Director, Subotnick Financial Services Center, Baruch College, and Richard Tibbetts, CTO of StreamBase. 

5 Best Practices of Execution Consulting Services

Yesterday, StreamBase hosted a webinar entitled “The 5 Best Practices of Execution Consulting Services” - an interview with StreamBase CTO Richard Tibbetts and Stephane Loiseau, Managing Director and Head of Equity Execution at Société Générale. The session described an important trend in the capital markets - the rise of execution consulting services (ECS).

ECS is defined as a sell-side service that helps buy-side clients achieve greater trade performance and efficiency. As electronic trading space gets more competitive, broker-dealers like Société Générale need to differentiate their trading services from their competitors. ECS is where the rubber meets the road, and provides the link between a broker and their clients to help them understand the market, market data, improve their trading performance and achieve their benchmarks.

ECS isn’t just one person’s job - it requires a mix of trading, technology and people. The 5 best practices Stephane presented are:

1. Find the Right People for the Job. Staffing the ECS function is one of the biggest challenges Stephane has at Société Générale. Today buy side has become more sophisticated with the use of electronic trading. They are not only familiar with the self-service tools and algorithms provided by sell-side, also exploring new technology and strategies internally. The expectation of ECS has increased. ECS used to be simply a technology job, because algorithms were new, and the service was new. But over the years, the trader’s role has evolved. An ECS team needs staff with a background in trading, but someone with experience in trade operations is critical to help understand the processing of an order from A to Z. Finally, the client relationship role is essential. Trading isn’t just about playing with technology; you’re interacting with the client to provide a service. Finally, there’s the intersection of these people skills and tools - the right ECS tools enable an ECS team to focus on providing great service. These people and tools are normally found in different organizational silos. So the best practice of ECS is to craft an ECS team with the right skills, team composition, and tools to get the job done.

2. Continually Re-Assess Market Micro Structure. Yes, the market is complex, liquidity is fragmented. These are true statements. But understanding all the aspects of microstructure can lead to better trading quality. If your ECS team understands where liquidity is coming from, what market events are affecting the market in real-time, and how speed affects trading, then they can help clients get better trading performance. For example, if your platform aggressively deals with dark pools in Europe, you can’t build trading strategies around where the liquidity is. The challenge is that market micro-structure knowledge has a shelf life of a few weeks. For example, if a venue changes its pricing structure, that’s a major event - that changes monthly. News changes daily. Market data changes in real-time. So best practice #2 is that ECS staff must continually re-assess market microstructure in order to help clients get the best trading results.

3. Establish a Dynamic, Real-Time System for Monitoring, Reporting, and Action. Historically, traders decided the actions they are going take, then carried on with those actions throughout the day, and results are assessed at the end of the day. This no longer the case - you need to do all of this in real-time. In order to accomplish this, ECS teams need a great monitoring system; it’s just too much data for the human brain to consume. At Société Générale, they analyze the market in real-time, based on microstructure and also macro level news events. But monitoring tools can’t be stale and static, which is why a robust alerting system is so important - you need to have some form of triggers that will help you know when to change your strategies - they could be based on volatility, or any other metric the ECS team decides is important on that day, or moment. So Stephane’s third best practice is to provide killer tools to the ECS team to monitor, assess, and adjust trading strategies in real-time.

4. Find a Balance Between Technology and High Touch Human Interaction. Over the last month, where volumes have been so low, Société Générale’s ECS team has adjusted the trading triggers they consider. ECS is not just about low touch (fully algo trading, all technology based) or just high touch (human trader interaction); these models are converging very quickly. Now ECS teams provide a lot more human interaction to change an algorithm, tweak their parameters, or help a client change their strategy as conditions change. For example, if you have an order for a client, and you start to work the order with an algorithm, and you can see there is an opportunity to cross the order on the floor, you need to call the customer and ask him or her if you want to close the cross. So you are seeing an opportunity, consulting with your clients, and saving them money. Another example is if the algorithm is not performing the way it should be - this is when the ECS consultant would reach out to the customer - should we split up the order into 2 and work them independently? So the fourth best practice of ECS is to strike a new balance between high and low touch service.

5. Understand that TCA is a Process Throughout the Trading Cycle, Not Just a Metric. Stephane feels that the notion of TCA must transform from an end-of-day metric to a trading lifecycle driver - the ECS team must think about the TCA throughout the lifecycle of trading. The ECS team must think about how their actions will affect the TCA grade you want to achieve. Then actions should be made carefully based on your data, which tools you use, which venues to trade, and so on. For example, on a specific order, the ECS team can decide to work a million shares on its own, but also to drop 500,000 in a VWAP algo for 2 hours with a specific metric just to make sure they are participating in the market. And the team must constantly monitor and adjust that strategy during the day with their monitoring tools (best practice 3). Stephane provided another simple example - when the trading day gets really busy, ECS staff can put more flow into the algorithm than manually trading. So you’re going to use data from previous days and from real-time to make your decisions and use as much data as you can throughout the lifecycle of the order. So changing the way the ECS team views TCA is the fifth best practice of ECS.

The session concluded with some Q&A, including how Stephane views ECS developing outside of the U.S. and Europe, how important it is on the trading floor, and how customized algorithms fit into the ECS strategy. For the Q&A, and to hear the full interview, click here.

Betfair Adopts StreamBase for Real-Time Analytics

(By Richard Tibbetts, StreamBase CTO)

Today we announced that Betfair has selected StreamBase for Real Time Analytics. I’m excited about this news as it demonstrates the importance of real-time data and analytics for driving the best possible consumer user experience for streaming, online services, and the role of StreamBase as the ideal platform for event-driven environments like Betfair.

American readers of this blog may be unfamiliar with Betfair.  Betfair is the world’s largest betting community and one of the world’s leading online betting and gaming operators. At the heart of Betfair is its pioneering Betting Exchange, where customers come together to bet on odds set by Betfair or offered by other customers, instead of with a traditional bookmaker. The Betting Exchange provides better pricing, flexibility, and a level playing field in a massive market. In April of this year, Betfair had over 4 million registered customers worldwide and processed, on average, more than 7 million transactions per day on the Betting Exchange. 

Betfair provides this service via Internet, mobile, and tablet devices that maximize the playing experience, and the Betfair infrastructure implements a service that implements its sophisticated betting market, real-time data processing, and real time clearing of bets.

Betfair is well known for its technically ambitious support for betting during a match. Players can bet while watching a live match, at the last possible moment. When I first met Alex Kozlenkov (Head of Research at Betfair) he described his challenge of executing 100,000 transactions per second against a single product, each transaction (or event) potentially impacts the price of that product, and must be processed in order.

And it’s not just the transactions that are streaming in the Betfair environment, they are also a leader in using real-time data to make the customer experience more engaging, fun, and unique. We’re in the midst of a renaissance of innovation in data management due to “Big Data” – innovative tools and infrastructure designed to manage increasing volume, velocity, and variety of information.  The betting world is no different - new real-time data feeds provide streaming play-by-play data from cricket matches, and real-time social media data brings an opportunity to crowd-source context about the events from fans watching live (in related news, StreamBase recently announced integration with GNIP and connectivity to Hadoop via Flume that enables connectivity to Big Data)

By investing in real-time analytics, Betfair can tap into streaming information and from the betting ecosystem and leverage it to provide a better experience for users live on the site, on mobile devices, and on tablets, providing innovative context and experience to the Betfair experience.

Consumers are now conditioned by social media to expect information and data to be real–time, whether it be pictures of a night out, stock twits, or a location update. Betfair represents a step in the march of all businesses to become more social media aware, real-time, and web-scale. Customers expect more than just a scrolling Twitter feed in the sidebar – they expect real-time context and relevance. The winning firms will integrate a variety of data sources - structured and unstructured, historical and real-time - and distill them down to insight that matters, when it matters. The real-time business of the future requires all of this, and requires real-time analytics and event processing as the new IT computing fabric.

Gain Capital: A Glimpse into the Future of Analytics?

Imagine trying to drive your car while ONLY looking in the rear-view mirror? If you're like most businesses, you make your decisions by looking at reports, which is like driving while only looking in the rear-view mirror.

First generation analytics and BI tools - Excel, Tableau, Qliktech - provide a map of where you've been. They're so essential that they are a $10 billion software market. But using old data to drive your business only tells you what potholes you already hit and what turns you could have made.

StreamBase LiveView provides a live dashboard of where you should go next. It analyzes live data; not old data. LiveView absorbs data streams with just milliseconds of latency - sensor data, high-speed market data, client orders, streaming news, tweets, FIX messages, network messages, emails, log data. LiveView gives you a live view of data - before everyone else sees the report tomorrow!

Live insight helps you see potholes before you run over them, and decide where to turn next. For a more complete overview of LiveView, watch the 4-minute intro presentation here. 

Today, GAIN Capital went public about their use of LiveView and how they use it to steer their business differently. Rob Perez, VP of Trading and Risk Management Systems at GAIN Capital, shared a terrific example of the transformation live analytics can bring. GAIN's business is trading, and success is linked to the quality of connectivity to the electronic markets. Gain services over 140 countries, with an average monthly transaction volume of over $275 billion. 

Previously, GAIN managed their trading operations with traditional tools - databases, log files and business intelligence. At the end of the day or week, reports are generated, problems are discovered, and addressed.

Now, LiveView helps GAIN understand and optimize their trading operations in real-time. Perez said in EuromoneyFXNews: "We an see if any metrics are moving outside of our comfort zone, and we can immediately reach out to our liquidity providers to understand what's going on."

At any time, GAIN clients can trade on any of 10 or more markets. It's essential to GAIN that problems with prices, connectivity, or quality are elimintated. Now, GAIN can now determine:

• What deals are being rejected now?
• Which liquidity providers are giving good prices now?
• Which liquidity providers are having problems now?

Live insight helps GAIN decide to call a poorly performing partner the moment there's a problem. Finding a problem as it's happening makes it easier to describe and easier to fix. And finding problems in real-time gives GAIN the opportunity to correct it before their clients notice. If the issue is with a partner who can't fix it on the spot, Gain can adjust and route around that market. 

StreamBase LiveView is the first platform of its kind to provide a complete analytics solution for live data and real-time action, which we believe could be a new disruptive physics that allows companies to broaden the decisions they make by looking ahead, not only behind. 

So, what decisions would YOU make differently if you stopped looking only in the rear-view mirror?

 

Chasing an Ambulance Full of BATS

The high-profile trading failure at BATS this week sparked a rash of negative publicity, gracing the front page of media outlets from the Wall Street Journal to Forbes. In the software industry, more software companies have proclaimed their techology would have sheltered BATS from their storm than cheap umbrella sidewalk vendors on a rainy day in New York City.

An especially troubling rash of "thought leadership" came from my own industry - real-time analytics and complex event processing. Forbes, for example, published an article by SAP (who sells a CEP product) BATS hits the rocks, which may have lifted its sales that proclaims that the BATS incident portends that "CEP technology is about to move from the backwaters of data management technology to center stage."

In Wall Street and Technology, John Bates, the CTO of Progress Software was quoted in the artice Flash Crash at BATS renews Market Concerns to say firms "clearly lacked adequate risk controls such as real-time surveillance and monitoring. Hopefully this incident serves as a wake up call." What kind of software do you think Bates sells?  Drum roll please... real-time surveillence and monitoring!"

My company sells these kinds of solutions too. And I know BATS well (disclosure: BATS is not a StreamBase client), and they are as technologically advanced as any firm in the world. So this kind of ambulance chasing is not only unbecoming, it's also dead wrong. It's like suggesting Formula One racing will be safer if the cars can be made to go faster (CEP) and we do a better job of filming them as they go careening into a wall (monitoring). Indeed, BATS quickly reported that they had a bug in their system, so comments about risk management and surveillance are irrelevent.

That said, there's an issue here for the capital markets - but the issue is cultural, not technological. 

What's happening in the capital markets is that we are in the midst of real-time automation renaissance. 10 years ago, less than 5% of stock trading in the U.S. markets was done algorithmically; today, as much as 70% is done on computer-implemented, and human designed, algorithms.

Larry Tabb, the founder and chief executive of the Tabb Group, said in the Times that “The markets basically gutted their high-cost, nonstop infrastructures for very fast, low-cost infrastructures." Now we are talking - Larry is on the right path!

In software, mistakes happen frequently. Professor Neil Johnson at the University of Miami analyzed 18,520 "ultrafast Black Swan [trading] events" from 2005-2011. In Stock Market Flaws Not So Rare, the New York Times that just one of the 13 U.S. stock exchanges - NASDAQ - declared 139 breakdowns at other exchanges last year alone, up from 69 the year before.

Software breaks. We have a word for it: bugs. But that's not new. What is new is that the errors can do damage so much more quickly, and, because the world is now so hyperconnected, everyone knows about it more quickly than ever before. 

So what is the magic elixir to making the capital markets fast, fair, and safe?  

There are still voices out there that seem to think that the answer is to discontinue the use of computers for automation. Ha! The whining about the evils of capitalism and sophisticated technologies continues to rain down from dissenters posting their admonations from inside their pup tents via shiny new iPad 3's. These dissenters need to Get Over It - technology is here to stay, and its use is expanding quickly from Wall Street to Main Street.

Making automation safe and reliable requires complex, cultural change:

1. Continue to refine reporting requirements. You can't manage what you can't measure. In the United States, exchanges didn't report outages on other exchanges until 2007. Nasdaq reports that there have been 110 outages in the last year like the one that triggered the BATS mishap. Continued measurement and transparency will lead the industry toward greater reliability. With more analysis and reporting, more accountability is possible, and regulators could focus on correlating events among exchanges and brokerages to look for systemic flaws.

2. Refine and lighten regulation; don't just pile it on. Dodd Frank and MIFID 2 have put regulation on the must-do agenda of all firms. But it's still largely up to the reader to separate what's essential from what's fluff, and that's not effective. Exchanges like BATS are extrodinarily regulated - so more, heavier regulation is not the answer.

3. More systemic attention to software quality.  Moore's Law - the idea that the power of computing doubles approximately every two years, does not apply to software quality. The BATS crash shines a light on the issue that is inbred in innovation; the rising importance of software testing, quality, and simulation. Most firms have ad-hoc processes for quality and should have continuous testing, regression testing, and real world simulation. CEP can make systems go faster, and surveillence helps spot problems more quicky, but when more attention is paid to testing and simulation, less errors occur. 

These changes are complex and cultural, not technological. Tools will help, but not solve, the challenge of ensuring a safe, reliable, and fair market.