The StreamBase Event Processing Blog

Best Practice #5: Balance Man and Machine

This post is #5 in a series of lessons learned from leading StreamBase customers who have effectively deployed real-time business systems. 

The prospect of automating a business often sparks a debate about what is more effective, man or the machine. It turns out that the debate presents a false choice, and effective automation lies in the proper balance between the two.

Best practice #5, "Balance Man and Machine," is about how successful firms strike a balance between machine-driven and human decision- making.

THE SITUATION: FIRMS MUST AUTOMATE TO SURVIVE

In the capital markets, open outcry exchanges have been closing down for years. Last October, the IntercontinentalExchange closed down open outcry trading of soft commondities in New York, a practice that goes back at least 142 years. When it closed, the 1,000 traders left working on the floor "mostly monitored their market using handheld pricing screens even when they're sitting in the pits." That is, ironically, the manual brokers were literally using the automated tools that were making their jobs obsolete.

And the impact went beyond the survival of some jobs to the survival of some firms. For example, from 2008 to 2012, the once dominant NYSE and NASDAQ exchanges saw their combined market share slip from 67% to 45%, in large part due to upstart rivals BATS and Direct Edge known for their technology-based innovation. 

Obviously, automation has changed Wall Street forever, and it's about to change other industries.

THE PROBLEM: AUTOMATION THREATENS EXISTING CULTURE AND CAN ADD RISK

In the capital markets, the debate has raged on for years about what's more effective - human traders or algorithms.  A quick glance at a time-restricted Google search in 2007 for "Man Versus Machine" reminds us about how massive the debate was! 

Underneath this media coverage was a real issue: culture.

"Manual" traders, and the hierarchy of power over trading systems that was built over years by those who made their mark as manual traders, resisted the ideas that computers could do better than humans. Technologists, eager to make their mark on the P&L of the bank, fought to prove that computers could do a better job. 

At the same time, automation can introduce risk. Events like today's Twitter Hoax that President Obama had been injured at 1:07PM caused the markets to plunge 200 points in about 2 minutes. At 1:10PM, the first denials came out on Twitter, and by 1:13, the Dow had recoverd most of the ground it had lost.  So risk can no longer be done with traditional tools, which evaluate positions at the end of the day. By the end of the delay, it can be too late.

This polarized view of the battle between man and machine was wrong. 10 years on, automated trading is here to stay, and so are the people who guide its use. The truth is that neither man nor machine alone is better at trading; the truth is, a balanced approach is what was created. 

THE SOLUTION: FIVE STEPS TO BALANCE MAN & MACHINE

Automating businesses must change. Here are four observations from the transformation in the capital markets, and the best practices other firms in other industries can learn to facilitate their own change.

1) Business-geek hybrids are common

Traders used to have mostly degrees in business; today, they either additionally have math and science degrees, or have learned more about the implications of algorithms on trading. 

Best Practice: Proactively hire data scientists and put them into leadership positions so that they can lead a different, quantitative way of thinking about business transformation.

2) Humans are more comfortable taking cues from real-time analytics

Yes, lots of trading is automated. But automation also signals humans to make the big decisions. For example, Geneva Trading, one of the largest oil futures traders in the world, uses automation to monitor "millions of trading opportunities" simultaneously. Some opportunities are no-brainers, and the computer acts automatically. But sometimes the computer has no idea how to act, and punts the decision to a human by sending a text message, email, or alerting them on their desktop. 

Best Practice: Couple data scientists with operational staff that are similarly willing to embrace a technologically-driven customer resource management approach. These are customer service personelle who are used to analyzing information supplied by automated systems, and working with clients to solve high-value-add problems. The recent trend in capital markets toward execution consulting services is a good example of this trend. 

3) Real-time risk and surveillance becomes more important

One great myth about algorithmic trading is that it's done without oversight. In fact, all the clients I meet with use real-time analytics to temper, or stop, trading decisions when the data, statistically, does not make sense. One of our customers, PhaseCapital, was the inspiration for the blog post "Fast is Good, But Fast and Dumb is Dangerous" that describes the firms' real-time analytics that helped them trade more safely on the day of the May, 2010 Flash Crash. 

Best Practice: Employ real-time risk, surveilence, and compliance systems. One of the most prevalent applications of real-time platforms in the last 2 years has been real-time risk, compliance, and surveillence applications. 

4) Low-touch and high-touch functions must be integrated

Automation is useful to handle well known, low risk decisions, the ones that require little over site other than a big red panic button to make systems stop. Bigger, high touch decisions are ones handled by humans. By using computers to automate the low touch decisions, humans are empowered to make the tough, big decisions. 

Best Practice: Start with a balanced sense of technical requirements, rather than a world separated by automated and manual workflow. 

5) Infuse events with historical context

In the world of automation, the critical technology stack is event driven; the technology stack for manual processes is historical data driven. These two systems must coexist. Event-driven messaging, event-oriented programming languages (e.g., CEP), event-driven user interfaces, and event-oriented databases (e.g., tick stores) and streaming business intelligence must all sit along side of their historical counterparts. 

Best Practice: Integrate real-time and historical data to give humans oversight and insight into what automated algorithms are doing.  

THE IMPLICATION: AUTOMATION EMPOWERS HUMANS

Balanced well, automation can be a huge competitive weapon for a business. By applying lessons from one of the most deeply automated industries in the business world, firms embarking on automation initiatives  can increase their chance of success and reduce their risk of failure. 

Gain Capital: A Glimpse into the Future of Analytics?

Imagine trying to drive your car while ONLY looking in the rear-view mirror? If you're like most businesses, you make your decisions by looking at reports, which is like driving while only looking in the rear-view mirror.

First generation analytics and BI tools - Excel, Tableau, Qliktech - provide a map of where you've been. They're so essential that they are a $10 billion software market. But using old data to drive your business only tells you what potholes you already hit and what turns you could have made.

StreamBase LiveView provides a live dashboard of where you should go next. It analyzes live data; not old data. LiveView absorbs data streams with just milliseconds of latency - sensor data, high-speed market data, client orders, streaming news, tweets, FIX messages, network messages, emails, log data. LiveView gives you a live view of data - before everyone else sees the report tomorrow!

Live insight helps you see potholes before you run over them, and decide where to turn next. For a more complete overview of LiveView, watch the 4-minute intro presentation here. 

Today, GAIN Capital went public about their use of LiveView and how they use it to steer their business differently. Rob Perez, VP of Trading and Risk Management Systems at GAIN Capital, shared a terrific example of the transformation live analytics can bring. GAIN's business is trading, and success is linked to the quality of connectivity to the electronic markets. Gain services over 140 countries, with an average monthly transaction volume of over $275 billion. 

Previously, GAIN managed their trading operations with traditional tools - databases, log files and business intelligence. At the end of the day or week, reports are generated, problems are discovered, and addressed.

Now, LiveView helps GAIN understand and optimize their trading operations in real-time. Perez said in EuromoneyFXNews: "We an see if any metrics are moving outside of our comfort zone, and we can immediately reach out to our liquidity providers to understand what's going on."

At any time, GAIN clients can trade on any of 10 or more markets. It's essential to GAIN that problems with prices, connectivity, or quality are elimintated. Now, GAIN can now determine:

• What deals are being rejected now?
• Which liquidity providers are giving good prices now?
• Which liquidity providers are having problems now?

Live insight helps GAIN decide to call a poorly performing partner the moment there's a problem. Finding a problem as it's happening makes it easier to describe and easier to fix. And finding problems in real-time gives GAIN the opportunity to correct it before their clients notice. If the issue is with a partner who can't fix it on the spot, Gain can adjust and route around that market. 

StreamBase LiveView is the first platform of its kind to provide a complete analytics solution for live data and real-time action, which we believe could be a new disruptive physics that allows companies to broaden the decisions they make by looking ahead, not only behind. 

So, what decisions would YOU make differently if you stopped looking only in the rear-view mirror?

 

Chasing an Ambulance Full of BATS

The high-profile trading failure at BATS this week sparked a rash of negative publicity, gracing the front page of media outlets from the Wall Street Journal to Forbes. In the software industry, more software companies have proclaimed their techology would have sheltered BATS from their storm than cheap umbrella sidewalk vendors on a rainy day in New York City.

An especially troubling rash of "thought leadership" came from my own industry - real-time analytics and complex event processing. Forbes, for example, published an article by SAP (who sells a CEP product) BATS hits the rocks, which may have lifted its sales that proclaims that the BATS incident portends that "CEP technology is about to move from the backwaters of data management technology to center stage."

In Wall Street and Technology, John Bates, the CTO of Progress Software was quoted in the artice Flash Crash at BATS renews Market Concerns to say firms "clearly lacked adequate risk controls such as real-time surveillance and monitoring. Hopefully this incident serves as a wake up call." What kind of software do you think Bates sells?  Drum roll please... real-time surveillence and monitoring!"

My company sells these kinds of solutions too. And I know BATS well (disclosure: BATS is not a StreamBase client), and they are as technologically advanced as any firm in the world. So this kind of ambulance chasing is not only unbecoming, it's also dead wrong. It's like suggesting Formula One racing will be safer if the cars can be made to go faster (CEP) and we do a better job of filming them as they go careening into a wall (monitoring). Indeed, BATS quickly reported that they had a bug in their system, so comments about risk management and surveillance are irrelevent.

That said, there's an issue here for the capital markets - but the issue is cultural, not technological. 

What's happening in the capital markets is that we are in the midst of real-time automation renaissance. 10 years ago, less than 5% of stock trading in the U.S. markets was done algorithmically; today, as much as 70% is done on computer-implemented, and human designed, algorithms.

Larry Tabb, the founder and chief executive of the Tabb Group, said in the Times that “The markets basically gutted their high-cost, nonstop infrastructures for very fast, low-cost infrastructures." Now we are talking - Larry is on the right path!

In software, mistakes happen frequently. Professor Neil Johnson at the University of Miami analyzed 18,520 "ultrafast Black Swan [trading] events" from 2005-2011. In Stock Market Flaws Not So Rare, the New York Times that just one of the 13 U.S. stock exchanges - NASDAQ - declared 139 breakdowns at other exchanges last year alone, up from 69 the year before.

Software breaks. We have a word for it: bugs. But that's not new. What is new is that the errors can do damage so much more quickly, and, because the world is now so hyperconnected, everyone knows about it more quickly than ever before. 

So what is the magic elixir to making the capital markets fast, fair, and safe?  

There are still voices out there that seem to think that the answer is to discontinue the use of computers for automation. Ha! The whining about the evils of capitalism and sophisticated technologies continues to rain down from dissenters posting their admonations from inside their pup tents via shiny new iPad 3's. These dissenters need to Get Over It - technology is here to stay, and its use is expanding quickly from Wall Street to Main Street.

Making automation safe and reliable requires complex, cultural change:

1. Continue to refine reporting requirements. You can't manage what you can't measure. In the United States, exchanges didn't report outages on other exchanges until 2007. Nasdaq reports that there have been 110 outages in the last year like the one that triggered the BATS mishap. Continued measurement and transparency will lead the industry toward greater reliability. With more analysis and reporting, more accountability is possible, and regulators could focus on correlating events among exchanges and brokerages to look for systemic flaws.

2. Refine and lighten regulation; don't just pile it on. Dodd Frank and MIFID 2 have put regulation on the must-do agenda of all firms. But it's still largely up to the reader to separate what's essential from what's fluff, and that's not effective. Exchanges like BATS are extrodinarily regulated - so more, heavier regulation is not the answer.

3. More systemic attention to software quality.  Moore's Law - the idea that the power of computing doubles approximately every two years, does not apply to software quality. The BATS crash shines a light on the issue that is inbred in innovation; the rising importance of software testing, quality, and simulation. Most firms have ad-hoc processes for quality and should have continuous testing, regression testing, and real world simulation. CEP can make systems go faster, and surveillence helps spot problems more quicky, but when more attention is paid to testing and simulation, less errors occur. 

These changes are complex and cultural, not technological. Tools will help, but not solve, the challenge of ensuring a safe, reliable, and fair market.

 

b-next Selects StreamBase: Regulation Goes Real-Time

Today's announcement by StreamBase partner b-next is part of steady shift toward real-time in the capital markets. b-next, a pioneering provider of multi-venue, multi- asset class capital markets surveillance and compliance solutions, announced their selection of StreamBase for their CMC:Suite solution set. By incorporating event processing, b-next's customers, such as regulators, exchanges and banks, will be able to rapidly implement near real-time market abuse surveillance solutions. This will enable relevant market participants to comply with recent ESMA guidelines and future proof themselves against additional regulation expected in the coming years.

Although event processing has often been associated with sub millisecond decision making and algorithmic trading, trading was indeed, just the tip of the iceberg. Pre-trade risk management followed, and now the entire infrastructure of the capital markets applying automation technologies like event processing. b-next’s systems is further evidence of that shift, as the CMC:Suite solution helps reduce compliance officers' response period by helping them immediately identify suspicious trading patterns and generating alerts. All in real-time.

"We're one flash crash away from being required to understand our risk in real-time," the head of electronic trading at a top-tier firm told me recently. His firm has deployed event processing globally to manage their trade flow, risk, and positions. After years of being deployed in the heart of electronic trading systems, recently the technology has been increasingly used to manage the same automated infrastructure its often used to power. 

We welcome our newest partner b-next to the StreamBase family!

About b-next

b-next is a provider of multi-venue, multi-asset class, Capital Markets Surveillance and Compliance software solutions to manage risk, meet regulatory mandates and drive trading efficiencies. The b-next Capital Markets Compliance (CMC) solution offers  a single integrated compliance platform for Market Abuse, Insider Trading, Conflict of Interest, Derivatives/OTC Monitoring, Best Execution, Fraud Detection and Prevention, AML and more. With over 150 customers, b-next has a 20 year history of delivering solutions to market participants, exchanges and capital markets regulators.

Santiago Stock Exchange Selects StreamBase

Too busy to keep up on the news! Last week, the Santiago Stock Exchange (SSE) (Bolsa de Comercio de Santiago) announced a strategic partnership with StreamBase Systems to provide algorithmic trading tools for the Chilean market. StreamBase CEP will be integrated into SSE's electronic trading platform, Telepregon, offering local brokers advanced order execution algorithms.

Time to market and ease of use, again, were the key reasons for the SSE's choice of StreamBase: "We chose StreamBase as a partner for their rapid developer environment StreamBase Studio in addition to their extensive connectivity options," said Andrés Araya Falcone, CIO at the Santiago Stock Exchange. "StreamBase's graphical development environment allows us to easily provide software as a service to local brokers and other market participants for building their own algorithms. The Chilean market is still at an early stage of tremendous growth, and it is critical to us as an exchange to bring in the best technology available to our local market."

Danielle Tierney, Senior Analyst at AITE Group commented, is one of the leading analysts about the capital markets in South America. She was quoted as saying this about the news: "As the region's third largest market after Mexico and Brazil, Chile is an important Latin American market with a history of strong performance based on solid fundamentals. The Santiago Stock Exchange is also making significant progress in the development of its trading infrastructure to facilitate the expansion of investment opportunities. SSE's participation in the MILA initiative and its efforts to support growth in electronic trading are only the beginning of many promising things to come."

Lots more big customer announcements to come in 2012! Stay tuned!

Introducing Live BI and StreamBase LiveView

For some time now, StreamBase has been working on a new product concept that's a little bit CEP, a little bit data warehouse, and a little bit business intelligence. Today we've unveiled it as StreamBase LiveView. Our customers have called it a "game changer," and a product that "fills a massive hole in the BI space." Hyperbole like that needs to be backed up, so here we go.

Complex event processing (introduction on Google+) was envisioned by its inventors as a "database for moving data" as firms increasingly deal with data in motion from mobile devices, social media and digitized data of all types, and in all industries. In its first decade of commercial use, CEP has come to power automated trading on Wall Street. A who's who list of firms have been public about their adoption of CEP, so let's examine the live BI problem in the context of Wall Street.

Industry analysts estimate that over 60% of U.S. equities trading is automated. The problem is, with so much automation, it's impossible to see and act on what's going on, because a typical trading environment can generate 10's of millions of events a day. Existing business intelligence and data warehouse tools aren't used in the front office of Wall Street, because they're designed to summarize what's already happened, not what's happening now. 

StreamBase LiveView brings BI and streaming data together, for the first time. Applications that need Live BI are everywhere on Wall Street - trading risk management, position management, P&L management, surveillance, compliance, and trading itself, as Waters Technology magazine describes in the article that StreamBase Seeks 'Radical Transparancy' in Business Intelligence.

And applications in other markets can be found in any environment that has big data in motion: telecom (location aware insight and live network operations), security (live fraud detection, live surveillance), and e-commerce (live social media analytics, ad placement, big streaming analytics for Hadoop and Flume). These apps can be deployed on premise or distributed in the cloud. 

StreamBase LiveView is built with StreamBase CEP, which provides connectivity to over 125 event-based sources of data and the extreme scalability required by live BI. The server-based architecture is designed to process hundreds of millions of events per node, absorbed from a network of event sources such as an enterprise message bus or scalable event absorption layer like Flume or IBM Streams. At the front end, users can slice and dice streams of data with ad-hoc query tools in the StreamBase LiveView Desktop (shown above). But here's the key: once users ask the questions they want answered, StreamBase LiveView keeps answering their questions - as conditions change in real-time; users see an always-live view of streaming data.

For the user, all this technology means just one thing: they see business intelligence that's constantly fresh and relevant to their business operations. It's not a view of 5 days ago, or even 5 minutes ago. The data is live, the insight is live, and live insight can lead to live action.

Facebook, Twitter, Google+ give us a live view of what's happening in our social networks; StreamBase LiveVew gives firms a live view of their enterprise data.

To learn more, watch the StreamBase LiveView presentation, follow new materials, articles, and resources on Google+ (+StreamBase and CEO +Mark Palmer), read industry analyst Curt Monash, DMBS2.0, or follow StreamBase Systems (@streambase), CEO Mark Palmer (@mrkwpalmer ), and CTO Richard Tibbetts (@tibbets) on Twitter.

 

StreamBase News at SIFMA

The SIFMA show in New York is always a big week for news for firms that do business in the capital markets. This year, we had so much new news  that we issued a series of announcements over the last several weeks. In case you missed any of them, here's a summary of what we're talking about at SIFMA this year.

1. Thomson Reuters Selects StreamBase CEP. This is a huge endorsement from a leader in the capital markets. Thomson Reuters will utilize StreamBase in their open, global, ultra-high speed and resilient network for the financial industry, Elektron, as an integrated element in their Enterprise Platform for Velocity Analytics, as an embedded CEP engine in Thomson Reuters applications, and in their Thomson Reuters Dealing Aggregator FX Trading Solution. We're very honored to be selected by Thomson Reuters!

2. BMF&BOVESPA Selects StreamBase CEP. The largest exchange operator in South America selected StreamBase as their enterprise-wide deployment platform for event-based applications - yet another huge endorsement and evidence of the increased use of CEP for enterprise-wide deployments. 

3. StreamBase in Asia: Complex Event Processing in Trading with SunGard. Our partner SunGard recently held a online panel discussion about algorithmic trading, market data management, risk management, and other other applications in Hong Kong with Benjamin Becar, algorithmic trading product management for SunGard in Hong Kong, and Richard Tibbetts, StreamBase CTO. You can watch the replay of the session here.

4. New product release: StreamBase 7. Fantastic new usability, and performance improvements in four different dimensions.

5. StreamBase University Program. Our new program to provide StreamBase software and certification resources to universities to help students learn about the growing field of CEP in school. The StreamBase University Program is integrated with our StreamBase Developer Certification Program and StreamBase Component Exchange community for sharing StreamBase integrations, code examples, frameworks, and adapters. 

6. Erjang / Erlang integration and the growing StreamBase Component Exchange Community. The latest contribution from Darach Ennis (@darachennis) has been a big hit in the Erlang community and on SlideShare. Watch it here: Complex Erjang / Erlang Processing with @StreamBase. 

7. CEP in Risk Management and Surveillance. CEP isn't just for algo trading anymore! Broad adoption by firms like Thomson Reuters and BMF&BOVESPA, and increasing use in applications such as risk management and surveillance illustrate that CEP is becoming a strategic platform for firms on Wall Street, and for third party developed applications with "CEP Inside."

8. "Faster than a Flash" - MIT Technology Review on CEP for market data risk management. MIT Technology Review published a story on how proprietary trading firm PhaseCapital used CEP to have ZERO bad trades on the day of the flash crash. While other pundits promote fear of a looming "splash crash" - bigger and worse than the "flash crash," we're simply trying to promote firms that are actually doing something about it with innovative technology approaches to make trading safer.

I'll be at the show in New York this week and addressing questions on Twitter about these developments and others at @mrkwpalmer.

5 Trends Revealed by BM&FBOVESPA's Selection of CEP

Today StreamBase announced that BM&FBOVESPA, the largest exchange operator in Latin America, has selected CEP (for an introduction to complex event processing watch this short video) as their enterprise-wide deployment platform for event-based applications.

Big, public declarations like this one beg the simple question: "what's going on here?" I think BM&FBOVESPA's decision to use CEP points to 5 key trends:

1. Brazil continues their development as an economic power and an innovation power. The Brazilian economy has been surging in past years, and BM&FBOVESPA's use of CEP further reflects the innovation in this increasingly important market. According to analyst AITE group, high frequency trading is up 52% since last November (read their report on the Brazilian market Brazil: The Best of the BRIC?). CEP will be used to build innovative applications that will help the markets operate more efficiently, and more safely. While the U.S. struggles with increased regulation, the conditions in Brazil continue to become more attractive in terms of liquidity and fairness for worldwide investors. 

2. Visual CEP enables "Smart Jobs." NPR and Wired Magazine fascinating study this month on areas where the economy is doing well found that "Smart Jobs" are they key to current and future growth and will continue to grow in importance for years to come. The Smart Jobs on Wall Street are jobs where traditional IT AND business analysts work together to innovate by building smarter trading infrastructure. Marcio Castro, Chief Technology Officer of BM&FBOVESPA, said it this way: “The development of capital markets in Latin America is experiencing the fastest growing phase in decades. Innovation is the key for us to develop a liquid, low-latency and fair marketplace for our worldwide investors. [StreamBase was selected to help enable this innovation].” To learn more about how the StreamBase visual programming environment, StreamBase Studio, enables Smart Jobs, watch "Build a CEP App in 46 Seconds." 

3. CEP is an increasingly strategic piece of the enterprise IT landscape. BM&FBOVESPA's commitment to CEP isn't for an algo trading app or two - that was the first generation of CEP use in Brazil. Bovespa shows that CEP is becoming an essential platform for applications of all types, from front to back office. CEP is growing up and increasingly critical in the most critical IT environments.

4. IT infrastructure continues to become more event-based. Traditional IT architecture utilize a static computing model; they store a bunch of data, and allow humans to query that data to make decisions. Modern IT architecture is event-based: repetitive decisions are automated - decisions don't have to be left to the end of the day, or end of the month. In trading, tasks that used to be performed at the end of the day, like risk management, are now performed during the day - in real-time.  BM&FBovespa's move shows that the static platforms of the past are being augmented by the event-based platforms of the future.

5. StreamBase is taking Brazil by storm. OK, I have to brag just a little! It doesn't get much bigger than an endorsement by a global leader like BM&FBOVESPA. Other CEP companies moved first in Brazil, but when customers look for the leader in CEP, they turn to StreamBase. 

Fascinating things are going on in Brazil - now, with the largest exchange in Latin America featuring CEP to power innovation, new innovation lies ahead - stay tuned!

Fast is Good, But Fast and Dumb is Dangerous

Last week I was on a panel at the Bloomberg Enterprise Technology Summit (@BBGlink) with Steve Goldman from CME Group (@CMEGroup), the CTO of NYSE Technologies, Bloomberg TV Markets Reporter Dominic Chu (@The_Domino) and we talked about how speed is important, but that the real innovation going on today is about making applications smarter, not just faster. The conversation was constructive and productive and was re-tweeted all over the internet. 

Then, this weekend, Barron's article "Next Danger: 'Splash Crash'" featured an interview with Progress Software's CTO Dr. John Bates. Barron's cites his view that high frequency trading poses an "extreme risk" of creating a "splash crash," his own term, which he describes as a worse than the flash crash: "trading systems could clog up, limited bandwidth could choke orders, exchanges could freeze up -- splashing across all of the affected asset classes. Pandemonium."

Bates "frets that, absent more oversight, terrorists wielding smart machines could attack the markets in an attempt to cripple our economy,"  and has indeed been fretting publicly for a while now in interviews blog posts, such as:

• "The trouble with algorithms: wild children or reckless parents?"
• "Beware the splash crash"
• "Red flags in morning, firms take warning"
• "When does a rogue trader become a scoundrel?"

It's unhelpful to focus on the fear; we should focus on solutions. Every automated application - from kiosk-based airport check-in to security in a retail store - needs surveillance. Trading is no different. This is not new. What's new is that trading occurs at speeds never before seen, and when things go wrong, it's easy to blame the bankers of nefarious activity.

So today's article in MIT Technology Review, "Faster than a Flash," was a refreshing break from provocative predictions of doom and gloom. MIT features Boston-based proprietary trading firm PhaseCapital (disclosure: PhaseCapital is a StreamBase customer). CEO Eric Pritchett described to MIT reporter Erica Naone that his algorithmically-driven trading firm made NO erroneous trades on the day of the flash crash. 

How did Phase Capital use algorithms and provide a safe trading harbor to their investors on the day of the flash crash? They use the same technology to analyze their market data in real-time that they use to trade in real-time: complex event processing, or CEP (intro here).

Corwin Yu, director of electronic trading for PhaseCapital, explained their innovation this way:

Typically, PhaseCapital processes 30,000 to 40,000 'ticks,' or pieces of market data, per second. During the flash crash, that number jumped to more than 289,000 ticks per second— much of it representing stocks swinging wildly. "A lot of scrubs kicked in and realized that this data didn't make sense," Yu says. For example, some large publicly held companies, such as Accenture, traded for less than $1 during the flash crash. PhaseCapital continued trading, but filtered data that appeared suspicious. Because it didn't act on that data, it was spared erroneous trades.

Yu concluded that speed is essential in the world of trading today, but that the trick is to be "fast and smart." Read about PhaseCapital's innovation in MIT Technology Review.

The solution for a safer market isn't heavy-handed regulation of "wild, reckless algorithms" in order to catch the "scoundrels" of Wall Street. Fast does good on Wall Street; but fast and dumb is dangerous. So we should tune out the fear mongers and tune in to the innovators that can help make Wall Street be fast - and smart.

Thomson Reuters Selects StreamBase CEP for Elektron High-Speed Cloud, Quant Platform, Apps, and FX

Today we're excited to report that Thomson Reuters has selected StreamBase as its complex event processing (CEP) platform for a broad set of initiatives. CEP "flips the database on its head" by providing the ability to monitor, analyze, and act on events happening in real-time, rather than looking in the rear-view-mirror of what happened yesterday, last week, or last month. CEP powers real-time, big data applications on Wall Street including high frequency trading, pre-trade risk management, surveillance, market data management, smart order routing, pricing, settlement, real-time P&L, monitoring - applications from front to back office on Wall Street.

Why did Thomson Reuters partner with StreamBase for CEP? Initially, they announced four reasons: 

StreamBase in Elektron Brings CEP to Thomson Reuters' High-Speed Cloud

The StreamBase platform lies at the heart of a new generation of real-time, automated applications on Wall Street. As the use of CEP continues to grow, our customers increasingly ask to deploy some of their applications in the cloud. They want their automated processing to happen as close to the exchange as possible for the lowest possible latency, and they want secure and reliable access to their market data.

Thomson Reuters announced that they will provide StreamBase CEP in Elektron, their open, global, ultra-high speed and resilient network for the financial industry. By adding StreamBase CEP to Elektron, Thomson Reuters is creating the industry’s only platform that includes comprehensive real-time and historical data, high capacity network connectivity, managed hosting services, and CEP for rapid application development and strategy deployment.

The First End-to-End Quant Platform for Back Testing, Algorithmic Development and Deployment

Although StreamBase is all about acting on what's happening now, the knowledge of what happened yesterday is essential for any automated system. History is essential to discover trends and patterns in event streams. History is essential to back test automated strategies. History is essential to gain perspective on real-time actions at execution time. Over 90% of applications that use StreamBase also use some form of historical event data management.

Thomson Reuters will integrate StreamBase with the Enterprise Platform for Velocity Analytics, which integrates all Thomson Reuters content, including tick history, reference data, corporate actions, condition code processing and cancellation and correction messages. It also brings together many disparate third party applications including data management solutions, EMS, OMS, price discovery, back testing, trade surveillance and low latency execution capabilities across the quantitative research and trading workflow.

By integrating Velocity Analytics with StreamBase CEP, Thomson Reuters has created the first end-to-end quant platform for back testing, algorithmic development and deployment. 

Thomson Reuters Applications "Powered By" StreamBase

Thomson Reuters will deliver a suite of applications that are “powered by” StreamBase CEP to address low-latency workflow. CEP inside will provide Thomson Reuters clients the benefit of time to market usually associated with turnkey solutions, while at the same time allowing full customizability of the underlying application. One application Thomson Reuters already provides is the real-time surveillance application from Redkite financial which is based on the StreamBase CEP platform. 

StreamBase and Thomson Reuters FX Solutions

The FX markets have rapidly emerged as an important market for the capabilities of CEP. Thomson Reuters announced that StreamBase CEP will complement the existing Thomson Reuters Dealing Aggregator FX trading solution, available exclusively to Thomson Reuters Dealing customers, by providing CEP capability in tandem with the low latency algo trading platform. 

Why did Thomson Reuters choose StreamBase? Mike Powell, Global Head of Enterprise Real-Time Solutions for Thomson Reuters, said it this way: “The financial market landscape continues to evolve rapidly, meaning that nimbleness and time to market have become essential for market participants. Elektron empowers our customers to innovate and achieve high performance through a seamless infrastructure that brings together global market participants, delivers world-class content and creates a suite of new added-value services. The combination of StreamBase’s market leading CEP engine together with Elektron will provide a platform for us to deliver flexible and sophisticated solutions to our customers faster and more efficiently.”

For StreamBase, we’re excited about the vast resources and capabilities of Thomson Reuters can provide our customers. We’re honored to be selected as Thomson Reuters’ CEP platform of choice - their selection validates our focused approach to CEP, the ever increasing importance of the cloud and real-time big data, the disruption of simplicity, and the continued need for low latency and scalability on Wall Street.