The StreamBase Event Processing Blog

Gain Capital: A Glimpse into the Future of Analytics?

Imagine trying to drive your car while ONLY looking in the rear-view mirror? If you're like most businesses, you make your decisions by looking at reports, which is like driving while only looking in the rear-view mirror.

First generation analytics and BI tools - Excel, Tableau, Qliktech - provide a map of where you've been. They're so essential that they are a $10 billion software market. But using old data to drive your business only tells you what potholes you already hit and what turns you could have made.

StreamBase LiveView provides a live dashboard of where you should go next. It analyzes live data; not old data. LiveView absorbs data streams with just milliseconds of latency - sensor data, high-speed market data, client orders, streaming news, tweets, FIX messages, network messages, emails, log data. LiveView gives you a live view of data - before everyone else sees the report tomorrow!

Live insight helps you see potholes before you run over them, and decide where to turn next. For a more complete overview of LiveView, watch the 4-minute intro presentation here. 

Today, GAIN Capital went public about their use of LiveView and how they use it to steer their business differently. Rob Perez, VP of Trading and Risk Management Systems at GAIN Capital, shared a terrific example of the transformation live analytics can bring. GAIN's business is trading, and success is linked to the quality of connectivity to the electronic markets. Gain services over 140 countries, with an average monthly transaction volume of over $275 billion. 

Previously, GAIN managed their trading operations with traditional tools - databases, log files and business intelligence. At the end of the day or week, reports are generated, problems are discovered, and addressed.

Now, LiveView helps GAIN understand and optimize their trading operations in real-time. Perez said in EuromoneyFXNews: "We an see if any metrics are moving outside of our comfort zone, and we can immediately reach out to our liquidity providers to understand what's going on."

At any time, GAIN clients can trade on any of 10 or more markets. It's essential to GAIN that problems with prices, connectivity, or quality are elimintated. Now, GAIN can now determine:

• What deals are being rejected now?
• Which liquidity providers are giving good prices now?
• Which liquidity providers are having problems now?

Live insight helps GAIN decide to call a poorly performing partner the moment there's a problem. Finding a problem as it's happening makes it easier to describe and easier to fix. And finding problems in real-time gives GAIN the opportunity to correct it before their clients notice. If the issue is with a partner who can't fix it on the spot, Gain can adjust and route around that market. 

StreamBase LiveView is the first platform of its kind to provide a complete analytics solution for live data and real-time action, which we believe could be a new disruptive physics that allows companies to broaden the decisions they make by looking ahead, not only behind. 

So, what decisions would YOU make differently if you stopped looking only in the rear-view mirror?

 

Three Stories from StreamBase LiveView Customers

A year and a half in the making, we're thrilled to release StreamBase LiveView 1.0 (press release). Since we released the concept video last November (which drew almost 4,000 views on YouTube), our early access program has been a huge success. We'll announce an exciting set of customers in the coming weeks from that program.  

What's all the excitement about? While the technology is awesome (download this great StreamBase LiveView Technical Paper by CTO Richard Tibbets, Steve Yang, Rob MacNeil, and Dave Rydzewski), it's really all about the value our customers get from the stuff we build.

Here were three "aha" moments I had with those visionary adopters that, for me, crystallize why LiveView is so special.

1. Real-Time Service Experience Management. Recently the head of trading at a big bank told me that when a big buy side client like Fidelity calls to ask about the performance of a trade, they:

• Issue a request to IT
• IT searches log files, databases, and reports to gather raw data
• IT analyzes the data with one or more of: a quant, a risk expert, an execution expert, and a market structure expert
• The team writes a report
• The team briefs the sales trader on the account
• Finally, after 3-5 DAYS - they explain the situation to the client
• They decide on a course of action
• Adjustments are made - algorithms or strategies are changed

No more. Now, LiveView helps them make many of these decisions in seconds. LiveView creates a live, 360 degree view of order flow, executions, risk, transaction cost, risk, and operations. LiveView lets sales staff see what's happening before the data is lost in an IT black hole of log files, databases, and reports. 

Now, the moment the client calls, a salesman, risk manager, or trade execution specialist can answer if orders are being received correctly? Is the risk profile in line? Are the trades in line with the cost benchmarks? Are my 5 EMSs up and running? Are connections to the exchanges fast? Does the client want to adjust their strategies? 

The transformation is exciting: a typical reactive support function becomes proactive, opportunities lead to bigger opportunities, and problems become sales opportunities.

2. Real-Time Selling. In January, at a meeting at another bank, a sales trader asked: "Could I ask LiveView in the morning to alert me whenever a big order comes in from an important client using the new algorithm we released last week so I can bird-dog it?"

This was the perfect question, because it strikes at the heart of what LiveView does: it's like setting hundreds or thousands of business probes, and being alerted proactively when one is tripped. The probes can be turned on, turned off, and adjusted on the fly. Any system, internal or external, packaged app or custom built.

Firms that provide a highly automated service to clients need a real-time intelligence system for sales people. The ability to sell more depends on the ability to anticipate that a key client is headed for trouble, or when the time is right to encourage more sales.

And by building real-time intelligence into the system, everyone becomes a sales person, because the conditions that encourage more sales are right in their face.

3. Real-Time Fraud Detection. My last story took 3 months to unfold. Another LiveView early adopter wanted to implement a "fraud detection" analytic with LiveView. The firm needs to monitor machine-driven algorithms, and look for suspicious patterns. Sounds simple, right? Not so simple!

In this case, the patterns of interest required correlation of 5 streams of real-time data. Patterns happen within 15-30 second windows, during which thousands of dollars could be lost. Attacks come in bursts.

The data required to find these patterns was loaded into a data warehouse and reports were checked each day. Decisions to act were made every day.

LiveView now intercepts the data before it hit the warehouse by connecting LiveView to the source of data. It took 3 days to integrate these sources because it took that long to find someone who knew where 3 of the data streams came from!

StreamBase detects fraud patterns in milliseconds. But the really interesting part came next.

Once this firm could see patterns of fraud, they were faced with a new challenge: what to DO about it? How many times did the pattern need to be repeated until active surveillance is started?  Should the action be quarantined for a period, or halted immediately? All these questions were new, and the answers to them keeps changing.

The fact that the answers keep changing highlights the importance of ease of use. Analytics must be changed quicky and be made available to fraud experts - in some cases, in hours - as understanding deepens, and as the bad guys change their tactics.

Better, higher value-add customer service for highly automated industries. Knowledge workers who anticipate sales opportunities. Spotting fraud in high-speed transactions streams and taking action. These stories are the stories of the future for the real-time enterprise. These stories tell the story of why StreamBase LiveView is so special.

Chasing an Ambulance Full of BATS

The high-profile trading failure at BATS this week sparked a rash of negative publicity, gracing the front page of media outlets from the Wall Street Journal to Forbes. In the software industry, more software companies have proclaimed their techology would have sheltered BATS from their storm than cheap umbrella sidewalk vendors on a rainy day in New York City.

An especially troubling rash of "thought leadership" came from my own industry - real-time analytics and complex event processing. Forbes, for example, published an article by SAP (who sells a CEP product) BATS hits the rocks, which may have lifted its sales that proclaims that the BATS incident portends that "CEP technology is about to move from the backwaters of data management technology to center stage."

In Wall Street and Technology, John Bates, the CTO of Progress Software was quoted in the artice Flash Crash at BATS renews Market Concerns to say firms "clearly lacked adequate risk controls such as real-time surveillance and monitoring. Hopefully this incident serves as a wake up call." What kind of software do you think Bates sells?  Drum roll please... real-time surveillence and monitoring!"

My company sells these kinds of solutions too. And I know BATS well (disclosure: BATS is not a StreamBase client), and they are as technologically advanced as any firm in the world. So this kind of ambulance chasing is not only unbecoming, it's also dead wrong. It's like suggesting Formula One racing will be safer if the cars can be made to go faster (CEP) and we do a better job of filming them as they go careening into a wall (monitoring). Indeed, BATS quickly reported that they had a bug in their system, so comments about risk management and surveillance are irrelevent.

That said, there's an issue here for the capital markets - but the issue is cultural, not technological. 

What's happening in the capital markets is that we are in the midst of real-time automation renaissance. 10 years ago, less than 5% of stock trading in the U.S. markets was done algorithmically; today, as much as 70% is done on computer-implemented, and human designed, algorithms.

Larry Tabb, the founder and chief executive of the Tabb Group, said in the Times that “The markets basically gutted their high-cost, nonstop infrastructures for very fast, low-cost infrastructures." Now we are talking - Larry is on the right path!

In software, mistakes happen frequently. Professor Neil Johnson at the University of Miami analyzed 18,520 "ultrafast Black Swan [trading] events" from 2005-2011. In Stock Market Flaws Not So Rare, the New York Times that just one of the 13 U.S. stock exchanges - NASDAQ - declared 139 breakdowns at other exchanges last year alone, up from 69 the year before.

Software breaks. We have a word for it: bugs. But that's not new. What is new is that the errors can do damage so much more quickly, and, because the world is now so hyperconnected, everyone knows about it more quickly than ever before. 

So what is the magic elixir to making the capital markets fast, fair, and safe?  

There are still voices out there that seem to think that the answer is to discontinue the use of computers for automation. Ha! The whining about the evils of capitalism and sophisticated technologies continues to rain down from dissenters posting their admonations from inside their pup tents via shiny new iPad 3's. These dissenters need to Get Over It - technology is here to stay, and its use is expanding quickly from Wall Street to Main Street.

Making automation safe and reliable requires complex, cultural change:

1. Continue to refine reporting requirements. You can't manage what you can't measure. In the United States, exchanges didn't report outages on other exchanges until 2007. Nasdaq reports that there have been 110 outages in the last year like the one that triggered the BATS mishap. Continued measurement and transparency will lead the industry toward greater reliability. With more analysis and reporting, more accountability is possible, and regulators could focus on correlating events among exchanges and brokerages to look for systemic flaws.

2. Refine and lighten regulation; don't just pile it on. Dodd Frank and MIFID 2 have put regulation on the must-do agenda of all firms. But it's still largely up to the reader to separate what's essential from what's fluff, and that's not effective. Exchanges like BATS are extrodinarily regulated - so more, heavier regulation is not the answer.

3. More systemic attention to software quality.  Moore's Law - the idea that the power of computing doubles approximately every two years, does not apply to software quality. The BATS crash shines a light on the issue that is inbred in innovation; the rising importance of software testing, quality, and simulation. Most firms have ad-hoc processes for quality and should have continuous testing, regression testing, and real world simulation. CEP can make systems go faster, and surveillence helps spot problems more quicky, but when more attention is paid to testing and simulation, less errors occur. 

These changes are complex and cultural, not technological. Tools will help, but not solve, the challenge of ensuring a safe, reliable, and fair market.

 

Tech Q&A with CEP Developers, and StreamBase

Today StreamBase hosted an online roundtable entitled “How to Empower Your Development Team with CEP and Visual Programming” featuring developers from J.P. Morgan, GAIN Capital, and Datasports. Combined, the panelists have 35+ years of development experience, and years of CEP experience, and they were kind enough to share those experiences with us, discussing the adoption of CEP technology, developing applications with CEP, and the best practices they’ve learned along the way.

For developers, development managers, architects, and business stakeholders this roundtable tells an interesting story about how CEP technology can truly transform the role of the developer. Take a look at some of the highlights below, watch the replay, then comment below and tell us about your experiences with CEP. 

Talk about your experience with CEP

Phil Martin, Datasports: I like using StreamBase because I like tools that let the computers do stuff that the computers are good at, and let me do stuff that requires thought. As a developer, you wind up doing some tasks over and over again; the structure of the application, domain specific plumbing, etc. Everytime feels like wasted effort. StreamBase lets me focus on the stuff that requires thought, not the details of implementation, connectivity, processing events, and so on. Another comment is around the aggregate operator. It is analogous to another API to learn. It’s an incredibly rich and flexible operator that I think almost qualifies as its own sub language or sub paradigm.  There’s a lot there and a lot of subtlety in its behavior, learn about and study it!

Mike Grandilone, J.P. Morgan Asset Management: The shift to a message passing structure is sometimes a shift in thinking. It's key to understanding how StreamBase handles the life of a tuple; we gained that through experience.  

Talk about the applications you've built with StreamBase

Joseph Chua, GAIN Capital: After 4 days of training and going through the examples, I built my first StreamBase application in just three hours, and it was fully deployed in a week. It connects to market data, manipulates the data, and stores it in a high-speed database. We've also built market making and real-time surveillence applications, and many more to come.

Phil: I've built several applications with StreamBase. One is an "indication of interest" (IOI) app that monitors trading interest and compares it against a CRM system and sends alerts to individual sales traders because they have a customer who either has traded on the symbol, transacted above a given threshold, or has a real-time order for it. The correlation of these factors in real-time also includes real-time news sources. Also, I've worked on two algorithmic trading systems that host their strategies in StreamBase and perform real-time risk checks on the algos, and an ETF arbitrage system where there was only one strategy, but it was a very involved strategy.

Mike: We brought a smart order routing system up and running in 6 months with 2 developers, which we thought was very fast. We also got the business people involved in the canvas and the development - this is a very important factor in getting applications done quickly.

Talk about the benefits of StreamBase and visual programming

Phil: StreamBase helps facilitate communication. Whenever you want to convey information about a system, you always draw a picture. And they almost always look like StreamBase Event Flow. If you have taken the necessary care to clearly delineate modules, then, depending on the technical sophistication of who you are talking to, you can choose how far to drill down. It's extremely helpful in faciliating technical discourse; with the business users, it saves time.

Mike: Visual development has really improved the speed of our development. Especially when you're doing a proof of concept. It helps you understand the application's logic, and also communicate with the business; in typical programming environments like Java, you're not going to show many people the code, but you can do it with StreamBase.

Joseph: The big value is that all the technical details are abstracted away in terms of connectivity and low-level code; we can then communicate with the business owners, so development cycles are shorter and more efficient. The other value we have found is they we can very quickly build a prototype solution so the business can make decisions more quickly with respect to requirements and decisions to develop the applications.

Phil: One thing that's helpful: with StreamBase, you can tell at a glance that spaghetti code is spaghetti code. You can tell if an application was thoughtfully designed without hours and hours and hours of code review.

What best practices have you learned from your experiences with CEP?

Mike: You need to shift your thinking, to think in terms of events, rather than traditional static applications. And, just like any application, build a testing environment at the same time as your application logic.

Joseph: Identify the connection points quickly, be open minded and willing to unlearn the more traditional ways of development, use StreamBase interfaces, and make sure you continuously refactor the application.

Phil: There are lot of resources for ramping up StreamBase skills - invest the effort to develop your skills. The speed of putting together something quickly is a great asset. The thing to keep in mind is that although the development speed of StreamBase is a benefit in the short term, it can, if you're not careful, work against you in the long term.

Phil: While the principles aren't different, the practice IS different. The first thing that sold me is seeing the ease with which you can change and tune your threading model, take things in and out of the threading part of your system, with multiplicity and dispatch methods. Scalability and performance is about taking more advantage of the cores in a system, and careful management of it becomes more and more important.

What was your experience integrating other code into CEP?

Mike: We have separate routing logic written in Java for routing logic that we integrated with our StreamBase application. We've done a lot of work with custom Java operators in StreamBase - it's very straight forward - no problem in doing it. There's lots of power in developing in Java for code that isn't event driven that allowed us to bring in the power of algorithmic trading as a drop-in to the StreamBase environment. Also, we have a C# front end GUI, which uses the .NET interface from StreamBase - this was easy, although the challenge was how to take a GUI that isn't event-based and make it work with an event-driven application. We're in production with that now.

The StreamBase community

Phil: I've put two components onto the StreamBase Component Exchange - mostly adapters. The publishing process was straightforward. One is an adapter that watches for events on the file system and creates StreamBase events when files are created, renamed, modified, or deleted, and another one listens for calls on a web service interface and creates events for the applciation. I suggest other developers look at the utility classes – see what are other developers are doing, and pick up some tips and techniques and helpful hands-on examples of how they are building their applications. 

StreamBase Resources for Developers

Below are the StreamBase resources mentioned in this roundtable:

• Trial version of StreamBase: www.streambase.com/download
• Product documentation: docs.streambase.com
• Personalized demonstrations: www.streambase.com/forms/request-a-demonstration
• “How To” developer webinars: www.streambase.com/webinars
• StreamBase Component Exchange: www.streambase.com/sbx
• StreamBase University: www.streambase.com/sbu
• StreamBase Training and Certification: www.streambase.com/community/training-and-certification
• Datasports blog for StreamBase CEP developers: http://datasports.ca

b-next Selects StreamBase: Regulation Goes Real-Time

Today's announcement by StreamBase partner b-next is part of steady shift toward real-time in the capital markets. b-next, a pioneering provider of multi-venue, multi- asset class capital markets surveillance and compliance solutions, announced their selection of StreamBase for their CMC:Suite solution set. By incorporating event processing, b-next's customers, such as regulators, exchanges and banks, will be able to rapidly implement near real-time market abuse surveillance solutions. This will enable relevant market participants to comply with recent ESMA guidelines and future proof themselves against additional regulation expected in the coming years.

Although event processing has often been associated with sub millisecond decision making and algorithmic trading, trading was indeed, just the tip of the iceberg. Pre-trade risk management followed, and now the entire infrastructure of the capital markets applying automation technologies like event processing. b-next’s systems is further evidence of that shift, as the CMC:Suite solution helps reduce compliance officers' response period by helping them immediately identify suspicious trading patterns and generating alerts. All in real-time.

"We're one flash crash away from being required to understand our risk in real-time," the head of electronic trading at a top-tier firm told me recently. His firm has deployed event processing globally to manage their trade flow, risk, and positions. After years of being deployed in the heart of electronic trading systems, recently the technology has been increasingly used to manage the same automated infrastructure its often used to power. 

We welcome our newest partner b-next to the StreamBase family!

About b-next

b-next is a provider of multi-venue, multi-asset class, Capital Markets Surveillance and Compliance software solutions to manage risk, meet regulatory mandates and drive trading efficiencies. The b-next Capital Markets Compliance (CMC) solution offers  a single integrated compliance platform for Market Abuse, Insider Trading, Conflict of Interest, Derivatives/OTC Monitoring, Best Execution, Fraud Detection and Prevention, AML and more. With over 150 customers, b-next has a 20 year history of delivering solutions to market participants, exchanges and capital markets regulators.

Santiago Stock Exchange Selects StreamBase

Too busy to keep up on the news! Last week, the Santiago Stock Exchange (SSE) (Bolsa de Comercio de Santiago) announced a strategic partnership with StreamBase Systems to provide algorithmic trading tools for the Chilean market. StreamBase CEP will be integrated into SSE's electronic trading platform, Telepregon, offering local brokers advanced order execution algorithms.

Time to market and ease of use, again, were the key reasons for the SSE's choice of StreamBase: "We chose StreamBase as a partner for their rapid developer environment StreamBase Studio in addition to their extensive connectivity options," said Andrés Araya Falcone, CIO at the Santiago Stock Exchange. "StreamBase's graphical development environment allows us to easily provide software as a service to local brokers and other market participants for building their own algorithms. The Chilean market is still at an early stage of tremendous growth, and it is critical to us as an exchange to bring in the best technology available to our local market."

Danielle Tierney, Senior Analyst at AITE Group commented, is one of the leading analysts about the capital markets in South America. She was quoted as saying this about the news: "As the region's third largest market after Mexico and Brazil, Chile is an important Latin American market with a history of strong performance based on solid fundamentals. The Santiago Stock Exchange is also making significant progress in the development of its trading infrastructure to facilitate the expansion of investment opportunities. SSE's participation in the MILA initiative and its efforts to support growth in electronic trading are only the beginning of many promising things to come."

Lots more big customer announcements to come in 2012! Stay tuned!

Introducing Live BI and StreamBase LiveView

For some time now, StreamBase has been working on a new product concept that's a little bit CEP, a little bit data warehouse, and a little bit business intelligence. Today we've unveiled it as StreamBase LiveView. Our customers have called it a "game changer," and a product that "fills a massive hole in the BI space." Hyperbole like that needs to be backed up, so here we go.

Complex event processing (introduction on Google+) was envisioned by its inventors as a "database for moving data" as firms increasingly deal with data in motion from mobile devices, social media and digitized data of all types, and in all industries. In its first decade of commercial use, CEP has come to power automated trading on Wall Street. A who's who list of firms have been public about their adoption of CEP, so let's examine the live BI problem in the context of Wall Street.

Industry analysts estimate that over 60% of U.S. equities trading is automated. The problem is, with so much automation, it's impossible to see and act on what's going on, because a typical trading environment can generate 10's of millions of events a day. Existing business intelligence and data warehouse tools aren't used in the front office of Wall Street, because they're designed to summarize what's already happened, not what's happening now. 

StreamBase LiveView brings BI and streaming data together, for the first time. Applications that need Live BI are everywhere on Wall Street - trading risk management, position management, P&L management, surveillance, compliance, and trading itself, as Waters Technology magazine describes in the article that StreamBase Seeks 'Radical Transparancy' in Business Intelligence.

And applications in other markets can be found in any environment that has big data in motion: telecom (location aware insight and live network operations), security (live fraud detection, live surveillance), and e-commerce (live social media analytics, ad placement, big streaming analytics for Hadoop and Flume). These apps can be deployed on premise or distributed in the cloud. 

StreamBase LiveView is built with StreamBase CEP, which provides connectivity to over 125 event-based sources of data and the extreme scalability required by live BI. The server-based architecture is designed to process hundreds of millions of events per node, absorbed from a network of event sources such as an enterprise message bus or scalable event absorption layer like Flume or IBM Streams. At the front end, users can slice and dice streams of data with ad-hoc query tools in the StreamBase LiveView Desktop (shown above). But here's the key: once users ask the questions they want answered, StreamBase LiveView keeps answering their questions - as conditions change in real-time; users see an always-live view of streaming data.

For the user, all this technology means just one thing: they see business intelligence that's constantly fresh and relevant to their business operations. It's not a view of 5 days ago, or even 5 minutes ago. The data is live, the insight is live, and live insight can lead to live action.

Facebook, Twitter, Google+ give us a live view of what's happening in our social networks; StreamBase LiveVew gives firms a live view of their enterprise data.

To learn more, watch the StreamBase LiveView presentation, follow new materials, articles, and resources on Google+ (+StreamBase and CEO +Mark Palmer), read industry analyst Curt Monash, DMBS2.0, or follow StreamBase Systems (@streambase), CEO Mark Palmer (@mrkwpalmer ), and CTO Richard Tibbetts (@tibbets) on Twitter.

 

Trading on Macroeconomic News

Last Thursday, we hosted a webinar – “Trading on Macroeconomic News” with Need to Know News (NTKN), a subsidiary of Deutsche Bourse Group Company. With pre-built connectivity to NTKN’s AlphaFlash, we demonstrated how a trading firm could easily incorporate low-latency news algo data feeds into their trading systems. In the 50-minute webinar, we built a basic algo that will trade EUR/USD based on the data of non-farm payroll employment report. 

Trading on unstructured or non-traditional data is a topic that is gaining more and more attention. Adam Honoré (@adhonore) at Aite Group reported that 35% of quant firms are now exploring machine-readable news feeds in some way, compared to only 2% in 2008. The development of electronic trading and explosion in data volumes over the past decade play key roles in this phenomenon. Yet, most market participants are still trying to figure out effective and efficient ways of transforming unstructured data into increased profits or prevented losses.

One fund, Derwent Capital has reported success already, announcing a 2% profit from its $40 million dollar fund, with investment strategies based on Twitter mood - derived from a paper published by Johan Bollen and Huina Mao at Indiana University, Bloomington, and Xiao-Jun Zeng at the University of Manchester. In 2009, Zhi Da and Pengjie Gao at Notre Dame and Joseph Engelberg of UNC also proposed correlation between Google Search Volume Index (SIV) and stock prices.

Here at StreamBase, we were one of the first financial technology firms to offer clients a way to connect to unstructured social media content, such as Twitter. Media interest in this area has been high, with coverage from CNBC, the FT, the WSJ, and USA Today; though enabling customers to trading on social media, is really only a small percentage of what we do. The scalability and flexibility of StreamBase CEP makes it a great platform to handle unstructured data in real-time.

Leading economic indicators, such as GDP, consumer confidence index, interest rates and unemployment reports are commonly used for signal generation but a firm needs to make extra efforts to really gain a competitive edge with unstructured data. Simply categorizing news into positive, neutral or negative might not be sufficient enough for effective algorithmic trading. In 2010 we demonstrated how StreamBase can be used with a LingPipe, a natural language processing tool kit that can understand the implications of a news story vs. market expectation.

Although it is rare to see signal generation or alpha seeking strategies solely based on news events, some sophisticated firms incorporate unstructured data and sentiment analysis to develop algos that react to market volatility after a particular news item breaks. 

Processing of unstructured data has relevance not only for alpha-generating strategies but also for risk managers and regulators who could use machine-readable news and sentiment analysis to simulate a firms’ exposure or market fairness under extreme events, for example, the recent downgrade of government credit ratings.

Trading on unstructured data will continue to be a popular topic. If you have any new ideas for using other non-traditional data for trading, please let us know. Below are some additional links which might prove interesting.

Further reading: 

Using Twitter to Predict Stock Market Swings, New York Times

Twitter Mood Predicts The Stock Market, Technology Review

Hedge Fund Will Track Twitter to Predict Stock Moves, Bloomberg

Which Stocks Will Rise? Ask Google, SmartMoney

In Search of Attention, Zhi Da, Joseph Engelberg and Pengjie Gao

Computers That Trade on the News, The New York Times

On IBM, Unstructured Data, and CEP, StreamBase Event Processing Blog

Twitter for those dirty, alpha-seeking strategies, FT Alphaville

The Future of FX Trading Technology - Indicators from the 2011 StreamBase FX Survey

This year's third annual StreamBase survey on technology trends in the FX trading market had some interesting findings. The survey was conducted from March to May 2011, drawing 135 FX industry respondents, 55% sell-side and 36% buy side.  43% of the respondents use algorithms for trading, and of those 64% actually use their own in-house developed algorithms (that is, they don't use algorithms supplied by a vendor or bank). The survey was designed with input from the TABB Group and AITE Group. Here are the three main areas that leapt out to me as the CEO of StreamBase - many of our customers using CEP as part of their FX technology platform. 

1) Increased Regulation is the Leading Fear, and Real-Time Risk Management for FX Will Soon Emerge

52% of the respondents said they believe regulation will have the biggest impact on their business, and 37% of buy-side firms expect to see an increased call for market transparency as a result of recent lawsuits regarding FX transaction costs. But what are the implications of this expectation of increasing regulation?

In the past 10 years, the FX markets have lagged the equities market in terms of technology adoption. In many cases, innovation spreads from equities groups to other asset class such as FX - In the past 18 months, pre-trade risk management has become a leading initiative for equities trading groups – but will it soon become common for FX trading? Sang Lee, cofounder and managing partner of Aite Group, thinks so: "it's critical that firms manage their risk in real-time," said Sang, in reaction to our survey results.

2) FX Liquidity Aggregation is Becoming Increasingly Complicated

94% of the survey respondents said they are doing some form of FX liquidity aggregation, up from 65% last year. This finding was initially surprising to me; but we see a lot of innovation in FX with CEP, and plenty of firms are adding new aggregation functionality with more connectivity to banks as well as the ECNs. This is undoubtedly leading to more sophisticated real-time aggregation of FX liquidity sources.  A skeptic might say, “that sort of result is to be expected from a CEP vendor”, yet less than 10% of the survey respondents were actually StreamBase users, so we believe that this is reflective of an industry trend, not our sample.

What’s more is that 34% of firms are planning to improve or add aggregation functionality - this suggests continued innovation in aggregation applications and platforms – such as demands for new FX sources, direct bank connectivity, and new features to be coming from FX venues.  So, keep watching the FX space for more sophisticated aggregation technology and techniques.

3) It's Better to be (a Little) Slower and Right than Fast and Wrong

The need for speed is always paramount in trading, but this year the survey revealed that speed is no longer job #1 in FX trading. #1 in FX, is the ongoing struggle with the complexity of integrating and normalizing FX data formats, and #2 is accessing low-latency sources. The votes for #1 and #2 were totaled over half of all respondents.

This finding is consistent with what a head of FX trading recently told me, that "it's better to be a little slower and right than fast and wrong." The point he was making wasn't that it's OK to be slow, only that getting smarter algorithms deployed given the complexity of pricing and trading FX is just as important, if not more important, than being the fastest.

For more details about the 2011 FX Survey, read the press release, download the survey results, watch the online roundtable, and watch for coverage in the media this week.

Real-Time Big Data and the 11 Principles of Modern Surveillance Systems

(This article is also posted in the TABB Group's TABBForum, and generated a broad discussion in the comments there. To participate, visit the original post here.)

In his TabbFORUM article, Dave Tolladay eloquently argues that real-time surveillance is crucial in today's high-frequency trading landscape, and I agree. Ten years ago, real-time computing began to influence trading; today, HFT is pervasive and existing surveillance techniques, designed to run at the speed of click trading, should be considered to be legacy architectures in need of a radically new, modern approach. 

But managing big data in real-time is a new problem of physics, and those physics come center stage when it comes to surveillance. A recent panel on modern surveillance systems, Miranda Mizen from TABB Group, Justin Amos from Redkite Financial Systems and Richard Tibbetts from StreamBase compared modern surveillance systems to “anti-virus software for trading.” 

In the course of their conversation, Justin, Miranda and Richard touched on 11 key points about the “next generation” architectural requirements for surveillance, including: 

1. Real-time analytics. Surveillance systems must analyze market data at the speed of automated trading. That means real-time computing and complex event processing (CEP) as one of the critical technology enablers of modern surveillance systems. 

2. Historical “big data” analytics. Surveillance systems must put activity in context – suspicious activity can be an aberration or a pattern, based on historical actions. The big data movement has spurred a renaissance in the database industry with firms like Netezza (now IBM), Vertica (now HP), MongoDB (open source) as the flag bearers for the NoSQL (Not Only SQL) database revolution that aims to supplant legacy RDBMS systems to deal with the rising deluge of data described in the Economist piece. In surveillance, “big data analytics” allow on-line access to trade-by-trade history for investigation, trending, and discovery to be mashed up with real-time data to provide a real-time and historical context to behavior. 

3. Inclusion of new types of data sources. Justin suggested that the inclusion of real-time news – and even social media data sources – can be a benefit in correlating, for example, suspicious trading on news. 

4. Data fusion. Real-time, historical and new data sources must be fused for a complete picture of intra-day trading behavior, and a 360 degree view of market data, activity and sentiment. 

5. Multi-asset class support. Since HFT is becoming increasingly cross-asset class, so must modern surveillance systems. 

6. Pre-built algorithms. Todayʼs surveillance systems must support standard algorithms like front running, spiking, tailgating, and ramping that can mistakenly or purposefully occur in computer-based HFT systems. 

7. Adaptive algorithms. The ability to adapt algorithms on the fly to reduce false positives is important, due to the increasingly dynamic capability of modern HFT systems to adapt their behavior. 

8. Algorithm extensibility. The ability to quickly add new algorithms to the system as market dynamics evolve is also a key modern requirement for systems – they must keep pace with the rate of change of modern trading algorithms. 

9. Flexible workflow management and case management. Surveillance systems generally donʼt make automated decisions to change trading behavior or stop trading on their own; instead, they tend to guide a series of workflow steps to alert compliance officers, portfolio managers or traders and manage the process of alert analysis, resolution and tracking. 

10. Accessibility for the business analyst. With regulation changing so quickly, more firms want their own control over surveillance behavior, so some provide tools that allow compliance personnel to discover and implement surveillance algorithms on their own. 

11. Cloud-based or on-premise deployment. Surveillance is not typically the core expertise of trading groups, so the ability to deploy systems either in the cloud or on premise allows for more rapid and global rollout of surveillance tools. 

Real-time big data is a change in computing physics on Wall Street. New requirements motivate new solutions, and the new solutions and innovation coming from the surveillance community continues to prove that innovation never sleeps on Wall Street.