The StreamBase Event Processing Blog

Tech Q&A with CEP Developers, and StreamBase

Today StreamBase hosted an online roundtable entitled “How to Empower Your Development Team with CEP and Visual Programming” featuring developers from J.P. Morgan, GAIN Capital, and Datasports. Combined, the panelists have 35+ years of development experience, and years of CEP experience, and they were kind enough to share those experiences with us, discussing the adoption of CEP technology, developing applications with CEP, and the best practices they’ve learned along the way.

For developers, development managers, architects, and business stakeholders this roundtable tells an interesting story about how CEP technology can truly transform the role of the developer. Take a look at some of the highlights below, watch the replay, then comment below and tell us about your experiences with CEP. 

Talk about your experience with CEP

Phil Martin, Datasports: I like using StreamBase because I like tools that let the computers do stuff that the computers are good at, and let me do stuff that requires thought. As a developer, you wind up doing some tasks over and over again; the structure of the application, domain specific plumbing, etc. Everytime feels like wasted effort. StreamBase lets me focus on the stuff that requires thought, not the details of implementation, connectivity, processing events, and so on. Another comment is around the aggregate operator. It is analogous to another API to learn. It’s an incredibly rich and flexible operator that I think almost qualifies as its own sub language or sub paradigm.  There’s a lot there and a lot of subtlety in its behavior, learn about and study it!

Mike Grandilone, J.P. Morgan Asset Management: The shift to a message passing structure is sometimes a shift in thinking. It's key to understanding how StreamBase handles the life of a tuple; we gained that through experience.  

Talk about the applications you've built with StreamBase

Joseph Chua, GAIN Capital: After 4 days of training and going through the examples, I built my first StreamBase application in just three hours, and it was fully deployed in a week. It connects to market data, manipulates the data, and stores it in a high-speed database. We've also built market making and real-time surveillence applications, and many more to come.

Phil: I've built several applications with StreamBase. One is an "indication of interest" (IOI) app that monitors trading interest and compares it against a CRM system and sends alerts to individual sales traders because they have a customer who either has traded on the symbol, transacted above a given threshold, or has a real-time order for it. The correlation of these factors in real-time also includes real-time news sources. Also, I've worked on two algorithmic trading systems that host their strategies in StreamBase and perform real-time risk checks on the algos, and an ETF arbitrage system where there was only one strategy, but it was a very involved strategy.

Mike: We brought a smart order routing system up and running in 6 months with 2 developers, which we thought was very fast. We also got the business people involved in the canvas and the development - this is a very important factor in getting applications done quickly.

Talk about the benefits of StreamBase and visual programming

Phil: StreamBase helps facilitate communication. Whenever you want to convey information about a system, you always draw a picture. And they almost always look like StreamBase Event Flow. If you have taken the necessary care to clearly delineate modules, then, depending on the technical sophistication of who you are talking to, you can choose how far to drill down. It's extremely helpful in faciliating technical discourse; with the business users, it saves time.

Mike: Visual development has really improved the speed of our development. Especially when you're doing a proof of concept. It helps you understand the application's logic, and also communicate with the business; in typical programming environments like Java, you're not going to show many people the code, but you can do it with StreamBase.

Joseph: The big value is that all the technical details are abstracted away in terms of connectivity and low-level code; we can then communicate with the business owners, so development cycles are shorter and more efficient. The other value we have found is they we can very quickly build a prototype solution so the business can make decisions more quickly with respect to requirements and decisions to develop the applications.

Phil: One thing that's helpful: with StreamBase, you can tell at a glance that spaghetti code is spaghetti code. You can tell if an application was thoughtfully designed without hours and hours and hours of code review.

What best practices have you learned from your experiences with CEP?

Mike: You need to shift your thinking, to think in terms of events, rather than traditional static applications. And, just like any application, build a testing environment at the same time as your application logic.

Joseph: Identify the connection points quickly, be open minded and willing to unlearn the more traditional ways of development, use StreamBase interfaces, and make sure you continuously refactor the application.

Phil: There are lot of resources for ramping up StreamBase skills - invest the effort to develop your skills. The speed of putting together something quickly is a great asset. The thing to keep in mind is that although the development speed of StreamBase is a benefit in the short term, it can, if you're not careful, work against you in the long term.

Phil: While the principles aren't different, the practice IS different. The first thing that sold me is seeing the ease with which you can change and tune your threading model, take things in and out of the threading part of your system, with multiplicity and dispatch methods. Scalability and performance is about taking more advantage of the cores in a system, and careful management of it becomes more and more important.

What was your experience integrating other code into CEP?

Mike: We have separate routing logic written in Java for routing logic that we integrated with our StreamBase application. We've done a lot of work with custom Java operators in StreamBase - it's very straight forward - no problem in doing it. There's lots of power in developing in Java for code that isn't event driven that allowed us to bring in the power of algorithmic trading as a drop-in to the StreamBase environment. Also, we have a C# front end GUI, which uses the .NET interface from StreamBase - this was easy, although the challenge was how to take a GUI that isn't event-based and make it work with an event-driven application. We're in production with that now.

The StreamBase community

Phil: I've put two components onto the StreamBase Component Exchange - mostly adapters. The publishing process was straightforward. One is an adapter that watches for events on the file system and creates StreamBase events when files are created, renamed, modified, or deleted, and another one listens for calls on a web service interface and creates events for the applciation. I suggest other developers look at the utility classes – see what are other developers are doing, and pick up some tips and techniques and helpful hands-on examples of how they are building their applications. 

StreamBase Resources for Developers

Below are the StreamBase resources mentioned in this roundtable:

• Trial version of StreamBase: www.streambase.com/download
• Product documentation: docs.streambase.com
• Personalized demonstrations: www.streambase.com/forms/request-a-demonstration
• “How To” developer webinars: www.streambase.com/webinars
• StreamBase Component Exchange: www.streambase.com/sbx
• StreamBase University: www.streambase.com/sbu
• StreamBase Training and Certification: www.streambase.com/community/training-and-certification
• Datasports blog for StreamBase CEP developers: http://datasports.ca

StreamBase Loves Erlang

The philosophy of Erlang and StreamBase align quite well. Quoting Mike Williams, one of the three inventors of Erlang:

• Find the right methods—Design by Prototyping
• It is not good enough to have ideas, you must also be able to implement them and know they work
• Make mistakes on a small scale, not in a production project

These philosophies of rapid development also apply to the way we've designed StreamBase; additionally, our customers also demand that much less technical users participate in the rapid implementation of ideas. StreamBase has a visual development approach that allows less technical, more "business" users - with mathematics, analytics, and business domain expertise - use the same tools as professional software developers.

What if we could add Erlang developers to that party? Event-driven, visual development, coupled with the power and rapid development of Erlang? 

Well you can. Darach Ennis (@darachennis) shows us how in his recent presentation at Erlang UG in London. 

StreamBase has a graphical, compiler-oriented programming language, targeted at serious developers, that can also be used by the less technical. Erlang programmers, and the Erjang folks with whom we share a virtual machine, do their thing in Erlang, and that code becomes a graphical "operator" on the StreamBase Studio pallet. In Darach's example he's used Erlang to implement a MACD analytic, and integrate that into a StreamBase-based trading system so it operates in the event-driven CEP application. 

The combination of StreamBase and Erlang combines the best of both worlds: StreamBase for rapid, graphical development to develop low latency applications that process 100's of thousands of events a second. Combining "how," "why," and "under what conditions," real time streams are matched and processed. StreamBase allows business analysts, actuaries, quantitative contributors and developers to share a common language. An event-driven integration whiteboard, if you will, that compiles to efficient optimized native binary code. Add Erlang for development of analytics like MACD and you have a powerful combination. 

Erlang support will be contributed soon to the StreamBase Component Exchange (SBX), our public "app exchange" for CEP integrations, applications, frameworks, and sample code. You'll already find support for Java, C++, R, Matlab, python, .NET. If you want to plug in some cool analytics written in C++, just do it. You want to exploit Basho's Riak? Sure, plug it in.

As a result, StreamBase allows developers control over how they exploit the facilities at their disposal. Use our graphical DSL for flow oriented algorithms and real time low latency high frequency data management. It's high performance from a human productivity and systemic perspective. 

Learn more by watching Darach's presentation here, and follow him on Twitter at @darachennis.

 

The Flash Crash & the Role of Financial Software Engineering

Even though the flash crash happened seven months ago, the debate rages on. And since CEP is a common piece of our modern financial computing fabric many firms have entered the debate with vested interest. Unfortunately, most vendors wrap flash crash observations with veiled sales pitches for CEP-based risk management, or disingenuous calls to "catch the rogues and scoundrels" with CEP-based surveillance systems. Although these applications are important, to suggest they are a magical elixir to ward off future breakdowns is misleading at best. 

But thoughtful discourse does continue, and the recent Securities and Exchange Commission report has reinvigorated debate. Richard Tibbetts, StreamBase CTO, weighed in with a piece called "What Does the Flash Crash Mean For Financial Software Engineering?" on Tabb Forum this week. Richard's view on the flash crash examines the role of financial software engineering and the efficacy of data processing, integrity, and reliability within the fabric of the infrastructure. The role of financial engineering is an essential, overlooked, and misunderstood piece of the puzzle. Richard's thoughts go like this: 

"In all the conversation about the crash, the real historic change is going unmentioned. In the very last paragraph of the Lessons Learned section of the report, the authors note: “another area of focus going forward should be on the integrity and reliability of market centers’ data processes, especially those that involve the publication of trades and quotes to the consolidated market data feeds.”

This is a key insight – late in coming – that will expand over time. Traditionally the SEC has focused on regulating the behavior of market participants, identifying deliberately malicious behavior or patterns of behavior. But now they are beginning to realize that data processing, integrity, and reliability can be just as important. The efficient functioning of markets is in just as much danger from software system failures as it is from malicious traders..."

The role of financial engineering is much broader than CEP, and deserves deeper discourse. To read more, continue reading Richard's piece on Tabb Forum (which requires a free registration) here, and watch for more over the coming months on Tabb Forum, on this blog, and on Richard's blog on software engineering and innovation, Hyperextended Metaphor.

Gain Capital's Selection of CEP and "Whiteboard Programming"

In the late 90's, visual programming was The Next Big Thing. Instead of writing millions and millions of lines of incomprehensible computer instructions, visual programming tools promised a world where you could "paint" applications using high-level objects. Visual programming promised a future where applications would be built more quickly, more easily, with result in an application that would be less expensive to maintain. Unfortunately that era of visual programming did not deliver on many of it's promises. Tools were primitive, workstations weren't up to the task, many potential users didn't know why they would want to write programs, and the languages didn't line up well with the use cases. In the end, the tool with the most staying power was UML, which is a modeling language rather than a programming language, and which used traditional textual programming languages to create the "real" program. UML became fodder for big-project architecture and requirements documents, not a tool for real programmers.

Last week I was reminded that I'm fortunate to be running a company that delivers on the promise of visual programming. This struck me as I was listening to press interviews with Andrew Haines, the CTO of Gain Capital. Andrew discussed the reasons behind their selection of StreamBase. He talked about how they had evaluated all the major CEP vendors and selected StreamBase for its speed, ease of use, and our being a good partner. And he talked about how strategic their use is - Gain processes almost $200 billion in trades a month. All good stuff.

But the comment that struck me was when Andrew said Gain has found StreamBase Event Flow programming language to be 50% faster than traditional programming techniques (citing an experiment they did that measured StreamBase versus .NET). This comment reminded me of the 90's and the promise of visual programming, or what we sometimes call "Whiteboard Programming." Whiteboard programming (watch a short video here) is the right visual metaphor for event processing because developers design event-driven applications naturally on a whiteboard, with data flows, icons for business logic, and arcs that connect logic to next part of their event driven architecture. StreamBase's visual development tools allow developers to "paint" their application. But the visual representation of their whiteboard isn't just a picture of the application, it is the application. The application is tested, profiled, and debugged visually, and the promise of visual programming, for event-based applications, is fulfilled.

StreamBase EventFlow benefits from several improvements in the state of the art since the 90s, as well as our own innovation. First, it's built into a modern IDE based on the Eclipse platform, already a powerful and flexible tool before StreamBase starts with it. StreamBase also takes many of its ideas from the agile programming methodology developed over the last 15 years, including automated refactoring, unit testing, and a focus on quick iterations. More specific to StreamBase, the StreamSQL Event Flow language derives from declarative data processing languages developed at MIT, Brown, Brandeis, and Stanford universities, with rigorous models behind them. The language is designed to not only be visual, but also fundamentally to be directly executable (unlike UML) and to have powerful facilities for extensibility and abstraction (unlike end-user oriented visual languages). The result is a language that is accessible to financial engineers, quantitative analysts, and business analysts, but powerful enough that application and infrastructure developers are comfortable working on the same platform. Despite the ability to add functionality to the platform in many traditional languages (Java, C++, .NET, Python, textual SQL), StreamBase developers universally adopt the visual programming model for it's productivity and teamwork benefits.

Andrew's comment reminded me why StreamBase is on a run of amazing accolades as one of the most innovative technologies in the world, including Time Magazine, the World Economic Forum Technology Pioneer program, and MIT Technology Review. These esteemed technologies realize that, as Clayton Christensen suggests in The Innovator’s Dilemma, simplicity is always a disruptive innovation.  And the proof lies in clients like Gain Capital, who get to market 50% faster, with 50% less cost, with 50% less cost of ownership over time. 

But at the same time, other CEP venders aren't gaining customer success stories like Gain Capital. Why not? Gain evaluated numerous other CEP products from Progress Software, Aleri, and TIBCO as reported by FX Week. Coral8 was acquired by Aleri, then Aleri was acquired by Sybase. After the acquisition, Aleri engineers revealed that although their product was fast, customer implementations took years and lots of consulting to build. Apama, the early pioneer in CEP, tried to develop a visual programming tool after they developed a textual language. The textual language works, but the visual one doesn't, which is why Apama's visual tool is being scrapped as they integrate the CEP language with Savvion's visual BPM development environment.

What Gain Capital's selection of StreamBase reminds us is that simplicity is always disruptive, and that the promise of visual programming is just as relevant today as it was 10 years ago, and being fulfilled by the leading CEP companies.

Q&A with CMC Markets: Trading $1.4 Trillion on CEP

This week CMC Markets announced that they have deployed complex event processing (CEP) in their global trading infrastructure that processes $1.4 trillion a year on over 5,000 trading instruments and over 115,000 trades a day. 

You might be thinking - $1.4 trillion in trading? Who is this CMC Markets? CMC is the leading global provider of retail contracts for difference (CFDs) and one of the largest financial spread betting leaders. CMC is a sophisticated, distributed organization - according to Inside Market Data, "the StreamBase platform is being rolled out across CMC's global operations, including offices in Australia, Canada, China, Germany and the UK, and is supporting a wide range of applications including its market-making, risk management, and hedging tools to support trading in FX, indices, futures, commodities, and equities."

This kind of global, mission critical commitment to CEP is important news, so we spoke with CMC’s Chief Information Officer, Asif Adatia, to find out more about CMC, their needs, why they chose CEP, their business results, and their future plans with CEP.

Q1: The AITE Group report from July, 2007 said your average trade volume was about $100 billion monthly.  Is that correct now?

That’s an old number. Now we’re at about $200 billion a month or $10 billion a day.  For example, if I look at today’s stats we’ve done $7.5 billion now at trading shops.

Q2: Have your trading volumes increased a lot over the last couple of years?

I think all of our stats have increased.  We’ve got more clients, more trades per day, which explains some of the re-architecture work we’re doing.  Yeah, we’re doing very well at the moment.  This business fundamentally is doing very well from a key metrics perspective.

Q3: Right.  Can I ask, were your upgrade plans hampered at all this last year by the recession or have you been able to carry on?

Peter Cruddas, our CEO, gave us massive support from a technology perspective.  If anything I’ve got more to do. We put a plan together, the board bought into it and it’s pretty exciting.  For me it’s phenomenal, we’ve been able to do everything we wanted to do.  This re-architecture is an effort to take our business to another level.

Q4: So there hasn’t been anything that had to be put on hold because of the crisis?

No, definitely not from a technology perspective, no.  If anything, the question is “can I do more and get it done faster.”  I’ve certainly not had restrictions.

Q5: This re-architecture is replacing your existing in-house built architecture.  What other technology upgrades are you making?

We’ve got a whole style change from our front end and our transactional engines to our back office.  There’s a program over the coming months where we are upgrading all of our systems.

Today, StreamBase is doing pricing, and it’s far more sophisticated than our legacy market maker platform we previously had.  Ultimately, StreamBase will replace a lot of the intelligence behind the old system.

Q6: Can you give me a couple of examples?  And over what period you will conclude?

So the back office as an example, we started to work on that. We’ve got a batch orientated back office currently, and we’re in the process of creating a real time P&L and cash balances being realized in P&L.  Today you wait until the end of the day, you mark the market and you adjust your P&L’s and you realize your P&L then. We’re moving a real time back office.  The timing is probably end of year. And then again because it’s so different, there will be a coordinated effect with our marketing and our client service guys.  Because it is a big difference from the way we have been working. That’s just one example of many.

Q7: Can you describe the project StreamBase is supporting?

StreamBase supports FX for all of our major currencies, exotics, as well as forwards.  It will support indices futures, futures on indices, cash commodities, treasuries and equities, single stock.

Q8: Can you describe how the StreamBase application processes data and pricing for CMC?

We cleanse our prices as they come through. We get multiple feeds from various brokers.  We cleanse our prices and we’ve got a reasonably sophisticated model where we look at groupings - if there are multiple feeds coming in and you want to take the best price, you want to take out the ones that are outliers. So the mathematics behind it are sophisticated - it doesn’t just do a standard mean, it takes a look at standard deviation of all of these prices and then looks at the volatility over multi-factor, high frequency scenarios. Then we start to look at which ones are outliers.

As I said StreamBase helps us look at what’s happening and very quickly take algorithms in and out - fix them and put them back in as well.

Q9: What are your FX market data sources?

I can’t give you the names but it’s primarily the usual suspects.  Tier one brokers provide liquidity and pricing to firms like ourselves.  We’ve got quite a few brokers that we deal with and their price feeds. We’ve also got feeds for other asset classes.  And along with the brokers, we have multiple data feeds from vendors, exchanges, alternative ECNs and OTC price providers

Q10: Are you using the StreamBase adapters to connect directly to those price feeds?

Yes, it’s actually is a combination.  StreamBase provides feed handlers.  So very quickly we were able to connect and use the StreamBase feed handlers. And then we’re in the process with some of our brokers, again, StreamBase allows us to connect to our own feed handlers so we do that.  So as I said at the beginning flexibility was one of the key criteria.  StreamBase offers that to us as well.  So we can mix and match.

Q11: Are you live? 

Yes, we’re live now.

Q12: You mentioned scalability and performance - what kind of demands are you putting on the StreamBase system now?  I’m thinking updates, latency, that kind of thing?

So at the moment we’re able to process 17,000 ticks per second on FX.  It doesn’t really hurt the system or the processes that much.  And we’ve got it scaled up to for twice as large as that. Can it handle our peak? Our peak to date has been nowhere close to 17,000 at the moment.  We’ve got what we call a non-functional regression test environment where we just overload it.  We overload our environment and haven’t broken it.  We've got so much going on right now that if it can cope with double our peak, we can focus on other things.

Q13: So what kind of a difference will users notice?  What’s going to be different for them? 

If you look at FX pricing live right now our spread is pretty much in line and competitive.  We used to have an issue back in before we implemented some of this.  We used to have issues with latency.  Those problems have disappeared now.  We’re able to process a lot more - about 17,000 ticks a second, which is pretty cool. We’ve still got plenty of headroom as well.  And we’re also now taking multiple feeds in.

From a usability and user experience, clients will just get better pricing, no re-quotes as we roll this out across all of our asset classes.  Re-quotes will disappear, type of spreads, better spreads.  We’re able to handle volatile markets - Fed announcements; all of that kind of stuff is no longer an issue for us. We’re probably one of the few retail players out there that if you’ll notice during any highly volatility period, CMC markets are usually there providing two way quotes.  So that, I think, is the upside for clients that our infrastructure is solid in times of high volatility.

Q14: Can you describe two way pricing?  It’s one of your differentiators right?

The ability to quote a two way price is not the issue - the ability to quote and allow you to trade at that price is the difficult part and our systems can do that - especially for FX, as we add more asset classes to our new infrastructure we should be able to do the same for all asset classes, hence the urgency to get everything done yesterday.

Q15: So what makes your pricing better?  Is it just the lower latency?

I think it’s a number of things.  Yeah, it’s latency, it’s flow. It’s lower latency, robust feeds, and having multiple feeds coming in.  The result is to offer the retail client almost institutional pricing, which is unique.

Q16: Can you put your latency improvements in numeric terms?

We’re able to handle 17,000 ticks per second from multiple sources.  Previously we looked at probably 3000 ticks per second.  This doesn’t make a big difference during normal trading days, but when volumes start to pick up - and this is only one asset class - we’re able to process this number of ticks and still be able to provide a two way price, a clean price you’re able to trade on in milliseconds.  That’s the upside.  Market data is a big issue for all the vendors - even the big players out there.  I won’t mention any names but during summer last year you will notice that there were massive delays in all of the larger players out there with respect to providing prices to clients in general.  And, again, we believe that now those kinds of delays have disappeared from our architecture.  So can I quantify the number of ticks per second?  We can handle the latency that we’ve got, yes, but it’s an improving target.  And one where, because of the StreamBase multi threaded architecture, we can run it across multiple CPU’s now.  If we ever get a bottleneck we just plug in another blade, pull up another service, and we get better latency and better pricing.

Q17: So the scalability is obviously great. Do you have any metrics you want to offer that quantify the latency you’re experiencing?

No, we don’t give them out. Speed is a strategic advantage for us, so we think those numbers are competitive information.

That’s all right, fair enough.

It’s good though and you can compare us on FX pricing to a bunch of our competitors.  And you will see our performance is noticeably better at times. Absolutely.

Q18: Okay.  You mentioned initially in the beginning the broader investment in your architecture.  Is that correct?

Yeah, we are in the process of rethinking a lot of our systems and our core systems at the moment.  This is just part of that.

Q19: What other applications will you use StreamBase for?

Yeah, the flexibility of the platform is the key thing for us.  There are lots of ideas - for example, surveillance.  Again, the concept behind surveillance is that you want to process a large number of trades and constantly be able to look at patterns. From what we know of StreamBase, it could lend itself well to that.  But, again, we’re going through pricing first then risk - the classics.  We’ll do the classics first and then all the other added value things we may want, because we have the infrastructure in place now.

Q20: And when you say when you look into the future you might use it for surveillance purposes, will that be looking at clients?  Or is that more for looking at potential opportunities that you might have been missing, or is it a bit of both?

It’s a bit of both actually.  Surveillance is a broad area. One of the things we do is look for people behaving badly. So you want to be able to protect the firm from people doing financial fraud.  You want to be able to stop fraud before it happens because you want to be able to flag that in real time.  StreamBase will be capable of that as well.  So there’s a number of projects, again, that we would look at after we’ve done the basics.

What makes this possible is that because the StreamBase infrastructure is there, the tools are there, the adapters are there to feed the platform. Then it’s a matter of just gluing it together.

Q21: Right, okay.  Very interesting.  Are you going to then be incorporating this into your market maker platform?  That’s the trading platforms that you guys have.

Yes.

Q22: So this is deployed across your entire enterprise?

Yes, it is global deployment.

Q23: How long did it take you to deploy the system?

6 months.

Q24: What were your requirements when you were looking for a solution and why did you choose StreamBase? 

We wanted two things: a scalable solution, and a flexible solution.  We’ve got a bunch of financial engineers that want to be able to prototype or even test different models and scenarios.  And we wanted that to be as easy as possible, so ease of use was a big part of the criteria. 

Q25: So you selected StreamBase CEP over other CEP products for its simplicity of development, performance, and scalability.  Is it working?

Yes - the flexibility of StreamBase is key to us - so much so that we’re finding new uses for it.  We originally brought it in for pricing and risk, and now that we understand the concept and some of the things it can do, it should facilitate other projects as well.

Q26: Can you talk about performance - was that about latency or capacity management?

With performance it was a combination of things.  The number of ticks, as you are probably well aware, of market data, isn’t going down.  So it’s the number of ticks you can process and how you can scale across multiple different environments.  It’s the number of different feeds you can then handle.  That was part of the criteria. 

And then it was the ease of managing all of that performance infrastructure.  Clearly there are definitely products out there that can do some of these things, but it was the combination of speed and ease of use that was critical.  And at the end of the day for us, the overriding factor actually was the financial engineers signing off, because they could participate in the process and work with the StreamBase tools.  They felt we can make this product work for us.  We like this product.

Q27: When you were doing an evaluation of different CEP platforms, what did the evaluation entail?

There was performance, scalability and ease of use and how our financial engineers found the products in general.  So when we did look at the other guys, one of the nice things we liked about StreamBase was the development environment, StreamBase Studio.  The Studio environment is intuitive; it allows the financial engineers to put models together. It’s graphical. You can connect it together quickly.  It’s quite novel and we like that. 

So StreamBase had to meet both the technical requirements as well as the financial engineers’ requirements.  And StreamBase met both those needs quite well.

Q28: You mentioned that you initially used Apama.  Why did you drop Apama? 

As I mentioned, the ease of development for both financial engineers and technical engineers, the performance, and scalability of the platform were our key criteria. We dropped Apama because we liked StreamBase better from all of those criteria.

Winning Paper Evaluations, Good; Winning Customer Evaluations, Better!

Today, Forrester released their evaluation of CEP vendors, and declared StreamBase a leader in the space. Wonderful news - we're very proud. 

That said, analysts typically perform paper evaluations.  Forrester's paper evaluation is a good tool, and helps prospects create short lists for competitive evaluations.  But as a technologist, I'm thankful for the recent public disclosure of findings from some deep-dive customer evaluations from the last six months.  These evaluations were performed by firms that installed the products and ran, in many cases, stringent benchmarks and scalability tests. They evaluated the debugging tools by using them, and used deployment features like high availability.  They looked at how easily threading or partitioning behavior can be designed and controlled by doing it.  

In some cases, these firms released hard numbers.  In others, we have generic performance characterization information that we provide on an as-needed basis.  And in many cases, they put their money where their mouths are - they made one choice and bought the product. In case you missed them, here's the public take on some of these evaluations:

• CME Group (Press release, CME Group by the Numbers article, follow @CMEGroup on Twitter)
• RBC (Press coverage in Inside Market Data)
• BNY ConvergEx (Press coverage in The Trade)
• PhaseCapital (Blog post: 5 Observations about PhaseCapital's selection of CEP, blog post: 8 Lessons for Entrepreneurs - Based on PhaseCapital)
• BlueCrest (BlueCrest case study, interview with Justo Ruiz-Ferrer from BlueCrest)   

Among the more interesting evaluations was the CME Group's work.  They performed the most exhaustive CEP evaluation I've seen in years - perhaps ever - and their findings reflect what we typically see when you dig in deep and stand StreamBase, Aleri, Coral8, IBM, Oracle, TIBCO, and Progress Apama against each other.

Steve Goldman, the director of Technology Architecture from CME Group, said it nicely and even put it on video for us, where he cited usability ("The business users and technology users felt the StreamBase technology was heads above everybody else"), horizontal scalability & performance ("This was very, very important"), and sustainability ("The ability to run the platform reliably and with fault tolerance was key.") as the key reasons why the CME Group chose StreamBase as the leader, and the choice for their enterprise CEP deployment.

Here's the clip of Steve from CME Group, in his own words:

  

This endorsement was so strong and decisive that some of our competitors have started making crazy claims, including that StreamBase gave their product to CME Group for the endorsement.  In some cases the claims are so unprofessional that I enjoy nothing more than personally responding to them (with appropriate confidentiality of course).  In fact, I enjoy doing this so much, I decided to put it right out here on the blog and offer my direct contact.  I can be reached at mpalmer@streambase.com, or directly in my office:  +1 781 761-0812.  Naturally, sales solicitations are not welcome and will not be returned. 

On IBM, Unstructured Data, and CEP

(Richard Tibbetts is Chief Technology Officer at StreamBase Systems. Follow him on Twitter as @tibbetts)

Recently IBM System S (aka Infosphere Streams) has been getting a lot of attention in the press. As is the custom in the news business, writers try to focus on what is new or novel in the technology. Some have talked about the automatic query generation or the massive parallel scalability, which are some of the contributions of the System S research project. Unfortunately others have picked up on the distinction between structured and unstructured data processing and identified that as unique to System S. Philip Howard writing at IT Director is one example, and another comes in the comments on a StreamBase blog post.

The reality is that CEP engines have been processing unstructured data for a long time, and doing it well. For example, unstructured data processing is a major aspect of applications in the defense and intelligence space. As one architect at an intelligence agency put it, you can’t standardize messaging protocols when the other side doesn’t want you to listen.

Unstructured data processing means processing data that won’t fit into a standard relational model. The most unstructured data is media, such as text, audio, or video. Other kinds of unstructured or semi-structured data include XML or packet capture data. These data formats may come as raw streams, or as part of the data payload in events that also contain structured data.

Specialized algorithms and libraries exist to extract meaning from this data, ranging from XPath queries and protocol analyzers to natural language processing (NLP) and machine vision. Different kinds of analysis require different algorithms, but they all share some characteristics. A system for real-time unstructured data processing requires four things:

• Unstructured Data Objects — Data processing begins with data representation. The platform must have an efficient mechanism for representing large data object, integrated with memory management, persistence, and messaging systems.
• Extensible Language — Most unstructured data processing involves specialized algorithms. The language must be extensible so that these domain-specific algorithms can coexist with built-in functionality. Since the algorithms are often already available as libraries, it is important that these APIs support mainstream languages.
• Unstructured-Data Aware Authoring Tools — Authoring tools are a critical part of any modern platform, and it is important that the tools be designed for both real-time processing and for both structured and unstructured data processing.
• Clustering Capabilities — Unstructured data processing applications are often resource intensive. Spreading the load across large numbers of compute servers is a critical scaling technique. Being able to scale via multi-threading across a single large server is also required.

Many commercial CEP systems already have some or all of these capabilities; StreamBase, for example, has all of them. Unstructured data objects can be ingested over the network at very high speed. Advanced text processing plugins can be developed in Java or C++ - and can be developed and debugged in the CEP development, debugging, and testing tools (i.e., StreamBase Studio.) And event processing applications that analyze structured data can be deployed and scaled across large clusters.

One reasonably infers that System S has these capabilities, though for the moment it is difficult to find information on authoring tools.

Now Philip Howard suggests that System S is more capable in unstructured data processing because it does not use SQL. This is a non sequitur. The additional power afforded StreamBase by using SQL does not harm developers of unstructured data processing applications. Instead, it increases the flexibility of their systems, and the speed with which they can learn the system. One system, developed using StreamBase, first extracts meaning from unstructured text, and then uses that meaning data to identify which analysts need to be alerted to this data. The conditional alerting is expressed best in SQL, while the text processing uses unstructured facilities. Developers can build alerting logic using StreamSQL EventFlow and integrate the text processing algorithms without learning an entirely new language.

The IBM SPADE language is not based on SQL or any other language familiar to developers. This may have given IBM researchers a lot of freedom to experiment when designing the system, but it won’t help enterprise programmers learn the language quickly, and it certainly is not a silver bullet for unstructured data processing.

Design for Debugging (by a debugger)

(Matt Fowles is a Senior Software Engineer for StreamBase, whose interests include programming language, compiler, and virtual machine design.)

As the ponytail (and the tiny bio above) imply, my posts will focus on technical details such as language semantics, general programming issues, compiler design, or anything else I feel is nifty.

Bob Hagmann from Aleri recently posted a great piece on designing and debugging CEP apps; unfortunately, he leaves out what that process of debugging might actually look like.  While some languages force you to spelunk through printf output or trace files to track down a bug, I have found that the returns from an integrated debugger are immeasurable (and greatly magnified by good design).  StreamBase ships with a visual debugger built into its development environment, so I will expand on Bob's post with a concrete example using the StreamBase visual debugger.  Consider the following sample application that continuously calculates Bollinger Bands for stock quotes (for those playing along at home this sample ships with StreamBase and can be loaded with "File -> Load StreamBase Sample...")

 

This app actually calculates Bollinger bands in two different ways, one using a simple moving average and the other using an exponential moving average.  For simplicity, the two paths are separate branches of the main app, allowing each path to be debugged separately.  Within each path, the calculation is divided into a first step, where the moving average and standard deviation are calculated, and a second step, where the intermediate results are used to calculate the Bollinger bands.  This allows us to use the StreamBase debugger to inspect intermediate results and inspect their values.

With the app paused at a breakpoint, we can inspect the fields of the current tuple or at any earlier point in the active data flow, allowing easy inspection of transformations and invariants that need to be maintained.  The StreamBase debugger also lets the user modify the value of fields in the current frame before resuming execution, providing a simple way to test downstream components with hand crafted data.

Of course in an application this simple, one doesn't even need a debugger or careful design.  But, when debugging a larger application like a smart order routing system, organizing your code into staged transformations where each stage is a submodule can be invaluable.

This sort of design not only allows you to step into each module and debug its inner workings separately, but it also provides clean abstraction barriers for unit testing.  And once you have unit tests for each stage and a debugger to help you fix regressions, you will be free to refactor, optimize, and extend your application with impunity.

Building an FX Aggregation System

We just finished designing a presentation on FX Aggregation and complex event processing (CEP) that we'll deliver as a webcast on Thursday, April 16th at 10AM (register here).  Over 700 people signed up, which shows how much interest there is automated, computer-based FX trading!

The FX market is ripe with electronic trading opportunity because the markets are so fragmented, unregulated, and a lot of trading isn't electronic at all - so it feels a bit like a trading "gold rush" as firms employ new technologies for FX trading.  My personal experience is with equities-based trading, and today's FX markets feels like the equities market from about 7 years ago.  

The FX webinar will be a short tutorial on how to build an FX system with CEP - specifically, StreamBase of course.  We'll run through the 5 major technology elements of such a system:

1. FX Market Visualization.  Front end technologies that are ideal for displaying aggregated FX market views, algorithmic trading strategies, and integrating the typical client desktop technologies you'll find on an FX trading desk.  In our session, we feature Eclipse RCP technologies.  We've seen RCP take off recently as the open-source, community-based graphical user interface technology of choice with large and small firms.  
2. FX Market Connectivity.   How do you connect to CurrenEx, Hotspot, direct bank market data, etc., and connect to execution venues via FIX or other protocols? 
3. Real-Time Aggregated Market Data.  Creating a consolidated view of the market that can be used to implement FX pricing models, algorithmic trading strategies, or simply visualize FX market data. 
4. Development tools.  To design, implement, test, backtest, debug, and evolve FX trading systems.  
5. Algorithm development.   What tools are best to express FX trading algorithms?  How can you change them quickly?  We'll use a triangular arbitrage algorithm to illustrate.   

Naturally we feature our CEP development tools as the example of how you would build such a system, and show some example of interesting technologies like RCP for client front-end development, and take some questions and answers.  

Hope you enjoy the webinar!

Language Design Lessons 2 - Lists and Custom Functions

(Matt Fowles is a Senior Software Engineer for StreamBase, whose interests include programming language, compiler, and virtual machine design.)

As the ponytail (and the tiny bio above) imply, my posts will focus on technical details such as language semantics, general programming issues, compiler design, or anything else I feel is nifty.

This post is the second in a series exploring how new features in a language often impact unexpected corners of the language and can lead to a better whole.

In StreamBase 6.3, to be announced in early April, we added lists as a parameterized data type to StreamSQL, allowing easy representation of data of unknown length:

CREATE SCHEMA offer_details (
  symbol string,
  num_shares double,
  share_price double,
  user_id string
);

CREATE SCHEMA order_book (
  bids list(offer_details),
  asks list(offer_details),
)

It is worth noting that StreamSQL's parameterized data types are more like C++'s than Java's: a parameterized data type is not complete until it has been given a parameter. For example, list(int) or list(list(string)) are both complete types, but list alone does not specify a type. While Java's generics suffer because of their use of type erasure, StreamBase's parameterized types provide strong, static typechecking.

One of StreamBase's selling points is the ease of extending the StreamBase Expression Language with custom functions written in Java or C++.  When integrating with Java, we automatically map between StreamBase types and their Java equivalent (e.g. StreamBase int to Java int or StreamBase string to Java String) for both arguments to and returns types of custom functions.

Obviously, the natural representation for a list(int) or list(string) in Java is a List<Integer> or List<String> respectively; unfortunately, at the JVM level these types don't exists. Both types become merely List after type erasure. Thus, we cannot automatically infer the type of an argument or return for parameterized data types. We could require users to explicitly declare the types of their custom functions in the configuration file, but that is a bit inelegant (and also prevents users from writing a single Java function that can take several different list types like list(int), list(long), list(double)).

After some discussion, we decided that the users needed a more direct hook into the typechecking process for their functions. Thus, we added the concept of a function resolver. This is probably easiest to describe with an example:

class ExprUtil {
  @CustomFunctionResolver("concatenateResolver")
  public static List<?> concatenate(List<?> v1, List<?> v2) {
    List<?> res = new ArrayList<?>();
    if (v1 != null) { res.addAll(v1); }
    if (v2 != null) { res.addAll(v2); }
    return res;
  }

  public static CompleteDataType concatenateResolver(
        CompleteDataType arg1, CompleteDataType arg2) {
    if (arg1.getElementType() == null) {
      return null; // not a list
    }
    if (arg1.equals(arg2)) { return arg1; }
    return null;
  }
}

The concatenate function above will take 2 lists (provided they are the same type) and return a list of that type. We use the @CustomFunctionResolver to annotate the method with just such a resolver, and everything else is taken care of.  During typecheck time, the resolver will be called as necessary to determine whether a function is a legitimate candidate. Then at runtime, the concatenate function will be called appropriately. One can, of course, do more complicated manipulation of the types involved if inclined:

class ExprUtil {
  @CustomFunctionResolver("flattenResolver")
 public static List<?> flatten(List<List<?>> top) {
    List<?> res = new ArrayList<?>();
    if (top != null) {
      for (List<?> sub : top) {
        if (sub != null) { res.addAll(sub); }
      }
    }
    return res;
  }


  public static CompleteDataType flattenResolver(
        CompleteDataType arg) {
    CompleteDataType inner = arg.getElementType();
    if (inner == null) {
      return null; // arg not a list
    }
    if (inner.getElementType() != null) {
      return null; // arg not a list of lists
    }
    return inner;
  }
}

As an added bonus, while implementing this feature, we realized that hierarchical data is just another parameterized data type.   Prior to adding type resolvers we didn't allow tuples as arguments to custom functions.  But now that we have a simple, clean API for interfacing parameterized types between Java and StreamSQL, we can (and do).