The StreamBase Event Processing Blog

Guangfa Building The Fastest Trading System in China with StreamBase CEP and Thomson Reuters

Guangfa (GF) Securities, one of the largest securities brokers in China, announced this week that they are building out a StreamBase CEP-based trading system, which will be the fastest in China, and includes innovative rapid pairs trading and order execution algorithms.

The GF system will be an end-to-end trading system, but what's remarkable about their system is that it's not being built with legions of programmers and consultants.  GF has a small specialist team that have re-used open-source components from the StreamBase Component Exchange and extended them with their own "secret sauce" and knowledge of the Chinese micro-market structure. 

GF's rapid time to market is made possible due to the real differences in how the leading products are built. Esper is a simple, "roll your own" toolkit, and lacks enterprise features and connectivity. Apama is a low-level scripting language with lots of pre-built applications for FX and surveillance, but doesn't have the robust development tools or visual programming support offered by StreamBase CEP.

StreamBase CEP is built for the professional developer and quantitative analyst. It's a comprehensive, event-based system platform, featuring a visual CEP development platform, seamless integration to event data, Java, Matlab, and over 150 environments typically found on the trading floor. This makes it possible for a small team of professional staff to build a complete system in months, not years.

StreamBase CEP is designed for speed. Speed of development - when measured, usually 40%-75% faster than other approaches. Speed of deployment - out of the box, applications that have sub-millisecond latency. Speed to work with Java - with the ability to quickly snap in existing Javacode, or extend event-based logic with Java. And finally, speed of integration, with an ecosystem of over 150 external integration points including our partners in China, Thomson Reuters.

Learning about this difference is repaid to developers, since time to market has a tangible impact on the business bottom line. We recently produced a 20 minute CEP programming tutorial by StreamBase Solutions Architect Mark Hudson that gives developers a good introduction to the power of visual programming with StreamBase Studio. Watch it and learn how you can get systems to market more quickly and with higher quality, leveraging what you have and the development practices you already use.

Do more with less and do it quickly.

 

Guest Blog by Peter Rossi on NETIQ, Aegis and CEP integration

In the increasingly complicated world of IT – Big Data, Cloud, Super Datacentres, DevOps and Agile Development legacy IT support teams are overwhelmed with the new world that they live in.

Traditional server architectures on a 3 or 5 year refresh cycle are being replaced with cloud solutions which may only exist for a matter of hours, but are equally important to the success of the organisations they underpin.

In most organisations IT monitoring systems are designed in from the ground up, as most CIO’s and CTO’s have been caught out by an unexplained outage at some point. However, many IT departments still live blissfully unaware of what is happening in their infrastructure…. Until it’s too late!

Regardless of the actual monitoring technology you will generally see a team of IT professionals attempting to understand this raft of new data. They have to assess multiple streams of monitoring data, understand its context, point the finger at the potential problem and then perform some form of remediation to resolve it. 

This has led to the rise of Complex Event Processing (CEP) engines coupled with IT work flow automation tools, such as NetIQ Aegis which utilises StreamBase as its CEP engine.

NetIQ Aegis allows you to listen to multiple streams of data from your IT monitoring systems simultaneously, for example:

• Server Alerts
• Storage Systems
• Network Switches
• Network Intrusion Detection Systems
• Firewalls
• Audit Logs

You can then create “triggers” using the StreamBase CEP to perform data correlation in real time; this is referred to as a temporal query. In the old world of querying static data, imagine this to be a static query that the data washes over.

This kind of “data in flight” analysis has changed the financial markets – you can now analyse events as they are happening – this type of processing allows you to watch for events across all available data streams in real time – e.g.

• If you see a missing ping alert within any 5 minute window:
  • AND the URL www.myurl.com stops responding
  • AND there is an alert from the storage subsystem
  • THEN Trigger

Every time the CEP engine sees a group of matching events it raises a new alert or trigger…. This is where it gets interesting….

NetIQ Aegis can use event triggers to kick off workflow processes. These workflow’s can contain any number of steps to remediate known faults or raise service desk tickets or make any number of changes to your systems to resolve the problem, automatically, with no human input.

This isn’t to say that we don’t need skilled IT professionals anymore, far from it, but why have a highly skilled worker repeating very standard and similar fixes all day long, when they could be put to use making your business more successful? Complex Event Processing can help you achieve this.

About the author

Peter Rossi is an IT automation expert in the UK currently working on exciting new highly automated cloud project Skyscape Cloud Services (www.skyscapecloud.com). Independently of this, Peter is the author of a well known Microsoft scripting blog www.poshpete.com which focusses on the use of PowerShell in automation. 

Follow Peter on Twitter 

 

Betfair Adopts StreamBase for Real-Time Analytics

(By Richard Tibbetts, StreamBase CTO)

Today we announced that Betfair has selected StreamBase for Real Time Analytics. I’m excited about this news as it demonstrates the importance of real-time data and analytics for driving the best possible consumer user experience for streaming, online services, and the role of StreamBase as the ideal platform for event-driven environments like Betfair.

American readers of this blog may be unfamiliar with Betfair.  Betfair is the world’s largest betting community and one of the world’s leading online betting and gaming operators. At the heart of Betfair is its pioneering Betting Exchange, where customers come together to bet on odds set by Betfair or offered by other customers, instead of with a traditional bookmaker. The Betting Exchange provides better pricing, flexibility, and a level playing field in a massive market. In April of this year, Betfair had over 4 million registered customers worldwide and processed, on average, more than 7 million transactions per day on the Betting Exchange. 

Betfair provides this service via Internet, mobile, and tablet devices that maximize the playing experience, and the Betfair infrastructure implements a service that implements its sophisticated betting market, real-time data processing, and real time clearing of bets.

Betfair is well known for its technically ambitious support for betting during a match. Players can bet while watching a live match, at the last possible moment. When I first met Alex Kozlenkov (Head of Research at Betfair) he described his challenge of executing 100,000 transactions per second against a single product, each transaction (or event) potentially impacts the price of that product, and must be processed in order.

And it’s not just the transactions that are streaming in the Betfair environment, they are also a leader in using real-time data to make the customer experience more engaging, fun, and unique. We’re in the midst of a renaissance of innovation in data management due to “Big Data” – innovative tools and infrastructure designed to manage increasing volume, velocity, and variety of information.  The betting world is no different - new real-time data feeds provide streaming play-by-play data from cricket matches, and real-time social media data brings an opportunity to crowd-source context about the events from fans watching live (in related news, StreamBase recently announced integration with GNIP and connectivity to Hadoop via Flume that enables connectivity to Big Data)

By investing in real-time analytics, Betfair can tap into streaming information and from the betting ecosystem and leverage it to provide a better experience for users live on the site, on mobile devices, and on tablets, providing innovative context and experience to the Betfair experience.

Consumers are now conditioned by social media to expect information and data to be real–time, whether it be pictures of a night out, stock twits, or a location update. Betfair represents a step in the march of all businesses to become more social media aware, real-time, and web-scale. Customers expect more than just a scrolling Twitter feed in the sidebar – they expect real-time context and relevance. The winning firms will integrate a variety of data sources - structured and unstructured, historical and real-time - and distill them down to insight that matters, when it matters. The real-time business of the future requires all of this, and requires real-time analytics and event processing as the new IT computing fabric.