The StreamBase Event Processing Blog

Introducing Live BI and StreamBase LiveView

For some time now, StreamBase has been working on a new product concept that's a little bit CEP, a little bit data warehouse, and a little bit business intelligence. Today we've unveiled it as StreamBase LiveView. Our customers have called it a "game changer," and a product that "fills a massive hole in the BI space." Hyperbole like that needs to be backed up, so here we go.

Complex event processing (introduction on Google+) was envisioned by its inventors as a "database for moving data" as firms increasingly deal with data in motion from mobile devices, social media and digitized data of all types, and in all industries. In its first decade of commercial use, CEP has come to power automated trading on Wall Street. A who's who list of firms have been public about their adoption of CEP, so let's examine the live BI problem in the context of Wall Street.

Industry analysts estimate that over 60% of U.S. equities trading is automated. The problem is, with so much automation, it's impossible to see and act on what's going on, because a typical trading environment can generate 10's of millions of events a day. Existing business intelligence and data warehouse tools aren't used in the front office of Wall Street, because they're designed to summarize what's already happened, not what's happening now. 

StreamBase LiveView brings BI and streaming data together, for the first time. Applications that need Live BI are everywhere on Wall Street - trading risk management, position management, P&L management, surveillance, compliance, and trading itself, as Waters Technology magazine describes in the article that StreamBase Seeks 'Radical Transparancy' in Business Intelligence.

And applications in other markets can be found in any environment that has big data in motion: telecom (location aware insight and live network operations), security (live fraud detection, live surveillance), and e-commerce (live social media analytics, ad placement, big streaming analytics for Hadoop and Flume). These apps can be deployed on premise or distributed in the cloud. 

StreamBase LiveView is built with StreamBase CEP, which provides connectivity to over 125 event-based sources of data and the extreme scalability required by live BI. The server-based architecture is designed to process hundreds of millions of events per node, absorbed from a network of event sources such as an enterprise message bus or scalable event absorption layer like Flume or IBM Streams. At the front end, users can slice and dice streams of data with ad-hoc query tools in the StreamBase LiveView Desktop (shown above). But here's the key: once users ask the questions they want answered, StreamBase LiveView keeps answering their questions - as conditions change in real-time; users see an always-live view of streaming data.

For the user, all this technology means just one thing: they see business intelligence that's constantly fresh and relevant to their business operations. It's not a view of 5 days ago, or even 5 minutes ago. The data is live, the insight is live, and live insight can lead to live action.

Facebook, Twitter, Google+ give us a live view of what's happening in our social networks; StreamBase LiveVew gives firms a live view of their enterprise data.

To learn more, watch the StreamBase LiveView presentation, follow new materials, articles, and resources on Google+ (+StreamBase and CEO +Mark Palmer), read industry analyst Curt Monash, DMBS2.0, or follow StreamBase Systems (@streambase), CEO Mark Palmer (@mrkwpalmer ), and CTO Richard Tibbetts (@tibbets) on Twitter.

 

Big Companies Can Innovate, if they Act Small

(Mark Palmer is the CEO of StreamBase Systems.  Follow him at twitter.com/mrkwpalmer. This post was originally released in 2009, but rewritten in January of 2010)

The American spirit is an entrepreneurial spirit. So it’s no surprise that the recent economic upheaval has spurred a rash of discussion about entrepreneurship and innovation. Today, for example, the New York Times reported that we are in the midst of a shift where innovation is moving to big companies, and away from small companies. But the article missed the heart of the entrepreneurial matter - it’s approach, not size, that forms fertile ground for innovation.

Size of a company is a specious way to characterize innovation. But is there a better way? Gifford Pinchot's classic book Intrapreneuring analyzed the leading big innovative companies of his day (the book was published in 1985), and his findings have stood the test of time. 

To begin with, he found that big companies, as a group, aren't good at innovation:

You might think that larger organizations would be stodgy in thinking up new ideas, but, because of a wealth of management talent, would be very good at executing them. It turns out, however, that just the reverse is true:  our large organizations are producing large numbers of good ideas but generally are unable to implement them." 

Unfortunately, I think this finding still stands, 25 years later.

But he went on to describe how big companies can adopt the right spirit, hire the right people, and design the right incentives to innovate. A few central points are helpful to review.

To innovate, big companies must form small, autonomous, focused groups. These groups must be incentivized to act like small companies, by adopting the "Intrapreneur's 10 Commandments": 

1. Come to work each day willing to be fired
   
2. Circumvent any orders aimed at stopping your mission
   
3. Do any job needed to make your project work, regardless of your job description
   
4. Find people to help you
   
5. Follow your intuition about the people you choose, and work only with the best
   
6. Work underground as long as you can – publicity triggers the corporate immune mechanism
   
7. Never bet on a race unless you are running in it
   
8. Remember it is easier to ask for forgiveness than for permission
   
9. Be true to your goal, but be realistic about the ways to achieve them
   
10. Honor your sponsors
    

These commandments will seem obvious to entrepreneurs, but they are both rare and threatening in most big corporations.

So if you're the CEO of a big company and you're serious about innovation, what should you promote? First the CEO must check his or her own personality. Not all CEOs are alike. Some personality types are best suited for the "cash cow" part of a corporate lifecycle, and some are best suited for the early stages.  

What are the characteristics of people who can thrive in innovative firms? Pinchot provided a tool for this too, with his breakdown of 19 characteristics of intrapreneurs versus traditional managers. Here's an exerpt from that table, (or download the entire list here, published with permission from the author).

	Traditional Manager	Intrapreneur
Primary Motives	Wants promotion and other traditional corporate rewards.  Power-motivated.	Wants freedom and access to corporate resources.  Goal-oriented and self-motivated, but also responds to corporate rewards and recognition.
Decisions	Agrees with those in power.  Delays decision until gets a feel of what bosses want.	Adept at getting others to agree to private vision.
Time Orientation	Responds to quotas and budgets, weekly, monthly, quarterly, annual planning horizons, the next promotion or transfer	End goals of 3-15 years, depending on type of venture.  Urgency to meet self-imposed and corporate timetables.
Action	Delegates action.  Supervising and reporting take most of energy.	Has a bias for action. May know how to delegate, but when necessary does what needs to be done.
Courage and Destiny	Sees others in charge of his or her destiny.  Can be forceful and ambitious, but may be fearful of others’ ability to do him or her in.	Self-confident and courageous.  Many Intrapreneurs are cynical about the system, but optimistic about their ability to outwit it.
Attention	Primarily on events inside corporation	Both inside and outside.  Sells insiders on need of venture and marketplace, but also focuses on customers.
Risk	Careful.	Likes moderate risk.  Generally not afraid of being fired so sees little personal risk.
Market Research	Has market studies done to discover needs and guide product conceptualization.	Does own market research and intuitive market evaluation like the entrepreneur.sponsors

These are just a few key key elements a culture that's designed for innovation. So many more are critical, including compensation plans, equity structure, organizational structure, and so on. But I find that for people with the entrepreneurial mindset and some experience, those things become obvious.

Since our economic future depends on getting innovation right, both in big companies and small, we're lucky that we already know how to innovate. We just need more companies and leaders that have the guts to get out there and do it.

Nine Predictions for the Future of Event Processing (CEP)

(Mark Palmer is the CEO of StreamBase Systems - follow him at twitter.com/mrkwpalmer)

The information technology world is buzzing about event processing (or complex event processing — CEP) as one of the hottest areas in software innovation. For example, Time Magazine declared event processing to be one of eight “Innovations that will change your life,” the World Economic Forum at Davos named event processing as one of the 26 top innovations in the world, Gartner analyst Roy Schulte calls event processing the key to “agile companies,” and event processing is all over top trends lists for 2010, including Entrepreneur / VentureBeat, CIO Magazine, and well-respected IT industry analyst Brenda Michelson.

Before getting to predictions for the future, let’s step back and look at what the hype is all about.

What’s the problem? Why event processing? What is event processing?

Let’s start with the problem: recently, and for the foreseeable future, the physics of data has fundamentally changed. The first 50 years of data processing managed data at rest; the next 50 years of data processing will manage information in motion. Sensor data, digitized stock market feeds, click-stream data from online commerce, and events from real-time social media activity — all this information is in motion, streaming through the global high-speed network of computers (watch this 1-minute video tutorial on event processing for an overview).

Not only is today’s data in motion, there’s more of it. In two minutes, the data stream from Twitter generates the amount of text found in the entire works of William Shakespeare. A single feed of U.S. stock market data generates the same volume in under three seconds.

Some industries, like financial services, have quietly figured out how to harness the power of this streaming data — for example, 70% of the trading done in the U.S. stock market every day depends on streaming data, and much of it uses event processing technology.  The federal government uses event processing broadly, and uses of event processing outside of these markets are growing.

The early impact of event processing has been profound because it presents an opportunity to innovate by applying a new computing physics for moving data. In the late 90‘s the inventors of the relational database (Dr. Michael Stonebraker and others) had a simple idea: to process moving data with a language (SQL) borrowed from the lingua franca of databases, but which implements its processing in a way that eliminates the latency and restrictions of storing data to disk. The objective was to help applications find needles before they go into the haystack. Metaphorically speaking, a business that uses event processing is like the bear in the photo above: the bear can know — in real-time — where the salmon is. If the bear had to wait for a database, he’d be finding out next week where the salmon was.

It’s a simple idea, but not easy to implement. But ten years on and driven by pioneers in financial services, event processing now processes quadrillions of dollars a year on today’s version of the virtual, electronic Wall Street. The natural gestation period of ten years is ending, the technology is proven, and it’s fun to see it proclaimed to be the next big thing.

Predictions for the Evolution of Event Processing 

OK, so lots of smart people think event processing is hot and important. But history is littered with dead technologies that were once proclaimed as “the next big thing.” What does the future of event processing hold? Since a streambase sounds as important as a database, will Oracle own this space? What role will open source play? Will the existing IT infrastructure providers be the gorillas or the monkeys of the event processing era? 

As the CEO of what most consider to be the leading pure play event processing company, I'm fortunate to see signs that foreshadow the emergence of a great industry. So here are my predictions for the future of event processing, beginning with the dramatic change in the nature of event processing customer deployments that has occurred in just the past eighteen months.

Prediction #1: 100+ Event Processing Applications Per Firm Milestone will be Passed. Event processing turned an important corner in 2009; it went from an innovative enigma to a key architectural elixir in some of the world’s most innovative companies. Previously, early adopters used event processing in applications like algorithmic trading, fraud detection, location-based services, and homeland security. These applications were an important first steps, and a sign of things to come.

But the turning point in 2009 was news that some of the world's biggest companies went big with event processing. The watershed event was when the CME Group, one of the largest financial exchanges in the world, which trades $1.3 quadrillion (1,300,000,000,000) in assets a year, adopted event processing enterprise-wide. And other big names made big announcements about event processing like RBC, ConvergEx, CMC Markets, and City Index. These firms aren't just building a single application with event processing; they made broad, multi-application, enterprise-wide commitments to a new technological backbone.

Beginning in 2010, watch for the next step — for individual firms to deploy over a hundred event processing applications throughout their new IT foundation — a sign that event processing is coming of age as the backbone for the real-time enterprise.

Prediction #2: The "Deluge of Real-Time Data" Will Not Drive Event Processing Adoption 

Yes, it’s true that three seconds of stock market data can generate more data than Shakespeare did in his entire life’s work. Yes, the best event processing platforms can process that volume with latency measured in microseconds (a microsecond is a 1/millionth of a second; it takes light 5.4 microseconds to travel a mile in a vacuum). Yes, many times, speed is the main selection criteria among CEP products (for example, trading specialist CMC Markets processed 3,000 events with one vendor, switched, and currently processes 17,000 events a second). Yes, StreamBase is known for its blinding speed we make it better with every release.

But will speed be the most dominant reason why event processing is adopted? No.

Still, headlines proclaim that the raison d’etre of event processing is to "tame the data deluge," but these stories miss the point. No, most often, event processing will be adopted for different reasons.

Prediction #3: Cognitive Physics Will Be as Important as Computing Physics

As Clayton Christensen suggests in The Innovator’s Dilemma, simplicity is always a disruptive innovation, Today this is more true than ever. The half-life of software is weeks and days, not years and decades. In computing’s first 50 years of technology, frictionless electronic distribution wasn’t the norm; now, with high-speed networks, cloud computing, and software as a service, customers expect new features immediately, and because switching costs are so much lower, they'll leave — or be acquired — more quickly than before.

If you want to do a job faster and better, you need the right tools. For knowledge-based work, the right tool aligns the "cognitive physics" — that is, good tool choice aligns the way of thinking about a problem with the way the solution is built. When cognitive physics matches implementation physics, barriers to implementation fall. Event processing tools can break down these barriers in an organization that has moving data. For example, PhaseCapital, a Boston-based hedge fund, built their entire trading system with event processing in four months — CEO Eric Pritchett estimates it would have taken them four years with traditional programming tools. But more importantly, PhaseCapital changes their system every day. They test and refine their existing ideas. Invent new ones. 

So the future of innovation with event processing will be governed by its ability to reduce the friction between the way you think about a problem and the way you solve a problem. For an event-based business, ideas turn into applications more quickly, improvements are made more quickly, and problems are fixed more quickly. And this kind of speed — the speed to find, implement, and get a new idea to market — is a key element of a truly "agile" company.

Prediction #4: Quantitative Thinking will Trump Traditional Thinking

Earlier this month, the Bureau of Labor Statistics forecasted the 2 of the top 3 biggest growth areas of labor in the U.S. to be tech-driven: a whopping 1.5 million jobs over the next 10 years (coverage from the Huffington Post). Yet the vast majority of Fortune 100 CEOs are still technology slackers(as measured by participation in social media. Hardly a perfect measure, but it's scary).

Modern tech-driven innovation will be led by a new kind of business leader who has a keen sense of technology, and barriers between C-Level executives and IT must fall. Two industries already illustrate the trend that will prevail in all industries over the next 50 years: capital markets and baseball.

Yup, that's right, let's start with baseball. In the past 20 years, the organizational principles of baseball have changed from gut feel and folklore to robust statistical analysis of every single element of the game (then applying gut feel). Bill James began his statistical analysis of what made teams win in 1977, and it was brought to the popular consciousness by the best seller Moneyball in 2003. The Boston Red Sox and their fabled 86-year losing streak was not just broken after they adopted a quantitative approach to baseball, they became the most successful team of the decade. And its no accident that the main owner of the Red Sox is financial guru John Henry, which leads us to the other exemplar for future business: financial services.

Financial services is already 20 years into this transition. In the past 10 years, mathematical, real-time trading has risen from less than 5% to over 70% of trading activity on U.S. stocks alone, and the rest of the industry is becoming electronic. 20 years ago, in my first job outof college, I worked on a trading floor. The traders were MBAs from Harvard and Stanford - brilliant, personable, and competitive. The ultimate knowledge workers. Now, the top traders are mathematicians. They are competitive but in a more cerebral way. 20 years ago, trading floors were loud and boisterous - traders shouted down phones, and at each other; now, they are intense mission control centers. Shouting has been replaced by the clicking of keys on keyboards. Analytical acumen is now just as important as intense physical passion.

Over the next 50 years, other industries will evolve in much the same way: the winners of the modern economy will adopt quantitative strategies as much as traditional business strategies.

Event processing facilitates quantitative thinking with its modern, visual development tools. These tools help facilitate analytical thinking, and also lower the barrier between business concept and implementation (see the related prediction that cognitive physics will be as important as computing physics)

For example, a trading strategist can describe a strategy like: “If the price of IBM goes up more than 2% in any 10 minute time window, but HP doesn’t, then buy HP, but only if the spread holds.” These words, this strategy, this business logic, can be expressed directly with visual tools by business analysts and IT. And the modern business leader will need to   communicate fluently in this language - they won't have to code it, but they'll have to understand the math of their business deeply, and in detail.

By lowering the barrier to express quantitative business strategies, innovation cycles accelerate.

But with only 2 of the Fortune 100 CEOs on Twitter, the opportunity to out-think your competition is wide open. So the winners of tomorrow will be led by those who get comfortable with technology today.

Prediction #5: Software Stacks will Continue to Miss the Mark

Rupert Murdoch said: “The world is changing fast. Big will not beat small anymore. It will be the fast beating the slow.” This is certainly true in the technology space - big companies can innovate, if they stay small.

In the world of enterprise software, Oracle, IBM, Microsoft, and TIBCO are big, and they don't act small. In fact, over the last decade, they have acquired their way to an ever more complex portfolio of products - they act even bigger than they actually are. Moreover, these big product stacks are designed for traditional computing - computing that looks at what happened in the past, not what's happening in real-time. 

Recently, these big vendors became aware of event processing. What’s the natural thing to do? Buy some technology and slap it on the stack! Even Progress Software, who acquired CEP pioneer Apama, lost its focus to chase the big stack companies (for example, this week they acquired a small business process management technology (Savvion) as the centerpiece of yet-another-stack-strategy).

These stacks are to event processing as the record industry is to iTunes - a different industry for a different era. Customers don't want big stacks; they want tools that solve their problems.

The truth lies in the results, as measured by customers and the specific business value they achieve. In the past year, the endorsements have gone to small, focused event processing companies, not the big stack vendors. Consider the last 12 months of customer endorsements:

In the past year, TIBCO (quoting the TIBCO CEO) claims to have won the following customers:  A “western states utility”, a “major Asian bank”, and a “major Indian mobile company.”

By contrast, in the past year, StreamBase customers aren't anonymous; they are serious corporations. They are enterprise-wide endorsements. They manage quadrillions of dollars of business every year. And they furnish technical details. The partial list of public endorsements include: CME Group, RBC, BNY ConvergEx,City Index, CMC Markets (who switched from a stack vendor (Progress Apama)), Curex, PhaseCapital,BlueCrest and Kairos. And these are public endorsements in one of the most secretive industries there is: the capital markets.

Credible customer stories don't lie, and they show that in CEP, small is beating big. Watch for this trend to continue as the nimble, pure-play players continue their astounding growth and Hollywood endorsements, and run circles around their less nimble competitors pushing big stacks of software designed for yesterday's applications.

Prediction #6: A New Event-Based Software Stack will Emerge

Just because the traditional stack vendors aren't players in event processing, that doesn't mean there isn't a complete software solution required for event processing. The new event processing stack has elements that the big stack vendors don't want you to know about (because they can't sell it to you).

The elements of any software infrastructure are:

1. Connectivity
2. Business logic
3. Presentation layer (GUI)
4. Database

For event-based systems, connectivity is the most important element. It includes legacy middleware like TIBCO, but more importantly includes new innovative messaging like high-speed solutions Solace and Tervela, Google Chrome (namely websockets), and standards like AMQP and DDS. AMQP is very attractive as a standards based messaging wire protocol. Solace, Tervela, RedHat, Apache (QPID), RabbitMQ and various other messaging providers/vendors are working hard on AMQP interoperability. CEP is great at analytics. AMQP is great at distribution. CEP over AMQP simplifies integration. Industry specific connectivity is also critical. For example, in the capital markets, connectivity to FIX, Reuters, Wombat, FX venues, and so on - this connectivity helps speed integration and bring CEP online.

Business logic is the domain of event processing like StreamBase. 

At the database level, relational databases still play a role to store history, but just as important, if not more so, are column-oriented databases like Vertica. 

Finally, some of the most innovative presentation layer technology for time-series data is coming from the Eclipse open source community. 

What’s the pattern in all this stack innovation?  None of it comes from the big companies - it’s coming from small innovative startups that are breaking down the physics of traditional computing as we form a new stack for the next 50 years of innovation.

Prediction #7: Stream-Based Platforms will Continue to Lead a Siege Against the $15B Data-Base Market

It’s flattering that Entrepreneur / VentureBeat chose StreamBase as one of the technologies that will continue to lead a siege against the $15 billlion database market, but the truth is that there are multiple technologies in the real-time computing stack required to make this happen, including high speed connectivity, column-oriented data stores and CEP.

Event-based platforms are about real-time computing; databases, business intelligence, and data warehouses are about looking at what happened in the past. By combining the two to operate on streaming data, you create an optimal platform for the real-time enterprise.

So watch for the seige on this market to continue as data management techniques help corporations find needles before they go into the haystack of corporate databases.

Prediction #8: Open Source CEP Won't Impact the Market, But Open Source Will

Although event processing platforms is robust and proven, there is still constant innovation in the space. Although the efforts of open source projects Esper and Cayuga are admirable they both miss the importance of simplicity through graphical event languages and powerful development tools. They also aren't proven for mission critical usage, or optimized for the high performance, low latency demands.

However, open source is helping spur the growth of event processing in important ways:

1. The rise of Eclipse provides instant access to a vast community of development tool plug-ins for event processing development. 
2. Connectivity is commoditizing with developments in QuickFix, AMQP implementations like RabbitMQ, and DDS implementations like OpenSplice. 
3. Events-in-the-cloud is becoming easier with open source cloud computing and web based messaging protocols.

The rise of open source for development, connectivity, and cloud computing is also why software stacks will continue to miss the mark with their proprietary stacks of software.

So as the new event-based software stack begins to emerge, event processing providers that embrace this chance will continue to grow as the enterprise platform of choice.

Prediction #9: Event processing will Yield a Great New Software Powerhouse

The ninth and final prediction for event processing is that the processing of moving information will yield a great independent software company. 

While it's nice that Time Magazine declared event processing is a technology that will "change your life," and the World Economic Forum at Davos named event processing as one of the 26 top innovations in the world, the previous 8 predictions explain why event processing will prove fertile ground for a great software sector:

• The business case - to enable the rapid construction of real-time applications  - has already impacted two of our most critical markets: the capital markets and the intelligence agencies. 
• The solution requires a fundamentally new computing physics where traditional static computing technologies don't fit,
• The technology is proven, with major companies already using CEP enterprise-wide
• The revolutionary simplicity of event processing is, as Clayton Christensen suggests, "always disruptive," and therefore facilitates disruptive innovation
• This innovation has become even more critical in an era where competitive advantage is becoming more quantitative.
• The market for data management is huge and the big software stack vendors continue to miss the mark with their technology offerings.
• The possibility of an entirely new software stack is upon us - open source is a big part of it, and innovation in the field should continue for the next 10 years.

Put all of the conditions together, and it's ripe territory for the emergence of a new software powerhouse that disrupts the old guard.

Hire Great People: 6 "How To" Tips

This week, I attended Accel Partner's CEO Summit at Stanford University, along with 150 CEOs from Accel's portfolio companies. We were joined by some well-known public CEOs and the Accel partners in a series of "fire side chats" and networking sessions to talk about leadership and entrepreneurship. During a morning session, Michael Dell was asked for his best advice for entrepreneurs, and reminded us to: hire great people.

During the day, the theme of hiring great people raised some interesting ideas from the audience and the panelists, and even on Twitter as I tweeted online during the event.

1. Spend HOURS with candidates and insist on a good cultural fit.  Michael Dell went on to share a simple, but powerful idea: toss out the idea that interviews should last an hour, and spend HOURS with new candidates. He told a story about a time where he picked up a candidate and drove to work together for an extended period of time, getting to know each other. Apparently Dell is a very "aggressive" driver. The candidate's reaction under stress told him something about his character.  

2. Go to a U2 concert.  Mark Zukenburg (founder) and Sheryl Sandberg (COO BusinessWeek profile), co-leaders of Facebook, talked about going to a U2 Concert together, and spending nights and evenings talking about the future and culture of Facebook before committing to work together.

3. If there's no cultural fit, fire away.  Lee Scott, recently retired CEO of Wal-Mart, said that there were two things at Wal-Mart that would get you fired immediately, regardless of performance: stealing and not "matching the Wal-Mart culture," which is:

    a) The customer is the boss, 

    b) Treat people with dignity,

    c) Improve every day

4. Hire people better than you.  Lee Scott's story of a key point in his career illustrated this well. He was a logistics manager, and was suddenly promoted to VP and handed one of the most critical new roles in Wal-Mart. The move ignited his rise to the top at Wal-Mart.  The reason he was handed the job?  "Because you were the only one on our list who had hired someone strong enough below him that we knew would do as good or better job than you would, so we could promote you," he was told.

5. Ask candidates: "Describe your sense of humor." In response to my Twitter tweet about this (follow me at @mrkwpalmer), my former colleague and CEO Peter Beard said that his favorite interview question was to ask candidates to describe their sense of humor, or to tell their favorite joke.  Leanne Browne from Progress said she thinks the best way to interview someone is to take them to the pub. I was reminded how one of my most insightful interviews happened on the golf course, where I learned that the candidate was a great golfer (he was very skilled at work too), he was late for the tee time (he was consistently late for meetings), he was lazy and always drove in the cart instead of walking a short distance (he cut corners at work too), and he was very creative in his shot selection (he was creative at work too).

6. Get great investors. These comments came in the context of Accel's CEO Summit. Accel was just named number one on Red Herring's List of the Top 100 Global Venture Capitalists. Enrique Salem, CEO of Symantec, explained how his VC's helped him decide whether to sell BrightMail or go public, eventually deciding on a $370M sale to Symantec (on $25M trailing revenues) that catapulted their business in a way he didn't think they could do on his own.

6 great reminders about one of the most important things we all do: choose the people we surround ourselves with.

Trading on Twitter: Opportunity, Danger, or Folly?

This week my company issued what I thought was an innocuous announcement - our software now allows applications to process Twitter tweets with an application and analyze the contents of the messages in real-time. A firestorm of debate ensued, and the issue gained mainstream media coverage in the Financial Times, the Telegraph, CNBC,and dozens of others. The debate gives us pause to consider the role of the sweeping adoption of Twitter as a real-time, global information source, and how that information might be used beyond the mostly social nature of Twitter today.

One of the examples we cited for this technology is automated trading on Wall Street, where any of the 16 million tweets a day can be analyzed in real time as a means to inform trading decisions.  Wall Street tends to be an innovator with new technologies, so it’s useful to use it as a petri dish to examine some of the opportunities and issues around Twitter in a business context.

Trading on rumor is a time-honored trading technique.   “Buy on rumor; sell on news” is one of the oldest mantras on Wall Street and Main Street alike.  Any tool that improves the speed or quality of transmitting information is of interest to traders.  Twitter does that.   But be careful what you ask for – lots of the tweets on Twitter are garbage – scam artists and manipulators.  On the other hand, there are thousands of credible sources of information, from the SEC (@SEC_Investor_Ed), to the New York Times (@NYTimes), and even exchanges like CMEGroup (@CMEGroup), which has over 500,000 people subscribed to its tweets about the futures and options markets, and more. 

And beyond any individual source, and probably more uniquely, Twitter allows instant access to the sentiment of the masses.

In the Wall Street & Technology article, Todd C. Mirabella, chief investment officer and principal of New York hedge fund QAT, says one of the uses he has found for Twitter is to help look at market volatility and compare it. "So how we get our information, that's the trick. Twitter can give us information on retail " [Tweeters] are the herd," he says.

For example, anything new that Apple does throws Apple’s stock into volatile gyrations.  The recent launch of the iPhone 3GS set off a rash of Twitter tweets about what the herd thinks – taken together, they can reveal sentiment – good and bad. Getting that feedback a few seconds faster, or a little better can make a big difference in trading returns. 

Business leaders, from CEOs to inventors, use Twitter.  For example, Richard Branson (@richardbranson), the CEO of Virgin, tweets regularly.  Now I’m pretty sure the news in the first tweet about Jimmy Fallon isn’t actionable for trading Virgin stock, but the fact that he’s ordering $2.1 billion worth of new planes might be.

Nasir Zubairi, former product manager for algorithmic trading and FX E-commerce, RBS, put it this way in the Wall Street & Technology article:  "[Twitter can be] the fastest source of news, particularly in reference to trading and algorithmic trading," he says. "News feeds are gaining a lot of popularity. Twitter provides the medium in that people, who are there, experiencing things, are able to broadcast their sentiments to their user base via tweets. This provides lots of competitive advantage to firms with trading strategies."

News about world events moves the foreign currency markets as Mr. Mirabella also pointed out.  He’s not yet trading on Twitter but said he sees real use for Twitter in currency trading: "We like to trade in ranges," he explains. "That helps us characterize the extreme edges in any specific currency. We've been trying to follow more individualized Twitter sites to see if we can get extreme views [of Tweeters]. It does give us flavor for what types of ranges we're looking at."

But real use of Twitter has some serious issues.  Technically, it wasn’t built for the kind of quality of service that a robust news source, like Reuters (@Reuters), provides – what happens when Twitter is unavailable?  There are questions about latency of the transmissions – is it fast enough?  Can it be faster?  Authentication and forgery are challenges – there’s a big difference between the real Richard Branson (@richardbranson), and the fake Richard Branson (@sirdickbranson) - who is responsible for the quality and reliability of the information transmitted via Twitter?  Should it be regulated?

And regulatory issues abound, as well.  Trading decisions are moving faster, so regulators have to move more quickly, too.  In December the unfortunate trading run on United Airlines (UAL) (read the story from the Wall Street Journal) stock to drop from $12 to $3 in just 15 minutes only served to remind us how quickly conditions can change in an era of sweeping automation.  The chain reaction was started by an erroneous trading signal from an old story that the Google news crawler picked up from 2002.  How can we guard against these unintended technology-triggered mistakes?

The adoption of Twitter in a social context has been swift, massive, and global.   As technology like event processing makes the content contained in Tweets readily available to business applications, the questions of what to do with this power and responsibility looms large.

Postscript and continued debate (July 1, 2009)

Since this writing, there has been a flurry of response - positive, negative, and vitriolic.  

My favorite coverage was the excellent, balanced story by Melanie Rodier at Wall Street & Technology: Algo Traders Connect to Twitter. She did her homework, including comments by some well-respected people from the trading industry. Read it first.

CNBC's Day-Trading 2.0:  Twitter Takes on Stocks took a day-trader's view on using Twitter to understand market sentiment.  

The Telegraph's coverage, Hedge fund managers betting Twitter will give them an edge in rapid trading also covered the intelligence and military side of the potential use of Twitter.

A spirited message board debate on Server Side was provoked by John Davis (@jtdavis) from OnixS. Like many message board debates, there are rat holes, but there's good stuff there, too.

There was an assertion that StreamBase's announcement of Twitter connectivity is a cheap marketing gimmick. It isn't. This announcement was driven by current and anticipated customer demand, as you can see from our history of connecting to Alpha Trading, Interactive Data, Lime Brokerage, Reuters, other news and data sources of many types, and so on. The article you're reading illustrates the balanced way we're talking about Twitter.  

One blogger, who used to work for one of our competitors, TIBCO, even took the opportunity to predict that StreamBase will be trounced by the "big boys" as they enter our market! To that, I say: "bring it on." Dr. Michael Stonebraker is one of the inventors and thought leaders in event processing, and StreamBase is known as one of the leading businesses in the CEP market, renowned for its performance and ease of use. Are we wary of their deep pockets? Yes. But the big companies have yet to field compelling products - in fact, we rarely compete with them. Furthermore, the "big boys" have not showed signs of innovation and intrapreneurship as I wrote in When Microsoft enters your market:  cause for worry or cause for celebration? And in Innovation: why size doesn't matter, I countered the argument that big companies are the most effective at innovating in a market like CEP. Richard Tibbetts, StreamBase CTO, clarified IBM's recent claims of innovation as misleading in On IBM, unstructured data, and CEP.  Finally, and most importantly, our customer endorsements from CME Group, BNY ConvergEx, BlueCrest, and PhaseCapital and many more say it best - directly from our customer's mouths.

Finding Alpha: Be Fast, Be Smart, or Be Dirty

(Richard Tibbetts is Chief Technology Officer at StreamBase Systems. Follow him on Twitter as @tibbetts)

When it comes to algorithmic trading, everyone is looking for alpha, outsized returns which exceed your risk. In an efficient market, there would never be any alpha, so seeking alpha means seeking inefficiencies in the market. The problem with market inefficiencies is that they don't last. The mere act of using them depletes the available revenue - when others join in, the opportunity can go away very quickly.

So not only do you have to find alpha, you have to find sustainable alpha. Since a lot of customers use StreamBase and complex event processing for algorithmic trading, I see a lot of alpha seekers. Three attributes seem to consistently contribute to sustainable alpha: being smart, being fast, or being dirty. To create a sustainable strategy, mix in at least one, and preferably more of these components.

Smart — The best-known way to create sustainable alpha is to be smarter than the competition. Unfortunately, on Wall Street, genius is a commodity, so trading techniques must be both creative, and evolve. Industry analyst AITE Group found that some algorithms have a shelf life of only a few weeks. And, since market conditions are constantly changing, “smart” is never “static.” Finally, if you can discover something based on easily available public information, the other geniuses won't be far behind. So smart, the best way to achieve alpha, must also be creative, and must also constantly evolve and get smarter as market conditions evolve.

Fast — Another way of securing alpha is to be faster than the competition. This means making decisions quickly, then executing your trades before competitors can execute theirs. Efficiency and speed is important for any system, but if you are betting on being the fastest, ultra-low-latency execution management will be keeping you up at night. The challenge with fast is that it's easy to spend money to get faster. If you are a small company, larger competitors with will always be able to outspend you, so you must take advantage of your size. In 8 Lessons for Entrepreneurs – A Case Study Based on Phase Capital, the lessons from this small, innovative firm included leveraging automation, building incrementally and quickly, and choosing the right partner. On the other hand, large companies can fight off more nimble competitors with their scale, reach, and ability to choose and deploy bleeding edge technology that smaller IT departments cannot support. In either case, you must exploit your natural advantages to stay fast.

Dirty — Finally, you can be dirtier than the competition. I don't mean being ethically questionable. Instead, I mean working with markets, information, and technology that others avoid. Developers and quants both like systems that make sense, clean systems that have been developed by like-minded engineers and are easier to work with. Getting your hands dirty with legacy systems, systems that haven't yet been polished to industry standards, or systems which were never designed for trading exploitation is one way to distinguish yourself. For example, trading in emerging markets often requires getting your hands dirty. But it also lets you trade securities that others aren't following as closely. Trading uncommon or unknown products is one strategy for finding alpha. So is trading based on non-financial information. This has famously been done with weather data and Amazon sales information. Steve Steinberg gave a great presentation on this at O'Reilly's Money:Tech. Working with these data sources may require checking your engineering sensibilities at the door, but the rewards can be substantial.

Lots of trading firms use complex event processing as their platform to create alpha-seeking strategies. But not all CEP platforms are the same. So, when selecting an alpha-generating technology platform, it is important to know what kind of strategy you are building, and what key technical elements you’ll need to meet your requirements.

To build smart strategies, a development platform must have a flexible programming language, good development tools (debugger, profiling, testing), and a good backtesting environment so that you can design, test, deploy, and evolve smart alpha-seeking strategies. There is no one more frustrated than a smart quant who is told by developers that his new strategy can't be built in their system. For example, can your platform handle multi-security strategies? Cross-asset strategies? Plug in analytics written in your quant’s' favorite programming languages? All these features and more may be required to support smart strategies.

To build fast strategies, you will need a platform and deployment environment that supports ultra-low latency. This will generally require collocation at the site of your favorite exchange. If your strategy requires interaction with multiple exchanges, than your deployment architecture must take into account the wide area network latencies involved. Your development platform should also support low-latency data transports like Infiniband, and have a multi-threaded architecture that optimizes the processing of individual events. And for some fast strategies, throughput and scalability will also be important, so multi-core technologies, clustering, and cloud computing techniques may be required.

To build dirty strategies, your platform needs to ingest any kind of data you might need. This could include unstructured data processing for handling news, analyst reports, underlying data that effects, for example, a derivative like weather information, and so on. An effective platform for dirty alpha-seeking strategies requires APIs and development tools which support building connectivity quickly. Pre-built vendor connectivity is nice to have, but for a dirty strategy you are likely to connect to arcane information sources or technologies.

If you are planning to find alpha, know what your goals are. When selecting a platform, make sure it supports the kind of alpha you will generate. If you choose correctly, the platform will help secure your alpha against the competition.

27 Twitter Lessons for CEOs

(Mark Palmer is the president and CEO of StreamBase Systems.  Follow him at twitter.com/mrkwpalmer)

I’m a CEO, and I use Twitter.  I started “tweeting,” or using Twitter, as a test a few months ago. When I told one of my board members, he was skeptical, and wondered about my focus. So as I experimented with Twitter, I kept a list of the pros and cons. At first, Twitter seemed like a waste of time - but slowly, my list of pros grew, and my list of cons shrunk. Now, I’m convinced that Twitter is an essential business tool.

I looked, but I couldn't find a list of guidelines for CEOs from other CEOs. So, even though I was loath to spend time writing Yet-Another-Article-About-Twitter, I thought my list might be helpful for someone else. And best of all, it was essentially done.  So here's my list of Twitter lessons for CEOs:

1. The business case of Twitter is compelling.   Twitter, when combined with other social networking tools, deliver measurable business value.  Lots of it.

Since my companies started using social networking a few years ago, (blogging, LinkedIn, Twitter, and so on), we've reduced marketing spend by over 50% and roughly tripled marketing effectiveness (measured by lead generation volume, cost per lead, press coverage, and other quantitative metrics) - at the same time.  We showed the chart at the right at my last board meeting.  Marketing spend is the green area chart in back, and leads are in the bar chart. 

Although these numbers don't lie, they only tell part of the story. Twitter's great, but not if you don't use it right. The rest of these Twitter lessons are about using Twitter effectively.

2. Do use Twitter to transmit values, standards, and ethics.  Peter Drucker said an essential job of the CEO is “to set the values, the standards, and the ethics of an organization.” Social networking helps me communicate my values, standards, and ethics to hundreds of people at once.  It’s like having hundreds of water cooler chats in real-time, from wherever I am - while I’m visiting a customer, speaking at a conference, or meeting with my board. 

For example, when I tweet about the exciting things I see on the road, you can feel our enthusiasm for our customers. When our CTO tweets, we reveal how we think about innovation. Recently I tweeted while I ran the Boston Marathon (video on www.notallceosarejerks.com) for Dana Farber and cancer research, which shows that we care about charity.

Twitter and blogging are unique in that they are intensely personal. With care and attention, it's the perfect medium for transmitting our ethics.

3. Do use Twitter for press relations – As I wrote in press releases: dead or reborn?, I think the old ways of communicating are forever changed. Traditional PR agencies have role, but need to evolve.  Recently I evaluated almost 20 press agencies. They all paid lip service to social media, but in my view only two had it deeply engrained in their culture. PR firms must evolve, or they will die (I didn't hire any of them and brought it in house).

4. Do make the mundane funny and constructive. Great tweets can be about mundane moments.  Zappos CEO Tony Hsieh (@zappos) is a master at it - here are three bad tweets I recently received, and 3 tweets from Tony about the same topics:

• BAD TWEET:  “Having coffee”
• GOOD TWEET:  “Had coffee w/ my mom, told her I gave a presentation @ Tony Robbins conference. She was really excited. In her words: "Wow! Robin Williams!"

• BAD TWEET: “Going into a meeting now” 
• GOOD TWEET: “Visited meetup.com office, met w/ CEO. Thankfully the meetup at meetup did not cause a rift in the space-time continuum”  

• BAD TWEET: “My allergies are killing me!”
• GOOD TWEET: “Taking allergy pills is like having Snow White multiple personality disorder. You go from Sneezy/Grumpy to Sleepy/Dopey/Happy.”

6. Do use Twitter for employee communication – Social networking is like a virtual water cooler, creating a steady, informal stream of communication about market events, customer stories, and the day-to-day developments in the company.

7. Do use Twitter to augment investor and analyst relations. Industry and financial analysts like getting a view direct from the CEO. Blogging is especially good for this kind of communication, and Twitter alerts them that you've got something new to say.

8. Do use Twitter for peer CEO learning. I follow other CEOs to see their observations - the good ones help shine a light on my own day.  I made a Twitter list of the top 50 CEOs here: @mrkwpalmer/top50twitterCEOs, as ranked by Twellow.

9. Do use Twitter to promote rising stars. We recently promoted one of our founders, Richard Tibbetts (@tibbetts) to CTO.  He’s an amazing technologist, and getting him out and speaking is great for us, and great for the industry. Matt Fowles, one of our top engineers, has written some great technical blog entries. Twitter helps raise their visibility, and educate the market.

10. Define your Twitter goals carefully. Signing up for Twitter is easy.  Writing tweets is easy.  But figuring out your goals for tweeting is hard. I already blog, write magazine articles, and speak at conferences, so it took me a while to figure out my list of goals for Twitter.  My main goal, of course, was lesson #2 - to transmit StreamBase's values, standards, and ethics.

11. Do use Twitter to make you a better communicator. Strunk and White's rule: “omit meaningless words” is a great one, and Twitter forces you to omit meaningless words by its very structure - the discipline imposed by 140 characters is a good discipline.

12. Don’t use Twitter for traditional marketing. Some companies seem to think Twitter’s just another vehicle for traditional marketing. H&R Block’s Stacey Gratz, marketing manager, explained: "[On Twitter,] we soon realized that we needed to listen and share, rather than pushing out marketing messages." Read about the H&R Block Twitter case study here.

13. Do use Twitter for thought leadership.  My company, StreamBase, is a leading visionary in an enterprise software market called event processing. The market is still less than $100M, but it's growing, and controversial. In a new area such as ours, education is important, and social networking provides another vehicle to educate prospects and customers. 

14. Do use LinkedIn and Facebook to connect. I have links to my Facebook, LinkedIn, and Twitter profiles on my business cards. (If you liked this article, feel free to connect with me: on Facebook, on LinkedIn, or on Twitter)

15. Do use Twitter to transmit your brand's conscience.  Habit #1 of Bruce Philip's 5 Habits of successful executives is: “[executives are] their brand’s conscience.”  Michael Hyatt (@MichaelHyatt) of Thomas Nelson Publishers and Richard Branson (@sirdickbranson) of Virgin do this well. I think all leaders should establish this direct link to transmit the essence of your corporate brand and culture. 

16. Do listen to Twitter for feedback. Great leaders feed on feedback. But a lot of people shy away from saying too much to "the boss." On Twitter, I'm just like everyone else, so I get unfiltered feedback whenever I want.

17. Do use Twitter to respond to rumors quickly.  Recently, we had to fire a well-known employee, and our competitor started spreading false rumors that we were downsizing.  Twitter helped me stop the rumor mill in real-time.

18. Don’t use Twitter to sell. Tempted to tweet: “Buy my product now for $19.99!”  Don't.

19. Don’t say too much. Sometimes basic stuff like where I am is sensitive. For example, I couldn’t tweet for a whole week because if the competition knew I was in Chicago, I would have jeopardized a sales situation.

20. Remember that content is still king on Twitter. Even though Twitter messages are only 140 characters, the quality of those 140 characters is still king. The link between content quality and business value (lesson #1) is direct.

21. Turn your communications approach upside down. If you've gotten this far and any of this stuff shocks you, start reading. The classic, and first place to start, is: Cluetrain Manifesto (about the book, short form, the entire ebook). Cluetrain was published over 10 years ago, and boldly declared that a “powerful global conversation had begun.” And this was before Facebook and Twitter. 

22. Don’t compete with SpongeBob.  My 8-year-old son, Jack, asked what I was doing one day.  I told him I was tweeting, and told him what Twitter is.  He asked:  “How many people follow you?” 

I said:  “about 1,200.” 

Jack asked: “How many people follow Sponge Bob?”

I looked it up.  “28,000.” 

Jack said:  “Dad - if you tell more jokes, maybe you’ll be more popular.”

I don’t want to compete with SpongeBob.  I’m don’t care how many followers I have.  Neither should you!

23. Don’t tweet too much; don’t tweet too little. I think it’s important not to tweet too much - I recently saw someone who had tweeted 26,000 times – yikes! My rule of thumb is to only put something on Twitter that I find truly remarkable. That happens between 0-5 times a day.

24. Twitter helps keep me informed in real-time, personally, with less effort, and dynamically.  Today one of our engineers asked:  “doesn’t Twitter just give you too much information?”  My response was that it’s the opposite – it helps me cull the information deluge down to the stream I care about.  Currently, I watch Ted Talks, Seth Godin, and David Armano (search on TweetDeck), my favorite industry analysts (@bmichelson), my competitors (you know who you are), my customers (CME Group, and many more), my cousin (@marissapage who’s really, really funny), and my list of other CEOs.

25. Don’t start using Twitter unless you love to communicate, and are willing to stay committed.  It’s hard to get in the habit of tweeting, and CEOs are busy!  Moreover, tweeting well is an art, and takes some thinking.  You have to want to communicate, and be willing to stay committed to creating ideas on your own; you can’t outsource the job, either.

26. Learn how to Twublish.  I used to write byline articles for industry journals. Traditional publishing is static, formal, unidirectional, and controlled by an external editor.  Then I started to blog, connect via LinkedIn, share photos with FlickR, and message with Twitter.  Taken together - and they must be used together - these tools form a new way of publishing - I think of it as "Twublishing."  

To publish with Twitter, or Twublish, involves writing to our blog.  Then I tweet about points in the article on Twitter.  Comments in the blog and direct messages on Twitter help me get feedback on the quality of my thinking.  Often, this leads me to new places, and adjacent topics.  Or I more fully develop the original idea, and more deeply hyperlink the original to new sources I discover.  The feedback deepens the original thought and creates articles that evolve into portals for other information on the same topic.  This post, for example, has links to 30 other information sources about the use of Twitter from a CEO's perspective.  

Twublishing is dynamic, informal, bi-directional, and unconstrained by an editor.  I can mix in video and images.  It's hyperlinked so my thoughts can become portals to other people's ideas that were the building blocks of my own.  And it's temporal - that is, as information changes over time and others respond, I can update or augment the original to refer to the new information.  

Other than the whopping $250 I used to occasionally get for a magazine article, publishing with social networking tools is better in every way.  

27. I’m more aware of why I love my job because of Twitter.  As I was returning from a sales trip, I tweeted:  “Anyone who thinks the capital markets are collapsing should go out and spend a day with the traders – lots of innovation going on out here!”  The act of paying attention to the ironic, funny, shocking, and curious things that happen from day to day makes me more aware of why I love my job.

 

When Microsoft Enters Your Market (CEP): Cause for Worry or Celebration?

(Mark Palmer is president and CEO of StreamBase Systems)

This week, Microsoft announced they will enter the technology space inhabited by my company, StreamBase.  Naturally, I got a flurry of questions, ranging from: "Is Microsoft's entrance going to crush your business?" to "Could Microsoft entering your market be validating?"

The question behind these questions is:  "How can a company with less than 100 people compete with Microsoft?"

First let's start with some background:  what is complex event processing, or CEP?  CEP is an enterprise software technology - big companies buy it to do big things.  CEP is kind of like a database, but turned on its head for real-time decision-making.  For example, Orbitz uses CEP to monitor their business in real time - when there's a problem, Orbitz knows about it in milliseconds, which means they can do something about it before its too late.  And they can change the logic for detecting and responding to problems on the fly. Databases, in contrast to CEP, are designed to help a company look at what already happened, not what's happening now.  Clearly the database is an essential tool that can inform future decisions.  But knowing what's happening now helps identify and therefore seize business opportunities that exist for just a short time, like fixing your problem on Orbitz before you leave and go to Travelocity.  CEP is increasingly becoming a critical tool for the enterprise.

The most extreme, and common, use of CEP is in the capital markets - firms like Deutsche Bank, JP Morgan, and RBC use CEP to automate their trading decisions.  The financial markets produce millions of events every second, and opportunities to make money last, in some cases, for minutes, or, in some cases, for milliseconds. The systems that identify and exploit those opportunities are under constant development. In trading, rapid development and instant action aren't luxuries;  they directly impact revenue and competitive differentiation.

So, can Microsoft dominate CEP?  In his groundbreaking book Intrapreneuring, Gifford Pinchot researched the success and failures of innovative efforts in large U.S. corporations.  At a macro level, he found that big companies, statistically, are terrible at innovation: 

You might think that larger organizations would be stodgy in thinking up new ideas, but, because of a wealth of management talent, would be very good at executing them.  It turns out, however, that just the reverse is true:  our large organizations are producing large numbers of good ideas but generally are unable to implement them." 

This, of course, is why the venture capital industry is alive and well - innovation naturally comes from small firms.  However, Pinchot went on to describe that when big companies act like little companies, they can innovate.  But they must create small groups, accept and expect failure, let these groups be autonomous, hire different kinds of people, compensate them differently, manage them differently, and get close to customers.  All these attributes fly in the face of traditional big company culture.  

But Microsoft has pulled off innovation before - are they doing it here according to Pinchot's findings? It's a CEOs job to watch for competitive threats, and I see no evidence to suggest Microsoft is approaching CEP in a way that portends success.  Nimble moves by Microsoft could include:  recruit top visionary CEP engineers (they haven't); create a small, independent group (they didn't - the product is coming from the SQL server group, which has hundreds of engineers); get out and talk to customers directly and co-develop with visionary partners (we see no evidence of this); recruit and empower entrepreneurs and empower them to break down corporate structures (Microsoft didn't do this either).

So the physics that would make an entrepreneur worry just aren't there at Microsoft's CEP effort.

Which leads me to my current answer: the fact that Microsoft is trying to get into CEP is a big, important endorsement for CEP.  And since we think we have the best CEP product available (as many others who have evaluated us have found), we invite the competition.  If Microsoft's endorsement makes corporations more aware of CEP, it's better for StreamBase.  

Welcome, Microsoft!

Design for Debugging (by a debugger)

(Matt Fowles is a Senior Software Engineer for StreamBase, whose interests include programming language, compiler, and virtual machine design.)

As the ponytail (and the tiny bio above) imply, my posts will focus on technical details such as language semantics, general programming issues, compiler design, or anything else I feel is nifty.

Bob Hagmann from Aleri recently posted a great piece on designing and debugging CEP apps; unfortunately, he leaves out what that process of debugging might actually look like.  While some languages force you to spelunk through printf output or trace files to track down a bug, I have found that the returns from an integrated debugger are immeasurable (and greatly magnified by good design).  StreamBase ships with a visual debugger built into its development environment, so I will expand on Bob's post with a concrete example using the StreamBase visual debugger.  Consider the following sample application that continuously calculates Bollinger Bands for stock quotes (for those playing along at home this sample ships with StreamBase and can be loaded with "File -> Load StreamBase Sample...")

 

This app actually calculates Bollinger bands in two different ways, one using a simple moving average and the other using an exponential moving average.  For simplicity, the two paths are separate branches of the main app, allowing each path to be debugged separately.  Within each path, the calculation is divided into a first step, where the moving average and standard deviation are calculated, and a second step, where the intermediate results are used to calculate the Bollinger bands.  This allows us to use the StreamBase debugger to inspect intermediate results and inspect their values.

With the app paused at a breakpoint, we can inspect the fields of the current tuple or at any earlier point in the active data flow, allowing easy inspection of transformations and invariants that need to be maintained.  The StreamBase debugger also lets the user modify the value of fields in the current frame before resuming execution, providing a simple way to test downstream components with hand crafted data.

Of course in an application this simple, one doesn't even need a debugger or careful design.  But, when debugging a larger application like a smart order routing system, organizing your code into staged transformations where each stage is a submodule can be invaluable.

This sort of design not only allows you to step into each module and debug its inner workings separately, but it also provides clean abstraction barriers for unit testing.  And once you have unit tests for each stage and a debugger to help you fix regressions, you will be free to refactor, optimize, and extend your application with impunity.