Wednesday, December 22, 2010

Using Twitter to Predict the Stock Market

Yes, we do this in our Fund.  And we sell this data to quants at

My inbox has been filling up with articles about using Twitter moods to predict the stock market and this New York Times overview.  This strategy is effective for trading.  We know because we've been using these techniques in our own Fund:

Most revolutionary is that we can predict collective human behavior by monitoring conversations and expressions.  That's ground-breaking in itself and speaks to a forthcoming transformation of economic forecasting.  We've already seen psychologists and behavioral economists advising economic policy makers.

The applications of this go much beyond stock market forecasting. You can also predict consumer behavior - shopping and spending and risk-taking in general. And of interest to corporations, you can use mood data to predict what types of products comsumers are willing to buy (and thus what they should display).

For our fund we scan text data including news media, social media (chat rooms, blogs, and twitter), SEC filings, and executive commentary (interviews and earnings conference calls).  In all we track 240 psychological variables in various forms of granularity.  We've got data back to 1997 for social media, and we've been trading these strategies since we launched our Fund (the MarketPsy Long-Short Fund LP) on September 2, 2008.  And we have a great track record with low volatility.

Lest you doubt that this works, consider a more familiar scenario:  You had a bad day at work.  When you get home, everyone there seems so inappropriately upbeat.  So you maybe sulk a little, slam a door, say something rude.  You're acting out your unconscious emotions, and by definition you don't even know it.  How you were feeling - your irritation at work - changed your behavior.  And in the markets, little bits of news and information change collective behavior all the time.  My book Inside the Investor's Brain discusses this phenomenon in detail, and more for the layperson is our book MarketPsych:  How to Mange Fear and Build You Investor Identity.

We've got more information about our Fund available to accredited investors.   Also please see our charting service at

Best wishes,
Richard L Peterson MD
richardpeterson - AT -

Wednesday, December 08, 2010

The MarketPsych Bubble-ometer: How To Know When There is a Market Bubble

I read a recent WSJ article on predicting bubbles.  Turns out there are many forecasting techniques from psychoanalysis to econometrics, but none is widely accepted.  I'd like to throw my hat in the ring here.

We've got our own technique for predicting bubbles, which naturally I think is the best :) 

As you may know, we're monitoring online conversations about stocks, and we're able to parse what people are talking about and their sentiment.

Here are our assumptions: 
1)  When investors (traders) are speculating, they are aiming for capital gains.  They want an appreciation in the value of the shares (or decline if they're a short-seller).  This focus on gains and losses is as true of "day-traders" as it is of "house-flippers."  Such speculators talk about stock prices and their likelihood of going up or down.
2)  When investors are investing, they are concerned with a business' viability, valuation, and its income - dividends, rents, cash flows, earnings, etc...  Investors talk about accounting issues and fundamentals.

Based on the dominant themes in online conversation (price speculation versus fundamentals-focus) we've created an index (the MarketPsych Bubble-ometer) to see which of these two groups is dominant in the markets.  An image of this index for the U.S. stock market is at the top.   It shows that based on this measure we are not currently in a speculative bubble for U.S. equities.

We've also got Bubble-ometers for individual market sectors and industries.  In a few weeks we'll post charts for the most and least-bubbly sectors and industries.  (Gold, anyone?)

If you avoid a market during a bubble, you can rest assured it will eventually fall below the current price. If you know a market is in a bubble, you may want to buy the momentum, betting on further inflation of prices (and your wallet).  It's a dangerous game, along the lines of the greater fool theory.

Happy Investing!

Richard Peterson
rpeterson - AT -

Tuesday, December 07, 2010

Declining Trust in the U.S. Financial Markets

Friedrich Nietzsche

Below I've posted a chart of the levels of trust expressed by investors chatting in online financial social media since 1998.  As you can see, the level of trust has dropped 33% since 1998 in a series of steps.  It has not made a real attempt at recovery.

Economic studies have found that levels of interpersonal trust in a society are related to rates of economic growth.  (Others are less confirmatory).  In any case, it's fairly intuitive that we are less likely to take business risk if we don't think we can trust the person we are entrusting to grow, manage, or invest our money.  So the decline in trust we've measured is problematic for the future prosperity of the U.S. 

So the next question is, how can we restore trust in the financial system? 
1.  First of all, we've got to stop the bleeding.  We need fewer Madoff's, Jeff Skilling's, et al through better oversight and regulation of the financial system.  The Dodd-Frank bill addresses some of the relevant areas. 
2.  We've got to change incentives for economic players to violate trust.  For now, most financial firms use an annual bonus system.  This system incentivizes taking high short term risks with OPM (other people's money) in order to get a higher annual bonus for the individual risk taker.  This incentive system should morph into a longer term, stake-driven protocal.  Otherwise we'll bubbles rising again and again fueled by individual incentives.  Consider how many people have been punished for making bad bets during the financial crisis?  (None as far as I'm aware, and they've all kept their nice bonuses from pre-2008).
3.  We've got to punish breaches of trust (legal enforcement).  When the legal system enforces arbitrary, irrelevant, or outdated laws, or when it hands out punishments that don't do justice to the actual suffering caused, it loses credibility.  Recall that Madoff remained free for many months before his trial while the case was prepared.  To most victims, that appeared as a failure of justice. 
4.  A shared social mission.  Seeing others selflessly work towards a goal that benefits others restores trust.  Historically major wars and religious revivals served this function, but it's also possible through leadership (Roosevelt with the WPA and Reagan's "morning in America" were able to create such a sense in much of the population).  Perhaps shared sacrifice towards cutting the budget, educational reform, or healthcare overhaul could do it?  I don't know - once distrust is used as a political tool, then we're in a different stage of the game.

Certainly, there is a social devaluation of trust that is seen in recent politics.  This may be a by-product of new forms of communication and media.  When Elliot Spitzer becomes the host his own show on CNN after prosecuting many financial leaders as attorney general of NY (allegedly unfairly) and then being caught breaking the law systematically himself while Governor of NY, what does that say about our assessment of character versus desire for entertainment? 

Whatever the solution, I imagine we'll know the harbingers of returning trust when we feel them.  For now, the erosion will likely continue as more and more municipalities, states, and sovereigns come looking for debt relief, and mistrustful investors balk at bailing them out.

Richard Peterson M.D.
rpeterson "at'