Archive for the 'Analysis (Meta)' Category
Google’s enterprise prediction markets — Using Prediction Markets to Track Information Flows: Evidence from Google
Chris F. Masse March 26th, 2008
Using Prediction Markets to Track Information Flows: Evidence from Google - (PDF file - PDF file) - by Bo Cowgill, Justin Wolfers, and Eric Zitwewitz - 2008-01-06
-
VIDEO — Bo Cowgill on Google’s enterprise prediction markets — O’Reilly Money:Tech
-
-
-
-
Using Prediction Markets to Track Information Flows: Evidence from Google - (PDF file - PDF file) - by Bo Cowgill, Justin Wolfers, and Eric Zitwewitz - 2008-01-06
-
ABSTRACT: In the last 2.5 years, Google has conducted the largest corporate experiment with prediction markets we are aware of. In this paper, we illustrate how markets can be used to study how an organization processes information. We document a number of biases in Google’s markets, most notably an optimistic bias. Newly hired employees are on the optimistic side of these markets, and optimistic biases are significantly more pronounced on days when Google stock is appreciating. We find strong correlations in trading for those who sit within a few feet of one another; social networks and work relationships also play a secondary explanatory role. The results are interesting in light of recent research on the role of optimism in entrepreneurial firms, as well as recent work on the importance of geographical and social proximity in explaining information flows in firms and markets.
-
DISCUSSION: In the past few years, many companies have experimented with prediction markets. In this paper, we analyze the largest such experiment we are aware of. We find that prices in Google’s markets closely approximated event probabilities, but did contain some biases, especially early in our sample. The most interesting of these was an optimism bias, which was more pronounced for subjects under the control of Google employees, such as would a project be completed on time or would a particular office be opened. Optimism was more present in the trading of newly hired employees, and was significantly more pronounced on and immediately following days with Google stock price appreciation. Our optimism results are interesting given the role that optimism is often thought to play in motivation and the success of entrepreneurial firms. They raise the possibility of a “stock price-optimism-performance-stock price” feedback that may be worthy of further investigation. We also examine how information and beliefs about prediction market topics move around an organization. We find a significant role for micro-geography. The trading of physically proximate employees is correlated, and only becomes correlated after the employees begin to sit near each other, suggesting a causal relationship. Work and social connections play a detectable but significantly smaller role.
An important caveat to our results is that they tell us about information flows about prediction market subjects, many of which are ancillary to employees’ main jobs. This may explain why physical proximity matters so much more than work relationships – if prediction market topics are lower-priority subjects on which to exchange information, then information exchange may require the opportunities for low-opportunity-cost communication created by physical proximity. Of course, introspection suggests that genuinely creative ideas often arise from such low-opportunity-cost communication. Google’s frequent office moves and emphasis on product innovation may provide an ideal testing ground in which to better understand the creative process.
-
PAPER BODY: In the last 4 years, many large firms have begun experimenting with internal prediction markets run among their employees. The primary goal of these markets is to generate predictions that efficiently aggregate many employees’ information and augment existing forecasting methods. [...] In this paper, we argue that in addition to making predictions, internal prediction can provide insight into how organizations process information. Prediction markets provide employees with incentives for truthful revelation and can capture changes in opinion at a much higher frequency than surveys, allowing one to track how information moves around an organization and how it responds to external events. [...]
We can draw two main conclusions. The first is that Google’s markets, while reasonably efficient, reveal some biases. During our study period, the internal markets overpriced securities tied to optimistic outcomes by 10 percentage points. The optimistic bias in Google’s markets was significantly greater on and following days when Google stock appreciated. Securities tied to extreme outcomes were underpriced by a smaller magnitude, and favorites were also overpriced slightly. These biases in prices were partly driven by the trading of newly hired employees; Google employees with longer tenure and more experience trading in the markets were better calibrated. Perhaps as a result, the pricing biases in Google’s markets declined over our sample period, suggesting that corporate prediction markets may perform better as collective experience increases.
The second conclusion is that opinions on specific topics are correlated among employees who are proximate in some sense. Physical proximity was the most important of the forms of proximity we studied. Physical proximity needed to be extremely close for it to matter. Using data on the precise latitude and longitude of employees’ offices, we found that prediction market positions were most correlated among employees sharing an office, that correlations declined with distance for employees on the same floor of a building, and that employees on different floors of the same building were no more correlated than employees in different cities.4 Google employees moved offices extremely frequently during our sample period (in the US, approximately once every 90 days), and we are able to use these office moves to show that our results are not simply the result of like-minded individuals being seated together. [...]
Our findings contribute to three quite different literatures: on the role of optimism in entrepreneurial firms, on employee communication in organizations, and on social networks and information flows among investors. [...]
The lessons of the literature informed Google CEO Eric Schmidt and Chief Economist Hal Varian’s (2005) third rule for managing knowledge workers: “Pack Them In.” Indeed, the fact that Google employees moved so frequently during our sample period suggests that considerable thought is put into optimizing physical locations. To this literature, which has largely relied on retrospective surveys to track communication, we illustrate how prediction markets can be used as high-frequency, market-incentivized surveys to track information flows in real-time. [...]
Google’s prediction markets were launched in April 2005. The [Google prediction] markets are patterned on the Iowa Electronic Markets (Berg, et. al., 2001). In Google’s terminology, a market asks a question (e.g., “how many users will Gmail have?”) that has 2-5 possible mutually exclusive and completely exhaustive answers (e.g., “Fewer than X users”, “Between X and Y”, and “More than Y”). Each answer corresponds to a security that is worth a unit of currency (called a “Gooble”) if the answer turns out to be correct (and zero otherwise). Trade is conducted via a continuous double auction in each security. As on the IEM, short selling is not allowed; traders can instead exchange a Gooble for a complete set of securities and then sell the ones they choose. Likewise, they can exchange complete set of securities for currency. There is no automated market maker, but several employees did create robotic traders that sometimes played this role.
Each calendar quarter from 2005Q2 to 2007Q3 about 25-30 different markets were created. Participants received a fresh endowment of Goobles which they could invest in securities. The markets’ questions were designed so that they could all be resolved by the end of the quarter. At the end of the quarter, Goobles were converted into raffle tickets and prizes were raffled off. The prize budget was $10,000 per quarter, or about $25-100 per active trader (depending on the number active in a particular quarter). Participation was open to active employees and some contractors and vendors; out of 6,425 employees who had a prediction market account, 1,463 placed at least one trade. [...]
Common types of markets included those forecasting demand (e.g., the number of users for a product) and internal performance (e.g., a product’s quality rating, whether a product would leave beta on time). [...]
In addition, about 30 percent of Google’s markets were so-called “fun” markets –markets on subjects of interest to its employees but with no clear connection to its business (e.g., the quality of Star Wars Episode III, gas prices, the federal funds rate). Other firms experimenting with prediction markets that we are aware of have avoided these markets, perhaps out of fear of appearing unserious. Interestingly, we find that volume in “fun” and “serious” markets are positively correlated (at the daily, weekly, and monthly frequencies), suggesting that the former might help create, rather than crowd out, liquidity for the latter. [...]
Google’s prediction markets are reasonably efficient, but did exhibit four specific biases: an overpricing of favorites, short aversion, optimism, and an underpricing of extreme outcomes. New employees and inexperienced traders appear to suffer more from these biases, and as market participants gained experience over the course of our sample period, the biases become less pronounced. [...]
-
FOOT NOTE: One trader in Google’s markets wrote a trading robot that was extremely prolific and ended up participating in about half of all trades. Many of these trades exploited arbitrage opportunities available from simultaneously selling all securities in a bundle. In order to avoid having this trader dominate the (trade-weighted) results in Table 9, we include a dummy variable to control for him or her. None of the results discussed in the above paragraph are sensitive to removing this dummy variable.
-
APPENDIX:
-
More info on Google’s enterprise prediction markets on Midas Oracle .ORG…
-
Using Prediction Markets to Track Information Flows: Evidence from Google - (PDF file - PDF file) - by Bo Cowgill, Justin Wolfers, and Eric Zitwewitz - 2008-01-06
-
Sphere: Related ContentWrap-up reviews of the 2007 Consensus Point conference on prediction markets
Chris F. Masse October 1st, 2007
—
#1. Justin Wolfers on the 2007 Consensus Point conference
Robin Hanson tells me that he is now (back to) bullish on prediction markets - he saw real evidence of real firms implementing prediction markets and taking them seriously.
#2. Jed Christiansen on the 2007 Consensus Point conference
Sphere: Related ContentRobin [Hanson] gave a fairly standard introduction to prediction markets lecture that some may have seen at other events or downloaded from his website. It was a good overview of the topic.
The question and answer period was the most interesting part with Robin. He was asked about manipulation, and provided some fairly convincing answers that manipulation shouldn’t be a worry (at least with the correct incentives.) Robin described the situation in terms of sheep and wolves. Sheep aren’t that knowledgeable; they are trading for any number of reasons, and are the “noise” in the marketplace. Wolves take advantage of that, and consequently they look for markets with lots of sheep. With better information, the wolves will easily have plenty to “eat.” The net result is that those noisy markets are accurate markets.
Another concept he talked about was creating a “fudge” account. Let’s say you want to weight one set of traders more than another, or simply want to “move” the forecast in one direction or another; create a “fudge” account to conduct those transactions. If after the account has been running for a while and it’s positive, you’ll know you’ve done a good job fudging. But if the fudge account is negative, you don’t know more than the market so just stop fudging and leave the market to itself. It’s a great idea, and fairly easy to implement.
Intel Corporation found their information aggregation mechanism to be accurate and useful.
Chris F. Masse September 17th, 2007
WARNING: Even though the Intel director uses 15 times the term “prediction markets” in this paper, the forecasting tool they have been using is another form of information aggregation mechanism.
—
The Spectrum of Risk Management in a Technology Company - Using Forecasting Markets to Manage Demand Risk - (PDF) - by Intel Corporation’s Jay W. Hopman - 2007-05-16
—
- Abstract
Intel completed a study of several generations of products to learn how product forecasts and plans are managed, how demand risks manifest themselves, and how business processes contend with, and sometimes contribute to, demand risk. The study identified one critical area prone to breakdown: the aggregation of market insight from customers. Information collected from customers and then rolled up through sales, marketing, and business planning teams is often biased, and it can lead to inaccurate forecasts, as evidenced by historical results. A research effort launched in 2005 sought to introduce new methodologies that might help crack the bias in demand signals. We worked with our academic partners to develop a new application, a form [???] of prediction market, integrated with Intel’s regular short-term forecasting processes. The process enables product and market experts to dynamically negotiate product forecasts in an environment offering anonymity and performance-based incentives. To the extent these conditions curb bias and motivate improved performance, the system should alleviate demand miscalls that have resulted in inventory surpluses or shortages in the past. Results of early experiments suggest that market-developed forecasts are meeting or beating traditional forecasts in terms of increased accuracy and decreased volatility, while responding well to demand shifts. In addition, the new process is training Intel’s experts to improve their use and interpretation of information.
[...] Tackling demand risk and other challenges requires moving information around decentralized organizations in new ways. If employees across Intel’s many functional groups have information and insights that can help inform our planning and forecasting decisions, we need a way to aggregate that information and turn it into intelligence. Prediction markets are a potential solution to this problem and have been written about extensively for the past five to ten years. Our research discovered that, despite the buzz around prediction markets, the integration of prediction markets and similar Information Aggregation Mechanisms (IAMs) into organizational forecasting processes is still in its infancy. Popular stories on prediction markets still frame the potential as being greater than the demonstrated value, and reports of usage at companies such as Hewlett Packard, Microsoft, Google, Eli Lilly, and others suggest that application is often viewed as experimental and that markets are largely separate from other organizational forecasting processes.
- Challenges to Anticipating Market Demand
[...] Decentralized organizations must find a means of transmitting business context; in other words, instead of transmitting mere data sets, they must transmit information and intelligence from employees who have it to employees who need it to make decisions and plans. We learned that Intel has many informal networks that attempt to move that knowledge across the organization, but these networks have many failure modes: turnover of employees in key positions, limited bandwidth of each individual and team, and difficulty systematically discovering the important information to be learned (stated differently, whom to include in the network). [...]
- Market Mechanisms as Forecasting Tools
[...] In our research at Intel we are extending the idea of prediction markets to create “forecasting markets,” which are essentially prediction markets or similar IAMs integrated into the company’s standard, ongoing forecasting processes. Participants reveal not just an expected outcome but a series of expected outcomes [???] for the same variable over time. So, the forecasting market captures individual and collective assessments about trends such as increasing or decreasing demand just as weather forecasts anticipate warming and cooling trends. [...] Anonymity helps prevent biases created by the presence of formal or informal power, the social norms of group interaction, and expectations of management. [...]
- Design Considerations and Elections
[...] Our overall design structures each investment as a decision based on both the individual’s expectations for the outcome and the aggregate group prediction. Participants weigh owning lower percentages of more likely outcomes against higher percentages of less likely outcomes. [...]
- Results
We are using three primary measures to assess the performance of our markets: accuracy, stability, and timely response to genuine demand shifts. Having run pilot markets for approximately 18 months, we are starting to get a sense for how the markets are performing. Although the market forecasts and official company forecasts are not independent, it is nonetheless interesting to compare the signals and then assess how effectively they are working together. In terms of accuracy, the markets are producing forecasts at least the equal of the official figures and as much as 20% better (20% less error), an impressive result given that the official forecasts have set a rather high standard during this time period with errors of only a few percent. In the longest sample to date, six of eight market forecasts fell within 2.7% of actual sales. The accuracy of the official and market forecasts has been remarkably good, well within the stated goal of +/- 5% error for all but a few individual monthly forecasts. [...] We are also amused that although we never publish the list of participants and winners, everyone knows who participated and who won. [...]
[...] As we propose market mechanisms to aid with forecasting, potential participants and managers have most often expressed three concerns: incentives, anonymity, and groupthink. [...]
[...] The key drivers that we believe have led to strong performance are 1) anonymity and incentives, which encourage honest, unbiased information, 2) the averaging of multiple opinions, which produces smooth, accurate signals, and 3) feedback, which enables participants to evaluate past performance and learn how to weigh information and produce better forecasts. [...] [Prediction markets] are a new approach toward business management, promising, and at the same time frightening to potential adopters. As with many such innovations, starting small and running in parallel to existing processes are keys to success. As our trials are demonstrating excellent results at remarkably low cost, expanding their use at Intel is a natural and expected outcome.
- Sidebar: Five Categories of Considerations for Designing Information Aggregation Mechanisms
Sphere: Related ContentInformation - Integration - Inclusion - Interface - Incentives
