Data and privacy, social network analysis, sensors, wearable computing, and location intelligence: Name a technological trend that’s revolutionizing commerce and society, and Sandy Pentland and MIT’s Media Lab were likely doing it before anyone else was even writing about it.
For a lifelong technology enthusiast like me, visiting MIT’s fabled Media Lab is tantamount to touring Willy Wonka’s chocolate factory. Five years ago, when I was first invited to visit Sandy Pentland’s Human Dynamics Laboratory at the Media Lab, I couldn’t help but hum a few lines of I’ve Got a Golden Ticket in a quiet fit of geeky jubilation.
Soon after I arrived, Sandy showed me a film of office workers interacting with each other. There’s nothing exceptional about that, except for one peculiarity: They were all wearing badges with tiny cameras around their necks. These “sociometric badges” allowed Pentland and his team of braniacs to convert millions of face-to-face interactions into data that can be analyzed to improve group communication.
The second time I called on Sandy a few years later, he projected onto the wall a giant map of San Francisco that showed thousands of tiny dots moving around in real time. And yes, these “dots” were actually cell phone signals. By tracking each unique signal over a period of weeks, his team could identify which of dozens of different “tribes” tens of thousands of different San Franciscans belonged to, and they could fairly accurately predict each of their future movements. It was impressive, and it was also a wee bit unnerving.
One of the world’s most quoted computer scientists, Sandy Pentland has long been at the intellectual center of many of society’s biggest technology issues: data and privacy, social network analysis, sensors, wearable computing, location intelligence, you name it. Name a major technological trend that’s revolutionizing commerce and society, and there’s a good chance that Pentland and his lab were doing it before most anyone else was even writing about it.
Sandy’s new book, Social physics: How good ideas spread—the lessons from a new science, lays out his ideas for a broader audience—the masses, so to speak. On a series of visits to his computer lab, I had the distinct pleasure of picking Pentland’s brain.
Bill Eggers: You call your work “reality mining.” What is it, and how does it differ from data mining?
Sandy Pentland: In traditional social science, you ask people questions using surveys. Typically, this just gets you responses that are socially acceptable but do not really reflect reality. Similarly, scientists talk about mining Twitter feeds and Facebook, but that’s really just mining the socially constructed version of you. Reality mining is about what you actually do; it’s not about how you imagine yourself. It uses the digital breadcrumbs left behind by cell phones and credit cards to quantify your life. For instance, how long is your commute? You may think it’s an hour when it’s really only 45 minutes—it just seems longer because you hate it. The bottom line is that the data used in reality mining is more valuable because it’s how you actually live your life. The name is also a little scary, and that’s on purpose. People generally don’t realize how many breadcrumbs they leave behind in daily life.
Bill: Aren’t there instances where credit card data or cell phone data can reveal someone’s health better than doctors’ visits?
Sandy: Well, maybe not better than doctors’ visits, but it’s remarkable what you can deduce. We evolved as a social species, and so we have a talent for looking at somebody and “reading” them. If someone is unhealthy, then how they act and how they respond tells you that they’re coming down with something. You can see similar cues from cell phone data, from purchasing data, and from movement data. It’s not better than a doctor, but because it knows your habits, it can read when you are acting a bit off.
“We’ve had social science for centuries, but the science was historically limited to things you could measure in a lab, collect on questionnaires, etc. With the advent of big data and machine learning, researchers actually have enough data and sufficient mathematical tools to build predictive mathematical models.”
Bill: Your work is based on a mathematical model, but isn’t the behavior of social networks very unpredictable? Can you really use math to predict things like financial bubbles or the Arab Spring?
Sandy: We’ve had social science for centuries, but the science was historically limited to things you could measure in a lab, collect on questionnaires, etc. With the advent of big data and machine learning, researchers actually have enough data and sufficient mathematical tools to build predictive mathematical models. One example is our research with eToro, a financial social network where we found we can identify when people are making good or poor decisions based on the mathematics of idea flow in their social networks.
Here’s the common-sense version of this research: If you talk to other people and see what they are doing, you can improve your own performance, and as you talk to more and more people, you continue to do better and better. For instance, social learning on eToro’s trading network can improve the user’s return on investment by about 30 percent, which is huge. And this has been tested over millions of people, several years, and tens of millions of trades.
However, this social learning effect can lead you astray. When people get a little too enthusiastic, they form echo chambers. The idea flow becomes a circle, swamping any new strategies people are experimenting with and leaving just a few strategies to circulate. As a consequence, everybody adopts the same set of strategies. But the world is constantly changing, so even the best strategies go stale very quickly, and everybody in the echo chamber loses. That pretty much defines what it means to be a bubble.
By examining the flow of ideas between people, we can detect and even deflate these echo chambers. So, the answer is, yes, you can predict things like bubbles and panics. In fact, we are beginning to help a number of financial institutions avoid bubbles as a commercial service. It works.
Bill: Relatedly, you found that exposure to the behavior of peers, even indirectly, predicts idea flow and shapes opinions and habits better than anything else—economic incentives included. So if you hang around people who take a second slice of pizza, you will too. Was this surprising? What are the implications?
Sandy: Social learning is amazingly powerful. If you show up at a new job where everybody wears a tie, then you are going to wear a tie. It’s part of being compatible with the group. Similarly, if your new job has a bunch of people that are big risk takers, you will move toward more risky behavior. So while group learning can produce echo chambers, more often, exposure to other people’s experiences enhances your thinking process, as in the eToro example. So social learning—learning from observing peers—generally improves your own decisions. It’s not surprising at all: In fact, you also see it in ape troops. It just makes sense to learn from the people around you. It makes so much sense that most of it happens almost unconsciously.
Bill: Does that mean the best way to alter negative behavior is to get a new group of peers? Or can incentives or exposure from outside your peer group alter network dynamics?
Sandy: Peer behaviors are usually the strongest influences. This is because you don’t choose your habits. Your community does. When several peers play with a new behavior and it seems to be working, you are extremely likely to pick it up without even thinking about it.
You can harness this social mechanism by using social network incentives to get a group to experiment with new behaviors. Instead of asking individuals to change by themselves, you can incentivize them to encourage other people to try new behaviors. If we all want to lose weight, then giving everyone in the group a few bucks to help their buddies lose weight becomes fun and interesting. Social network incentives can cause new ideas to enter the community in sufficient numbers and with sufficient force that people almost automatically adopt the new behavior.
Bill: You also wrote about the relationship between wealth and social exploration patterns. Wealthier people tend to explore much more outside their immediate social circles. On the other hand, low-income individuals are much more stuck in routines, leading to negative effects. So what can you do if you are not already wealthy?
Sandy: As people become wealthy, they have more opportunities to explore and return with new ideas. Exploration benefits an entire group more than it seems to benefit an individual. So in a community that faces limitations from economic constraints, exploration has to take different forms. Individuals could explore each other’s family traditions, for instance, a process sometimes known as finding your roots.
Research strongly suggests that both greater levels of engagement within a community and exploration outside a community benefit the community over time. As people begin to communicate, they begin to establish compatible ways of behaving. Infant mortality declines, GDP goes up, life expectancy goes up, and crime goes down. But if a community limits conversation to within its membership, they only perpetuate behaviors already in the community, which may not be very adaptive. By exploring what other communities are doing, you can begin to incorporate some of those new behaviors.
A death spiral happens when everybody is so disconnected that they give up on the community and begin to exploit each other. In those instances, the more you interact with people, the more you are convinced that people are going to exploit you. Individuals and communities require a certain amount of social capital or positive attitude to climb up the ladder.
So what can you do in a community devoid of social capital or trust? As an individual, you can go somewhere else where people have a cooperative attitude. Here at MIT, it is common to hear, “You guys don’t know how good you have it, because everybody here wants you to succeed. Back where I come from,” they say, “people are not trying to be helpful. It’s like crabs in a box. If one of the crabs tries to escape, the other crabs pull it back.” You need to have a community that has somehow reached a positive place where engagement and exploration build trust. But if you are not there, you have to reset the community.
Obesity and social network incentives
Bill: What does your work tell us about how obesity spreads, and how might we be able to use social physics to address it?
Sandy: I found that a lot of the habits that result in obesity are picked up by exposure. They are not particularly conscious habits, so they are very, very hard to change consciously. It’s difficult to catch yourself when everyone around you is taking extra slices of pizza.
Obesity is a hard one. Smoking was also considered a hard one until social pressure turned against smoking. By banning it within buildings and making people stand outside, communities altered the conditions. Everyone would walk by and look at smokers like they were freaks. The social pressure to stop increased, and smoking dropped off dramatically.
The social physics answer is not to directly incentivize people to change themselves, but to create social pressure for change. In a small community of young families, for instance, we used this approach to encourage being active. It worked surprisingly well.
Bill: With 35 percent of adults in America obese, does tackling obesity require regulatory and statutory changes?
Sandy: I don’t think so. Imagine that everyone in your workgroup could get a hundred bucks off of health insurance if they lost five pounds, but the catch was that everybody had to lose five pounds. It would be a little uncomfortable, but people would end up talking to each other and even helping each other, so my expectation is that people would lose those five pounds. It wouldn’t have to be an obnoxious, government-imposed thing. It’s really about getting people to work together around a goal, like in The Biggest Loser. People compete, but they also support each other. They see other people succeed and realize it’s possible. They transform, and it starts with social pressure and social learning.
Social network incentives versus market incentives
Bill: You are pretty tough on market and economic incentives in your book, positing that social incentives are far superior for behavioral change. Is it really an either-or? Couldn’t you use a combination of the two?
Sandy: It is not either-or, but our math shows that social incentives can be at least twice as efficient in a typical situation under fairly general assumptions. Twice as efficient is huge, and in the experiments, we get numbers that are more like eight times more efficient.
Bill: More efficient than economic incentives?
Sandy: Correct, for behavior change. A lot of literature says economic incentives are often counterproductive. Beyond people living at starvation wages, economic incentives often don’t work the way people think.
In fact, we have an assumption in our society about how markets automatically distribute things optimally. But Adam Smith himself said that it’s the social interactions interacting with the market that cause equitable distribution. In his view, the invisible hand suggests that the exchanges between people establish the norms that divide market opportunity. So it is the social exchanges that are the main actor, not the market.
In our society, efficient allocation generally comes from the regulator or market maker setting rules that they believe are efficient and not from the market itself. Despite a number of Nobel Prizes around market efficiency, the math says that it is very difficult to achieve efficient allocation using markets. So despite having almost a religion around markets as mechanisms for equitable distribution, it’s not generally true. We have ignored the fact that there are constant exchanges going on between people and that, very often, the most powerful and important kind of economic incentive comes from peer-to-peer exchanges, rather than from markets.
Bill: Researchers studied Indian reservations after casinos came in and families received $6,000 or $9,000 each. The children did much better because there was less stress in the households. Similar results have occurred in developing countries, leading social scientists to believe such cash economic incentives can have a bigger impact than a lot of the social work that has been done.
Sandy: Yes, I believe that. You need a certain amount of money to be able to eat, and if you can’t eat, that’s stressful. But let’s imagine a group of people that are beyond those minimal levels of living. Now, if I ask them to exercise more, cash incentives almost uniformly fail. Evidence suggests that social incentives, when applied correctly, are far more successful.
This is similar to microcredit loans. It works because entrepreneurs sign up as a group. It works because you have to sign up with four buddies, and your buddies encourage you to pay the loan back because then they get a small reward. Many traditional social structures that enable people to improve themselves or help society depend on similar social connections. I am not saying that economics is wrong or that cash incentives are wrong, but at the very birth of economics, it was recognized that the peer-to-peer exchanges are more important. We have forgotten that in our policy work today.
“In social physics, we use mathematical tools similar to the tools used in economics. In economics, you can measure people’s price sensitivity. But we can do that, plus ask how responsive people are to social incentives. As a consequence, you can design a campaign to change behavior that includes social considerations.”
Bill: You say that social physics can dramatically improve policymaking by making it more scientific and evidence based. Can you explain?
Sandy: In many business situations, you can apply the lesson from our eToro experiment. You can ask mathematically, “Have I explored diverse enough opinions, or am I trapped in an echo chamber?”
You ask that by studying the links between different opinions. For instance, auditing produces cases. Each case refers to another case, which refers to another. By examining search history, you can tell if the audit team looked at a sufficiently diverse set of ideas. You can analyze an organization’s social network and literally ask whether it is diverse enough to produce good decisions.
You can also enable better patterns of idea flow within the group, through better patterns of exploration and engagement. You don’t even have to be mathematical about it; just having employees being aware of idea flow tends to produce better outcomes.
In social physics, we use mathematical tools similar to the tools used in economics. In economics, you can measure people’s price sensitivity. But we can do that, plus ask how responsive people are to social incentives. As a consequence, you can design a campaign to change behavior that includes social considerations. In marketing, by measuring a person’s exposure to a new product, you can then calculate how likely they are to adopt that product, and you can calculate the optimal incentive to get them to tip. So this whole new set of tools are like economic tools, but they also account for the social fabric, and so they are much more powerful.
Social physics in companies
Bill: You note that organizational charts don’t really tell you very much about how an organization actually works compared to idea flows and social network analysis. What are some of the implications for organizations that want to redo their charts and use social physics?
Sandy: You really have to distribute leadership, and that distributed leadership needs to improve communication with the entire organization. Organizations have goals, and they have to create an internal plan in a way that is compatible with the other parts of the organization that touch or affect what they do. It’s like the planning process that a leader or a management team might do, but it is done in a much more peer-to-peer way.
Bill: Certain types of organizations seem to distribute leadership better than others. Technology companies come to mind.
Sandy: Tech companies are naturally that way up to a certain size because they have that startup-type culture. It’s not like people are coming in and punching the clocks. As the company gets older, it gets more ossified and siloed. Management complains, but they’re afraid of letting employees experiment. Newer companies tend to be a lot more agile because they rely much more on peer-to-peer communication, on people actually setting behavioral norms, and on peer-to-peer allocation of processes and bonuses.
Bill: How far away from optimally distributed models are the average organizations that you’ve worked with?
Sandy: Many of them are not too bad, but these tend to be special or different, like a drug discovery unit or a creative unit. Management leaves them alone to a degree. They say, “What you do is special. Do it your way, and we will just check in with you at the end of the day.”
The places where social physics have been most transformative have been in places like call centers, where management believes they understand their employees’ tasks, and as a result, everybody acts like a little robot. But call center work is actually very creative at a microscale, with a lot of practical knowledge that will never appear in the operating manual. You need to trade tips and know the reasoning behind each part of the job, and that requires communication. When we helped renovate a call center by changing the patterns of communication, there was a dramatic improvement not only in the calls themselves, but also in employee retention.
Bill: You shifted how people structured their coffee breaks and saw a big impact. Can you explain?
Sandy: It opened up paths of communication. It’s just that simple. The call center was previously managed in such a way that people were not supposed to talk to each other, which meant they couldn’t trade practical knowledge like what you do when the boss is being irrational, what you do with a new campaign that doesn’t work, or what you do when the babysitter is going to be late. In order to promote sharing of this type of knowledge, I convinced management to give people shared coffee breaks.
Bill: And that had a huge impact on productivity.
Sandy: Yes, an absolutely huge impact. In another company, we made the lunch tables longer so that you had to sit with other people. This led to new social ties, which enabled improved intergroup communication, which then made the team much more resilient to issues that would have otherwise disturbed the organization.
Bill: Your work with organizations involves some large-scale experiments with real people you’ve equipped with sociometric badges, mobile devices, or other technologies that allow you to track their behavior. Then you aggregate results using big data techniques. How did you develop this approach, and why?
Sandy: In the early 1990s, everyone was talking about pervasive computing in buildings. But I thought that computing would move to the body, so I created the first race of cyborgs and built wearable computers just to test what such a life would be like. We had 20 students running around with the precursors to Google Glass and modern cell phones, and these same students later developed Google Glass and other wearable computing devices. From this experiment, we discovered a couple of things: You can actually tell when people are healthy or sick by their behavior. You can also deduce many characteristics from a social interaction—whether somebody is fearful or excited. And we realized that you could judge whether, for instance, a meeting was a good meeting. Good meetings have a certain energy to them, and it turns out you can pick that up.
Big Data and privacy
Bill: Your work for the World Economic Forum focuses on ways we can have more control over our personal data. What’s your take on government, surveillance, and privacy?
Sandy: My work on personal data relates to trust. We notice that people will share data if they trust that recipients will use the data appropriately. In today’s system, it’s difficult to trust that people aren’t using it for bad things. As a result, people don’t want to share data. This impedes collecting important information—which drug treatments work, for example, or facts about community welfare—because people won’t share the data.
To improve sharing, we built encrypted computer networks call trust networks that are rather like the networks that banks use to transfer money. Trust networks allow people to control where their data goes and what it is used for. Unsurprisingly, when people have the option to use such trusted networks, they are open to greater sharing of data.
While trust networks give the intelligence community problems, the homeland security people like them. They’re worried about building a resilient infrastructure that resists cyber-attacks. They want a system where hackers can’t bring down the power grid or the public health system. And guess what, trust networks also provide them with a good solution, through encryption and personal control of data. To build resilient homeland security is to build trusted exchanges. It is going to make some intelligence agencies’ life harder, but I don’t feel too much sympathy for them.
Bill: You’ve called for a “New Deal on Data” in which control of personal data collected by private companies has to be shared between consumers and companies.
Sandy: My New Deal on Data proposal has been quite successful in helping form the Consumer Privacy Bill of Rights here in the United States and the Data Protection Acts in the European Union. I feel good about that. It establishes ownership and control over data in a way that lets consumers have greater trust in sharing data. It permits companies to monetize personal data, but it requires informed consent. Companies have to engage with customers: describe the data, and describe what they intend to do with it. They also have to allow the customer to back out if they are unhappy with the deal. The deal also gives companies clear license to aggregate data, which frees up companies to utilize the aggregate data they’ve compiled. Frankly, I think the aggregate data is the data that really matters, not so much the personal data.
2020 and beyond
Bill: If you look out toward the future, say 2020, how do you see social physics playing out?
Sandy: We are already seeing people designing cities, campuses, and corporations to improve the idea flow. The science is in early days, but it’s growing. The designs work; they engender innovative, productive organizations and cities.
Social incentives are permitting peer-to-peer organization. We are going to see much more agile organizations with much more distributed power. Consider, for example, the Cambridge Innovation Center, which has 400 little companies in it. The ecosystem of expertise is amazing, and new technical ideas flow easily across company boundaries. Silicon Valley is like that, too. People change companies all the time, so you get this creative churn, and, as a consequence, ideas flow more fluidly than in most places. As a result of these social physics ideas, I think you are going to see large organizations becoming much more entrepreneurial. Expect more distributed responsibility where everyone helps invent the future.