Balancing big data analytics with intuition

Sure, Big Data Is Great. But So Is Intuition. With the growing potential of data and analytics, where is the shifting line between analytics and intuition?

[Reproduced from MIT Sloan management Review]
Analytics and Intuition: Finding Equilibrium
[By Renee Boucher Ferguson 03.05.13]

As the practice of using data analytics to make organizational decisions grows, where is the line between analytics and intuition? Is there a perfect balance between experience versus data, or data versus experience?

With the growing potential of data and analytics, how should managers balance analytics and intuition?

Just before the New Year, New York Times writer Steve Lohr wrote a blog post, Sure, Big Data Is Great. But So Is Intuition, which addresses a question we here at MIT Sloan Management Reviewhave been researching for the past year: With the growing potential of data and analytics, where is the shifting line between analytics and intuition?

In other words, is there a “correct” balance between analytics and intuition in making good business decisions?

Lohr bases his post on comments made by a number of speakers who presented at a recent MIT Center for Digital Business conference, Big Data: The Management Revolution, including Claudia Perlich, chief scientist at Media6Degrees and Rachel Schutt, a senior statistician at Google Research. His central premise:

Personally, my…concern is that the algorithms that are shaping my digital world are too simple-minded, rather than too smart.

It’s encouraging that thoughtful data scientists like Ms. Perlich and Ms. Schutt recognize the limits and shortcomings of the Big Data technology that they are building. Listening to the data is important, they say, but so is experience and intuition. After all, what is intuition at its best but large amounts of data of all kinds filtered through a human brain rather than a math model?

As I read Lohr’s article, I couldn’t help but think of our own findings around the theme of analytics versus intuition.

In conjunction with our corporate partner SAS, we have conducted research into how organizations are using data and analytics to both compete and innovate. As part of that research, we asked 2,500 survey respondents to rate the extent to which they rely on intuition versus analytics to make decisions in a number of areas, from enhancing customer experience to allocating annual budgets.

We found that while organizations have access to more useful data than they did a year ago — another finding from our survey — the extent to which managers are using that data to make business decisions varies greatly (see “Addressing Analytics Equilibrium”).

ADDRESSING ANALYTICS EQUILIBRIUM

Not surprisingly, many respondents said that for traditionally quantitative functions — establishing financial forecasts, managing supply chain or logistics, and allocating annual budgets — they use “mostly or purely” analytics.

In contrast, only a third of respondents said they rely more on data and analytics (rather than intuition and experience) to address issues connected with organizational risk, new product development, marketing campaigns, and identifying strategic objectives. And less than a third of our respondents said they rely on data and analytics to address the “softer” processes: enhancing customer experience, assessing employee performance and matching sales representatives with customers.

We also found is that there is an interesting dynamic playing out among organizations across the globe: a struggle to find an analytics equilibrium — a balance between the use of data and intuition to make important business decisions.
In our interviews with executives from across industries, the balance issue elicited several points — potential roadblocks, really — to finding analytics equilibrium. These points suggest how complicated the process is at the organizational level, not to mention among individual managers:

·         Culture Clash: In some cases, managers have different perspectives and experiences with data and analytics. These differences create organizational tension. A top-down approach — a CEO who understands and values data — may be necessary to reduce this tension.

·         Data Hiders: Executives often win their positions based on experience, skill and knowhow. Not only do they keep this intellectual property close to the vest, they often rely on it in place of data.

·         Data Mistrusters: An older generation of executives — those not brought up on the Internet — are often distrusting of data. They require education about the benefits of utilizing (or at least considering) data to make decisions.

But here’s what we’ve found: In those situations where there is reliable data to utilize, and a culture that embraces analytics, the analytics equilibrium is skewing towards data. In our qualitative research, for example, we spoke with data scientists from such “digital native” companies as LinkedIn, PayPal and Match.com. These organizations overwhelmingly rely on data to make major strategic decisions, from new products and services to mergers and acquisitions.

We believe that there is no right way to apply the “correct” percentage of analytics over intuition — or even that there is a correct percentage. What counts as an appropriate analytics equilibrium can vary, depending on a number of different factors including the skill level and experience of those utilizing data, the prevalence of data usage in the organization, the data goals of the organization — and, of course, the data culture.

The key lies in finding what is right for your organization, with this thought in mind: While an analytics equilibrium is determined, in part, by the individual manager’s comfort level and experience with data, the organizational environment plays a significant role as well.

That was Rachel Schutt’s point at the MIT conference: “I don’t worship the machine,” she said, even though as a data scientist, Schutt lives and breathes analytics. Her prescription for a good data scientist, as Lohr points out, is “someone who has a deep, wide-ranging curiosity, is innovative, and is guided by experience as well as data.”