Where Good Ideas Come From
Charles Duhigg says that habits will take you to the next level. But he isn't telling you the whole story. Welcome to Exhibit A in the case against habit.
At the height of the company’s expansion, in the mid-2000s, Starbucks had a problem. By 2007, they were opening, on average, seven new stores per day. These stores required employees, and the company was hiring up to fifteen-hundred new ones each week.
For lots of these new workers, Starbucks was their first job. Many of them were fresh from a high school classroom. It’s a tough transition. An eight hour shift serving hot drinks to over-caffeinated consumers is not a uniformly gratifying experience. There were certain skills they needed to learn. Most of them had to do with the graceful reception of verbal abuse from angry customers.
As a solution, Starbucks developed a series of employee training programs. Under this new regiment, employees would spend up to fifty hours in Starbucks classrooms during their first year alone. They would complete take-home exercises, filling out page after page of a company-wide workbook.
What did they learn? Strategies like the LATTE method. When a customer presents an issue to an employee, the appropriate response is to: Listen, Acknowledge, Take action, Thank the customer for identifying the problem, and Explain why it occurred in the first place. The exercises were designed to cultivate self-discipline, the ability to keep cool under pressure. The training served the same purpose for baristas behind the bar that Rocky Balboa’s did in the boxing ring. They know they’re going to take a pummeling; it’s all about having a plan for what to do in response.
Starbucks had on their hands a tricky problem of alignment. The company’s goal was that a customer should be able to enter any one of their tens of thousands of stores around the world and have approximately the same quality of experience. They needed to align this high-end homogeneity with the messy heterogeneity of human behavior. And there’s only one way to make that happen. Habit.
This is the argument made by Charles Duhigg in Chapter 5 of his best-selling book, The Power of Habit. Duhigg looks at this anecdote and sees a story about how far habit can take you or your company. Starbucks, he writes, “spent millions of dollars developing curriculums to train employees on self-discipline. Executives wrote workbooks that, in effect, serve as guides to how to make willpower a habit in workers’ lives. Those curriculums are, in part, why Starbucks has grown from a sleepy Seattle company into a behemoth with more than seventeen thousand stores and revenues of more than $10 billion a year.” What the whole case study boils down to is, in Duhigg’s words, that “Starbucks taught their employees how to handle moments of adversity by giving them willpower habit loops.”
The overall thrust of Duhigg’s book is that “there’s nothing you can’t do if you get the habits right.” He is by no means the first or only one to make this claim. In his blockbuster Atomic Habits, James Clear writes that “Success is the product of daily habits—not once-in-a-lifetime transformations.” For Clear, the habitual mode is something to be celebrated, an occasion for giddy exultation. “The only way to become excellent,” he writes, “is to be endlessly fascinated by doing the same thing over and over. You have to fall in love with boredom.” Operating from a similar premise, Cal Newport, in Deep Work, views the culmination of his enterprise as “transforming your work habits” to reflect the principles outlined in his book. Don’t even get me started on Stephen Covey's Seven Habits of Highly Effective People. The name says it all, as if “effective people” were mere bundles of habitually executed actions. Together, millions of copies of these books are sold each year. If we are what we repeatedly do, then writers of self-help books are a promise to improve one’s habits.
But these authors aren’t the only ones who are smitten with habit. William James, the father of experimental psychology, came to this conclusion long before the modern self-helpers, writing in 1887 that “we must make automatic and habitual, as early as possible, as many useful actions as we can, and guard against the growing into ways that are likely to be disadvantageous to us, as we should guard against the plague.”
Pop quiz: Which James—William or Clear—wrote that “the quality of our lives often depends on the quality of our habits”? The line was penned by the latter. But the former would probably have agreed.
Our culture venerates habit. And in an important sense, people like Charles Duhigg are right. There is power in habit. It’s absolutely true that the most efficient way to accomplish any particular task is to consolidate optimal actions into habits. This is what makes habit so seductive. Ideal habits imply frictionless productivity. It is the cultivation of a mental environment in which thinking about what you are doing never gets in the way of actually doing it. There is no doubt that habits are important, and that good ones get you further than bad ones.
But also there’s something off about this argument. It seems strange: the idea that, as Duhigg and his contemporaries argue, success is a simple function of our habits. After all, the ultimate habitual agent is not a human. It’s a machine. It would take as input those tasks it needs to do. Then it would neatly, without contemplation or complaint, execute the most efficient set of actions. A machine like that could churn out caramel macchiatos all day, no questions asked.
C’mon, Mr Duhigg. Does that sound like success to you?
In general, psychologists divide personality into five categories: conscientiousness, neuroticism, openness to experience, extraversion, and agreeableness. These are known as the “Big Five” inventory of traits. Together, these traits make for a rough description of an individual’s personality. In general, the Big Five inventory is considered by psychologists to be a useful and legitimate construct—as opposed to, say, the Myers-Briggs test, which is totally meaningless.
The most thoroughly studied of all these traits is conscientiousness. Here’s the definition of conscientiousness on Wikipedia. I think it sums it up pretty well. Conscientiousness is:
the personality trait of being careful, or diligent. Conscientiousness implies a desire to do a task well, and to take obligations to others seriously. Conscientious people tend to be efficient and organized as opposed to easy-going and disorderly. They exhibit planned rather than spontaneous behavior; and they are generally dependable. It is manifested in characteristic behaviors such as being neat, and systematic; also including such elements as carefulness, thoroughness, and deliberation (the tendency to think before acting).
Study after study suggests that this dimension of personality is a key predictor of important personal outcomes. For example, a recent meta-analysis of studies across one hundred years showed that conscientiousness is the most powerful non-cognitive predictor—as opposed to cognitive predictors, such as IQ—of positive outcomes in the workplace.
This meta-analysis combined the results of over 2,500 individual studies, which together included 1.1 million participants. And what did the authors of this study find? In the 175 work-performance categories they looked at, they found that higher conscientiousness scores predicted better job performance outcomes 98% of the time. They spend the majority of their paper breaking down the different aspects of job performance that conscientiousness affects. But the overall point is clear. Conscientiousness is key.
Now, this is precisely in line with Charles Duhigg’s argument. Did you notice the first category on the left-hand column of the figure? Study habits. It’s also the strongest correlation of all those categories (besides whatever “commendable behavior” is, shown at the bottom). Conscientiousness is effectively the personality trait that measures our ability to form and stick with good habits.
Following that line of reasoning, Starbucks’ program to develop the “habit of self-discipline” is an exercise in conscientiousness. Diligent? Efficient and organized? Planned rather than spontaneous behavior? Replace “conscientiousness” with “an ideal barista” in the Wikipedia definition and the paragraph still makes sense. These are exactly the muscles of conscientiousness which that program is trying to strengthen.
Okay, then. Score one for the ‘higg. That’s the #PowerOfHabit. Right?
Here’s the part of the story Charles Duhigg doesn’t tell.
In 1983, before the global proliferation of Starbucks, one of their employees took a trip. Starbucks was then a small regional coffee roaster, with four locations in the Seattle area. The company’s big plans for expansion topped out at opening a few stores in nearby Portland, Oregon. In the early eighties, you couldn’t go into Starbucks and order a cup of coffee. They only sold the beans.
This employee had been tasked by one of the founders with attending a coffee trade show in Milan, as a representative of the company. As he recalls: “That’s when I was walking the streets of Milan and every twenty, thirty yards you are intercepted, literally, with an Italian coffee bar.”
“I walked in to all of these places and kept seeing the theatre, the romance, and the nectar of the gods which is espresso... What really struck me was the sense of community. I started going to these stores almost the same time every day and I would start seeing the same people. I started realizing this was a third place between home and work. But the beverage was the draw.”
The lesson this employee drew from the experience was that “Starbucks was in the coffee business but perhaps the wrong part of the business.”
That employee? Starbucks CEO and visionary Howard Schultz. When he returned to Seattle, Schultz told the founders that Starbucks should begin making espresso drinks, like they do in Italy. The founders demurred. They didn’t buy Schultz’s vision. It wasn’t what they had in mind for the company. Their trajectory at the time was totally out of alignment with what Schultz was proposing. Besides, espresso machines were expensive. It seemed unlikely that they’d be able to make back their initial up-front investment on the margins of a few cups of coffee. In short, they wanted to continue doing things the way they’d been doing them. They wanted to stick with habit.
But for more than a year, Schultz kept on insisting and eventually the founders relented.
“I think they just said, ‘Okay, get this kid out of our face. We’re opening a new store on the corner of 4th and Spring,’ in 1985. It was fifteen-hundred square feet, and they said, ‘We’ll give you five hundred feet of the fifteen-hundred to open up an Italian coffee bar in the store.’ And that was it. I mean, we opened the 4th and Spring Starbucks store with our first coffee bar. Overnight, we had hundreds of customers every day. And we introduced caffè latte to the city of Seattle, basically.”
The coffee bar was so successful that it soon superseded the income from Starbuck’s whole-bean sales. But the results of the experiment failed to move the founders. It wasn’t how they envisioned their company being run. It was such a stark departure from their current mode of operation. And so they told Schultz, “We don’t want to repeat this.” Schultz was taken aback. “I was devastated. I almost didn’t believe it at first. Like, really. Are you serious?”
Schultz decided to quit Starbucks. He informed the founders of his decision to leave and start his own coffee bar. To their credit, the founders said, “Well, if you do that, we’d like to invest.” The deal was, of course, that he had to use Starbucks-roasted beans in his coffee bar. Schultz named it Il Giornale, after the Milanese newspaper.
After opening his first coffee bar, in Seattle’s Columbia Tower in 1986, Schultz went back to Italy. Over the next five years, he toured the country, visiting by his own estimation over 500 espresso bars. When in the late 80s Howard Schultz bought Starbucks and took over as CEO, he built the company on his concept of the “third place”—beyond home and work—inspired by the coffee bars of Milan. It was only then that Starbucks began to expand into the ubiquitous coffee retailer as we know it today.
So, was habit important for Starbucks going from nine-thousand stores to ten, or fourteen-thousand to fifteen? Absolutely. But if habit reigned supreme, the company would never have gotten to that point. If habits had won out, Starbucks would still be a half dozen shops in the Pacific Northwest. They would’ve never even begun selling cups of coffee.
How do we square this with our findings about conscientiousness? Isn’t conscientiousness the most important non-cognitive predictor of work-place success? Yes, but there’s an exception. I pulled a Duhigg on you. I left out an important part of the story.
There was a further layer to the findings from the meta-analysis of conscientiousness. The exception to the rule had to do with the complexity of the problems faced in the workplace. Conscientiousness is the single most powerful non-cognitive predictor in the workplace—but only in low-complexity environments. In high-complexity jobs, conscientiousness matters far less.
Here’s what that figure looks like:
This was one of the key distinctions the authors drew in the paper. They argued that the reason conscientiousness is such a strong predictor of success has to do with things like: a preference for a stable environment, the ability to persevere in the face of difficulty, a well-developed muscle of self-restraint to avoid distractions, and the motivation to achieve conventional goals. They boil all these considerations down to the low- versus high-complexity distinction.
Working in a job that is “high-complexity” is neither a good thing nor a bad thing. “Low-complexity” does not mean that it’s not difficult or doesn’t require intelligence. It simply means that the rules don’t change. In some jobs the kinds of problems that you have to solve are similar from day-to-day, while in others the structure of those problems can be drawn from a much larger range. Making coffee is, under this rubric, a “low-complexity” job. That’s not to say that the job isn’t exhausting or demanding or even different every day. It totally is. But the kinds of problems that you face occur over and over again. That’s the environment in which conscientiousness matters most, and in which habit is king.
Another example of a “low-complexity” job is professional athlete. This makes sense. The rules of a game or competition are stable, so almost by definition the problems faced by athletes are repetitive. The hoop doesn’t move, and the goal never gets any bigger or smaller. So we would expect to see that professional athletes score high in conscientiousness.
In a 2019 study, a group of Australian researchers conducted a large-scale analysis of Twitter data. They used people’s tweets to predict their personality scores on the Big Five inventory. Then they matched those scores with the occupations listed in their bio. As we’d expect, they found that professional athletes, like good baristas, score high on conscientiousness. Meanwhile, software engineers were, by comparison, much lower on conscientiousness. They showed a similar trend on a number of personality traits, with software engineers also being low in agreeableness and extraversion. The only personality trait on which the trend flipped was openness to experience.
In general, athletes were extremely low on openness. This also makes sense. In order to be good at what they do, they have to abide by the same strict training regiment, day after day, week after week. You don’t want a swimmer to take a month off from training to wonder what life might be like as a tennis player. You want them to stay the course.
But while the software developers scored near the bottom on the other personality dimensions, they scored near the top of the chart on openness. Why? It’s hard to say for sure from these data. But as a software developer, the nature of the problems you face varies dramatically. To import a metaphor from a different problem-solving domain, one day you could be doing sudoku, another day you might be faced with a crossword, and on yet another you might find yourself unscrambling anagrams. It’s a high-complexity job. And in high-complexity environments the key is not to be maximally self-disciplined and to cultivate the ultimate habitual routine. You need to be open to new things, new ideas, and ready for whatever comes your way in the moment. After all, isn’t that the trait Schultz was relying on when he brought back his experiences from Italy? It definitely wasn’t his propensity to stay within prescribed bounds and stay the current course. It was his openness to the ideas he was encountering abroad.
Whether conscientiousness or openness is more important to your job depends entirely on what you do.
In our effort to cultivate optimally efficient habits, I think we’ve begun to overlook some of its downsides. There is a price to be paid for habit’s seductive promise of frictionless productivity. Habit can also be a prison. We get locked in the same way of doing things, the same way of thinking about others, the same beliefs about the way things ought to work and the way they in fact do. The price of eliminating friction is that we become inflexible. We can’t change course. It becomes hard to escape from the habits of our daily routine. As efficient as it is to rely on habit, the opposite mode is also necessary.
In my work on this Substack, I will argue that we get just as much out of this radically unhabitual mode as we do from the frictionless habitual mode. I will also argue that many of the issues that come up in the course of everyday existence are due to entrenchment in habit. Habit is in many ways the enemy of creativity—a demon on our shoulder whispering to us that the best way to do things is the way we did them yesterday. It encourages us to rely on habits of mind in prejudging others, in our beliefs about the world, and in the stories we tell about ourselves and others. Most of all, habit is antithetical to meaning. Pure habitual routine is not the realm of the human, but the mechanical. Habits allow us to go through life without actually appreciating it.
Mr Duhigg is correct that habits are powerful. But they are only powerful in situations where the problems are well-defined—where you know what you want, and it’s fairly straightforward how to achieve it. But that hardly suggests that “there’s nothing you can’t do if you get the habits right.” Not all jobs ask you do something for which optimal habits are the answer. Even less so is that the case in other areas of life.
To me, the larger point here is that modern society tells us that in order to be successful, we need to form optimal habits. In most cases, we’re given some sort of socially-accepted story about what “success” looks like and what sort of habits are required to obtain it. Rarely are we given the space to re-evaluate the validity of the instructions we’ve received. We just go along with it. What other option do we have?
Perhaps what we need isn’t a more efficient route to success. Perhaps what we need is a better definition of what it means to succeed.