Why Diversity Programs Fail

Businesses started caring a lot more about diversity after a series of high-profile lawsuits rocked the financial industry. In the late 1990s and early 2000s, Morgan Stanley shelled out $54 million—and Smith Barney and Merrill Lynch more than $100 million each—to settle sex discrimination claims. In 2007, Morgan was back at the table, facing a new class action, which cost the company $46 million. In 2013, Bank of America Merrill Lynch settled a race discrimination suit for $160 million. Cases like these brought Merrill’s total 15-year payout to nearly half a billion dollars.

It’s no wonder that Wall Street firms now require new hires to sign arbitration contracts agreeing not to join class actions. They have also expanded training and other diversity programs. But on balance, equality isn’t improving in financial services or elsewhere. Although the proportion of managers at U.S. commercial banks who were Hispanic rose from 4.7% in 2003 to 5.7% in 2014, white women’s representation dropped from 39% to 35%, and black men’s from 2.5% to 2.3%. The numbers were even worse in investment banks (though that industry is shrinking, which complicates the analysis). Among all U.S. companies with 100 or more employees, the proportion of black men in management increased just slightly—from 3% to 3.3%—from 1985 to 2014. White women saw bigger gains from 1985 to 2000—rising from 22% to 29% of managers—but their numbers haven’t budged since then. Even in Silicon Valley, where many leaders tout the need to increase diversity for both business and social justice reasons, bread-and-butter tech jobs remain dominated by white men.

It shouldn’t be surprising that most diversity programs aren’t increasing diversity. Despite a few new bells and whistles, courtesy of big data, companies are basically doubling down on the same approaches they’ve used since the 1960s—which often make things worse, not better. Firms have long relied on diversity training to reduce bias on the job, hiring tests and performance ratings to limit it in recruitment and promotions, and grievance systems to give employees a way to challenge managers. Those tools are designed to preempt lawsuits by policing managers’ thoughts and actions. Yet laboratory studies show that this kind of force-feeding can activate bias rather than stamp it out. As social scientists have found, people often rebel against rules to assert their autonomy. Try to coerce me to do X, Y, or Z, and I’ll do the opposite just to prove that I’m my own person.

In analyzing three decades’ worth of data from more than 800 U.S. firms and interviewing hundreds of line managers and executives at length, we’ve seen that companies get better results when they ease up on the control tactics. It’s more effective to engage managers in solving the problem, increase their on-the-job contact with female and minority workers, and promote social accountability—the desire to look fair-minded. That’s why interventions such as targeted college recruitment, mentoring programs, self-managed teams, and task forces have boosted diversity in businesses. Some of the most effective solutions aren’t even designed with diversity in mind.

Here, we dig into the data, the interviews, and company examples to shed light on what doesn’t work and what does.

Why You Can’t Just Outlaw Bias

Executives favor a classic command-and-control approach to diversity because it boils expected behaviors down to dos and don’ts that are easy to understand and defend. Yet this approach also flies in the face of nearly everything we know about how to motivate people to make changes. Decades of social science research point to a simple truth: You won’t get managers on board by blaming and shaming them with rules and reeducation. Let’s look at how the most common top-down efforts typically go wrong.

Diversity training. Do people who undergo training usually shed their biases? Researchers have been examining that question since before World War II, in nearly a thousand studies. It turns out that while people are easily taught to respond correctly to a questionnaire about bias, they soon forget the right answers. The positive effects of diversity training rarely last beyond a day or two, and a number of studies suggest that it can activate bias or spark a backlash. Nonetheless, nearly half of midsize companies use it, as do nearly all the Fortune 500.

Many firms see adverse effects. One reason is that three-quarters use negative messages in their training. By headlining the legal case for diversity and trotting out stories of huge settlements, they issue an implied threat: “Discriminate, and the company will pay the price.” We understand the temptation—that’s how we got your attention in the first paragraph—but threats, or “negative incentives,” don’t win converts.

Another reason is that about three-quarters of firms with training still follow the dated advice of the late diversity guru R. Roosevelt Thomas Jr. “If diversity management is strategic to the organization,” he used to say, diversity training must be mandatory, and management has to make it clear that “if you can’t deal with that, then we have to ask you to leave.” But five years after instituting required training for managers, companies saw no improvement in the proportion of white women, black men, and Hispanics in management, and the share of black women actually decreased by 9%, on average, while the ranks of Asian-American men and women shrank by 4% to 5%. Trainers tell us that people often respond to compulsory courses with anger and resistance—and many participants actually report more animosity toward other groups afterward.

But voluntary training evokes the opposite response (“I chose to show up, so I must be pro-diversity”), leading to better results: increases of 9% to 13% in black men, Hispanic men, and Asian-American men and women in management five years out (with no decline in white or black women). Research from the University of Toronto reinforces our findings: In one study white subjects read a brochure critiquing prejudice toward blacks. When people felt pressure to agree with it, the reading strengthened their bias against blacks. When they felt the choice was theirs, the reading reduced bias.

Companies too often signal that training is remedial. The diversity manager at a national beverage company told us that the top brass uses it to deal with problem groups. “If there are a number of complaints…or, God forbid, some type of harassment case…leaders say, ‘Everyone in the business unit will go through it again.’” Most companies with training have special programs for managers. To be sure, they’re a high-risk group because they make the hiring, promotion, and pay decisions. But singling them out implies that they’re the worst culprits. Managers tend to resent that implication and resist the message.

Hiring tests. Some 40% of companies now try to fight bias with mandatory hiring tests assessing the skills of candidates for frontline jobs. But managers don’t like being told that they can’t hire whomever they please, and our research suggests that they often use the tests selectively. Back in the 1950s, following the postwar migration of blacks northward, Swift & Company, Chicago meatpackers, instituted tests for supervisor and quality-checking jobs. One study found managers telling blacks that they had failed the test and then promoting whites who hadn’t been tested. A black machine operator reported: “I had four years at Englewood High School. I took an exam for a checker’s job. The foreman told me I failed” and gave the job to a white man who “didn’t take the exam.”

This kind of thing still happens. When we interviewed the new HR director at a West Coast food company, he said he found that white managers were making only strangers—most of them minorities—take supervisor tests and hiring white friends without testing them. “If you are going to test one person for this particular job title,” he told us, “you need to test everybody.”

But even managers who test everyone applying for a position may ignore the results. Investment banks and consulting firms build tests into their job interviews, asking people to solve math and scenario-based problems on the spot. While studying this practice, Kellogg professor Lauren Rivera played a fly on the wall during hiring meetings at one firm. She found that the team paid little attention when white men blew the math test but close attention when women and blacks did. Because decision makers (deliberately or not) cherry-picked results, the testing amplified bias rather than quashed it.

Managers made only strangers—most of them minorities—take tests and hired white friends without testing them.

Companies that institute written job tests for managers—about 10% have them today—see decreases of 4% to 10% in the share of managerial jobs held by white women, African-American men and women, Hispanic men and women, and Asian-American women over the next five years. There are significant declines among white and Asian-American women—groups with high levels of education, which typically score well on standard managerial tests. So group differences in test-taking skills don’t explain the pattern.

Performance ratings. More than 90% of midsize and large companies use annual performance ratings to ensure that managers make fair pay and promotion decisions. Identifying and rewarding the best workers isn’t the only goal—the ratings also provide a litigation shield. Companies sued for discrimination often claim that their performance rating systems prevent biased treatment.

But studies show that raters tend to lowball women and minorities in performance reviews. And some managers give everyone high marks to avoid hassles with employees or to keep their options open when handing out promotions. However managers work around performance systems, the bottom line is that ratings don’t boost diversity. When companies introduce them, there’s no effect on minority managers over the next five years, and the share of white women in management drops by 4%, on average.

Grievance procedures. This last tactic is meant to identify and rehabilitate biased managers. About half of midsize and large firms have systems through which employees can challenge pay, promotion, and termination decisions. But many managers—rather than change their own behavior or address discrimination by others—try to get even with or belittle employees who complain. Among the nearly 90,000 discrimination complaints made to the Equal Employment Opportunity Commission in 2015, 45% included a charge of retaliation—which suggests that the original report was met with ridicule, demotion, or worse.

Once people see that a grievance system isn’t warding off bad behavior in their organization, they may become less likely to speak up. Indeed, employee surveys show that most people don’t report discrimination. This leads to another unintended consequence: Managers who receive few complaints conclude that their firms don’t have a problem. We see this a lot in our interviews. When we talked with the vice president of HR at an electronics firm, she mentioned the widely publicized “difficulties other corporations are having” and added, “We have not had any of those problems…we have gone almost four years without any kind of discrimination complaint!” What’s more, lab studies show that protective measures like grievance systems lead people to drop their guard and let bias affect their decisions, because they think company policies will guarantee fairness.

Things don’t get better when firms put in formal grievance systems; they get worse. Our quantitative analyses show that the managerial ranks of white women and all minority groups except Hispanic men decline—by 3% to 11%—in the five years after companies adopt them.

Still, most employers feel they need some sort of system to intercept complaints, if only because judges like them. One strategy that is gaining ground is the “flexible” complaint system, which offers not only a formal hearing process but also informal mediation. Since an informal resolution doesn’t involve hauling the manager before a disciplinary body, it may reduce retaliation. As we’ll show, making managers feel accountable without subjecting them to public rebuke tends to help.

Tools for Getting Managers on Board

If these popular solutions backfire, then what can employers do instead to promote diversity?

A number of companies have gotten consistently positive results with tactics that don’t focus on control. They apply three basic principles: engage managers in solving the problem, expose them to people from different groups, and encourage social accountability for change.

Engagement. When someone’s beliefs and behavior are out of sync, that person experiences what psychologists call “cognitive dissonance.” Experiments show that people have a strong tendency to “correct” dissonance by changing either the beliefs or the behavior. So, if you prompt them to act in ways that support a particular view, their opinions shift toward that view. Ask them to write an essay defending the death penalty, and even the penalty’s staunch opponents will come to see some merits. When managers actively help boost diversity in their companies, something similar happens: They begin to think of themselves as diversity champions.

Take college recruitment programs targeting women and minorities. Our interviews suggest that managers willingly participate when invited. That’s partly because the message is positive: “Help us find a greater variety of promising employees!” And involvement is voluntary: Executives sometimes single out managers they think would be good recruiters, but they don’t drag anyone along at gunpoint.

Managers who make college visits say they take their charge seriously. They are determined to come back with strong candidates from underrepresented groups—female engineers, for instance, or African-American management trainees. Cognitive dissonance soon kicks in—and managers who were wishy-washy about diversity become converts.

The effects are striking. Five years after a company implements a college recruitment program targeting female employees, the share of white women, black women, Hispanic women, and Asian-American women in its management rises by about 10%, on average. A program focused on minority recruitment increases the proportion of black male managers by 8% and black female managers by 9%.

Mentoring is another way to engage managers and chip away at their biases. In teaching their protégés the ropes and sponsoring them for key training and assignments, mentors help give their charges the breaks they need to develop and advance. The mentors then come to believe that their protégés merit these opportunities—whether they’re white men, women, or minorities. That is cognitive dissonance—“Anyone I sponsor must be deserving”—at work again.

While white men tend to find mentors on their own, women and minorities more often need help from formal programs. One reason, as Georgetown’s business school dean David Thomas discovered in his research on mentoring, is that white male executives don’t feel comfortable reaching out informally to young women and minority men. Yet they are eager to mentor assigned protégés, and women and minorities are often first to sign up for mentors.

Mentoring programs make companies’ managerial echelons significantly more diverse: On average they boost the representation of black, Hispanic, and Asian-American women, and Hispanic and Asian-American men, by 9% to 24%. In industries where plenty of college-educated nonmanagers are eligible to move up, like chemicals and electronics, mentoring programs also increase the ranks of white women and black men by 10% or more.

Only about 15% of firms have special college recruitment programs for women and minorities, and only 10% have mentoring programs. Once organizations try them out, though, the upside becomes clear. Consider how these programs helped Coca-Cola in the wake of a race discrimination suit settled in 2000 for a record $193 million. With guidance from a court-appointed external task force, executives in the North America group got involved in recruitment and mentoring initiatives for professionals and middle managers, working specifically toward measurable goals for minorities. Even top leaders helped to recruit and mentor, and talent-sourcing partners were required to broaden their recruitment efforts. After five years, according to former CEO and chairman Neville Isdell, 80% of all mentees had climbed at least one rung in management. Both individual and group mentoring were open to all races but attracted large numbers of African-Americans (who accounted for 36% of protégés). These changes brought important gains. From 2000 to 2006, African-Americans’ representation among salaried employees grew from 19.7% to 23%, and Hispanics’ from 5.5% to 6.4%. And while African-Americans and Hispanics respectively made up 12% and 4.9% of professionals and middle managers in 2002, just four years later those figures had risen to 15.5% and 5.9%.

This began a virtuous cycle. Today, Coke looks like a different company. This February, Atlanta Tribune magazine profiled 17 African-American women in VP roles and above at Coke, including CFO Kathy Waller.

Contact. Evidence that contact between groups can lessen bias first came to light in an unplanned experiment on the European front during World War II. The U.S. army was still segregated, and only whites served in combat roles. High casualties left General Dwight Eisenhower understaffed, and he asked for black volunteers for combat duty. When Harvard sociologist Samuel Stouffer, on leave at the War Department, surveyed troops on their racial attitudes, he found that whites whose companies had been joined by black platoons showed dramatically lower racial animus and greater willingness to work alongside blacks than those whose companies remained segregated. Stouffer concluded that whites fighting alongside blacks came to see them as soldiers like themselves first and foremost. The key, for Stouffer, was that whites and blacks had to be working toward a common goal as equals—hundreds of years of close contact during and after slavery hadn’t dampened bias.

Business practices that generate this kind of contact across groups yield similar results. Take self-managed teams, which allow people in different roles and functions to work together on projects as equals. Such teams increase contact among diverse types of people, because specialties within firms are still largely divided along racial, ethnic, and gender lines. For example, women are more likely than men to work in sales, whereas white men are more likely to be in tech jobs and management, and black and Hispanic men are more likely to be in production.

As in Stouffer’s combat study, working side-by-side breaks down stereotypes, which leads to more equitable hiring and promotion. At firms that create self-managed work teams, the share of white women, black men and women, and Asian-American women in management rises by 3% to 6% over five years.

The Downside of the Diversity Label

Why can mentoring, self-managed teams, and cross-training increase diversity without the backlash prompted by mandatory training? One reason may be that these programs aren’t usually branded as diversity efforts. Diversity language in company policy can stress white men out, as researchers at UC Santa Barbara and the University of Washington found when they put young white men through a simulated job interview—half of them for a company that touted its commitment to diversity, and half for a company that did not. In the explicitly pro-diversity company, subjects expected discrimination against whites, showed cardiovascular distress, and did markedly worse in the taped interview.

Rotating management trainees through departments is another way to increase contact. Typically, this kind of cross-training allows people to try their hand at various jobs and deepen their understanding of the whole organization. But it also has a positive impact on diversity, because it exposes both department heads and trainees to a wider variety of people. The result, we’ve seen, is a bump of 3% to 7% in white women, black men and women, and Asian-American men and women in management.

About a third of U.S. firms have self-managed teams for core operations, and nearly four-fifths use cross-training, so these tools are already available in many organizations. Though college recruitment and mentoring have a bigger impact on diversity—perhaps because they activate engagement in the diversity mission and create intergroup contact—every bit helps. Self-managed teams and cross-training have had more positive effects than mandatory diversity training, performance evaluations, job testing, or grievance procedures, which are supposed to promote diversity.

Social accountability. The third tactic, encouraging social accountability, plays on our need to look good in the eyes of those around us. It is nicely illustrated by an experiment conducted in Israel. Teachers in training graded identical compositions attributed to Jewish students with Ashkenazic names (European heritage) or with Sephardic names (African or Asian heritage). Sephardic students typically come from poorer families and do worse in school. On average, the teacher trainees gave the Ashkenazic essays Bs and the Sephardic essays Ds. The difference evaporated, however, when trainees were told that they would discuss their grades with peers. The idea that they might have to explain their decisions led them to judge the work by its quality.

In the workplace you’ll see a similar effect. Consider this field study conducted by Emilio Castilla of MIT’s Sloan School of Management: A firm found it consistently gave African-Americans smaller raises than whites, even when they had identical job titles and performance ratings. So Castilla suggested transparency to activate social accountability. The firm posted each unit’s average performance rating and pay raise by race and gender. Once managers realized that employees, peers, and superiors would know which parts of the company favored whites, the gap in raises all but disappeared.

Corporate diversity task forces help promote social accountability. CEOs usually assemble these teams, inviting department heads to volunteer and including members of underrepresented groups. Every quarter or two, task forces look at diversity numbers for the whole company, for business units, and for departments to figure out what needs attention.

After investigating where the problems are—recruitment, career bottlenecks, and so on—task force members come up with solutions, which they then take back to their departments. They notice if their colleagues aren’t volunteering to mentor or showing up at recruitment events. Accountability theory suggests that having a task force member in a department will cause managers in it to ask themselves, “Will this look right?” when making hiring and promotion decisions.

Deloitte has seen how powerful social accountability can be. In 1992, Mike Cook, who was then the CEO, decided to try to stanch the hemorrhaging of female associates. Half the company’s hires were women, but nearly all of them left before they were anywhere near making partner. As Douglas McCracken, CEO of Deloitte’s consulting unit at the time, later recounted in HBR, Cook assembled a high-profile task force that “didn’t immediately launch a slew of new organizational policies aimed at outlawing bad behavior” but, rather, relied on transparency to get results.

The task force got each office to monitor the career progress of its women and set its own goals to address local problems. When it became clear that the CEO and other managing partners were closely watching, McCracken wrote, “women started getting their share of premier client assignments and informal mentoring.” And unit heads all over the country began getting questions from partners and associates about why things weren’t changing faster. An external advisory council issued annual progress reports, and individual managers chose change metrics to add to their own performance ratings. In eight years turnover among women dropped to the same level as turnover among men, and the proportion of female partners increased from 5% to 14%—the highest percentage among the big accounting firms. By 2015, 21% of Deloitte’s global partners were women, and in March of that year, Deloitte LLP appointed Cathy Engelbert as its CEO—making her the first woman to head a major accountancy.

Task forces are the trifecta of diversity programs. In addition to promoting accountability, they engage members who might have previously been cool to diversity projects and increase contact among the women, minorities, and white men who participate. They pay off, too: On average, companies that put in diversity task forces see 9% to 30% increases in the representation of white women and of each minority group in management over the next five years.

Once it was clear that top managers were watching, women started to get more premier assignments.

Diversity managers, too, boost inclusion by creating social accountability. To see why, let’s go back to the finding of the teacher-in-training experiment, which is supported by many studies: When people know they might have to explain their decisions, they are less likely to act on bias. So simply having a diversity manager who could ask them questions prompts managers to step back and consider everyone who is qualified instead of hiring or promoting the first people who come to mind. Companies that appoint diversity managers see 7% to 18% increases in all underrepresented groups—except Hispanic men—in management in the following five years. Those are the gains after accounting for both effective and ineffective programs they put in place.

Only 20% of medium and large employers have task forces, and just 10% have diversity managers, despite the benefits of both. Diversity managers cost money, but task forces use existing workers, so they’re a lot cheaper than some of the things that fail, such as mandatory training.

Leading companies like Bank of America Merrill Lynch, Facebook, and Google have placed big bets on accountability in the past couple of years. Expanding on Deloitte’s early example, they’re now posting complete diversity numbers for all to see. We should know in a few years if that moves the needle for them.

. . .

Strategies for controlling bias—which drive most diversity efforts—have failed spectacularly since they were introduced to promote equal opportunity. Black men have barely gained ground in corporate management since 1985. White women haven’t progressed since 2000. It isn’t that there aren’t enough educated women and minorities out there—both groups have made huge educational gains over the past two generations. The problem is that we can’t motivate people by forcing them to get with the program and punishing them if they don’t.

The numbers sum it up. Your organization will become less diverse, not more, if you require managers to go to diversity training, try to regulate their hiring and promotion decisions, and put in a legalistic grievance system.

The very good news is that we know what does work—we just need to do more of it.

Copyright 2023 Harvard Business School Publishing Corporation. Distributed by The New York Times Syndicate.