Firm managers may benefit from transparency in machine-learning algorithms

In today’s business world, machine-learning algorithms are increasingly being applied to decision-making processes, which affects employment, education, and access to credit. But firms usually keep algorithms secret, citing concerns over gaming by users that can harm the predictive power of algorithms. Amid growing calls to require firms to make their algorithms transparent, a new study developed an analytical model to compare the profit of firms with and without such transparency. The study concluded that there are benefits but also risks in algorithmic transparency.

Conducted by researchers at Carnegie Mellon University (CMU) and the University of Michigan, the study appears in Management Science.

“As managers face calls to boost transparency, our findings can help them make decisions to benefit their firms,” says Param Vir Singh, Professor of Business Technologies and Marketing at CMU’s Tepper School of Business, who coauthored the study.

Researchers investigated how algorithmic transparency affects firms and applicants for a job (also called agents) by developing and analyzing a game-theory model that captures how both parties act under opaque and transparent scenarios. In so doing, the authors sought to answer four questions: 1) From the perspective of the firm (the decision maker), are there advantages in making an algorithm transparent even when it could be manipulated by agents? 2) How would agents be affected if firms made their algorithms transparent? 3) How would the results be affected by the predictive power of those features that are more susceptible to gaming by agents? 4) How would market composition (in terms of desirable and undesirable agents) affect these results?

The study concluded that algorithmic transparency could have positive effects for managers and firms and negative effects for agents. Under a broad set of conditions, transparency benefits firms, allowing them to motivate agents to invest in improving features valuable to the firm and in some situations, increasing the predictive power of the algorithm. This challenges the conventional wisdom that making algorithms transparent will always hurt firms economically.

But the study also concluded that agents may not always be better off under algorithmic transparency. Firms use algorithms to separate high-type (more desirable) agents from low-type (less desirable) agents. These algorithms use desirable features (i.e., causal features that directly affect a firm’s performance, such as relevant education or training received in the case of hiring) and typically gameable correlational features (i.e., features that are correlated with the agent’s type but do not affect the firm’s performance, such as that high-type agents may be more likely to wear glasses).

High-type agents can get away with underinvesting in costly features that are desirable by the firm when the correlational features used by the firm’s opaque algorithms provide them a classification advantage. When a firm makes its algorithm transparent, then all agents would game on the correlational features and the predictive power of the correlational features would disappear. As a result, high-type agents have to invest in the costly desirable feature to separate themselves from low-type agents.

“Our analysis suggests that firms should not always worry about the potential loss of the predictive power under transparency when facing strategic individuals,” says Qiaochu Wang, a Ph.D. student in business technologies at CMU’s Tepper School of Business, who coauthored the study. “Rather, they can use algorithmic transparency as a lever to motivate agents to invest in more desirable features.”

One of the limitations of the study, note the authors, is that while it shows the economic benefits that transparent algorithms can bring to firms, there may be reasons firms do not want to make their algorithm transparent. These reasons—including privacy and competition—were not addressed in the study.

“The results of our model, which focused on a job-hiring scenario, are generalizable to other scenarios in which a firm attempts to gain further insight or knowledge into individuals’ private information,” notes Yan Huang, Associate Professor of Business Technologies at CMU’s Tepper School of Business, who coauthored the study.