Who Does a Better Job with ESG ratings, Global or Local Research Firms?

Contradictory ESG ratings can make life confusing for ESG-focused investors. This may be especially true for those who invest globally, since they’re likely to encounter additional divergences between the assessments from the major global ESG raters (who are generally based in the U.S.) and those from raters with a country-specific focus (who may be based in that country). Whom to believe?

In a new paper, Aaron Yoon, an assistant professor of accounting and information management at the Kellogg School, continues his multi-paper investigation of the most useful ESG information sources for investors. His focus here is on comparing the accuracy of the ESG ratings generated by a global rater (MSCI, a behemoth in the game with 30 percent market share and three decades of experience in ESG rating) with those produced by a Chinese rater (SINO Securities, who began issuing ESG ratings for Chinese firms in 2009). To determine which rater comes out on top, Yoon and his colleagues used a sample of Chinese firms rated by both MSCI and SINO, and analyzed whose ratings better predict incidents of companies’ bad ESG behavior (like violating national laws or polluting) over the subsequent year.

Yoon and his coauthors—Jeff Chen at Texas Christian University; Zengquan Li, at Shanghai University of Finance and Economics; and Ting Mao, a PhD student at Chinese University of Hong Kong—found that despite its shorter tenure and more-limited resources, the Shanghai-based SINO proved superior in rating Chinese companies.

Yoon says the finding reveals how ESG ratings, like so much else, can be inflected by cultural lenses. An ESG rater deeply familiar with the society, legal structure, and culture in which a company operates is likely to produce more useful ESG assessments of those companies.

“A lot of ESG information is very society-specific and context-specific,” Yoon says. “We need to look at the overall institutional framework, the local context, to make sense of that information.”

Modeling ESG Performance

In considering the relative advantages in local vs. global ESG raters, Yoon explains that it’s important to understand how their ESG scores are tabulated. Much more than, say, equity analysts—who tend to be deeply familiar with a small group of companies under their coverage—ESG raters rely on models that are uniform across their coverage universes. In other words, much less human discretion goes into generating individual companies’ ESG ratings vis-à-vis generating forecasts made by financial analysts.

“Whether they’re local or global, the ESG raters each have a model,” Yoon explains. “They input a bunch of different ESG disclosures and other information, and then the model spits out a score.”

This means that for a global ESG rater, they have the benefit of a model that can compare companies in different countries, but the model can only include the inputs that will be relevant to large companies across the world. The global rating models aren’t able to fully incorporate information that’s specific to a certain country’s laws or culture, such local environmental laws or cultural norms around succession planning. A local ESG rater, on the other hand, can model these country-specific dynamics.

Still, there are also compelling reasons to suspect that a global rater like MSCI would produce more-informative ratings, particularly as it pertains to creating value for shareholders. One could argue that local raters in China might feel pressured by the government to alter their ratings, or that the bigger, international raters might attract top talent and resources.

To establish which ESG researcher produced better ratings, Yoon and his colleagues drew data from RepRisk, an ESG data company based in Switzerland that tracks companies’ ESG-related incidents, using machine learning to power a daily review of 100,000 multilanguage information sources.

Focusing on SINO’s and MSCI’s ESG ratings of the same Chinese companies, the researchers analyzed which firms’ ESG ratings better aligned with the ESG-related incidents that occurred over the next twelve months.

A Surprising Advantage

Yoon admits that before conducting the research, he expected that MSCI’s ratings would emerge as the stronger ones.

“I thought that they would better predict value-relevant issues, and that it would place more emphasis on shareholder value, since it’s a U.S.-based rater,” he says. “None of that panned out, which was very surprising to me.”

In fact, MSCI’s ratings did not, statistically speaking, predict future negative ESG incidents at all. SINO’s, on the other hand, predicted such incidents within the social and governance categories—though they did not predict environment-related incidents.

The researchers found that, within the social and governance domains, the local rater’s strength seemed to be concentrated within ESG incidents related to corruption and bribery, poor employment conditions, and law violations. This “suggests that local raters have a greater advantage over global raters in issues where ESG performance measurement is less quantifiable (i.e., social issues rather than metrics like CO2 emissions), more subjective, and requires more context,” the researchers write.

Another way to think about it, says Yoon, is that SINO, as the local rater, is better able to shape its model to account for its local context and the issues specific to it.

The researchers dug even deeper to better understand the channels through which SINO achieved its advantage. To do this, they subdivided the sample of Chinese companies according to whether or not they issued ESG reports and were politically connected. In both cases, SINO’s ratings better predicted future negative ESG events than MSCI’s.

The researchers also looked at companies that relied on relational contracts (that is, whether their sales and operations are heavily reliant on relationships with just a few firms).

Yoon explains that focusing on firms that rely on relational contracts is one way to test for the effects of possible cultural bias in ESG rating. While China has a highly relational economy, he explains, U.S.-based raters are quicker to flag businesses relying on relational contracts because those raters are looking through a U.S-centric lens. That perspective could lead them to believe that companies that operate this way are weaker and potentially more problematic than those with strictly market-based, price-oriented relationships. On the other hand, a rater deeply familiar with the Chinese context would likely better discern between problematic relational behavior and healthy relational behavior among the country’s firms.

Indeed, among the Chinese firms that relied on relational contracts, SINO’s ratings did prove more accurate than MSCI’s.

“When a firm’s information environment is opaque (i.e., when firms rely on relationships and political connections), the locals have the knowledge and context to piece information together,” the authors write.

Leaving Room for Interpretation

Regulators interested in standardizing how ESG reporting and rating is done globally should take note of these findings, Yoon says. While there are benefits to standardization, there could be downsides to being too heavy-handed and specific in dictating how ESG should be defined and measured across jurisdictions.

“The call for global standardization of ESG certainly has merits given that it would make ESG-related information comparable,” Yoon says. “The punchline of our paper is, when it comes to ESG, there are a lot of local, specific issues that also need to be considered.”