ChatNetZero: Harnessing Large-Language Models for Synthesizing Evidence to Evaluate Non-State Actors’ Climate Actions

Large language models (LLMs) that power generative AI technologies like ChatGPT and Bard can analyze vasts amounts of textual data to generate easy to understand insights in human prose. But these technologies have also been demonstrated to hallucinate, generating convincing but false answers; prone to bias, and difficult to unpack due to the “black box” nature of their underlying models. Particularly with respect to complex domain-specific issues like climate change, generic LLMs have been demonstrated to produce questionable and often inaccurate interpretations, highlighting the need for specialized models or fine-tuning to ensure reliable insights and informed analyses. Specifically, addressing the challenge of parsing and evaluating climate actions by corporations and other non-state actors, which can range from greenwashing to mere window dressing, proves challenging due to the heterogeneous nature and time-consuming process of assessing diverse documents. Here we introduce ChatNetZero, a fine-tuned LLM, trained on verified analyses and third-party, scientific assessments of non-state climate actions and policies, to demonstrate the potential and limitations of utilizing LLMs and generative AI more broadly as an evidence synthesis tool. We will introduce the methodology, highlight specific use cases, and share validation metrics for ChatNetZero’s frameworks addressing hallucination and greenwashing, employing human-in-the-loop validation and perplexity testing.