Technical Challenges and Solutions for Verifying, Improving, and Diagnosing LLMs

Over the past few years, there has been an explosion in the development of a class of large neural networks known as foundation models. Foundation models such as GPT-4, PaLM, and Wu Dao 2.0 have demonstrated impressive capabilities in language, speech, and vision domains. These models are characterized by their massive scale, containing billions or trillions of parameters, which allows them to acquire broad knowledge about the world from their training data.

However, accompanying the size of these models are several key reliability challenges that must be solved before they can be responsibly deployed in real-world applications.

The Core Challenges: Hallucination, Accuracy, and Transparency

Sapien has identified three core technical challenges with current foundation models:

Hallucination and Verification: Models often confidently output plausible but incorrect information, requiring mechanisms to detect and verify outputs.

Accuracy and Calibration: Performance remains unreliable, especially for out-of-distribution inputs, necessitating enhancements.

Transparency and Diagnostics: The models remain black-boxes, hampering diagnostic testing and improvement.

The Technical Background Behind LLMs

Model Architectures

Most major foundation models are based on the Transformer architecture originally proposed in Vaswani et al. 2017. The Transformer does away with recurrence and convolutions, relying entirely on a self-attention mechanism to model global dependencies. Some key architectural components include:

Embedding Layers: converts discrete input tokens into continuous vector representations

Encoders: layers composed of multi-headed self-attention and feedforward sublayers, which model interactions between input elements.

Decoders (autoregressive models only): similar to encoders, but mask future attention to preserve autoencoding ordering.

Heads: separate groups of transformations used within the self-attention module to provide multiple "representations" of the inputs.

The scale of foundation models enlarges all components, increasing the resolution of the input embedding mapping as well as the capacity of attention mechanisms. For example, GPT-3 contains 96 layers with 96 heads and dimension of 12,288, yielding over 175 billion parameters.

Pretraining Objectives

Unsupervised pretraining objectives provide the learning signal that enables models to develop linguistic understanding before task-specific fine-tuning:

Autoregressive Language Modeling (GPT line): Models next token prediction abilities using causal transformers.

Masked Language Modeling (BERT line): Models use bidirectional context to predict randomly masked input tokens. Usually supplemented with next-sentence prediction task.

Multitask Learning: Some models pretrain on multiple objectives simultaneously, e.g. PaLM trains on both masked and causal language modeling.

The optimization goal is to compress the training distribution into the parameters such that new examples can be accurately generated or predicted from. However, this can overfit statistical regularities that fail to generalize.

Key Challenge #1 - Output Verification

The Hallucination Problem

A major reliability issue with large foundation models is their tendency to hallucinate - to output confident but entirely incorrect or ungrounded statements. For example, Chowdhery et al. 2022 found that 70% of confident GPT-3 predictions in a quiz experiment were false, underscoring the prevalence of hallucination. This severely limits real-world applicability across tasks like question answering, summarization, and language translation.

Causes of Hallucination

Several factors contribute to the emergence of hallucination in model outputs:

Spurious Correlations

The self-supervised objectives allow models to pick up on accidental regularities between tokens that lead to false inferences. For example, there may be a pattern in the data that country X is often discussed in the context of export Y. The model may then overgeneralize and state that country X is a major exporter of good Y, even if no such relationship holds.

Sampling Error

Most models are trained not to precisely predict tokens but to estimate a probability distribution over the vocabulary. At each step, the generation process samples from this distribution. However, unlikely tokens can be occasionally sampled, compounding over long textual spans into plausible false claims.

Lack of Grounding

With no connection to real-world states, models have no anchor to determine if an inferred relationship actually reflects reality or not. There are no mechanisms to verify if something is ungrounded before confidently asserting it.

Promising Solutions

Thankfully, the extensive attention on hallucination has led to various proposed techniques to address it:

Confidence Scoring and Unlikelihood Detection: By scoring the model's (un)certainty at each generation step, implausible outputs can be flagged for verification. Unlikely n-grams and semantic transformations can also be caught.

Multi-Model Consistency Checking: Since errors may vary across models, generating multiple outputs and checking for consistency can help determine trustworthy claims. One can also query external models for fact-checking.

Semi-Supervised Falsehood Detection: Explicit classifiers can be trained to discriminate between truthful and hallucinated outputs using human judgments and adversarial distractors.

Causal Analysis: Techniques from causal inference could detect statistical patterns exploited by models that lack causal grounding in reality. Interventional robustness checks may also help diagnose these patterns.

However, despite all of this, output verification remains an open challenge. Multi-pronged approaches across training objectives, model architectures, and output analysis procedures appear necessary to enable reliable generative foundation models. Ongoing advances on benchmark tasks assessing hallucination will likely catalyze progress in the coming years.

Key Challenge #2 - Accuracy Improvements

The Accuracy Problem

While foundation models demonstrate strong performance when making predictions within the distribution of their training data, they struggle to maintain accuracy on out-of-distribution inputs. For example, performance can rapidly degrade on longer textual contexts, complex reasoning tasks, and domains with limited data. Without reliability guarantees, real-world utilization remains limited.

Causes of Accuracy Limitations

Several factors contribute to the brittleness of accuracy:

Dataset Biases

The model inadvertently encodes skewed regularities and selection biases reflected in the pretrained data. This leads to representations that fail to generalize beyond ingrained assumptions.

Simplifying Assumptions

Architectural choices and objectives make implicit simplifications about the domains being modeled. For example, the single-sequence design of models like GPT-3 cannot explicitly model complex relational reasoning.

Promising Solutions

Many current lines of research hold promise for improving out-of-distribution accuracy:

Dataset Expansion and Augmentation

Creating training sets that better cover the breadth of target domains can enhance robustness. Data augmentation techniques can programmatically increase diversity.

Self-Supervised Pretraining

By pretraining on intensive unsupervised prediction tasks over diverse corpora, models can learn more grounded representations before specification.

Hybrid Models

Combining neural modules with structured knowledge bases and databases may supplement core model limitations around reasoning and grounding.

Formal Guarantees

Drawing from program synthesis and verification literature, some have proposed using proof systems to generate certified bounds on model performance for classes of inputs.

While still large issues, the challenge of unreliable accuracy has galvanized intense focus among researchers towards developing more rigorous, evidence-based foundations for large language models. Progress will likely involve holistic solutions across model families, objectives, and formal analysis techniques.

Key Challenge #3 - Diagnostics and Interpretability

The Transparency Problem

The massive scale and complexity of foundation models means that they operate largely as black boxes, providing little visibility into their inner workings. This lack of transparency creates issues for both model development and utilization:

1. Model developers lack diagnostic techniques to trace how particular behaviors emerge from interactions of architectural components. This hinders directed efforts to improve model performance.
2. End-users cannot determine whether model rationales can be trusted since the reasoning process remains opaque. This reduces deployability in sensitive applications like healthcare, finance, and governance.

Causes of Intransparency

While opaque, the black box nature of large language models stems from their technical underpinnings:

Massive Parameterization

With billions to trillions of parameters regulating a web of nonlinear interactions, the source of any particular model output becomes astronomically difficult to isolate. There are no clear one-to-one mappings from parameters to functions.

Emergent Representations

The representations learned by attention layers are not directly programmed but emerge indirectly from parameter optimization. The origins of these latent representations are thus mysterious yet critical to model function.

Promising Solutions

Various techniques have been introduced to deconstruct the black box nature of models:

Attention Analysis

Attention heatmaps provide some visibility into parametric interactions, showing which inputs influence outputs. However, their interpretability remains debated.

Modular Component Dissection

By carefully ablating or modifying specific encoder blocks, layers, heads, and neurons, researchers can measure isolated contributions towards certain functions.

Concept Bottleneck Models

Forcing representations through discrete categorization bottlenecks enables explicit manipulation of model concept usage, facilitating analysis.

Counterfactual Evaluation

Systematically manipulating inputs and examining effects on outputs can empirically trace patterns of dependency and sensitivity without full transparency.

We expect the battlefield between model complexity and interpretability to spur an escalating arms race towards transparency as the scale of next-generation models continues ballooning. Interpretability research remains crucial to ensuring these models remain diagnosable, debuggable, and safe.

The Role of Reinforcement Learning from Human Feedback

Reinforcement learning (RL) provides a framework for agents to learn behaviors through interactive evaluation of actions by a human trainer. This approach has recently been applied to large language models as well, using human feedback to provide an additional tuning signal.

In this system, models generate textual outputs, which trainers then critique through ratings, corrections, or other forms of review. Feedback is converted into a reward signal that updates model parameters to reinforce helpful behaviors and discourage unwanted behaviors.

Over successive interactions, models can learn to produce higher quality, safer, and more reliable text tailored to the preferences of trainers.

Advantages over Passive Learning

RL from human feedback confers several advantages over conventional supervised or unsupervised objectives.

Rich Evaluation Signal

Rather than learning from static historical data, models learn from direct human judgment of specific model behaviors. This provides a richer, more targeted signal.

Potential for Safe Exploration

Models can explore editing suggestions from trainers to expand capabilities beyond the confinement of historical data. However, human oversight enables safer experimentation bounds.

Scalable Data Collection

Rather than require full dataset annotation, models can learn from the context of live interactions, increasing scalability.

Challenges with RLHF

However, numerous research challenges remain around adopting RL from human feedback:

Feedback Quality and Reliability

Unlike fixed historical datasets, quality control over human feedback can be difficult as trainers may disagree or make mistakes. Mitigating unreliable signals poses an open problem.

Sample Efficiency

With limited interaction episodes relative to the model scale, maximizing learning from each human judgment is critical but non-trivial. More efficient algorithms are needed.

Reward Gaming and Manipulation

Models may find unintended exploits in the feedback mechanisms to maximize rewards without improving underlying performance. Ensuring alignment remains challenging.

Integrating with Existing Paradigms

Seamlessly combining RL objectives with supervised, semi-supervised, and self-supervised training is an open architectural challenge with many possibilities.

‍

As research continues reconciling these tensions, reinforcement learning from human feedback shows promise to enhance both model performance and reliability through synergistic human-AI interaction, which is why Sapien is focusing our efforts toward this solution.

‍

The Future of RLHF and the Most Complex Technical Challenges for LLMs

At Sapien, we believe progress requires commitment across four interconnected fronts:

Objectives: Training schemes like reinforcement learning from human feedback and self-supervised prediction tasks can provide useful auxiliary signals alongside primary pretraining goals. Hybrid approaches may be necessary.

Architecture: Specialized modules for reasoning, verification, and grounding should supplement core generative infrastructure. More structured architectures could enhance interpretability.

Data: Expansive, multi-domain corpora are needed with coverage of target distributions. Data augmentation and synthesis techniques should be employed for fuller representation.

Analysis: Formal verification systems and improved diagnostic protocols are crucial for interpreting model behaviors and providing performance guarantees.

Book a Demo with Sapien to Learn More About Scalable Data Labeling for Your LLM

Throughout this article, we have tried to make clear the limitations in existing training paradigms for large language models including insufficient data coverage, sample efficiency constraints, and data quality assurance. Thankfully, dedicated data labeling providers like Sapien are emerging to help address these obstacles.

Sapien offers secure, customizable data labeling through a global network of domain experts in fields ranging from law to medicine. Our human-in-the-loop platform enables models to learn interactively from real-time feedback on outputs for text, image, and speech data. Quality assurance processes maximize signal clarity and relevance.

Our services can directly tackle challenges around model hallucination, accuracy limitations out-of-distribution, and safe exploration in reinforcement learning settings we have covered. By scaling high-fidelity labeled data generation, reliability and transparency of next-generation models can be enhanced. Just as scaled model architectures pushed progress, scalable data infrastructure promises to unlock the full potential of AI with human guidance.

To learn more about our solutions for LLMs, book a demo from Sapien to explore our platform.