A Survey of Techniques for Maximizing LLM Performance

This OpenAI Dev Day talk from over a year ago (around the time of GPT-4) discusses the common techniques for maximizing LLM performance, and it’s remarkable on the way they went through them in this video.

A recommended watch for anyone working with LLMs.

---

Based on the provided source, here are some best practices for maximizing LLM performance when building effective AI agents:

• Start with Prompt Engineering: Begin with prompt engineering techniques as they are low investment and allow for quick iteration and validation of LLMs for the problem.
  • Write clear instructions.
  • Split complex tasks into simpler subtasks.
  • Give the model time to think.
  • Test changes systematically.
• Establish a Baseline: Before fine-tuning, establish a performance baseline to understand the strengths and weaknesses of the models.
• Evaluate: Create a solid set of evaluations and use LLMOps to systematically measure changes.
• Use Few-Shot Examples: Provide input and output pairs to show the model how to behave.
• Address Context or Instruction Issues: Determine whether the problem is a lack of context or an issue with how the model needs to act.
• Consider Retrieval-Augmented Generation (RAG): If the model needs more context or access to domain-specific content, use RAG to introduce new information and reduce hallucinations.
  • Optimize the search and retrieval of relevant content.
  • Use metrics like faithfulness, answer relevancy, context precision, and context recall to evaluate RAG.
• Consider Fine-tuning: If the model needs more consistent instruction following or a specific output format or style, use fine-tuning.
  • Use high-quality training data.
  • Start with a small dataset.
  • Iterate and evaluate the model's performance.
  • Be aware of the limitations of fine-tuning for introducing new knowledge.
• Combine RAG and Fine-tuning: For some use cases, combining RAG and fine-tuning can maximize performance.
  • Fine-tune for instruction following and then use RAG to inject relevant knowledge.
• Iterate: Be prepared to iterate and jump back and forth between prompt engineering, RAG, and fine-tuning.
• Evaluate the Loss Function: Understand that the loss function may not correlate with performance on downstream tasks.
• Pay attention to Data Quality: When fine-tuning, focus on high-quality examples, not large quantities of data.

---

How do prompt engineering, RAG and fine-tuning compare?

Prompt Engineering:

• Best for: Starting point for testing and learning; establishing a baseline. Good for simple tasks.
• How it works: Involves writing clear instructions, splitting complex tasks, giving the model time to think, and systematically testing changes.
• Limitations: Not scalable for introducing large amounts of new information, limited by context window, can lead to increased token usage.
• Goal: To tell the model how to act.

Retrieval-Augmented Generation (RAG):

• Best for: Introducing new information and domain-specific content; reducing hallucinations.
• How it works: Retrieves relevant content from a knowledge base and combines it with a prompt.
• Limitations: Cannot teach a broad domain or new style, increases token usage, relies on quality of search and retrieval.
• Goal: To give the model the right context to answer a question.
• Evaluation: Requires evaluation of both the LLM's answers and the relevance of the retrieved content.

Fine-tuning:

• Best for: Achieving higher levels of performance, modifying output structure or tone, emphasizing existing knowledge in the base model.
• How it works: Continues the training process of an existing model on a smaller, domain-specific dataset.
• Limitations: Not good for adding new knowledge or quick iterations on new use cases, can be a slow feedback loop.
• Goal: To specialize a model for a specific task.
• Efficiency: Fine-tuned models are often more efficient to interact with, requiring fewer prompt tokens and enabling the use of smaller models.
• Data: Requires high quality, task-specific data

Comparison:

• Prompt engineering: Tells the model how to act, good for quick iterations, but limited in scalability.
• RAG: Provides the model with necessary context, enhances accuracy, but adds complexity in evaluation.
• Fine-tuning: Specializes the model's behavior and improves performance, but requires more investment and is less suitable for introducing new knowledge.
• They are not linear steps but rather different techniques that can be used independently or together depending on the problem.
• A typical optimization flow starts with prompt engineering, then moves to RAG if context is needed, and fine-tuning if instruction following is the issue.
• They can be combined. For example, fine-tuning can improve the ability to follow complex instructions, allowing more room in the context window for RAG-provided context.

---

Based on the sources, here's how fine-tuning is done:

• Data Preparation:
  • A dataset is needed. This dataset is smaller and more domain-specific than the data the model was originally trained on.
  • Data can be obtained from open-source datasets, private markets, human labelers, or by distilling from a larger model.
  • The dataset consists of examples of the desired input and output, without complex instructions, formal schemas or in-context examples.
• Training:
  • The training process continues training an existing model on the new dataset.
  • This process can be done using a turnkey solution like the OpenAI fine-tuning API, or by using open-source models with your own GPUs and frameworks.
  • It is important to understand the available hyperparameters to tune during the training process, to avoid overfitting or catastrophic forgetting.
• Loss Function:
  • The loss function is a proxy for next token prediction, which may not always correlate with performance on the downstream tasks.
• Evaluation:
  • The model is evaluated using expert human evaluations, by ranking against other models or using a more powerful model to rank the outputs.
• Deployment and Iteration:
  • The fine-tuned model is deployed and used for inference.
  • Samples from the deployed model can be used to create new datasets for further fine-tuning. This can create a feedback loop.
• Best Practices:
  • Start with prompt engineering and few-shot learning.
  • Establish a baseline before fine-tuning.
  • Start with a small, high-quality dataset.
  • Use an active learning approach to target areas where the model struggles.
  • Prioritize data quality over quantity.

The goal of fine-tuning is to specialize a general model to be better suited for a specific task.