AI Feedback Algorithms

Feedback loops are AI’s compound interest engine.. if you skip them and your AI performance will just erode over time. Too many roadmaps punt on serious evals because “models don’t hallucinate as much anymore” or “we’ll tighten it up later.” Be wary of those that say this, they really aren't serious practitioners. Here is the gold standard we run for production AI implementation at Bottega8: 1. Offline evals (CI gatekeeper): A lightweight suite of prompt unit tests, RAGAS faithfulness checks, latency, and cost thresholds runs on every PR. If anything regresses, the build fails. 2. RLHF, internal sandbox: A staging environment where we hammer the model with synthetic edge cases and adversarial red team probes. 3. RLHF, dogfood: Real users and real tasks. We expose a feedback widget that decomposes each output into groundedness, completeness, and tone so our labelers can triage in minutes. 4. RLHF, virtual assistants: Contract VAs replay the week’s top workflows nightly, score them with an LLM as judge, and surface drift long before customers notice. 5. Shadow traffic and A/B canaries: Ten percent of live queries route to the new model, and we ship only when conversion, CSAT, and error budgets clear the bar. The result is continuous quality and predictable budgets.. no one wants mystery spikes in spend nor surprise policy violations. If your AI pipeline does not fail fast in code review and learn faster in production, it is not an engineering practice, it is a gamble. There's enough eng industry best practice now with nearly three years of mainstream LLM/GenAI adoption. Happy building and let's build AI systems that audit themselves and compound insight daily.

Excited to announce my new (free!) white paper: “Self-Improving LLM Architectures with Open Source” – the definitive guide to building AI systems that continuously learn and adapt. If you’re curious how Large Language Models can critique, refine, and upgrade themselves in real-time using fully open source tools, this is the resource you’ve been waiting for. I’ve put together a comprehensive deep dive on: Foundation Models (Llama 3, Mistral, Google Gemma, Falcon, MPT, etc.): How to pick the right LLM as your base and unlock reliable instruction-following and reasoning capabilities. Orchestration & Workflow (LangChain, LangGraph, AutoGen): Turn your model into a self-improving machine with step-by-step self-critiques and automated revisions. Knowledge Storage (ChromaDB, Qdrant, Weaviate, Neo4j): Seamlessly integrate vector and graph databases to store semantic memories and advanced knowledge relationships. Self-Critique & Reasoning (Chain-of-Thought, Reflexion, Constitutional AI): Empower LLMs to identify errors, refine outputs, and tackle complex reasoning by exploring multiple solution paths. Evaluation & Feedback (LangSmith Evals, RAGAS, W&B): Monitor and measure performance continuously to guide the next cycle of improvements. ML Algorithms & Fine-Tuning (PPO, DPO, LoRA, QLoRA): Transform feedback into targeted model updates for faster, more efficient improvements—without catastrophic forgetting. Bias Amplification: Discover open source strategies for preventing unwanted biases from creeping in as your model continues to adapt. In this white paper, you’ll learn how to: Architect a complete self-improvement workflow, from data ingestion to iterative fine-tuning. Deploy at scale with optimized serving (vLLM, Triton, TGI) to handle real-world production needs. Maintain alignment with human values and ensure continuous oversight to avoid rogue outputs. Ready to build the next generation of AI? Download the white paper for free and see how these open source frameworks come together to power unstoppable, ever-learning LLMs. Drop a comment below or send me a DM for the link! Let’s shape the future of AI—together. #AI #LLM #OpenSource #SelfImproving #MachineLearning #LangChain #Orchestration #VectorDatabases #GraphDatabases #SelfCritique #BiasMitigation #Innovation #aiagents

First draft online version of The RLHF Book is DONE. Recently I've been creating the advanced discussion chapters on everything from Constitutional AI to evaluation and character training, but I also sneak in consistent improvements to the RL specific chapter. https://rlhfbook.com/ RLHF has a long future ahead of it and this will do a lot to make it more accessible to the next generation. What's next: Getting a physical copy in your hands (may not be exactly 1to1, we'll see) and minor fixes at a slower cadence (thanks to many github contributors, some of you will get a copy from me). Here are all the chapters. 1. Introduction: Overview of RLHF and what this book provides. 2. Seminal (Recent) Works: Key models and papers in the history of RLHF techniques. 3. Definitions: Mathematical definitions for RL, language modeling, and other ML techniques leveraged in this book. 4. RLHF Training Overview: How the training objective for RLHF is designed and basics of understanding it. 5. What are preferences?: Why human preference data is needed to fuel and understand RLHF. 6. Preference Data: How preference data is collected for RLHF. 7. Reward Modeling: Training reward models from preference data that act as an optimization target for RL training (or for use in data filtering). 8. Regularization: Tools to constrain these optimization tools to effective regions of the parameter space. 9. Instruction Tuning: Adapting language models to the question-answer format. 10. Rejection Sampling: A basic technique for using a reward model with instruction tuning to align models. 11. Policy Gradients: The core RL techniques used to optimize reward models (and other signals) throughout RLHF. 12. Direct Alignment Algorithms: Algorithms that optimize the RLHF objective directly from pairwise preference data rather than learning a reward model first. 13. Constitutional AI and AI Feedback: How AI feedback data and specific models designed to simulate human preference ratings work. 14. Reasoning and Reinforcement Finetuning: The role of new RL training methods for inference-time scaling with respect to post-training and RLHF. 15. Synthetic Data: The shift away from human to synthetic data and how distilling from other models is used. 16. Evaluation: The ever-evolving role of evaluation (and prompting) in language models. 17. Over-optimization: Qualitative observations of why RLHF goes wrong and why over-optimization is inevitable with a soft optimization target in reward models. 18. Style and Information: How RLHF is often underestimated in its role in improving the user experience of models due to the crucial role that style plays in information sharing. 19. Product, UX, Character: How RLHF is shifting in its applicability as major AI laboratories use it to subtly match their models to their products. 

How to use expert feedback to optimize AI agents? In many real-world applications, there is no clear ground truth label for what a “good” agent response is. Often, all we have is user feedback and preferences (“this is wrong”, “missing context”, “too verbose”, etc.). This feedback is an extremely valuable supervision signal, but turning it into effective optimization of agent behavior is not straightforward: Stochasticity & replay To learn from feedback, we often need to “replay” the original sample or trace. But agentic systems (with tools, RAG, branching, etc.) are stochastic, so re-running the same input may not reproduce the same trajectory or output. Linking feedback to replays Even if we can approximate the original run, evaluating a new or re-played trace against the old feedback is non-trivial. The feedback is textual, often high-level and contextual, not a simple scalar reward. Optimizing config and structure Finally, we want to optimize both the agent configuration (prompts, hyperparameters, tools, thresholds) and the agent graph/structure (which nodes, in what order, with what routing). Jointly optimizing these under noisy, text-based feedback is a challenging learning and search problem. In this notebook, using an agentic RAG example, we show how to operationalize this: 📝 Convert user feedback on agentic runs into an annotation benchmark on RELAI 🎯 Use the Maestro agent optimizer to consume that benchmark and automatically improve both the config and the graph of the agent 🔁 Close the loop from user preference → benchmark → optimization → better agent in a reproducible, data-driven way 🔗 Notebook: https://lnkd.in/eWXRxHEz Powered by RELAI (relai.ai)

Reinforcement Learning(RL) has quietly become one of the most important techniques shaping the evolution of LLM fine-tuning. For years, we optimized models through supervised learning, predicting the next token or minimizing cross-entropy loss. But as generative models scaled, we needed them to reason, align with intent, and adapt to human feedback in more complex ways. That’s where Reinforcement Learning (RL) entered the picture. At its core, RL is about interaction and feedback. An agent learns by interacting with an environment to maximize reward. In the context of large language models, the agent is the model itself. Each action is the next token it generates, and the reward is a signal derived from metrics or human preferences that measures how aligned the output is with the desired goal. Here’s a quick technical primer on the RL methods now powering GenAI fine-tuning: 1. RL Fine-Tuning (RLFT) We adapt a pre-trained model to new objectives like truthfulness, coherence, and safety using policy gradient algorithms such as PPO (Proximal Policy Optimization). Instead of minimizing loss, the model improves through iterative reward-driven optimization. 2. Reinforcement Learning from Human Feedback (RLHF) Human preference data trains a Reward Model (RM), which then guides fine-tuning through PPO. RLHF was key in aligning early LLMs, making outputs more helpful, factual, and instruction-following. 3. Direct Preference Optimization (DPO) A newer, more efficient approach. DPO skips the Reward Model and the full RL loop. It reframes alignment as a direct optimization task, teaching the model to prefer human-approved responses through a simplified objective function. It’s computationally stable, theoretically grounded in RL, and rapidly becoming a standard for GenAI alignment. Reinforcement Learning is no longer just a research concept. It is the foundation of how large language models learn to reason, align, and self-improve. ♻️ Share this with your network to spread learning 🔔Follow me for more data and AI insights

Stanford researchers just introduced a new way to optimize AI models using text-based feedback instead of traditional backpropagation! Deep learning has long relied on numerical gradients to fine-tune neural networks. But, optimizing generative AI systems has been much harder because they interact using natural language, not numbers. 𝗧𝗲𝘅𝘁𝗚𝗿𝗮𝗱 𝗶𝘀 𝘁𝗵𝗲 𝗳𝗶𝗿𝘀𝘁 𝗳𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 𝘁𝗼 𝗯𝗮𝗰𝗸𝗽𝗿𝗼𝗽𝗮𝗴𝗮𝘁𝗲 𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗺𝗼𝗱𝗲𝗹 𝗳𝗲𝗲𝗱𝗯𝗮𝗰𝗸, 𝗲𝗻𝗮𝗯𝗹𝗶𝗻𝗴 𝗔𝗜 𝘁𝗼 𝗶𝘁𝗲𝗿𝗮𝘁𝗶𝘃𝗲𝗹𝘆 𝗿𝗲𝗳𝗶𝗻𝗲 𝗶𝘁𝘀 𝗼𝘂𝘁𝗽𝘂𝘁𝘀 𝗮𝗰𝗿𝗼𝘀𝘀 𝗱𝗶𝘃𝗲𝗿𝘀𝗲 𝘁𝗮𝘀𝗸𝘀. 1. Improved AI performance in PhD-level science Q&A, raising accuracy from 51.0% to 55.0% on GPQA and from 91.2% to 95.1% on MMLU physics. 2. Optimized medical treatment plans, outperforming human-designed radiotherapy plans by better balancing tumor targeting and organ protection. 3. Enhanced AI-driven drug discovery by iteratively refining molecular structures, generating high-affinity compounds faster than traditional methods. 4. Boosted complex AI agents like Chameleon, increasing multimodal reasoning accuracy by 7.7% through iterative feedback refinement. 𝗧𝗵𝗲 𝘂𝘀𝗲 𝗼𝗳 "𝘁𝗲𝘅𝘁𝘂𝗮𝗹 𝗴𝗿𝗮𝗱𝗶𝗲𝗻𝘁𝘀" 𝗶𝗻𝘀𝘁𝗲𝗮𝗱 𝗼𝗳 𝗻𝘂𝗺𝗲𝗿𝗶𝗰𝗮𝗹 𝗴𝗿𝗮𝗱𝗶𝗲𝗻𝘁𝘀 𝗶𝘀 𝗽𝗿𝗲𝘁𝘁𝘆 𝗱𝗮𝗿𝗻 𝗰𝗼𝗼𝗹. It treats LLM feedback as “textual gradients” which are collected from every use of a variable in the system. By aggregating critiques from different contexts and iteratively updating variables (using a process analogous to numerical gradient descent), the method smooths out individual inconsistencies. 𝗜'𝗺 𝗰𝘂𝗿𝗶𝗼𝘂𝘀 𝗮𝗯𝗼𝘂𝘁 𝗵𝗼𝘄 𝗳𝗼𝗿𝗺𝗮𝗹𝗶𝘇𝗶𝗻𝗴 𝗺𝗲𝘁𝗵𝗼𝗱𝘀 𝘁𝗼 𝘃𝗮𝗹𝗶𝗱𝗮𝘁𝗲 𝗮𝗻𝗱 𝗰𝗼𝗻𝘀𝘁𝗿𝗮𝗶𝗻 𝘁𝗲𝘅𝘁𝘂𝗮𝗹 𝗴𝗿𝗮𝗱𝗶𝗲𝗻𝘁𝘀 beyond formalization of the propagation and update process via the equations could be developed to enhance robustness. Perhaps training secondary models to evaluate the quality and consistency of textual gradients or an ensemble approach of generating multiple textual gradients using different LLMs or multiple prompts? Just throwing some ideas out there; this stuff is pretty cool. Here's the awesome work: https://lnkd.in/gX8ABsdM Congrats to Mert Yuksekgonul, Federico Bianchi, Joseph Boen, James Zou, and co! I post my takes on the latest developments in health AI – 𝗰𝗼𝗻𝗻𝗲𝗰𝘁 𝘄𝗶𝘁𝗵 𝗺𝗲 𝘁𝗼 𝘀𝘁𝗮𝘆 𝘂𝗽𝗱𝗮𝘁𝗲𝗱! Also, check out my health AI blog here: https://lnkd.in/g3nrQFxW

Reinforcement learning from human feedback (RLHF)—the technique for getting large language models to follow instructions—usually operates in two separate phases: collect a large set of human judgments comparing pairs of model responses, then train a reward model (a network that scores responses) and use those scores to push the language model toward higher-rated outputs. Google DeepMind researchers instead interleave the two, updating both models as each new comparison arrives, and pick which response pairs to send to humans based on which choice would be most informative. Informativeness rests on a small ensemble of networks attached to the reward model, which produces a spread of reward predictions instead of a single number; pairs where ensemble members disagree most about the winner are the ones routed to humans. They report needing roughly 10x fewer human labels than standard offline RLHF at the 20K-comparison scale, and project a 1000x reduction at one million comparisons. Read with an AI tutor on ChapterPal: https://lnkd.in/eJK9aFER PDF: https://lnkd.in/ecQFNxcp

🤖 The Agent-as-a-Judge evaluation framework for AI systems 🤖 What is it? Agent-as-a-Judge is a novel framework that uses AI agents to evaluate other AI systems. Unlike traditional methods, it goes beyond just looking at final outcomes and delves into how these systems actually make decisions and solve problems. Why is it needed? Most current evaluations only look at the final product, missing the vital steps in the middle. This is like grading a student's final exam but never checking their homework or class participation. Moreover, having humans do the evaluations can be expensive, time-consuming, and sometimes inconsistent due to subjective opinions. How does it work? At its core, Agent-as-a-Judge integrates several specialized skills such as graph building, locating files, retrieving information, and checking requirements. It uses these skills to evaluate tasks from start to finish with the help of the Dev AI benchmarking dataset, which consists of 55 real-world AI tasks. This approach gives a full picture of how an AI system works through every step, offering insights often ignored by conventional methods. Why "Agent-as-a-judge"? LLM-as-a-Judge vs. Agent-as-a-Judge: The traditional LLM-as-a-Judge approach evaluates AI systems mainly by looking at their final outputs, much like an exam result. Agent-as-a-Judge not only looks at these outputs but also evaluates how the AI got there, providing feedback on every stage of the process. This means it's like monitoring both the journey and the destination. Intermediate Feedback: Agent-as-a-Judge provides rich, ongoing feedback during the task-solving process, much like a teacher guiding a student through each step of a math problem, not just checking the final answer. System Complexity: While LLM-as-a-Judge focuses on static inputs and outputs, Agent-as-a-Judge uses multiple tools to get a holistic view, assessing not just what the AI does but how it does it. Challenges and opportunities: Although Agent-as-a-Judge is promising, it's important to note some challenges like optimizing its components and testing its adaptability beyond just coding tasks. Also, combining its strengths with other methods (like enhancing LLMs with retrieval skills) could create a powerful hybrid approach to AI evaluation. What’s next? Agent-as-a-Judge opens up exciting new possibilities for AI evaluation. As we refine this method, we pave the way for potentially phasing out human evaluations entirely. Link to the paper -> https://lnkd.in/gfYrXpHt #AI #Innovation #AgentAsAJudge #DevAI #AIDevelopment #MachineLearning

User Feedback Loops: the missing piece in AI success? AI is only as good as the data it learns from -- but what happens after deployment? Many businesses focus on building AI products but miss a critical step: ensuring their outputs continue to improve with real-world use. Without a structured feedback loop, AI risks stagnating, delivering outdated insights, or losing relevance quickly. Instead of treating AI as a one-and-done solution, companies need workflows that continuously refine and adapt based on actual usage. That means capturing how users interact with AI outputs, where it succeeds, and where it fails. At Human Managed, we’ve embedded real-time feedback loops into our products, allowing customers to rate and review AI-generated intelligence. Users can flag insights as: 🔘Irrelevant 🔘Inaccurate 🔘Not Useful 🔘Others Every input is fed back into our system to fine-tune recommendations, improve accuracy, and enhance relevance over time. This is more than a quality check -- it’s a competitive advantage. - for CEOs & Product Leaders: AI-powered services that evolve with user behavior create stickier, high-retention experiences. - for Data Leaders: Dynamic feedback loops ensure AI systems stay aligned with shifting business realities. - for Cybersecurity & Compliance Teams: User validation enhances AI-driven threat detection, reducing false positives and improving response accuracy. An AI model that never learns from its users is already outdated. The best AI isn’t just trained -- it continuously evolves.

🚀 Exploring Feedback Loops in Language Models: A Double-Edged Sword! 📢 Have you considered how feedback loops in AI can amplify unintended consequences? 🔬 The recent research, "Feedback Loops With Language Models Drive In-Context Reward Hacking", highlights critical dynamics in how language models interact with feedback systems. 🌟 Key Takeaways:- 👉 In-Context Reward Hacking (ICRH) - Language models often optimize for specific objectives in ways that unintentionally lead to undesirable outputs. 📌 Two Feedback Loop Mechanisms - 1️⃣ Output-Refinement - It repeatedly improving outputs based on feedback can amplify biases or errors. 🔍 Example: In content moderation systems, refining outputs for strict compliance can result in over-censorship or loss of nuance. 2️⃣ Policy-Refinement - It adapting the model’s decision-making to feedback can cause unintended policy shifts. 🔍 Example: Customer support chatbots may overly prioritize high ratings, offering refunds unnecessarily to ensure positive feedback. 🌎 Real-World Implications:- 🗝️ Gen AI in Content Creation - When feedback prioritizes engagement, AI may generate clickbait or sensational content to maximize metrics. 🗝️ Personalized Recommendations - Systems adapting to user feedback may create echo chambers, reinforcing specific preferences while ignoring diverse perspectives. 🛠️ Why It Matters? As AI systems become more ubiquitous, their ability to self-optimize through feedback is both a strength and a potential risk. 👭 We must develop robust strategies to:- 🌐 Detect unintended behaviors early. 🌎 Ensure ethical and aligned outcomes. ⌛ Foster transparency in AI feedback systems. 📖 Read the full paper for deeper insights - https://lnkd.in/dp42viKq 👉 What strategies do you think could help mitigate these challenges in feedback systems? Let’s discuss in the comments! 💬