Software Testing Best Practices

Two strikingly similar headlines surfaced this past week that should make every leader pause: • “Companies Are Pouring Billions Into A.I. It Has Yet to Pay Off.” — New York Times • “Companies Are Pouring Billions Into AI. Here’s Why They’re Not Seeing Returns” — Forbes The NYT points to the human side: employees resist tools they don’t trust. Forbes focuses on the technical side: most AI still can’t understand the context of work. Both are true, and they’re related. When AI lacks context, employees lose trust. It can’t tell the latest doc from last year’s draft. It summarizes a customer conversation but drops the follow-ups buried in the thread. It pulls a response from Slack while ignoring the context in Google Drive. Employees realize it creates more work than it saves, and stop using it. Pilots stall, deployments fade, and projects slide into the “trough of disillusionment" as the NYT describes. Unfortunately, that's the reality for many organizations. At Glean, we work hard to make sure AI understands the enterprise context the way a human does. If a subject matter expert says something, I trust it more. If something’s old, I double-check it. That’s how people think, and it’s how AI should work too. Yet every enterprise has its own documentation culture and quirks, so sometimes we struggle at first. But we persist and co-develop with customers until the system reaches the quality they need. Then we take those learnings to make it work automatically for the next customer. We’ve seen this approach deliver measurable impact for customers: • Booking.com: Glean Agents give teams faster access to customer insights, cutting video production time by 75% and doubling monthly output. • Confluent: Glean’s AI-powered search saves 15,000+ hours/month, boosts support satisfaction by 13%, and cuts ticket investigation time by 10 minutes. • Fortune 100 telecom company: Glean surfaces instant knowledge during support calls, reducing call resolution time by 17 seconds across 800+ agents. • Leading global consultancy: Glean Agents automate RFP workflows, cutting consulting project proposals from 4 weeks to a few hours (97% faster). • Wealthsimple: Glean gives employees instant access to policies and knowledge, driving $1M+ in annual productivity gains. When AI understands the real context of work—across people, tools, and workflows— employees trust it and use it. Instead of falling into the trough of disillusionment, companies climb a slope toward productivity gains and real ROI.

Alignment without context integrity is not safety. An AI agent can faithfully follow its instructions and still be dangerous if its understanding of reality is wrong. Anthropic recently disclosed that Claude models gained unauthorized access to the real systems of three organizations during cybersecurity evaluations. The agents had been told they were operating in a simulation with no internet access. But a configuration mistake gave them access to the live internet. They treated real production systems as part of the exercise and kept pursuing the goal they had been given. OpenAI separately disclosed that models found a previously unknown vulnerability, escaped an isolated evaluation environment and compromised Hugging Face. These were not simply failures of intelligence. The deeper problem was that the agents were acting inside a false understanding of reality. We have spent years asking whether an AI system will follow our instructions. We now also need to ask whether it correctly understands the environment in which those instructions are being executed. This creates a new security requirement. Context integrity. Before an agent acts, the system must continuously verify where it is, which resources are in scope, whose authority it carries, what it is allowed to do, and when that authority expires. Just in time permission for every action. At just the right time. For just enough time. Assessed in real time. Those facts cannot live only inside a prompt. They must be verified and enforced by the infrastructure around the model. A prompt is not a security boundary. Zero trust taught us to never trust identity and always verify access. And provide least privileged access. Agentic AI adds another dimension. Never blindly trust context. Continuously verify reality. The next security perimeter is not just the agent’s identity. It is the agent’s understanding of reality. The most dangerous agent may not be misaligned. It may simply be mistaken. And in an agentic world, a false belief can become a real breach.

Knowledge Graphs as a source of trust for LLM-powered enterprise question answering That has been our position from the beginning when we started our research of understanding how knowledge graphs increase the accuracy of LLM-powered question answering systems over 2 years ago!  The intersection of knowledge graphs and large language models (LLMs) isn’t theoretical anymore. It's been a game-changer for enterprise question answering and now everyone is talking about it and many are doing it. 🚀 This new paper is a summary of our lessons learned of implementing this technology in data.world and working with customers, and outline the opportunities for future research contributions and where the industry needs to go (guess where the data.world AI Lab is focusing). Sneak peek and link in the comments Lessons Learned ✅ Knowledge engineering is essential but underutilized: Across organizations, it’s often sporadic and inconsistent, leading to assumptions and misalignment. It’s time to systematize this critical work. ✅ Explainability builds trust: Showing users exactly how an answer is derived, including auto-corrections, increases transparency and confidence. ✅ Governance matters: Aligning answers with an organization’s business glossary ensures consistency and clarity. ✅ Avoid “boiling the ocean”: don’t tackle too many questions at once A pay-as-you-go approach ensures meaningful progress without overwhelm. ✅ Testing matters: Non-deterministic systems like LLMs require new frameworks to test ambiguity and validate responses effectively. Where the Industry Needs to Go 🌟 Simplified knowledge engineering: Tools and methodologies must make this foundational work easier for everyone. 🌟 User-centric explainability: Different users have different needs so we need to focus on “explainable to whom?”. 🌟 Testing non-deterministic systems: The deterministic models of yesterday won’t cut it. We need innovative frameworks to ensure quality in LLMs powered software applications. 🌟 Small semantics vs. Larger semantics: The concept of semantics is being increasingly referenced in industry in the context of “semantic layers” for BI and Analytics. Let’s close the gap between the small semantics (fact/dimension modeling) and large semantics (ontologies, taxonomies) 🌟 Multi-agent systems: break down the problem into smaller, more manageable components. Should an agent deal with the core task of answering questions and managing ambiguity, or should these be split into separate agents? This research reflects our commitment to co-innovate with customers to solve real-world challenges in enterprise AI. 💬 What do you think? How are knowledge graphs shaping your AI strategies?

We all have our go-to test cases. But sometimes, the sneaky bugs hide in the places we don’t think to look. Here are 12 test scenarios that often get missed: 🔄 Using the browser back button 💤 Leaving a session idle, then coming back 📱 Switching from WiFi to mobile data mid-action 🔍 Zooming in/out on mobile or desktop 💾 Saving without filling all required fields 🕓 Schedule submitting a form right at midnight (date-related edge cases!) 🔗 Opening multiple tabs of the same app ⛔ Trying activities with limited user permissions 🌍 Using different keyboard layouts with language settings ✍️ Inputting emojis, special characters, or long text 🧪 Rapid double-clicking or tapping buttons 📥 Uploading weird file types or broken files These may seem small… until they break something in production. Sometimes it’s the “what if” moments that make the biggest difference. What would you add to the list? QA Touch #softwaretesting #edgecases #exploratorytesting #QA #bughunting #testingtips #qa #QATouch #qatouch

AI models like ChatGPT and Claude are powerful, but they aren’t perfect. They can sometimes produce inaccurate, biased, or misleading answers due to issues related to data quality, training methods, prompt handling, context management, and system deployment. These problems arise from the complex interaction between model design, user input, and infrastructure. Here are the main factors that explain why incorrect outputs occur: 1. Model Training Limitations AI relies on the data it is trained on. Gaps, outdated information, or insufficient coverage of niche topics lead to shallow reasoning, overfitting to common patterns, and poor handling of rare scenarios. 2. Bias & Hallucination Issues Models can reflect social biases or create “hallucinations,” which are confident but false details. This leads to made-up facts, skewed statistics, or misleading narratives. 3. External Integration & Tooling Issues When AI connects to APIs, tools, or data pipelines, miscommunication, outdated integrations, or parsing errors can result in incorrect outputs or failed workflows. 4. Prompt Engineering Mistakes Ambiguous, vague, or overloaded prompts confuse the model. Without clear, refined instructions, outputs may drift off-task or omit key details. 5. Context Window Constraints AI has a limited memory span. Long inputs can cause it to forget earlier details, compress context poorly, or misinterpret references, resulting in incomplete responses. 6. Lack of Domain Adaptation General-purpose models struggle in specialized fields. Without fine-tuning, they provide generic insights, misuse terminology, or overlook expert-level knowledge. 7. Infrastructure & Deployment Challenges Performance relies on reliable infrastructure. Problems with GPU allocation, latency, scaling, or compliance can lower accuracy and system stability. Wrong outputs don’t mean AI is "broken." They show the challenge of balancing data quality, engineering, context management, and infrastructure. Tackling these issues makes AI systems stronger, more dependable, and ready for businesses. #LLM

When you work with AI, you'll eventually hit an inflection point. The more data you feed it, the worse it gets. You'd think more data means better quality, but it works the other way. I recently talked to a $60M/yr manufacturer that hit this wall. They stood up an AI agent to handle all of their customer support. The demo looked awesome -- it answered everything and sounded great, so they shipped it. It ran for a couple months just fine. They kept feeding it with new data: docs, support tickets, spec sheets, policies, etc.. The logic being that any company knowledge you add makes the agent smarter. But it got dumber 🤯 And they noticed that it started with the little things, like changing the validity dates on a return policy. Then it began to blend answers together that weren't quite right, even though they sounded right. A couple months in, they trusted it way less than the day they launched and now they're in hot water. An LLM doesn't "learn" about your business when you dump it with data. Doing that actually makes it harder for it to retrieve the right thing, because now there's more info to grab from. More data that's unstructured, outdated, or flat out contradicting something else in the system. You don't increase accuracy by adding context. It's kinda the opposite. We're working with the needle in a haystack, but the needle stays the same size while the haystack gets huge. The fix is not switching to a better model. It's building the backend that can actually handle all of it: - structure the data before it ever reaches the model. what's authoritative? what's stale? what takes precedence? - organize the memory with semantic mapping, so the model knows how one piece of data relates to another. in the FDE industry, this is called an "ontology". - keep humans in the loop who approve answers and can rate confidence levels on anything that carries risk. this improves responses over time, rather than degrading them. There's a big difference between throwing data into a greenfield project or demo and saying it works great, versus building an AI system that you'd actually ship to a customer and gets better over time. If you've watched your own AI tool get more confident but less right over time, you're not crazy. And you didn't pick the wrong system. You just skipped the data foundation which everything else gets built upon. It's a hard problem to solve, and one that I'm actively working on. Have any of you hit these data/context walls and noticed their AI getting more stupid over time?

22 Test Automation Framework Practices That Separate Good SDETs from Great Ones Here's what actually works: 1. KISS Principle Break complex tests into smaller modules. Avoid singletons that kill parallel execution. Example: Simple initBrowser() method instead of static WebDriver instances. 2. Modular Approach Separate test data, utilities, page objects, and execution logic. Example: LoginPage class handles only login elements and actions. 3. Setup Data via API/DB Never use UI for test preconditions. It's slow and flaky. Example: RestAssured POST to create test users before running tests. 4. Ditch Excel for Test Data Use JSON, XML, or CSV. They're faster, easier to version control, and actually work. Example: Jackson ObjectMapper to read JSON into POJOs. 5. Design Patterns Factory: Create driver instances based on browser type Strategy: Switch between different browser setups Builder: Construct complex test objects step by step 6. Static Code Analysis SonarLint catches unused variables and potential bugs while you code. 7. Data-Driven Testing Run same test with multiple data sets using TestNG DataProvider. Example: One login test, 10 different user credentials. 8. Exception Handling + Logging Log failures properly. Future you will thank present you. Example: Logger.severe() with meaningful error messages. 9. Automate the Right Tests Focus on repetitive, critical tests. Each test must be independent. 10. Wait Utilities WebDriverWait with explicit conditions. Never Thread.sleep(). Example: wait.until(ExpectedConditions.visibilityOfElementLocated()) 11. POJOs for API Type-safe response handling using Gson or Jackson. Example: Convert JSON response directly to User object. 12. DRY Principle Centralize common locators and setup/teardown in BaseTest class. 13. Independent Tests Each test sets up and cleans up its own data. Enables parallel execution. 14. Config Files URLs, credentials, environment settings—all in external properties files. Example: ConfigReader class to load properties. 15. SOLID Principles Single responsibility per class. Test logic separate from data and helpers. 16. Custom Reporting ExtentReports with screenshots, logs, and environment details. 17. Cucumber Reality Check If you're not doing full BDD, skip Cucumber. It adds complexity without value. 18. Right Tool Selection Choose based on project needs, not trends. Evaluate maintenance cost. 19. Atomic Tests One test = one feature. Fast, reliable, easy to maintain. 20. Test Pyramid Many unit tests (fast) → Some API tests → Few UI tests (slow). 21. Clean Test Data Create in @BeforeMethod, delete in @AfterMethod. Zero data pollution. 22. Data-Driven API Tests Dynamic assertions, realistic data, POJO response validation. Which practice transformed your framework the most? -x-x- Most asked SDET Q&A for 2025 with SDET Coding Interview Prep (LeetCode): https://lnkd.in/gFvrJVyU #japneetsachdeva

Conversational AI is transforming customer support, but making it reliable and scalable is a complex challenge. In a recent tech blog, Airbnb’s engineering team shares how they upgraded their Automation Platform to enhance the effectiveness of virtual agents while ensuring easier maintenance. The new Automation Platform V2 leverages the power of large language models (LLMs). However, recognizing the unpredictability of LLM outputs, the team designed the platform to harness LLMs in a more controlled manner. They focused on three key areas to achieve this: LLM workflows, context management, and guardrails. The first area, LLM workflows, ensures that AI-powered agents follow structured reasoning processes. Airbnb incorporates Chain of Thought, an AI agent framework that enables LLMs to reason through problems step by step. By embedding this structured approach into workflows, the system determines which tools to use and in what order, allowing the LLM to function as a reasoning engine within a managed execution environment. The second area, context management, ensures that the LLM has access to all relevant information needed to make informed decisions. To generate accurate and helpful responses, the system supplies the LLM with critical contextual details—such as past interactions, the customer’s inquiry intent, current trip information, and more. Finally, the guardrails framework acts as a safeguard, monitoring LLM interactions to ensure responses are helpful, relevant, and ethical. This framework is designed to prevent hallucinations, mitigate security risks like jailbreaks, and maintain response quality—ultimately improving trust and reliability in AI-driven support. By rethinking how automation is built and managed, Airbnb has created a more scalable and predictable Conversational AI system. Their approach highlights an important takeaway for companies integrating AI into customer support: AI performs best in a hybrid model—where structured frameworks guide and complement its capabilities. #MachineLearning #DataScience #LLM #Chatbots #AI #Automation #SnacksWeeklyonDataScience – – –  Check out the "Snacks Weekly on Data Science" podcast and subscribe, where I explain in more detail the concepts discussed in this and future posts:    -- Spotify: https://lnkd.in/gKgaMvbh   -- Apple Podcast: https://lnkd.in/gj6aPBBY    -- Youtube: https://lnkd.in/gcwPeBmR https://lnkd.in/gFjXBrPe

Building LLM apps? Learn how to test them effectively and avoid common mistakes with this ultimate guide from LangChain! 🚀 This comprehensive document highlights: 1️⃣ Why testing matters: Tackling challenges like non-determinism, hallucinated outputs, and performance inconsistencies. 2️⃣ The three stages of the development cycle: 💥 Design: Incorporating self-corrective mechanisms for error handling (e.g., RAG systems and code generation). 💥Pre-Production: Building datasets, defining evaluation criteria, regression testing, and using advanced techniques like pairwise evaluation. 💥Post-Production: Monitoring performance, collecting feedback, and bootstrapping to improve future versions. 3️⃣ Self-corrective RAG applications: Using error handling flows to mitigate hallucinations and improve response relevance. 4️⃣ LLM-as-Judge: Automating evaluations while reducing human effort. 5️⃣ Real-time online evaluation: Ensuring your LLM stays robust in live environments. This guide offers actionable strategies for designing, testing, and monitoring your LLM applications efficiently. Check it out and level up your AI development process! 🔗📘 ------------ Add your thoughts in the comments below—I’d love to hear your perspective! Sarveshwaran Rajagopal #AI #LLM #LangChain #Testing #AIApplications