Data Analysis Techniques For Engineers

Over the last year, I’ve seen many people fall into the same trap: They launch an AI-powered agent (chatbot, assistant, support tool, etc.)… But only track surface-level KPIs — like response time or number of users. That’s not enough. To create AI systems that actually deliver value, we need 𝗵𝗼𝗹𝗶𝘀𝘁𝗶𝗰, 𝗵𝘂𝗺𝗮𝗻-𝗰𝗲𝗻𝘁𝗿𝗶𝗰 𝗺𝗲𝘁𝗿𝗶𝗰𝘀 that reflect: • User trust • Task success • Business impact • Experience quality    This infographic highlights 15 𝘦𝘴𝘴𝘦𝘯𝘵𝘪𝘢𝘭 dimensions to consider: ↳ 𝗥𝗲𝘀𝗽𝗼𝗻𝘀𝗲 𝗔𝗰𝗰𝘂𝗿𝗮𝗰𝘆 — Are your AI answers actually useful and correct? ↳ 𝗧𝗮𝘀𝗸 𝗖𝗼𝗺𝗽𝗹𝗲𝘁𝗶𝗼𝗻 𝗥𝗮𝘁𝗲 — Can the agent complete full workflows, not just answer trivia? ↳ 𝗟𝗮𝘁𝗲𝗻𝗰𝘆 — Response speed still matters, especially in production. ↳ 𝗨𝘀𝗲𝗿 𝗘𝗻𝗴𝗮𝗴𝗲𝗺𝗲𝗻𝘁 — How often are users returning or interacting meaningfully? ↳ 𝗦𝘂𝗰𝗰𝗲𝘀𝘀 𝗥𝗮𝘁𝗲 — Did the user achieve their goal? This is your north star. ↳ 𝗘𝗿𝗿𝗼𝗿 𝗥𝗮𝘁𝗲 — Irrelevant or wrong responses? That’s friction. ↳ 𝗦𝗲𝘀𝘀𝗶𝗼𝗻 𝗗𝘂𝗿𝗮𝘁𝗶𝗼𝗻 — Longer isn’t always better — it depends on the goal. ↳ 𝗨𝘀𝗲𝗿 𝗥𝗲𝘁𝗲𝗻𝘁𝗶𝗼𝗻 — Are users coming back 𝘢𝘧𝘵𝘦𝘳 the first experience? ↳ 𝗖𝗼𝘀𝘁 𝗽𝗲𝗿 𝗜𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻 — Especially critical at scale. Budget-wise agents win. ↳ 𝗖𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻 𝗗𝗲𝗽𝘁𝗵 — Can the agent handle follow-ups and multi-turn dialogue? ↳ 𝗨𝘀𝗲𝗿 𝗦𝗮𝘁𝗶𝘀𝗳𝗮𝗰𝘁𝗶𝗼𝗻 𝗦𝗰𝗼𝗿𝗲 — Feedback from actual users is gold. ↳ 𝗖𝗼𝗻𝘁𝗲𝘅𝘁𝘂𝗮𝗹 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 — Can your AI 𝘳𝘦𝘮𝘦𝘮𝘣𝘦𝘳 𝘢𝘯𝘥 𝘳𝘦𝘧𝘦𝘳 to earlier inputs? ↳ 𝗦𝗰𝗮𝗹𝗮𝗯𝗶𝗹𝗶𝘁𝘆 — Can it handle volume 𝘸𝘪𝘵𝘩𝘰𝘶𝘵 degrading performance? ↳ 𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆 — This is key for RAG-based agents. ↳ 𝗔𝗱𝗮𝗽𝘁𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗦𝗰𝗼𝗿𝗲 — Is your AI learning and improving over time? If you're building or managing AI agents — bookmark this. Whether it's a support bot, GenAI assistant, or a multi-agent system — these are the metrics that will shape real-world success. 𝗗𝗶𝗱 𝗜 𝗺𝗶𝘀𝘀 𝗮𝗻𝘆 𝗰𝗿𝗶𝘁𝗶𝗰𝗮𝗹 𝗼𝗻𝗲𝘀 𝘆𝗼𝘂 𝘂𝘀𝗲 𝗶𝗻 𝘆𝗼𝘂𝗿 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀? Let’s make this list even stronger — drop your thoughts 👇

Many Digital Twin projects fail. Why? The #1 killer of DT projects is: Data Preprocessing. A true Digital Twin isn't a model. It's an engine. And the fuel for that engine is data. But how do you build the plumbing? How do you get data from your physical asset into your virtual model and then get valuable insights back out? Here’s the 5-step breakdown of the engine you actually need to build: Step 1: Data Acquisition Your engine is useless without fuel. This starts at the source. - IIoT Sensors: These are the nerves of your asset. They measure pressure, temperature, vibration, location—whatever matters. If you can't sense it, you can't twin it 😂 - Real-time Transmission: The data can't be a day old. You need a high-speed data bus (like MQTT, OPC-UA) to transmit that sensor data now. - Data Preprocessing: Again, this is the #1 killer of DT projects. Raw sensor data is dirty. It's noisy, full of gaps, and in the wrong format. You MUST clean, normalize, and filter it before it goes anywhere else. Step 2: The Modeling Now your clean data has somewhere to go. - Digital Twin Construction: You map the data streams to the virtual asset. "Sensor 1A" is now officially the "vibration reading for Pump 7." - Virtual Model: This isn't just a 3D drawing. This is a physics-based or ML model. It understands thermodynamics, material fatigue, or fluid dynamics. This is where the data gets context. Step 3: Analytics This is where the ROI lives. The engine is running. Now, what does it do? Predictive Analytics: Your model takes the data and simulates "what if?" What happens if I increase the load by 20%? When will this specific component fail? - High-Performance Computing (HPC): These complex simulations can't run on a laptop. You need the horsepower to process massive data streams and run complex algorithms instantly. Your data is no longer just describing the past. It's actively predicting and optimizing the future. Step 4 & 5: Security & Standards Your high-performance engine needs a chassis to hold it together. Amateurs forget this. Pros build it first. - Cybersecurity & Privacy: You just connected your most critical physical assets to the cloud. Securing this isn't an afterthought; it's priority #1. - Interoperability Standards: Your sensors, software, and platforms must speak the same language. If you build a proprietary, closed system, you're building technical debt. Plan for an open architecture, always. -------- Follow me for #digitaltwins Links in my profile Florian Huemer

🔬 Thermogravimetric Analysis (TGA) & Differential Scanning Calorimetry (DSC): Understanding Thermal Stability & Phase Behavior of Materials Thermal analysis techniques such as TGA and DSC are indispensable tools in materials science, catalysis, polymers, pharmaceuticals, and energy-related research. These techniques help us understand how materials respond to temperature in terms of mass changes, thermal stability, phase transitions, and reaction energetics. 🔹 What is TGA? Thermogravimetric Analysis (TGA) measures the change in mass of a sample as a function of temperature or time under a controlled atmosphere. 📌 Key Information from TGA: 👉 Moisture and volatile content 👉Thermal stability range 👉Decomposition temperatures 👉Oxidation/reduction behavior 👉Coke or carbon deposition on catalysts 👉Ash or residue content 📌 Common Atmospheres Used: Nitrogen / Argon → inert conditions Air / Oxygen → oxidation studies Hydrogen → reduction behavior 📌 Typical Applications in Catalysis: Determination of coke formation after reaction Stability of fresh vs spent catalysts Decomposition of precursor salts Calcination temperature optimization 🔹 What is DSC? Differential Scanning Calorimetry (DSC) measures the heat flow associated with physical or chemical transitions in a material as a function of temperature. 📌 Information Obtained from DSC: Glass transition temperature (Tg) Melting temperature (Tm) Crystallization temperature (Tc) Phase transitions Reaction enthalpy (endothermic/exothermic events) 📌 Why DSC Matters: Understanding phase purity Identifying polymorphic transformations Studying crystallinity and amorphous content Thermal behavior of polymers and composites 🔹 How to Interpret TGA Curve? A typical TGA curve consists of mass (%) vs temperature: Initial weight loss → moisture or adsorbed species Major weight loss step → decomposition of material Final plateau → residual stable phase 👉 Derivative TGA (DTG) peaks help pinpoint exact decomposition temperatures. 🔹 How to Interpret DSC Curve? DSC plots heat flow vs temperature: Endothermic peaks → melting, evaporation, desorption Exothermic peaks → crystallization, oxidation, curing Peak area → enthalpy change (ΔH) 🔹 Combining TGA + DSC When TGA and DSC are used together: ✅ Correlate mass loss with heat events ✅ Distinguish physical vs chemical transitions ✅ Obtain deeper insight into reaction mechanisms This combined approach is extremely powerful for catalyst development, material design, and process optimization. 💡 Key Takeaway TGA tells how much mass changes, while DSC tells how much energy is involved. Together, they provide a complete picture of a material’s thermal behavior. ✍️ Kanchan Guru DST INSPIRE Fellow (SRF) Department of Chemistry, Manipal University Jaipur Subscribe to Research Decoded newsletter for more insights on characterization & catalysis https://lnkd.in/g74ryQ66

Over the weekend, Andrej Karpathy shared this tweet & it inspired me to conduct the 2024 GTM Survey Analysis this way. I use a language called R to analyze data because of its ability to generate pretty charts & the depth of its statistical analysis tools. Within 90 minutes I had found 9 key data points in the data that were statistically significant & runs more than 50 analyses. In the past this kind of work would have taken me 30 to 40 hours. Programming is becoming prompting. I used to write something like this to generate a chart : ggplot(data) + geom_bar(aes(x = variable, y = value), stat = "identity") + theme_minimal() + labs(title = "Title", caption = "Caption") Copilot autocompleted the different fields. But using Sonnet & Cursor, I first wrote “Perform a conjoined analysis, comparing the correlation across all variables within the data frame. Plot this on a bar chart using my particular theme, with an insightful title & a caption for Theory Ventures.” Then I wrote “Run the same analysis for sales quota compared to company size.” Next, “how about NDR for company size?” Each time, the robot produced 150 lines of code in seconds. More than just the code, I request a test for statistical significance. I remembered from statistics class in college to perform a t-test for comparing two means when the sample size is greater than 35. But I had forgotten how to compare the means across more than two groups. ANOVA to the rescue. All of the code is formatted according to proper syntax & it works. The only errors I found concerned color palette specifications. English is the new programming language. Coding this way, I explored the data much more deeply, more rigorously, & more quickly than I would have otherwise. The user still needs to be aware of the underlying syntax to fix errors & some statistical tests to verify the computer is doing the right thing, but gone are the days of memorizing the functional arcana of individual programming libraries. In other words, I’m operating at a higher level of abstraction. Though it may not seem this way, the user interface of data exploration has changed. It’s a back & forth with the computer, a conversation, a dialogue with ongoing output. I’m thinking about the next analysis, not the next functional argument.

The Technology Stack Behind a Real #Digital #Twin Every impressive Digital Twin demo usually hides one important question: Where does all that data actually come from? A Digital Twin isn't a single application. It's an ecosystem of technologies continuously exchanging data to model the physical world. A simplified architecture looks like this: #Layer #1 — #Physical #World Everything starts with real assets. • Buildings • Roads • Bridges • Power Lines • Wind Turbines • Factories • Water Pipelines These assets generate enormous amounts of data. #Layer #2 — #Data #Acquisition Multiple technologies observe the same asset from different perspectives. • IoT Sensors → Temperature, vibration, pressure, strain • Drone Mapping → Orthomosaics, point clouds, 3D meshes • LiDAR → High-density geometry • Satellite Imagery → Large-scale monitoring • CCTV Cameras → Visual inspection • Mobile Mapping → Street-level updates • SCADA Systems → Operational telemetry No single sensor tells the whole story. Sensor fusion creates the complete picture. #Layer #3 — #Data #Engineering Raw data is rarely usable. It must be: • Cleaned • Registered • Georeferenced • Time synchronized • Converted into common coordinate systems • Indexed • Version controlled Without this layer, your Digital Twin becomes inconsistent within weeks. #Layer #4 — #Spatial #Data #Platform This is where everything connects. Typical datasets include: • GIS Layers • BIM Models • Point Clouds • Meshes • Terrain Models • Utility Networks • Asset Inventories • Time-series Sensor Data Every object receives a unique identity. Now a bridge isn't just geometry. It's linked to inspections, maintenance logs, sensor history, drawings, documents, and operational events. #Layer #5 — #Intelligence This is where AI becomes valuable. Machine Learning models can: • Detect structural defects • Predict equipment failures • Estimate Remaining Useful Life (RUL) • Forecast maintenance costs • Detect anomalies • Simulate future scenarios • Optimize operations Instead of dashboards, you begin receiving recommendations. #Layer #6 — #Applications Finally, different teams consume the Digital Twin. • Operations • Maintenance • Engineering • Asset Management • Emergency Response • City Planning • Executives Everyone works from the same continuously updated source of truth. A mature Digital Twin is less about visualization and more about data architecture. The hardest challenge isn't rendering millions of points in 3D. It's integrating dozens of heterogeneous systems into a reliable, real-time representation of reality. That's what transforms a collection of datasets into a Digital Twin. #DigitalTwin #AI #MachineLearning #IoT #GIS #LiDAR #DroneMapping #ComputerVision #SpatialComputing #SmartInfrastructure #Engineering #DataEngineering #AssetManagement

Your model is trained. But is it actually good? Most ML engineers default to accuracy. Then wonder why their model fails in production. Here are 20 evaluation metrics — and when to actually use each one: Classification: - Accuracy → Balanced datasets only. - Precision → When false positives are costly. - Recall → When false negatives matter more. - F1 Score → Imbalanced datasets. Balances both. - ROC-AUC → Binary classification evaluation. - Log Loss → Probabilistic models. Penalizes confident wrong predictions. - Confusion Matrix → Error analysis. See exactly where it breaks. - Specificity → When detecting negatives correctly matters. - Balanced Accuracy → Uneven datasets. Don't trust plain accuracy here. Regression: - MAE → Simple, interpretable error measurement. - MSE → Penalizes larger errors more heavily. - RMSE → Error in original scale. Most interpretable. - R² Score → How much variance your model explains. - Adjusted R² → Feature-heavy models. Adjusts for complexity. - MAPE → Business forecasting. Error as a percentage. - Explained Variance → Model consistency evaluation. Clustering: - Silhouette Score → Cluster cohesion and separation. Cluster validation. - Davies-Bouldin Index → Lower is better clustering. NLP: - BLEU Score → Machine translation quality. - ROUGE Score → Text summarization quality. Accuracy is not a strategy. Picking the right metric for the right problem is. A model that looks great on accuracy can destroy real-world outcomes when the wrong metric guided its evaluation. Save this. 📌 Which metric do most engineers misuse? 👇

🎯 Why Most Business Problems Remain Unsolved (And How to Fix That) Last week, I had the privilege of facilitating a Problem Solving & Business Acumen workshop for our teams at L'Oréal Indonesia. 💡 The Problem We All Face (But Rarely Talk About) Here's an uncomfortable truth: we're wired to jump to solutions. In business, this looks like: ✔️ Launching promotions without understanding why sales declined ✔️ Hiring more people without diagnosing process inefficiencies ✔️ Copying competitor tactics without validating if they fit our context The cost? Wasted resources, frustrated teams, and recurring problems that never truly go away. According to the World Economic Forum's Future of Jobs Report 2023, analytical and critical thinking are the #1 and #2 most important skills for workers. Yet, most of us were never formally taught how to think critically or solve problems systematically. 🛠️ The Problem-Solving Process: A Step-by-Step Guide Step 1: Define the Problem (Don't Jump to Judgment!) 📝 Craft a Problem Statement with 6 components: "How can [responsible party] improve/reduce [reality] to meet [expectation] within [timeline] without [anti-goals], in order to fulfill [reason]?" Example: "How can the product team launch a new product on time in Q4 2024 without sacrificing key processes, in order to meet the sales target?" Step 2: Find Alternatives (Issue Tree + MECE) Once the problem is clear, break it down using an Issue Tree. For instance, if mascara sales dropped -14% YoY: 📦 Placement → Gondola compliance, visibility, signage 🎁 Promotion → BOGO mechanics, POS materials 💰 Price → Elasticity, perceived value 🎨 Product Claims → Content freshness, reviews 🔥 Competition → Share of voice, endcap presence ✅ Ensure hypotheses are MECE (Mutually Exclusive, Collectively Exhaustive)—no overlaps, no gaps. Step 3: Test Your Hypotheses Don't fall in love with your first idea. Run quick tests: 📊 For a skincare serum declining in pharmacies, we tested: ✔️ Hypothesis A: Reduced pharmacist advocacy is the issue → Micro-detailing pilot in 10 stores ✔️ Hypothesis B: Cold chain OOS drives lost sales → Warehouse SOP audit + temperature logs ✔️ Hypothesis C: Execution gaps suppress promo ROI → Endcap compliance audit Each hypothesis had clear KPIs and timelines—no guessing, just data. Step 4: Make the Decision (Impact vs. Effort Matrix) Not all solutions are equal. Prioritize: 🟩 Quick wins—do this! 🟦 Strategic bets 🟨 Fill-ins 🟥 Avoid Focus on low effort, high impact moves first. Build momentum, then tackle the big bets. 🚨 What Happens When We Skip These Steps? A mascara brand saw sales drop -14% YoY. The reaction? "Let's run a BOGO promo!" The result? Sales stayed flat. Why? Because the real issues were: ❌ Poor gondola compliance (only 68% correct facings) ❌ Weak influencer share of voice ❌ Competitor secured prime endcap space The lesson: Solutions applied to the wrong problem = wasted budget and missed targets.

For years, we talked about Digital Twins as a visualisation tool. A smarter, live version of the BIM model. Something impressive to show clients in a project review. That conversation has shifted dramatically in 2026. AI-driven Digital Twins are moving beyond dashboards toward self-learning systems that continuously refine predictions as more data is collected. We are not talking about a model that reflects reality. We are talking about one that anticipates it. What does that actually mean on the ground? It means maintenance schedules driven by live sensor data, not assumption. It means risk thresholds triggering automated recommendations before a problem becomes an incident. It means the gap between design intent and operational reality finally starting to close. Interoperability is becoming a priority, with increasing focus on open standards and integration across BIM, GIS, IoT, and asset management systems. The siloed platform era is ending. The connected data ecosystem era is beginning. Digital models are no longer ready to be built. They are being developed as long-term operational resources on which maintenance plans, financial plans, and sustainability performance are based. This is the lifecycle shift our industry has been talking about for a decade. It is now happening in practice. The question is not whether your organisation needs a Digital Twin strategy. The question is whether your data is structured well enough to feed one. Is your information ready for what comes next? #DigitalTwin #BIM #InformationManagement #AssetManagement #DigitalConstruction #AI

From Raw Engineering Data to Real-Time Optimization - How is an AI model built? 🔍 This visual from Narnia Labs breaks it down beautifully, and it’s not just a generic ML pipeline. It’s specifically designed for engineering data like 3D CAD models, simulation outputs, and performance metrics. What stood out to me: → Engineering-first data pipeline: It all starts with 1D/2D/3D design or simulation data - no text or tabular shortcuts here. → AI without parameter chaos: Instead of manually defining design variables, generative models learn directly from shapes and performance outcomes. →From model to action: Once trained and tested, the AI can be deployed as an API or GUI for real-time design evaluation and optimization. 🔍 This approach is grounded in a recent peer-reviewed paper from KAIST/Narnia Labs exploring eight application scenarios for generative AI in engineering - from 3D shape generation to simulation prediction and optimization. Instead of optimizing in a high-dimensional parameter space, they compress the design into a low-dimensional latent space - which makes real-time generative optimization possible. That’s a game-changer for anyone working on simulation-heavy or geometry-rich products. 📄 Full paper: Generative AI-driven Design Optimization (Kang, JMSTA 2025) || 🔗 DOI: 10.1007/s42791-025-00097-1 #engineering #ai #generativedesign

For centuries, scientific progress was driven by observation. Early astronomers charted the sky, physicians recorded anatomy, and natural philosophers catalogued the world. Then, in the 1600s came a pivotal transformation, an awakening of deep curiosity in a period referred to as the Enlightenment. During this time observation evolved into hypothesis, experimentation, and prediction. Newton’s laws did not only describe falling apples; they enabled humanity to understand and even predict the forces at play. Science shifted from observing the natural world to theory and hypotheses of it, and through that change many of the modern conveniences we enjoy today were born. Business is undergoing a similar evolution. Operational excellence and performance analysis began with observation, measuring outputs, identifying inefficiencies, and standardising processes. Frameworks such as Lean and Six Sigma remain grounded in empirical observation and correlation. They excel at explaining what happens and, to a degree, why. Yet much of this remains retrospective. We monitor, we record, and we improve incrementally. In scientific terms, many organisations remain at the stage of saying, “If I drop this apple, it will fall.” Business cases, budgets, and cash flow forecasts are all forms of modelling. However, they extrapolate from established patterns and are based on the assumption that tomorrow will behave much like today. Digital twins and advanced simulations represent this progression. A digital twin replicates a real-world process or system, ingesting data and enabling changes to be tested virtually. These models are increasingly powered by artificial intelligence, including inference models that learn from vast datasets and forecast complex outcomes with growing accuracy. Looking ahead, the potential of quantum computing promises to accelerate this capability further, making it possible to simulate scenarios of previously unmanageable scale and complexity. As in science experiments, these tools could reveal how a change might ripple through a network before any adjustment is made in reality. Today, when we combine data with predictive analytics and simulation it allows organisations to shift from reactive observation to proactive change. Continuous improvement becomes continuous simulation. Rather than waiting for failure to surface opportunity, leaders can test “what if” scenarios in real time. Just as scientific theory enabled experimentation without incurring the full costs of trial and error, predictive modelling allows decision-makers to explore options, optimise outcomes, and allocate resources more effectively before committing to action. Science advanced when people began to theorise and not merely observe. Business now stands at a similar inflection point. Those who embrace predictive experimentation will not only understand their operations more deeply but, like Newton, begin to shape the very principles that define their success.