Scientific Methodological Standards

When I first embarked on my PhD journey, constructing a theoretical research framework felt like scaling Mount Everest in flipflops. Today, I want to break it down into manageable steps, so you can transform your research from a chaotic jumble to a coherent narrative. Let's dive into the process: 1️⃣ Identify Your Research Question Your research question is your North Star. It guides your entire study, so clarity is crucial. Ask yourself: What problem am I really trying to solve? 💡 For instance, in psychology, you might ask, How does social media usage impact adolescent self-esteem? 2️⃣ Dive Into the Literature This is more than just reading; it's detective work. Look for patterns, contradictions, and gaps. Creating a literature map can help visualize connections between different studies. 💡 Perhaps many studies link social media use to decreased selfesteem, but some suggest the opposite in certain contexts. 3️⃣ Choose a Theoretical Lens Your theoretical lens is like the glasses you view your research through. Your choice will shape your approach. 💡 Are you examining social media through social comparison theory or uses and gratifications theory? 4️⃣ Build a Conceptual Model Think of this as a 'mind map' for your research. Draw boxes for key concepts and arrows to show relationships. 💡For example, you might have boxes for Social Media Usage, SelfEsteem, and Peer Comparison, with arrows showing their interactions. 5️⃣ Define Your Constructs Precision is key. Clear definitions prevent confusion later. 💡 What do you mean by selfesteem in your study? Is it global self-worth or specific domains like academic or social self-esteem? 6️⃣ Establish Relationships Connect the dots between your concepts. Make these relationships explicit in your framework. 💡 You might hypothesize that increased social media usage leads to more peer comparison, affecting self-esteem. 7️⃣ Validate Your Framework Don't work in isolation. Share your framework with peers, mentors, and researchers in related fields for feedback. Be open to constructive criticism—it's your framework's immune system! 👉 Ongoing step: Iterate and Refine Your framework isn't set in stone. As you gather data and delve deeper, be ready to adjust. Incorporate new insights to strengthen your framework. Your theoretical framework isn't just a box to tick off. It's the backbone of your study, the lens through which you'll interpret your findings, and your unique contribution to your field. What challenges have you faced in developing your framework? #research #researcher #academia #phd #postdoc

How to Build a Conceptual Theoretical Framework Your PhD Supervisor Can’t Tear Apart   A weak conceptual framework won’t just raise the blood pressure of your most loving supervisor; it will jeopardize your proposal or entire thesis.   BTW… a conceptual framework is not a diagram you sketch out at 1am because your supervisor said you need one. It’s the logic engine of your entire thesis, it indicates you really know what you’re doing.   For mixed methods, your conceptual theoretical framework is even more important. It explains how your qualitative insights and quantitative results speak and relate to each other.   Here’s what I often share with my registered students…   1) Anchor your framework in real theory, not “it sounds right”. a] Don’t construct a framework based on your preferences. b] Select theories that actually match your topic, variables, and context. c] For example: If your topic is about the importance of a technology, such as using TAM in the framework, explain how people accept or reject technology. d] Likewise, if about JD-R as a framework to examine staff retention issues, articulate about burnout, stress, and staff engagement. e] Similarly, we use TPB to describe how attitudes, norms, and control shape behaviour. f] If you can’t explain why a theory belongs in your framework, bin it.   2) Why these theories belong in your study a] Your supervisor will want to know… Why THIS theory for THIS study in THIS context? b] Be explicit… c] What does the theory help you measure, explain, or predict? d] What gaps does it fill? e] How does it help you understand your variables? f] Use logic. Weak logic = a framework that collapses under scrutiny.   3) Map your variables like a researcher, not a graphic designer a] A solid conceptual framework clearly indicates… b] Independent variables. c] Dependent variables. d] Mediators/moderators (only if they're tested). e] Theoretical relationships. f] If mixed methods: How the qualitative and quantitative phases connect. g] If your conceptual pathways look like a bowl of noodles and create more questions than answers, start again.   4) Connect theory with the method with the analysis (triangulate) a] The best frameworks show alignment all the way through. Why? Because… b] Theory informs your variables. c] Variables shape your research questions. d] RQs shape your methodology. e] Methodology shapes your instruments. f] Instruments shape your analysis. g] This pathway may convince your most loving supervisor that you may know what you’re doing.   5) End with one powerful sentence a] A strong conceptual framework often looks something like… b] This framework integrates X and Y theories to explain how A influences B within Z context, guiding both the qualitative and quantitative phases of this mixed-methods study. c] That one sentence alone will inform your most loving supervisor… d] Your entire study is coherent, not a random collection of frameworks or chapters.   Need help? Check my comments below…

Evaluating LLMs is not like testing traditional software. Traditional systems are deterministic → pass/fail. LLMs are probabilistic → same input, different outputs, shifting behaviors over time. That makes model selection and monitoring one of the hardest engineering problems today. This is where Eval Protocol (EP) developed by Fireworks AI is so powerful. It’s an open-source framework for building an internal model leaderboard, where you can define, run, and track evals that actually reflect your business needs. → Simulated Users – generate synthetic but realistic user interactions to stress-test models under lifelike conditions. → evaluation_test – pytest-compatible evals (pointwise, groupwise, all) so you can treat model behavior like unit tests in CI/CD. → MCP Extensions – evaluate agents that use tools, multi-step reasoning, or multi-turn dialogue via Model Context Protocol. → UI Review – a dashboard to visualize eval results, compare across models, and catch regressions before they ship. Instead of relying on generic benchmarks, EP lets you encode your own success criteria and continuously measure models against them. If you’re serious about scaling LLMs in production, this is worth a look: evalprotocol.io

🚨 Several months ago, I shared my perspective on the paper “Hidden flaws behind expert-level accuracy of multimodal GPT-4 vision in medicine” to emphasize the need to avoid testing LLMs and LMMs on web-sourced data and called for more rigorous research methods to prevent overestimating these models' capabilities. 🧐 Recently, a paper in Radiology also used the NEJM Image Challenge (the authors referenced the aforementioned paper)—you could already guess some of the things I am going to say in this post. Here’s my take on the paper, with the hope of improving the quality of radiology research using foundation models in the future. ❌ The authors stated, “NEJM website was searched.” This poses a significant risk of data contamination. Ironically, they ensured “none of the human readers had experience with the NEJM Image Challenge cases.” This same rigor was not applied to the test set. To minimize data contamination risks, don’t simply copy and use test questions (❗️that also have answers) from the web verbatim. If you can Google it, the LMM you’re using probably has seen it. ❌ The authors noted, “LLMs achieved similar accuracies regardless of the image input,” which should have triggered concerns about data contamination and limited vision capabilities. Studies have shown LLM performance declines when evaluation problems are paraphrased or recontextualized. Modifying questions and answers could have helped assess contamination risks. Also, while it may take considerable effort (good research often does), one should try to develop their own test cases. As I discussed during the RSNA 2024 Radiology AI Fireside Chat, while MCQs can serve as a testing method, we need to explore beyond this type of model evaluation methodology, especially in medicine, to better reflect how we practice medicine. ❓ What is disappointing is that the authors knew and acknowledged these limitations that their previous paper on Radiology Diagnosis Please cases also described. Why continue repeating these issues? 🔍 The use of foundation models in radiology presents exciting opportunities, but to drive the field forward responsibly, we must ensure that evaluations are rigorous and free from contamination. 🔗 to all mentioned resources are in the first comment. 👇🏼 #GenAI #radiology #RadiologyResearch #LLMs #LMMs

Researchers at UC San Diego and Tsinghua just solved a major challenge in making LLMs reliable for scientific tasks: knowing when to use tools versus solving problems directly. Their method, called Adapting While Learning (AWL), achieves this through a novel two-component training approach: (1) World knowledge distillation - the model learns to solve problems directly by studying tool-generated solutions (2) Tool usage adaptation - the model learns to intelligently switch to tools only for complex problems it can't solve reliably The results are impressive: * 28% improvement in answer accuracy across scientific domains * 14% increase in tool usage precision * Strong performance even with 80% noisy training data * Outperforms GPT-4 and Claude on custom scientific datasets Current approaches either make LLMs over-reliant on tools or prone to hallucinations when solving complex problems. This method mimics how human experts work - first assessing if they can solve a problem directly before deciding to use specialized tools. Paper https://lnkd.in/g37EK3-m — Join thousands of world-class researchers and engineers from Google, Stanford, OpenAI, and Meta staying ahead on AI http://aitidbits.ai

As I did my PhD in network science, this field within data science is particularly dear to me. Complementing my two courses << links in comments >> I am recapping this ever-growing list of network analytics tools. Comment/add your favorite ones! What else would you add? 𝐏𝐨𝐢𝐧𝐭-𝐚𝐧𝐝-𝐜𝐥𝐢𝐜𝐤 𝐬𝐨𝐟𝐭𝐰𝐚𝐫𝐞: - Cytoscape - https://cytoscape.org - Gephi - https://gephi.org - Graphia - https://graphia.app - GraphInsight - https://lnkd.in/d5XnkWJr - NodeXL - https://nodexl.com - Orange - https://lnkd.in/dZU8Zx3D - SemSpect - https://www.semspect.de - SocNetV - https://socnetv.org - Tulip - https://lnkd.in/dtc_BD33 - Ucinet - https://lnkd.in/dE8k34v7 - VOSviewer - https://www.vosviewer.com 𝐎𝐧𝐥𝐢𝐧𝐞 𝐭𝐨𝐨𝐥𝐬: - Gephisto - https://lnkd.in/diSp3BWN - Gephi Lite - https://lnkd.in/dHJ3F-r6 - Kumu - https://kumu.io - Graphistry - https://www.graphistry.com - Cosmograph - https://lnkd.in/dUBJS4w3 𝐏𝐲𝐭𝐡𝐨𝐧 𝐥𝐢𝐛𝐫𝐚𝐫𝐢𝐞𝐬: - networkx - https://lnkd.in/dKCCXjif - graph-tool - https://lnkd.in/dvytUzdu - graphviz - https://lnkd.in/d3GqtmQn - ipycytoscape - https://lnkd.in/dvTwmySk - ipydagred3 - https://lnkd.in/diXgFWMD - ipysigma - https://lnkd.in/dP55J5et - ipyvolume - https://lnkd.in/dq52_wdr - netwulf - https://lnkd.in/dsgKDHPh - nxviz - https://lnkd.in/duHbKGPN - Py3Plex - https://lnkd.in/dhwe7f_g - py4cytoscape - https://lnkd.in/d7NwU8_Y - pydot - https://lnkd.in/d8w6VfyP - pyGraphistry - https://lnkd.in/dz-NfFf7 - pygsp - https://lnkd.in/dS7s-A_v - python-igraph - https://lnkd.in/dCGsRXh2 - PyTorch Geometric - https://lnkd.in/duT3y8-U - pyvis - https://lnkd.in/duJ5kWAd - scikit-network - https://lnkd.in/dKPXenCk - SNAP - https://lnkd.in/duM5uHnr - visjs.org - https://visjs.org - visNetwork - https://zurl.co/zY3O - 3D Force-Directed Graph - https://zurl.co/AYks More info reading: https://lnkd.in/dWYgERtK #datascience #networkscience #datavisualization #data #analytics#networkvisualization #ai

DON’T rely on AI to do your research… Large language models (LLMs) are often praised for their ability to process information and assist with problem-solving, but can they really reason like ourselves? The latest study by Apple researchers reveals significant limitations in their capacity for genuine mathematical reasoning - and raises important questions about their reliability in research contexts. What Apple Found: 1. Inconsistent results: LLMs struggle with variations of the same problem, even at a basic grade-school math level. This variability challenges the validity of current benchmarks like GSM8K, which rely on single-point accuracy metrics. 2. Fragility to complexity: As questions become slightly more challenging, performance drops drastically, exposing a fragile reasoning process. 3. Susceptibility to irrelevant information: When distracting but inconsequential details were included in problems, model performance plummeted by up to 65%. Even repeated exposure to similar questions or fine-tuning couldn’t fix this. 4. Pattern matching ≠ reasoning: The models often “solve” problems by sophisticated pattern matching, not genuine logical understanding. What this means for research: While LLMs are powerful tools for speeding up certain tasks, their inability to discern critical from irrelevant information, and their reliance on pattern recognition, makes them unreliable for rigorous, logic-based research. This is particularly true in fields like mathematics, engineering, and data-driven sciences, where accuracy and reasoning are non-negotiable. As exciting as these tools are, they’re not ready to replace human critical thinking (yet?). How do you see AI evolving in research applications? #research #chemicalengineering #scientist #engineering #professor PS. Full paper available on ArXiv under 2410.05229

What actually happens when LLMs evaluate LLM-generated research? 🐍 Scientific quality quietly collapses. New research analyzing 125,000+ paper-review pairs from ICLR, NeurIPS, and [ICML] Int'l Conference on Machine Learning just dropped on arXiv, and the findings are a wake-up call for scientific integrity. When LLMs review research papers, the core problem isn’t hallucination. It’s Rating Compression. LLM reviewers are trained to be helpful and polite. That makes them very bad at giving strong rejections and strong endorsements. Everything gets squeezed into a beige middle - grammatically perfect, low-variance, low-conviction reviews. This creates three dangerous illusions: → It looks like LLM reviewers prefer LLM-written papers. In reality, weaker papers tend to use more AI writing, and LLM reviewers are simply too “nice” to flag mediocrity. → The signal that separates breakthrough research from plausible-sounding work disappears. → Worst of all, LLM-assisted meta-reviews are significantly more likely to flip a decision to “Accept” given the same underlying scores than a human meta-reviewer would. If we use LLMs to write papers and to grade papers, we don’t just lose the human touch - we lose the ability to distinguish insight from polish. Some takeaways that I found useful... • Authors: If you use an LLM to pre-review your paper, ignore the score. Focus only on critiques of logic, novelty, and assumptions. • Reviewers: Watch for beige reviews, polished language with no strong stance on novelty or impact. • Chairs: High confidence + low variance is classic bot behavior. Evaluation systems need variance-aware checks. This isn’t about banning LLMs from peer review. It’s about understanding their systematic biases and designing processes that compensate for them. As an engineer, I love automation, especially when there are 20k+ submissions. But judgment? That still has to stay human. IMO LLMs can assist, but should never be the final arbiter. Curious where others draw the line 💭 #AIEthics #PeerReview #ICML #ICLR #NeurIPS

🚀 Introducing the Generate README Eval Today we unveil a new evaluation method that challenges LLMs to summarize entire GitHub repositories into comprehensive README files – a task that demands deep understanding of complex codebases and the ability to synthesize information effectively. What sets this benchmark apart is its holistic approach to evaluation. We've gone beyond traditional NLP metrics like BLEU and ROUGE scores, incorporating critical dimensions such as structural similarity, code consistency, readability (using the Flesch Reading Ease Score), and information retrieval. This multifaceted evaluation provides a more nuanced and practical assessment of an LLM's capabilities in real-world scenarios. Our initial findings are interesting: The current state-of-the-art performer is Gemini-1.5-Flash-Exp-0827, showcasing impressive capabilities across various metrics. We have uncovered an interesting trade-off: as we increase the context length to include more examples (for few-shot evaluation), we see a decline in information retrieval and readability scores. This suggests that LLMs struggle with perfect recall in larger contexts, potentially missing crucial information – a critical insight for those working on improving model performance at scale. The benchmark is designed to handle repositories up to 100k tokens in size, allowing for comprehensive evaluation while remaining within the context limits of most frontier LLMs. This design choice enables us to test models on real-world, substantial codebases, providing insights that are directly applicable to practical scenarios. Explore the benchmark here: https://lnkd.in/gMn5dehf #AI #MachineLearning #Benchmarks #NLP #AIEvaluation #LargeLanguageModels