Developing Review Criteria

Struggling to turn piles of papers into a strong, insightful literature review? This 5 C’s of writing a literature review (Cite, Compare, Contrast, Critique, Connect) narrow it down well. But how do you actually do these well in practice? Here’s how I teach my PhD mentees to move beyond theory and into action: 1. CITE: Be strategic, not exhaustive. ✅ Don’t just collect papers, curate them. Prioritize studies that shape the field, influence your thinking or set up your argument. 📌 Use citation mapping tools (like Connected Papers or ResearchRabbit to visually trace foundational works and identify key influencers fast. 2. COMPARE: Patterns matter more than papers. ✅ Group studies by themes: methods, theories, findings, not by author name or publication date. 📌 Create a simple comparison matrix in Excel, SciSpace or Anara to spot patterns across studies. You’ll see trends (and gaps) much faster this way. 3. CONTRAST: Don’t be afraid to question the giants. ✅ Highlight conflicting evidence, contradictory findings or evolving theories. This shows depth. 📌 Always ask yourself: Why might these studies disagree? Sample? Method? Context? Theory? This leads to stronger insights. 4. CRITIQUE: Not all papers deserve equal weight. ✅ Evaluate studies for quality, not just relevance. Weak studies make weak foundations. 📌 Apply a simple checklist when reading: clarity of aim, appropriateness of method, robustness of findings. Highlight these in your notes for easy reference. 5. CONNECT: Your review needs to lead somewhere. ✅ Your literature review is a bridge to your research question. 📌 After reviewing each group of studies, explicitly write: “What does this mean for my study?” This helps transition from review to rationale. A literature review is not about how much you’ve read. It’s about how clearly you can show your reader: 📍 What’s known 📍 What’s contested 📍 What’s missing 📍 And why your study matters PS: Which of these 5 C’s do YOU find the trickiest to apply? Share in the comments REPOST this to help others.

Most startups make the same performance review mistakes. They either skip them entirely ("we're too small") or mimic what Big Tech does ("let's do 360s for our 15-person team"). Here's the framework I use to design review cycles that fit for different sizes and stages: Frequency by stage: 0-35 people: Quarterly development reviews 35-200 people: Semi-annual reviews (with optional off-cycle developmental check-ins). Combine compensation and performance reviews at this point. 200+ people: Annual reviews (with optional off-cycle developmental check-ins) Timing matters more than you think. Don't plan promotions and merit increases during low cash flow periods. While January and July are classic review times, if March and September are bad cash flow months for your business model, move your performance review schedule. Review complexity should scale with headcount AND experience: Start with self + manager reviews (180°), even at 1,000 people if you've never run reviews before. Add peer feedback (270°) after 1-2 successful cycles. Full 360s only when you're ready for the complexity. Once you’ve moved to semi-annual reviews with compensation considerations included, it helps to run a tight schedule with clear week-by-week deadlines: Week 1: Self and upward reviews due Weeks 2-3: Downward reviews due — 2 weeks help here for managers with 5+ direct reports and stragglers who were OOO for the whole self and upward review week Weeks 4-5: Manager calibrations Weeks 6-7: Real-time performance review conversations and compensation decision-making Week 8: Compensation and promotion decisions shared Week 9: Title and compensation changes go into effect Tool threshold: 20 people. Below that, Google Forms works. Above that, invest in something like Lattice (my recommended tool)—the time savings on scheduling and reminders alone pays for itself. The biggest mistake? Treating performance reviews like an HR project instead of a business process. Get the sequencing and feedback mechanisms right, and they become a strategic advantage. More tactical details on building review cycles that scale coming in my next newsletter. __ 👋 I'm Melissa Theiss, 4x Head of People and Business Operations and advisor for bootstrapped and VC-backed SaaS companies. 🗞️ In my newsletter, “The Business of People,” I share tips and tricks that help People leaders think like business leaders.

Many evaluations fail to influence decision-making because organisations often treat evaluation as a single standard exercise with one universal purpose. Yet the type of evaluation selected shapes the questions asked, the evidence collected, the methods used, and the decisions that follow. A project facing low community participation during implementation does not only need a judgement on whether results are weak. It may need a deeper look at mobilisation strategies, communication channels, inclusion barriers, field presence, seasonal constraints and the way implementation is actually unfolding. This is why choosing the right evaluation type matters. A formative evaluation helps improve the intervention while there is still time to adjust. A process evaluation examines whether implementation is happening as intended and where bottlenecks are emerging. A summative evaluation supports accountability and overall performance judgement, while an impact evaluation explores whether observed changes can credibly be linked to the intervention. In complex or evolving contexts, developmental evaluation can also support adaptation when the intervention itself is still being refined. Good evaluation practice starts with one essential question: what decision should this evaluation inform? When that question is clear, evaluation becomes more than a reporting requirement. It becomes a practical tool for learning, accountability, strategic correction and better programme performance.

𝗜𝘁 𝘁𝗼𝗼𝗸 𝗺𝗲 𝗮 𝗹𝗼𝗻𝗴 𝘁𝗶𝗺𝗲 𝘁𝗼 𝗿𝗲𝗮𝗹𝗶𝘇𝗲 𝘁𝗵𝗮𝘁 𝗜 𝗮𝗹𝘀𝗼 𝗻𝗲𝗲𝗱𝗲𝗱 𝘁𝗼 𝗯𝗲 𝘀𝗲𝗹𝗲𝗰𝘁𝗶𝘃𝗲 𝗮𝗻𝗱 𝗽𝗶𝗰𝗸𝘆 𝗮𝗯𝗼𝘂𝘁 𝘁𝗵𝗲 𝗳𝘂𝗻𝗱𝗶𝗻𝗴 𝗼𝗽𝗽𝗼𝗿𝘁𝘂𝗻𝗶𝘁𝗶𝗲𝘀 𝗜 𝗽𝘂𝗿𝘀𝘂𝗲𝗱. Early on, I chased every funding opportunity that vaguely aligned with our mission. When resources are tight, it’s easy to reshape your work to meet funders’ interests—even if it feels like squeezing a round peg into a square hole. Over time, I learned that this approach comes with costs that can be more detrimental than the reward they bring. These include: 🍃 𝗠𝗶𝘀𝘀𝗶𝗼𝗻 𝗗𝗿𝗶𝗳𝘁: We move away from our original purpose when we adjust our programs to fit a funder’s requirements. This “mission drift” can dilute our core impact, spreading us thin and lessening our unique value. 💪🏿𝗧𝗲𝗮𝗺 𝗠𝗼𝗿𝗮𝗹𝗲: Constantly pivoting to satisfy funders’ priorities rather than focusing on a clear mission can lead to burnout and disillusionment, making retaining talented, passionate staff harder. 🎯𝗟𝗮𝗰𝗸 𝗼𝗳 𝗙𝗼𝗰𝘂𝘀: Casting a wide net without a strategy leads to scattered efforts and less productive results. This especially affects the development team, making them less efficient and the relationships they build more surface-level and less impactful. So, how do you ensure funder alignment? I use a weighted rubric that keeps us focused on impact. I rate each funder on key criteria—like mission alignment, application ease, and grant size—scoring them as low, medium, or high. We only pursue funders who meet our threshold so we can focus on partnerships that genuinely support our mission and goals. The criteria include: 🚀 𝗠𝗶𝘀𝘀𝗶𝗼𝗻 𝗔𝗹𝗶𝗴𝗻𝗺𝗲𝗻𝘁 (𝟮𝟬%): Does the funder have a history of supporting causes like yours? Funders interested in your mission area will likely be a better fit. 💰 𝗚𝗿𝗮𝗻𝘁 𝗦𝗶𝘇𝗲 (𝟮𝟱%): Does the grant amount align with your financial needs? You also need to factor in the costs of applying for the opportunity. Does the team time pay off? 👥 𝗖𝗼𝗻𝗻𝗲𝗰𝘁𝗶𝗼𝗻 𝘁𝗼 𝗬𝗼𝘂𝗿 𝗡𝗲𝘁𝘄𝗼𝗿𝗸 (𝟭𝟬%): Is there an existing link through board members or mutual partners? Familiarity can create a trust-based relationship, often leading to a smoother collaboration. 🧘🏿♀️ 𝗘𝗮𝘀𝗲 𝗼𝗳 𝗚𝗿𝗮𝗻𝘁 𝗣𝗿𝗼𝗰𝗲𝘀𝘀 (𝟮𝟬%): A clear, grantee-focused application process means your team can focus more on impact than on admin. 🧩 𝗦𝘁𝗿𝗮𝘁𝗲𝗴𝗶𝗰 𝗔𝗹𝗶𝗴𝗻𝗺𝗲𝗻𝘁 (𝟮𝟱%): Does the funder’s mission support your core priorities? Funding that aligns naturally with your main programs allows you to focus on impact without significant shifts in strategy. 💬 How do you evaluate funding opportunities? What would you add to the above criteria? #internationaldevelopment #fundraising #nonprofitafrica #fundingafrica

If we want research to have societal impact, we need to think beyond communication and dissemination. In my conversations with researchers across public health, clinical practice, and related fields, a recurring issue is that we need fewer controlled trials and more implementation-focused research. Researchers know this. Practitioners and policy-makers definitely know this. And yet, implementation research is harder to get funded, harder to publish, and often seen as a career risk, despite being the type of research needed to drive societal change. When we talk about optimising societal impact, we often focus on the need for more and better dissemination and communication of research to non-academic audiences. And, while this is important, we know that the real issue starts much earlier in the research process. How research is generated shapes whether it can be translated into practice or policy. To close the research–practice–policy gap, we need to reward the use of designs and methods that are fit for purpose. We need to reward those that attend to complexity, context, and the realities of implementation. As someone who works at the intersection of research, policy and practice, I’ve seen the disconnect. Research designs with strong internal validity (like RCTs) are often prioritised, but they rarely tell us how or why something works, for whom, or under what conditions. To improve uptake and usefulness of research, we need to: ✔️ Balance internal and external validity ✔️ Use designs that make sense for complex, context-dependent challenges ✔️ Elevate methodologies that support both rigour and relevance It's also time for a shift in how we assess funding applications. What if review criteria explicitly considered: ✔️ Implementation feasibility ✔️ Scalability ✔️Relevance to practice and policy ✔️Appropriateness of the design to the research problem This would require: ✔️Review panels with diverse methodological expertise ✔️Quality indicators for translational potential ✔️Training for reviewers in knowledge translation assessment 💡 If we’re serious about societal impact, then it’s not just about what we do after the research is completed, it’s about how we design it in the first place. I would love to hear how others are navigating these tensions. How do you balance methodological rigour with relevance in your research? And what changes would you like to see in how research is funded and published? If you're working on, or interested in, developing quality indicators for the translational potential of research, I’d love to connect. It’s an area that needs more attention. Image designed using Chat GPT. #ResearchImpact #ImplementationScience #PublicHealth #ResearchFunding #KnowledgeTranslation #ComplexityInResearch #EvaluationDesign #ImpactMatters #HealthPromotion Aurélie Pankowiak (PhD)

Most performance reviews try to do two jobs at once: 1️⃣ Pick between people for pay, promotion, and roles. 2️⃣ Develop people by finding strengths and gaps. These goals pull in opposite directions. Why this clash happens (brain + math): 🧠 Brain: When a review affects your pay or job, your brain reads it as a threat. Stress goes up. Learning shuts down. Feedback feels like a warning, not help. 🔢 Math: If you focus on ranking people clearly, everyone’s profile looks the same and you lose detail about strengths and weaknesses. If you focus on rich, detailed feedback, clear rankings get fuzzy. You can’t optimize both at the same time. The fix isn’t “blend them better.” You need a third way. Build two separate tracks with different goals, timing, and rules. Track A — Allocate (between people) - Purpose: pay, promotion, role, and staffing decisions. - Timing: set times (e.g., twice a year). - Evidence: common criteria and comparisons across people. - Norms: fairness, consistency, clear documentation. Track B — Develop (within people) - Purpose: growth, new skills, behavior change. - Timing: ongoing, low‑stakes coaching in regular 1:1s. - Evidence: specific behaviors and goals; focus on the future (“feedforward”). - Norms: psychological safety, curiosity, experimentation. Design moves that make it work: 👉 Separate the moments: Never mix ratings or money talks with coaching time. 👉 Separate the artifacts: Use different forms and language for each track. 👉 Separate the roles: Talent review leaders handle Track A; managers/peers coach in Track B. 👉 Give employees a voice: Enable upward feedback and self‑nominations for growth or promotion. 👉 Aim at behavior and the future: Be specific about what to try next, not who someone “is.” Employee gut‑check: “Is this feedback or a warning?” If people can’t tell, the system isn’t truly separate yet. When we honor the polarity—allocate separately, develop safely—performance management can actually serve both business goals. #EmployeeExperience #PerformanceManagement #Leadership #HR

Peer reviewing is a voluntary task, but it’s also one of the most critical responsibilities in academia. It requires time, expertise, care, and a strong sense of academic integrity. And yet, very few of us were ever formally trained for it. We often learn by doing—or by watching others—frequently under pressure and without much guidance. Some of my PhD students are now receiving their first invitations to review papers—a major milestone in their academic journey! But naturally, they're unsure where to start. So, what do I recommend they keep in mind when stepping into this role? ✅ 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱 𝘁𝗵𝗲 𝗺𝗮𝗻𝘂𝘀𝗰𝗿𝗶𝗽𝘁 𝘁𝘆𝗽𝗲 Before diving into the review, understand the manuscript's type and check the journal’s specific guidelines for that category. This will help you apply the appropriate evaluation criteria. ✅ 𝗠𝗮𝗶𝗻𝘁𝗮𝗶𝗻 𝗮 𝗰𝗼𝗻𝘀𝘁𝗿𝘂𝗰𝘁𝗶𝘃𝗲 (𝗮𝗻𝗱 𝗽𝗼𝗹𝗶𝘁𝗲) 𝘁𝗼𝗻𝗲 Feedback should be direct, respectful, and aimed at helping the authors improve their work. Even when identifying significant flaws, your tone should remain collegial—remember, you're offering support, not judgment. Talk to the authors in the same way you would like a reviewer to talk to you when the paper under evaluation is yours! ✅ 𝗙𝗼𝗰𝘂𝘀 𝗼𝗻 𝘁𝗵𝗲 𝗰𝗼𝗻𝘁𝗲𝗻𝘁, 𝗻𝗼𝘁 𝗹𝗮𝗻𝗴𝘂𝗮𝗴𝗲 Don't get overly focused on grammatical issues. Your role is to assess ideas, methodology, overall clarity, logic, and contributions. ✅ 𝗢𝗿𝗴𝗮𝗻𝗶𝘇𝗲 𝗳𝗲𝗲𝗱𝗯𝗮𝗰𝗸 𝗰𝗹𝗲𝗮𝗿𝗹𝘆 Use headings or bullet points to separate your feedback into major comments (e.g., theoretical gaps, methodological concerns) and minor comments (e.g., missing references, formatting issues). This structure helps authors understand your review more easily. ✅ 𝗠𝗮𝗻𝗮𝗴𝗲 𝘆𝗼𝘂𝗿 𝘁𝗶𝗺𝗲 𝘄𝗶𝘀𝗲𝗹𝘆 Don’t rush through a review, but also avoid perfectionism. Aim for a thorough yet timely response—delayed reviews stall the entire publication process. Personally, I always write a draft and then wait a day or two before revisiting it. Coming back with a fresh perspective helps me refine my comments. ✅ 𝗥𝗲𝗰𝗼𝗴𝗻𝗶𝘇𝗲 𝘄𝗵𝗲𝗻 𝘁𝗼 𝗱𝗲𝗰𝗹𝗶𝗻𝗲 If the paper is outside your expertise, or you’re unable to review it within the requested time frame, it’s best to decline. The integrity of the review process depends on the match between reviewers and subject matter. ✅ 𝗔𝘀𝘀𝗲𝘀𝘀 𝘁𝗵𝗲 𝗰𝗼𝗻𝘁𝗿𝗶𝗯𝘂𝘁𝗶𝗼𝗻, 𝗻𝗼𝘁 𝗷𝘂𝘀𝘁 𝘁𝗵𝗲 𝗽𝗿𝗲𝘀𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻 Ask yourself: Does this manuscript make a meaningful contribution to the field, from a theoretical perspective and in managerial, practical, and/or policy terms? Sometimes a messy draft still holds valuable insights that are worth refining. 👇 [The list continues in the comments] #PeerReview #AcademicPublishing #PhDLife #EarlyCareerResearchers #AcademicTips #ResearchCommunity #Reviewers #ScholarlyPublishing #ResponsibleResearch #ReviewerGuidance #AcademicSupport #PublishingAdvice

Evaluation is both an art and a science, balancing systematic analysis with the nuanced understanding of complex interventions. The "Applying Evaluation Criteria Thoughtfully" guide by the OECD Development Assistance Committee (DAC) introduces a refined framework for using the six evaluation criteria—relevance, coherence, effectiveness, efficiency, impact, and sustainability—in a way that goes beyond checklist approaches. Instead, it emphasizes the importance of critical thinking, adaptability, and context sensitivity in every stage of the evaluation process. This document integrates three decades of global evaluation practice with contemporary priorities such as the Sustainable Development Goals (SDGs) and human rights frameworks. It underscores the need to consider interconnections, equity gaps, and the holistic impacts of interventions. By providing examples, insights, and practical guidance, the manual ensures that evaluators and decision-makers can navigate diverse contexts, addressing complexities in implementation and fostering meaningful accountability and learning. Tailored for policymakers, evaluators, and development practitioners, this resource elevates evaluation practice to ensure that interventions not only meet their objectives but also generate transformative and sustainable impacts. By adopting its principles, users can advance evidence-based strategies, improving global collaboration and the effectiveness of development cooperation.

EA and ARB I recently had the opportunity to review a number of different ARBs (Architecture Review Boards). It was rather interesting to see the ways in which they differed (or were similar). On LinkedIn I have seen some very different views expressed about ARBs, from "they are evil" types of views (is that bad though? 😉 ) to "that's the only way to guide solution architecture in the right direction" types of views. Of course, every org is different, but if you do have an ARB I would say the following principles should be followed: 1. Proportion Review to Risk Some orgs have a "one size fits all" type of review process. This usually doesn't work well. Obviously low or no risk projects get too much review, and potentially high-risk projects don't get enough scrutiny. A better approach is to have some kind of initial "risk assessment" based on some set criteria, and then tailor the amount of review accordingly. 2. Clearly Identify Review Criteria Some ARBs are very informal "chats" between the ARB and the SA. Others require reams of documentation. You should choose what works best for you, but either way, I think it should be very clear what actually is being reviewed and for what. And by who - generally an ARB should include a broad set of stakeholder, not just EA. 3. Solution PoV vs. Enterprise PoV A review could focus on just the Solution itself (e.g. is it well architected), and/or it could focus on Enterprise concerns (is this architecture the best choice for the Enterprise "all things considered"). Those two PoVs may yield very different results. Generally, I think an ARB should be focused on the Enterprise PoV. A well architected Solution should be the job of the SA. However many ARBs do both, which is fine, if that's what needed. But I would argue that an ARB which only has a Solution PoV is not really an ARB. (Some people distinguish between an ARB and a TDA or some such, personally I haven't seen that) 4. Review Early (and often?) Assuming the point of the review is to make sure that the fundamental architecture choices are good, this would imply that an early high level architecture review should be done. Waiting too long to review will mean it's too late (or very costly) to make any fundamental changes. Some orgs assign an EA to the Solution Team, but that only works if EA can represent all Stakeholders (which is often not the case). 5. Log Decisions, Recommendations and Risks I've seen some ARBs where the process is that the Solution Architecture Document (the "SAD") is basically used as the review tool - i.e. reviewers write comments on the side, and then the SA updates the document (or maybe makes changes to the architecture). That's a convenient process but you end up with no log of what the ARB recommended, and then ultimately it's hard to see what value (if any) the ARB is adding. I would suggest that any ARB (even a more informal one) should always keep a log of all key recommendations, decisions, risks etc.

𝗠𝗼𝘀𝘁 𝗿𝗲𝘃𝗶𝗲𝘄 𝗽𝗮𝗽𝗲𝗿𝘀 𝗳𝗮𝗶𝗹 𝗯𝗲𝗳𝗼𝗿𝗲 𝘁𝗵𝗲 𝘄𝗿𝗶𝘁𝗶𝗻𝗴 𝘀𝘁𝗮𝗿𝘁𝘀. A student collects 40 papers. Then 70 papers. Then 120 papers. Then opens a Word document and starts summarising one article after another. That is not a strong review paper. That is usually an annotated bibliography with a conclusion attached. A publishable review needs more than reading. It needs a method. 𝗔 𝘀𝘁𝗿𝗼𝗻𝗴 𝗿𝗲𝘃𝗶𝗲𝘄 𝗽𝗮𝗽𝗲𝗿 𝗻𝗲𝗲𝗱𝘀 𝘀𝗶𝘅 𝘁𝗵𝗶𝗻𝗴𝘀: 1. Define the scope before you search Start with a real research problem, not a vague topic. Decide whether you are writing a systematic review, scoping review, narrative review, or meta-analysis. Each one has a different purpose, standard, and level of evidence. 2. Build a search that can be defended Your search strategy is part of your methodology. Use the right databases. Write proper Boolean strings. Document the date, database, keywords, filters, and number of results. If your search cannot be repeated, your review is already weak. 3. Screen studies with consistency Remove duplicates first. Screen titles and abstracts against the same criteria. Then read full texts carefully before deciding what stays. Do not change the rules halfway because a paper feels useful. 4. Extract data into a structure Do not rely on memory. Create an extraction table with the same fields for every paper: - design - sample - method - findings - limitations - quality indicators This is where patterns, gaps, and contradictions start becoming visible. 5. Synthesize instead of summarising Weak reviews say: "Author A found this. Author B found that. Author C argued this." Strong reviews ask: "What does this body of literature collectively show?" Your job is to build an argument from the literature, not list the literature. 6. Write with transparency Your methods section should allow another researcher to understand exactly what you did. Your results should show patterns across studies. Your discussion should explain what the evidence means, where it conflicts, and where future research should go. 𝗠𝘆 𝗳𝗶𝗻𝗮𝗹 𝘁𝗲𝘀𝘁: If another researcher cannot reproduce your search, screening, extraction, and synthesis logic from your paper, the review is not ready. Save this before you start writing your review paper. www.researchcrave.com | tayyab@researchcrave.com #ResearchCrave #ReviewPaper #SystematicReview #ScopingReview #LiteratureReview #AcademicWriting #ResearchWriting #ResearchMethods #ResearchMethodology #PhDStudents #MastersStudents #GraduateStudents #DissertationWriting #ThesisWriting #DoctoralResearch #HigherEducation #ResearchSupport #AcademicResearch #PRISMA #MetaAnalysis