Managing expectations in the application of quality and strength of evidence assessments in qualitative synthesis

Approaches to assessing the strengths and limitations of qualitative studies included in evidence syntheses continue to be discussed in the synthesis community (e.g., Garside 2014, Noyes et al. 2023). This presentation discusses such strengths and limitations in the application of evidence quality and confidence in findings assessments we used in a review of food production interventions with a climate change component.

We applied the Critical Appraisal Skills Program (CASP) checklist to assess individual papers on the appropriateness of methodology, study design, recruitment strategy, data collection methods, consideration of researcher’s role and ethical issues, data analysis, statement of findings, and value of research. We applied GRADE-CERQual (Confidence in the Evidence from Reviews of Qualitative Research) to assess the level of confidence in our review findings based on methodological limitations, coherence, data adequacy, and contextual relevance.

Overall, evaluations in our study tended to receive lower ratings because they lacked a thorough description of the data analysis process or research ethics considerations, or failed to acknowledge the potential biases of the researcher or research team. Given the focus of the synthesis (to understand barriers and facilitators to climate adaptive practices in agriculture), the potential for evaluations to provide insight into these questions, and that cutoff points for exclusion are not well established in qualitative evidence synthesis, we decided not to exclude any evaluations despite considerable variation in CASP ratings. Further, our application of GRADE-CERQual may have weighted some findings as lower confidence because of these quality ratings in addition to the number of studies that addressed a question. We discuss the merits of these assessments given considerations such as variation in methodological reporting and study questions.