Dr Laura Downie1, Dr Michael Pianta1
1Department of Optometry and Vision Sciences, The University Of Melbourne, Parkville, Australia
We developed a free, online tool that teaches critical appraisal and facilitates the sharing of appraisals amongst a global community (CrowdCARE, crowdcare.unimelb.edu.au). Our aim was to investigate the rigour of crowdsourced appraisals from trained novice raters.
Systematic reviews (n=71) were appraised in CrowdCARE by five trained novices and two expert raters. Appraisals were performed using a validated tool (AMSTAR) to yield: (i) an aggregate quality score, and (ii) domain-specific responses for each of the 11 assessment items. Appraisal quality was investigated by assessing the variability in AMSTAR scoring and calculating the concordance of ratings using Cohen’s Kappa (κ).
Variability in aggregate AMSTAR scores was similar between expert raters, and between the expert consensus and mean novice ratings. A strong correlation was evident between the expert consensus and mean novice rating (r2=0.89, p<0.0001). There was good agreement (κ=0.67, 95%CI: 0.61 to 0.73) between the aggregate score of the expert consensus rating and mean novice rating.
Significance of the findings to allied health:
These data demonstrate the merit of our novel crowdsourcing approach for appraising research quality. Allied health students can be trained to critically appraise systematic reviews and there is moderate agreement between expert and novice raters. CrowdCARE provides students and clinicians with the skills to appraise research quality and contributes to making evidence-based practice more efficient by removing the substantial duplication of effort made by individual clinicians across the globe.
Dr Laura Downie is a tenured Senior Lecturer in the Department of Optometry and Vision Sciences at the University of Melbourne. In this role, she provides didactic and clinical training to Doctor of Optometry students, leads the sub-specialty cornea clinic at University of Melbourne eyecare clinic and heads her own research laboratory, the ‘Downie Anterior Eye, Clinical Trials and Research Translation Unit.’ Laura completed both her undergraduate optometry degree (2003) and doctorate (2008) at the University of Melbourne. In 2014, she was awarded two fellowships from the National Health and Medical Research Council and achieved international recognition for her clinical research achievements with the award of the Irvin and Beatrice Borish Award from the American Academy of Optometry. Dr Downie’s research laboratory adopts an integrated approach to research in ocular disease that combines laboratory, clinical and behavioural science as a foundation for evidence-based practice, to improve clinical outcomes. She possesses high level expertise in critical appraisal, research synthesis, translation and implementation science.
Dr Michael Pianta is a Senior Lecturer in the Department of Optometry and Vision Sciences and the Director of Learning and Teaching in the Melbourne School of Health Sciences at the University of Melbourne. Michael completed his undergraduate degree (1991), masters degree (1994), and doctorate (2000) at the University of Melbourne. He is the coordinator of the Doctor of Optometry (OD) program, and has led innovative curriculum design in that program, including use of the CrowdCARE platform. His reputation in competency-based assessment led to his appointment as a consultant for the Optometric Council of Australia & New Zealand, the independent accreditation body for optometry programs in the region.