The whole is more than the sum of its parts – assessing writing using the consensual assessment technique

Evaluating the impact of Academic Literacies teaching (Lea and Street [1998. “Student Writing in Higher Education: An Academic Literacies Approach.” Studies in Higher Education 23 (2): 157–72. doi:10.1080/03075079812331380364]) is difficult, as it involves gauging whether writers (1) gain better understanding of what influences written social interactions, and (2) improve their ability to manipulate language to address readers. Self-report can assess the first; the second can only be evaluated by examining texts and their effect on readers. Texts are commonly assessed with rubrics-based tools focussing on textual features, but their insensitivity to communicative context and readers’ perception makes these inappropriate for an Academic Literacies framework (Canton [2018. “‘It’s Hard to Define Good Writing, but I Recognise it When I See it’: Can Consensus-Based Assessment Evaluate the Teaching of Writing?” Journal of Academic Writing 8 (1): 13–27. doi:10.18552/joaw.v8i1.450]). Consensual assessment used by Amabile ([1996. Creativity in Context. Boulder, CL: Westview Press]) for creativity, offers a potential solution (Canton [2018. “‘It’s Hard to Define Good Writing, but I Recognise it When I See it’: Can Consensus-Based Assessment Evaluate the Teaching of Writing?” Journal of Academic Writing 8 (1): 13–27. doi:10.18552/joaw.v8i1.450]). This paper presents a new instrument based on consensual assessment and empirically tests it. Intra-class Correlation Coefficients (ICCs) found moderate (Koo and Li [2016. “A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research.” Journal of Chiropractic Medicine 15 (2): 155–63. doi:10.1016/j.jcm.2016.02.012]) to good (Cicchetti [1994. “Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology.” Psychological Assessment 6 (4): 284–90. https://pdfs.semanticscholar.org/50d7/f68422d0c0424674f6b235ac23be8300da38.pdf]) agreement among raters, which offers proof of concept for capturing the readers’ perception of the complex interactions in writing.