The whole is more than the sum of its parts – assessing writing using the consensual assessment technique

Research output: Contribution to journalArticle

Abstract

Evaluating the impact of Academic Literacies teaching (Lea and Street [1998. “Student Writing in Higher Education: An Academic Literacies Approach.” Studies in Higher Education 23 (2): 157–72. doi:10.1080/03075079812331380364]) is difficult, as it involves gauging whether writers (1) gain better understanding of what influences written social interactions, and (2) improve their ability to manipulate language to address readers. Self-report can assess the first; the second can only be evaluated by examining texts and their effect on readers. Texts are commonly assessed with rubrics-based tools focussing on textual features, but their insensitivity to communicative context and readers’ perception makes these inappropriate for an Academic Literacies framework (Canton [2018. “‘It’s Hard to Define Good Writing, but I Recognise it When I See it’: Can Consensus-Based Assessment Evaluate the Teaching of Writing?” Journal of Academic Writing 8 (1): 13–27. doi:10.18552/joaw.v8i1.450]). Consensual assessment used by Amabile ([1996. Creativity in Context. Boulder, CL: Westview Press]) for creativity, offers a potential solution (Canton [2018. “‘It’s Hard to Define Good Writing, but I Recognise it When I See it’: Can Consensus-Based Assessment Evaluate the Teaching of Writing?” Journal of Academic Writing 8 (1): 13–27. doi:10.18552/joaw.v8i1.450]). This paper presents a new instrument based on consensual assessment and empirically tests it. Intra-class Correlation Coefficients (ICCs) found moderate (Koo and Li [2016. “A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research.” Journal of Chiropractic Medicine 15 (2): 155–63. doi:10.1016/j.jcm.2016.02.012]) to good (Cicchetti [1994. “Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology.” Psychological Assessment 6 (4): 284–90. https://pdfs.semanticscholar.org/50d7/f68422d0c0424674f6b235ac23be8300da38.pdf]) agreement among raters, which offers proof of concept for capturing the readers’ perception of the complex interactions in writing.
Original languageEnglish
Number of pages13
JournalStudies in Higher Education
Early online date24 Jan 2020
DOIs
Publication statusE-pub ahead of print - 24 Jan 2020

Fingerprint

canton
creativity
Teaching
interaction
education
psychology
writer
medicine
ability
language
student

Keywords

  • consensus
  • writing
  • academic literacies
  • assessment
  • ICC

Cite this

@article{d8b58454287340a297a6d61679e8f5fc,
title = "The whole is more than the sum of its parts – assessing writing using the consensual assessment technique",
abstract = "Evaluating the impact of Academic Literacies teaching (Lea and Street [1998. “Student Writing in Higher Education: An Academic Literacies Approach.” Studies in Higher Education 23 (2): 157–72. doi:10.1080/03075079812331380364]) is difficult, as it involves gauging whether writers (1) gain better understanding of what influences written social interactions, and (2) improve their ability to manipulate language to address readers. Self-report can assess the first; the second can only be evaluated by examining texts and their effect on readers. Texts are commonly assessed with rubrics-based tools focussing on textual features, but their insensitivity to communicative context and readers’ perception makes these inappropriate for an Academic Literacies framework (Canton [2018. “‘It’s Hard to Define Good Writing, but I Recognise it When I See it’: Can Consensus-Based Assessment Evaluate the Teaching of Writing?” Journal of Academic Writing 8 (1): 13–27. doi:10.18552/joaw.v8i1.450]). Consensual assessment used by Amabile ([1996. Creativity in Context. Boulder, CL: Westview Press]) for creativity, offers a potential solution (Canton [2018. “‘It’s Hard to Define Good Writing, but I Recognise it When I See it’: Can Consensus-Based Assessment Evaluate the Teaching of Writing?” Journal of Academic Writing 8 (1): 13–27. doi:10.18552/joaw.v8i1.450]). This paper presents a new instrument based on consensual assessment and empirically tests it. Intra-class Correlation Coefficients (ICCs) found moderate (Koo and Li [2016. “A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research.” Journal of Chiropractic Medicine 15 (2): 155–63. doi:10.1016/j.jcm.2016.02.012]) to good (Cicchetti [1994. “Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology.” Psychological Assessment 6 (4): 284–90. https://pdfs.semanticscholar.org/50d7/f68422d0c0424674f6b235ac23be8300da38.pdf]) agreement among raters, which offers proof of concept for capturing the readers’ perception of the complex interactions in writing.",
keywords = "consensus, writing, academic literacies, assessment, ICC",
author = "Daniela Zahn and Ursula Canton and Victoria Boyd and Laura Hamilton and Josianne Mamo and Jane McKay and Linda Proudfoot and Dickson Telfer and Kim Williams and Colin Wilson",
note = "Acceptance in SAN AAM: 18m embargo",
year = "2020",
month = "1",
day = "24",
doi = "10.1080/03075079.2019.1711044",
language = "English",
journal = "Studies in Higher Education",
issn = "0307-5079",
publisher = "Routledge",

}

TY - JOUR

T1 - The whole is more than the sum of its parts – assessing writing using the consensual assessment technique

AU - Zahn, Daniela

AU - Canton, Ursula

AU - Boyd, Victoria

AU - Hamilton, Laura

AU - Mamo, Josianne

AU - McKay, Jane

AU - Proudfoot, Linda

AU - Telfer, Dickson

AU - Williams, Kim

AU - Wilson, Colin

N1 - Acceptance in SAN AAM: 18m embargo

PY - 2020/1/24

Y1 - 2020/1/24

N2 - Evaluating the impact of Academic Literacies teaching (Lea and Street [1998. “Student Writing in Higher Education: An Academic Literacies Approach.” Studies in Higher Education 23 (2): 157–72. doi:10.1080/03075079812331380364]) is difficult, as it involves gauging whether writers (1) gain better understanding of what influences written social interactions, and (2) improve their ability to manipulate language to address readers. Self-report can assess the first; the second can only be evaluated by examining texts and their effect on readers. Texts are commonly assessed with rubrics-based tools focussing on textual features, but their insensitivity to communicative context and readers’ perception makes these inappropriate for an Academic Literacies framework (Canton [2018. “‘It’s Hard to Define Good Writing, but I Recognise it When I See it’: Can Consensus-Based Assessment Evaluate the Teaching of Writing?” Journal of Academic Writing 8 (1): 13–27. doi:10.18552/joaw.v8i1.450]). Consensual assessment used by Amabile ([1996. Creativity in Context. Boulder, CL: Westview Press]) for creativity, offers a potential solution (Canton [2018. “‘It’s Hard to Define Good Writing, but I Recognise it When I See it’: Can Consensus-Based Assessment Evaluate the Teaching of Writing?” Journal of Academic Writing 8 (1): 13–27. doi:10.18552/joaw.v8i1.450]). This paper presents a new instrument based on consensual assessment and empirically tests it. Intra-class Correlation Coefficients (ICCs) found moderate (Koo and Li [2016. “A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research.” Journal of Chiropractic Medicine 15 (2): 155–63. doi:10.1016/j.jcm.2016.02.012]) to good (Cicchetti [1994. “Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology.” Psychological Assessment 6 (4): 284–90. https://pdfs.semanticscholar.org/50d7/f68422d0c0424674f6b235ac23be8300da38.pdf]) agreement among raters, which offers proof of concept for capturing the readers’ perception of the complex interactions in writing.

AB - Evaluating the impact of Academic Literacies teaching (Lea and Street [1998. “Student Writing in Higher Education: An Academic Literacies Approach.” Studies in Higher Education 23 (2): 157–72. doi:10.1080/03075079812331380364]) is difficult, as it involves gauging whether writers (1) gain better understanding of what influences written social interactions, and (2) improve their ability to manipulate language to address readers. Self-report can assess the first; the second can only be evaluated by examining texts and their effect on readers. Texts are commonly assessed with rubrics-based tools focussing on textual features, but their insensitivity to communicative context and readers’ perception makes these inappropriate for an Academic Literacies framework (Canton [2018. “‘It’s Hard to Define Good Writing, but I Recognise it When I See it’: Can Consensus-Based Assessment Evaluate the Teaching of Writing?” Journal of Academic Writing 8 (1): 13–27. doi:10.18552/joaw.v8i1.450]). Consensual assessment used by Amabile ([1996. Creativity in Context. Boulder, CL: Westview Press]) for creativity, offers a potential solution (Canton [2018. “‘It’s Hard to Define Good Writing, but I Recognise it When I See it’: Can Consensus-Based Assessment Evaluate the Teaching of Writing?” Journal of Academic Writing 8 (1): 13–27. doi:10.18552/joaw.v8i1.450]). This paper presents a new instrument based on consensual assessment and empirically tests it. Intra-class Correlation Coefficients (ICCs) found moderate (Koo and Li [2016. “A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research.” Journal of Chiropractic Medicine 15 (2): 155–63. doi:10.1016/j.jcm.2016.02.012]) to good (Cicchetti [1994. “Guidelines, Criteria, and Rules of Thumb for Evaluating Normed and Standardized Assessment Instruments in Psychology.” Psychological Assessment 6 (4): 284–90. https://pdfs.semanticscholar.org/50d7/f68422d0c0424674f6b235ac23be8300da38.pdf]) agreement among raters, which offers proof of concept for capturing the readers’ perception of the complex interactions in writing.

KW - consensus

KW - writing

KW - academic literacies

KW - assessment

KW - ICC

U2 - 10.1080/03075079.2019.1711044

DO - 10.1080/03075079.2019.1711044

M3 - Article

JO - Studies in Higher Education

JF - Studies in Higher Education

SN - 0307-5079

ER -