Paper Review: Assertion-reason multiple-choice testing

Following my work related to the Audience Response System (ARS) pilot at the University of Bath, I recently came across a paper by Jeremy B. Williams from Universitas 21 Global, Singpore entitled Assertation-reason multiple-choice testing as a tool for deep learning: a qualitative analysis (2006). [download]

Assertion-reason questions (ARQs) are a developed from of Multiple Choice Questions (MCQs) which aim to ‘encourage higher-order thinking on the part of the student’. As Berk (1998 ) remarks, the MCQ format ‘holds world records in the categories of most popular, most unpopular, most used, most misused, most loved, and most hated’. Quite a statement! Williams states that whilst the ARQ format has not been particularly used or embraced, it ‘constitutes a useful assessment tool and one that appears to be superior to the traditional MCQ format in terms of student learning outcomes’.

The aim of the development of ARQs, as outlined by Williams’ project team, was to develop a question set which would test reasoning (procedural knowledge) rather than recall (declarative knowledge). The paper refers to Bloom’s taxonomy (Bloom, 1956) which puts ARQs in the context of aiming to focus on the highest levels of learning within the cognitive domain, namely, Application, Analysis and Synthesis.

As with a traditional set of MCQs, ARQs present students with a given set of possible solutions. However, the key difference in ARQs is that they also include a true/false element. ‘Specifically, each item consists of two statements, an assertion and a rason, that are linked by the word ‘because’. Traditional MCQs will usually only test one particular issue or concept – ARQs will test two per question (the assertion and reason statements) as well as the validity of the ‘because’ statement. As Williams observes, ‘…judging the correctness of two statements must be harder than judging the correctness of one’.

However, the construction and use of ARQs are not without their drawbacks. For example, Connelly (2004) observes that it took some time for the students to become accustomed to this format. In addition, some students (for whom English was not their first language) found that the ARQs were testing their ‘English skills rather than knowledge of the subject being studied’.

Given my work with the ARS at the University of Bath, this ARQ format certainly lends itself to the type of questions that could potentially be asked with the hardware. The aim for any course of ARS-related PowerPoint slides should be limit the number of slides where students already know the answer, but instead getting them to think and consider the arguments put in front of them, thereby using ‘higher order skills’. I wrote a blog post in a similar vain late last year on the ARS project website.

ARQs could well be one way to achieving this, though I feel that I should first undertake additional research to assess the validity of Williams’ claims. For example, has further research been done in this area since 2006? If so, what were the results? Additionally, have ARQs been successfully applied to ARS-related formative (or summative) assessments?

Exploring models for activities, collaboration and assessment in wikis

In preparation for some research project work that I’m going to be doing in relation to collaborating online using wikis, I’ve been reading 2007 a short paper by Edna Tal-Elhasid and Hagit Meishar-Tal from the The Open University of Israel.

The authors put forward a short description of teaching coordinators at the University who expressed an interest in integrating wiki usage into their courses. During the pilot project, teaching coordinators were given the freedom to design their own learning activities based on their own approaches and understanding. Following this work, authors put forward models developed during the course of the pilot, analyse the differences between them and the nature of the activities that took place.

In particular, I was interested in the collaborative models that the authors put forward. A summary of each appears below.

• Cooperation – In this model, most of the work is performed individually by students, who each create a wiki page. Only the individual writes or edits their own page. Their peers use it solely as a resource or product.
• Collaboration and Cooperation - All students work together on the same content, therefore increasing the level of collaboration. As a result of this, the context is edited and improved upon by the group. This is referred to as the process.
• Cooperation, Collaboration and Peer-assessment - Collaboration is implemented with respect to all dimensions: product, process and assessment. Students work in individually or together, upload information to the wiki, edit each other’s work as well as providing peer feedback. Given the complexity of the collaboration aspect, detailed planning of the assessment part of the assignment is required. Each student should feel comfortable enough to contribute, and the ability for one student to monopolise the assignment should be avoided.

Using this model, it should be relatively easy to identify which category a particular wiki falls into. At my own institution, The University of Bath, we’re looking to construct a matrix which will make classification of such wikis easier and possibly more fine-grained. It is the peer-assessment part of this model which I believe is the most difficult for individuals and the teacher to support. I remember hearing a tale of wiki use at another University where a particular group member became so protective of their contribution to a wiki, that they restored an original version of their writing (after changes had been made by a peer), marking it with “This is mine – please do not delete”.

A couple of questions begin to emerge. How can we begin to encourage students to work collaboratively online together, where they take ownership of the space as a collective group, rather than as individuals? What is the best way of promoting a “safe” environment in which students can do this?