Paper Review: Assertion-reason multiple-choice testing

Following my work related to the Audience Response System (ARS) pilot at the University of Bath, I recently came across a paper by Jeremy B. Williams from Universitas 21 Global, Singpore entitled Assertation-reason multiple-choice testing as a tool for deep learning: a qualitative analysis (2006). [download]

Assertion-reason questions (ARQs) are a developed from of Multiple Choice Questions (MCQs) which aim to ‘encourage higher-order thinking on the part of the student’. As Berk (1998 ) remarks, the MCQ format ‘holds world records in the categories of most popular, most unpopular, most used, most misused, most loved, and most hated’. Quite a statement! Williams states that whilst the ARQ format has not been particularly used or embraced, it ‘constitutes a useful assessment tool and one that appears to be superior to the traditional MCQ format in terms of student learning outcomes’.

The aim of the development of ARQs, as outlined by Williams’ project team, was to develop a question set which would test reasoning (procedural knowledge) rather than recall (declarative knowledge). The paper refers to Bloom’s taxonomy (Bloom, 1956) which puts ARQs in the context of aiming to focus on the highest levels of learning within the cognitive domain, namely, Application, Analysis and Synthesis.

As with a traditional set of MCQs, ARQs present students with a given set of possible solutions. However, the key difference in ARQs is that they also include a true/false element. ‘Specifically, each item consists of two statements, an assertion and a rason, that are linked by the word ‘because’. Traditional MCQs will usually only test one particular issue or concept – ARQs will test two per question (the assertion and reason statements) as well as the validity of the ‘because’ statement. As Williams observes, ‘…judging the correctness of two statements must be harder than judging the correctness of one’.

However, the construction and use of ARQs are not without their drawbacks. For example, Connelly (2004) observes that it took some time for the students to become accustomed to this format. In addition, some students (for whom English was not their first language) found that the ARQs were testing their ‘English skills rather than knowledge of the subject being studied’.

Given my work with the ARS at the University of Bath, this ARQ format certainly lends itself to the type of questions that could potentially be asked with the hardware. The aim for any course of ARS-related PowerPoint slides should be limit the number of slides where students already know the answer, but instead getting them to think and consider the arguments put in front of them, thereby using ‘higher order skills’. I wrote a blog post in a similar vain late last year on the ARS project website.

ARQs could well be one way to achieving this, though I feel that I should first undertake additional research to assess the validity of Williams’ claims. For example, has further research been done in this area since 2006? If so, what were the results? Additionally, have ARQs been successfully applied to ARS-related formative (or summative) assessments?

Learning Futures Festival ‘09

It’s been a while since I last posted. First off, Happy New Year!I guess a New Year’s resolution should be post a little more than I have been doing of late. Anyway…

Last Thursday, I attended the University of Leicester-hosted Learning Futures Festival ‘09 at the Walkers Stadium, the home of Leicester City Football Club.

After a short introduction from the University of Leicester Vice Chancellor, Professor Robert Burgess, the opening keynote was delivered by Professor Sugata Mitra from Newcastle University, who spoke on the title An Alternative Schooling: Self Organising Systems in Education.  I managed to jot down a few thoughts from the first part of Professor Mitra’s keynote:

• If good schools don’t exist, good teachers don’t want to go there. This in turn has a knock on effect on the student experience.
• Education technology – perceived to be over-hyped and under-performing in schools that do indeed have good students and teachiners.
• Teachers are borrowing technologies that were designed for the corporate boardroom, for example, laptops, projectors… and Microsoft PowerPoint!

I felt a particular resonance to that final point which seemed to reinforce earlier thoughts that I have had that very few hardware and software applications have been designed with teachers/lecturers in mind. I think of the case of my experience with interactive whiteboards – some of the software that I have come across in the past has been particularly cumbersome, so much so, that I have given up using this technology and found an alternative.  In turn, and relating this to my own work at the University of Bath, I think about the number of colleagues who could be resisting engaging with our institutional VLE and/or Web 2.0 technologies, simply because the learning curve at the beginning is too steep. Google Docs is a fantastic application (which I use almost daily), but does take a little time to get to grips with.. and certainly was not designed with teachers (or students) in mind.

Professor Mitra then moved on to the notion of self-organising systems, before talking through his involvement with the Hole in the Wall (HIWEL) research work. As a study of how technology can have an impact on people in a developing country (India), I found this project absolutely fascinating. Through HIWEL, a single computer was set up in an outdoor playground setting which children can access at any time. With monitor and keypad (and trackball), was setup with an English language internet browser. Often, the children had never seen a computer before and could not understand the English language, so this was a rather interesting concept which could have failed spectacularly. However, as Professor Mitra explained, this was where the term “self instruction” was coined – the students simply taught themselves the skills that they needed to use the hardware that was put in front of them. In one particular case, Professor Mitra returned to the scene of his experiment three months of placing the hardware in a location, only to be greeted with the following from the children: “We want a faster processor and a better mouse”. Impressive! The children not only learned how to use computers, but only learned how to speak English. If children can teach themselves how to use both computers and the Internet, what else can they teach themselves? In another experience, non-English speaking children were given some Biochemsitry-related material, and within three months could explain, in English, a key concept related to DNA and genetics [sorry,  I can't remember the specifics!]. This illustrated clearly that grasp of a language was certainly not a barrier to learning.

The second keynote of the day was delivery by Dr Ralph Schroder from the Oxford Internet Institute from the University of Oxford. On the title of Virtual Environments and the Future of Distance Collaboration, Ralph focused on virtual environments and virtual worlds, and technologies for collaborating, educating and training at a distance. Ralph spoke about two particular approaches, [i] Blue-C: the video captured immersive end-state and [ii] CAVE-live systems: the computer generated end-state. For further exploration of the issues raised in this keynote, please head over to my colleague Roger Gardner’s blog.

Following these two keynotes, I headed to the first ‘Intervention’ session of the day entitled Strand 2 – Future Technologies, which revolved around the theme of personal[ised] learning. The most enthusiastic of the presentations came from Professor Rozhan M. Idrus from the Universiti Sains Malaysia, who spoke about the concept of Technogogy,  defined as “the convergence of technology, pedagogy and content in the transformative use of technology to foster learning” [Idrus 2008]. The diagram on the right is the visual representation of this content.

Professor Idrus argued [unfortunately for only five minutes!] that by taking the buffett approach, the same content can be taught in lots of different ways, drawing on differing levels of content, technology and pedagogy. Indeed, his vision is to move institutions from placing learning objects in a content repository into a directory of learning. Indeed, the concept of technogogy is an emerging one and one that is yet to be fully explored and discussed by others.

Following the lunch break, during which time Roger and I were given a rather short tour of the Walkers Stadium [photos on Flickr], I attended the second ‘Intervention’ sesson, entitled Strand 4 – Creating the Future. Once again, this session featured four panel members including Dave Hall, Registrar and Secretary at the University of Leiecster. Dave session focused on the notion of change management, and the requirement to make it central to institutional objectives if it is to be successful. From my own work prior to joining the University of Bath, I can appreciate the strategic importance of Dave’s message. To make change a success in an institution, it needs to be supported, through the following five key points:

1. Prove it [the change] works
2. Catch them young – target new lecturers
3. Pay for it(!)
4. Encourage students to demand change [could this be done through mechanisms such as Online Unit Evaluation and surveys such as the National Students Survey?]
5. Do it together – make it as easy as possible [for staff] to access resources, facilities, training and support

Overall, a day at the Learning Futures Festival ‘09 provided an excellent forum to hear about some of the issue issues currently facing learning technologies in the UK and elsewhere. I came away with some useful thoughts and reflections about the evolving nature of personalised learning and how I can apply these to my own current work. Indeed, I would interested to hear from colleagues at other institutions who have explored in notion of self-organising systems in an HE level, in the UK or otherwise.