Learning about EVAF in Edinburgh

3316159385_3b72550e1e_mA couple of weeks ago, I travelled up to Edinburgh [photos on Flickr] for a meeting to discuss the JISC funded EVAF4All: Electronic Voting Analysis and Feedback for All project. Led by Simon Bates and Keith Brunton from the e-Learning unit in the School of Physics and Astronomy at the University of Edinburgh, colleagues from Imperial College London, City University, the Universities of Bath, East Anglia, Leicester and Manchester were in attendance, along with a representative from the JISC-RSC for Scotland.

The current EVAF system allows students to obtain a record of their progress in formative assessment using an Electronic Voting System (refered to an Audience Response System, ARS, at the University of Bath). When viewing their results, students will be directed back to relevant content and presented statistics regarding whole class performance. This feedback can thus aid their developing self-management of their own studying and provide valuable diagnostic feedback. The staff view of the system will enable aggregated data to be extracted and analyse aggregated student performance. [see EVAF website for further information].

The aim of the EVAF4All project is be create an EVS-neutral solution the application to be installed at a variety of institutions irrespective of the VLE and EVS handset (“clicker”) loan scheme. This process in turn would in turn close the ‘feedback loop’ (see Nicol and Mcfarlane-Dick (2006) for further information about this concept) for students engaged in learning activities.

Over the course of the day, a variety of topics were covered in relation to moving the project forward. I have divided these conversations into three main areas, as follows.

1. “Loanership” schemes

The conversation moved onto the different forms of clicker “loanership” scheme currently employed at the different institutions. At my own institution for example, 200 TurningPoint clickers are held by a central AV team. Bags of clickers can be booked out in multiples of 40 clickers [and a single USB RF receiver]. Generally speaking, student participation in formative assessments are completed anonymously.

Other institutions had developed a variety of mechanisms for supporting EVS-related work within learning and teaching. For example, at the University of Edinburgh, students borrow the handsets from their library, where the handset is lent out using a similar process for when taking out a book. Once the barcode on the back of the handset is scanned, the library system, the students records system and the VLE (as well as the EVAF software!) all know that the student has that particular handset. This in turn makes tracking student progression that much easier. The University of Edinburgh currently have over 2500 handsets, across a wide range of disciplines, which are loaned out in such a way. Students can then track their progress using the EVAF application.

In the School of Biological Sciences at the University of Leicester, first year students are lent a handset for the academic year in return for a £10 refundable deposit – the overall loss rate in the first year was 10% of handsets broken or not returned (the latter mainly from students who dropped out). The handsets have been used in the past to track student attendance, meaning that a clicker can be tied to a particular student.

It is because of such loanership models, that these two institutions are in a better place to use their existing e-Learning/MIS infrastructure to participate in the EVAF pilot, and thereby provide a case study of usage. That said, the University of Bath could do so too… it might just take a little longer to get to the end point, and require a little more human intervention to make the process work. [I’ll blog about this further as the project progresses.]

2. Technical infrastructre

As I alluded to in the section above, the are a number of technical requirements that will need to be addressed before the University of Bath will be able to connect to the relevant EVAF instance. For example, at the University of Bath, we have the following systems: Moodle (our VLE), SITS (the student records system), TurningPoint (the EVS system). I would expect that these details will vary across the institutions involved in the project currently. Both Simon and Keith stressed that they will make best efforts to integrate EVAF into our existing structures, but given the timescales, this may be tricky without additional support at a local level. Thankfully, the e-Learning team at the University of Bath have two excellent and enthusastic software developers in house!

3. Getting feedback to students

The key element of this project is of how to best get feedback back to students once the EVS has been used in a lecture. Whilst initial, whole class feedback can be give within the assessment context (e.g. the lecture theatre), the EVAF application seeks to allow students to review the questions post-lecture in their own time. Aligned with this, it is about stressing the importance to students of formative feedback in the learning process. For staff, the software provideds for an easy and effective way to present and view large volumes of user data in a simple way, and begin to assess the effectiveness of the questions used.

Using the TurningPoint 2008 software, colleagues at the University of Bath can already make some question data available to students in a proprietry format, which the students can then analyse themselves. However, this is whole class feedback rather than individual feedback. EVAF takes this process further by personalising feedback and giving the individual student the opportunity to examine their strengths and weaknesses.

At Bath, where no loanership scheme is in place, we still have the problem of how to make this individual feedback available to students (and staff) in an easy and quick way, without a significant additional overhead of time or resource.

I am encouraged by the successes of the EVAF at the University of Edinburgh to date, and applaud both Simon and Keith for having the vision and committment to attempt to roll this project out to additional partner institutions. In addition, both were very receptive to feedback about the EVAF application, and I look forward to becoming more involved with the project as it develops.

I have no doubts that EVAF would be a success if it was implemented at the University of Bath, and in turn, used by the students. I would hope that it would have an impact on any surveys such as the NSS, where students often complain about the lack of feedback from assessment!

Introducing Audio Feedback

Yesterday, I attended a presentation by Dr. Jo Phoenix, from the School of Applied Social Sciences at Durham University. Coming back to the University of Bath for a number of  other meetings, Jo spoke enthusiastically to former colleagues about developing a mechanism at her new institution when providing feedback to students for formative assignments.

Using a digital dictaphone (with a USB cable attachment), Jo recorded audio feedback for formative student assignments and made the files available via duo,  Durham University’s Blackboard-based VLE. Feedback took the form of c. 8 minutes recordings, where Jo would pause/record feedback as she worked her way through the essay.

The majority of students were pleased with the results, particularly as the marker was able to convey tone in her voice, indicating a variety of emotions. One student did comment that she was downbeat about the feedback form as she could “sense the disappointment in [Jo’s] voice”.

The current format allows for all students to hear each others feedback, though as Jo quite rightly pointed out, the feedback would only make sense if the listener had the essay in front of them. A follow-up podcast could then be made avaiable for generalised feedback (for example, covering essay structure, grammar etc.), though this could just as easily be done at some point during a face-to-face lecture.

This approach does raise questions about accessibility of the feedback mechanism to students, and whilst no problems were encountered during Jo’s work, she did mention that feedback could be recorded through Dragon NaturallySpeaking, which would allow feedback to be given in text format with no additional overhead to the marker.

As the feedback for this assessment mentioned above audio-based, no extended feedback is available in written form. This does raise a variety of questions on the Quality Assurance (QA) side, particularly if this approach is extended to include summative assessment.  I would need to defer to my colleague Ellie Clewlow, Head of Enhancement and Development in the LTEO, who might be able to shed more light on the issues to do with the use of audio feedback for summative assignments/assessment. Is it even allowed?

We use Moodle as our VLE at the University of Bath, so a colleagues will have a couple of different options available to them if they wish to setup their own mechanisms for replicating this approach:

  1. If you wish to allow students to hear each others feedback, create a new folder within the Files section of your Moodle course. Upload the files and then link to the relevant folder from your Moodle course home page. An alternative is to make the files available through the Learning Materials Filestore.
  2. If you do not wish to allow students to hear each others feedback,
    • Create a Moodle assignment activity (choose the Advanced uploading of files option) and ask students to submit their assignments online.
    • Send the audio file as a response file, leaving a short amount of written feedback (and grade, if applicable) using the relevant dialog boxes.

Of course, the relevant dictaphones would be available from the Audio Visual unit at the University of Bath.

This inevitably raises questions about the effectiveness of audio feedback to students about work, over other methods. My next task would be to find out if any other institutions, UK and/or abroad, have used this approach and, if applicable, the degree tio which the approach has been successful. Moreover, it would be useful to know if any good practice advice exists.


Get every new post delivered to your Inbox.

Join 1,575 other followers

%d bloggers like this: