The Opposite

Just do the opposite

Kevin Kelley: How does technology evolve? Like we did

Kevin Kelley asks "What does technology want?" and posits that its movement toward ubiquity and complexity is much like the evolution of life.

Eric Reiss: user centered design or user driven design?

"If you are truly trying to be innovative, you have got to solve a problem. You don't want to just be different for the sake of being different"

University of Glamorgan - JISC Case Study

Phased online summative assessment in undergraduate accounting

This project aimed to address a situation where students were performing poorly not participating in the learning process, and only engaging superficially with the course by setting up a phased online assessment which would provide students with timely feedback to facilitate reflection and encourage active learning, and identify areas wehre additional support was needed .

This project used QMP (Question Mark Perception) a web based tool for management of quizzes suveys and assessments to create question banks, an expert in the software created the question banks, and provided training in the use of QMP software. QMP is a mature product.

Students were offered a choice of paper or online, with the exception of a visually impaired student, all but one students opted for online and the one student who didn't changed his mind for the next test.

Hard copies prepared as contingency in case technical issues provented online delivery of test

Limitations on space in the computer rooms meant that there had to be multiple deliveries of the test, and delivery of the questions were randomised to avoide cheating. Measures were taken to ensure equal distribution of difficulty within the randomised question sets.

Student performance increased on the financial accounting module and students stated that they preferred to be assessed online, that they preferred small tests covering specific areas of the syllabus, and that it helped with time management. Students found the feedback useful for identifying areas of weakness.

The positive comments received from students indicated that they recognised the benefits to be derived from this phased or continuous assessment, suggest that a change in assessment practice has been effective.

The project was also successful in measuring student development and identifying students that were experiencing difficulties in understanding certain topic areas which enabled the tutors to provide appropriate and timely support.

This project did result in an improvement student learning and showed how technology can be used successfully for assessment purposes.

The main costs of this were the time spent in setting up question banks and the cost of the software.

University of Birmingham - JISC Case Study

Facilitating the use of WinEcon for interactive learning

WinEcon is a toolbook based interactive learning package available to higher education institutions containing learning materials for Economics. The objective of the project was to get students and lecturers to engage more with the software and exploit the benefit that such a package offers.

The project team worked with 18 universities (22 modules and 19 lecturers) who subscribed to WinEcon, and found that in general there was a very low adoption and use of WinEcon amongst lecturers at institutions who subscribe to it, generally used as an add on resource.

This was partly due to the 'not invented here' syndrome and that lecturers thought it would take too long to learn, and if they did spend the time learning it, then this time may well be wasted because they did not know how useful the learning materials would be to them, and were not sure that students would actually use them.

The project team trained seven people, mainly postgraduate students from English universities to communicate the benefits of WinEcon.

The project identified that there were many technical barriers to implementation, and that fear of such problems was a large factor in discouraging participation. The team therefore encouraged WinEcon Consortium to improve their product by making linking easier and improving the user interface. The team also provided educational material on setting up links to WinEcon.

They arranged for the product to be made available at more attractive cost to students.

The dissemination model used, provided a model for other activities.

There was an unanticipated need for a dedicated member of staff to maintain the project.

University of Warwick -

Use of e-assessment in medicine

The project at University of Warkwick was to take a a blueprint approach - a 'take action' approach - to assessment where assessment was centred around GMC competency areas, and feedback given on their assessment which would directly guide them to plan learning strategies to help them improve their performance. Assessing by module did not encourage 'integrated thinking' and previously assessments did not use integrated questions (questions relating to subject matter learned ouside that module) teaching and feedback was in terms of a score for each module.

Feedback about learning was a high priority, give students a better insight into their exam performance.

Some staff had to be persuaded of the value of this exericse and not everybody understood or was aware of the GMC competency areas, including Module co-ordinators.

Necessary IT was not in place

Technology

Presented a range of technical challenges, questions were coded against GMC competency areas by use of spreadsheets, record sheets were used by markers that could be scanned by an optical mark reader.

A database was used to correlate exam marks against competency areas.

Students received scores in integrated questions along with attainment against competency areas. Both students and tutors received feedback emails, so tutors could help students interpret results.

System yet to be evaluated, but has influenced assessment policy

Required a new IT Post to help create the systems necessary to roll out and to train admin staff.

Further work:

In order to develop a competency framework for students to cover all four years of the course then all assessments must be 'blueprinted'.

Need to develop an assessment item (question?) bank

A better (than email) automated feedback system

To add a competency element to the portfolio system

No pedagogical drawbacks to providing students with better, individualised feedback on exam performance, however on first implementation students were unsure of how to interpret their results and tutors did not feel equipped to help them.

Leeds Metropolitan University - JISC Case Study

Computer Assisted Summative Assessment

This case study concerns the use of summative computer assisted assessment (CAA) in an Applied Technology and Finance module taught to students of Tourism and Events Management.

Aims:

• Support the learning and teaching strategy
• Save staff time marking 350 scripts
• Provide earlier feedback to students
• Familiarlise students with rigid deadlines and HE assessment procedures. Students arriving from FE often felt deadlines could be negotiated.

The module - Applied Finance and Technology - semester one level one course taken by (350) students on two different courses. Finance and Technology are both subjects which can cause stress some students may have struggled with numeracy and IT literacy.

Previous assessment was a paper submission - marking and giving feedback took time (3-4 weeks), only one single feedback opportunity.

Proposal

5 fortnightly stage tests @ 10% (50%)
Final Exam 50%

Few challenges to implementation since staff teaching the module were key drivers of change.

Concern over whether the VLE system would cope with the traffic level of 350 students taking test concurrently.

Students had full details of what was expected of them over the course of the module

Many students not used to strict university processes and during the trials wanted to choose when they did their test. so guidelines had to be established about how this would work with mitigation requied for missing final exam.

Some students complained that others could look over their shoulders and copy their answers, so question banks were extended and 20 questions were presented from a bank of 30 to each student and the order they were presented was also randomised.

Used WebCT which is a proprietory VLE system used within the organisation. A tool called Respondus was also used to aid the writing of questions.

Tangible Benefits

Acheivement of strategy target

Students received final marks within three hours of finishing the exam - it would have been possible to give the marks sooner, but course staff reviewed stats for each question and standardisation.

Staff time for assessment reduced from 120 hours to 20 hours.

Increased use of reusable learning objects and question banks could reduce time costs even further, but there is a large upfront development time investment required - important for this to be acknowledged by institution.

Testing students every two weeks improved attendance during test weeks, and results sent directly to tutors to monitor progress and follow up absences.

Project was led by staff who would be teaching the module, and who were enthusastic advocates 'champions' of the WebCT eLearning system.

Students had a lot of time to practice the test, so would obviuosly have deveoped their test taking skills.

mean average exam results rose from 53% to 65%.

Forced exposure to something they were resistant to built confidece, engagement and possibly opened their mind to broader study.

Drawbacks

This type of learning suited to Level One students - not so suitable for students at higher level

Preparation and testing of large numbers of questions represented a considerable up front effort, but good scalability since more students can be added to the course with little extra effort.

Additional staff development for new staff teaching the module.

Initial expectations were that computer assessments would be formative, not summative, but strategic approach of students at this level meant that students might not engage with something that apparently carried no credit.

This project demonstrated that small summative assessments are something that students will engage with at this level.