IATEFL 2015: Testing times all-round

I think Geoff Jordan1 has already outlined the general concerns about the lack of adequate examination of issues in testing at IATEFL 2015, mainly based  on Jeremy Harmer’s presentation2.

One key area that this presentation overlooked is that of high-stakes testing, although Harmer may well have felt that passing a music exam on the tuba was very important for him it does not really compare to test taking candidates whose scores can mean getting a place at university, gaining employment or passing immigration requirements.

The Pearson academic test which Harmer was promoting is used in such high-stakes situations. So glossing over this massive issue was telling. The example of brain surgery in the talk was not comparable as medical education is a long process, and the airline pilots case ignores that such tests cover more or less the whole of a pilot’s syllabus something language tests are rarely able to do.

The lack of critical discussion of automated scoring was another key area. One would be none the wiser from Harmer’s talk that automated scoring is the site of great debate and controversy. For example there’s the Human Readers movement3 which campaigns for the dropping of automated scoring for high-stakes testing.

Further some researchers in the field such as Xiaoming Xi4 claim that automated scoring systems are not ready for high-stakes decision making. She lists two main reasons – they can’t score for “coherence, logic or content like human raters” and the “vulnerability to cheating and test scheming”.

In addition Harmer claimed that “algorithms that are built into the software will grade and evaluate what you say more reliably and as accurately as any human being can.” This confuses the consistency of automated scoring with validity and accuracy, plus such scoring is dependent on human raters in any case.

Controversies such as tests being used to evaluate teachers in the US adds to paint a very contested picture of this area.

It is evident as Harmer says that testing is not going away and that teachers will need to engage with the issues so it was a missed opportunity to help teachers do this instead of simply pushing the onus onto the audience.

One could turn to the following internet sources to get engaged:

Language Testing Bytes – Podcasts on testing compiled by Glen Fulcher.

Thinking about tests and testing: a short primer in “assessment literacy” – A pdf of some useful basics on assessment.

Assessment is – ITDI blog with straight and useful talk from the chalkface, they have some other issues on this also worth checking.

Finally for a bit of fun and look away if you are sensitive to male body parts:

Thanks for reading (and watching).

References:

1. IATEFL post mortem part 2

2. IATEFL 2015 video – An uncertain and approximate business? Why teachers should love testing
https://web.archive.org/web/20180429055948/http://iatefl.britishcouncil.org/2015/session/uncertain-and-approximate-business-why-teachers-should-love-testing

3. Human readers research findings

4. LTJ 27 3 Automated scoring transcript