IATEFL 2015: Testing times all-round

I think Geoff Jordan1 has already outlined the general concerns about the lack of adequate examination of issues in testing at IATEFL 2015, mainly based  on Jeremy Harmer’s presentation2.

One key area that this presentation overlooked is that of high-stakes testing, although Harmer may well have felt that passing a music exam on the tuba was very important for him it does not really compare to test taking candidates whose scores can mean getting a place at university, gaining employment or passing immigration requirements.

The Pearson academic test which Harmer was promoting is used in such high-stakes situations. So glossing over this massive issue was telling. The example of brain surgery in the talk was not comparable as medical education is a long process, and the airline pilots case ignores that such tests cover more or less the whole of a pilot’s syllabus something language tests are rarely able to do.

The lack of critical discussion of automated scoring was another key area. One would be none the wiser from Harmer’s talk that automated scoring is the site of great debate and controversy. For example there’s the Human Readers movement3 which campaigns for the dropping of automated scoring for high-stakes testing.

Further some researchers in the field such as Xiaoming Xi4 claim that automated scoring systems are not ready for high-stakes decision making. She lists two main reasons – they can’t score for “coherence, logic or content like human raters” and the “vulnerability to cheating and test scheming”.

In addition Harmer claimed that “algorithms that are built into the software will grade and evaluate what you say more reliably and as accurately as any human being can.” This confuses the consistency of automated scoring with validity and accuracy, plus such scoring is dependent on human raters in any case.

Controversies such as tests being used to evaluate teachers in the US adds to paint a very contested picture of this area.

It is evident as Harmer says that testing is not going away and that teachers will need to engage with the issues so it was a missed opportunity to help teachers do this instead of simply pushing the onus onto the audience.

One could turn to the following internet sources to get engaged:

Language Testing Bytes – Podcasts on testing compiled by Glen Fulcher.

Thinking about tests and testing: a short primer in “assessment literacy” – A pdf of some useful basics on assessment.

Assessment is – ITDI blog with straight and useful talk from the chalkface, they have some other issues on this also worth checking.

Finally for a bit of fun and look away if you are sensitive to male body parts:

Thanks for reading (and watching).

References:

1. IATEFL post mortem part 2

2. IATEFL 2015 video – An uncertain and approximate business? Why teachers should love testing

3. Human readers research findings

4. LTJ 27 3 Automated scoring transcript

8 thoughts on “IATEFL 2015: Testing times all-round

  1. Hi Mura,

    I was going to follow up my short rant with some suggestions about where to find a Harmer antidote, but you’ve saved me the trouble. Good to see that you’ve put a link to Glenn Fulcher’s really excellent website. Allow me to add a plug for the journal Language Testing, always worth browsing, and a book review in this month’s issue “Criterial Features in L2 English: Specifying the Reference Levels of the Common European Framework” . A very important book, which you, Mura, will love, if you haven’t already got a copy. http://ltj.sagepub.com.ezproxy3.lib.le.ac.uk/content/current

  2. Hi Mura,

    Sorry – the link should have been: http://ltj.sagepub.com/ Then click on “current issue” and the book review is at the bottom. You MUST have a look at this book!

    Yes, I saw the Pearson signature event. They correctly identify some of the problems of the CEFR proficiency stages, but fail to deal with the equally important problem posed by the use of their own coursebooks. Thus, they argue that everything would be OK if the European “levels” corresponded better to their own series of coursebooks, which offer grammar-based, product syllabuses. The presenters proceeded (totally unchallenged by anybody in the audience) on the false assumption that in classrooms where learners are led through successive units of successive coursebooks, they will learn what they’re taught when they’re taught it.

    1. Hi Geoff,
      Couldn’t find anywhere to read a review of Criterial Features in L2 English: Specifying the Reference Levels of the Common European Framework without first subscribing to some expensive academic journal. Maybe it’s just too soon?
      Cheers,
      Glenys

      PS: Love a good academic ding dong – thanks for starting this one.

  3. Hi Glenys,
    I’m lucky in that my work for Leicester University gives me free access to al academic journals. I’m afraid you’re right – hefty payments are required otherwise. Email me and I’ll send you a copy,

Penny for your thoughts

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s