Friday, October 26, 2012

CHI reviewing: Some reflections

Reviewing for a quality conference such as CHI is an excellent way to find out what is going on in the field. What is even more beneficial is that I have to read things I would never otherwise read.  I promised to review 8 papers this year since I felt bad being on several submitted papers. (We should all pay our dues by reviewing at least as many papers as we submit.)

Anyway, I have now reviewed all of them (well, working on the last one). One interesting aspect that I saw in many of the papers is a mismatch between the way theory is said to be used and how it is actually used.

The typical mistake looks like this. The authors start with an introduction, usually quite good. Then comes the "theory" part, also in many cases surprisingly good. Several papers have impressed me by taking on quite ambitious theoretical perspectives in relation to their research. In some cases I read excellent reviews of existing theory with quite interesting reflections on how it relates to the topic in question. But then the problems start. In almost all the papers, after the theory has been introduced, it disappears. It is seldom possible to find any trace of the theory in the study or experiment that is usually the core of the paper. The analysis and interpretation of the results and findings are not at all done based on or informed by the theory. Another problem is that the authors do not return to the theory at the end of the paper. The findings does not lead to any revisiting of the theory or to any reflections of the use of the theory, for instance, in relation to how well it worked or what the theory failed to support.

I find it ok if researchers use a theory as a tool, that is how I see theories myself. They are tools that can help us to describe and analyze something. So, it is fine if the research does not necessary lead to any final or revolutionary reflections on the theory used, or to any form theoretical development (even though if that is the case the paper usually becomes quite boring). However, it is definitely not necessary to introduce a theory or a conceptual framework in a paper if it is not used in the actual research process. If it is not used, why do we have to read about it?

This means that many papers start out with grand plans on a fairly advanced theoretical level while it ends on a low practical level that in no way correspond to the claims and issues introduced in the beginning. It would be wonderful to read a paper once that had the opposite structure, that is, that started out with a simple and practical problem but ended in an advanced discussion about theoretical implications and developments. Maybe they are our there. Maybe I just did not get any to review this year.

[In a paper by me and Mikael Wiberg we discuss the idea of teory driven research as an opportunity for improvement in HCI research, http://www.tandfonline.com/doi/abs/10.1080/07370020903586696]

1 comment:

GK said...

Thank you Erik. When you say "theory" do you mean theory appropriated from other disciplines or the author(s) themselves proposing new/extended theories for HCI? I ask this because in the case of the former, the goal might be to provide an orientation - how to read and make sense of the work - instead of having to legitimate such a theory (which might actually be less relevant to the venue considering that it is from other discipline).