Thursday, June 25, 2015

HCI research and the problems with the scientific method

A few years ago I read an article in the New Yorker about the phenomenon of 'declining truth'. I have been thinking about this article since then and today my PhD student found it and I had the chance to read it again (thanks Jordan). It is an article that asked critical questions about the scientific method in general and specifically about replicability. Reading this article today makes me reflect upon the present status of HCI research of course and its relation to the scientific method. Will come back to that.

The article is "The truth wears off -- is there something wrong with the scientific method?" by Jonah Lehrer. The article was published in 2010 so things may have changed a bit since then. The questions asked in the article are interesting and challenging to any science practitioner. The topic of the article is a phenomenon that has been described and discussed by several scientists over the last decades. It is by some called the "decline effect". The phenomenon is that scientific results that are highly significant seems to wear off over time. It is not easy to understand what could be the cause of this effect. Lehrer discusses several potential answers to why this effect is showing up, for instance, it could be that scientists are biased when they conduct experiments, maybe there is a bias in the publication system towards studies with 'positive' results, etc. However, it is not easy to find one simple explanation.

Lehrer describes the history of the idea of 'fluctuating asymmetry' that was discovered in 1991. The discovery showed that barn swallow females prefer males that show a symmetrical appearance. The idea drawn from this was, among other things, that aesthetics is genetics. Over several years there were many studies around the world that confirmed the theory by studying other animals, even humans. But, a few years later the theory started to become questioned. More and more studies could not find any results that supported the study. Even so, the theory is still today found in textbooks and has influenced other areas. Lehrer shows that this is not a one time thing, there are large studies that show this effect in other areas and disciplines.

I find this article interesting and I think it should be read by anyone who deals with research and science. It is not that the article destroys science in any way, but it does add a healthy dose of skepticism when it comes to our general trust in the scientific method. And especially when it comes to the role of empirical research and its relation to truth. As the author ends, it reminds us of how difficult it is to prove anything and it asks some good questions about what evidence is.

I think this article is also raises good questions for HCI research. Our field is of course not a field based on scientific inquiry in the same way as physics or biology but there is a strong belief in our field that at the end of the day ideas have to be tested against nature, that is, empirical studies are needed to make convincing claims.

What is maybe disturbing for HCI research is that if the fields in the article, which are disciplines with a strong history of scientific empirical practice, have problems, then the problem in HCI might be much worse. Despite the problems in these other fields their tradition and culture of doing scientific studies is something HCI is not close to. Replicability, which is a core concept in these disciplines, is barely mentioned in HCI research and not taken seriously. I am not arguing that HCI research should become more rigorous or scientific, actually the opposite. I do not believe that almost any research in HCI is of a kind that either require or is suitable for scientific research in the way that the article discusses. However, I see a serious problem in HCI research that is caused by a conflict or internal dilemma that stems from a shared view that HCI is not really a scientific enterprise while at the same time scientific research  is still valued and rewarded. This dilemma is easy to see not only on the level of the discipline but in individuals. It is possible in HCI to see condescending remarks about science that are strangely mixed with a scientific practice that seems to aspire to become 'real' science.

HCI research will of course over time develop a position in relation to science as practiced in other disciplines in a way that hopefully is adequate for the purposes and context of the field. I am quite optimistic in the sense that I see more debate about HCI research today than just a few years ago. I also find it intriguing to see where the field will go. As far as I can see it, there are so many potential directions open right now. HCI can move towards a more scientific tradition, a humanistic tradition, or towards a designerly tradition, or towards a new and particular understanding of what research in this field is all about.