I realized I forgot to write anything about the last question for the pre-reflection, so here it goes:
I read a paper titled "Creative thinking in digital game design and development: a case study" that concerned itself with project in an american middle school where students got to take a class in game development.
The was a couple of things I reacted to. First off, the choice of which school to study didn't come off as especially theoretical. Maybe it was, but it wasn't thoroughly explained.
The data collected was purely qualitative: interviews with students, classroom observations and analysis of the games produced. Only 12 interviews were conducted and those students were chosen by the teachers according to loose criteria that they had gotten from the researcher.
There wasn't any substantial shift in perspective, or arguing from other points of view in the paper. It all felt very much at rest on top of several other previous studies and seemed, to me, to rather be an excercise in affirming a hypothesis in a rather weak way rather than putting it to a test with the possibility of falsifying it.
This is another case of a study where I believe in the hypothesis, but can't see how the design of the study and the methods chosen could actually make the hypothesis more likely to be true than what was already confirmed in previous studies. At first glance I thought this would be a good paper to discuss in relation to theory building, but after reading it thoroughly I don't feel that is the case.
For the case study to be a better basis for building theory upon, I think it would have needed a design that enabled the collection of some quantitative data and more qualitative data, on different aspects of the human-computer interaction. Also, the researcher seemed to have gone into the study with multiple hypotheses, all summed up pretty well in the sentence "game development in the curriculum is great because it is open-ended, fosters creativity and technological proficiency". There was never any room for any more specific hypotheses about correlations or explanations of how different factors affected the outcome. The sample-size did also seem very small for being able to predict anything beyond loosely affirming the already formed hypotheses from the beginning of the study.
As far as reflection goes, I enjoyed the seminar we had on Monday. The discussions were interesting, even though they didn't stray very far from the papers we had each read. All in all I feel like I have gotten more comfortable reading papers in an objective way, and having gained experience in reading papers, also gotten more comfortable in analysing research methodology and study design as separate from content.
Hej Edvard!
ReplyDeleteI like the way that you critically reflect on the research paper that you chose. You note that is was an exercise of affirming a hypothesis in a rather weak way. How would you suggest that the researchers would test it in a better way? Do you think that more qualitative data would be enough? Or is it by the combination of quantitative and qualitative methodologies, as you discuss in your reflection?
Regarding your comment on sample sizes being small, I would like to say that I have the same feeling about many case studies. But on the other hand I understand that it is difficult, time consuming and confusing to examine and evaluate the findings of large samples. Furthermore, most of the qualitative methodologies that we discussed during the first seminar were criticized for the same reason-small sample sizes.
Lot's of what you describe I also found in my paper. I often find that the summing at the end is actually inconclusive though at first glance it seems that the hypotheses and conclusions seemed somewhat controlled. After dissecting a paper and critically reviewing it, as you have, one finds the shortcomings of the research. I have a sneaky suspicion that this is a growing phenomenon.
ReplyDeleteI too believe that you have gotten better in being more objective in you critique since I've read earlier reflections from you. Great job! I also enjoy your small rants on academic papers not contributing to the field, we need more of those critiques ;) But to add a comment on this, In some cases it might as well be the results of a research. The hope is of course to contribute to the research in a specific area by asking different questions and changing the methodology perhaps. And you could end up stating: "well this didn't help". But nonetheless it is research which show another way which yields the same result.
ReplyDeleteOf course you can argue in this case that the way they were carrying out the research can't yield different results because the method is too similar. But in that case, you have another source which can confirm the previous research on the same topic.