Even though concerns have been raised regarding self-report instruments such as the PISA student-questionnaire, self-report surveys still seem to predominate the educational research, psychology and social sciences in general (Schwartz 1999). Research aimed at improving such measurements is therefore needed. Cognitive research methods have gained acceptance for improving questionnaires and survey research (Sudman, Bradburn et al. 1996). One of these methods, cognitive interviews, is described as a promising tool for improving surveys and self-reports (Karabenick et al. 2007).

After the PISA results were released in 2000 and 2003, the results regarding the measurement of self-regulation and learning strategies in Norway have been widely debated. Norwegian students reported to use learning strategies less than the OECD mean, and researchers analysed the measurement to further investigate the reliability and validity of the results.

To give valid answers to the questionnaire, the students have to understand the items according to the test makers’ intention. The question raised in the present paper is concerned with whether students actually understand what we mean when they answer the questionnaire. The main research question in the present investigation is therefore:

• How can we know that the 15-year-olds responding to the PISA questionnaire have metacognitive knowledge of their own learning processes in such a way that they are able to give valid responses to the questionnaire?

 To answer the research question, data has been collected through a triangulation of questionnaires and interviews. The results indicate that low performing students did not give valid answers about their self-regulation, since they had a greater tendency to check off for one response category in the questionnaire. More research is needed to further understand self-regulation among these students. Cognitive interviews seem to be one way of contribute to better instruments.

Background

Even though concerns have been raised regarding self-report instruments such as the PISA student-questionnaire, self-report surveys still seem to predominate the educational research, psychology and social sciences in general. Research aimed at improving such measurements is therefore needed. Cognitive research methods have gained acceptance for improving questionnaires and survey research (Sudman, Bradburn et al. 1996). One of these methods, cognitive interviews, is described as a promising tool for improving surveys and self-reports (Karabenick et al. 2007). A call has been made for the improved rigor of educational research, policy analysis and evaluation (Desimone and Carlson 2004) and a significant response would be the use of cognitive interviews to increase the reliability of surveys and the construct validity of the specific items (Gorin 2006).

Use of cognitive interviews has provided insight into respondent errors, and this research has led to better questionnaires (Jobe and Mingay 1989). Some of these investigations have revealed how simpler wording can increase the comprehension of items, even in items were the terminology was judged as understandable for the respondents. Despite the importance of this research, cognitive interviews have been neglected in educational research. Both Desimone and Carlson (2004) and Karabenick et al. (2007) argue that this method should be given more attention as a way of improving survey research.

After the PISA results were released in 2000 and 2003, the results regarding the measurement of self-regulation and learning strategies in Norway have been widely debated. Norwegian students reported to use learning strategies less than the OECD mean, and researchers analysed the measurement to further investigate the reliability and validity of the results.

One critique has been the use of global measurement in the PISA questionnaire in 2000, which asked students to report their strategies across different tasks and subjects, and therefore do not capture what self-regulation is usually considered to be (Samuelstuen and Bråten 2007).The self-regulated student adapts strategy use to the specific context, and hence it is difficult to report strategies on a general level  (Hadwin, Winnie et al. 2001; Samuelstuen and Bråten 2007). To give valid answers to the questionnaire, the students have to understand the items according to the test makers’ intention. The question raised in the present paper is concerned with whether students actually understand what we mean when they answer questionnaire. The main research question in the present investigation is therefore:

• How can we know that the 15-year-olds responding to the PISA questionnaire have metacognitive knowledge of their own learning processes in such a way that they are able to give valid responses to the questionnaire?

Method

To answer the research question, data has been collected through a triangulation of questionnaires and interviews. Correlation analysis and regression analysis of 4770 students use of learning strategies and achievement scores on the PISA test has been conduced, as well as mean scores of different student groups, based  upon their achievement on the PISA test in 2006. In addition, 22 students from 5 different schools have been interviewed about their understanding of the questionnaire, their motivation for the PISA test and their perception of the assessment culture in the classroom. High-and low-achieving students on the PISA test have been compared and an analysis of the interviews has been conducted with special focus upon students’ ability to explain how they approach their learning, and how they prepare for tests.

Results

Analysis of the questionnaire data showed that the 20% weakest performing students on the PISA test, showed a less valid response pattern than the 20% highest performing students. One hypothesis is that low performing students will have more comprehension problems understanding the items – and hence a greater tendency to check off for just one answer. In the weakest performing group, 23,5% (184 out of 782) of the students checked off for the same response category. In the highest performing group, only 6,8% (62 of 918) of the students checked off for the same response category. The higher students performed on the PISA test, the more likely they were to check off for different response categories. The cognitive interviews showed a pattern whereby the poorer their results on the PISA test, the more problems they had with interpreting and explaining the questionnaire items. 

Discussion

The results indicate that low performing students did not give valid answers about their self-regulation. We still do not know enough about why low performing students have a greater tendency to check off for one response category, and more research is needed to further understand self-regulation among these students. Cognitive interviews seem to be one way of contribute to better instruments.

 References

Desimone, L. M. and K. L. F. Carlson (2004). "Are we Asking the Right Questions? Using Cognitive Interviews to Improve Surveys in Education Research." Educational Evaluation and Policy Analysis 26(1): 1 - 22.

Gorin, J. S. (2006). "Test Design with Cognition in Mind." Educational Measurements: Issues and Practice: 21 - 35.

Hadwin, A., Fiona, P. H. Winnie, et al. (2001). "Context Moderates Student's Self-Reports About How they Study." Journal of Educational Psychology 93(3): 477 - 487.

Jobe, J. B. and D. Mingay (1989). "Cognitive Research improves Questionnaires." American Journal of Public Health 79(8).

Samuelstuen, M. and I. Bråten (2007). "Examining the validity of self-reports on scales measuring students' strategic processing." British Journal of Educational Psychology 77: 351 - 378.

Sudman, s., N. Bradburn, et al. (1996). Thinking about answers:The application of cognitive processes to survey methodology. San Fransisco, Jossey-Bass.