While the need for multi-method approaches in research on self-regulated learning has already been claimed (e.g. Veenman, van Hout-Wolters & Afflerbach, 2006; Winne & Perry, 2000) this symposium will present and discuss approaches of that kind.

Based on two experimental studies (Glogger et al.; Marschner et al.), this symposium will present good reasons for using multi-method approaches in research on self-regulated learning. They range from methodical reasons such as validation of instruments to theoretical reasons such as revealing relationships between process and product measures of self-regulated learning.

Based on two review studies (Afflerbach, Cho & Veenman; Azevedo & Witherspoon), the symposium will discuss the specific benefits and risks of combining multiple methods in research on self-regulated learning. The discussion will range from theoretical and methodical aspects such as adequacy of operationalisation and advantages of triangulation of instruments to practical implications such as revealing promising approaches for support of self-regulated learning.

Finally, the symposium aims at encouraging the further use and development of multiple methods in research on self-regulated learning.

Learning with a hypermedia environment requires a student to regulate his or her learning; that is, to make decisions about what to learn, how to learn it, how much time to spend on it, how to access other instructional materials, and to determine whether he or she understands the material. Specifically, students need to analyze the learning situation, set meaningful learning goals, determine which strategies to use, assess whether the strategies are effective in meeting the learning goal(s), and evaluate their emerging understanding of the topic. They also need to monitor their understanding and modify their plans, goals, strategies, and effort in relation to changing contextual conditions (e.g., cognitive, motivational, and task conditions; Moos & Azevedo, in press; Pintrich, 2000; Winne, 2001; Winne & Hadwin, 2008; Zimmerman, 2006). However, most students experience some difficulty regulating their learning, which severely affects their learning about challenging topics (Azevedo, 2007; Hmelo-Silver & Azevedo, 2006; Jacobson, 2008; Schraw, in press; Veenman, 2007). In sum, we conceive of SRL as an event (see also Leelawong & Biswas, 2008; Witherspoon, Azevedo, & D’Mello, 2008; Winne & Perry, 2000) rather than as an aptitude. As such, the use multiple methods and analytical techniques are required to capture, trace, model, and infer the temporal deployment of SRL processes during hypermedia learning.

This presentation will focus on critically analyzing the benefits of mixed-methods approaches to examining the temporal deployment of self-regulatory processes during hypermedia learning. We will approach this task by presenting and discussing the strengths and weaknesses in mixed-methods studies by using data from several of our laboratory studies on learning about complex science topics with hypermedia. In a series of studies, students at different developmental levels and with little knowledge of the topic are randomly assigned either to a control or some other experimental condition.  Learners in the control condition regulated their own learning, while learners in the some other experimental condition (e.g., with a human tutor) who have access to a human tutor who facilitates their self-regulated learning. Across studies, we converged product (pretest-posttest declarative knowledge and qualitative shifts in students’ mental models) and process (think-aloud) data, and log-file data collected in real-time to examine the effectiveness of several experimental conditions during extended hypermedia learning tasks. Our presentation will present the advantage of using different types of quantitative analytical procedures (e.g., inferential tests, logistic regression) to analyze the various types of data. In addition, we will also discuss and present the computational methods from the field of AI that can be used to make inferences about the probability of certain SRL state transitions, assessment of essay collected during hypermedia learning to assess learners’ metal model shifts. Our presentation will also focus on the development of theoretically-driven and empirically-derived coding schemes to analyze the think-aloud protocols. For example, we will discuss issues related to levels of granularity, valence used to denote accuracy and feedback mechanisms is learners’ use of metacognitive monitoring and control processes, and qualitative and quantitative issues related to analyzing differences in the deployment of self-regulatory processes between experimental groups. Finally, we discuss implications for future research in SRL.

Azevedo, R. (2007). Understanding the complex nature of self-regulated learning processes in learning with computer-based learning environments: An introduction. Metacognition and Learning, 2(2/3), 57-65.

Hmelo-Silver, C. E., & Azevedo, R. (2006). Understanding complex systems: Some core challenges. Journal of the Learning Sciences, 15(1), 53-61.

Jacobson, M. J. (2008) A design framework for educational hypermedia systems: Theory, research, and learning emerging scientific conceptual perspectives. ETR&D, 56(1), 5-28.

Leelawong, K., & Biswas, G. (2008). Designing learning by teaching agents: The Betty's Brain system. International Journal of AI in Education, 18(3), 181-208.

Moos, D.C., & Azevedo, R. (in press). Learning with computer-based learning environments: A literature review of computer self-efficacy. Review of Educational Research.

Pintrich, P. R. (2000). The role of goal orientation in self-regulated learning. In M. Boekaerts, P. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 451–502). San Diego, CA: Academic Press.

Schraw, G. (in press). Measuring metacogniitve judgements. In D. Hacker, J. Dunlosky, & A. Graesser (Eds.), Handbook of metacognition and education.

Veenman, M. (2007). The assessment and instruction of self-regulation in computer-based environments: A discussion. Metacognition and Learning, 2, 177-183.

Witherspoon, A., Azevedo, R., & D’Mello, S. (2008). The dynamics of self-regulatory processes within self- and externally-regulated learning episodes. Proceedings of the International Conference on  Intelligent Tutoring Systems (pp. 260-269). Berlin: Springer.

Winne, P. H. (2001). Self-regulated learning viewed from models of information processing. In B. Zimmerman & D. Schunk (Eds.), Self-regulated learning and academic achievement: Theoretical perspectives (pp. 153–189). Mahwah, NJ: Erlbaum.

Winne, P., & Hadwin, A. (2008). The weave of motivation and self-regulated learning. In D. Schunk & B. Zimmerman (Eds.), Motivation and self-regulated learning: Theory, research, and applications (pp. 297–314). Mahwah, NJ: Erlbaum.

Winne, P., & Perry, N.E. (2000). Measuring self-regulated learning. In M. Boekaerts, P. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 531-566). San Diego, CA: Academic Press.

Zimmerman, B. (2006). Development and adaptation of expertise: The role of self-regulatory processes and beliefs. In K. Ericsson, N. Charness, P. Feltovich, & R. Hoffman (Eds.), The Cambridge handbook of expertise and expert performance (pp. 705–722). NY:  Cambridge University Press.

Aims of the study

The intention of this study is to review existing studies on self-regulation in reading on the Internet and to describe both the nature of self-regulation strategies and the means with which researchers gather and interpret data relevant to these strategies.  As well, the paper examines the methodologies used to inquire about self-regulation strategies, and focuses on the idea that think-aloud protocol data can be complemented by various data-gathering methods.

Methodology/research design

The paper synthesizes research conducted on Internet reading, and involved identification of studies that focus on Internet reading and self-regulation of that reading.  Twenty-nine studies were identified, published in the time frame of 1987-2008.   We conducted a grounded theory of self-regulation strategy use and Internet reading as reported in the 29 publications.  We used Pressley and Afflerbach’s (1995) strategy categories of Monitoring and Evaluating to initially identify self-regulation.  Following this initial identification, we built descriptions of common and idiosyncratic self-regulation strategies across the study results.  The synthesis allowed us to determine that successful reading on the Internet requires that readers use self-regulation strategies, some of which derive from reading traditional (i.e., single, printed) texts and others that appear to be unique to strategic Internet reading.    

Results

There are two sets of results.  First, synthesis of the research demonstrates that readers on the Internet engage in strategic behavior that has much in common with reading in more traditional texts.  Accomplished readers regularly set goals, monitor their progress towards these goals, detect problems with comprehending when they occur, determine a means to address the problems and then fix the problem (Pressley & Afflerbach, 1995).  Clearly different self-regulation strategies occur in what we describe as Internet-specific reading events (Azevedo, Guthrie, & Siebert, 2004).  These strategies address the need to self-regulate reading in relation to the diversity and volume of possible texts that may be encountered in Internet reading, the challenges to self-regulating in managing Internet reading which is often conducted in ill-defined problem spaces, exhibiting patience and self-control when confronted with “click throughs,” and re-orienting to particular texts when a promising path on the Internet is not helpful (e.g., Balcytiene, 1999; Kaakinen, Hyone, & Keenan, 2002; McNamara & Shapiro, 2005).

Second, the synthesis describes effective means to combine methods to develop multiple perspectives on Internet strategy use.  The reviewed literature demonstrates that researchers have used innovative approaches to complementing think-aloud protocol data on Internet reading.  Using combinations of data from verbal reports (Afflerbach, 2000), eye movements (Rayner et al, 2006), theoretical task analysis (Magliano & Graesser, 1991), self-reports (Segev-Miller, 2007), videotape and computer logs (Coiro & Dobler, 2007), researchers are able to generate rich inferences about Internet reading strategies, including self-regulation.  Researchers are able to combine data from multi-methods to triangulate their findings and gain confidence in the interpretations of data.

Theoretical and educational significance of the results

We believe the results of this analysis and synthesis are important for several reasons.  First, they offer the means to compare readers’ self-regulation strategies as they engage in Internet or traditional reading.  This contributes to the increased understanding of strategies that are common between the two, and strategies that are unique to Internet reading.  Second, as the world becomes increasingly online and Internet capable, more readers will be challenged to self-regulate in the sometimes vexing context of Internet reading.  Ideas for instruction that addresses the new strategy demands of Internet reading can be derived from this synthesis.  Third, as investigations of self-regulation Internet reading and other strategies continue, the results of this synthesis provide ideas for researchers who are seeking to construct experimental designs that provide triangulated strategy data.

References

Afflerbach, P. (2000). Verbal reports and protocol analysis. In M. Kamil, P. Mosenthal, P. Pearson, & R. Barr (Eds.), Handbook of reading research (Vol. 3, pp. 163-179). Mahwah, NJ: Lawrence Erlbaum Associates.

Azevedo, R., Guthrie, J. T., & Seibert, D. (2004). The role of self-regulated learning in fostering students’ conceptual understanding of complex systems with hypermedia. Journal of Educational Computing Research, 30, 87-111.

Balcytiene, A. (1999). Exploring individual processes of knowledge construction with hypertext. Instructional Science, 27, 303-328.

Coiro, J., & Dobler, B. (2007).  Exploring the online comprehension strategies

used by sixth-grade skilled readers to search for and locate information on

the Internet.  Reading Research Quarterly, 42, 214-257.

Kaakinen, J., Hyona, J., & Keenan, J. (2002). Perspective effects on online text processing. Discourse Processes, 33, 159-173.

Magliano, J., & Graesser, A. (1991). A three-pronged  method for studying inference generation in literacy text. Poetics, 20, 193-232.

McNamara, D., & Shapiro, A. (2005). Multimedia and hypermedia solutions for promoting metacognitive engagement, coherence, and learning. Journal of Educational Computing Research, 33, 1-29.

Pressley, M., & Afflerbach, P. (1995). Verbal protocols of reading: The nature of constructively responsive reading. Hillsdale, NJ: Lawrence Erlbaum Associates.

Rayner, K., Chace, K., Slattery, T., & Ashby, J. (2006). Eye movements as reflections of comprehension processes in reading. Scientific Studies of Reading, 10, 241-255.

As opposed to learning strategy questionnaires, learning journals and interviews can capture student’s cognitive and metacognitive learning strategies firstly ‘near to action’ and secondly they can assess not only the quantity, but also the quality of the applied strategies. Strong and stable correlations between learning strategy use in learning journals and learning outcomes in experimental studies indicate that learning journals could be applicable as assessment method for learning strategies. Interviews have already shown to be satisfactory valid. In the present study, we asked (1) if the quantity and quality of learning strategy use assessed by an interview and by a learning journal, respectively, are related to each other, (2) if both measurements can predict learning outcomes and (3) if the two types of assessment methods are "redundant" or "complementary" predictors of learning outcomes?

Ninth-grade students (N = 255) of a German medium-track high school wrote learning journals in mathematics as homework during six weeks. Two students of each classroom were interviewed about their use of cognitive and metacognitive learning strategies during class directly after a mathematics lesson, based on the lesson's learning materials (stimulated recall) (n = 18). Prior knowledge and learning outcomes were assessed.

To assess these learning strategies, learning journals or interviews can be used. As opposed to learning strategies questionnaires, these methods capture student’s learning strategies firstly ‘near to action’. Secondly, they can assess not only the quantity, but also the quality of the applied strategies.

A learning journal typically represents a written explication of one’s own learning processes and outcomes. In writing a learning protocol, students are asked to organize main ideas, explain concepts with concrete examples, and link new learning material to everyday life, that is, they should apply cognitive learning strategies during writing. In addition, students are asked to monitor their understanding and to formulate questions which, for example, they could ask in the next lesson to remediate comprehension difficulties (i.e., metacognitive learning strategies). Strong and stable correlations between learning strategy use in learning journals and learning outcomes in experimental studies (e.g., Schwonke, Hauser, Nückles, & Renkl, 2006) indicate that learning journals could be applicable as assessment method for learning strategies. However, only first steps were taken in testing the validity of this method (Boekarts & Corno, 2005).

Interviews can ask the application of the same learning strategies and, in contrast, have shown to be satisfactory valid (Zimmerman & Martinez-Pons, 1986). Certainly, interviews and learning journals refer to different learning and assessment situations: the interview is oral and refers in this study to the previous mathematics lesson; the learning journal is individually written as homework about the learning content of previous week. Against this background, we addressed the following research questions: 1. Are the quantity and quality of learning strategy use assessed by an interview and by a learning journal, respectively, related to each other? 2. Can the application of learning strategies measured by learning journals and by interviews predict learning outcomes? 3. Are the two types of strategy assessment methods "redundant" or "complementary" predictors of learning outcomes?

Methods

To address these research questions 255 ninth-grade students from 10 classrooms of a German medium-track high school wrote learning journals in mathematics (topic: probability). Two students of each classroom were interviewed about their cognitive and metacognitive learning strategies during class directly after a mathematics lesson, based on the lesson's learning materials (stimulated recall) (n = 18).

In an introductory lesson, we conducted a topic-specific pretest and we informed students on why they should use elaboration and organization (i.e., cognitive) and metacognitive learning strategies and we showed how these strategies can be realized in writing learning journals by cognitive modelling. Students wrote learning journals weekly as homework during six weeks. In the fifth week, the interviews were conducted. In the end of the six weeks, learning outcomes were assessed.

Learning journals and interviews were coded and rated. The number of coded segments provided a measure of the quantity of learning strategies. Ratings (six-point-scale) of the best elaborative, organizational, and metacognitive strategy in each journal were taken as a measure of quality.

Results.

Correlations of the two methods, learning journal and interview, were small (between r = .07 and r = .20), except for the quantity of metacognitive strategies (r = .46, p = .05). For the quantity of learning strategies in learning journals, we found significant partial correlations with the posttest, controlled for the pretest (organization: r = .28, p < .01; elaboration: r = .36, p < .01; metacognition: r = .18, p = .01). For the quality of learning strategies we found a similar pattern (organization: r = .31, p < .01; elaboration: r = .24, p < .01; metacognition r = .13, p = .07). For the quantity of learning strategies in interviews, partial correlations with the posttest, controlled for the pretest, did not reach significance (organization: r = .40, p = .11; elaboration: r = .40, p = .11; metacognition: r = .44, p = .07). For the quality of learning strategies we found a significant partial correlation of elaboration with the posttest (organization: r = .39, p = .17; elaboration: r = .61, p = .01; metacognition r = .09, p = .75). A multiple regression with interview, learning journal, and posttest revealed that the quantity of elaboration in the learning journal as well as in the interviews explain a significant part of variance of the posttest (learning journal: b = 0.08, p = .01; interview: b = 0.18, p = .04; R2 = .50). The multiple regression with organization also reveals significant results (learning journals: b = 0.06, p = .02; interview: b = 0.34, p = .06; R2 = .46).

Discussion.

In summary, the two measurement methods independently explain a significant amount of variance of the posttest. They are "complementary" predictors of learning outcomes and obviously measure different aspects of learning strategy behaviour. Thus, the necessity of a multi-method approach to measure self-regulated learning is underlined (Boekarts, 2005). As the methods refer to different learning situations, our results also suggest that learning strategy assessments should not be or just cautiously generalized to different learning situations.

       On the whole, the present pattern of results suggests that learning journals as used in this study can offer an important complementary perspective when looking at learning strategy use. In comparison to interviews, learning journals can be more easily used by classrooms teachers or in field studies and when observing learning strategy use over a longer period of time.

References

Boekarts, M. & Corno, L. (2005). Self-Regulation in the Classroom: A Perspective on Assessment and Intervention. Applied Psychology: An International Review, 54, 199–231.

Schwonke, R., Hauser, S., Nückles, M., & Renkl, A. (2006). Enhancing computer-supported writing of learning protocols by adaptive prompts. Computers in Human Behavior, 22, 77-92.

Appropriate strategies for scientific discovery learning situations are described within the Scientific Discovery as Dual Search (SDDS) model (Klahr & Dunbar, 1988). It suggests that strategies should not only be applied in the “hypothesis space” or in the “experiment space”. Rather, the interaction between these two spaces is supposed to be crucial for learning. In order to test this assumption we investigated whether this interaction has a stronger influence on learning outcome than applying strategies in merely one of the spaces. Additionally, we examined which learner characteristics predict this interaction. 86 students participated in our study. We used four different measures for strategy knowledge and strategy use: a strategy knowledge test and behavior-based logfile data for the conduction of unsystematic experiments, experiments with control of variable strategy and the interaction between the spaces. Before and after learning, content knowledge was assessed. Results revealed that the systematic interaction between the two spaces had a stronger impact on learning than the systematic or unsystematic use of the experiment space, only. In addition, content knowledge and strategy knowledge both positively influenced the interaction between spaces. From a theoretical point of view, the core assumption of the SDDS model could be confirmed. Furthermore, our study shows that it is necessary to assess different aspects of self-regulated learning with appropriate instruments. From a practical point of view, the results can be used to design adaptive methods to support self-regulated scientific discovery learning and thereby learning outcome.

Aims of the study

The use of appropriate learning strategies is crucial in all kinds of learning situations (e.g. Pressley, Borkowski & Schneider, 1987). Appropriate strategies for scientific discovery learning situations are described within the Scientific Discovery as Dual Search (SDDS) model by Klahr and Dunbar (1988). It suggests that cognitive strategies can be applied in two “spaces”: In a “hypothesis space”, which comprises all rules describing the phenomenon that can be observed within a certain domain and an “experiment space”, which comprises all experiments within a domain. According to the SDDS model, it is crucial that strategies are not only executed in one of the spaces but also interact with and influence each other in order to learn successfully. However, to our knowledge, there is no empirical evidence supporting this assumption, yet. Most studies only investigated parts of the model by focusing on single strategies of merely one of the spaces (e.g. Chen & Klahr, 1999; Njoo & de Jong, 1993). So far, the impact of the interaction between two spaces on learning has not been investigated. This is probably due to missing methods assessing the interaction between the two spaces. Thus, we developed a measure for the interaction between spaces and investigated whether it has a stronger influence on learning outcome than applying strategies in one of the spaces, only.

As the interaction between spaces should be the key to learning outcome, we additionally examined which learner characteristics predict the interaction between spaces. We expected that both, content knowledge as well as strategy knowledge are needed in order to generate systematic hypotheses and test them by systematic experiments.

Methodology and research design

A sample of 86 German high school students from eighth to ninth grade took part in this study (mean age: 14.11 years (SD = 0.63); 53.5 % male, 46.5% female). All students learned self-regulated within a computer-based scientific discovery learning environment on ‘buoyancy in fluids’. The learning environment consisted of a hypothesis space, in which hypotheses could be charted, and an experiment space, in which simulated experiments could be conducted. Students had to find out as much as possible about ‘buoyancy in fluids’. During the 15-minute-phase of self-regulated learning all mouse-clicks a learner made were automatically registered and written into logfiles.

We used four different measures for strategy knowledge and strategy use. (1) Before learning, all students had to fill in one of two versions of a newly developed strategy knowledge test. The strategy knowledge test included tasks referring to generating systematic hypotheses, conducting systematic experiments (control-of-variables strategy; CVS), and drawing conclusions. Each task consisted of a concrete description of an experimental situation and three possible experimental procedures. Each item consisted of a pair of two of the three procedures. In a forced-choice format learners had to choose which of the two procedures they favored. The test score was build by comparing the individual learner’s rating with an aggregated expert rating.

Based on the behavioral log-file data, measures of strategy use were calculated. They included (2) a measure for conducting unsystematic experiments (unsystematic use of the experiment space), (3) for conducting experiments with CVS (systematic use of the experiment space) and (4) for the interaction between spaces, i.e. for conducting experiments with CVS according to stated hypotheses.

Additionally, content knowledge (pre and post), interest and demographical data were assessed.

Results

First, we examined whether working with the learning environment led to a gain in content knowledge. Results indicate a significant difference between pre and post content knowledge tests (t(82) = -2.38, p < .05, d = 0,25).

Next, we tested the assumptions of the SDDS-model. As expected, a path analysis (see Figure 1) showed that the systematic interaction between the two spaces had a stronger impact on learning than the systematic or unsystematic use of the experiment space, only.

Finally, we tested which learner characteristics best predict the interaction between spaces. Analyses are based on a reduced sample size (n = 40), because only one version of the strategy knowledge test proved to be reliable. A path analysis (see Figure 2) showed that content knowledge and strategy knowledge both positively influence the interaction between spaces. However, results do not reach statistical significance due to a reduced sample size.

Theoretical and educational significance

Until now, most studies on scientific discovery learning have analyzed isolated strategies rather than the core assumption of the SDDS model (e.g. Chen & Klahr, 1999). Results of our study confirm this core assumption. Furthermore, our study shows that it is necessary to assess different aspects of self-regulated learning, such as knowledge about strategies and use of strategies with appropriate instruments. From a practical point of view, the results on relevant learner characteristics can be used to design adaptive methods to support self-regulated scientific discovery learning and thereby learning outcome.

References

Chen, Z. & Klahr, D. (1999). All other things being equal: Acquisition and transfer of the control variable strategy. Child Development, 70, 1098-1120.

De Jong, T. & van Joolingen, W.R. (1998). Scientific discovery learning with computer simulations of conceptual domains. Review of Educational Research, 68, 179-201.

Klahr, D. & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1-48.

 Njoo, M. & de Jong, T. (1993). Supporting exploratory learning by offering structured overviews of hypotheses. In  D.M. Towne, T. de Jong & H. Spada (Eds.), Simulation-based experiential learning (pp. 207-223). Berlin: Springer.

Pressley, M., Borkowski, J.G., & Schneider, W. (1987). Cognitive strategies: Good strategy users coordinate metacognition and knowledge. In R. Vasta, & G. Whilehurst (Eds.), Annals of child development, 4, 80-129. Greenwich, CT: JAI Press.