Construct validity research is necessary to clarify research on self-regulation and provide a stronger basis for future research. We used multitrait-multimethod (MTMM) analysis to assess convergent and discriminant validity of three self-report self-regulation measures: the Learning and Study Strategies Inventory (LASSI; Weinstein, 1987), the Motivated Strategies for Learning Questionnaire (MSLQ; Pintrich, Smith, Garcia, & McKeachie, 1993), and the Metacognitive Awareness Inventory (MAI; Schraw & Dennison, 1994). Method bias across all inventories also was examined. Three hundred eighteen undergraduate students were recruited to participate in research on perceptions about studying and study methods. Participants spent 30-60 minutes completing all three inventories. Evidence for convergent validity was found at the matrix level but was attenuated at the individual parameter level. Evidence for discriminant validity among traits was modest. Common method bias was evident across all measures. Results revealed these inventories yielded different results. Accordingly, researchers should be selective in the inventory they use to assess self-regulated learning.

      Research investigating self-regulated learning is prominent in educational psychology. Self-regulated learners are more strategic and perform better than less self-regulated learners (Zimmerman, 1990). Theoretically, SRL can be defined as an aptitude (e.g., Snow, 1996) and as an event (e.g., Winne, 2001). As an aptitude, SRL is considered to be relatively trait-like. From this view, learners have relatively stable and contextualized approaches to selecting learning strategies. For SRL as an event, learners regulate learning “on-the-fly” (Winne, Jamieson-Noel, & Muis, 2002) by metacognitively monitoring features of their engagement and adjusting learning strategies as work on a task unfolds.

Generally, SRL is measured as an aptitude using self-report inventories (Winne & Perry, 2000) wherein students rate statements about various learning strategies. Three widely used self-report measures are the Learning and Study Strategies Inventory (LASSI) (Weinstein, 1987), the Metacognitive Awareness Inventory (MAI; Schraw & Dennison, 1994), and the Motivated Strategies for Learning Questionnaire (MSLQ) (Pintrich, Smith, Garcia, & McKeachie, 1993).

In light of widespread use of these inventories, it is important to inform the field about their psychometric qualities. Because operational definitions and theoretical frameworks that generate measures of SRL can substantially affect its relations with other constructs, Winne et al. (2002) questioned: What influences do response formats or other methodological features such as instructions have on measuring SRL? They called for multitrait-multimethod (MTMM) studies to contribute to answering these questions (Winne et al., 2002). We respond to this call by comparing these three frequently used inventories within the MTMM tradition.

Prior to analyzing data, we examined each instrument to identify similarities in theoretically defined constructs. Based on our analysis, we chose four subscales on each instrument that represented the same theoretical constructs. Accordingly, our MTMM model includes four traits and three methods.

      Undergraduate students (N = 318) participated. Participants spent 60 minutes completing the LASSI, MAI, and MSLQ for a typical course.

We applied three methods to examine convergent validity, discriminant validity, and method bias: Campbell and Fiske’s (1959) traditional correlation matrix method, matrix-level CFAs, and individual parameter estimates. Due to space limits, we report the first MTMM analysis only.

Our MTMM model has four factors—Main Ideas/Organization, Elaboration, Self-Regulation, and Evaluation—and three methods: a 5-point Likert LASSI scale, a 7-point Likert MSLQ scale, and 100-point MAI scale. Convergent validity is supported when correlations across instruments for similar constructs are moderate to high; discriminant validity is supported when correlations between different constructs within an instrument are low. Using Campbell and Fiske’s (1959) criteria, evidence of convergent validity among corresponding subscales across the LASSI, MSLQ and the MAI was modest. Correlations among the Main Ideas/Organization subscales across the inventories were low, .29 to .37. Correlations across the inventories for Elaboration were moderate, .54 to .60. Self-Regulation-related subscale correlations across the inventories were low to moderate, .27 to .55. Finally, subscales for Evaluation had moderate correlations, .39 to .50.

Evidence of discriminant validity within inventories was weak to modest. The MAI evidenced the highest correlations across the traits, .51 to .70. Correlations across traits in the MSLQ were lower, .20 to .52. Finally, correlations across the LASSI were lowest, .04 to .44. The LASSI provided greatest evidence of discriminant validity.

To test method bias, correlations are averaged over traits within instruments. High values signal method bias. Averages for the three measures were .29 for the LASSI, .43 for the MSLQ, and .63 for the MAI. The LASSI exhibits little method bias. 

Discussion

      Researchers must wrestle with variations in methods used to measure SRL when they synthesize results of research and then design studies to advance theory about SRL. Among studies that use self-reports about SRL, the MAI, MSLQ, and LASSI instruments are prevalent. Each inventory, however, was developed from modestly different theoretical standpoints. The developer(s) of these inventories also operationalized different instructions and differing response formats to elicit respondents’ self-reports about elements of SRL. Our multitrait-multimethod study informs researchers about the comparability of data generated when using different inventories.

Based on our analyses, we argue it is imprudent to assume convergent validity within facets we identified as common across these inventories. Lack of strong convergence is not due to variation among the inventories’ methods, i.e., the instructions and response formats that are unique to each instrument. Instead, data generated when learners answer items on these inventories are undifferentiated by the variations across the inventories’ instructions and response formats. Our results raise questions about the dependability of prior reviews of research wherein convergence of constructs across these inventories was assumed. On a positive note, this calls for more research that can illuminate the convergence and divergence among subscales on these inventories and across other operational definitions of SRL, such as traces of actual learning activities. This aligns with recommendations made by Winne et al. (2002) that, over time, will advance the field.

Campbell, D., & Fiske, D. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.

Pintrich, P., Smith, D., Garcia, T., & McKeachie, W. (1993). A manual for the use of the motivated strategies for learning questionnaire. MI: U. Michigan.

Schraw, G., & Dennison, R. (1994). Assessing metacognitive awareness. Contemporary Educational Psychology, 19, 460-475.

Snow, R. (1996). Self-regulation as meta-conation? Learning and Individual Differences, 8, 261-267.

Weinstein, C. (1987). LASSI User’s Manual. FL: H&H.

Winne, P. (2001). Self-regulated learning viewed from models of information processing. In B. Zimmerman and D. Schunk (Eds.), Self-regulated learning and academic achievement: Theoretical perspectives (2^nd ed, pp. 153-189). Hillsdale: LEA.

Winne, P., Jamieson-Noel, D, & Muis, K. (2002). Methodological issues and advances in researching tactics, strategies, and self-regulated learning. In P. R. Pintrich & M. L. Maehr (Eds.), Advances in motivation and achievement: New directions in measures and methods (Vol. 12, pp. 121-155). CT: JAI Press.

Winne, P. H, & Perry, N. (2000). Measuring self-regulated learning. In M. Boekaerts, P. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 531-566). FL: Academic Press.

Zimmerman. B. (1990). Self-regulated learning: An overview. Educational Psychologist, 25, 3-18.

 

In this study, 216 education undergraduates were administered a 15-item task-specific strategy-use inventory immediately following their reading of seven texts representing partly conflicting views about climate change. The inventory included items focusing on memorizing as much information as possible from the texts and items focusing on cross-text elaboration in the form of integrating, comparing, and contrasting information across texts. Maximum likelihood exploratory factor analysis with oblique rotation using this sample identified two factors, with all items written to assess cross-text elaboration loading on the first factor and all items written to assess memorization loading on the second factor. In a sample of 68 education undergraduates, who were also administered a measure of topic knowledge about climate change before reading and a measure of intertextual comprehension after completing the strategy inventory, we found that topic knowledge was positively correlated with self-reported elaboration and negatively correlated with self-reported memorization. Moreover, self-reported elaboration was positively and self-reported memorization negatively correlated with intertextual comprehension. Hierarchical regression analysis showed that scores on the self-report strategy measures contributed to intertextual comprehension over and above topic knowledge, with self-reported elaboration being a positive and self-reported memorization a negative predictor of comprehension. Finally, in a sample of 19 undergraduates who also took notes while studying the texts, self-reported cross-text elaboration was positively correlated with traced cross-text elaboration and negatively correlated with traced paraphrasing. In contrast, self-reported memorization was negatively correlated with traced cross-text elaboration and positively correlated with traced paraphrasing. Given that our knowledge of the strategic competence required to build integrated representations of controversial topics through the reading of multiple information sources is rather limited and mainly based on very labor-intensive think-aloud methodologies, less suitable for large scale investigations, our preliminary validation data are promising and speak for continued work on this instrument.

 

            The aim was to design and validate a task-specific self-report measure of strategic processing during the reading of multiple texts.

Introduction

Most studies of reading comprehension strategy use have employed self-report questionnaires. Typically, students answer on a Likert-type scale how frequently they use each strategy (e.g., Mokhtari & Reichard, 2002). In addition to self-reporting frequency of use, some questionnaires present scenarios and ask students which strategy they would use (e.g., Schmitt, 1990). However, a number of authors have challenged the validity of these types of measures Baker & Cerro, 2000). Data suggest that students are not accurate at reporting the strategies they “usually use” or “just used,” but are relatively accurate at reporting the strategies they are currently using.

Several alternatives are available, all of which require the student to actually enact strategies while reading. In addition to error detection, collecting computer traces or paper-and-pencil artifacts, think-aloud protocols, or eye tracking data, another approach is to ask students to enact a strategy, and then measure whether they have done this accurately by posing multiple-choice questions (Kozminsky & Kozminsky, 2001).

 

Method

In the three studies presented below, students answered paper-and-pencil, multiple choice measures during a single session of approximately 90 min.

 

In study 1 (Cromley & Azevedo, 2007), we created a domain-general strategy use measure, as well as measures of inference and word reading fluency, and administered these together with standardized reading comprehension and vocabulary items.

 Participants. Participants were 177 ninth-grade students, selected to be at a wide range of reading comprehension proficiency. [Details omitted for reasons of space.]

Measures. Measures were identified or developed for each of the five predictor variables and for reading comprehension. 1) The background knowledge measure was a 13-item multiple-choice measure developed by the researchers based on the content of the Gates-MacGinitie comprehension subtest (MacGinitie, MacGinitie, Marie & Dreyer, 2001). 2) The strategy measure was a 10-item multiple-choice measure developed by the researchers based on Kozminsky and Kozminsky (2001). We chose neighboring passages from original material used in the Gates-MacGinitie and constructed questions that asked students to apply the strategies of hypothesizing, inference, prior knowledge activation, self-questioning, summarizing, and text search to the passages. The vocabulary measure consisted of the 23 odd-numbered questions from the Gates-MacGinitie 7/9 Form S vocabulary subtest. The reading comprehension measure was the comprehension subtest of the Gates-MacGinitie, consisting of 48 items. The inference measure was a 10-item multiple-choice measure developed by the researchers based on the content of the Gates comprehension subtest, using new paragraphs from the same texts as the Gates. The word reading measure was a principal components composite of Woodcock Letter Word Identification and Word Attack (Woodcock, 1997) and a one-minute timed curriculum-based passage reading fluency task.

In study 2 (Cromley & Snyder, 2007), we created an analogous set of domain-general measures.

Participants.  Participants were 185 students from a large public university in a large mid-Atlantic US city, recruited from the College of Education. [Details omitted for reasons of space.]

Materials. Reading comprehension and component measures included background knowledge, vocabulary, reading comprehension, inference, strategies, and word reading. [Details omitted for reasons of space.]

Strategy use measure. A researcher-developed 10-item, test was developed based on Kozminsky and Kozminsky (2001) following the same approach as in Study 1 and to measure the same strategies.

 

In study 3 (Cromley, Snyder, Luciw, & Tanaka, 2008), we created an analogous set of domain-specific measures.

Participants. The 737 participants were current or prospective biology majors enrolled in a first-semester introductory biology course at a large, moderately-selective urban university in the Mid-Atlantic region of the US. [Details omitted for reasons of space.]

Measures. We developed six measures: one for each of the five predictor variables and one for reading comprehension. [Details omitted for reasons of space.]

The strategy use measure was a 13-item multiple-choice measure based on Kozminsky and Kozminsky (2001) following the same approach as in Studies 1 and 2 and to measure the same strategies.

 

Results

            For reasons of space, we report in Table 1 only correlations between the strategy use measure and each of the other measures.

 

Table 1

Note: all Sig at p < .05; different superscripts represent significant differences across studies.

 

            The correlations suggest good concurrent validity for the measures. The pattern across age groups is stronger correlations of strategy use to the other variables for the high school sample, and weaker correlations for the two undergraduate samples. With regard to domain-specificity, correlations with inference, vocabulary, and word reading are parallel for the samples answering domain-general questions, and the patterns in these correlations diverge for the biology students.

 

Discussion

            We created a series of new measures of strategy use based on work by Kozminsky and Kozminsky (2001), measures which show better concurrent validity than the most-frequently-used self-report instruments. They are relatively easy to administer, in that they do not require individual administration nor do they require expert scoring.

            Theoretically, these results suggest the importance of all of the components for comprehension—accurate strategy use is related to comprehension, but it is also related to vocabulary, inference, and background knowledge; these also have strong relationships to comprehension. The trends suggest that strategy use is more strongly related to the component measures for high school students than for undergraduate students with either type of text. However, this could be due to differences between the student groups or the measures: first, undergraduate students self-select to attend university and have high achievement. Second, we have not worked with large enough samples to equate measures or create scale scores.

We believe that for large-scale administration requiring efficiencies of group administration and multiple-choice format, this approach has many advantages. Multiple-choice measures do not capture the richness of comprehension processes in real time, so we are also continuing to collect think-aloud and other process data for smaller-scale studies.