Research on competencies gains more and more attention in the fields of vocational, occupational and professional education and training (VET). As the problem of an effective and efficient transition from school to work is central for educational policy of the EU – but also for other industrialized countries too – questions of how to develop VET and to measure its results have to be answered. The Symposium deals with important aspects of such research. The topic is also actually important as al large-scale assessment study on VET is in preparation on the European level.

The contributions to this Symposium shall exemplarily help to demonstrate the width and depth of research on VET competencies. Two presentations are dedicated to more content-specific themes: measuring the quality of teaching competencies and assessing the competence development of the learners at the workplace. In both cases different learning places (school; worksite) are given as organizational conditions for the development of competencies. The other two deal with research methodology. Firstly, the development of adequate tasks which form the basis of test items for an international measurement of competencies. Secondly, possibilities of an external measurement of competencies, as performance-driven approach, are discussed and presented with hints to necessary statistical operations.

All four presentations give decisive marks and operationalizations for the construction of a PISA-VET (cf. Baethge et al., 2006). They all are of high scientific relevance as they especially demonstrate which different and sophisticated procedures are urged to measure validly and reliably VET competencies of all individuals involved – also with regard to international comparisons.

Literature

Baethge, M., Achtenhagen, F., Arends, L., Babic, E., Baethge-Kinsky, V., & Weber , S. (2006). PISA-VET. Stuttgart: Steiner.

Is it possible to judge indirectly on the quality of single teacher competences, thus using the own judgement on this respective teacher? Such a method we call advocatory in the sense that we use the judgement of the judger as indicator for valuing in a certain way the quality of his own action system. Even if so far no clear agreement about a sole way of measurement has been found we believe that this way could be seen a break through for quality generalizability. We will on the one hand present an overview on trials and arrors in measuring teaching quality and on the other hand we will discuss whether a bottom-up procedure is contradictory to the assessment of quality in vocational education. We will show that the bottom-up way and the inclusion of the vocational teachers themselves into the process of competence profile development is a precondition for a new discourse about quality. The current project is part of a 12-year reseach program, called "Professional Minds" in oder to enhance the quality of instruction by using tailored film vignetts on comptence profiles in VET classrooms. We believe that the quality question cannot be discussed separately of the teaching pracitce and the emergency situation (vocational) teachers are confronted with. Therefore, a multiple perspective is necessary. We will present our stepwise approach to the measurement of quality by means of a Delphi-Study, diagnostic instruments, model testing, diagnosing quality by means of video vignettes and expert ratings as well as intervention studies.  

(1) Aim and research procedure

We will present a stepwise and innovative approximation to the measurement of teaching quality. The final goal will be a new concept called “advocatory approach”. In order to make the measurement of quality possible and valuable, contents have to be determined by means of the definition of profession-specific acting situations with high validity and the definition of important dimensions in order to comprehend the concrete acting. Thus, findings of differences in regard to the quality of teacher acting have to be possible. As Hascher and Thonhauser (2004) emphasize, lacking teaching competencies prevent from successful performance and are a typical catalyst for more demands, stress and finally burnout syndromes. Teaching quality cannot be defined without knowing and understanding the emergency room a teacher is daily confronted with. Therefore, we worked bottom-up oriented together with vocational teachers by means of a Quasi-Delphi-Study. The Quasi-Delphi-Study contained four phases in total. In a first step, relevant action situations and possible reaction patterns of vocational teachers were defined from bottom-up in collaboration with the actors. During the second meeting the collected situations were evaluated in their relevance and they were supplemented by teaching students. Consequently, also the perspective of the novices was included. On the basis of these collected situations and the evaluation, 45 competence profiles required for the teaching profession were developed. At the end, a representative sample (n=853) validated them on the following dimensions by means of a four scale questionnaire (see Oser et al., 2006):

• Importance: “how important is the described competence profile for your class at the vocational school?”
• Frequency of use: “how frequently do you act on the described competence profile at the vocational school?”
• Difficulty of implementation in class: “how difficult is the implementation of the described competence profile in the vocational school?”

·        Desired/expected value place in the education: “what place value should have the described competence profile within vocational teacher education?”

(2) Some results of the Delphi-Study and the validation of the diagnostic instrument

As a surprise all of the 45 competence profiles were rated as rather to be very important and they also seem to be rather or frequently used in the teaching profession. Some of the competence profiles are seen as more difficult to implement as others but all should be known and trained during teachers’ education. After this first analysis and some additional invest­tigations on the felt necessity (‘needed’ teaching competence profiles within vocational school education), the measurement of the quality of the competencies was tried to be realised. Because of the disadvantages of the mostly used competence evaluation methods: a) self evaluation and b) external evaluation, a new approach – the so-called advocatory approach – was developed. By the advocatory approach it is assumed, that competence measurement becomes possible, if a person A judges the teaching behaviour of a person B - observed by means of a three-dimensional film vignette – with respect to different quality indicators, and then the judgment of person A is used as an inference measure to his/her own cognitive competence structure. If the quality-specific dimensions are theoretically elaborated for each competence profile then we can use some cross-standard dimensions that are understood as quality aspects of teaching and instruction in general. These quality cross-standard dimensions are f.i. empathy, commitment/motiva­tion, task difficulty and flexibility, autonomy/control, differentiation and complexity as well as vision and task significance (Darling-Hammond, 2005, Helmke 2003). The respective theoretically developed models for each competence profile with standard-specific dimen­sions and cross-standard dimensions have been tested empirically by means of confirmatory factor analyses as well as higher order factor analyses.

An important aspect of quality measurement consists of substantial differences with respect to target groups. With our diagnostic instrument we could visualize that teachers and non-teachers significantly differ in their evaluation of the observed teaching behaviour in (almost) all dimensions. These differences are an indicator to a teacher professionalism-based knowledge. Still, the question about quality is not answered. The obtained differences do not state anything about the direction of quality. Fixing quality norms in teaching is neither easy nor uniformly traded. Different possibilities and ways are conceivable. In our case, due to the film-vignettes and the advocatory approach working with experts is one adequate way. Thus, for each film-vignette expert-ratings are realized. Because the competence measurement is related to the observation of complex and concrete situations, it is not possible to generalize the experts’ comments, but they count as substantial indicator of existing or missing quality. During each expert-rating, different experts are discussing the items on the different dimensions in respect to the same observed film-vignette as the participants did. For each competence profile a different film vignette exists. The experts discuss as long as a consensus is reached. Significant differences between the judgment of the non-teachers, teachers and experts have been found. More important than the exact number is the qualitative statement of the experts. Finally, we not only have a qualitative setting but also a qualitative argumentation and reflexive explication. The quantitative setting gives the quality reference and the direction of competence development. The qualitative statements have the function of stimulating the competence development.

(3) Conclusion

In this presentation we present several aspects of the advocatory approach to measuring the quality of teacher competences. We will elicit the methods we used and the results we found. Special concern will be also given to possible structural modelling of the quality indicators and to correlational measures with respect to specific and general criteria judgments.

Literature

Darling-Hammond, L. & Bransford, J. (Hrsg) (2005). Preparing Teachers for a Changing World. San Francisco: Jossey-Bass, S.232-274..

Helmke, A. (2003). Unterrichtsqualität – erfassen, bewerten, verbessern. Seelze: Kallmeyersche Verlagsbuchhandlung.

Hascher, T. & Thonhauser, J. (2004). Die Entwicklung von Kompetenzen beurteilen. Journal für Lehrerinnen- und lehrerbildung, 1, S. 4-9.

Oser, F. (2000). Emergency Room Schule: Erschwerende Rahmenbedingungen pädagogischer Professionalität. Beiträge zur Lehrerbildung, 18 (1), 82 - 84.

Background and aims

Recent developments in working life have challenged not only work organisations but also educational institutions. For example, Finnish vocational education and training (VET) system which has traditionally been school-centred underwent remarkable transformations introduced workplace learning periods to all vocational qualifications. Thus, Finnish vocational students acquire at least one-sixth of their vocational qualification outside of vocational schools, in authentic workplace. This offers students a possibility to learn those skills and knowledge that are needed in real working life.

There are studies on the dualism of theoretical school learning Eraut 2004b; Le Maistre and Paré 2006). Griffiths and Guile (2003) have presented a connective model of work experience. In this connective model, schools and workplaces together create learning environments and develop work practices, and students are encouraged to critically reflect workplace practices and their personal experiences with the help of conceptual and theoretical tools provided by school learning. The model of integrative pedagogy by Tynjälä (2008; Tynjälä et al., 2006) suggests that the connective model can be realised by integrating theoretical, practical and self-re­gulative knowledge in students’ learning environments. In the present study, we will examine the recently reformed Finnish model of organising workplace learning for vocational students: (1) what kind of skills and knowledge do students learn at work (i.e. during their workplace learning periods)? and (2) how students and their teachers and workplace trainers see the above-mentioned integration between school learning and workplace learning?. In addition, we will also pay attention to the differences between vocational fields.

Data and methods

The data from the students (N = 1824) and teachers (N = 796) was collected with Internet questionnaires. In total, 1071 students (59 %) and 330 teachers (42 %) answered the questionnaires. A sample (n = 800) was taken from the population of workplace trainers (N = 2484). The sample was weighted according to the relative amount of the students by regions and subfields. The workplace trainers were administered an ordinary hard copy questionnaire, and in total, 53 % of them (n = 420) answered it. When we compare the results between different vocational fields, we use the data from the four biggest fields of Finnish VET (1) technology and transport, 2) social services and health care, 3) commerce and administration, and 4) tourism, catering and domestic services. The data were analysed using quantitative methods (i.e. factor analysis for forming aggregate scales, comparison of mean values, analyses of variance ANOVA).

Results and conclusions

Vocational students reported that they had acquired a lot of different knowledge and skills during their workplace learning periods. The students felt that in particular they had learnt skills that involved independence (an aggregated scale with Cronbach’s alpha 0.83; mean value 4.04 on a 5-point scale). The independence scale included variables such as self-confidence, working independently, and initiative. The students also reported that they had learnt collaboration skills (alpha 0.83; mean 3.87), learning skills (alpha 0.73; mean 3.87), vocational skills (alpha 0.83; mean 3.86), self-assessment skills (single variable, mean 3.70), thinking skills (single variable, mean 3.23) and communications skills (alpha 0.76; mean 3.07). In addition to these learning outcomes that were in line with the aims of workplace learning, the students reported that they had also learned some things at work that were undesirable, such as bad practices, shirking duties, and disadvantages of the field. The aggregated scale including these things was named as negative learning outcomes (alfa 0.74; mean 2.13). There were significant differences between vocational fields in self-reported learning outcomes.

The connection and integration of school learning and workplace learning was seen differently by different actors. The findings showed that the teachers and the workplace trainers saw this integration much stronger than students (p<0.001). Also the views of the integration of school learning and workplace learning varied significantly between different vocational fields. According to the students, school learning and workplace learning were most connected in the field of social services and health care, second most in the field of tourism, catering and domestic services and the least in the fields of technology and transport and commerce and administration (p<0.001). Among teachers, these assessments were parallel with students’ assessments (p<0.001). According to the workplace trainers, similarly, the integration between school learning and workplace learning was strongest in the field of social service and health care. The second highest assessments of the integration of school learning and workplace learning gave the workplace trainers from the fields of commerce and administration and technology and transport and the lowest the workplace trainers from the field of tourism, catering and domestic services (p=0.002).

(1) Objectives

The contexts for VET differ markedly: VET operate in multiple ways, even in the same sector or country. The content of learning contains job-specific skills and key compentences to handle job-specific tasks. On these perspectives an important feature of VET is how it is designed to offer equality of opportunity for learning to students in different personal circumstances (e.g. Leney & Green, 2005). In this way it sounds up-and-coming for judging effective VET at an international level to focus on the needs of specific users particularly with regard to specific occupational profiles and corresponding tasks. In our opinion that is a promising option to describe and compare modern VET programmes in terms of competences associated with professional behaviour. This shall be done by developing authentic vocational tasks as preparation for a VET-PISA-Study.

(2) Theoretical Framework

The first challenge of a large-scale assessment in VET is to specify an agreement on occupational profiles and to clarify whether or not the national educational outcomes meet the actual requirements of labour market. Therefore we have to check how the VET programmes assess students’ ability to act on the labour market. Analysis shows that VET systems in countries or even in sectors are unlikely to be drawn into convergence through the European process. Results of our pre-feasibility study suggested a variety of outcomes rather than conformity (Baethge et al., 2006). Regarding these results there is a need for a frame of reference. We propose a catalogue of measures: (1) for the identification of typical jobs and corresponding tasks including the generalization for one sector we suggest O*NET criteria (Peterson et al. 1995). O*NET On-line from the US Department of Labour reports on a number of employer surveys from different countries that aim at identifying skills that employees should possess. Competences are defined very broadly and frequently emphasise communication, cooperation, and other specific knowledge requirements. One of the greatest benefits of O*NET is that competences, skills and corresponding knowledge requirements are defined within the context in which they are practiced (e.g. Leney, 2004). (2) In addition to the national assessment procedures we want to collect necessary information for the design and the difficulty of test situation. An analysis of different tasks for assessing educational outcomes at the end of VET gives insights into the educational level, in terms of

§         requirements at the end of VET programme(s),

§         expected levels of students performance, and

§         range of final examinations and assessment procedures.

(3) Methods of inquiry

Using that framework the research group has started to construct an instrument and invest its measurement properties (specifically, validity and reliability). The act of measuring will be positioned as a link between qualitative observations in the labour market and quantitative measures using authentic vocational tasks (Winther, & Achtenhagen, 2008). In the world of work, occupational tasks are defined by organisations as requirements for their employees. In this regard, tasks are stimuli given in an organisational context. Contrary to occupational tasks, competences are bound to an individual. Individuals are carriers of competences; they have developed the ability and willingness to successfully apply their knowledge, abilities and experiences to authentic occupational situations. In this regard, competences cannot be developed per se; competences deal with occupational tasks in a specific situation in a particular occupational context. Therefore vocational tasks are implemented in authentic computer-based simulations. For measuring competences in the world of work we set in both classical and modern testing approaches.

(4) Results

The empirical collection started this October; so first results (based on IRT-models) of our study are expected in March 2009. According to TIMSS and PISA vocational competences will be reported mainly through scale scores derived using Item Response Theory (IRT) scaling (Wilson, 2005). This approach allows the performance of a sample of test takers in a subject area to be summarized on a common scale or series of scales even when different persons have been administered different items. This also allows the performance of test takers from different cohorts to be summarized on a common scale even when they have worked on different test forms. The international comparison of competences in real-word situations is new. That is, new tests for vocational competences have to be developed. The proposed item format is a realistic task in a computer-simulated work environment. The main advantage of this format is (face) validity. That is, the test reflects professional tasks that are common in the vocational area. It ensures that the test measures the concept which is intended to be measured. Another advantage of the computer-simulated format is that students enjoy doing the test, and this will enhance the data quality. The main disadvantage of the computer-simulated format is the relatively long assessment time. At least twice the time is needed to finish the test, in order to get the same measurement precision as with an abstract multiple-choice format for the items. Since the requirements on measurement error are relatively low, the test length will still fit in the time restrictions which are reasonable in a large scale assessment.

References

Baethge, M., Achtenhagen, F., Arends, L., Babic, E., Baethge-Kinsky, V., & Weber, S. (2006). VET-PISA. Stuttgart: Steiner.

Leney, T. et al. (2004). Achieving the Lisbon goal: The contribution of VET. Final Report to the European Commission. The Lisbon-to-Copenhagen-to-Maastricht Consortium Partners. London: QC.

Peterson, N., Mumford, M., Borman, W., Jeanneret, P., & Fleishman, E., 1995. Development of prototype Occupational Information Network (O*NET) content model (Vols.I & II). Salt Lake City, UT: Utah Department of Workforce Services.

Wilson, M. (2005). Constructing Measures. An Item Response Modeling Approach. Mahwah, NJ: Lawrence Erlbaum.

Competencies can be understood as cognitive abilities and skills with strong and stable inter-individual differences.  One of the major problems in research and practice is how to measure them with regard to their general and specialized character. 

Construct Definition

Competence in VET spans general education to job-specific skills. Beck and Achtenhagen (2007) identify four points along the continuum: general subject-specific competence in reading, mathematics and problem solving, generic occupational competence in organizations and labor markets, vocational subject-specific competence in occupational identity and work-task problem solving, and domain-specific skills necessary for successful job performance. 

Approach to Measurement Construction

The criterion-sampling model seems most appropriate for assessing many aspects of VET competence.  It is quite simple and logical.  If you want to know what a person knows and can do, sample tasks from the domain in which that person is to act, observe performance, and infer competence.  For example, if you want to know if a person knows the laws for driving a car and has the ability to drive a car, don’t give only a multiple-choice test. That works fine for knowledge of laws. Rather, give the person a driving test in which he has to start the car, pull into traffic, turn right and left in traffic, back up, and park.  Based on this sample of performance it is possible to draw more general inferences about his driving performance.

This criterion-sampling model differs from the usual ability model.  With the ability model, complex performance is divided into component abilities (e.g., quantitative, spatial, visual, critical thinking), those abilities are tested, and inferences drawn from an additive combination of the abilities to overall complex performance.

So, the criterion-sampling model: Samples tasks from the domain of tasks to which you want to infer knowledge. As a person learns, performance on these “real-world” tasks should improve and it is alright to show people how to perform the tasks. Indeed, if a person “cheats” by learning to perform well on the tasks, that person has acquired competence in the domain of tasks of interest. Moreover most real-world tasks rely on a person operating on the world and not selecting alternatives as the world does not present itself as such—life is not a multiple-choice test with only one right answer.  Consequently, the responses on the performance measurement should match those normally expected in real-world performance.

The criterion-sampling approach differs, then, with the ability approach we are all so very familiar with where complex tasks are divided into component parts, multiple-choice items are written in subtests, and the test items must be kept secret because improving performance on them doesn’t necessarily mean improved performance in the domain—cheating is a major concern.

Evaluation of the Quality of Measurements

The notion of sampling tasks and operant responses from an often “real-world” domain can be modeled using Generalizability (G) Theory, a statistical theory of measurement (Shavelson & Webb, 1991).  G theory pinpoints major sources of error in a behavioral measurement, provides a reliability-like coefficient, and models alternative revisions in the measurement to improve its quality.

As an example of the proposed approach to measuring competence, consider the measurement of military job performance (Wigdor & Green, 1991).  A criterion-sampling approach was used. Military doctrine was used to identify and enumerate the universe of job tasks in a military occupational specialty.  Tasks were sample from the job and a performance assessment created with these tasks and the operant responses embedded in the tasks.  Human raters evaluated complex performance in real time.

In order to evaluate the quality of the military performance measurements, generalizability theory was used.  Two major questions were of great concern.  First, could human raters reliably rate job incumbents’ complex performance in a military occupational specialty?  Second, how much error was introduced by variation in the difficulty, importance, and frequency of the tasks included in the assessment?

To get at sources of measurement error and the reliability of the job performance assessment, we partitioned variability in enlistees’ performance into 3 sources: person (machinist mate or infantryman), task and examiner (rater).  Person (job incumbent) variability was considered “desirable variability”—people differ systematically in how well they perform a job and we wanted to measure differences in performance.  Task and rater sampling, in contrast, introduce unwanted error into estimating a person’s level of performance or competence.

With respect to whether raters could accurately evaluate complex military performance in real time, they could.  Indeed what was surprising was how well they could do so (small or zero variability due to rater, and rater interactions with person and task). With respect to inconsistency in performance due to the difficulty (etc.) of job tasks, task sampling variability accounted for the major share of measurement error—person x task and also task sampling.  Increasing the number of tasks would reduce this error and increase reliability. Surprisingly, raters were well calibrated, even when evaluating complex performance.  These findings have been replicated over and over in the past 20 years (e.g., Shavelson, Baxter & Gao, 1993).

The usual validity indices indicated that the job-performance observed correlated with supervisors’ ratings (to the limit of reduced variability in the ratings) and with the military entrance examination as would be predicted by theory and past research.

Concluding Point

Advances in measurement of performance have been made over the past 100 years. And now with computer technology, costs, logistics, and data collection can be carried out within practical bounds, and scoring can be automated.  These advances would serve the study and evaluation of VET competencies well.

References

Beck, K., & Achtenhagen, F. (2007).  Vocational education and training in a globalized world.  Berichte: Band 29. Göttingen, Germany: Seminar für Wirtschaftspädagogik der Georg-August-Universität Göttingen.

Shavelson, R.J., Baxter, G.P., & Gao, X. (1993). Sampling variability of performance assessments. Journal of Educational Measurement, 30(3), 215-232.

Shavelson, R.J., & Webb, N.M. (1991). Generalizability theory: A primer. Newbury Park, CA:  SAGE.