Think-aloud Protocol Analysis as a Measure of Students' Science Learning through Design Assessment
Abstract
"Research has indicated that authentic engineering design tasks hold the attention and interest of students and can lead to deeper levels of science engagement (Fortus, Dershimer, Krajcik, Marx, & Mamlok-Naaman, 2004; Roth, 1996; 1997; 1998). However, there is a need for student assessments that can measure how students recall and possibly apply scientific concepts in the context engineering design-based tasks. Furthermore, there is a need for assessments that can identify students' transfer of learning from classroom to real-world science and engineering. One promising assessment methodology is the use of the concurrent think-aloud (CTA) protocol analysis (Atman & Bursic, 1998; Ericsson & Simon, 1993) to assess students' transfer of learning from classroom engineering design-based experiences. Although this assessment process is limited to small purposeful sample sizes, the technique allows researchers to observe in real time triads of students engaged in an authentic design task and identify which science concepts naturally emerge in the design team's dialogue. Using a coding process (Halfin, 1973), researchers in this study have successfully piloted a coding scheme for different cognitive strategies exhibited by elementary school students and furthermore, have systematically identified gaps and areas of overemphasis of the design process. Results were organized using frequency and percentages for time on code. Preliminary analysis of data collected from pilot studies indicated limited use of science terms within design thinking dialogue; however the science terms that did emerge were used accurately and appropriately. Implications of the study reveal that CTAs provide significant insight into students' design and inquiry thinking and if triangulated with other student measures, CTAs can enhance case study research."
