First, a big thanks to Science Borealis for highlighting my blog as today’s #cdnsciblog. I’m flattered to be in their company and happy to contribute.
I write mostly about teaching and learning science at the university level. This blog started as a way for me to archive interesting things I’d seen or done while working in the Carl Wieman Science Education Initiative at UBC. It was for me, so I wouldn’t have to rely on my fading memory and chicken-scratch notes. Then other people started reading it, and leaving comments. Wow! There’s a community out there!
I’m hoping to combine my passions for science education and science communication at the next Science Online Together meeting, February 26 – March 1, 2014 in in Raleigh, NC. If everything goes as planned (c’mon, program committee, you really like my proposal, right?) I’ll meet you there!
And just in case you’re wondering about my Twitter handle, @polarisdotca: that’s Polaris, the North Star for my love of astronomy, and “dot ca” to let everyone know I’m proudly Canadian. Strangely enough, dot CA also work here in my new home, San Diego, California.
Thanks for dropping in. And again, to the Science Borealis team for igniting this community.
I spend a lot of time thinking about how and why peer instruction works and helping instructors improve their technique. The other day, I had an experience that crystallized for me the difference between peer instruction and students merely clicking their clickers.
The instructor I was observing, who gave me permission to tell this story, was teaching a political science class about gender and politics. During the class, she asked a clicker question something like this:
Where does the US rank in the world when it comes to
the percentage of women in elected positions?
A) 12th
B) 34th
C) 78th
D) 112th
Here’s what was supposed to happen…
The students would vote. Then they’d “turn to your neighbors and convince them you’re right.” The students would help each other remember the correct answer, 78th, and then spontaneously launch into a discussion about how that’s surprising because of ABC and interesting because it’s DEF but not GHI and so on, bringing in all the interesting, conceptually challenging ideas that political scientists explore and debate.
That’s not what happened, though. Sure, they voted. The even split across all four answers showed they didn’t know the answer and were guessing. And when she asked them to turn to their neighbors, the room didn’t crescendo with conversation. There was some brief murmuring
“What’d you pick?”
“78th.”
“Huh. I picked 34th.”
And that was it. There was no interest, no conversation, no debate.
The problem, I think, was not with the students but with the question. It didn’t require students to confront their understanding, take a stance and be prepared to defend it. It simply required them to remember some fact from some book somewhere. In other words, just the kind of low-level knowledge we often scoff at when we say, “Oh, this course isn’t about fact and memorization. No, this course is about gathering evidence to form and then defend arguments.” Not to pick on the Humanities, I often hear from STEM instructors, “…No, this course is about problem-solving and applying the theory to real world applications.”
Here’s a better clicker question:
The US ranks 78th in the world when it comes to the percentage of women
in elected positions. In your opinion, why is this surprising?
A) Because it shows ABC
B) Because it’s an example of DEF
C) Because it contradicts GHI
D) Something else
The fact that requires memorization? Just give it to them. Most importantly, seed the conversation with the concepts you want them to grapple with. They’re not expert enough, not yet, to spontaneously come up with ABC, DEF and GHI, so spark the conversation, especially if one of the choices is a common misconception that needs confronting.
With a question like this, and effective peer instruction “choreography”, the students first do a solo vote where they have to decide, each in their own heads, what they believe about the concept. This solo vote is critical because it prepares them to contribute to the discussion with their peers that follows. For 30 seconds or a minute, the room will be filled with political scientists. Or at least, students practicing to become political scientists, getting immediate feedback from their peers and the instructor. That’s what peer instruction is about.
[Update Jul 5, 2013] One of my Summer projects is (finally) reading Ken Bain’s What the Best College Teachers Do. In summarizing how the best college teachers conduct their classes, Bain describes the “natural critical learning environment”
More than anything else, the best teachers try to create a natural critical learning environment: “natural” because students encounter the skills, habits, attitudes, and information they are trying to learn embedded in questions and tasks they find fascinating — authentic tasks that arouse curiosity and become intrinsically interesting; “critical” because students learn to think critically, to reason from evidence, to examine the quality of their reasoning using a variety of intellectual standards, to make improvements while thinking, and to ask probing and insightful questions about the thinking of other people. (p. 99)
Where I see the clearest connection with the questions used in peer instruction is one of the characteristics of a natural critical learning environment”
[T]he natural critical learning environment also encourages students in some higher-order intellectual activity: encouraging them to compare, apply, evaluate, analyze, and synthesize, but never only to listen and remember. Often that means asking students to make and defend judgments and them providing them with some basis for making the decision. (p. 102)
The original peer instruction question, “Where does the US rank in the world…” asks students only to remember. A good peer instruction question (and a well-choreographed episode of peer instruction) forces students climb higher into Blooms’s taxonomy.
“Misconception” is one of those words that makes you slump your shoulders and sigh. It’s not inspiring like “creativity” or “glee.” In fact, in education circles we often resort to “alternate conception” so we’re not starting the conversation at a bad place.
In this post, I want to share with you some beautiful new research on how misconception affects teaching and learning.
Here’s what Sadler et al. did in the present study.
They created a 20-question, multiple-choice quiz based on concepts taught in middle school science: properties and changes in properties of matter, motion and forces, and transfer of energy. They chose concepts where kids have a common misconception, for example,
Electrical circuits provide a means of transferring electrical energy when heat, light, sound and chemical changes are produced (with common misconception that electricity behaves in the same way as a fluid.) (p. 12)
With the test in hand, they recruited 100’s of seventh and eighth grade science teachers in 589 schools across the U.S. They asked them to give the test at the beginning, middle and end of the year, to look for signs of learning. By the end of testing, there were matching sets of tests from 9556 students and 181 teachers. In other words, a big enough N that the data could mean something.
By looking at the students’ responses, the authors were able to classify the 20 questions into 2 types:
for 8 questions, some students got them right, some got them wrong, with no pattern in the wrong answers. They call these “no misconception” questions.
for 12 questions, when students got them wrong, 50% or more chose the same incorrect answer, a carefully chosen distractor. These questions are called “strong misconception” questions.
Sadler et al. also had the students write math and reading tests. From their scores, the students were classified as “high math and reading” or “low math and reading”.
They did something else, too, and this is what makes this study interesting. They asked the teachers to write the test. Twice. The first time, the teachers answered as best they could. Their scores are a measure of their subject matter knowledge (SMK). The second time, the teachers were asked to identify the most common wrong answer for each question. How often they could identify the common wrong answer in the strong misconception questions is the teachers’ knowledge of student misconception (KoSM) score.
With me so far? Students with high or low math and reading skills have pre- and post-scores to measure their science learning gain. Teachers have SMK and KoSM scores.
Do you see where this is going? Good.
There’s a single graph in the article that encapulates all the relationships between student learning and teachers SMK and KOSM. And it’s a doozie of a graph. Teaching students how to read graphs, or more precisely, teaching instructors how to present graphs so students learn how to interpret them, is something I often think about. So, if you’ll permit me, I’m going to present Sadler’s graph like I’d present it to students.
First, let’s look at the “architecture” of the axes before we grapple with the data.
Let’s look at the axes of the graph first, before the data overwhelm us. SMK = teachers’ subect matter knowledge; KoSM is the teachers’ knowledge of student misconceptions. (Adapted from Sadler et al. (2013))The x-axis give the characteristics of the science teachers (no SMK,…, SMK & KoSM) who taught the concepts for which students show no misconception or strong misconception. Why are there 3 categories for Strong Misconception but only 2 for No Misconception? Because there is no misconception and no KoSM on the No Misconception questions. What about the missing “KoSM only” condition? There were no teachers who had knowledge of the misconceptions but no subject matter knowledge. Good questions, thanks.
The y-axis measures how much the students learned compared to their knowledge on the pre-test given at the beginning of the school year. This study does not use the more common normalized learning gain, popularized by Hake in his “Six-thousand student” study. Instead, student learning is measured by effect size, in units of the standard deviation of the pretest. An effect size of 1, for example, means the average of the post-test is 1 standard deviation higher than the average of the pre-test, illustrated in the d=1 panel from Wikipedia. Regardless of the units, the bigger the number on the y-axis, the more the students learned from their science teachers.
And now, the results
This is my post so I get to choose in which order I describe the results, in a mixture of the dramatic and the logical. Here’s the first of 4 cases:
The students who scored low on the reading and math tests didn’t do great on the science test either, though the ones who had knowledgeable teachers (SMK) did better. Oh, don’t be mislead into thinking the dashed line between the circles represents a time series, showing students’ scores before and after. No, the dashed line is there to help us match the corresponding data points when the graph gets busy. The size of the circles, by the way, encodes the number of teachers with students in the condition. In this case, there were not very many teachers with no SMK (small white circle).
Next, here are the learning gains for the students with low math and reading scores on the test questions with strong misconceptions:
Students with low math and reading scores did poorly on the strong misconception questions, regardless of the knowledge of their teachers. (Adapted from Sadler et al. (2013))
Uh-oh, low gains across the board, regardless of the knowledge of their teachers. Sadler et al. call this “particularly troubling” and offer these explanations:
These [strong misconception questions] may simply have been misread, or they may be cognitively too sophisticated for these students at this point in their education, or they many not have tried their hardest on a low-stakes test. (p. 22)
Fortunately, the small size of the circles indicates there were not many of these.
What about the students who scored high on the math and reading tests? First, let’s look at their learning gains on the no-misconception questions. [Insert dramatic drum-roll here because the results are pretty spectaculars.]
Students with knowledgeable teachers exhibited huge learning gains. (Adapted from Sadler et al. (2013))
Both black circles are higher than all the white circles: Even the students with less-knowledgeable teachers (“no SMK”) did better than all the students with low math and reading scores. The important result is how much higher students with knowledgeable teachers scored, represented by the big, black circle just north of effect size 0.9. Science teachers with high subject matter knowledge helped their students improve by almost a full standard deviation. Rainbow cool! The large size of that black circle says this happened a lot. Double rainbow cool!
Finally we get to the juicy part of the study: how does a teacher’s knowledge of the students’ misconceptions (KoSM) affect their students’ learning?
Subject matter knowledge alone isn’t enough. To get significant learning gains in their students, teachers also need knowledge of the misconceptions. (Adapted from Sadler et al. (2013))
Here, students with knowledgeable teachers (I guess-timate the effect size is about 0.52) do only slightly better than students with less knowledgeable teachers (effect size around 0.44). In other words, on concepts with strong misconceptions, subject matter knowledge alone isn’t enough. To get significant learning on these strong misconception concepts, way up around 0.70, teachers must also have knowledge of those misconceptions.
Turning theory into practice
Some important results from this ingenious study:
students with low math and reading skills did poorly on all the science questions, despite the knowledge of their teachers, once again demonstrating that math and reading skills are predictors of success in other fields.
Teachers with subject matter knowledge can do a terrific job teaching the concepts without misconceptions, dare we say, the straightforward concepts. On the trickier concepts, though, SMK is not enough.
Students bring preceptions to the classroom. To be effective, teachers must have knowledge of their students’ misconceptions so they can integrate that (mis)knowledge into the lesson. It’s not good enough to know how to get a question right — you also have to know how to get it wrong.
Others, like Ed Prather and Gina Brissenden (2008), have studied the importance of teachers’ pedagogical content knowledge (PCK). This research by Sadler et al. shows that knowledge of students’ misconceptions most definitely contributes to a teacher’s PCK.
If you use peer instruction in your classroom and you follow what Eric Mazur, Carl Wieman, Derek Bruff and others suggest, the results of this study reinforce the importance of using common misconceptions as distractors in your clicker questions. I’ll save it for another time, though; this post is long enough already.
Epilogue
Interestingly, knowledge of misconceptions is just what Derek Muller has been promoting over at Veritasium. The first minute of this video is about Khan Academy but after that, Derek describes his Ph.D. research and how teachers need to confront students’ misconceptions in order to get them to sit up and listen.
If you’ve got 8 more minutes, I highly recommend you watch. Then, if you want to see how Derek puts it into practice, check out his amazing “Where Do Trees Get Their Mass From?” video:
Update 6/6/2013 – I’ve been thinking about this paper and post for 3 months and only today finally had time to finish writing it. An hour after I clicked Publish, Neil Brown (@twistedsq on Twitter) tweeted me to say he also, today, posted a summary of Sadler’s paper. You should read his post, too, “The Importance of Teachers’ Knowledge.” He’s got a great visual for the results.
First, a big thanks to Science Borealis for highlighting my blog as today’s #cdnsciblog. I’m flattered to be in their company and happy to contribute.![Let's look at the axes of the graph first, before the data blind us. (Adapted from [1])](https://peternewbury.org/wp-content/uploads/2018/03/SadlerMisconceptionGraphs_0.png)
