Tag: research

The Power of Misconception

“Misconception” is one of those words that makes you slump your shoulders and sigh. It’s not inspiring like “creativity” or “glee.” In fact, in education circles we often resort to “alternate conception” so we’re not starting the conversation at a bad place.

In this post, I want to share with you some beautiful new research on how misconception affects teaching and learning.

In the 6 March 2013 issue of the American Education Research Journal, Philip M. Sadler, Gerhard Sonnert, Harold P. Coyle, Nancy Cook-Smith and Jaimie L. Miller describe “The Influence of Teachers’ Knowledge on Student Learning in Middle School Physical Science Classrooms”. Those of us in astronomy education immediately recognize Phil Sadler. His “A Private Universe” video is must-see for every astronomy instructor, K-12 and beyond.

Here’s what Sadler et al. did in the present study.

They created a 20-question, multiple-choice quiz based on concepts taught in middle school science: properties and changes in properties of matter, motion and forces, and transfer of energy. They chose concepts where kids have a common misconception, for example,

Electrical circuits provide a means of transferring electrical energy when heat, light, sound and chemical changes are produced (with common misconception that electricity behaves in the same way as a fluid.) (p. 12)

With the test in hand, they recruited 100’s of seventh and eighth grade science teachers in 589 schools across the U.S. They asked them to give the test at the beginning, middle and end of the year, to look for signs of learning. By the end of testing, there were matching sets of tests from 9556 students and 181 teachers. In other words, a big enough N that the data could mean something.

By looking at the students’ responses, the authors were able to classify the 20 questions into 2 types:

  • for 8 questions, some students got them right, some got them wrong, with no pattern in the wrong answers. They call these “no misconception” questions.
  • for 12 questions, when students got them wrong, 50% or more chose the same incorrect answer, a carefully chosen distractor. These questions are called “strong misconception” questions.

Sadler et al. also had the students write math and reading tests. From their scores, the students were classified as “high math and reading” or “low math and reading”.

They did something else, too, and this is what makes this study interesting. They asked the teachers to write the test. Twice. The first time, the teachers answered as best they could. Their scores are a measure of their subject matter knowledge (SMK). The second time, the teachers were asked to identify the most common wrong answer for each question. How often they could identify the common wrong answer in the strong misconception questions is the teachers’ knowledge of student misconception (KoSM) score.

With me so far? Students with high or low math and reading skills have pre- and post-scores to measure their science learning gain. Teachers have SMK and KoSM scores.

Do you see where this is going? Good.

There’s a single graph in the article that encapulates all the relationships between student learning and teachers SMK and KOSM. And it’s a doozie of a graph. Teaching students how to read graphs, or more precisely, teaching instructors how to present graphs so students learn how to interpret them, is something I often think about. So, if you’ll permit me, I’m going to present Sadler’s graph like I’d present it to students.

First, let’s look at the “architecture” of the axes before we grapple with the data.

Let's look at the axes of the graph first, before the data blind us. (Adapted from [1])
Let’s look at the axes of the graph first, before the data overwhelm us. SMK = teachers’ subect matter knowledge; KoSM is the teachers’ knowledge of student misconceptions.  (Adapted from Sadler et al. (2013))
The x-axis give the characteristics of the science teachers (no SMK,…, SMK & KoSM) who taught the concepts for which students show no misconception or strong misconception. Why are there 3 categories for Strong Misconception but only 2 for No Misconception? Because there is no misconception and no KoSM on the No Misconception questions. What about the missing “KoSM only” condition? There were no teachers who had knowledge of the misconceptions but no subject matter knowledge. Good questions, thanks.

Cohens_d_4panel_wikipedia_CCThe y-axis measures how much the students learned compared to their knowledge on the pre-test given at the beginning of the school year. This study does not use the more common normalized learning gain, popularized by Hake in his “Six-thousand student” study. Instead, student learning is measured by effect size, in units of the standard deviation of the pretest. An effect size of 1, for example, means the average of the post-test is 1 standard deviation higher than the average of the pre-test, illustrated in the d=1 panel from Wikipedia. Regardless of the units, the bigger the number on the y-axis, the more the students learned from their science teachers.

And now, the results

This is my post so I get to choose in which order I describe the results, in a mixture of  the dramatic and the logical. Here’s the first of 4 cases:

Students who scored low on the reading and math tests didn't do great on the science test, though the ones who had knowledgeable teachers did better. (Graph adapted from Sadler et al. (2013))

Students who scored low on the reading and math tests didn’t do great on the science test, though the ones who had knowledgeable teachers did better. (Adapted from Sadler et al. (2013))

The weaker students who scored low on the reading and math tests didn’t do great on the science test either, though the ones who had knowledgeable teachers (SMK) did better. Oh, don’t be mislead into thinking the dashed line between the circles represents a time series, showing students’ scores before and after. No, the dashed line is there to help us match the corresponding data points when the graph gets busy. The size of the circles, by the way, encodes the number of teachers with students in the condition. In this case, there were not very many teachers with no SMK (small white circle).

Next, here are the learning gains for the students with low math and reading scores on the test questions with strong misconceptions:

Students with low math and reading scores did poorly on the strong misconception questions, regardless of the skill of their teachers. (Adapted from Sadler et al. (2013))
Students with low math and reading scores did poorly on the strong misconception questions, regardless of the knowledge of their teachers. (Adapted from Sadler et al. (2013))

Uh-oh, low gains across the board, regardless of the knowledge of their teachers. Sadler et al. call this “particularly troubling” and offer these explanations:

These [strong misconception questions] may simply have been misread, or they may be cognitively too sophisticated for these students at this point in their education, or they many not have tried their hardest on a low-stakes test. (p. 22)

Fortunately, the small size of the circles indicates there were not many of these.

What about the students who scored high on the math and reading tests? First, let’s look at their learning gains on the no-misconception questions. [Insert dramatic drum-roll here because the results are pretty spectaculars.]

Students with knowledgeable teachers exhibited huge learning gains. (Adapted from Sadler et al. (2013))
Students with knowledgeable teachers exhibited huge learning gains. (Adapted from Sadler et al. (2013))

Both black circles are higher than all the white circles: Even the students with less-knowledgeable teachers (“no SMK”) did better than all the students with low math and reading scores. The important result is how much higher students with knowledgeable teachers scored, represented by the big, black circle just north of effect size 0.9. Science teachers with high subject matter knowledge helped their students improve by a half a standard deviation. Rainbow cool! The large size of that black circle says this happened a lot. Double rainbow cool!

Finally we get to the juicy part of the study: how does a teacher’s knowledge of the students’ misconceptions (KoSM) affect their students’ learning?

Subject matter knowledge alone isn't enough. To get significant learning gains in their students, teachers also need knowledge of the misconceptions. (Adapted from Sadler et al. (2013))
Subject matter knowledge alone isn’t enough. To get significant learning gains in their students, teachers also need knowledge of the misconceptions. (Adapted from Sadler et al. (2013))

Here, students with knowledgeable teachers (I guess-timate the effect size is about 0.52) do only slightly better than students with less knowledgeable teachers (effect size around 0.44). In other words, on concepts with strong misconceptions, subject matter knowledge alone isn’t enough. To get significant learning on these strong misconception concepts, way up around 0.70, teachers must also have knowledge of those misconceptions.

Turning theory into practice

Some important results from this ingenious study:

  • students with low math and reading skills did poorly on all the science questions, despite the knowledge of their teachers, once again demonstrating that math and reading skills are predictors of success in other fields.
  • Teachers with subject matter knowledge can do a terrific job teaching the concepts without misconceptions, dare we say, the straightforward concepts. On the trickier concepts, though, SMK is not enough.
  • Students bring preceptions to the classroom. To be effective, teachers must have knowledge of their students’ misconceptions so they can integrate that (mis)knowledge into the lesson. It’s not good enough to know how to get a question right — you also have to know how to get it wrong.

Others, like Ed Prather and Gina Brissenden (2008), have studied the importance of teachers’ pedagogical content knowledge (PCK). This research by Sadler et al. shows that knowledge of students’ misconceptions most definitely contributes to a teacher’s PCK.

If you use peer instruction in your classroom and you follow what Eric Mazur, Carl Wieman, Derek Bruff and others suggest, the results of this study reinforce the importance of using common misconceptions as distractors in your clicker questions. I’ll save it for another time, though; this post is long enough already.

Epilogue

Interestingly, knowledge of misconceptions is just what Derek Muller has been promoting over at Veritasium. The first minute of this video is about Khan Academy but after that, Derek describes his Ph.D. research and how teachers need to confront students’ misconceptions in order to get them to sit up and listen. If you’ve got 8 more minutes, I highly recommend you watch. Then, if you want to see how Derek puts it into practice, check out his amazing “Where Do Trees Get Their Mass From?” video.

Update 6/6/2013 – I’ve been thinking about this paper and post for 3 months and only today finally had time to finish writing it. An hour after I clicked Publish, Neil Brown (@twistedsq on Twitter) tweeted me to say he also, today, posted a summary of Sadler’s paper. You should read his post, too, “The Importance of Teachers’ Knowledge.” He’s got a great visual for the results.

Another Update 6/6/2013  – Neil pointed me to another summary of Sadler et al. by Mark Guzdial (@guzdial on Twitter) “The critical part of PCK: What students get wrong” with links to computer science education.

Self-enhancement and imposter syndrome: neither is good for your teaching

I read a terrific paper this week by Jennifer McCrickerd (Drake University) called, “Understanding and Reducing Faculty Reluctance to Improve Teaching.” In it, the author lists 6 reasons why some post-secondary (#highered) instructors are not interested in improving the way they teach:

  1. instructors’ self-identification as members of a discipline (sociologists, biologists, etc.) instead of as members of the teaching profession;
  2. emphasis early in instructors’ careers (graduate school, when working to attain jobs and then tenure) on research and publishing;
  3. instructors’ resistance to being told what to do;
  4. instructors’ unwillingness to sacrifice content delivery for better teaching;
  5. instructors’ momentum and no perception that current practices need to change;
  6. risk to sense of self involve with change by change by instructors

These are succinct descriptions of the anecdotes and grumblings I hear all the time, from instructors who have transformed to student-centered instruction, from instructors who see no need to switch away from traditional lectures and from my colleagues and peers in the teaching and learning community whose enable and support change.

What makes McCrickerd’s paper so good, in my opinion, is she connects the motivation behind these 6 reasons  to research in psychology. In particular, to Dweck’s work [1] on fixed- and mutable-mindsets (with fixed-mindset, you can either teach or you can’t, just like some people can do math and some can’t) and to Fischer’s work [2] on dynamic skill theory (which posits, “skill acquisition always includes drops in proficiency before progress in proficiency returns”).

I won’t go into all the details because McCrickerd’s paper is very nice — you should read it yourself. But there’s one facet that I want to examine because of how it relates to a blog post I recently read, “How I cured my imposter syndrome,” by Jacquelyn Gill (@jacquelyngill on Twitter). She writes,

I felt like I’d somehow fooled everyone into thinking I was qualified to get into graduate school, and couldn’t shake the anxiety that someone would ultimately figure out the error. When something good would happen– a grant, or an award– I subconsciously chalked it up to luck, rather than merit.

With that resonating resonating in my head (yes, resonating: I often feel imposter syndrome), I read that McCrickerd traces some instructors’ reluctance to “self-enhancement” which she describes as follows:

Most Westerners tend, when assessing our own abilities, character or behavior, to judge ourselves to be above average in ability. In particular, we view ourselves as crucial to the success of our accomplishments but when not successful, we attribute the lack of success to things other than our actual abilities.

The streams crossed and I scratched out a little table in the margins of the paper:

McCrickerd points out it is only through dissatisfaction that we change our behavior. An instructor with an overly-enhanced self sees no reason to change when something bad happens in class. “Not my fault they didn’t learn…”

And who else does a lot of teaching? Teaching assistants, that’s who. Graduate students with a raging case of imposter syndrome. When something goes wrong in their classes, “It’s my fault. I shouldn’t even be here in the first place…”

Yeah, that’s a real motivator.

So, what do we do about it.Again, McCrickerd has some excellent ideas:

[I]nstructors need to be understood to be learners with good psychological reasons for their choices and if different choices are going to be encouraged, these reasons must be addressed.

The delicate job of those tasked with helping to improve teaching and learning is to engage these reluctant instructors so they begin to look at learning objectively, then to demonstrate there are more effective ways to teach, to closely support their first attempts (which are likely to result in decrease in proficiency) and to continue to support incremental steps forward. It’s not always easy to start the process but if there’s one thing I’ve learned in my job, it’s the importance of making a connection and then earning the trust of the instructor.

Now, go read the McCrickerd paper. It’s really good.

 

References

[1] Dweck, C. 2000. Self-theories: Their roles in motivation, personality and development. New York, NY: Taylor and Francis Group.

This Scientific American article by Dweck is a nice introduction to fixed and mutable minds-sets

[2] Fischer, K., Z. Yan, & J. Stewart. 2003. Adult cognitive development: Dynamics in the developmental web. In Handbook of developmental psychology, ed. J. Valsiner & K. Connelly, 491-516. Thousand Oaks, CA: Sage Publications. [pdf from gse.harvard.edu]

 Image “The Show Off. Part 2” by Sister72 on flicker (CC)

Making memories stick. With Play-Doh.

My boss, Carl Wieman, likes to describe what we do as “looking for the pattern of how people learn science” (as he does in this video.) And the places to look are classroom studies, brain research and cognitive psychology. I certainly agree with the first place – that’s teachers and teaching. And research like this, that and this other thing about how the brain physically changes while you learn in very cool – that’s science. But cognitive psychology? I’ve been a science geek since, well, since before I can remember anything else, so I really haven’t been exposed to psychology and those other disciplines they teach on the “Arts” side of campus.

Carl says it’s important, though, and I trust him, so my colleagues and I read a cognitive psychology paper for our CWSEI Reading Group “What College Teachers Should Know About Memory: A Perspective From Cognitive Psychology” by Michelle D. Miller (College Teaching, 59, 3, 117, (2011)). Here’s a link, if you have access from where you’re clicking.

The paper is a nice summary of the models of memory. Short term, long term, working memory, ecological (or adaptive) memory. Here’s my interpretation. Every bit of information that’s stored in memory is accompanied by “cues”. Think “tags”, like the ones that accompany this blog post. When you see the cues, you recall the memory, just like finding blog posts by clicking on a tag. Without the tags, finding posts means paging through the archive. With a tag, you can zero in on the post. And the more tags on the post, the easier it is to find. Same with memories: the more cues linked the memory, the easier it will be to recall later.

Not all cues are created equal, though. As Miller puts it,

[u]nderstanding the role and importance of cues enables a richer and more accurate understanding of why people remember — and forget — what they do. (p.119)

Miller carefully crafted descriptions of the kinds of cues that trigger recall, so while I’m cutting them into a list and adding some bold, these are Miller’s words (p. 120):

Here are what I believe to be the cues that trigger us to “tag” information as being survival-relevant:

sensory impact, termed vividness: Concrete information that comes accompanied by sound, visual qualities, even tactile sensation tends to be more memorable than abstract information. Visual information is particularly salient to human beings, so that anything that can be visualized tends to be particularly memorable.

emotional impact is another cue that incoming information warrants long-term storage. Consider situations that relate to survival in a “natural” setting—a sudden danger, a new food source, encountering an enemy—and all would come accompanied with an emotional “charge.”

relevance to one’s own personal history is another indication that information will be useful in the future

structure and meaning—the ability to interpret information and put it into context—helps us distinguish useless background clutter from information that we need to keep

personal participation, as contrasted with passive exposure. This will come as no surprise to those familiar with the “active learning” trend. If we watch someone else do something, that activity may or may not be relevant to us, and it we will likely opt not to form a detailed memory of it. However, if we ourselves carry out the action, there is a greater likelihood that we will need to learn from and recall that experience later. We may also encode a richer set of cues when we are actively involved, which increases the likelihood of retrieving the information later.

Don’t you love it when you read an article that concisely and explicitly describes all those things you feel, in your gut, are important? It’s times like this that make me re-evaluate my naive and, frankly, prejudiced view of psychology, “C’mon, how can you possibly know how humans work?” “Oh, like that, ” he says, sheepishly. “Um, thanks. That’s cool!”

The week my colleagues and I read this paper, I was preparing the next activity for an introductory, general-education astronomy course I work on. This activity, like the others I’ve written and am sharing through the Astro Labs page on this blog, is a chance for “Astro 101” students to get some hands-on interaction with astronomy. Up next was the activity on black holes, especially spaghettification.

“Spaghettification”?

Talk about a made-up word, huh. Not by me, mind you. Chat with any astronomy instructor and you’ll find we all know exactly what it means because it’s the perfect word to describe what happens if you fall into a black hole.

<astronomy lesson>

A black hole with the mass of the Earth would only be about the size of a grape. Imagine it this way: if you could pack together and compress the entire Earth down to the size of a grape, the force of gravity would be so strong curvature of spacetime would be so high that not even light, traveling outwards as the speed of light, could escape.

That describes trying to get out a black hole. What about falling in? Let’s imagine you’re 2 metres tall and your lying on your back with your feet 2 metres from the black hole and your head 4 metres from the black hole. You can see it down there, between your feet, a little shiny grape a couple of metres away. It’s okay to think classically here, for a moment. Gravity is very strong but, being an inverse square law, it drops off quickly: your head is 2 times farther from the black hole than your feet so the force of gravity is only 1/4  as strong. What do you suppose happens when the black hole pulls 4 times harder on your feet? They get ripped off, that’s what. Your body gets stretched out as your feet accelerate towards the black hole, leaving your knees, hands, chest and head behind. This difference-in-forces is called a tidal force because these same kinds of forces occur in the Earth-Moon system where the Moon yanks on the water on Earth’s near-side and leaves the far-side water behind, giving us the tides. Newton worked that one out for us, more than 300 years ago.

The force of gravity between the Moon and the water on the near-side of the Earth is stronger than the force between the Moon and the more distance, far-side water. Earth's watery skin is deformed, giving us the tides. (Graphic: Peter Newbury CC)

Meanwhile, back at the black hole, the hapless astronaut is being pulled down a little funnel that ends up on the grape-sized black hole. Happy astronaut one second, long and skinny piece of spaghetti the next. Spaghettification, baby!

</astronomy lesson>

Ouch, that’s gotta hurt! LOL. Yeah. But how do we get Astro 101 students to remember it a month from now on their exam? Play-Doh, that’s how. Our activity progresses from setting up the phenomenon of tidal forces, to sample calculations demonstrating tidal forces are real, to recreating the spaghettification of a Play-Doh astronaut.

An astronaut falling into a black hole, before spaghettification...
...and after!

Here’s where the part about memory comes in. Students are potentially reluctant to play with Play-Doh. This is University. We’re not Children anymore. Teaching assistants and instructors are equally reluctant to ask students to play with Play-Doh. “Why,” they wonder, “should I?”

Because, I tell the teaching assistants who, if necessary, relay it to the students, it will help you remember. Playing with Play-Doh, stretching the poor astronaut’s legs, often pulling them right off his body, and squishing the Play-Doh into to a narrow strip, is tactile. And emotional – you just ripped his head off, dude! It gives relevance and a physical structure to those calculations. And it takes personal participation – oops, I just pulled his leg off!

Good in theory but how about in practice? The activity ran. The teaching assistants sold it. The students did it. All of them! Now we just have to see if they (1) learned anything and (2) can remember it. For (1), one of the questions they answer at the end of the activity is, “In your own words, describe what happens to the astronaut. Why do you think it’s called ‘spaghettification’?” Here’s one student’s answer, typical of many I thumbed through:

as the astronaut falls toward the black hole, feet first, its body stretches as it nears the black hole. the closer body parts (feet, then hands) stretch faster and fall faster than the head and body. It’s called spaghettification because the legs and hands stretch elongate like spaghetti.

Yep, I’ll take that. Would have been nice to see the word “tidal” in there but he did make the connection between closer and faster. For (2), we’ll be sure to put something on the final exam that tests this material. I’ll let you know in 4 weeks.

As my pop likes to say, “learn by doing.” Let’s update that to, “remember by doing.”

Navigation