Supporting SoTL

Scholarly teaching. Education research. Scholarship of Teaching and Learning. These are all activities related to applying valid research methods – typically developed in other disciplines – to study teaching and learning.

For faculty members who’s merit, tenure, and promotion is based, in part, on their research output, publishing articles about education can’t hurt but it may not be seen as important as their disciplinary research. UBC, like a growing number of universities, has a tenure-track stream of Assistant, Associate, and (full) Professor of Teaching. We call it the Educational Leadership stream because success and promotion requires demonstrating impact and leadership beyond your classroom. For faculty in this stream, engaging in SoTL is a powerful way to demonstrate that leadership.

It’s my Centre for Teaching and Learning’s mission to “promote, inspire, and support excellence, leadership, scholarship, and technologies in teaching and learning.” I find supporting scholarship is one of most difficult part of our mission because when we start talking about research, each faculty member immediately snaps to the kinds of disciplinary research they do – if any – and tries to force education into that methodology. I struggle to support them because (i) I don’t know what kind of research they do and (ii) I’m most familiar with research methods found in STEM.

I’m writing this post because something happened last week, something good, that’s changed my approach and, I hope, the success of the faculty members I work with. Here’s the story. Dr. Jasmin Hristov, a research-stream Assistant Professor in the Department of History & Sociology, Irving K. Barber School of Arts and Sciences gave me her permission to tell it.

Professor Hristov teaches upper-level sociology. She plans to bring in a series of guest speakers via video conference and asked if she could use my Centre’s workshop room. “Yes, of course,” I replied. And then, thinking about my Centre’s mission, I added, “You’re doing something innovative – would you be interested in talking about how you could study whether or not it’s effective?” She was, and we met.

First, Professor Hristov described her motivation: introduce the students to six experts from around the World, with careful attention to diversity of gender, race, location, and rank. For each guest speaker, the students do some background reading, prepare questions to ask the speaker, and lead a discussion. After class, the students write a reflection about the experience.

“How can we tell if it was effective? How can we tell if students learned anything?”

We nearly got lost down a dead end. Professor Hristov: “I’ve taught this course before without the video conferencing but with different students and, obviously, without the reflection.” Both of us nearly concluded, “Without a control group to compare grades against, I don’t see how we can study this.”

We didn’t go there, though, because serendipitously, I started the conversation with,

How can we find evidence of impact?

This question opened up whole new ways of thinking, without sending us on that narrow “research = A/B study with statistical significance” path. It led quickly to a couple of possibilities that could produce interesting results that don’t rely on the success or failure of p < 0.05.

Text analysis of students’ reflection

4-page reflection × 6 reflections × 30 students = huge amount of text

Imagine examining all that text with powerful tools like Voyant or NVivo. Will students naturally comment on the diversity of the speakers? That was one of the elements deliberately built into this intervention, recall. Do they need a prompt? Not a heavy prompt like, “Please comment on the diversity of the speakers.” That will only get the answers the students think Professor Hristov wants to hear. Something more subtle, like, um, not sure yet.

But imagine the kind of evidence of impact she could include in the SoTL article:

“I carefully chose the speakers to expose my students to a wide range of races, locations, genders, and ranks. In their reflections, students made the following associations…”

This isn’t cherry-picking an individual student’s comments – that’s a helpful exemplar or supporting anecdote but it’s not evidence. Instead, we have legitimate connections and insight students are making.

Quantitative analysis of reflection grades

Just because we can’t do a controlled A/B study doesn’t mean we can’t do quantitative analysis. Imagine this: imagine we compare the students’ marks on the reflections with their marks on the rest of the course. The reflections are worth around 1/3 of the total mark, so the reflections are worth enough that students will put legitimate care and effort into them. In other words, the reflections are not some incidental marks students can blow off, and they’re not so important that nothing else matters in the course. I made up some data (thx, to see what kinds of conclusions we could make (click to enlarge):

Hypothetical student marks on the reflection and other course assignments, with a range of correlations and conclusions about impact. (Data via Graphic: Peter Newbury)

The left graph shows there’s a relationship between the students’ success on the reflections and the rest of the course. Do the reflections help them succeed with the other assignments? Do the other assignments help them write better reflections? Can’t tell. Better look at the text analysis…

The center graph isn’t telling a compelling story. Success on the reflections doesn’t seem to have any connection to success on the rest of the course. We can probably conclude the same about what the students are getting out of the video conferences. Time to rethink how the video conferences are integrated and supported.

The right graph is a worst-case scenario: success on the reflections comes at the expense of the their success in the rest of the course. Oh c’mon, this would never happen, right? Well, I’ve seen courses where there’s a “capstone project” that takes all the students’ time. If the capstone is that important, it should probably represent a significant fraction of the overall course mark, so success on the capstone guarantees success in the course. I’ve also seen cases where success on the capstone requires sacrificing the other courses you’re taking – time for the Department Head to get the course instructors together to coordinate their assignments!

No matter the scenario, there’s something here for Professor Hristov to share in the discussion of her SoTL paper. The conclusions will be useful to others thinking about integrating video conferencing into their courses.

Evidence of Impact

This will be my new conversation starter when promoting, inspiring, and supporting scholarship. It’s also a good prompt for the faculty members, themselves, who want to (need to?) demonstrate educational leadership. This prompt invites us to be curious and creative, instead of trying to jam teaching and learning into the same research methods that we’re familiar with from disciplinary research.

Open Classroom pop-up thank-you card

Every Fall, I follow along as Derek Bruff @derekbruff tweets out inspiring stories from the open classroom event his Center for Teaching runs at Vanderbilt. Course instructors from across campus volunteer to open their classrooms and welcome their peers to come observe. While we may have 25, 60, 300, or more students in our classrooms, it’s rare to have a colleague, and open classroom events provide an opportunity for all the educators in the room – the ones at the front and the ones at the back – to share some formative feedback.

We hosted our first Open Classroom Week at UBC Okanagan October 1-5, 2018. Twelve course instructors from across the campus, across disciplines, and from 1st-year to 4th-year invited their peers into their classrooms.

Opening your classroom to your colleagues takes courage and confidence and demonstrates educational leadership. So, I wanted to thank those twelve course instructors. Sure, I could send SW-S, RT, RP, WSM, CL, RF, GD, TE, TF, NL, CS, and AK a letter (or a letter to their Department Heads) on Centre letterhead, formally thanking them for participating in the event. But I wanted something they could put on the shelf in their office so remind them, and any visitors, that they did something valuable. Combine that with my obsessi–, er, interest in pop-up cards and you get this:

The pop-up thank-you card I made for the UBC Okanagan course instructors who opened their classrooms and welcomed their peers and colleagues to come and observe. A template and instructions available below so you can make your own.
The pop-up thank-you card I made for the UBC Okanagan course instructors who opened their classrooms and welcomed their peers and colleagues to come and observe. A template and instructions available below so you can make your own. (Photo: Peter Newbury CC-BY)

Do it yourself

Want to make one for your Center? Here’s the PPT file I used to create the card, plus a set of directions for editing the text, printing the card, and making it. I think the instructions are clear but by the time I wrote them, I’d already made 5 prototypes and then the dozen cards I gave to my UBC Okanagan colleagues so I could pretty well do it in my sleep. If you get stuck, feel free to tweet me at @polarisdotca. And then send me a picture of your finished card (and permission to share it)!

Planning your first day of class

As Will Rogers once said, “you never get a second chance to make a first impression.” What you do in your first day of class establishes the learning environment for the rest of your course, so it’s critical to think and plan carefully.

Nobel prize winner and science educator, Carl Wieman, reminds us the goals of the first class are to

  • motivate learning – why should your students engage and invest their time and energy?
  • personalize the experience – how can each student find your course meaningful?
  • establish expectations – how will your course run and what will happen in class?

That’s a lot to accomplish in 50 or 80 minutes, especially if you also want to (and you probably do!) start teaching your students about the content and concepts of your course.

Why is this important?

Why is it important to think about and plan your first class, on top of planning your syllabus, assessments, and lessons?

  1. You want every student to leave the first class thinking

    This will be a good course.
    I’m okay, I’m safe being here.
    I have something valuable to contribute.

  2. No matter how much you prepare, when the clock strikes and finally stand up at the front of the room and flip on your wireless mic, you are not at your best. You’re anxious and exhausted and nervous and excited. And that is NOT the moment you want to be making important decisions and setting precedents that will impact the rest of the course. Now, before the course starts, is the time to think and make decisions.
  3. More from Wieman: If you don’t spend time establishing the learning environment but instead, simply “go over the syllabus” or launch right into Topic 1,

    students who are most likely to see the subject as worth learning are those whose backgrounds, and corresponding attitudes, are most like that of the instructor. Those students whose backgrounds are different, which by definition (usually) includes most members of under-represented groups, will be less likely to understand the appeal of the subject and consequently more inclined to put their efforts into pursuing some other discipline.

Do-It-Yourself first class

This excellent resource (PDF) from the Carl Wieman Science Education Initiative contains dozens of suggestions for what you could do. There isn’t time to do them all.

I invite you to download this list and print this Venn diagram. For each item A, B, C,… (and others you add to the list) decide for yourself if the item motivates learning, personalizes the experience for the student, and/or establishes expectations. When you’re done, perhaps the items at the center of the Venn diagram – the items that do all three simultaneously – are the ones to build into your first class. That way, you can be the most efficient and effective in the limited time you have with your students.

What should you do in the first day of class? Things that simultaneously motive learning, personalize the experience, and establish expectations. (Graphic by Peter Newbury CC-BY)

I think you’ll find, for example, that when an item clearly establishes expectations and personalizes the experience, with just a small change in how you present it or build it into your class syllabus or policies, you can also motivate learning.

Do’s and Don’ts

What you do (and don’t do) in your first class is up to you, of course. As a helpful reminder from people who’ve been there before and seen it happen, here are some first day of class do’s and don’ts for you to consider.

Check out the classroom before the first class

  • fully connect and test your laptop
  • using clickers? connect and test the hardware and software
  • how do you log into the podium/lecturn computer, if needed?
  • what’s the wifi like, even in the back corners?
  • how do the classroom lights work?
  • try the lapel (“lav”) mic
  • are you using a presentation remote to advance your slides? Does it work from the back of the room?
  • Assume you can figure it out at the time
  • let a technical problem ruin your only chance to make a first impression
Start the class on time (establish expectations!)
  • arrive late (what expectation does that establish!)
  • have “intimate” conversations with the (enthusiastic) students who arrive early and sit in the front row. This can signal to the rest of the class who will be getting special attention. Instead, circulate around the room and speak with lots of students, or greet everyone at the door.
Tell students you think they can all succeed if they put in the effort (growth mindset). It’s fine to say the course is challenging (after all, shouldn’t it be?) as long as you also let them know the course is

  • interesting
  • valuable
  • achievable with appropriate effort
Say threatening things like

  • you expect some of them to fail (“Look left, look right – one of you won’t be here by the end of the course.”)
  • this is a “weed-out” or “gatekeeping” course (to get rid of students who shouldn’t continue to the next course)
  • students don’t usually like this course
  • this course is really hard
Give them an authentic experience of what the class will be like.

  • If you’re going to use peer instruction with clickers, do it even though not everyone has a clicker yet. If awarding participation points is part of your plan, don’t start that until Week 2.
  • If you’re going to flip your class, send them a pre-reading assignment (and welcome) before the first day.
  • If you’ll be asking them to discuss challenging issues and items in small groups throughout the course, do it in the first class, too, maybe as an icebreaker.
Use teaching practices that are inconsistent with how you’ll teach the rest of the course.
Model academic integrity, today and every day. Address it when it’s needed: discuss plagiarism in Week 3 when you assign the first essay. Emphasize penalties for academic misconduct and all the ways a student can be kicked out of the university.

  • It establishes a feeling of distrust
  • Now is not the time they need to be hearing this. It’s important, yes, but not right now.
End the class on time with a slide containing the most valuable information, just in case a lost student missed the first few minutes of the class:

  • your preferred name
  • office location and hours
  • contact info
  • course website
  • Important Thing
End the class early (establishes the wrong expectation) or
end the class late (be kind to your anxious, exhausted colleague who’s trying to get into the classroom to set up their first class!)
Repeat vital information (your preferred name, contact info, Important Thing) at the begin of second class Assume everyone was there in the first class.

You got this

Taking the time now to think and plan doesn’t mean you won’t be anxious and exhausted on your first day of class. But you can be confident in what you say and do. Through your actions and inactions (h/t, @ddmeyer, for that excellent phrase), you can support your students and not intensify their struggles.