Ixchell Reyes
The DIESOL podcast

Brent Warner
Digital Integration in English as a Second or Other Language,

Ixchell Reyes
Episode 42. Using data and teaching and language learning with Mark Makino

Brent Warner
Welcome to DIESOL. This is Episode 42. We are your hosts. I am Brent Warner.

Ixchell Reyes
And I’m Ixchell Reyes Woohoo: 42!

Brent Warner
We’ll leave that one to people to those who know. How are you?

Ixchell Reyes
I just started a teaching a TOEFL course and it is kicking my butt because it’s so clunky to teach it online.

Brent Warner
Yeah, yeah. Well, it’s been a while since you’ve taught like a proper class to write like you were doing teacher training stuff for a while. Right?

Ixchell Reyes
Right teacher training and an odd little teaching jobs that were very short term. So this is going to be a 16 weaker. So week one down 15 word ago, and I survived four days. congratulate my students survived me for four days.

Brent Warner
That’s okay. You’re making it work. I’m sure

Ixchell Reyes
about you. are you closing in on the end of the semester? Or you guys?

Brent Warner
Yeah, this will be the last episode for for this semester. And so I’ll be transitioning to prepping for summer school by the time this episode comes out. And, you know, just trucking along. We’re still we’re still online, fully online. So looking at for maybe coming back for like part part time in person part time online, something like that. Moving forward, so we’ll, we’ll catch up with that as it comes along. So here we go. So today we have a guest, Mark Makino. How are you mark?

Mark Makino
Pretty good. How are you guys?

Brent Warner
Good. Thank you for coming. So Mark, we Mark’s voice may be familiar to serious listeners of the show. From Drinks with DIESOL and a few other times and you’ve come in the clubhouse some of the times too. So Mark, brief introduction, Mark started working in the English teaching industry in Japan after graduating from UCI with an unrelated degree in 2004. That sounds familiar. He started his own a Kiowa, which is an English conversation school and ran that from 2005 to 2016, and returned to the US with his master’s degree in hand before grad or sorry, before adjuncts around the California Community Colleges and other universities for a couple of years. And now he teaches at the American Language and Culture Center at Southern Utah University, where he teaches alternating task based and project based courses. So, Mark, you got to kind of a traveled history of teaching all around in different types of settings. Yeah, you like being on different places?

Mark Makino
It’s, it’s good to have a good good breadth of experience. I guess I miss teaching kids sometimes. You know, I don’t know if you do that. So Brent and I both used to teach in Japan, right. And we both used to teach at IVC.

Brent Warner
Right, but we are in Japan.

Mark Makino
Now. Now. We did not know each other when we will solve math. Mystery Science Theater 3000 the movie at the same movie theater three were teenagers either. But yeah, so um, I, I have taught quite a variety of people by all the student ever, I guess, was 72. And my youngest ever was three, which is a pretty big standard deviation as far as teaching ages go. And, yeah, it’s, it’s been interesting. I missed all of them. But I’m also happy where I am now, which is, I guess, a good sign of a good career, I guess.

Brent Warner
Yeah, that’s, that’s all you can really ask for, right. So Mark, a part of the things that we wanted to talk to you and there are plenty of things that we could talk to you about. But from your time working together with me, you were like always doing interesting stuff around data and around, you know, collecting information from people, whether it be Twitter followers, or students online or si students in class and kind of track up some of the information there. They’re all these sorts of interesting things. And I always thought that it’s like, you’re very diligent about being able to do that and to use data to improve the quality of your teaching, I think and to improve your understanding of what you’re doing in the classroom. And so I wanted to just have an open conversation about that like about data. And, again, we had our little you gave us the warning. Be careful because this could be very boring podcasts content, if we just start talking about Things like numbers and, you know, spreads, spreads of data points and things like that. But at the same time, I think it is really valuable to understand how to use this information because all we get presented with these days is like, what’s the data say? What are you? What are you doing with that data like, and it’s like, well, I don’t think most of us are stuck Titian’s. And I don’t think most of us really have a good sense of how to properly collected data if we want to use it for our classes, or for our own growth in our teaching. And so that’s kind of what I was hoping to talk with you about, there’s a couple other things that we could we could go wherever the conversation goes. But I guess I just think of you as like a data minded person, and I, I’m interested in how you see yourself and then also how, how you use what you collect for your teaching and for your, you know, your practices as a teacher,

Mark Makino
for my own. What’s the word I’m looking for, for my own? I don’t know, my own pure aesthetic pleasure of looking at lots of numbers. In addition to my career, it’s just a hobby I enjoy aesthetically. Yeah, it’s it’s odd that I have this predilection, this is sounding so twisted, we’re just talking about numbers here, right. And I ended up in this career, which is not a very numbers, heavy career. I mean, there are people who do study, quantitatively teaching in all its forms, including language teaching, but it’s not a common personality, I want to say defect among ESL teachers at large, you know what I mean, right. So I always am able to find kind of a niche doing that, me and maybe one other people in any given department. And there’s, you know, at IVC, there was me, and maybe one or two other people who really liked doing this kind of thing. But usually, it’s just in the humanities, in general, it’s something people do in their grad school, and then probably never touch again. Right. And I don’t know when this started, but like, sometime in grad school, I just figured, you know, I should be doing this more regularly. When I was collecting data to write papers for a master’s thesis and, and other papers during grad school, I just thought I this is a way of looking at the world that’s less prone to leaving you fooling yourself thinking something is true when it’s not. So I just started doing it sort of habitually then collecting data on all my classes, and then just doing pretty basic statistical analysis, just to see if the things that I thought were true, were actually true. Or if there were any surprises that I wasn’t noticing, just on my, you know, first person recollection of the class, I just taught where I was teaching.

Brent Warner
Yeah, so let’s talk a little bit about that. Because you just wrapped up a class or two, I guess, and you kind of have done some stuff, maybe recently. So let’s just get some examples. Cuz I think people might be going well, wait, what are we talking about? Including you, Ixchell

Ixchell Reyes
I know. I mean, I, I have I love spreadsheets, and I like numbers and data and sorting things, but I’m not quite good at looking for patterns. So I’m actually very curious as to how you use this data or what when you weren’t talking about this what what it actually means for the teacher, I guess, like, explain it, like a five.

Mark Makino
You know, it’s like, you probably tell your students when it’s when they’re writing, like you can start writing and get the ideas, as you’re writing is sort of like that, for putting things in a spreadsheet. And just, once you have a habit of doing that, you don’t even have to have a hypothesis. To start with, you’ll start looking at the data and being like, Oh, I wonder if this assignment actually predicted final grades as much as I thought I did. But once you have the habit of just dumping all your classes, so I use Canvas, and canvas lets you export grading sheets as Excel files, or CSV. I don’t even know it’s CSV, but it’s compatible with Excel. And Blackboard lets you do the same thing. But you know, I started the habit of doing that back in 2015, or 16. And I’ve just done it ever since, regardless of whether I thought there was anything to look for. And half the time, I’ll have an idea, I’ll think, do the final drafts. Let’s see, were the rough draft grades and the final draft grades correlated with each other, which is a question that I often have. And then other times, I’ll just do it out of habit, and then not even look at it for a couple months and then come back later, or even at the end of the whole academic year and look across the whole year and look for patterns. But you know, sort of like writing it’s having the inclination to sit down and do it in the first place is sort of maybe more important.

Brent Warner
Yeah. So let’s talk a little bit about what this process is. So if you’re saying hey, I’m going to export grains from Canvas, right? Here’s all my, here’s all the grades that I get what types of So by now you have ideas of what you either have found, or what you might tend to want yourself looking for, with a collection of grades, what would you? What have you learned just by looking at a big collection of grades from all of your students, right?

Mark Makino
So I get a lot of mileage out of just four, maybe five functions on Google Sheets, which are average, standard deviation, correlation, t test and filter. And the ones that I named I think are 90% of what I use on any Google Sheet.

Brent Warner
And I have pozzi there, because

Ixchell Reyes
I know I know what average is. The other ones I don’t. And again, I love Google Sheets, and I do a lot of stuff on there. But I’ve never actually gone back to use any of that. I know the filter. Well. Yeah,

Brent Warner
this is interesting to me. So like, I am interested, because like, standard deviation, for example, is something we all hear when people like when we listen to NPR. But do we know?

Ixchell Reyes
Or read of those research papers…

Brent Warner
Right, right. And so like, I’m always like, Well, hold on a second, like, at what point do I know if this is like, you know, we hear people talk about things like statistically irrelevant, or statistics, you know, like, and I’m always like, well, at what point? Like, where did they decide? Is it like a point? Five 2.6? Or is it a 1.3? Like, at what level? Do we set? What do we understand statistics to actually be relevant? Are these numbers to matter? Or, or B, correlation or causation? Right? I understand correlation and causation as ideas, but I don’t totally understand the other ones, like the T point or even, you know, standard deviation or what these things mean. So let’s, can we define a couple of these just to help us? Maybe anyone who’s listening could go, oh, if I know what this means, then I can start playing with it. Right?

Mark Makino
Well, t pain is a rapper. That’s not a statistic.

Brent Warner
I don’t have my voice filter to get my, my auto tune on.

Mark Makino
Right, right. I’m so I’m okay.

Brent Warner
Tpain was one of the things to

Mark Makino
T test. T tests. He hasn’t broken and he’s up and coming. He’ll be he’ll be. Um, yeah, some nerd is probably making a YouTube channel. That’s like statistical wraps right now. Probably you probably already exists. But But anyway, so. So if you know, you have, say, one, one variable that you’re testing, say, like the average temperature in summer? The mean is just, you know, the number? Well, let us say. So it’s all the numbers added together divided by how many numbers there were. So it’s basically a number right in the middle, right. And the standard deviation will tell you how far apart those numbers were from each other. Okay, right. So why it’s a big standard deviation means the answers were not similar to each other. If you’re testing something like summer temperature, in a city like that I’m living in now where it the beginning of summer, it might even still snow, some mornings, and then it’ll be like 100 degrees, sometime in August, the standard deviation will be will be a large number. Got it? Right. What else that I mentioned. So filter is just a Google Sheets function that lets you narrow down a big range of answers to just the answers that you’re interested in. So you could take one Google Sheet that has, like all the summer temperatures, and use the filter function to to copy those answers to another Google Sheet, but only the answers that are in August, or only the answers that are above 80 degrees or something like that. I’m in filter service. Yes, it filters. It’s a very well named function. Some of the functions are not very well named. But anyway. Let’s see what else that I mentioned t test, t test and correlation. Right. Okay, so so correlation is between negative one and one, it’s how closely two, two variables are related. So if you have sort of a canonical example is ice cream sales and then temperature. So if you have if you test that the amount of ice cream that I give in store in a city sells throughout the year, and then you also you’ll have like $1 amount for that and you compare that to the temperature outside in the same city, you’ll find that those two numbers tend to rise at the same time and tend to fall at the same time. And we call that a positive correlation. Right? And basically, and this is the point that’s most gives you Using for people is is just between negative one and one. One is the highest possible correlation. And that means basically you’re looking at two numbers that are exactly the same, or two variables that are exactly the same, right? And negative one is just the opposite. It’s basically it’s called an inverse correlation. It’s when one number rises when the other one falls. So if you can imagine, like, number of boy bands and the top 20 versus average length of beard of the average man.

Brent Warner
I was gonna go chilly on a hot summer day. Okay,

Mark Makino
right. That’s a better example more intuitive. And you have to be a certain age to understand that boy bands have been, you know, they’ve had their heyday, and they’re gone now. But they’ll come back right when everyone loses their beards.

Brent Warner
Well, yeah, I don’t know how BTS feels about that

Mark Makino
means Yes, that’s true. That’s true. That’s true. So like, yeah, that’s interesting, isn’t it? So outsourcing a boy bands correlates positively with beard length.

Brent Warner
You’re gonna have to put it in the spreadsheet for me.

Mark Makino
And then t test is what? What most people refer to as a test of statistical significance. So you’ve heard that term, like, if you just read the news casually, you’ve heard of statistical significance before, right? It says like, you know, people who took the Johnson and Johnson vaccine had some adverse health effects, but it wasn’t statistically significant. That means that if you compare the people who took the vaccine to people who didn’t take the vaccine, the number of people who had that adverse health effect wasn’t enough to show that those two populations are different in that respect. Okay? Okay. Right. So it’s a way of telling if the populations that you’re comparing so like, students in one class versus students in another class, it’s, if, if you rent a T test, and you get a value that’s very low, it means those populations are actually different in the respect or are very likely to be different in respect to the variable that you’re comparing. And if the T tests, and this is unintuitive, if the t test is very high, it means they’re likely to be the same population. Okay.

Brent Warner
Okay. So I think I get the basics of it. I would need to dive into this to actually like, properly understand it, but I understand the concepts here what we’re talking about the broad concept seashell. Are you on board? are we are we

Ixchell Reyes
gonna say? Yeah, it makes me want to open up my spreadsheet. I like spreadsheets, and I like data. But yeah, it’s like, I want kind of want to go look right now. Yeah,

Mark Makino
I mean, all of us probably did something with these during their, regarding our master’s degrees, and just probably just by a little way, somewhere in our whatever part of our brain that we don’t touch anymore. Right?

Brent Warner
Well, it’s interesting, because I was thinking, you know, they taught us to do some of this stuff in our master’s programs are like, we had to do them for our, you know, for our capstone projects, or whatever, whatever we turned in right at the end. And we spend this time like learning and figuring out these things. But I wonder if they really expected teachers to be using this on a regular basis, or if it’s just because that’s how the education system is set up to test us to, for us to understand like, it seems more like it’s kind of setting people up to be researchers, which Mark that’s your hobby at this point, right. But also, but like for the vast majority of teachers, there’s still value in knowing how to do this, but we don’t do it right.

Mark Makino
I’m just at the cusp of like, I’m at the barrier between people who know very little about statistics and people who know like actual statistics, to the point where I can have a conversation with like people from my psychology department, but they very quickly overwhelm me with new terms. So I’m not a researcher by their lights and my my aunt, who is a real PhD linguist, frequently comments on my blog, that I am not using statistics quite the way that she would. And well, she points to a danger that is very real, which is over reading statistics and what they what they really are telling you over reading the numbers that you do get, but I mean, I tend to I’m like the I this is really a self aggrandizement comparison, I’m sort of the Black Sabbath of, of using statistics, I use a few statistics a lot, sort of like Black Sabbath uses, you know, power chords 135 a lot. And I get the most mileage that I can out of those.

Brent Warner
Right, perfect.

Mark Makino
I like using analogies that make my explanations more difficult to understand.

Brent Warner
Okay, so We get it. So how do you take this information then? So what are you using this for, like with your classes with your students? How are you actually applying it then? So we get some of these basic ideas on what these statistics might mean, or what these numbers are, how this data might get collected? But then what is the output for your What do you end up getting out of it?

Mark Makino
So, um, I mean, this, I really should have been the answer to an earlier question. But I have a Google Sheet Set up where the first page is just me copying and pasting whatever the output from the latest class was the export a grade sheet from Canvas, and then sheets, two through 10 are just computed automatically from the first sheet. So I have one sheet that I just dump all my, my class grades in, and it computes everything that I might be interested in automatically. And I shared this on my blog, but you know, there aren’t very many people who are interested in this kind of thing. But if you want it, yeah, I mean, the link is on my blog. And I guess we can put it in the show notes or whatever. I’m not allowed to say things like that. I’m not a host. I can’t put things in the show. No,

Brent Warner
we’ll put it in the show notes. For sure. Yeah. And you can also you can also just speak it out right now. So people are driving, they can go look at it on their phone.

Mark Makino
Wait till you get to a stoplight and then go to a future real is real.wordpress.com. Which is Yeah, obviously I’m not paying for a premium account future real Israel. It’s not a reference to Israel, it is real. So and then yeah, so let’s see that that’s, that’s that Google Sheet that you can download and make a copy of and just dump your own class grades into the first sheet after the first sheet. That’d be the second sheet. Yes, the second sheet, I forgot the word. Second, thank you. That just figures out correlations of every assignment that’s in your in your grade sheet with final grades. And it gives you all the positive, the very high correlations and the negative correlations. And again, a correlation a negative correlation isn’t something that you would normally expect to find in your grade sheet because it means that the final grade was going down when the grade for this assignment was going up, right or the the inverse, right, so there’s grade for this assignment going down, is associated with higher final grades in the class. And you don’t ever expect that to happen, because it’s not like the Mad Magazine board game, or you’re trying to lose points. And that gets you a higher final score. Good reference mark, that’ll bring in a lot of listeners. But, um, that does happen sometimes. And it’s interesting to see when it does in my class is most commonly it’s extra credit. That gets negative correlations with final grades. And that’s because people who have low final scores are more likely to do the extra credit assignments. So it winds up being that students who have low final grades are more likely to get more points for extra credit assignments. But not always. And just as often, in other classes, I will find that extra credit is correlated with high final grades, which means it’s not serving its function in the function that I designed it for. For extra credit, I wanted it for students who missed a class because of some kind of emergency or something like that, to be able to make up points that they last sort of unavoidably, but what it ends up being used for. And I found this out by looking at my Google Sheets by students who have like a high B and are trying to get an A or have an A minus and are trying to just get an A to you know, sort of nickel and dime that way up to a slightly higher grade from an already high grade. So across all of my classes extra credit is actually not correlated with final grades at all. Because there are just about even amounts of those kinds of students students with low grades trying to make up and pass the class and students with high grades we’re trying to get even higher grade. Okay, I guess you guys have probably anecdotally noticed this in your own classes too. Like you give like an extra credit tutoring session and you expect it to be the people with like low SES were showing up but it’s actually people with a minuses.

Brent Warner
Yes, yes. Yeah. Yeah, like why are you here?

Ixchell Reyes
This extra credit Oh, but I really want I don’t want to be plus I want the A for the eight nine is I’ll take Where can I turn my A minus into the A because you know, the minus horrible next it’s a bummer.

Mark Makino
Yeah. Yeah. So I mean, I that’s one sort of hypothesis that I had when I was looking at my grades over the last year. Yeah, the last year, but it turned out to be it was serving both functions but I don’t give as much extra credit now because I I would rather just give the students who are not passing the opportunity to make up old work by just turning it in. Rather than giving them a zero for that and then giving them this sort of you know, nickel and dime me extra credit assignment which is likely to be abused by I mean, not abused by but you know, used for the purpose that I didn’t design it for by people who are already doing well,

Brent Warner
but I like this because now you’re, you’re saying like, Hey, I have a legitimate reason for the the pedagogical shift that I made for, for my classes or for offering these types of things. So it’s not just like, I don’t want to offer extra credit because it doesn’t go to the right students, right, you can actually say, Well, look at all this information, this is why I made this choice, I can make a better choice, instead of this to say, you know, it’s, it is an option for you to make up a single assignment that you have failed, or that you didn’t turn in properly, or whatever it is, then it is to go and do some other potentially less related or, you know, kind of like, you know, not to say that you’re doing wrong, extra credit assignments, but it’s like, you know, oh, go, you know, some teachers would do like, go watch the movie and report back on this, like, Well, okay, how is that gonna help me with my writing class?

Mark Makino
You know, that’s exactly what I was doing, though. Yeah, I mean, I had a bunch of assignments they could choose from one of them was like, you know, talk to someone and write about your conversation. One of them was like, you know, leave a comment on YouTube copy pasted here, something like that. And it’s sort of it’s tertiary Lee related to the learning outcomes of the class, but not as directly as the original assignment that they missed. You know, I, I tend to have someone in my mind that I’m explaining these curricular changes to when I’m making them. And I always in my mind, I always have to have a good reason for making them. By the way, when I was at IVC, it was you that I was imagining, like, I find changing this in my syllabus, I have to be able to explain it to brain why I’m doing this. But you know, it’s sort of just finding myself and then this imaginary other very strict supervisor that I have not that Brent was

Brent Warner
about to find out. Yeah,

Mark Makino
I like to know that I’m not just misremembering what the patterns actually were in assignments being turned in, you know, in a class, because you do tend to remember some assignments better than others, or some students work better than others work for reasons that are not necessarily related to their final grades. I, you know, I like to use the Google Sheets to make sure that I’m not just, you know, selectively remembering as especially not selectively remembering in order to make the planning for next semester easier and to give me less work. Yeah.

Brent Warner
Well, I like that too, though. Because, I mean, do you feel that it helps you remove bias from your decisions? Or at least not remove? not eliminate, but like, maybe reduce? Or or that your, your decisions are more justified?

Mark Makino
I hope it makes them more justified, I wouldn’t say it’s made my decisions. Well, certain decisions are less biased. I mean, yeah, like the decision we just described had to be fair to, or I made it by consulting everyone’s grades, not just the ones that I remember. But let’s see, it can. Let me tell you another story. another statistic story, and it’ll be sort of a cautionary tale about how you can introduce new bias in your classes by maybe, let’s see, I would say reading statistics correctly, but, you know, people could disagree, I guess. But, um, so basically, I have a recurring pattern. That’s, let’s say, it’s, I started looking for these after moving to my current University, but you it’s, I don’t know how long it existed before that. But it’s basically that people don’t fail my classes for getting bad grades on assignments, they only fail for missing assignments. Right. And for me, this sticks in my craw a bit because I want grades to be valid. I want if you get a B, you have B English or written English or oral English, oral English or whatever, if you have an A, you’re better than the B student in English and oral English or written things or whatever, according to the learning outcomes that are in the syllabus. And if someone’s failing my classes because they play too much fortnight it’s a validity issue for my grades because that’s not a learning outcome in my syllabus, right? You have to have you cannot have friends in Taiwan who play fortnight which keeps you up all night because they’re in a different

Ixchell Reyes
light. Yeah, I know the stories.

Mark Makino
Right. And I don’t like this aspect of, of my, well, I don’t like this pattern that emerges in my grades and I don’t know how to get rid of it. But you know, people fail my classes more because they miss assignments and get zeros then because they turn in assignments that are not good. Right that don’t meet, you know, the objectives of the class. Right. And I I’ve been looking for this. Well, I’ve had evidence of this piling up over the last year and I especially was interested in it because of remote teaching. Like you guys, we started doing all remote classes last spring, and carrying through fall now, less, let’s see this spring, they were sort of half remote half in person. But anyway, it turned out that. So two very interesting findings for me was one, the number of zeros almost doubled when we went to remote instruction. Okay. And I don’t know if you guys have anecdotally noticed this in your classes too, but people are less likely to turn in assignments, if they don’t have to hand them in on paper, or if they don’t see your face, like in person. It would be weird if a teacher were not turning on their camera on zoom. So yeah, I mean, see your face in person. And I don’t know what causes it. But basically, in remote teaching, the number of zeros doubled. And in my classes, getting a certain threshold of zeros, basically guarantees you’re going to fail. And I mean, that’s statistically not in terms of the number of points that zeros would necessarily lose you. But in my classes, if you miss 10% of your assignments, statistically, you’re in a group of people who has an average grade of D. Right? Okay. And if you don’t, if you fall above that threshold, I mean, sorry, below that threshold, so you get zeros on less than 10% of your assignments. Statistically speaking, you’re, you’re in a group of people whose average rate is B. B as in boy, right? So 10% of assignment seems to be a very key threshold in my classes, for some reason, and I don’t know exactly why. But people who miss 10% of classes almost Oh, sorry. 10% of assignments almost always fail. People who miss less than 10% almost always pass.

Brent Warner
That’s interesting. Okay. Yeah. And so so I can see your life. Interesting. thing. Go ahead, Brent. I was Yeah, I was just thinking like, cuz now you’re saying, Well hold on a second, just doing your work. doesn’t necessarily mean that you have the language skill that we’re looking for. But you’re turning everything

Mark Makino
Yeah. Right. It’s, it’s frustrating for me, because let’s say you would hope that the difference between a B and A d student would be that B student is better at English than the D student. But it turns out that that’s at least not directly showing up in my class grades. And I don’t know, you know, what other teachers might be having this problem, because we don’t all go looking for these in our class grade sheets, but I’m getting the feeling it’s it’s probably a pretty common problem. And I mean, realistically, it’s common in every college class, because, right, it’s never in the learning outcomes. Yeah. But it was interesting to me that 10% was the dividing line. Because, I mean, theoretically, you could miss 10% of assignments and still get 90% of the points in the class. Right. But that doesn’t happen. And then the last thing that I noticed, and let’s see. So I, the last major blog post that I made about these kinds of statistics was just looking at where the zeros occurred. And if zeros that happened, at particular points in the class were especially meaningful. And basically any zeros during the first third of my class is correlated with very big drop in grades.

Brent Warner
Any zeros meaning like a student doesn’t turn in a single, like anything, even even only one assignment is missed.

Mark Makino
Yeah. So like, if they missed a, like, you know, kind of a throwaway worksheet, like a classwork worksheet, like they watched a TED talk in class and wrote a reflection in class, and just didn’t hand you the assignment at the end. I mean, that’s maybe a cherry picked example. But any zero on any assignment in the first two, two weeks, which is the first third of my classes, my classes are only six weeks. Statistical statistically drops your average grade by something like 20 points.

Brent Warner
Right. Interesting.

Mark Makino
It’s… interesting.

Brent Warner
Like you kind of have to process that because it’s like, you would not imagine that right? Like, that’s something I would probably only pull out of getting that data, right.

Mark Makino
Yeah, I think so. I mean, I looked for it, and there it was. So if I had looked for it, I probably wouldn’t have found. Well, that’s, that’s the good tautology. If I hadn’t looked for it, I probably wouldn’t have found it.

Ixchell Reyes
So you’re talking about individual or things that are individual assignments and how that affects a student’s grades or what you thought would happen, but what about group work? What is your What have you found there?

Mark Makino
So this is another thing I found just by looking at the statistics sheet that I produce at the end of every term, but basically group work is a lot more likely to be completed. And so keeping in mind my my own Earlier warning about over reading statistics, I don’t know what the solution to this problem is, I can’t just make every assignment group work because there’s sort of a validity problem associated with that. Um, you want to know that your students can also work by themselves. So you can’t just make every assignment group work because they’re more likely to be turned in. But basically, group work is a lot more likely to be turned in. And over the last year, I’ve had a couple of instances where there were students who I pretty sure checked out by week to like, they decided they weren’t going to pass and we’re going to do any working to make any effort turn in the rest days, any of that stuff. But they would still show up to group discussions, which the students held on their own time in my classes, even like the last week of the term, when they were, you know, well below the threshold where they could possibly come back and make up and pass the class, they would still show up to group discussions, and just talk to their classmates. Yeah, and even into survey

Brent Warner
why, like, what’s going on? I’m interested, because it’s sometimes I get that too. I’m like, there’s no way you’re gonna pass this class. And then they just keep on showing up and sitting in the class and listening in. It’s like, I am happy to have you if you think you can get some value out of it. But I’m like, I don’t know what the value is to you.

Mark Makino
Yeah. I mean, it’s purely on a human level, I guess, like the

Brent Warner
socialization aspect.

Mark Makino
Yeah, they want to still feel like they’re part of the group. And you know, those, those students can end up being pretty disappointed when they don’t end up passing it going on with their cohort to the next level. You know, it’s interesting, it there’s a bit of irony, because I, as the teacher watch their discussion, and I know that those people tend to be quite lively in discussions, and their classmates don’t know that those students, I don’t know if we’re going to keep this in the I don’t want to make a FERPA violation. Those students? Yeah, no names. Yeah, there’s a bit of irony in Washington, because they’re often kind of oblique personalities, like they’re bringing everyone up to a higher level in the discussion. And no one else in the discussion besides me, the teacher and that student know that they’re not actually passing that class.

Brent Warner
Yeah, that’s, that is an interesting thing. I had a conversation recently with a student who was quite clearly failing from my, my, you know, viewing of it. And I assumed that the student also understood that right. And so then I went and talked to them. And I said, Well, hey, like, I’m glad you’re here. Like, I’m glad you’re doing all this stuff like, and do you recognize that you’re not likely to pass this class? And they said, Yes. I know that. And I said, Okay, then like, you’re welcome to participate as long as you want for the rest of the semester. You’re, you’re showing up, you’re doing things, but you’re never turning in assignments. Right. And then we had that conversation. And then the next class, they never showed up again. So it was like, it was, you know, so yeah, so I don’t know how aware they actually were of their status at that point, either, I guess.

Mark Makino
Yeah, there’s, I mean, it’s a very small group of students. But everyone, I mean, you remember students like this, Ixchell I’m sure, you know, students like this too, right. They just, they sort of dedicate themselves to the group that they’re with more so than their academic concerns. Right, they’re really big in participating in the, you know, the community of practice that emerges in the classroom. That’s all my TESOL buzzwords used for the day. But they’re not, they’re not dedicated to, you know, getting a high grade and then applying to transfer to a university or anything like that, or they don’t seem to be or at least that’s not where they put their, like, their, their eggs are not in that basket, in terms of their dedication to the class.

Ixchell Reyes
I’ve had students in the past higher ed students who told me, I’m coming to the class because otherwise, you know, their visa will be jeopardized. But I’m not going to complete the essays. So therefore, they did all that they, you know, partook in the group work and all of that, but nothing like drafts or anything for peer review days or anything like that was ever turned in. And I guess it was less work for me. But it also sort of, you know, it doesn’t feel good as a teacher to know.

Mark Makino
Now, and it’s, it’s funny, there are some middle you in the ESL world where teachers and students will be very upfront about expecting a class to be like that, like, everyone here is just here for a visa and they’re working under their table somewhere for cash, illegally, right. And if you can manage to turn around a class like that and get some real academic good discussions out of it, you feel like an amazing teacher. But yes, most of the time. I don’t know. It’s it’s the expectations are clear all around that. It’s not Going to be an academic endeavor.

Brent Warner
So, there’s a lot to kind of consider here. And I want to just mark because you have an expertise in one other area that’s kind of related, which is the corpus and COCA. In particular, I want to talk just a little bit about this. Because I think it’s really useful for some teachers that maybe don’t understand or, or haven’t applied, using it to their classes. And the reason it really fits in with what you’re talking about, because you’re saying, Oh, well, all these things you learn in your master’s program, and then you can just kind of you say goodbye when you start to actually teach. But coca is like straight up. Let’s learn how to use this thing in the master’s program. I had a, you know, a whole unit, like subdivision class on it, and all these types of things. But you actively use it with your students. So let’s talk for a minute. First of all, what is coca? What is the corpus? And then how do you use it with your students?

Mark Makino
So if you Google it, it’s the one that’s not the plant and not the soda. It’s a corpus, which means it’s a collection of real world language use data collected as texts. So the they collect from textbooks? Actually, no, no, sorry, not textbooks. Because textbooks are copyrighted materials. So they can’t use that. They collect from, let’s say, news websites, newspapers, not

Brent Warner
real language.

Mark Makino
Well, I think they’re someone I met at an IT SO conference was making a corpus of textbooks like and looking at them for academic language, but he couldn’t make it publicly available, because all of the data in the corpus was copyrighted by the peer center, you know, somebody else. Anyway. So corpus Coke, coca collects from American sources that are newspapers, television, fiction, TV and movies, academic journals, and a few other categories, and makes them easy to search. So when you do a search on coca, you’re basically doing a Google search for instances of whatever you look for in that real world language data. So if you search for a word like crunk, you can see that it’s used more often on television than in academic journals. If you look for a word like, heretofore, you’ll find the opposite. And you’ll see exactly, you know, at least out of the data that’s in the corpus, how many times it was used, what percentage of where it’s like, out of sorry, out of a million words, how often it occurred. So it’s very useful for sort of supplementing a dictionary because it tells you everything about a word except for the meaning. Right? So it doesn’t tell you what the word here to four means. But it does tell you what kind of sentence they use it in, and what kind of punctuation follows it? Or is followed by it? And then, of course, you know, is it a very stuffy, epic Academic Word? Or is it a word you can use with your your friends and your host family?

Brent Warner
Right? And so when students access this, like, what kind of activities are you doing with your students to use it?

Mark Makino
I mean, I always introduce it in class, preferably in a computer lab, or if everyone has a laptop, then that’s, that’s fine, too. And we do the first few searches together. And that’s where I explained to them what it is how it’s similar to a dictionary, how it’s different from a dictionary. And we usually have one or two problems to solve at the end of that first introduction. Like for example, I will have them let’s see, find the most common adjective after looks. Right? So looks good, looks sweet, looks nice, etc. And that’s the kind of thing you can find on on in by searching a corpus like coca, you can’t find in a dictionary because the dictionary doesn’t tell you collocations right. collocations, meaning the words that often occur with other words, right. And then another one I have them do is which one is more common, despite that, or despite the fact that and there I’m specifically telling the look under academic because, let’s see, it’s a chunk that occurs almost exclusively Well, not in almost exclusively, but very, very commonly in academic writing, and then not very commonly in like spoken English or on television. And they have to solve those puzzles on their own. And when they saw them all, then I feel like they’re ready to do their first, like, real COCA assignment. But after the first week’s homework, or so the only way that I use coca in my classes is by having them review their own writing. So in sort of keeping with this ongoing movement, to have more writing classes be kind of workshopping where you’re doing more review in class peer review, trying to make the revision process more aboveboard, like giving special tipping points to specific aspects of the, the you know, the self editing process the self. What’s the word I’m looking for the process of revision, right of drafting and revision, I give them an assignment between their first and final drafts, that’s just look for the turns that I highlighted and put a comment on Google Drive, Google Docs and coca, find the problem with them, and write what the problem was, and then change it in your essay and then change every other similar error in your essay. And they get points for just using coca to accomplish that. And that’s, let’s see, it’s sorted out for me as a way of addressing the fact that you, you want to address grammar, still in an ESL class in a world in which Grammarly and Google Translate still exist. And you can’t in any way verify that they’re not using those. And you know, some teachers think that you shouldn’t try to stop them from using them at all. Right, but I want them to have some kind of active involvement in their grammar revision besides looking for the blue or red underline on Google Docs, and right clicking the first suggestion, or running everything through Grammarly. And Koch has really the way I do that. And it sort of addresses issues that aren’t commonly taught in grammar classes anyway, like colocation. So I find it a useful sort of consciousness raising exercise, partly for that reason, too.

Brent Warner
I like that too, because it lets them kind of focus in for themselves, like, Hey, this is an area where I can be more academic, or I can, I can upgrade my level to the next, you know, the next step or something, but it can be customized for each student, right? If they learn how to use it as a tool for themselves, rather than like, okay, here’s the, we’re all going to look up the same form every single time, right? It’s like Now hold on a second, you can go in and do searches for what the language you’re trying to use, and then do multiple instances of it after you understand it better.

Mark Makino
Yeah, it puts it a lot more in their hands. And it works also for other genres, too. So you want them to write a short story, not an academic paper. You can help them choose more genre appropriate language, say they’re writing like a sort of Lord of the Rings style, where they use words like tone and scroll. And what’s another sort of the rings the word? alien? Yeah, I wonder how often that’s in the corpus. That would actually would probably be more under academic, I guess, because people talking about Tom bombadil. But anyway, yeah, it’s very flexible that way. It gives them a choice. You know, it doesn’t say this word is, this word is wrong. It says this word is, you know, not genre appropriate, or you know, not saying not, let’s see dressing up your essay the way you want it to be dressed up, like you’re putting it in a straw hat and a and a sari, where you want it to be in a tuxedo or something like that, you know what I mean? Not that saris are informal or something like that and saris an interesting combination.

Ixchell Reyes
Yeah, you know, you mentioned that, and I remember, especially students who were, maybe they were used to getting very high scores on on English tests overseas, but then they came here, and they had memorized all sorts of word lists, but in their writing, they were putting them in the wrong place. And they would tell me, I swear, my teacher, my teacher told me this was a real English word. And it’s like, I’m not arguing that it’s not an English word. Let’s take a look at the Copa and let’s see how often it appears and where, where it appears. And so then they could see for themselves that no, I’m not trying to convince them otherwise. Yeah, and, and it’s a really good habit to instill in them. Because if you if they’re not sure how to use it on their own, they wouldn’t they wouldn’t go out and analyze, well, what type of word form follows this? This comes before it, what comes after it? What can you know, those kinds of skills, but that’s how I used to use it. And I like that you’re having them write comments to figure out what, what’s wrong with it after looking at cocoa because that definitely gives them more onus over their editing.

Mark Makino
Yeah, it’s, it’s two things that you said I really like one is it? It’s very good at addressing things that a traditional, you know, I’m putting up scare quotes here, grammar, education does not cover like colocation, like genre, sorry, like register or genre. And then also, you know, I’m very big on making them show their work as far as the changes that they make, because Coke is like, like we all acknowledged, not that intuitive to us, and it’s capable of giving you some sort of red herring type answers. So I really want them to show that They understood what they were seeing, and then helped me understand what why they made the change that they made given what they saw.

Brent Warner
There is a ton here to think about I think we’re gonna let people just float and think. Yeah, that just digest.

Mark Makino
Yes. In a bomba DeLeon fashion after coca

Ixchell Reyes
it’s time for our fun finds and my fun find this time around is some is a tool that we had talked about before but now I’ve actually gone in and really gone in depth with it. It’s writable we talked about it in one of our episodes. Yeah. And and now that I’m teaching this TOEFL class, and we don’t we don’t have an LMS. So it’s really, really difficult to have the students that don’t have a lot of experience you using Google Docs or opening links I actually have it’s a it’s a wide range and abilities and I don’t have time to teach them. So I was just kind of had a headache on when I was thinking how am I going to have them turn in their, their writing samples, because it’s actually really frustrating for a couple of my students to just to learn Google Docs, they just arrived here from another country and everything is brand new, like everything technology is brand new to them. So I can already sense on the first week the frustration. So I went in, I remembered, hey, we’ve talked about writable. Let me go in and see how much of it I how much of the prompts, the writing prompts I can customize. Can I put directions in there? And is it something where the student doesn’t have to open another window to then send to me or upload. And I know that sounds to many of us because we’re techies and we we do that already naturally, where maybe it’s required by our institution. It sounds like it’s a second nature type thing, but not so for many students from other countries. So rideable was my solution, you can go in there, and you can put in the prompt, you can add a video of explanation if you want. So everything’s within one window, and all the student needs to do is log in, the teacher can grade it from there, and I just thought, oh, that just saved that just saved me because I was going I was already I was frustrated about how I was going to frustrate my students by trying to teach them this clunky version of you know, submitting your assignments when they’re they’re already under a lot of pressure to to pass a test. So writable. And I also found that the the free version lasts for I think it’s 90 days. And then you can purchase additional licenses for your students, but it’s $1 a month per student.

Brent Warner
If you can log in with a different email.

Ixchell Reyes
You could do that as well. Not that I’m adding to that support. No matter what I’m saying is $1 that’s the price of I have happened to have a small class. It’s five students. That’s $5 that’s cheaper than what my coffee would be daily, right? So if I need it for another month, I can purge out I’d be happy to purchase the licenses at $1 per student. So very, very useful. So writable

Brent Warner
writable cool. I went back to Trader Joe’s. Sometimes Sometimes my friend finds from Trader Joe’s and I got greenlit granola Have you tried this one yet? This is no I mean it’s exactly what it sounds like it’s granola without tasty It’s really good. It’s really good. And so they put it together in like chunks. I don’t know what that whatever they’re using as their binding agent, you know, like if it’s some sort of honey honey or something but but it’s it’s they’re good little chunks and it’s a nice snack. And then also what you can do is like I also crumble it over my yogurt in the morning and so it makes it a little bit more, a little bit more filling of a yogurt too. So if you’re swinging by a Trader Joe’s grab up some grainless granola, it’s very good.

Mark Makino
What is the what is the filler if it’s not grains?

Brent Warner
Uh.. sweet and salty clusters made with almonds, coconut sunflower seeds and pumpkin seeds. It’s mostly like nuts and seeds.

Mark Makino
So it’s kind of like a high protein snack. Yeah.

Ixchell Reyes
So basically it’s a nut cluster called the granola.

Mark Makino
It’s a rebranding. It’s good. Successful rebranding. Yeah, you reminded me Actually, I was going to talk about a video game, but I’ll talk about something else instead. I let’s see. I’ve got way more students now from the DRC. Not the Democratic Republic of China. That’s not the DRC is it? It’s Congo. Yeah. DRC, Congo, Congo. And then there’s another Congo. Did you know that? There’s a whole other Congo and Right across the river. But anyway, most of my students are from the Dr. Congo. And then a few are also from the Ivory Coast. I have a lot of students now from Francophone Africa. And they introduced me to foods that I had never heard of, let alone tasted before. Did you get some ones that I didn’t? I haven’t had Well, I don’t eat chicken. But the beauty part sounds okay. So, but sort of related to that, that one of my students from Ivory Coast introduced me to a dish called sauce other sheet, which is basically peanut sauce, but it’s not like the Thai peanut sauce. It’s, it’s a very rich savory. It’s like, I mean, this is probably a Philistines rendition of it. But it’s like, if you can imagine French cooking with a lot more peanuts in it. It’s really, really good. It’s very heavy. But it’s, it’s very tasty, too. And it’s, it’s, it can go with a lot of things. I think it’s usually served with chicken, but I just, I think I cook, I cook it with eggplants, it’s still really good that

Ixchell Reyes
is that something they brought from their home country, or you could easily find it and they don’t share with me the stuff that they bring, because it’s too valuable.

Mark Makino
But they told me about it. And I looked it up and I made a recipe from Epicurious or something. But it was still good despite the fact that an obvious non expert was making it But yeah, I recommend trying it and how they sometimes sauce arashid. I think arashid just means peanuts.

Ixchell Reyes
I have to look it up.

Mark Makino
It’s French but the dish is Ivorian. Yeah, but um, man, it’s, it’s a good way to use a lot of peanut butter. And if you if you’re like me, you go to Costco every couple of months and stock up on peanut butter. And then you think I’m not gonna eat this many sandwiches. You know, good way to use the SS sheet. Okay,

Ixchell Reyes
I got it. All right. Thanks so much for listening to the show. You could win a one of a kind DIESOL pin by leaving us a review on Apple podcasts. And if you’re giving us a shout out any other way tag us on social media. I don’t think we had any recent ones. And yeah. But Mark, you’re getting one for for being on the show. And I think you’ve you’ve won one before. I think you’ve got like a collection now. Maybe? Yeah, we’re not these drinks with DIESOL. That’s right.

Mark Makino
I got some there in my office at school, actually. So I haven’t seen them in a while.

Ixchell Reyes
Well, now you need one for home. All right. Leave us reviews. Yeah, we’re

Brent Warner
on Patreon if you want to support the show $1 $3 $6 or you can support it at $0 just by listening. You can find the show notes at DIESOL.org/42. And you can find us at voicED Canada. That’s voicEd.ca. We’re on Twitter. You can find the show @DIESOLpod and you can find me @BrentGWarner.

Ixchell Reyes
You can find me show at @Ixy_Pixy that’s I x y underscore p i x y and, Mark?

Mark Makino
@futurealisreal

Brent Warner
And we’ll also make sure that there are links to your blog and to all the things that you’ve talked about in the show notes as well.

Mark Makino
In Japanese Thank you is Arigatou Gozaimasu so Arigatou Gozaimasu for tuning in to the DIESOL podcast.