Evaluations, workshops, podcasts, tweets, mobile apps, concept maps and Christmas Carols. What do these have in common? One of British Columbia’s most interesting evaluators!
Back in September I had the opportunity to talk with Kylie Hutchinson of Community Solutions Planning and Evaluation. I was familiar with her popular Adventures in Evaluation Podcasts with James Coyle, and her famous evaluation Christmas Carols. Most likely, my introduction to her work was through one of her many aea365 posts. Kylie has been a frequent contributor on topics ranging from needs assessments to effective reporting techniques to presentation tips. A recent aea365 post introducing two new resources particularly caught my interest – a newly released Evaluation Glossary Mobile App, and a Working Typology of Evaluation Terms – and I wanted to ask Kylie how these came about.
I had the pleasure of meeting Kylie in person at the AEA Summer Evaluation Institute in Atlanta, GA last summer where we both presented courses, and found with our shared interests in evaluation and instructional design, we got on swimmingly. Shortly after that, I approached her to ask for an interview to which, despite her busy schedule, she readily agreed.
On our scheduled day, we Skyped – she from British Columbia, and me from New York State.
*
*Sheila: Hi Kylie! I’ve explored your website, been a big fan of your Christmas Carols and love that you readily share resources. I’m fascinated with the podcasts, and wonder how those came about. I’m interested in what comprises most of your work, whether it’s external evaluations, evaluation capacity building, or training. I saw your really impressive list of clients and all the training you do and wonder if training is built in to evaluation work, or always a separate endeavor? I’d like to know more about the glossary and typology, and finally, I’d like to know what is the most rewarding, enjoyable or energizing part of evaluation work for you.
Having asked Kylie essentially a barrage of questions at once, I just sat back and let her tell me about how her evaluation career evolved:
Kylie: I took my first course in evaluation back in 1988 and it was the only course on program evaluation in Canada. I just really enjoyed it – I really liked my professor and she kind of became a mentor of mine. I did program evaluation as my undergrad thesis. That was outside of Toronto. Then I moved out west and started working for a number of not-for-profits – I worked for an immigrant settlement agency and for a environmental organization – and whenever they found out that I had a background in program evaluation, they got really interested. They tried to pull me off of whatever project I’d been hired for and tried to put me on evaluation and I thought, ‘there might be something here’ and so I came back to evaluation.
For a long time I was just doing straight evaluations, and then I fell into training – probably around the same time that the term ‘evaluation capacity building’ started getting thrown around and I think I had a client who said ‘you know I would really like it if you could give some training to my staff so that some of this stays with them.’
There was a community college that I submitted a proposal to and I started doing workshops for them – all the time winging it with my instructional skills. The Canadian Evaluation Society has an Essential Skill Series – it’s like a 4-day boot camp in evaluation and I got hired to do that for about seven years in British Columbia.
You can probably relate to this, Sheila: I would go into those sessions before 9 o’clock in the morning and come out at 4 going ‘what the hell just happened?’ because I didn’t have that nice structure of instructional design to support me and that was really bothering me. So, I went back to school for an intensive course on instructional design, which was the best thing I ever did! The more evaluation training I did, the more I developed my own resources, materials, real-life examples, and case studies.
I got to the point that I was feeling pretty comfortable with evaluation training. I had enough materials, I had curriculum – I even held workshops for not-for-profit groups and things like that to the point where training was starting to take up about 30% of my work and I liked that, but it was never just for one organization. I would always take the initiative and I would put it on at a local college or something like that and then with the advent of all these wonderful event management tools, I started being able to run them myself.
I used to just wait for somebody to hire me but then I started running them myself and then it got to point where it was getting harder and harder to get people out to them – the face-to-face workshops. I’d beat the bushes and beat the bushes but I’d get only eight people. People said, ‘you know I’m just really busy.’ By then I had taken two webinars on my own and thought, ‘I’m going to try this.’ So, I don’t need to do face to face now; the only face-to-face workshops I do now are at conferences.
Image credit: patriziasoliani via Flickr
What I found was when I got that diploma in instructional design, my confidence increased. I had the structure and I knew what I was doing, so then I started mining my skills, my knowledge and my experience for evaluation topics because everything that I had done had been really well received. I noticed that after workshops – you’ve got this in your class right, Sheila? If you’ve given a really great class, students stick around and they come up to you and they ask you questions after, right? I realized these people are really interested in evaluation – they’re hungry for more information on evaluation and what they really want to know is what’s in here (she points to her head). They want to know what I’m thinking, and so I did two things – I started the Twitter account and started tweeting and I started mining all of my work for the things that I could share with other people.
By this point, I’d been in evaluation since 1988 and out on my own since 1997 and got the point where I realized, I’ve got a lot of experience in here. I’ve worked with enough organizations and there’s things there that I can share with people, so when I tweet I decided early on that I was only going to tweet once a week and I was not going to tweet another website, but what was going on in my evaluation brain at the time.
Image credit: Matt Hamm via Flickr
Sheila: What can you tell me about the resources on your website?
Kylie: With the resources it was kind of the same thing. For example, I have a resource on 30 ideas for building a culture of evaluation. That came out of some brainstorming I was doing with some students in a workshop. It was just an unformatted list in Word and I decided there was no point in having it just sit on my computer. Similarly, I did some research on what are the things that make programs more sustainable and I had this list of 34 factors and thought, ‘why don’t I put that together?’ It’s often from the training that I come up with resources for students.
I think I’ve been in evaluation long enough that I’m always kind of thinking, ‘what would help new evaluators?’ or students come up to me and ask me questions.
Image credit: klepas via Flickr
To emphasize, it all comes from the training – it’s either that ideas that come from questions that participants or learners’ have asked me, or I’m looking for a resource to communicate some kind of evaluation concept and I just don’t find it so I think let’s just develop it.
I’m developing two e-learnings right now: one for the public health agency of Canada on a basic introduction to evaluation, and then I’m also doing a mobile learning for senior executives and senior decision-makers in evaluation. The idea is that these guys are so busy they can’t go to a webinar, they can’t do a self-paced course, so we’ve got it on tablets that they can use when they’re there sitting in an airport lounge waiting for their flight they just kind of flip through this tablet and it’s structured for 2 to 5 minutes of attention span.
Sheila: And the glossary?
Kylie: The glossary idea came from a woman who did her own in instructional design and I saw it I went ahhh! We don’t have anything like that for evaluation! So, the first thing I did was -I knew that I wasn’t going to come up with all these terms, there’s no way – I went looking for online glossaries. I wanted to know what was there already and I found 3 – one with the OECD [The Organisation for Economic Co-operation and Development], one with the US EPA [United States Environmental Protection Agency], and one for the Evaluation Center at the University of Michigan, and I got permission from all three of them to combine them into one glossary. As I went through them I found that if I didn’t like their definition I didn’t use it, so for “logic model” I had three choices and I picked the one that I liked. Out of the 600, there might be 100 or so that I just added from other sources.
Sheila: How did you know when the glossary was done?
Kylie: It’s not done. I already have about 30 new terms for version two. I got an email today from a guy who said, ‘hey Kylie this is great, but I want to suggest these terms for the next version that are pertinent for people who working in the US Federal government.’ Wow, that’s great, so the more people that can send those in, that will really help. My hope is that someone will fund a multi-lingual version, so that somebody who is working on the ground in Angola can look up the Portuguese term for logic model or somebody who finds themselves in working in Columbia can say “evaluation framework” in Spanish.
So, have you shown them [she refers to my graduate students] the bubble.us typology?
Sheila: No, not yet, but I’ve added the evaluation glossary to the syllabus.
Kylie: That’s great. That’s really novel. The typology – that’s just me kind of lumping similar terms in our field together and there’s definitely people who say, ‘no, this term is distinct from that one’, and that’s fine. I try to put it out there that it’s just out there to promote discussion, and to show visually that this is what we’re dealing with here. That’s kind of challenging for people. I did a one-hour webinar on evaluation terminology. Maybe I should do it again. Just kind of trying to put their fears to rest – put their minds at rest about it. Just kind of explain that this is the way it is.
I’ve been in the field long enough where, I’m still obviously really learning, and that’s one reason why – and I think you’ve probably had the same thing, Sheila. I love teaching program evaluation because it keeps you on your toes. You have to have your fundamentals down, but you have to be really on top of what’s changing, and what’s trending. I think I’ve been in it so long, I can identify when there’s a gap- when there’s no practical resources out there in this. I really find the training and thinking of resources a diversion from writing final reports.
Sheila: So, with all this going on, what do you find the most rewarding, enjoyable, or energizing about evaluation work?
Kylie: I really like the way you put your question – what is the most rewarding, enjoyable or most energizing. I love meeting with the client, I love scoping out the evaluation, and I love – I really love scoping it out and thinking up the framework and how we’re going to do the data collection. The data analysis? I’m OK with that as well. But the data collection – meh, I really don’t need to do another focus group…and writing the report? Ugh!
That’s how I fell into the workshops on effective reporting. I’m always asking the client if they want a traditional final report or can I do a more practical two page summary instead? I hate writing the final report, I’m always praying they’ll say no.
I’m still doing evaluation but the breakdown is always changing. It’s anywhere from 50-70% of what I’m doing.
Sheila: How did you get started podcasting?
Kylie: I had this colleague who is in a different part of British Columbia. He works about 4 hours away – James Coyle. James and I met because he hired me to do some evaluation workshops in his organization about 10 years ago or so, and we just always really enjoyed getting together and talking evaluation. James is a lot more restrained on the podcasts but he’s an extremely funny guy. Extremely. I think I got off the phone with him one day and we were talking about something very simple but it went on for 45 minutes, and I thought, y’know? Wouldn’t it be cool if James and I could do this podcast and people would have a chance to listen to two evaluators who like their work and work in different areas – internal vs external?
It’s casual- it’s two evaluators kind of standing by the water cooler, just shooting the breeze, talking about whatever’s trending or challenging or whatever’s going on in their worlds right now, and so I asked him. James is actually an avid podcast listener and he was really excited and so we started. He does all the technical stuff and I just kind of laugh at his jokes, and it just seems to work really well.
Although we’ve gotten a bit diverted. Originally the intent was just the two of us, ‘Hey James, Hey Kylie, what’s up? What are you going through his week?’ What we didn’t anticipate was that there are so many interesting guests out there. So, all of a sudden, every podcast is an interview with somebody. I think we want to bring it back to a little bit of both, because we want people to hear two evaluators talk about what are they struggling with that week or not. We have limitations – I’ve got confidentiality issues with my clients and James has to be discreet with his organization as well. We want to try to balance that out a little more. But it’s going really well. We’re over 30,000 hits on the podcast site. The reason we went with podcasting, A) it was my idea, and B) I hate writing, remember that? I know people love to write but I can’t stand it. I find it painful and try to avoid it at all costs, so that’s why we went with the podcasting.
Wow, what a conversation! I learned so much and really enjoyed talking with Kylie. As an educator/evaluator, I very much relate to her journey as evaluator/educator. As an avid list maker and collector, I can relate as well, and I so admire her ingenuity with regard to the glossary and typology.
I conducted this interview in September, prior to Evaluation 2013 in October, where I found a session on case studies of evaluators’ lives. I guess I’m far from the only one interested in evaluator’s stories, and hearing Kylie’s story was a great start for me.
Sheila B Robinson 