Making a Change in your Course: Determining Effectiveness
Have you ever asked yourself, “How effective is a certain strategy I’m using, and how would I know?” or said, “I received some comments on my SRTEs that I’d like to address, but I’m not sure how to go about changing things so that it would make a difference.” “How can I effectively communicate (at important junctures) the scholarly effort that goes into my instructional decisions?”
Consider using an action research cycle to guide the design of the changes you want to make. The steps in the cycle give you opportunities to gather data about the effectiveness of your instructional decisions, so you have the confidence and knowledge you need to describe the rationale behind your decisions as well as data to describe their effectiveness.
Taking the time to contemplate your teaching effectiveness and having a process by which to analyze what you are doing, can be rewarding and confidence-building, and can be counted as another type of scholarly activity. The Carnegie Foundation for the Advancement of Teaching has been shaping and contributing to the body of work known as The Scholarship of Teaching and Learning (SoTL) since the late 1990′s following the publication of the Boyer Commission’s groundbreaking paper, Reinventing Undergraduate Education: A Blueprint for America’s Research Universities.
This semester, participants in the Course in College Teaching are implementing mini action research projects in their classes. As we are working through our projects, it occurred to me that others might benefit from this process as well, so I’m sharing it in abbreviated form here. This article contains an overview of the action research process followed by an extended example that applies the steps to a classroom scenario.
The basic process is this:
- Identify a question, problem, or challenging situation in a course
- Research possible strategies to address the issue
- Plan for implementation of the strategy (including what data to collect to determine effectiveness)
- Implement the strategy & gather information
- Analyze and evaluate for effectiveness
- Record your findings and reflections (go through the IRB if you plan to publish the results!
- Share what you are learning with colleagues!
- Adjust the strategy as needed for the next round… or… if it was successful…
- Begin the process again with a new question
Briefly then, action research is a reflective process that guides the implementation of a chosen instructional change, giving the instructor important information about the effectiveness of the change. In the next section, each step is presented in greater detail.
Identify a question, problem, or challenging situation in a course
- Be really focused and concise. An issue that reads, I want students to be motivated to learn, might be too broad. Ask yourself what lies behind that statement – Are students coming to class unprepared? Are they disengaged from discussions or group activities? Get to the root of the issue. Form your question at the underlying issues.
- Pick an authentic issue that you want to address. It should be interesting to you as well as have impact on learning. For example, don’t just add a new piece of technology for the sake of it – there should be a real issue/need driving the changes you want to make.
- Design with observable behaviors in mind. You need to get data to evaluate effectiveness, so as you think through your possible implementations, start to think about this. You want to implement a strategy that will allow you to “see” the changes you are looking for. So, if by being motivated, you really mean you want students to do more substantive work… then ask yourself what you can devise to foster and see that desired behavior.
- Ask yourself whether the question you want to address can be connected to your course objectives. While there are some things, like developing a positive classroom climate, that aren’t directly related to content objectives, we know that they impact learning in the classroom. So these questions are also legitimate to address… At the end of the process, you want to be able to say that whatever change you implemented helped students to reach the course objectives more successfully… just check the alignment – don’t spend time implementing a strategy that doesn’t help you to do this in some way – either directly or indirectly.
Research possible strategies to address the issue
Look through journals, books, and university teaching and learning center resources for tested ideas to address your issue. A good literature search can provide powerful information about what research and experience say are effective strategies. This information becomes just one part of the data you present to support the changes you make in your classroom.
- Many content areas have professional journals dedicated to the teaching issues of that content. Math, English, engineering, communications, foreign languages are just a few that come to mind.
- A list of content specific journals and resources
- There are also wonderful journals about teaching in higher education in general. International Journal of Teaching and Learning in Higher Education is one.
- Check with Barb Eshbach in the library for further assistance in finding specific journals.
- Many universities have wonderful teaching and learning resources already available – Here is an extensive list
- Look through some of these important texts for teaching in higher education:
- McKeachie’s Teaching Tips – Wadsworth (2009)
- Barbara Gross Davis’s Tools for Teaching- Jossey Bass (2009)
- PSU Teacher II – available at Schreyer Institute
- Classroom Assessment Techniques – Angelo & Cross – Jossey Bass (1993)
- Collaborative Learning Techniques- Barkley, Cross, & Major – Jossey Bass (2004)
- Engaging Ideas - Bean – Jossey Bass (1996)
- Creating Significant Learning Experiences – Fink – Jossey Bass (2003)
Plan for implementation of the strategy (including data collection for effectiveness)
So you’ve narrowed your question and you’ve come up with some strategy you think will be effective in addressing the issue… so how are you going to implement it and how are you going to continue to gather data about its effectiveness?
- Lesson Planning – start with a good general plan for the implementation. Examples of different models that could be used:
- Gagne’s 9 Events of Instruction
- L. Dee Fink’s Taxonomy of Significant Learning
- Basic constructivist lesson plan
Planning is key – What is your learning objective? What strategy are you going to implement? How are you going to implement it? How will you collect data? What data will you collect?
- Data Collection – Here are some typical types of classroom data that you could collect:
- Student surveys, interviews, focus groups
- Pre/post assessments
- Instructor reflection journals
- Student artifacts – homework, journal entries, other classroom documents
Resources about data collection
- Effective Use of Assessment Data in Action Research – Learning Sciences (2001) shared with permission
- Documenting and Evaluating Effective Teaching
Implement the strategy & gather information
So you’ve planned the session and created all the data collection instruments you’ll need. Now it’s time to try it out and gather the data!
Analyze and evaluate for effectiveness… and next steps
Take time soon after the session to jot down your thoughts about your initial observations. Overall, how do you think it went? How does it compare to other sessions? What differences did you notice? Now look at the data you collected. What is it telling you? Anything significant?
Take a day or two to let it all sink in. Based on all of the different kinds of data you collected: observations, student surveys, assessment or homework data, participation, your own reflection, literature searches – do you think the strategy worked? How do you know? Would you do anything differently the next time? Were there any gaps in the information you needed to analyze the effectiveness? How could you go about getting that information next time? Overall, what did this exercise tell you about your own teaching… about your students… about the strategies used? What are some follow-up or new questions you have about your teaching?
And the process begins anew…
Make a written record of your analysis – Don’t rely on your memory later on. Keep a teaching journal – electronic or hard copy to capture what you learned. Penn State is now using WordPress http://sites.psu.edu Consider sharing your findings with colleagues!
Consider how much more confident you could be (after working through this process) to talk about your instructional decisions and their effectiveness/impact on student learning! Not only can you discuss what you are doing in the classroom, but why… and how effective it is… and how you know that…
Having worked this process several times this semester, I can honestly say that it has made a real difference in my own satisfaction levels in the classroom and in the learning gains of my students. I hope you get a chance to try it out and use it to share what you are doing in the classroom. Keep me posted on how it goes!!!
What follows is an extended example to further illustrate the process.
Time for an example…
What does this all look like in practice? Here is an example…
You’ve noticed that students are coming to class and don’t seem really prepared with the readings you’ve assigned. They aren’t participating in discussions and group work activities are not producing substantive results. You think this also ties back into preparedness issues… so your action research question might be something like this: How can I design reading assignments that improve the substantive nature of student participation in class?
In preparation, you’ve done some reading in several texts and content-related journals about the subject. Based on your reading, you’ve decided to implement 2 strategies in the next unit: RATs (Readiness Assessment Test) and RAFTs (Role – Audience – Format – Topic)
RATs are usually quizzes that test students’ readiness to participate in the next step of a process – so in this case – readiness to participate in a discussion and to do other in-class activities – They need to know X,Y, Z before they can do the next step. You can allow multiple attempts in online quizzes so students really get the important points by the end.
A RAFT requires students to produce an authentic document – a memo, letter, web site, etc. that has to be written from a certain perspective to a certain group of people – related to your content area. See a more detailed description at:
For this class, you plan to assign a reading section for homework and you tell students that there will be a quiz at the beginning of class (or in ANGEL before class) = RAT. The purpose of the quiz is to help them get ready for in-class activities = discussion of the reading and construction (in group) of a related RAFT.
N.B. – the nature of the RAFT assignment should be something that connects to your course objectives. If you don’t see this connection, re-work it until it aligns.
So what kind of data can you plan to collect to see whether the RATs and RAFTs are effective?
- Observations – count the number of substantive (showing evidence of having done the actual readings)participations in the class discussions BEFORE you implement the strategy and then AFTER… look for meaningful increases
- Quiz data from RATs – Do you see that students did understand the material before coming to class?
- Group project – RAFT – Do you see evidence of more substantive participation in the group work/product compared to other group projects?
- Teacher journal – what is your perception of participation in group work before implementation of RATs/RAFT and afterwards.
- Student survey – do an anonymous survey asking students about whether they think the RAT and RAFT design helped them to achieve the learning goal – What aspects of each did they think were helpful/unhelpful. To what degree did they feel they were prepared and participating as a result of the design? Did the RAT help them to be better prepared to complete the RAFT?
- Literature search – you already did research about effective strategies – this is good supporting information in your explanation of effectiveness – the rational for your choice….
PLAN – Do you need to create anything to help you gather the data before the session? In this case, a survey, some way to gather your observations – using the seating chart with tick marks for each substantive participation in the discussion, the RAFT details – creating the assignment, a rubric…., a place to record your own reflections, data and analysis ( a blog, teaching journal, Word doc) some place to capture all of your work to share later (either in your FAR, in your teaching portfolio, or in your tenure packet) Remember, if you are planning to publish, you need to complete the IRB process!!!
Use a lesson planning model to support your entire session design.
Now you go to class, implement the strategies, gather the data and come back to analyze what you found. Perhaps you initially noticed more enthusiasm in the class. You notice that you had 17 substantive comments ( from 75% of class members)during the in-class discussion. The ANGEL quiz data (RAT given before the class online) showed an average class score of 85%, indicating overall readiness. In a previous session, you observed only 5 substantive comments and some uncomfortable silences or off-topic comments.
During the group work, you noted that all groups were working throughout the class period on the task assigned. Within groups, you noted that 80% of group members were on-task. After instructor prompts, this number rose to 90%. Students turned in their RAFT assignments at the end of the session and you noted the average class grade of 85%(based on the criteria specified in your rubric) indicating substantive completion as well as successful application of course concepts into the project.
The student survey indicated that most students felt confident of the material by the end of the completion of the RAFT. They also enjoyed the activity and felt overall better prepared for the class discussion. Students indicated a preference for being allowed to take the RAT multiple times before class.
Your overall reflection then might include information about your lit search on the strategies used, your findings, and your interpretation. Would you continue with the strategies, change them, or try something new and why? How did you, as the instructor feel at the end of the session? Was there a higher degree of satisfaction for you?
When you look at the overall session, you can determine a strong alignment with the attainment of at least one of your course objectives.
After this action research process is completed, you should have a more complete and detailed understanding of the processes happening in your classroom and therefore enough information to make a much stronger case for your instructional decisions.