How do you know if the student service initiatives at your institution are making a difference?
How do you measure the efficacy and efficiency of programs such as new student orientation, first-year experience courses, academic advising, and tutoring? When you do measure the impact of these services, how do you control for differences in personality traits, situation-in-life factors, computing access, and technical competency?
Gateway Technical College utilized data from the SmarterMeasure Learning Readiness Indicator to control for these extraneous variables. To quantify the impact of their tutoring program on course completion rates, retention, and graduation, they crafted a case-control methodology in a quasi-experimental design that controlled for these non-cognitive factors so that any improvement in outcomes among the tutoring group could be directly attributed to the tutoring intervention.
During our recent webinar, Dr. Mac Adkins, Founder and Chief Academic Officer of SmarterServices, sat down with Gateway Technical College’s Michelle Borckardt and Mary Xiong as they explained the effectiveness of their tutoring program, and how SmarterMeasure data can be used to help measure the influence of retention efforts.
Take a look at the webinar recording below or continue reading to review the webinar blog recap.
Resources from the Webinar:
- Slide deck: Follow along with the slides from the webinar or send the slide deck to one of your colleagues.
- Free download: Learn about what SmarterMeasure can do for your institution and about answers to some of our top frequently asked questions.
- Related blogs:
Webinar Recap
SmarterMeasure Implementation
Located in Southeast Wisconsin, Gateway Technical College is a two-year, open-access public college that actively supports at-risk students facing academic challenges. Currently, 52% of their students are economically disadvantaged, 18% are single parents, 40% are housing insecure, and 13% are food insecure.
The institution strives to provide student success resources and foster their personal and academic growth.
However, with declining retention and enrollment rates, they knew they needed to adopt a solution to increase retention. So they adopted a multi-pronged approach of offering a mandatory 1 credit first-year seminar course that allowed students to explore the Gateway Technical College community. In the course, students examine college resources and services, investigate skills that lead to academic success, and identify strategies for achieving educational and personal goals.
In this course, they also leveraged the SmarterMeasure Learner Readiness assessment.
The assessment measures several different scales. The first scale measures individual attributes. Those attributes include things like: 
- Motivation over procrastination
- Willingness to ask for help
- Locus of control
- Learning personality
- Identifying the student's learning style
This assessment also incorporates multiple intelligence models to help evaluate student learning styles such as visual, verbal, social, or solitary. Along with that, it evaluates crucial student life factors such as available study time, family support, health, and finances. Additionally, the assessment includes a skill set analysis covering on-screen reading, technical knowledge, and keyboarding accuracy.
By implanting SmarterMeasure within the first-year experience, many of their students took it as an assignment with a reflection. Within that reflection, instructors were able to evaluate how else the institution could help their students.
Key to Tutoring Success
Assessing students earlier on with SmarterMeasure helped obtain higher response rates and gain insight into their online readiness. This method allowed Gateway Technical College to address risks and establish effective referral systems which ultimately supported students on their path to success. Tutoring is one of those pathways.
At Gateway, tutoring is more than a service, it’s a valuable resource for their students. With professional instructors who oversee the tutoring centers, they can ensure that their students receive the best possible guidance to enhance their learning.
Their primary goal is to ensure the best possible tutoring experience for students. The tutors begin by assessing the student's goals for the tutoring session, which allows them to understand what the students aim to achieve.
They also provide empowering strategies to facilitate the learning and application of new concepts, including reciprocal questioning. With an emphasis on retrieval practice, they’re able to provide feedback centered around a positive, growth-oriented learning environment.
Click here to listen to the webinar and learn more about Gateway Technical College’s tutoring program.
Program Evaluation and Research Design
When it comes to program evaluation, it’s important to make sure that what your institution is doing is effective and makes sense. It’s also important to evaluate whether there are any areas that need improvement.
So when Gateway Technical College’s tutoring program approached Borckardt and Xiong to understand whether their tutoring was an effective intervention for student success, they applied data in order to help them achieve program evaluation.
When you start any research project, you need to ask, what is your primary research question — how can you define whether tutoring is effective or not?
The best way Gateway Technical College found to define success was to look at course pass rates or course completion rates. They wanted to see if tutoring would improve the student's ability to pass the class.
At the same time, they thought this might have an impact on retention or graduation rates. So, they need to decide how they were going to answer these questions as well.
In order to answer these questions, Borckardt and Xiong chose to evaluate the SmarterMeasure data taken by the students at their institution.
Click here to listen to how they conducted the research.
Three Step Process
As Borckardt and Xiong gathered research to enhance student success, they had a three-step process to help them find results. This process involved preparing the data by consolidating information, running it through the matching method that is used to evaluate the effect of a treatment by comparing the students receiving tutoring and the students who are not, and finally comparing the outcomes to reveal the benefits of tutoring.
1. Prepare the data
To prepare the data, Borckardt and Xiong needed to consolidate information from three different data sources into a single comprehensive spreadsheet. The process of merging the data involved matching each student based on their unique student ID, which was sometimes added as a custom question, and was available automatically in the SmarterMeasure data source.
The initial data source consisted of student data containing all enrolled individuals within a three-year timeframe. This dataset included outcome data such as course completion rates, retention, and graduation rates.
2. Run the data through the matching method by SmarterMeasure
The next step involved incorporating tutoring data, highlighting which students were utilizing the tutoring services. Furthermore, Borckardt and Xiong integrated the SmarterMeasure data, which was used to identify matching comparison groups.
Gateway used SPSS, a statistical software, for case-control matching. This process consisted of obtaining the data and identifying suitable comparison groups. The data was then transferred into the SPSS.
Then, the dataset was divided into three groups:
- The intervention group, which included students who received tutoring.
- The comparison group, which comprised matched students who did not utilize tutoring.
- The group of a few students who did not receive tutoring and were not part of the matching process.
3. Compare the outcomes
After gathering the data and comparing the outcomes between students who used tutoring and the matched group that did not, a striking difference emerged.
The intervention group experienced an impressive 8.2 percentage point increase in course completion rates and the comparison group saw an 8.7 percentage point decline in the same rates.
The gap between the two groups was almost 20 percentage points. These results demonstrated the substantial benefits of tutoring and student success.
Overall, the research concluded that tutoring demonstrates a positive impact on course completion rates. On a course level, they found this to be especially true for English Comp 1, but also for Quantitative Reasoning to a lesser extent.
Future Goals and Next Steps
After evaluating the data, Gateway Technical College set several future goals and next steps to continue to improve institutional research and retention rates. These goals and next steps included the following.
Future Institutional Research Goals:
- Implement a data warehouse to improve the data collection and transformation process
- Perform similar data assessments as they did with their tutoring department for their:
- Orientation program
- Academic advising
- HEADS UP mentoring program
- Promise Program
- Scholarship/Emergency grant recipients
- Perform a regression analysis for students who receive more than one form of intervention
Future Retention Goals:
- Continue to develop early risk assessment
- Automate service referrals
- Integrate into advising and support processes
Let’s Continue Learning
Learn more about how Gateway Technical College used SmarterMeasure to determine the success of their tutoring program and how your institution can measure the influence of your retention efforts by watching the full on-demand webinar here.
For more information about SmarterMeasure or to schedule a demo, click here to get started.
