Measuring Online Language Teacher Quality: Stakeholder Metrics
Measuring teacher quality in online language schools is a very complex task. In a brick-and-mortar school, the Director of Studies / Head Teacher will regularly observe teachers and create training programs for their teachers in order to judge teacher quality and create tailored training programs.
With the shift from offline classes taught by employed teachers to online classes, often taught by independent contractors, observing large volumes of classes to determine teacher quality has become more of a challenge. Additionally, there has been a shift from more subjective ‘assessment’ to data-driven ‘measurement.’
In this article, we will look at how we can examine teacher quality in large volumes of classes by taking data inputs from a range of stakeholders and sources. While every school has different stakeholders and requirements, this can provide a basic framework for how to use data when measuring teacher quality.
This article is adapted from the book Language School Teacher Quality & Training by Peter Sommerville available on TEFLConsultants.com
How is Data Used to Manage Language Teacher Quality?
As we increasingly use data for feedback and assessment, language schools are creating scoring metrics that are then used as an indicator of overall teacher quality. This data is used for:
Assignment of classes to higher scoring teachers
Giving parents an indication of teacher quality when booking classes
Teacher payments and bonuses
Deciding on teacher contract renewal
Determining future training programs
Should Data be Used for Measuring Teacher Quality?
Many important questions arise when we start using data to measure quality, especially given that this data has a very direct impact on teacher payments and livelihoods:
What is a good online teacher and how do different stakeholders view and assess teacher quality?
Is it fair to incorporate student and parent feedback into teacher quality?
Is this data objective and based on actual teaching quality or is it influenced by other factors including expectations when buying a course?
Is retention only influenced by teacher quality or are other factors internal and external to the language school impacting quality metrics?
Are internal observations subjective? Do we have clear rubrics to ensure consistency and fairness when observing classes?
Are we weighing the metrics fairly to give an accurate reflection of teacher quality?
These are just a few of the complex and difficult questions that have serious repercussions and should be considered very carefully when using data to measure teacher quality. Much greater detail is covered in our book ‘Language School Teacher Quality and Training.’
Stakeholders
With the increasing use of feedback systems, the measurement of teacher quality is no longer solely determined by an academic manager but needs to take into account the opinions and needs of various stakeholders and data sources:
Students’ feedback
Parents’ feedback
Business needs
Quality Assurance Teams
Academic Operations Teams
Customer Service Teams
Artificial Intelligence
Measurements of Online Teacher Quality:
How Students Measure Online Teacher Quality
In most online schools students will give a feedback score after each class. The way it is scored will vary by the school or Learning Management System (LMS) used. The most common usage is a score out of 5. In addition to scores, students are often able to leave written comments either through a comment box or a drop-down list of comments in their native language which are shown to teachers in English. These comments may also be categorized into positive / needs improvement or by topic. Analysis of comments can also be used as a qualitative data source. However, not all students will choose to leave feedback and the amount of feedback will drop over time as the students become familiar with a teacher. It is important therefore we don’t use single classes or student feedback in isolation and look at scores over a longer time period (e.g. the last three months).
This kind of feedback emphasises the classroom experience over the outcome. It generally does not indicate if a student was able to produce the target language or passed any exams or tests. Additionally, it can lead to teachers prioritising high feedback scores over student outcomes.
How Parents Measure Online Teacher Quality
In K-12 online English classes, parents are also frequently asked to give feedback on teacher quality. This is done in much the same way as with the students — through a combination of a numerical score and comments system. This feedback is usually given after parents have watched a recording of an online class. It can be extremely beneficial as it provides insight into how parents are perceiving the classes they have paid for and gives them a sense of agency in their child’s learning.
There are however some areas to be mindful of when using this data. It is common that parents don’t watch the classes and give feedback based on what the child has told them about what they did in the class. Be that an accurate representation of the class or not. Feedback is often given based on an assumption of what ‘quality’ teaching is and does not necessarily match what teachers or academic departments believe it to be. For example, in Asia, many parents want to see a focus on accuracy which would involve the introduction of grammatical structures or an emphasis on error correction. Conversely, teachers and academic departments are more likely to have a focus on fluency or task achievement.
As with student data we need to ensure that we understand the underlying assumptions parents are making and use this data carefully.
How Businesses Measure Online Teacher Quality
Language School Cost of Customer Acquisition (CAC) and Lifetime Value (LTV)
Online English language teaching is very competitive due to the ease with which students can switch to our competitors. There is generally a very high Customer Acquisition Cost (CAC) of new students and low margins. This can often mean that it can take several courses for a school to break even and start making a profit from a student.
Therefore it is critical for an English language teaching business to retain students past the breakeven point. A major KPI for schools is their retention rate. While retention is driven by many factors, teacher quality is usually the main driver. However, retention is also influenced by many other factors including price, content quality, customer service etc.
A key metric for each teacher is their retention rate as this determines whether giving that teacher classes will lead to profit or loss. For example, if the school needs a retention rate of 75% and a teacher has a retention rate of 80% then, from a purely business perspective, they could be considered high quality regardless of skill, student outcomes, etc.
This model can also be extended to measure the profitability of each teacher over time to calculate a Teacher’s Lifetime Value (TLTV).
How Quality Assurance (QA) Departments Measure Online Teacher Quality
Quality assurance in online language schools can vary widely by the organisation depending on the scale. It may be the Head Teacher or Director of Studies watching online classes as they did with their teachers in offline classes, up to large teams watching hundreds or thousands of classes per month. In both cases, the goals are to check classes to ensure that teachers are providing ‘quality’ education, record what the teacher did in the class and provide feedback and areas for improvement & further training. However, the legal issues surrounding providing feedback and training to independent contractors can make this a difficult task.
Furthermore, high volumes of classes make observations a very high-cost exercise. Where a Director of Studies could once watch one class per teacher per month, having potentially thousands of teachers makes this impractical. QA departments must then decide on what strategy to take when deciding on classes to observe. This is usually a combination of the following factors:
Assessing new teachers
Regularly scheduled observations
Corrective action observations based on negative feedback or complaints
Assessing teachers due for contract renewal or payment changes
In assessing online classes a QA team will use a rubric to identify areas of strength and weakness of the teacher. This will usually evaluate the kind of skills that have traditionally been observed but give an additional numerical score in the form of an overall lesson score. Areas that are measured may include:
Student engagement
Equipment and background
Achievement of lesson aims
Time efficiency
Instructions
Error correction
Use of feedback
Total physical response (TPR)
How Operations Teams Measure Online Teacher Quality
As with the QA team, the operations team is also part of the business but will tend to focus on teacher attendance as a critical online teacher quality measurement. Teacher attendance is one of the key drivers of student retention, especially in 1vN group classes where sales teams have likely promised a fixed teacher for the duration of a course or semester. Teachers not attending classes and needing to be replaced can have a dramatic impact on retention as students and parents view teacher changes or non-attendance as having a major effect on the quality of service. Operations teams will therefore measure the overall attendance rate of a teacher including:
Missed classes (no-show)
Teacher late
Teacher leaving early
Cancellation of classes before the scheduled start time
It is also common to measure technical issues such as drop-out from classes or audio and video problems.
How Artificial Intelligence is Used to Measure Online Teacher Quality
Given the difficulties of observing multiple classes, there is a growing trend in larger online language schools to develop ‘AI’ systems that use data from online classrooms to measure both student engagement and teacher quality. While this is in its infancy and is arguably not true AI, its use in measuring teacher quality is becoming increasingly widespread. Examples of how it may be used are:
Measuring Teacher Talking Time (TTT) vs. Student Talking Time (STT)
Facial recognition to measure student (and teacher) engagement
Student mouse click data for measurement of engagement
Speech recognition to measure target language usage and pronunciation
Tool use data (e.g. the frequency of a teacher’s use of online classroom tools)
While not widely available due to the high cost and difficulty in building accurate tools, AI is likely to be increasingly used as a measurement of teacher quality.
Using Multiple Data Points
Once the data is gathered it can then be combined to give an overall teacher quality score. At a basic level, online teaching organisations will convert these measurements to a percentage value and then average them out to create an overall teacher quality score. Some metrics such as retention rate and QA score could be considered as being more important than others and are thus given a greater weighting. The total weighted average then gives a total quality score, in this case, 50%.
Note that the metrics you use will depend on your situation. You may for example want to add more outcome-based data such as exam results, test score changes etc. or financial data such as the Teacher Lifetime Value (TLTV).
The Online Teacher Quality Bell Curve
Once the school has created total quality scores for each teacher the data can be plotted out to form a bell curve. With this data, high-performing teachers can be prioritised for the assignment of classes and potentially pay rises or bonuses to ensure that they can be retained by the school. Lower performing teachers can be given additional training or potentially not have contracts renewed.
Measuring Success
The ultimate goal of teacher quality management is to ensure high-quality teaching is delivered to students so they are able to achieve their goals. This in turn should lead to increased retention rates and higher margins. Much as a teacher’s quality is measured in part by their retention, the overall quality metrics are also used to measure the success of teacher quality managers and teams.
It is important to note however that while student retention rate can be increased through increased teacher quality, there are additional factors that will flatten out a retention rate curve which makes 100% retention unfeasible. In addition to the internal factors mentioned above, there are external factors beyond a language school’s control. Students may leave the school when they have achieved their goals, such as passing an exam, being admitted to a university or completing a study abroad course.
The Dangers of Data
“You can’t manage what you can’t measure.”
This quote by Deming is often cited as a justification for reducing teacher quality down to a purely data-driven exercise. In fact, not only is the quote misattributed but Deming believed it was dangerous. What makes a good teacher can’t be reduced down entirely to numbers. It is always vital when managing teacher quality to perform ‘sanity checks.’ This involves selecting some teachers from across the bell curve and observing their classes. Does their teacher quality score reflect their actual practice and skills? Are there other factors that could be impacting their scores? For example, is a non-native teacher being unfairly given lower scores in parent feedback? In K-12 in particular is it really sensible to rely on student data from children to assess a professional and determine their pay rates and job security?
“I am seen as a number, a cog in a machine.”
We must also be very careful about how we communicate and use data with teachers. Data needs to be transparent and accessible if it is used to measure performance and set goals. As managers, we manage people, not data. As language schools get larger and use independent contractors on the other side of the world, it is all too easy to lose sight of this. Consider what the root cause of the data is and constantly test and adjust to find better ways to utilize that data. Not doing so will cause the school to lose teachers and reduce teacher retention rates.
Finally, as language school managers, we must practice what we preach. If your teachers’ pay is based on data-driven quality metrics, so should yours!
Doing so will lead you to question and refine your use of data for quality management.
