Everyday Automation · Cognitive Offloading
Engcall
Auto-grades a spoken English lesson from its recording.
- Year
- 2024~now
- Role
- Solo — product, backend, frontend, operations
- Status
- Live — operating
TL;DR
Engcall is an English education service. This page is about one part of it: automatic speaking evaluation. After each 20-minute lesson the recording is transcribed, only the learner's speech is scored on complexity, accuracy and fluency (CAF), and a report that quotes the learner's own sentences is published.
What can you do with it?
Every lesson ends with a graded report, without a tutor writing it.
- Score every lesson on one scale — complexity, accuracy and fluency, each from 1 to 9, from beginner to near-native.
- See the evidence — every point quotes the learner's own English with the fix beside it, so the learner can check the AI instead of simply trusting it.
- Get corrections — grammar corrections, better expressions in casual, business and academic tones, and vocabulary upgrades.
- Share one report — the same analysis reaches both the learner and the tutor.
Why it matters
Tutors found it hard to grade by the same standard, the quality of feedback varied from tutor to tutor, and writing it took 10–15 minutes after every lesson.
- In a 20–30 minute lesson a tutor is both an assessor of language and a person looking after motivation. Doing both is hard, so feedback stayed brief and general.
- Scoring on the CAF framework used to need trained human raters, so it was used in research, not in everyday lessons.
- Automatic scoring makes the standard the same for every lesson, and hands tutors time back for what a machine cannot do.
How it works
Five steps, from a lesson recording to a graded report.
- Upload — after the lesson, the tutor uploads the recording through the portal.
- Transcribe and label the voices — AssemblyAI transcribes the English and separates the two speakers.
- Score the learner only — GPT-4o picks out the learner's speech and scores it on the 1–9 CAF scale. The rubric is written into the prompt, and every score must be backed by the learner's actual words.
- Check the speakers — a rule looks for lesson-closing phrases at the end of the transcript to confirm who the tutor was, and swaps the labels if GPT got it wrong.
- Publish — the report, written in Korean around quotes of the learner's English, appears in the learner's and tutor's portals.
Two design choices
- Three dimensions, not one score — CAF separates complexity, accuracy and fluency, so a learner sees which one is holding them back, and the feedback has something to act on.
- Korean explanations, English evidence — the feedback is in Korean so it is easy to follow, but every point quotes the learner's own English. The learner can check each claim against what they said — a user-side guard against the model making things up.
How was it tested, and how does it perform?
Reported in a paper at the Korean EdTech Society's 2026 spring conference: a pilot in live operation from 30 March to 17 April 2026, asking two questions. Does the pipeline run reliably in real use? And how does automatic scoring change the way tutors spend their time?
- Reliable in operation — 123 of 128 recordings completed and none failed with an error.
- Consistent, not random — across 109 lessons the three dimensions differed significantly, and still did when averaged per learner. Accuracy scored lowest, in line with research finding that accuracy is the last of the three to become automatic in a second language.
- Time handed back to tutors — grammar, expression and vocabulary feedback, once written by hand after every lesson, is now automatic, and CAF scoring, never done before for lack of time, became standard. Tutors said they could now "listen to the student to the end, instead of hunting for errors."
- 128
- lesson recordings, from 34 learners and 4 tutors
- 96.1%
- graded automatically, end to end (123 of 128)
- 0
- failures — the other 5 stopped at a transcription time-out and can be retried
| Dimension | Mean | SD |
|---|---|---|
| Complexity | 5.12 | 1.00 |
| Fluency | 4.79 | 1.02 |
| Accuracy | 4.42 | 0.86 |
| Pair | t(108) | Cohen's d | d, per learner |
|---|---|---|---|
| Complexity vs Accuracy | 22.80 | 2.18 | 2.49 |
| Fluency vs Accuracy | 9.30 | 0.89 | 1.00 |
| Complexity vs Fluency | 8.66 | 0.83 | 0.96 |
Limits, as stated in the paper
- The significant differences are not taken as proof that the scores are valid. GPT-4o was trained on the research literature about CAF, and may be reproducing the typical pattern of Korean learners rather than measuring this learner.
- Agreement with human raters (ICC) has not been measured, and is needed before the scores can be called valid.
- Whether the 5 unfinished recordings succeeded on retry was not measured, so the true completion rate may be higher.
- The findings about tutors come from informal tracking and short conversations with 4 tutors, not from a measured study.
Applications
- Tutoring services at scale — one standard across many tutors, from any recorded lesson.
- Tracking progress — a learner's CAF scores lesson after lesson give tutors data for choosing the next topic, where they used to rely on impression alone.
- Language schools and research — CAF scoring at a scale that human raters cannot reach.
Future directions
- Check against human raters — measure agreement with trained raters before relying on the scores.
- From checking to reasoning — before showing a fix, ask the learner why the other expression is more natural, and look for the answer in the next lesson's recording.
- Measure the tutor side — surveys and work-log analysis to test whether the time handed back actually goes to learners.
Behind the project — how it started and how it changed
The problem
Engcall connects Korean learners with tutors based in the Philippines for one-to-one daily English conversation. I built it because the services I could find were expensive and rigid, and I have run it since — booking, payment, tutor management and the lesson flow are all systems I designed and wrote alone.
Talking every day is the easy part of learning a language. Knowing whether you are getting better is the hard part, and it was the bottleneck in the service: assessment is slow, subjective, and does not scale with the number of tutors. Two versions of the product were attempts to fix that, and only one of them survived.
How it changed
Each version below exists because the one before it was wrong about something specific.
- v1retired
Build the phone call ourselves
- Why
- A phone-English service that depends on a third-party call app does not own its own core experience. I wanted the call inside the product.
- Change
- Built in-app voice calling on the Agora SDK — real-time token authentication, session lifecycle management over HTTP polling, and AI noise suppression so a lesson survives a noisy room on either end.
- Result
- It worked. It was also the wrong thing to own. Per-minute media costs scaled directly with lessons, a bespoke call stack meant every connection problem became my support ticket, and the tutors already lived in Zoom and were fluent in it. The return did not justify the surface area.
- v2
Drop it, and move the effort to assessment
- Why
- The call was solved by everyone. Assessment was solved by nobody, and it was the part learners kept asking about.
- Change
- Replaced the in-house call with Zoom, and spent the reclaimed effort on an automatic evaluation pipeline. AssemblyAI transcribes the lesson and labels the two voices, GPT-4o works out which one is the tutor, and only the learner is graded. GPT-4o then scores that transcript on complexity, accuracy and fluency — the CAF measures used in second-language research — and writes the feedback that goes with each score.
- Result
- Feedback stopped depending on which tutor you happened to get or on how much time they had after a lesson. Every learner gets the same structured read on the same day, and tutors get to spend the lesson teaching instead of grading.
engcall.kr — live, in Korean, with paying users. 
Daily expressions in the learner-facing product. 
Booking and scheduling, built in-house.
The part worth keeping
The call feature was the piece of engineering I was proudest of and the piece I cut. Deleting working code is the decision I would make again: the question was never whether I could build it, it was whether owning it made the service better for the person on the call. It did not.
The evaluation pipeline is what actually changed the product. It is also the piece I would not have had time to build if I were still maintaining a call stack.
Built with
- React
- TypeScript
- Spring Boot
- Java
- MariaDB
- AWS EC2
- AssemblyAI
- OpenAI Whisper
- GPT-4o
- Toss Payments
Deep dives
Individual pieces of this project, written up on their own.