In classrooms, teachers put AI tutoring bots to the test

NYT News Service

Tito Rodriguez leads his 6th grade math class in Newark, First Avenue Elementary School, with the camera behind him. (Gabriela Bhaskar/The New York Times).

Cheryl Drakeford, an elementary school teacher in Newark who teaches third grade, recently projected on the whiteboard of her classroom a difficult math question: "What percentage of the letters in MATHEMATICIAN is consonants?"

Drakeford was aware that the word "consonant", for some students, might not be familiar. She suggested that they use Khanmigo, an online tutoring tool.

artificial intelligence

, for help.

She paused a moment while 15 children typed in the same question "What are consonants?" She paused for a minute while about 15 schoolchildren dutifully typed the same question - "What are consonants?" - into their mathematics software. She then asked the third-graders to share the tutoring robot's answer. One student read out loud: "Consonants in the alphabet are letters that aren't vowels." "The vowels include A, E I, O, and U. Consonants include all the other alphabet letters."

The hype in the tech industry and the doomsday predictions surrounding AI-enhanced Chatbots such as ChatGPT have sent many schools scrambling to limit or block the use of these tools in the classroom. Newark Public Schools has a different strategy. It's one of the United States' first school systems to test Khanmigo - an automated teaching tool developed by Khan Academy.

Education

Nonprofit whose online lesson are used by hundreds districts.

Newark is essentially volunteering to be the guinea-pig for schools around the country who are trying to differentiate the real use of AI-assisted tutoring robots from their marketing claims.

Advocates claim that chatbots in classrooms can democratize tutoring because they automatically customize responses for students and allow them to learn at their own pace. The bots are trained using vast databases of texts and critics warn that they can create plausible-sounding false information. They say this is a risky gamble for schools. Newark officials, the largest district of New Jersey, have said that they are cautiously testing out the tutoring bot at three schools. The findings of the Newark officials could have an impact on districts in the United States who are testing AI tools for the coming school year this summer. Timothy Nellegar is the director of Educational Technology at Newark Public Schools. He said that it was important to introduce students to AI-assisted technologies because they will not disappear. But we have to understand how it works and what the risks are. We also need to know the positives and negatives. Khan Academy, one of a few online learning companies to have developed new tutoring robots based on OpenAI's language models (the research lab behind ChatGPT), is among the handful. Khan Academy, which has received high-level donations from Google, Bill and Melinda Gates Foundation and Elon Musk Foundation in the past year, was given access to AI models. The tutoring bot is designed specifically for schools and often guides students through the steps necessary to solve a particular problem. Newark was one of the districts that volunteered to test out Khan Academy's experimental tutorbot when it began seeking pilot testing districts this spring. Newark's elementary schools already used the online math lessons of the education organization to monitor students' understanding of concepts such as grouping numbers. During the first phase of pilot testing, the AI tool will be provided free to these schools. Officials from the district said that they were interested in seeing if Khanmigo would improve student engagement and math education. First Avenue, a school attended by children from low-income families was also eager to offer their students a chance to test a new AI-assisted learning aid. Districts such as Newark, which use Khan Academy’s online lessons and analytics along with other school services (excluding Khanmigo), pay an annual fee per student of $10. The nonprofit stated that districts who want to test Khanmigo in the coming school year would have to pay an extra fee of $60 for each student. It noted that the computing costs for AI models are "significant". Newark's students started using Khan's automated learning aid in May. The reviews so far have been mixed.

Sixth graders from First Avenue Elementary spent a morning working on a project.

Statistics

The students were given the task of creating their own surveys for consumers. Tito Rodriguez suggested that the students begin by asking Khanmigo a couple of background questions. What makes a statistical question?

Rodriguez called the bot a "co-teacher", which allowed him to spend more time with children who were in need of guidance, while still allowing other students to be self-motivated. He said, "They don't need to wait for Mr. Rodriguez." They can ask Khanmigo. The bot's responses in Drakeford Math class seemed to be more direct than suggestions. Khanmigo answered the fraction question on the whiteboard of the classroom by saying that the word "mathematician", which contained 13 letters, had seven consonants. The bot then wrote that the fraction of consonants is 7 out of 13 or 7/13. Alan Usherenko is the special assistant of the district for First Avenue in Newark’s North Ward. He said, "That's what we're most concerned about, that Khanmigo is doing too much thinking work." He said that the district didn't want the bot to guide students through a task step-by-step. "We want to teach them how to solve the problem on their own, using critical thinking skills," he added. Khan Academy stated in an email that students sometimes needed help to solve problems. By practicing, they could learn how to do it automatically. The group said that the tutoring robot was designed to assist students in solving problems and not provide them with answers. The organization claimed that Khanmigo had "helped to much, too quickly" in Newark's fraction problem. Khan Academy wrote in an email on Tuesday that "our engineering team corrected AI a few months ago" so it "no longer gives the correct answer to this problem." A reporter asked Khanmigo to answer the same fractional question on Wednesday. The tutoring bot provided the correct answer in student mode: "The fraction of consonants within the word MATHEMATICIAN is 7/13." The bot gave a different, incorrect response in teacher mode. This mode is intended to guide educators through questions and answers. Khanmigo incorrectly stated that the word "mathematician" contained eight consonants. The bot then gave the wrong answer, "8 consonants/14 total letters = 8/14". In an email to students, Khan Academy stated that it had resolved the problem in its "tutor-me: math and sciences" section. It noted that the reporter asked the question on a different page of the website. The email stated that "sometimes Khanmigo can make mistakes." Usherenko was still hopeful. The district suggested to Khan Academy, that rather than relying solely on students to ask Khanmigo questions, it would make more sense for the bot to ask students open-ended and analyze their responses. Usherenko told Khanmigo that it was not yet where he wanted it to be. When it is able to identify students' misconceptions that will be a game-changer. Khan Academy stated that the tutoring bot asked open-ended questions to students and the group worked on improving the AI models' ability to identify misconceptions. The nonprofit said it would continue to improve Khanmigo based on feedback from schools districts. It remains to be determined whether schools will be capable of affording AI-assisted tutorialbots. Khan Academy announced that it will offer a discount to districts where more than half the students qualify for free or reduced price lunch. The financial obstacles suggest that AI-enhanced chatbots in classrooms will not democratize tutoring anytime soon. Nellegar said that Newark's Ed Tech Director was seeking outside funding for Khanmigo costs this fall. He said that the long-term costs of AI are a concern.