Two Dartmouth computer scientists have won National Science Foundation CAREER awards for two different approaches to artificial intelligence.
SouYoung Jin and Yujun Yan, assistant professors in the Department of Computer Science, each received a Faculty Early Career Development award, known as a CAREER award. The NSF aims to support scientists and engineers with a strong record in both research and teaching, and who, in the agency’s words, “have the potential to serve as academic role models in research and education and to lead advances in the mission of their department or organization.”
“Jin and Yan are tackling very different but exciting problems in AI,” says Devin Balkcom, the Dartmouth Professor of Computer Science and chair of the department. “Jin’s work on AI that senses, sees, and feels opens a broader channel between the physical, human, and computational worlds. Yan’s work focuses on building computational models of complex systems including the brain, social networks, and multi-agent systems, allowing AIs to reason and draw conclusions. This range is part of what makes our department strong, and our students are fortunate to learn from both of them while that work is taking shape.”
Teaching AI to understand video
Jin works on computer vision, teaching machines to interpret images and video, and draws on cognitive science to do it. Her award supports a system that can make better use of all the information packed into a video, while protecting the privacy of the people in it.
“Basically, we want to develop AI systems that can understand videos, capturing the emotional content of the videos and contextualizing what is happening in the video in order to align more closely with human understanding,” says Jin. “This includes understanding emotions and social context in situations where emotional signals can be subtle or ambiguous. And to make sure the AI model works the way we want it to, we will also be establishing a way to test how rational the model is. Is it actually reasoning out what happened in the video? How can we tell? This is critical, particularly if our long term goal is for people to use the model in real world settings.”
One of those real-world uses is accessibility: the project aims to improve tools like audio descriptions for blind and low-vision audiences.
The award also extends Jin’s work on training methods that do a better job of preserving people’s privacy.
The work will shape and inform Jin’s teaching. “I’ll be exploring questions related to accessibility and privacy with undergraduate and graduate students in my video understanding courses,” she says. “They’ll have a chance to engage with these challenges in a hands-on way. And I'm hoping to incorporate some of what we learn here to Dartmouth Science Day, where we can share it with the community.”
Borrowing the brain’s playbook
Yan studies where AI meets fields like neuroscience, looking at how a deeper grasp of other disciplines can shape new machine learning models. With the award, Yan will study how people organize knowledge, reason, and collaborate, then use what she learns to make AI systems easier to interpret and better at working together.
“Centuries of research have given us a set of established principles that help us understand how people think and behave,” says Yan. “And recent work from my lab finds there are significant similarities between how our brains and AI systems process data to solve problems. My goal with this award is to build on those findings to expand these cognitive and social principles and incorporate them into AI models and systems.”
The project points in a few concrete directions: computational tools for neuroscience research, software that tailors feedback to individual students as they learn, designs that enable large language models to understand logical rules in more human-like ways, and collaborative principles that help AI systems work together more effectively.
“I’m excited about the opportunities this project creates for my students,” says Yan.
Graduate and undergraduate students will be actively involved with the research. Findings will support an ongoing collaboration between Yan’s lab and Dartmouth's DALI Lab, a student-driven technology and design agency that builds early-stage mobile, web, AR/VR, and hardware products. The project will also extend ongoing mentorship efforts with both local schools and the New Hampshire Academy of Science.