Crowd-AI Teams

Problem / Research Question

How can humans and AI work together to evaluate the creativity and quality of thousands of design ideas accurately and at scale?

Why It Matters

Design studies produce vast, varied ideas — sketches, CAD, text, prototypes — that are slow and inconsistent to rate by hand. Reliable human-AI evaluation speeds innovation and reveals how experts think.

What britelab Is Doing

This NSF-funded research project advances engineering design by developing a framework for rating the novelty and quality of design artifacts using a combination of human and machine expertise. The project addresses a significant challenge: how to accurately evaluate the vast amounts of complex, heterogeneous ideas generated in design studies — including CAD drawings, text, sketches, and prototypes — against established design metrics at scale. It investigates three distinct means of evaluation: (1) human expert designers and crowd-sourced raters, (2) pure ML-based methods, and (3) expert designer-assisted ML methods. The team also uses deep learning, eye-tracking, and physiological sensing to measure and understand how engineers think, create, and assess during design tasks — mapping cognitive patterns like attention, anticipation, and motion efficiency that were previously invisible.

• Understand the impact of design representation on the reliability of crowd-based design ratings
• Establish and validate multi-modal representation learning methods for design evaluation
• Create methods for deploying crowd-ML collaborations for scalable design evaluation
• Use eye-tracking to understand expert vs. novice cognitive strategies during design
• Create objective measures of design quality, creativity, and innovation
• Contribute to ML research on what, when, and how to learn from humans when evaluating heterogeneous datasets

Methods & Technology

Research Team

• Dr. Scarlett Miller (PI)
• Dr. Farnaz Tehranchi (Co-PI)
• Dr. Faez Ahmed (PI, MIT)
• Duk Hee Ka (PhD Student, MIT)

Funding Agency

Collaborators

• Massachusetts Institute of Technology (MIT)

Outcomes / Impact

Contributing to improved product design by effectively leveraging human and computational expertise in evaluating design artifacts. The team partners with the Women in Science and Engineering Research (WISER) program and the Multicultural Engineering Program (MEP) at Penn State to encourage participation of underrepresented groups.

Related Publications

• NSF Award #2231261 — Official Award Page
• How Should We Measure Creativity in Engineering Design? A Comparison Between Social Science and Engineering Approaches
• Forming Diverse Teams From Sequentially Arriving People
• Design Variety Measurement using Sharma-Mittal Entropy
• Measuring and Optimizing Design Variety Using Herfindahl Index
• Interpreting Idea Maps: Pairwise Comparisons Reveal What Makes Ideas Novel
• Unpacking Subjective Creativity Ratings: Using Embeddings to Explain and Measure Idea Novelty

Related News

• Measuring the Surgical Mind: Where AI Actually Adds Value

  Endoscopy Superstore · Mar 24, 2026