[Paper] GenExam: A Multidisciplinary Text-to-Image Exam – A New Benchmark for Testing AI’s Ability to Draw Like an Expert

Exams have long served as the gold standard for measuring human intelligence. They don’t just test memorization but comprehensively assess understanding, reasoning, and creative application of knowledge. This is why researchers from OpenGVLab have introduced GenExam, the first benchmark specifically designed to evaluate how well AI models can handle multidisciplinary text-to-image exams, pushing artificial general intelligence (AGI) toward expert-level capabilities in visual creation.

Unlike existing benchmarks that focus primarily on testing AI’s understanding or simple concept illustration, GenExam presents a significantly more challenging task: can AI systems accurately draw complex diagrams that require deep subject knowledge? The benchmark consists of 1,000 samples spanning 10 academic subjects including mathematics, physics, chemistry, biology, computer science (for those interested in diving deeper into the field, Invitation to Computer Science provides an excellent foundation), geography, economics, engineering, music, and history.

What makes GenExam particularly rigorous is its approach to evaluation. Each prompt is crafted like a real exam question, requiring the AI to not just understand the text but to reason through it and generate a precise visual response. For instance, a chemistry problem might ask the model to draw the step-by-step reaction pathway for an organic synthesis (students of chemistry might find Advanced Organic Chemistry: Part B: Reaction and Synthesis particularly helpful), including proper molecular structures, reaction conditions, and intermediate products. Each question comes with detailed scoring points to evaluate semantic correctness and visual plausibility.

The results from testing current state-of-the-art models reveal significant limitations in today’s AI systems. Even advanced commercial models like GPT-Image-1 (based on GPT-4o) and Gemini-2.5-Flash-Image achieve strict scores of less than 15% on GenExam. Most models, including many recent open-source text-to-image generators and unified multimodal language models, score nearly 0% when judged by strict criteria. This demonstrates the substantial challenge that multidisciplinary image generation presents for current AI technology.

The gap between closed-source and open-source models is particularly striking. While GPT-Image-1 achieved the highest strict score at 12.1%, open-source models like BAGEL (developed by ByteDance’s Seed team), Show-o2, and BLIP3o couldn’t generate a single image that met all the strict requirements. These results highlight the long path ahead before AI can truly integrate understanding, reasoning, and generation at an expert level across multiple disciplines.

GenExam’s evaluation framework is comprehensive, measuring both semantic correctness (does the image accurately represent what was asked?) and visual plausibility (is the image well-formed with correct spelling, logical consistency, and readability?). The benchmark offers both strict scores, where any flaw results in failure, and relaxed scores that provide a more nuanced measure of performance.

The researchers validated their evaluation framework through human alignment studies, showing strong correlation between automatic assessment and human judgment. This ensures that the benchmark provides meaningful insights into model capabilities rather than just arbitrary metrics.

GenExam reveals common failure patterns in current text-to-image models. These include lack of domain knowledge (misunderstanding key concepts or structures), reasoning errors (failing to correctly infer spatial relationships or interactions), and execution problems (producing illogical or poorly rendered images). For example, models might draw chemical structures (visualizing these concepts is easier with tools like the Molecular Model Kit for Organic Chemistry) with incorrect bonds or mathematical graphs with misplaced coordinates. For students studying these mathematical concepts, Introduction to Graph Theory offers valuable insights into this field.

As AI continues to advance toward more general capabilities, benchmarks like GenExam provide crucial guideposts for progress. The ability to integrate deep knowledge across disciplines with complex reasoning and precise generation remains a frontier challenge. By framing image generation as an expert-level exam, GenExam raises the bar for what we expect from artificial intelligence and offers a roadmap for future development.

The benchmark is open-source and available on GitHub and Hugging Face, allowing researchers to evaluate their own models and contribute to advancing the field of multidisciplinary AI image generation. The dataset features detailed evaluation metrics and scoring criteria that enable systematic assessment of model performance across diverse academic domains.

As models improve on GenExam scores in the coming years, we’ll likely see significant progress toward AI systems that can truly think and create like human experts across diverse domains of knowledge. This benchmark represents an important step in the journey toward artificial general intelligence that can not only understand but also visually communicate complex ideas with expert-level precision.