benchmarks.wiki

benchmarks.wiki / Public workspace

MathVista

Mathematics / Multimodal

MathVista gathers 6,141 mathematical problems that come with figures, charts or geometric diagrams, of which 1,000 form the public testmini split. It measures whether visual information can be carried into mathematical reasoning.

mathvista

Official source

Download answersVisit the official benchmarkWhere the answers areStart a benchmark discussion

History

The MathVista research team introduced the benchmark in 2023 to evaluate mathematical reasoning that depends on visual information. The AI4Math source distribution recorded here was published on Hugging Face in October of that year. Its motivation was to test whether models could connect evidence in charts, figures and geometric diagrams to the calculations or deductions needed for an answer.

How it was built

MathVista combines problems from existing visual question-answering resources with newly collected tasks focused on mathematical reasoning. The source contains 6,141 problems, with 1,000 in the public testmini split. Each item pairs an image with a mathematical question, using either multiple-choice or free-form answers, and evaluation measures answer accuracy.

At a glance

Metric
accuracy over testmini and the full test set
Size
6,141 problems in the source, 1,000 in testmini; none imported.
License
cc-by-sa-4.0 (declared on the Hugging Face dataset card)
Released
Dataset published on Hugging Face 2023-10
Maintainer
AI4Math (source organisation)
Answer access
No problem text is imported, so there is no answer storage for this entry. The source distributes the answers under its own terms.

What a problem looks like

A mathematical question with an accompanying image, in either multiple-choice or free-form answer format.

How it is scored

The source ships the answer; nothing is imported here.

Metric: accuracy over testmini and the full test set

Why it is hard

The quantities are in the picture. Reading a chart or a geometric figure precisely enough to compute with it is a different skill from solving the equation afterwards.

Limitations and criticism

A final-answer score combines visual perception, mathematical reasoning and answer extraction, so it does not identify which stage caused an error. The mixture of problem types also means that an aggregate score can hide weaknesses on particular visual formats or mathematical skills. Results on the public 1,000-question testmini split should be distinguished from results on the separate test split, and public examples create a potential exposure risk.

Cite

@article{lu2023mathvista,
  title={MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts},
  author={Pan Lu and Hritik Bansal and Tony Xia and Jiacheng Liu and Chunyuan Li and Hannaneh Hajishirzi and Hao Cheng and Kai-Wei Chang and Michel Galley and Jianfeng Gao},
  journal={arXiv preprint arXiv:2310.02255},
  year={2023}
}

Discussion

1 thread and 0 posts on this page. Anyone may reply; no account is required.

Contribute

Share a reproduction, report an evaluation issue, or suggest a correction to this benchmark overview.

Public problem summary

Catalogue metadata only; public problem statistics are not available.

Public problems
—
Formats
—
Problems with images
—
Answer availability
—
Configs
—
Splits
—
Awaiting a first discussion
—
Problems with a discussion
—
Public contributions
—