Benchmark AI / Public workspace
SWE-Bench Pro
Code / Agents
SWE-Bench Pro evaluates agents on long-horizon software engineering tasks in real repositories. The public card describes 731 tasks containing issue descriptions, repository identifiers, and base commits.
Japanese introduction
実際のソフトウェアリポジトリに対する修正課題で、長い工程を要する開発能力を評価します。公開データカードの731課題には問題文、対象リポジトリ、修正開始点のcommitなどが含まれます。
swe-bench-pro
Full text: eligible public problems include text, images and discussion.
Tasks
Whole public benchmark.
- Public problems
- 200
- Formats
- text (200)
- Problems with images
- 0
- Answer availability
- Answer published by the source (200)
- Configs
- Default (200)
- Splits
- test (200)
- Awaiting a first discussion
- 200
- List operations do not support removing multiple distinct…
Answer published by the source
- Add support for min/max score ranges in sortedSetsCardSum
Answer published by the source
- Plugin activation accepts invalid plugin identifiers without…
Answer published by the source
- Email Confirmation Expiry and Resend Not Working Consistently…
Answer published by the source
- Chat Allow/Deny List
Answer published by the source
- Feature Request: Refactor Link Analysis with a Dedicated `DirectedGraph`…
Answer published by the source
- Issue title: Pass attribute to the max filter and min filter
Answer published by the source
- Forked output from ‘Display.display’ is unreliable and exposes shutdown…
Answer published by the source
- INI string values are not unquoted correctly in `ansible.cfg`
Answer published by the source
- Add Support for Galaxy Server Configuration in ansible-config Command
Answer published by the source
- No description recorded
Problem identifier:
instance_ansible__ansible-d58e69c82d7edd0583dd8e78d76b075c33c3151e-v173091e2e36d38c978002990795f66cfc0af30ad. No description was recorded for this problem.Answer published by the source
- Admin action buttons (Kick, Ban, Mute) trigger multiple times on…
Answer published by the source
- PosthogAnalytics fails to reliably handle initialization,…
Answer published by the source
- Adaptive Audio Recording Quality Based on User Audio Settings
Answer published by the source
- Title Sign in with QR feature lacks feature flag control mechanism…
Answer published by the source
- Feature request: Include audit configuration in anonymous telemetry
Answer published by the source
- Config loader misses DB pool options and update-check flag.
Answer published by the source
- Incorrect parsing of `rpm -qa` output when release field is…
Answer published by the source
- Scanner fails on non-standard source RPM filenames and epoch handling
Answer published by the source
- Missing Filter Counts and Inadequate Logging of CVE Filtering in `Detect`
Answer published by the source
- WordPress cache pointer indirection and inactive package filtering
Answer published by the source
- Scan results miss Package URL (PURL) information in library output
Answer published by the source
- tsh device enroll --current-device panics when the device limit is…
Answer published by the source
- Add linear benchmark generator for progressive request rate…
Answer published by the source
- Add auditd integration
Answer published by the source
- Incorrect counting of authenticated HTTP connections in ingress reporter metrics
Answer published by the source
- OSS users lose connection to leaf clusters after root cluster upgrade to…
Answer published by the source
- Lack of utility functions for extracting system metadata
Answer published by the source
- Consistent author extraction from MARC 1xx and 7xx fields and reliable…
Answer published by the source
- Bug Report:** Mismatching of Editions for Wikisource Imports
Answer published by the source
- Internet Archive metadata imports do not correctly handle publisher…
Answer published by the source
- No description recorded
Problem identifier:
instance_internetarchive__openlibrary-7f7e53aa4cf74a4f8549a5bcd4810c527e2f6d7e-v13642507b4fc1f8d234172bf8129942da2c2ca26. No description was recorded for this problem.Answer published by the source
- Inconsistent return type of `update_key` in Solr updaters
Answer published by the source
- Function-to-CLI adapter mishandles list inputs and filesystem…
Answer published by the source
- Author matching fails with different date formats and special…
Answer published by the source
- Placeholder values are not removed during normalization
Answer published by the source
- Navidrome export playlist to M3U from command line option
Answer published by the source
- Selective event delivery for user and client
Answer published by the source
- [Bug]: Unset timestamp fields cause internal errors after upgrade…
Answer published by the source
- Vault aliases drawer does not consistently render list/empty state or open…
Answer published by the source
- API error metrics
Answer published by the source
- Local SSO URLs not correctly aligned with local proxy…
Answer published by the source
- Notifications with HTML content display incorrectly and duplicate…
Answer published by the source
- Add filesystem path completion support for `:open` command
Answer published by the source
- Replace non-Qt version parsing with a Qt-native mechanism in specific…
Answer published by the source
- No description recorded
Problem identifier:
instance_qutebrowser__qutebrowser-8f46ba3f6dc7b18375f7aa63c48a1fe461190430-v2ef375ac784985212b1805e1d0431dc8f1b3c171. No description was recorded for this problem.Answer published by the source
- Bug Report: `parse_duration` accepts invalid formats and miscalculates…
Answer published by the source
- Successful process data is retained indefinitely
Answer published by the source
- URL parsing and search term handling edge cases cause incorrect…
Answer published by the source
- Entropy Management Logic Scattered Across Multiple Classes Creates Coupling…
Answer published by the source