{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"0054a0ef-e3be-51e2-b514-56740fb0d34f","task_key":"bbeh~5fgeometric~5fshapes--full--0054a0ef-e3be-51e2-b514-56740fb0d34f","task_revision_id":"1","upstream_id":"","short_description":"Suppose we draw this SVG path element: M 19.03578,0.0 L 16.93423,0.0 M…","config":"bbeh_geometric_shapes","split":"full","body":"{\"input\":\"Suppose we draw this SVG path element: M 19.03578,0.0 L 16.93423,0.0 M -2.88675,5.0 L -2.02073,3.5 M -9.6135,-22.85723 L -8.8071,-21.4605 M 8.75646,4.32 L 19.03578,0.0 M -4.92,-24.72784 L -9.6636,-32.944 M -13.4487,-29.49999 L -12.29814,-27.50716 M -5.7735,-6.22316 L -5.7735,-3.24123 M -12.29814,-27.50716 L -9.6135,-22.85723 M 19.03578,-3.50053 L 19.03578,0.0 M -2.07846,-16.20615 L -5.7735,-16.20615 M 9.89861,0.0 L 7.42396,0.0 M -12.0,-36.99076 L -16.21466,-36.99076 M 14.08648,-16.20615 L 19.03578,-16.20615 M -17.7735,-36.99076 L -13.4487,-29.49999 M -5.7735,-0.0 L -2.88675,5.0 M 0.0,-16.20615 L 14.08648,-16.20615 M 16.93423,0.0 L 9.89861,0.0 M -2.02073,3.5 L -1.23265,2.135 M 0.0,0.0 L 0.0,8.0 M -8.8071,-21.4605 L -5.7735,-16.20615 M -5.7735,-10.41602 L -5.7735,-6.22316 M -5.7735,-16.20615 L -5.7735,-10.41602 M -5.7735,-3.24123 L -5.7735,-2.04197 M 0.0,-10.37194 L 0.0,-16.20615 L -3.1488,-21.66003 M 19.03578,-13.77523 L 19.03578,-12.96492 M -3.1488,-21.66003 L -4.92,-24.72784 M 19.03578,-16.20615 L 19.03578,-14.4802 M 0.0,0.0 L 0.0,-10.37194 M 0.0,-16.20615 L -2.07846,-16.20615 M -5.7735,-2.04197 L -5.7735,-0.85763 M 19.03578,-12.96492 L 19.03578,-3.50053 M 0.0,8.0 L 8.75646,4.32 M 19.03578,-14.4802 L 19.03578,-13.77523 M -5.7735,-0.85763 L -5.7735,-0.0 M -5.7735,-16.20615 L 0.0,-16.20615 M -1.23265,2.135 L 0.0,0.0 M 7.42396,0.0 L 0.0,0.0 M -16.21466,-36.99076 L -17.7735,-36.99076 M -9.6636,-32.944 L -12.0,-36.99076 . Out of the following shapes:\\n1. rectangle that is not a square and with no diagonals drawn\\n2. square with no diagonals drawn\\n3. triangle that is not a right triangle\\n4. right triangle\\n5. parallelogram that is not a rectangle and with no diagonals drawn\\n6. trapezoid with exactly one pair of parallel sides and with no diagonals drawn\\n7. irregular convex pentagon with no diagonals drawn\\n8. irregular concave pentagon with no diagonals drawn\\n9. regular pentagon with no diagonals drawn\\n10. regular hexagon with no diagonals drawn\\nwhich one(s) can be viewed when the lines in the SVG are visualized? Note that a shape with n sides should not necessarily be drawn by n lines; e.g., a triangle might be drawn by 4 lines, two of which collinear.\\nOptions:\\n(A) 4 and 5\\n(B) 5, 3 and 6\\n(C) 1, 5, 7 and 6\\n(D) 1 and 5\\n(E) 5, 2 and 10\\n(F) 1, 4, 5 and 7\\n(G) 4, 5, 7, 3, 6 and 9\\n(H) 1, 4, 5, 8 and 10\\n(I) 1 and 4\\n(J) 1, 4 and 5\\n(K) 1, 4 and 7\\n(L) 5, 7 and 6\\n(M) only 5\\n(N) 1, 4, 5 and 6\\n(O) 1, 5, 7, 6 and 10\\n(P) 5 and 7\\nCoordinates have been rounded to 5 decimal places so ignore slight differences.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}