{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a8acbff0-fb4f-5cb1-b4d6-7ec40fe301a2","task_key":"logical~5fdeduction~5ffive~5fobjects--main--a8acbff0-fb4f-5cb1-b4d6-7ec40fe301a2","task_revision_id":"1","upstream_id":"","short_description":"The following paragraphs each describe a set of five objects arranged in a fixed…","config":"logical_deduction_five_objects","split":"main","body":"{\"input\":\"The following paragraphs each describe a set of five objects arranged in a fixed order. The statements are logically consistent within each paragraph. A fruit stand sells five fruits: watermelons, oranges, loquats, plums, and kiwis. The plums are less expensive than the kiwis. The plums are more expensive than the watermelons. The loquats are more expensive than the kiwis. The oranges are the most expensive.\\nOptions:\\n(A) The watermelons are the second-most expensive\\n(B) The oranges are the second-most expensive\\n(C) The loquats are the second-most expensive\\n(D) The plums are the second-most expensive\\n(E) The kiwis are the second-most expensive\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}