{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"9d806a3a-db8f-5bf0-a886-af34d8d9aa57","task_key":"bbeh~5fmultistep~5farithmetic--full--9d806a3a-db8f-5bf0-a886-af34d8d9aa57","task_revision_id":"1","upstream_id":"","short_description":"Consider the following new operations:","config":"bbeh_multistep_arithmetic","split":"full","body":"{\"input\":\"Consider the following new operations:\\n\\n$a # b$ equals $(a + b)$ if $a * b > 0$; otherwise,  it equals $-a + -b$.\\n$a : b$ equals $(a * a) + b$ if $b < 2$ and $(b * b) + a$ otherwise.\\n$a [] b$ equals $(a + b)$ if $a : b > 0$; otherwise,  it equals $a - b$.\\n$a <> b$ equals $(2 * b) + a$ if $a > b$ and $b$ otherwise.\\n$a ~ b$ equals $(a - b)$ if $a <> b > 0$; otherwise,  it equals $a + b$.\\n$a & b$ equals $(b + a) # b$ if $|a-b| < 2$ and $(a + b) # a$ otherwise.\\nFor brevity, we use $a <op1><op2> b$ to denote $(a op1 b) op2 b$. For example, $4 +* -5$ means $(4 + -5) * -5$ and $4 *-- -5$ means $(4 * -5) -- -5$.\\nLet A = ((((((((((((-7 -<> -3) #* three) ~ -1) ~ -1) []&[] 10) : three) [] -7) #+ -10) []&: 6) ~[]~ -1) - -10) []+~ five)\\nLet B = ((((-8 <>:# 5) -~ (-5 &#* -10)) [] ((8 &<>[] -7) #*+ (-4 *- two))) <><>+ (((-2 ~<>~ 7) -~[] (9 : -6)) -# ((-1 []<> -10) +#[] (-8 <>+ -3))))\\nLet C = (((((((((((((7 <>- -6) -: -10) ++* 10) :<> -2) [] -7) & -3) -* -4) <> nine) ~#: four) *# 8) & 3) :: five) <> one).\\nCompute A + B - C. Your final answer must be in number form.\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}