{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"8ba5e000-059b-53e7-bb92-ec1b5ed26718","task_key":"causal~5fjudgement--main--8ba5e000-059b-53e7-bb92-ec1b5ed26718","task_revision_id":"1","upstream_id":"","short_description":"How would a typical person answer each of the following questions about…","config":"causal_judgement","split":"main","body":"{\"input\":\"How would a typical person answer each of the following questions about causation?\\nA sniper has been ordered to kill an enemy commander. So, after getting himself into position, he finally has the enemy commander in his sights. Before he pulls the trigger, however, the sniper realizes that the gunfire will definitely alert the other enemy soldiers to his presence. But the sniper doesn't care at all about that -- he just wants to shoot his target. So, he pulls the trigger -- thereby shooting and killing the commander. And, as he expected, the enemy soldiers are alerted to his presence. Did the sniper intentionally alert the enemies to his presence?\\nOptions:\\n- Yes\\n- No\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}