{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"ddb26c86-ff98-5200-be43-b431ae42ffc8","task_key":"causal~5fjudgement--main--ddb26c86-ff98-5200-be43-b431ae42ffc8","task_revision_id":"1","upstream_id":"","short_description":"How would a typical person answer each of the following questions about…","config":"causal_judgement","split":"main","body":"{\"input\":\"How would a typical person answer each of the following questions about causation?\\nThe city planner's assistant came to him and said, \\\"We finally developed a plan to address our pollution problem. We have a new reclamation project that will clean up the toxic waste polluting the former industrial area. However, if we actually do implement the project, it will also increase the levels of joblessness.\\\" The city planner answered, \\\"I feel terrible about increasing joblessness. But we have to do something about our pollution problem. Let's start the project.\\\" They started the project. Sure enough, the toxic waste polluting the former industrial area was cleaned up, and joblessness level increased. Did the city planner increase levels of joblessness intentionally?\\nOptions:\\n- Yes\\n- No\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}