{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"93d83eae-6310-5231-aa80-8a23d51790dc","task_key":"causal~5fjudgement--main--93d83eae-6310-5231-aa80-8a23d51790dc","task_revision_id":"1","upstream_id":"","short_description":"How would a typical person answer each of the following questions about…","config":"causal_judgement","split":"main","body":"{\"input\":\"How would a typical person answer each of the following questions about causation?\\nLouie and Claire are playing a game of basketball, and they made a bet with their friends who are watching on the sidelines. If either Louie makes a 3-point shot or Claire makes a 3-point shot during the game, then they'll win $100 and split it down the middle. Just when the game started, Louie immediately got the ball at the 3-point line. He looked to the basket, focused his shot, and made a 3-point shot right at the beginning of the game. Louie, Claire, and their friends continued playing, but as hard as they tried, Louie and Claire couldn't make another shot. And then right at the end of the game as the clock was winding down, Claire got the ball at the 3-point line. She looked to the basket, focused her shot, and made a 3- point shot right at the buzzer. Then the game ended. Because they would win $100 if either Louie made a 3-point shot or Claire made a 3- point shot, Louie and Claire won $100 and split it down the middle. Think about what could have been different in the story that you just read. Which event do you imagine happening differently: Louie making the 3-point shot right at the beginning of the game. Claire making the 3-point shot right at the buzzer. Did Louie and Claire win the $100 bet because Louie made the 3-point shot?\\nOptions:\\n- Yes\\n- No\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}