{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"85d90c77-ea41-5b57-836a-2428c054e74e","task_key":"bbeh~5fsarc~5ftriples--full--85d90c77-ea41-5b57-836a-2428c054e74e","task_revision_id":"1","upstream_id":"","short_description":"Here are three (post, reply) pairs from Reddit. Your task is to decide whether…","config":"bbeh_sarc_triples","split":"full","body":"{\"input\":\"Here are three (post, reply) pairs from Reddit. Your task is to decide whether each reply is sarcastic. Specifically, label each pair with a \\\"0\\\" or \\\"1\\\", wherea \\\"1\\\" indicates that the reply is sarcastic, and a \\\"0\\\" indicates that the reply does not contain sarcasm, and provide your final answer as a comma-separated set of labels (e.g., \\\"1,0,0\\\" or \\\"0,0,0\\\").\\n\\nPOST 1: Props to Nine News for picturing the schoolboy at the target of Latham's comments, I'm sure that this is in the public interest and the media is fulfilling all of their responsibilities to those who the story covers and certainly won't be used for anything negative\\nREPLY 1: This is a man who was brought up badly, who displays none of the characteristics which made the older generation admirable.\\n\\nPOST 2: oh no.... what a nightmare... The horror of letting the era of Cruz and Mitch McConnell controlling the senate go to Bernie Sanders.\\nREPLY 2: Master mind of the New Left, Bernie Sanders reveals himself to be his own political splinter group and the reincarnation of Noam Chomsky.\\n\\nPOST 3: Think of all the glitter bombs they use during their parades and the connection is all the more apparent!\\nREPLY 3: Looking forward to laughing at Kenya, Uganda and Russia in 50 years when they're better educated and begging for forgiveness\\n\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}