{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"a5634102-5bb9-5343-abe1-3966d1784a7b","task_key":"bbeh~5fsarc~5ftriples--full--a5634102-5bb9-5343-abe1-3966d1784a7b","task_revision_id":"1","upstream_id":"","short_description":"Here are three (post, reply) pairs from Reddit. Your task is to decide whether…","config":"bbeh_sarc_triples","split":"full","body":"{\"input\":\"Here are three (post, reply) pairs from Reddit. Your task is to decide whether each reply is sarcastic. Specifically, label each pair with a \\\"0\\\" or \\\"1\\\", wherea \\\"1\\\" indicates that the reply is sarcastic, and a \\\"0\\\" indicates that the reply does not contain sarcasm, and provide your final answer as a comma-separated set of labels (e.g., \\\"1,0,0\\\" or \\\"0,0,0\\\").\\n\\nPOST 1: Complain about how you said \\\"Well, that's the The International then\\\" and make fun of you until new hatz\\nREPLY 1: Let's wait for the inevitable C9 disband and possible Secret disband, this year's post-TI roster changes will be the best.\\n\\nPOST 2: so when are the criminal charges for perpetrating this massive fraud scheme going to be handed down?\\nREPLY 2: Good, I hope they win, fraudulent crony banksters won't go to jail, but hopefully they will get hit where it hurts\\n\\nPOST 3: There should be a \\\"friendly horn\\\" in cars that makes a little chirp or something to thank other drivers.\\nREPLY 3: Horns and other warning devices You must not use the horn or any other warning device unless: Road rule in NSW say you can't use a horn unless You need to warn other road users that your vehicle is approaching You need to warn animals to get off the road The horn is being used as part of an anti-theft or alcohol interlock device fitted to your vehicle.\\n\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}