{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbeh","formal_name":"BIG-Bench Extra Hard (BBEH)","introduction":"BIG-Bench Extra Hard evaluates language-model reasoning through challenging task categories. It offers full and mini distributions; this import contains the full distribution: 4,520 questions across all 23 task categories.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/google-deepmind/bbeh","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"af9422eb-4d74-508c-85f5-817c052e9a4c","task_key":"bbeh~5fsarc~5ftriples--full--af9422eb-4d74-508c-85f5-817c052e9a4c","task_revision_id":"1","upstream_id":"","short_description":"Here are three (post, reply) pairs from Reddit. Your task is to decide whether…","config":"bbeh_sarc_triples","split":"full","body":"{\"input\":\"Here are three (post, reply) pairs from Reddit. Your task is to decide whether each reply is sarcastic. Specifically, label each pair with a \\\"0\\\" or \\\"1\\\", wherea \\\"1\\\" indicates that the reply is sarcastic, and a \\\"0\\\" indicates that the reply does not contain sarcasm, and provide your final answer as a comma-separated set of labels (e.g., \\\"1,0,0\\\" or \\\"0,0,0\\\").\\n\\nPOST 1: might want to hot glue the bottle down or something, it's just a matter of time till the filament gets stuck on the spool or something and the bottle gets pulled over.\\nREPLY 1: You should swap to a bottle of everclear so you know if something goes wrong, the everclear will sterilize the area.\\n\\nPOST 2: Really sucks for teenagers who live with their parents who won't let them use their debit/credit card\\nREPLY 2: Steam is the ONLY way I can contact anyone of my gaming friends, I'm too hipster for Facebook Messages, too new school for email, too old school to tweet, too cool for text messaging or the apps that give me free texting to anyone.\\n\\nPOST 3: Endless line of workers getting artillery shells ready for attack at the National Shell Filling Factory at Chilwell, Nottinghamshire, UK 1917-07\\nREPLY 3: Imagine what humanity could accomplish if we possessed the will to mobilize like this to combat climate change or to explore the other planets.\\n\\n\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/google-deepmind/bbeh","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}