{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"48b3f563-07e7-5d65-9fad-2d0e8b70a89e","task_key":"salient~5ftranslation~5ferror~5fdetection--main--48b3f563-07e7-5d65-9fad-2d0e8b70a89e","task_revision_id":"1","upstream_id":"","short_description":"The following translations from German to English contain a particular error.…","config":"salient_translation_error_detection","split":"main","body":"{\"input\":\"The following translations from German to English contain a particular error. That error will be one of the following types: Named Entities: An entity (names, places, locations, etc.) is changed to a different entity. Numerical Values: Numerical values (ordinals or cardinals), dates, and/or units are changed. Modifiers or Adjectives: The modifiers and adjectives pertaining to a noun are changed. Negation or Antonyms: Introduce or remove a negation or change comparatives to their antonyms. Facts: Trivial factual errors not pertaining to the above classes are introduced in the translations. Dropped Content: A significant clause in the translation is removed. Please identify that error.  Source: Die Liste der Stolpersteine in Bochum enthält Stolpersteine, die im Rahmen des gleichnamigen Kunst-Projekts von Gunter Demnig in Bochum verlegt wurden.\\nTranslation: The list of stumbling blocks in Hokum contains stumbling blocks, which were laid as part of the art project of the same name by Hunter Demnig in Bochum.\\nThe translation contains an error pertaining to\\nOptions:\\n(A) Modifiers or Adjectives\\n(B) Numerical Values\\n(C) Negation or Antonyms\\n(D) Named Entities\\n(E) Dropped Content\\n(F) Facts\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}