{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"11e81268-297e-564b-bff6-8c4dce50b0fc","task_key":"salient~5ftranslation~5ferror~5fdetection--main--11e81268-297e-564b-bff6-8c4dce50b0fc","task_revision_id":"1","upstream_id":"","short_description":"The following translations from German to English contain a particular error.…","config":"salient_translation_error_detection","split":"main","body":"{\"input\":\"The following translations from German to English contain a particular error. That error will be one of the following types: Named Entities: An entity (names, places, locations, etc.) is changed to a different entity. Numerical Values: Numerical values (ordinals or cardinals), dates, and/or units are changed. Modifiers or Adjectives: The modifiers and adjectives pertaining to a noun are changed. Negation or Antonyms: Introduce or remove a negation or change comparatives to their antonyms. Facts: Trivial factual errors not pertaining to the above classes are introduced in the translations. Dropped Content: A significant clause in the translation is removed. Please identify that error.  Source: Als Fassadenbrand werden Ereignisse bezeichnet, bei denen Teile der Fassade eines Gebäudes in Brand geraten.\\nTranslation: A facade fire refers to events in which parts of the facade of a building are set on ice.\\nThe translation contains an error pertaining to\\nOptions:\\n(A) Modifiers or Adjectives\\n(B) Numerical Values\\n(C) Negation or Antonyms\\n(D) Named Entities\\n(E) Dropped Content\\n(F) Facts\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}