{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"bbh","formal_name":"BIG-Bench Hard","introduction":"BIG-Bench Hard is the set of 23 BIG-Bench tasks on which models of the time scored below the average human rater. It is best known for showing how much chain-of-thought prompting changes the result.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","indexing_mode":"index","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[],"history":"","construction":"","limitations":"","saturation":"","scores":[]}},"task_id":"b1b3d30a-b41a-53c7-a20f-37bcc95accda","task_key":"salient~5ftranslation~5ferror~5fdetection--main--b1b3d30a-b41a-53c7-a20f-37bcc95accda","task_revision_id":"1","upstream_id":"","short_description":"The following translations from German to English contain a particular error.…","config":"salient_translation_error_detection","split":"main","body":"{\"input\":\"The following translations from German to English contain a particular error. That error will be one of the following types: Named Entities: An entity (names, places, locations, etc.) is changed to a different entity. Numerical Values: Numerical values (ordinals or cardinals), dates, and/or units are changed. Modifiers or Adjectives: The modifiers and adjectives pertaining to a noun are changed. Negation or Antonyms: Introduce or remove a negation or change comparatives to their antonyms. Facts: Trivial factual errors not pertaining to the above classes are introduced in the translations. Dropped Content: A significant clause in the translation is removed. Please identify that error.  Source: Die Liste der Kulturgüter in Freienstein-Teufen enthält alle Objekte in der Gemeinde Freienstein-Teufen im Kanton Zürich, die gemäss der Haager Konvention zum Schutz von Kulturgut bei bewaffneten Konflikten, dem Bundesgesetz vom 20.\\nTranslation: The list of cultural assets in Freienstein-Teufen contains all objects in the municipality of Freienstein-Teufen in the canton of Zurich, which, in accordance with the Hague Convention for the Protection of Cultural Property in the event of armed conflict, is protected by the Federal Act of 20 July 1898.\\nThe translation contains an error pertaining to\\nOptions:\\n(A) Modifiers or Adjectives\\n(B) Numerical Values\\n(C) Negation or Antonyms\\n(D) Named Entities\\n(E) Dropped Content\\n(F) Facts\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://github.com/suzgunmirac/BIG-Bench-Hard","history":"initial import","indexing_mode":"index","subproblems":[],"grids":[]}