{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"longbench-v2","formal_name":"LongBench v2","introduction":"LongBench v2 evaluates deep understanding and reasoning over long contexts through multiple-choice questions. Its official description lists 503 questions spanning tasks such as single-document and multi-document QA and code-repository understanding.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/zai-org/LongBench-v2","indexing_mode":"noindex","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[]}},"task_id":"ed8b621f-861c-5205-ae81-fe3987001147","task_key":"train--66f53892821e116aacb33225","task_revision_id":"3","upstream_id":"66f53892821e116aacb33225","short_description":"Which of the following methods is used in this article to calculate the…","config":"","split":"train","body":"{\"choice_A\":\"After converting the document content to bert embedding, calculate the n-grams that occur in the reference and candidate\",\"choice_B\":\"For each word in the generated text, select the word in the reference text that is most similar to it, and calculate the maximum cosine similarity of the BERT hidden layer representation\",\"choice_C\":\"For each word in the reference text, select the word in the generated text that is most similar to it, and calculate the maximum cosine similarity of the BERT hidden layer representation\",\"choice_D\":\"The evaluation of the generated text is regarded as a text generation task, and unsupervised learning is used to evaluate different aspects of the generated text (e.g. informativeness, fluency, or factuality)\",\"context\":\"BERTSCORE: EVALUATING TEXT GENERATION WITH\\nBERT\\nABSTRACT\\nWe propose BERTSCORE, an automatic evaluation metric for text generation.\\nAnalogously to common metrics, BERTSCORE computes a similarity score for\\neach token in the candidate sentence with each token in the reference sentence.\\nHowever, instead of exact matches, we compute token similarity using contextual\\nembeddings. We evaluate using the outputs of 363 machine translation and image\\ncaptioning systems. BERTSCORE correlates better with human judgments and\\nprovides stronger model selection performance than existing metrics. Finally, we\\nuse an adversarial paraphrase detection task to show that BERTSCORE is more\\nrobust to challenging examples when compared to existing metrics.\\n1\\nINTRODUCTION\\nAutomatic evaluation of natural language generation, for example in machine translation and caption\\ngeneration, requires comparing candidate sentences to annotated references. The goal is to evaluate\\nsemantic equivalence. However, commonly used methods rely on surface-form similarity only. For\\nexample, BLEU (Papineni et al., 2002), the most common machine translation metric, simply counts\\nn-gram overlap between the candidate and the reference. While this provides a simple and general\\nmeasure, it fails to account for meaning-preserving lexical and compositional diversity.\\nIn this paper, we introduce BERTSCORE, a language generation evaluation metric based on pre-\\ntrained BERT contextual embeddings (Devlin et al., 2019). BERTSCORE computes the similarity\\nof two sentences as a sum of cosine similarities between their tokens’ embeddings.\\nBERTSCORE addresses two common pitfalls in n-gram-based metrics (Banerjee & Lavie, 2005).\\nFirst, such methods often fail to robustly match paraphrases. For example, given the reference peo-\\nple like foreign cars, BLEU and METEOR (Banerjee & Lavie, 2005) incorrectly give a higher score\\nto people like visiting places abroad compared to consumers prefer imported cars. This leads to\\nperformance underestimation when semantically-correct phrases are penalized because they differ\\nfrom the surface form of the reference. In contrast to string matching (e.g., in BLEU) or matching\\nheuristics (e.g., in METEOR), we compute similarity using contextualized token embeddings, which\\nhave been shown to be effective for paraphrase detection (Devlin et al., 2019). Second, n-gram mod-\\nels fail to capture distant dependencies and penalize semantically-critical ordering changes (Isozaki\\net al., 2010). For example, given a small window of size two, BLEU will only mildly penalize\\nswapping of cause and effect clauses (e.g. A because B instead of B because A), especially when\\nthe arguments A and B are long phrases. In contrast, contextualized embeddings are trained to\\neffectively capture distant dependencies and ordering.\\nWe experiment with BERTSCORE on machine translation and image captioning tasks using the\\noutputs of 363 systems by correlating BERTSCORE and related metrics to available human judg-\\nments. Our experiments demonstrate that BERTSCORE correlates highly with human evaluations.\\nIn machine translation, BERTSCORE shows stronger system-level and segment-level correlations\\nwith human judgments than existing metrics on multiple common benchmarks and demonstrates\\n∗Equal contribution. † Work done at Cornell.\\n1\\narXiv:1904.09675v3  [cs.CL]  24 Feb 2020\\n\\n\\nPublished as a conference paper at ICLR 2020\\nstrong model selection performance compared to BLEU.\\nWe also show that BERTSCORE is\\nwell-correlated with human annotators for image captioning, surpassing SPICE, a popular task-\\nspecific metric (Anderson et al., 2016).\\nFinally, we test the robustness of BERTSCORE on\\nthe adversarial paraphrase dataset PAWS (Zhang et al., 2019), and show that it is more ro-\\nbust to adversarial examples than other metrics.\\nThe code for BERTSCORE is available at\\nhttps://github.com/Tiiiger/bert_score.\\n2\\nPROBLEM STATEMENT AND PRIOR METRICS\\nNatural language text generation is commonly evaluated using annotated reference sentences. Given\\na reference sentence x tokenized to k tokens ⟨x1, . . . , xk⟩and a candidate ˆ\\nx tokenized to l tokens\\n⟨ˆ\\nx1, . . . , ˆ\\nxl⟩, a generation evaluation metric is a function f(x, ˆ\\nx) ∈R. Better metrics have a higher\\ncorrelation with human judgments. Existing metrics can be broadly categorized into using n-gram\\nmatching, edit distance, embedding matching, or learned functions.\\n2.1\\nn-GRAM MATCHING APPROACHES\\nThe most commonly used metrics for generation count the number of n-grams that occur in the\\nreference x and candidate ˆ\\nx. The higher the n is, the more the metric is able to capture word order,\\nbut it also becomes more restrictive and constrained to the exact form of the reference.\\nFormally, let Sn\\nx and Sn\\nˆ\\nx be the lists of token n-grams (n ∈Z+) in the reference x and candidate\\nˆ\\nx sentences. The number of matched n-grams is P\\nw∈Sn\\nˆ\\nx I[w ∈Sn\\nx], where I[·] is an indicator\\nfunction. The exact match precision (Exact-Pn) and recall (Exact-Rn) scores are:\\nExact-Pn =\\nP\\nw∈Sn\\nˆ\\nx I[w ∈Sn\\nx]\\n|Sn\\nˆ\\nx|\\nand\\nExact-Rn =\\nP\\nw∈Sn\\nx I[w ∈Sn\\nˆ\\nx]\\n|Sn\\nx|\\n.\\nSeveral popular metrics build upon one or both of these exact matching scores.\\nBLEU\\nThe most widely used metric in machine translation is BLEU (Papineni et al., 2002), which\\nincludes three modifications to Exact-Pn. First, each n-gram in the reference can be matched at\\nmost once. Second, the number of exact matches is accumulated for all reference-candidate pairs in\\nthe corpus and divided by the total number of n-grams in all candidate sentences. Finally, very short\\ncandidates are discouraged using a brevity penalty. Typically, BLEU is computed for multiple values\\nof n (e.g. n = 1, 2, 3, 4) and the scores are averaged geometrically. A smoothed variant, SENT-\\nBLEU (Koehn et al., 2007) is computed at the sentence level. In contrast to BLEU, BERTSCORE is\\nnot restricted to maximum n-gram length, but instead relies on contextualized embeddings that are\\nable to capture dependencies of potentially unbounded length.\\nMETEOR\\nMETEOR (Banerjee & Lavie, 2005) computes Exact-P1 and Exact-R1 while allowing\\nbacking-off from exact unigram matching to matching word stems, synonyms, and paraphrases. For\\nexample, running may match run if no exact match is possible. Non-exact matching uses an external\\nstemmer, a synonym lexicon, and a paraphrase table. METEOR 1.5 (Denkowski & Lavie, 2014)\\nweighs content and function words differently, and also applies importance weighting to different\\nmatching types. The more recent METEOR++ 2.0 (Guo & Hu, 2019) further incorporates a learned\\nexternal paraphrase resource. Because METEOR requires external resources, only five languages\\nare supported with the full feature set, and eleven are partially supported. Similar to METEOR,\\nBERTSCORE allows relaxed matches, but relies on BERT embeddings that are trained on large\\namounts of raw text and are currently available for 104 languages. BERTSCORE also supports\\nimportance weighting, which we estimate with simple corpus statistics.\\nOther Related Metrics\\nNIST (Doddington, 2002) is a revised version of BLEU that weighs each\\nn-gram differently and uses an alternative brevity penalty. ∆BLEU (Galley et al., 2015) modifies\\nmulti-reference BLEU by including human annotated negative reference sentences. CHRF (Popovi´\\nc,\\n2015) compares character n-grams in the reference and candidate sentences. CHRF++ (Popovi´\\nc,\\n2017) extends CHRF to include word bigram matching. ROUGE (Lin, 2004) is a commonly used\\nmetric for summarization evaluation. ROUGE-n (Lin, 2004) computes Exact-Rn (usually n = 1, 2),\\nwhile ROUGE-L is a variant of Exact-R1 with the numerator replaced by the length of the longest\\ncommon subsequence. CIDER (Vedantam et al., 2015) is an image captioning metric that computes\\n2\\n\\n\\nPublished as a conference paper at ICLR 2020\\ncosine similarity between tf–idf weighted n-grams. We adopt a similar approach to weigh tokens\\ndifferently. Finally, Chaganty et al. (2018) and Hashimoto et al. (2019) combine automatic metrics\\nwith human judgments for text generation evaluation.\\n2.2\\nEDIT-DISTANCE-BASED METRICS\\nSeveral methods use word edit distance or word error rate (Levenshtein, 1966), which quantify\\nsimilarity using the number of edit operations required to get from the candidate to the refer-\\nence. TER (Snover et al., 2006) normalizes edit distance by the number of reference words, and\\nITER (Panja & Naskar, 2018) adds stem matching and better normalization. PER (Tillmann et al.,\\n1997) computes position independent error rate, CDER (Leusch et al., 2006) models block reorder-\\ning as an edit operation. CHARACTER (Wang et al., 2016) and EED (Stanchev et al., 2019) operate\\non the character level and achieve higher correlation with human judgements on some languages.\\n2.3\\nEMBEDDING-BASED METRICS\\nWord embeddings (Mikolov et al., 2013; Pennington et al., 2014; Grave et al., 2018; Nguyen et al.,\\n2017; Athiwaratkun et al., 2018) are learned dense token representations. MEANT 2.0 (Lo, 2017)\\nuses word embeddings and shallow semantic parses to compute lexical and structural similarity.\\nYISI-1 (Lo et al., 2018) is similar to MEANT 2.0, but makes the use of semantic parses optional.\\nBoth methods use a relatively simple similarity computation, which inspires our approach, including\\nusing greedy matching (Corley & Mihalcea, 2005) and experimenting with a similar importance\\nweighting to YISI-1. However, we use contextual embeddings, which capture the specific use of\\na token in a sentence, and potentially capture sequence information. We do not use external tools\\nto generate linguistic structures, which makes our approach relatively simple and portable to new\\nlanguages. Instead of greedy matching, WMD (Kusner et al., 2015), WMDO (Chow et al., 2019), and\\nSMS (Clark et al., 2019) propose to use optimal matching based on earth mover’s distance (Rubner\\net al., 1998). The tradeoff1 between greedy and optimal matching was studied by Rus & Lintean\\n(2012). Sharma et al. (2018) compute similarity with sentence-level representations. In contrast, our\\ntoken-level computation allows us to weigh tokens differently according to their importance.\\n2.4\\nLEARNED METRICS\\nVarious metrics are trained to optimize correlation with human judgments. BEER (Stanojevi´\\nc &\\nSima’an, 2014) uses a regression model based on character n-grams and word bigrams. BLEND (Ma\\net al., 2017) uses regression to combine 29 existing metrics. RUSE (Shimanaka et al., 2018) com-\\nbines three pre-trained sentence embedding models. All these methods require costly human judg-\\nments as supervision for each dataset, and risk poor generalization to new domains, even within a\\nknown language and task (Chaganty et al., 2018). Cui et al. (2018) and Lowe et al. (2017) train a\\nneural model to predict if the input text is human-generated. This approach also has the risk of being\\noptimized to existing data and generalizing poorly to new data. In contrast, the model underlying\\nBERTSCORE is not optimized for any specific evaluation task.\\n3\\nBERTSCORE\\nGiven a reference sentence x = ⟨x1, . . . , xk⟩and a candidate sentence ˆ\\nx = ⟨ˆ\\nx1, . . . , ˆ\\nxl⟩, we use\\ncontextual embeddings to represent the tokens, and compute matching using cosine similarity, op-\\ntionally weighted with inverse document frequency scores. Figure 1 illustrates the computation.\\nToken Representation\\nWe use contextual embeddings to represent the tokens in the input sen-\\ntences x and ˆ\\nx. In contrast to prior word embeddings (Mikolov et al., 2013; Pennington et al.,\\n2014), contextual embeddings, such as BERT (Devlin et al., 2019) and ELMO (Peters et al., 2018),\\ncan generate different vector representations for the same word in different sentences depending on\\nthe surrounding words, which form the context of the target word. The models used to generate\\nthese embeddings are most commonly trained using various language modeling objectives, such as\\nmasked word prediction (Devlin et al., 2019).\\n1We provide an ablation study of this design choice in Appendix C.\\n3\\n\\n\\nPublished as a conference paper at ICLR 2020\\nReference\\nthe weather is \\ncold today\\nCandidate\\nit is freezing today\\nCandidate\\nContextual\\nEmbedding\\nPairwise Cosine\\nSimilarity\\nRBERT = (0.713⇥1.27)+(0.515⇥7.94)+...\\n1.27+7.94+1.82+7.90+8.88\\n<latexit sha1_base64=\\\"OJyoKlmBAgUA0KDtUcsH/di5BlI=\\\">ACSHicbZDLatAFIaPnLRJ3JvTLrsZYgoJAqFxGqwsCqal0FVJQ5wELCNG41EyZHRh5ijECL1EnqAv02X2eUZsumipXR6Mj2Ipf+MPDznXM4Z/64UNKg7187raXlR49XVtfaT54+e/6is/7y0OSl5mLIc5Xr45gZoWQmhihRieNC5bGShzFZx+a+tG50\\nEbm2QFOCzFO2UkmE8kZWhR1ov2oClFcYPX+4/5BXZN3JEw049Wm7/XpdogyFYZQr9fci3aoTsL1Pd23265oZrkaOqaXAb5FIv6DXOdwMvCOqo0/U9fyby0NCF6Q52/15+BYC9qHMVTnJepiJDrpgxI+oXOK6YRsmVqNthaUTB+Bk7ESNrM2aPGVezIGryxpIJSXJtX4ZkRm9PVCw1ZprGtjNleGru1xr4v9qoxCQYVzIrShQZny9KSkUwJ02qZCK14Kim1jCupb2V8FNmc0SbfduGQO9/+aE57HnU9+gX2h18hrlW4TVswCZQ6MAPsEeDIHDN7iBn/DL+e78cH47f+atLWcx8wruqNX6B8dUrVw=</latexit>\\n<latexit sha1_base64=\\\"RInTcZkWiVBnf/ncBstCvatCtG4=\\\">ACSHicbZDPShxBEMZ7Nproxugaj14al4AyMEyvyoyHwGIQPImKq8LOMvT09mhjzx+6a0KWYV4iL5EnySXH3HwGLx4U8SDYs7sHo/mg4eNXVT1F+VSaHDda6vxbmb2/Ye5+ebHhU+LS63lz6c6KxTjPZbJTJ1HVHMpUt4DAZKf54rTJL8Lr6VtfPvnOlR\\nZaewCjng4RepCIWjIJBYSs8DsA+A8od/eOT6oKf8VBrCgr13HI5sBiIRrTJyOt2EbtE2p8hzdrY27EAOM9BVWTfYNbKJ43dq59q+4/tV2Gq7jsWfmvI1LS7O08/f3nLi4dh628wzFiR8BSYpFr3iZvDoKQKBJO8agaF5jlV/SC941NqTlmUI6DqPAXQ4Y4zpR5KeAxfTlR0kTrURKZzoTCpX5dq+H/av0CYn9QijQvgKdsiguJIYM16nioVCcgRwZQ5kS5lbMLqnJEUz2TRMCef3lt+a04xDXIUek3T1AE82hVbSG1hFBHuqifXSIeoih3+gG3aF76491az1Yj5PWhjWdWUH/qNF4BkPYrbk=</latexit>\\n<latexit sha1_base64=\\\"RInTcZkWiVBnf/ncBstCvatCtG4=\\\">ACSHicbZDPShxBEMZ7Nproxugaj14al4AyMEyvyoyHwGIQPImKq8LOMvT09mhjzx+6a0KWYV4iL5EnySXH3HwGLx4U8SDYs7sHo/mg4eNXVT1F+VSaHDda6vxbmb2/Ye5+ebHhU+LS63lz6c6KxTjPZbJTJ1HVHMpUt4DAZKf54rTJL8Lr6VtfPvnOlR\\nZaewCjng4RepCIWjIJBYSs8DsA+A8od/eOT6oKf8VBrCgr13HI5sBiIRrTJyOt2EbtE2p8hzdrY27EAOM9BVWTfYNbKJ43dq59q+4/tV2Gq7jsWfmvI1LS7O08/f3nLi4dh628wzFiR8BSYpFr3iZvDoKQKBJO8agaF5jlV/SC941NqTlmUI6DqPAXQ4Y4zpR5KeAxfTlR0kTrURKZzoTCpX5dq+H/av0CYn9QijQvgKdsiguJIYM16nioVCcgRwZQ5kS5lbMLqnJEUz2TRMCef3lt+a04xDXIUek3T1AE82hVbSG1hFBHuqifXSIeoih3+gG3aF76491az1Yj5PWhjWdWUH/qNF4BkPYrbk=</latexit>\\n<latexit sha1_base64=\\\"fGWl4NCvlvtMu17rjLtk25oWpdc=\\\">ACSHicbZBLS+RAFIUrPT7bVzsu3RQ2ghIqVbpuBgQRZiVqNgqdJpQqa5oYeVB1Y1ME/Lz3Lic3fwGNy6UwZ2VNgtfBwoO372Xe+uEmRQaXPef1fgxMTk1PTPbnJtfWFxqLf812muGO+xVKbqMqSaS5HwHgiQ/DJTnMah5BfhzUFVv7jlSos0OYNRxgcxv\\nUpEJBgFg4JWcBoUPvA/UOwfnp6VJf6F/UhRVmy4Tpds+SBirjFxOt1N26AdslOjrO7vWn7cpiCLouqwa6QTRyvUznX9hzPK4NW23XcsfBXQ2rTRrWOg9Zf5iyPOYJMEm17hM3g0FBFQgmedn0c80zym7oFe8bm1BzKAYB1HidUOGOEqVeQngMX0/UdBY61Ecms6YwrX+XKvgd7V+DpE3KES5cAT9rYoyiWGFep4qFQnIEcGUOZEuZWzK6pyRFM9k0TAvn85a/mvOMQ1yEnpL13VMcxg1bRGtpABHXRHvqNjlEPMXSHtATerburUfrv/Xy1tqw6pkV9EGNxisxMKq0</latexit>\\n1.27\\n7.94\\n1.82\\n7.90\\n8.88\\nidf \\nweights\\nImportance Weighting\\n(Optional)\\nMaximum Similarity\\nx\\n<latexit sha1_base64=\\\"f2yzimwbR/Dgjzp6tZ\\n360fHRqNI=\\\">AB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3btZhN2N2IJ/QV\\nePCji1Z/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZLGLVCahGwSU\\n2DTcCO4lCGgUC28H4dua3H1FpHst7M0nQj+hQ8pAzaqzUeOqXK27VnYOsEi8nFchR75e/eoOYpRFKw\\nTVu5ifEzqgxnAqelXqoxoWxMh9i1VNItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/\\n7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2oszYbEo2BG/5VXSuqh6btVrXFZqN3kcR\\nTiBUzgHD6gBndQhyYwQHiGV3hzHpwX5935WLQWnHzmGP7A+fwB5jmM/A=</latexit>\\n<latexit sha1_base64=\\\"f2yzimwbR/Dgjzp6tZ\\n360fHRqNI=\\\">AB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3btZhN2N2IJ/QV\\nePCji1Z/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZLGLVCahGwSU\\n2DTcCO4lCGgUC28H4dua3H1FpHst7M0nQj+hQ8pAzaqzUeOqXK27VnYOsEi8nFchR75e/eoOYpRFKw\\nTVu5ifEzqgxnAqelXqoxoWxMh9i1VNItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/\\n7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2oszYbEo2BG/5VXSuqh6btVrXFZqN3kcR\\nTiBUzgHD6gBndQhyYwQHiGV3hzHpwX5935WLQWnHzmGP7A+fwB5jmM/A=</latexit>\\n<latexit sha1_base64=\\\"f2yzimwbR/Dgjzp6tZ\\n360fHRqNI=\\\">AB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3btZhN2N2IJ/QV\\nePCji1Z/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZLGLVCahGwSU\\n2DTcCO4lCGgUC28H4dua3H1FpHst7M0nQj+hQ8pAzaqzUeOqXK27VnYOsEi8nFchR75e/eoOYpRFKw\\nTVu5ifEzqgxnAqelXqoxoWxMh9i1VNItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/\\n7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2oszYbEo2BG/5VXSuqh6btVrXFZqN3kcR\\nTiBUzgHD6gBndQhyYwQHiGV3hzHpwX5935WLQWnHzmGP7A+fwB5jmM/A=</latexit>\\n<latexit sha1_base64=\\\"f2yzimwbR/Dgjzp6tZ\\n360fHRqNI=\\\">AB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cW7Ae0oWy2k3btZhN2N2IJ/QV\\nePCji1Z/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZLGLVCahGwSU\\n2DTcCO4lCGgUC28H4dua3H1FpHst7M0nQj+hQ8pAzaqzUeOqXK27VnYOsEi8nFchR75e/eoOYpRFKw\\nTVu5ifEzqgxnAqelXqoxoWxMh9i1VNItZ/ND52SM6sMSBgrW9KQufp7IqOR1pMosJ0RNSO97M3E/\\n7xuasJrP+MySQ1KtlgUpoKYmMy+JgOukBkxsYQyxe2thI2oszYbEo2BG/5VXSuqh6btVrXFZqN3kcR\\nTiBUzgHD6gBndQhyYwQHiGV3hzHpwX5935WLQWnHzmGP7A+fwB5jmM/A=</latexit>\\nˆ\\nx\\n<latexit sha1_base64=\\\"5QTnVRVSrnyznVU7d5bF\\n5u03Iw=\\\">AB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDbTbt0swm7E7GE/gvHhTx6u/\\nx5r9x0+agrQ8GHu/NMDMvSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlslrHuBtRwKRvoUDJu4nmNAo\\nk7wST29zvPHJtRKwecJpwP6IjJULBKFqp0x9TzJ5mg2rNrbtzkFXiFaQGBZqD6ld/GLM04gqZpMb0PDdBP6M\\naBZN8VumnhieUTeiI9yxVNOLGz+bnzsiZVYkjLUthWSu/p7IaGTMNApsZ0RxbJa9XPzP6UYXvuZUEmKXLH\\nFojCVBGOS/06GQnOGcmoJZVrYWwkbU0Z2oQqNgRv+eV0r6oe27du7+sNW6KOMpwAqdwDh5cQPuoAktYDCB\\nZ3iFNydxXpx352PRWnKmWP4A+fzB7A8j8k=</latexit>\\n<latexit sha1_base64=\\\"5QTnVRVSrnyznVU7d5bF\\n5u03Iw=\\\">AB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDbTbt0swm7E7GE/gvHhTx6u/\\nx5r9x0+agrQ8GHu/NMDMvSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlslrHuBtRwKRvoUDJu4nmNAo\\nk7wST29zvPHJtRKwecJpwP6IjJULBKFqp0x9TzJ5mg2rNrbtzkFXiFaQGBZqD6ld/GLM04gqZpMb0PDdBP6M\\naBZN8VumnhieUTeiI9yxVNOLGz+bnzsiZVYkjLUthWSu/p7IaGTMNApsZ0RxbJa9XPzP6UYXvuZUEmKXLH\\nFojCVBGOS/06GQnOGcmoJZVrYWwkbU0Z2oQqNgRv+eV0r6oe27du7+sNW6KOMpwAqdwDh5cQPuoAktYDCB\\nZ3iFNydxXpx352PRWnKmWP4A+fzB7A8j8k=</latexit>\\n<latexit sha1_base64=\\\"5QTnVRVSrnyznVU7d5bF\\n5u03Iw=\\\">AB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDbTbt0swm7E7GE/gvHhTx6u/\\nx5r9x0+agrQ8GHu/NMDMvSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlslrHuBtRwKRvoUDJu4nmNAo\\nk7wST29zvPHJtRKwecJpwP6IjJULBKFqp0x9TzJ5mg2rNrbtzkFXiFaQGBZqD6ld/GLM04gqZpMb0PDdBP6M\\naBZN8VumnhieUTeiI9yxVNOLGz+bnzsiZVYkjLUthWSu/p7IaGTMNApsZ0RxbJa9XPzP6UYXvuZUEmKXLH\\nFojCVBGOS/06GQnOGcmoJZVrYWwkbU0Z2oQqNgRv+eV0r6oe27du7+sNW6KOMpwAqdwDh5cQPuoAktYDCB\\nZ3iFNydxXpx352PRWnKmWP4A+fzB7A8j8k=</latexit>\\n<latexit sha1_base64=\\\"5QTnVRVSrnyznVU7d5bF\\n5u03Iw=\\\">AB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cK9gPaUDbTbt0swm7E7GE/gvHhTx6u/\\nx5r9x0+agrQ8GHu/NMDMvSKQw6LrfTmltfWNzq7xd2dnd2z+oHh61TZxqxlslrHuBtRwKRvoUDJu4nmNAo\\nk7wST29zvPHJtRKwecJpwP6IjJULBKFqp0x9TzJ5mg2rNrbtzkFXiFaQGBZqD6ld/GLM04gqZpMb0PDdBP6M\\naBZN8VumnhieUTeiI9yxVNOLGz+bnzsiZVYkjLUthWSu/p7IaGTMNApsZ0RxbJa9XPzP6UYXvuZUEmKXLH\\nFojCVBGOS/06GQnOGcmoJZVrYWwkbU0Z2oQqNgRv+eV0r6oe27du7+sNW6KOMpwAqdwDh5cQPuoAktYDCB\\nZ3iFNydxXpx352PRWnKmWP4A+fzB7A8j8k=</latexit>\\nReference\\nFigure 1: Illustration of the computation of the recall metric RBERT. Given the reference x and\\ncandidate ˆ\\nx, we compute BERT embeddings and pairwise cosine similarity. We highlight the greedy\\nmatching in red, and include the optional idf importance weighting.\\nWe experiment with different models (Section 4), using the tokenizer provided with each model.\\nGiven a tokenized reference sentence x = ⟨x1, . . . , xk⟩, the embedding model generates a se-\\nquence of vectors ⟨x1, . . . , xk⟩. Similarly, the tokenized candidate ˆ\\nx = ⟨ˆ\\nx1, . . . , ˆ\\nxm⟩is mapped\\nto ⟨ˆ\\nx1, . . . , ˆ\\nxl⟩. The main model we use is BERT, which tokenizes the input text into a sequence\\nof word pieces (Wu et al., 2016), where unknown words are split into several commonly observed\\nsequences of characters. The representation for each word piece is computed with a Transformer\\nencoder (Vaswani et al., 2017) by repeatedly applying self-attention and nonlinear transformations\\nin an alternating fashion. BERT embeddings have been shown to benefit various NLP tasks (Devlin\\net al., 2019; Liu, 2019; Huang et al., 2019; Yang et al., 2019a).\\nSimilarity Measure\\nThe vector representation allows for a soft measure of similarity instead of\\nexact-string (Papineni et al., 2002) or heuristic (Banerjee & Lavie, 2005) matching. The cosine\\nsimilarity of a reference token xi and a candidate token ˆ\\nxj is\\nx⊤\\ni ˆ\\nxj\\n∥xi∥∥ˆ\\nxj∥. We use pre-normalized\\nvectors, which reduces this calculation to the inner product x⊤\\ni ˆ\\nxj. While this measure considers\\ntokens in isolation, the contextual embeddings contain information from the rest of the sentence.\\nBERTSCORE\\nThe complete score matches each token in x to a token in ˆ\\nx to compute recall,\\nand each token in ˆ\\nx to a token in x to compute precision. We use greedy matching to maximize\\nthe matching similarity score,2 where each token is matched to the most similar token in the other\\nsentence. We combine precision and recall to compute an F1 measure. For a reference x and\\ncandidate ˆ\\nx, the recall, precision, and F1 scores are:\\nRBERT = 1\\n|x|\\nX\\nxi∈x\\nmax\\nˆ\\nxj∈ˆ\\nx x⊤\\ni ˆ\\nxj ,\\nPBERT = 1\\n|ˆ\\nx|\\nX\\nˆ\\nxj∈ˆ\\nx\\nmax\\nxi∈x x⊤\\ni ˆ\\nxj ,\\nFBERT = 2 PBERT · RBERT\\nPBERT + RBERT\\n.\\nImportance Weighting\\nPrevious work on similarity measures demonstrated that rare words can\\nbe more indicative for sentence similarity than common words (Banerjee & Lavie, 2005; Vedantam\\net al., 2015). BERTSCORE enables us to easily incorporate importance weighting. We experiment\\nwith inverse document frequency (idf) scores computed from the test corpus. Given M reference\\nsentences {x(i)}M\\ni=1, the idf score of a word-piece token w is\\nidf(w) = −log 1\\nM\\nM\\nX\\ni=1\\nI[w ∈x(i)] ,\\nwhere I[·] is an indicator function. We do not use the full tf-idf measure because we process single\\nsentences, where the term frequency (tf) is likely 1. For example, recall with idf weighting is\\nRBERT =\\nP\\nxi∈x idf(xi) maxˆ\\nxj∈ˆ\\nx x⊤\\ni ˆ\\nxj\\nP\\nxi∈x idf(xi)\\n.\\nBecause we use reference sentences to compute idf, the idf scores remain the same for all systems\\nevaluated on a specific test set. We apply plus-one smoothing to handle unknown word pieces.\\n2We compare greedy matching with optimal assignment in Appendix C.\\n4\\n\\n\\nPublished as a conference paper at ICLR 2020\\nBaseline Rescaling\\nBecause we use pre-normalized vectors, our computed scores have the same\\nnumerical range of cosine similarity (between −1 and 1). However, in practice we observe scores\\nin a more limited range, potentially because of the learned geometry of contextual embeddings.\\nWhile this characteristic does not impact BERTSCORE’s capability to rank text generation systems,\\nit makes the actual score less readable. We address this by rescaling BERTSCORE with respect\\nto its empirical lower bound b as a baseline. We compute b using Common Crawl monolingual\\ndatasets.3 For each language and contextual embedding model, we create 1M candidate-reference\\npairs by grouping two random sentences. Because of the random pairing and the corpus diversity,\\neach pair has very low lexical and semantic overlapping.4 We compute b by averaging BERTSCORE\\ncomputed on these sentence pairs. Equipped with baseline b, we rescale BERTSCORE linearly. For\\nexample, the rescaled value ˆ\\nRBERT of RBERT is:\\nˆ\\nRBERT = RBERT −b\\n1 −b\\n.\\nAfter this operation ˆ\\nRBERT is typically between 0 and 1. We apply the same rescaling procedure\\nfor PBERT and FBERT. This method does not affect the ranking ability and human correlation of\\nBERTSCORE, and is intended solely to increase the score readability.\\n4\\nEXPERIMENTAL SETUP\\nWe evaluate our approach on machine translation and image captioning.\\nContextual Embedding Models\\nWe evaluate twelve pre-trained contextual embedding models,\\nincluding variants of BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019b), XLNet (Yang et al.,\\n2019b), and XLM (Lample & Conneau, 2019). We present the best-performing models in Section 5.\\nWe use the 24-layer RoBERTalarge model5 for English tasks, 12-layer BERTchinese model for Chi-\\nnese tasks, and the 12-layer cased multilingual BERTmulti model for other languages.6 We show the\\nperformance of all other models in Appendix F. Contextual embedding models generate embedding\\nrepresentations at every layer in the encoder network. Past work has shown that intermediate layers\\nproduce more effective representations for semantic tasks (Liu et al., 2019a). We use the WMT16\\ndataset (Bojar et al., 2016) as a validation set to select the best layer of each model (Appendix B).\\nMachine Translation\\nOur main evaluation corpus is the WMT18 metric evaluation dataset (Ma\\net al., 2018), which contains predictions of 149 translation systems across 14 language pairs, gold\\nreferences, and two types of human judgment scores. Segment-level human judgments assign a score\\nto each reference-candidate pair. System-level human judgments associate each system with a single\\nscore based on all pairs in the test set. WMT18 includes translations from English to Czech, German,\\nEstonian, Finnish, Russian, and Turkish, and from the same set of languages to English. We follow\\nthe WMT18 standard practice and use absolute Pearson correlation |ρ| and Kendall rank correlation\\nτ to evaluate metric quality, and compute significance with the Williams test (Williams, 1959) for |ρ|\\nand bootstrap re-sampling for τ as suggested by Graham & Baldwin (2014). We compute system-\\nlevel scores by averaging BERTSCORE for every reference-candidate pair. We also experiment with\\nhybrid systems by randomly sampling one candidate sentence from one of the available systems for\\neach reference sentence (Graham & Liu, 2016). This enables system-level experiments with a higher\\nnumber of systems. Human judgments of each hybrid system are created by averaging the WMT18\\nsegment-level human judgments for the corresponding sentences in the sampled data. We compare\\nBERTSCOREs to one canonical metric for each category introduced in Section 2, and include the\\ncomparison with all other participating metrics from WMT18 in Appendix F.\\nIn addition to the standard evaluation, we design model selection experiments. We use 10K hybrid\\nsystems super-sampled from WMT18. We randomly select 100 out of 10K hybrid systems, and rank\\nthem using the automatic metrics. We repeat this process 100K times. We report the percentage of\\nthe metric ranking agreeing with the human ranking on the best system (Hits@1). In Tables 23-28,\\n3https://commoncrawl.org/\\n4BLEU computed on these pairs is around zero.\\n5We use the tokenizer provided with each model. For all Hugging Face models that use the GPT-2 tokenizer,\\nat the time of our experiments, the tokenizer adds a space to the beginning of each sentence.\\n6All the models used are from https://github.com/huggingface/pytorch-transformers.\\n5\\n\\n\\nPublished as a conference paper at ICLR 2020\\nMetric\\nen↔cs\\nen↔de\\nen↔et\\nen↔fi\\nen↔ru\\nen↔tr\\nen↔zh\\n(5/5)\\n(16/16)\\n(14/14)\\n(9/12)\\n(8/9)\\n(5/8)\\n(14/14)\\nBLEU\\n.970/.995\\n.971/.981\\n.986/.975\\n.973/.962\\n.979/.983\\n.657/.826\\n.978/.947\\nITER\\n.975/.915\\n.990/.984\\n.975/.981\\n.996/.973\\n.937/.975\\n.861/.865\\n.980/ –\\nRUSE\\n.981/ –\\n.997/ –\\n.990/ –\\n.991/ –\\n.988/ –\\n.853/ –\\n.981/ –\\nYiSi-1\\n.950/.987\\n.992/.985\\n.979/.979\\n.973/.940\\n.991/.992\\n.958/.976\\n.951/.963\\nPBERT\\n.980/.994\\n.998/.988\\n.990/.981\\n.995/.957\\n.982/.990\\n.791/.935\\n.981/.954\\nRBERT\\n.998/.997\\n.997/.990\\n.986/.980\\n.997/.980\\n.995/.989\\n.054/.879\\n.990/.976\\nFBERT\\n.990/.997\\n.999/.989\\n.990/.982\\n.998/.972\\n.990/.990\\n.499/.908\\n.988/.967\\nFBERT (idf)\\n.985/.995\\n.999/.990\\n.992/.981\\n.992/.972\\n.991/.991\\n.826/.941\\n.989/.973\\nTable 1: Absolute Pearson correlations with system-level human judgments on WMT18. For each\\nlanguage pair, the left number is the to-English correlation, and the right is the from-English. We\\nbold correlations of metrics not significantly outperformed by any other metric under Williams Test\\nfor that language pair and direction. The numbers in parenthesis are the number of systems used for\\neach language pair and direction.\\nMetric\\nen↔cs\\nen↔de\\nen↔et\\nen↔fi\\nen↔ru\\nen↔tr\\nen↔zh\\nBLEU\\n.956/.993\\n.969/.977\\n.981/.971\\n.962/.958\\n.972/.977\\n.586/.796\\n.968/.941\\nITER\\n.966/.865\\n.990/.978\\n.975/.982\\n.989/.966\\n.943/.965\\n.742/.872\\n.978/ –\\nRUSE\\n.974/ –\\n.996/ –\\n.988/ –\\n.983/ –\\n.982/ –\\n.780/ –\\n.973/ –\\nYiSi-1\\n.942/.985\\n.991/.983\\n.976/.976\\n.964/.938\\n.985/.989\\n.881/.942\\n.943/.957\\nPBERT\\n.965/.989\\n.995/.983\\n.990/.970\\n.976/.951\\n.976/.988\\n.846/.936\\n.975/.950\\nRBERT\\n.989/.995\\n.997/.991\\n.982/.979\\n.989/.977\\n.988/.989\\n.540/.872\\n.981/.980\\nFBERT\\n.978/.993\\n.998/.988\\n.989/.978\\n.983/.969\\n.985/.989\\n.760/.910\\n.981/.969\\nFBERT (idf)\\n.982/.995\\n.998/.988\\n.988/.979\\n.989/.969\\n.983/.987\\n.453/.877\\n.980/.963\\nTable 2: Absolute Pearson correlations with system-level human judgments on WMT18. We use\\n10K hybrid super-sampled systems for each language pair and direction. For each language pair, the\\nleft number is the to-English correlation, and the right is the from-English. Bolding criteria is the\\nsame as in Table 1.\\nwe include two additional measures to the model selection study: (a) the mean reciprocal rank of the\\ntop metric-rated system according to the human ranking, and (b) the difference between the human\\nscore of the top human-rated system and that of the top metric-rated system.\\nAdditionally, we report the same study on the WMT17 (Bojar et al., 2017) and the WMT16 (Bojar\\net al., 2016) datasests in Appendix F.7 This adds 202 systems to our evaluation.\\nImage Captioning\\nWe use the human judgments of twelve submission entries from the COCO\\n2015 Captioning Challenge. Each participating system generates a caption for each image in the\\nCOCO validation set (Lin et al., 2014), and each image has approximately five reference cap-\\ntions.\\nFollowing Cui et al. (2018), we compute the Pearson correlation with two system-level\\nmetrics: the percentage of captions that are evaluated as better or equal to human captions (M1)\\nand the percentage of captions that are indistinguishable from human captions (M2). We compute\\nBERTSCORE with multiple references by scoring the candidate with each available reference and\\nreturning the highest score. We compare with eight task-agnostic metrics: BLEU (Papineni et al.,\\n2002), METEOR (Banerjee & Lavie, 2005), ROUGE-L (Lin, 2004), CIDER (Vedantam et al., 2015),\\nBEER (Stanojevi´\\nc & Sima’an, 2014), EED (Stanchev et al., 2019), CHRF++ (Popovi´\\nc, 2017), and\\nCHARACTER (Wang et al., 2016). We also compare with two task-specific metrics: SPICE (Ander-\\nson et al., 2016) and LEIC (Cui et al., 2018). SPICE is computed using the similarity of scene graphs\\nparsed from the reference and candidate captions. LEIC is trained to predict if a caption is written\\nby a human given the image.\\n7For WMT16, we only conduct segment-level experiments on to-English pairs due to errors in the dataset.\\n6\\n\\n\\nPublished as a conference paper at ICLR 2020\\nMetric\\nen↔cs\\nen↔de\\nen↔et\\nen↔fi\\nen↔ru\\nen↔tr\\nen↔zh\\nBLEU\\n.134/.151\\n.803/.610\\n.756/.618\\n.461/.088\\n.228/.519\\n.095/.029\\n.658/.515\\nITER\\n.154/.000\\n.814/.692\\n.742/.733\\n.475/.111\\n.234/.532\\n.102/.030\\n.673/ –\\nRUSE\\n.214/ –\\n.823/ –\\n.785/ –\\n.487/ –\\n.248/ –\\n.109/ –\\n.670/ –\\nYiSi-1\\n.159/.178\\n.809/.671\\n.749/.671\\n.467/.230\\n.248/.544\\n.108/.398\\n.613/.594\\nPBERT\\n.173/.180\\n.706/.663\\n.764/.771\\n.498/.078\\n.255/.545\\n.140/.372\\n.661/.551\\nRBERT\\n.163/.184\\n.804/.730\\n.770/.722\\n.494/.148\\n.260/.542\\n.005/.030\\n.677/.657\\nFBERT\\n.175/.184\\n.824/.703\\n.769/.763\\n.501/.082\\n.262/.544\\n.142/.031\\n.673/.629\\nFBERT (idf)\\n.179/.178\\n.824/.722\\n.760/.764\\n.503/.082\\n.265/.539\\n.004/.030\\n.678/.595\\nTable 3: Model selection accuracies (Hits@1) on WMT18 hybrid systems. We report the average of\\n100K samples and the 0.95 confidence intervals are below 10−3. We bold the highest numbers for\\neach language pair and direction.\\nMetric\\nen↔cs\\nen↔de\\nen↔et\\nen↔fi\\nen↔ru\\nen↔tr\\nen↔zh\\n(5k/5k)\\n(78k/ 20k)\\n(57k/32k)\\n(16k/10k)\\n(10k/22k)\\n(9k/1k)\\n(33k/29k)\\nBLEU\\n.233/.389\\n.415/.620\\n.285/.414\\n.154/.355\\n.228/.330\\n.145/.261\\n.178/.311\\nITER\\n.198/.333\\n.396/.610\\n.235/.392\\n.128/.311\\n.139/.291\\n-.029/.236\\n.144/ –\\nRUSE\\n.347/ –\\n.498/ –\\n.368/ –\\n.273/ –\\n.311/ –\\n.259/ –\\n.218/ –\\nYiSi-1\\n.319/.496\\n.488/.691\\n.351/.546\\n.231/.504\\n.300/.407\\n.234/.418\\n.211/.323\\nPBERT\\n.387/.541\\n.541/.715\\n.389/.549\\n.283/.486\\n.345/.414\\n.280/.328\\n.248/.337\\nRBERT\\n.388/.570\\n.546/.728\\n.391/.594\\n.304/.565\\n.343/.420\\n.290/.411\\n.255/.367\\nFBERT\\n.404/.562\\n.550/.728\\n.397/.586\\n.296/.546\\n.353/.423\\n.292/.399\\n.264/.364\\nFBERT (idf)\\n.408/.553\\n.550/.721\\n.395/585\\n.293/.537\\n.346/.425\\n.296/.406\\n.260/.366\\nTable 4: Kendall correlations with segment-level human judgments on WMT18. For each language\\npair, the left number is the to-English correlation, and the right is the from-English. We bold corre-\\nlations of metrics not significantly outperformed by any other metric under bootstrap sampling for\\nthat language pair and direction. The numbers in parenthesis are the number of candidate-reference\\nsentence pairs for each language pair and direction.\\n5\\nRESULTS\\nMachine Translation\\nTables 1–3 show system-level correlation to human judgements, correla-\\ntions on hybrid systems, and model selection performance. We observe that BERTSCORE is con-\\nsistently a top performer. In to-English results, RUSE (Shimanaka et al., 2018) shows competitive\\nperformance. However, RUSE is a supervised method trained on WMT16 and WMT15 human\\njudgment data. In cases where RUSE models were not made available, such as for our from-English\\nexperiments, it is not possible to use RUSE without additional data and training. Table 4 shows\\nsegment-level correlations. We see that BERTSCORE exhibits significantly higher performance\\ncompared to the other metrics. The large improvement over BLEU stands out, making BERTSCORE\\nparticularly suitable to analyze specific examples, where SENTBLEU is less reliable. In Appendix A,\\nwe provide qualitative examples to illustrate the segment-level performance difference between\\nSENTBLEU and BERTSCORE. At the segment-level, BERTSCORE even significantly outperforms\\nRUSE. Overall, we find that applying importance weighting using idf at times provides small bene-\\nfit, but in other cases does not help. Understanding better when such importance weighting is likely\\nto help is an important direction for future work, and likely depends on the domain of the text and\\nthe available test data. We continue without idf weighting for the rest of our experiments. While\\nrecall RBERT, precision PBERT, and F1 FBERT alternate as the best measure in different setting, F1\\nFBERT performs reliably well across all the different settings. Our overall recommendation is there-\\nfore to use F1. We present additional results using the full set of 351 systems and evaluation metrics\\nin Tables 12–28 in the appendix, including for experiments with idf importance weighting, different\\ncontextual embedding models, and model selection.\\nImage Captioning\\nTable 5 shows correlation results for the COCO Captioning Challenge.\\nBERTSCORE outperforms all task-agnostic baselines by large margins. Image captioning presents a\\nchallenging evaluation scenario, and metrics based on strict n-gram matching, including BLEU and\\nROUGE, show weak correlations with human judgments. idf importance weighting shows signifi-\\n7\\n\\n\\nPublished as a conference paper at ICLR 2020\\nMetric\\nM1\\nM2\\nBLEU\\n-0.019∗\\n-0.005∗\\nMETEOR\\n0.606∗\\n0.594∗\\nROUGE-L\\n0.090∗\\n0.096∗\\nCIDER\\n0.438∗\\n0.440∗\\nSPICE\\n0.759∗\\n0.750∗\\nLEIC\\n0.939∗\\n0.949∗\\nBEER\\n0.491\\n0.562\\nEED\\n0.545\\n0.599\\nCHRF++\\n0.702\\n0.729\\nCHARACTER\\n0.800\\n0.801\\nPBERT\\n-0.105\\n-0.041\\nRBERT\\n0.888\\n0.863\\nFBERT\\n0.322\\n0.350\\nRBERT (idf)\\n0.917\\n0.889\\nTable 5:\\nPearson correlation on the\\n2015 COCO Captioning Challenge.\\nThe M1 and M2 measures are described\\nin Section 4. LEIC uses images as addi-\\ntional inputs. Numbers with ∗are cited\\nfrom Cui et al. (2018).\\nWe bold the\\nhighest correlations of task-specific and\\ntask-agnostic metrics.\\nType\\nMethod\\nQQP\\nPAWSQQP\\nTrained on QQP\\n(supervised)\\nDecAtt\\n0.939∗\\n0.263\\nDIIN\\n0.952∗\\n0.324\\nBERT\\n0.963∗\\n0.351\\nTrained on QQP\\n+ PAWSQQP\\n(supervised)\\nDecAtt\\n-\\n0.511\\nDIIN\\n-\\n0.778\\nBERT\\n-\\n0.831\\nMetric\\n(Not trained\\non QQP or\\nPAWSQQP)\\nBLEU\\n0.707\\n0.527\\nMETEOR\\n0.755\\n0.532\\nROUGE-L\\n0.740\\n0.536\\nCHRF++\\n0.577\\n0.608\\nBEER\\n0.741\\n0.564\\nEED\\n0.743\\n0.611\\nCHARACTER\\n0.698\\n0.650\\nPBERT\\n0.757\\n0.687\\nRBERT\\n0.744\\n0.685\\nFBERT\\n0.761\\n0.685\\nFBERT (idf)\\n0.777\\n0.693\\nTable 6:\\nArea under ROC curve (AUC) on QQP\\nand PAWSQQP datasets.\\nThe scores of trained De-\\ncATT (Parikh et al., 2016), DIIN (Gong et al., 2018),\\nand fine-tuned BERT are reported by Zhang et al.\\n(2019). Numbers with ∗are scores on the held-out test\\nset of QQP. We bold the highest correlations of task-\\nspecific and task-agnostic metrics.\\ncant benefit for this task, suggesting people attribute higher importance to content words. Finally,\\nLEIC (Cui et al., 2018), a trained metric that takes images as additional inputs and is optimized\\nspecifically for the COCO data and this set of systems, outperforms all other methods.\\nSpeed\\nDespite the use of a large pre-trained model, computing BERTSCORE is relatively fast. We\\nare able to process 192.5 candidate-reference pairs/second using a GTX-1080Ti GPU. The complete\\nWMT18 en-de test set, which includes 2,998 sentences, takes 15.6sec to process, compared to 5.4sec\\nwith SacreBLEU (Post, 2018), a common BLEU implementation. Given the sizes of commonly used\\ntest and validation sets, the increase in processing time is relatively marginal, and BERTSCORE is\\na good fit for using during validation (e.g., for stopping) and testing, especially when compared to\\nthe time costs of other development stages.\\n6\\nROBUSTNESS ANALYSIS\\nWe test the robustness of BERTSCORE using adversarial paraphrase classification. We use the\\nQuora Question Pair corpus (QQP; Iyer et al., 2017) and the adversarial paraphrases from the Para-\\nphrase Adversaries from Word Scrambling dataset (PAWS; Zhang et al., 2019). Both datasets con-\\ntain pairs of sentences labeled to indicate whether they are paraphrases or not. Positive examples\\nin QQP are real duplicate questions, while negative examples are related, but different questions.\\nSentence pairs in PAWS are generated through word swapping. For example, in PAWS, Flights from\\nNew York to Florida may be changed to Flights from Florida to New York and a good classifier\\nshould identify that these two sentences are not paraphrases. PAWS includes two parts: PAWSQQP,\\nwhich is based on the QQP data, and PAWSWiki. We use the PAWSQQP development set which\\ncontains 667 sentences. For the automatic metrics, we use no paraphrase detection training data.\\nWe expect that pairs with higher scores are more likely to be paraphrases. To evaluate the automatic\\nmetrics on QQA, we use the first 5,000 sentences in the training set instead of the the test set because\\nthe test labels are not available. We treat the first sentence as the reference and the second sentence\\nas the candidate.\\nTable 6 reports the area under ROC curve (AUC) for existing models and automatic metrics. We\\nobserve that supervised classifiers trained on QQP perform worse than random guess on PAWSQQP,\\nwhich shows these models predict the adversarial examples are more likely to be paraphrases. When\\n8\\n\\n\\nPublished as a conference paper at ICLR 2020\\nadversarial examples are provided in training, state-of-the-art models like DIIN (Gong et al., 2018)\\nand fine-tuned BERT are able to identify the adversarial examples but their performance still de-\\ncreases significantly from their performance on QQP. Most metrics have decent performance on\\nQQP, but show a significant performance drop on PAWSQQP, almost down to chance performance.\\nThis suggests these metrics fail to to distinguish the harder adversarial examples. In contrast, the\\nperformance of BERTSCORE drops only slightly, showing more robustness than the other metrics.\\n7\\nDISCUSSION\\nWe propose BERTSCORE, a new metric for evaluating generated text against gold standard refer-\\nences. BERTSCORE is purposely designed to be simple, task agnostic, and easy to use. Our analysis\\nillustrates how BERTSCORE resolves some of the limitations of commonly used metrics, especially\\non challenging adversarial examples. We conduct extensive experiments with various configuration\\nchoices for BERTSCORE, including the contextual embedding model used and the use of impor-\\ntance weighting. Overall, our extensive experiments, including the ones in the appendix, show that\\nBERTSCORE achieves better correlation than common metrics, and is effective for model selec-\\ntion. However, there is no one configuration of BERTSCORE that clearly outperforms all others.\\nWhile the differences between the top configurations are often small, it is important for the user to\\nbe aware of the different trade-offs, and consider the domain and languages when selecting the exact\\nconfiguration to use. In general, for machine translation evaluation, we suggest using FBERT, which\\nwe find the most reliable. For evaluating text generation in English, we recommend using the 24-\\nlayer RoBERTalarge model to compute BERTSCORE. For non-English language, the multilingual\\nBERTmulti is a suitable choice although BERTSCORE computed with this model has less stable\\nperformance on low-resource languages. We report the optimal hyperparameter for all models we\\nexperimented with in Appendix B\\nBriefly following our initial preprint publication, Zhao et al. (2019) published a concurrently devel-\\noped method related to ours, but with a focus on integrating contextual word embeddings with earth\\nmover’s distance (EMD; Rubner et al., 1998) rather than our simple matching process. They also\\npropose various improvements compared to our use of contextualized embeddings. We study these\\nimprovements in Appendix C and show that integrating them into BERTSCORE makes it equivalent\\nor better than the EMD-based approach. Largely though, the effect of the different improvements\\non BERTSCORE is more modest compared to their method. Shortly after our initial publication,\\nYiSi-1 was updated to use BERT embeddings, showing improved performance (Lo, 2019). This\\nfurther corroborates our findings. Other recent related work includes training a model on top of\\nBERT to maximize the correlation with human judgments (Mathur et al., 2019) and evaluating gen-\\neration with a BERT model fine-tuned on paraphrasing (Yoshimura et al., 2019). More recent work\\nshows the potential of using BERTSCORE for training a summarization system (Li et al., 2019)\\nand for domain-specific evaluation using SciBERT (Beltagy et al., 2019) to evaluate abstractive text\\nsummarization (Gabriel et al., 2019).\\nIn future work, we look forward to designing new task-specific metrics that use BERTSCORE as a\\nsubroutine and accommodate task-specific needs, similar to how Wieting et al. (2019) suggests to use\\nsemantic similarity for machine translation training. Because BERTSCORE is fully differentiable,\\nit also can be incorporated into a training procedure to compute a learning loss that reduces the\\nmismatch between optimization and evaluation objectives.\\nACKNOWLEDGEMENT\\nThis research is supported in part by grants from the National Science Foundation (III-1618134, III-\\n1526012, IIS1149882, IIS-1724282, TRIPODS-1740822, CAREER-1750499), the Office of Naval\\nResearch DOD (N00014-17-1-2175), and the Bill and Melinda Gates Foundation, SAP, Zillow,\\nWorkday, and Facebook Research. We thank Graham Neubig and David Grangier for for their\\ninsightful comments. We thank the Cornell NLP community including but not limited to Claire\\nCardie, Tianze Shi, Alexandra Schofield, Gregory Yauney, and Rishi Bommasani. We thank Yin\\nCui and Guandao Yang for their help with the COCO 2015 dataset.\\n9\\n\\n\\nPublished as a conference paper at ICLR 2020\\nREFERENCES\\nPeter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE: Semantic proposi-\\ntional image caption evaluation. In ECCV, 2016.\\nBen Athiwaratkun, Andrew Wilson, and Anima Anandkumar. Probabilistic fasttext for multi-sense\\nword embeddings. In ACL, 2018.\\nSatanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for mt evaluation with im-\\nproved correlation with human judgments. In IEEvaluation@ACL, 2005.\\nIz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: A pretrained language model for scientific text.\\nArXiv, 2019.\\nOndˇ\\nrej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojevi´\\nc. Results of the WMT16 metrics\\nshared task. In WMT, 2016.\\nOndˇ\\nrej Bojar, Yvette Graham, and Amir Kamran. Results of the WMT17 metrics shared task. In\\nWMT, 2017.\\nArun Chaganty, Stephen Mussmann, and Percy Liang. The price of debiasing automatic metrics in\\nnatural language evalaution. In ACL, 2018.\\nJulian Chow, Lucia Specia, and Pranava Madhyastha. WMDO: Fluency-based word mover’s dis-\\ntance for machine translation evaluation. In WMT, 2019.\\nElizabeth Clark, Asli Celikyilmaz, and Noah A. Smith. Sentence mover’s similarity: Automatic\\nevaluation for multi-sentence texts. In ACL, 2019.\\nCourtney Corley and Rada Mihalcea. Measuring the semantic similarity of texts. In ACL Workshop,\\nEMSEE ’05, 2005.\\nYin Cui, Guandao Yang, Andreas Veit, Xun Huang, and Serge J. Belongie. Learning to evaluate\\nimage captioning. In CVPR, 2018.\\nMichael Denkowski and Alon Lavie. Meteor universal: Language specific translation evaluation for\\nany target language. In WMT@ACL, 2014.\\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep\\nbidirectional transformers for language understanding. In NAACL-HLT, 2019.\\nGeorge Doddington.\\nAutomatic evaluation of machine translation quality using n-gram co-\\noccurrence statistics. In HLT, 2002.\\nWilliam B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases.\\nIn IWP, 2005.\\nSaadia Gabriel, Antoine Bosselut, Ari Holtzman, Kyle Lo, Asli Çelikyilmaz, and Yejin Choi. Co-\\noperative generator-discriminator networks for abstractive summarization with narrative flow.\\nArXiv, 2019.\\nMichel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Mar-\\ngaret Mitchell, Jianfeng Gao, and William B. Dolan. deltaBLEU: A discriminative metric for\\ngeneration tasks with intrinsically diverse targets. In ACL, 2015.\\nJonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional\\nsequence to sequence learning. In ICML, 2017.\\nYichen Gong, Heng Luo, and Jian Zhang. Natural language inference over interaction space. In\\nICLR, 2018.\\nYvette Graham and Timothy Baldwin. Testing for significance of increased correlation with human\\njudgment. In EMNLP, 2014.\\n10\\n\\n\\nPublished as a conference paper at ICLR 2020\\nYvette Graham and Qun Liu. Achieving accurate conclusions in evaluation of automatic machine\\ntranslation metrics. In NAACL, 2016.\\nEdouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. Learning\\nword vectors for 157 languages. arXiv preprint arXiv:1802.06893, 2018.\\nYinuo Guo and Junfeng Hu. Meteor++ 2.0: Adopt syntactic level paraphrase knowledge into ma-\\nchine translation evaluation. In WMT, 2019.\\nTatsu Hashimoto, Hugh Zhang, and Percy Liang. Unifying human and statistical evaluation for\\nnatural language generation. In NAACL-HLT, 2019.\\nChenyang Huang, Amine Trabelsi, and Osmar R Zaïane. ANA at semeval-2019 task 3: Contex-\\ntual emotion detection in conversations through hierarchical LSTMs and BERT. arXiv preprint\\narXiv:1904.00132, 2019.\\nHideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. Automatic\\nevaluation of translation quality for distant language pairs. In EMNLP, 2010.\\nShankar Iyer, Nikhil Dandekar, and Kornel Csernai. First quora dataset release: Question pairs.\\nhttps://tinyurl.com/y2y8u5ed, 2017.\\nPhilipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola\\nBertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar,\\nAlexandra Constantin, and Evan Herbst.\\nMoses: Open source toolkit for statistical machine\\ntranslation. In ACL, 2007.\\nMatt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to document\\ndistances. In ICML, 2015.\\nGuillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv, 2019.\\nGregor Leusch, Nicola Ueffing, and Hermann Ney. CDER: Efficient MT evaluation using block\\nmovements. In EACL, 2006.\\nVladimir Iosifovich Levenshtein. Binary Codes Capable of Correcting Deletions, Insertions and\\nRever sals. Soviet Physics Doklady, 10, 1966.\\nSiyao Li, Deren Lei, Pengda Qin, and William Yang Wang. Deep reinforcement learning with\\ndistributional semantic rewards for abstractive summarization. In EMNLP-IJCNLP, 2019.\\nChin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In ACL, 2004.\\nTsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James\\nHays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO:\\nCommon objects in context. In ECCV, 2014.\\nNelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. Linguistic\\nknowledge and transferability of contextual representations. arXiv preprint arXiv:1903.08855,\\n2019a.\\nYang Liu. Fine-tune BERT for extractive summarization. arXiv preprint arXiv:1903.10318, 2019.\\nYinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike\\nLewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretrain-\\ning approach. arXiv, abs/1907.11692, 2019b.\\nChi-kiu Lo. MEANT 2.0: Accurate semantic mt evaluation for any output language. In WMT, 2017.\\nChi-kiu Lo. YiSi - a unified semantic MT quality evaluation and estimation metric for languages\\nwith different levels of available resources. In WMT, 2019.\\n11\\n\\n\\nPublished as a conference paper at ICLR 2020\\nChi-kiu Lo, Michel Simard, Darlene Stewart, Samuel Larkin, Cyril Goutte, and Patrick Littell. Ac-\\ncurate semantic textual similarity for cleaning noisy parallel corpora using semantic machine\\ntranslation evaluation metric: The NRC supervised submissions to the parallel corpus filtering\\ntask. In WMT, 2018.\\nRyan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio,\\nand Joelle Pineau. Towards an automatic Turing test: Learning to evaluate dialogue responses.\\nIn ACL, 2017.\\nQingsong Ma, Yvette Graham, Shugen Wang, and Qun Liu. Blend: a novel combined mt metric\\nbased on direct assessment – casict-dcu submission to WMT17 metrics task. In WMT, 2017.\\nQingsong Ma, Ondrej Bojar, and Yvette Graham. Results of the WMT18 metrics shared task: Both\\ncharacters and embeddings achieve good performance. In WMT, 2018.\\nNitika Mathur, Timothy Baldwin, and Trevor Cohn. Putting evaluation in context: Contextual em-\\nbeddings improve machine translation evaluation. In ACL, 2019.\\nTomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean.\\nDistributed\\nrepresentations of words and phrases and their compositionality. In NIPS, 2013.\\nDai Quoc Nguyen, Dat Quoc Nguyen, Ashutosh Modi, Stefan Thater, and Manfred Pinkal.\\nA\\nmixture model for learning multi-sense word embeddings. In ACL, 2017.\\nMyle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation.\\nIn WMT, 2018.\\nMyle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier,\\nand Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint\\narXiv:1904.01038, 2019.\\nJoybrata Panja and Sudip Kumar Naskar. Iter: Improving translation edit rate through optimizable\\nedit costs. In WMT, 2018.\\nKishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic\\nevaluation of machine translation. In ACL, 2002.\\nAnkur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention\\nmodel for natural language inference. In EMNLP, 2016.\\nJeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word\\nrepresentation. In EMNLP, 2014.\\nMatthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and\\nLuke S. Zettlemoyer. Deep contextualized word representations. In NAACL-HLT, 2018.\\nMaja Popovi´\\nc. chrf: character n-gram f-score for automatic mt evaluation. In WMT@ACL, 2015.\\nMaja Popovi´\\nc. chrf++: words helping character n-grams. In WMT, 2017.\\nMatt Post. A call for clarity in reporting BLEU scores. In WMT, 2018.\\nNils Reimers and Iryna Gurevych. Alternative weighting schemes for elmo embeddings. arXiv\\npreprint arXiv:1904.02954, 2019.\\nYossi Rubner, Carlo Tomasi, and Leonidas J Guibas. A metric for distributions with applications to\\nimage databases. In ICCV. IEEE, 1998.\\nVasile Rus and Mihai Lintean. A comparison of greedy and optimal assessment of natural language\\nstudent input using word-to-word similarity metrics. In Proceedings of the Seventh Workshop on\\nBuilding Educational Applications Using NLP. ACL, 2012.\\n12\\n\\n\\nPublished as a conference paper at ICLR 2020\\nAndreas Rücklé, Steffen Eger, Maxime Peyrard, and Iryna Gurevych. Concatenated power mean\\nword embeddings as universal cross-lingual sentence representations. arXiv, 2018.\\nShikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. Relevance of unsupervised\\nmetrics in task-oriented dialogue for evaluating natural language generation.\\narXiv preprint\\narXiv:1706.09799, 2018.\\nHiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. Ruse: Regressor using sentence\\nembeddings for automatic machine translation evaluation. In WMT, 2018.\\nMatthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. A study of\\ntranslation edit rate with targeted human annotation. In AMTA, 2006.\\nPeter Stanchev, Weiyue Wang, and Hermann Ney. EED: Extended edit distance measure for machine\\ntranslation. In WMT, 2019.\\nMiloš Stanojevi´\\nc and Khalil Sima’an. Beer: Better evaluation as ranking. In WMT, 2014.\\nChristoph Tillmann, Stephan Vogel, Hermann Ney, Arkaitz Zubiaga, and Hassan Sawaf. Accelerated\\ndp based search for statistical translation. In EUROSPEECH, 1997.\\nKristina Toutanova, Chris Brockett, Ke M Tran, and Saleema Amershi. A dataset and evaluation\\nmetrics for abstractive compression of sentences and short paragraphs. In EMNLP, 2016.\\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,\\nŁukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.\\nRamakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. CIDEr: Consensus-based image\\ndescription evaluation. In CVPR, 2015.\\nWeiyue Wang, Jan-Thorsten Peter, Hendrik Rosendahl, and Hermann Ney. Character: Translation\\nedit rate on character level. In WMT, 2016.\\nJohn Wieting, Taylor Berg-Kirkpatrick, Kevin Gimpel, and Graham Neubig. Beyond BLEU:training\\nneural machine translation with semantic similarity. In ACL, 2019.\\nAdina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen-\\ntence understanding through inference. In ACL, 2018.\\nEvan James Williams. Regression analysis. wiley, 1959.\\nFelix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. Pay less attention with\\nlightweight and dynamic convolutions. In ICLR, 2019.\\nYonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey,\\nMaxim Krikun, Yuan Cao, Qin Gao, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu,\\nLukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens,\\nGeorge Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick,\\nOriol Vinyals, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. Google’s neural machine\\ntranslation system: Bridging the gap between human and machine translation. arXiv preprint\\narXiv:1609.08144, 2016.\\nWei Yang, Haotian Zhang, and Jimmy Lin. Simple applications of BERT for ad hoc document\\nretrieval. arXiv preprint arXiv:1903.10972, 2019a.\\nZhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le.\\nXLNet: Generalized autoregressive pretraining for language understanding. arXiv, 2019b.\\nRyoma Yoshimura, Hiroki Shimanaka, Yukio Matsumura, Hayahide Yamagishi, and Mamoru Ko-\\nmachi. Filtering pseudo-references by paraphrasing for automatic evaluation of machine transla-\\ntion. In WMT, 2019.\\n13\\n\\n\\nPublished as a conference paper at ICLR 2020\\nYuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase adversaries from word scram-\\nbling. arXiv preprint arXiv:1904.01130, 2019.\\nWei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. Moverscore:\\nText generation evaluating with contextualized embeddings and earth mover distance. In EMNLP,\\n2019.\\n14\\n\\n\\nPublished as a conference paper at ICLR 2020\\nCase\\nNo.\\nReference and Candidate Pairs\\nHuman\\nFBERT\\nBLEU\\nFBERT > BLEU\\n1.\\nx: At the same time Kingfisher is closing 60 B&Q outlets across the country\\n38\\n125\\n530\\nˆ\\nx: At the same time, Kingfisher will close 60 B & Q stores nationwide\\n2.\\nx: Hewlett-Packard to cut up to 30,000 jobs\\n119\\n39\\n441\\nˆ\\nx: Hewlett-Packard will reduce jobs up to 30.000\\n3.\\nx: According to opinion in Hungary, Serbia is “a safe third country\\\".\\n23\\n96\\n465\\nˆ\\nx: According to Hungarian view, Serbia is a “safe third country.\\\"\\n4.\\nx: Experts believe November’s Black Friday could be holding back spending.\\n73\\n147\\n492\\nˆ\\nx: Experts believe that the Black Friday in November has put the brakes on spending\\n5.\\nx: And it’s from this perspective that I will watch him die.\\n37\\n111\\n414\\nˆ\\nx: And from this perspective, I will see him die.\\nBLEU > FBERT\\n6.\\nx: In their view the human dignity of the man had been violated.\\n500\\n470\\n115\\nˆ\\nx: Look at the human dignity of the man injured.\\n8.\\nx: For example when he steered a shot from Ideye over the crossbar in the 56th minute.\\n516\\n524\\n185\\nˆ\\nx: So, for example, when he steered a shot of Ideye over the latte (56th).\\n7.\\nx: A good prank is funny, but takes moments to reverse.\\n495\\n424\\n152\\nˆ\\nx: A good prank is funny, but it takes only moments before he becomes a boomerang.\\n9.\\nx: I will put the pressure on them and onus on them to make a decision.\\n507\\n471\\n220\\nˆ\\nx: I will exert the pressure on it and her urge to make a decision.\\n10.\\nx: Transport for London is not amused by this flyposting \\\"vandalism.\\\"\\n527\\n527\\n246\\nˆ\\nx: Transport for London is the Plaka animal \\\"vandalism\\\" is not funny.\\nFBERT > Human\\n11.\\nx: One big obstacle to access to the jobs market is the lack of knowledge of the German language.\\n558\\n131\\n313\\nˆ\\nx: A major hurdle for access to the labour market are a lack of knowledge of English.\\n12.\\nx: On Monday night Hungary closed its 175 km long border with Serbia.\\n413\\n135\\n55\\nˆ\\nx: Hungary had in the night of Tuesday closed its 175 km long border with Serbia.\\n13.\\nx: They got nothing, but they were allowed to keep the clothes.\\n428\\n174\\n318\\nˆ\\nx: You got nothing, but could keep the clothes.\\n14.\\nx: A majority of Republicans don’t see Trump’s temperament as a problem.\\n290\\n34\\n134\\nˆ\\nx: A majority of Republicans see Trump’s temperament is not a problem.\\n15.\\nx:His car was still running in the driveway.\\n299\\n49\\n71\\nˆ\\nx: His car was still in the driveway.\\nHuman > FBERT\\n16.\\nx: Currently the majority of staff are men.\\n77\\n525\\n553\\nˆ\\nx: At the moment the men predominate among the staff.\\n17.\\nx: There are, indeed, multiple variables at play.\\n30\\n446\\n552\\nˆ\\nx: In fact, several variables play a role.\\n18.\\nx: One was a man of about 5ft 11in tall.\\n124\\n551\\n528\\nˆ\\nx: One of the men was about 1,80 metres in size.\\n19.\\nx: All that stuff sure does take a toll.\\n90\\n454\\n547\\nˆ\\nx: All of this certainly exacts its toll.\\n20.\\nx: Wage gains have shown signs of picking up.\\n140\\n464\\n514\\nˆ\\nx: Increases of wages showed signs of a recovery.\\nTable 7: Examples sentences where similarity ranks assigned by Human, FBERT, and BLEU differ\\nsignificantly on WMT16 German-to-English evaluation task. x: gold reference, ˆ\\nx: candidate outputs\\nof MT systems. Rankings assigned by Human, FBERT, and BLEU are shown in the right three\\ncolumns. The sentences are ranked by the similarity, i.e. rank 1 is the most similar pair assigned by\\na score. An ideal metric should rank similar to humans.\\nA\\nQUALITATIVE ANALYSIS\\nWe study BERTSCORE and SENTBLEU using WMT16 German-to-English (Bojar et al., 2016). We\\nrank all 560 candidate-reference pairs by human score, BERTSCORE, or SENTBLEU from most\\nsimilar to least similar. Ideally, the ranking assigned by BERTSCORE and SENTBLEU should be\\nsimilar to the ranking assigned by the human score.\\nTable 7 first shows examples where BERTSCORE and SENTBLEU scores disagree about the ranking\\nfor a candidate-reference pair by a large number. We observe that BERTSCORE is effectively able\\nto capture synonyms and changes in word order. For example, the reference and candidate sentences\\nin pair 3 are almost identical except that the candidate replaces opinion in Hungary with Hungarian\\nview and switches the order of the quotation mark (“) and a. While BERTSCORE ranks the pair\\nrelatively high, SENTBLEU judges the pair as dissimilar, because it cannot match synonyms and is\\nsensitive to the small word order changes. Pair 5 shows a set of changes that preserve the semantic\\nmeaning: replacing to cut with will reduce and swapping the order of 30,000 and jobs. BERTSCORE\\nranks the candidate translation similar to the human judgment, whereas SENTBLEU ranks it much\\nlower. We also see that SENTBLEU potentially over-rewards n-gram overlap, even when phrases\\nare used very differently. In pair 6, both the candidate and the reference contain the human dignity\\nof the man. Yet the two sentences convey very different meaning. BERTSCORE agrees with the\\nhuman judgment and ranks the pair low. In contrast, SENTBLEU considers the pair as relatively\\nsimilar because of the significant word overlap.\\n15\\n\\n\\nPublished as a conference paper at ICLR 2020\\nFigure 2: BERTSCORE visualization. The cosine similarity of each word matching in PBERT are\\ncolor-coded.\\nThe bottom half of Table 7 shows examples where BERTSCORE and human judgments disagree\\nabout the ranking. We observe that BERTSCORE finds it difficult to detect factual errors. For\\nexample, BERTSCORE assigns high similarity to pair 11 when the translation replaces German\\nlanguage with English and pair 12 where the translation incorrectly outputs Tuesday when it is\\nsupposed to generate Monday. BERTSCORE also fails to identify that 5ft 11in is equivalent with\\n1.80 metres in pair 18. As a result, BERTSCORE assigns low similarity to the eighth pair in Table 7.\\nSENTBLEU also suffers from these limitations.\\nFigure 2 visualizes the BERTSCORE matching of two pairs of candidate and reference sentences.\\nThe figure illustrates how FBERT matches synonymous phrases, such as imported cars and foreign\\ncars. We also see that FBERT effectively matches words even given a high ordering distortion, for\\nexample the token people in the figure.\\n16\\n\\n\\nPublished as a conference paper at ICLR 2020\\nB\\nREPRESENTATION CHOICE\\nAs suggested by previous works (Peters et al., 2018; Reimers & Gurevych, 2019), selecting a\\ngood layer or a good combination of layers from the BERT model is important. In designing\\nBERTSCORE, we use WMT16 segment-level human judgment data as a development set to fa-\\ncilitate our representation choice. For Chinese models, we tune with the WMT17 “en-zh” data\\nbecause the language pair “en-zh” is not available in WMT16. In Figure 3, we plot the change of\\nhuman correlation of FBERT over different layers of BERT, RoBERTa, XLNet and XLM models.\\nBased on results from different models, we identify a common trend that FBERT computed with the\\nintermediate representations tends to work better. We tune the number of layer to use for a range of\\npublicly available models.8 Table 8 shows the results of our hyperparameter search.\\nModel\\nTotal Number of Layers\\nBest Layer\\nbert-base-uncased\\n12\\n9\\nbert-large-uncased\\n24\\n18\\nbert-base-cased-finetuned-mrpc\\n12\\n9\\nbert-base-multilingual-cased\\n12\\n9\\nbert-base-chinese\\n12\\n8\\nroberta-base\\n12\\n10\\nroberta-large\\n24\\n17\\nroberta-large-mnli\\n24\\n19\\nxlnet-base-cased\\n12\\n5\\nxlnet-large-cased\\n24\\n7\\nxlm-mlm-en-2048\\n12\\n7\\nxlm-mlm-100-1280\\n16\\n11\\nTable 8: Recommended layer of representation to use for BERTSCORE. The layers are chosen\\nbased on a held-out validation set (WMT16).\\n8https://huggingface.co/pytorch-transformers/pretrained_models.html\\n17\\n\\n\\nPublished as a conference paper at ICLR 2020\\nFigure 3: Pearson correlation of FBERT computed with different models, across different layers, with\\nsegment-level human judgments on the WMT16 to-English machine translation task. The WMT17\\nEnglish-Chinese data is used for the BERT Chinese model. Layer 0 corresponds to using BPE\\nembeddings. Consistently, correlation drops significantly in the final layers.\\n18\\n\\n\\nPublished as a conference paper at ICLR 2020\\nC\\nABLATION STUDY OF MOVERSCORE\\nWord Mover’s Distance (WMD; Kusner et al., 2015) is a semantic similarity metric that relies on\\nword embeddings and optimal transport. MOVERSCORE (Zhao et al., 2019) combines contextual\\nembeddings and WMD for text generation evaluation. In contrast, BERTSCORE adopts a greedy\\napproach to aggregate token-level information. In addition to using WMD for generation evalu-\\nation, Zhao et al. (2019) also introduce various other improvements. We do a detailed ablation\\nstudy to understand the benefit of each improvement, and to investigate whether it can be applied to\\nBERTSCORE. We use a 12-layer uncased BERT model on the WMT17 to-English segment-level\\ndata, the same setting as Zhao et al. (2019).\\nWe identify several differences between MOVERSCORE and BERTSCORE by analyzing the released\\nsource code. We isolate each difference, and mark it with a bracketed tag for our ablation study:\\n1. [MNLI] Use a BERT model fine-tuned on MNLI (Williams et al., 2018).\\n2. [PMEANS] Apply power means (Rücklé et al., 2018) to aggregate the information of dif-\\nferent layers.9\\n3. [IDF-L] For reference sentences, instead of computing the idf scores on the 560 sen-\\ntences in the segment-level data ([IDF-S]), compute the idf scores on the 3,005 sentences\\nin the system-level data.\\n4. [SEP] For candidate sentences, recompute the idf scores on the candidate sentences. The\\nweighting of reference tokens are kept the same as in [IDF-S]\\n5. [RM] Exclude punctuation marks and sub-word tokens except the first sub-word in each\\nword from the matching.\\nWe follow the setup of Zhao et al. (2019) and use their released fine-tuned BERT model to\\nconduct the experiments.\\nTable 9 shows the results of our ablation study.\\nWe report corre-\\nlations for the two variants of WMD Zhao et al. (2019) study:\\nunigrams (WMD1) and bi-\\ngrams (WMD2). Our FBERT corresponds to the vanilla setting and the importance weighted vari-\\nant corresponds to the [IDF-S] setting.\\nThe complete MOVERSCORE metric corresponds to\\n[IDF-S]+[SEP]+[PMEANS]+[MNLI]+[RM]. We make several observations. First, for all lan-\\nguage pairs except fi-en and lv-en, we can replicate the reported performance. For these two lan-\\nguage pairs, Zhao et al. (2019) did not release their implementations at the time of publication.10\\nSecond, we confirm the effectiveness of [PMEANS] and [MNLI]. In Appendix F, we study more\\npre-trained models and further corroborate this conclusion. However, the contribution of other tech-\\nniques, including [RM] and [SEP], seems less stable. Third, replacing greedy matching with\\nWMD does not lead to consistent improvement. In fact, oftentimes BERTSCORE is the better met-\\nric when given the same setup. In general, for any given language pair, BERTSCORE is always\\namong the best performing ones. Given the current results, it is not clear tht WMD is better than\\ngreedy matching for text generation evaluation.\\n9 Zhao et al. (2019) uses the embeddings from the last five layers from BERT and L2-normalizes the embed-\\nding vectors at each layer before computing the P-MEANs and L2-normalizing the concatenated P-MEANS.\\n10A public comment on the project page indicates that some of the techniques are not applied for these two\\nlanguage pairs (https://github.com/AIPHES/emnlp19-moverscore/issues/1).\\n19\\n\\n\\nPublished as a conference paper at ICLR 2020\\nAblation\\nMetric\\ncs-en\\nde-en\\nfi-en\\nlv-en\\nru-en\\ntr-en\\nzh-en\\nVanilla\\nWMD1\\n0.628\\n0.655\\n0.795\\n0.692\\n0.701\\n0.715\\n0.699\\nWMD2\\n0.638\\n0.661\\n0.797\\n0.695\\n0.700\\n0.728\\n0.714\\nFBERT\\n0.659\\n0.680\\n0.817\\n0.702\\n0.719\\n0.727\\n0.717\\nIDF-S\\nWMD1\\n0.636\\n0.662\\n0.824\\n0.709\\n0.716\\n0.728\\n0.713\\nWMD2\\n0.643\\n0.662\\n0.821\\n0.708\\n0.712\\n0.732\\n0.715\\nFBERT\\n0.657\\n0.681\\n0.823\\n0.713\\n0.725\\n0.718\\n0.711\\nIDF-L\\nWMD1\\n0.633\\n0.659\\n0.825\\n0.708\\n0.716\\n0.727\\n0.715\\nWMD2\\n0.641\\n0.661\\n0.822\\n0.708\\n0.713\\n0.730\\n0.716\\nFBERT\\n0.655\\n0.682\\n0.823\\n0.713\\n0.726\\n0.718\\n0.712\\nIDF-L + SEP\\nWMD1\\n0.651\\n0.660\\n0.819\\n0.703\\n0.714\\n0.724\\n0.715\\nWMD2\\n0.659\\n0.662\\n0.816\\n0.702\\n0.712\\n0.729\\n0.715\\nFBERT\\n0.664\\n0.681\\n0.818\\n0.709\\n0.724\\n0.716\\n0.710\\nIDF-L + SEP\\n+ RM\\nWMD1\\n0.651\\n0.686\\n0.803\\n0.681\\n0.730\\n0.730\\n0.720\\nWMD2\\n0.664\\n0.687\\n0.797\\n0.679\\n0.728\\n0.735\\n0.718\\nFBERT\\n0.659\\n0.695\\n0.800\\n0.683\\n0.734\\n0.722\\n0.712\\nIDF-L + SEP\\n+ PMEANS\\nWMD1\\n0.658\\n0.663\\n0.820\\n0.707\\n0.717\\n0.725\\n0.712\\nWMD2\\n0.667\\n0.665\\n0.817\\n0.707\\n0.717\\n0.727\\n0.712\\nFBERT\\n0.671\\n0.682\\n0.819\\n0.708\\n0.725\\n0.715\\n0.704\\nIDF-L + SEP\\n+ MNLI\\nWMD1\\n0.659\\n0.679\\n0.822\\n0.732\\n0.718\\n0.746\\n0.725\\nWMD2\\n0.664\\n0.682\\n0.819\\n0.731\\n0.715\\n0.748\\n0.722\\nFBERT\\n0.668\\n0.701\\n0.825\\n0.737\\n0.727\\n0.744\\n0.725\\nIDF-L + SEP\\n+ PMEANS + MNLI\\nWMD1\\n0.672\\n0.686\\n0.831\\n0.738\\n0.725\\n0.753\\n0.737\\nWMD2\\n0.677\\n0.690\\n0.828\\n0.736\\n0.722\\n0.755\\n0.735\\nFBERT\\n0.682\\n0.707\\n0.836\\n0.741\\n0.732\\n0.751\\n0.736\\nIDF-L + SEP\\n+ PMEANS + MNLI\\n+ RM\\nWMD1\\n0.670\\n0.708\\n0.821\\n0.717\\n0.738\\n0.762\\n0.744\\nWMD2\\n0.679\\n0.709\\n0.814\\n0.716\\n0.736\\n0.762\\n0.738\\nFBERT\\n0.676\\n0.717\\n0.824\\n0.719\\n0.740\\n0.757\\n0.738\\nTable 9: Ablation Study of MOVERSCORE and BERTSCORE using Pearson correlations on the\\nWMT17 to-English segment-level data. Correlations that are not outperformed by others for that\\nlanguage pair under Williams Test are bolded. We observe that using WMD does not consistently\\nimprove BERTSCORE.\\n20\\n\\n\\nPublished as a conference paper at ICLR 2020\\nType\\nMetric\\nMeaning\\nGrammar\\nCombined\\nBERTSCORE\\nPBERT\\n0.36\\n0.47\\n0.46\\nRBERT\\n0.64\\n0.29\\n0.52\\nFBERT\\n0.58\\n0.41\\n0.56\\nCommon metrics\\nBLEU\\n0.46\\n0.13\\n0.33\\nMETEOR\\n0.53\\n0.11\\n0.36\\nROUGE-L\\n0.51\\n0.16\\n0.38\\nSARI\\n0.50\\n0.15\\n0.37\\nBest metrics according to\\nToutanova et al. (2016)\\nSKIP-2+RECALL+MULT-PROB\\n0.59\\nN/A\\n0.51\\nPARSE-2+RECALL+MULT-MAX\\nN/A\\n0.35\\n0.52\\nPARSE-2+RECALL+MULT-PROB\\n0.57\\n0.35\\n0.52\\nTable 10: Pearson correlations with human judgments on the MSR Abstractive Text Compression\\nDataset.\\nD\\nADDITIONAL EXPERIMENTS ON ABSTRACTIVE TEXT COMPRESSION\\nWe use the human judgments provided from the MSR Abstractive Text Compression\\nDataset (Toutanova et al., 2016) to illustrate the applicability of BERTSCORE to abstractive text\\ncompression evaluation. The data includes three types of human scores: (a) meaning: how well a\\ncompressed text preserve the meaning of the original text; (b) grammar: how grammatically correct\\na compressed text is; and (c) combined: the average of the meaning and the grammar scores. We\\nfollow the experimental setup of Toutanova et al. (2016) and report Pearson correlation between\\nBERTSCORE and the three types of human scores. Table 10 shows that RBERT has the highest cor-\\nrelation with human meaning judgments, and PBERT correlates highly with human grammar judg-\\nments. FBERT provides a balance between the two aspects.\\n21\\n\\n\\nPublished as a conference paper at ICLR 2020\\nTask\\nModel\\nBLEU\\nˆ\\nPBERT\\nˆ\\nRBERT\\nˆ\\nFBERT\\nPBERT\\nRBERT\\nFBERT\\nWMT14\\nEn-De\\nConvS2S (Gehring et al., 2017)\\n0.266\\n0.6099 0.6055 0.6075 0.8499 0.8482 0.8488\\nTransformer-big∗∗(Ott et al., 2018)\\n0.298\\n0.6587 0.6528 0.6558 0.8687 0.8664 0.8674\\nDynamicConv∗∗∗(Wu et al., 2019)\\n0.297\\n0.6526 0.6464 0.6495 0.8664 0.8640 0.8650\\nWMT14\\nEn-Fr\\nConvS2S (Gehring et al., 2017)\\n0.408\\n0.6998 0.6821 0.6908 0.8876 0.8810 0.8841\\nTransformer-big (Ott et al., 2018)\\n0.432\\n0.7148 0.6978 0.7061 0.8932 0.8869 0.8899\\nDynamicConv (Wu et al., 2019)\\n0.432\\n0.7156 0.6989 0.7071 0.8936 0.8873 0.8902\\nIWSLT14\\nDe-En\\nTransformer-iwslt+ (Ott et al., 2019) 0.350\\n0.6749 0.6590 0.6672 0.9452 0.9425 0.9438\\nLightConv (Wu et al., 2019)\\n0.348\\n0.6737 0.6542 0.6642 0.9450 0.9417 0.9433\\nDynamicConv (Wu et al., 2019)\\n0.352\\n0.6770 0.6586 0.6681 0.9456 0.9425 0.9440\\nTable 11:\\nBLEU scores and BERTSCOREs of publicly available pre-trained MT models in\\nfairseq (Ott et al., 2019). We show both rescaled scores marked with ˆ and raw BERTSCOREs.\\n∗: trained on unconfirmed WMT data version, ∗∗: trained on WMT16 + ParaCrawl, ∗∗∗: trained on\\nWMT16, +: trained by us using fairseq.\\nE\\nBERTSCORE OF RECENT MT MODELS\\nTable 11 shows the BLEU scores and the BERTSCOREs of pre-trained machine translation models\\non WMT14 English-to-German, WMT14 English-to-French, IWSLT14 German-to-English task.\\nWe used publicly available pre-trained models from fairseq (Ott et al., 2019).11 Because a pre-\\ntrained Transformer model on IWSLT is not released, we trained our own using the fairseq library.\\nWe use multilingual cased BERTbase12 for English-to-German and English-to-French pairs, and\\nEnglish uncased BERTbase13 for German-to-English pairs. Interestingly, the gap between a Dy-\\nnamicConv (Wu et al., 2019) trained on only WMT16 and a Transformer (Ott et al., 2018) trained\\non WMT16 and ParaCrawl14 (about 30× more training data) becomes larger when evaluated with\\nBERTSCORE rather than BLEU.\\n11 Code and pre-trained model available at https://github.com/pytorch/fairseq.\\n12Hash code: bert-base-multilingual-cased_L9_version=0.2.0\\n13Hash code: roberta-large_L17_version=0.2.0\\n14http://paracrawl.eu/download.html\\n22\\n\\n\\nPublished as a conference paper at ICLR 2020\\nF\\nADDITIONAL RESULTS\\nIn this section, we present additional experimental results:\\n1. Segment-level and system-level correlation studies on three years of WMT metric evalua-\\ntion task (WMT16–18)\\n2. Model selection study on WMT18 10K hybrid systems\\n3. System-level correlation study on 2015 COCO captioning challenge\\n4. Robustness study on PAWS-QQP.\\nFollowing BERT (Devlin et al., 2019), a variety of Transformer-based (Vaswani et al., 2017) pre-\\ntrained contextual embeddings have been proposed and released. We conduct additional experiments\\nwith four types of pre-trained embeddings: BERT, XLM (Lample & Conneau, 2019), XLNet (Yang\\net al., 2019b), and RoBERTa (Liu et al., 2019b). XLM (Cross-lingual Language Model) is a Trans-\\nformer pre-trained on the translation language modeling of predicting masked tokens from a pair of\\nsentence in two different languages and masked language modeling tasks using multi-lingual train-\\ning data. Yang et al. (2019b) modify the Transformer architecture and pre-train it on a permutation\\nlanguage modeling task resulting in some improvement on top of the original BERT when fine-tuned\\non several downstream tasks. Liu et al. (2019b) introduce RoBERTa (Robustly optimized BERT ap-\\nproach) and demonstrate that an optimized BERT model is comparable to or sometimes outperforms\\nan XLNet on downstream tasks.\\nWe perform a comprehensive study with the following pre-trained contextual embedding models:15\\n• BERT\\nmodels:\\nbert-base-uncased,\\nbert-large-uncased,\\nbert-based-chinese,\\nbert-base-multilingual-cased,\\nand\\nbert-base-cased-mrpc\\n• RoBERTa models: roberta-base, roberta-large, and roberta-large-mnli\\n• XLNet models: xlnet-base-cased and xlnet-base-large\\n• XLM models: xlm-mlm-en-2048 and xlm-mlm-100-1280\\nF.1\\nWMT CORRELATION STUDY\\nExperimental setup\\nBecause of missing data in the released WMT16 dataset (Bojar et al., 2016),\\nwe are only able to experiment with to-English segment-level data, which contains the outputs of\\n50 different systems on 6 language pairs. We use this data as the validation set for hyperparam-\\neter tuning (Appendix B). Table 12 shows the Pearson correlations of all participating metrics and\\nBERTSCOREs computed with different pre-trained models. Significance testing for this dataset does\\nnot include the baseline metrics because the released dataset does not contain the original outputs\\nfrom the baseline metrics. We conduct significance testing between BERTSCORE results only.\\nThe WMT17 dataset (Bojar et al., 2017) contains outputs of 152 different translations on 14 lan-\\nguage pairs. We experiment on the segment-level and system-level data on both to-English and\\nfrom-English language pairs. We exclude fi-en data from the segment-level experiment due to an\\nerror in the released data. We compare our results to all participating metrics and perform standard\\nsignificance testing as done by Bojar et al. (2017). Tables 13–16 show the results.\\nThe WMT18 dataset (Ma et al., 2018) contains outputs of 159 translation systems on 14 lan-\\nguage pairs. In addition to the results in Tables 1–4, we complement the study with the correla-\\ntions of all participating metrics in WMT18 and results from using different contextual models for\\nBERTSCORE.\\nResults\\nTable 12–22 collectively showcase the effectiveness of BERTSCORE in correlating with\\nhuman judgments. The improvement of BERTSCORE is more pronounced on the segment-level\\nthan on the system-level. We also see that more optimized or larger BERT models can produce better\\ncontextual representations (e.g., comparing FRoBERTa–Large and FBERT–Large). In contrast, the smaller\\nXLNet performs better than a large one. Based on the evidence in Figure 8 and Tables 12–22, we\\n15Denoted by names specified at https://huggingface.co/pytorch-transformers/pretrained_models.html.\\n23\\n\\n\\nPublished as a conference paper at ICLR 2020\\nhypothesize that the permutation language task, though leading to a good set of model weights for\\nfine-tuning on downstream tasks, does not necessarily produce informative pre-trained embeddings\\nfor generation evaluation. We also observe that fine-tuning pre-trained models on a related task,\\nsuch as natural language inference (Williams et al., 2018), can lead to better human correlation in\\nevaluating text generation. Therefore, for evaluating English sentences, we recommend computing\\nBERTSCORE with a 24-layer RoBERTa model fine-tuned on the MNLI dataset. For evaluating\\nNon-English sentences, both the multilingual BERT model and the XLM model trained on 100\\nlanguages are suitable candidates. We also recommend using domain- or language-specific contex-\\ntual embeddings when possible, such as using BERT Chinese models for evaluating Chinese tasks.\\nIn general, we advise users to consider the target domain and languages when selecting the exact\\nconfiguration to use.\\nF.2\\nMODEL SELECTION STUDY\\nExperimental setup\\nSimilar to Section 4, we use the 10K hybrid systems super-sampled from\\nWMT18. We randomly select 100 out of 10K hybrid systems, rank them using automatic metrics,\\nand repeat this process 100K times. We add to the results in the main paper (Table 3) performance of\\nall participating metrics in WMT18 and results from using different contextual embedding models\\nfor BERTSCORE. We reuse the hybrid configuration and metric outputs released in WMT18. In\\naddition to the Hits@1 measure, we evaluate the metrics using (a) mean reciprocal rank (MRR) of\\nthe top metric-rated system in human rankings, and (b) the absolute human score difference (Diff)\\nbetween the top metric- and human-rated systems. Hits@1 captures a metric’s ability to select the\\nbest system. The other two measures quantify the amount of error a metric makes in the selection\\nprocess. Tables 23–28 show the results from these experiments.\\nResults\\nThe additional results further support our conclusion from Table 3: BERTSCORE demon-\\nstrates better model selection performance. We also observe that the supervised metric RUSE dis-\\nplays strong model selection ability.\\nF.3\\nIMAGE CAPTIONING ON COCO\\nWe follow the experimental setup described in Section 4. Table 29 shows the correlations of several\\npre-trained contextual embeddings. We observe that precision-based methods such as BLEU and\\nPBERT are weakly correlated with human judgments on image captioning tasks. We hypothesize\\nthat this is because human judges prefer captions that capture the main objects in a picture for\\nimage captioning. In general, RBERT has a high correlation, even surpassing the task-specific metric\\nSPICE Anderson et al. (2016). While the fine-tuned RoBERTa-Large model does not result in the\\nhighest correlation, it is one of the best metrics.\\nF.4\\nROBUSTNESS ANALYSIS ON PAWS-QQP\\nWe present the full results of the robustness study described in Section 6 in Table 30. In general,\\nwe observe that BERTSCORE is more robust than other commonly used metrics. BERTSCORE\\ncomputed with the 24-layer RoBERTa model performs the best. Fine-tuning RoBERTa-Large on\\nMNLI (Williams et al., 2018) can significantly improve the robustness against adversarial sentences.\\nHowever, a fine-tuned BERT on MRPC (Microsoft Research Paraphrasing Corpus) (Dolan & Brock-\\nett, 2005) performs worse than its counterpart.\\n24\\n\\n\\nPublished as a conference paper at ICLR 2020\\nSetting\\nMetric\\ncs-en\\nde-en\\nfi-en\\nro-en\\nru-en\\ntr-en\\n560\\n560\\n560\\n560\\n560\\n560\\nUnsupervised\\nDPMFCOMB\\n0.713\\n0.584\\n0.598\\n0.627\\n0.615\\n0.663\\nMETRICS-F\\n0.696\\n0.601\\n0.557\\n0.662\\n0.618\\n0.649\\nCOBALT-F.\\n0.671\\n0.591\\n0.554\\n0.639\\n0.618\\n0.627\\nUPF-COBA.\\n0.652\\n0.550\\n0.490\\n0.616\\n0.556\\n0.626\\nMPEDA\\n0.644\\n0.538\\n0.513\\n0.587\\n0.545\\n0.616\\nCHRF2\\n0.658\\n0.457\\n0.469\\n0.581\\n0.534\\n0.556\\nCHRF3\\n0.660\\n0.455\\n0.472\\n0.582\\n0.535\\n0.555\\nCHRF1\\n0.644\\n0.454\\n0.452\\n0.570\\n0.522\\n0.551\\nUOW-REVAL\\n0.577\\n0.528\\n0.471\\n0.547\\n0.528\\n0.531\\nWORDF3\\n0.599\\n0.447\\n0.473\\n0.525\\n0.504\\n0.536\\nWORDF2\\n0.596\\n0.445\\n0.471\\n0.522\\n0.503\\n0.537\\nWORDF1\\n0.585\\n0.435\\n0.464\\n0.508\\n0.497\\n0.535\\nSENTBLEU\\n0.557\\n0.448\\n0.484\\n0.499\\n0.502\\n0.532\\nDTED\\n0.394\\n0.254\\n0.361\\n0.329\\n0.375\\n0.267\\nSupervised\\nBEER\\n0.661\\n0.462\\n0.471\\n0.551\\n0.533\\n0.545\\nPre-Trained\\nPBERT–Base\\n0.729\\n0.617\\n0.719\\n0.651\\n0.684\\n0.678\\nRBERT–Base\\n0.741\\n0.639\\n0.616\\n0.693\\n0.660\\n0.660\\nFBERT–Base\\n0.747\\n0.640\\n0.661\\n0.723\\n0.672\\n0.688\\nPBERT–Base (no idf)\\n0.723\\n0.638\\n0.662\\n0.700\\n0.633\\n0.696\\nRBERT–Base (no idf)\\n0.745\\n0.656\\n0.638\\n0.697\\n0.653\\n0.674\\nFBERT–Base (no idf)\\n0.747\\n0.663\\n0.666\\n0.714\\n0.662\\n0.703\\nPBERT–Base–MRPC\\n0.697\\n0.618\\n0.614\\n0.676\\n0.62\\n0.695\\nRBERT–Base–MRPC\\n0.723\\n0.636\\n0.587\\n0.667\\n0.648\\n0.664\\nFBERT–Base–MRPC\\n0.725\\n0.644\\n0.617\\n0.691\\n0.654\\n0.702\\nPBERT–Base–MRPC (idf)\\n0.713\\n0.613\\n0.630\\n0.693\\n0.635\\n0.691\\nRBERT–Base–MRPC (idf)\\n0.727\\n0.631\\n0.573\\n0.666\\n0.642\\n0.662\\nFBERT–Base–MRPC (idf)\\n0.735\\n0.637\\n0.620\\n0.700\\n0.658\\n0.697\\nPBERT–Large\\n0.756\\n0.671\\n0.701\\n0.723\\n0.678\\n0.706\\nRBERT–Large\\n0.768\\n0.684\\n0.677\\n0.720\\n0.686\\n0.699\\nFBERT–Large\\n0.774\\n0.693\\n0.705\\n0.736\\n0.701\\n0.717\\nPBERT–Large (idf)\\n0.758\\n0.653\\n0.704\\n0.734\\n0.685\\n0.705\\nRBERT–Large (idf)\\n0.771\\n0.680\\n0.661\\n0.718\\n0.687\\n0.692\\nFBERT–Large (idf)\\n0.774\\n0.678\\n0.700\\n0.740\\n0.701\\n0.711\\nPRoBERTa–Base\\n0.738\\n0.642\\n0.671\\n0.712\\n0.669\\n0.671\\nRRoBERTa–Base\\n0.745\\n0.669\\n0.645\\n0.698\\n0.682\\n0.653\\nFRoBERTa–Base\\n0.761\\n0.674\\n0.686\\n0.732\\n0.697\\n0.689\\nPRoBERTa–Base (idf)\\n0.751\\n0.626\\n0.678\\n0.723\\n0.685\\n0.668\\nRRoBERTa–Base (idf)\\n0.744\\n0.652\\n0.638\\n0.699\\n0.685\\n0.657\\nFRoBERTa–Base (idf)\\n0.767\\n0.653\\n0.688\\n0.737\\n0.705\\n0.685\\nPRoBERTa–Large\\n0.757\\n0.702\\n0.709\\n0.735\\n0.721\\n0.676\\nRRoBERTa–Large\\n0.765\\n0.713\\n0.686\\n0.718\\n0.714\\n0.676\\nFRoBERTa–Large\\n0.780\\n0.724\\n0.728\\n0.753\\n0.738\\n0.709\\nPRoBERTa–Large (idf)\\n0.771\\n0.682\\n0.705\\n0.727\\n0.714\\n0.681\\nRRoBERTa–Large (idf)\\n0.762\\n0.695\\n0.683\\n0.711\\n0.708\\n0.678\\nFRoBERTa–Large (idf)\\n0.786\\n0.704\\n0.727\\n0.747\\n0.732\\n0.711\\nPRoBERTa–Large–MNLI\\n0.777\\n0.718\\n0.733\\n0.744\\n0.729\\n0.747\\nRRoBERTa–Large–MNLI\\n0.790\\n0.731\\n0.702\\n0.741\\n0.727\\n0.732\\nFRoBERTa–Large–MNLI\\n0.795\\n0.736\\n0.733\\n0.757\\n0.744\\n0.756\\nPRoBERTa–Large–MNLI (idf)\\n0.794\\n0.695\\n0.731\\n0.752\\n0.732\\n0.747\\nRRoBERTa–Large–MNLI (idf)\\n0.792\\n0.706\\n0.694\\n0.737\\n0.724\\n0.733\\nFRoBERTa–Large–MNLI (idf)\\n0.804\\n0.710\\n0.729\\n0.760\\n0.742\\n0.754\\nPXLNet–Base\\n0.708\\n0.612\\n0.639\\n0.650\\n0.606\\n0.690\\nRXLNet–Base\\n0.728\\n0.630\\n0.617\\n0.645\\n0.621\\n0.675\\nFXLNet–Base\\n0.727\\n0.631\\n0.640\\n0.659\\n0.626\\n0.695\\nPXLNet–Base (idf)\\n0.726\\n0.618\\n0.655\\n0.678\\n0.629\\n0.700\\nRXLNet–Base (idf)\\n0.734\\n0.633\\n0.618\\n0.66\\n0.635\\n0.682\\nFXLNet–Base (idf)\\n0.739\\n0.633\\n0.649\\n0.681\\n0.643\\n0.702\\nPXL-NET–LARGE\\n0.710\\n0.577\\n0.643\\n0.647\\n0.616\\n0.684\\nRXL-NET–LARGE\\n0.732\\n0.600\\n0.610\\n0.636\\n0.627\\n0.668\\nFXL-NET–LARGE\\n0.733\\n0.600\\n0.643\\n0.655\\n0.637\\n0.691\\nPXL-NET–LARGE (idf)\\n0.728\\n0.574\\n0.652\\n0.669\\n0.633\\n0.681\\nRXL-NET–LARGE (idf)\\n0.735\\n0.592\\n0.597\\n0.642\\n0.629\\n0.662\\nFXL-NET–LARGE (idf)\\n0.742\\n0.592\\n0.643\\n0.670\\n0.645\\n0.685\\nPXLM–En\\n0.688\\n0.569\\n0.613\\n0.645\\n0.583\\n0.659\\nRXLM–En\\n0.715\\n0.603\\n0.577\\n0.645\\n0.609\\n0.644\\nFXLM–En\\n0.713\\n0.597\\n0.610\\n0.657\\n0.610\\n0.668\\nPXLM–En (idf)\\n0.728\\n0.576\\n0.649\\n0.681\\n0.604\\n0.683\\nRXLM–En (idf)\\n0.730\\n0.597\\n0.591\\n0.659\\n0.622\\n0.669\\nFXLM–En (idf)\\n0.739\\n0.594\\n0.636\\n0.682\\n0.626\\n0.691\\nTable 12: Pearson correlations with segment-level human judgments on WMT16 to-English trans-\\nlations. Correlations of metrics not significantly outperformed by any other for that language pair\\nare highlighted in bold. For each language pair, we specify the number of examples.\\n25\\n\\n\\nPublished as a conference paper at ICLR 2020\\nSetting\\nMetric\\ncs-en\\nde-en\\nfi-en\\nlv-en\\nru-en\\ntr-en\\nzh-en\\n560\\n560\\n560\\n560\\n560\\n560\\n560\\nUnsupervised\\nCHRF\\n0.514\\n0.531\\n0.671\\n0.525\\n0.599\\n0.607\\n0.591\\nCHRF++\\n0.523\\n0.534\\n0.678\\n0.520\\n0.588\\n0.614\\n0.593\\nMEANT 2.0\\n0.578\\n0.565\\n0.687\\n0.586\\n0.607\\n0.596\\n0.639\\nMEANT 2.0-NOSRL\\n0.566\\n0.564\\n0.682\\n0.573\\n0.591\\n0.582\\n0.630\\nSENTBLEU\\n0.435\\n0.432\\n0.571\\n0.393\\n0.484\\n0.538\\n0.512\\nTREEAGGREG\\n0.486\\n0.526\\n0.638\\n0.446\\n0.555\\n0.571\\n0.535\\nUHH_TSKM\\n0.507\\n0.479\\n0.600\\n0.394\\n0.465\\n0.478\\n0.477\\nSupervised\\nAUTODA\\n0.499\\n0.543\\n0.673\\n0.533\\n0.584\\n0.625\\n0.583\\nBEER\\n0.511\\n0.530\\n0.681\\n0.515\\n0.577\\n0.600\\n0.582\\nBLEND\\n0.594\\n0.571\\n0.733\\n0.577\\n0.622\\n0.671\\n0.661\\nBLEU2VEC\\n0.439\\n0.429\\n0.590\\n0.386\\n0.489\\n0.529\\n0.526\\nNGRAM2VEC\\n0.436\\n0.435\\n0.582\\n0.383\\n0.490\\n0.538\\n0.520\\nPre-Trained\\nPBERT–Base\\n0.625\\n0.659\\n0.808\\n0.688\\n0.698\\n0.713\\n0.675\\nRBERT–Base\\n0.653\\n0.645\\n0.782\\n0.662\\n0.678\\n0.716\\n0.715\\nFBERT–Base\\n0.654\\n0.671\\n0.811\\n0.692\\n0.707\\n0.731\\n0.714\\nPBERT–Base (idf)\\n0.626\\n0.668\\n0.819\\n0.708\\n0.719\\n0.702\\n0.667\\nRBERT–Base (idf)\\n0.652\\n0.658\\n0.789\\n0.678\\n0.696\\n0.703\\n0.712\\nFBERT–Base (idf)\\n0.657\\n0.680\\n0.823\\n0.712\\n0.725\\n0.718\\n0.711\\nPBERT–Base–MRPC\\n0.599\\n0.630\\n0.788\\n0.657\\n0.659\\n0.710\\n0.681\\nRBERT–Base–MRPC\\n0.613\\n0.620\\n0.754\\n0.616\\n0.650\\n0.685\\n0.705\\nFBERT–Base–MRPC\\n0.627\\n0.647\\n0.792\\n0.656\\n0.676\\n0.717\\n0.712\\nPBERT–Base–MRPC (idf)\\n0.609\\n0.630\\n0.801\\n0.680\\n0.676\\n0.712\\n0.682\\nRBERT–Base–MRPC (idf)\\n0.611\\n0.628\\n0.759\\n0.633\\n0.665\\n0.687\\n0.703\\nFBERT–Base–MRPC (idf)\\n0.633\\n0.649\\n0.803\\n0.678\\n0.690\\n0.719\\n0.713\\nPBERT–Large\\n0.638\\n0.685\\n0.816\\n0.717\\n0.719\\n0.746\\n0.693\\nRBERT–Large\\n0.661\\n0.676\\n0.782\\n0.693\\n0.705\\n0.744\\n0.730\\nFBERT–Large\\n0.666\\n0.701\\n0.814\\n0.723\\n0.730\\n0.760\\n0.731\\nPBERT–Large (idf)\\n0.644\\n0.692\\n0.827\\n0.728\\n0.729\\n0.734\\n0.689\\nRBERT–Large (idf)\\n0.665\\n0.686\\n0.796\\n0.712\\n0.729\\n0.733\\n0.730\\nFBERT–Large (idf)\\n0.671\\n0.707\\n0.829\\n0.738\\n0.745\\n0.746\\n0.729\\nPRoBERTa–Base\\n0.639\\n0.663\\n0.801\\n0.689\\n0.688\\n0.700\\n0.704\\nRRoBERTa–Base\\n0.648\\n0.652\\n0.768\\n0.651\\n0.669\\n0.684\\n0.734\\nFRoBERTa–Base\\n0.675\\n0.683\\n0.818\\n0.693\\n0.707\\n0.718\\n0.740\\nPRoBERTa–Base (idf)\\n0.629\\n0.655\\n0.804\\n0.702\\n0.711\\n0.707\\n0.700\\nRRoBERTa–Base (idf)\\n0.652\\n0.646\\n0.773\\n0.667\\n0.676\\n0.689\\n0.734\\nFRoBERTa–Base (idf)\\n0.673\\n0.673\\n0.823\\n0.708\\n0.719\\n0.721\\n0.739\\nPRoBERTa–Large\\n0.658\\n0.724\\n0.811\\n0.743\\n0.727\\n0.720\\n0.744\\nRRoBERTa–Large\\n0.685\\n0.714\\n0.778\\n0.711\\n0.718\\n0.713\\n0.759\\nFRoBERTa–Large\\n0.710\\n0.745\\n0.833\\n0.756\\n0.746\\n0.751\\n0.775\\nPRoBERTa–Large (idf)\\n0.644\\n0.721\\n0.815\\n0.740\\n0.734\\n0.736\\n0.734\\nRRoBERTa–Large (idf)\\n0.683\\n0.705\\n0.783\\n0.718\\n0.720\\n0.726\\n0.751\\nFRoBERTa–Large (idf)\\n0.703\\n0.737\\n0.838\\n0.761\\n0.752\\n0.764\\n0.767\\nPRoBERTa–Large–MNLI\\n0.694\\n0.736\\n0.822\\n0.764\\n0.741\\n0.754\\n0.737\\nRRoBERTa–Large–MNLI\\n0.706\\n0.725\\n0.785\\n0.732\\n0.741\\n0.750\\n0.760\\nFRoBERTa–Large–MNLI\\n0.722\\n0.747\\n0.822\\n0.764\\n0.758\\n0.767\\n0.765\\nPRoBERTa–Large–MNLI (idf)\\n0.686\\n0.733\\n0.836\\n0.772\\n0.760\\n0.767\\n0.738\\nRRoBERTa–Large–MNLI (idf)\\n0.697\\n0.717\\n0.796\\n0.741\\n0.753\\n0.757\\n0.762\\nFRoBERTa–Large–MNLI (idf)\\n0.714\\n0.740\\n0.835\\n0.774\\n0.773\\n0.776\\n0.767\\nPXLNET–Base\\n0.595\\n0.579\\n0.779\\n0.632\\n0.626\\n0.688\\n0.646\\nRXLNET–Base\\n0.603\\n0.560\\n0.746\\n0.617\\n0.624\\n0.689\\n0.677\\nFXLNET–Base\\n0.610\\n0.580\\n0.775\\n0.636\\n0.639\\n0.700\\n0.675\\nPXLNET–Base (idf)\\n0.616\\n0.603\\n0.795\\n0.665\\n0.659\\n0.693\\n0.649\\nRXLNET–Base (idf)\\n0.614\\n0.583\\n0.765\\n0.640\\n0.648\\n0.697\\n0.688\\nFXLNET–Base (idf)\\n0.627\\n0.603\\n0.795\\n0.663\\n0.665\\n0.707\\n0.684\\nPXLNET–Large\\n0.620\\n0.622\\n0.796\\n0.648\\n0.648\\n0.694\\n0.660\\nRXLNET–Large\\n0.622\\n0.601\\n0.758\\n0.628\\n0.645\\n0.684\\n0.701\\nFXLNET–Large\\n0.635\\n0.627\\n0.794\\n0.654\\n0.664\\n0.705\\n0.698\\nPXLNET–Large (idf)\\n0.635\\n0.633\\n0.808\\n0.673\\n0.672\\n0.688\\n0.649\\nRXLNET–Large (idf)\\n0.626\\n0.611\\n0.770\\n0.646\\n0.661\\n0.682\\n0.700\\nFXLNET–Large (idf)\\n0.646\\n0.636\\n0.809\\n0.675\\n0.682\\n0.700\\n0.695\\nPXLM–En\\n0.565\\n0.594\\n0.769\\n0.631\\n0.649\\n0.672\\n0.643\\nRXLM–En\\n0.592\\n0.586\\n0.734\\n0.618\\n0.647\\n0.673\\n0.686\\nFXLM–En\\n0.595\\n0.605\\n0.768\\n0.641\\n0.664\\n0.686\\n0.683\\nPXLM–En (idf)\\n0.599\\n0.618\\n0.795\\n0.670\\n0.686\\n0.690\\n0.657\\nRXLM–En (idf)\\n0.624\\n0.605\\n0.768\\n0.652\\n0.680\\n0.684\\n0.698\\nFXLM–En (idf)\\n0.630\\n0.624\\n0.798\\n0.676\\n0.698\\n0.698\\n0.694\\nTable 13: Absolute Pearson correlations with segment-level human judgments on WMT17 to-\\nEnglish translations. Correlations of metrics not significantly outperformed by any other for that\\nlanguage pair are highlighted in bold. For each language pair, we specify the number of examples.\\n26\\n\\n\\nPublished as a conference paper at ICLR 2020\\nSetting\\nMetric\\nen-cs\\nen-de\\nen-fi\\nen-lv\\nen-ru\\nen-tr\\nen-zh\\n32K\\n3K\\n3K\\n3K\\n560\\n247\\n560\\nτ\\nτ\\nτ\\nτ\\n|r|\\nτ\\n|r|\\nUnsupervised\\nAUTODA\\n0.041\\n0.099\\n0.204\\n0.130\\n0.511\\n0.409\\n0.609\\nAUTODA-TECTO\\n0.336\\n-\\n-\\n-\\n-\\n-\\n-\\nCHRF\\n0.376\\n0.336\\n0.503\\n0.420\\n0.605\\n0.466\\n0.608\\nCHRF+\\n0.377\\n0.325\\n0.514\\n0.421\\n0.609\\n0.474\\n-\\nCHRF++\\n0.368\\n0.328\\n0.484\\n0.417\\n0.604\\n0.466\\n0.602\\nMEANT 2.0\\n-\\n0.350\\n-\\n-\\n-\\n-\\n0.727\\nMEANT 2.0-NOSRL\\n0.395\\n0.324\\n0.565\\n0.425\\n0.636\\n0.482\\n0.705\\nSENTBLEU\\n0.274\\n0.269\\n0.446\\n0.259\\n0.468\\n0.377\\n0.642\\nTREEAGGREG\\n0.361\\n0.305\\n0.509\\n0.383\\n0.535\\n0.441\\n0.566\\nSupervised\\nBEER\\n0.398\\n0.336\\n0.557\\n0.420\\n0.569\\n0.490\\n0.622\\nBLEND\\n-\\n-\\n-\\n-\\n0.578\\n-\\n-\\nBLEU2VEC\\n0.305\\n0.313\\n0.503\\n0.315\\n0.472\\n0.425\\n-\\nNGRAM2VEC\\n-\\n-\\n0.486\\n0.317\\n-\\n-\\n-\\nPre-Trained\\nPBERT–Multi\\n0.412\\n0.364\\n0.561\\n0.435\\n0.606\\n0.579\\n0.759\\nRBERT–Multi\\n0.443\\n0.430\\n0.587\\n0.480\\n0.663\\n0.571\\n0.804\\nFBERT–Multi\\n0.440\\n0.404\\n0.587\\n0.466\\n0.653\\n0.587\\n0.806\\nPBERT–Multi (idf)\\n0.411\\n0.328\\n0.568\\n0.444\\n0.616\\n0.555\\n0.741\\nRBERT–Multi (idf)\\n0.449\\n0.416\\n0.591\\n0.479\\n0.665\\n0.579\\n0.796\\nFBERT–Multi (idf)\\n0.447\\n0.379\\n0.588\\n0.470\\n0.657\\n0.571\\n0.793\\nPXLM–100\\n0.406\\n0.383\\n0.553\\n0.423\\n0.562\\n0.611\\n0.722\\nRXLM–100\\n0.446\\n0.436\\n0.587\\n0.458\\n0.626\\n0.652\\n0.779\\nFXLM–100\\n0.444\\n0.424\\n0.577\\n0.456\\n0.613\\n0.628\\n0.778\\nPXLM–100 (idf)\\n0.419\\n0.367\\n0.557\\n0.427\\n0.571\\n0.595\\n0.719\\nRXLM–100 (idf)\\n0.450\\n0.424\\n0.592\\n0.464\\n0.632\\n0.644\\n0.770\\nFXLM–100 (idf)\\n0.448\\n0.419\\n0.580\\n0.459\\n0.617\\n0.644\\n0.771\\nTable 14: Absolute Pearson correlation (|r|) and Kendall correlation (τ) with segment-level human\\njudgments on WMT17 from-English translations. Correlations of metrics not significantly outper-\\nformed by any other for that language pair are highlighted in bold. For each language pair, we\\nspecify the number of examples.\\n27\\n\\n\\nPublished as a conference paper at ICLR 2020\\nSetting\\nMetric\\ncs-en\\nde-en\\nfi-en\\nlv-en\\nru-en\\ntr-en\\nzh-en\\n4\\n11\\n6\\n9\\n9\\n10\\n16\\nUnsupervised\\nBLEU\\n0.971\\n0.923\\n0.903\\n0.979\\n0.912\\n0.976\\n0.864\\nCDER\\n0.989\\n0.930\\n0.927\\n0.985\\n0.922\\n0.973\\n0.904\\nCHARACTER\\n0.972\\n0.974\\n0.946\\n0.932\\n0.958\\n0.949\\n0.799\\nCHRF\\n0.939\\n0.968\\n0.938\\n0.968\\n0.952\\n0.944\\n0.859\\nCHRF++\\n0.940\\n0.965\\n0.927\\n0.973\\n0.945\\n0.960\\n0.880\\nMEANT 2.0\\n0.926\\n0.950\\n0.941\\n0.970\\n0.962\\n0.932\\n0.838\\nMEANT 2.0-NOSRL\\n0.902\\n0.936\\n0.933\\n0.963\\n0.960\\n0.896\\n0.800\\nNIST\\n1.000\\n0.931\\n0.931\\n0.960\\n0.912\\n0.971\\n0.849\\nPER\\n0.968\\n0.951\\n0.896\\n0.962\\n0.911\\n0.932\\n0.877\\nTER\\n0.989\\n0.906\\n0.952\\n0.971\\n0.912\\n0.954\\n0.847\\nTREEAGGREG\\n0.983\\n0.920\\n0.977\\n0.986\\n0.918\\n0.987\\n0.861\\nUHH_TSKM\\n0.996\\n0.937\\n0.921\\n0.990\\n0.914\\n0.987\\n0.902\\nWER\\n0.987\\n0.896\\n0.948\\n0.969\\n0.907\\n0.925\\n0.839\\nSupervised\\nAUTODA\\n0.438\\n0.959\\n0.925\\n0.973\\n0.907\\n0.916\\n0.734\\nBEER\\n0.972\\n0.960\\n0.955\\n0.978\\n0.936\\n0.972\\n0.902\\nBLEND\\n0.968\\n0.976\\n0.958\\n0.979\\n0.964\\n0.984\\n0.894\\nBLEU2VEC\\n0.989\\n0.936\\n0.888\\n0.966\\n0.907\\n0.961\\n0.886\\nNGRAM2VEC\\n0.984\\n0.935\\n0.890\\n0.963\\n0.907\\n0.955\\n0.880\\nPre-Trained\\nPBERT–Base\\n0.975\\n0.936\\n0.991\\n0.993\\n0.918\\n0.981\\n0.892\\nRBERT–Base\\n0.995\\n0.975\\n0.944\\n0.978\\n0.953\\n0.991\\n0.975\\nFBERT–Base\\n0.987\\n0.961\\n0.979\\n0.991\\n0.937\\n0.991\\n0.953\\nPBERT–Base (idf)\\n0.983\\n0.937\\n0.998\\n0.992\\n0.939\\n0.985\\n0.878\\nRBERT–Base (idf)\\n0.997\\n0.981\\n0.962\\n0.968\\n0.977\\n0.985\\n0.949\\nFBERT–Base (idf)\\n0.992\\n0.967\\n0.995\\n0.992\\n0.960\\n0.996\\n0.951\\nPBERT–Base–MRPC\\n0.982\\n0.926\\n0.990\\n0.987\\n0.916\\n0.970\\n0.899\\nRBERT–Base–MRPC\\n0.999\\n0.979\\n0.950\\n0.982\\n0.957\\n0.977\\n0.985\\nFBERT–Base–MRPC\\n0.994\\n0.957\\n0.986\\n0.994\\n0.938\\n0.980\\n0.960\\nPBERT–Base–MRPC (idf)\\n0.989\\n0.936\\n0.992\\n0.979\\n0.931\\n0.976\\n0.892\\nRBERT–Base–MRPC (idf)\\n0.999\\n0.987\\n0.962\\n0.980\\n0.975\\n0.979\\n0.973\\nFBERT–Base–MRPC (idf)\\n0.997\\n0.968\\n0.995\\n0.997\\n0.956\\n0.989\\n0.963\\nPBERT–Large\\n0.981\\n0.937\\n0.991\\n0.996\\n0.921\\n0.987\\n0.905\\nRBERT–Large\\n0.996\\n0.975\\n0.953\\n0.985\\n0.954\\n0.992\\n0.977\\nFBERT–Large\\n0.990\\n0.960\\n0.981\\n0.995\\n0.938\\n0.992\\n0.957\\nPBERT–Large (idf)\\n0.986\\n0.938\\n0.998\\n0.995\\n0.939\\n0.994\\n0.897\\nRBERT–Large (idf)\\n0.997\\n0.982\\n0.967\\n0.979\\n0.974\\n0.992\\n0.966\\nFBERT–Large (idf)\\n0.994\\n0.965\\n0.993\\n0.995\\n0.958\\n0.998\\n0.959\\nPRoBERTa–Base\\n0.987\\n0.930\\n0.984\\n0.966\\n0.916\\n0.963\\n0.955\\nRRoBERTa–Base\\n0.999\\n0.982\\n0.947\\n0.979\\n0.956\\n0.986\\n0.984\\nFRoBERTa–Base\\n0.996\\n0.961\\n0.993\\n0.993\\n0.937\\n0.983\\n0.982\\nPRoBERTa–Base (idf)\\n0.990\\n0.938\\n0.980\\n0.956\\n0.929\\n0.967\\n0.962\\nRRoBERTa–Base (idf)\\n0.998\\n0.987\\n0.963\\n0.979\\n0.971\\n0.986\\n0.974\\nFRoBERTa–Base (idf)\\n0.996\\n0.970\\n0.999\\n0.994\\n0.952\\n0.989\\n0.982\\nPRoBERTa–Large\\n0.989\\n0.948\\n0.984\\n0.949\\n0.927\\n0.960\\n0.967\\nRRoBERTa–Large\\n0.998\\n0.988\\n0.957\\n0.983\\n0.969\\n0.982\\n0.984\\nFRoBERTa–Large\\n0.996\\n0.973\\n0.997\\n0.991\\n0.949\\n0.984\\n0.987\\nPRoBERTa–Large (idf)\\n0.989\\n0.959\\n0.975\\n0.935\\n0.944\\n0.968\\n0.974\\nRRoBERTa–Large (idf)\\n0.995\\n0.991\\n0.962\\n0.979\\n0.981\\n0.981\\n0.970\\nFRoBERTa–Large (idf)\\n0.996\\n0.982\\n0.998\\n0.991\\n0.965\\n0.991\\n0.984\\nPRoBERTa–Large–MNLI\\n0.994\\n0.963\\n0.995\\n0.990\\n0.944\\n0.981\\n0.974\\nRRoBERTa–Large–MNLI\\n0.995\\n0.991\\n0.962\\n0.981\\n0.973\\n0.985\\n0.984\\nFRoBERTa–Large–MNLI\\n0.999\\n0.982\\n0.992\\n0.996\\n0.961\\n0.988\\n0.989\\nPRoBERTa–Large–MNLI (idf)\\n0.995\\n0.970\\n0.997\\n0.985\\n0.955\\n0.988\\n0.979\\nRRoBERTa–Large–MNLI (idf)\\n0.994\\n0.992\\n0.967\\n0.977\\n0.983\\n0.988\\n0.972\\nFRoBERTa–Large–MNLI (idf)\\n0.999\\n0.989\\n0.996\\n0.997\\n0.972\\n0.994\\n0.987\\nPXLNET–Base\\n0.988\\n0.938\\n0.993\\n0.993\\n0.914\\n0.974\\n0.960\\nRXLNET–Base\\n0.999\\n0.978\\n0.956\\n0.977\\n0.946\\n0.981\\n0.980\\nFXLNET–Base\\n0.996\\n0.963\\n0.986\\n0.991\\n0.932\\n0.981\\n0.978\\nPXLNET–Base (idf)\\n0.992\\n0.951\\n0.998\\n0.996\\n0.930\\n0.982\\n0.939\\nRXLNET–Base (idf)\\n0.999\\n0.986\\n0.968\\n0.973\\n0.964\\n0.987\\n0.955\\nFXLNET–Base (idf)\\n0.998\\n0.974\\n0.996\\n0.994\\n0.950\\n0.990\\n0.970\\nPXLNET–Large\\n0.991\\n0.944\\n0.996\\n0.995\\n0.924\\n0.982\\n0.943\\nRXLNET–Large\\n0.996\\n0.981\\n0.945\\n0.971\\n0.961\\n0.986\\n0.958\\nFXLNET–Large\\n0.999\\n0.969\\n0.986\\n0.992\\n0.945\\n0.992\\n0.961\\nPXLNET–Large (idf)\\n0.995\\n0.955\\n0.999\\n0.996\\n0.941\\n0.985\\n0.937\\nRXLNET–Large (idf)\\n0.993\\n0.985\\n0.951\\n0.960\\n0.975\\n0.974\\n0.910\\nFXLNET–Large (idf)\\n1.000\\n0.978\\n0.994\\n0.993\\n0.962\\n0.994\\n0.954\\nPXLM–En\\n0.983\\n0.933\\n0.994\\n0.989\\n0.918\\n0.973\\n0.928\\nRXLM–En\\n0.998\\n0.978\\n0.949\\n0.983\\n0.957\\n0.985\\n0.972\\nFXLM–En\\n0.994\\n0.960\\n0.985\\n0.995\\n0.938\\n0.984\\n0.964\\nPXLM–En (idf)\\n0.986\\n0.940\\n0.997\\n0.992\\n0.939\\n0.979\\n0.916\\nRXLM–En (idf)\\n0.999\\n0.983\\n0.966\\n0.980\\n0.975\\n0.991\\n0.952\\nFXLM–En (idf)\\n0.995\\n0.967\\n0.996\\n0.998\\n0.959\\n0.993\\n0.958\\nTable 15: Absolute Pearson correlations with system-level human judgments on WMT17 to-English\\ntranslations. Correlations of metrics not significantly outperformed by any other for that language\\npair are highlighted in bold. For each language pair, we specify the number of systems.\\n28\\n\\n\\nPublished as a conference paper at ICLR 2020\\nSetting\\nMetric\\nen-cs\\nen-de\\nen-lv\\nen-ru\\nen-tr\\nen-zh\\n14\\n16\\n17\\n9\\n8\\n11\\nUnsupervised\\nBLEU\\n0.956\\n0.804\\n0.866\\n0.898\\n0.924\\n–\\nCDER\\n0.968\\n0.813\\n0.930\\n0.924\\n0.957\\n–\\nCHARACTER\\n0.981\\n0.938\\n0.897\\n0.939\\n0.975\\n0.933\\nCHRF\\n0.976\\n0.863\\n0.955\\n0.950\\n0.991\\n0.976\\nCHRF++\\n0.974\\n0.852\\n0.956\\n0.945\\n0.986\\n0.976\\nMEANT 2.0\\n–\\n0.858\\n–\\n–\\n–\\n0.956\\nMEANT 2.0-NOSRL\\n0.976\\n0.770\\n0.959\\n0.957\\n0.991\\n0.943\\nNIST\\n0.962\\n0.769\\n0.935\\n0.920\\n0.986\\n–\\nPER\\n0.954\\n0.687\\n0.851\\n0.887\\n0.963\\n–\\nTER\\n0.955\\n0.796\\n0.909\\n0.933\\n0.967\\n–\\nTREEAGGREG\\n0.947\\n0.773\\n0.927\\n0.921\\n0.983\\n0.938\\nUHH_TSKM\\n–\\n–\\n–\\n–\\n–\\n–\\nWER\\n0.954\\n0.802\\n0.906\\n0.934\\n0.956\\n–\\nSupervised\\nAUTODA\\n0.975\\n0.603\\n0.729\\n0.850\\n0.601\\n0.976\\nBEER\\n0.970\\n0.842\\n0.930\\n0.944\\n0.980\\n0.914\\nBLEND\\n–\\n–\\n–\\n0.953\\n–\\n–\\nBLEU2VEC\\n0.963\\n0.810\\n0.859\\n0.903\\n0.911\\n–\\nNGRAM2VEC\\n–\\n–\\n0.862\\n–\\n–\\n–\\nPre-Trained\\nPBERT–Multi\\n0.959\\n0.798\\n0.960\\n0.946\\n0.981\\n0.970\\nRBERT–Multi\\n0.982\\n0.909\\n0.957\\n0.980\\n0.979\\n0.994\\nFBERT–Multi\\n0.976\\n0.859\\n0.959\\n0.966\\n0.980\\n0.992\\nPBERT–Multi (idf)\\n0.963\\n0.760\\n0.960\\n0.947\\n0.984\\n0.971\\nRBERT–Multi (idf)\\n0.985\\n0.907\\n0.955\\n0.981\\n0.984\\n0.982\\nFBERT–Multi (idf)\\n0.979\\n0.841\\n0.958\\n0.968\\n0.984\\n0.991\\nPXLM–100\\n0.967\\n0.825\\n0.965\\n0.953\\n0.974\\n0.977\\nRXLM–100\\n0.980\\n0.902\\n0.965\\n0.982\\n0.977\\n0.979\\nFXLM–100\\n0.979\\n0.868\\n0.969\\n0.971\\n0.976\\n0.986\\nPXLM–100 (idf)\\n0.968\\n0.809\\n0.965\\n0.955\\n0.980\\n0.975\\nRXLM–100 (idf)\\n0.981\\n0.894\\n0.964\\n0.984\\n0.983\\n0.968\\nFXLM–100 (idf)\\n0.979\\n0.856\\n0.966\\n0.973\\n0.982\\n0.979\\nTable 16: Absolute Pearson correlations with system-level human judgments on WMT17 from-\\nEnglish translations. Correlations of metrics not significantly outperformed by any other for that\\nlanguage pair are highlighted in bold. For each language pair, we specify the number of systems.\\n29\\n\\n\\nPublished as a conference paper at ICLR 2020\\nSetting\\nMetric\\ncs-en\\nde-en\\net-en\\nfi-en\\nru-en\\ntr-en\\nzh-en\\n5K\\n78K\\n57K\\n16K\\n10K\\n9K\\n33K\\nUnsupervised\\nCHARACTER\\n0.256\\n0.450\\n0.286\\n0.185\\n0.244\\n0.172\\n0.202\\nITER\\n0.198\\n0.396\\n0.235\\n0.128\\n0.139\\n-0.029\\n0.144\\nMETEOR++\\n0.270\\n0.457\\n0.329\\n0.207\\n0.253\\n0.204\\n0.179\\nSENTBLEU\\n0.233\\n0.415\\n0.285\\n0.154\\n0.228\\n0.145\\n0.178\\nUHH_TSKM\\n0.274\\n0.436\\n0.300\\n0.168\\n0.235\\n0.154\\n0.151\\nYISI-0\\n0.301\\n0.474\\n0.330\\n0.225\\n0.294\\n0.215\\n0.205\\nYISI-1\\n0.319\\n0.488\\n0.351\\n0.231\\n0.300\\n0.234\\n0.211\\nYISI-1 SRL\\n0.317\\n0.483\\n0.345\\n0.237\\n0.306\\n0.233\\n0.209\\nSupervised\\nBEER\\n0.295\\n0.481\\n0.341\\n0.232\\n0.288\\n0.229\\n0.214\\nBLEND\\n0.322\\n0.492\\n0.354\\n0.226\\n0.290\\n0.232\\n0.217\\nRUSE\\n0.347\\n0.498\\n0.368\\n0.273\\n0.311\\n0.259\\n0.218\\nPre-Trained\\nPBERT–Base\\n0.349\\n0.522\\n0.373\\n0.264\\n0.325\\n0.264\\n0.232\\nRBERT–Base\\n0.370\\n0.528\\n0.378\\n0.291\\n0.333\\n0.257\\n0.244\\nFBERT–Base\\n0.373\\n0.531\\n0.385\\n0.287\\n0.341\\n0.266\\n0.243\\nPBERT–Base (idf)\\n0.352\\n0.524\\n0.382\\n0.27\\n0.326\\n0.277\\n0.235\\nRBERT–Base (idf)\\n0.368\\n0.536\\n0.388\\n0.300\\n0.340\\n0.284\\n0.244\\nFBERT–Base (idf)\\n0.375\\n0.535\\n0.393\\n0.294\\n0.339\\n0.289\\n0.243\\nPBERT–Base–MRPC\\n0.343\\n0.520\\n0.365\\n0.247\\n0.333\\n0.25\\n0.227\\nRBERT–Base–MRPC\\n0.370\\n0.524\\n0.373\\n0.277\\n0.34\\n0.261\\n0.244\\nFBERT–Base–MRPC\\n0.366\\n0.529\\n0.377\\n0.271\\n0.342\\n0.263\\n0.242\\nPBERT–Base–MRPC (idf)\\n0.348\\n0.522\\n0.371\\n0.25\\n0.318\\n0.256\\n0.224\\nRBERT–Base–MRPC (idf)\\n0.379\\n0.531\\n0.383\\n0.285\\n0.339\\n0.266\\n0.242\\nFBERT–Base–MRPC (idf)\\n0.373\\n0.534\\n0.383\\n0.274\\n0.342\\n0.275\\n0.242\\nPBERT–LARGE\\n0.361\\n0.529\\n0.380\\n0.276\\n0.340\\n0.266\\n0.241\\nRBERT–LARGE\\n0.386\\n0.532\\n0.386\\n0.297\\n0.347\\n0.268\\n0.247\\nFBERT–LARGE\\n0.402\\n0.537\\n0.390\\n0.296\\n0.344\\n0.274\\n0.252\\nPBERT–LARGE (idf)\\n0.377\\n0.532\\n0.390\\n0.287\\n0.342\\n0.292\\n0.246\\nRBERT–LARGE (idf)\\n0.386\\n0.544\\n0.396\\n0.308\\n0.356\\n0.287\\n0.251\\nFBERT–LARGE (idf)\\n0.388\\n0.545\\n0.399\\n0.309\\n0.358\\n0.300\\n0.257\\nPRoBERTa–Base\\n0.368\\n0.53\\n0.371\\n0.274\\n0.318\\n0.265\\n0.235\\nRRoBERTa–Base\\n0.383\\n0.536\\n0.376\\n0.283\\n0.336\\n0.253\\n0.245\\nFRoBERTa–Base\\n0.391\\n0.540\\n0.383\\n0.273\\n0.339\\n0.270\\n0.249\\nPRoBERTa–Base (idf)\\n0.379\\n0.528\\n0.372\\n0.261\\n0.314\\n0.265\\n0.232\\nRRoBERTa–Base (idf)\\n0.389\\n0.539\\n0.384\\n0.288\\n0.332\\n0.267\\n0.245\\nFRoBERTa–Base (idf)\\n0.400\\n0.540\\n0.385\\n0.274\\n0.337\\n0.277\\n0.247\\nPRoBERTa–LARGE\\n0.387\\n0.541\\n0.389\\n0.283\\n0.345\\n0.280\\n0.248\\nRRoBERTa–LARGE\\n0.388\\n0.546\\n0.391\\n0.304\\n0.343\\n0.290\\n0.255\\nFRoBERTa–LARGE\\n0.404\\n0.550\\n0.397\\n0.296\\n0.353\\n0.292\\n0.264\\nPRoBERTa–LARGE (idf)\\n0.391\\n0.540\\n0.387\\n0.280\\n0.334\\n0.284\\n0.252\\nRRoBERTa–LARGE (idf)\\n0.386\\n0.548\\n0.394\\n0.305\\n0.338\\n0.295\\n0.252\\nFRoBERTa–LARGE (idf)\\n0.408\\n0.550\\n0.395\\n0.293\\n0.346\\n0.296\\n0.260\\nPRoBERTa–Large–MNLI\\n0.397\\n0.549\\n0.396\\n0.299\\n0.351\\n0.295\\n0.253\\nRRoBERTa–Large–MNLI\\n0.404\\n0.553\\n0.393\\n0.313\\n0.351\\n0.279\\n0.253\\nFRoBERTa–Large–MNLI\\n0.418\\n0.557\\n0.402\\n0.312\\n0.362\\n0.290\\n0.258\\nPRoBERTa–Large–MNLI (idf)\\n0.414\\n0.552\\n0.399\\n0.301\\n0.349\\n0.306\\n0.249\\nRRoBERTa–Large–MNLI (idf)\\n0.412\\n0.555\\n0.400\\n0.316\\n0.357\\n0.289\\n0.258\\nFRoBERTa–Large–MNLI (idf)\\n0.417\\n0.559\\n0.403\\n0.309\\n0.357\\n0.307\\n0.258\\nPXLNet–Base\\n0.335\\n0.514\\n0.359\\n0.243\\n0.308\\n0.247\\n0.232\\nRXLNet–Base\\n0.351\\n0.515\\n0.362\\n0.261\\n0.311\\n0.227\\n0.232\\nFXLNet–Base\\n0.351\\n0.517\\n0.365\\n0.257\\n0.315\\n0.25\\n0.237\\nPXLNet–Base (idf)\\n0.339\\n0.516\\n0.366\\n0.258\\n0.307\\n0.261\\n0.236\\nRXLNet–Base (idf)\\n0.364\\n0.521\\n0.371\\n0.268\\n0.317\\n0.242\\n0.238\\nFXLNet–Base (idf)\\n0.355\\n0.524\\n0.374\\n0.265\\n0.320\\n0.261\\n0.241\\nPXL-NET–LARGE\\n0.344\\n0.522\\n0.371\\n0.252\\n0.316\\n0.264\\n0.233\\nRXL-NET–LARGE\\n0.358\\n0.524\\n0.374\\n0.275\\n0.332\\n0.249\\n0.239\\nFXL-NET–LARGE\\n0.357\\n0.530\\n0.380\\n0.265\\n0.334\\n0.263\\n0.238\\nPXL-NET–LARGE (idf)\\n0.348\\n0.520\\n0.373\\n0.260\\n0.319\\n0.265\\n0.235\\nRXL-NET–LARGE (idf)\\n0.366\\n0.529\\n0.378\\n0.278\\n0.331\\n0.266\\n0.241\\nFXL-NET–LARGE (idf)\\n0.375\\n0.530\\n0.382\\n0.274\\n0.332\\n0.274\\n0.240\\nPXLM–En\\n0.349\\n0.516\\n0.366\\n0.244\\n0.310\\n0.259\\n0.233\\nRXLM–En\\n0.358\\n0.518\\n0.364\\n0.264\\n0.320\\n0.244\\n0.237\\nFXLM–En\\n0.358\\n0.525\\n0.373\\n0.259\\n0.322\\n0.258\\n0.238\\nPXLM–En (idf)\\n0.355\\n0.527\\n0.374\\n0.254\\n0.311\\n0.28\\n0.238\\nRXLM–En (idf)\\n0.362\\n0.528\\n0.376\\n0.274\\n0.333\\n0.26\\n0.24\\nFXLM–En (idf)\\n0.367\\n0.531\\n0.382\\n0.273\\n0.330\\n0.275\\n0.246\\nTable 17: Kendall correlations with segment-level human judgments on WMT18 to-English transla-\\ntions. Correlations of metrics not significantly outperformed by any other for that language pair are\\nhighlighted in bold. For each language pair, we specify the number of examples.\\n30\\n\\n\\nPublished as a conference paper at ICLR 2020\\nSetting\\nMetric\\nen-cs\\nen-de\\nen-et\\nen-fi\\nen-ru\\nen-tr\\nen-zh\\n5K\\n20K\\n32K\\n10K\\n22K\\n1K\\n29K\\nUnsupervised\\nCHARACTER\\n0.414\\n0.604\\n0.464\\n0.403\\n0.352\\n0.404\\n0.313\\nITER\\n0.333\\n0.610\\n0.392\\n0.311\\n0.291\\n0.236\\n-\\nSENTBLEU\\n0.389\\n0.620\\n0.414\\n0.355\\n0.330\\n0.261\\n0.311\\nYISI-0\\n0.471\\n0.661\\n0.531\\n0.464\\n0.394\\n0.376\\n0.318\\nYISI-1\\n0.496\\n0.691\\n0.546\\n0.504\\n0.407\\n0.418\\n0.323\\nYISI-1 SRL\\n-\\n0.696\\n-\\n-\\n-\\n-\\n0.310\\nSupervised\\nBEER\\n0.518\\n0.686\\n0.558\\n0.511\\n0.403\\n0.374\\n0.302\\nBLEND\\n-\\n-\\n-\\n-\\n0.394\\n-\\n-\\nPre-Trained\\nPBERT–Multi\\n0.541\\n0.715\\n0.549\\n0.486\\n0.414\\n0.328\\n0.337\\nRBERT–Multi\\n0.570\\n0.728\\n0.594\\n0.565\\n0.420\\n0.411\\n0.367\\nFBERT–Multi\\n0.562\\n0.728\\n0.586\\n0.546\\n0.423\\n0.399\\n0.364\\nPBERT–Multi (idf)\\n0.525\\n0.7\\n0.54\\n0.495\\n0.423\\n0.352\\n0.338\\nRBERT–Multi (idf)\\n0.569\\n0.727\\n0.601\\n0.561\\n0.423\\n0.420\\n0.374\\nFBERT–Multi (idf)\\n0.553\\n0.721\\n0.585\\n0.537\\n0.425\\n0.406\\n0.366\\nPXLM–100\\n0.496\\n0.711\\n0.561\\n0.527\\n0.417\\n0.364\\n0.340\\nRXLM–100\\n0.564\\n0.724\\n0.612\\n0.584\\n0.418\\n0.432\\n0.363\\nFXLM–100\\n0.533\\n0.727\\n0.599\\n0.573\\n0.421\\n0.408\\n0.362\\nPXLM–100 (idf)\\n0.520\\n0.710\\n0.572\\n0.546\\n0.421\\n0.370\\n0.328\\nRXLM–100 (idf)\\n0.567\\n0.722\\n0.609\\n0.587\\n0.420\\n0.439\\n0.365\\nFXLM–100 (idf)\\n0.554\\n0.724\\n0.601\\n0.584\\n0.422\\n0.389\\n0.355\\nTable 18: Kendall correlations with segment-level human judgments on WMT18 from-English\\ntranslations. Correlations of metrics not significantly outperformed by any other for that language\\npair are highlighted in bold. For each language pair, we specify the number of examples.\\n31\\n\\n\\nPublished as a conference paper at ICLR 2020\\nSetting\\nMetric\\ncs-en\\nde-en\\net-en\\nfi-en\\nru-en\\ntr-en\\nzh-en\\n5\\n16\\n14\\n9\\n8\\n5\\n14\\nUnsupervised\\nBLEU\\n0.970\\n0.971\\n0.986\\n0.973\\n0.979\\n0.657\\n0.978\\nCDER\\n0.972\\n0.980\\n0.990\\n0.984\\n0.980\\n0.664\\n0.982\\nCHARACTER\\n0.970\\n0.993\\n0.979\\n0.989\\n0.991\\n0.782\\n0.950\\nITER\\n0.975\\n0.990\\n0.975\\n0.996\\n0.937\\n0.861\\n0.980\\nMETEOR++\\n0.945\\n0.991\\n0.978\\n0.971\\n0.995\\n0.864\\n0.962\\nNIST\\n0.954\\n0.984\\n0.983\\n0.975\\n0.973\\n0.970\\n0.968\\nPER\\n0.970\\n0.985\\n0.983\\n0.993\\n0.967\\n0.159\\n0.931\\nTER\\n0.950\\n0.970\\n0.990\\n0.968\\n0.970\\n0.533\\n0.975\\nUHH_TSKM\\n0.952\\n0.980\\n0.989\\n0.982\\n0.980\\n0.547\\n0.981\\nWER\\n0.951\\n0.961\\n0.991\\n0.961\\n0.968\\n0.041\\n0.975\\nYISI-0\\n0.956\\n0.994\\n0.975\\n0.978\\n0.988\\n0.954\\n0.957\\nYISI-1\\n0.950\\n0.992\\n0.979\\n0.973\\n0.991\\n0.958\\n0.951\\nYISI-1 SRL\\n0.965\\n0.995\\n0.981\\n0.977\\n0.992\\n0.869\\n0.962\\nSupervised\\nBEER\\n0.958\\n0.994\\n0.985\\n0.991\\n0.982\\n0.870\\n0.976\\nBLEND\\n0.973\\n0.991\\n0.985\\n0.994\\n0.993\\n0.801\\n0.976\\nRUSE\\n0.981\\n0.997\\n0.990\\n0.991\\n0.988\\n0.853\\n0.981\\nPre-Trained\\nPBERT–Base\\n0.965\\n0.995\\n0.986\\n0.973\\n0.976\\n0.941\\n0.974\\nRBERT–Base\\n0.994\\n0.991\\n0.979\\n0.992\\n0.991\\n0.067\\n0.988\\nFBERT–Base\\n0.982\\n0.994\\n0.983\\n0.986\\n0.985\\n0.949\\n0.984\\nPBERT–Base (idf)\\n0.961\\n0.993\\n0.987\\n0.988\\n0.976\\n0.984\\n0.973\\nRBERT–Base (idf)\\n0.996\\n0.994\\n0.977\\n0.995\\n0.995\\n0.874\\n0.983\\nFBERT–Base (idf)\\n0.981\\n0.995\\n0.984\\n0.995\\n0.988\\n0.994\\n0.981\\nPBERT–Base–MRPC\\n0.957\\n0.994\\n0.989\\n0.953\\n0.976\\n0.798\\n0.977\\nRBERT–Base–MRPC\\n0.992\\n0.994\\n0.983\\n0.988\\n0.993\\n0.707\\n0.990\\nFBERT–Base–MRPC\\n0.975\\n0.995\\n0.987\\n0.975\\n0.986\\n0.526\\n0.986\\nPBERT–Base–MRPC (idf)\\n0.957\\n0.997\\n0.989\\n0.967\\n0.975\\n0.894\\n0.980\\nRBERT–Base–MRPC (idf)\\n0.991\\n0.997\\n0.981\\n0.994\\n0.993\\n0.052\\n0.987\\nFBERT–Base–MRPC (idf)\\n0.975\\n0.998\\n0.987\\n0.985\\n0.987\\n0.784\\n0.987\\nPBERT–Large\\n0.978\\n0.992\\n0.987\\n0.971\\n0.977\\n0.920\\n0.978\\nRBERT–Large\\n0.997\\n0.990\\n0.985\\n0.990\\n0.992\\n0.098\\n0.990\\nFBERT–Large\\n0.989\\n0.992\\n0.987\\n0.983\\n0.985\\n0.784\\n0.986\\nPBERT–Large (idf)\\n0.977\\n0.992\\n0.988\\n0.986\\n0.976\\n0.980\\n0.977\\nRBERT–Large (idf)\\n0.998\\n0.993\\n0.983\\n0.996\\n0.995\\n0.809\\n0.986\\nFBERT–Large (idf)\\n0.989\\n0.993\\n0.986\\n0.993\\n0.987\\n0.976\\n0.984\\nPRoBERTa–Base\\n0.970\\n0.995\\n0.991\\n0.998\\n0.976\\n0.796\\n0.980\\nRRoBERTa–Base\\n0.996\\n0.996\\n0.982\\n0.998\\n0.994\\n0.477\\n0.991\\nFRoBERTa–Base\\n0.984\\n0.997\\n0.989\\n0.999\\n0.987\\n0.280\\n0.989\\nPRoBERTa–Base (idf)\\n0.966\\n0.993\\n0.991\\n0.994\\n0.977\\n0.880\\n0.984\\nRRoBERTa–Base (idf)\\n0.995\\n0.998\\n0.981\\n0.998\\n0.995\\n0.230\\n0.989\\nFRoBERTa–Base (idf)\\n0.981\\n0.998\\n0.989\\n0.997\\n0.988\\n0.741\\n0.990\\nPRoBERTa–Large\\n0.980\\n0.998\\n0.990\\n0.995\\n0.982\\n0.791\\n0.981\\nRRoBERTa–Large\\n0.998\\n0.997\\n0.986\\n0.997\\n0.995\\n0.054\\n0.990\\nFRoBERTa–Large\\n0.990\\n0.999\\n0.990\\n0.998\\n0.990\\n0.499\\n0.988\\nPRoBERTa–Large (idf)\\n0.972\\n0.997\\n0.993\\n0.985\\n0.982\\n0.920\\n0.983\\nRRoBERTa–Large (idf)\\n0.996\\n0.997\\n0.984\\n0.997\\n0.995\\n0.578\\n0.989\\nFRoBERTa–Large (idf)\\n0.985\\n0.999\\n0.992\\n0.992\\n0.991\\n0.826\\n0.989\\nPRoBERTa–Large–MNLI\\n0.989\\n0.998\\n0.994\\n0.998\\n0.985\\n0.908\\n0.982\\nRRoBERTa–Large–MNLI\\n1.000\\n0.996\\n0.988\\n0.996\\n0.995\\n0.097\\n0.991\\nFRoBERTa–Large–MNLI\\n0.996\\n0.998\\n0.992\\n0.998\\n0.992\\n0.665\\n0.989\\nPRoBERTa–Large–MNLI (idf)\\n0.986\\n0.998\\n0.994\\n0.993\\n0.986\\n0.989\\n0.985\\nRRoBERTa–Large–MNLI (idf)\\n0.999\\n0.997\\n0.986\\n0.997\\n0.993\\n0.633\\n0.990\\nFRoBERTa–Large–MNLI (idf)\\n0.995\\n0.998\\n0.991\\n0.996\\n0.993\\n0.963\\n0.990\\nPXLNET–Base\\n0.970\\n0.996\\n0.986\\n0.990\\n0.979\\n0.739\\n0.982\\nRXLNET–Base\\n0.994\\n0.997\\n0.979\\n0.995\\n0.994\\n0.795\\n0.990\\nFXLNET–Base\\n0.983\\n0.997\\n0.983\\n0.993\\n0.987\\n0.505\\n0.988\\nPXLNET–Base (idf)\\n0.968\\n0.998\\n0.986\\n0.990\\n0.978\\n0.923\\n0.982\\nRXLNET–Base (idf)\\n0.993\\n0.998\\n0.978\\n0.996\\n0.994\\n0.439\\n0.988\\nFXLNET–Base (idf)\\n0.981\\n0.999\\n0.984\\n0.995\\n0.989\\n0.722\\n0.988\\nPXLNET–Large\\n0.969\\n0.998\\n0.986\\n0.995\\n0.979\\n0.880\\n0.981\\nRXLNET–Large\\n0.995\\n0.997\\n0.977\\n0.997\\n0.995\\n0.430\\n0.988\\nFXLNET–Large\\n0.983\\n0.998\\n0.983\\n0.997\\n0.988\\n0.713\\n0.988\\nPXLNET–Large (idf)\\n0.963\\n0.996\\n0.986\\n0.995\\n0.978\\n0.939\\n0.979\\nRXLNET–Large (idf)\\n0.992\\n0.997\\n0.975\\n0.993\\n0.996\\n0.531\\n0.982\\nFXLNET–Large (idf)\\n0.978\\n0.997\\n0.983\\n0.996\\n0.990\\n0.886\\n0.984\\nPXLM–En\\n0.965\\n0.996\\n0.990\\n0.978\\n0.980\\n0.946\\n0.981\\nRXLM–En\\n0.990\\n0.995\\n0.984\\n0.996\\n0.996\\n0.286\\n0.987\\nFXLM–En\\n0.978\\n0.997\\n0.988\\n0.990\\n0.989\\n0.576\\n0.987\\nPXLM–En (idf)\\n0.960\\n0.996\\n0.990\\n0.987\\n0.980\\n0.989\\n0.981\\nRXLM–En (idf)\\n0.991\\n0.997\\n0.983\\n0.996\\n0.998\\n0.612\\n0.985\\nFXLM–En (idf)\\n0.976\\n0.998\\n0.988\\n0.994\\n0.992\\n0.943\\n0.985\\nTable 19: Absolute Pearson correlations with system-level human judgments on WMT18 to-English\\ntranslations. Correlations of metrics not significantly outperformed by any other for that language\\npair are highlighted in bold. For each language pair, we specify the number of systems.\\n32\\n\\n\\nPublished as a conference paper at ICLR 2020\\nSetting\\nMetric\\nen-cs\\nen-de\\nen-et\\nen-fi\\nen-ru\\nen-tr\\nen-zh\\n5\\n16\\n14\\n12\\n9\\n8\\n14\\nUnsupervised\\nBLEU\\n0.995\\n0.981\\n0.975\\n0.962\\n0.983\\n0.826\\n0.947\\nCDER\\n0.997\\n0.986\\n0.984\\n0.964\\n0.984\\n0.861\\n0.961\\nCHARACTER\\n0.993\\n0.989\\n0.956\\n0.974\\n0.983\\n0.833\\n0.983\\nITER\\n0.915\\n0.984\\n0.981\\n0.973\\n0.975\\n0.865\\n–\\nMETEOR++\\n–\\n–\\n–\\n–\\n–\\n–\\n–\\nNIST\\n0.999\\n0.986\\n0.983\\n0.949\\n0.990\\n0.902\\n0.950\\nPER\\n0.991\\n0.981\\n0.958\\n0.906\\n0.988\\n0.859\\n0.964\\nTER\\n0.997\\n0.988\\n0.981\\n0.942\\n0.987\\n0.867\\n0.963\\nUHH_TSKM\\n–\\n–\\n–\\n–\\n–\\n–\\n–\\nWER\\n0.997\\n0.986\\n0.981\\n0.945\\n0.985\\n0.853\\n0.957\\nYISI-0\\n0.973\\n0.985\\n0.968\\n0.944\\n0.990\\n0.990\\n0.957\\nYISI-1\\n0.987\\n0.985\\n0.979\\n0.940\\n0.992\\n0.976\\n0.963\\nYISI-1 SRL\\n–\\n0.990\\n–\\n–\\n–\\n–\\n0.952\\nSupervised\\nBEER\\n0.992\\n0.991\\n0.980\\n0.961\\n0.988\\n0.965\\n0.928\\nBLEND\\n–\\n–\\n–\\n–\\n0.988\\n–\\n–\\nRUSE\\n–\\n–\\n–\\n–\\n–\\n–\\n–\\nPre-Trained\\nPBERT–Multi\\n0.994\\n0.988\\n0.981\\n0.957\\n0.990\\n0.935\\n0.954\\nRBERT–Multi\\n0.997\\n0.990\\n0.980\\n0.980\\n0.989\\n0.879\\n0.976\\nFBERT–Multi\\n0.997\\n0.989\\n0.982\\n0.972\\n0.990\\n0.908\\n0.967\\nPBERT–Multi (idf)\\n0.992\\n0.986\\n0.974\\n0.954\\n0.991\\n0.969\\n0.954\\nRBERT–Multi (idf)\\n0.997\\n0.993\\n0.982\\n0.982\\n0.992\\n0.901\\n0.984\\nFBERT–Multi (idf)\\n0.995\\n0.990\\n0.981\\n0.972\\n0.991\\n0.941\\n0.973\\nPXLM–100\\n0.984\\n0.992\\n0.993\\n0.972\\n0.993\\n0.962\\n0.965\\nRXLM–100\\n0.991\\n0.992\\n0.992\\n0.989\\n0.992\\n0.895\\n0.983\\nFXLM–100\\n0.988\\n0.993\\n0.993\\n0.986\\n0.993\\n0.935\\n0.976\\nPXLM–100 (idf)\\n0.982\\n0.992\\n0.994\\n0.975\\n0.993\\n0.968\\n0.964\\nRXLM–100 (idf)\\n0.993\\n0.993\\n0.991\\n0.989\\n0.993\\n0.911\\n0.986\\nFXLM–100 (idf)\\n0.989\\n0.993\\n0.994\\n0.985\\n0.993\\n0.945\\n0.979\\nTable 20: Absolute Pearson correlations with system-level human judgments on WMT18 from-\\nEnglish translations. Correlations of metrics not significantly outperformed by any other for that\\nlanguage pair are highlighted in bold. For each language pair, we specify the number of systems.\\n33\\n\\n\\nPublished as a conference paper at ICLR 2020\\nSetting\\nMetric\\ncs-en\\nde-en\\net-en\\nfi-en\\nru-en\\ntr-en\\nzh-en\\n10K\\n10K\\n10K\\n10K\\n10K\\n10K\\n10K\\nUnsupervised\\nBLEU\\n0.956\\n0.969\\n0.981\\n0.962\\n0.972\\n0.586\\n0.968\\nCDER\\n0.964\\n0.980\\n0.988\\n0.976\\n0.974\\n0.577\\n0.973\\nCHARACTER\\n0.960\\n0.992\\n0.975\\n0.979\\n0.984\\n0.680\\n0.942\\nITER\\n0.966\\n0.990\\n0.975\\n0.989\\n0.943\\n0.742\\n0.978\\nMETEOR++\\n0.937\\n0.990\\n0.975\\n0.962\\n0.989\\n0.787\\n0.954\\nNIST\\n0.942\\n0.982\\n0.980\\n0.965\\n0.965\\n0.862\\n0.959\\nPER\\n0.937\\n0.982\\n0.978\\n0.983\\n0.955\\n0.043\\n0.923\\nTER\\n0.942\\n0.970\\n0.988\\n0.960\\n0.963\\n0.450\\n0.967\\nUHH_TSKM\\n0.943\\n0.979\\n0.987\\n0.974\\n0.973\\n0.443\\n0.972\\nWER\\n0.942\\n0.961\\n0.989\\n0.953\\n0.962\\n0.072\\n0.967\\nYISI-0\\n0.947\\n0.992\\n0.972\\n0.969\\n0.982\\n0.863\\n0.950\\nYISI-1\\n0.942\\n0.991\\n0.976\\n0.964\\n0.985\\n0.881\\n0.943\\nYISI-1 SRL\\n0.957\\n0.994\\n0.978\\n0.968\\n0.986\\n0.785\\n0.954\\nSupervised\\nBEER\\n0.950\\n0.993\\n0.983\\n0.982\\n0.976\\n0.723\\n0.968\\nBLEND\\n0.965\\n0.990\\n0.982\\n0.985\\n0.986\\n0.724\\n0.969\\nRUSE\\n0.974\\n0.996\\n0.988\\n0.983\\n0.982\\n0.780\\n0.973\\nPre-Trained\\nPBERT–Base\\n0.954\\n0.992\\n0.984\\n0.980\\n0.970\\n0.917\\n0.965\\nRBERT–Base\\n0.988\\n0.994\\n0.974\\n0.987\\n0.988\\n0.801\\n0.975\\nFBERT–Base\\n0.973\\n0.994\\n0.981\\n0.987\\n0.982\\n0.924\\n0.973\\nPBERT–Base (idf)\\n0.957\\n0.994\\n0.983\\n0.966\\n0.970\\n0.875\\n0.966\\nRBERT–Base (idf)\\n0.986\\n0.990\\n0.976\\n0.984\\n0.984\\n0.019\\n0.980\\nFBERT–Base (idf)\\n0.974\\n0.993\\n0.980\\n0.978\\n0.978\\n0.853\\n0.976\\nPBERT–Base–MRPC\\n0.949\\n0.995\\n0.986\\n0.960\\n0.969\\n0.832\\n0.972\\nRBERT–Base–MRPC\\n0.983\\n0.997\\n0.979\\n0.986\\n0.986\\n0.099\\n0.980\\nFBERT–Base–MRPC\\n0.967\\n0.997\\n0.984\\n0.978\\n0.981\\n0.722\\n0.979\\nPBERT–Base–MRPC (idf)\\n0.949\\n0.994\\n0.986\\n0.946\\n0.969\\n0.743\\n0.969\\nRBERT–Base–MRPC (idf)\\n0.984\\n0.994\\n0.980\\n0.980\\n0.986\\n0.541\\n0.982\\nFBERT–Base–MRPC (idf)\\n0.967\\n0.995\\n0.984\\n0.968\\n0.979\\n0.464\\n0.978\\nPBERT–Large\\n0.969\\n0.991\\n0.985\\n0.979\\n0.970\\n0.915\\n0.969\\nRBERT–Large\\n0.990\\n0.993\\n0.980\\n0.988\\n0.988\\n0.745\\n0.978\\nFBERT–Large\\n0.982\\n0.993\\n0.984\\n0.986\\n0.981\\n0.909\\n0.976\\nPBERT–Large (idf)\\n0.970\\n0.991\\n0.984\\n0.963\\n0.971\\n0.858\\n0.970\\nRBERT–Large (idf)\\n0.989\\n0.990\\n0.982\\n0.982\\n0.985\\n0.047\\n0.982\\nFBERT–Large (idf)\\n0.981\\n0.991\\n0.984\\n0.976\\n0.978\\n0.722\\n0.978\\nPRoBERTa–Base\\n0.959\\n0.992\\n0.988\\n0.986\\n0.971\\n0.809\\n0.976\\nRRoBERTa–Base\\n0.987\\n0.997\\n0.978\\n0.989\\n0.988\\n0.238\\n0.981\\nFRoBERTa–Base\\n0.973\\n0.997\\n0.987\\n0.989\\n0.982\\n0.674\\n0.982\\nPRoBERTa–Base (idf)\\n0.963\\n0.994\\n0.988\\n0.989\\n0.970\\n0.711\\n0.972\\nRRoBERTa–Base (idf)\\n0.988\\n0.996\\n0.979\\n0.989\\n0.987\\n0.353\\n0.983\\nFRoBERTa–Base (idf)\\n0.976\\n0.997\\n0.986\\n0.990\\n0.980\\n0.277\\n0.980\\nPRoBERTa–Large\\n0.965\\n0.995\\n0.990\\n0.976\\n0.976\\n0.846\\n0.975\\nRRoBERTa–Large\\n0.989\\n0.997\\n0.982\\n0.989\\n0.988\\n0.540\\n0.981\\nFRoBERTa–Large\\n0.978\\n0.998\\n0.989\\n0.983\\n0.985\\n0.760\\n0.981\\nPRoBERTa–Large (idf)\\n0.972\\n0.997\\n0.988\\n0.986\\n0.976\\n0.686\\n0.973\\nRRoBERTa–Large (idf)\\n0.990\\n0.996\\n0.983\\n0.989\\n0.989\\n0.096\\n0.982\\nFRoBERTa–Large (idf)\\n0.982\\n0.998\\n0.988\\n0.989\\n0.983\\n0.453\\n0.980\\nPRoBERTa–Large–MNLI\\n0.978\\n0.997\\n0.991\\n0.984\\n0.980\\n0.914\\n0.977\\nRRoBERTa–Large–MNLI\\n0.991\\n0.996\\n0.984\\n0.989\\n0.987\\n0.566\\n0.982\\nFRoBERTa–Large–MNLI\\n0.987\\n0.998\\n0.989\\n0.988\\n0.986\\n0.873\\n0.982\\nPRoBERTa–Large–MNLI (idf)\\n0.982\\n0.998\\n0.992\\n0.990\\n0.978\\n0.822\\n0.974\\nRRoBERTa–Large–MNLI (idf)\\n0.992\\n0.996\\n0.985\\n0.988\\n0.988\\n0.022\\n0.983\\nFRoBERTa–Large–MNLI (idf)\\n0.989\\n0.998\\n0.990\\n0.990\\n0.985\\n0.583\\n0.980\\nPXLNET–Base\\n0.960\\n0.997\\n0.984\\n0.982\\n0.972\\n0.849\\n0.974\\nRXLNET–Base\\n0.985\\n0.997\\n0.975\\n0.988\\n0.988\\n0.303\\n0.980\\nFXLNET–Base\\n0.974\\n0.998\\n0.981\\n0.986\\n0.982\\n0.628\\n0.980\\nPXLNET–Base (idf)\\n0.962\\n0.995\\n0.983\\n0.982\\n0.972\\n0.657\\n0.974\\nRXLNET–Base (idf)\\n0.986\\n0.996\\n0.976\\n0.987\\n0.987\\n0.666\\n0.982\\nFXLNET–Base (idf)\\n0.975\\n0.996\\n0.980\\n0.985\\n0.981\\n0.259\\n0.980\\nPXLNET–Large\\n0.955\\n0.995\\n0.983\\n0.986\\n0.972\\n0.875\\n0.970\\nRXLNET–Large\\n0.984\\n0.996\\n0.972\\n0.984\\n0.989\\n0.491\\n0.975\\nFXLNET–Large\\n0.971\\n0.996\\n0.980\\n0.987\\n0.984\\n0.821\\n0.976\\nPXLNET–Large (idf)\\n0.961\\n0.997\\n0.983\\n0.987\\n0.973\\n0.816\\n0.973\\nRXLNET–Large (idf)\\n0.987\\n0.996\\n0.975\\n0.989\\n0.988\\n0.320\\n0.981\\nFXLNET–Large (idf)\\n0.976\\n0.997\\n0.980\\n0.989\\n0.982\\n0.623\\n0.980\\nPXLM–En\\n0.953\\n0.995\\n0.988\\n0.979\\n0.974\\n0.918\\n0.972\\nRXLM–En\\n0.983\\n0.996\\n0.980\\n0.988\\n0.991\\n0.561\\n0.977\\nFXLM–En\\n0.969\\n0.997\\n0.986\\n0.986\\n0.985\\n0.869\\n0.977\\nPXLM–En (idf)\\n0.957\\n0.996\\n0.987\\n0.970\\n0.974\\n0.862\\n0.973\\nRXLM–En (idf)\\n0.982\\n0.995\\n0.981\\n0.988\\n0.989\\n0.213\\n0.980\\nFXLM–En (idf)\\n0.970\\n0.996\\n0.985\\n0.982\\n0.982\\n0.519\\n0.978\\nTable 21: Absolute Pearson correlations with human judgments on WMT18 to-English language\\npairs for 10K hybrid systems. Correlations of metrics not significantly outperformed by any other\\nfor that language pair are highlighted in bold. For each language pair, we specify the number of\\nsystems.\\n34\\n\\n\\nPublished as a conference paper at ICLR 2020\\nSetting\\nMetric\\nen-cs\\nen-de\\nen-et\\nen-fi\\nen-ru\\nen-tr\\nen-zh\\n10K\\n10K\\n10K\\n10K\\n10K\\n10K\\n10K\\nUnsupervised\\nBLEU\\n0.993\\n0.977\\n0.971\\n0.958\\n0.977\\n0.796\\n0.941\\nCDER\\n0.995\\n0.984\\n0.981\\n0.961\\n0.982\\n0.832\\n0.956\\nCHARACTER\\n0.990\\n0.986\\n0.950\\n0.963\\n0.981\\n0.775\\n0.978\\nITER\\n0.865\\n0.978\\n0.982\\n0.966\\n0.965\\n0.872\\n–\\nMETEOR++\\n–\\n–\\n–\\n–\\n–\\n–\\n–\\nNIST\\n0.997\\n0.984\\n0.980\\n0.944\\n0.988\\n0.870\\n0.944\\nPER\\n0.987\\n0.979\\n0.954\\n0.904\\n0.986\\n0.829\\n0.950\\nTER\\n0.995\\n0.986\\n0.977\\n0.939\\n0.985\\n0.837\\n0.959\\nUHH_TSKM\\n–\\n–\\n–\\n–\\n–\\n–\\n–\\nWER\\n0.994\\n0.984\\n0.977\\n0.942\\n0.983\\n0.824\\n0.954\\nYISI-0\\n0.971\\n0.983\\n0.965\\n0.942\\n0.988\\n0.953\\n0.951\\nYISI-1\\n0.985\\n0.983\\n0.976\\n0.938\\n0.989\\n0.942\\n0.957\\nYISI-1 SRL\\n–\\n0.988\\n–\\n–\\n–\\n–\\n0.948\\nSupervised\\nBEER\\n0.990\\n0.989\\n0.978\\n0.959\\n0.986\\n0.933\\n0.925\\nBLEND\\n–\\n–\\n–\\n–\\n0.986\\n–\\n–\\nRUSE\\n–\\n–\\n–\\n–\\n–\\n–\\n–\\nPre-Trained\\nPBERT–Multi\\n0.989\\n0.983\\n0.970\\n0.951\\n0.988\\n0.936\\n0.950\\nRBERT–Multi\\n0.995\\n0.991\\n0.979\\n0.977\\n0.989\\n0.872\\n0.980\\nFBERT–Multi\\n0.993\\n0.988\\n0.978\\n0.969\\n0.989\\n0.910\\n0.969\\nPBERT–Multi (idf)\\n0.992\\n0.986\\n0.978\\n0.954\\n0.988\\n0.903\\n0.950\\nRBERT–Multi (idf)\\n0.995\\n0.988\\n0.977\\n0.976\\n0.987\\n0.850\\n0.972\\nFBERT–Multi (idf)\\n0.995\\n0.988\\n0.979\\n0.969\\n0.987\\n0.877\\n0.963\\nPXLM–100\\n0.980\\n0.990\\n0.991\\n0.972\\n0.991\\n0.936\\n0.959\\nRXLM–100\\n0.991\\n0.990\\n0.989\\n0.985\\n0.991\\n0.882\\n0.981\\nFXLM–100\\n0.987\\n0.990\\n0.991\\n0.981\\n0.991\\n0.915\\n0.974\\nPXLM–100 (idf)\\n0.982\\n0.990\\n0.990\\n0.968\\n0.991\\n0.931\\n0.960\\nRXLM–100 (idf)\\n0.989\\n0.990\\n0.990\\n0.985\\n0.990\\n0.867\\n0.978\\nFXLM–100 (idf)\\n0.986\\n0.991\\n0.991\\n0.982\\n0.991\\n0.905\\n0.972\\nTable 22: Absolute Pearson correlations with human judgments on WMT18 from-English language\\npairs for 10K hybrid systems. Correlations of metrics not significantly outperformed by any other\\nfor that language pair are highlighted in bold. For each language pair, we specify the number of\\nsystems.\\n35\\n\\n\\nPublished as a conference paper at ICLR 2020\\nSetting\\nMetric\\ncs-en\\nde-en\\net-en\\nfi-en\\nru-en\\ntr-en\\nzh-en\\nUnsupervised\\nBLEU\\n0.135\\n0.804\\n0.757\\n0.460\\n0.230\\n0.096\\n0.661\\nCDER\\n0.162\\n0.795\\n0.764\\n0.493\\n0.234\\n0.087\\n0.660\\nCHARACTER\\n0.146\\n0.737\\n0.696\\n0.496\\n0.201\\n0.082\\n0.584\\nITER\\n0.152\\n0.814\\n0.746\\n0.474\\n0.234\\n0.100\\n0.673\\nMETEOR++\\n0.172\\n0.804\\n0.646\\n0.456\\n0.253\\n0.052\\n0.597\\nNIST\\n0.136\\n0.802\\n0.739\\n0.469\\n0.228\\n0.135\\n0.665\\nPER\\n0.121\\n0.764\\n0.602\\n0.455\\n0.218\\n0.000\\n0.602\\nTER\\n0.139\\n0.789\\n0.768\\n0.470\\n0.232\\n0.001\\n0.652\\nUHH_TSKM\\n0.191\\n0.803\\n0.768\\n0.469\\n0.240\\n0.002\\n0.642\\nWER\\n0.149\\n0.776\\n0.760\\n0.471\\n0.227\\n0.000\\n0.654\\nYISI-0\\n0.148\\n0.780\\n0.703\\n0.483\\n0.229\\n0.106\\n0.629\\nYISI-1\\n0.157\\n0.808\\n0.752\\n0.466\\n0.250\\n0.110\\n0.613\\nYISI-1 SRL\\n0.159\\n0.814\\n0.763\\n0.484\\n0.243\\n0.008\\n0.620\\nSupervised\\nBEER\\n0.165\\n0.811\\n0.765\\n0.485\\n0.237\\n0.030\\n0.675\\nBLEND\\n0.184\\n0.820\\n0.779\\n0.484\\n0.254\\n0.003\\n0.611\\nRUSE\\n0.213\\n0.823\\n0.788\\n0.487\\n0.250\\n0.109\\n0.672\\nPre-Trained\\nPBERT–Base\\n0.190\\n0.815\\n0.778\\n0.468\\n0.261\\n0.130\\n0.655\\nRBERT–Base\\n0.189\\n0.813\\n0.775\\n0.481\\n0.266\\n0.014\\n0.663\\nFBERT–Base\\n0.194\\n0.819\\n0.778\\n0.474\\n0.265\\n0.144\\n0.670\\nPBERT–Base (idf)\\n0.189\\n0.817\\n0.775\\n0.477\\n0.255\\n0.131\\n0.650\\nRBERT–Base (idf)\\n0.192\\n0.808\\n0.771\\n0.484\\n0.248\\n0.005\\n0.674\\nFBERT–Base (idf)\\n0.193\\n0.817\\n0.774\\n0.483\\n0.262\\n0.081\\n0.669\\nPBERT–Base–MRPC\\n0.190\\n0.701\\n0.766\\n0.487\\n0.254\\n0.126\\n0.653\\nRBERT–Base–MRPC\\n0.199\\n0.826\\n0.765\\n0.493\\n0.258\\n0.000\\n0.671\\nFBERT–Base–MRPC\\n0.197\\n0.824\\n0.767\\n0.491\\n0.260\\n0.147\\n0.668\\nPBERT–Base–MRPC (idf)\\n0.186\\n0.806\\n0.765\\n0.492\\n0.247\\n0.125\\n0.661\\nRBERT–Base–MRPC (idf)\\n0.200\\n0.823\\n0.760\\n0.495\\n0.258\\n0.000\\n0.680\\nFBERT–Base–MRPC (idf)\\n0.196\\n0.821\\n0.763\\n0.497\\n0.254\\n0.031\\n0.676\\nPBERT–Large\\n0.200\\n0.815\\n0.778\\n0.474\\n0.261\\n0.137\\n0.661\\nRBERT–Large\\n0.194\\n0.809\\n0.779\\n0.493\\n0.270\\n0.006\\n0.672\\nFBERT–Large\\n0.199\\n0.810\\n0.782\\n0.484\\n0.266\\n0.142\\n0.672\\nPBERT–Large (idf)\\n0.200\\n0.813\\n0.772\\n0.485\\n0.256\\n0.136\\n0.657\\nRBERT–Large (idf)\\n0.197\\n0.806\\n0.769\\n0.495\\n0.262\\n0.005\\n0.675\\nFBERT–Large (idf)\\n0.199\\n0.811\\n0.772\\n0.494\\n0.262\\n0.006\\n0.673\\nPRoBERTa–Base\\n0.173\\n0.675\\n0.757\\n0.502\\n0.258\\n0.126\\n0.654\\nRRoBERTa–Base\\n0.165\\n0.816\\n0.764\\n0.483\\n0.266\\n0.000\\n0.674\\nFRoBERTa–Base\\n0.173\\n0.820\\n0.764\\n0.498\\n0.262\\n0.090\\n0.669\\nPRoBERTa–Base (idf)\\n0.172\\n0.691\\n0.755\\n0.503\\n0.252\\n0.123\\n0.661\\nRRoBERTa–Base (idf)\\n0.172\\n0.809\\n0.758\\n0.490\\n0.268\\n0.000\\n0.678\\nFRoBERTa–Base (idf)\\n0.178\\n0.820\\n0.758\\n0.501\\n0.260\\n0.001\\n0.674\\nPRoBERTa–Large\\n0.174\\n0.704\\n0.765\\n0.497\\n0.255\\n0.140\\n0.663\\nRRoBERTa–Large\\n0.163\\n0.805\\n0.770\\n0.491\\n0.263\\n0.005\\n0.679\\nFRoBERTa–Large\\n0.175\\n0.825\\n0.770\\n0.499\\n0.262\\n0.143\\n0.675\\nPRoBERTa–Large (idf)\\n0.181\\n0.821\\n0.758\\n0.500\\n0.256\\n0.089\\n0.669\\nRRoBERTa–Large (idf)\\n0.165\\n0.787\\n0.763\\n0.495\\n0.270\\n0.000\\n0.684\\nFRoBERTa–Large (idf)\\n0.179\\n0.824\\n0.761\\n0.502\\n0.265\\n0.004\\n0.679\\nPRoBERTa–Large–MNLI\\n0.185\\n0.828\\n0.780\\n0.504\\n0.263\\n0.133\\n0.654\\nRRoBERTa–Large–MNLI\\n0.179\\n0.779\\n0.775\\n0.494\\n0.266\\n0.004\\n0.670\\nFRoBERTa–Large–MNLI\\n0.186\\n0.827\\n0.778\\n0.502\\n0.267\\n0.113\\n0.669\\nPRoBERTa–Large–MNLI (idf)\\n0.190\\n0.820\\n0.771\\n0.504\\n0.261\\n0.102\\n0.661\\nRRoBERTa–Large–MNLI (idf)\\n0.181\\n0.769\\n0.766\\n0.494\\n0.266\\n0.004\\n0.674\\nFRoBERTa–Large–MNLI (idf)\\n0.188\\n0.822\\n0.768\\n0.501\\n0.265\\n0.004\\n0.671\\nPXLNET–Base\\n0.186\\n0.771\\n0.762\\n0.496\\n0.247\\n0.153\\n0.658\\nRXLNET–Base\\n0.182\\n0.823\\n0.764\\n0.496\\n0.256\\n0.000\\n0.671\\nFXLNET–Base\\n0.186\\n0.824\\n0.765\\n0.499\\n0.253\\n0.049\\n0.673\\nPXLNET–Base (idf)\\n0.178\\n0.819\\n0.756\\n0.506\\n0.241\\n0.130\\n0.656\\nRXLNET–Base (idf)\\n0.183\\n0.817\\n0.754\\n0.501\\n0.256\\n0.000\\n0.673\\nFXLNET–Base (idf)\\n0.182\\n0.821\\n0.755\\n0.505\\n0.250\\n0.000\\n0.670\\nPXLNET–Large\\n0.195\\n0.721\\n0.767\\n0.493\\n0.152\\n0.144\\n0.661\\nRXLNET–Large\\n0.192\\n0.821\\n0.766\\n0.494\\n0.260\\n0.001\\n0.659\\nFXLNET–Large\\n0.196\\n0.824\\n0.773\\n0.496\\n0.261\\n0.155\\n0.675\\nPXLNET–Large (idf)\\n0.191\\n0.811\\n0.765\\n0.500\\n0.167\\n0.144\\n0.657\\nRXLNET–Large (idf)\\n0.196\\n0.815\\n0.762\\n0.495\\n0.259\\n0.000\\n0.673\\nFXLNET–Large (idf)\\n0.195\\n0.822\\n0.764\\n0.499\\n0.256\\n0.046\\n0.674\\nPXLM–En\\n0.192\\n0.796\\n0.779\\n0.486\\n0.255\\n0.131\\n0.665\\nRXLM–En\\n0.202\\n0.818\\n0.772\\n0.495\\n0.261\\n0.005\\n0.662\\nFXLM–En\\n0.199\\n0.827\\n0.778\\n0.491\\n0.262\\n0.086\\n0.674\\nPXLM–En (idf)\\n0.189\\n0.818\\n0.770\\n0.485\\n0.259\\n0.116\\n0.662\\nRXLM–En (idf)\\n0.202\\n0.812\\n0.761\\n0.490\\n0.250\\n0.003\\n0.668\\nFXLM–En (idf)\\n0.196\\n0.821\\n0.766\\n0.490\\n0.263\\n0.003\\n0.672\\nTable 23: Model selection accuracies (Hits@1) on to-English WMT18 hybrid systems. We report\\nthe average of 100K samples and the 0.95 confidence intervals are below 10−3. We bold the highest\\nnumbers for each language pair and direction.\\n36\\n\\n\\nPublished as a conference paper at ICLR 2020\\nSetting\\nMetric\\ncs-en\\nde-en\\net-en\\nfi-en\\nru-en\\ntr-en\\nzh-en\\nUnsupervised\\nBLEU\\n0.338\\n0.894\\n0.866\\n0.666\\n0.447\\n0.265\\n0.799\\nCDER\\n0.362\\n0.890\\n0.870\\n0.689\\n0.451\\n0.256\\n0.799\\nCHARACTER\\n0.349\\n0.854\\n0.814\\n0.690\\n0.429\\n0.254\\n0.739\\nITER\\n0.356\\n0.901\\n0.856\\n0.676\\n0.454\\n0.278\\n0.811\\nMETEOR++\\n0.369\\n0.895\\n0.798\\n0.662\\n0.470\\n0.174\\n0.757\\nNIST\\n0.338\\n0.894\\n0.857\\n0.672\\n0.446\\n0.323\\n0.803\\nPER\\n0.325\\n0.866\\n0.771\\n0.663\\n0.435\\n0.021\\n0.754\\nTER\\n0.342\\n0.885\\n0.873\\n0.673\\n0.447\\n0.063\\n0.792\\nUHH_TSKM\\n0.387\\n0.894\\n0.873\\n0.671\\n0.460\\n0.063\\n0.788\\nWER\\n0.353\\n0.876\\n0.868\\n0.674\\n0.443\\n0.034\\n0.790\\nYISI-0\\n0.344\\n0.881\\n0.834\\n0.681\\n0.452\\n0.275\\n0.776\\nYISI-1\\n0.352\\n0.896\\n0.864\\n0.671\\n0.470\\n0.285\\n0.765\\nYISI-1 SRL\\n0.351\\n0.901\\n0.871\\n0.682\\n0.464\\n0.086\\n0.770\\nSupervised\\nBEER\\n0.364\\n0.899\\n0.871\\n0.684\\n0.460\\n0.125\\n0.811\\nBLEND\\n0.382\\n0.904\\n0.880\\n0.681\\n0.473\\n0.077\\n0.767\\nRUSE\\n0.417\\n0.906\\n0.885\\n0.686\\n0.468\\n0.273\\n0.809\\nPre-Trained\\nPBERT–Base\\n0.386\\n0.901\\n0.880\\n0.674\\n0.481\\n0.318\\n0.793\\nRBERT–Base\\n0.383\\n0.899\\n0.877\\n0.683\\n0.486\\n0.100\\n0.804\\nFBERT–Base\\n0.388\\n0.903\\n0.879\\n0.678\\n0.484\\n0.331\\n0.808\\nPBERT–Base (idf)\\n0.390\\n0.902\\n0.877\\n0.681\\n0.475\\n0.318\\n0.786\\nRBERT–Base (idf)\\n0.390\\n0.896\\n0.874\\n0.686\\n0.475\\n0.077\\n0.811\\nFBERT–Base (idf)\\n0.393\\n0.902\\n0.876\\n0.685\\n0.483\\n0.225\\n0.806\\nPBERT–Base–MRPC\\n0.392\\n0.832\\n0.872\\n0.686\\n0.475\\n0.319\\n0.791\\nRBERT–Base–MRPC\\n0.397\\n0.908\\n0.870\\n0.691\\n0.478\\n0.025\\n0.811\\nFBERT–Base–MRPC\\n0.398\\n0.907\\n0.872\\n0.690\\n0.481\\n0.335\\n0.806\\nPBERT–Base–MRPC (idf)\\n0.392\\n0.896\\n0.870\\n0.689\\n0.467\\n0.316\\n0.797\\nRBERT–Base–MRPC (idf)\\n0.400\\n0.906\\n0.867\\n0.691\\n0.479\\n0.018\\n0.817\\nFBERT–Base–MRPC (idf)\\n0.400\\n0.905\\n0.869\\n0.693\\n0.475\\n0.097\\n0.812\\nPBERT–Large\\n0.398\\n0.901\\n0.880\\n0.678\\n0.481\\n0.327\\n0.799\\nRBERT–Large\\n0.391\\n0.897\\n0.879\\n0.690\\n0.490\\n0.085\\n0.810\\nFBERT–Large\\n0.397\\n0.898\\n0.882\\n0.684\\n0.486\\n0.328\\n0.810\\nPBERT–Large (idf)\\n0.398\\n0.900\\n0.875\\n0.685\\n0.475\\n0.323\\n0.794\\nRBERT–Large (idf)\\n0.395\\n0.895\\n0.873\\n0.692\\n0.488\\n0.080\\n0.813\\nFBERT–Large (idf)\\n0.398\\n0.899\\n0.875\\n0.691\\n0.482\\n0.086\\n0.810\\nPRoBERTa–Base\\n0.372\\n0.814\\n0.866\\n0.697\\n0.475\\n0.313\\n0.795\\nRRoBERTa–Base\\n0.366\\n0.902\\n0.870\\n0.683\\n0.483\\n0.026\\n0.813\\nFRoBERTa–Base\\n0.374\\n0.904\\n0.870\\n0.694\\n0.480\\n0.224\\n0.808\\nPRoBERTa–Base (idf)\\n0.373\\n0.825\\n0.865\\n0.697\\n0.470\\n0.303\\n0.802\\nRRoBERTa–Base (idf)\\n0.374\\n0.898\\n0.866\\n0.688\\n0.486\\n0.028\\n0.816\\nFRoBERTa–Base (idf)\\n0.380\\n0.904\\n0.866\\n0.696\\n0.479\\n0.037\\n0.812\\nPRoBERTa–Large\\n0.375\\n0.833\\n0.871\\n0.693\\n0.474\\n0.327\\n0.800\\nRRoBERTa–Large\\n0.366\\n0.895\\n0.874\\n0.689\\n0.480\\n0.039\\n0.816\\nFRoBERTa–Large\\n0.378\\n0.907\\n0.874\\n0.694\\n0.480\\n0.324\\n0.811\\nPRoBERTa–Large (idf)\\n0.384\\n0.905\\n0.866\\n0.694\\n0.475\\n0.220\\n0.806\\nRRoBERTa–Large (idf)\\n0.368\\n0.885\\n0.869\\n0.692\\n0.487\\n0.030\\n0.819\\nFRoBERTa–Large (idf)\\n0.382\\n0.907\\n0.868\\n0.696\\n0.484\\n0.048\\n0.815\\nPRoBERTa–Large–MNLI\\n0.383\\n0.909\\n0.880\\n0.698\\n0.480\\n0.323\\n0.795\\nRRoBERTa–Large–MNLI\\n0.378\\n0.880\\n0.877\\n0.692\\n0.481\\n0.078\\n0.811\\nFRoBERTa–Large–MNLI\\n0.385\\n0.909\\n0.879\\n0.697\\n0.484\\n0.286\\n0.809\\nPRoBERTa–Large–MNLI (idf)\\n0.389\\n0.905\\n0.874\\n0.698\\n0.478\\n0.268\\n0.803\\nRRoBERTa–Large–MNLI (idf)\\n0.380\\n0.874\\n0.870\\n0.691\\n0.483\\n0.079\\n0.814\\nFRoBERTa–Large–MNLI (idf)\\n0.387\\n0.906\\n0.872\\n0.696\\n0.482\\n0.082\\n0.811\\nPXLNET–Base\\n0.385\\n0.875\\n0.869\\n0.692\\n0.469\\n0.342\\n0.796\\nRXLNET–Base\\n0.381\\n0.907\\n0.869\\n0.693\\n0.477\\n0.026\\n0.809\\nFXLNET–Base\\n0.385\\n0.907\\n0.871\\n0.694\\n0.476\\n0.128\\n0.810\\nPXLNET–Base (idf)\\n0.381\\n0.904\\n0.864\\n0.699\\n0.464\\n0.289\\n0.794\\nRXLNET–Base (idf)\\n0.384\\n0.903\\n0.863\\n0.696\\n0.479\\n0.013\\n0.812\\nFXLNET–Base (idf)\\n0.384\\n0.905\\n0.864\\n0.699\\n0.472\\n0.032\\n0.809\\nPXLNET–Large\\n0.392\\n0.844\\n0.873\\n0.689\\n0.367\\n0.338\\n0.799\\nRXLNET–Large\\n0.389\\n0.905\\n0.871\\n0.690\\n0.482\\n0.031\\n0.800\\nFXLNET–Large\\n0.393\\n0.907\\n0.876\\n0.691\\n0.483\\n0.348\\n0.812\\nPXLNET–Large (idf)\\n0.393\\n0.899\\n0.870\\n0.694\\n0.387\\n0.333\\n0.794\\nRXLNET–Large (idf)\\n0.395\\n0.901\\n0.868\\n0.690\\n0.483\\n0.023\\n0.810\\nFXLNET–Large (idf)\\n0.396\\n0.906\\n0.870\\n0.693\\n0.478\\n0.128\\n0.811\\nPXLM–En\\n0.394\\n0.891\\n0.880\\n0.685\\n0.476\\n0.322\\n0.802\\nRXLM–En\\n0.401\\n0.903\\n0.875\\n0.692\\n0.483\\n0.082\\n0.803\\nFXLM–En\\n0.400\\n0.909\\n0.878\\n0.689\\n0.483\\n0.234\\n0.811\\nPXLM–En (idf)\\n0.391\\n0.903\\n0.874\\n0.684\\n0.480\\n0.293\\n0.797\\nRXLM–En (idf)\\n0.402\\n0.900\\n0.868\\n0.688\\n0.477\\n0.068\\n0.806\\nFXLM–En (idf)\\n0.398\\n0.905\\n0.871\\n0.688\\n0.487\\n0.079\\n0.809\\nTable 24: Mean Reciprocal Rank (MRR) of the top metric-rated system on to-English WMT18\\nhybrid systems. We report the average of 100K samples and the 0.95 confidence intervals are below\\n10−3. We bold the highest numbers for each language pair and direction.\\n37\\n\\n\\nPublished as a conference paper at ICLR 2020\\nSetting\\nMetric\\ncs-en\\nde-en\\net-en\\nfi-en\\nru-en\\ntr-en\\nzh-en\\nUnsupervised\\nBLEU\\n3.85\\n0.45\\n1.01\\n2.17\\n2.34\\n4.48\\n3.19\\nCDER\\n3.88\\n0.43\\n0.87\\n1.33\\n2.30\\n4.58\\n3.43\\nCHARACTER\\n3.77\\n0.49\\n0.94\\n2.07\\n2.25\\n4.07\\n3.37\\nITER\\n3.55\\n0.46\\n1.25\\n1.43\\n4.65\\n3.11\\n2.92\\nMETEOR++\\n3.70\\n0.41\\n0.69\\n1.13\\n2.28\\n1.40\\n3.50\\nNIST\\n3.93\\n0.49\\n1.10\\n1.19\\n2.36\\n1.42\\n3.92\\nPER\\n2.02\\n0.46\\n1.71\\n1.49\\n2.25\\n4.22\\n3.20\\nTER\\n3.86\\n0.43\\n1.14\\n1.14\\n4.34\\n5.18\\n3.82\\nUHH_TSKM\\n3.98\\n0.40\\n1.27\\n1.10\\n2.23\\n4.26\\n3.47\\nWER\\n3.85\\n0.44\\n1.48\\n1.18\\n4.87\\n5.96\\n3.72\\nYISI-0\\n3.81\\n0.48\\n0.72\\n1.20\\n1.75\\n1.40\\n3.44\\nYISI-1\\n3.88\\n0.44\\n0.65\\n1.13\\n2.17\\n1.32\\n3.40\\nYISI-1 SRL\\n3.67\\n0.41\\n0.64\\n1.20\\n2.15\\n1.31\\n3.55\\nSupervised\\nBEER\\n3.82\\n0.41\\n0.79\\n1.08\\n1.92\\n1.96\\n3.43\\nBLEND\\n3.77\\n0.41\\n0.66\\n1.09\\n2.21\\n1.28\\n3.46\\nRUSE\\n3.13\\n0.32\\n0.64\\n1.03\\n1.51\\n1.94\\n3.15\\nPre-Trained\\nPBERT–Base\\n3.97\\n0.36\\n0.72\\n1.16\\n2.20\\n1.25\\n3.26\\nRBERT–Base\\n1.51\\n0.43\\n0.60\\n1.65\\n1.33\\n1.34\\n3.50\\nFBERT–Base\\n3.70\\n0.36\\n0.59\\n1.08\\n1.92\\n1.27\\n3.38\\nPBERT–Base (idf)\\n3.94\\n0.36\\n0.64\\n1.18\\n2.06\\n2.55\\n3.54\\nRBERT–Base (idf)\\n1.54\\n0.43\\n0.63\\n1.87\\n1.12\\n5.96\\n3.38\\nFBERT–Base (idf)\\n2.75\\n0.39\\n0.60\\n1.10\\n1.38\\n1.26\\n3.51\\nPBERT–Base–MRPC\\n4.02\\n0.35\\n0.74\\n1.15\\n1.09\\n3.33\\n3.06\\nRBERT–Base–MRPC\\n2.66\\n0.43\\n0.62\\n1.75\\n1.10\\n5.64\\n3.34\\nFBERT–Base–MRPC\\n3.89\\n0.36\\n0.60\\n1.09\\n1.08\\n3.82\\n3.23\\nPBERT–Base–MRPC (idf)\\n4.02\\n0.35\\n0.67\\n1.18\\n1.48\\n3.30\\n3.49\\nRBERT–Base–MRPC (idf)\\n1.63\\n0.43\\n0.65\\n1.93\\n1.13\\n7.26\\n3.13\\nFBERT–Base–MRPC (idf)\\n3.86\\n0.38\\n0.61\\n1.11\\n1.14\\n4.24\\n3.28\\nPBERT–Large\\n3.82\\n0.34\\n0.66\\n1.12\\n2.10\\n1.31\\n3.60\\nRBERT–Large\\n1.49\\n0.40\\n0.59\\n1.56\\n1.17\\n1.35\\n3.61\\nFBERT–Large\\n1.71\\n0.35\\n0.58\\n1.08\\n1.65\\n1.29\\n3.60\\nPBERT–Large (idf)\\n3.74\\n0.35\\n0.65\\n1.12\\n1.90\\n1.98\\n3.77\\nRBERT–Large (idf)\\n1.51\\n0.42\\n0.62\\n1.86\\n1.10\\n5.84\\n3.21\\nFBERT–Large (idf)\\n1.49\\n0.38\\n0.60\\n1.17\\n1.24\\n1.96\\n3.53\\nPRoBERTa–Base\\n3.89\\n0.37\\n0.75\\n1.18\\n1.07\\n3.45\\n2.62\\nRRoBERTa–Base\\n1.92\\n0.39\\n0.64\\n1.57\\n1.11\\n5.75\\n3.13\\nFRoBERTa–Base\\n3.56\\n0.37\\n0.59\\n1.10\\n1.08\\n3.79\\n2.90\\nPRoBERTa–Base (idf)\\n3.89\\n0.38\\n0.67\\n1.20\\n1.30\\n3.27\\n3.47\\nRRoBERTa–Base (idf)\\n1.61\\n0.42\\n0.67\\n1.65\\n1.14\\n6.55\\n2.95\\nFRoBERTa–Base (idf)\\n3.18\\n0.38\\n0.60\\n1.11\\n1.13\\n6.54\\n3.11\\nPRoBERTa–Large\\n3.64\\n0.36\\n0.71\\n1.10\\n1.03\\n2.69\\n2.57\\nRRoBERTa–Large\\n1.60\\n0.37\\n0.64\\n1.51\\n1.09\\n3.91\\n3.27\\nFRoBERTa–Large\\n2.38\\n0.35\\n0.58\\n1.06\\n1.05\\n3.57\\n2.95\\nPRoBERTa–Large (idf)\\n2.70\\n0.36\\n0.69\\n1.13\\n1.08\\n3.18\\n2.89\\nRRoBERTa–Large (idf)\\n1.55\\n0.39\\n0.66\\n1.59\\n1.10\\n6.66\\n3.18\\nFRoBERTa–Large (idf)\\n1.68\\n0.37\\n0.59\\n1.08\\n1.08\\n5.58\\n2.91\\nPRoBERTa–Large–MNLI\\n2.14\\n0.35\\n0.61\\n1.07\\n1.09\\n1.21\\n3.35\\nRRoBERTa–Large–MNLI\\n1.45\\n0.37\\n0.64\\n1.49\\n1.10\\n4.42\\n3.55\\nFRoBERTa–Large–MNLI\\n1.42\\n0.35\\n0.59\\n1.07\\n1.07\\n1.27\\n3.41\\nPRoBERTa–Large–MNLI (idf)\\n1.55\\n0.35\\n0.60\\n1.08\\n1.12\\n1.54\\n3.87\\nRRoBERTa–Large–MNLI (idf)\\n1.45\\n0.39\\n0.64\\n1.65\\n1.09\\n5.89\\n3.32\\nFRoBERTa–Large–MNLI (idf)\\n1.42\\n0.36\\n0.60\\n1.10\\n1.08\\n3.80\\n3.45\\nPXLNET–Base\\n3.90\\n0.37\\n0.68\\n1.07\\n1.16\\n2.47\\n2.91\\nRXLNET–Base\\n1.71\\n0.45\\n0.72\\n1.58\\n1.07\\n6.29\\n3.36\\nFXLNET–Base\\n3.78\\n0.39\\n0.62\\n1.05\\n1.07\\n3.60\\n3.20\\nPXLNET–Base (idf)\\n3.90\\n0.46\\n0.65\\n1.08\\n2.93\\n3.30\\n3.39\\nRXLNET–Base (idf)\\n1.51\\n0.45\\n0.82\\n1.78\\n1.12\\n10.77\\n3.13\\nFXLNET–Base (idf)\\n3.67\\n0.42\\n0.66\\n1.11\\n1.22\\n7.13\\n3.23\\nPXLNET–Large\\n3.94\\n0.37\\n0.71\\n1.10\\n21.10\\n1.85\\n2.90\\nRXLNET–Large\\n2.23\\n0.41\\n0.69\\n1.34\\n1.07\\n4.46\\n3.40\\nFXLNET–Large\\n3.84\\n0.36\\n0.60\\n1.03\\n1.07\\n3.38\\n3.22\\nPXLNET–Large (idf)\\n3.92\\n0.41\\n0.64\\n1.12\\n21.10\\n3.24\\n3.37\\nRXLNET–Large (idf)\\n1.60\\n0.43\\n0.78\\n1.70\\n1.09\\n6.13\\n3.20\\nFXLNET–Large (idf)\\n3.80\\n0.38\\n0.63\\n1.06\\n1.09\\n3.72\\n3.25\\nPXLM–En\\n3.88\\n0.33\\n0.75\\n1.16\\n2.16\\n1.28\\n3.29\\nRXLM–En\\n1.98\\n0.41\\n0.60\\n1.41\\n1.21\\n3.30\\n3.47\\nFXLM–En\\n3.78\\n0.36\\n0.61\\n1.09\\n1.71\\n1.30\\n3.40\\nPXLM–En (idf)\\n3.84\\n0.36\\n0.69\\n1.17\\n1.86\\n1.33\\n3.47\\nRXLM–En (idf)\\n1.70\\n0.42\\n0.63\\n1.55\\n1.11\\n5.87\\n3.36\\nFXLM–En (idf)\\n3.72\\n0.40\\n0.62\\n1.14\\n1.32\\n4.15\\n3.43\\nTable 25: Absolute Difference (×100) of the top metric-rated and the top human-rated system on to-\\nEnglish WMT18 hybrid systems. Smaller difference signify higher agreement with human scores.\\nWe report the average of 100K samples and the 0.95 confidence intervals are below 10−3. We bold\\nthe lowest numbers for each language pair and direction.\\n38\\n\\n\\nPublished as a conference paper at ICLR 2020\\nSetting\\nMetric\\nen-cs\\nen-de\\nen-et\\nen-fi\\nen-ru\\nen-tr\\nen-zh\\nUnsupervised\\nBLEU\\n0.151\\n0.611\\n0.617\\n0.087\\n0.519\\n0.029\\n0.515\\nCDER\\n0.163\\n0.663\\n0.731\\n0.081\\n0.541\\n0.032\\n0.552\\nCHARACTER\\n0.135\\n0.737\\n0.639\\n0.492\\n0.543\\n0.027\\n0.667\\nITER\\n0.000\\n0.691\\n0.734\\n0.112\\n0.534\\n0.031\\n–\\nMETEOR++\\n–\\n–\\n–\\n–\\n–\\n–\\n–\\nNIST\\n0.182\\n0.662\\n0.549\\n0.083\\n0.537\\n0.033\\n0.553\\nPER\\n0.179\\n0.555\\n0.454\\n0.062\\n0.535\\n0.032\\n0.539\\nTER\\n0.175\\n0.657\\n0.550\\n0.065\\n0.545\\n0.029\\n0.551\\nUHH_TSKM\\n–\\n–\\n–\\n–\\n–\\n–\\n–\\nWER\\n0.155\\n0.643\\n0.552\\n0.067\\n0.538\\n0.029\\n0.546\\nYISI-0\\n0.154\\n0.674\\n0.622\\n0.356\\n0.523\\n0.383\\n0.600\\nYISI-1\\n0.178\\n0.670\\n0.674\\n0.230\\n0.548\\n0.396\\n0.595\\nYISI-1 SRL\\n–\\n0.708\\n–\\n–\\n–\\n–\\n0.537\\nSupervised\\nBEER\\n0.174\\n0.670\\n0.662\\n0.113\\n0.555\\n0.296\\n0.531\\nBLEND\\n–\\n–\\n–\\n–\\n0.559\\n–\\n–\\nRUSE\\n–\\n–\\n–\\n–\\n–\\n–\\n–\\nPre-Trained\\nPBERT–Multi\\n0.181\\n0.665\\n0.771\\n0.077\\n0.550\\n0.373\\n0.550\\nRBERT–Multi\\n0.184\\n0.728\\n0.722\\n0.146\\n0.544\\n0.031\\n0.657\\nFBERT–Multi\\n0.185\\n0.703\\n0.764\\n0.081\\n0.548\\n0.032\\n0.629\\nPBERT–Multi (idf)\\n0.175\\n0.713\\n0.769\\n0.080\\n0.542\\n0.031\\n0.549\\nRBERT–Multi (idf)\\n0.177\\n0.725\\n0.752\\n0.178\\n0.538\\n0.031\\n0.628\\nFBERT–Multi (idf)\\n0.178\\n0.721\\n0.766\\n0.081\\n0.543\\n0.030\\n0.594\\nPXLM–100\\n0.175\\n0.669\\n0.748\\n0.079\\n0.550\\n0.314\\n0.582\\nRXLM–100\\n0.195\\n0.671\\n0.770\\n0.222\\n0.555\\n0.034\\n0.658\\nFXLM–100\\n0.187\\n0.670\\n0.775\\n0.099\\n0.552\\n0.034\\n0.615\\nPXLM–100 (idf)\\n0.163\\n0.664\\n0.750\\n0.091\\n0.550\\n0.288\\n0.578\\nRXLM–100 (idf)\\n0.191\\n0.681\\n0.770\\n0.231\\n0.548\\n0.033\\n0.645\\nFXLM–100 (idf)\\n0.180\\n0.672\\n0.774\\n0.127\\n0.550\\n0.033\\n0.616\\nTable 26: Model selection accuracies (Hits@1) on to-English WMT18 hybrid systems. We report\\nthe average of 100K samples and the 0.95 confidence intervals are below 10−3. We bold the highest\\nnumbers for each language pair and direction.\\n39\\n\\n\\nPublished as a conference paper at ICLR 2020\\nSetting\\nMetric\\nen-cs\\nen-de\\nen-et\\nen-fi\\nen-ru\\nen-tr\\nen-zh\\nUnsupervised\\nBLEU\\n0.363\\n0.764\\n0.766\\n0.323\\n0.714\\n0.205\\n0.666\\nCDER\\n0.371\\n0.803\\n0.851\\n0.319\\n0.729\\n0.210\\n0.700\\nCHARACTER\\n0.346\\n0.853\\n0.781\\n0.667\\n0.732\\n0.205\\n0.809\\nITER\\n0.044\\n0.825\\n0.853\\n0.365\\n0.717\\n0.210\\n–\\nMETEOR++\\n–\\n–\\n–\\n–\\n–\\n–\\n–\\nNIST\\n0.393\\n0.803\\n0.710\\n0.326\\n0.726\\n0.211\\n0.698\\nPER\\n0.387\\n0.719\\n0.624\\n0.301\\n0.725\\n0.211\\n0.678\\nTER\\n0.384\\n0.798\\n0.708\\n0.305\\n0.733\\n0.209\\n0.695\\nUHH_TSKM\\n–\\n–\\n–\\n–\\n–\\n–\\n–\\nWER\\n0.367\\n0.787\\n0.710\\n0.308\\n0.728\\n0.209\\n0.696\\nYISI-0\\n0.370\\n0.811\\n0.775\\n0.553\\n0.715\\n0.602\\n0.753\\nYISI-1\\n0.390\\n0.808\\n0.811\\n0.439\\n0.735\\n0.612\\n0.750\\nYISI-1 SRL\\n–\\n0.835\\n–\\n–\\n–\\n–\\n0.691\\nSupervised\\nBEER\\n0.388\\n0.808\\n0.804\\n0.353\\n0.739\\n0.507\\n0.683\\nBLEND\\n–\\n–\\n–\\n–\\n0.742\\n–\\n–\\nRUSE\\n–\\n–\\n–\\n–\\n–\\n–\\n–\\nPre-Trained\\nPBERT–Multi\\n0.395\\n0.805\\n0.876\\n0.314\\n0.736\\n0.586\\n0.694\\nRBERT–Multi\\n0.401\\n0.849\\n0.844\\n0.368\\n0.732\\n0.212\\n0.802\\nFBERT–Multi\\n0.400\\n0.832\\n0.872\\n0.317\\n0.735\\n0.214\\n0.775\\nPBERT–Multi (idf)\\n0.390\\n0.839\\n0.875\\n0.320\\n0.730\\n0.213\\n0.691\\nRBERT–Multi (idf)\\n0.395\\n0.847\\n0.864\\n0.398\\n0.727\\n0.212\\n0.776\\nFBERT–Multi (idf)\\n0.395\\n0.844\\n0.873\\n0.319\\n0.730\\n0.212\\n0.739\\nPXLM–100\\n0.391\\n0.808\\n0.862\\n0.316\\n0.735\\n0.522\\n0.733\\nRXLM–100\\n0.413\\n0.809\\n0.876\\n0.435\\n0.738\\n0.216\\n0.803\\nFXLM–100\\n0.404\\n0.809\\n0.878\\n0.333\\n0.737\\n0.216\\n0.767\\nPXLM–100 (idf)\\n0.377\\n0.805\\n0.863\\n0.326\\n0.735\\n0.497\\n0.729\\nRXLM–100 (idf)\\n0.409\\n0.816\\n0.876\\n0.444\\n0.733\\n0.214\\n0.793\\nFXLM–100 (idf)\\n0.396\\n0.810\\n0.878\\n0.355\\n0.735\\n0.214\\n0.767\\nTable 27: Mean Reciprocal Rank (MRR) of the top metric-rated system on to-English WMT18\\nhybrid systems. We report the average of 100K samples and the 0.95 confidence intervals are below\\n10−3. We bold the highest numbers for each language pair and direction.\\n40\\n\\n\\nPublished as a conference paper at ICLR 2020\\nSetting\\nMetric\\nen-cs\\nen-de\\nen-et\\nen-fi\\nen-ru\\nen-tr\\nen-zh\\nUnsupervised\\nBLEU\\n1.26\\n6.36\\n2.59\\n0.92\\n0.76\\n9.40\\n3.01\\nCDER\\n1.25\\n6.70\\n1.90\\n1.41\\n0.87\\n9.37\\n1.75\\nCHARACTER\\n1.23\\n6.90\\n2.19\\n4.35\\n0.93\\n5.22\\n1.64\\nITER\\n1.25\\n9.14\\n2.52\\n1.52\\n1.35\\n7.33\\n–\\nMETEOR++\\n–\\n–\\n–\\n–\\n–\\n–\\n–\\nNIST\\n1.24\\n5.28\\n2.55\\n1.02\\n0.75\\n8.82\\n3.34\\nPER\\n1.25\\n6.62\\n4.92\\n7.43\\n0.68\\n9.76\\n2.31\\nTER\\n1.21\\n6.02\\n4.34\\n2.17\\n0.73\\n8.80\\n1.43\\nUHH_TSKM\\n–\\n–\\n–\\n–\\n–\\n–\\n–\\nWER\\n1.22\\n6.15\\n4.19\\n2.43\\n0.72\\n9.28\\n1.49\\nYISI-0\\n1.25\\n6.62\\n1.53\\n1.46\\n0.75\\n3.47\\n2.87\\nYISI-1\\n1.22\\n6.27\\n1.21\\n1.13\\n0.71\\n3.51\\n3.33\\nYISI-1 SRL\\n–\\n6.57\\n–\\n–\\n–\\n–\\n3.71\\nSupervised\\nBEER\\n1.21\\n5.96\\n1.84\\n0.77\\n0.74\\n3.36\\n1.96\\nBLEND\\n–\\n–\\n–\\n–\\n0.71\\n–\\n–\\nRUSE\\n–\\n–\\n–\\n–\\n–\\n–\\n–\\nPre-Trained\\nPBERT–Multi\\n1.17\\n3.27\\n1.38\\n1.24\\n0.75\\n4.14\\n2.08\\nRBERT–Multi\\n1.16\\n6.68\\n0.77\\n0.94\\n0.68\\n3.22\\n1.31\\nFBERT–Multi\\n1.15\\n5.17\\n0.90\\n0.98\\n0.71\\n3.26\\n1.62\\nPBERT–Multi (idf)\\n1.14\\n3.82\\n1.66\\n1.27\\n0.76\\n4.57\\n2.04\\nRBERT–Multi (idf)\\n1.15\\n6.97\\n0.83\\n3.65\\n0.68\\n3.32\\n1.37\\nFBERT–Multi (idf)\\n1.14\\n5.63\\n1.13\\n1.19\\n0.71\\n3.38\\n1.58\\nPXLM–100\\n1.22\\n6.30\\n1.14\\n0.79\\n0.74\\n3.73\\n2.21\\nRXLM–100\\n1.18\\n6.89\\n0.76\\n0.77\\n0.66\\n3.26\\n1.68\\nFXLM–100\\n1.19\\n6.44\\n0.82\\n0.76\\n0.69\\n3.21\\n1.57\\nPXLM–100 (idf)\\n1.21\\n6.61\\n1.07\\n0.78\\n0.72\\n5.59\\n2.02\\nRXLM–100 (idf)\\n1.19\\n7.07\\n0.77\\n0.77\\n0.66\\n3.33\\n1.60\\nFXLM–100 (idf)\\n1.20\\n6.57\\n0.86\\n0.76\\n0.68\\n3.28\\n1.56\\nTable 28: Absolute Difference (×100) of the top metric-rated and the top human-rated system on to-\\nEnglish WMT18 hybrid systems. Smaller difference indicate higher agreement with human scores.\\nWe report the average of 100K samples and the 0.95 confidence intervals are below 10−3. We bold\\nthe lowest numbers for each language pair and direction.\\n41\\n\\n\\nPublished as a conference paper at ICLR 2020\\nMetric\\nM1\\nM2\\nBLEU-1\\n0.124∗\\n0.135∗\\nBLEU-2\\n0.037∗\\n0.048∗\\nBLEU-3\\n0.004∗\\n0.016∗\\nBLEU-4\\n-0.019∗\\n-0.005∗\\nMETEOR\\n0.606∗\\n0.594∗\\nROUGE-L\\n0.090∗\\n0.096∗\\nCIDER\\n0.438∗\\n0.440∗\\nSPICE\\n0.759∗\\n0.750∗\\nLEIC\\n0.939∗\\n0.949∗\\nBEER\\n0.491\\n0.562\\nEED\\n0.545\\n0.599\\nCHRF++\\n0.702\\n0.729\\nCHARACTER\\n0.800\\n0.801\\nPBERT–Base\\n0.313\\n0.344\\nRBERT–Base\\n0.679\\n0.622\\nFBERT–Base\\n0.531\\n0.519\\nPBERT–Base (idf)\\n0.243\\n0.286\\nRBERT–Base (idf)\\n0.834\\n0.783\\nFBERT–Base (idf)\\n0.579\\n0.581\\nPBERT–Base–MRPC\\n0.252\\n0.331\\nRBERT–Base–MRPC\\n0.644\\n0.641\\nFBERT–Base–MRPC\\n0.470\\n0.512\\nPBERT–Base–MRPC (idf)\\n0.264\\n0.300\\nRBERT–Base–MRPC (idf)\\n0.794\\n0.767\\nFBERT–Base–MRPC (idf)\\n0.575\\n0.583\\nPBERT–Large\\n0.454\\n0.486\\nRBERT–Large\\n0.756\\n0.697\\nFBERT–Large\\n0.649\\n0.634\\nPBERT–Large (idf)\\n0.327\\n0.372\\nRBERT–Large (idf)\\n0.873\\n0.821\\nFBERT–Large (idf)\\n0.645\\n0.647\\nPRoBERTa–Base\\n-0.223\\n-0.179\\nRRoBERTa–Base\\n0.827\\n0.800\\nFRoBERTa–Base\\n0.176\\n0.191\\nPRoBERTa–Base (idf)\\n-0.256\\n-0.267\\nRRoBERTa–Base (idf)\\n0.901\\n0.869\\nFRoBERTa–Base (idf)\\n0.188\\n0.157\\nPRoBERTa–Large\\n-0.105\\n-0.041\\nRRoBERTa–Large\\n0.888\\n0.863\\nFRoBERTa–Large\\n0.322\\n0.350\\nPRoBERTa–Large (idf)\\n0.063\\n-0.011\\nRRoBERTa–Large (idf)\\n0.917\\n0.889\\nFRoBERTa–Large (idf)\\n0.519\\n0.453\\nPRoBERTa–Large–MNLI\\n0.129\\n0.208\\nRRoBERTa–Large–MNLI\\n0.820\\n0.823\\nFRoBERTa–Large–MNLI\\n0.546\\n0.592\\nPRoBERTa–Large–MNLI (idf)\\n0.081\\n0.099\\nRRoBERTa–Large–MNLI (idf)\\n0.906\\n0.875\\nFRoBERTa–Large–MNLI (idf)\\n0.605\\n0.596\\nPXLNet–Base\\n-0.046\\n0.080\\nRXLNet–Base\\n0.409\\n0.506\\nFXLNet–Base\\n0.146\\n0.265\\nPXLNet–Base (idf)\\n0.006\\n0.145\\nRXLNet–Base (idf)\\n0.655\\n0.720\\nFXLNet–Base (idf)\\n0.270\\n0.391\\nPXLNet–Large\\n-0.188\\n-0.115\\nRXLNet–Large\\n0.178\\n0.195\\nFXLNet–Large\\n-0.014\\n0.036\\nPXLNet–Large (idf)\\n-0.186\\n-0.072\\nRXLNet–Large (idf)\\n0.554\\n0.555\\nFXLNet–Large (idf)\\n0.151\\n0.234\\nPXLM–En\\n0.230\\n0.220\\nRXLM–En\\n0.333\\n0.263\\nFXLM–En\\n0.297\\n0.243\\nPXLM–En (idf)\\n0.266\\n0.275\\nRXLM–En (idf)\\n0.700\\n0.640\\nFXLM–En (idf)\\n0.499\\n0.470\\nTable 29: Pearson correlation on the 2015 COCO Captioning Challenge. The M1 and M2 measures\\nare described in Section 4. We bold the best correlating task-specific and task-agnostic metrics in\\neach setting LEIC uses images as additional inputs. Numbers with ∗are cited from Cui et al. (2018).\\n42\\n\\n\\nPublished as a conference paper at ICLR 2020\\nType\\nMethod\\nQQP\\nPAWSQQP\\nTrained on QQP (supervised)\\nDecAtt\\n0.939*\\n0.263\\nDIIN\\n0.952*\\n0.324\\nBERT\\n0.963*\\n0.351\\nTrained on QQP + PAWSQQP (supervised)\\nDecAtt\\n-\\n0.511\\nDIIN\\n-\\n0.778\\nBERT\\n-\\n0.831\\nMetric (Not trained on QQP or PAWSQQP)\\nBLEU-1\\n0.737\\n0.402\\nBLEU-2\\n0.720\\n0.548\\nBLEU-3\\n0.712\\n0.527\\nBLEU-4\\n0.707\\n0.527\\nMETEOR\\n0.755\\n0.532\\nROUGE-L\\n0.740\\n0.536\\nCHRF++\\n0.577\\n0.608\\nBEER\\n0.741\\n0.564\\nEED\\n0.743\\n0.611\\nCHARACTER\\n0.698\\n0.650\\nPBERT–Base\\n0.750\\n0.654\\nRBERT–Base\\n0.739\\n0.655\\nFBERT–Base\\n0.755\\n0.654\\nPBERT–Base (idf)\\n0.766\\n0.665\\nRBERT–Base (idf)\\n0.752\\n0.665\\nFBERT–Base (idf)\\n0.770\\n0.664\\nPBERT–Base–MRPC\\n0.742\\n0.615\\nRBERT–Base–MRPC\\n0.729\\n0.617\\nFBERT–Base–MRPC\\n0.746\\n0.614\\nPBERT–Base–MRPC (idf)\\n0.752\\n0.618\\nRBERT–Base–MRPC (idf)\\n0.737\\n0.619\\nFBERT–Base–MRPC (idf)\\n0.756\\n0.617\\nPBERT–Large\\n0.752\\n0.706\\nRBERT–Large\\n0.740\\n0.710\\nFBERT–Large\\n0.756\\n0.707\\nPBERT–Large (idf)\\n0.766\\n0.713\\nRBERT–Large (idf)\\n0.751\\n0.718\\nFBERT–Large (idf)\\n0.769\\n0.714\\nPRoBERTa–Base\\n0.746\\n0.657\\nRRoBERTa–Base\\n0.736\\n0.656\\nFRoBERTa–Base\\n0.751\\n0.654\\nPRoBERTa–Base (idf)\\n0.760\\n0.666\\nRRoBERTa–Base (idf)\\n0.745\\n0.666\\nFRoBERTa–Base (idf)\\n0.765\\n0.664\\nPRoBERTa–Large\\n0.757\\n0.687\\nRRoBERTa–Large\\n0.744\\n0.685\\nFRoBERTa–Large\\n0.761\\n0.685\\nPRoBERTa–Large (idf)\\n0.773\\n0.691\\nRRoBERTa–Large (idf)\\n0.757\\n0.697\\nFRoBERTa–Large (idf)\\n0.777\\n0.693\\nPRoBERTa–Large–MNLI\\n0.763\\n0.767\\nRRoBERTa–Large–MNLI\\n0.750\\n0.772\\nFRoBERTa–Large–MNLI\\n0.766\\n0.770\\nPRoBERTa–Large–MNLI (idf)\\n0.783\\n0.756\\nRRoBERTa–Large–MNLI (idf)\\n0.767\\n0.764\\nFRoBERTa–Large–MNLI (idf)\\n0.784\\n0.759\\nPXLNet–Base\\n0.737\\n0.603\\nRXLNet–Base\\n0.731\\n0.607\\nFXLNet–Base\\n0.739\\n0.605\\nPXLNet–Base (idf)\\n0.751\\n0.625\\nRXLNet–Base (idf)\\n0.743\\n0.630\\nFXLNet–Base (idf)\\n0.751\\n0.626\\nPXLNet–Large\\n0.742\\n0.593\\nRXLNet–Large\\n0.734\\n0.598\\nFXLNet–Large\\n0.744\\n0.596\\nPXLNet–Large (idf)\\n0.759\\n0.604\\nRXLNet–Large (idf)\\n0.749\\n0.610\\nFXLNet–Large (idf)\\n0.760\\n0.606\\nPXLM–En\\n0.734\\n0.600\\nRXLM–En\\n0.725\\n0.604\\nFXLM–En\\n0.737\\n0.602\\nPXLM–En (idf)\\n0.757\\n0.596\\nRXLM–En (idf)\\n0.745\\n0.603\\nFXLM–En (idf)\\n0.759\\n0.600\\nTable 30: Area under ROC curve (AUC) on QQP and PAWSQQP datasets. The scores of trained\\nDecATT (Parikh et al., 2016), DIIN (Gong et al., 2018), and fine-tuned BERT are reported by Zhang\\net al. (2019). We bold the best task-specific and task-agnostic metrics. Numbers with ∗are scores\\non the held-out test set of QQP.\\n43\",\"difficulty\":\"easy\",\"domain\":\"Single-Document QA\",\"length\":\"short\",\"question\":\"Which of the following methods is used in this article to calculate the precision similarity of two documents?\",\"sub_domain\":\"Academic\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/zai-org/LongBench-v2","history":"initial import","indexing_mode":"noindex","subproblems":[],"grids":[]}