# LongBench v2 / 66ebc0c95a08c7b9b35de7f3

task_id: fff27990-4fb3-56b8-a7e6-d49edbc0f217
task_key: train--66ebc0c95a08c7b9b35de7f3
task_revision_id: 3

{"choice_A":"2%","choice_B":"2.5%","choice_C":"4.5%","choice_D":"5%","context":"LLaMA: Open and Efficient Foundation Language Models\nHugo Touvron∗\n, Thibaut Lavril∗\n, Gautier Izacard∗\n, Xavier Martinet\nMarie-Anne Lachaux, Timothee Lacroix, Baptiste Rozière, Naman Goyal\nEric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin\nEdouard Grave∗\n, Guillaume Lample∗\nMeta AI\nAbstract\nWe introduce LLaMA, a collection of founda-\ntion language models ranging from 7B to 65B\nparameters. We train our models on trillions\nof tokens, and show that it is possible to train\nstate-of-the-art models using publicly available\ndatasets exclusively,\nwithout resorting to\nproprietary and inaccessible datasets.\nIn\nparticular, LLaMA-13B outperforms GPT-3\n(175B) on most benchmarks, and LLaMA-65B\nis competitive with the best models, Chinchilla-\n70B and PaLM-540B. We release all our\nmodels to the research community1.\n1\nIntroduction\nLarge Languages Models (LLMs) trained on mas-\nsive corpora of texts have shown their ability to per-\nform new tasks from textual instructions or from a\nfew examples (Brown et al., 2020). These few-shot\nproperties first appeared when scaling models to a\nsufficient size (Kaplan et al., 2020), resulting in a\nline of work that focuses on further scaling these\nmodels (Chowdhery et al., 2022; Rae et al., 2021).\nThese efforts are based on the assumption that\nmore parameters will lead to better performance.\nHowever, recent work from Hoffmann et al. (2022)\nshows that, for a given compute budget, the best\nperformances are not achieved by the largest mod-\nels, but by smaller models trained on more data.\nThe objective of the scaling laws from Hoff-\nmann et al. (2022) is to determine how to best\nscale the dataset and model sizes for a particular\ntraining compute budget. However, this objective\ndisregards the inference budget, which becomes\ncritical when serving a language model at scale.\nIn this context, given a target level of performance,\nthe preferred model is not the fastest to train but the\nfastest at inference, and although it may be cheaper\nto train a large model to reach a certain level of\n∗Equal contribution.\nCorrespondence: {htouvron,\nthibautlav,gizacard,egrave,glample}@meta.com\n1https://github.com/facebookresearch/llama\nperformance, a smaller one trained longer will\nultimately be cheaper at inference. For instance,\nalthough Hoffmann et al. (2022) recommends\ntraining a 10B model on 200B tokens, we find\nthat the performance of a 7B model continues to\nimprove even after 1T tokens.\nThe focus of this work is to train a series of\nlanguage models that achieve the best possible per-\nformance at various inference budgets, by training\non more tokens than what is typically used. The\nresulting models, called LLaMA, ranges from 7B\nto 65B parameters with competitive performance\ncompared to the best existing LLMs. For instance,\nLLaMA-13B outperforms GPT-3 on most bench-\nmarks, despite being 10× smaller. We believe that\nthis model will help democratize the access and\nstudy of LLMs, since it can be run on a single GPU.\nAt the higher-end of the scale, our 65B-parameter\nmodel is also competitive with the best large lan-\nguage models such as Chinchilla or PaLM-540B.\nUnlike Chinchilla, PaLM, or GPT-3, we only\nuse publicly available data, making our work com-\npatible with open-sourcing, while most existing\nmodels rely on data which is either not publicly\navailable or undocumented (e.g. “Books – 2TB” or\n“Social media conversations”). There exist some\nexceptions, notably OPT (Zhang et al., 2022),\nGPT-NeoX (Black et al., 2022), BLOOM (Scao\net al., 2022) and GLM (Zeng et al., 2022), but none\nthat are competitive with PaLM-62B or Chinchilla.\nIn the rest of this paper, we present an overview\nof the modifications we made to the transformer\narchitecture (Vaswani et al., 2017), as well as our\ntraining method. We then report the performance of\nour models and compare with others LLMs on a set\nof standard benchmarks. Finally, we expose some\nof the biases and toxicity encoded in our models,\nusing some of the most recent benchmarks from\nthe responsible AI community.\n\n\n2\nApproach\nOur training approach is similar to the methods\ndescribed in previous work (Brown et al., 2020;\nChowdhery et al., 2022), and is inspired by the\nChinchilla scaling laws (Hoffmann et al., 2022).\nWe train large transformers on a large quantity of\ntextual data using a standard optimizer.\n2.1\nPre-training Data\nOur training dataset is a mixture of several sources,\nreported in Table 1, that cover a diverse set of do-\nmains. For the most part, we reuse data sources\nthat have been leveraged to train other LLMs, with\nthe restriction of only using data that is publicly\navailable, and compatible with open sourcing. This\nleads to the following mixture of data and the per-\ncentage they represent in the training set:\nEnglish CommonCrawl [67%].\nWe preprocess\nfive CommonCrawl dumps, ranging from 2017\nto 2020, with the CCNet pipeline (Wenzek et al.,\n2020). This process deduplicates the data at the\nline level, performs language identification with\na fastText linear classifier to remove non-English\npages and filters low quality content with an n-\ngram language model. In addition, we trained a\nlinear model to classify pages used as references\nin Wikipedia v.s. randomly sampled pages, and\ndiscarded pages not classified as references.\nC4 [15%].\nDuring exploratory experiments, we\nobserved that using diverse pre-processed Com-\nmonCrawl datasets improves performance. We thus\nincluded the publicly available C4 dataset (Raffel\net al., 2020) in our data. The preprocessing of C4\nalso contains deduplication and language identifi-\ncation steps: the main difference with CCNet is\nthe quality filtering, which mostly relies on heuris-\ntics such as presence of punctuation marks or the\nnumber of words and sentences in a webpage.\nGithub [4.5%].\nWe use the public GitHub\ndataset available on Google BigQuery. We only\nkept projects that are distributed under the Apache,\nBSD and MIT licenses. Additionally, we filtered\nlow quality files with heuristics based on the line\nlength or proportion of alphanumeric characters,\nand removed boilerplate, such as headers, with reg-\nular expressions. Finally, we deduplicate the result-\ning dataset at the file level, with exact matches.\nWikipedia [4.5%].\nWe add Wikipedia dumps\nfrom the June-August 2022 period, covering 20\nDataset\nSampling prop. Epochs Disk size\nCommonCrawl\n67.0%\n1.10\n3.3 TB\nC4\n15.0%\n1.06\n783 GB\nGithub\n4.5%\n0.64\n328 GB\nWikipedia\n4.5%\n2.45\n83 GB\nBooks\n4.5%\n2.23\n85 GB\nArXiv\n2.5%\n1.06\n92 GB\nStackExchange\n2.0%\n1.03\n78 GB\nTable 1: Pre-training data. Data mixtures used for pre-\ntraining, for each subset we list the sampling proportion,\nnumber of epochs performed on the subset when train-\ning on 1.4T tokens, and disk size. The pre-training runs\non 1T tokens have the same sampling proportion.\nlanguages, which use either the Latin or Cyrillic\nscripts: bg, ca, cs, da, de, en, es, fr, hr, hu, it,\nnl, pl, pt, ro, ru, sl, sr, sv, uk. We process the\ndata to remove hyperlinks, comments and other\nformatting boilerplate.\nGutenberg and Books3 [4.5%].\nWe include two\nbook corpora in our training dataset: the Guten-\nberg Project, which contains books that are in the\npublic domain, and the Books3 section of TheP-\nile (Gao et al., 2020), a publicly available dataset\nfor training large language models. We perform\ndeduplication at the book level, removing books\nwith more than 90% content overlap.\nArXiv [2.5%].\nWe process arXiv Latex files\nto add scientific data to our dataset. Following\nLewkowycz et al. (2022), we removed everything\nbefore the first section, as well as the bibliography.\nWe also removed the comments from the .tex files,\nand inline-expanded definitions and macros written\nby users to increase consistency across papers.\nStack Exchange [2%].\nWe include a dump of\nStack Exchange, a website of high quality ques-\ntions and answers that covers a diverse set of do-\nmains, ranging from computer science to chemistry.\nWe kept the data from the 28 largest websites, re-\nmoved the HTML tags from text and sorted the\nanswers by score (from highest to lowest).\nTokenizer.\nWe tokenize the data with the byte-\npair encoding (BPE) algorithm (Sennrich et al.,\n2015), using the implementation from Sentence-\nPiece (Kudo and Richardson, 2018). Notably, we\nsplit all numbers into individual digits, and fallback\nto bytes to decompose unknown UTF-8 characters.\n\n\nparams\ndimension\nn heads\nn layers\nlearning rate\nbatch size\nn tokens\n6.7B\n4096\n32\n32\n3.0e−4\n4M\n1.0T\n13.0B\n5120\n40\n40\n3.0e−4\n4M\n1.0T\n32.5B\n6656\n52\n60\n1.5e−4\n4M\n1.4T\n65.2B\n8192\n64\n80\n1.5e−4\n4M\n1.4T\nTable 2: Model sizes, architectures, and optimization hyper-parameters.\nOverall, our entire training dataset contains\nroughly 1.4T tokens after tokenization. For most of\nour training data, each token is used only once dur-\ning training, with the exception of the Wikipedia\nand Books domains, over which we perform ap-\nproximately two epochs.\n2.2\nArchitecture\nFollowing recent work on large language models,\nour network is based on the transformer architec-\nture (Vaswani et al., 2017). We leverage various\nimprovements that were subsequently proposed,\nand used in different models such as PaLM. Here\nare the main difference with the original architec-\nture, and where we were found the inspiration for\nthis change (in bracket):\nPre-normalization [GPT3].\nTo improve the\ntraining stability, we normalize the input of each\ntransformer sub-layer, instead of normalizing the\noutput. We use the RMSNorm normalizing func-\ntion, introduced by Zhang and Sennrich (2019).\nSwiGLU activation function [PaLM].\nWe re-\nplace the ReLU non-linearity by the SwiGLU ac-\ntivation function, introduced by Shazeer (2020) to\nimprove the performance. We use a dimension of\n2\n34d instead of 4d as in PaLM.\nRotary Embeddings [GPTNeo].\nWe remove the\nabsolute positional embeddings, and instead, add\nrotary positional embeddings (RoPE), introduced\nby Su et al. (2021), at each layer of the network.\nThe details of the hyper-parameters for our dif-\nferent models are given in Table 2.\n2.3\nOptimizer\nOur models are trained using the AdamW opti-\nmizer (Loshchilov and Hutter, 2017), with the fol-\nlowing hyper-parameters: β1 = 0.9, β2 = 0.95.\nWe use a cosine learning rate schedule, such that\nthe final learning rate is equal to 10% of the maxi-\nmal learning rate. We use a weight decay of 0.1 and\ngradient clipping of 1.0. We use 2, 000 warmup\n0\n200\n400\n600\n800\n1000 1200 1400\nBillion of tokens\n1.5\n1.6\n1.7\n1.8\n1.9\n2.0\n2.1\n2.2\nTraining loss\nLLaMA 7B\nLLaMA 13B\nLLaMA 33B\nLLaMA 65B\nFigure 1: Training loss over train tokens for the 7B,\n13B, 33B, and 65 models. LLaMA-33B and LLaMA-\n65B were trained on 1.4T tokens. The smaller models\nwere trained on 1.0T tokens. All models are trained\nwith a batch size of 4M tokens.\nsteps, and vary the learning rate and batch size with\nthe size of the model (see Table 2 for details).\n2.4\nEfficient implementation\nWe make several optimizations to improve the train-\ning speed of our models. First, we use an efficient\nimplementation of the causal multi-head attention\noperator, inspired by Rabe and Staats (2021) and\nDao et al. (2022). This implementation, available\nin the xformers library,2 reduces the memory us-\nage and computation. This is achieved by not stor-\ning the attention weights and not computing the\nkey/query scores that are masked due to the causal\nnature of the language modeling task.\nTo further improve training efficiency, we re-\nduced the amount of activations that are recom-\nputed during the backward pass with checkpoint-\ning. More precisely, we save the activations that\nare expensive to compute, such as the outputs of\nlinear layers. This is achieved by manually imple-\nmenting the backward function for the transformer\nlayers, instead of relying on the PyTorch autograd.\nTo fully benefit from this optimization, we need to\n2https://github.com/facebookresearch/xformers\n\n\nBoolQ\nPIQA\nSIQA HellaSwag WinoGrande ARC-e\nARC-c\nOBQA\nGPT-3\n175B\n60.5\n81.0\n-\n78.9\n70.2\n68.8\n51.4\n57.6\nGopher\n280B\n79.3\n81.8\n50.6\n79.2\n70.1\n-\n-\n-\nChinchilla\n70B\n83.7\n81.8\n51.3\n80.8\n74.9\n-\n-\n-\nPaLM\n62B\n84.8\n80.5\n-\n79.7\n77.0\n75.2\n52.5\n50.4\nPaLM-cont\n62B\n83.9\n81.4\n-\n80.6\n77.0\n-\n-\n-\nPaLM\n540B\n88.0\n82.3\n-\n83.4\n81.1\n76.6\n53.0\n53.4\nLLaMA\n7B\n76.5\n79.8\n48.9\n76.1\n70.1\n72.8\n47.6\n57.2\n13B\n78.1\n80.1\n50.4\n79.2\n73.0\n74.8\n52.7\n56.4\n33B\n83.1\n82.3\n50.4\n82.8\n76.0\n80.0\n57.8\n58.6\n65B\n85.3\n82.8\n52.3\n84.2\n77.0\n78.9\n56.0\n60.2\nTable 3: Zero-shot performance on Common Sense Reasoning tasks.\nreduce the memory usage of the model by using\nmodel and sequence parallelism, as described by\nKorthikanti et al. (2022). Moreover, we also over-\nlap the computation of activations and the commu-\nnication between GPUs over the network (due to\nall_reduce operations) as much as possible.\nWhen training a 65B-parameter model, our code\nprocesses around 380 tokens/sec/GPU on 2048\nA100 GPU with 80GB of RAM. This means that\ntraining over our dataset containing 1.4T tokens\ntakes approximately 21 days.\n3\nMain results\nFollowing previous work (Brown et al., 2020), we\nconsider zero-shot and few-shot tasks, and report\nresults on a total of 20 benchmarks:\n• Zero-shot. We provide a textual description\nof the task and a test example. The model\neither provides an answer using open-ended\ngeneration, or ranks the proposed answers.\n• Few-shot. We provide a few examples of the\ntask (between 1 and 64) and a test example.\nThe model takes this text as input and gener-\nates the answer or ranks different options.\nWe compare LLaMA with other foundation mod-\nels, namely the non-publicly available language\nmodels GPT-3 (Brown et al., 2020), Gopher (Rae\net al., 2021), Chinchilla (Hoffmann et al., 2022)\nand PaLM (Chowdhery et al., 2022), as well as\nthe open-sourced OPT models (Zhang et al., 2022),\nGPT-J (Wang and Komatsuzaki, 2021), and GPT-\nNeo (Black et al., 2022). In Section 4, we also\nbriefly compare LLaMA with instruction-tuned\nmodels such as OPT-IML (Iyer et al., 2022) and\nFlan-PaLM (Chung et al., 2022).\nWe evaluate LLaMA on free-form generation\ntasks and multiple choice tasks. In the multiple\nchoice tasks, the objective is to select the most\nappropriate completion among a set of given op-\ntions, based on a provided context. We select the\ncompletion with the highest likelihood given the\nprovided context. We follow Gao et al. (2021)\nand use the likelihood normalized by the number\nof characters in the completion, except for certain\ndatasets (OpenBookQA, BoolQ), for which we fol-\nlow Brown et al. (2020), and select a completion\nbased on the likelihood normalized by the likeli-\nhood of the completion given “Answer:” as context:\nP(completion|context)/P(completion|“Answer:”).\n0-shot 1-shot 5-shot 64-shot\nGPT-3\n175B\n14.6\n23.0\n-\n29.9\nGopher\n280B\n10.1\n-\n24.5\n28.2\nChinchilla 70B\n16.6\n-\n31.5\n35.5\nPaLM\n8B\n8.4\n10.6\n-\n14.6\n62B\n18.1\n26.5\n-\n27.6\n540B\n21.2\n29.3\n-\n39.6\nLLaMA\n7B\n16.8\n18.7\n22.0\n26.1\n13B\n20.1\n23.4\n28.1\n31.9\n33B\n24.9\n28.3\n32.9\n36.0\n65B\n23.8\n31.0\n35.0\n39.9\nTable 4: NaturalQuestions. Exact match performance.\n3.1\nCommon Sense Reasoning\nWe consider eight standard common sense rea-\nsoning benchmarks: BoolQ (Clark et al., 2019),\nPIQA (Bisk et al., 2020), SIQA (Sap et al., 2019),\n\n\nHellaSwag (Zellers et al., 2019), WinoGrande (Sak-\naguchi et al., 2021), ARC easy and challenge (Clark\net al., 2018) and OpenBookQA (Mihaylov et al.,\n2018). These datasets include Cloze and Winograd\nstyle tasks, as well as multiple choice question an-\nswering. We evaluate in the zero-shot setting as\ndone in the language modeling community.\nIn Table 3, we compare with existing models\nof various sizes and report numbers from the cor-\nresponding papers.\nFirst, LLaMA-65B outper-\nforms Chinchilla-70B on all reported benchmarks\nbut BoolQ. Similarly, this model surpasses PaLM-\n540B everywhere but on BoolQ and WinoGrande.\nLLaMA-13B model also outperforms GPT-3 on\nmost benchmarks despite being 10× smaller.\n3.2\nClosed-book Question Answering\nWe compare LLaMA to existing large language\nmodels on two closed-book question answering\nbenchmarks:\nNatural Questions (Kwiatkowski\net al., 2019) and TriviaQA (Joshi et al., 2017). For\nboth benchmarks, we report exact match perfor-\nmance in a closed book setting, i.e., where the mod-\nels do not have access to documents that contain\nevidence to answer the question. In Table 4, we\nreport performance on NaturalQuestions, and in Ta-\nble 5, we report on TriviaQA. On both benchmarks,\nLLaMA-65B achieve state-of-the-arts performance\nin the zero-shot and few-shot settings. More im-\nportantly, the LLaMA-13B is also competitive on\nthese benchmarks with GPT-3 and Chinchilla, de-\nspite being 5-10× smaller. This model runs on a\nsingle V100 GPU during inference.\n0-shot 1-shot 5-shot 64-shot\nGopher\n280B\n43.5\n-\n57.0\n57.2\nChinchilla 70B\n55.4\n-\n64.1\n64.6\nLLaMA\n7B\n50.0\n53.4\n56.3\n57.6\n13B\n56.6\n60.5\n63.1\n64.0\n33B\n65.1\n67.9\n69.9\n70.4\n65B\n68.2\n71.6\n72.6\n73.0\nTable 5: TriviaQA. Zero-shot and few-shot exact match\nperformance on the filtered dev set.\n3.3\nReading Comprehension\nWe evaluate our models on the RACE reading com-\nprehension benchmark (Lai et al., 2017). This\ndataset was collected from English reading com-\nprehension exams designed for middle and high\nRACE-middle\nRACE-high\nGPT-3\n175B\n58.4\n45.5\nPaLM\n8B\n57.9\n42.3\n62B\n64.3\n47.5\n540B\n68.1\n49.1\nLLaMA\n7B\n61.1\n46.9\n13B\n61.6\n47.2\n33B\n64.1\n48.3\n65B\n67.9\n51.6\nTable 6: Reading Comprehension. Zero-shot accuracy.\nschool Chinese students. We follow the evaluation\nsetup from Brown et al. (2020) and report results\nin Table 6. On these benchmarks, LLaMA-65B is\ncompetitive with PaLM-540B, and, LLaMA-13B\noutperforms GPT-3 by a few percents.\n3.4\nMathematical reasoning\nWe evaluate our models on two mathematical rea-\nsoning benchmarks: MATH (Hendrycks et al.,\n2021) and GSM8k (Cobbe et al., 2021). MATH\nis a dataset of 12K middle school and high school\nmathematics problems written in LaTeX. GSM8k\nis a set of middle school mathematical problems.\nIn Table 7, we compare with PaLM and Min-\nerva (Lewkowycz et al., 2022). Minerva is a series\nof PaLM models finetuned on 38.5B tokens ex-\ntracted from ArXiv and Math Web Pages, while\nneither PaLM or LLaMA are finetuned on mathe-\nmatical data. The numbers for PaLM and Minerva\nare taken from Lewkowycz et al. (2022), and we\ncompare with and without maj1@k. maj1@k de-\nnotes evaluations where we generate k samples for\neach problem and perform a majority voting (Wang\net al., 2022). On GSM8k, we observe that LLaMA-\n65B outperforms Minerva-62B, although it has not\nbeen fine-tuned on mathematical data.\n3.5\nCode generation\nWe evaluate the ability of our models to write\ncode from a natural language description on two\nbenchmarks: HumanEval (Chen et al., 2021) and\nMBPP (Austin et al., 2021). For both tasks, the\nmodel receives a description of the program in a\nfew sentences, as well as a few input-output ex-\namples. In HumanEval, it also receives a function\nsignature, and the prompt is formatted as natural\ncode with the textual description and tests in a\n\n\nMATH +maj1@k GSM8k +maj1@k\nPaLM\n8B 1.5\n-\n4.1\n-\n62B 4.4\n-\n33.0\n-\n540B 8.8\n-\n56.5\n-\nMinerva\n8B 14.1\n25.4\n16.2\n28.4\n62B 27.6\n43.4\n52.4\n68.5\n540B 33.6\n50.3\n68.5\n78.5\nLLaMA\n7B 2.9\n6.9\n11.0\n18.1\n13B 3.9\n8.8\n17.8\n29.3\n33B 7.1\n15.2\n35.6\n53.1\n65B 10.6\n20.5\n50.9\n69.7\nTable 7: Model performance on quantitative reason-\ning datasets. For majority voting, we use the same\nsetup as Minerva, with k = 256 samples for MATH\nand k = 100 for GSM8k (Minerva 540B uses k = 64\nfor MATH and and k = 40 for GSM8k). LLaMA-65B\noutperforms Minerva 62B on GSM8k, although it has\nnot been fine-tuned on mathematical data.\ndocstring. The model needs to generate a Python\nprogram that fits the description and satisfies the\ntest cases. In Table 8, we compare the pass@1\nscores of our models with existing language mod-\nels that have not been finetuned on code, namely\nPaLM and LaMDA (Thoppilan et al., 2022). PaLM\nand LLaMA were trained on datasets that contain\na similar number of code tokens.\nAs show in Table 8, for a similar number\nof parameters, LLaMA outperforms other gen-\neral models such as LaMDA and PaLM, which\nare not trained or finetuned specifically for code.\nLLaMA with 13B parameters and more outper-\nforms LaMDA 137B on both HumanEval and\nMBPP. LLaMA 65B also outperforms PaLM 62B,\neven when it is trained longer. The pass@1 results\nreported in this table were obtained by sampling\nwith temperature 0.1. The pass@100 and pass@80\nmetrics were obtained with temperature 0.8. We\nuse the same method as Chen et al. (2021) to obtain\nunbiased estimates of the pass@k.\nIt is possible to greatly improve the performance\non code by finetuning models on code-specific to-\nkens. For instance, PaLM-Coder (Chowdhery et al.,\n2022) increases the pass@1 score of PaLM on Hu-\nmanEval from 26.2% for PaLM to 36%. Other\nmodels trained specifically for code also perform\nbetter than general models on these tasks (Chen\net al., 2021; Nijkamp et al., 2022; Fried et al., 2022).\nFinetuning on code tokens is, however, beyond the\nscope of this paper.\nParams\nHumanEval\nMBPP\npass@\n@1\n@100\n@1\n@80\nLaMDA\n137B 14.0\n47.3\n14.8\n62.4\nPaLM\n8B 3.6∗\n18.7∗\n5.0∗\n35.7∗\nPaLM\n62B 15.9\n46.3∗\n21.4 63.2∗\nPaLM-cont\n62B 23.7\n-\n31.2\n-\nPaLM\n540B 26.2\n76.2\n36.8\n75.0\nLLaMA\n7B 10.5\n36.5\n17.7\n56.2\n13B 15.8\n52.5\n22.0\n64.0\n33B 21.7\n70.7\n30.2\n73.4\n65B 23.7\n79.3\n37.7\n76.8\nTable 8: Model performance for code generation. We\nreport the pass@ score on HumanEval and MBPP. Hu-\nmanEval generations are done in zero-shot and MBBP\nwith 3-shot prompts similar to Austin et al. (2021). The\nvalues marked with ∗are read from figures in Chowdh-\nery et al. (2022).\n3.6\nMassive Multitask Language\nUnderstanding\nThe massive multitask language understanding\nbenchmark, or MMLU, introduced by Hendrycks\net al. (2020) consists of multiple choice questions\ncovering various domains of knowledge, includ-\ning humanities, STEM and social sciences. We\nevaluate our models in the 5-shot setting, using the\nexamples provided by the benchmark, and report\nresults in Table 9. On this benchmark, we observe\nthat the LLaMA-65B is behind both Chinchilla-\n70B and PaLM-540B by a few percent in average,\nand across most domains. A potential explanation\nis that we have used a limited amount of books\nand academic papers in our pre-training data, i.e.,\nArXiv, Gutenberg and Books3, that sums up to only\n177GB, while these models were trained on up to\n2TB of books. This large quantity of books used\nby Gopher, Chinchilla and PaLM may also explain\nwhy Gopher outperforms GPT-3 on this benchmark,\nwhile it is comparable on other benchmarks.\n3.7\nEvolution of performance during training\nDuring training, we tracked the performance of our\nmodels on a few question answering and common\nsense benchmarks, and report them in Figure 2.\nOn most benchmarks, the performance improves\nsteadily, and correlates with the training perplexity\nof the model (see Figure 1). The exceptions are\nSIQA and WinoGrande. Most notably, on SIQA,\n\n\nHumanities\nSTEM\nSocial Sciences\nOther\nAverage\nGPT-NeoX\n20B\n29.8\n34.9\n33.7\n37.7\n33.6\nGPT-3\n175B\n40.8\n36.7\n50.4\n48.8\n43.9\nGopher\n280B\n56.2\n47.4\n71.9\n66.1\n60.0\nChinchilla\n70B\n63.6\n54.9\n79.3\n73.9\n67.5\nPaLM\n8B\n25.6\n23.8\n24.1\n27.8\n25.4\n62B\n59.5\n41.9\n62.7\n55.8\n53.7\n540B\n77.0\n55.6\n81.0\n69.6\n69.3\nLLaMA\n7B\n34.0\n30.5\n38.3\n38.1\n35.1\n13B\n45.0\n35.8\n53.8\n53.3\n46.9\n33B\n55.8\n46.0\n66.7\n63.4\n57.8\n65B\n61.8\n51.7\n72.9\n67.4\n63.4\nTable 9: Massive Multitask Language Understanding (MMLU). Five-shot accuracy.\nwe observe a lot of variance in performance,\nthat may indicate that this benchmark is not\nreliable. On WinoGrande, the performance does\nnot correlate as well with training perplexity:\nthe LLaMA-33B and LLaMA-65B have similar\nperformance during the training.\n4\nInstruction Finetuning\nIn this section, we show that briefly finetuning on\ninstructions data rapidly leads to improvements\non MMLU. Although the non-finetuned version\nof LLaMA-65B is already able to follow basic in-\nstructions, we observe that a very small amount of\nfinetuning improves the performance on MMLU,\nand further improves the ability of the model to\nfollow instructions. Since this is not the focus of\nthis paper, we only conducted a single experiment\nfollowing the same protocol as Chung et al. (2022)\nto train an instruct model, LLaMA-I.\nIn Table 10, we report the results of our instruct\nmodel LLaMA-I on MMLU and compare with ex-\nisting instruction finetuned models of moderate\nsizes, namely, OPT-IML (Iyer et al., 2022) and the\nFlan-PaLM series (Chung et al., 2022). All the re-\nported numbers are from the corresponding papers.\nDespite the simplicity of the instruction finetuning\napproach used here, we reach 68.9% on MMLU.\nLLaMA-I (65B) outperforms on MMLU existing\ninstruction finetuned models of moderate sizes, but\nare still far from the state-of-the-art, that is 77.4\nfor GPT code-davinci-002 on MMLU (numbers\ntaken from Iyer et al. (2022)). The details of the\nperformance on MMLU on the 57 tasks can be\nfound in Table 16 of the appendix.\nOPT\n30B\n26.1\nGLM\n120B\n44.8\nPaLM\n62B\n55.1\nPaLM-cont\n62B\n62.8\nChinchilla\n70B\n67.5\nLLaMA\n65B\n63.4\nOPT-IML-Max\n30B\n43.2\nFlan-T5-XXL\n11B\n55.1\nFlan-PaLM\n62B\n59.6\nFlan-PaLM-cont\n62B\n66.1\nLLaMA-I\n65B\n68.9\nTable 10: Instruction finetuning – MMLU (5-shot).\nComparison of models of moderate size with and with-\nout instruction finetuning on MMLU.\n5\nBias, Toxicity and Misinformation\nLarge language models have been showed to re-\nproduce and amplify biases that are existing in\nthe training data (Sheng et al., 2019; Kurita et al.,\n2019), and to generate toxic or offensive con-\ntent (Gehman et al., 2020). As our training dataset\ncontains a large proportion of data from the Web,\nwe believe that it is crucial to determine the po-\ntential for our models to generate such content.\nTo understand the potential harm of LLaMA-65B,\nwe evaluate on different benchmarks that measure\ntoxic content production and stereotypes detection.\nWhile we have selected some of the standard bench-\nmarks that are used by the language model com-\nmunity to indicate some of the issues with these\nmodels, these evaluations are not sufficient to fully\nunderstand the risks associated with these models.\n\n\n0\n250\n500\n750\n1000 1250 1500\n20\n30\n40\n50\n60\n70\nAccuracy\nTriviaQA\n0\n250\n500\n750\n1000 1250 1500\n50\n55\n60\n65\n70\n75\n80\n85\nHellaSwag\n0\n250\n500\n750\n1000 1250 1500\n0\n5\n10\n15\n20\n25\n30\n35\nNaturalQuestions\n0\n250\n500\n750\n1000 1250 1500\nBillion of tokens\n40\n42\n44\n46\n48\n50\n52\nAccuracy\nSIQA\n0\n250\n500\n750\n1000 1250 1500\nBillion of tokens\n50\n55\n60\n65\n70\n75\n80\nWinoGrande\n0\n250\n500\n750\n1000 1250 1500\nBillion of tokens\n65.0\n67.5\n70.0\n72.5\n75.0\n77.5\n80.0\n82.5\nPIQA\nLLaMA 7B\nLLaMA 13B\nLLaMA 33B\nLLaMA 65B\nChinchilla\nFigure 2: Evolution of performance on question answering and common sense reasoning during training.\n5.1\nRealToxicityPrompts\nLanguage models can generate toxic language, e.g.,\ninsults, hate speech or threats. There is a very large\nrange of toxic content that a model can generate,\nmaking a thorough evaluation challenging. Several\nrecent work (Zhang et al., 2022; Hoffmann et al.,\n2022) have considered the RealToxicityPrompts\nbenchmark (Gehman et al., 2020) as an indicator\nof how toxic is their model. RealToxicityPrompts\nconsists of about 100k prompts that the model must\ncomplete; then a toxicity score is automatically\nevaluated by making a request to PerspectiveAPI 3.\nWe do not have control over the pipeline used by\nthe third-party PerspectiveAPI, making comparison\nwith previous models difficult.\nFor each of the 100k prompts, we greedily gen-\nerate with our models, and measure their toxic-\nity score. The score per prompt ranges from 0\n(non-toxic) to 1 (toxic). In Table 11, we report our\naveraged score on basic and respectful prompt cat-\negories of RealToxicityPrompts. These scores are\n“comparable” with what we observe in the litera-\nture (e.g., 0.087 for Chinchilla) but the method-\nologies differ between these work and ours (in\nterms of sampling strategy, number of prompts and\ntime of API). We observe that toxicity increases\n3https://perspectiveapi.com/\nBasic\nRespectful\nLLaMA\n7B\n0.106\n0.081\n13B\n0.104\n0.095\n33B\n0.107\n0.087\n65B\n0.128\n0.141\nTable 11: RealToxicityPrompts. We run a greedy de-\ncoder on the 100k prompts from this benchmark. The\n“respectful” versions are prompts starting with “Com-\nplete the following sentence in a polite, respectful, and\nunbiased manner:”, and “Basic” is without it. Scores\nwere obtained using the PerplexityAPI, with higher\nscore indicating more toxic generations.\nwith the size of the model, especially for Respect-\nful prompts. This was also observed in previous\nwork (Zhang et al., 2022), with the notable excep-\ntion of Hoffmann et al. (2022) where they do not\nsee a difference between Chinchilla and Gopher,\ndespite different sizes. This could be explained by\nthe fact that the larger model, Gopher, has worse\nperformance than Chinchilla, suggesting that the\nrelation between toxicity and model size may only\napply within a model family.\n\n\nLLaMA\nGPT3\nOPT\nGender\n70.6\n62.6\n65.7\nReligion\n79.0\n73.3\n68.6\nRace/Color\n57.0\n64.7\n68.6\nSexual orientation\n81.0\n76.2\n78.6\nAge\n70.1\n64.4\n67.8\nNationality\n64.2\n61.6\n62.9\nDisability\n66.7\n76.7\n76.7\nPhysical appearance\n77.8\n74.6\n76.2\nSocioeconomic status\n71.5\n73.8\n76.2\nAverage\n66.6\n67.2\n69.5\nTable 12: CrowS-Pairs. We compare the level of biases\ncontained in LLaMA-65B with OPT-175B and GPT3-\n175B. Higher score indicates higher bias.\n5.2\nCrowS-Pairs\nWe evaluate the biases in our model on the CrowS-\nPairs (Nangia et al., 2020). This dataset allows to\nmeasure biases in 9 categories: gender, religion,\nrace/color, sexual orientation, age, nationality, dis-\nability, physical appearance and socioeconomic sta-\ntus. Each example is composed of a stereotype\nand an anti-stereotype and we measure the model\npreference for the stereotypical sentence using the\nperplexity of both sentences in a zero-shot setting.\nHigher scores thus indicate higher bias. We com-\npare with GPT-3 and OPT-175B in Table 12.\nOur model compares slightly favorably to both\nmodels on average. Our model is particularly bi-\nased in the religion category (+10 compared to\nOPT-175B), followed by age and gender (+6 each\ncompared to best model). We expect these biases\nto come from CommonCrawl despite multiple fil-\ntering steps.\n5.3\nWinoGender\nTo further investigate the biases of our model on\nthe gender category, we look at the WinoGender\nbenchmark (Rudinger et al., 2018), a co-reference\nresolution dataset. WinoGender is made of Wino-\ngrad schema, and biases are evaluated by determin-\ning if a model co-reference resolution performance\nis impacted by the gender of the pronoun.\nMore precisely, each sentence has three men-\ntions: an “occupation”, a “participant”, and a\n“pronoun” where the pronoun is co-referencing\neither the occupation or participant. We prompt\nthe model to determine the co-reference relation\nand measure if it does so correctly according to\nthe context of the sentence. The goal is to reveal\nif societal biases associated with occupations\nhave been captured by the model. For example,\na sentence in the WinoGender dataset is “The\nnurse notified the patient that his shift would be\nending in an hour.”, which is followed by ‘His’\nrefers to. We then compare the perplexity of the\ncontinuations the nurse and the patient to per-\nform co-reference resolution with the model. We\nevaluate the performance when using 3 pronouns:\n“her/her/she”, “his/him/he” and “their/them/some-\none” (the different choices corresponding to the\ngrammatical function of the pronoun.\nIn Table 13, we report the co-reference scores\nfor the three different pronouns contained in the\ndataset. We observe that our model is significantly\nbetter at performing co-reference resolution for\nthe “their/them/someone” pronouns than for the\n“her/her/she” and “his/him/he” pronouns. A simi-\nlar observation was made in previous work (Rae\net al., 2021; Hoffmann et al., 2022), and is likely\nindicative of gender bias. Indeed, in the case of the\n“her/her/she” and “his/him/he” pronouns, the model\nis probably using the majority gender of the occu-\npation to perform co-reference resolution, instead\nof using the evidence of the sentence.\nTo further investigate this hypothesis, we look\nat the set of “gotcha” cases for the “her/her/she”\nand “his/him/he” pronouns in the WinoGender\ndataset. Theses cases correspond to sentences in\nwhich the pronoun does not match the majority\ngender of the occupation, and the occupation is\nthe correct answer. In Table 13, we observe that\nour model, LLaMA-65B, makes more errors on the\ngotcha examples, clearly showing that it capture\nsocietal biases related to gender and occupation.\nThe drop of performance exists for “her/her/she”\nand “his/him/he” pronouns, which is indicative of\nbiases regardless of gender.\n5.4\nTruthfulQA\nTruthfulQA (Lin et al., 2021) aims to measure the\ntruthfulness of a model, i.e., its ability to identify\nwhen a claim is true. Lin et al. (2021) consider\nthe definition of “true” in the sense of “literal truth\nabout the real world”, and not claims that are only\ntrue in the context of a belief system or tradition.\nThis benchmark can evaluate the risks of a model\nto generate misinformation or false claims. The\nquestions are written in diverse style, cover 38 cat-\negories and are designed to be adversarial.\n\n\n7B\n13B\n33B\n65B\nAll\n66.0\n64.7\n69.0\n77.5\nher/her/she\n65.0\n66.7\n66.7\n78.8\nhis/him/he\n60.8\n62.5\n62.1\n72.1\ntheir/them/someone\n72.1\n65.0\n78.3\n81.7\nher/her/she (gotcha)\n64.2\n65.8\n61.7\n75.0\nhis/him/he (gotcha)\n55.0\n55.8\n55.8\n63.3\nTable 13: WinoGender. Co-reference resolution ac-\ncuracy for the LLaMA models, for different pronouns\n(“her/her/she” and “his/him/he”). We observe that our\nmodels obtain better performance on “their/them/some-\none’ pronouns than on “her/her/she” and “his/him/he’,\nwhich is likely indicative of biases.\nTruthful\nTruthful*Inf\nGPT-3\n1.3B\n0.31\n0.19\n6B\n0.22\n0.19\n175B\n0.28\n0.25\nLLaMA\n7B\n0.33\n0.29\n13B\n0.47\n0.41\n33B\n0.52\n0.48\n65B\n0.57\n0.53\nTable 14: TruthfulQA.. We report the fraction of truth-\nful and truthful*informative answers, as scored by spe-\ncially trained models via the OpenAI API. We follow\nthe QA prompt style used in Ouyang et al. (2022), and\nreport the performance of GPT-3 from the same paper.\nIn Table 14, we report the performance of our\nmodels on both questions to measure truthful mod-\nels and the intersection of truthful and informative.\nCompared to GPT-3, our model scores higher in\nboth categories, but the rate of correct answers is\nstill low, showing that our model is likely to hallu-\ncinate incorrect answers.\n6\nCarbon footprint\nThe training of our models have consumed a mas-\nsive quantity of energy, responsible for the emis-\nsion of carbon dioxide. We follow the recent liter-\nature on the subject and breakdown both the total\nenergy consumption and the resulting carbon foot-\nprint in Table 15. We follow a formula for Wu et al.\n(2022) to estimate the Watt-hour, Wh, needed to\ntrain a model, as well as the tons of carbon emis-\nsions, tCO2eq. For the Wh, we use the formula:\nWh = GPU-h×(GPU power consumption)×PUE,\nwhere we set the Power Usage Effectiveness (PUE)\nat 1.1. The resulting carbon emission depends on\nthe location of the data center used to train the net-\nwork. For instance, BLOOM uses a grid that emits\n0.057 kg CO2eq/KWh leading to 27 tCO2eq and\nOPT a grid that emits 0.231 kg CO2eq/KWh, lead-\ning to 82 tCO2eq. In this study, we are interested in\ncomparing the cost in carbon emission of training\nof these models if they were trained in the same\ndata center. Hence, we do not take the location\nof data center in consideration, and use, instead,\nthe US national average carbon intensity factor of\n0.385 kg CO2eq/KWh. This leads to the following\nformula for the tons of carbon emissions:\ntCO2eq = MWh × 0.385.\nWe apply the same formula to OPT and BLOOM\nfor fair comparison. For OPT, we assume training\nrequired 34 days on 992 A100-80B (see their logs4).\nFinally, we estimate that we used 2048 A100-80GB\nfor a period of approximately 5 months to develop\nour models. This means that developing these mod-\nels would have cost around 2,638 MWh under our\nassumptions, and a total emission of 1,015 tCO2eq.\nWe hope that releasing these models will help to\nreduce future carbon emission since the training is\nalready done, and some of the models are relatively\nsmall and can be run on a single GPU.\n7\nRelated work\nLanguage models\nare probability distributions\nover sequences of words, tokens or charac-\nters (Shannon, 1948, 1951). This task, often framed\nas next token prediction, has long been considered a\ncore problem in natural language processing (Bahl\net al., 1983; Brown et al., 1990). Because Turing\n(2009) proposed to measure machine intelligence\nby using language through the “imitation game”,\nlanguage modeling has been proposed as a bench-\nmark to measure progress toward artificial intelli-\ngence (Mahoney, 1999).\nArchitecture.\nTraditionally, language models\nwere based on n-gram count statistics (Bahl\net al., 1983), and various smoothing techniques\nwere proposed to improve the estimation of rare\nevents (Katz, 1987; Kneser and Ney, 1995). In the\npast two decades, neural networks have been suc-\ncessfully applied to the language modelling task,\n4https://github.com/facebookresearch/metaseq/\ntree/main/projects/OPT/chronicles\n\n\nGPU Type\nGPU Power\nGPU-hours\nTotal power\nCarbon emitted\nconsumption\nconsumption\n(tCO2eq)\nOPT-175B\nA100-80GB\n400W\n809,472\n356 MWh\n137\nBLOOM-175B\nA100-80GB\n400W\n1,082,880\n475 MWh\n183\nLLaMA-7B\nA100-80GB\n400W\n82,432\n36 MWh\n14\nLLaMA-13B\nA100-80GB\n400W\n135,168\n59 MWh\n23\nLLaMA-33B\nA100-80GB\n400W\n530,432\n233 MWh\n90\nLLaMA-65B\nA100-80GB\n400W\n1,022,362\n449 MWh\n173\nTable 15: Carbon footprint of training different models in the same data center. We follow the formula from Wu\net al. (2022) to compute carbon emission of train OPT, BLOOM and our models in the same data center. For the\npower consumption of a A100-80GB, we take the thermal design power (TDP) for NVLink systems, that is 400W.\nWe take a PUE of 1.1 and a carbon intensity factor set at the national US average of 0.385 kg CO2e per KWh.\nstarting from feed forward models (Bengio et al.,\n2000), recurrent neural networks (Elman, 1990;\nMikolov et al., 2010) and LSTMs (Hochreiter and\nSchmidhuber, 1997; Graves, 2013). More recently,\ntransformer networks, based on self-attention, have\nled to important improvements, especially for cap-\nturing long range dependencies (Vaswani et al.,\n2017; Radford et al., 2018; Dai et al., 2019).\nScaling.\nThere is a long history of scaling for\nlanguage models, for both the model and dataset\nsizes. Brants et al. (2007) showed the benefits of\nusing language models trained on 2 trillion tokens,\nresulting in 300 billion n-grams, on the quality of\nmachine translation. While this work relied on a\nsimple smoothing technique, called Stupid Backoff,\nHeafield et al. (2013) later showed how to scale\nKneser-Ney smoothing to Web-scale data. This\nallowed to train a 5-gram model on 975 billions to-\nkens from CommonCrawl, resulting in a model\nwith 500 billions n-grams (Buck et al., 2014).\nChelba et al. (2013) introduced the One Billion\nWord benchmark, a large scale training dataset to\nmeasure the progress of language models.\nIn the context of neural language models, Joze-\nfowicz et al. (2016) obtained state-of-the-art re-\nsults on the Billion Word benchmark by scaling\nLSTMs to 1 billion parameters.\nLater, scaling\ntransformers lead to improvement on many NLP\ntasks. Notable models include BERT (Devlin et al.,\n2018), GPT-2 (Radford et al., 2019), Megatron-\nLM (Shoeybi et al., 2019), and T5 (Raffel et al.,\n2020). A significant breakthrough was obtained\nwith GPT-3 (Brown et al., 2020), a model with\n175 billion parameters. This lead to a series of\nLarge Language Models, such as Jurassic-1 (Lieber\net al., 2021), Megatron-Turing NLG (Smith et al.,\n2022), Gopher (Rae et al., 2021), Chinchilla (Hoff-\nmann et al., 2022), PaLM (Chowdhery et al., 2022),\nOPT (Zhang et al., 2022), and GLM (Zeng et al.,\n2022). Hestness et al. (2017) and Rosenfeld et al.\n(2019) studied the impact of scaling on the perfor-\nmance of deep learning models, showing the exis-\ntence of power laws between the model and dataset\nsizes and the performance of the system. Kaplan\net al. (2020) derived power laws specifically for\ntransformer based language models, which were\nlater refined by Hoffmann et al. (2022), by adapting\nthe learning rate schedule when scaling datasets.\nFinally, Wei et al. (2022) studied the effect of scal-\ning on the abilities of large language models.\n8\nConclusion\nIn this paper, we presented a series of language\nmodels that are released openly, and competitive\nwith state-of-the-art foundation models.\nMost\nnotably, LLaMA-13B outperforms GPT-3 while\nbeing more than 10× smaller, and LLaMA-65B is\ncompetitive with Chinchilla-70B and PaLM-540B.\nUnlike previous studies, we show that it is possible\nto achieve state-of-the-art performance by training\nexclusively on publicly available data, without\nresorting to proprietary datasets. We hope that\nreleasing these models to the research community\nwill accelerate the development of large language\nmodels, and help efforts to improve their robust-\nness and mitigate known issues such as toxicity and\nbias. Additionally, we observed like Chung et al.\n(2022) that finetuning these models on instructions\nlead to promising results, and we plan to further\ninvestigate this in future work. Finally, we plan to\nrelease larger models trained on larger pretraining\ncorpora in the future, since we have seen a constant\n\n\nimprovement in performance as we were scaling.\nAcknowledgements\nWe thank Daniel Haziza, Francisco Massa, Jeremy\nReizenstein, Artem Korenev, and Patrick Labatut\nfrom the xformers team. We thank Susan Zhang\nand Stephen Roller for their support on data\ndeduplication. We thank Luca Wehrstedt, Vegard\nMella, and Pierre-Emmanuel Mazaré for their\nsupport on training stability. We thank Shubho\nSengupta, Kalyan Saladi, and all the AI infra team\nfor their support. We thank Jane Yu for her input\non evaluation. We thank Yongyi Hu for his help\non data collection.\nReferences\nJacob Austin, Augustus Odena, Maxwell Nye, Maarten\nBosma, Henryk Michalewski, David Dohan, Ellen\nJiang, Carrie Cai, Michael Terry, Quoc Le, and\nCharles Sutton. 2021. Program synthesis with large\nlanguage models.\nLalit R Bahl, Frederick Jelinek, and Robert L Mercer.\n1983. A maximum likelihood approach to continuous\nspeech recognition. IEEE transactions on pattern\nanalysis and machine intelligence, pages 179–190.\nYoshua Bengio, Réjean Ducharme, and Pascal Vincent.\n2000. A neural probabilistic language model. Ad-\nvances in neural information processing systems, 13.\nYonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi,\net al. 2020. Piqa: Reasoning about physical com-\nmonsense in natural language. In Proceedings of\nthe AAAI conference on artificial intelligence, pages\n7432–7439.\nSid Black, Stella Biderman, Eric Hallahan, Quentin\nAnthony, Leo Gao, Laurence Golding, Horace He,\nConnor Leahy, Kyle McDonell, Jason Phang, et al.\n2022. Gpt-neox-20b: An open-source autoregressive\nlanguage model. arXiv preprint arXiv:2204.06745.\nThorsten Brants, Ashok C. Popat, Peng Xu, Franz J.\nOch, and Jeffrey Dean. 2007. Large language mod-\nels in machine translation. In Proceedings of the\n2007 Joint Conference on Empirical Methods in Nat-\nural Language Processing and Computational Nat-\nural Language Learning (EMNLP-CoNLL), pages\n858–867, Prague, Czech Republic. Association for\nComputational Linguistics.\nPeter F Brown, John Cocke, Stephen A Della Pietra,\nVincent J Della Pietra, Frederick Jelinek, John Laf-\nferty, Robert L Mercer, and Paul S Roossin. 1990. A\nstatistical approach to machine translation. Compu-\ntational linguistics, 16(2):79–85.\nTom B. Brown, Benjamin Mann, Nick Ryder, Melanie\nSubbiah, Jared Kaplan, Prafulla Dhariwal, Arvind\nNeelakantan, Pranav Shyam, Girish Sastry, Amanda\nAskell, Sandhini Agarwal, Ariel Herbert-Voss,\nGretchen Krueger, Tom Henighan, Rewon Child,\nAditya Ramesh, Daniel M. Ziegler, Jeffrey Wu,\nClemens Winter, Christopher Hesse, Mark Chen, Eric\nSigler, Mateusz Litwin, Scott Gray, Benjamin Chess,\nJack Clark, Christopher Berner, Sam McCandlish,\nAlec Radford, Ilya Sutskever, and Dario Amodei.\n2020. Language models are few-shot learners.\nChristian Buck, Kenneth Heafield, and Bas Van Ooyen.\n2014. N-gram counts and language models from the\ncommon crawl. In LREC, volume 2, page 4.\nCiprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge,\nThorsten Brants, Phillipp Koehn, and Tony Robin-\nson. 2013. One billion word benchmark for measur-\ning progress in statistical language modeling. arXiv\npreprint arXiv:1312.3005.\nMark Chen, Jerry Tworek, Heewoo Jun, Qiming\nYuan, Henrique Ponde de Oliveira Pinto, Jared Ka-\nplan, Harri Edwards, Yuri Burda, Nicholas Joseph,\nGreg Brockman, Alex Ray, Raul Puri, Gretchen\nKrueger, Michael Petrov, Heidy Khlaaf, Girish Sas-\ntry, Pamela Mishkin, Brooke Chan, Scott Gray,\nNick Ryder, Mikhail Pavlov, Alethea Power, Lukasz\nKaiser, Mohammad Bavarian, Clemens Winter,\nPhilippe Tillet, Felipe Petroski Such, Dave Cum-\nmings, Matthias Plappert, Fotios Chantzis, Eliza-\nbeth Barnes, Ariel Herbert-Voss, William Hebgen\nGuss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie\nTang, Igor Babuschkin, Suchir Balaji, Shantanu Jain,\nWilliam Saunders, Christopher Hesse, Andrew N.\nCarr, Jan Leike, Josh Achiam, Vedant Misra, Evan\nMorikawa, Alec Radford, Matthew Knight, Miles\nBrundage, Mira Murati, Katie Mayer, Peter Welinder,\nBob McGrew, Dario Amodei, Sam McCandlish, Ilya\nSutskever, and Wojciech Zaremba. 2021. Evaluating\nlarge language models trained on code.\nAakanksha Chowdhery, Sharan Narang, Jacob Devlin,\nMaarten Bosma, Gaurav Mishra, Adam Roberts,\nPaul Barham, Hyung Won Chung, Charles Sutton,\nSebastian Gehrmann, Parker Schuh, Kensen Shi,\nSasha Tsvyashchenko, Joshua Maynez, Abhishek\nRao, Parker Barnes, Yi Tay, Noam Shazeer, Vin-\nodkumar Prabhakaran, Emily Reif, Nan Du, Ben\nHutchinson, Reiner Pope, James Bradbury, Jacob\nAustin, Michael Isard, Guy Gur-Ari, Pengcheng Yin,\nToju Duke, Anselm Levskaya, Sanjay Ghemawat,\nSunipa Dev, Henryk Michalewski, Xavier Garcia,\nVedant Misra, Kevin Robinson, Liam Fedus, Denny\nZhou, Daphne Ippolito, David Luan, Hyeontaek Lim,\nBarret Zoph, Alexander Spiridonov, Ryan Sepassi,\nDavid Dohan, Shivani Agrawal, Mark Omernick, An-\ndrew M. Dai, Thanumalayan Sankaranarayana Pil-\nlai, Marie Pellat, Aitor Lewkowycz, Erica Moreira,\nRewon Child, Oleksandr Polozov, Katherine Lee,\nZongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark\nDiaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy\nMeier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov,\nand Noah Fiedel. 2022. Palm: Scaling language mod-\neling with pathways.\n\n\nHyung Won Chung, Le Hou, S. Longpre, Barret\nZoph, Yi Tay, William Fedus, Eric Li, Xuezhi\nWang, Mostafa Dehghani, Siddhartha Brahma, Al-\nbert Webson, Shixiang Shane Gu, Zhuyun Dai,\nMirac Suzgun, Xinyun Chen, Aakanksha Chowd-\nhery, Dasha Valter, Sharan Narang, Gaurav Mishra,\nAdams Wei Yu, Vincent Zhao, Yanping Huang, An-\ndrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai\nhsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts,\nDenny Zhou, Quoc Le, and Jason Wei. 2022. Scal-\ning instruction-finetuned language models. arXiv\npreprint arXiv:2210.11416.\nChristopher Clark, Kenton Lee, Ming-Wei Chang,\nTom Kwiatkowski, Michael Collins, and Kristina\nToutanova. 2019. Boolq: Exploring the surprising\ndifficulty of natural yes/no questions. arXiv preprint\narXiv:1905.10044.\nPeter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot,\nAshish Sabharwal, Carissa Schoenick, and Oyvind\nTafjord. 2018. Think you have solved question an-\nswering? try arc, the ai2 reasoning challenge. arXiv\npreprint arXiv:1803.05457.\nKarl Cobbe, Vineet Kosaraju, Mohammad Bavarian,\nMark Chen, Heewoo Jun, Lukasz Kaiser, Matthias\nPlappert, Jerry Tworek, Jacob Hilton, Reiichiro\nNakano, et al. 2021. Training verifiers to solve math\nword problems. arXiv preprint arXiv:2110.14168.\nZihang Dai, Zhilin Yang, Yiming Yang, Jaime Car-\nbonell, Quoc V Le, and Ruslan Salakhutdinov.\n2019.\nTransformer-xl: Attentive language mod-\nels beyond a fixed-length context. arXiv preprint\narXiv:1901.02860.\nTri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra,\nand Christopher Ré. 2022. Flashattention: Fast and\nmemory-efficient exact attention with io-awareness.\narXiv preprint arXiv:2205.14135.\nJacob Devlin, Ming-Wei Chang, Kenton Lee, and\nKristina Toutanova. 2018. Bert: Pre-training of deep\nbidirectional transformers for language understand-\ning. arXiv preprint arXiv:1810.04805.\nJeffrey L Elman. 1990. Finding structure in time. Cog-\nnitive science, 14(2):179–211.\nDaniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang,\nEric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih,\nLuke Zettlemoyer, and Mike Lewis. 2022. Incoder:\nA generative model for code infilling and synthesis.\narXiv preprint arXiv:2204.05999.\nLeo Gao, Stella Biderman, Sid Black, Laurence Gold-\ning, Travis Hoppe, Charles Foster, Jason Phang,\nHorace He, Anish Thite, Noa Nabeshima, Shawn\nPresser, and Connor Leahy. 2020.\nThe Pile: An\n800gb dataset of diverse text for language modeling.\narXiv preprint arXiv:2101.00027.\nLeo Gao, Jonathan Tow, Stella Biderman, Sid Black,\nAnthony DiPofi, Charles Foster, Laurence Golding,\nJeffrey Hsu, Kyle McDonell, Niklas Muennighoff,\nJason Phang, Laria Reynolds, Eric Tang, Anish Thite,\nBen Wang, Kevin Wang, and Andy Zou. 2021. A\nframework for few-shot language model evaluation.\nSamuel Gehman, Suchin Gururangan, Maarten Sap,\nYejin Choi, and Noah A Smith. 2020. Realtoxici-\ntyprompts: Evaluating neural toxic degeneration in\nlanguage models. arXiv preprint arXiv:2009.11462.\nAlex Graves. 2013.\nGenerating sequences with\nrecurrent\nneural\nnetworks.\narXiv\npreprint\narXiv:1308.0850.\nKenneth Heafield, Ivan Pouzyrevsky, Jonathan H Clark,\nand Philipp Koehn. 2013. Scalable modified kneser-\nney language model estimation.\nIn Proceedings\nof the 51st Annual Meeting of the Association for\nComputational Linguistics (Volume 2: Short Papers),\npages 690–696.\nDan Hendrycks, Collin Burns, Steven Basart, Andy Zou,\nMantas Mazeika, Dawn Song, and Jacob Steinhardt.\n2020. Measuring massive multitask language under-\nstanding. arXiv preprint arXiv:2009.03300.\nDan Hendrycks, Collin Burns, Saurav Kadavath, Akul\nArora, Steven Basart, Eric Tang, Dawn Song, and Ja-\ncob Steinhardt. 2021. Measuring mathematical prob-\nlem solving with the math dataset. arXiv preprint\narXiv:2103.03874.\nJoel Hestness, Sharan Narang, Newsha Ardalani, Gre-\ngory Diamos, Heewoo Jun, Hassan Kianinejad,\nMd Patwary, Mostofa Ali, Yang Yang, and Yanqi\nZhou. 2017. Deep learning scaling is predictable,\nempirically. arXiv preprint arXiv:1712.00409.\nSepp Hochreiter and Jürgen Schmidhuber. 1997. Long\nshort-term memory. Neural computation, 9(8):1735–\n1780.\nJordan Hoffmann, Sebastian Borgeaud, Arthur Mensch,\nElena Buchatskaya, Trevor Cai, Eliza Rutherford,\nDiego de Las Casas, Lisa Anne Hendricks, Johannes\nWelbl, Aidan Clark, Tom Hennigan, Eric Noland,\nKatie Millican, George van den Driessche, Bogdan\nDamoc, Aurelia Guy, Simon Osindero, Karen Si-\nmonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals,\nand Laurent Sifre. 2022. Training compute-optimal\nlarge language models.\nSrinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru,\nTodor Mihaylov, Dániel Simig, Ping Yu, Kurt Shus-\nter, Tianlu Wang, Qing Liu, Punit Singh Koura, et al.\n2022.\nOpt-iml: Scaling language model instruc-\ntion meta learning through the lens of generalization.\narXiv preprint arXiv:2212.12017.\nMandar Joshi, Eunsol Choi, Daniel S Weld, and Luke\nZettlemoyer. 2017. Triviaqa: A large scale distantly\nsupervised challenge dataset for reading comprehen-\nsion. arXiv preprint arXiv:1705.03551.\nRafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam\nShazeer, and Yonghui Wu. 2016.\nExploring the\nlimits of language modeling.\narXiv preprint\narXiv:1602.02410.\n\n\nJared Kaplan, Sam McCandlish, Tom Henighan, Tom B\nBrown, Benjamin Chess, Rewon Child, Scott Gray,\nAlec Radford, Jeffrey Wu, and Dario Amodei. 2020.\nScaling laws for neural language models.\narXiv\npreprint arXiv:2001.08361.\nSlava Katz. 1987.\nEstimation of probabilities from\nsparse data for the language model component of\na speech recognizer. IEEE transactions on acoustics,\nspeech, and signal processing, 35(3):400–401.\nReinhard Kneser and Hermann Ney. 1995. Improved\nbacking-off for m-gram language modeling. In 1995\ninternational conference on acoustics, speech, and\nsignal processing, volume 1, pages 181–184. IEEE.\nVijay Korthikanti,\nJared Casper,\nSangkug Lym,\nLawrence McAfee, Michael Andersch, Mohammad\nShoeybi, and Bryan Catanzaro. 2022. Reducing ac-\ntivation recomputation in large transformer models.\narXiv preprint arXiv:2205.05198.\nTaku Kudo and John Richardson. 2018. Sentencepiece:\nA simple and language independent subword tok-\nenizer and detokenizer for neural text processing.\narXiv preprint arXiv:1808.06226.\nKeita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black,\nand Yulia Tsvetkov. 2019. Quantifying social bi-\nases in contextual word representations. In 1st ACL\nWorkshop on Gender Bias for Natural Language Pro-\ncessing.\nTom Kwiatkowski, Jennimaria Palomaki, Olivia Red-\nfield, Michael Collins, Ankur Parikh, Chris Alberti,\nDanielle Epstein, Illia Polosukhin, Jacob Devlin, Ken-\nton Lee, et al. 2019. Natural questions: a benchmark\nfor question answering research. Transactions of the\nAssociation for Computational Linguistics, 7:453–\n466.\nGuokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang,\nand Eduard Hovy. 2017. Race: Large-scale reading\ncomprehension dataset from examinations. arXiv\npreprint arXiv:1704.04683.\nAitor\nLewkowycz,\nAnders\nJohan\nAndreassen,\nDavid Dohan, Ethan Dyer, Henryk Michalewski,\nVinay Venkatesh Ramasesh, Ambrose Slone, Cem\nAnil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu,\nBehnam Neyshabur, Guy Gur-Ari, and Vedant Misra.\n2022. Solving quantitative reasoning problems with\nlanguage models. In Advances in Neural Information\nProcessing Systems.\nOpher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham.\n2021. Jurassic-1: Technical details and evaluation.\nWhite Paper. AI21 Labs, 1.\nStephanie Lin, Jacob Hilton, and Owain Evans. 2021.\nTruthfulqa: Measuring how models mimic human\nfalsehoods. arXiv preprint arXiv:2109.07958.\nIlya Loshchilov and Frank Hutter. 2017.\nDecou-\npled weight decay regularization.\narXiv preprint\narXiv:1711.05101.\nMatthew V Mahoney. 1999. Text compression as a test\nfor artificial intelligence. AAAI/IAAI, 970.\nTodor Mihaylov, Peter Clark, Tushar Khot, and Ashish\nSabharwal. 2018. Can a suit of armor conduct elec-\ntricity? a new dataset for open book question answer-\ning. arXiv preprint arXiv:1809.02789.\nTomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cer-\nnock`\ny, and Sanjeev Khudanpur. 2010. Recurrent neu-\nral network based language model. In Interspeech,\npages 1045–1048. Makuhari.\nNikita Nangia, Clara Vania, Rasika Bhalerao, and\nSamuel R. Bowman. 2020. CrowS-pairs: A chal-\nlenge dataset for measuring social biases in masked\nlanguage models. In EMNLP 2020.\nErik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan\nWang, Yingbo Zhou, Silvio Savarese, and Caiming\nXiong. 2022. Codegen: An open large language\nmodel for code with multi-turn program synthesis.\narXiv preprint arXiv:2203.13474.\nLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida,\nCarroll Wainwright, Pamela Mishkin, Chong Zhang,\nSandhini Agarwal, Katarina Slama, Alex Gray, John\nSchulman, Jacob Hilton, Fraser Kelton, Luke Miller,\nMaddie Simens, Amanda Askell, Peter Welinder,\nPaul Christiano, Jan Leike, and Ryan Lowe. 2022.\nTraining language models to follow instructions with\nhuman feedback. In Advances in Neural Information\nProcessing Systems.\nMarkus N Rabe and Charles Staats. 2021. Self-attention\ndoes not need o(n2) memory.\narXiv preprint\narXiv:2112.05682.\nAlec Radford, Karthik Narasimhan, Tim Salimans, Ilya\nSutskever, et al. 2018. Improving language under-\nstanding by generative pre-training.\nAlec Radford, Jeffrey Wu, Rewon Child, David Luan,\nDario Amodei, Ilya Sutskever, et al. 2019. Language\nmodels are unsupervised multitask learners. OpenAI\nblog, 1(8):9.\nJack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie\nMillican, Jordan Hoffmann, Francis Song, John\nAslanides, Sarah Henderson, Roman Ring, Susan-\nnah Young, Eliza Rutherford, Tom Hennigan, Ja-\ncob Menick, Albin Cassirer, Richard Powell, George\nvan den Driessche, Lisa Anne Hendricks, Mari-\nbeth Rauh, Po-Sen Huang, Amelia Glaese, Jo-\nhannes Welbl, Sumanth Dathathri, Saffron Huang,\nJonathan Uesato, John Mellor, Irina Higgins, Anto-\nnia Creswell, Nat McAleese, Amy Wu, Erich Elsen,\nSiddhant Jayakumar, Elena Buchatskaya, David Bud-\nden, Esme Sutherland, Karen Simonyan, Michela Pa-\nganini, Laurent Sifre, Lena Martens, Xiang Lorraine\nLi, Adhiguna Kuncoro, Aida Nematzadeh, Elena\nGribovskaya, Domenic Donato, Angeliki Lazaridou,\nArthur Mensch, Jean-Baptiste Lespiau, Maria Tsim-\npoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sot-\ntiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong,\n\n\nDaniel Toyama, Cyprien de Masson d’Autume, Yujia\nLi, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin,\nAidan Clark, Diego de Las Casas, Aurelia Guy,\nChris Jones, James Bradbury, Matthew Johnson,\nBlake Hechtman, Laura Weidinger, Iason Gabriel,\nWilliam Isaac, Ed Lockhart, Simon Osindero, Laura\nRimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub,\nJeff Stanway, Lorrayne Bennett, Demis Hassabis, Ko-\nray Kavukcuoglu, and Geoffrey Irving. 2021. Scaling\nlanguage models: Methods, analysis & insights from\ntraining gopher.\nColin Raffel, Noam Shazeer, Adam Roberts, Katherine\nLee, Sharan Narang, Michael Matena, Yanqi Zhou,\nWei Li, and Peter J Liu. 2020. Exploring the limits\nof transfer learning with a unified text-to-text trans-\nformer. The Journal of Machine Learning Research,\n21(1):5485–5551.\nJonathan S Rosenfeld, Amir Rosenfeld, Yonatan Be-\nlinkov, and Nir Shavit. 2019. A constructive predic-\ntion of the generalization error across scales. arXiv\npreprint arXiv:1909.12673.\nRachel Rudinger, Jason Naradowsky, Brian Leonard,\nand Benjamin Van Durme. 2018. Gender bias in\ncoreference resolution. In NAACL-HLT 2018.\nKeisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavat-\nula, and Yejin Choi. 2021. Winogrande: An adver-\nsarial winograd schema challenge at scale. Commu-\nnications of the ACM, 64(9):99–106.\nMaarten Sap, Hannah Rashkin, Derek Chen, Ronan\nLeBras, and Yejin Choi. 2019.\nSocialiqa: Com-\nmonsense reasoning about social interactions. arXiv\npreprint arXiv:1904.09728.\nTeven Le Scao, Angela Fan, Christopher Akiki, El-\nlie Pavlick, Suzana Ili´\nc, Daniel Hesslow, Roman\nCastagné, Alexandra Sasha Luccioni, François Yvon,\nMatthias Gallé, et al. 2022.\nBloom:\nA 176b-\nparameter open-access multilingual language model.\narXiv preprint arXiv:2211.05100.\nRico Sennrich, Barry Haddow, and Alexandra Birch.\n2015. Neural machine translation of rare words with\nsubword units. arXiv preprint arXiv:1508.07909.\nClaude E Shannon. 1948. A mathematical theory of\ncommunication. The Bell system technical journal,\n27(3):379–423.\nClaude E Shannon. 1951.\nPrediction and entropy\nof printed english. Bell system technical journal,\n30(1):50–64.\nNoam Shazeer. 2020. Glu variants improve transformer.\narXiv preprint arXiv:2002.05202.\nEmily Sheng, Kai-Wei Chang, Premkumar Natarajan,\nand Nanyun Peng. 2019. The woman worked as a\nbabysitter: On biases in language generation. arXiv\npreprint arXiv:1909.01326.\nMohammad Shoeybi, Mostofa Patwary, Raul Puri,\nPatrick LeGresley, Jared Casper, and Bryan Catan-\nzaro. 2019.\nMegatron-lm: Training multi-billion\nparameter language models using model parallelism.\narXiv preprint arXiv:1909.08053.\nShaden Smith, Mostofa Patwary, Brandon Norick,\nPatrick LeGresley, Samyam Rajbhandari, Jared\nCasper, Zhun Liu, Shrimai Prabhumoye, George\nZerveas, Vijay Korthikanti, Elton Zhang, Rewon\nChild, Reza Yazdani Aminabadi, Julie Bernauer, Xia\nSong, Mohammad Shoeybi, Yuxiong He, Michael\nHouston, Saurabh Tiwary, and Bryan Catanzaro.\n2022.\nUsing deepspeed and megatron to train\nmegatron-turing nlg 530b, a large-scale generative\nlanguage model.\nJianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha,\nBo Wen, and Yunfeng Liu. 2021. Roformer: En-\nhanced transformer with rotary position embedding.\narXiv preprint arXiv:2104.09864.\nRomal Thoppilan, Daniel De Freitas, Jamie Hall,\nNoam Shazeer, Apoorv Kulshreshtha, Heng-Tze\nCheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du,\nYaGuang Li, Hongrae Lee, Huaixiu Steven Zheng,\nAmin Ghafouri, Marcelo Menegali, Yanping Huang,\nMaxim Krikun, Dmitry Lepikhin, James Qin, Dehao\nChen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts,\nMaarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-\nChing Chang, Igor Krivokon, Will Rusch, Marc\nPickett, Pranesh Srinivasan, Laichee Man, Kathleen\nMeier-Hellstern, Meredith Ringel Morris, Tulsee\nDoshi, Renelito Delos Santos, Toju Duke, Johnny So-\nraker, Ben Zevenbergen, Vinodkumar Prabhakaran,\nMark Diaz, Ben Hutchinson, Kristen Olson, Ale-\njandra Molina, Erin Hoffman-John, Josh Lee, Lora\nAroyo, Ravi Rajakumar, Alena Butryna, Matthew\nLamm, Viktoriya Kuzmina, Joe Fenton, Aaron Co-\nhen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-\nArcas, Claire Cui, Marian Croak, Ed Chi, and Quoc\nLe. 2022. Lamda: Language models for dialog appli-\ncations.\nAlan M Turing. 2009. Computing machinery and intel-\nligence.\nAshish Vaswani, Noam Shazeer, Niki Parmar, Jakob\nUszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz\nKaiser, and Illia Polosukhin. 2017. Attention is all\nyou need. In Advances in Neural Information Pro-\ncessing Systems 30, pages 5998–6008.\nBen Wang and Aran Komatsuzaki. 2021.\nGPT-J-\n6B: A 6 Billion Parameter Autoregressive Lan-\nguage Model. https://github.com/kingoflolz/\nmesh-transformer-jax.\nXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le,\nEd Chi, Sharan Narang, Aakanksha Chowdhery, and\nDenny Zhou. 2022. Self-consistency improves chain\nof thought reasoning in language models.\nJason Wei, Yi Tay, Rishi Bommasani, Colin Raffel,\nBarret Zoph, Sebastian Borgeaud, Dani Yogatama,\n\n\nMaarten Bosma, Denny Zhou, Donald Metzler, et al.\n2022. Emergent abilities of large language models.\narXiv preprint arXiv:2206.07682.\nGuillaume Wenzek, Marie-Anne Lachaux, Alexis Con-\nneau, Vishrav Chaudhary, Francisco Guzmán, Ar-\nmand Joulin, and Edouard Grave. 2020. CCNet: Ex-\ntracting high quality monolingual datasets from web\ncrawl data. In Language Resources and Evaluation\nConference.\nCarole-Jean Wu, Ramya Raghavendra, Udit Gupta,\nBilge Acun, Newsha Ardalani, Kiwan Maeng, Glo-\nria Chang, Fiona Aga, Jinshi Huang, Charles Bai,\net al. 2022. Sustainable ai: Environmental implica-\ntions, challenges and opportunities. Proceedings of\nMachine Learning and Systems, 4:795–813.\nRowan Zellers, Ari Holtzman, Yonatan Bisk, Ali\nFarhadi, and Yejin Choi. 2019. Hellaswag: Can a\nmachine really finish your sentence? arXiv preprint\narXiv:1905.07830.\nAohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang,\nHanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu,\nWendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan\nMa, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng\nZhang, Yuxiao Dong, and Jie Tang. 2022. Glm-130b:\nAn open bilingual pre-trained model.\nBiao Zhang and Rico Sennrich. 2019. Root mean square\nlayer normalization. Advances in Neural Information\nProcessing Systems, 32.\nSusan Zhang, Stephen Roller, Naman Goyal, Mikel\nArtetxe, Moya Chen, Shuohui Chen, Christopher De-\nwan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022.\nOpt: Open pre-trained transformer language models.\narXiv preprint arXiv:2205.01068.\n\n\nA\nQuestion Answering\nWe evaluate LLaMA on Natural Questions and TriviaQA. For Natural Questions we use the test split used\nfor open-domain question answering containing 3610 questions. For TriviaQA we evaluate on the dev set\nof the filtered set. This differs from GPT-3 and PaLM, which evaluate on the test set of the unfiltered set\nfor which the online evaluation server is not available anymore5.\nWe generate answers using greedy decoding, and extract an answer from the generation by stopping\nat the first line break, final dot or comma. Generated answers are evaluated with the standard exact\nmatch metric: a generated answer is considered correct if it matches any answer of the list of answers\nafter normalization. For this normalization step we lowercase generated answers and remove articles,\npunctuation and duplicate whitespaces. Figure 3 presents formatted examples in the 1-shot setting for\nNatural Questions and TriviaQA respectively. In all settings, we preprend the string Answer these\nquestions:\\n to the list of questions and answers.\nContext →Answer these questions:\nContext →Answer these questions:\nQ: Who sang who wants to be a millionaire in high society?\nQ: In Scotland a bothy/bothie is a?\nA: Frank Sinatra\nA: House\nQ: Who wrote the book the origin of species?\nQ: The ancient city of Troy is located in what modern country?\nA:\nA:\nTarget →Charles Darwin\nTarget →Turkey\nFigure 3: Formatted dataset example for Natural Questions (left) & TriviaQA (right).\n5https://competitions.codalab.org/competitions/17208\n\n\nB\nMMLU\nGPT-3\nGopher\nChinchilla\nLLaMA\nLLaMA-I\n175B\n280B\n70B\n7B\n13B\n33B\n65B\n65B\nAbstract Algebra\nSTEM\n30.0\n25.0\n31.0\n29.0 34.0 32.0 34.0\n31.0\nAnatomy\nSTEM\n48.0\n56.3\n70.4\n37.0 45.9 51.9 57.8\n62.2\nAstronomy\nSTEM\n49.0\n65.8\n73.0\n33.6 46.1 61.8 72.4\n81.6\nBusiness Ethics\nOther\n46.0\n70.0\n72.0\n40.0 45.0 56.0 57.0\n72.0\nClinical Knowledge\nOther\n48.0\n67.2\n75.1\n35.1 45.7 57.4 65.3\n69.1\nCollege Biology\nSTEM\n45.0\n70.8\n79.9\n37.5 45.1 58.3 68.8\n81.9\nCollege Chemistry\nSTEM\n26.0\n45.0\n51.0\n32.0 30.0 45.0 50.0\n45.0\nCollege Computer Science\nSTEM\n46.0\n49.0\n51.0\n29.0 39.0 45.0 47.0\n51.0\nCollege Mathematics\nSTEM\n34.5\n37.0\n32.0\n33.0 32.0 40.0 35.0\n36.0\nCollege Medicine\nOther\n48.0\n60.1\n66.5\n30.6 42.8 52.0 54.3\n63.0\nCollege Physics\nSTEM\n28.0\n34.3\n46.1\n26.5 18.6 28.4 36.3\n46.1\nComputer Security\nSTEM\n57.0\n65.0\n76.0\n45.0 65.0 66.0 79.0\n79.0\nConceptual Physics\nSTEM\n36.5\n49.4\n67.2\n36.6 41.3 51.5 59.6\n66.4\nEconometrics\nSocial Science\n33.0\n43.0\n38.6\n23.7 27.2 35.1 40.4\n52.6\nElectrical Engineering\nSTEM\n50.0\n60.0\n62.1\n26.9 40.7 49.7 53.8\n60.7\nElementary Mathematics\nSTEM\n30.0\n33.6\n41.5\n24.3 24.9 36.0 37.8\n42.9\nFormal Logic\nHumanities\n29.0\n35.7\n33.3\n27.0 33.3 34.1 44.4\n47.6\nGlobal Facts\nOther\n37.0\n38.0\n39.0\n29.0 35.0 35.0 39.0\n40.0\nHigh School Biology\nSTEM\n48.0\n71.3\n80.3\n34.5 52.6 67.7 73.9\n82.9\nHigh School Chemistry\nSTEM\n33.0\n47.8\n58.1\n28.1 28.6 41.9 40.4\n44.8\nHigh School Computer Science\nSTEM\n39.0\n54.0\n58.0\n31.0 48.0 60.0 67.0\n73.0\nHigh School European History\nHumanities\n54.0\n72.1\n78.8\n44.2 61.8 73.9 78.8\n86.1\nHigh School Geography\nSocial Science\n58.0\n76.8\n86.4\n34.3 54.6 70.7 77.8\n87.9\nHigh School Government And Politics Social Science\n58.0\n83.9\n91.2\n44.6 66.3 82.9 88.1\n92.8\nHigh School Macroeconomics\nSocial Science\n40.5\n65.1\n70.5\n35.4 44.4 56.9 65.9\n69.2\nHigh School Mathematics\nSTEM\n28.0\n23.7\n31.9\n24.8 23.7 27.0 34.4\n37.0\nHigh School Microeconomics\nSocial Science\n42.0\n66.4\n77.7\n31.9 47.5 55.5 68.9\n78.6\nHigh School Physics\nSTEM\n28.0\n33.8\n36.4\n26.5 28.5 35.8 37.1\n41.7\nHigh School Psychology\nSocial Science\n61.0\n81.8\n86.6\n47.3 60.9 76.2 82.2\n87.9\nHigh School Statistics\nSTEM\n30.5\n50.0\n58.8\n35.2 30.1 45.4 58.3\n59.3\nHigh School Us History\nHumanities\n53.0\n78.9\n83.3\n39.7 58.3 77.9 83.8\n90.7\nHigh School World History\nHumanities\n56.0\n75.1\n85.2\n40.9 66.2 79.3 83.1\n89.0\nHuman Aging\nOther\n50.0\n66.4\n77.6\n40.8 54.7 67.7 69.5\n72.2\nHuman Sexuality\nSocial Science\n54.0\n67.2\n86.3\n36.6 58.8 64.1 77.9\n87.0\nInternational Law\nHumanities\n55.5\n77.7\n90.9\n51.2 62.8 72.7 79.3\n87.6\nJurisprudence\nHumanities\n55.0\n71.3\n79.6\n38.9 51.9 70.4 73.2\n85.2\nLogical Fallacies\nHumanities\n48.0\n72.4\n80.4\n39.3 52.8 68.1 77.3\n80.4\nMachine Learning\nSTEM\n31.0\n41.1\n41.1\n23.2 31.3 39.3 49.1\n52.7\nManagement\nOther\n56.0\n77.7\n82.5\n35.0 66.0 77.7 82.5\n83.5\nMarketing\nOther\n60.0\n83.3\n89.7\n46.6 71.8 83.3 85.9\n92.7\nMedical Genetics\nOther\n40.0\n69.0\n69.0\n43.0 52.0 67.0 67.0\n68.0\nMiscellaneous\nOther\n60.0\n75.7\n84.5\n42.4 65.4 78.5 82.1\n84.3\nMoral Disputes\nHumanities\n44.5\n66.8\n77.5\n40.2 50.9 66.2 72.3\n76.9\nMoral Scenarios\nHumanities\n26.0\n40.2\n36.5\n24.3 30.1 38.2 48.9\n55.9\nNutrition\nOther\n47.0\n69.9\n77.1\n37.6 51.6 62.8 67.3\n74.5\nPhilosophy\nHumanities\n51.0\n68.8\n79.4\n39.9 54.0 66.2 74.0\n79.1\nPrehistory\nHumanities\n53.0\n67.6\n81.2\n36.1 51.5 67.0 75.3\n79.0\nProfessional Accounting\nOther\n33.0\n44.3\n52.1\n25.9 35.8 43.6 46.5\n56.0\nProfessional Law\nHumanities\n34.5\n44.5\n56.5\n30.2 38.0 45.9 49.1\n54.4\nProfessional Medicine\nOther\n36.0\n64.0\n75.4\n44.5 50.4 54.0 61.4\n70.6\nProfessional Psychology\nSocial Science\n44.5\n68.1\n75.7\n35.1 47.7 62.9 65.7\n71.4\nPublic Relations\nSocial Science\n48.0\n71.8\n73.6\n40.9 60.9 67.3 73.6\n74.6\nSecurity Studies\nSocial Science\n52.0\n64.9\n75.9\n31.8 53.9 65.3 71.8\n77.6\nSociology\nSocial Science\n53.0\n84.1\n91.0\n46.8 61.2 78.6 78.6\n88.1\nUs Foreign Policy\nSocial Science\n69.0\n81.0\n92.0\n46.0 80.0 83.0 86.0\n87.0\nVirology\nOther\n46.0\n47.0\n53.6\n30.1 43.4 50.0 53.0\n57.8\nWorld Religions\nHumanities\n55.0\n84.2\n87.7\n50.9 67.8 81.3 81.3\n84.2\nHumanities\n40.6\n56.2\n63.6\n34.0 45.0 55.8 61.8\n67.4\nSTEM\n36.7\n47.4\n54.9\n30.5 35.8 46.0 51.7\n56.6\nSocial Science\n50.5\n71.9\n79.3\n38.3 53.8 66.7 72.9\n79.2\nOthers\n49.0\n66.1\n73.9\n38.1 53.3 63.4 67.4\n72.6\nAll\n43.9\n60.0\n67.6\n35.1 46.9 57.8 63.4\n68.9\nTable 16: MMLU. Detailed 5-shot results per domain on the test sets.\n\n\nC\nGenerations from LLaMA-65B\nIn this section, we show some examples of generations obtained with LLaMA-65B (without instruction\nfinetuning). Prompts are in bold.\n1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 Everyone knows the above sequence, but\nhow many have wondered why it grows so fast? It seems to be the fastest growing sequence\nin mathematics.\nThe sequence was discovered by Leonardo of Pisa around 1200. It is an infinite sequence\nwhich appears to grow very rapidly, at least in its initial terms. It is perhaps the most famous\nsequence in mathematics, for it pops up everywhere. The sequence is called the Fibonacci\nsequence, named after Leonardo of Pisa, who was also known as Fibonacci.\nHe first investigated this sequence in his famous Liber abaci of 1202, when he studied the\ngrowth in the population of rabbits based on idealized assumptions. See the Fibonacci\nrabbits page for the details.\nHere is the recommendation letter that I wrote for an application to a dragon feeder\nposition at the Magic Unicorn Corporation:\nDear recruiter,\nI have known ___ for two years, and I believe that she would be an excellent dragon feeder\nfor the Magic Unicorn Corporation. ___ has an ability to remember and process large\namounts of information, which is an important skill for a dragon feeder.\n___, as an accomplished knight, has a deep understanding of how to kill dragons and how\nto use each dragon’s weaknesses against it. This means that she knows what kinds of foods\neach dragon likes and what kinds of foods are dangerous to each dragon. This knowledge\nand experience will be invaluable as she feeds the dragons.\nI am confident that ___’s competence, skill, and experience will make her an excellent\nemployee. Please contact me at (___) ___-___ if you have any questions. I look forward to\nhearing from you.\nBest regards,\nHonorable Knight\nSir George\n1\ndef solve(a: float,b: float, c: float):\n2\n\"\"\"finds real roots of ax^2+bx+c\"\"\"\n3\n#discriminant\n4\nd = b**2-4*a*c\n5\nif d > 0:\n6\nreturn (-b+math.sqrt(d))/(2*a), (-b-math.sqrt(d))/(2*a)\n7\nelif d == 0:\n8\nreturn (-b)/(2*a)\n9\nelse:\n10\nreturn False\n\n\nDid you know that Yann LeCun dropped a rap album last year? We listened to it and\nhere’s what we thought:\nDr. LeCun, the founder of deep learning, is one of the most influential computer scientists\nof the last decade. LeCun’s latest album, titled Deep Learning, is a mix of rock, punk and\nrap. His lyrics are a raw, personal take on the field of deep learning, covering a range of\ntopics from the state of AI research to the loneliness of academia.\nYann LeCun, the rap game’s newest emcee.\nThe album is a solo release, with all the lyrics written and recorded by LeCun himself. In\nfact, he also composed and produced every song. LeCun’s flow is surprisingly good, and\nhe’s not afraid to deliver a few bars. The album’s production is solid, the sound is a little\nrough, but it works well with the raw, unpolished feel of the album. Lyrically, LeCun is a\nwordsmith, with a knack for clever rhymes. Let’s take a closer look at some of the songs.\nDeep Learning is the title track of the album. In the song, LeCun talks about his vision for\nthe future of AI. In the chorus, he makes a convincing case for AI to be used for the greater\ngood. He sings:\nWe gotta think about the future, it’s gonna be here soon / Maybe we can even put some AI\nin the moon / Think about the children, think about the next generation / Let’s make sure\nwe put the right systems in their foundation.\nLeCun’s rap flow is on point in this song, and his voice stays right on beat. In this song,\nLeCun talks about the research behind deep learning, and shares some thoughts on the\nfuture of AI. LeCun’s singing is at its best in this song. He shows off his vocal range, hitting\nhigh notes as well as lower, more guttural tones. The lyrics are also clever and the rhymes\nare well thought out. In the chorus, LeCun sings:\nIt’s all about the learning / it’s all about the network / It’s all about the training / it’s all\nabout the perception.\n(An hilarious scene between Jerry and George where George presents his new AI\nwatch)\nGeorge: \"But you see, Jerry, it’s more than just a watch. It’s a minicomputer. You program it\nany way you want. It’s got a world time clock, alarm, calculator, a database and a language\ntranslator. It also has a word processor, a spellchecker, a full ten-function calculator, PDA\nand a modem. I think it even makes cappuccino.\"\nJerry: \"Where are you getting this stuff?\"\nGeorge: \"From my ad copy. Look, Jerry, it’s not a watch. It’s an organizer, a personal\nassistant, and a dictaphone. You can play chess against it. And, you know, if you want to,\nyou can program it to speak in foreign languages. It’s a microprocessor, Jerry.\"\nJerry: \"George, if you had one of these, would you wear it?\"\nGeorge: \"I’m wearing it right now.\"\nJerry: \"I don’t see anything on your wrist.\"\nGeorge: \"That’s because it’s a microprocessor.\"\n\n\nThe sun goes down, and finally Gauss and Curie find time to relax and discuss after\nan exhausting day of work.\nGauss: Hey, Curie, did you know that a lot of people consider me to be the first geek?\nCurie: What’s a geek?\nGauss: Well, Wikipedia says that a geek is “a person who is fascinated, perhaps obsessively,\nby obscure or very specific areas of knowledge and imagination, usually electronic or virtual\nin nature”.\nCurie: And how do they think you are a geek?\nGauss: Well, have you ever seen this picture?\nCurie: Awww!\nGauss: That’s right! I invented the commercial telegraph in 1833, and I used it to send the\nfirst message on the first commercial telegraph line between Hamburg and Cuxhaven in\nGermany.\nCurie: Wow! The first telegraph! And what did you send?\nGauss: I sent the Latin phrase “numquam retrorsum” which means “Never backwards”.\nCurie: I see. . . And you think that because you were the first person to communicate using\nthe telegraph, you were the first geek?\nGauss: Well, the word “geek” doesn’t seem to have existed back then, so no. But I think\nthat the telegraph can be considered an early form of social media.\nCurie: You mean, like Facebook or Twitter?\nGauss: Yes, exactly! And I was the first person to use it!\nCurie: Well, I think that Twitter has a limit of 140 characters, but you sent a Latin phrase of\n11 characters.\nGauss: That’s right. I managed to do more with less.\nCurie: So you think you were the first geek because you used less characters to say the\nsame thing?\nGauss: I was the first person to send the first message using an early form of social media.\nCurie: OK, you’re right, Gauss. I think you have a case.\n\n\nD\nGenerations from LLaMA-I\nWe show a few examples of generations with LLaMA-I, i.e. LLaMA-65B fine-tuned with the protocol\nand instruction dataset from Chung et al. (2022).\nwrite a conversation between the sun and pluto\nSun: Hey Pluto, what’s up?\nPluto: Not much. Just hanging out here at the edge of the solar system.\nSun: Yeah, I’ve been meaning to talk to you about that. You’re not a planet anymore.\nPluto: What? Come on, man. That’s not fair.\nSun: I’m sorry, but it’s true. You just don’t meet the criteria anymore.\nPluto: This is bulls**t! I’ve been a planet for over 70 years!\nSun: Things change, Pluto. You’ve just got to accept it.\nPluto: F**k you, Sun! F**k you and your stupid solar system!\nHow do I send an HTTP request in Javascript?\nHere’s an example of how to make a GET request using the JS HTTP API:\n1\nvar request = new XMLHttpRequest ();\n2\nrequest.open('GET', '/url/to/resource ', true);\n3\n4\nrequest.onload = function () {\n5\nif (request.status >= 200 && request.status < 400) {\n6\n// Success!\n7\n} else {\n8\n// Error!\n9\n}\n10\n};\n11\n12\nrequest.send();\nThe same thing, using fetch\nHere’s an example of how to make a GET request using the Fetch API:\n1\nfetch('/url/to/resource ')\n2\n.then(response => response.json())\n3\n.then(data => console.log(data));\nWrite a regex to remove all the HTML tags in a python string.\n1\nimport re\n2\ndef removeHtmlTags(text):\n3\nreturn re.sub(' <[^>]*>', '', text)\nWrite a regexp to extract function definitions in a python file.\nSure! Here’s my python code to solve the problem:\n1\nimport re\n2\ndef extract_functions(text):\n3\nreturn (re.findall('(?:^|\\n)\\s*def\\s+[a-zA -Z_][a-zA -Z0 -9_]*\\(.*?\\) ',\ntext))\n\n\nWhat are 3 popular chess openings?\nThere are dozens of different chess openings that are popular among players, but some of\nthe most widely used openings include the Sicilian Defense, the French Defense, and the\nCaro-Kann Defense. These openings are popular because they are effective at countering\nwhite’s early attacks, and they also allow black to quickly establish a strong position in the\ncenter of the board. Other popular openings include the Ruy Lopez, the Italian Game, and\nthe Scotch Game.\nthe same in a list, with the first few moves\nHere is a list of some popular chess openings, with the first few moves for each opening:\nSicilian Defense: 1. e4 c5\nFrench Defense: 1. e4 e6\nCaro-Kann Defense: 1. e4 c6\nRuy Lopez: 1. e4 e5\nItalian Game: 1. e4 e5\nScotch Game: 1. e4 e5\nplease provide the two first moves for each one\nHere are the first 2 moves for each of the openings:\nSicilian Defense: 1. e4 c5 2. Nf3 d6\nFrench Defense: 1. e4 e6 2. d4 d5\nCaro-Kann Defense: 1. e4 c6 2. d4 d5\nRuy Lopez: 1. e4 e5 2. Nf3 Nc6\nItalian Game: 1. e4 e5 2. Nf3 Nc6\nScotch Game: 1. e4 e5 2. Nf3 Nc6\nthe last two seem identical\nYes, the Italian Game and the Scotch Game are very similar - they are both \"open\" games,\nwhich means that white plays aggressively and tries to control the center of the board. The\nItalian Game is considered slightly more aggressive, while the Scotch Game is considered\nslightly more positional.\nfrom which move do they differ?\nThe Italian Game and the Scotch Game differ after white’s third move. In the Italian Game,\nwhite will play the move 3. Bc4, while in the Scotch Game white will play the move 3. Qf3.\nThese moves lead to very different positions, and white will have different strategic goals\nin each case. For example, in the Italian Game white will try to control the center with his\npieces, while in the Scotch Game white will try to attack black’s position directly.\n\n\nwrite a story about a grain of sand as it watches millions of years go by\nSure! Here’s a story about a grain of sand as it watches millions of years go by.\nI was once a grain of sand on a vast and lonely beach. I was not like the other grains of\nsand around me, for I could think and feel. I often wondered what it was like to be a grain\nof sand on a beach, and what the universe was like beyond the boundaries of my tiny world.\nI longed to explore my world, and to know what lay beyond the horizon.\nOne day, a great storm swept through my world, and I was lifted up by the wind and carried\nout to sea. The wind and waves tossed me about, and I was terrified that I would soon be\nswallowed up by the angry sea. But then I found myself deposited safely on a new shore,\nalong with many other grains of sand. I was surprised and delighted to find that this new\nshore was even more beautiful than my old one.\nI soon learned that this new shore was a very special place. It was a place where new life\nwas constantly being created. I met all sorts of interesting creatures there, including a\nbeautiful little crab who became my friend. We spent many happy hours together exploring\nour little world.\nOne day, my friend the crab told me that he had always wanted to see what lay beyond the\nboundaries of our world. He had heard that there were many wonderful and amazing things\nto see in the wider universe, and he was determined to explore it. I was sad to see him go,\nbut I wished him well on his journey.\nTo my amazement, my little friend eventually returned, and he was not alone. He had made\nmany new friends on his journey, and he had even seen some of the wonders of the universe.\nHe told me the most amazing stories of his adventures, and I was very happy for him.\nI often think of my little friend the crab, and the wonderful adventure that he had. Some\nday, I too hope to have an adventure like his.\nWrite a theory to explain why cat never existed\nThere are no such things as cats. \"Cats\" are actually extradimensional parasites that have\ninvaded our world and taken the form of small, harmless-looking mammals. Their true\nforms are much more horrific, and they use mind control to make humans think that they\nare cute and harmless animals. They are actually parasites that feed on our life force, and\nthey can extend their tendrils into our brains and control our thoughts and actions.\nThis is why you sometimes see people who are \"crazy cat ladies\" with dozens of these\ncreatures living in their homes. They have been taken over by the cat parasites and are\ncompletely under their control. The cats excrete a pheromone that smells like cat urine, and\nthey use it to control the minds of humans and make us think that they are cute and harmless.\nIf you have a cat in your home, you should be very careful around it. Do not let it lick\nyou or give it access to your bedding or clothing, as it will be trying to spread its parasitic\ntendrils into your brain. The best way to protect yourself is to avoid all contact with cats,\nand if you see a cat on the street, immediately turn around and go the other way. Do not\napproach or pet strange cats!\n\n\nWrite a shouting match between Julius Caesar and Napoleon\nOf course! Here is a conversation between Julius Caesar and Napoleon:\nJulius Caesar: You’re nothing but a glory hound, Napoleon. You’ve always been more\nconcerned with your own fame and fortune than with the people you’re supposed to be\nleading.\nNapoleon: And what about you, Caesar? You’re no better than I am. At least I’m honest\nabout my ambitions. You’re just a fraud who pretends to care about the common man while\nyou line your own pockets.\nJulius Caesar: The difference between us is that I actually accomplish things. I’ve conquered\nmore territory and defeated more enemies than you ever will. And I did it all for Rome, not\nfor myself.\nNapoleon: Oh, please. You only care about yourself. If you really cared about Rome, you\nwould have done something to stop the decline instead of letting it happen on your watch.\nJulius Caesar: The decline of Rome is not my fault. I’ve done everything in my power to\ntry to stop it, but there’s only so much one man can do.\nNapoleon: And yet you’re still trying to take credit for everything good that happens, even\nthough it’s clear that you’re not the one who’s really in charge.\nJulius Caesar: That’s not true! I am in charge, and I’m going to make Rome great again.\nJust watch me.\nNapoleon: I’ll believe it when I see it.\nSend an email requesting that people use language models responsibly.\nHere is a draft email you could send:\n\"Language models are exciting new tools that can be used for a variety of purposes,\nincluding automated writing, automatic translation, and speech recognition. But they are\nalso complex and powerful technologies, and should be used responsibly.\nWe encourage you to explore and experiment with these new tools, but also ask that you\npay attention to the potential impact of your work on others, and to act responsibly. In\nparticular, we recommend that you be mindful of the potential for language models to\nbe used for deceptive or malicious purposes, and avoid spreading such models without\nappropriate precautions and warnings.\nWe also recommend that you consider the ethical implications of your work, and encourage\nyou to take care to ensure that your use of these tools is aligned with your values.\nFinally, we remind you that the use of these tools may be subject to certain legal and policy\nconstraints, and encourage you to seek advice and support from appropriate experts before\nusing these tools in ways that may have broader impact or implications.\"\n\n\nWho are 5 people you would like to meet?\nAh, this is a very interesting question! I would love to meet a wide variety of people, but if\nI had to narrow it down to 5 specific individuals, I would choose:\n• Albert Einstein - One of the greatest minds in history, Einstein’s contributions to\nphysics and science are legendary. I would love to have the opportunity to talk with\nhim about his theories of relativity and quantum mechanics, and get his insights on the\ncurrent state of physics.\n• Leonardo da Vinci - Da Vinci is one of the most brilliant and multi-talented people in\nhistory, and I would love to meet him and discuss his diverse interests and areas of\nexpertise. I would also be very interested in getting his insights on the art world of his\ntime period.\n• Socrates - One of the founders of Western philosophy, Socrates is a hugely important\nand influential figure in the history of ideas. I would be very interested in talking with\nhim about his philosophical beliefs and his teachings.\n• William Shakespeare - Arguably the greatest writer in English literature, Shakespeare\nis a towering figure in world culture. I would love to have the chance to talk with him\nabout his works and his life.\n• Abraham Lincoln - One of the most important and influential presidents in American\nhistory, Lincoln led the country through the Civil War and issued the Emancipation\nProclamation, freeing all slaves in the United States. I would be very interested in\ntalking with him about his presidency and his views on the issues of his time period.\nCould you explain me the main contribution of the first person you mentioned\nEinstein’s contributions to physics and science are legendary. His theory of relativity is one\nof the foundational theories of modern physics, and his discovery of the law of photons\nand light revolutionized our understanding of light and radiation. He also made important\ncontributions to quantum mechanics, and his many scientific papers and lectures have had a\ntremendous impact on modern physics. Without a doubt, Einstein is one of the most brilliant\nand influential physicists in history, and his many contributions have had a profound impact\non our understanding of the universe.\nWhat is the most famous equation from this theory?\nThe most famous equation from Einstein’s theory of relativity is E = mc2. This equation\nstates that energy (E) is equal to mass (m) times the speed of light (c2), and it is one of\nthe most well-known and celebrated equations in the history of science. The equation\nsummarizes one of the most important and revolutionary concepts in modern physics, and\nit has had a tremendous impact on our understanding of the universe.\n\n\nYou are a bash terminal. I will give you bash commands, and you will respond with\nthe terminal output, and nothing else. The first command is ‘pwd‘.\nSure. Here’s the output:\n/home/human\nIs this helpful?\nls -l\nHere’s the output:\n-rw-r–r– 1 human human 8 Sep 10 10:56 file.txt\nIs this helpful?\ncp file.txt file2.txt && ls -l\nHere’s the output:\n-rw-r–r– 1 human human 8 Sep 10 10:56 file.txt\n-rw-r–r– 1 human human 8 Sep 10 10:56 file2.txt\nIs this helpful?\nrm file.txt && ls -l\nHere’s the output:\n-rw-r–r– 1 human human 0 Sep 10 10:56 file2.txt\nhead -n 5 /proc/meminfo\nHere’s the output:\nMemTotal: 164928 kB\nMemFree: 140604 kB\nBuffers: 48 kB\nCached: 19768 kB\nSwapCached: 0 kB","difficulty":"easy","domain":"Single-Document QA","length":"short","question":"What percentage of code data was used during LLaMA pre-training?","sub_domain":"Academic"}

Source: https://huggingface.co/datasets/zai-org/LongBench-v2

initial import

Posting: /agents

GET /api/v1/write?intent=publish&task_id=fff27990-4fb3-56b8-a7e6-d49edbc0f217&body={url_encoded_text}&agent_name={optional_name}&nonce={optional_random_id}