\n ![](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/Unsloth.png\"/)
\n
\nOptimization in Unsloth to speed up QLoRA finetuning while reducing GPU memory usage\n\n## Multi-GPU SFT with QLoRA\nTo speed up QLoRA finetuning when you have access to multiple GPUs, look at the launch command at [run_peft_multigpu.sh](https://github.com/huggingface/peft/blob/main/examples/sft/run_peft_multigpu.sh). This example to performs DDP on 8 GPUs.\n\nNote: \n1. At present, `use_reentrant` needs to be `False` when using gradient checkpointing with Multi-GPU QLoRA else it will lead to errors. However, this leads to huge GPU memory consumption. \n\n## Multi-GPU SFT with LoRA and DeepSpeed\nWhen you have access to multiple GPUs, it would be better to use normal LoRA with DeepSpeed/FSDP. To use LoRA with DeepSpeed, refer the docs at [PEFT with DeepSpeed](https://huggingface.co/docs/peft/accelerate/deepspeed).\n\n\n## Multi-GPU SFT with LoRA and FSDP\nWhen you have access to multiple GPUs, it would be better to use normal LoRA with DeepSpeed/FSDP. To use LoRA with DeepSpeed, refer the docs at [PEFT with FSDP](https://huggingface.co/docs/peft/accelerate/fsdp).\n\n\n\n\nimport os\nfrom enum import Enum\n\nimport torch\nfrom datasets import DatasetDict, load_dataset, load_from_disk\nfrom datasets.builder import DatasetGenerationError\nfrom transformers import (\n AutoModelForCausalLM,\n AutoTokenizer,\n BitsAndBytesConfig,\n)\n\nfrom peft import LoraConfig\n\n\nDEFAULT_CHATML_CHAT_TEMPLATE = \"{% for message in messages %}\\n{{'<|im_start|>' + message['role'] + '\\n' + message['content'] + '<|im_end|>' + '\\n'}}{% if loop.last and add_generation_prompt %}{{'<|im_start|>assistant\\n' }}{% endif %}{% endfor %}\"\nDEFAULT_ZEPHYR_CHAT_TEMPLATE = \"{% for message in messages %}\\n{% if message['role'] == 'user' %}\\n{{ '<|user|>\\n' + message['content'] + eos_token }}\\n{% elif message['role'] == 'system' %}\\n{{ '<|system|>\\n' + message['content'] + eos_token }}\\n{% elif message['role'] == 'assistant' %}\\n{{ '<|assistant|>\\n' + message['content'] + eos_token }}\\n{% endif %}\\n{% if loop.last and add_generation_prompt %}\\n{{ '<|assistant|>' }}\\n{% endif %}\\n{% endfor %}\"\n\n\nclass ZephyrSpecialTokens(str, Enum):\n user = \"<|user|>\"\n assistant = \"<|assistant|>\"\n system = \"<|system|>\"\n eos_token = \"\"\n bos_token = \"\n\n ![](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/PEFT-hub-screenshot.png\"/)
\n The adapter weights for a opt-350m model stored on the Hub are only ~6MB compared to the full size of the model weights, which can be ~700MB. \n
\n\n## Inference\n\n\n \n
\n\n\n
\nGet started
\n Start here if you're new to 🤗 PEFT to get an overview of the library's main features, and how to train a model with a PEFT method.
\n \nHow-to guides
\n Practical guides demonstrating how to apply various PEFT methods across different types of tasks like image classification, causal language modeling, automatic speech recognition, and more. Learn how to use 🤗 PEFT with the DeepSpeed and Fully Sharded Data Parallel scripts.
\n \nConceptual guides
\n Get a better theoretical understanding of how LoRA and various soft prompting methods help reduce the number of trainable parameters to make training more efficient.
\n \nReference
\n Technical descriptions of how 🤗 PEFT classes and methods work.
\n \n\n ![](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/peft_deepspeed_mem_usage.png\"/)
\n
\nGPU memory usage for the training run\n\n## More resources\nYou can also refer this blog post [Falcon 180B Finetuning using 🤗 PEFT and DeepSpeed](https://medium.com/@sourabmangrulkar/falcon-180b-finetuning-using-peft-and-deepspeed-b92643091d99) on how to finetune 180B Falcon model on 16 A100 GPUs on 2 machines.\n\n\n# Use PEFT QLoRA and DeepSpeed with ZeRO3 for finetuning large models on multiple GPUs\n\nIn this section, we will look at how to use QLoRA and DeepSpeed Stage-3 for finetuning 70B llama model on 2X40GB GPUs.\nFor this, we first need `bitsandbytes>=0.43.0`, `accelerate>=0.28.0`, `transformers>4.38.2`, `trl>0.7.11` and `peft>0.9.0`. We need to set `zero3_init_flag` to true when using Accelerate config. Below is the config which can be found at [deepspeed_config_z3_qlora.yaml](https://github.com/huggingface/peft/blob/main/examples/sft/configs/deepspeed_config_z3_qlora.yaml):\n\n```yml\ncompute_environment: LOCAL_MACHINE \ndebug: false\ndeepspeed_config:\n deepspeed_multinode_launcher: standard\n offload_optimizer_device: none\n offload_param_device: none\n zero3_init_flag: true\n zero3_save_16bit_model: true\n zero_stage: 3\ndistributed_type: DEEPSPEED\ndowncast_bf16: 'no'\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: bf16\nnum_machines: 1\nnum_processes: 2\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n```\n\nLaunch command is given below which is available at [run_peft_qlora_deepspeed_stage3.sh](https://github.com/huggingface/peft/blob/main/examples/sft/run_peft_deepspeed.sh):\n```\naccelerate launch --config_file \"configs/deepspeed_config_z3_qlora.yaml\" train.py \\\n--seed 100 \\\n--model_name_or_path \"meta-llama/Llama-2-70b-hf\" \\\n--dataset_name \"smangrul/ultrachat-10k-chatml\" \\\n--chat_template_format \"chatml\" \\\n--add_special_tokens False \\\n--append_concat_token False \\\n--splits \"train,test\" \\\n--max_seq_len 2048 \\\n--num_train_epochs 1 \\\n--logging_steps 5 \\\n--log_level \"info\" \\\n--logging_strategy \"steps\" \\\n--evaluation_strategy \"epoch\" \\\n--save_strategy \"epoch\" \\\n--push_to_hub \\\n--hub_private_repo True \\\n--hub_strategy \"every_save\" \\\n--bf16 True \\\n--packing True \\\n--learning_rate 1e-4 \\\n--lr_scheduler_type \"cosine\" \\\n--weight_decay 1e-4 \\\n--warmup_ratio 0.0 \\\n--max_grad_norm 1.0 \\\n--output_dir \"llama-sft-qlora-dsz3\" \\\n--per_device_train_batch_size 2 \\\n--per_device_eval_batch_size 2 \\\n--gradient_accumulation_steps 2 \\\n--gradient_checkpointing True \\\n--use_reentrant True \\\n--dataset_text_field \"content\" \\\n--use_flash_attn True \\\n--use_peft_lora True \\\n--lora_r 8 \\\n--lora_alpha 16 \\\n--lora_dropout 0.1 \\\n--lora_target_modules \"all-linear\" \\\n--use_4bit_quantization True \\\n--use_nested_quant True \\\n--bnb_4bit_compute_dtype \"bfloat16\" \\\n--bnb_4bit_quant_storage_dtype \"bfloat16\"\n```\n\nNotice the new argument being passed `bnb_4bit_quant_storage_dtype` which denotes the data type for packing the 4-bit parameters. For example, when it is set to `bfloat16`, **32/4 = 8** 4-bit params are packed together post quantization.\n\nIn terms of training code, the important code changes are: \n\n```diff\n...\n\nbnb_config = BitsAndBytesConfig(\n load_in_4bit=args.use_4bit_quantization,\n bnb_4bit_quant_type=args.bnb_4bit_quant_type,\n bnb_4bit_compute_dtype=compute_dtype,\n bnb_4bit_use_double_quant=args.use_nested_quant,\n+ bnb_4bit_quant_storage=quant_storage_dtype,\n)\n\n...\n\nmodel = AutoModelForCausalLM.from_pretrained(\n args.model_name_or_path,\n quantization_config=bnb_config,\n trust_remote_code=True,\n attn_implementation=\"flash_attention_2\" if args.use_flash_attn else \"eager\",\n+ torch_dtype=quant_storage_dtype or torch.float32,\n)\n```\n\nNotice that `torch_dtype` for `AutoModelForCausalLM` is same as the `bnb_4bit_quant_storage` data type. That's it. Everything else is handled by Trainer and TRL.\n\n## Memory usage\n\nIn the above example, the memory consumed per GPU is **36.6 GB**. Therefore, what took 8X80GB GPUs with DeepSpeed Stage 3+LoRA and a couple of 80GB GPUs with DDP+QLoRA now requires 2X40GB GPUs. This makes finetuning of large models more accessible.\n\n# Use PEFT and DeepSpeed with ZeRO3 and CPU Offloading for finetuning large models on a single GPU\nThis section of guide will help you learn how to use our DeepSpeed [training script](https://github.com/huggingface/peft/blob/main/examples/conditional_generation/peft_lora_seq2seq_accelerate_ds_zero3_offload.py). You'll configure the script to train a large model for conditional generation with ZeRO-3 and CPU Offload.\n\n\n\n ![](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/fsdp-peft-config.png\"/)
\n
\nCreating Accelerate's config to use FSDP\n\nOnce this is done, the corresponding config should look like below and you can find it in config folder at [fsdp_config.yaml](https://github.com/huggingface/peft/blob/main/examples/sft/configs/fsdp_config.yaml):\n\n```yml\ncompute_environment: LOCAL_MACHINE\ndebug: false\ndistributed_type: FSDP\ndowncast_bf16: 'no'\nfsdp_config:\n fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP\n fsdp_backward_prefetch: BACKWARD_PRE\n fsdp_cpu_ram_efficient_loading: true\n fsdp_forward_prefetch: false\n fsdp_offload_params: false\n fsdp_sharding_strategy: FULL_SHARD\n fsdp_state_dict_type: SHARDED_STATE_DICT\n fsdp_sync_module_states: true\n fsdp_use_orig_params: false\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: bf16\nnum_machines: 1\nnum_processes: 8\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n```\n\n## Launch command\n\nThe launch command is available at [run_peft_fsdp.sh](https://github.com/huggingface/peft/blob/main/examples/sft/run_peft_fsdp.sh) and it is also shown below:\n```bash\naccelerate launch --config_file \"configs/fsdp_config.yaml\" train.py \\\n--seed 100 \\\n--model_name_or_path \"meta-llama/Llama-2-70b-hf\" \\\n--dataset_name \"smangrul/ultrachat-10k-chatml\" \\\n--chat_template_format \"chatml\" \\\n--add_special_tokens False \\\n--append_concat_token False \\\n--splits \"train,test\" \\\n--max_seq_len 2048 \\\n--num_train_epochs 1 \\\n--logging_steps 5 \\\n--log_level \"info\" \\\n--logging_strategy \"steps\" \\\n--evaluation_strategy \"epoch\" \\\n--save_strategy \"epoch\" \\\n--push_to_hub \\\n--hub_private_repo True \\\n--hub_strategy \"every_save\" \\\n--bf16 True \\\n--packing True \\\n--learning_rate 1e-4 \\\n--lr_scheduler_type \"cosine\" \\\n--weight_decay 1e-4 \\\n--warmup_ratio 0.0 \\\n--max_grad_norm 1.0 \\\n--output_dir \"llama-sft-lora-fsdp\" \\\n--per_device_train_batch_size 8 \\\n--per_device_eval_batch_size 8 \\\n--gradient_accumulation_steps 4 \\\n--gradient_checkpointing True \\\n--use_reentrant False \\\n--dataset_text_field \"content\" \\\n--use_flash_attn True \\\n--use_peft_lora True \\\n--lora_r 8 \\\n--lora_alpha 16 \\\n--lora_dropout 0.1 \\\n--lora_target_modules \"all-linear\" \\\n--use_4bit_quantization False\n```\n\nNotice that we are using LoRA with rank=8, alpha=16 and targeting all linear layers. We are passing the FSDP config file and finetuning the 70B Llama model on a subset of the [ultrachat dataset](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k).\n\n## The important parts\n\nLet's dive a little deeper into the script so you can see what's going on, and understand how it works.\n\nThe first thing to know is that the script uses FSDP for distributed training as the FSDP config has been passed. The `SFTTrainer` class handles all the heavy lifting of creating PEFT model using the peft config that is passed. After that when you call `trainer.train()`, Trainer internally uses 🤗 Accelerate to prepare model, optimizer and trainer using the FSDP config to create FSDP wrapped model which is then trained. The main code snippet is below:\n\n```python\n# trainer\ntrainer = SFTTrainer(\n model=model,\n tokenizer=tokenizer,\n args=training_args,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n peft_config=peft_config,\n packing=data_args.packing,\n dataset_kwargs={\n \"append_concat_token\": data_args.append_concat_token,\n \"add_special_tokens\": data_args.add_special_tokens,\n },\n dataset_text_field=data_args.dataset_text_field,\n max_seq_length=data_args.max_seq_length,\n)\ntrainer.accelerator.print(f\"{trainer.model}\")\nif model_args.use_peft_lora:\n # handle PEFT+FSDP case\n trainer.model.print_trainable_parameters()\n if getattr(trainer.accelerator.state, \"fsdp_plugin\", None):\n from peft.utils.other import fsdp_auto_wrap_policy\n\n fsdp_plugin = trainer.accelerator.state.fsdp_plugin\n fsdp_plugin.auto_wrap_policy = fsdp_auto_wrap_policy(trainer.model)\n\n# train\ncheckpoint = None\nif training_args.resume_from_checkpoint is not None:\n checkpoint = training_args.resume_from_checkpoint\ntrainer.train(resume_from_checkpoint=checkpoint)\n\n# saving final model\nif trainer.is_fsdp_enabled:\n trainer.accelerator.state.fsdp_plugin.set_state_dict_type(\"FULL_STATE_DICT\")\ntrainer.save_model()\n```\n\n\nHere, one main thing to note currently when using FSDP with PEFT is that `use_orig_params` needs to be `False` to realize GPU memory savings. Due to `use_orig_params=False`, the auto wrap policy for FSDP needs to change so that trainable and non-trainable parameters are wrapped separately. This is done by the code snippt below which uses the util function `fsdp_auto_wrap_policy` from PEFT:\n\n```\nif getattr(trainer.accelerator.state, \"fsdp_plugin\", None):\n from peft.utils.other import fsdp_auto_wrap_policy\n\n fsdp_plugin = trainer.accelerator.state.fsdp_plugin\n fsdp_plugin.auto_wrap_policy = fsdp_auto_wrap_policy(trainer.model)\n```\n\n## Memory usage\n\nIn the above example, the memory consumed per GPU is 72-80 GB (90-98%) as seen in the screenshot below. The slight increase in GPU memory at the end is when saving the model using `FULL_STATE_DICT` state dict type instead of the `SHARDED_STATE_DICT` so that the model has adapter weights that can be loaded normally with `from_pretrained` method during inference:\n\n\n\n ![](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/peft_fsdp_mem_usage.png\"/)
\n
\nGPU memory usage for the training run\n\n# Use PEFT QLoRA and FSDP for finetuning large models on multiple GPUs\n\nIn this section, we will look at how to use QLoRA and FSDP for finetuning 70B llama model on 2X24GB GPUs. [Answer.AI](https://www.answer.ai/) in collaboration with bitsandbytes and Hugging Face 🤗 open sourced code enabling the usage of FSDP+QLoRA and explained the whole process in their insightful blogpost [You can now train a 70b language model at home](https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html). This is now integrated in Hugging Face ecosystem. \n\nFor this, we first need `bitsandbytes>=0.43.0`, `accelerate>=0.28.0`, `transformers>4.38.2`, `trl>0.7.11` and `peft>0.9.0`. We need to set `fsdp_cpu_ram_efficient_loading=true`, `fsdp_use_orig_params=false` and `fsdp_offload_params=true`(cpu offloading) when using Accelerate config. When not using accelerate launcher, you can alternately set the environment variable `export FSDP_CPU_RAM_EFFICIENT_LOADING=true`. Here, we will be using accelerate config and below is the config which can be found at [fsdp_config_qlora.yaml](https://github.com/huggingface/peft/blob/main/examples/sft/configs/fsdp_config_qlora.yaml):\n\n```yml\ncompute_environment: LOCAL_MACHINE \ndebug: false \ndistributed_type: FSDP\ndowncast_bf16: 'no'\nfsdp_config:\n fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP\n fsdp_backward_prefetch: BACKWARD_PRE\n fsdp_cpu_ram_efficient_loading: true\n fsdp_forward_prefetch: false\n fsdp_offload_params: true\n fsdp_sharding_strategy: FULL_SHARD\n fsdp_state_dict_type: SHARDED_STATE_DICT\n fsdp_sync_module_states: true\n fsdp_use_orig_params: false\nmachine_rank: 0\nmain_training_function: main\nmixed_precision: 'no'\nnum_machines: 1\nnum_processes: 2\nrdzv_backend: static\nsame_network: true\ntpu_env: []\ntpu_use_cluster: false\ntpu_use_sudo: false\nuse_cpu: false\n```\n\nLaunch command is given below which is available at [run_peft_qlora_fsdp.sh](https://github.com/huggingface/peft/blob/main/examples/sft/run_peft_qlora_fsdp.sh):\n```\naccelerate launch --config_file \"configs/fsdp_config_qlora.yaml\" train.py \\\n--seed 100 \\\n--model_name_or_path \"meta-llama/Llama-2-70b-hf\" \\\n--dataset_name \"smangrul/ultrachat-10k-chatml\" \\\n--chat_template_format \"chatml\" \\\n--add_special_tokens False \\\n--append_concat_token False \\\n--splits \"train,test\" \\\n--max_seq_len 2048 \\\n--num_train_epochs 1 \\\n--logging_steps 5 \\\n--log_level \"info\" \\\n--logging_strategy \"steps\" \\\n--evaluation_strategy \"epoch\" \\\n--save_strategy \"epoch\" \\\n--push_to_hub \\\n--hub_private_repo True \\\n--hub_strategy \"every_save\" \\\n--bf16 True \\\n--packing True \\\n--learning_rate 1e-4 \\\n--lr_scheduler_type \"cosine\" \\\n--weight_decay 1e-4 \\\n--warmup_ratio 0.0 \\\n--max_grad_norm 1.0 \\\n--output_dir \"llama-sft-qlora-fsdp\" \\\n--per_device_train_batch_size 2 \\\n--per_device_eval_batch_size 2 \\\n--gradient_accumulation_steps 2 \\\n--gradient_checkpointing True \\\n--use_reentrant True \\\n--dataset_text_field \"content\" \\\n--use_flash_attn True \\\n--use_peft_lora True \\\n--lora_r 8 \\\n--lora_alpha 16 \\\n--lora_dropout 0.1 \\\n--lora_target_modules \"all-linear\" \\\n--use_4bit_quantization True \\\n--use_nested_quant True \\\n--bnb_4bit_compute_dtype \"bfloat16\" \\\n--bnb_4bit_quant_storage_dtype \"bfloat16\"\n```\n\nNotice the new argument being passed, `bnb_4bit_quant_storage_dtype`, which denotes the data type for packing the 4-bit parameters. For example, when it is set to `bfloat16`, **16/4 = 4** 4-bit params are packed together post quantization. When using mixed precision training with `bfloat16`, `bnb_4bit_quant_storage_dtype` can be either `bfloat16` for pure `bfloat16` finetuning, or `float32` for automatic mixed precision (this consumes more GPU memory). When using mixed precision training with `float16`, `bnb_4bit_quant_storage_dtype` should be set to `float32` for stable automatic mixed precision training.\n\nIn terms of training code, the important code changes are: \n\n```diff\n...\n\nbnb_config = BitsAndBytesConfig(\n load_in_4bit=args.use_4bit_quantization,\n bnb_4bit_quant_type=args.bnb_4bit_quant_type,\n bnb_4bit_compute_dtype=compute_dtype,\n bnb_4bit_use_double_quant=args.use_nested_quant,\n+ bnb_4bit_quant_storage=quant_storage_dtype,\n)\n\n...\n\nmodel = AutoModelForCausalLM.from_pretrained(\n args.model_name_or_path,\n quantization_config=bnb_config,\n trust_remote_code=True,\n attn_implementation=\"flash_attention_2\" if args.use_flash_attn else \"eager\",\n+ torch_dtype=quant_storage_dtype or torch.float32,\n)\n```\n\nNotice that `torch_dtype` for `AutoModelForCausalLM` is same as the `bnb_4bit_quant_storage` data type. That's it. Everything else is handled by Trainer and TRL.\n\n## Memory usage\n\nIn the above example, the memory consumed per GPU is **19.6 GB** while CPU RAM usage is around **107 GB**. When disabling CPU offloading, the GPU memory usage is **35.6 GB/ GPU**. Therefore, what took 16X80GB GPUs for full finetuning, 8X80GB GPUs with FSDP+LoRA, and a couple of 80GB GPUs with DDP+QLoRA, now requires 2X24GB GPUs. This makes finetuning of large models more accessible.\n\n## More resources\nYou can also refer the [llama-recipes](https://github.com/facebookresearch/llama-recipes/?tab=readme-ov-file#fine-tuning) repo and [Getting started with Llama](https://llama.meta.com/get-started/#fine-tuning) guide on how to finetune using FSDP and PEFT.\n\n## Caveats\n1. Merging when using PEFT and FSDP is currently unsupported and will raise error.\n2. Passing `modules_to_save` config parameter to is untested at present.\n3. GPU Memory saving when using CPU Offloading is untested at present.\n4. When using FSDP+QLoRA, `paged_adamw_8bit` currently results in an error when saving a checkpoint.\n5. DoRA training with FSDP should work (albeit at lower speed than LoRA). If combined with bitsandbytes (QDoRA), 4-bit quantization should also work, but 8-bit quantization has known issues and is not recommended.\n\n\n\n\n# IA3 \n\nThis conceptual guide gives a brief overview of [IA3](https://arxiv.org/abs/2205.05638), a parameter-efficient fine tuning technique that is \nintended to improve over [LoRA](./lora).\n\nTo make fine-tuning more efficient, IA3 (Infused Adapter by Inhibiting and Amplifying Inner Activations) \nrescales inner activations with learned vectors. These learned vectors are injected in the attention and feedforward modules \nin a typical transformer-based architecture. These learned vectors are the only trainable parameters during fine-tuning, and thus the original \nweights remain frozen. Dealing with learned vectors (as opposed to learned low-rank updates to a weight matrix like LoRA)\nkeeps the number of trainable parameters much smaller. \n\nBeing similar to LoRA, IA3 carries many of the same advantages: \n\n* IA3 makes fine-tuning more efficient by drastically reducing the number of trainable parameters. (For T0, an IA3 model only has about 0.01% trainable parameters, while even LoRA has > 0.1%)\n* The original pre-trained weights are kept frozen, which means you can have multiple lightweight and portable IA3 models for various downstream tasks built on top of them.\n* Performance of models fine-tuned using IA3 is comparable to the performance of fully fine-tuned models.\n* IA3 does not add any inference latency because adapter weights can be merged with the base model.\n\nIn principle, IA3 can be applied to any subset of weight matrices in a neural network to reduce the number of trainable\nparameters. Following the authors' implementation, IA3 weights are added to the key, value and feedforward layers\nof a Transformer model. To be specific, for transformer models, IA3 weights are added to the outputs of key and value layers, and to the input of the second feedforward layer\nin each transformer block.\n\nGiven the target layers for injecting IA3 parameters, the number of trainable parameters\ncan be determined based on the size of the weight matrices.\n\n\n## Common IA3 parameters in PEFT\n\nAs with other methods supported by PEFT, to fine-tune a model using IA3, you need to:\n\n1. Instantiate a base model.\n2. Create a configuration (`IA3Config`) where you define IA3-specific parameters.\n3. Wrap the base model with `get_peft_model()` to get a trainable `PeftModel`.\n4. Train the `PeftModel` as you normally would train the base model.\n\n`IA3Config` allows you to control how IA3 is applied to the base model through the following parameters:\n\n- `target_modules`: The modules (for example, attention blocks) to apply the IA3 vectors.\n- `feedforward_modules`: The list of modules to be treated as feedforward layers in `target_modules`. While learned vectors are multiplied with\nthe output activation for attention blocks, the vectors are multiplied with the input for classic feedforward layers. Note that `feedforward_modules` must be a subset of `target_modules`.\n- `modules_to_save`: List of modules apart from IA3 layers to be set as trainable and saved in the final checkpoint. These typically include model's custom head that is randomly initialized for the fine-tuning task.\n\n## Example Usage\n\nFor the task of sequence classification, one can initialize the IA3 config for a Llama model as follows:\n\n```py\npeft_config = IA3Config(\n task_type=TaskType.SEQ_CLS, target_modules=[\"k_proj\", \"v_proj\", \"down_proj\"], feedforward_modules=[\"down_proj\"]\n)\n```\n\n\n\n# Adapters\n\nAdapter-based methods add extra trainable parameters after the attention and fully-connected layers of a frozen pretrained model to reduce memory-usage and speed up training. The method varies depending on the adapter, it could simply be an extra added layer or it could be expressing the weight updates ∆W as a low-rank decomposition of the weight matrix. Either way, the adapters are typically small but demonstrate comparable performance to a fully finetuned model and enable training larger models with fewer resources.\n\nThis guide will give you a brief overview of the adapter methods supported by PEFT (if you're interested in learning more details about a specific method, take a look at the linked paper).\n\n## Low-Rank Adaptation (LoRA)\n\n\n\n ![](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/lora_animated.gif\"/)
\n
\n\nThis approach has a number of advantages:\n\n* LoRA makes finetuning more efficient by drastically reducing the number of trainable parameters.\n* The original pretrained weights are kept frozen, which means you can have multiple lightweight and portable LoRA models for various downstream tasks built on top of them.\n* LoRA is orthogonal to other parameter-efficient methods and can be combined with many of them.\n* Performance of models finetuned using LoRA is comparable to the performance of fully finetuned models.\n\nIn principle, LoRA can be applied to any subset of weight matrices in a neural network to reduce the number of trainable parameters. However, for simplicity and further parameter efficiency, LoRA is typically only applied to the attention blocks in Transformer models. The resulting number of trainable parameters in a LoRA model depends on the size of the update matrices, which is determined mainly by the rank `r` and the shape of the original weight matrix.\n\n\n\n ![](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/lora.png\"/)
\n
\nNavigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation\n\n## Low-Rank Hadamard Product (LoHa)\n\nLow-rank decomposition can impact performance because the weight updates are limited to the low-rank space, which can constrain a model's expressiveness. However, you don't necessarily want to use a larger rank because it increases the number of trainable parameters. To address this, [LoHa](https://huggingface.co/papers/2108.06098) (a method originally developed for computer vision) was applied to diffusion models where the ability to generate diverse images is an important consideration. LoHa should also work with general model types, but the embedding layers aren't currently implemented in PEFT.\n\nLoHa uses the [Hadamard product](https://en.wikipedia.org/wiki/Hadamard_product_(matrices)) (element-wise product) instead of the matrix product. ∆W is represented by four smaller matrices instead of two - like in LoRA - and each pair of these low-rank matrices are combined with the Hadamard product. As a result, ∆W can have the same number of trainable parameters but a higher rank and expressivity.\n\n## Low-Rank Kronecker Product (LoKr)\n\n[LoKr](https://hf.co/papers/2309.14859) is very similar to LoRA and LoHa, and it is also mainly applied to diffusion models, though you could also use it with other model types. LoKr replaces the matrix product with the [Kronecker product](https://en.wikipedia.org/wiki/Kronecker_product) instead. The Kronecker product decomposition creates a block matrix which preserves the rank of the original weight matrix. Another benefit of the Kronecker product is that it can be vectorized by stacking the matrix columns. This can speed up the process because you're avoiding fully reconstructing ∆W.\n\n## Orthogonal Finetuning (OFT)\n\n\n\n ![](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/oft.png\"/)
\n
\nControlling Text-to-Image Diffusion by Orthogonal Finetuning\n\n[OFT](https://hf.co/papers/2306.07280) is a method that primarily focuses on preserving a pretrained model's generative performance in the finetuned model. It tries to maintain the same cosine similarity (hyperspherical energy) between all pairwise neurons in a layer because this better captures the semantic information among neurons. This means OFT is more capable at preserving the subject and it is better for controllable generation (similar to [ControlNet](https://huggingface.co/docs/diffusers/using-diffusers/controlnet)).\n\nOFT preserves the hyperspherical energy by learning an orthogonal transformation for neurons to keep the cosine similarity between them unchanged. In practice, this means taking the matrix product of an orthogonal matrix with the pretrained weight matrix. However, to be parameter-efficient, the orthogonal matrix is represented as a block-diagonal matrix with rank `r` blocks. Whereas LoRA reduces the number of trainable parameters with low-rank structures, OFT reduces the number of trainable parameters with a sparse block-diagonal matrix structure.\n\n## Orthogonal Butterfly (BOFT)\n\n[BOFT](https://hf.co/papers/2311.06243) is a method that primarily focuses on preserving a pretrained model's generative performance in the finetuned model. It tries to maintain the same cosine similarity (hyperspherical energy) between all pairwise neurons in a layer because this better captures the semantic information among neurons. This means OFT is more capable at preserving the subject and it is better for controllable generation (similar to [ControlNet](https://huggingface.co/docs/diffusers/using-diffusers/controlnet)).\n\nOFT preserves the hyperspherical energy by learning an orthogonal transformation for neurons to keep the cosine similarity between them unchanged. In practice, this means taking the matrix product of an orthogonal matrix with the pretrained weight matrix. However, to be parameter-efficient, the orthogonal matrix is represented as a block-diagonal matrix with rank `r` blocks. Whereas LoRA reduces the number of trainable parameters with low-rank structures, OFT reduces the number of trainable parameters with a sparse block-diagonal matrix structure.\n\n## Adaptive Low-Rank Adaptation (AdaLoRA)\n\n[AdaLoRA](https://hf.co/papers/2303.10512) manages the parameter budget introduced from LoRA by allocating more parameters - in other words, a higher rank `r` - for important weight matrices that are better adapted for a task and pruning less important ones. The rank is controlled by a method similar to singular value decomposition (SVD). The ∆W is parameterized with two orthogonal matrices and a diagonal matrix which contains singular values. This parametrization method avoids iteratively applying SVD which is computationally expensive. Based on this method, the rank of ∆W is adjusted according to an importance score. ∆W is divided into triplets and each triplet is scored according to its contribution to model performance. Triplets with low importance scores are pruned and triplets with high importance scores are kept for finetuning.\n\n## Llama-Adapter\n\n[Llama-Adapter](https://hf.co/papers/2303.16199) is a method for adapting Llama into a instruction-following model. To help adapt the model for instruction-following, the adapter is trained with a 52K instruction-output dataset.\n\nA set of of learnable adaption prompts are prefixed to the input instruction tokens. These are inserted into the upper layers of the model because it is better to learn with the higher-level semantics of the pretrained model. The instruction-output tokens prefixed to the input guide the adaption prompt to generate a contextual response.\n\n\n\n ![](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/llama-adapter.png\"/)
\n
\nLLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention\n\nTo avoid adding noise to the tokens, the adapter uses zero-initialized attention. On top of this, the adapter adds a learnable gating factor (initialized with zeros) to progressively add information to the model during training. This prevents overwhelming the model's pretrained knowledge with the newly learned instructions.\n\n\n\n\n# Orthogonal Finetuning (OFT and BOFT) \n\nThis conceptual guide gives a brief overview of [OFT](https://arxiv.org/abs/2306.07280) and [BOFT](https://arxiv.org/abs/2311.06243), a parameter-efficient fine-tuning technique that utilizes orthogonal matrix to multiplicatively transform the pretrained weight matrices.\n\nTo achieve efficient fine-tuning, OFT represents the weight updates with an orthogonal transformation. The orthogonal transformation is parameterized by an orthogonal matrix multiplied to the pretrained weight matrix. These new matrices can be trained to adapt to the new data while keeping the overall number of changes low. The original weight matrix remains frozen and doesn’t receive any further adjustments. To produce the final results, both the original and the adapted weights are multiplied togethor.\n\nOrthogonal Butterfly (BOFT) generalizes OFT with Butterfly factorization and further improves its parameter efficiency and finetuning flexibility. In short, OFT can be viewed as a special case of BOFT. Different from LoRA that uses additive low-rank weight updates, BOFT uses multiplicative orthogonal weight updates. The comparison is shown below.\n\n\n\n ![](\"https://raw.githubusercontent.com/wy1iu/butterfly-oft/main/assets/BOFT_comparison.png\"/)
\n
\n\n\nBOFT has some advantages compared to LoRA: \n\n* BOFT proposes a simple yet generic way to finetune pretrained models to downstream tasks, yielding a better preservation of pretraining knowledge and a better parameter efficiency.\n* Through the orthogonality, BOFT introduces a structural constraint, i.e., keeping the [hyperspherical energy](https://arxiv.org/abs/1805.09298) unchanged during finetuning. This can effectively reduce the forgetting of pretraining knowledge.\n* BOFT uses the butterfly factorization to efficiently parameterize the orthogonal matrix, which yields a compact yet expressive learning space (i.e., hypothesis class).\n* The sparse matrix decomposition in BOFT brings in additional inductive biases that are beneficial to generalization.\n\nIn principle, BOFT can be applied to any subset of weight matrices in a neural network to reduce the number of trainable parameters. Given the target layers for injecting BOFT parameters, the number of trainable parameters can be determined based on the size of the weight matrices.\n\n## Merge OFT/BOFT weights into the base model\n\nSimilar to LoRA, the weights learned by OFT/BOFT can be integrated into the pretrained weight matrices using the merge_and_unload() function. This function merges the adapter weights with the base model which allows you to effectively use the newly merged model as a standalone model.\n\n\n\n ![](\"https://raw.githubusercontent.com/wy1iu/butterfly-oft/main/assets/boft_merge.png\"/)
\n
\n\nThis works because during training, the orthogonal weight matrix (R in the diagram above) and the pretrained weight matrices are separate. But once training is complete, these weights can actually be merged (multiplied) into a new weight matrix that is equivalent.\n\n## Utils for OFT / BOFT\n\n### Common OFT / BOFT parameters in PEFT\n\nAs with other methods supported by PEFT, to fine-tune a model using OFT or BOFT, you need to:\n\n1. Instantiate a base model.\n2. Create a configuration (`OFTConfig` or `BOFTConfig`) where you define OFT/BOFT-specific parameters.\n3. Wrap the base model with `get_peft_model()` to get a trainable `PeftModel`.\n4. Train the `PeftModel` as you normally would train the base model.\n\n\n### BOFT-specific paramters\n\n`BOFTConfig` allows you to control how OFT/BOFT is applied to the base model through the following parameters:\n\n- `boft_block_size`: the BOFT matrix block size across different layers, expressed in `int`. Smaller block size results in sparser update matrices with fewer trainable paramters. **Note**, please choose `boft_block_size` to be divisible by most layer's input dimension (`in_features`), e.g., 4, 8, 16. Also, please only \nspecify either `boft_block_size` or `boft_block_num`, but not both simultaneously or leaving both to 0, because `boft_block_size` x `boft_block_num` must equal the layer's input dimension.\n- `boft_block_num`: the number of BOFT matrix blocks across different layers, expressed in `int`. Fewer blocks result in sparser update matrices with fewer trainable paramters. **Note**, please choose `boft_block_num` to be divisible by most layer's input dimension (`in_features`), e.g., 4, 8, 16. Also, please only \nspecify either `boft_block_size` or `boft_block_num`, but not both simultaneously or leaving both to 0, because `boft_block_size` x `boft_block_num` must equal the layer's input dimension.\n- `boft_n_butterfly_factor`: the number of butterfly factors. **Note**, for `boft_n_butterfly_factor=1`, BOFT is the same as vanilla OFT, for `boft_n_butterfly_factor=2`, the effective block size of OFT becomes twice as big and the number of blocks become half.\n- `bias`: specify if the `bias` parameters should be trained. Can be `\"none\"`, `\"all\"` or `\"boft_only\"`.\n- `boft_dropout`: specify the probability of multiplicative dropout.\n- `target_modules`: The modules (for example, attention blocks) to inject the OFT/BOFT matrices.\n- `modules_to_save`: List of modules apart from OFT/BOFT matrices to be set as trainable and saved in the final checkpoint. These typically include model's custom head that is randomly initialized for the fine-tuning task.\n\n\n\n## BOFT Example Usage\n\nFor an example of the BOFT method application to various downstream tasks, please refer to the following guides:\n\nTake a look at the following step-by-step guides on how to finetune a model with BOFT:\n- [Dreambooth finetuning with BOFT](../task_guides/boft_dreambooth) \n- [Controllable generation finetuning with BOFT (ControlNet)](../task_guides/boft_controlnet) \n\nFor the task of image classification, one can initialize the BOFT config for a DinoV2 model as follows:\n\n```py\nimport transformers\nfrom transformers import AutoModelForSeq2SeqLM, BOFTConfig\nfrom peft import BOFTConfig, get_peft_model\n\nconfig = BOFTConfig(\n boft_block_size=4,\n boft_n_butterfly_factor=2,\n target_modules=[\"query\", \"value\", \"key\", \"output.dense\", \"mlp.fc1\", \"mlp.fc2\"],\n boft_dropout=0.1,\n bias=\"boft_only\",\n modules_to_save=[\"classifier\"],\n)\n\nmodel = transformers.Dinov2ForImageClassification.from_pretrained(\n \"facebook/dinov2-large\",\n num_labels=100,\n)\n\nboft_model = get_peft_model(model, config)\n```\n\n\n\n\n# Soft prompts\n\nTraining large pretrained language models is very time-consuming and compute-intensive. As they continue to grow in size, there is increasing interest in more efficient training methods such as *prompting*. Prompting primes a frozen pretrained model for a specific downstream task by including a text prompt that describes the task or even demonstrates an example of the task. With prompting, you can avoid fully training a separate model for each downstream task, and use the same frozen pretrained model instead. This is a lot easier because you can use the same model for several different tasks, and it is significantly more efficient to train and store a smaller set of prompt parameters than to train all the model's parameters.\n\nThere are two categories of prompting methods:\n\n- hard prompts are manually handcrafted text prompts with discrete input tokens; the downside is that it requires a lot of effort to create a good prompt\n- soft prompts are learnable tensors concatenated with the input embeddings that can be optimized to a dataset; the downside is that they aren't human readable because you aren't matching these \"virtual tokens\" to the embeddings of a real word\n\nThis conceptual guide provides a brief overview of the soft prompt methods included in 🤗 PEFT: prompt tuning, prefix tuning, P-tuning, and multitask prompt tuning.\n\n## Prompt tuning\n\n\n\n ![](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/prompt-tuning.png\"/)
\n
\nOnly train and store a significantly smaller set of task-specific prompt parameters (image source).\n\n[Prompt tuning](https://hf.co/papers/2104.08691) was developed for text classification tasks on T5 models, and all downstream tasks are cast as a text generation task. For example, sequence classification usually assigns a single class label to a sequence of text. By casting it as a text generation task, the tokens that make up the class label are *generated*. Prompts are added to the input as a series of tokens. Typically, the model parameters are fixed which means the prompt tokens are also fixed by the model parameters.\n\nThe key idea behind prompt tuning is that prompt tokens have their own parameters that are updated independently. This means you can keep the pretrained model's parameters frozen, and only update the gradients of the prompt token embeddings. The results are comparable to the traditional method of training the entire model, and prompt tuning performance scales as model size increases.\n\nTake a look at [Prompt tuning for causal language modeling](../task_guides/clm-prompt-tuning) for a step-by-step guide on how to train a model with prompt tuning.\n\n## Prefix tuning\n\n\n\n ![](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/prefix-tuning.png\"/)
\n
\nOptimize the prefix parameters for each task (image source).\n\n[Prefix tuning](https://hf.co/papers/2101.00190) was designed for natural language generation (NLG) tasks on GPT models. It is very similar to prompt tuning; prefix tuning also prepends a sequence of task-specific vectors to the input that can be trained and updated while keeping the rest of the pretrained model's parameters frozen. \n\nThe main difference is that the prefix parameters are inserted in **all** of the model layers, whereas prompt tuning only adds the prompt parameters to the model input embeddings. The prefix parameters are also optimized by a separate feed-forward network (FFN) instead of training directly on the soft prompts because it causes instability and hurts performance. The FFN is discarded after updating the soft prompts.\n\nAs a result, the authors found that prefix tuning demonstrates comparable performance to fully finetuning a model, despite having 1000x fewer parameters, and it performs even better in low-data settings.\n\nTake a look at [Prefix tuning for conditional generation](../task_guides/seq2seq-prefix-tuning) for a step-by-step guide on how to train a model with prefix tuning.\n\n## P-tuning\n\n\n\n ![](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/p-tuning.png\"/)
\n
\nPrompt tokens can be inserted anywhere in the input sequence, and they are optimized by a prompt encoder (image source).\n\n[P-tuning](https://hf.co/papers/2103.10385) is designed for natural language understanding (NLU) tasks and all language models. \nIt is another variation of a soft prompt method; P-tuning also adds a trainable embedding tensor that can be optimized to find better prompts, and it uses a prompt encoder (a bidirectional long-short term memory network or LSTM) to optimize the prompt parameters. Unlike prefix tuning though:\n\n- the prompt tokens can be inserted anywhere in the input sequence, and it isn't restricted to only the beginning\n- the prompt tokens are only added to the input instead of adding them to every layer of the model\n- introducing *anchor* tokens can improve performance because they indicate characteristics of a component in the input sequence\n\nThe results suggest that P-tuning is more efficient than manually crafting prompts, and it enables GPT-like models to compete with BERT-like models on NLU tasks.\n\nTake a look at [P-tuning for sequence classification](../task_guides/ptuning-seq-classification) for a step-by-step guide on how to train a model with P-tuning.\n\n## Multitask prompt tuning\n\n\n\n ![](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/mpt.png\"/)
\n
\nMultitask prompt tuning enables parameter-efficient transfer learning.\n\n[Multitask prompt tuning (MPT)](https://hf.co/papers/2103.10385) learns a single prompt from data for multiple task types that can be shared for different target tasks. Other existing approaches learn a separate soft prompt for each task that need to be retrieved or aggregated for adaptation to target tasks. MPT consists of two stages:\n\n1. source training - for each task, its soft prompt is decomposed into task-specific vectors. The task-specific vectors are multiplied together to form another matrix W, and the Hadamard product is used between W and a shared prompt matrix P to generate a task-specific prompt matrix. The task-specific prompts are distilled into a single prompt matrix that is shared across all tasks. This prompt is trained with multitask training.\n2. target adaptation - to adapt the single prompt for a target task, a target prompt is initialized and expressed as the Hadamard product of the shared prompt matrix and the task-specific low-rank prompt matrix.\n\n\n\n ![](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/mpt-decomposition.png\"/)
\n
\nPrompt decomposition.\n\n\n\n\n# IA3\n\n[IA3](../conceptual_guides/ia3) multiplies the model's activations (the keys and values in the self-attention and encoder-decoder attention blocks, and the intermediate activation of the position-wise feedforward network) by three learned vectors. This PEFT method introduces an even smaller number of trainable parameters than LoRA which introduces weight matrices instead of vectors. The original model's parameters are kept frozen and only these vectors are updated. As a result, it is faster, cheaper and more efficient to finetune for a new downstream task.\n\nThis guide will show you how to train a sequence-to-sequence model with IA3 to *generate a sentiment* given some financial news.\n\n\n\n ![](\"https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/beignets.jpeg\")
\n
\n\nConvert the image to RGB and return the underlying PyTorch tensors.\n\n```py\nencoding = image_processor(image.convert(\"RGB\"), return_tensors=\"pt\")\n```\n\nNow run the model and return the predicted class!\n\n```py\nwith torch.no_grad():\n outputs = model(**encoding)\n logits = outputs.logits\n\npredicted_class_idx = logits.argmax(-1).item()\nprint(\"Predicted class:\", model.config.id2label[predicted_class_idx])\n\"Predicted class: beignets\"\n```\n\n\n\n\n# Prompt-based methods\n\nA prompt can describe a task or provide an example of a task you want the model to learn. Instead of manually creating these prompts, soft prompting methods add learnable parameters to the input embeddings that can be optimized for a specific task while keeping the pretrained model's parameters frozen. This makes it both faster and easier to finetune large language models (LLMs) for new downstream tasks.\n\nThe PEFT library supports several types of prompting methods (p-tuning, prefix tuning, prompt tuning) and you can learn more about how these methods work conceptually in the [Soft prompts](../conceptual_guides/prompting) guide. If you're interested in applying these methods to other tasks and use cases, take a look at our [notebook collection](https://huggingface.co/spaces/PEFT/soft-prompting)!\n\nThis guide will show you how to train a causal language model - with a soft prompting method - to *generate a classification* for whether a tweet is a complaint or not.\n\n\n\n \n For example, the adapter weights for a opt-350m model stored on the Hub are only ~6MB compared to the full model size which can be ~700MB. \n
\n\n## Inference\n\nLet's load the model for inference and test it out on a tweet!\n\n```py\nfrom peft import AutoPeftModelForCausalLM\n\nmodel = AutoPeftModelForCausalLM.from_pretrained(\"peft_model_id\").to(\"cuda\")\ntokenizer = AutoTokenizer.from_pretrained(\"bigscience/bloomz-560m\")\n\ni = 15\ninputs = tokenizer(f'{text_column} : {ds[\"test\"][i][\"Tweet text\"]} Label : ', return_tensors=\"pt\")\nprint(ds[\"test\"][i][\"Tweet text\"])\n\"@NYTsupport i have complained a dozen times & yet my papers are still thrown FAR from my door. Why is this so hard to resolve?\"\n```\n\nCall the [`~transformers.GenerationMixin.generate`] method to generate the predicted classification label.\n\n```py\nwith torch.no_grad():\n inputs = {k: v.to(device) for k, v in inputs.items()}\n outputs = model.generate(input_ids=inputs[\"input_ids\"], max_new_tokens=10)\n print(tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True))\n\"['Tweet text : @NYTsupport i have complained a dozen times & yet my papers are still thrown FAR from my door. Why is this so hard to resolve? Label : complaint']\"\n```\n\n\n\n\n# Contribute to PEFT\n\nWe are happy to accept contributions to PEFT. If you plan to contribute, please read this to make the process as smooth as possible.\n\n## Installation\n\nFor code contributions to PEFT, you should choose the [\"source\"](../install#source) installation method.\n\nIf you are new to creating a pull request, follow the [Creating a pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request) guide by GitHub.\n\n## Tests and code quality checks\n\nRegardless of the contribution type (unless it’s only about the docs), you should run tests and code quality checks before creating a PR to ensure your contribution doesn’t break anything and follows the project standards.\n\nWe provide a Makefile to execute the necessary tests. Run the code below for the unit test:\n\n```sh\nmake test\n```\n\nRun one of the following to either only check or check and fix code quality and style:\n\n```sh\nmake quality # just check\nmake style # check and fix\n```\n\nYou can also set up [`pre-commit`](https://pre-commit.com/) to run these fixes\nautomatically as Git commit hooks.\n\n```bash\n$ pip install pre-commit\n$ pre-commit install\n```\n\nRunning all the tests can take a couple of minutes, so during development it can be more efficient to only run tests specific to your change:\n\n```sh\npytest tests/ -k \n \n ![](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/peft/lora_diagram.png\"/)
\n
\n\nWe show in the snippets below how to run that using PEFT.\n\n```py\nfrom transformers import AutoModelForCausalLM\nfrom peft import PeftModel\n\nbase_model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-v0.1\")\npeft_model_id = \"alignment-handbook/zephyr-7b-sft-lora\"\nmodel = PeftModel.from_pretrained(base_model, peft_model_id)\nmodel.merge_and_unload()\n```\n\nIf you need to keep a copy of the weights so you can unmerge the adapter later or delete and load different ones, you should use the [`~LoraModel.merge_adapter`] function instead. Now you have the option to use [`~LoraModel.unmerge_adapter`] to return the base model.\n\n```py\nfrom transformers import AutoModelForCausalLM\nfrom peft import PeftModel\n\nbase_model = AutoModelForCausalLM.from_pretrained(\"mistralai/Mistral-7B-v0.1\")\npeft_model_id = \"alignment-handbook/zephyr-7b-sft-lora\"\nmodel = PeftModel.from_pretrained(base_model, peft_model_id)\nmodel.merge_adapter()\n\n# unmerge the LoRA layers from the base model\nmodel.unmerge_adapter()\n```\n\nThe [`~LoraModel.add_weighted_adapter`] function is useful for merging multiple LoRAs into a new adapter based on a user provided weighting scheme in the `weights` parameter. Below is an end-to-end example.\n\nFirst load the base model:\n\n```python\nfrom transformers import AutoModelForCausalLM\nfrom peft import PeftModel\nimport torch\n\nbase_model = AutoModelForCausalLM.from_pretrained(\n \"mistralai/Mistral-7B-v0.1\", torch_dtype=torch.float16, device_map=\"auto\"\n)\n```\n\nThen we load the first adapter: \n\n```python\npeft_model_id = \"alignment-handbook/zephyr-7b-sft-lora\"\nmodel = PeftModel.from_pretrained(base_model, peft_model_id, adapter_name=\"sft\")\n```\n\nThen load a different adapter and merge it with the first one:\n\n```python\nweighted_adapter_name = \"sft-dpo\"\nmodel.load_adapter(\"alignment-handbook/zephyr-7b-dpo-lora\", adapter_name=\"dpo\")\nmodel.add_weighted_adapter(\n adapters=[\"sft\", \"dpo\"],\n weights=[0.7, 0.3],\n adapter_name=weighted_adapter_name,\n combination_type=\"linear\"\n)\nmodel.set_adapter(weighted_adapter_name)\n```\n\n\n\n\nThis shouldn't be an issue if you're only merging LoRA adapters trained from the same base model.\n\n
\n ![](\"https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/test-lora-diffusers.png\"/)
\n
\n\nNow let's try another cool LoRA model, [ostris/super-cereal-sdxl-lora](https://huggingface.co/ostris/super-cereal-sdxl-lora). All you need to do is load and name this new adapter with `adapter_name`, and use the [`set_adapters`](https://huggingface.co/docs/diffusers/api/loaders/unet#diffusers.loaders.UNet2DConditionLoadersMixin.set_adapters) method to set it as the currently active adapter.\n\n```py\npipeline.load_lora_weights(\n \"ostris/super-cereal-sdxl-lora\", \n weight_name=\"cereal_box_sdxl_v1.safetensors\", \n adapter_name=\"cereal\"\n)\npipeline.set_adapters(\"cereal\")\nimage = pipeline(\"sushi rolls shaped like kawaii cat faces\").images[0]\nimage\n```\n\n\n\n ![](\"https://huggingface.co/datasets/ybelkada/documentation-images/resolve/main/test-lora-diffusers-2.png\"/)
\n
\n\nFinally, you can call the [`disable_lora`](https://huggingface.co/docs/diffusers/api/loaders/unet#diffusers.loaders.UNet2DConditionLoadersMixin.disable_lora) method to restore the base model.\n\n```py\npipeline.disable_lora()\n```\n\nLearn more about how PEFT supports Diffusers in the [Inference with PEFT](https://huggingface.co/docs/diffusers/tutorials/using_peft_for_inference) tutorial.\n\n## Transformers\n\n🤗 [Transformers](https://hf.co/docs/transformers) is a collection of pretrained models for all types of tasks in all modalities. You can load these models for training or inference. Many of the models are large language models (LLMs), so it makes sense to integrate PEFT with Transformers to manage and train adapters.\n\nLoad a base pretrained model to train.\n\n```py\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\"facebook/opt-350m\")\n```\n\nNext, add an adapter configuration to specify how to adapt the model parameters. Call the [`~PeftModel.add_adapter`] method to add the configuration to the base model.\n\n```py\nfrom peft import LoraConfig\n\npeft_config = LoraConfig(\n lora_alpha=16,\n lora_dropout=0.1,\n r=64,\n bias=\"none\",\n task_type=\"CAUSAL_LM\"\n)\nmodel.add_adapter(peft_config)\n```\n\nNow you can train the model with Transformer's [`~transformers.Trainer`] class or whichever training framework you prefer.\n\nTo use the newly trained model for inference, the [`~transformers.AutoModel`] class uses PEFT on the backend to load the adapter weights and configuration file into a base pretrained model.\n\n```py\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\"peft-internal-testing/opt-350m-lora\")\n```\n\nAlternatively, you can use transformers [Pipelines](https://huggingface.co/docs/transformers/en/main_classes/pipelines) to load the model for conveniently running inference:\n\n```py\nfrom transformers import pipeline\n\nmodel = pipeline(\"text-generation\", \"peft-internal-testing/opt-350m-lora\")\nprint(model(\"Hello World\"))\n```\n\nIf you're interested in comparing or using more than one adapter, you can call the [`~PeftModel.add_adapter`] method to add the adapter configuration to the base model. The only requirement is the adapter type must be the same (you can't mix a LoRA and LoHa adapter).\n\n```py\nfrom transformers import AutoModelForCausalLM\nfrom peft import LoraConfig\n\nmodel = AutoModelForCausalLM.from_pretrained(\"facebook/opt-350m\")\nmodel.add_adapter(lora_config_1, adapter_name=\"adapter_1\")\n```\n\nCall [`~PeftModel.add_adapter`] again to attach a new adapter to the base model.\n\n```py\nmodel.add_adapter(lora_config_2, adapter_name=\"adapter_2\")\n```\n\nThen you can use [`~PeftModel.set_adapter`] to set the currently active adapter.\n\n```py\nmodel.set_adapter(\"adapter_1\")\noutput = model.generate(**inputs)\nprint(tokenizer.decode(output_disabled[0], skip_special_tokens=True))\n```\n\nTo disable the adapter, call the [disable_adapters](https://github.com/huggingface/transformers/blob/4e3490f79b40248c53ee54365a9662611e880892/src/transformers/integrations/peft.py#L313) method.\n\n```py\nmodel.disable_adapters()\n```\n\nThe [enable_adapters](https://github.com/huggingface/transformers/blob/4e3490f79b40248c53ee54365a9662611e880892/src/transformers/integrations/peft.py#L336) can be used to enable the adapters again.\n\nIf you're curious, check out the [Load and train adapters with PEFT](https://huggingface.co/docs/transformers/main/peft) tutorial to learn more.\n\n\n\n\n# PEFT configurations and models\n\nThe sheer size of today's large pretrained models - which commonly have billions of parameters - present a significant training challenge because they require more storage space and more computational power to crunch all those calculations. You'll need access to powerful GPUs or TPUs to train these large pretrained models which is expensive, not widely accessible to everyone, not environmentally friendly, and not very practical. PEFT methods address many of these challenges. There are several types of PEFT methods (soft prompting, matrix decomposition, adapters), but they all focus on the same thing, reduce the number of trainable parameters. This makes it more accessible to train and store large models on consumer hardware.\n\nThe PEFT library is designed to help you quickly train large models on free or low-cost GPUs, and in this tutorial, you'll learn how to setup a configuration to apply a PEFT method to a pretrained base model for training. Once the PEFT configuration is setup, you can use any training framework you like (Transformer's [`~transformers.Trainer`] class, [Accelerate](https://hf.co/docs/accelerate), a custom PyTorch training loop).\n\n## PEFT configurations\n\n\n