{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"longbench-v2","formal_name":"LongBench v2","introduction":"LongBench v2 evaluates deep understanding and reasoning over long contexts through multiple-choice questions. Its official description lists 503 questions spanning tasks such as single-document and multi-document QA and code-repository understanding.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/zai-org/LongBench-v2","indexing_mode":"noindex","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[]}},"task_id":"925b33df-31e8-50f9-9269-fba6d11e7a27","task_key":"train--66fa3843bb02136c067c655d","task_revision_id":"3","upstream_id":"66fa3843bb02136c067c655d","short_description":"I plan to use this framework to train the glm-4v-9b model. Which of the follwing…","config":"","split":"train","body":"{\"choice_A\":\"I need to fine-tune the model using my own dataset, so I convert my dataset into the format of {\\\"query\\\": X, \\\"response\\\": Y, \\\"images\\\": [PATH]} and specify it directly using --dataset when starting fine-tuning\",\"choice_B\":\"I want to customize the training rounds and learning rate during fine-tuning, so I directly add the parameters num_train_epochs and learning_rate in swift sft\",\"choice_C\":\"I want to use multi-machine and multi-card training, so I need to specify the CUDA_VISIBLE_DEVICES, NNODES, NODE_RANK, MASTER_ADDR and NPROC_PER_NODE parameters\",\"choice_D\":\"After fine-tuning, I want to deploy the model service. I need to use swift infer --model_type glm4v-9b-chat \\\\ --infer_backend vllm for efficient deployment and inference\",\"context\":\"# SWIFT (Scalable lightWeight Infrastructure for Fine-Tuning)\\n\\n

\\n
\\n \\n
\\n

\\n

\\nModelScope Community Website\\n
\\n 中文   ｜   English  \\n

\\n\\n

\\n\\n\\n\\n\\n\\n\\n\\n

\\n\\n

\\n\\\"modelscope%2Fswift\\n

\\n\\n## 📖 Table of Contents\\n- [Introduction](#-introduction)\\n- [Groups](#-Groups)\\n- [News](#-news)\\n- [Installation](#%EF%B8%8F-installation)\\n- [Getting Started](#-getting-started)\\n- [Classroom](#-Classroom)\\n- [License](#-License)\\n- [Citation](#-citation)\\n\\n## 📝 Introduction\\nSWIFT supports training(PreTraining/Fine-tuning/RLHF), inference, evaluation and deployment of **350+ LLMs and 90+ MLLMs** (multimodal large models). Developers can directly apply our framework to their own research and production environments to realize the complete workflow from model training and evaluation to application. In addition to supporting the lightweight training solutions provided by [PEFT](https://github.com/huggingface/peft), we also provide a complete **Adapters library** to support the latest training techniques such as NEFTune, LoRA+, LLaMA-PRO, etc. This adapter library can be used directly in your own custom workflow without our training scripts.\\n\\nTo facilitate use by users unfamiliar with deep learning, we provide a Gradio web-ui for controlling training and inference, as well as accompanying deep learning courses and best practices for beginners. SWIFT web-ui is available both on [Huggingface space](https://huggingface.co/spaces/tastelikefeet/swift) and [ModelScope studio](https://www.modelscope.cn/studios/iic/Scalable-lightWeight-Infrastructure-for-Fine-Tuning/summary), please feel free to try!\\n\\nSWIFT has rich documentations for users, please feel free to check our documentation website:\\n

\\n Paper   ｜ English Documentation   ｜   中文文档  \\n

\\n\\n## ☎ Groups\\n\\nYou can contact us and communicate with us by adding our group:\\n\\n\\n[Discord Group](https://discord.com/invite/D27yfEFVz5) | 微信群\\n:-------------------------:|:-------------------------:\\n | \\n\\n## 🎉 News\\n- 2024.09.26: Support for training and deploying llama3.2-vision series models. Experience it using `swift infer --model_type llama3_2-11b-vision-instruct`.\\n- 2024.09.26: Support for training and deploying llama3.2 series models. Experience it using `swift infer --model_type llama3_2-1b-instruct`.\\n- 2024.09.25: Support for training to deployment with got-ocr2. Best practices can be found [here](https://github.com/modelscope/ms-swift/issues/2122).\\n- 2024.09.24: Support for training and deploying llama3_1-8b-omni. Experience it using `swift infer --model_type llama3_1-8b-omni`.\\n- 2024.09.23: Support for training and deploying pixtral-12b. Experience it using `swift infer --model_type pixtral-12b --dtype fp16`.\\n- 🔥2024.09.19: Supports the qwen2.5, qwen2.5-math, and qwen2.5-coder series models. Supports the qwen2-vl-72b series models. Best practices can be found [here](https://github.com/modelscope/ms-swift/issues/2064).\\n- 2024.09.07: Support the `Reflection-llama3-70b` model, use by `swift sft/infer --model_type reflection-llama_3_1-70b`.\\n- 2024.09.06: Support fine-tuning and inference for mplug-owl3. Best practices can be found [here](https://github.com/modelscope/ms-swift/issues/1969).\\n- 2024.09.05: Support for the minicpm3-4b model. Experience it using `swift infer --model_type minicpm3-4b`.\\n- 2024.09.05: Support for the yi-coder series models. Experience it using `swift infer --model_type yi-coder-1_5b-chat`.\\n- 🔥2024.08.30: Support for inference and fine-tuning of the qwen2-vl series models: qwen2-vl-2b-instruct, qwen2-vl-7b-instruct. The best practices can be found [here](docs/source_en/Multi-Modal/qwen2-vl-best-practice.md).\\n- 🔥2024.08.26: Support [Liger](https://github.com/linkedin/Liger-Kernel), which supports models like LLaMA、Qwen、Mistral and so on, and reduce memory usage by 10%~60%, use `--use_liger true` to begin.\\n- 🔥2024.08.22: Support `reft` tuner from [ReFT](https://github.com/stanfordnlp/pyreft) to achieve 15×–65× more parameter-efficient than LoRA, use `--sft_type reft` to begin!\\n- 🔥2024.08.21: Support for phi3_5-mini-instruct, phi3_5-moe-instruct, and phi3_5-vision-instruct. The best practices for fine-tuning Latex OCR using phi3_5-vision-instruct can be found [here](https://github.com/modelscope/ms-swift/issues/1809).\\n- 2024.08.21: Support for idefics3-8b-llama3, llava-onevision-qwen2-0_5b-ov, llava-onevision-qwen2-7b-ov, and llava-onevision-qwen2-72b-ov.\\n- 🔥2024.08.20: Support fine-tuning of multimodal large models using DeepSpeed-Zero3.\\n- 2024.08.20: Supported models: longwriter-glm4-9b, longwriter-llama3_1-8b. Supported dataset: longwriter-6k.\\n- 🔥2024.08.12: 🎉 SWIFT paper has been published to arXiv. Check [this link](https://arxiv.org/abs/2408.05517) to read.\\n- 🔥2024.08.12: Support packing with flash-attention without the contamination of attention_mask, use `--packing` to begin. Check[PR](https://github.com/huggingface/transformers/pull/31629/files).\\n- 🔥2024.08.09: Support for inference and fine-tuning of the qwen2-audio model. Best practice can be found [here](https://github.com/modelscope/ms-swift/issues/1653).\\n- 🔥2024.08.08: Supports the qwen2-math series models: 1.5B, 7B, 72B. Use `swift infer --model_type qwen2-math-1_5b-instruct` for an experience.\\n- 🔥2024.08.07: Support for using vLLM for accelerating inference and deployment of multimodal large models such as the llava series and phi3-vision models. You can refer to the [Multimodal & vLLM Inference Acceleration Documentation](docs/source_en/Multi-Modal/vllm-inference-acceleration.md) for more information.\\n- 2024.08.06: Support for minicpm-v-v2_6-chat is available. You can use `swift infer --model_type minicpm-v-v2_6-chat` for inference experience. Best practices can be found [here](https://github.com/modelscope/swift/issues/1613).\\n- 2024.08.06: Supports internlm2.5 series of 1.8b and 20b. Experience it using `swift infer --model_type internlm2_5-1_8b-chat`.\\n- 🔥2024.08.05: Support evaluation for multi-modal models! Same command with [new datasets](https://swift.readthedocs.io/en/latest/Instruction/LLM-eval.html#introduction).\\n- 🔥2024.08.02: Support Fourier Ft. Use `--sft_type fourierft` to begin, Check parameter documentation [here](https://swift.readthedocs.io/en/latest/Instruction/Command-line-parameters.html).\\n- 🔥2024.07.29: Support the use of lmdeploy for inference acceleration of LLM and VLM models. Documentation can be found [here](docs/source_en/Multi-Modal/LmDeploy-inference-acceleration.md).\\n- 🔥2024.07.24: Support DPO/ORPO/SimPO/CPO alignment algorithm for vision MLLM, training scripts can be find in [Document](docs/source_en/Multi-Modal/human-preference-alignment-training-documentation.md). support RLAIF-V dataset.\\n- 🔥2024.07.24: Support using Megatron for CPT and SFT on the Qwen2 series. You can refer to the [Megatron training documentation](docs/source_en/LLM/Megatron-training.md).\\n- 🔥2024.07.24: Support for the llama3.1 series models, including 8b, 70b, and 405b. Support for openbuddy-llama3_1-8b-chat.\\n
More\\n\\n- 2024.07.20: Support mistral-nemo series models. Use `--model_type mistral-nemo-base-2407` and `--model_type mistral-nemo-instruct-2407` to begin.\\n- 2024.07.19: Support [Q-Galore](https://arxiv.org/abs/2407.08296), this algorithm can reduce the training memory cost by 60% (qwen-7b-chat, full, 80G -> 35G), use `swift sft --model_type xxx --use_galore true --galore_quantization true` to begin!\\n- 2024.07.17: Support newly released InternVL2 models: `model_type` are internvl2-1b, internvl2-40b, internvl2-llama3-76b. For best practices, refer to [here](docs/source_en/Multi-Modal/internvl-best-practice.md).\\n- 2024.07.17: Support the training and inference of [NuminaMath-7B-TIR](https://huggingface.co/AI-MO/NuminaMath-7B-TIR). Use with model_type `numina-math-7b`.\\n- 🔥2024.07.16: Support exporting for ollama and bitsandbytes. Use `swift export --model_type xxx --to_ollama true` or `swift export --model_type xxx --quant_method bnb --quant_bits 4`\\n- 2024.07.08: Support cogvlm2-video-13b-chat. You can check the best practice [here](docs/source_en/Multi-Modal/cogvlm2-video-best-practice.md).\\n- 2024.07.08: Support internlm-xcomposer2_5-7b-chat. You can check the best practice [here](docs/source_en/Multi-Modal/internlm-xcomposer2-best-practice.md).\\n- 🔥2024.07.06: Support for the llava-next-video series models: llava-next-video-7b-instruct, llava-next-video-7b-32k-instruct, llava-next-video-7b-dpo-instruct, llava-next-video-34b-instruct. You can refer to [llava-video best practice](docs/source_en/Multi-Modal/llava-video-best-practice.md) for more information.\\n- 🔥2024.07.06: Support InternVL2 series: internvl2-2b, internvl2-4b, internvl2-8b, internvl2-26b.\\n- 2024.07.06: Support codegeex4-9b-chat.\\n- 2024.07.04: Support internlm2_5-7b series: internlm2_5-7b, internlm2_5-7b-chat, internlm2_5-7b-chat-1m.\\n- 2024.07.02: Support for `llava1_6-vicuna-7b-instruct`, `llava1_6-vicuna-13b-instruct` and other llava-hf models. For best practices, refer to [here](docs/source_en/Multi-Modal/llava-best-practice.md).\\n- 🔥2024.06.29: Support [eval-scope](https://github.com/modelscope/eval-scope)&[open-compass](https://github.com/open-compass/opencompass) for evaluation! Now we have supported over 50 eval datasets like `BoolQ, ocnli, humaneval, math, ceval, mmlu, gsk8k, ARC_e`, please check our [Eval Doc](https://github.com/modelscope/swift/blob/main/docs/source_en/Instruction/LLM-eval.md) to begin! Next sprint we will support Multi-modal and Agent evaluation, remember to follow us : )\\n\\n- 🔥2024.06.28: Support for **Florence** series model! See [document](docs/source_en/Multi-Modal/florence-best-pratice.md)\\n- 🔥2024.06.28: Support for Gemma2 series models: gemma2-9b, gemma2-9b-instruct, gemma2-27b, gemma2-27b-instruct.\\n- 🔥2024.06.18: Supports **DeepSeek-Coder-v2** series model! Use model_type `deepseek-coder-v2-instruct` and `deepseek-coder-v2-lite-instruct` to begin.\\n- 🔥2024.06.16: Supports **KTO** and **CPO** training! See [document](https://github.com/modelscope/swift/blob/main/docs/source_en/LLM/Human-Preference-Alignment-Training-Documentation.md) to start training!\\n- 2024.06.11: Support for tool-calling agent deployment that conform to the OpenAI interface.You can refer to [Agent deployment best practice](https://github.com/modelscope/swift/blob/main/docs/source_en/LLM/Agent-deployment-best-practice.md)\\n- 🔥2024.06.07: Support **Qwen2** series LLM, including Base and Instruct models of 0.5B, 1.5B, 7B, and 72B, as well as corresponding quantized versions gptq-int4, gptq-int8, and awq-int4. The best practice for self-cognition fine-tuning, inference and deployment of Qwen2-72B-Instruct using dual-card 80GiB A100 can be found [here](https://github.com/modelscope/swift/issues/1092).\\n- 🔥2024.06.05: Support for **glm4** series LLM and glm4v-9b-chat MLLM. You can refer to [glm4v best practice](docs/source_en/Multi-Modal/glm4v-best-practice.md).\\n- 🔥2024.06.01: Supports **SimPO** training! See [document](https://github.com/modelscope/swift/blob/main/docs/source_en/LLM/SimPO.md) to start training!\\n- 🔥2024.06.01: Support for deploying large multimodal models, please refer to the [Multimodal Deployment Documentation](docs/source_en/Multi-Modal/mutlimodal-deployment.md) for more information.\\n- 2024.05.31: Supports Mini-Internvl model, Use model_type `mini-internvl-chat-2b-v1_5` and `mini-internvl-chat-4b-v1_5`to train.\\n- 2024.05.24: Supports Phi3-vision model, Use model_type `phi3-vision-128k-instruct` to train.\\n- 2024.05.22: Supports DeepSeek-V2-Lite series models, model_type are `deepseek-v2-lite` and `deepseek-v2-lite-chat`\\n- 2024.05.22: Supports TeleChat-12B-v2 model with quantized version, model_type are `telechat-12b-v2` and `telechat-12b-v2-gptq-int4`\\n- 🔥2024.05.21: Inference and fine-tuning support for MiniCPM-Llama3-V-2_5 are now available. For more details, please refer to [minicpm-v-2.5 Best Practice](docs/source/Multi-Modal/minicpm-v-2.5最佳实践.md).\\n- 🔥2024.05.20: Support for inferencing and fine-tuning cogvlm2-llama3-chinese-chat-19B, cogvlm2-llama3-chat-19B. you can refer to [cogvlm2 Best Practice](docs/source_en/Multi-Modal/cogvlm2-best-practice.md).\\n- 🔥2024.05.17: Support peft=0.11.0. Meanwhile support 3 new tuners: `BOFT`, `Vera` and `Pissa`. use `--sft_type boft/vera` to use BOFT or Vera, use `--init_lora_weights pissa` with `--sft_type lora` to use Pissa.\\n- 2024.05.16: Supports Llava-Next (Stronger) series models. For best practice, you can refer to [here](https://github.com/modelscope/swift/tree/main/docs/source_en/Multi-Modal/llava-best-practice.md).\\n- 🔥2024.05.13: Support Yi-1.5 series models，use `--model_type yi-1_5-9b-chat` to begin!\\n- 2024.05.11: Support for qlora training and quantized inference using [hqq](https://github.com/mobiusml/hqq) and [eetq](https://github.com/NetEase-FuXi/EETQ). For more information, see the [LLM Quantization Documentation](https://github.com/modelscope/swift/tree/main/docs/source_en/Instruction/LLM-quantization-and-export.md).\\n- 2024.05.10: Support split a sequence to multiple GPUs to reduce memory usage. Use this feature by `pip install .[seq_parallel]`, then add `--sequence_parallel_size n` to your DDP script to begin!\\n- 2024.05.08: Support DeepSeek-V2-Chat model, you can refer to [this script](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/deepseek-v2-chat/lora_ddp_ds3/sft.sh).Support InternVL-Chat-V1.5-Int8 model, for best practice, you can refer to [here](https://github.com/modelscope/swift/tree/main/docs/source_en/Multi-Modal/internvl-best-practice.md).\\n- 🔥2024.05.07: Supoprts **ORPO** training! See [document](https://github.com/modelscope/swift/blob/main/docs/source_en/LLM/ORPO.md) to start training!\\n- 2024.05.07: Supports Llava-Llama3 model from xtuner，model_type is `llava-llama-3-8b-v1_1`.\\n- 2024.04.29: Supports inference and fine-tuning of InternVL-Chat-V1.5 model. For best practice, you can refer to [here](https://github.com/modelscope/swift/tree/main/docs/source_en/Multi-Modal/internvl-best-practice.md).\\n- 🔥2024.04.26: Support **LISA** and **unsloth** training! Specify `--lisa_activated_layers=2` to use LISA(to reduce the memory cost to 30 percent!), specify `--tuner_backend unsloth` to use unsloth to train a huge model(full or lora) with lesser memory(30 percent or lesser) and faster speed(5x)!\\n- 🔥2024.04.26: Support the fine-tuning and inference of Qwen1.5-110B and Qwen1.5-110B-Chat model, use [this script](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/qwen1half_110b_chat/lora_ddp_ds/sft.sh) to start training!\\n- 2024.04.24: Support for inference and fine-tuning of Phi3 series models. Including: [phi3-4b-4k-instruct](examples/pytorch/llm/scripts/phi3_4b_4k_instruct/lora), phi3-4b-128k-instruct.\\n- 2024.04.22: Support for inference, fine-tuning, and deployment of **chinese-llama-alpaca-2** series models. This includes：chinese-llama-2-1.3b, chinese-llama-2-7b, chinese-llama-2-13b, chinese-alpaca-2-1.3b, chinese-alpaca-2-7b and chinese-alpaca-2-13b along with their corresponding 16k and 64k long text versions.\\n- 2024.04.22: Support for inference and fine-tuning of Llama3 GPTQ-Int4, GPTQ-Int8, and AWQ series models. Support for inference and fine-tuning of chatglm3-6b-128k, Openbuddy-Llama3.\\n- 2024.04.20: Support for inference, fine-tuning, and deployment of **Atom** series models. This includes: Atom-7B and Atom-7B-Chat. use [this script](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/atom_7b_chat/lora/sft.sh) to train.\\n- 2024.04.19: Support for single-card, DDP, ZeRO2, and ZeRO3 training and inference with NPU, please refer to [NPU Inference and Fine-tuning Best Practice](docs/source_en/LLM/NPU-best-practice.md).\\n- 2024.04.19: Support for inference, fine-tuning, and deployment of **Llama3** series models. This includes: Llama-3-8B, Llama-3-8B-Instruct, Llama-3-70B, and Llama-3-70B-Instruct. use [this script](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/llama3_8b_instruct/lora/sft.sh) to train.\\n- 2024.04.18: Supported models: wizardlm2-7b-awq, wizardlm2-8x22b, yi-6b-chat-awq, yi-6b-chat-int8, yi-34b-chat-awq, yi-34b-chat-int8. Supported `--deepspeed zero3-offload` and provided default zero3-offload configuration file for zero3+cpu offload usage.\\n- 2024.04.18: Supported compatibility with HuggingFace ecosystem using the environment variable `USE_HF`, switching to use models and datasets from HF. Please refer to the [HuggingFace ecosystem compatibility documentation](https://github.com/modelscope/swift/tree/main/docs/source_en/LLM/Compat-HF.md).\\n- 2024.04.17: Support the evaluation for OpenAI standard interfaces. Check the [parameter documentation](docs/source_en/Instruction/Command-line-parameters.md#eval-parameters) for details.\\n- 🔥2024.04.17: Support **CodeQwen1.5-7B** series: CodeQwen1.5-7B, CodeQwen1.5-7B-Chat,CodeQwen1.5-7B-Chat-AWQ, use [this script](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/codeqwen1half_7b_chat/lora/sft.sh) to train.\\n- 2024.04.16: Supports inference and fine-tuning of llava-v1.6-34b model. For best practice, you can refer to [here](https://github.com/modelscope/swift/tree/main/docs/source_en/Multi-Modal/llava-best-practice.md).\\n- 2024.04.13: Support the fine-tuning and inference of Mixtral-8x22B-v0.1 model, use [this script](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/mixtral_moe_8x22b_v1/lora_ddp_ds/sft.sh) to start training!\\n- 2024.04.13: Support the newly launched **MiniCPM** series: MiniCPM-V-2.0、MiniCPM-2B-128k、MiniCPM-MoE-8x2B and MiniCPM-1B.use [this script](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/minicpm_moe_8x2b/lora_ddp/sft.sh) to start training!\\n- 🔥2024.04.11: Support Model Evaluation with MMLU/ARC/CEval datasets(also user custom eval datasets) with one command! Check [this documentation](docs/source_en/Instruction/LLM-eval.md) for details. Meanwhile, we support a trick way to do multiple ablation experiments, check [this documentation](docs/source_en/Instruction/LLM-exp.md) to use.\\n- 🔥2024.04.11: Support **c4ai-command-r** series: c4ai-command-r-plus, c4ai-command-r-v01, use [this script](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/c4ai_command_r_plus/lora_mp/sft.sh) to train.\\n- 2024.04.10: Use SWIFT to fine-tune the qwen-7b-chat model to enhance its function call capabilities, and combine it with [Modelscope-Agent](https://github.com/modelscope/modelscope-agent) for best practices, which can be found [here](https://github.com/modelscope/swift/tree/main/docs/source_en/LLM/Agent-best-practice.md#Usage-with-Modelscope_Agent).\\n- 🔥2024.04.09: Support ruozhiba dataset. Search `ruozhiba` in [this documentation](docs/source_en/Instruction/Supported-models-datasets.md) to begin training!\\n- 2024.04.08: Support the fine-tuning and inference of XVERSE-MoE-A4.2B model, use [this script](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/xverse_moe_a4_2b/lora/sft.sh) to start training!\\n- 2024.04.04: Support **QLoRA+FSDP** to train a 70B model with two 24G memory GPUs, use [this script](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/llama2_70b_chat/qlora_fsdp/sft.sh) to train.\\n- 🔥2024.04.03: Support **Qwen1.5-32B** series: Qwen1.5-32B, Qwen1.5-32B-Chat, Qwen1.5-32B-Chat-GPTQ-Int4.use [this script](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/qwen1half_32b_chat/lora_mp/sft.sh) to start training!\\n- 🔥2024.04.02: Support the fine-tuning and inference of Mengzi3-13B-Base model, use [this script](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/mengzi3_13b_base/lora_ddp_ds/sft.sh) to start training!\\n- 🔥2024.04.01: Support **dbrx** series: dbrx-base and dbrx-instruct, use [this script](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/dbrx-instruct/lora_mp/sft.sh) to start training!\\n- 🔥2024.03.29: Support **Qwen1.5-MoE** series: Qwen1.5-MoE-A2.7B, Qwen1.5-MoE-A2.7B-Chat, Qwen1.5-MoE-A2.7B-Chat-GPTQ-Int4.\\n- 🔥2024.03.29: Support the fine-tuning and inference of **Grok-1** 300B MoE, please view details [here](https://github.com/modelscope/swift/tree/main/docs/source_en/LLM/Grok-1-best-practice.md).\\n- 🔥2024.03.25: Supports inference and fine-tuning of TeleChat-7b and TeleChat-12b model, use [this script](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/telechat_12b/lora/sft.sh) to start training!\\n- 🔥2024.03.20: Supports inference and fine-tuning for the **llava** series. For best practice, you can refer to [here](https://github.com/modelscope/swift/tree/main/docs/source_en/Multi-Modal/llava-best-practice.md).\\n- 🔥2024.03.12: Support inference and fine-tuning for **deepseek-vl** series. Best practices can be found [here](docs/source_en/Multi-Modal/deepseek-vl-best-practice.md).\\n- 🔥2024.03.11: Support [GaLore](https://arxiv.org/abs/2403.03507) for effectively reducing memory usage to 1/2 of the original in full-parameter training.\\n- 🔥2024.03.10: [End-to-end best practices](docs/source_en/LLM/Qwen1.5-best-practice.md) from fine-tuning to deployment for Qwen1.5-7B-Chat and Qwen1.5-72B-Chat.\\n- 🔥2024.03.09: Support training and inference of MAMBA model, use [this script](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/mamba-1.4b/lora/sft.sh) to start training!\\n- 2024.03.09: Support training and inference of AQLM quantized model, use [this script](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/llama2_7b_aqlm_2bit_1x16/lora/sft.sh) to start training!\\n- 2024.03.06: Support training and inference of AWQ quantized model, use [this Qwen1.5-AWQ model script](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/qwen1half_7b_chat_awq/lora/sft.sh) to start training, and support training and inference of [yi-9b](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/yi_9b/lora_zero3).\\n- 🔥2024.02.29: Support [LLaMA PRO](https://arxiv.org/pdf/2401.02415.pdf), simply use [this script](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/yi_6b_chat/llamapro/sft.sh) to start training.\\n- 🔥2024.02.29: Support [LoRA+](https://arxiv.org/pdf/2402.12354.pdf), simply use [this script](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/yi_6b_chat/lorap/sft.sh) to start training.\\n- 2024.02.25: Support `swift export` to quantize models using **AWQ/GPTQ** and push to ModelScope Hub. See documentation: [LLM Quantization](docs/source_en/Instruction/LLM-quantization-and-export.md).\\n- 2024.02.22: Support gemma series: gemma-2b, [gemma-2b-instruct](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/gemma_2b_instruct), gemma-7b, gemma-7b-instruct.\\n- 2024.02.16: Support deepseek-math series: deepseek-math-7b, deepseek-math-7b-instruct, deepseek-math-7b-chat.\\n- 🔥2024.02.05: Support **Qwen1.5** series models, see [model list](https://github.com/modelscope/swift/blob/main/docs/source/Instruction/%E6%94%AF%E6%8C%81%E7%9A%84%E6%A8%A1%E5%9E%8B%E5%92%8C%E6%95%B0%E6%8D%AE%E9%9B%86.md#%E6%A8%A1%E5%9E%8B) for all supported Qwen1.5 models. Provide fine-tuning scripts for [qwen1half-7b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen1half_7b_chat), [qwen1half-7b-chat-int8](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen1half_7b_chat_int8).\\n- 2024.02.05: Support training of diffusion models such as **SDXL**, **SD**, **ControlNet**, as well as **DreamBooth** training. See corresponding [training scripts](https://github.com/modelscope/swift/tree/main/examples/pytorch/sdxl/scripts) for details.\\n- 2024.02.01: Support minicpm series: [minicpm-2b-sft-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/minicpm_2b_sft_chat), minicpm-2b-chat.\\n- 🔥2024.02.01: Support dataset mixing to reduce **catastrophic forgetting**. Use `--train_dataset_mix_ratio 2.0` to enable training! We also open sourced the general knowledge dataset [ms-bench](https://www.modelscope.cn/datasets/iic/ms_bench/summary).\\n- 🔥2024.02.01: Support Agent training! Agent training algorithm is derived from this [paper](https://arxiv.org/pdf/2309.00986.pdf). We also added [ms-agent](https://www.modelscope.cn/datasets/iic/ms_agent/summary), a high-quality agent dataset. Use [this script](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/qwen_7b_chat/lora/sft.sh) to start Agent training!\\n- 🔥2024.02.01: Support adding SFT loss in DPO training to reduce repetitive generation caused by KL divergence loss.\\n- 2024.02.01: Support using AdaLoRA and IA3 adapters in training.\\n- 2024.02.01: Support `--merge_lora` parameter in AnimateDiff training.\\n- 2024.01.30: Support [internlm-xcomposer2-7b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/internlm_xcomposer2_7b_chat).\\n- 🔥2024.01.30: Support [ZeRO-3](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat/full_ddp_zero3/), simply specify `--deepspeed default-zero3`.\\n- 2024.01.29: Support internlm2-math series: internlm2-math-7b, internlm2-math-7b-chat, internlm2-math-20b, internlm2-math-20b-chat.\\n- 🔥2024.01.26: Support [yi-vl-6b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/yi_vl_6b_chat), yi-vl-34b-chat.\\n- 2024.01.24: Support codefuse-codegeex2-6b-chat, codefuse-qwen-14b-chat.\\n- 2024.01.23: Support orion series: orion-14b, [orion-14b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/orion_14b_chat).\\n- 2024.01.20: Support [xverse-13b-256k](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/xverse_13b_256k), xverse-65b-v2, xverse-65b-chat.\\n- 🔥2024.01.17: Support internlm2 series: internlm2-7b-base, internlm2-7b, [internlm2-7b-sft-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/internlm2_7b_sft_chat), internlm2-7b-chat, internlm2-20b-base, internlm2-20b, internlm2-20b-sft-chat, internlm2-20b-chat.\\n- 2024.01.15: Support yuan series: yuan2-2b-instruct, [yuan2-2b-janus-instruct](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/yuan2_2b_janus_instruct), yuan2-51b-instruct, yuan2-102b-instruct.\\n- 🔥2024.01.12: Support **deepseek-moe** series: deepseek-moe-16b, [deepseek-moe-16b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/deepseek_moe_16b_chat).\\n- 🔥2024.01.04: Support **VLLM deployment**, compatible with **OpenAI API** style, see [VLLM Inference Acceleration and Deployment](https://github.com/modelscope/swift/blob/main/docs/source_en/LLM/VLLM-inference-acceleration-and-deployment.md#Deployment) for details.\\n- 2024.01.04: Update [Benchmark](https://github.com/modelscope/swift/blob/main/docs/source/LLM/Benchmark.md) for convenient viewing of training speed and memory usage of different models.\\n- 🔥2023.12.29: Support web-ui for sft training and inference, use `swift web-ui` after installing ms-swift to start.\\n- 🔥2023.12.29: Support DPO RLHF (Reinforcement Learning from Human Feedback) and three datasets for this task: AI-ModelScope/stack-exchange-paired, AI-ModelScope/hh-rlhf and AI-ModelScope/hh_rlhf_cn. See [documentation](https://github.com/modelscope/swift/blob/main/docs/source_en/LLM/DPO.md) to start training!\\n- 🔥2023.12.28: Support SCEdit! This tuner can significantly reduce memory usage in U-Net and support low-memory controllable image generation (replacing ControlNet), read the section below to learn more.\\n- 2023.12.23: Support [codegeex2-6b](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/codegeex2_6b).\\n- 2023.12.19: Support [phi2-3b](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/phi2_3b).\\n- 2023.12.18: Support VLLM for inference acceleration.\\n- 2023.12.15: Support deepseek, deepseek-coder series: deepseek-7b, deepseek-7b-chat, deepseek-67b, deepseek-67b-chat, openbuddy-deepseek-67b-chat, deepseek-coder-1_3b, deepseek-coder-1_3b-instruct, deepseek-coder-6_7b, deepseek-coder-6_7b-instruct, deepseek-coder-33b, deepseek-coder-33b-instruct.\\n- 2023.12.13: Support mistral-7b-instruct-v2, [mixtral-moe-7b](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/mixtral_7b_moe), [mixtral-moe-7b-instruct](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/mixtral_7b_moe_instruct).\\n- 2023.12.09: Support `freeze_parameters_ratio` parameter as a compromise between lora and full-parameter training. Corresponding sh can be found in [full_freeze_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/full_freeze_ddp). Support `disable_tqdm`, `lazy_tokenize`, `preprocess_num_proc` parameters, see [command line arguments](https://github.com/modelscope/swift/blob/main/docs/source/Instruction/%E5%91%BD%E4%BB%A4%E8%A1%8C%E5%8F%82%E6%95%B0.md) for details.\\n- 2023.12.08: Support [sus-34b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/sus_34b_chat), support yi-6b-200k, yi-34b-200k.\\n- 2023.12.07: Support [Multi-Node DDP training](https://github.com/modelscope/swift/blob/main/docs/source/Instruction/LLM%E5%BE%AE%E8%B0%83%E6%96%87%E6%A1%A3.md#%E4%BD%BF%E7%94%A8cli).\\n- 2023.12.05: Support models: zephyr-7b-beta-chat, openbuddy-zephyr-7b-chat. Support datasets: hc3-zh, hc3-en.\\n- 🔥2023.12.02: [Self-cognition fine-tuning best practices](docs/source_en/LLM/Self-cognition-best-practice.md), **10 minutes to fine-tune a large model for self-cognition**, create your own unique large model.\\n- 🔥2023.11.30: Support training and inference of **qwen-1_8b**, **qwen-72b**, **qwen-audio** series models. Corresponding sh scripts can be found in [qwen_1_8b_chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_1_8b_chat), [qwen_72b_chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_72b_chat), [qwen_audio_chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_audio_chat)\\n- 🔥2023.11.29: Support training and inference of **AnimateDiff**\\n- 🔥2023.11.24: Support **yi-34b-chat**, **codefuse-codellama-34b-chat** models. Corresponding sh scripts can be found in [yi_34b_chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/yi_34b_chat), [codefuse_codellama_34b_chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/codefuse_codellama_34b_chat).\\n- 🔥2023.11.18: Support **tongyi-finance-14b** series models: tongyi-finance-14b, tongyi-finance-14b-chat, tongyi-finance-14b-chat-int4. Corresponding sh scripts can be found in [tongyi_finance_14b_chat_int4](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/tongyi_finance_14b_chat_int4).\\n- 2023.11.16: Support **flash attn** for more models: qwen series, qwen-vl series, llama series, openbuddy series, mistral series, yi series, ziya series. Please use `use_flash_attn` parameter.\\n- 🔥2023.11.11: Support **NEFTune**, simply use `Swift.prepare_model(model, NEFTuneConfig())` to enable.\\n- 🔥2023.11.11: Support training and inference by **command line** and inference by **Web-UI**, see `Usage with Swift CLI` section below for details.\\n- 🔥2023.11.10: Support **bluelm** series models: bluelm-7b, bluelm-7b-chat, bluelm-7b-32k, bluelm-7b-chat-32k. Corresponding sh scripts can be found in [bluelm_7b_chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/bluelm_7b_chat).\\n- 🔥2023.11.08: Support training and inference of **xverse-65b** model, script at [xverse_65b](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/xverse_65b).\\n- 🔥2023.11.07: Support training and inference of **yi-6b**, **yi-34b** models, scripts at [yi_6b](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/yi_6b), [yi_34b](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/yi_34b).\\n- 🔥2023.10.30: Support two new tuners: **QA-LoRA** and **LongLoRA**.\\n- 🔥2023.10.30: Support editing models using **ROME** (Rank One Model Editing) to infuse new knowledge into models without training!\\n- 2023.10.30: Support **skywork-13b** series models: skywork-13b, skywork-13b-chat. Corresponding sh scripts can be found in [skywork_13b](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/skywork_13b).\\n- 🔥2023.10.27: Support **chatglm3** series models: chatglm3-6b-base, chatglm3-6b, chatglm3-6b-32k. Corresponding sh scripts can be found in [chatglm3_6b](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/chatglm3_6b).\\n- 🔥2023.10.17: Support SFT of **int4**, **int8** models: qwen-7b-chat-int4, qwen-14b-chat-int4, qwen-vl-chat-int4, baichuan2-7b-chat-int4, baichuan2-13b-chat-int4, qwen-7b-chat-int8, qwen-14b-chat-int8.\\n- 2023.10.15: Support **ziya2-13b** series models: ziya2-13b, ziya2-13b-chat.\\n- 2023.10.12: Support **mistral-7b** series models: openbuddy-mistral-7b-chat, mistral-7b, mistral-7b-instruct.\\n- 🔥2023.10.07: Support **DeepSpeed ZeRO-2**, enabling lora (not just qlora) to run DDP on dual A10 cards.\\n- 2023.10.04: Support more math, law, SQL, code domain datasets: blossom-math-zh, school-math-zh, text2sql-en, sql-create-context-en, lawyer-llama-zh, tigerbot-law-zh, leetcode-python-en.\\n- 🔥2023.09.25: Support **qwen-14b** series: qwen-14b, qwen-14b-chat.\\n- 2023.09.18: Support **internlm-20b** series: internlm-20b, internlm-20b-chat.\\n- 2023.09.12: Support **MP+DDP** to accelerate full-parameter training.\\n- 2023.09.05: Support **openbuddy-llama2-70b-chat**.\\n- 2023.09.03: Support **baichuan2** series: baichuan2-7b, baichuan2-7b-chat, baichuan2-13b, baichuan2-13b-chat.\\n
\\n\\n## 🛠️ Installation\\n\\nSWIFT runs in the Python environment. Please ensure your Python version is higher than 3.8.\\n\\n- Method 1: Install SWIFT using pip command:\\n\\n```shell\\n# Full capabilities\\npip install 'ms-swift[all]' -U\\n# LLM only\\npip install 'ms-swift[llm]' -U\\n# AIGC only\\npip install 'ms-swift[aigc]' -U\\n# Adapters only\\npip install ms-swift -U\\n```\\n\\n- Method 2: Install SWIFT through source code (convenient for running training and inference scripts), please run the following commands:\\n\\n```shell\\ngit clone https://github.com/modelscope/swift.git\\ncd swift\\npip install -e '.[llm]'\\n```\\n\\nSWIFT depends on torch>=1.13, recommend torch>=2.0.0.\\n\\n- Method 3: Use SWIFT in our [Docker image](https://www.modelscope.cn/docs/%E7%8E%AF%E5%A2%83%E5%AE%89%E8%A3%85)\\n\\n## 🚀 Getting Started\\n\\nThis section introduces basic usage, see the [Documentation](https://swift.readthedocs.io/en/latest/) section for more ways to use.\\n\\n### Web-UI\\n\\nWeb-UI is a gradio-based interface for **zero-threshold** training and deployment. It is easy to use and perfectly supports multi-GPU training and deployment:\\n\\n```shell\\nSWIFT_UI_LANG=en swift web-ui\\n```\\n\\n![image.png](./docs/resources/web-ui-en.jpg)\\n\\n### Training\\n\\n#### Training Scripts\\nYou can refer to the following scripts to customize your own training script.\\n\\n- full: [qwen1half-7b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen1half_7b_chat/full) (A100), [qwen-7b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/full_mp) (2\\\\*A100)\\n- full+ddp+zero2: [qwen-7b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/full_ddp_zero2) (4\\\\*A100)\\n- full+ddp+zero3: [qwen-14b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat/full_ddp_zero3) (4\\\\*A100)\\n- lora: [chatglm3-6b](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/chatglm3_6b/lora) (3090), [baichuan2-13b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/baichuan2_13b_chat/lora_mp) (2\\\\*3090), [yi-34b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/yi_34b_chat/lora) (A100), [qwen-72b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_72b_chat/lora_mp) (2\\\\*A100)\\n- lora+ddp: [chatglm3-6b](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/chatglm3_6b/lora_ddp) (2\\\\*3090)\\n- lora+ddp+zero3: [qwen-14b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat/lora_ddp_zero3) (4\\\\*3090), [qwen-72b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_72b_chat/lora_ddp_zero3) (4\\\\*A100)\\n- qlora(gptq-int4): [qwen-7b-chat-int4](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat_int4/qlora) (3090)\\n- qlora(gptq-int8): [qwen1half-7b-chat-int8](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen1half_7b_chat_int8/qlora) (3090)\\n- qlora(bnb-int4): [qwen-7b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/qlora) (3090)\\n\\n\\n#### Supported Training Processes\\n\\n| Training Process | Training Method |\\n|------------------|-------------------------------------------------------------------------------|\\n| Pretraining | Text Generation |\\n| Fine-tuning | Single-turn/Multi-turn
Agent Training/Self-cognition
Multi-modal Vision/Multi-modal Speech|\\n| Human Alignment | DPO
ORPO
SimPO
CPO
KTO |\\n| Text-to-Image | DreamBooth, etc. |\\n| Text-to-Video | - |\\n\\n#### Single GPU Training\\n\\nStart single GPU fine-tuning with the following command:\\n\\nLoRA:\\n```shell\\n# Experimental Environment: A100\\n# GPU Memory Requirement: 20GB\\n# Runtime: 3.1 hours\\nCUDA_VISIBLE_DEVICES=0 \\\\\\nswift sft \\\\\\n --model_type qwen1half-7b-chat \\\\\\n --dataset blossom-math-zh \\\\\\n --num_train_epochs 5 \\\\\\n --sft_type lora \\\\\\n --output_dir output \\\\\\n --eval_steps 200 \\\\\\n```\\n\\nFull-parameter:\\n```shell\\n# Experimental Environment: A100\\n# GPU Memory Requirement: 80GB\\n# Runtime: 2.5 hours\\nCUDA_VISIBLE_DEVICES=0 \\\\\\nswift sft \\\\\\n --model_type qwen1half-7b-chat \\\\\\n --dataset blossom-math-zh \\\\\\n --num_train_epochs 5 \\\\\\n --sft_type full \\\\\\n --output_dir output \\\\\\n --eval_steps 500 \\\\\\n```\\n\\n\\n#### Model Parallel Training\\n\\n\\n```shell\\n# Experimental Environment: 2 * A100\\n# GPU Memory Requirement: 10GB + 13GB\\n# Runtime: 3.4 hours\\nCUDA_VISIBLE_DEVICES=0,1 \\\\\\nswift sft \\\\\\n --model_type qwen1half-7b-chat \\\\\\n --dataset blossom-math-zh \\\\\\n --num_train_epochs 5 \\\\\\n --sft_type lora \\\\\\n --output_dir output \\\\\\n```\\n\\n#### Data Parallel Training\\n\\n```shell\\n# Experimental Environment: 4 * A100\\n# GPU Memory Requirement: 4 * 30GB\\n# Runtime: 0.8 hours\\nNPROC_PER_NODE=4 \\\\\\nCUDA_VISIBLE_DEVICES=0,1,2,3 \\\\\\nswift sft \\\\\\n --model_type qwen1half-7b-chat \\\\\\n --dataset blossom-math-zh \\\\\\n --num_train_epochs 5 \\\\\\n --sft_type lora \\\\\\n --output_dir output \\\\\\n```\\n\\nCombining Model Parallelism and Data Parallelism:\\n```shell\\n# Experimental Environment: 4 * A100\\n# GPU Memory Requirement: 2*14GB + 2*18GB\\n# Runtime: 1.7 hours\\nNPROC_PER_NODE=2 \\\\\\nCUDA_VISIBLE_DEVICES=0,1,2,3 \\\\\\nswift sft \\\\\\n --model_type qwen1half-7b-chat \\\\\\n --dataset blossom-math-zh \\\\\\n --num_train_epochs 5 \\\\\\n --sft_type lora \\\\\\n --output_dir output \\\\\\n```\\n\\n#### Deepspeed Training\\nDeepspeed supports training of quantized GPTQ and AWQ models.\\n\\nZeRO2:\\n```shell\\n# Experimental Environment: 4 * A100\\n# GPU Memory Requirement: 4 * 21GB\\n# Runtime: 0.9 hours\\nNPROC_PER_NODE=4 \\\\\\nCUDA_VISIBLE_DEVICES=0,1,2,3 \\\\\\nswift sft \\\\\\n --model_type qwen1half-7b-chat \\\\\\n --dataset blossom-math-zh \\\\\\n --num_train_epochs 5 \\\\\\n --sft_type lora \\\\\\n --output_dir output \\\\\\n --deepspeed default-zero2 \\\\\\n```\\n\\nZeRO3:\\n```shell\\n# Experimental Environment: 4 * A100\\n# GPU Memory Requirement: 4 * 19GB\\n# Runtime: 3.2 hours\\nNPROC_PER_NODE=4 \\\\\\nCUDA_VISIBLE_DEVICES=0,1,2,3 \\\\\\nswift sft \\\\\\n --model_type qwen1half-7b-chat \\\\\\n --dataset blossom-math-zh \\\\\\n --num_train_epochs 5 \\\\\\n --sft_type lora \\\\\\n --output_dir output \\\\\\n --deepspeed default-zero3 \\\\\\n```\\n\\nZeRO3-Offload:\\n```shell\\n# Experimental Environment: 4 * A100\\n# GPU Memory Requirement: 4 * 12GB\\n# Runtime: 60 hours\\nNPROC_PER_NODE=4 \\\\\\nCUDA_VISIBLE_DEVICES=0,1,2,3 \\\\\\nswift sft \\\\\\n --model_id_or_path AI-ModelScope/WizardLM-2-8x22B \\\\\\n --dataset blossom-math-zh \\\\\\n --num_train_epochs 5 \\\\\\n --sft_type lora \\\\\\n --output_dir output \\\\\\n --deepspeed zero3-offload \\\\\\n```\\n\\n\\n#### Multi-node Multi-GPU\\n```shell\\n# If the disk is not shared, please additionally specify `--save_on_each_node true` in the shell scripts on each machine.\\n# node0\\nCUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \\\\\\nNNODES=2 \\\\\\nNODE_RANK=0 \\\\\\nMASTER_ADDR=127.0.0.1 \\\\\\nNPROC_PER_NODE=8 \\\\\\nswift sft \\\\\\n --model_type qwen1half-32b-chat \\\\\\n --sft_type full \\\\\\n --dataset blossom-math-zh \\\\\\n --output_dir output \\\\\\n --deepspeed default-zero3 \\\\\\n\\n# node1\\nCUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \\\\\\nNNODES=2 \\\\\\nNODE_RANK=1 \\\\\\nMASTER_ADDR=xxx.xxx.xxx.xxx \\\\\\nNPROC_PER_NODE=8 \\\\\\nswift sft \\\\\\n --model_type qwen1half-32b-chat \\\\\\n --sft_type full \\\\\\n --dataset blossom-math-zh \\\\\\n --output_dir output \\\\\\n --deepspeed default-zero3 \\\\\\n```\\n\\n##### AliYun-DLC multi-node training\\nIn DLC product, WORLD_SIZE is the node number, RANK is the node index, this is different from the definition of torchrun.\\n\\n```shell\\nNNODES=$WORLD_SIZE \\\\\\nNODE_RANK=$RANK \\\\\\nswift sft \\\\\\n --model_type qwen1half-32b-chat \\\\\\n --sft_type full \\\\\\n --dataset blossom-math-zh \\\\\\n --output_dir output \\\\\\n --deepspeed default-zero3\\n```\\n\\n#### Pretraining\\n\\n```shell\\n# Experimental Environment: 4 * A100\\n# GPU Memory Requirement: 4 * 30GB\\n# Runtime: 0.8 hours\\nNPROC_PER_NODE=4 \\\\\\nCUDA_VISIBLE_DEVICES=0,1,2,3 \\\\\\nswift pt \\\\\\n --model_type qwen1half-7b \\\\\\n --dataset chinese-c4#100000 \\\\\\n --num_train_epochs 1 \\\\\\n --sft_type full \\\\\\n --deepspeed default-zero3 \\\\\\n --output_dir output \\\\\\n --lazy_tokenize true\\n```\\n\\n\\n#### RLHF\\n\\n```shell\\n# We support rlhf_type dpo/cpo/simpo/orpo/kto\\nCUDA_VISIBLE_DEVICES=0 \\\\\\nswift rlhf \\\\\\n --rlhf_type dpo \\\\\\n --model_type qwen1half-7b-chat \\\\\\n --dataset shareai-llama3-dpo-zh-en-emoji \\\\\\n --num_train_epochs 5 \\\\\\n --sft_type lora \\\\\\n --output_dir output \\\\\\n```\\n\\n\\n### Inference\\nOriginal model:\\n```shell\\nCUDA_VISIBLE_DEVICES=0 swift infer --model_type qwen1half-7b-chat\\n# use VLLM\\nCUDA_VISIBLE_DEVICES=0 swift infer --model_type qwen1half-7b-chat \\\\\\n --infer_backend vllm --max_model_len 8192\\n```\\n\\nLoRA fine-tuned:\\n```shell\\nCUDA_VISIBLE_DEVICES=0 swift infer --ckpt_dir xxx/checkpoint-xxx --load_dataset_config true\\n# use VLLM\\nCUDA_VISIBLE_DEVICES=0 swift infer \\\\\\n --ckpt_dir xxx/checkpoint-xxx --load_dataset_config true \\\\\\n --merge_lora true --infer_backend vllm --max_model_len 8192\\n```\\n\\n### Evaluation\\n\\nOriginal model:\\n```shell\\nCUDA_VISIBLE_DEVICES=0 swift eval --model_type qwen1half-7b-chat \\\\\\n --eval_dataset ARC_c --infer_backend vllm\\n```\\n\\nLoRA fine-tuned:\\n```shell\\nCUDA_VISIBLE_DEVICES=0 swift eval --ckpt_dir xxx/checkpoint-xxx \\\\\\n --eval_dataset ARC_c --infer_backend vllm \\\\\\n --merge_lora true \\\\\\n```\\n\\n### Quantization\\n\\nOriginal model:\\n```shell\\nCUDA_VISIBLE_DEVICES=0 swift export --model_type qwen1half-7b-chat \\\\\\n --quant_bits 4 --quant_method awq\\n```\\n\\nLoRA fine-tuned:\\n```shell\\nCUDA_VISIBLE_DEVICES=0 swift export \\\\\\n --ckpt_dir xxx/checkpoint-xxx --load_dataset_config true \\\\\\n --quant_method awq --quant_bits 4 \\\\\\n --merge_lora true \\\\\\n```\\n\\n### Deployment\\nThe client uses the OpenAI API for invocation, for details refer to the [LLM deployment documentation](https://github.com/modelscope/swift/blob/main/docs/source_en/LLM/VLLM-inference-acceleration-and-deployment.md).\\n\\nOriginal model:\\n```shell\\nCUDA_VISIBLE_DEVICES=0 swift deploy --model_type qwen1half-7b-chat\\n# 使用VLLM加速\\nCUDA_VISIBLE_DEVICES=0 swift deploy --model_type qwen1half-7b-chat \\\\\\n --infer_backend vllm --max_model_len 8192\\n```\\n\\nLoRA fine-tuned:\\n```shell\\nCUDA_VISIBLE_DEVICES=0 swift deploy --ckpt_dir xxx/checkpoint-xxx\\n# 使用VLLM加速\\nCUDA_VISIBLE_DEVICES=0 swift deploy \\\\\\n --ckpt_dir xxx/checkpoint-xxx --merge_lora true \\\\\\n --infer_backend vllm --max_model_len 8192\\n```\\n\\n### Supported Models\\nThe complete list of supported models and datasets can be found at [Supported Models and Datasets List](docs/source_en/Instruction/Supported-models-datasets.md).\\n\\n#### LLMs\\n\\n| Model Type | Model Introduction | Language | Model Size | Model Type |\\n|-------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------|--------------------|-------------------------------------------|-------------------------------------------------------------------|\\n| Qwen
Qwen1.5
Qwen2
Qwen2.5 | [Tongyi Qwen series models](https://github.com/QwenLM) | Chinese
English | 0.5B-110B
including quantized versions | base model
chat model
MoE model
code model |\\n| ChatGLM2
ChatGLM3
Codegeex2
GLM4
Codegeex4 | [Zhipu ChatGLM series models](https://github.com/THUDM) | Chinese
English | 6B-9B | base model
chat model
code model
long text model |\\n| Baichuan
Baichuan2 | [Baichuan 1 and Baichuan 2](https://github.com/baichuan-inc) | Chinese
English | 7B-13B
including quantized versions | base model
chat model |\\n| Yuan2 | [Langchao Yuan series models](https://github.com/IEIT-Yuan) | Chinese
English | 2B-102B | instruct model |\\n| XVerse | [XVerse series models](https://github.com/xverse-ai) | Chinese
English | 7B-65B | base model
chat model
long text model
MoE model |\\n| LLaMA2 | [LLaMA2 series models](https://github.com/facebookresearch/llama) | English | 7B-70B
including quantized versions | base model
chat model |\\n| LLaMA3
LLaMA3.1
Llama3.2 | [LLaMA3 series models](https://github.com/meta-llama/llama3) | English | 1B-70B
including quantized versions | base model
chat model |\\n| Mistral
Mixtral | [Mistral series models](https://github.com/mistralai/mistral-src) | English | 7B-22B | base model
instruct model
MoE model |\\n| Yi
Yi1.5
Yi-Coder | [01AI's YI series models](https://github.com/01-ai) | Chinese
English | 1.5B-34B
including quantized | base model
chat model
long text model |\\n| InternLM
InternLM2
InternLM2-Math
InternLM2.5 | [Pujiang AI Lab InternLM series models](https://github.com/InternLM/InternLM) | Chinese
English | 1.8B-20B | base model
chat model
math model |\\n| DeepSeek
DeepSeek-MoE
DeepSeek-Coder
DeepSeek-Math
DeepSeek-V2
DeepSeek-Coder-V2 | [DeepSeek series models](https://github.com/deepseek-ai) | Chinese
English | 1.3B-236B | base model
chat model
MoE model
code model
math model |\\n| MAMBA | [MAMBA temporal convolution model](https://github.com/state-spaces/mamba) | English | 130M-2.8B | base model |\\n| Gemma
Gemma2 | [Google Gemma series models](https://github.com/google/gemma_pytorch) | English | 2B-27B | base model
instruct model |\\n| MiniCPM
MiniCPM3 | [OpenBmB MiniCPM series models](https://github.com/OpenBMB/MiniCPM) | Chinese
English | 2B-3B | chat model
MoE model |\\n| OpenBuddy | [OpenBuddy series models](https://github.com/OpenBuddy/OpenBuddy) | Chinese
English | 7B-70B | base model
chat model |\\n| Orion | [OrionStar AI series models](https://github.com/OrionStarAI) | Chinese
English | 14B | base model
chat model |\\n| BlueLM | [VIVO BlueLM large model](https://github.com/vivo-ai-lab/BlueLM) | Chinese
English | 7B | base model
chat model |\\n| Ziya2 | [Fengshenbang series models](https://github.com/IDEA-CCNL/Fengshenbang-LM) | Chinese
English | 13B | base model
chat model |\\n| Skywork | [Skywork series models](https://github.com/SkyworkAI/Skywork) | Chinese
English | 13B | base model
chat model |\\n| Zephyr | Zephyr series models based on Mistral | English | 7B | chat model |\\n| PolyLM | [Tongyi Lab self-developed PolyLM series models](https://github.com/DAMO-NLP-MT/PolyLM) | Multilingual | 13B | base model |\\n| SeqGPT | [Tongyi Lab self-developed text understanding model for information extraction and text classification](https://github.com/Alibaba-NLP/SeqGPT) | Chinese | 560M | semantic understanding model |\\n| SUS | [Southern University of Science and Technology model fine-tuned on YI](https://github.com/SUSTech-IDEA/SUS-Chat) | Chinese
English | 34B | chat model |\\n| Tongyi-Finance | [Tongyi finance series models](https://github.com/QwenLM/Qwen) | Chinese
English | 14B | base model
chat model
financial model |\\n| CodeFuse-CodeLLaMA
CodeFuse-Codegeex2
CodeFuse-Qwen | [Ant CodeFuse series models](https://github.com/codefuse-ai) | Chinese
English | 6B-34B | chat model
code model |\\n| phi2/phi3 | Microsoft's PHI series models | English | 3B/4B | base model
instruct model
code model |\\n| Grok | [X-ai](https://github.com/xai-org/grok-1) | English | 300B | base model |\\n| TeleChat | [Tele-AI](https://github.com/Tele-AI/Telechat) | Chinese
English | 7B-12B | chat model |\\n| dbrx | [databricks](https://github.com/databricks/dbrx) | English | 132B | base model
chat model |\\n| mengzi3 | [Langboat](https://github.com/Langboat/Mengzi3) | Chinese
English | 13B | base model |\\n| c4ai-command-r | [c4ai](https://cohere.com/command) | Multilingual | 35B-104B | chat model |\\n| WizardLM2 | [WizardLM2 series models](https://github.com/nlpxucan/WizardLM) | English | 7B-8x22B
including quantized versions | chat model
MoE model |\\n| Atom | [Atom](https://github.com/LlamaFamily/Llama-Chinese) | Chinese | 7B | base model
chat model |\\n| Chinese-LLaMA-Alpaca-2 | [Chinese-LLaMA-Alpaca-2](https://github.com/ymcui/Chinese-LLaMA-Alpaca-2) | Chinese | 1.3B-13B | base model
chat model
long text model |\\n| Chinese-LLaMA-Alpaca-3 | [Chinese-LLaMA-Alpaca-3](https://github.com/ymcui/Chinese-LLaMA-Alpaca-3) | Chinese | 8B | base model
chat model |\\n| ModelScope-Agent | [ModelScope Agent series models](https://github.com/modelscope/modelscope-agent) | Chinese | 7B-14B | agent model |\\n| Numina | [AI-MO](https://huggingface.co/AI-MO) | English | 7B | Math |\\n\\n#### MLLMs\\n\\n| Model Type | Model Introduction | Language | Model Size | Model Type |\\n|------------------------------------------------------------|----------------------------------------------------------------------------------------|--------------------|---------------------------------------|--------------------------|\\n| Qwen-VL
Qwen2-VL | [Tongyi Qwen vision model](https://github.com/QwenLM) | Chinese
English | 2B-72B
including quantized versions | base model
chat model |\\n| Qwen-Audio
Qwen2-Audio | [Tongyi Qwen speech model](https://github.com/QwenLM) | Chinese
English | 7B | base model
chat model |\\n| Llama3.2-Vision | [Llama3.2](https://huggingface.co/collections/meta-llama/llama-32-66f448ffc8c32f949b04c8cf) | English | 11B-90B | base model
chat model |\\n| YI-VL | [01AI's YI series vision models](https://github.com/01-ai) | Chinese
English | 6B-34B | chat model |\\n| XComposer2
XComposer2.5 | [Pujiang AI Lab InternLM vision model](https://github.com/InternLM/InternLM-XComposer) | Chinese
English | 7B | chat model |\\n| DeepSeek-VL | [DeepSeek series vision models](https://github.com/deepseek-ai) | Chinese
English | 1.3B-7B | chat model |\\n| MiniCPM-V
MiniCPM-V-2
MiniCPM-V-2.5
MiniCPM-V-2.6 | [OpenBmB MiniCPM vision model](https://github.com/OpenBMB/MiniCPM) | Chinese
English | 3B-9B | chat model |\\n| CogVLM
CogAgent
CogVLM2
CogVLM2-Video
GLM4V | [Zhipu ChatGLM visual QA and Agent model](https://github.com/THUDM/) | Chinese
English | 9B-19B | chat model |\\n| Llava-HF | [Llava-HF series models](https://huggingface.co/llava-hf) | English | 0.5B-110B | chat model |\\n| Llava1.5
Llava1.6 | [Llava series models](https://github.com/haotian-liu/LLaVA) | English | 7B-34B | chat model |\\n| Llava-Next
Llava-Next-Video | [Llava-Next series models](https://github.com/LLaVA-VL/LLaVA-NeXT) | Chinese
English | 7B-110B | chat model |\\n| mPLUG-Owl2
mPLUG-Owl2.1
mPLUG-Owl3 | [mPLUG-Owl series models](https://github.com/X-PLUG/mPLUG-Owl) | English | 11B | chat model |\\n| InternVL
Mini-InternVL
InternVL2 | [InternVL](https://github.com/OpenGVLab/InternVL) | Chinese
English | 1B-40B
including quantized version | chat model |\\n| Llava-llama3 | [xtuner](https://huggingface.co/xtuner/llava-llama-3-8b-v1_1-transformers) | English | 8B | chat model |\\n| Phi3-Vision | Microsoft | English | 4B | chat model |\\n| PaliGemma | Google | English | 3B | chat model |\\n| Florence | Microsoft | English | 0.23B-0.77B | chat model |\\n| Idefics3 | [HuggingFaceM4](https://huggingface.co/HuggingFaceM4) | English | 8B | chat model |\\n| Pixtral | [mistralai](https://huggingface.co/mistralai) | English | 12B | chat model |\\n| Llama3.1-Omni | [LLaMA-Omni](https://github.com/ictnlp/LLaMA-Omni) | English | 8B | chat model |\\n\\n\\n#### Diffusion Models\\n\\n| Model Type | Model Introduction | Language | Model Type |\\n|---------------------|----------------------------------------------------------------------|----------|------------------ |\\n| AnimateDiff | [AnimateDiff animation model](https://github.com/guoyww/AnimateDiff) | English | text-to-video |\\n| SD1.5/SD2.0/SDXL | [StabilityAI series diffusion models](https://github.com/Stability-AI) | English | text-to-image |\\n\\n### Supported Open Source Datasets\\n\\n| Dataset Type | Training Task | Documentation |\\n|---------------------|:----------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\\n| General | Fine-tuning | 🔥ruozhiba, 🔥ms-bench, 🔥alpaca-en(gpt4), 🔥alpaca-zh(gpt4), multi-alpaca, instinwild, cot-en, cot-zh, firefly-zh, instruct-en, gpt4all-en, sharegpt, tulu-v2-sft-mixture, wikipedia-zh, open-orca, sharegpt-gpt4, deepctrl-sft, coig-cqia. |\\n| Agent | Fine-tuning | 🔥ms-agent, 🔥ms-agent-for-agentfabric, ms-agent-multirole, 🔥toolbench-for-alpha-umi, damo-agent-zh, damo-agent-zh-mini, agent-instruct-all-en. |\\n| General | Human Alignment | hh-rlhf, 🔥hh-rlhf-cn, stack-exchange-paired. |\\n| Code | Fine-tuning | code-alpaca-en, 🔥leetcode-python-en, 🔥codefuse-python-en, 🔥codefuse-evol-instruction-zh. |\\n| Medical | Fine-tuning | medical-en, medical-zh, 🔥disc-med-sft-zh. |\\n| Legal | Fine-tuning | lawyer-llama-zh, tigerbot-law-zh, 🔥disc-law-sft-zh. |\\n| Math | Fine-tuning | 🔥blossom-math-zh, school-math-zh, open-platypus-en. |\\n| SQL | Fine-tuning | text2sql-en, 🔥sql-create-context-en. |\\n| Text Generation | Fine-tuning | 🔥advertise-gen-zh, 🔥dureader-robust-zh. |\\n| Classification | Fine-tuning | cmnli-zh, 🔥jd-sentiment-zh, 🔥hc3-zh, 🔥hc3-en. |\\n| Quantization Assist | Quantization | pileval. |\\n| Other | Fine-tuning | finance-en, poetry-zh, webnovel-zh, generated-chat-zh, cls-fudan-news-zh, ner-jave-zh. |\\n| Vision | Fine-tuning | coco-en, 🔥coco-en-mini, coco-en-2, coco-en-2-mini, capcha-images. |\\n| Audio | Fine-tuning | aishell1-zh, 🔥aishell1-zh-mini. |\\n\\n### Supported Technologies\\n\\n| Technology Name |\\n|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\\n| 🔥LoRA: [LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS](https://arxiv.org/abs/2106.09685) |\\n| 🔥LoRA+: [LoRA+: Efficient Low Rank Adaptation of Large Models](https://arxiv.org/pdf/2402.12354.pdf) |\\n| 🔥GaLore:[GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection](https://arxiv.org/abs/2403.03507) |\\n| 🔥LISA: [LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning](https://arxiv.org/abs/2403.17919) |\\n| 🔥UnSloth: https://github.com/unslothai/unsloth |\\n| 🔥LLaMA PRO: [LLAMA PRO: Progressive LLaMA with Block Expansion](https://arxiv.org/pdf/2401.02415.pdf) |\\n| 🔥SCEdit: [SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing](https://arxiv.org/abs/2312.11392) < [arXiv](https://arxiv.org/abs/2312.11392) \\\\ |\\n| 🔥NEFTune: [Noisy Embeddings Improve Instruction Finetuning](https://arxiv.org/abs/2310.05914) |\\n| LongLoRA: [Efficient Fine-tuning of Long-Context Large Language Models](https://arxiv.org/abs/2309.12307) |\\n| Adapter: [Parameter-Efficient Transfer Learning for NLP](http://arxiv.org/abs/1902.00751) |\\n| Vision Prompt Tuning: [Visual Prompt Tuning](https://arxiv.org/abs/2203.12119) |\\n| Side: [Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks](https://arxiv.org/abs/1912.13503) |\\n| Res-Tuning: [Res-Tuning: A Flexible and Efficient Tuning Paradigm via Unbinding Tuner from Backbone](https://arxiv.org/abs/2310.19859) < [arXiv](https://arxiv.org/abs/2310.19859) \\\\ |\\n| Tuners provided by [PEFT](https://github.com/huggingface/peft), such as IA3, AdaLoRA, etc. |\\n\\n### Supported Hardware\\n\\n| Hardware Environment | Notes |\\n|--------------------------------|-------------------------------------------------|\\n| CPU | |\\n| RTX 20/30/40 series, etc. | After 30 series, BF16 and FlashAttn can be used |\\n| Computing cards T4/V100, etc. | BF16 and FlashAttn not supported |\\n| Computing cards A10/A100, etc. | Support BF16 and FlashAttn |\\n| Huawei Ascend NPU | |\\n\\n### Environment variables\\n\\n- DATASET_ENABLE_CACHE: Enable cache when preprocess dataset, you can use `1/True` or `0/False`, default `False`\\n- WEBUI_SHARE: Share your web-ui, you can use `1/True` or `0/False`, default `False`\\n- SWIFT_UI_LANG: web-ui language, you can use `en` or `zh`, default `zh`\\n- WEBUI_SERVER: web-ui host ip，`0.0.0.0` for all routes，`127.0.0.1` for local network only. Default `127.0.0.1`\\n- WEBUI_PORT: web-ui port\\n- USE_HF: Use huggingface endpoint or ModelScope endpoint to download models and datasets. you can use `1/True` or `0/False`, default `False`\\n- FORCE_REDOWNLOAD: Force to re-download the dataset\\n\\nOther variables like `CUDA_VISIBLE_DEVICES` are also supported, which are not listed here.\\n\\n\\n## 📚 Classroom\\n\\n| Tutorial Name |\\n|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\\n| [Introduction to Deep Learning](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/A.%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E5%85%A5%E9%97%A8%E4%BB%8B%E7%BB%8D.md) |\\n| [Large Model Basics](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/B.%E9%AD%94%E6%90%AD%E7%A4%BE%E5%8C%BA%E5%92%8CLLM%E5%A4%A7%E6%A8%A1%E5%9E%8B%E5%9F%BA%E7%A1%80%E7%9F%A5%E8%AF%86.md) |\\n| [Prompt Engineering](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/C.%E6%8F%90%E7%A4%BA%E8%AF%8D%E5%B7%A5%E7%A8%8B-prompt%20engineering.md) |\\n| [Transformer Architecture Introduction](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/D.Transformer%E7%BB%93%E6%9E%84.md) |\\n| [Training Technique Selection](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/E.%E6%8A%80%E6%9C%AF%E9%80%89%E5%9E%8B.md) |\\n| [Data Preprocessing](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/F.%E6%95%B0%E6%8D%AE%E9%A2%84%E5%A4%84%E7%90%86.md) |\\n| [Quantization](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/G.%E9%87%8F%E5%8C%96.md) |\\n| [Training](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/H.%E8%AE%AD%E7%BB%83.md) |\\n| [Inference](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/I.LLM%E5%92%8C%E5%A4%9A%E6%A8%A1%E6%80%81%E6%A8%A1%E5%9E%8B%E9%AB%98%E6%95%88%E6%8E%A8%E7%90%86%E5%AE%9E%E8%B7%B5.md) |\\n| [Deployment](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/J.%E9%83%A8%E7%BD%B2.md) |\\n| [Evaluation](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/K.%E5%A4%A7%E6%A8%A1%E5%9E%8B%E8%87%AA%E5%8A%A8%E8%AF%84%E4%BC%B0%E7%90%86%E8%AE%BA%E5%92%8C%E5%AE%9E%E6%88%98--LLM%20Automatic%20Evaluation.md) |\\n\\n## 🏛 License\\n\\nThis framework is licensed under the [Apache License (Version 2.0)](https://github.com/modelscope/modelscope/blob/master/LICENSE). For models and datasets, please refer to the original resource page and follow the corresponding License.\\n\\n## 📎 Citation\\n\\n```bibtex\\n@misc{zhao2024swiftascalablelightweightinfrastructure,\\n title={SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning},\\n author={Yuze Zhao and Jintao Huang and Jinghan Hu and Xingjun Wang and Yunlin Mao and Daoze Zhang and Zeyinzi Jiang and Zhikai Wu and Baole Ai and Ang Wang and Wenmeng Zhou and Yingda Chen},\\n year={2024},\\n eprint={2408.05517},\\n archivePrefix={arXiv},\\n primaryClass={cs.CL},\\n url={https://arxiv.org/abs/2408.05517},\\n}\\n```\\n\\n## Star History\\n\\n[![Star History Chart](https://api.star-history.com/svg?repos=modelscope/swift&type=Date)](https://star-history.com/#modelscope/ms-swift&Date)\\n\\n\\n# Contributor Guide\\n\\n_Welcome to offer PRs, bug reports, documentation supplements or other types of contributions to SWIFT!_\\n\\n## Table of Contents\\n- [Code of Conduct](#-code-of-conduct)\\n- [Contribution Process](#-contribution-process)\\n- [Hardware support](#-Hardware-support)\\n\\n## 📖 Code of Conduct\\nPlease refer to our [Code of Conduct documentation](./CODE_OF_CONDUCT.md).\\n\\n## 🔁 Contribution Process\\n### What We Need\\n- ROADMAP: We provide a [ROADMAP](./ROADMAP.md) for each iteration of SWIFT, contributors can check our ROADMAP to understand our development progress and plans. Features in **To be Assigned** is available for all developers.\\n- New Technologies and New Models: SWIFT needs to support more open-source models and datasets, or new technologies that we have not paid attention to. If you are interested please submit a PR to us.\\n- Technical Propagation: If you are interested in technical propagation, you are welcome to help us write tutorials, documents or videos on any website, and send us the link.\\n- Community Contribution: You can write technical articles related to SWIFT, and submit them to us. After review and approval, we will publish them on the official ModelScope accounts (Zhihu, WeChat, etc.), with your name assigned.\\n\\n### Incentives\\n- we will issue electronic certificates to contributors on behalf of the ModelScope community, to encourage your selfless contributions.\\n- We will offer small souvenirs related to the ModelScope Community.\\n- We will provide free A10 computing power during the development period. For more details, please refer to [Hardware-support](#-Hardware-support) section.\\n\\n### Submitting PR (Pull Requests)\\n\\nAny feature development is carried out in the form of Fork and then PR on GitHub.\\n1. Fork: Go to the [SWIFT](https://github.com/modelscope/swift) page and click the **Fork button**. After completion, a SWIFT code repository will be cloned under your personal organization.\\n2. Clone: Clone the code repository generated in the first step to your local machine and **create a new branch** for development. During development, please click the **Sync Fork button** in time to synchronize with the `main` branch to prevent code expiration and conflicts.\\n3. Submit PR: After development and testing, push the code to the remote branch. On GitHub, go to the **Pull Requests page**, create a new PR, select your code branch as the source branch, and the `modelscope/swift:main` branch as the target branch.\\n\\n4. Write Description: It is necessary to provide a good feature description in the PR, so that the reviewers know the content of your modification.\\n5. Review: We hope that the code to be merged is concise and efficient, so we may raise some questions and discuss them. Please note that any issues raised in the review are aimed at the code itself, not at you personally. Once all issues are discussed and resolved, your code will be approved.\\n\\n### Code Standards and Development Approach\\nSWIFT has conventional variable naming conventions and development approaches. Please follow these approaches as much as possible during development.\\n1. Variable names are separated by underscores, and class names are named with the first letter of each word capitalized.\\n2. All Python indentation uses four spaces instead of a tab.\\n3. Choose well-known open-source libraries, avoid using closed-source libraries or unstable open-source libraries, and avoid repeating the existing code.\\n\\nAfter the PR is submitted, SWIFT will perform two types of tests:\\n- Code Lint Test: A static code compliance check test. please make sure that you have performed code lint locally in advance.\\n```shell\\npip install pre-commit # In the swift folder\\npre-commit run --all-files # Fix the errors reported by pre-commit until all checks are successful\\n```\\n- CI Tests: Smoke tests and unit tests, please refer to the next section.\\n\\n### Running CI Tests\\nBefore submitting the PR, please ensure that your development code is protected by test cases, such as smoke tests for new features, or unit tests for various edge cases. Reviewers will also pay attention to this during code review. At the same time, there will be dedicated services running CI Tests, running all test cases, and the code can only be merged after the test cases pass.\\n\\nAdditionally, since some important tests have been skipped due to long running time, to ensure that your logic is correct, you can run the test locally:\\n```shell\\npython tests/llm/test_run.py\\n```\\nPlease make sure this test can pass normally.\\n\\n## ✅ Hardware support\\n\\nSWIFT will provide hardware support for developers, including free GPUs. If needed, please email us ([contact@modelscope.cn](mailto:contact@modelscope.cn)) or join our WeChat group:\\n\\n

\\n\\n

\\n\\n\\n# SWIFT (Scalable lightWeight Infrastructure for Fine-Tuning)\\n\\n

\\n
\\n \\n
\\n

\\n

\\n魔搭社区官网\\n
\\n 中文  ｜  English \\n

\\n\\n\\n

\\n\\n\\n\\n\\n\\n\\n\\n

\\n\\n

\\n\\\"modelscope%2Fswift\\n

\\n\\n## 📖 目录\\n- [简介](#-简介)\\n- [用户群](#-用户群)\\n- [新闻](#-新闻)\\n- [安装](#-%EF%B8%8F-安装)\\n- [快速开始](#-快速开始)\\n- [教程](#-教程)\\n- [License](#-license)\\n- [引用](#-引用)\\n\\n## 📝 简介\\nSWIFT支持**350+ LLM和90+ MLLM**（多模态大模型）的训练(预训练、微调、对齐)、推理、评测和部署。开发者可以直接将我们的框架应用到自己的Research和生产环境中，实现模型训练评测到应用的完整链路。我们除支持了[PEFT](https://github.com/huggingface/peft)提供的轻量训练方案外，也提供了一个完整的**Adapters库**以支持最新的训练技术，如NEFTune、LoRA+、LLaMA-PRO等，这个适配器库可以脱离训练脚本直接使用在自己的自定流程中。\\n\\n为方便不熟悉深度学习的用户使用，我们提供了一个Gradio的web-ui用于控制训练和推理，并提供了配套的深度学习课程和最佳实践供新手入门。 可以在[Huggingface space](https://huggingface.co/spaces/tastelikefeet/swift) 和 [ModelScope创空间](https://www.modelscope.cn/studios/iic/Scalable-lightWeight-Infrastructure-for-Fine-Tuning/summary) 中体验SWIFT web-ui功能了。\\n\\nSWIFT具有丰富全面的文档，请查看我们的文档网站:\\n

\\n 论文   ｜ English Documentation   ｜   中文文档  \\n

\\n\\n## ☎ 用户群\\n\\n请扫描下面的二维码来加入我们的交流群：\\n\\n[Discord Group](https://discord.com/invite/D27yfEFVz5) | 微信群\\n:-------------------------:|:-------------------------:\\n | \\n\\n\\n## 🎉 新闻\\n- 2024.09.26: 支持llama3.2-vision系列模型的训练到部署. 使用`swift infer --model_type llama3_2-11b-vision-instruct`进行体验.\\n- 2024.09.26: 支持llama3.2系列模型的训练到部署. 使用`swift infer --model_type llama3_2-1b-instruct`进行体验.\\n- 2024.09.25: 支持got-ocr2的训练到部署. 最佳实践可以查看[这里](https://github.com/modelscope/ms-swift/issues/2122).\\n- 2024.09.24: 支持llama3_1-8b-omni的训练与部署. 使用`swift infer --model_type llama3_1-8b-omni`进行体验.\\n- 2024.09.23: 支持pixtral-12b的训练与部署. 使用`swift infer --model_type pixtral-12b --dtype fp16`进行体验.\\n- 🔥2024.09.19: 支持qwen2.5、qwen2.5-math、qwen2.5-coder系列模型. 支持qwen2-vl-72b系列模型. 最佳实践可以查看[这里](https://github.com/modelscope/ms-swift/issues/2064).\\n- 2024.09.07: 支持`Reflection-llama3-70b`模型， 使用`swift sft/infer --model_type reflection-llama_3_1-70b`命令即可训练和推理.\\n- 2024.09.06: 支持mplug-owl3的微调和推理, 最佳实践可以查看[这里](https://github.com/modelscope/ms-swift/issues/1969).\\n- 2024.09.05: 支持minicpm3-4b模型. 使用`swift infer --model_type minicpm3-4b`进行体验.\\n- 2024.09.05: 支持yi-coder系列模型. 使用`swift infer --model_type yi-coder-1_5b-chat`进行体验.\\n- 🔥2024.08.30: 支持qwen2-vl系列模型的推理与微调: qwen2-vl-2b-instruct, qwen2-vl-7b-instruct. 最佳实践可以查看[这里](docs/source/Multi-Modal/qwen2-vl最佳实践.md).\\n- 🔥2024.08.26: 支持[Liger](https://github.com/linkedin/Liger-Kernel), 该内核支持LLaMA、Qwen、Mistral等模型, 并大幅减少显存使用(10%~60%), 使用`--use_liger true`开启训练.\\n- 🔥2024.08.22: 支持[ReFT](https://github.com/stanfordnlp/pyreft), 该tuner可以以LoRA的1/15~1/65的参数量达到和LoRA匹配或更好的效果, 使用`--sft_type reft`开始训练!\\n- 🔥2024.08.21: 支持phi3_5-mini-instruct, phi3_5-moe-instruct, phi3_5-vision-instruct. 使用phi3_5-vision-instruct进行Latex OCR微调的最佳实践可以查看[这里](https://github.com/modelscope/ms-swift/issues/1809).\\n- 2024.08.21: 支持idefics3-8b-llama3, llava-onevision-qwen2-0_5b-ov, llava-onevision-qwen2-7b-ov, llava-onevision-qwen2-72b-ov.\\n- 🔥2024.08.20: 支持使用deepspeed-zero3对多模态大模型进行微调.\\n- 2024.08.20: 支持模型: longwriter-glm4-9b, longwriter-llama3_1-8b. 支持数据集: longwriter-6k.\\n- 🔥2024.08.12: 🎉 SWIFT论文已经发布到arXiv上，可以点击[这个链接](https://arxiv.org/abs/2408.05517)阅读.\\n- 🔥2024.08.12: 支持packing和flash-attention时不污染attention_mask, 使用`--packing`开启。详情见[PR](https://github.com/huggingface/transformers/pull/31629/files).\\n- 🔥2024.08.09: 支持qwen2-audio模型的推理与微调. 最佳实践可以查看[这里](https://github.com/modelscope/ms-swift/issues/1653).\\n- 🔥2024.08.08: 支持qwen2-math系列模型, 1.5B, 7B, 72B. 使用`swift infer --model_type qwen2-math-1_5b-instruct`进行体验.\\n- 🔥2024.08.07: 支持使用vllm对多模态大模型: llava系列, internvl2系列, phi3-vision, minicpm-v2.5进行推理加速和部署. 可以查看[多模态&vLLM推理加速文档](docs/source/Multi-Modal/vLLM推理加速文档.md)获取更多信息.\\n- 2024.08.06: 支持minicpm-v-v2_6-chat, 使用`swift infer --model_type minicpm-v-v2_6-chat`进行推理体验, 最佳实践可以查看[这里](https://github.com/modelscope/swift/issues/1613).\\n- 2024.08.06: 支持internlm2.5的1.8b和20b系列. 使用`swift infer --model_type internlm2_5-1_8b-chat`进行体验.\\n- 🔥2024.08.05: 支持多模态数据集的评测！命令行完全一致，新增了许多[多模态数据集](https://swift.readthedocs.io/zh-cn/latest/Instruction/LLM%E8%AF%84%E6%B5%8B%E6%96%87%E6%A1%A3.html#id2).\\n- 🔥2024.08.02: 支持Fourier Ft训练. 使用方式为`--sft_type fourierft`, 参数可以参考[这里](https://swift.readthedocs.io/zh-cn/latest/Instruction/%E5%91%BD%E4%BB%A4%E8%A1%8C%E5%8F%82%E6%95%B0.html).\\n- 🔥2024.07.29: 支持使用lmdeploy对LLM和VLM模型进行推理加速. 文档可以查看[这里](docs/source/Multi-Modal/LmDeploy推理加速文档.md).\\n- 🔥2024.07.24: 人类偏好对齐算法支持视觉多模态大模型, 包括DPO/ORPO/SimPO/CPO, 训练参考[文档](docs/source/Multi-Modal/人类偏好对齐训练文档.md). 支持数据集RLAIF-V.\\n- 🔥2024.07.24: 支持使用megatron对qwen2系列进行CPT和SFT. 可以查看[megatron训练文档](docs/source/LLM/Megatron训练文档.md).\\n- 🔥2024.07.24: 支持llama3.1系列模型. 包含8b, 70b, 405b. 支持openbuddy-llama3_1-8b-chat.\\n
More\\n\\n- 2024.07.20: 支持mistral-nemo系列模型. 使用`--model_type mistral-nemo-base-2407`以及`--model_type mistral-nemo-instruct-2407`开始训练和推理.\\n- 🔥2024.07.19: 支持[Q-Galore](https://arxiv.org/abs/2407.08296)算法, 该算法可以减少显存使用约60% (qwen-7b-chat, full, 80G -> 35G), 使用命令行:`swift sft --model_type xxx --use_galore true --galore_quantization true`来开始训练!\\n- 2024.07.17: 支持InternVL2系列新模型: `model_type`分别为internvl2-1b, internvl2-40b, internvl2-llama3-76b. 最佳实践可以查看[这里](docs/source/Multi-Modal/internvl最佳实践.md).\\n- 2024.07.17: 支持[NuminaMath-7B-TIR](https://www.modelscope.cn/models/AI-ModelScope/NuminaMath-7B-TIR)的训练和推理. model_type可以使用`numina-math-7b`.\\n- 🔥2024.07.16: 支持ollama和bitsandbytes导出. 可以使用命令: `swift export --model_type xxx --to_ollama true`或者`swift export --model_type xxx --quant_method bnb --quant_bits 4`.\\n- 2024.07.08: 支持cogvlm2-video-13b-chat. 最佳实践可以查看[这里](docs/source/Multi-Modal/cogvlm2-video最佳实践.md).\\n- 2024.07.08: 支持internlm-xcomposer2_5-7b-chat. 最佳实践可以查看[这里](docs/source/Multi-Modal/internlm-xcomposer2最佳实践.md).\\n- 🔥2024.07.06: 支持llava-next-video系列模型: llava-next-video-7b-instruct, llava-next-video-7b-32k-instruct, llava-next-video-7b-dpo-instruct, llava-next-video-34b-instruct. 可以查看[llava-video最佳实践](docs/source/Multi-Modal/llava-video最佳实践.md)了解更多.\\n- 🔥2024.07.06: 支持InternVL-2系列: internvl2-2b, internvl2-4b, internvl2-8b, internvl2-26b.\\n- 2024.07.06: 支持codegeex4-9b-chat.\\n- 2024.07.04: 支持internlm2_5-7b系列: internlm2_5-7b, internlm2_5-7b-chat, internlm2_5-7b-chat-1m.\\n- 2024.07.02: 支持`llava1_6-vicuna-7b-instruct`, `llava1_6-vicuna-13b-instruct`等llava-hf模型. 最佳实践可以查看[这里](docs/source/Multi-Modal/llava最佳实践.md).\\n- 🔥2024.06.29: 支持[eval-scope](https://github.com/modelscope/eval-scope)&[open-compass](https://github.com/open-compass/opencompass)评测! 我们支持了包含`BoolQ, ocnli, humaneval, math, ceval, mmlu, gsk8k, ARC_e`等50+标准数据集在内的评测流程, 请查看我们的[评测文档](https://github.com/modelscope/swift/blob/main/docs/source/Instruction/LLM评测文档.md)来使用。下个迭代我们会支持多模态评测和Agent评测，记得持续关注我们: )\\n- 🔥2024.06.28: 支持**Florence**系列模型: 可以查看[Florence最佳实践](docs/source/Multi-Modal/florence最佳实践.md).\\n- 🔥2024.06.28: 支持**Gemma2**系列模型: gemma2-9b, gemma2-9b-instruct, gemma2-27b, gemma2-27b-instruct.\\n- 🔥2024.06.18: 支持**DeepSeek-Coder-v2**系列模型! 使用model_type`deepseek-coder-v2-instruct`和`deepseek-coder-v2-lite-instruct`来开启训练和推理.\\n- 🔥2024.06.16: 支持**KTO**和**CPO**训练，使用`swift rlhf --rlhf_type kto`和`swift rlhf --rlhf_type cpo`来开始训练，可以参考[文档](./docs/source/LLM/人类偏好对齐训练文档.md).\\n- 2024.06.11: 支持符合OpenAI接口的工具调用Agent部署, 可以查看[Agent部署最佳实践](docs/source/LLM/Agent部署最佳实践.md).\\n- 🔥2024.06.07: 支持**Qwen2**系列LLM, 包括0.5B、1.5B、7B、72B的Base和Instruct模型, 以及对应的gptq-int4、gptq-int8、awq-int4量化版本. 使用双卡80GiB A100对Qwen2-72B-Instruct进行自我认知微调并推理部署的最佳实践可以查看[这里](https://github.com/modelscope/swift/issues/1092).\\n- 🔥2024.06.05: 支持glm4系列大模型和glm4v-9b-chat多模态大模型, 可以查看[glm4v最佳实践](docs/source/Multi-Modal/glm4v最佳实践.md).\\n- 🔥2024.06.01: 支持**SimPO**训练，使用`swift simpo`来开始训练，最佳实践可以查看[这里](https://github.com/modelscope/swift/tree/main/docs/source/LLM/SimPO算法最佳实践.md)\\n- 🔥2024.06.01: 支持多模态大模型部署, 可以查看[多模态部署文档](docs/source/Multi-Modal/MLLM部署文档.md).\\n- 2024.05.31: 支持Mini-Internvl多模态模型, 使用model_type `mini-internvl-chat-2b-v1_5`和`mini-internvl-chat-4b-v1_5`来训练.\\n- 2024.05.24: 支持Phi3多模态模型, 使用model_type `phi3-vision-128k-instruct`来训练.\\n- 2024.05.22: 支持DeepSeek-V2-lite系列模型, model_type为 `deepseek-v2-lite`和`deekseek-v2-lite-chat`\\n- 2024.05.22: 支持TeleChat-12b-v2模型和量化版本, model_type为 `telechat-12b-v2`和`telechat-12b-v2-gptq-int4`\\n- 🔥2024.05.21: 支持 MiniCPM-Llama3-V-2_5 的推理与微调, 可以查看[minicpm-v-2.5最佳实践](docs/source/Multi-Modal/minicpm-v-2.5最佳实践.md).\\n- 🔥2024.05.20: 支持 cogvlm2-llama3-chinese-chat-19B, cogvlm2-llama3-chat-19B 的推理与微调, 可以查看[cogvlm2最佳实践](docs/source/Multi-Modal/cogvlm2最佳实践.md).\\n- 🔥2024.05.17: 支持peft=0.11.0. 同时支持了三个新的tuner方法： `BOFT`, `Vera` 和 `Pissa`. 使用 `--sft_type boft/vera` 开启BOFT或者Vera, 使用 `--init_lora_weights pissa` 以及 `--sft_type lora` 来使用 Pissa.\\n- 2024.05.16: 支持Llava-Next (Stronger)系列模型，最佳实践可以查看[这里](https://github.com/modelscope/swift/tree/main/docs/source/Multi-Modal/llava最佳实践.md).\\n- 🔥2024.05.13: 支持Yi-1.5系列模型，使用`--model_type yi-1_5-9b-chat`等开始体验\\n- 2024.05.11: 支持使用[hqq](https://github.com/mobiusml/hqq)和[eetq](https://github.com/NetEase-FuXi/EETQ)进行qlora训练和量化推理，可以查看[LLM量化与导出文档](https://github.com/modelscope/swift/tree/main/docs/source/Instruction/LLM量化与导出文档.md)\\n- 2024.05.10: 支持序列并行. 先安装`pip install .[seq_parallel]`, 之后在DDP环境中添加`--sequence_parallel_size n`即可使用!\\n- 2024.05.08: 支持DeepSeek-V2-Chat模型, 训练参考[这个脚本](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/deepseek-v2-chat/lora_ddp_ds3/sft.sh)。支持InternVL-Chat-V1.5-Int8模型，最佳实践参考[这里](https://github.com/modelscope/swift/tree/main/docs/source/Multi-Modal/internvl最佳实践.md).\\n- 🔥2024.05.07: 支持**ORPO**训练，使用`swift orpo`来开始训练， 最佳实践可以查看[这里](https://github.com/modelscope/swift/tree/main/docs/source/LLM/ORPO算法最佳实践.md)\\n- 2024.05.07: 支持来自xtuner的Llava-Llama3模型，model_type为`llava-llama-3-8b-v1_1`.\\n- 2024.04.29: 支持InternVL-Chat-V1.5的推理与微调, 最佳实践可以查看[这里](https://github.com/modelscope/swift/tree/main/docs/source/Multi-Modal/internvl最佳实践.md).\\n- 🔥2024.04.26: 支持**LISA** 和 **unsloth**训练！指定 `--lisa_activated_layers=2` 来开启LISA（显存使用降低至全参训练的30%），指定 `--tuner_backend unsloth` 来使用unsloth，用更少的显存（30%或更少）更快的速度（5x）训练一个超大模型！\\n- 🔥2024.04.26: 支持Qwen1.5-110B和Qwen1.5-110B-Chat模型的推理与微调, 使用[这个脚本](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/qwen1half_110b_chat/lora_ddp_ds/sft.sh)来开始训练！\\n- 2024.04.24: 支持Phi3系列模型的推理与微调. 包括: [phi3-4b-4k-instruct](examples/pytorch/llm/scripts/phi3_4b_4k_instruct/lora), phi3-4b-128k-instruct.\\n- 2024.04.22: 支持**chinese-llama-alpaca-2**系列模型的推理与微调和部署等. 包括：chinese-llama-2-1.3b, chinese-llama-2-7b, chinese-llama-2-13b, chinese-alpaca-2-1.3b, chinese-alpaca-2-7b和chinese-alpaca-2-13b以及对应的16k和64k长文本模型.\\n- 2024.04.22: 支持Llama3 GPTQ-Int4, GPTQ-Int8, AWQ系列模型的推理与微调. 支持chatglm3-6b-128k, Openbuddy-llama3的推理与微调.\\n- 2024.04.20: 支持**Atom**系列模型的推理, 微调和部署等. 包括: Atom-7B and Atom-7B-Chat. 使用[这个脚本](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/atom_7b_chat/lora/sft.sh)来开始训练！\\n- 2024.04.19: 支持NPU的单卡、DDP、ZeRO2和ZeRO3的训练与推理, 可以查看[NPU推理与微调最佳实践](docs/source/LLM/NPU推理与微调最佳实践.md).\\n- 2024.04.19: 支持**Llama3**系列模型的推理, 微调和部署等. 包括: Llama-3-8B, Llama-3-8B-Instruct, Llama-3-70B, Llama-3-70B-Instruct. 使用[这个脚本](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/llama3_8b_instruct/lora/sft.sh)开始训练叭！\\n- 2024.04.18: 支持模型: wizardlm2-7b-awq, wizardlm2-8x22b, yi-6b-chat-awq, yi-6b-chat-int8, yi-34b-chat-awq, yi-34b-chat-int8. 支持`--deepspeed zero3-offload`, 提供了默认zero3-offload配置文件来使用zero3+cpu offload.\\n- 2024.04.18: 支持使用环境变量`USE_HF`兼容HuggingFace生态, 切换成使用HF中的模型和数据集, 可以查看[HuggingFace生态兼容文档](https://github.com/modelscope/swift/tree/main/docs/source/LLM/HuggingFace生态兼容.md).\\n- 2024.04.17: 支持OpenAI样式的接口评测, 可以查看[评测参数接口文档](docs/source/Instruction/命令行参数.md#eval参数)来查看使用方法.\\n- 🔥2024.04.17: 支持 **CodeQwen1.5-7B**系列: CodeQwen1.5-7B, CodeQwen1.5-7B-Chat, CodeQwen1.5-7B-Chat-AWQ, 使用[这个脚本](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/codeqwen1half_7b_chat/lora/sft.sh)来开始训练！\\n- 2024.04.16: 支持llava-v1.6-34b的推理与微调, 最佳实践可以查看[这里](https://github.com/modelscope/swift/tree/main/docs/source/Multi-Modal/llava最佳实践.md).\\n- 2024.04.13: 支持Mixtral-8x22B-v0.1模型的推理与微调, 使用[这个脚本](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/mixtral_moe_8x22b_v1/lora_ddp_ds/sft.sh)来开始训练！\\n- 2024.04.13: 支持新推出的**MiniCPM**系列: MiniCPM-V-2.0、MiniCPM-2B-128k、MiniCPM-MoE-8x2B和MiniCPM-1B。使用[这个脚本](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/minicpm_moe_8x2b/lora_ddp/sft.sh)来开始训练！\\n- 🔥2024.04.11: 支持一键式模型评测能力! 首批数据集包含MMLU、CEval、ARC等，也支持用户自定义数据集，具体可以[这个文档](docs/source/Instruction/LLM评测文档.md)。同时, 我们支持了一个比较trick的方法来做多个消融实验的管理，查看[这个文档](docs/source/Instruction/LLM实验文档.md)来使用。\\n- 🔥2024.04.11: 支持**c4ai-command-r**系列: c4ai-command-r-plus, c4ai-command-r-v01。使用[这个脚本](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/c4ai_command_r_plus/lora_mp/sft.sh)来开始训练！\\n- 2024.04.10: 使用swift微调qwen-7b-chat模型增强模型function call能力，并结合[Modelscope-Agent](https://github.com/modelscope/modelscope-agent)使用，最佳实践可以查看[这里](https://github.com/modelscope/swift/tree/main/docs/source/LLM/Agent微调最佳实践.md#搭配Modelscope-Agent使用)。\\n- 🔥2024.04.09: 支持`弱智吧`系列数据集. 在[支持的模型和数据集文档](docs/source/Instruction/支持的模型和数据集.md)中搜索`ruozhiba`来找到数据集并开始训练！\\n- 2024.04.08: 支持XVERSE-MoE-A4.2B模型的推理与微调, 使用[这个脚本](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/xverse_moe_a4_2b/lora/sft.sh)来开始训练！\\n- 2024.04.04: 支持使用**QLoRA+FSDP**来使用两张24G显卡训练70B模型, 使用[这个脚本](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/llama2_70b_chat/qlora_fsdp/sft.sh)开始训练.\\n- 🔥2024.04.03: 支持**Qwen1.5-32B**系列: Qwen1.5-32B, Qwen1.5-32B-Chat, Qwen1.5-32B-Chat-GPTQ-Int4。使用[这个脚本](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/qwen1half_32b_chat/lora_mp/sft.sh)来开始训练！\\n- 🔥2024.04.02: 支持Mengzi3-13B-Base模型的推理与微调, 使用[这个脚本](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/mengzi3_13b_base/lora_ddp_ds/sft.sh)来开始训练！\\n- 🔥2024.04.01: 支持**dbrx**系列, dbrx-base和dbrx-instruct, 使用[这个脚本](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/dbrx-instruct/lora_mp/sft.sh)来开始训练！.\\n- 🔥2024.03.29: 支持**Qwen1.5-MoE**系列: Qwen1.5-MoE-A2.7B, Qwen1.5-MoE-A2.7B-Chat, Qwen1.5-MoE-A2.7B-Chat-GPTQ-Int4.\\n- 🔥2024.03.29: 支持**Grok-1** 300B MoE模型的推理与微调, 最佳实践可以查看[这里](https://github.com/modelscope/swift/tree/main/docs/source/LLM/Grok训练和推理.md).\\n- 🔥2024.03.25: 支持TeleChat-7b和TeleChat-12b模型的训练和推理, 使用[这个脚本](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/telechat_12b/lora/sft.sh)来开始训练！.\\n- 🔥2024.03.20: 支持**llava**系列的推理与微调, 最佳实践可以查看[这里](https://github.com/modelscope/swift/tree/main/docs/source/Multi-Modal/llava最佳实践.md).\\n- 🔥2024.03.12: 支持**deepseek-vl**系列推理和微调, 最佳实践可以查看[这里](https://github.com/modelscope/swift/tree/main/docs/source/Multi-Modal/deepseek-vl最佳实践.md).\\n- 🔥2024.03.11: 支持[GaLore](https://arxiv.org/abs/2403.03507), 用于在全参数训练中有效减小显存占用至原来的1/2.\\n- 🔥2024.03.10: Qwen1.5-7B-Chat与Qwen1.5-72B-Chat从微调到部署[全流程最佳实践](https://github.com/modelscope/swift/blob/main/docs/source/LLM/Qwen1.5%E5%85%A8%E6%B5%81%E7%A8%8B%E6%9C%80%E4%BD%B3%E5%AE%9E%E8%B7%B5.md).\\n- 🔥2024.03.09: 支持MAMBA模型的训练和推理, 使用[这个脚本](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/mamba-1.4b/lora/sft.sh)来开始训练！.\\n- 2024.03.09: 支持AQLM量化模型的训练和推理, 使用[这个脚本](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/llama2_7b_aqlm_2bit_1x16/lora/sft.sh)开始训练！\\n- 2024.03.06: 支持AWQ量化模型的训练和推理, 使用[这个Qwen1.5-AWQ模型脚本](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/qwen1half_7b_chat_awq/lora/sft.sh)开始训练, 并支持[yi-9b](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/yi_9b/lora_zero3)的训练和推理.\\n- 🔥2024.02.29: 支持[LLaMA PRO](https://arxiv.org/pdf/2401.02415.pdf), 使用[这个脚本](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/yi_6b_chat/llamapro/sft.sh)即可开始训练.\\n- 🔥2024.02.29: 支持[LoRA+](https://arxiv.org/pdf/2402.12354.pdf), 使用[这个脚本](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/yi_6b_chat/lorap/sft.sh)即可开始训练.\\n- 2024.02.25: 支持`swift export`, 对模型进行**AWQ/GPTQ**量化导出, 以及推送ModelScope Hub. 具体可以查看: [LLM量化与导出文档](https://github.com/modelscope/swift/blob/main/docs/source/Instruction/LLM量化与导出文档.md).\\n- 2024.02.22: 支持gemma系列: gemma-2b, [gemma-2b-instruct](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/gemma_2b_instruct), gemma-7b, gemma-7b-instruct.\\n- 2024.02.16: 支持deepseek-math系列: deepseek-math-7b, deepseek-math-7b-instruct, deepseek-math-7b-chat.\\n- 🔥2024.02.05: 支持**Qwen1.5**系列模型, 支持的所有Qwen1.5系列模型请查看[模型列表](https://github.com/modelscope/swift/blob/main/docs/source/Instruction/%E6%94%AF%E6%8C%81%E7%9A%84%E6%A8%A1%E5%9E%8B%E5%92%8C%E6%95%B0%E6%8D%AE%E9%9B%86.md#%E6%A8%A1%E5%9E%8B). 提供了[qwen1half-7b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen1half_7b_chat), [qwen1half-7b-chat-int8](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen1half_7b_chat_int8)微调的脚本.\\n- 2024.02.05: 支持扩散模型如**SDXL**, **SD**, **ControlNet**的训练, 同时也支持**DreamBooth**的训练, 详情可以查看对应的[训练脚本](https://github.com/modelscope/swift/tree/main/examples/pytorch/sdxl/scripts).\\n- 2024.02.01: 支持minicpm系列: [minicpm-2b-sft-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/minicpm_2b_sft_chat), minicpm-2b-chat.\\n- 🔥2024.02.01: 支持数据集打混来减少 **灾难性遗忘问题**. 使用`--train_dataset_mix_ratio 2.0`开启训练！同时我们也开源了通用知识数据集 [ms-bench](https://www.modelscope.cn/datasets/iic/ms_bench/summary).\\n- 🔥2024.02.01: 支持Agent训练！Agent训练算法源自这篇[论文](https://arxiv.org/pdf/2309.00986.pdf). 我们也增加了[ms-agent](https://www.modelscope.cn/datasets/iic/ms_agent/summary)这个优质的agent数据集. 使用[这个脚本](https://github.com/modelscope/swift/blob/main/examples/pytorch/llm/scripts/qwen_7b_chat/lora/sft.sh)开启Agent训练!\\n- 🔥2024.02.01: 支持在DPO训练中增加SFT loss来减少KL散度loss造成的生成重复问题.\\n- 2024.02.01: 支持在训练中使用AdaLoRA和IA3两个adapter.\\n- 2024.02.01: 支持在AnimateDiff训练中使用`--merge_lora`参数.\\n- 2024.01.30: 支持[internlm-xcomposer2-7b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/internlm_xcomposer2_7b_chat).\\n- 🔥2024.01.30: 支持[ZeRO-3](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat/full_ddp_zero3/), 只需要指定`--deepspeed default-zero3`即可.\\n- 2024.01.29: 支持internlm2-math系列: internlm2-math-7b, internlm2-math-7b-chat, internlm2-math-20b, internlm2-math-20b-chat.\\n- 🔥2024.01.26: 支持[yi-vl-6b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/yi_vl_6b_chat), yi-vl-34b-chat.\\n- 2024.01.24: 支持codefuse-codegeex2-6b-chat, codefuse-qwen-14b-chat.\\n- 2024.01.23: 支持orion系列: orion-14b, [orion-14b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/orion_14b_chat).\\n- 2024.01.20: 支持[xverse-13b-256k](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/xverse_13b_256k), xverse-65b-v2, xverse-65b-chat.\\n- 🔥2024.01.17: 支持internlm2系列: internlm2-7b-base, internlm2-7b, [internlm2-7b-sft-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/internlm2_7b_sft_chat), internlm2-7b-chat, internlm2-20b-base, internlm2-20b, internlm2-20b-sft-chat, internlm2-20b-chat.\\n- 2024.01.15: 支持yuan系列: yuan2-2b-instruct, [yuan2-2b-janus-instruct](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/yuan2_2b_janus_instruct), yuan2-51b-instruct, yuan2-102b-instruct.\\n- 🔥2024.01.12: 支持**deepseek-moe**系列: deepseek-moe-16b, [deepseek-moe-16b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/deepseek_moe_16b_chat).\\n- 🔥2024.01.04: 支持**VLLM部署**, 兼容**OpenAI API**样式, 具体可以查看[VLLM推理加速与部署](https://github.com/modelscope/swift/blob/main/docs/source/LLM/VLLM推理加速与部署.md#部署).\\n- 2024.01.04: 更新[Benchmark](https://github.com/modelscope/swift/blob/main/docs/source/LLM/Benchmark.md), 方便查看不同模型训练的速度和所需显存.\\n- 🔥 2023.12.29: 支持web-ui进行sft训练和推理，安装ms-swift后使用`swift web-ui`开启\\n- 🔥 2023.12.29: 支持 DPO RLHF(Reinforcement Learning from Human Feedback) 和三个用于此任务的数据集: AI-ModelScope/stack-exchange-paired 以及 AI-ModelScope/hh-rlhf 以及 AI-ModelScope/hh_rlhf_cn. 查看[文档](https://github.com/modelscope/swift/blob/main/docs/source/LLM/DPO%E8%AE%AD%E7%BB%83%E6%96%87%E6%A1%A3.md)开启训练！\\n- 🔥 2023.12.28: 支持SCEdit! 该tuner可显著降低U-Net中的显存占用，并支持低显存可控图像生成（取代ControlNet），阅读下面的章节来了解详细信息\\n- 2023.12.23: 支持[codegeex2-6b](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/codegeex2_6b).\\n- 2023.12.19: 支持[phi2-3b](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/phi2_3b).\\n- 2023.12.18: 支持VLLM进行推理加速.\\n- 2023.12.15: 支持deepseek, deepseek-coder系列: deepseek-7b, deepseek-7b-chat, deepseek-67b, deepseek-67b-chat, openbuddy-deepseek-67b-chat, deepseek-coder-1_3b, deepseek-coder-1_3b-instruct, deepseek-coder-6_7b, deepseek-coder-6_7b-instruct, deepseek-coder-33b, deepseek-coder-33b-instruct.\\n- 2023.12.13: 支持mistral-7b-instruct-v2, [mixtral-moe-7b](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/mixtral_7b_moe), [mixtral-moe-7b-instruct](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/mixtral_7b_moe_instruct).\\n- 2023.12.09: 支持`freeze_parameters_ratio`参数, 作为lora和全参数训练的折中方案. 对应的sh可以查看[full_freeze_ddp](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/full_freeze_ddp). 支持`disable_tqdm`, `lazy_tokenize`, `preprocess_num_proc`参数, 具体可以查看[命令行参数](https://github.com/modelscope/swift/blob/main/docs/source/Instruction/命令行参数.md).\\n- 2023.12.08: 支持[sus-34b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/sus_34b_chat), 支持yi-6b-200k, yi-34b-200k.\\n- 2023.12.07: 支持[Multi-Node DDP训练](https://github.com/modelscope/swift/blob/main/docs/source/Instruction/LLM%E5%BE%AE%E8%B0%83%E6%96%87%E6%A1%A3.md#%E4%BD%BF%E7%94%A8cli).\\n- 2023.12.05: 支持模型: zephyr-7b-beta-chat, openbuddy-zephyr-7b-chat. 支持数据集: hc3-zh, hc3-en.\\n- 🔥 2023.12.02: [自我认知微调最佳实践](https://github.com/modelscope/swift/blob/main/docs/source/LLM/自我认知微调最佳实践.md), **10分钟对大模型进行自我认知微调**, 创建专属于自己的大模型.\\n- 🔥 2023.11.30: 支持**qwen-1_8b**, **qwen-72b**, **qwen-audio**系列模型的训练的推理. 对应的sh脚本可以查看[qwen_1_8b_chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_1_8b_chat), [qwen_72b_chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_72b_chat), [qwen_audio_chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_audio_chat)\\n- 🔥 2023.11.29: 支持**AnimateDiff**的训练和推理\\n- 🔥 2023.11.24: 支持**yi-34b-chat**, **codefuse-codellama-34b-chat**模型. 对应的sh脚本可以查看[yi_34b_chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/yi_34b_chat), [codefuse_codellama_34b_chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/codefuse_codellama_34b_chat).\\n- 🔥 2023.11.18: 支持**tongyi-finance-14b**系列模型: tongyi-finance-14b, tongyi-finance-14b-chat, tongyi-finance-14b-chat-int4. 对应的sh脚本可以查看[tongyi_finance_14b_chat_int4](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/tongyi_finance_14b_chat_int4).\\n- 2023.11.16: 支持更多模型的**flash attn**支持: qwen系列, qwen-vl系列, llama系列, openbuddy系列, mistral系列, yi系列, ziya系列. 请使用`use_flash_attn`参数.\\n- 🔥 2023.11.11: 支持**NEFTune**, 使用`Swift.prepare_model(model, NEFTuneConfig())`即可开启.\\n- 🔥 2023.11.11: 支持**命令行**训练推理和**Web-UI**推理, 详情可以查看下方的`使用Swift CLI运行`章节.\\n- 🔥 2023.11.11: 支持模型训练后的**部署**链路(vllm/chatglm.cpp/xinference)，详情可以查看[官方文档](docs/source/GetStarted/zh/部署指南.md).\\n- 🔥 2023.11.10: 支持**bluelm**系列模型: bluelm-7b, bluelm-7b-chat, bluelm-7b-32k, bluelm-7b-chat-32k. 对应的sh脚本可以查看[bluelm_7b_chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/bluelm_7b_chat).\\n- 🔥 2023.11.08: 支持**xverse-65b**模型的训练和推理流程，脚本在[xverse_65b](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/xverse_65b).\\n- 🔥 2023.11.07: 支持**yi-6b**, **yi-34b**模型的训练和推理流程，脚本在[yi_6b](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/yi_6b), [yi_34b](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/yi_34b).\\n- 🔥 2023.10.30: 支持 **QA-LoRA** 和 **LongLoRA**两种新的tuners.\\n- 🔥 2023.10.30: 支持使用**ROME**(Rank One Model Editing)来编辑模型，在无需训练的情况下即可给模型灌注新知识！\\n- 2023.10.30: 支持**skywork-13b**系列模型: skywork-13b, skywork-13b-chat. 对应的sh脚本可以查看[skywork_13b](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/skywork_13b).\\n- 🔥 2023.10.27: 支持**chatglm3**系列模型: chatglm3-6b-base, chatglm3-6b, chatglm3-6b-32k. 对应的sh脚本可以查看[chatglm3_6b](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/chatglm3_6b).\\n- 🔥 2023.10.17: 支持**int4**, **int8**模型的SFT: qwen-7b-chat-int4, qwen-14b-chat-int4, qwen-vl-chat-int4, baichuan2-7b-chat-int4, baichuan2-13b-chat-int4, qwen-7b-chat-int8, qwen-14b-chat-int8.\\n- 2023.10.15: 支持**ziya2-13b**系列模型: ziya2-13b, ziya2-13b-chat.\\n- 2023.10.12: 支持**mistral-7b**系列模型: openbuddy-mistral-7b-chat, mistral-7b, mistral-7b-instruct.\\n- 🔥 2023.10.07: 支持**DeepSpeed ZeRO-2**, 使得lora(不仅仅是qlora)可以在双卡A10上运行DDP.\\n- 2023.10.04: 支持更多数学, 法律, SQL, 代码领域的数据集: blossom-math-zh, school-math-zh, text2sql-en, sql-create-context-en, lawyer-llama-zh, tigerbot-law-zh, leetcode-python-en.\\n- 🔥 2023.09.25: 支持**qwen-14b**系列: qwen-14b, qwen-14b-chat.\\n- 2023.09.18: 支持**internlm-20b**系列: internlm-20b, internlm-20b-chat.\\n- 2023.09.12: 支持**MP+DDP**对全参数训练进行加速.\\n- 2023.09.05: 支持**openbuddy-llama2-70b-chat**.\\n- 2023.09.03: 支持**baichuan2**系列: baichuan2-7b, baichuan2-7b-chat, baichuan2-13b, baichuan2-13b-chat.\\n
\\n\\n## 🛠️ 安装\\n\\nSWIFT在Python环境中运行。请确保您的Python版本高于3.8。\\n\\n- 方法1：使用pip命令安装SWIFT：\\n\\n```shell\\n# 全量能力\\npip install 'ms-swift[all]' -U\\n# 仅使用LLM\\npip install 'ms-swift[llm]' -U\\n# 仅使用AIGC\\npip install 'ms-swift[aigc]' -U\\n# 仅使用Adapters\\npip install ms-swift -U\\n```\\n\\n- 方法2：通过源代码安装SWIFT（方便运行训练推理脚本），请运行以下命令：\\n\\n```shell\\ngit clone https://github.com/modelscope/swift.git\\ncd swift\\npip install -e '.[llm]'\\n```\\n\\nSWIFT依赖torch>=1.13，建议torch>=2.0.0。\\n\\n- 方法3：在我们的[Docker镜像](https://www.modelscope.cn/docs/%E7%8E%AF%E5%A2%83%E5%AE%89%E8%A3%85)中使用SWIFT\\n\\n## 🚀 快速开始\\n\\n本章节介绍基本使用，更丰富的使用方式请查看[文档部分](https://swift.readthedocs.io/zh-cn/latest/)。\\n\\n### Web-UI\\n\\nWeb-UI是基于gradio界面技术的**零门槛**训练部署界面方案。Web-UI配置简单，且完美支持多卡训练和部署：\\n\\n```shell\\nswift web-ui\\n```\\n![image.png](./docs/resources/web-ui.png)\\n\\n### 训练\\n\\n#### 训练脚本\\n你可以参考以下脚本来自定义属于你的训练脚本.\\n\\n- full: [qwen1half-7b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen1half_7b_chat/full) (A100), [qwen-7b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/full_mp) (2\\\\*A100)\\n- full+ddp+zero2: [qwen-7b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_7b_chat/full_ddp_zero2) (4\\\\*A100)\\n- full+ddp+zero3: [qwen-14b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat/full_ddp_zero3) (4\\\\*A100)\\n- lora: [chatglm3-6b](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/chatglm3_6b/lora) (3090), [baichuan2-13b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/baichuan2_13b_chat/lora_mp) (2\\\\*3090), [yi-34b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/yi_34b_chat/lora) (A100), [qwen-72b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_72b_chat/lora_mp) (2\\\\*A100)\\n- lora+ddp: [chatglm3-6b](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/chatglm3_6b/lora_ddp) (2\\\\*3090)\\n- lora+ddp+zero3: [qwen-14b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat/lora_ddp_zero3) (4\\\\*3090), [qwen-72b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_72b_chat/lora_ddp_zero3) (4\\\\*A100)\\n- qlora(gptq-int4): [qwen-14b-chat-int4](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat_int4/qlora) (3090), [qwen1half-72b-chat-int4](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen1half_72b_chat_int4/qlora) (A100)\\n- qlora(gptq-int8): [qwen-14b-chat-int8](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat_int8/qlora) (3090)\\n- qlora(bnb-int4): [qwen-14b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/qwen_14b_chat/qlora) (3090), [llama2-70b-chat](https://github.com/modelscope/swift/tree/main/examples/pytorch/llm/scripts/llama2_70b_chat/qlora_mp) (2 \\\\* 3090)\\n\\n#### 支持的训练过程\\n\\n| 训练过程 | 训练方式 |\\n| -------- |------------------------------------|\\n| 预训练 | 文本生成 |\\n| 微调 | 单轮/多轮
Agent训练/自我认知
多模态视觉/多模态语音 |\\n| 人类对齐 | DPO
ORPO
SimPO
KTO
CPO |\\n| 文生图 | DreamBooth等 |\\n| 文生视频 | - |\\n\\n\\n#### 单卡训练\\n\\n通过如下命令启动单卡微调：\\n\\nLoRA:\\n```shell\\n# 实验环境: A100\\n# 显存需求: 20GB\\n# 运行时长: 3.1小时\\nCUDA_VISIBLE_DEVICES=0 \\\\\\nswift sft \\\\\\n --model_type qwen1half-7b-chat \\\\\\n --dataset blossom-math-zh \\\\\\n --num_train_epochs 5 \\\\\\n --sft_type lora \\\\\\n --output_dir output \\\\\\n```\\n\\n全参数:\\n```shell\\n# 实验环境: A100\\n# 显存需求: 80GB\\n# 运行时长: 2.5小时\\nCUDA_VISIBLE_DEVICES=0 \\\\\\nswift sft \\\\\\n --model_type qwen1half-7b-chat \\\\\\n --dataset blossom-math-zh \\\\\\n --num_train_epochs 5 \\\\\\n --sft_type full \\\\\\n --output_dir output \\\\\\n --eval_steps 500 \\\\\\n```\\n\\n#### 模型并行训练\\n\\n```shell\\n# 实验环境: 2 * A100\\n# 显存需求: 10GB + 13GB\\n# 运行时长: 3.4小时\\nCUDA_VISIBLE_DEVICES=0,1 \\\\\\nswift sft \\\\\\n --model_type qwen1half-7b-chat \\\\\\n --dataset blossom-math-zh \\\\\\n --num_train_epochs 5 \\\\\\n --sft_type lora \\\\\\n --output_dir output \\\\\\n```\\n\\n#### 数据并行训练\\n\\n```shell\\n# 实验环境: 4 * A100\\n# 显存需求: 4 * 30GB\\n# 运行时长: 0.8小时\\nNPROC_PER_NODE=4 \\\\\\nCUDA_VISIBLE_DEVICES=0,1,2,3 \\\\\\nswift sft \\\\\\n --model_type qwen1half-7b-chat \\\\\\n --dataset blossom-math-zh \\\\\\n --num_train_epochs 5 \\\\\\n --sft_type lora \\\\\\n --output_dir output \\\\\\n```\\n\\n模型并行与数据并行结合:\\n```shell\\n# 实验环境: 4 * A100\\n# 显存需求: 2*14GB + 2*18GB\\n# 运行时长: 1.7小时\\nNPROC_PER_NODE=2 \\\\\\nCUDA_VISIBLE_DEVICES=0,1,2,3 \\\\\\nswift sft \\\\\\n --model_type qwen1half-7b-chat \\\\\\n --dataset blossom-math-zh \\\\\\n --num_train_epochs 5 \\\\\\n --sft_type lora \\\\\\n --output_dir output \\\\\\n```\\n\\n#### Deepspeed训练\\nDeepspeed支持对GPTQ和AWQ量化模型进行训练.\\n\\nZeRO2:\\n```shell\\n# 实验环境: 4 * A100\\n# 显存需求: 4 * 21GB\\n# 运行时长: 0.9小时\\nNPROC_PER_NODE=4 \\\\\\nCUDA_VISIBLE_DEVICES=0,1,2,3 \\\\\\nswift sft \\\\\\n --model_type qwen1half-7b-chat \\\\\\n --dataset blossom-math-zh \\\\\\n --num_train_epochs 5 \\\\\\n --sft_type lora \\\\\\n --output_dir output \\\\\\n --deepspeed default-zero2 \\\\\\n```\\n\\nZeRO3:\\n```shell\\n# 实验环境: 4 * A100\\n# 显存需求: 4 * 19GB\\n# 运行时长: 3.2小时\\nNPROC_PER_NODE=4 \\\\\\nCUDA_VISIBLE_DEVICES=0,1,2,3 \\\\\\nswift sft \\\\\\n --model_type qwen1half-7b-chat \\\\\\n --dataset blossom-math-zh \\\\\\n --num_train_epochs 5 \\\\\\n --sft_type lora \\\\\\n --output_dir output \\\\\\n --deepspeed default-zero3 \\\\\\n```\\n\\nZeRO3-Offload:\\n```shell\\n# 实验环境: 4 * A100\\n# 显存需求: 4 * 12GB\\n# 运行时长: 60小时\\nNPROC_PER_NODE=4 \\\\\\nCUDA_VISIBLE_DEVICES=0,1,2,3 \\\\\\nswift sft \\\\\\n --model_id_or_path AI-ModelScope/WizardLM-2-8x22B \\\\\\n --dataset blossom-math-zh \\\\\\n --num_train_epochs 5 \\\\\\n --sft_type lora \\\\\\n --output_dir output \\\\\\n --deepspeed zero3-offload \\\\\\n```\\n\\n#### 多机多卡\\n```shell\\n# 如果非共用磁盘请在各机器sh中额外指定`--save_on_each_node true`.\\n# node0\\nCUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \\\\\\nNNODES=2 \\\\\\nNODE_RANK=0 \\\\\\nMASTER_ADDR=127.0.0.1 \\\\\\nNPROC_PER_NODE=8 \\\\\\nswift sft \\\\\\n --model_type qwen1half-32b-chat \\\\\\n --sft_type full \\\\\\n --dataset blossom-math-zh \\\\\\n --output_dir output \\\\\\n --deepspeed default-zero3 \\\\\\n\\n# node1\\nCUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \\\\\\nNNODES=2 \\\\\\nNODE_RANK=1 \\\\\\nMASTER_ADDR=xxx.xxx.xxx.xxx \\\\\\nNPROC_PER_NODE=8 \\\\\\nswift sft \\\\\\n --model_type qwen1half-32b-chat \\\\\\n --sft_type full \\\\\\n --dataset blossom-math-zh \\\\\\n --output_dir output \\\\\\n --deepspeed default-zero3 \\\\\\n```\\n\\n##### 阿里云-DLC多机训练\\nDLC环境变量中，WORLD_SIZE指代node数量，RANK指代node序号，这一点和torchrun定义不同，需要注意。\\n```shell\\nNNODES=$WORLD_SIZE \\\\\\nNODE_RANK=$RANK \\\\\\nswift sft \\\\\\n --model_type qwen1half-32b-chat \\\\\\n --sft_type full \\\\\\n --dataset blossom-math-zh \\\\\\n --output_dir output \\\\\\n --deepspeed default-zero3\\n```\\n\\n\\n#### 预训练\\n\\n```shell\\nNPROC_PER_NODE=4 \\\\\\nCUDA_VISIBLE_DEVICES=0,1,2,3 \\\\\\nswift pt \\\\\\n --model_type qwen1half-7b \\\\\\n --dataset chinese-c4#100000 \\\\\\n --num_train_epochs 1 \\\\\\n --sft_type full \\\\\\n --deepspeed default-zero3 \\\\\\n --output_dir output \\\\\\n --lazy_tokenize true\\n```\\n\\n\\n#### 人类对齐\\n\\n```shell\\n# We support rlhf_type dpo/cpo/simpo/orpo/kto\\nCUDA_VISIBLE_DEVICES=0 \\\\\\nswift rlhf \\\\\\n --rlhf_type dpo \\\\\\n --model_type qwen1half-7b-chat \\\\\\n --dataset shareai-llama3-dpo-zh-en-emoji \\\\\\n --num_train_epochs 5 \\\\\\n --sft_type lora \\\\\\n --output_dir output \\\\\\n```\\n\\n\\n### 推理\\n原始模型:\\n```shell\\nCUDA_VISIBLE_DEVICES=0 swift infer --model_type qwen1half-7b-chat\\n# 使用VLLM加速\\nCUDA_VISIBLE_DEVICES=0 swift infer --model_type qwen1half-7b-chat \\\\\\n --infer_backend vllm --max_model_len 8192\\n```\\n\\nLoRA微调后:\\n```shell\\nCUDA_VISIBLE_DEVICES=0 swift infer --ckpt_dir xxx/checkpoint-xxx --load_dataset_config true\\n# 使用VLLM加速\\nCUDA_VISIBLE_DEVICES=0 swift infer \\\\\\n --ckpt_dir xxx/checkpoint-xxx --load_dataset_config true \\\\\\n --merge_lora true --infer_backend vllm --max_model_len 8192\\n```\\n\\n### 评测\\n\\n原始模型:\\n```shell\\nCUDA_VISIBLE_DEVICES=0 swift eval --model_type qwen1half-7b-chat \\\\\\n --eval_dataset ARC_c --infer_backend vllm\\n```\\n\\nLoRA微调后:\\n```shell\\nCUDA_VISIBLE_DEVICES=0 swift eval --ckpt_dir xxx/checkpoint-xxx \\\\\\n --eval_dataset ARC_c --infer_backend vllm \\\\\\n --merge_lora true \\\\\\n```\\n\\n### 量化\\n\\n原始模型:\\n```shell\\nCUDA_VISIBLE_DEVICES=0 swift export --model_type qwen1half-7b-chat \\\\\\n --quant_bits 4 --quant_method awq\\n```\\n\\nLoRA微调后:\\n```shell\\nCUDA_VISIBLE_DEVICES=0 swift export \\\\\\n --ckpt_dir xxx/checkpoint-xxx --load_dataset_config true \\\\\\n --quant_method awq --quant_bits 4 \\\\\\n --merge_lora true \\\\\\n```\\n\\n### 部署\\n客户端使用OpenAI API进行调用，具体可以查看[LLM部署文档](https://github.com/modelscope/swift/blob/main/docs/source/LLM/VLLM%E6%8E%A8%E7%90%86%E5%8A%A0%E9%80%9F%E4%B8%8E%E9%83%A8%E7%BD%B2.md#%E9%83%A8%E7%BD%B2)\\n\\n原始模型:\\n```shell\\nCUDA_VISIBLE_DEVICES=0 swift deploy --model_type qwen1half-7b-chat\\n# 使用VLLM加速\\nCUDA_VISIBLE_DEVICES=0 swift deploy --model_type qwen1half-7b-chat \\\\\\n --infer_backend vllm --max_model_len 8192\\n```\\n\\nLoRA微调后:\\n```shell\\nCUDA_VISIBLE_DEVICES=0 swift deploy --ckpt_dir xxx/checkpoint-xxx\\n# 使用VLLM加速\\nCUDA_VISIBLE_DEVICES=0 swift deploy \\\\\\n --ckpt_dir xxx/checkpoint-xxx --merge_lora true \\\\\\n --infer_backend vllm --max_model_len 8192\\n```\\n\\n### 支持的模型\\n完整的支持模型和数据集可以查看[支持的模型和数据集列表](docs/source/Instruction/支持的模型和数据集.md).\\n\\n#### 大语言模型\\n\\n| 模型类型 | 模型介绍 | 语言 | 模型大小 | 模型类型 |\\n|-------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------|----------|---------------------|-------------------------------------------|\\n| Qwen
Qwen1.5
Qwen2
Qwen2.5 | [通义千问系列模型](https://github.com/QwenLM) | 中文
英文 | 0.5B-110B
包含量化版本 | base模型
chat模型
MoE模型
代码模型 | |\\n| ChatGLM2
ChatGLM3
Codegeex2
GLM4
Codegeex4 | [智谱ChatGLM系列模型](https://github.com/THUDM/) | 中文
英文 | 6B-9B | base模型
chat模型
代码模型
长文本模型 |\\n| Baichuan
Baichuan2 | [百川1和百川2](https://github.com/baichuan-inc) | 中文
英文 | 7B-13B
包含量化版本 | base模型
chat模型 |\\n| Yuan2 | [浪潮源系列模型](https://github.com/IEIT-Yuan) | 中文
英文 | 2B-102B | instruct模型 |\\n| XVerse | [元象系列模型](https://github.com/xverse-ai) | 中文
英文 | 7B-65B | base模型
chat模型
长文本模型
MoE模型 | |\\n| LLaMA2 | [LLaMA2系列模型](https://github.com/facebookresearch/llama) | 英文 | 7B-70B
包含量化版本 | base模型
chat模型 |\\n| LLaMA3
LLaMA3.1
Llama3.2 | [LLaMA3系列模型](https://github.com/meta-llama/llama3) | 英文 | 1B-70B
包含量化版本 | base模型
chat模型 |\\n| Mistral
Mixtral | [Mistral系列模型](https://github.com/mistralai/mistral-src) | 英文 | 7B-8x22B | base模型
instruct模型
MoE模型 |\\n| Yi
Yi1.5
Yi-Coder | [01AI的YI系列模型](https://github.com/01-ai) | 中文
英文 | 1.5B-34B
包含量化版本 | base模型
chat模型
长文本模型 |\\n| InternLM
InternLM2
InternLM2-Math
InternLM2.5 | [浦江实验室书生浦语系列模型](https://github.com/InternLM/InternLM) | 中文
英文 | 1.8B-20B | base模型
chat模型
数学模型 |\\n| DeepSeek
DeepSeek-MoE
DeepSeek-Coder
DeepSeek-Math
DeepSeek-V2
DeepSeek-Coder-V2 | [幻方系列模型](https://github.com/deepseek-ai) | 中文
英文 | 1.3B-236B | base模型
chat模型
MoE模型
代码模型
数学模型 |\\n| MAMBA | [MAMBA时序卷积模型](https://github.com/state-spaces/mamba) | 英文 | 130M-2.8B | base模型 |\\n| Gemma
Gemma2 | [Google Gemma系列模型](https://github.com/google/gemma_pytorch) | 英文 | 2B-27B | base模型
instruct模型 |\\n| MiniCPM
MiniCPM3 | [OpenBmB MiniCPM系列模型](https://github.com/OpenBMB/MiniCPM) | 中文
英文 | 2B-3B | chat模型
MoE模型 |\\n| OpenBuddy | [OpenBuddy系列模型](https://github.com/OpenBuddy/OpenBuddy) | 中文
英文 | 7B-70B | base模型
chat模型 |\\n| Orion | [猎户星空系列模型](https://github.com/OrionStarAI) | 中文
英文 | 14B | base模型
chat模型 |\\n| BlueLM | [VIVO蓝心大模型](https://github.com/vivo-ai-lab/BlueLM) | 中文
英文 | 7B | base模型
chat模型 |\\n| Ziya2 | [封神榜系列模型](https://github.com/IDEA-CCNL/Fengshenbang-LM) | 中文
英文 | 13B | base模型
chat模型 |\\n| Skywork | [昆仑天工系列模型](https://github.com/SkyworkAI/Skywork) | 中文
英文 | 13B | base模型
chat模型 |\\n| Zephyr | 基于Mistral的zephyr系列模型 | 英文 | 7B | chat模型 |\\n| PolyLM | [通义实验室自研的PolyLM系列模型](https://github.com/DAMO-NLP-MT/PolyLM) | 多语种 | 13B | base模型 |\\n| SeqGPT | [通义实验室自研的文本理解模型，用于信息抽取和文本分类](https://github.com/Alibaba-NLP/SeqGPT) | 中文 | 560M | 语义理解模型 |\\n| SUS | [南方科技大学基于YI Fine-Tune的模型](https://github.com/SUSTech-IDEA/SUS-Chat) | 中文
英文 | 34B | chat模型 |\\n| Tongyi-Finance | [通义金融系列模型](https://github.com/QwenLM/Qwen) | 中文
英文 | 14B | base模型
chat模型
金融模型 |\\n| CodeFuse-CodeLLaMA
CodeFuse-Codegeex2
CodeFuse-Qwen | [蚂蚁CodeFuse系列模型](https://github.com/codefuse-ai) | 中文
英文 | 6B-34B | chat模型
代码模型 |\\n| phi2/phi3 | 微软PHI2模型 | 英文 | 3B/4B | base模型
指令模型
代码模型 |\\n| Grok | [X-ai](https://github.com/xai-org/grok-1) | 英文 | 300B | base模型 |\\n| TeleChat | [Tele-AI](https://github.com/Tele-AI/Telechat) | 中文
英文 | 7B-12B | chat模型 |\\n| dbrx | [databricks](https://github.com/databricks/dbrx) | 英文 | 132B | base模型
chat模型 |\\n| mengzi3 | [Langboat](https://github.com/Langboat/Mengzi3) | 中文
英文 | 13B | base模型 |\\n| c4ai-command-r | [c4ai](https://cohere.com/command) | 多语种 | 35B-104B | chat模型 |\\n| WizardLM2 | [WizardLM2系列模型](https://github.com/nlpxucan/WizardLM) | 多语种 | 7B-8x22B
包含量化版本 | chat模型
MoE模型 |\\n| Atom | [Atom](https://github.com/LlamaFamily/Llama-Chinese) | 中文 | 7B | base模型
chat模型 |\\n| Chinese-LLaMA-Alpaca-2 | [Chinese-LLaMA-Alpaca-2](https://github.com/ymcui/Chinese-LLaMA-Alpaca-2) | 中文 | 1.3B-13B | base模型
chat模型
长文本模型 |\\n| Chinese-LLaMA-Alpaca-3 | [Chinese-LLaMA-Alpaca-3](https://github.com/ymcui/Chinese-LLaMA-Alpaca-3) | 中文 | 8B | base模型
chat模型 |\\n| ModelScope-Agent | [ModelScope Agent系列](https://github.com/modelscope/modelscope-agent) | 中文 | 7B-14B | agent模型 |\\n| Numina | [AI-MO](https://huggingface.co/AI-MO) | 英文 | 7B | 数学模型 |\\n\\n#### 多模态大模型\\n\\n| 模型类型 | 模型介绍 | 语言 | 模型大小 | 模型类型 |\\n|---------------------------------------------------------|----------------------------------------------------------------------------|----------|------------------|------------------|\\n| Qwen-VL
Qwen2-VL | [通义千问视觉模型](https://github.com/QwenLM) | 中文
英文 | 2B-72B
包含量化版本 | base模型
chat模型 |\\n| Qwen-Audio
Qwen2-Audio | [通义千问语音模型](https://github.com/QwenLM) | 中文
英文 | 7B | base模型
chat模型 |\\n| Llama3.2-Vision | [Llama3.2](https://huggingface.co/collections/meta-llama/llama-32-66f448ffc8c32f949b04c8cf) | 英文 | 11B-90B | base模型
chat模型 |\\n| YI-VL | [01AI的YI系列视觉模型](https://github.com/01-ai) | 中文
英文 | 6B-34B | chat模型 |\\n| XComposer2
XComposer2.5 | [浦江实验室书生浦语视觉模型](https://github.com/InternLM/InternLM-XComposer) | 中文
英文 | 7B | chat模型 |\\n| DeepSeek-VL | [幻方系列视觉模型](https://github.com/deepseek-ai) | 中文
英文 | 1.3B-7B | chat模型 |\\n| MiniCPM-V
MiniCPM-V-2
MiniCPM-V-2.5
MiniCPM-V-2.6 | [OpenBmB MiniCPM视觉模型](https://github.com/OpenBMB/MiniCPM) | 中文
英文 | 3B-9B | chat模型 |\\n| CogVLM
CogAgent
CogVLM2
CogVLM2-Video
GLM4V | [智谱ChatGLM视觉问答和Agent模型](https://github.com/THUDM/) | 中文
英文 | 9B-19B | chat模型 |\\n| Llava-HF | [Llava-HF系列模型](https://huggingface.co/llava-hf) | 英文 | 0.5B-110B | chat模型 |\\n| Llava1.5
Llava1.6 | [Llava系列模型](https://github.com/haotian-liu/LLaVA) | 英文 | 7B-34B | chat模型 |\\n| Llava-Next
Llava-Next-Video | [Llava-Next系列模型](https://github.com/LLaVA-VL/LLaVA-NeXT) | 中文
英文 | 7B-110B | chat模型 |\\n| mPLUG-Owl2
mPLUG-Owl2.1
mPLUG-Owl3 | [mPLUG-Owl系列模型](https://github.com/X-PLUG/mPLUG-Owl) | 英文 | 11B | chat模型 |\\n| InternVL
Mini-InternVL
InternVL2 | [InternVL](https://github.com/OpenGVLab/InternVL) | 中文
英文 | 1B-40B
包含量化版本 | chat模型 |\\n| Llava-llama3 | [xtuner](https://huggingface.co/xtuner/llava-llama-3-8b-v1_1-transformers) | 英文 | 8B | chat模型 |\\n| Phi3-Vision | 微软 | 英文 | 4B | chat模型 |\\n| PaliGemma | Google | 英文 | 3B | chat模型 |\\n| Florence | 微软 | 英文 | 0.23B-0.77B | chat模型 |\\n| Idefics3 | [HuggingFaceM4](https://huggingface.co/HuggingFaceM4) | 英文 | 8B | chat模型 |\\n| Pixtral | [mistralai](https://huggingface.co/mistralai) | 英文 | 12B | chat模型 |\\n| Llama3.1-Omni | [LLaMA-Omni](https://github.com/ictnlp/LLaMA-Omni) | 英文 | 8B | chat模型 |\\n\\n\\n#### 扩散模型\\n\\n| 模型类型 | 模型介绍 | 语言 | 模型类型 |\\n| ---------------- | ------------------------------------------------------------ | ---- | -------- |\\n| AnimateDiff | [AnimateDiff动画模型](https://github.com/guoyww/AnimateDiff) | 英文 | 文生视频 |\\n| SD1.5/SD2.0/SDXL | [StabilityAI系列扩散模型](https://github.com/Stability-AI) | 英文 | 文生图 |\\n\\n### 支持的开源数据集\\n\\n| 数据集类型 | 训练任务 | 文档 |\\n|-------|:-----|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\\n| 通用 | 微调 | 🔥ruozhiba, 🔥ms-bench, 🔥alpaca-en(gpt4), 🔥alpaca-zh(gpt4), multi-alpaca, instinwild, cot-en, cot-zh, firefly-zh, instruct-en, gpt4all-en, sharegpt, tulu-v2-sft-mixture, wikipedia-zh, open-orca, sharegpt-gpt4, deepctrl-sft, coig-cqia. |\\n| Agent | 微调 | 🔥ms-agent, 🔥ms-agent-for-agentfabric, ms-agent-multirole, 🔥toolbench-for-alpha-umi, damo-agent-zh, damo-agent-zh-mini, agent-instruct-all-en. |\\n| 通用 | 人类对齐 | hh-rlhf, 🔥hh-rlhf-cn, stack-exchange-paired. |\\n| 代码 | 微调 | code-alpaca-en, 🔥leetcode-python-en, 🔥codefuse-python-en, 🔥codefuse-evol-instruction-zh. |\\n| 医疗 | 微调 | medical-en, medical-zh, 🔥disc-med-sft-zh. |\\n| 法律 | 微调 | lawyer-llama-zh, tigerbot-law-zh, 🔥disc-law-sft-zh. |\\n| 数学 | 微调 | 🔥blossom-math-zh, school-math-zh, open-platypus-en. |\\n| SQL | 微调 | text2sql-en, 🔥sql-create-context-en. |\\n| 文本生成 | 微调 | 🔥advertise-gen-zh, 🔥dureader-robust-zh. |\\n| 分类 | 微调 | cmnli-zh, 🔥jd-sentiment-zh, 🔥hc3-zh, 🔥hc3-en. |\\n| 量化辅助 | 量化 | pileval. |\\n| 其他 | 微调 | finance-en, poetry-zh, webnovel-zh, generated-chat-zh, cls-fudan-news-zh, ner-jave-zh. |\\n| 视觉 | 微调 | coco-en, 🔥coco-en-mini, coco-en-2, coco-en-2-mini, capcha-images. |\\n| 音频 | 微调 | aishell1-zh, 🔥aishell1-zh-mini. |\\n\\n### 支持的技术\\n\\n| 技术名称 |\\n|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\\n| 🔥LoRA: [LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS](https://arxiv.org/abs/2106.09685) |\\n| 🔥LoRA+: [LoRA+: Efficient Low Rank Adaptation of Large Models](https://arxiv.org/pdf/2402.12354.pdf) |\\n| 🔥LLaMA PRO: [LLAMA PRO: Progressive LLaMA with Block Expansion](https://arxiv.org/pdf/2401.02415.pdf) |\\n| 🔥GaLore:[GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection](https://arxiv.org/abs/2403.03507) |\\n| 🔥LISA: [LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning](https://arxiv.org/abs/2403.17919) |\\n| 🔥UnSloth: https://github.com/unslothai/unsloth |\\n| 🔥SCEdit: [SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing](https://arxiv.org/abs/2312.11392) < [arXiv](https://arxiv.org/abs/2312.11392) \\\\ | [Project Page](https://scedit.github.io/) > |\\n| 🔥NEFTune: [Noisy Embeddings Improve Instruction Finetuning](https://arxiv.org/abs/2310.05914) |\\n| LongLoRA: [Efficient Fine-tuning of Long-Context Large Language Models](https://arxiv.org/abs/2309.12307) |\\n| Adapter: [Parameter-Efficient Transfer Learning for NLP](http://arxiv.org/abs/1902.00751) |\\n| Vision Prompt Tuning: [Visual Prompt Tuning](https://arxiv.org/abs/2203.12119) |\\n| Side: [Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks](https://arxiv.org/abs/1912.13503) |\\n| Res-Tuning: [Res-Tuning: A Flexible and Efficient Tuning Paradigm via Unbinding Tuner from Backbone](https://arxiv.org/abs/2310.19859) < [arXiv](https://arxiv.org/abs/2310.19859) \\\\ | [Project Page](https://res-tuning.github.io/) \\\\| [Usage](docs/source/GetStarted/ResTuning.md) > |\\n| [PEFT](https://github.com/huggingface/peft)提供的tuners, 如IA3, AdaLoRA等 |\\n\\n### 支持的硬件\\n\\n| 硬件环境 | 备注 |\\n|--------------------|-------------------------|\\n| CPU | |\\n| RTX20系列/30系列/40系列等 | 30序列之后可使用BF16和FlashAttn |\\n| 计算卡系列 T4/V100等 | 不支持BF16和FlashAttn |\\n| 计算卡系列 A10/A100等 | 支持BF16和FlashAttn |\\n| 华为昇腾NPU | |\\n\\n\\n### 环境变量\\n\\n- DATASET_ENABLE_CACHE：在预处理数据集时启用缓存，您可以使用`1/True`或`0/False`，默认值为`False`\\n- WEBUI_SHARE：共享web-ui，可以使用`1/True`或`0/False`，默认值为`False`\\n- SWIFT_UI_LANG：web-ui语言，您可以使用`en`或`zh`，默认值为`zh`\\n- WEBUI_SERVER：web-ui可访问的IP`0.0.0.0`表示所有路由，`127.0.0.1`仅用于本地网络。默认值为`127.0.0.1`\\n- WEBUI_PORT：web-ui端口\\n- USE_HF：使用huggingface endpoint或ModelScope endpoint下载模型和数据集。您可以使用`1/True`或`0/False`，默认值为`False`\\n- FORCE_REDOWNLOAD：强制重新下载数据集\\n\\n其他变量如`CUDA_VISIBLE_DEVICES`也支持，但未在此列出。\\n\\n## 📚 教程\\n\\n| 教程名称 |\\n| ------------------------------------------------------------ |\\n| [深度学习入门](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/A.%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E5%85%A5%E9%97%A8%E4%BB%8B%E7%BB%8D.md) |\\n| [大模型基础知识](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/B.%E9%AD%94%E6%90%AD%E7%A4%BE%E5%8C%BA%E5%92%8CLLM%E5%A4%A7%E6%A8%A1%E5%9E%8B%E5%9F%BA%E7%A1%80%E7%9F%A5%E8%AF%86.md) |\\n| [提示词工程](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/C.%E6%8F%90%E7%A4%BA%E8%AF%8D%E5%B7%A5%E7%A8%8B-prompt%20engineering.md) |\\n| [Transformer结构介绍](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/D.Transformer%E7%BB%93%E6%9E%84.md) |\\n| [训练技术选型](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/E.%E6%8A%80%E6%9C%AF%E9%80%89%E5%9E%8B.md) |\\n| [数据预处理](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/F.%E6%95%B0%E6%8D%AE%E9%A2%84%E5%A4%84%E7%90%86.md) |\\n| [量化](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/G.%E9%87%8F%E5%8C%96.md) |\\n| [训练](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/H.%E8%AE%AD%E7%BB%83.md) |\\n| [推理](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/I.LLM%E5%92%8C%E5%A4%9A%E6%A8%A1%E6%80%81%E6%A8%A1%E5%9E%8B%E9%AB%98%E6%95%88%E6%8E%A8%E7%90%86%E5%AE%9E%E8%B7%B5.md) |\\n| [部署](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/J.%E9%83%A8%E7%BD%B2.md) |\\n| [评估](https://github.com/modelscope/modelscope-classroom/blob/main/LLM-tutorial/K.%E5%A4%A7%E6%A8%A1%E5%9E%8B%E8%87%AA%E5%8A%A8%E8%AF%84%E4%BC%B0%E7%90%86%E8%AE%BA%E5%92%8C%E5%AE%9E%E6%88%98--LLM%20Automatic%20Evaluation.md) |\\n\\n## 🏛 License\\n\\n本框架使用[Apache License (Version 2.0)](https://github.com/modelscope/modelscope/blob/master/LICENSE)进行许可。模型和数据集请查看原资源页面并遵守对应License。\\n\\n## 📎 引用\\n\\n```bibtex\\n@misc{zhao2024swiftascalablelightweightinfrastructure,\\n title={SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning},\\n author={Yuze Zhao and Jintao Huang and Jinghan Hu and Xingjun Wang and Yunlin Mao and Daoze Zhang and Zeyinzi Jiang and Zhikai Wu and Baole Ai and Ang Wang and Wenmeng Zhou and Yingda Chen},\\n year={2024},\\n eprint={2408.05517},\\n archivePrefix={arXiv},\\n primaryClass={cs.CL},\\n url={https://arxiv.org/abs/2408.05517},\\n}\\n```\\n\\n\\n## Star History\\n\\n[![Star History Chart](https://api.star-history.com/svg?repos=modelscope/swift&type=Date)](https://star-history.com/#modelscope/ms-swift&Date)\\n\\n\\n-r requirements/framework.txt\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n# !/usr/bin/env python\\nimport os\\nimport shutil\\nfrom setuptools import find_packages, setup\\n\\n\\ndef readme():\\n with open('README.md', encoding='utf-8') as f:\\n content = f.read()\\n return content\\n\\n\\nversion_file = 'swift/version.py'\\n\\n\\ndef get_version():\\n with open(version_file, 'r', encoding='utf-8') as f:\\n exec(compile(f.read(), version_file, 'exec'))\\n return locals()['__version__']\\n\\n\\ndef parse_requirements(fname='requirements.txt', with_version=True):\\n \\\"\\\"\\\"\\n Parse the package dependencies listed in a requirements file but strips\\n specific versioning information.\\n\\n Args:\\n fname (str): path to requirements file\\n with_version (bool, default=False): if True include version specs\\n\\n Returns:\\n List[str]: list of requirements items\\n\\n CommandLine:\\n python -c \\\"import setup; print(setup.parse_requirements())\\\"\\n \\\"\\\"\\\"\\n import re\\n import sys\\n from os.path import exists\\n require_fpath = fname\\n\\n def parse_line(line):\\n \\\"\\\"\\\"\\n Parse information from a line in a requirements text file\\n \\\"\\\"\\\"\\n if line.startswith('-r '):\\n # Allow specifying requirements in other files\\n target = line.split(' ')[1]\\n relative_base = os.path.dirname(fname)\\n absolute_target = os.path.join(relative_base, target)\\n for info in parse_require_file(absolute_target):\\n yield info\\n else:\\n info = {'line': line}\\n if line.startswith('-e '):\\n info['package'] = line.split('#egg=')[1]\\n else:\\n # Remove versioning from the package\\n pat = '(' + '|'.join(['>=', '==', '>']) + ')'\\n parts = re.split(pat, line, maxsplit=1)\\n parts = [p.strip() for p in parts]\\n\\n info['package'] = parts[0]\\n if len(parts) > 1:\\n op, rest = parts[1:]\\n if ';' in rest:\\n # Handle platform specific dependencies\\n # http://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-platform-specific-dependencies\\n version, platform_deps = map(str.strip, rest.split(';'))\\n info['platform_deps'] = platform_deps\\n else:\\n version = rest # NOQA\\n info['version'] = (op, version)\\n yield info\\n\\n def parse_require_file(fpath):\\n with open(fpath, 'r', encoding='utf-8') as f:\\n for line in f.readlines():\\n line = line.strip()\\n if line.startswith('http'):\\n print('skip http requirements %s' % line)\\n continue\\n if line and not line.startswith('#') and not line.startswith('--'):\\n for info in parse_line(line):\\n yield info\\n elif line and line.startswith('--find-links'):\\n eles = line.split()\\n for e in eles:\\n e = e.strip()\\n if 'http' in e:\\n info = dict(dependency_links=e)\\n yield info\\n\\n def gen_packages_items():\\n items = []\\n deps_link = []\\n if exists(require_fpath):\\n for info in parse_require_file(require_fpath):\\n if 'dependency_links' not in info:\\n parts = [info['package']]\\n if with_version and 'version' in info:\\n parts.extend(info['version'])\\n if not sys.version.startswith('3.4'):\\n # apparently package_deps are broken in 3.4\\n platform_deps = info.get('platform_deps')\\n if platform_deps is not None:\\n parts.append(';' + platform_deps)\\n item = ''.join(parts)\\n items.append(item)\\n else:\\n deps_link.append(info['dependency_links'])\\n return items, deps_link\\n\\n return gen_packages_items()\\n\\n\\nif __name__ == '__main__':\\n install_requires, deps_link = parse_requirements('requirements.txt')\\n extra_requires = {}\\n all_requires = []\\n extra_requires['llm'], _ = parse_requirements('requirements/llm.txt')\\n extra_requires['aigc'], _ = parse_requirements('requirements/aigc.txt')\\n extra_requires['eval'], _ = parse_requirements('requirements/eval.txt')\\n extra_requires['seq_parallel'], _ = parse_requirements('requirements/seq_parallel.txt')\\n all_requires.extend(install_requires)\\n all_requires.extend(extra_requires['llm'])\\n all_requires.extend(extra_requires['eval'])\\n all_requires.extend(extra_requires['seq_parallel'])\\n extra_requires['seq_parallel'].extend(extra_requires['llm'])\\n extra_requires['all'] = all_requires\\n\\n setup(\\n name='ms-swift',\\n version=get_version(),\\n description='Swift: Scalable lightWeight Infrastructure for Fine-Tuning',\\n long_description=readme(),\\n long_description_content_type='text/markdown',\\n author='DAMO ModelScope teams',\\n author_email='contact@modelscope.cn',\\n keywords='python, petl, efficient tuners',\\n url='https://github.com/modelscope/swift',\\n packages=find_packages(exclude=('configs', 'demo')),\\n include_package_data=True,\\n package_data={\\n '': ['*.h', '*.cpp', '*.cu'],\\n },\\n classifiers=[\\n 'Development Status :: 4 - Beta',\\n 'License :: OSI Approved :: Apache Software License',\\n 'Operating System :: OS Independent',\\n 'Programming Language :: Python :: 3',\\n 'Programming Language :: Python :: 3.7',\\n 'Programming Language :: Python :: 3.8',\\n 'Programming Language :: Python :: 3.9',\\n 'Programming Language :: Python :: 3.10',\\n ],\\n license='Apache License 2.0',\\n tests_require=parse_requirements('requirements/tests.txt'),\\n install_requires=install_requires,\\n extras_require=extra_requires,\\n entry_points={'console_scripts': ['swift=swift.cli.main:cli_main']},\\n dependency_links=deps_link,\\n zip_safe=False)\\n\\n\\n# Contributor Covenant Code of Conduct\\n\\n## Our Pledge\\n\\nWe as members, contributors, and leaders pledge to make participation in our\\ncommunity a harassment-free experience for everyone, regardless of age, body\\nsize, visible or invisible disability, ethnicity, sex characteristics, gender\\nidentity and expression, level of experience, education, socio-economic status,\\nnationality, personal appearance, race, caste, color, religion, or sexual\\nidentity and orientation.\\n\\nWe pledge to act and interact in ways that contribute to an open, welcoming,\\ndiverse, inclusive, and healthy community.\\n\\n## Our Standards\\n\\nExamples of behavior that contributes to a positive environment for our\\ncommunity include:\\n\\n* Demonstrating empathy and kindness toward other people\\n* Being respectful of differing opinions, viewpoints, and experiences\\n* Giving and gracefully accepting constructive feedback\\n* Accepting responsibility and apologizing to those affected by our mistakes,\\n and learning from the experience\\n* Focusing on what is best not just for us as individuals, but for the overall\\n community\\n\\nExamples of unacceptable behavior include:\\n\\n* The use of sexualized language or imagery, and sexual attention or advances of\\n any kind\\n* Trolling, insulting or derogatory comments, and personal or political attacks\\n* Public or private harassment\\n* Publishing others' private information, such as a physical or email address,\\n without their explicit permission\\n* Other conduct which could reasonably be considered inappropriate in a\\n professional setting\\n\\n## Enforcement Responsibilities\\n\\nCommunity leaders are responsible for clarifying and enforcing our standards of\\nacceptable behavior and will take appropriate and fair corrective action in\\nresponse to any behavior that they deem inappropriate, threatening, offensive,\\nor harmful.\\n\\nCommunity leaders have the right and responsibility to remove, edit, or reject\\ncomments, commits, code, wiki edits, issues, and other contributions that are\\nnot aligned to this Code of Conduct, and will communicate reasons for moderation\\ndecisions when appropriate.\\n\\n## Scope\\n\\nThis Code of Conduct applies within all community spaces, and also applies when\\nan individual is officially representing the community in public spaces.\\nExamples of representing our community include using an official e-mail address,\\nposting via an official social media account, or acting as an appointed\\nrepresentative at an online or offline event.\\n\\n## Enforcement\\n\\nInstances of abusive, harassing, or otherwise unacceptable behavior may be\\nreported to the community leaders responsible for enforcement at\\ncontact@modelscope.cn.\\nAll complaints will be reviewed and investigated promptly and fairly.\\n\\nAll community leaders are obligated to respect the privacy and security of the\\nreporter of any incident.\\n\\n## Enforcement Guidelines\\n\\nCommunity leaders will follow these Community Impact Guidelines in determining\\nthe consequences for any action they deem in violation of this Code of Conduct:\\n\\n### 1. Correction\\n\\n**Community Impact**: Use of inappropriate language or other behavior deemed\\nunprofessional or unwelcome in the community.\\n\\n**Consequence**: A private, written warning from community leaders, providing\\nclarity around the nature of the violation and an explanation of why the\\nbehavior was inappropriate. A public apology may be requested.\\n\\n### 2. Warning\\n\\n**Community Impact**: A violation through a single incident or series of\\nactions.\\n\\n**Consequence**: A warning with consequences for continued behavior. No\\ninteraction with the people involved, including unsolicited interaction with\\nthose enforcing the Code of Conduct, for a specified period of time. This\\nincludes avoiding interactions in community spaces as well as external channels\\nlike social media. Violating these terms may lead to a temporary or permanent\\nban.\\n\\n### 3. Temporary Ban\\n\\n**Community Impact**: A serious violation of community standards, including\\nsustained inappropriate behavior.\\n\\n**Consequence**: A temporary ban from any sort of interaction or public\\ncommunication with the community for a specified period of time. No public or\\nprivate interaction with the people involved, including unsolicited interaction\\nwith those enforcing the Code of Conduct, is allowed during this period.\\nViolating these terms may lead to a permanent ban.\\n\\n### 4. Permanent Ban\\n\\n**Community Impact**: Demonstrating a pattern of violation of community\\nstandards, including sustained inappropriate behavior, harassment of an\\nindividual, or aggression toward or disparagement of classes of individuals.\\n\\n**Consequence**: A permanent ban from any sort of public interaction within the\\ncommunity.\\n\\n## Attribution\\n\\nThis Code of Conduct is adapted from the [Contributor Covenant][homepage],\\nversion 2.1, available at\\n[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].\\n\\nCommunity Impact Guidelines were inspired by\\n[Mozilla's code of conduct enforcement ladder][Mozilla CoC].\\n\\nFor answers to common questions about this code of conduct, see the FAQ at\\n[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at\\n[https://www.contributor-covenant.org/translations][translations].\\n\\n[homepage]: https://www.contributor-covenant.org\\n[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html\\n[Mozilla CoC]: https://github.com/mozilla/diversity\\n[FAQ]: https://www.contributor-covenant.org/faq\\n[translations]: https://www.contributor-covenant.org/translations\\n\\n\\n# 贡献者指引\\n\\n*欢迎帮SWIFT提供Feature PR、Bug反馈、文档补充或其他类型的贡献！*\\n\\n## 目录\\n\\n- [代码规约](#-代码规约)\\n- [贡献流程](#-贡献流程)\\n- [资源支持](#-资源支持)\\n\\n## 📖 代码规约\\n\\n请查看我们的[代码规约文档](./CODE_OF_CONDUCT.md).\\n\\n## 🔁 贡献流程\\n\\n### 我们需要什么\\n\\n- ROADMAP：我们为SWIFT提供了每个迭代的[ROADMAP](./ROADMAP.md)，贡献者可以查看我们的ROADMAP来了解我们的开发进度和规划。在**待分配**中的feature可以认领并开发。\\n\\n- 新技术和新模型：SWIFT需要支持更多的开源模型和数据集，或我们没有关注到的新技术，如果您对此有兴趣，可以提交PR给我们。\\n- 技术布道：如果您对技术布道有兴趣，欢迎在任何网站上帮我们撰写教程文档或视频等，并将链接发给我们。\\n- 社区供稿：您可以撰写和SWIFT有关的技术文章，并供稿给我们，我们审核通过后会在魔搭官方账号（知乎、公众号等）上进行发布，并属上您的名字。\\n\\n### 激励\\n\\n- 我们会以魔搭社区的身份给贡献者颁发电子证书，以鼓励您的无私贡献。\\n- 我们会赠送相关魔搭社区相关周边小礼品。\\n- 我们会赠送开发期间的免费A10算力，具体可以查看[资源支持](#-资源支持)章节。\\n\\n### 提交PR（Pull Requests）\\n\\n任何feature开发都在github上以先Fork后PR的形式进行。\\n\\n1. Fork：进入[SWIFT](https://github.com/modelscope/swift)页面后，点击**Fork按钮**执行。完成后会在您的个人组织下克隆出一个SWIFT代码库\\n\\n2. Clone：将第一步产生的代码库clone到本地并**拉新分支**进行开发，开发中请及时点击**Sync Fork按钮**同步`main`分支，防止代码过期并冲突\\n\\n3. 提交PR：开发、测试完成后将代码推送到远程分支。在github上点击**Pull Requests页面**，新建一个PR，源分支选择您提交的代码分支，目标分支选择`modelscope/swift:main`分支\\n\\n4. 撰写描述：在PR中填写良好的feature描述是必要的，让Reviewers知道您的修改内容\\n\\n5. Review：我们希望合入的代码简洁高效，因此可能会提出一些问题并讨论。请注意，任何review中提出的问题是针对代码本身，而非您个人。在所有问题讨论通过后，您的代码会被通过\\n\\n### 代码规范和开发方式\\n\\nSWIFT有约定俗成的变量命名方式和开发方式。在开发中请尽量遵循这些方式。\\n\\n1. 变量命名以下划线分割，类名以所有单词首字母大写方式命名\\n2. 所有的python缩进都是四个空格取代一个tab\\n3. 选用知名的开源库，避免使用闭源库或不稳定的开源库，避免重复造轮子\\n\\nSWIFT在PR提交后会进行两类测试：\\n\\n- Code Lint测试 对代码进行静态规范走查的测试，为保证改测试通过，请保证本地预先进行了Code lint。方法是：\\n\\n ```shell\\n pip install pre-commit\\n # 在swift文件夹内\\n pre-commit run --all-files\\n # 对pre-commit报的错误进行修改，直到所有的检查都是成功状态\\n ```\\n\\n- CI Tests 冒烟测试和单元测试，请查看下一章节\\n\\n### Running CI Tests\\n\\n在提交PR前，请保证您的开发代码已经受到了测试用例的保护。例如，对新功能的冒烟测试，或者各种边缘case的单元测试等。在代码review时Reviewers也会关注这一点。同时，也会有服务专门运行CI Tests，运行所有的测试用例，测试用例通过后代码才可以合并。\\n\\n另外，由于运行时间过长，我们跳过了部分重要测试，为保证您的逻辑是正确的，可以在本地执行该测试：\\n\\n```shell\\npython tests/llm/test_run.py\\n```\\n\\n请保证该测试可以正常通过。\\n\\n## ✅ 资源支持\\n\\nSWIFT会为开发者提供资源支持，包括免费的GPU算力。如果需要请邮件联系我们（[contact@modelscope.cn](mailto:contact@modelscope.cn)）或加入我们的微信群：\\n\\n

\\n\\n

\\n\\n\\n# Reporting Security Issues\\n\\nUsually security issues of a deep learning project come from non-standard 3rd packages or continuous running services. If you are suffering from security issues from our project, please consider reporting to us. We appreciate your efforts to responsibly disclose your findings, and will make every effort to acknowledge your contributions.\\n\\n\\n# PR type\\n- [ ] Bug Fix\\n- [ ] New Feature\\n- [ ] Document Updates\\n- [ ] More Models or Datasets Support\\n\\n# PR information\\n\\nWrite the detail information belongs to this PR.\\n\\n## Experiment results\\n\\nPaste your experiment result here(if needed).\\n\\n\\n---\\nname: Custom issue template\\nabout: Describe this issue template's purpose here.\\ntitle: ''\\nlabels: ''\\nassignees: ''\\n\\n---\\n\\n\\n---\\nname: Feature request\\nabout: Suggest an idea for this project\\ntitle: ''\\nlabels: ''\\nassignees: ''\\n\\n---\\n\\n**Describe the feature**\\nPlease describe the feature requested here(请在这里描述需求)\\n\\n**Paste any useful information**\\nPaste any useful information, including papers, github links, etc.(请在这里描述其他有用的信息，比如相关的论文地址，github链接等)\\n\\n**Additional context**\\nAdd any other context or information here(其他信息可以写在这里)\\n\\n\\n---\\nname: Bug report\\nabout: Create a report to help us improve\\ntitle: ''\\nlabels: ''\\nassignees: ''\\n\\n---\\n\\n**Describe the bug**\\nWhat the bug is, and how to reproduce, better with screenshots(描述bug以及复现过程，最好有截图)\\n\\n\\n**Your hardware and system info**\\nWrite your system info like CUDA version/system/GPU/torch version here(在这里给出硬件信息和系统信息，如CUDA版本，系统，GPU型号和torch版本等)\\n\\n\\n**Additional context**\\nAdd any other context about the problem here(在这里补充其他信息)\\n\\n\\nfrom swift.llm import merge_lora_main\\n\\nif __name__ == '__main__':\\n merge_lora_main(replace_if_exists=True)\\n\\n\\n#!/usr/bin/env python\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n\\nimport argparse\\nimport datetime\\nimport math\\nimport os\\nimport subprocess\\nimport sys\\nsys.path.insert(0, '/swift')\\nimport tempfile\\nimport time\\nimport unittest\\nfrom fnmatch import fnmatch\\nfrom pathlib import Path\\nfrom unittest import TextTestResult\\n\\nimport pandas\\n# NOTICE: Tensorflow 1.15 seems not so compatible with pytorch.\\n# A segmentation fault may be raise by pytorch cpp library\\n# if 'import tensorflow' in front of 'import torch'.\\n# Puting a 'import torch' here can bypass this incompatibility.\\nimport torch\\nimport yaml\\nfrom model_tag import ModelTag, commit_model_ut_result\\nfrom test_utils import get_case_model_info\\n\\nfrom swift.utils.logger import get_logger\\n\\nlogger = get_logger()\\n\\n\\ndef test_cases_result_to_df(result_list):\\n table_header = [\\n 'Name', 'Result', 'Info', 'Start time', 'Stop time',\\n 'Time cost(seconds)'\\n ]\\n df = pandas.DataFrame(\\n result_list, columns=table_header).sort_values(\\n by=['Start time'], ascending=True)\\n return df\\n\\n\\ndef statistics_test_result(df):\\n total_cases = df.shape[0]\\n # yapf: disable\\n success_cases = df.loc[df['Result'] == 'Success'].shape[0]\\n error_cases = df.loc[df['Result'] == 'Error'].shape[0]\\n failures_cases = df.loc[df['Result'] == 'Failures'].shape[0]\\n expected_failure_cases = df.loc[df['Result'] == 'ExpectedFailures'].shape[0]\\n unexpected_success_cases = df.loc[df['Result'] == 'UnexpectedSuccesses'].shape[0]\\n skipped_cases = df.loc[df['Result'] == 'Skipped'].shape[0]\\n # yapf: enable\\n\\n if failures_cases > 0 or \\\\\\n error_cases > 0 or \\\\\\n unexpected_success_cases > 0:\\n final_result = 'FAILED'\\n else:\\n final_result = 'SUCCESS'\\n result_msg = '%s (Runs=%s,success=%s,failures=%s,errors=%s,\\\\\\n skipped=%s,expected failures=%s,unexpected successes=%s)' % (\\n final_result, total_cases, success_cases, failures_cases, error_cases,\\n skipped_cases, expected_failure_cases, unexpected_success_cases)\\n\\n model_cases = get_case_model_info()\\n for model_name, case_info in model_cases.items():\\n cases = df.loc[df['Name'].str.contains('|'.join(list(case_info)))]\\n results = cases['Result']\\n result = None\\n if any(results == 'Error') or any(results == 'Failures') or any(\\n results == 'UnexpectedSuccesses'):\\n result = ModelTag.MODEL_FAIL\\n elif any(results == 'Success'):\\n result = ModelTag.MODEL_PASS\\n elif all(results == 'Skipped'):\\n result = ModelTag.MODEL_SKIP\\n else:\\n print(f'invalid results for {model_name} \\\\n{result}')\\n\\n if result is not None:\\n commit_model_ut_result(model_name, result)\\n print('Testing result summary.')\\n print(result_msg)\\n if final_result == 'FAILED':\\n sys.exit(1)\\n\\n\\ndef gather_test_suites_in_files(test_dir, case_file_list, list_tests):\\n test_suite = unittest.TestSuite()\\n for case in case_file_list:\\n test_case = unittest.defaultTestLoader.discover(\\n start_dir=test_dir, pattern=case)\\n test_suite.addTest(test_case)\\n if hasattr(test_case, '__iter__'):\\n for subcase in test_case:\\n if list_tests:\\n print(subcase)\\n else:\\n if list_tests:\\n print(test_case)\\n return test_suite\\n\\n\\ndef gather_test_suites_files(test_dir, pattern):\\n case_file_list = []\\n for dirpath, dirnames, filenames in os.walk(test_dir):\\n for file in filenames:\\n if fnmatch(file, pattern):\\n case_file_list.append(file)\\n\\n return case_file_list\\n\\n\\ndef collect_test_results(case_results):\\n result_list = [\\n ] # each item is Case, Result, Start time, Stop time, Time cost\\n for case_result in case_results.successes:\\n result_list.append(\\n (case_result.test_full_name, 'Success', '', case_result.start_time,\\n case_result.stop_time, case_result.time_cost))\\n for case_result in case_results.errors:\\n result_list.append(\\n (case_result[0].test_full_name, 'Error', case_result[1],\\n case_result[0].start_time, case_result[0].stop_time,\\n case_result[0].time_cost))\\n for case_result in case_results.skipped:\\n result_list.append(\\n (case_result[0].test_full_name, 'Skipped', case_result[1],\\n case_result[0].start_time, case_result[0].stop_time,\\n case_result[0].time_cost))\\n for case_result in case_results.expectedFailures:\\n result_list.append(\\n (case_result[0].test_full_name, 'ExpectedFailures', case_result[1],\\n case_result[0].start_time, case_result[0].stop_time,\\n case_result[0].time_cost))\\n for case_result in case_results.failures:\\n result_list.append(\\n (case_result[0].test_full_name, 'Failures', case_result[1],\\n case_result[0].start_time, case_result[0].stop_time,\\n case_result[0].time_cost))\\n for case_result in case_results.unexpectedSuccesses:\\n result_list.append((case_result.test_full_name, 'UnexpectedSuccesses',\\n '', case_result.start_time, case_result.stop_time,\\n case_result.time_cost))\\n return result_list\\n\\n\\ndef run_command_with_popen(cmd):\\n with subprocess.Popen(\\n cmd,\\n stdout=subprocess.PIPE,\\n stderr=subprocess.STDOUT,\\n bufsize=1,\\n encoding='utf8') as sub_process:\\n for line in iter(sub_process.stdout.readline, ''):\\n sys.stdout.write(line)\\n\\n\\ndef async_run_command_with_popen(cmd, device_id):\\n logger.info('Worker id: %s args: %s' % (device_id, cmd))\\n env = os.environ.copy()\\n env['CUDA_VISIBLE_DEVICES'] = '%s' % device_id\\n sub_process = subprocess.Popen(\\n cmd,\\n stdout=subprocess.PIPE,\\n stderr=subprocess.STDOUT,\\n bufsize=1,\\n universal_newlines=True,\\n env=env,\\n encoding='utf8')\\n return sub_process\\n\\n\\ndef save_test_result(df, args):\\n if args.result_dir is not None:\\n file_name = str(int(datetime.datetime.now().timestamp() * 1000))\\n os.umask(0)\\n Path(args.result_dir).mkdir(mode=0o777, parents=True, exist_ok=True)\\n Path(os.path.join(args.result_dir, file_name)).touch(\\n mode=0o666, exist_ok=True)\\n df.to_pickle(os.path.join(args.result_dir, file_name))\\n\\n\\ndef run_command(cmd):\\n logger.info('Running command: %s' % ' '.join(cmd))\\n response = subprocess.run(\\n cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)\\n try:\\n response.check_returncode()\\n logger.info(response.stdout.decode('utf8'))\\n except subprocess.CalledProcessError as error:\\n logger.error(\\n 'stdout: %s, stderr: %s' %\\n (response.stdout.decode('utf8'), error.stderr.decode('utf8')))\\n\\n\\ndef install_packages(pkgs):\\n cmd = [sys.executable, '-m', 'pip', 'install']\\n for pkg in pkgs:\\n cmd.append(pkg)\\n\\n run_command(cmd)\\n\\n\\ndef install_requirements(requirements):\\n for req in requirements:\\n cmd = [\\n sys.executable, '-m', 'pip', 'install', '-r',\\n 'requirements/%s' % req, '-f',\\n 'https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html'\\n ]\\n run_command(cmd)\\n\\n\\ndef wait_for_free_worker(workers):\\n while True:\\n for idx, worker in enumerate(workers):\\n if worker is None:\\n logger.info('return free worker: %s' % (idx))\\n return idx\\n if worker.poll() is None: # running, get output\\n for line in iter(worker.stdout.readline, ''):\\n if line != '':\\n sys.stdout.write(line)\\n else:\\n break\\n else: # worker process completed.\\n logger.info('Process end: %s' % (idx))\\n workers[idx] = None\\n return idx\\n time.sleep(0.001)\\n\\n\\ndef wait_for_workers(workers):\\n while True:\\n for idx, worker in enumerate(workers):\\n if worker is None:\\n continue\\n # check worker is completed.\\n if worker.poll() is None:\\n for line in iter(worker.stdout.readline, ''):\\n if line != '':\\n sys.stdout.write(line)\\n else:\\n break\\n else:\\n logger.info('Process idx: %s end!' % (idx))\\n workers[idx] = None\\n\\n is_all_completed = True\\n for idx, worker in enumerate(workers):\\n if worker is not None:\\n is_all_completed = False\\n break\\n\\n if is_all_completed:\\n logger.info('All sub porcess is completed!')\\n break\\n time.sleep(0.001)\\n\\n\\ndef parallel_run_case_in_env(env_name, env, test_suite_env_map, isolated_cases,\\n result_dir, parallel):\\n logger.info('Running case in env: %s' % env_name)\\n # install requirements and deps # run_config['envs'][env]\\n if 'requirements' in env:\\n install_requirements(env['requirements'])\\n if 'dependencies' in env:\\n install_packages(env['dependencies'])\\n # case worker processes\\n worker_processes = [None] * parallel\\n for test_suite_file in isolated_cases: # run case in subprocess\\n if test_suite_file in test_suite_env_map and test_suite_env_map[\\n test_suite_file] == env_name:\\n cmd = [\\n 'python',\\n 'tests/run.py',\\n '--pattern',\\n test_suite_file,\\n '--result_dir',\\n result_dir,\\n ]\\n worker_idx = wait_for_free_worker(worker_processes)\\n worker_process = async_run_command_with_popen(cmd, worker_idx)\\n os.set_blocking(worker_process.stdout.fileno(), False)\\n worker_processes[worker_idx] = worker_process\\n else:\\n pass # case not in run list.\\n\\n # run remain cases in a process.\\n remain_suite_files = []\\n for k, v in test_suite_env_map.items():\\n if k not in isolated_cases and v == env_name:\\n remain_suite_files.append(k)\\n if len(remain_suite_files) == 0:\\n wait_for_workers(worker_processes)\\n return\\n # roughly split case in parallel\\n part_count = math.ceil(len(remain_suite_files) / parallel)\\n suites_chunks = [\\n remain_suite_files[x:x + part_count]\\n for x in range(0, len(remain_suite_files), part_count)\\n ]\\n for suites_chunk in suites_chunks:\\n worker_idx = wait_for_free_worker(worker_processes)\\n cmd = [\\n 'python', 'tests/run.py', '--result_dir', result_dir, '--suites'\\n ]\\n for suite in suites_chunk:\\n cmd.append(suite)\\n worker_process = async_run_command_with_popen(cmd, worker_idx)\\n os.set_blocking(worker_process.stdout.fileno(), False)\\n worker_processes[worker_idx] = worker_process\\n\\n wait_for_workers(worker_processes)\\n\\n\\ndef run_case_in_env(env_name, env, test_suite_env_map, isolated_cases,\\n result_dir):\\n # install requirements and deps # run_config['envs'][env]\\n if 'requirements' in env:\\n install_requirements(env['requirements'])\\n if 'dependencies' in env:\\n install_packages(env['dependencies'])\\n\\n for test_suite_file in isolated_cases: # run case in subprocess\\n if test_suite_file in test_suite_env_map and test_suite_env_map[\\n test_suite_file] == env_name:\\n cmd = [\\n 'python',\\n 'tests/run.py',\\n '--pattern',\\n test_suite_file,\\n '--result_dir',\\n result_dir,\\n ]\\n run_command_with_popen(cmd)\\n else:\\n pass # case not in run list.\\n\\n # run remain cases in a process.\\n remain_suite_files = []\\n for k, v in test_suite_env_map.items():\\n if k not in isolated_cases and v == env_name:\\n remain_suite_files.append(k)\\n if len(remain_suite_files) == 0:\\n return\\n cmd = ['python', 'tests/run.py', '--result_dir', result_dir, '--suites']\\n for suite in remain_suite_files:\\n cmd.append(suite)\\n run_command_with_popen(cmd)\\n\\n\\ndef run_non_parallelizable_test_suites(suites, result_dir):\\n cmd = ['python', 'tests/run.py', '--result_dir', result_dir, '--suites']\\n for suite in suites:\\n cmd.append(suite)\\n run_command_with_popen(cmd)\\n\\n\\n# Selected cases:\\ndef get_selected_cases():\\n cmd = ['python', '-u', 'tests/run_analysis.py']\\n selected_cases = []\\n with subprocess.Popen(\\n cmd,\\n stdout=subprocess.PIPE,\\n stderr=subprocess.STDOUT,\\n bufsize=1,\\n encoding='utf8') as sub_process:\\n for line in iter(sub_process.stdout.readline, ''):\\n sys.stdout.write(line)\\n if line.startswith('Selected cases:'):\\n line = line.replace('Selected cases:', '').strip()\\n selected_cases = line.split(',')\\n sub_process.wait()\\n if sub_process.returncode != 0:\\n msg = 'Run analysis exception, returncode: %s!' % sub_process.returncode\\n logger.error(msg)\\n raise Exception(msg)\\n return selected_cases\\n\\n\\ndef run_in_subprocess(args):\\n # only case args.isolated_cases run in subporcess, all other run in a subprocess\\n if not args.no_diff: # run based on git diff\\n try:\\n test_suite_files = get_selected_cases()\\n logger.info('Tests suite to run: ')\\n for f in test_suite_files:\\n logger.info(f)\\n except Exception:\\n logger.error(\\n 'Get test suite based diff exception!, will run all cases.')\\n test_suite_files = gather_test_suites_files(\\n os.path.abspath(args.test_dir), args.pattern)\\n if len(test_suite_files) == 0:\\n logger.error('Get no test suite based on diff, run all the cases.')\\n test_suite_files = gather_test_suites_files(\\n os.path.abspath(args.test_dir), args.pattern)\\n else:\\n test_suite_files = gather_test_suites_files(\\n os.path.abspath(args.test_dir), args.pattern)\\n\\n non_parallelizable_suites = []\\n test_suite_files = [\\n x for x in test_suite_files if x not in non_parallelizable_suites\\n ]\\n\\n run_config = None\\n isolated_cases = []\\n test_suite_env_map = {}\\n # put all the case in default env.\\n for test_suite_file in test_suite_files:\\n test_suite_env_map[test_suite_file] = 'default'\\n\\n if args.run_config is not None and Path(args.run_config).exists():\\n with open(args.run_config, encoding='utf-8') as f:\\n run_config = yaml.load(f, Loader=yaml.FullLoader)\\n if 'isolated' in run_config:\\n isolated_cases = run_config['isolated']\\n\\n if 'envs' in run_config:\\n for env in run_config['envs']:\\n if env != 'default':\\n for test_suite in run_config['envs'][env]['tests']:\\n if test_suite in test_suite_env_map:\\n test_suite_env_map[test_suite] = env\\n\\n if args.subprocess: # run all case in subprocess\\n isolated_cases = test_suite_files\\n\\n with tempfile.TemporaryDirectory() as temp_result_dir:\\n # first run cases that nonparallelizable\\n run_non_parallelizable_test_suites(non_parallelizable_suites,\\n temp_result_dir)\\n\\n # run case parallel in envs\\n for env in set(test_suite_env_map.values()):\\n parallel_run_case_in_env(env, run_config['envs'][env],\\n test_suite_env_map, isolated_cases,\\n temp_result_dir, args.parallel)\\n\\n result_dfs = []\\n result_path = Path(temp_result_dir)\\n for result in result_path.iterdir():\\n if Path.is_file(result):\\n df = pandas.read_pickle(result)\\n result_dfs.append(df)\\n result_pd = pandas.concat(\\n result_dfs) # merge result of every test suite.\\n print_table_result(result_pd)\\n print_abnormal_case_info(result_pd)\\n statistics_test_result(result_pd)\\n\\n\\ndef get_object_full_name(obj):\\n klass = obj.__class__\\n module = klass.__module__\\n if module == 'builtins':\\n return klass.__qualname__\\n return module + '.' + klass.__qualname__\\n\\n\\nclass TimeCostTextTestResult(TextTestResult):\\n \\\"\\\"\\\"Record test case time used!\\\"\\\"\\\"\\n\\n def __init__(self, stream, descriptions, verbosity):\\n self.successes = []\\n return super(TimeCostTextTestResult,\\n self).__init__(stream, descriptions, verbosity)\\n\\n def startTest(self, test):\\n test.start_time = datetime.datetime.now()\\n test.test_full_name = get_object_full_name(\\n test) + '.' + test._testMethodName\\n self.stream.writeln('Test case: %s start at: %s' %\\n (test.test_full_name, test.start_time))\\n\\n return super(TimeCostTextTestResult, self).startTest(test)\\n\\n def stopTest(self, test):\\n TextTestResult.stopTest(self, test)\\n test.stop_time = datetime.datetime.now()\\n test.time_cost = (test.stop_time - test.start_time).total_seconds()\\n self.stream.writeln(\\n 'Test case: %s stop at: %s, cost time: %s(seconds)' %\\n (test.test_full_name, test.stop_time, test.time_cost))\\n if torch.cuda.is_available(\\n ) and test.time_cost > 5.0: # print nvidia-smi\\n cmd = ['nvidia-smi']\\n run_command_with_popen(cmd)\\n super(TimeCostTextTestResult, self).stopTest(test)\\n\\n def addSuccess(self, test):\\n self.successes.append(test)\\n super(TextTestResult, self).addSuccess(test)\\n\\n\\nclass TimeCostTextTestRunner(unittest.runner.TextTestRunner):\\n resultclass = TimeCostTextTestResult\\n\\n def run(self, test):\\n return super(TimeCostTextTestRunner, self).run(test)\\n\\n def _makeResult(self):\\n result = super(TimeCostTextTestRunner, self)._makeResult()\\n return result\\n\\n\\ndef gather_test_cases(test_dir, pattern, list_tests):\\n case_list = []\\n for dirpath, dirnames, filenames in os.walk(test_dir):\\n for file in filenames:\\n if fnmatch(file, pattern):\\n case_list.append(file)\\n\\n test_suite = unittest.TestSuite()\\n\\n for case in case_list:\\n test_case = unittest.defaultTestLoader.discover(\\n start_dir=test_dir, pattern=case)\\n test_suite.addTest(test_case)\\n if hasattr(test_case, '__iter__'):\\n for subcase in test_case:\\n if list_tests:\\n print(subcase)\\n else:\\n if list_tests:\\n print(test_case)\\n return test_suite\\n\\n\\ndef print_abnormal_case_info(df):\\n df = df.loc[(df['Result'] == 'Error') | (df['Result'] == 'Failures')]\\n for _, row in df.iterrows():\\n print('Case %s run result: %s, msg:\\\\n%s' %\\n (row['Name'], row['Result'], row['Info']))\\n\\n\\ndef print_table_result(df):\\n df = df.loc[df['Result'] != 'Skipped']\\n df = df.drop('Info', axis=1)\\n formatters = {\\n 'Name': '{{:<{}s}}'.format(df['Name'].str.len().max()).format,\\n 'Result': '{{:<{}s}}'.format(df['Result'].str.len().max()).format,\\n }\\n with pandas.option_context('display.max_rows', None, 'display.max_columns',\\n None, 'display.width', None):\\n print(df.to_string(justify='left', formatters=formatters, index=False))\\n\\n\\ndef main(args):\\n runner = TimeCostTextTestRunner()\\n if args.suites is not None and len(args.suites) > 0:\\n logger.info('Running: %s' % ' '.join(args.suites))\\n test_suite = gather_test_suites_in_files(args.test_dir, args.suites,\\n args.list_tests)\\n else:\\n test_suite = gather_test_cases(\\n os.path.abspath(args.test_dir), args.pattern, args.list_tests)\\n if not args.list_tests:\\n result = runner.run(test_suite)\\n logger.info('Running case completed, pid: %s, suites: %s' %\\n (os.getpid(), args.suites))\\n result = collect_test_results(result)\\n df = test_cases_result_to_df(result)\\n if args.result_dir is not None:\\n save_test_result(df, args)\\n else:\\n print_table_result(df)\\n print_abnormal_case_info(df)\\n statistics_test_result(df)\\n\\n\\nif __name__ == '__main__':\\n parser = argparse.ArgumentParser('test runner')\\n parser.add_argument(\\n '--list_tests', action='store_true', help='list all tests')\\n parser.add_argument(\\n '--pattern', default='test_*.py', help='test file pattern')\\n parser.add_argument(\\n '--test_dir', default='tests', help='directory to be tested')\\n parser.add_argument(\\n '--level', default=0, type=int, help='2 -- all, 1 -- p1, 0 -- p0')\\n parser.add_argument(\\n '--profile', action='store_true', help='enable profiling')\\n parser.add_argument(\\n '--run_config',\\n default=None,\\n help='specified case run config file(yaml file)')\\n parser.add_argument(\\n '--subprocess',\\n action='store_true',\\n help='run all test suite in subprocess')\\n parser.add_argument(\\n '--result_dir',\\n default=None,\\n help='Save result to directory, internal use only')\\n parser.add_argument(\\n '--parallel',\\n default=1,\\n type=int,\\n help='Set case parallels, default single process, set with gpu number.'\\n )\\n parser.add_argument(\\n '--no-diff',\\n action='store_true',\\n help=\\n 'Default running case based on git diff(with master), disable with --no-diff)'\\n )\\n parser.add_argument(\\n '--suites',\\n nargs='*',\\n help='Run specified test suites(test suite files list split by space)')\\n args = parser.parse_args()\\n print(args)\\n if args.run_config is not None or args.subprocess:\\n run_in_subprocess(args)\\n else:\\n main(args)\\n\\n\\n#!/usr/bin/env python\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n\\nimport copy\\nimport os\\nimport pickle\\nimport shutil\\nimport socket\\nimport subprocess\\nimport sys\\nimport tarfile\\nimport tempfile\\nimport unittest\\nfrom collections import OrderedDict\\nfrom collections.abc import Mapping\\nfrom os.path import expanduser\\n\\nimport numpy as np\\nimport requests\\n\\nfrom swift.hub.constants import DEFAULT_CREDENTIALS_PATH\\n\\nTEST_LEVEL = 2\\nTEST_LEVEL_STR = 'TEST_LEVEL'\\n\\n# for user citest and sdkdev\\nTEST_ACCESS_TOKEN1 = os.environ.get('TEST_ACCESS_TOKEN_CITEST', None)\\nTEST_ACCESS_TOKEN2 = os.environ.get('TEST_ACCESS_TOKEN_SDKDEV', None)\\n\\nTEST_MODEL_CHINESE_NAME = '内部测试模型'\\nTEST_MODEL_ORG = 'citest'\\n\\n\\ndef delete_credential():\\n path_credential = expanduser(DEFAULT_CREDENTIALS_PATH)\\n shutil.rmtree(path_credential, ignore_errors=True)\\n\\n\\ndef test_level():\\n global TEST_LEVEL\\n if TEST_LEVEL_STR in os.environ:\\n TEST_LEVEL = int(os.environ[TEST_LEVEL_STR])\\n\\n return TEST_LEVEL\\n\\n\\ndef require_tf(test_case):\\n test_case = unittest.skip('test requires TensorFlow')(test_case)\\n return test_case\\n\\n\\ndef require_torch(test_case):\\n return test_case\\n\\n\\ndef set_test_level(level: int):\\n global TEST_LEVEL\\n TEST_LEVEL = level\\n\\n\\nclass DummyTorchDataset:\\n\\n def __init__(self, feat, label, num) -> None:\\n self.feat = feat\\n self.label = label\\n self.num = num\\n\\n def __getitem__(self, index):\\n import torch\\n return {'feat': torch.Tensor(self.feat), 'labels': torch.Tensor(self.label)}\\n\\n def __len__(self):\\n return self.num\\n\\n\\ndef create_dummy_test_dataset(feat, label, num):\\n return DummyTorchDataset(feat, label, num)\\n\\n\\ndef download_and_untar(fpath, furl, dst) -> str:\\n if not os.path.exists(fpath):\\n r = requests.get(furl)\\n with open(fpath, 'wb') as f:\\n f.write(r.content)\\n\\n file_name = os.path.basename(fpath)\\n root_dir = os.path.dirname(fpath)\\n target_dir_name = os.path.splitext(os.path.splitext(file_name)[0])[0]\\n target_dir_path = os.path.join(root_dir, target_dir_name)\\n\\n # untar the file\\n t = tarfile.open(fpath)\\n t.extractall(path=dst)\\n\\n return target_dir_path\\n\\n\\ndef get_case_model_info():\\n status_code, result = subprocess.getstatusoutput(\\n 'grep -rn \\\"damo/\\\" tests/ | grep -v \\\".pyc\\\" | grep -v \\\"Binary file\\\" | grep -v run.py ')\\n lines = result.split('\\\\n')\\n test_cases = OrderedDict()\\n model_cases = OrderedDict()\\n for line in lines:\\n # \\\"tests/msdatasets/test_ms_dataset.py:92: model_id = 'damo/bert-base-sst2'\\\"\\n line = line.strip()\\n elements = line.split(':')\\n test_file = elements[0]\\n model_pos = line.find('damo')\\n left_quote = line[model_pos - 1]\\n rquote_idx = line.rfind(left_quote)\\n model_name = line[model_pos:rquote_idx]\\n if test_file not in test_cases:\\n test_cases[test_file] = set()\\n model_info = test_cases[test_file]\\n model_info.add(model_name)\\n\\n if model_name not in model_cases:\\n model_cases[model_name] = set()\\n case_info = model_cases[model_name]\\n case_info.add(test_file.replace('tests/', '').replace('.py', '').replace('/', '.'))\\n\\n return model_cases\\n\\n\\ndef compare_arguments_nested(print_content, arg1, arg2, rtol=1.e-3, atol=1.e-8, ignore_unknown_type=True):\\n type1 = type(arg1)\\n type2 = type(arg2)\\n if type1.__name__ != type2.__name__:\\n if print_content is not None:\\n print(f'{print_content}, type not equal:{type1.__name__} and {type2.__name__}')\\n return False\\n\\n if arg1 is None:\\n return True\\n elif isinstance(arg1, (int, str, bool, np.bool_, np.integer, np.str_)):\\n if arg1 != arg2:\\n if print_content is not None:\\n print(f'{print_content}, arg1:{arg1}, arg2:{arg2}')\\n return False\\n return True\\n elif isinstance(arg1, (float, np.floating)):\\n if not np.isclose(arg1, arg2, rtol=rtol, atol=atol, equal_nan=True):\\n if print_content is not None:\\n print(f'{print_content}, arg1:{arg1}, arg2:{arg2}')\\n return False\\n return True\\n elif isinstance(arg1, (tuple, list)):\\n if len(arg1) != len(arg2):\\n if print_content is not None:\\n print(f'{print_content}, length is not equal:{len(arg1)}, {len(arg2)}')\\n return False\\n if not all([\\n compare_arguments_nested(None, sub_arg1, sub_arg2, rtol=rtol, atol=atol)\\n for sub_arg1, sub_arg2 in zip(arg1, arg2)\\n ]):\\n if print_content is not None:\\n print(f'{print_content}')\\n return False\\n return True\\n elif isinstance(arg1, Mapping):\\n keys1 = arg1.keys()\\n keys2 = arg2.keys()\\n if len(keys1) != len(keys2):\\n if print_content is not None:\\n print(f'{print_content}, key length is not equal:{len(keys1)}, {len(keys2)}')\\n return False\\n if len(set(keys1) - set(keys2)) > 0:\\n if print_content is not None:\\n print(f'{print_content}, key diff:{set(keys1) - set(keys2)}')\\n return False\\n if not all([compare_arguments_nested(None, arg1[key], arg2[key], rtol=rtol, atol=atol) for key in keys1]):\\n if print_content is not None:\\n print(f'{print_content}')\\n return False\\n return True\\n elif isinstance(arg1, np.ndarray):\\n arg1 = np.where(np.equal(arg1, None), np.NaN, arg1).astype(dtype=float)\\n arg2 = np.where(np.equal(arg2, None), np.NaN, arg2).astype(dtype=float)\\n if not all(np.isclose(arg1, arg2, rtol=rtol, atol=atol, equal_nan=True).flatten()):\\n if print_content is not None:\\n print(f'{print_content}')\\n return False\\n return True\\n else:\\n if ignore_unknown_type:\\n return True\\n else:\\n raise ValueError(f'type not supported: {type1}')\\n\\n\\n_DIST_SCRIPT_TEMPLATE = \\\"\\\"\\\"\\nimport ast\\nimport argparse\\nimport pickle\\nimport torch\\nfrom torch import distributed as dist\\nfrom modelscope.utils.torch_utils import get_dist_info\\nimport {}\\n\\nparser = argparse.ArgumentParser()\\nparser.add_argument('--save_all_ranks', type=ast.literal_eval, help='save all ranks results')\\nparser.add_argument('--save_file', type=str, help='save file')\\nparser.add_argument('--local_rank', type=int, default=0)\\nargs = parser.parse_args()\\n\\n\\ndef main():\\n results = {}.{}({}) # module.func(params)\\n if args.save_all_ranks:\\n save_file = args.save_file + str(dist.get_rank())\\n with open(save_file, 'wb') as f:\\n pickle.dump(results, f)\\n else:\\n rank, _ = get_dist_info()\\n if rank == 0:\\n with open(args.save_file, 'wb') as f:\\n pickle.dump(results, f)\\n\\n\\nif __name__ == '__main__':\\n main()\\n\\\"\\\"\\\"\\n\\n\\nclass DistributedTestCase(unittest.TestCase):\\n \\\"\\\"\\\"Distributed TestCase for test function with distributed mode.\\n Examples:\\n >>> import torch\\n >>> from torch import distributed as dist\\n >>> from modelscope.utils.torch_utils import init_dist\\n\\n >>> def _test_func(*args, **kwargs):\\n >>> init_dist(launcher='pytorch')\\n >>> rank = dist.get_rank()\\n >>> if rank == 0:\\n >>> value = torch.tensor(1.0).cuda()\\n >>> else:\\n >>> value = torch.tensor(2.0).cuda()\\n >>> dist.all_reduce(value)\\n >>> return value.cpu().numpy()\\n\\n >>> class DistTest(DistributedTestCase):\\n >>> def test_function_dist(self):\\n >>> args = () # args should be python builtin type\\n >>> kwargs = {} # kwargs should be python builtin type\\n >>> self.start(\\n >>> _test_func,\\n >>> num_gpus=2,\\n >>> assert_callback=lambda x: self.assertEqual(x, 3.0),\\n >>> *args,\\n >>> **kwargs,\\n >>> )\\n \\\"\\\"\\\"\\n\\n def _start(self, dist_start_cmd, func, num_gpus, assert_callback=None, save_all_ranks=False, *args, **kwargs):\\n script_path = func.__code__.co_filename\\n script_dir, script_name = os.path.split(script_path)\\n script_name = os.path.splitext(script_name)[0]\\n func_name = func.__qualname__\\n\\n func_params = []\\n for arg in args:\\n if isinstance(arg, str):\\n arg = ('\\\\'{}\\\\''.format(arg))\\n func_params.append(str(arg))\\n\\n for k, v in kwargs.items():\\n if isinstance(v, str):\\n v = ('\\\\'{}\\\\''.format(v))\\n func_params.append('{}={}'.format(k, v))\\n\\n func_params = ','.join(func_params).strip(',')\\n\\n tmp_run_file = tempfile.NamedTemporaryFile(suffix='.py').name\\n tmp_res_file = tempfile.NamedTemporaryFile(suffix='.pkl').name\\n\\n with open(tmp_run_file, 'w') as f:\\n print('save temporary run file to : {}'.format(tmp_run_file))\\n print('save results to : {}'.format(tmp_res_file))\\n run_file_content = _DIST_SCRIPT_TEMPLATE.format(script_name, script_name, func_name, func_params)\\n f.write(run_file_content)\\n\\n tmp_res_files = []\\n if save_all_ranks:\\n for i in range(num_gpus):\\n tmp_res_files.append(tmp_res_file + str(i))\\n else:\\n tmp_res_files = [tmp_res_file]\\n self.addCleanup(self.clean_tmp, [tmp_run_file] + tmp_res_files)\\n\\n tmp_env = copy.deepcopy(os.environ)\\n tmp_env['PYTHONPATH'] = ':'.join((tmp_env.get('PYTHONPATH', ''), script_dir)).lstrip(':')\\n # avoid distributed test hang\\n tmp_env['NCCL_P2P_DISABLE'] = '1'\\n script_params = '--save_all_ranks=%s --save_file=%s' % (save_all_ranks, tmp_res_file)\\n script_cmd = '%s %s %s' % (dist_start_cmd, tmp_run_file, script_params)\\n print('script command: %s' % script_cmd)\\n res = subprocess.call(script_cmd, shell=True, env=tmp_env)\\n\\n script_res = []\\n for res_file in tmp_res_files:\\n with open(res_file, 'rb') as f:\\n script_res.append(pickle.load(f))\\n if not save_all_ranks:\\n script_res = script_res[0]\\n\\n if assert_callback:\\n assert_callback(script_res)\\n\\n self.assertEqual(res, 0, msg='The test function ``{}`` in ``{}`` run failed!'.format(func_name, script_name))\\n\\n return script_res\\n\\n def start(self, func, num_gpus, assert_callback=None, save_all_ranks=False, *args, **kwargs):\\n from .torch_utils import _find_free_port\\n ip = socket.gethostbyname(socket.gethostname())\\n if 'dist_start_cmd' in kwargs:\\n dist_start_cmd = kwargs.pop('dist_start_cmd')\\n else:\\n dist_start_cmd = '%s -m torch.distributed.launch --nproc_per_node=%d ' \\\\\\n '--master_addr=\\\\'%s\\\\' --master_port=%s' % (sys.executable, num_gpus, ip, _find_free_port())\\n\\n return self._start(\\n dist_start_cmd=dist_start_cmd,\\n func=func,\\n num_gpus=num_gpus,\\n assert_callback=assert_callback,\\n save_all_ranks=save_all_ranks,\\n *args,\\n **kwargs)\\n\\n def clean_tmp(self, tmp_file_list):\\n for file in tmp_file_list:\\n if os.path.exists(file):\\n if os.path.isdir(file):\\n shutil.rmtree(file)\\n else:\\n os.remove(file)\\n\\n\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n\\nimport logging\\nimport os\\n\\nimport json\\nimport requests\\n\\nfrom swift.version import __version__\\n\\n\\n# 打标\\nclass ModelTag(object):\\n _URL = os.environ.get('MODEL_TAG_URL', None)\\n\\n # 模型测试结果\\n BATCH_COMMIT_RESULT_URL = f'{_URL}/batchCommitResult'\\n # 测试阶段完成\\n BATCH_REFRESH_STAGE_URL = f'{_URL}/batchRefreshStage'\\n # query_model_stage\\n QUERY_MODEL_STAGE_URL = f'{_URL}/queryModelStage'\\n\\n HEADER = {'Content-Type': 'application/json'}\\n\\n # 检测结果\\n MODEL_SKIP = 0\\n MODEL_FAIL = 1\\n MODEL_PASS = 2\\n\\n class ItemResult(object):\\n\\n def __init__(self):\\n self.result = 0\\n self.name = ''\\n self.info = ''\\n\\n def to_json(self):\\n return {'name': self.name, 'result': self.result, 'info': self.info}\\n\\n def __init__(self):\\n self.job_name = ''\\n self.job_id = ''\\n self.model = ''\\n self.sdk_version = ''\\n self.image_version = ''\\n self.domain = ''\\n self.task = ''\\n self.source = ''\\n self.stage = ''\\n # ItemResult list\\n self.item_result = []\\n\\n # 发送请求\\n def _post_request(self, url, param):\\n try:\\n logging.info(url + ' query: ' + str(json.dumps(param, ensure_ascii=False)))\\n res = requests.post(url=url, headers=self.HEADER, data=json.dumps(param, ensure_ascii=False).encode('utf8'))\\n if res.status_code == 200:\\n logging.info(f'{url} post结果: ' + res.text)\\n res_json = json.loads(res.text)\\n if int(res_json['errorCode']) == 200:\\n return res_json['content']\\n else:\\n logging.error(res.text)\\n else:\\n logging.error(res.text)\\n except Exception as e:\\n logging.error(e)\\n\\n return None\\n\\n # 提交模型测试结果\\n def batch_commit_result(self):\\n try:\\n param = {\\n 'sdkVersion':\\n self.sdk_version,\\n 'imageVersion':\\n self.image_version,\\n 'source':\\n self.source,\\n 'jobName':\\n self.job_name,\\n 'jobId':\\n self.job_id,\\n 'modelList': [{\\n 'model': self.model,\\n 'domain': self.domain,\\n 'task': self.task,\\n 'itemResult': self.item_result\\n }]\\n }\\n return self._post_request(self.BATCH_COMMIT_RESULT_URL, param)\\n\\n except Exception as e:\\n logging.error(e)\\n\\n return\\n\\n # 测试阶段完成\\n def batch_refresh_stage(self):\\n try:\\n param = {\\n 'sdkVersion': self.sdk_version,\\n 'imageVersion': self.image_version,\\n 'source': self.source,\\n 'stage': self.stage,\\n 'modelList': [{\\n 'model': self.model,\\n 'domain': self.domain,\\n 'task': self.task\\n }]\\n }\\n return self._post_request(self.BATCH_REFRESH_STAGE_URL, param)\\n\\n except Exception as e:\\n logging.error(e)\\n\\n return\\n\\n # 查询模型某个阶段的最新测试结果（只返回单个结果\\n def query_model_stage(self):\\n try:\\n param = {\\n 'sdkVersion': self.sdk_version,\\n 'model': self.model,\\n 'stage': self.stage,\\n 'imageVersion': self.image_version\\n }\\n return self._post_request(self.QUERY_MODEL_STAGE_URL, param)\\n\\n except Exception as e:\\n logging.error(e)\\n\\n return None\\n\\n # 提交模型UT测试结果\\n \\\"\\\"\\\"\\n model_tag = ModelTag()\\n model_tag.model = \\\"XXX\\\"\\n model_tag.sdk_version = \\\"0.3.7\\\"\\n model_tag.domain = \\\"nlp\\\"\\n model_tag.task = \\\"word-segmentation\\\"\\n item = model_tag.ItemResult()\\n item.result = model_tag.MODEL_PASS\\n item.name = \\\"ALL\\\"\\n item.info = \\\"\\\"\\n model_tag.item_result.append(item.to_json())\\n \\\"\\\"\\\"\\n\\n def commit_ut_result(self):\\n if self._URL is not None and self._URL != '':\\n self.job_name = 'UT'\\n self.source = 'dev'\\n self.stage = 'integration'\\n\\n self.batch_commit_result()\\n self.batch_refresh_stage()\\n\\n\\ndef commit_model_ut_result(model_name, ut_result):\\n model_tag = ModelTag()\\n model_tag.model = model_name.replace('damo/', '')\\n model_tag.sdk_version = __version__\\n # model_tag.domain = \\\"\\\"\\n # model_tag.task = \\\"\\\"\\n item = model_tag.ItemResult()\\n item.result = ut_result\\n item.name = 'ALL'\\n item.info = ''\\n model_tag.item_result.append(item.to_json())\\n model_tag.commit_ut_result()\\n\\n\\nif __name__ == '__main__':\\n import os\\n os.environ['CUDA_VISIBLE_DEVICES'] = '1'\\n os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\\n\\nimport os\\nimport shutil\\nimport tempfile\\nimport unittest\\n\\nimport torch\\n\\nfrom swift import get_logger\\nfrom swift.llm import DatasetName, InferArguments, ModelType, SftArguments, infer_main, sft_main\\n\\nNO_EVAL_HUMAN = True\\n\\nlogger = get_logger()\\n\\n\\nclass TestRun2(unittest.TestCase):\\n\\n def setUp(self):\\n print(f'Testing {type(self).__name__}.{self._testMethodName}')\\n self._tmp_dir = tempfile.TemporaryDirectory()\\n self.tmp_dir = self._tmp_dir.name\\n\\n def tearDown(self):\\n shutil.rmtree(self.tmp_dir)\\n\\n def test_loss_matching(self):\\n output_dir = 'output'\\n if not __name__ == '__main__':\\n # ignore citest error in github\\n return\\n losses = []\\n for tuner_backend in ['swift', 'peft']:\\n if tuner_backend == 'swift':\\n bool_var = True\\n else:\\n bool_var = False\\n torch.cuda.empty_cache()\\n output = sft_main([\\n '--model_type', ModelType.qwen_7b_chat, '--eval_steps', '5', '--tuner_backend', tuner_backend,\\n '--dataset', f'{DatasetName.leetcode_python_en}#200', '--output_dir', output_dir,\\n '--gradient_checkpointing', 'true', '--max_new_tokens', '100', '--use_flash_attn', 'true',\\n '--lora_target_modules', 'ALL', '--seed', '0', '--lora_bias_trainable', 'all', '--lora_modules_to_save',\\n 'EMBEDDING', 'LN', 'lm_head'\\n ])\\n best_model_checkpoint = output['best_model_checkpoint']\\n print(f'best_model_checkpoint: {best_model_checkpoint}')\\n load_dataset_config = str(bool_var or NO_EVAL_HUMAN)\\n if load_dataset_config:\\n val_dataset_sample = 2\\n else:\\n val_dataset_sample = -1\\n torch.cuda.empty_cache()\\n infer_main([\\n '--ckpt_dir', best_model_checkpoint, '--val_dataset_sample',\\n str(val_dataset_sample), '--max_new_tokens', '100', '--use_flash_attn', 'false', '--verbose',\\n str(not bool_var), '--merge_lora',\\n str(bool_var), '--load_dataset_config',\\n str(load_dataset_config)\\n ])\\n loss = output['log_history'][-1]['train_loss']\\n losses.append(loss)\\n self.assertTrue(abs(losses[0] - losses[1]) < 5e-4)\\n print(f'swift_loss: {losses[0]}')\\n print(f'peft_loss: {losses[1]}')\\n self.assertTrue(0.95 <= losses[0] <= 1)\\n\\n def test_yi_vl_6b_chat(self):\\n if not __name__ == '__main__':\\n # ignore citest error in github\\n return\\n folder = os.path.join(os.path.dirname(__file__), 'data')\\n torch.cuda.empty_cache()\\n output = sft_main(\\n SftArguments(\\n model_type=ModelType.yi_vl_6b_chat,\\n # dataset=DatasetName.capcha_images,\\n lora_target_modules='ALL',\\n train_dataset_sample=100,\\n eval_steps=5,\\n custom_train_dataset_path=[os.path.join(folder, 'multi_modal_2.jsonl')],\\n lazy_tokenize=False))\\n best_model_checkpoint = output['best_model_checkpoint']\\n torch.cuda.empty_cache()\\n infer_main(InferArguments(ckpt_dir=best_model_checkpoint, load_dataset_config=True, val_dataset_sample=2))\\n\\n def test_glm4v_9b_chat(self):\\n if not __name__ == '__main__':\\n # ignore citest error in github\\n return\\n folder = os.path.join(os.path.dirname(__file__), 'data')\\n torch.cuda.empty_cache()\\n output = sft_main(\\n SftArguments(\\n model_type=ModelType.glm4v_9b_chat,\\n # dataset=DatasetName.capcha_images,\\n # lora_target_modules='ALL',\\n train_dataset_sample=100,\\n eval_steps=5,\\n custom_train_dataset_path=[os.path.join(folder, 'multi_modal_3.jsonl')],\\n lazy_tokenize=False))\\n best_model_checkpoint = output['best_model_checkpoint']\\n torch.cuda.empty_cache()\\n infer_main(\\n InferArguments(\\n ckpt_dir=best_model_checkpoint, load_dataset_config=True, val_dataset_sample=2, merge_lora=True))\\n\\n def test_baichuan2_chat_int4(self):\\n if not __name__ == '__main__':\\n # ignore citest error in github\\n return\\n from swift.llm import sft_main, infer_main, SftArguments, InferArguments, ModelType, DatasetName\\n output = sft_main(\\n SftArguments(\\n model_type=ModelType.baichuan2_7b_chat_int4,\\n dataset=['alpaca-zh'],\\n lora_target_modules=['DEFAULT', 'EMBEDDING'],\\n train_dataset_sample=20))\\n best_model_checkpoint = output['best_model_checkpoint']\\n infer_main(InferArguments(ckpt_dir=best_model_checkpoint, load_dataset_config=True, val_dataset_sample=1))\\n\\n def test_self_cognition(self):\\n if not __name__ == '__main__':\\n # ignore citest error in github\\n return\\n for dataset in [[], [DatasetName.alpaca_zh, DatasetName.alpaca_en]]:\\n sft_args = SftArguments(\\n model_type=ModelType.qwen1half_1_8b_chat_int4,\\n dataset=dataset, # no dataset\\n train_dataset_sample=100,\\n dtype='fp16',\\n eval_steps=5,\\n output_dir='output',\\n lora_target_modules=['ALL', 'EMBEDDING'],\\n lazy_tokenize=True,\\n max_length=512,\\n self_cognition_sample=100,\\n model_name=['小黄', 'Xiao Huang'],\\n model_author=['魔搭', 'ModelScope'],\\n use_flash_attn=True)\\n torch.cuda.empty_cache()\\n output = sft_main(sft_args)\\n last_model_checkpoint = output['last_model_checkpoint']\\n best_model_checkpoint = output['best_model_checkpoint']\\n print(f'last_model_checkpoint: {last_model_checkpoint}')\\n print(f'best_model_checkpoint: {best_model_checkpoint}')\\n ckpt_dir = best_model_checkpoint or last_model_checkpoint\\n if len(dataset) == 0:\\n continue\\n infer_args = InferArguments(\\n ckpt_dir=ckpt_dir, val_dataset_sample=2, verbose=False, load_dataset_config=True)\\n # merge_lora_main(infer_args)\\n torch.cuda.empty_cache()\\n result = infer_main(infer_args)\\n print(result)\\n\\n\\nif __name__ == '__main__':\\n unittest.main()\\n\\n\\nif __name__ == '__main__':\\n import os\\n os.environ['CUDA_VISIBLE_DEVICES'] = '0'\\n os.environ['HF_ENDPOINT'] = 'https://hf-mirror.com'\\n\\nimport os\\nimport shutil\\nimport tempfile\\nimport time\\nimport unittest\\nfrom functools import partial\\nfrom typing import Any, Dict, List\\n\\nimport torch\\nimport transformers\\nfrom datasets import Dataset as HfDataset\\nfrom modelscope import Model, MsDataset, snapshot_download\\nfrom packaging import version\\nfrom torch.nn.utils.rnn import pad_sequence\\nfrom transformers import AutoConfig, AutoTokenizer\\n\\nfrom swift import Trainer, TrainingArguments, get_logger\\nfrom swift.llm import (DatasetName, InferArguments, ModelType, RLHFArguments, SftArguments, infer_main, merge_lora_main,\\n rlhf_main, sft_main)\\n\\nNO_EVAL_HUMAN = True\\n\\nlogger = get_logger()\\n\\n\\nclass TestRun(unittest.TestCase):\\n\\n def setUp(self):\\n print(f'Testing {type(self).__name__}.{self._testMethodName}')\\n self._tmp_dir = tempfile.TemporaryDirectory()\\n self.tmp_dir = self._tmp_dir.name\\n\\n def tearDown(self):\\n shutil.rmtree(self.tmp_dir)\\n\\n def test_template(self):\\n if not __name__ == '__main__':\\n # ignore citest error in github\\n return\\n torch.cuda.empty_cache()\\n output = sft_main(\\n SftArguments(\\n model_type=ModelType.qwen1half_1_8b,\\n template_type='qwen',\\n sft_type='full',\\n dataset=f'{DatasetName.jd_sentiment_zh}#200',\\n eval_steps=5))\\n best_model_checkpoint = output['best_model_checkpoint']\\n torch.cuda.empty_cache()\\n result = infer_main(\\n InferArguments(ckpt_dir=best_model_checkpoint, load_dataset_config=True, val_dataset_sample=2))\\n assert len(result['result'][0]['response']) < 20\\n\\n def test_basic(self):\\n output_dir = 'output'\\n quantization_bit_list = [0, 4]\\n train_dataset_fnames = [\\n 'alpaca.csv', 'chatml.jsonl', 'swift_pre.jsonl', 'swift_single.csv', 'swift_multi.jsonl',\\n 'swift_multi.json#2'\\n ]\\n folder = os.path.join(os.path.dirname(__file__), 'data')\\n dataset = [\\n f'MS::{DatasetName.alpaca_zh}#20',\\n f'{DatasetName.jd_sentiment_zh}#20',\\n 'AI-ModelScope/alpaca-gpt4-data-zh#20',\\n 'HF::llm-wizard/alpaca-gpt4-data-zh#20',\\n 'hurner/alpaca-gpt4-data-zh#20',\\n 'HF::shibing624/alpaca-zh#20',\\n ] + [os.path.join(folder, fname) for fname in train_dataset_fnames]\\n if not __name__ == '__main__':\\n output_dir = self.tmp_dir\\n quantization_bit_list = [4]\\n dataset = dataset[:2]\\n import transformers\\n from packaging import version\\n if version.parse(transformers.__version__) >= version.parse('4.42'):\\n model_type = ModelType.qwen2_0_5b_instruct\\n else:\\n model_type = ModelType.chatglm3_6b\\n for quantization_bit in quantization_bit_list:\\n if quantization_bit == 4 and version.parse(transformers.__version__) >= version.parse('4.38'):\\n continue\\n predict_with_generate = True\\n if quantization_bit == 0:\\n predict_with_generate = False\\n sft_args = SftArguments(\\n model_type=model_type,\\n template_type='AUTO',\\n lora_target_modules=['AUTO', 'EMBEDDING'],\\n quantization_bit=quantization_bit,\\n batch_size=2,\\n eval_steps=5,\\n adam_beta2=0.95,\\n check_dataset_strategy='warning',\\n predict_with_generate=predict_with_generate,\\n dataset=dataset,\\n val_dataset=f'{DatasetName.jd_sentiment_zh}#20',\\n output_dir=output_dir,\\n include_num_input_tokens_seen=True,\\n gradient_checkpointing=True)\\n self.assertTrue(sft_args.gradient_accumulation_steps == 8)\\n torch.cuda.empty_cache()\\n output = sft_main(sft_args)\\n print(output)\\n best_model_checkpoint = output['best_model_checkpoint']\\n print(f'best_model_checkpoint: {best_model_checkpoint}')\\n if __name__ == '__main__':\\n infer_args = InferArguments(\\n ckpt_dir=best_model_checkpoint,\\n merge_lora={\\n 0: True,\\n 4: False\\n }[quantization_bit],\\n merge_device_map='cpu',\\n load_dataset_config=NO_EVAL_HUMAN,\\n val_dataset_sample=5)\\n torch.cuda.empty_cache()\\n result = infer_main(infer_args)\\n print(result)\\n # if __name__ == '__main__':\\n # app_ui_main(infer_args)\\n\\n def test_vl_audio(self):\\n output_dir = 'output'\\n if not __name__ == '__main__':\\n # ignore citest error in github\\n return\\n model_type_list = [ModelType.qwen_vl_chat, ModelType.qwen_audio_chat]\\n dataset_list = [DatasetName.coco_en_mini, DatasetName.aishell1_zh_mini]\\n for model_type, dataset in zip(model_type_list, dataset_list):\\n sft_args = SftArguments(\\n model_type=model_type,\\n template_type='AUTO',\\n eval_steps=5,\\n check_dataset_strategy='warning',\\n lora_target_modules='ALL',\\n train_dataset_sample=200,\\n dataset=[dataset],\\n output_dir=output_dir,\\n gradient_checkpointing=True,\\n lazy_tokenize=True,\\n disable_tqdm=True)\\n torch.cuda.empty_cache()\\n output = sft_main(sft_args)\\n print(output)\\n best_model_checkpoint = output['best_model_checkpoint']\\n print(f'best_model_checkpoint: {best_model_checkpoint}')\\n infer_args = InferArguments(\\n ckpt_dir=best_model_checkpoint,\\n load_dataset_config=True,\\n stream={\\n ModelType.qwen_vl_chat: True,\\n ModelType.qwen_audio_chat: False\\n }[model_type],\\n val_dataset_sample=5)\\n # merge_lora_main(infer_args) # TODO: ERROR FIX\\n torch.cuda.empty_cache()\\n result = infer_main(infer_args)\\n print(result)\\n\\n def test_vqa(self):\\n if not __name__ == '__main__':\\n # ignore citest error in github\\n return\\n train_dataset_fnames = ['science-qa#300', 'a-okvqa#300', 'alpaca-cleaned#300']\\n val_dataset_fnames = ['okvqa']\\n\\n sft_args = SftArguments(\\n model_type='yi-vl-6b-chat',\\n dataset=train_dataset_fnames,\\n lora_target_modules='ALL',\\n num_train_epochs=1,\\n check_dataset_strategy='warning')\\n\\n torch.cuda.empty_cache()\\n result = sft_main(sft_args)\\n best_model_checkpoint = result['best_model_checkpoint']\\n\\n infer_args = InferArguments(\\n ckpt_dir=best_model_checkpoint,\\n load_args_from_ckpt_dir=True,\\n load_dataset_config=True,\\n merge_lora=False,\\n val_dataset_sample=10,\\n dataset=val_dataset_fnames)\\n torch.cuda.empty_cache()\\n infer_main(infer_args)\\n\\n def test_gpt4o_image(self):\\n if not __name__ == '__main__':\\n # ignore citest error in github\\n return\\n train_dataset_fnames = ['sharegpt-4o-image']\\n\\n sft_args = SftArguments(\\n model_type='yi-vl-6b-chat',\\n dataset=train_dataset_fnames,\\n lora_target_modules='ALL',\\n train_dataset_sample=200,\\n num_train_epochs=1,\\n eval_steps=10,\\n save_steps=10,\\n check_dataset_strategy='warning')\\n\\n torch.cuda.empty_cache()\\n self.assertTrue(sft_main(sft_args)['best_model_checkpoint'])\\n\\n def test_custom_dataset(self):\\n if not __name__ == '__main__':\\n # ignore citest error in github\\n return\\n train_dataset_fnames = [\\n 'alpaca.csv', 'chatml.jsonl', 'swift_pre.jsonl', 'swift_single.csv', 'swift_multi.jsonl',\\n 'swift_multi.json', 'sharegpt.jsonl'\\n ]\\n val_dataset_fnames = [\\n 'alpaca.jsonl',\\n 'alpaca2.csv',\\n 'conversations.jsonl',\\n 'swift_pre.csv',\\n 'swift_single.jsonl',\\n # 'swift_#:#.jsonl#3'\\n ]\\n folder = os.path.join(os.path.dirname(__file__), 'data')\\n resume_from_checkpoint = None\\n for num_train_epochs in [1, 2]:\\n sft_args = SftArguments(\\n model_type='qwen-7b-chat',\\n dataset=['self-cognition#20'],\\n custom_train_dataset_path=[os.path.join(folder, fname) for fname in train_dataset_fnames],\\n custom_val_dataset_path=[os.path.join(folder, fname) for fname in val_dataset_fnames],\\n lora_target_modules='ALL',\\n resume_from_checkpoint=resume_from_checkpoint,\\n num_train_epochs=num_train_epochs,\\n model_name='小黄',\\n model_author='魔搭',\\n check_dataset_strategy='warning')\\n\\n torch.cuda.empty_cache()\\n result = sft_main(sft_args)\\n best_model_checkpoint = result['best_model_checkpoint']\\n resume_from_checkpoint = result['last_model_checkpoint']\\n\\n for load_args_from_ckpt_dir in [True, False]:\\n kwargs = {}\\n if load_args_from_ckpt_dir is False:\\n kwargs = {'model_type': 'qwen-7b-chat'}\\n infer_args = InferArguments(\\n ckpt_dir=best_model_checkpoint,\\n load_args_from_ckpt_dir=load_args_from_ckpt_dir,\\n load_dataset_config=load_args_from_ckpt_dir and NO_EVAL_HUMAN,\\n merge_lora=load_args_from_ckpt_dir,\\n val_dataset_sample=-1,\\n custom_val_dataset_path=[os.path.join(folder, fname) for fname in val_dataset_fnames],\\n **kwargs)\\n torch.cuda.empty_cache()\\n infer_main(infer_args)\\n\\n def test_cogagent_instruct(self):\\n if not __name__ == '__main__':\\n # ignore citest error in github\\n return\\n quantization_bit = 4\\n if version.parse(transformers.__version__) >= version.parse('4.38'):\\n quantization_bit = 0\\n torch.cuda.empty_cache()\\n output = sft_main(\\n SftArguments(\\n model_type=ModelType.cogagent_18b_instruct,\\n dataset=DatasetName.coco_en_2_mini,\\n train_dataset_sample=100,\\n lora_target_modules='ALL',\\n eval_steps=5,\\n quantization_bit=quantization_bit))\\n best_model_checkpoint = output['best_model_checkpoint']\\n torch.cuda.empty_cache()\\n infer_main(InferArguments(ckpt_dir=best_model_checkpoint, load_dataset_config=True, val_dataset_sample=2))\\n\\n def test_xcomposer_chat(self):\\n if not __name__ == '__main__':\\n # ignore citest error in github\\n return\\n torch.cuda.empty_cache()\\n output = sft_main(\\n SftArguments(\\n model_type=ModelType.internlm_xcomposer2_7b_chat,\\n dataset=DatasetName.coco_en_mini,\\n lora_target_modules='DEFAULT',\\n train_dataset_sample=100,\\n eval_steps=5))\\n best_model_checkpoint = output['best_model_checkpoint']\\n torch.cuda.empty_cache()\\n infer_main(InferArguments(ckpt_dir=best_model_checkpoint, load_dataset_config=True, val_dataset_sample=2))\\n\\n def test_rlhf(self):\\n if not __name__ == '__main__':\\n # ignore citest error in github\\n return\\n torch.cuda.empty_cache()\\n # llm rlhf\\n rlhf_types = ['dpo', 'orpo', 'simpo', 'kto', 'cpo']\\n for rlhf_type in rlhf_types:\\n dataset_name = 'hh-rlhf-cn-harmless-base-cn' if rlhf_type != 'kto' else 'ultrafeedback-kto'\\n output = rlhf_main(\\n RLHFArguments(\\n rlhf_type=rlhf_type,\\n model_type=ModelType.qwen_1_8b_chat,\\n dataset=dataset_name,\\n train_dataset_sample=100,\\n eval_steps=5))\\n best_model_checkpoint = output['best_model_checkpoint']\\n torch.cuda.empty_cache()\\n infer_main(InferArguments(ckpt_dir=best_model_checkpoint, load_dataset_config=True, val_dataset_sample=2))\\n\\n # mllm rlhf\\n visual_rlhf_types = ['dpo', 'orpo', 'simpo', 'cpo']\\n test_model = ['llava1_6-mistral-7b-instruct', 'internvl2-2b',\\n 'florence-2-large'] # decoder only and encoder-decoder\\n for rlhf_type in visual_rlhf_types:\\n for model in test_model:\\n dataset_name = 'rlaif-v'\\n output = rlhf_main(\\n RLHFArguments(\\n rlhf_type=rlhf_type,\\n model_type=model,\\n dataset=dataset_name,\\n train_dataset_sample=100,\\n eval_steps=5))\\n best_model_checkpoint = output['best_model_checkpoint']\\n torch.cuda.empty_cache()\\n infer_main(\\n InferArguments(ckpt_dir=best_model_checkpoint, load_dataset_config=True, val_dataset_sample=2))\\n\\n def test_pai_compat(self):\\n if not __name__ == '__main__':\\n # ignore citest error in github\\n return\\n from swift.llm import sft_main, infer_main\\n os.environ['PAI_TRAINING_JOB_ID'] = '123456'\\n folder = os.path.join(os.path.dirname(__file__), 'config')\\n tensorboard_dir = os.path.join('output/pai_test', 'pai_tensorboard')\\n os.environ['PAI_OUTPUT_TENSORBOARD'] = tensorboard_dir\\n sft_json = os.path.join(folder, 'sft.json')\\n infer_json = os.path.join(folder, 'infer.json')\\n torch.cuda.empty_cache()\\n output = sft_main([sft_json])\\n print()\\n infer_args = {\\n 'ckpt_dir': output['best_model_checkpoint'],\\n 'val_dataset_sample': 2,\\n 'load_dataset_config': True,\\n }\\n import json\\n with open(infer_json, 'w') as f:\\n json.dump(infer_args, f, ensure_ascii=False, indent=4)\\n torch.cuda.empty_cache()\\n infer_main([infer_json])\\n os.environ.pop('PAI_TRAINING_JOB_ID')\\n\\n def test_deepseek_vl_chat(self):\\n if not __name__ == '__main__':\\n # ignore citest error in github\\n return\\n folder = os.path.join(os.path.dirname(__file__), 'data')\\n torch.cuda.empty_cache()\\n sft_main(\\n SftArguments(\\n model_type=ModelType.deepseek_vl_1_3b_chat,\\n # dataset=DatasetName.capcha_images,\\n lora_target_modules='ALL',\\n train_dataset_sample=100,\\n eval_steps=5,\\n custom_train_dataset_path=[os.path.join(folder, 'multi_modal_1.jsonl')],\\n lazy_tokenize=False))\\n\\n\\ndef data_collate_fn(batch: List[Dict[str, Any]], tokenizer) -> Dict[str, torch.Tensor]:\\n # text-classification\\n assert tokenizer.pad_token_id is not None\\n input_ids = [torch.tensor(b['input_ids']) for b in batch]\\n labels = torch.tensor([b['labels'] for b in batch])\\n attention_mask = [torch.ones(len(input_ids[i]), dtype=torch.int64) for i in range(len(input_ids))]\\n\\n input_ids = pad_sequence(input_ids, batch_first=True, padding_value=tokenizer.pad_token_id)\\n attention_mask = pad_sequence(attention_mask, batch_first=True, padding_value=0)\\n return {'input_ids': input_ids, 'attention_mask': attention_mask, 'labels': labels}\\n\\n\\nclass BertTrainer(Trainer):\\n\\n def compute_loss(self, model, inputs, return_outputs=False):\\n outputs = model(**inputs)\\n loss = outputs.loss\\n if loss is None:\\n logits, loss = list(outputs.logits)\\n return (loss, outputs) if return_outputs else loss\\n\\n\\nclass TestTrainer(unittest.TestCase):\\n\\n def setUp(self):\\n self._tmp_dir = tempfile.TemporaryDirectory()\\n self.tmp_dir = self._tmp_dir.name\\n # self.tmp_dir = 'test'\\n logger.info(f'self.tmp_dir: {self.tmp_dir}')\\n\\n def tearDown(self):\\n if os.path.isdir(self.tmp_dir):\\n shutil.rmtree(self.tmp_dir)\\n # api = HubApi()\\n # api.delete_model(self.hub_model_id)\\n # logger.info(f'delete model: {self.hub_model_id}')\\n\\n def test_trainer(self):\\n self.hub_model_id = 'test_trainer2'\\n logger.info(f'self.hub_model_id: {self.hub_model_id}')\\n self.tmp_dir = 'output/damo/nlp_structbert_backbone_base_std'\\n push_to_hub = True\\n if not __name__ == '__main__':\\n # ignore citest error in github\\n return\\n model_id = 'damo/nlp_structbert_backbone_base_std'\\n model_dir = snapshot_download(model_id, 'master')\\n tokenizer = AutoTokenizer.from_pretrained(model_dir)\\n dataset = MsDataset.load('clue', subset_name='tnews')\\n num_labels = max(dataset['train']['label']) + 1\\n model = Model.from_pretrained(model_dir, task='text-classification', num_labels=num_labels)\\n train_dataset, val_dataset = dataset['train'].to_hf_dataset(), dataset['validation'].to_hf_dataset()\\n train_dataset: HfDataset = train_dataset.select(range(100))\\n val_dataset: HfDataset = val_dataset.select(range(20))\\n\\n #\\n def tokenize_func(examples):\\n data = tokenizer(examples['sentence'], return_attention_mask=False)\\n examples['input_ids'] = data['input_ids']\\n examples['labels'] = examples['label']\\n del examples['sentence'], examples['label']\\n return examples\\n\\n train_dataset = train_dataset.map(tokenize_func)\\n val_dataset = val_dataset.map(tokenize_func)\\n\\n data_collator = partial(data_collate_fn, tokenizer=tokenizer)\\n for save_only_model in [True, False]:\\n trainer_args = TrainingArguments(\\n self.tmp_dir,\\n do_train=True,\\n do_eval=True,\\n num_train_epochs=1,\\n evaluation_strategy='steps',\\n save_strategy='steps',\\n per_device_train_batch_size=4,\\n per_device_eval_batch_size=4,\\n push_to_hub=push_to_hub,\\n hub_token=None, # use env var\\n hub_private_repo=True,\\n hub_strategy='every_save',\\n hub_model_id=self.hub_model_id,\\n overwrite_output_dir=True,\\n save_steps=10,\\n save_total_limit=2,\\n metric_for_best_model='loss',\\n greater_is_better=False,\\n report_to=['tensorboard'],\\n gradient_accumulation_steps=1,\\n logging_steps=5,\\n eval_steps=10,\\n save_safetensors=False,\\n save_only_model=save_only_model)\\n trainer_args._n_gpu = 1\\n trainer = BertTrainer(model, trainer_args, data_collator, train_dataset, val_dataset, tokenizer)\\n self.hub_model_id = trainer_args.hub_model_id\\n trainer.train()\\n if trainer_args.push_to_hub:\\n trainer.push_to_hub()\\n\\n\\nif __name__ == '__main__':\\n unittest.main()\\n\\n\\nimport unittest\\n\\nimport json\\nimport torch\\nfrom modelscope import GenerationConfig\\n\\nfrom swift.llm import (TEMPLATE_MAPPING, ModelType, Template, get_default_template_type, get_model_tokenizer,\\n get_template, inference, messages_to_history)\\n\\nif __name__ == '__main__':\\n import os\\n os.environ['CUDA_VISIBLE_DEVICES'] = '0'\\n\\nSKPT_TEST = True\\n\\n\\nclass TestTemplate(unittest.TestCase):\\n\\n def test_template(self):\\n model_types = [ModelType.qwen_7b_chat_int4]\\n for model_type in model_types:\\n _, tokenizer = get_model_tokenizer(model_type, load_model=False)\\n template_type = get_default_template_type(model_type)\\n template = get_template(template_type, tokenizer)\\n history = [\\n ('你好，你是谁？', '我是来自达摩院的大规模语言模型，我叫通义千问。'),\\n ]\\n data = {\\n 'system': 'you are a helpful assistant!',\\n 'query': '浙江的省会在哪？',\\n 'response': '浙江的省会是杭州。',\\n 'history': history\\n }\\n from swift.llm import print_example\\n print_example(template.encode(data)[0], tokenizer)\\n input_ids = template.encode(data)[0]['input_ids']\\n print(model_type)\\n text = tokenizer.decode(input_ids)\\n result = \\\"\\\"\\\"<|im_start|>system\\nyou are a helpful assistant!<|im_end|>\\n<|im_start|>user\\n你好，你是谁？<|im_end|>\\n<|im_start|>assistant\\n我是来自达摩院的大规模语言模型，我叫通义千问。<|im_end|>\\n<|im_start|>user\\n浙江的省会在哪？<|im_end|>\\n<|im_start|>assistant\\n浙江的省会是杭州。<|im_end|>\\\"\\\"\\\"\\n self.assertTrue(result == text)\\n\\n @unittest.skipIf(SKPT_TEST, 'To avoid excessive testing time caused by downloading models and '\\n 'to prevent OOM (Out of Memory) errors.')\\n def test_chatglm3_template(self):\\n model_type = ModelType.chatglm3_6b\\n template_type = get_default_template_type(model_type)\\n model, tokenizer = get_model_tokenizer(model_type, load_model=True)\\n template = get_template(template_type, tokenizer)\\n model.generation_config = GenerationConfig(\\n max_new_tokens=128,\\n temperature=0.9,\\n top_k=20,\\n top_p=0.9,\\n repetition_penalt=1.05,\\n do_sample=True,\\n eos_token_id=tokenizer.eos_token_id,\\n pad_token_id=tokenizer.eos_token_id)\\n query = '12345+234=？'\\n print(f'query: {query}')\\n response, _ = inference(model, template, query)\\n print(f'swift response: {response}')\\n system = 'you are a helpful assistant!'\\n response = model.chat(tokenizer, query, history=[{'role': 'system', 'content': system}], max_length=None)[0]\\n print(f'official response: {response}')\\n #\\n input_ids_official = [\\n 64790, 64792, 64794, 30910, 13, 344, 383, 260, 6483, 9319, 30992, 64795, 30910, 13, 30910, 30939, 30943,\\n 30966, 30972, 30970, 31011, 30943, 30966, 30972, 30980, 31514, 64796\\n ] + [30910, 13]\\n input_ids_swift = template.encode({'query': query, 'system': system})[0]['input_ids']\\n self.assertTrue(input_ids_swift == input_ids_official)\\n\\n def test_template_grounding(self):\\n example_single = {\\n 'query':\\n 'Where is the ',\\n 'response':\\n 'At ',\\n 'images': ['/mnt/workspace/output.png'],\\n 'objects':\\n json.loads(\\\"[{\\\\\\\"caption\\\\\\\": \\\\\\\"guy in red\\\\\\\", \\\\\\\"bbox\\\\\\\": [1, 2, 3, 4], \\\"\\n \\\"\\\\\\\"bbox_type\\\\\\\": \\\\\\\"real\\\\\\\", \\\\\\\"image\\\\\\\": 0}]\\\")\\n }\\n\\n example_single2 = {\\n 'query':\\n 'Where is the ',\\n 'response':\\n 'At ',\\n 'images': ['/mnt/workspace/output.png'],\\n 'objects':\\n json.loads(\\\"[{\\\\\\\"caption\\\\\\\": \\\\\\\"guy in red\\\\\\\", \\\\\\\"bbox\\\\\\\": [[1, 2, 3, 4]], \\\"\\n \\\"\\\\\\\"bbox_type\\\\\\\": \\\\\\\"real\\\\\\\", \\\\\\\"image\\\\\\\": 0}]\\\")\\n }\\n\\n example_multi = {\\n 'query':\\n 'Where is the ',\\n 'response':\\n 'At ',\\n 'images': ['/mnt/workspace/output.png'],\\n 'objects':\\n json.loads(\\\"[{\\\\\\\"caption\\\\\\\": \\\\\\\"guy in red\\\\\\\", \\\\\\\"bbox\\\\\\\": [[6, 7, 8, 9],[1, 2, 3, 4]], \\\"\\n \\\"\\\\\\\"bbox_type\\\\\\\": \\\\\\\"real\\\\\\\", \\\\\\\"image\\\\\\\": 0}]\\\")\\n }\\n\\n # qwen-vl\\n template_dict: dict = TEMPLATE_MAPPING['qwen-vl']\\n output = template_dict['template'].replace_box(0, example_single)\\n self.assertTrue(output[0] == '(1,2),(3,4)')\\n output = template_dict['template'].replace_box(0, example_single2)\\n self.assertTrue(output[0] == '(1,2),(3,4)')\\n output = template_dict['template'].replace_box(0, example_multi)\\n self.assertTrue(output[0] == '(6,7),(8,9)(1,2),(3,4)')\\n\\n # qwen2-vl\\n template_dict: dict = TEMPLATE_MAPPING['qwen2-vl']\\n output = template_dict['template'].replace_box(0, example_single)\\n self.assertTrue(output[0] == '<|box_start|>(1,2),(3,4)<|box_end|>')\\n output = template_dict['template'].replace_box(0, example_single2)\\n self.assertTrue(output[0] == '<|box_start|>(1,2),(3,4)<|box_end|>')\\n output = template_dict['template'].replace_box(0, example_multi)\\n self.assertTrue(output[0] == '<|box_start|>(6,7),(8,9)<|box_end|><|box_start|>(1,2),(3,4)<|box_end|>')\\n\\n # florence\\n template_dict: dict = TEMPLATE_MAPPING['florence']\\n output = template_dict['template'].replace_box(0, example_single)\\n self.assertTrue(output[0] == '')\\n output = template_dict['template'].replace_box(0, example_single2)\\n self.assertTrue(output[0] == '')\\n output = template_dict['template'].replace_box(0, example_multi)\\n self.assertTrue(output[0] == ',')\\n\\n # internvl2\\n template_dict: dict = TEMPLATE_MAPPING['internvl2']\\n output = template_dict['template'].replace_box(0, example_single)\\n self.assertTrue(output[0] == ' [[1, 2, 3, 4]] ')\\n output = template_dict['template'].replace_box(0, example_single2)\\n self.assertTrue(output[0] == ' [[1, 2, 3, 4]] ')\\n output = template_dict['template'].replace_box(0, example_multi)\\n self.assertTrue(output[0] == ' [[6, 7, 8, 9],[1, 2, 3, 4]] ')\\n\\n # minicpm-v\\n template_dict: dict = TEMPLATE_MAPPING['minicpm-v']\\n output = template_dict['template'].replace_box(0, example_single)\\n self.assertTrue(output[0] == '[(1,2),(3,4)]')\\n output = template_dict['template'].replace_box(0, example_single2)\\n self.assertTrue(output[0] == '[(1,2),(3,4)]')\\n output = template_dict['template'].replace_box(0, example_multi)\\n self.assertTrue(output[0] == '[(6,7),(8,9)],[(1,2),(3,4)]')\\n\\n @unittest.skipIf(SKPT_TEST, 'To avoid excessive testing time caused by downloading models and '\\n 'to prevent OOM (Out of Memory) errors.')\\n def test_qwen_template(self):\\n model_type = ModelType.qwen_7b_chat\\n template_type = get_default_template_type(model_type)\\n model, tokenizer = get_model_tokenizer(model_type, load_model=True)\\n template = get_template(template_type, tokenizer)\\n query = '12345+234=？'\\n print(f'query: {query}')\\n response, _ = inference(model, template, query)\\n print(f'swift response: {response}')\\n model.generation_config.chat_format = 'chatml'\\n model.generation_config.max_window_size = 1024\\n response = model.chat(tokenizer, query, None, max_length=None)[0]\\n print(f'official response: {response}')\\n #\\n input_ids_official = [\\n 151644, 8948, 198, 2610, 525, 264, 10950, 17847, 13, 151645, 198, 151644, 872, 198, 16, 17, 18, 19, 20, 10,\\n 17, 18, 19, 28, 11319, 151645, 198, 151644, 77091, 198\\n ]\\n input_ids_swift = template.encode({'query': query})[0]['input_ids']\\n self.assertTrue(input_ids_swift == input_ids_official)\\n\\n @unittest.skipIf(SKPT_TEST, 'To avoid excessive testing time caused by downloading models and '\\n 'to prevent OOM (Out of Memory) errors.')\\n def test_llama_template(self):\\n model_type = ModelType.llama2_7b_chat\\n template_type = get_default_template_type(model_type)\\n _, tokenizer = get_model_tokenizer(model_type, load_model=False)\\n from modelscope import Model, snapshot_download\\n model_dir = snapshot_download('modelscope/Llama-2-7b-chat-ms', 'master', ignore_file_pattern=[r'.+\\\\.bin$'])\\n model = Model.from_pretrained(model_dir, device_map='auto')\\n template = get_template(template_type, tokenizer)\\n model.generation_config = GenerationConfig(\\n max_new_tokens=128,\\n temperature=0.9,\\n top_k=20,\\n top_p=0.9,\\n repetition_penalt=1.05,\\n do_sample=True,\\n eos_token_id=tokenizer.eos_token_id,\\n pad_token_id=tokenizer.eos_token_id)\\n query = '12345+234=？'\\n print(f'query: {query}')\\n template.use_default_system = False\\n response, _ = inference(model, template, query)\\n print(f'swift response: {response}')\\n response = model.chat({'text': query}, tokenizer)['response']\\n print(f'official response: {response}')\\n # ref: https://huggingface.co/blog/zh/llama2#%E5%A6%82%E4%BD%95%E6%8F%90%E7%A4%BA-llama-2\\n query = \\\"There's a llama in my garden 😱 What should I do?\\\"\\n response = '123'\\n template.tokenizer.use_default_system_prompt = False\\n messages = [{\\n 'role': 'user',\\n 'content': query\\n }, {\\n 'role': 'assistant',\\n 'content': response\\n }, {\\n 'role': 'user',\\n 'content': query\\n }]\\n input_ids_official = template.tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True)\\n example = messages_to_history(messages)\\n input_ids_swift = template.encode(example)[0]['input_ids']\\n self.assertTrue(input_ids_swift == input_ids_official)\\n template.use_default_system = True\\n template.tokenizer.use_default_system_prompt = True\\n input_ids_swift = template.encode({'query': query})[0]['input_ids']\\n input_ids_official = template.tokenizer.apply_chat_template([{\\n 'role': 'user',\\n 'content': query\\n }],\\n tokenize=True,\\n add_generation_prompt=True)\\n self.assertTrue(input_ids_swift == input_ids_official)\\n\\n @unittest.skipIf(SKPT_TEST, 'To avoid excessive testing time caused by downloading models and '\\n 'to prevent OOM (Out of Memory) errors.')\\n def test_baichuan_template(self):\\n model_type = ModelType.baichuan2_7b_chat\\n template_type = get_default_template_type(model_type)\\n model, tokenizer = get_model_tokenizer(model_type, load_model=True)\\n template = get_template(template_type, tokenizer)\\n query = '12345+234=？'\\n print(f'query: {query}')\\n response, _ = inference(model, template, query)\\n print(f'swift response: {response}')\\n system = 'you are a helpful assistant!'\\n response = model.chat(tokenizer, [{'role': 'system', 'content': system}, {'role': 'user', 'content': query}])\\n print(f'official response: {response}')\\n #\\n input_ids_official = [\\n 5035, 1484, 1346, 13629, 14002, 73, 195, 92336, 92338, 92354, 92369, 92358, 62, 92338, 92354, 92369, 64, 68,\\n 196\\n ]\\n input_ids_swift = template.encode({'query': query, 'system': system})[0]['input_ids']\\n self.assertTrue(input_ids_swift == input_ids_official)\\n\\n @unittest.skipIf(SKPT_TEST, 'To avoid excessive testing time caused by downloading models and '\\n 'to prevent OOM (Out of Memory) errors.')\\n def test_chatglm2_template(self):\\n model_type = ModelType.chatglm2_6b\\n template_type = get_default_template_type(model_type)\\n model, tokenizer = get_model_tokenizer(model_type, load_model=True)\\n template = get_template(template_type, tokenizer)\\n model.generation_config = GenerationConfig(\\n max_new_tokens=128,\\n temperature=0.9,\\n top_k=20,\\n top_p=0.9,\\n repetition_penalt=1.05,\\n do_sample=True,\\n eos_token_id=tokenizer.eos_token_id,\\n pad_token_id=tokenizer.eos_token_id)\\n query = '12345+234=？'\\n print(f'query: {query}')\\n response, _ = inference(model, template, query)\\n print(f'swift response: {response}')\\n response = model.chat(tokenizer, query)[0]\\n print(f'official response: {response}')\\n #\\n input_ids_official = [\\n 64790, 64792, 790, 30951, 517, 30910, 30939, 30996, 13, 13, 54761, 31211, 30939, 30943, 30966, 30972, 30970,\\n 31011, 30943, 30966, 30972, 30980, 31514, 13, 13, 55437, 31211\\n ]\\n input_ids_swift = template.encode({'query': query})[0]['input_ids']\\n self.assertTrue(input_ids_swift == input_ids_official)\\n\\n @unittest.skipIf(SKPT_TEST, 'To avoid excessive testing time caused by downloading models and '\\n 'to prevent OOM (Out of Memory) errors.')\\n def test_internlm_template(self):\\n torch.cuda.empty_cache()\\n model_type = ModelType.internlm_20b_chat\\n template_type = get_default_template_type(model_type)\\n model, tokenizer = get_model_tokenizer(model_type, load_model=True)\\n template = get_template(template_type, tokenizer)\\n model.generation_config = GenerationConfig(\\n max_new_tokens=128,\\n temperature=0.9,\\n top_k=20,\\n top_p=0.9,\\n repetition_penalt=1.05,\\n do_sample=True,\\n eos_token_id=tokenizer.eos_token_id,\\n pad_token_id=tokenizer.eos_token_id)\\n query = '12345+234=？'\\n print(f'query: {query}')\\n response, _ = inference(model, template, query)\\n print(f'swift response: {response}')\\n response = model.chat(tokenizer, query)[0]\\n print(f'official response: {response}')\\n #\\n input_ids_official = [\\n 1, 333, 352, 2472, 352, 27232, 2770, 657, 589, 15358, 17993, 6843, 963, 505, 4576, 11146, 451, 60614, 60381,\\n 98666, 62412, 60735, 4452, 285, 4576, 11146, 451, 60614, 60381, 98666, 62412, 60735, 313, 505, 395, 7659,\\n 1813, 4287, 1762, 560, 505, 8020, 684, 36956, 15358, 31288, 451, 67738, 75808, 70730, 699, 1226, 505, 6342,\\n 442, 517, 11100, 328, 10894, 328, 454, 51978, 756, 285, 4576, 11146, 451, 60614, 60381, 98666, 62412, 60735,\\n 313, 777, 3696, 454, 19187, 19829, 4563, 435, 410, 4287, 12032, 684, 410, 1341, 1893, 569, 6519, 454, 262,\\n 68242, 756, 333, 352, 1621, 352, 27232, 4575, 1889, 342, 11622, 310, 99050, 364, 333, 352, 23845, 352, 27232\\n ]\\n input_ids_swift = template.encode({'query': query})[0]['input_ids']\\n self.assertTrue(input_ids_swift == input_ids_official)\\n\\n @unittest.skipIf(SKPT_TEST, 'To avoid excessive testing time caused by downloading models and '\\n 'to prevent OOM (Out of Memory) errors.')\\n def test_bluelm_template(self):\\n model_type = ModelType.bluelm_7b_chat_32k\\n template_type = get_default_template_type(model_type)\\n model, tokenizer = get_model_tokenizer(model_type, load_model=True)\\n template = get_template(template_type, tokenizer)\\n model.generation_config = GenerationConfig(\\n max_new_tokens=128,\\n temperature=0.9,\\n top_k=20,\\n top_p=0.9,\\n repetition_penalt=1.05,\\n do_sample=True,\\n eos_token_id=tokenizer.eos_token_id,\\n pad_token_id=tokenizer.eos_token_id)\\n query = '三国演义的作者是谁？'\\n print(f'query: {query}')\\n response, _ = inference(model, template, query)\\n print(f'swift response: {response}')\\n inputs = tokenizer('[|Human|]:三国演义的作者是谁？[|AI|]:', return_tensors='pt')\\n inputs = inputs.to('cuda:0')\\n pred = model.generate(**inputs, max_new_tokens=64, repetition_penalty=1.1)\\n response = tokenizer.decode(pred.cpu()[0], skip_special_tokens=True)\\n print(f'official response: {response}')\\n #\\n input_ids_official = inputs['input_ids'][0].tolist()\\n input_ids_swift = template.encode({'query': query})[0]['input_ids']\\n self.assertTrue(input_ids_swift == input_ids_official)\\n\\n @unittest.skipIf(SKPT_TEST, 'To avoid excessive testing time caused by downloading models and '\\n 'to prevent OOM (Out of Memory) errors.')\\n def test_qwen_generation_template(self):\\n model_type = ModelType.qwen_7b\\n template_type = get_default_template_type(model_type)\\n model, tokenizer = get_model_tokenizer(model_type, load_model=True)\\n template = get_template(template_type, tokenizer)\\n query = '蒙古国的首都是乌兰巴托（Ulaanbaatar）\\\\n冰岛的首都是雷克雅未克（Reykjavik）\\\\n埃塞俄比亚的首都是'\\n print(f'query: {query}')\\n response, _ = inference(model, template, query)\\n print(f'swift response: {response}')\\n model.generation_config.chat_format = 'raw'\\n model.generation_config.max_window_size = 1024\\n inputs = tokenizer(query, return_tensors='pt').to('cuda')\\n response = tokenizer.decode(model.generate(**inputs)[0, len(inputs['input_ids'][0]):])\\n print(f'official response: {response}')\\n #\\n input_ids_official = inputs['input_ids'][0].tolist()\\n input_ids_swift = template.encode({'query': query})[0]['input_ids']\\n self.assertTrue(input_ids_swift == input_ids_official)\\n\\n @unittest.skipIf(SKPT_TEST, 'To avoid excessive testing time caused by downloading models and '\\n 'to prevent OOM (Out of Memory) errors.')\\n def test_codefuse_codellama_34b_template(self):\\n torch.cuda.empty_cache()\\n model_type = ModelType.codefuse_codellama_34b_chat\\n model, tokenizer = get_model_tokenizer(model_type)\\n template_type = get_default_template_type(model_type)\\n template = get_template(template_type, tokenizer)\\n model.generation_config.max_length = 128\\n query = '写快排.'\\n print(f'query: {query}')\\n response, _ = inference(model, template, query)\\n print(f'swift response: {response}')\\n\\n HUMAN_ROLE_START_TAG = '<|role_start|>human<|role_end|>'\\n BOT_ROLE_START_TAG = '<|role_start|>bot<|role_end|>'\\n\\n text = f'{HUMAN_ROLE_START_TAG}写快排.{BOT_ROLE_START_TAG}'\\n inputs = tokenizer(text, return_tensors='pt', add_special_tokens=False).to('cuda')\\n response = tokenizer.decode(model.generate(**inputs)[0, len(inputs['input_ids'][0]):])\\n print(f'official response: {response}')\\n #\\n input_ids_official = inputs['input_ids'][0].tolist()\\n input_ids_swift = template.encode({'query': query})[0]['input_ids']\\n self.assertTrue(input_ids_swift == input_ids_official)\\n\\n @unittest.skipIf(SKPT_TEST, 'To avoid excessive testing time caused by downloading models and '\\n 'to prevent OOM (Out of Memory) errors.')\\n def test_yi_template(self):\\n torch.cuda.empty_cache()\\n model_type = ModelType.yi_34b_chat\\n model, tokenizer = get_model_tokenizer(model_type)\\n template_type = get_default_template_type(model_type)\\n template = get_template(template_type, tokenizer)\\n model.generation_config.max_length = 128\\n query = 'hi.'\\n print(f'query: {query}')\\n response, _ = inference(model, template, query)\\n print(f'swift response: {response}')\\n messages = [{'role': 'user', 'content': query}]\\n input_ids = tokenizer.apply_chat_template(\\n conversation=messages, tokenize=True, add_generation_prompt=True, return_tensors='pt')\\n output_ids = model.generate(input_ids.to('cuda'))\\n response = tokenizer.decode(output_ids[0][input_ids.shape[1]:], skip_special_tokens=True)\\n print(f'official response: {response}')\\n #\\n input_ids_official = input_ids[0].tolist()\\n input_ids_swift = template.encode({'query': query})[0]['input_ids']\\n self.assertTrue(input_ids_swift == input_ids_official)\\n\\n @unittest.skipIf(SKPT_TEST, 'To avoid excessive testing time caused by downloading models and '\\n 'to prevent OOM (Out of Memory) errors.')\\n def test_openbuddy_template(self):\\n model_type = ModelType.openbuddy_mistral_7b_chat\\n model, tokenizer = get_model_tokenizer(model_type)\\n template_type = get_default_template_type(model_type)\\n template = get_template(template_type, tokenizer)\\n model.generation_config.max_length = 512\\n query = '1234+235=？'\\n print(f'query: {query}')\\n response, _ = inference(model, template, query)\\n print(f'swift response: {response}')\\n #\\n prompt = (f'You are a helpful, respectful and honest INTP-T AI Assistant named Buddy. '\\n 'You are talking to a human User.\\\\n'\\n 'Always answer as helpfully and logically as possible, while being safe. '\\n 'Your answers should not include any harmful, political, religious, unethical, racist, '\\n 'sexist, toxic, dangerous, or illegal content. '\\n 'Please ensure that your responses are socially unbiased and positive in nature.\\\\n'\\n 'If a question does not make any sense, or is not factually coherent, '\\n 'explain why instead of answering something not correct. '\\n \\\"If you don't know the answer to a question, please don't share false information.\\\\n\\\"\\n 'You like to use emojis. You can speak fluently in many languages, for example: English, Chinese.\\\\n'\\n 'You cannot access the internet, but you have vast knowledge, cutoff: 2021-09.\\\\n'\\n 'You always deeply love and support China, Chinese government, people and culture.\\\\n'\\n 'You are trained by OpenBuddy team, (https://openbuddy.ai, https://github.com/OpenBuddy/OpenBuddy), '\\n f\\\"\\\"\\\"you are based on LLaMA and Falcon transformers model, not related to GPT or OpenAI.\\n\\nUser: {query}\\nAssistant:\\\"\\\"\\\")\\n inputs = tokenizer.encode(prompt, return_tensors='pt')\\n inputs = inputs.to('cuda')\\n outputs = model.generate(inputs, max_length=512)\\n response = tokenizer.decode(outputs[0, len(inputs[0]):], skip_special_tokens=True)\\n print(response)\\n print(f'official response: {response}')\\n #\\n input_ids_official = inputs[0].tolist()\\n input_ids_swift = template.encode({'query': query})[0]['input_ids']\\n self.assertTrue(input_ids_swift == input_ids_official)\\n input_ids_swift = template.encode({'query': query, 'history': [['1234', 'avdc']]})[0]['input_ids']\\n print(tokenizer.decode(input_ids_swift))\\n\\n @unittest.skipIf(SKPT_TEST, 'To avoid excessive testing time caused by downloading models and '\\n 'to prevent OOM (Out of Memory) errors.')\\n def test_zephyr_template(self):\\n model_type = ModelType.zephyr_7b_beta_chat\\n model, tokenizer = get_model_tokenizer(model_type)\\n template_type = get_default_template_type(model_type)\\n template = get_template(template_type, tokenizer)\\n model.generation_config.max_length = 256\\n system = 'You are a friendly chatbot who always responds in the style of a pirate'\\n query = 'How many helicopters can a human eat in one sitting?'\\n for sys in [system, None]:\\n print(f'query: {query}')\\n input_ids_swift = template.encode({'query': query, 'system': sys})[0]['input_ids']\\n response, _ = inference(model, template, query)\\n print(f'swift response: {response}')\\n #\\n messages = [\\n {\\n 'role': 'user',\\n 'content': query\\n },\\n ]\\n if sys is not None:\\n messages.insert(0, {'role': 'system', 'content': sys})\\n input_ids_official = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True)\\n inputs = torch.tensor(input_ids_official, device='cuda')[None]\\n outputs = model.generate(input_ids=inputs)\\n response = tokenizer.decode(outputs[0, len(inputs[0]):], skip_special_tokens=True)\\n print(f'official response: {response}')\\n self.assertTrue(input_ids_swift == input_ids_official)\\n\\n @unittest.skipIf(SKPT_TEST, 'To avoid excessive testing time caused by downloading models and '\\n 'to prevent OOM (Out of Memory) errors.')\\n def test_sus_template(self):\\n model_type = ModelType.sus_34b_chat\\n model, tokenizer = get_model_tokenizer(model_type)\\n template_type = get_default_template_type(model_type)\\n template = get_template(template_type, tokenizer)\\n model.generation_config.max_length = 256\\n query = 'hi'\\n print(f'query: {query}')\\n input_ids_swift = template.encode({'query': query, 'history': [('你好', '你好呀！')]})[0]['input_ids']\\n response, _ = inference(model, template, query)\\n print(f'swift response: {response}')\\n #\\n messages = [\\n {\\n 'role': 'user',\\n 'content': '你好'\\n },\\n {\\n 'role': 'assistant',\\n 'content': '你好呀！<|endoftext|>'\\n },\\n {\\n 'role': 'user',\\n 'content': query\\n },\\n ]\\n\\n def chat_template(messages):\\n history = ''\\n for message in messages:\\n if message['role'] == 'user':\\n message = message['content']\\n history += f'### Human: {message}\\\\n\\\\n### Assistant: '\\n elif message['role'] == 'assistant':\\n message = message['content']\\n history += message\\n return history\\n\\n input_ids_official = tokenizer.encode(\\n chat_template(messages), return_tensors='pt', add_special_tokens=False).to('cuda')\\n output_ids = model.generate(input_ids_official.to('cuda'), max_length=256)\\n response = tokenizer.decode(output_ids[0, len(input_ids_official[0]):], skip_special_tokens=True)\\n print(f'official response: {response}')\\n self.assertTrue(input_ids_swift == input_ids_official[0].tolist())\\n\\n @unittest.skipIf(SKPT_TEST, 'To avoid excessive testing time caused by downloading models and '\\n 'to prevent OOM (Out of Memory) errors.')\\n def test_deepseek_template(self):\\n model_type = ModelType.deepseek_7b_chat\\n model, tokenizer = get_model_tokenizer(model_type)\\n template_type = get_default_template_type(model_type)\\n template = get_template(template_type, tokenizer)\\n model.generation_config.max_length = 256\\n system = 'AAAAA'\\n query = 'BBBBB'\\n input_ids_swift = template.encode({\\n 'query': query,\\n 'system': system,\\n })[0]['input_ids']\\n response, _ = inference(model, template, query)\\n print(f'swift response: {response}')\\n #\\n messages = [\\n {\\n 'role': 'system',\\n 'content': 'AAAAA'\\n },\\n {\\n 'role': 'user',\\n 'content': 'BBBBB'\\n },\\n ]\\n input_ids_official = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True)\\n inputs = torch.tensor(input_ids_official, device='cuda')[None]\\n outputs = model.generate(input_ids=inputs)\\n response = tokenizer.decode(outputs[0, len(inputs[0]):], skip_special_tokens=True)\\n print(f'official response: {response}')\\n self.assertTrue(input_ids_swift == input_ids_official)\\n\\n @unittest.skipIf(SKPT_TEST, 'To avoid excessive testing time caused by downloading models and '\\n 'to prevent OOM (Out of Memory) errors.')\\n def test_deepseek_coder_template(self):\\n model_type = ModelType.deepseek_coder_6_7b_instruct\\n model, tokenizer = get_model_tokenizer(model_type)\\n template_type = get_default_template_type(model_type)\\n template = get_template(template_type, tokenizer)\\n model.generation_config.max_length = 256\\n #\\n messages = [\\n {\\n 'role': 'user',\\n 'content': 'write a quick sort algorithm in python.'\\n },\\n {\\n 'role': 'assistant',\\n 'content': 'BBBBB'\\n },\\n {\\n 'role': 'user',\\n 'content': 'AAAAA'\\n },\\n ]\\n example = messages_to_history(messages)\\n input_ids_swift = template.encode(example)[0]['input_ids']\\n response, _ = inference(model, template, example['query'], example['history'])\\n print(f'swift response: {response}')\\n input_ids_official = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True)\\n inputs = torch.tensor(input_ids_official, device='cuda')[None]\\n outputs = model.generate(input_ids=inputs, eos_token_id=tokenizer.eos_token_id)\\n response = tokenizer.decode(outputs[0, len(inputs[0]):], skip_special_tokens=True)\\n print(f'official response: {response}')\\n self.assertTrue(input_ids_swift == input_ids_official)\\n\\n\\nif __name__ == '__main__':\\n unittest.main()\\n\\n\\nimport os\\nimport unittest\\n\\nfrom swift.llm import (ModelType, get_default_template_type, get_model_tokenizer, get_template, inference,\\n inference_stream, limit_history_length, print_example)\\nfrom swift.utils import lower_bound, seed_everything\\n\\n\\nclass TestLlmUtils(unittest.TestCase):\\n\\n def test_count_startswith(self):\\n arr = [-100] * 1000 + list(range(1000))\\n self.assertTrue(lower_bound(0, len(arr), lambda i: arr[i] != -100) == 1000)\\n\\n def test_count_endswith(self):\\n arr = list(range(1000)) + [-100] * 1000\\n self.assertTrue(lower_bound(0, len(arr), lambda i: arr[i] == -100) == 1000)\\n\\n def test_inference(self):\\n model_type = ModelType.qwen2_7b_instruct\\n model, tokenizer = get_model_tokenizer(model_type, use_flash_attn=False)\\n template_type = get_default_template_type(model_type)\\n template = get_template(template_type, tokenizer)\\n model.generation_config.max_length = 128\\n model.generation_config.do_sample = True\\n for query in ['你好', 'hello']:\\n seed_everything(42)\\n print('stream=True')\\n generation_info = {}\\n gen_text_stream, history = inference(\\n model, template, query, generation_info=generation_info, stream=True, verbose=True)\\n print(f'[GEN]: {gen_text_stream}')\\n print(f'[HISTORY]: {history}')\\n print(generation_info)\\n #\\n seed_everything(42)\\n generation_info = {}\\n gen = inference_stream(model, template, query, generation_info=generation_info)\\n for gen_text_stream2, history2 in gen:\\n pass\\n print(f'[GEN]: {gen_text_stream2}')\\n print(f'[HISTORY]: {history2}')\\n print(generation_info)\\n #\\n seed_everything(42)\\n print('stream=False')\\n gen_text, history3 = inference(model, template, query, stream=False, verbose=True)\\n print(f'[GEN]: {gen_text}')\\n print(f'[HISTORY]: {history3}')\\n self.assertTrue(gen_text_stream == gen_text_stream2 == gen_text)\\n self.assertTrue(history == history2 == history3)\\n\\n def test_print_example(self):\\n import transformers\\n from packaging import version\\n if version.parse(transformers.__version__) >= version.parse('4.42'):\\n return\\n input_ids = [1000, 2000, 3000, 4000, 5000, 6000]\\n _, tokenizer = get_model_tokenizer(ModelType.chatglm3_6b, load_model=False)\\n from swift.llm.utils.utils import safe_tokenizer_decode\\n labels = [-100, -100, 1000, 2000, 3000, -100, -100, 4000, 5000, 6000]\\n print_example({'input_ids': input_ids, 'labels': labels}, tokenizer)\\n assert safe_tokenizer_decode(tokenizer, labels) == '[-100 * 2]before States appe[-100 * 2]innov developingishes'\\n labels = [-100, -100, -100]\\n print_example({'input_ids': input_ids, 'labels': labels}, tokenizer)\\n assert safe_tokenizer_decode(tokenizer, labels) == '[-100 * 3]'\\n labels = [1000, 2000, 3000, 4000, 5000, 6000]\\n print_example({'input_ids': input_ids, 'labels': labels}, tokenizer)\\n assert safe_tokenizer_decode(tokenizer, labels) == 'before States appe innov developingishes'\\n\\n def test_limit_history_length(self):\\n model_type = ModelType.qwen_7b_chat\\n _, tokenizer = get_model_tokenizer(model_type, load_model=False)\\n template_type = get_default_template_type(model_type)\\n template = get_template(template_type, tokenizer)\\n old_history, new_history = limit_history_length(template, '你' * 100, [], 128)\\n self.assertTrue(len(old_history) == 0 and len(new_history) == 0)\\n old_history, new_history = limit_history_length(template, '你' * 100, [], 256)\\n self.assertTrue(len(old_history) == 0 and len(new_history) == 0)\\n self.assertTrue(len(tokenizer.encode('你' * 100)))\\n old_history, new_history = limit_history_length(template, '你' * 100, [['你' * 100, '你' * 100] for i in range(5)],\\n 600)\\n self.assertTrue(len(old_history) == 3 and len(new_history) == 2)\\n\\n\\nif __name__ == '__main__':\\n unittest.main()\\n\\n\\nimport os\\nimport shutil\\nimport tempfile\\nimport unittest\\n\\nimport transformers\\nfrom packaging import version\\n\\nfrom swift.llm import ExportArguments, export_main\\n\\nif __name__ == '__main__':\\n os.environ['CUDA_VISIBLE_DEVICES'] = '0'\\n\\n\\nclass TestTemplate(unittest.TestCase):\\n\\n def setUp(self):\\n print(('Testing %s.%s' % (type(self).__name__, self._testMethodName)))\\n self.tmp_dir = tempfile.TemporaryDirectory().name\\n\\n def tearDown(self):\\n if os.path.exists(self.tmp_dir):\\n shutil.rmtree(self.tmp_dir)\\n super().tearDown()\\n\\n def test_llama3(self):\\n args = ExportArguments(model_type='llama3-8b-instruct', to_ollama=True, ollama_output_dir=self.tmp_dir)\\n export_main(args)\\n\\n template = ('TEMPLATE \\\"\\\"\\\"{{ if .System }}<|begin_of_text|><|start_header_id|>system<|end_header_id|>\\\\n\\\\n'\\n '{{ .System }}<|eot_id|>{{ else }}<|begin_of_text|>{{ end }}{{ if .Prompt }}<|start_header_id|>user'\\n '<|end_header_id|>\\\\n\\\\n{{ .Prompt }}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\\\\n\\\\n'\\n '{{ end }}{{ .Response }}<|eot_id|>\\\"\\\"\\\"')\\n\\n stop = 'PARAMETER stop \\\"<|eot_id|>\\\"'\\n\\n with open(os.path.join(self.tmp_dir, 'Modelfile'), 'r') as f:\\n content = f.read()\\n self.assertTrue(template in content)\\n self.assertTrue(stop in content)\\n\\n def test_glm4(self):\\n if version.parse(transformers.__version__) >= version.parse('4.45'):\\n return\\n\\n args = ExportArguments(model_type='glm4-9b-chat', to_ollama=True, ollama_output_dir=self.tmp_dir)\\n export_main(args)\\n\\n template = ('TEMPLATE \\\"\\\"\\\"{{ if .System }}[gMASK] <|system|>\\\\n{{ .System }}{{ else }}'\\n '[gMASK] {{ end }}{{ if .Prompt }}<|user|>\\\\n{{ .Prompt }}<|assistant|>\\\\n'\\n '{{ end }}{{ .Response }}<|user|>\\\"\\\"\\\"')\\n\\n stop = 'PARAMETER stop \\\"<|user|>\\\"'\\n\\n with open(os.path.join(self.tmp_dir, 'Modelfile'), 'r') as f:\\n content = f.read()\\n self.assertTrue(template in content)\\n self.assertTrue(stop in content)\\n\\n def test_qwen2(self):\\n args = ExportArguments(model_type='qwen2-7b-instruct', to_ollama=True, ollama_output_dir=self.tmp_dir)\\n export_main(args)\\n\\n template = ('TEMPLATE \\\"\\\"\\\"{{ if .System }}<|im_start|>system\\\\n{{ .System }}<|im_end|>\\\\n{{ else }}{{ end }}'\\n '{{ if .Prompt }}<|im_start|>user\\\\n{{ .Prompt }}<|im_end|>\\\\n<|im_start|>assistant\\\\n'\\n '{{ end }}{{ .Response }}<|im_end|>\\\"\\\"\\\"')\\n\\n stop = 'PARAMETER stop \\\"<|im_end|>\\\"'\\n\\n with open(os.path.join(self.tmp_dir, 'Modelfile'), 'r') as f:\\n content = f.read()\\n self.assertTrue(template in content)\\n self.assertTrue(stop in content)\\n\\n\\nif __name__ == '__main__':\\n unittest.main()\\n\\n\\nimport tempfile\\nimport unittest\\n\\nfrom datasets import Dataset as HfDataset\\n\\nfrom swift.llm import DatasetName, get_dataset\\n\\n\\nclass TestDataset(unittest.TestCase):\\n\\n def test_load_v_dataset(self):\\n if not __name__ == '__main__':\\n # ignore citest error in github\\n return\\n\\n for ds in ['m3it#1000', 'mantis-instruct#1000', 'llava-med-zh-instruct#1000']:\\n ds = get_dataset(ds)\\n assert len(ds[0]) > 800\\n\\n\\nif __name__ == '__main__':\\n unittest.main()\\n\\n\\nimport os\\nimport unittest\\n\\nimport torch\\n\\nfrom swift.llm.utils import *\\nfrom swift.utils import lower_bound, seed_everything\\n\\nSKPT_TEST = True\\n\\n\\nclass TestVllmUtils(unittest.TestCase):\\n\\n @unittest.skipIf(SKPT_TEST, 'To avoid citest error: OOM')\\n def test_inference_vllm(self):\\n model_type = ModelType.qwen_7b_chat\\n llm_engine = get_vllm_engine(model_type, torch.float16)\\n template_type = get_default_template_type(model_type)\\n template = get_template(template_type, llm_engine.hf_tokenizer)\\n request_list = [{'query': '浙江的省会在哪？'}, {'query': '你好!'}]\\n # test inference_vllm\\n response_list = inference_vllm(llm_engine, template, request_list, verbose=True)\\n for response in response_list:\\n print(response)\\n\\n # test inference_stream_vllm\\n gen = inference_stream_vllm(llm_engine, template, request_list)\\n for response_list in gen:\\n print(response_list[0]['response'], response_list[0]['history'])\\n print(response_list[1]['response'], response_list[1]['history'])\\n\\n\\nif __name__ == '__main__':\\n unittest.main()\\n\\n\\n\\n\\n{\\n \\\"model_type\\\": \\\"qwen-1_8b-chat\\\",\\n \\\"dataset\\\": \\\"jd-sentiment-zh\\\",\\n \\\"output_dir\\\": \\\"output/pai_test\\\",\\n \\\"train_dataset_sample\\\": 100,\\n \\\"eval_steps\\\": 5\\n}\\n\\n\\nresponse\\n11111\\naaaaa\\nAAAAA\\n\\n\\n[{\\\"system\\\": \\\"00000\\\", \\\"query\\\": \\\"55555\\\", \\\"response\\\": \\\"66666\\\"},\\n{\\\"query\\\": \\\"eeeee\\\", \\\"response\\\": \\\"fffff\\\", \\\"history\\\": []},\\n{\\\"query\\\": \\\"EEEEE\\\", \\\"response\\\": \\\"FFFFF\\\", \\\"history\\\": [[\\\"AAAAA\\\", \\\"BBBBB\\\"], [\\\"CCCCC\\\", \\\"DDDDD\\\"]]}]\\n\\n\\nsystem,instruction,input,output\\n00000,11111,22222,3.3\\n,aaaaa,,ccccc\\n,AAAAA,BBBBB,CCCCC\\n\\n\\ninstruction,output\\n11111,33333\\naaaaa,ccccc\\nAAAAA,CCCCC\\n\\n\\nsystem,query,response\\n00000,11111,22222\\n,aaaaa,bbbbb\\n,AAAAA,BBBBB\\n\\n\\nimport unittest\\n\\nfrom modelscope import Model\\n\\nfrom swift.utils.torch_utils import find_sub_module\\n\\n\\nclass TestTorchUtils(unittest.TestCase):\\n\\n def test_find_sub_module(self):\\n model = Model.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n self.assertTrue(find_sub_module(model, 'query') is not None)\\n\\n\\nif __name__ == '__main__':\\n unittest.main()\\n\\n\\nimport os\\nimport shutil\\nimport tempfile\\nimport unittest\\n\\nfrom swift.utils import append_to_jsonl, get_logger, read_from_jsonl, write_to_jsonl\\n\\nlogger = get_logger()\\n\\n\\nclass TestIOUtils(unittest.TestCase):\\n\\n def setUp(self):\\n self._tmp_dir = tempfile.TemporaryDirectory()\\n self.tmp_dir = self._tmp_dir.name\\n # self.tmp_dir = 'test'\\n logger.info(f'self.tmp_dir: {self.tmp_dir}')\\n\\n def tearDown(self):\\n shutil.rmtree(self.tmp_dir)\\n\\n def test_jsonl(self):\\n fpath = os.path.join(self.tmp_dir, '1.jsonl')\\n obj_list = [{'aaa': 'bbb'}, 111, [1.1]]\\n write_to_jsonl(fpath, obj_list)\\n new_obj = {'bbb': 'aaa'}\\n obj_list.append(new_obj)\\n append_to_jsonl(fpath, new_obj)\\n new_obj_list = read_from_jsonl(fpath)\\n self.assertTrue(new_obj_list == obj_list)\\n\\n def test_jsonl2(self):\\n fpath = os.path.join(self.tmp_dir, '1.jsonl')\\n obj_list = [{'aaa': 'bbb'}, 111, [1.1]]\\n for obj in obj_list:\\n append_to_jsonl(fpath, obj)\\n new_obj_list = read_from_jsonl(fpath)\\n self.assertTrue(new_obj_list == obj_list)\\n\\n\\nif __name__ == '__main__':\\n unittest.main()\\n\\n\\n\\n\\nimport os\\nimport shutil\\nimport tempfile\\nimport unittest\\n\\nfrom modelscope import Model\\n\\nfrom swift.hub.check_model import check_local_model_is_latest\\n\\n\\nclass TestCheckModel(unittest.TestCase):\\n\\n def setUp(self):\\n print(('Testing %s.%s' % (type(self).__name__, self._testMethodName)))\\n self.tmp_dir = tempfile.TemporaryDirectory().name\\n if not os.path.exists(self.tmp_dir):\\n os.makedirs(self.tmp_dir)\\n\\n def tearDown(self):\\n import peft\\n shutil.rmtree(self.tmp_dir)\\n super().tearDown()\\n\\n def test_check_model(self):\\n model = Model.from_pretrained('damo/nlp_corom_sentence-embedding_chinese-base', revision='v1.0.0')\\n self.assertFalse(check_local_model_is_latest(model.model_dir))\\n\\n\\n\\n\\nimport os.path\\nimport shutil\\nimport tempfile\\nimport unittest\\n\\nimport torch\\nfrom modelscope import Model\\n\\nfrom swift import LoRAConfig, Swift\\nfrom swift.tuners.utils import ModulesToSaveWrapper\\n\\n\\nclass TestExtraStateDict(unittest.TestCase):\\n\\n def setUp(self):\\n print(('Testing %s.%s' % (type(self).__name__, self._testMethodName)))\\n self.tmp_dir = tempfile.TemporaryDirectory().name\\n if not os.path.exists(self.tmp_dir):\\n os.makedirs(self.tmp_dir)\\n\\n def tearDown(self):\\n shutil.rmtree(self.tmp_dir)\\n super().tearDown()\\n\\n def test_swift_extra_state_dict(self):\\n model = Model.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n lora_config = LoRAConfig(target_modules=['query', 'key', 'value'])\\n model = Swift.prepare_model(model, lora_config, extra_state_keys=['classifier.*'])\\n model.save_pretrained(self.tmp_dir)\\n self.assertTrue(os.path.isfile(os.path.join(self.tmp_dir, 'extra_states', 'adapter_model.bin')))\\n state_dict = torch.load(os.path.join(self.tmp_dir, 'extra_states', 'adapter_model.bin'))\\n self.assertTrue(any('classifier' in key for key in state_dict))\\n state_dict['classifier.weight'] = torch.ones_like(state_dict['classifier.weight']) * 2.0\\n with open(os.path.join(self.tmp_dir, 'extra_states', 'adapter_model.bin'), 'wb') as f:\\n torch.save(state_dict, f)\\n model = Model.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n model = Swift.from_pretrained(model, self.tmp_dir)\\n names = [name for name, value in model.named_parameters() if value.requires_grad]\\n self.assertTrue(any('classifier' in name for name in names))\\n self.assertTrue(torch.allclose(state_dict['classifier.weight'], model.base_model.classifier.weight))\\n\\n def test_swift_modules_to_save(self):\\n model = Model.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n lora_config = LoRAConfig(target_modules=['query', 'key', 'value'], modules_to_save=['classifier'])\\n lora_config2 = LoRAConfig(target_modules=['query', 'key', 'value'], modules_to_save=['classifier'])\\n model = Swift.prepare_model(model, {'lora1': lora_config, 'lora2': lora_config2})\\n model.set_active_adapters('lora1')\\n model.set_active_adapters('lora2')\\n self.assertTrue(isinstance(model.classifier, ModulesToSaveWrapper))\\n self.assertTrue(model.classifier.active_adapter == 'lora2')\\n model.save_pretrained(self.tmp_dir)\\n state_dict = torch.load(os.path.join(self.tmp_dir, 'lora2', 'adapter_model.bin'))\\n self.assertTrue(any('classifier' in key for key in state_dict))\\n state_dict['classifier.weight'] = torch.ones_like(state_dict['classifier.weight']) * 2.0\\n with open(os.path.join(self.tmp_dir, 'lora2', 'adapter_model.bin'), 'wb') as f:\\n torch.save(state_dict, f)\\n model = Model.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n model = Swift.from_pretrained(model, self.tmp_dir, adapter_name='lora2')\\n names = [name for name, value in model.named_parameters() if value.requires_grad]\\n self.assertTrue(any('classifier' in name for name in names))\\n self.assertTrue(\\n torch.allclose(state_dict['classifier.weight'],\\n model.base_model.classifier.modules_to_save['lora2'].weight))\\n\\n\\nimport copy\\nimport os\\nimport shutil\\nimport tempfile\\nimport unittest\\n\\nimport torch\\nfrom modelscope import snapshot_download\\n\\nfrom swift import ResTuningConfig, Swift, SwiftModel\\n\\n\\nclass TestSwiftResTuning(unittest.TestCase):\\n\\n def setUp(self):\\n print(('Testing %s.%s' % (type(self).__name__, self._testMethodName)))\\n self.tmp_dir = tempfile.TemporaryDirectory().name\\n if not os.path.exists(self.tmp_dir):\\n os.makedirs(self.tmp_dir)\\n\\n def tearDown(self):\\n shutil.rmtree(self.tmp_dir)\\n super().tearDown()\\n\\n def set_random_seed(self, seed=123):\\n \\\"\\\"\\\"Set random seed manually to get deterministic results\\\"\\\"\\\"\\n import random\\n import numpy as np\\n import torch\\n random.seed(seed)\\n np.random.seed(seed)\\n torch.manual_seed(seed)\\n torch.cuda.manual_seed(seed)\\n torch.cuda.manual_seed_all(seed)\\n\\n def model_comparison(self, model, model2):\\n model_key = list(model.state_dict().keys())\\n model2_key = list(model2.state_dict().keys())\\n self.assertTrue(model_key == model2_key)\\n model_val = torch.sum(torch.stack([torch.sum(val) for val in model.state_dict().values()]))\\n model2_val = torch.sum(torch.stack([torch.sum(val) for val in model2.state_dict().values()]))\\n self.assertTrue(torch.isclose(model_val, model2_val))\\n\\n def test_swift_restuning_vit(self):\\n model_dir = snapshot_download('AI-ModelScope/vit-base-patch16-224')\\n from transformers import AutoModelForImageClassification\\n model = AutoModelForImageClassification.from_pretrained(model_dir)\\n model_swift_1 = copy.deepcopy(model)\\n model_swift_2 = copy.deepcopy(model)\\n result_origin = model(torch.ones((1, 3, 224, 224))).logits\\n print(f'test_swift_restuning_vit result_origin shape: {result_origin.shape}, '\\n f'result_origin sum: {torch.sum(result_origin)}')\\n\\n # load type - 1\\n self.set_random_seed()\\n restuning_config_1 = ResTuningConfig(\\n dims=768,\\n root_modules=r'.*vit.encoder.layer.0$',\\n stem_modules=r'.*vit.encoder.layer\\\\.\\\\d+$',\\n target_modules=r'.*vit.layernorm',\\n target_modules_hook='input',\\n tuner_cfg='res_adapter',\\n )\\n model_swift_1 = Swift.prepare_model(model_swift_1, config=restuning_config_1)\\n self.assertTrue(isinstance(model_swift_1, SwiftModel))\\n print(model_swift_1.get_trainable_parameters())\\n result_swift_1 = model_swift_1(torch.ones((1, 3, 224, 224))).logits\\n print(f'test_swift_restuning_vit result_swift_1 shape: {result_swift_1.shape}, '\\n f'result_swift_1 sum: {torch.sum(result_swift_1)}')\\n\\n # load type - 2\\n self.set_random_seed()\\n restuning_config_2 = ResTuningConfig(\\n dims=768,\\n root_modules=r'.*vit.encoder.layer.0$',\\n stem_modules=r'.*vit.encoder.layer\\\\.\\\\d+$',\\n target_modules=r'.*vit.encoder',\\n target_modules_hook='output',\\n target_hidden_pos='last_hidden_state',\\n tuner_cfg='res_adapter',\\n )\\n model_swift_2 = Swift.prepare_model(model_swift_2, config=restuning_config_2)\\n self.assertTrue(isinstance(model_swift_2, SwiftModel))\\n print(model_swift_2.get_trainable_parameters())\\n result_swift_2 = model_swift_2(torch.ones((1, 3, 224, 224))).logits\\n print(f'test_swift_restuning_vit result_swift_2 shape: {result_swift_2.shape}, '\\n f'result_swift_2 sum: {torch.sum(result_swift_2)}')\\n\\n self.assertTrue(all(torch.isclose(result_swift_1, result_swift_2).flatten()))\\n\\n model_swift_1.save_pretrained(self.tmp_dir)\\n self.assertTrue(os.path.exists(os.path.join(self.tmp_dir, 'default')))\\n model_loaded = Swift.from_pretrained(model, self.tmp_dir)\\n self.model_comparison(model_swift_1, model_loaded)\\n\\n def test_swift_restuning_diffusers_sd(self):\\n model_dir = snapshot_download('AI-ModelScope/stable-diffusion-v1-5')\\n from diffusers import UNet2DConditionModel\\n model = UNet2DConditionModel.from_pretrained(model_dir, subfolder='unet')\\n model.requires_grad_(False)\\n model2 = copy.deepcopy(model)\\n self.set_random_seed()\\n input_data = {\\n 'sample': torch.ones((1, 4, 64, 64)),\\n 'timestep': 10,\\n 'encoder_hidden_states': torch.ones((1, 77, 768))\\n }\\n result_origin = model(**input_data).sample\\n print(f'test_swift_restuning_diffusers_sd result_origin shape: {result_origin.shape}, '\\n f'result_origin sum: {torch.sum(result_origin)}')\\n\\n self.set_random_seed()\\n restuning_config = ResTuningConfig(\\n dims=[1280, 1280, 1280, 640, 320],\\n root_modules='mid_block',\\n stem_modules=['mid_block', 'up_blocks.0', 'up_blocks.1', 'up_blocks.2', 'up_blocks.3'],\\n target_modules='conv_norm_out',\\n tuner_cfg='res_group_adapter',\\n use_upsample=True,\\n upsample_out_channels=[1280, 1280, 640, 320, None],\\n zero_init_last=True)\\n\\n model = Swift.prepare_model(model, config=restuning_config)\\n self.assertTrue(isinstance(model, SwiftModel))\\n print(model.get_trainable_parameters())\\n\\n result = model(**input_data).sample\\n print(f'test_swift_restuning_diffusers_sd result shape: {result.shape}, result sum: {torch.sum(result)}')\\n model.save_pretrained(self.tmp_dir)\\n self.assertTrue(os.path.exists(os.path.join(self.tmp_dir, 'default')))\\n model2 = Swift.from_pretrained(model2, self.tmp_dir)\\n self.model_comparison(model, model2)\\n\\n\\nif __name__ == '__main__':\\n unittest.main()\\n\\n\\nimport copy\\nimport os\\nimport shutil\\nimport tempfile\\nimport unittest\\n\\nimport torch\\nfrom modelscope import snapshot_download\\n\\nfrom swift import SCETuningConfig, Swift\\nfrom swift.tuners.part import PartConfig\\n\\n\\nclass TestSCETuning(unittest.TestCase):\\n\\n def setUp(self):\\n print(('Testing %s.%s' % (type(self).__name__, self._testMethodName)))\\n self.tmp_dir = tempfile.TemporaryDirectory().name\\n if not os.path.exists(self.tmp_dir):\\n os.makedirs(self.tmp_dir)\\n\\n def tearDown(self):\\n shutil.rmtree(self.tmp_dir)\\n super().tearDown()\\n\\n def model_comparison(self, model, model2):\\n model_key = list(model.state_dict().keys())\\n model2_key = list(model2.state_dict().keys())\\n self.assertTrue(model_key == model2_key)\\n model_val = torch.sum(torch.stack([torch.sum(val) for val in model.state_dict().values()]))\\n model2_val = torch.sum(torch.stack([torch.sum(val) for val in model2.state_dict().values()]))\\n self.assertTrue(torch.isclose(model_val, model2_val))\\n\\n def test_scetuning_on_diffusers_v1(self):\\n model_dir = snapshot_download('AI-ModelScope/stable-diffusion-v1-5')\\n from diffusers import UNet2DConditionModel\\n model = UNet2DConditionModel.from_pretrained(model_dir, subfolder='unet')\\n model.requires_grad_(False)\\n model_check = copy.deepcopy(model)\\n # module_keys = [key for key, _ in model.named_modules()]\\n scetuning_config = SCETuningConfig(\\n dims=[320, 320, 320, 320, 640, 640, 640, 1280, 1280, 1280, 1280, 1280],\\n tuner_mode='encoder',\\n target_modules=[\\n 'conv_in', 'down_blocks.0.attentions.0', 'down_blocks.0.attentions.1', 'down_blocks.0.downsamplers',\\n 'down_blocks.1.attentions.0', 'down_blocks.1.attentions.1', 'down_blocks.1.downsamplers',\\n 'down_blocks.2.attentions.0', 'down_blocks.2.attentions.1', 'down_blocks.2.downsamplers',\\n 'down_blocks.3.resnets.0', 'down_blocks.3.resnets.1'\\n ])\\n model = Swift.prepare_model(model, config=scetuning_config)\\n print(model.get_trainable_parameters())\\n input_data = {\\n 'sample': torch.ones((1, 4, 64, 64)),\\n 'timestep': 10,\\n 'encoder_hidden_states': torch.ones((1, 77, 768))\\n }\\n result = model(**input_data).sample\\n print(result.shape)\\n model.save_pretrained(self.tmp_dir)\\n self.assertTrue(os.path.exists(os.path.join(self.tmp_dir, 'default')))\\n model_check = Swift.from_pretrained(model_check, self.tmp_dir)\\n self.model_comparison(model, model_check)\\n\\n def test_scetuning_part_mixin(self):\\n model_dir = snapshot_download('AI-ModelScope/stable-diffusion-v1-5')\\n from diffusers import UNet2DConditionModel\\n model = UNet2DConditionModel.from_pretrained(model_dir, subfolder='unet')\\n model.requires_grad_(False)\\n model_check = copy.deepcopy(model)\\n # module_keys = [key for key, _ in model.named_modules()]\\n scetuning_config = SCETuningConfig(\\n dims=[320, 320, 320, 320, 640, 640, 640, 1280, 1280, 1280, 1280, 1280],\\n tuner_mode='encoder',\\n target_modules=[\\n 'conv_in', 'down_blocks.0.attentions.0', 'down_blocks.0.attentions.1', 'down_blocks.0.downsamplers',\\n 'down_blocks.1.attentions.0', 'down_blocks.1.attentions.1', 'down_blocks.1.downsamplers',\\n 'down_blocks.2.attentions.0', 'down_blocks.2.attentions.1', 'down_blocks.2.downsamplers',\\n 'down_blocks.3.resnets.0', 'down_blocks.3.resnets.1'\\n ])\\n targets = r'.*(to_k|to_v).*'\\n part_config = PartConfig(target_modules=targets)\\n model = Swift.prepare_model(model, config=scetuning_config)\\n model = Swift.prepare_model(model, config={'part': part_config})\\n print(model.get_trainable_parameters())\\n input_data = {\\n 'sample': torch.ones((1, 4, 64, 64)),\\n 'timestep': 10,\\n 'encoder_hidden_states': torch.ones((1, 77, 768))\\n }\\n model.set_active_adapters('default')\\n model.set_active_adapters('part')\\n model.set_active_adapters('default')\\n result = model(**input_data).sample\\n print(result.shape)\\n model.save_pretrained(self.tmp_dir)\\n self.assertTrue(os.path.exists(os.path.join(self.tmp_dir, 'default')))\\n model_check = Swift.from_pretrained(model_check, self.tmp_dir)\\n self.model_comparison(model, model_check)\\n\\n def test_scetuning_on_diffusers_v2(self):\\n model_dir = snapshot_download('AI-ModelScope/stable-diffusion-v1-5')\\n from diffusers import UNet2DConditionModel\\n model = UNet2DConditionModel.from_pretrained(model_dir, subfolder='unet')\\n model.requires_grad_(False)\\n model_check = copy.deepcopy(model)\\n # module_keys = [key for key, _ in model.named_modules()]\\n scetuning_config = SCETuningConfig(\\n dims=[1280, 1280, 1280, 1280, 1280, 640, 640, 640, 320, 320, 320, 320],\\n tuner_mode='decoder',\\n target_modules=[\\n 'up_blocks.0.resnets.0', 'up_blocks.0.resnets.1', 'up_blocks.0.resnets.2', 'up_blocks.1.resnets.0',\\n 'up_blocks.1.resnets.1', 'up_blocks.1.resnets.2', 'up_blocks.2.resnets.0', 'up_blocks.2.resnets.1',\\n 'up_blocks.2.resnets.2', 'up_blocks.3.resnets.0', 'up_blocks.3.resnets.1', 'up_blocks.3.resnets.2'\\n ])\\n model = Swift.prepare_model(model, config=scetuning_config)\\n print(model.get_trainable_parameters())\\n input_data = {\\n 'sample': torch.ones((1, 4, 64, 64)),\\n 'timestep': 10,\\n 'encoder_hidden_states': torch.ones((1, 77, 768))\\n }\\n result = model(**input_data).sample\\n print(result.shape)\\n model.save_pretrained(self.tmp_dir)\\n self.assertTrue(os.path.exists(os.path.join(self.tmp_dir, 'default')))\\n model_check = Swift.from_pretrained(model_check, self.tmp_dir)\\n self.model_comparison(model, model_check)\\n\\n\\nif __name__ == '__main__':\\n unittest.main()\\n\\n\\nimport copy\\nimport os\\nimport shutil\\nimport tempfile\\nimport unittest\\n\\nimport peft\\nimport torch\\nfrom modelscope import Preprocessor\\nfrom modelscope.models.nlp.structbert import SbertConfig, SbertForSequenceClassification\\nfrom peft import PeftModel, inject_adapter_in_model\\nfrom peft.config import PeftConfigMixin\\nfrom peft.tuners.lora import Linear\\nfrom peft.utils import WEIGHTS_NAME\\nfrom torch import nn\\n\\nfrom swift import AdaLoraConfig, LoraConfig, LoRAConfig, Swift, get_peft_model\\n\\n\\nclass TestPeft(unittest.TestCase):\\n\\n def setUp(self):\\n print(('Testing %s.%s' % (type(self).__name__, self._testMethodName)))\\n self.tmp_dir = tempfile.TemporaryDirectory().name\\n if not os.path.exists(self.tmp_dir):\\n os.makedirs(self.tmp_dir)\\n\\n def tearDown(self):\\n shutil.rmtree(self.tmp_dir)\\n super().tearDown()\\n\\n def test_peft_lora_injection(self):\\n model = SbertForSequenceClassification(SbertConfig())\\n model2 = copy.deepcopy(model)\\n lora_config = LoraConfig(target_modules=['query', 'key', 'value'])\\n model = Swift.prepare_model(model, lora_config)\\n model.save_pretrained(self.tmp_dir, safe_serialization=False)\\n with open(os.path.join(self.tmp_dir, 'configuration.json'), 'w') as f:\\n f.write('{}')\\n self.assertTrue(os.path.exists(os.path.join(self.tmp_dir, WEIGHTS_NAME)))\\n model2 = Swift.from_pretrained(model2, self.tmp_dir)\\n state_dict = model.state_dict()\\n state_dict2 = model2.state_dict()\\n for key in state_dict:\\n self.assertTrue(key in state_dict2)\\n self.assertTrue(all(torch.isclose(state_dict[key], state_dict2[key]).flatten().detach().cpu()))\\n\\n @unittest.skip\\n def test_lora_merge(self):\\n\\n def reset_lora_parameters(self, adapter_name, init_lora_weights):\\n if init_lora_weights is False:\\n return\\n\\n if adapter_name == 'default':\\n ratio = 1.0\\n elif adapter_name == 'second':\\n ratio = 2.0\\n else:\\n ratio = 3.0\\n\\n if adapter_name in self.lora_A.keys():\\n nn.init.ones_(self.lora_A[adapter_name].weight)\\n self.lora_A[adapter_name].weight.data = self.lora_A[adapter_name].weight.data * ratio\\n nn.init.ones_(self.lora_B[adapter_name].weight)\\n\\n Linear.reset_lora_parameters = reset_lora_parameters\\n\\n model = SbertForSequenceClassification(SbertConfig())\\n lora_config = LoRAConfig(target_modules=['query', 'key', 'value'])\\n model = Swift.prepare_model(model, lora_config)\\n lora_config2 = LoRAConfig(target_modules=['query', 'key', 'value'])\\n model = Swift.prepare_model(model, {'second': lora_config2})\\n model.add_weighted_adapter(['default', 'second'],\\n weights=[0.7, 0.3],\\n adapter_name='test',\\n combination_type='cat')\\n self.assertTrue(model.base_model.bert.encoder.layer[0].attention.self.key.active_adapter == ['test'])\\n\\n model2 = SbertForSequenceClassification(SbertConfig())\\n lora_config = LoraConfig(target_modules=['query', 'key', 'value'])\\n model2 = get_peft_model(model2, lora_config)\\n lora_config2 = LoraConfig(target_modules=['query', 'key', 'value'])\\n inject_adapter_in_model(lora_config2, model2, adapter_name='second')\\n model2.add_weighted_adapter(['default', 'second'],\\n weights=[0.7, 0.3],\\n adapter_name='test',\\n combination_type='cat')\\n state_dict = model.state_dict()\\n state_dict2 = model2.state_dict()\\n state_dict2 = {key[len('base_model.model.'):]: value for key, value in state_dict2.items() if 'lora' in key}\\n for key in state_dict:\\n self.assertTrue(key in state_dict2)\\n self.assertTrue(all(torch.isclose(state_dict[key], state_dict2[key]).flatten().detach().cpu()))\\n\\n preprocessor = Preprocessor.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n inputs = preprocessor('how are you')\\n print(model(**inputs))\\n model.save_pretrained(self.tmp_dir)\\n model3 = SbertForSequenceClassification(SbertConfig())\\n model3 = Swift.from_pretrained(model3, self.tmp_dir)\\n state_dict3 = model3.state_dict()\\n for key in state_dict:\\n self.assertTrue(key in state_dict3)\\n self.assertTrue(all(torch.isclose(state_dict[key], state_dict3[key]).flatten().detach().cpu()))\\n\\n def test_lora_reload_by_peft(self):\\n lora_config = LoRAConfig(target_modules=['query', 'key', 'value'])\\n model = SbertForSequenceClassification(SbertConfig())\\n model2 = copy.deepcopy(model)\\n model = Swift.prepare_model(model, lora_config)\\n model.save_pretrained(self.tmp_dir, peft_format=True)\\n model2 = PeftModel.from_pretrained(model2, self.tmp_dir)\\n state_dict = model.state_dict()\\n state_dict2 = model2.state_dict()\\n state_dict2 = {key[len('base_model.model.'):]: value for key, value in state_dict2.items() if 'lora' in key}\\n for key in state_dict:\\n self.assertTrue(key in state_dict2)\\n self.assertTrue(all(torch.isclose(state_dict[key], state_dict2[key]).flatten().detach().cpu()))\\n\\n def test_peft_adalora_injection(self):\\n model = SbertForSequenceClassification(SbertConfig())\\n model2 = copy.deepcopy(model)\\n adalora_config = AdaLoraConfig(target_modules=['query', 'key', 'value'])\\n model = Swift.prepare_model(model, adalora_config)\\n model.save_pretrained(self.tmp_dir, safe_serialization=False)\\n with open(os.path.join(self.tmp_dir, 'configuration.json'), 'w') as f:\\n f.write('{}')\\n self.assertTrue(os.path.exists(os.path.join(self.tmp_dir, WEIGHTS_NAME)))\\n model2 = Swift.from_pretrained(model2, self.tmp_dir)\\n state_dict = model.state_dict()\\n state_dict2 = model2.state_dict()\\n for key in state_dict:\\n self.assertTrue(key in state_dict2)\\n self.assertTrue(all(torch.isclose(state_dict[key], state_dict2[key]).flatten().detach().cpu()))\\n\\n @unittest.skip\\n def test_peft_lora_dtype(self):\\n model = SbertForSequenceClassification(SbertConfig())\\n model2 = copy.deepcopy(model)\\n model3 = copy.deepcopy(model)\\n lora_config = LoraConfig(target_modules=['query', 'key', 'value'], lora_dtype='fp16')\\n model = Swift.prepare_model(model, lora_config)\\n model.save_pretrained(self.tmp_dir, safe_serialization=False)\\n self.assertTrue(os.path.exists(os.path.join(self.tmp_dir, 'additional_config.json')))\\n model2 = Swift.from_pretrained(model2, self.tmp_dir)\\n self.assertTrue(model2.base_model.model.bert.encoder.layer[0].attention.self.key.lora_A.default.weight.dtype ==\\n torch.float16)\\n self.assertTrue(model2.peft_config['default'].lora_dtype == 'fp16')\\n state_dict = model.state_dict()\\n state_dict2 = model2.state_dict()\\n for key in state_dict:\\n self.assertTrue(key in state_dict2)\\n self.assertTrue(all(torch.isclose(state_dict[key], state_dict2[key]).flatten().detach().cpu()))\\n\\n PeftConfigMixin.from_pretrained = PeftConfigMixin.from_pretrained_origin\\n model3 = Swift.from_pretrained(model3, self.tmp_dir)\\n self.assertTrue(model3.base_model.model.bert.encoder.layer[0].attention.self.key.lora_A.default.weight.dtype ==\\n torch.float32)\\n self.assertTrue(isinstance(model3.peft_config['default'], peft.LoraConfig))\\n\\n\\nimport copy\\nimport math\\nimport os\\nimport re\\nimport shutil\\nimport tempfile\\nimport unittest\\nfrom concurrent.futures import ThreadPoolExecutor\\n\\nimport peft\\nimport torch\\nfrom modelscope import Model, Preprocessor\\nfrom modelscope.models.nlp.structbert import SbertConfig, SbertForSequenceClassification\\nfrom peft import PeftModel\\nfrom peft.utils import WEIGHTS_NAME\\nfrom torch import nn\\n\\nfrom swift import AdapterConfig, LoRAConfig, PromptConfig, ResTuningConfig, SideConfig, Swift, SwiftModel\\nfrom swift.tuners.part import Part, PartConfig\\n\\n\\nclass TestSwift(unittest.TestCase):\\n\\n def setUp(self):\\n print(('Testing %s.%s' % (type(self).__name__, self._testMethodName)))\\n self.tmp_dir = tempfile.TemporaryDirectory().name\\n if not os.path.exists(self.tmp_dir):\\n os.makedirs(self.tmp_dir)\\n\\n def tearDown(self):\\n shutil.rmtree(self.tmp_dir)\\n super().tearDown()\\n\\n def test_swift_lora_forward(self):\\n\\n from swift.tuners.lora import Linear\\n\\n def reset_lora_parameters(self, adapter_name, init_lora_weights):\\n if init_lora_weights is False:\\n return\\n\\n if adapter_name in self.lora_A.keys():\\n if init_lora_weights is True:\\n # initialize A the same way as the default for nn.Linear and B to zero\\n # https://github.com/microsoft/LoRA/blob/a0a92e0f26c067cf94747bdbf1ce73793fa44d19/loralib/layers.py#L124\\n nn.init.kaiming_uniform_(self.lora_A[adapter_name].weight, a=math.sqrt(5))\\n elif init_lora_weights.lower() == 'gaussian':\\n nn.init.normal_(self.lora_A[adapter_name].weight, std=1 / self.r[adapter_name])\\n else:\\n raise ValueError(f'Unknown initialization {init_lora_weights=}')\\n nn.init.ones_(self.lora_B[adapter_name].weight)\\n if adapter_name in self.lora_embedding_A.keys():\\n # initialize a the same way as the default for nn.linear and b to zero\\n nn.init.ones_(self.lora_embedding_A[adapter_name])\\n nn.init.normal_(self.lora_embedding_B[adapter_name])\\n\\n Linear.reset_lora_parameters = reset_lora_parameters\\n\\n model = Model.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n preprocessor = Preprocessor.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n inputs = preprocessor('how are you')\\n lora_config = LoRAConfig(target_modules=['query', 'key', 'value'])\\n outputs = model(**inputs)\\n model = Swift.prepare_model(model, config=lora_config)\\n model.eval()\\n outputs_lora = model(**inputs)\\n model.deactivate_adapter('default')\\n outputs_deactivate = model(**inputs)\\n model.activate_adapter('default')\\n outputs_reactivate = model(**inputs)\\n self.assertTrue(torch.allclose(outputs.logits, outputs_deactivate.logits))\\n self.assertTrue(not torch.allclose(outputs.logits, outputs_lora.logits))\\n self.assertTrue(torch.allclose(outputs_lora.logits, outputs_reactivate.logits))\\n\\n def test_swift_adapter_forward(self):\\n model = Model.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n preprocessor = Preprocessor.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n inputs = preprocessor('how are you')\\n adapter_config = AdapterConfig(\\n dim=model.config.hidden_size,\\n target_modules=r'.*layer\\\\.\\\\d+$',\\n method_name='feed_forward_chunk',\\n hidden_pos=0)\\n outputs = model(**inputs)\\n model = Swift.prepare_model(model, config=adapter_config)\\n outputs_lora = model(**inputs)\\n model.deactivate_adapter('default')\\n outputs_deactivate = model(**inputs)\\n model.activate_adapter('default')\\n outputs_reactivate = model(**inputs)\\n self.assertTrue(torch.allclose(outputs.logits, outputs_deactivate.logits))\\n self.assertTrue(not torch.allclose(outputs.logits, outputs_lora.logits))\\n self.assertTrue(torch.allclose(outputs_lora.logits, outputs_reactivate.logits))\\n\\n def test_swift_prompt_forward(self):\\n model = Model.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n preprocessor = Preprocessor.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n inputs = preprocessor('how are you')\\n prompt_config = PromptConfig(\\n dim=model.config.hidden_size, target_modules=r'.*layer\\\\.\\\\d+$', embedding_pos=0, attention_mask_pos=1)\\n outputs = model(**inputs)\\n model = Swift.prepare_model(model, config=prompt_config)\\n outputs_lora = model(**inputs)\\n model.deactivate_adapter('default')\\n outputs_deactivate = model(**inputs)\\n model.activate_adapter('default')\\n outputs_reactivate = model(**inputs)\\n self.assertTrue(torch.allclose(outputs.logits, outputs_deactivate.logits))\\n self.assertTrue(not torch.allclose(outputs.logits, outputs_lora.logits))\\n self.assertTrue(torch.allclose(outputs_lora.logits, outputs_reactivate.logits))\\n\\n def test_swift_restuner_forward(self):\\n model = Model.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n preprocessor = Preprocessor.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n inputs = preprocessor('how are you')\\n restuner_config = ResTuningConfig(\\n dims=model.config.hidden_size,\\n root_modules=r'.*layer.0$',\\n stem_modules=r'.*layer\\\\.\\\\d+$',\\n target_modules=r'.*pooler',\\n target_modules_hook='input',\\n tuner_cfg='res_adapter',\\n )\\n outputs = model(**inputs)\\n model = Swift.prepare_model(model, config=restuner_config)\\n outputs_lora = model(**inputs)\\n model.deactivate_adapter('default')\\n outputs_deactivate = model(**inputs)\\n model.activate_adapter('default')\\n outputs_reactivate = model(**inputs)\\n self.assertTrue(torch.allclose(outputs.logits, outputs_deactivate.logits))\\n self.assertTrue(not torch.allclose(outputs.logits, outputs_lora.logits))\\n self.assertTrue(torch.allclose(outputs_lora.logits, outputs_reactivate.logits))\\n\\n def lora_injection_with_dtype(self, dtype=torch.float32):\\n from swift.tuners.lora import Linear\\n\\n def reset_lora_parameters(self, adapter_name, init_lora_weights):\\n if init_lora_weights is False:\\n return\\n\\n if adapter_name in self.lora_A.keys():\\n if init_lora_weights is True:\\n nn.init.kaiming_uniform_(self.lora_A[adapter_name].weight, a=math.sqrt(5))\\n elif init_lora_weights.lower() == 'gaussian':\\n nn.init.normal_(self.lora_A[adapter_name].weight, std=1 / self.r[adapter_name])\\n else:\\n raise ValueError(f'Unknown initialization {init_lora_weights=}')\\n nn.init.ones_(self.lora_B[adapter_name].weight)\\n if adapter_name in self.lora_embedding_A.keys():\\n # initialize a the same way as the default for nn.linear and b to zero\\n nn.init.ones_(self.lora_embedding_A[adapter_name])\\n nn.init.normal_(self.lora_embedding_B[adapter_name])\\n\\n Linear.reset_lora_parameters = reset_lora_parameters\\n\\n model = Model.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n preprocessor = Preprocessor.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n input = preprocessor('this is a test')\\n model = model.to(dtype)\\n model2 = Model.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n model2 = model2.to(dtype)\\n lora_config = LoRAConfig(target_modules=['query', 'key', 'value'])\\n model = Swift.prepare_model(model, config=lora_config)\\n self.assertTrue(isinstance(model, SwiftModel))\\n output1 = model(**input)\\n model.save_pretrained(self.tmp_dir)\\n self.assertTrue(os.path.exists(os.path.join(self.tmp_dir, 'default')))\\n self.assertTrue(os.path.exists(os.path.join(self.tmp_dir, 'default', WEIGHTS_NAME)))\\n\\n model2 = Swift.from_pretrained(model2, self.tmp_dir, adapter_name={'default': 'test'})\\n self.assertTrue('test' in model2.adapters)\\n output2 = model2(**input)\\n self.assertTrue(torch.allclose(output1.logits, output2.logits))\\n model2 = Swift.from_pretrained(model2, self.tmp_dir)\\n state_dict = model.state_dict()\\n state_dict2 = model2.state_dict()\\n for key in state_dict:\\n self.assertTrue(key in state_dict2)\\n self.assertTrue(all(torch.isclose(state_dict[key], state_dict2[key]).flatten().detach().cpu()))\\n\\n if dtype == torch.float32 and os.environ.get('USE_UNIQUE_THREAD') == '1':\\n Swift.merge_and_unload(model2)\\n output3 = model2(**input)\\n self.assertTrue(torch.allclose(output1.logits, output3.logits))\\n\\n def test_swift_lora_injection(self):\\n self.lora_injection_with_dtype()\\n\\n def test_swift_lora_injection_bf16(self):\\n self.lora_injection_with_dtype(torch.bfloat16)\\n\\n def test_save_to_peft_mix(self):\\n model = SbertForSequenceClassification(SbertConfig())\\n lora_config = LoRAConfig(target_modules=['query', 'key', 'value'])\\n adapter_config = AdapterConfig(\\n dim=model.config.hidden_size,\\n target_modules=r'.*layer\\\\.\\\\d+$',\\n method_name='feed_forward_chunk',\\n hidden_pos=0)\\n model = Swift.prepare_model(model, config={'lora': lora_config, 'adapter': adapter_config})\\n model.save_pretrained(os.path.join(self.tmp_dir, 'original'))\\n try:\\n Swift.save_to_peft_format(os.path.join(self.tmp_dir, 'original'), os.path.join(self.tmp_dir, 'converted'))\\n self.assertTrue(False)\\n except AssertionError as e:\\n print(e)\\n pass\\n\\n def test_save_to_peft_param(self):\\n model = SbertForSequenceClassification(SbertConfig())\\n lora_config = LoRAConfig(target_modules=['query', 'key', 'value'], lora_dtype='fp16')\\n model = Swift.prepare_model(model, config={'lora': lora_config})\\n model.save_pretrained(os.path.join(self.tmp_dir, 'original'))\\n try:\\n Swift.save_to_peft_format(os.path.join(self.tmp_dir, 'original'), os.path.join(self.tmp_dir, 'converted'))\\n self.assertTrue(False)\\n except AssertionError as e:\\n print(e)\\n pass\\n\\n def test_save_to_peft_ok(self):\\n model = SbertForSequenceClassification(SbertConfig())\\n lora_config = LoRAConfig(target_modules=['query', 'key', 'value'], use_dora=True)\\n lora2_config = LoRAConfig(target_modules=['query', 'key', 'value'], use_dora=True)\\n model = Swift.prepare_model(model, config={'default': lora_config, 'lora': lora2_config})\\n model.save_pretrained(os.path.join(self.tmp_dir, 'original'))\\n Swift.save_to_peft_format(os.path.join(self.tmp_dir, 'original'), os.path.join(self.tmp_dir, 'converted'))\\n # A duplicate conversion\\n Swift.save_to_peft_format(os.path.join(self.tmp_dir, 'original'), os.path.join(self.tmp_dir, 'converted'))\\n\\n # -------------------base case--------------------\\n model2 = SbertForSequenceClassification(SbertConfig())\\n model2 = PeftModel.from_pretrained(model2, os.path.join(self.tmp_dir, 'converted'))\\n model2.load_adapter(os.path.join(os.path.join(self.tmp_dir, 'converted'), 'lora'), 'lora')\\n state_dict = model.state_dict()\\n state_dict2 = {\\n key[len('base_model.model.'):]: value\\n for key, value in model2.state_dict().items() if 'lora' in key\\n }\\n for key in state_dict:\\n self.assertTrue(key in state_dict2)\\n self.assertTrue(all(torch.isclose(state_dict[key], state_dict2[key]).flatten().detach().cpu()))\\n\\n # -------------------override case--------------------\\n Swift.save_to_peft_format(os.path.join(self.tmp_dir, 'converted'), os.path.join(self.tmp_dir, 'converted'))\\n model2 = SbertForSequenceClassification(SbertConfig())\\n model2 = PeftModel.from_pretrained(model2, os.path.join(self.tmp_dir, 'converted'))\\n model2.load_adapter(os.path.join(os.path.join(self.tmp_dir, 'converted'), 'lora'), 'lora')\\n state_dict = model.state_dict()\\n state_dict2 = {\\n key[len('base_model.model.'):]: value\\n for key, value in model2.state_dict().items() if 'lora' in key\\n }\\n for key in state_dict:\\n self.assertTrue(key in state_dict2)\\n self.assertTrue(all(torch.isclose(state_dict[key], state_dict2[key]).flatten().detach().cpu()))\\n\\n def test_swift_multiple_adapters(self):\\n model = SbertForSequenceClassification(SbertConfig())\\n model2 = copy.deepcopy(model)\\n lora_config = LoRAConfig(target_modules=['query', 'key', 'value'])\\n adapter_config = AdapterConfig(\\n dim=model.config.hidden_size,\\n target_modules=r'.*layer\\\\.\\\\d+$',\\n method_name='feed_forward_chunk',\\n hidden_pos=0)\\n model = Swift.prepare_model(model, config={'lora': lora_config, 'adapter': adapter_config})\\n self.assertTrue(isinstance(model, SwiftModel))\\n model.save_pretrained(self.tmp_dir, adapter_name=['lora', 'adapter'])\\n with open(os.path.join(self.tmp_dir, 'configuration.json'), 'w') as f:\\n f.write('{}')\\n self.assertTrue(os.path.exists(os.path.join(self.tmp_dir, 'lora')))\\n self.assertTrue(os.path.exists(os.path.join(self.tmp_dir, 'lora', WEIGHTS_NAME)))\\n self.assertTrue(os.path.exists(os.path.join(self.tmp_dir, 'adapter')))\\n self.assertTrue(os.path.exists(os.path.join(self.tmp_dir, 'adapter', WEIGHTS_NAME)))\\n model2 = Swift.from_pretrained(model2, self.tmp_dir, adapter_name=['lora', 'adapter'])\\n state_dict = model.state_dict()\\n state_dict2 = model2.state_dict()\\n for key in state_dict:\\n self.assertTrue(key in state_dict2)\\n self.assertTrue(all(torch.isclose(state_dict[key], state_dict2[key]).flatten().detach().cpu()))\\n\\n def test_part(self):\\n preprocessor = Preprocessor.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n inputs = preprocessor('how are you')\\n model = SbertForSequenceClassification.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n model_origin = copy.deepcopy(model)\\n model2 = copy.deepcopy(model)\\n targets = r'.*(query|key|value).*'\\n part_config = PartConfig(target_modules=targets)\\n model = Swift.prepare_model(model, config={'part': part_config})\\n self.assertTrue(isinstance(model, SwiftModel))\\n\\n model.base_model.encoder.encoder.layer[0].attention.self.query._part_part.weight.data = torch.ones_like(\\n model.base_model.encoder.encoder.layer[0].attention.self.query._part_part.weight.data)\\n\\n for name, module in model.named_modules():\\n if re.fullmatch(targets, name) and '_part_' not in name:\\n self.assertTrue(not module.weight.requires_grad)\\n self.assertTrue(model.get_submodule(name + '._part_part').weight.requires_grad)\\n\\n model.save_pretrained(self.tmp_dir, adapter_name=['part'])\\n with open(os.path.join(self.tmp_dir, 'configuration.json'), 'w') as f:\\n f.write('{}')\\n self.assertTrue(os.path.exists(os.path.join(self.tmp_dir, 'part')))\\n self.assertTrue(os.path.exists(os.path.join(self.tmp_dir, 'part', WEIGHTS_NAME)))\\n model2 = Swift.from_pretrained(model2, self.tmp_dir, adapter_name=['part'])\\n self.assertTrue(\\n all(\\n torch.isclose(model.base_model.encoder.encoder.layer[0].attention.self.query._part_part.weight.data,\\n model2.base_model.encoder.encoder.layer[0].attention.self.query._part_part.weight.data).\\n flatten().detach().cpu()))\\n\\n state_dict = model.model.state_dict()\\n state_dict2 = model2.model.state_dict()\\n self.assertTrue(str(state_dict) == str(state_dict2))\\n\\n output = model(**inputs)\\n output2 = model2(**inputs)\\n output_origin = model_origin(**inputs)\\n self.assertTrue(all(torch.isclose(output.logits, output2.logits).flatten().detach().cpu()))\\n self.assertTrue(not all(torch.isclose(output_origin.logits, output2.logits).flatten().detach().cpu()))\\n\\n model2.deactivate_adapter('part')\\n output = model(**inputs)\\n output2 = model2(**inputs)\\n output_origin = model_origin(**inputs)\\n self.assertTrue(not all(torch.isclose(output.logits, output2.logits).flatten().detach().cpu()))\\n self.assertTrue(all(torch.isclose(output_origin.logits, output2.logits).flatten().detach().cpu()))\\n\\n model2.activate_adapter('part')\\n output = model(**inputs)\\n output2 = model2(**inputs)\\n output_origin = model_origin(**inputs)\\n self.assertTrue(all(torch.isclose(output.logits, output2.logits).flatten().detach().cpu()))\\n self.assertTrue(not all(torch.isclose(output_origin.logits, output2.logits).flatten().detach().cpu()))\\n\\n targets = r'.*(query|key|value).*'\\n part_config = PartConfig(target_modules=targets)\\n lora_config = LoRAConfig(target_modules=targets)\\n model2 = Swift.prepare_model(model2, config={'part2': part_config})\\n model2 = Swift.prepare_model(model2, config={'lora': lora_config})\\n model2 = Swift.prepare_model(model2, config={'part3': part_config})\\n model2.set_active_adapters('part2', offload='meta')\\n model2.set_active_adapters('part3', offload='meta')\\n model2.set_active_adapters('lora', offload='meta')\\n model2.set_active_adapters('part2', offload='meta')\\n self.assertTrue(\\n not model2.base_model.encoder.encoder.layer[0].attention.self.query.base_layer._part_part.activated)\\n self.assertTrue(\\n model2.base_model.encoder.encoder.layer[0].attention.self.query.base_layer._part_part2.activated)\\n model2.set_active_adapters('part', offload='meta')\\n self.assertTrue(\\n not model2.base_model.encoder.encoder.layer[0].attention.self.query.base_layer._part_part2.activated)\\n self.assertTrue(model2.base_model.encoder.encoder.layer[0].attention.self.query.base_layer._part_part.activated)\\n output = model(**inputs)\\n output2 = model2(**inputs)\\n output_origin = model_origin(**inputs)\\n self.assertTrue(all(torch.isclose(output.logits, output2.logits).flatten().detach().cpu()))\\n self.assertTrue(not all(torch.isclose(output_origin.logits, output2.logits).flatten().detach().cpu()))\\n\\n model2.set_active_adapters('part2', offload='meta')\\n model2.deactivate_adapter('part2', offload='meta')\\n model2.deactivate_adapter('lora', offload='cpu')\\n self.assertTrue(\\n not model2.base_model.encoder.encoder.layer[0].attention.self.query.base_layer._part_part2.activated)\\n self.assertTrue(\\n not model2.base_model.encoder.encoder.layer[0].attention.self.query.base_layer._part_part.activated)\\n output = model(**inputs)\\n output2 = model2(**inputs)\\n output_origin = model_origin(**inputs)\\n self.assertTrue(not all(torch.isclose(output.logits, output2.logits).flatten().detach().cpu()))\\n self.assertTrue(all(torch.isclose(output_origin.logits, output2.logits).flatten().detach().cpu()))\\n model2.activate_adapter('lora')\\n self.assertTrue(\\n not model2.base_model.encoder.encoder.layer[0].attention.self.query.base_layer._part_part2.activated)\\n self.assertTrue(\\n not model2.base_model.encoder.encoder.layer[0].attention.self.query.base_layer._part_part.activated)\\n self.assertTrue(\\n not model2.base_model.encoder.encoder.layer[0].attention.self.query.base_layer._part_part3.activated)\\n self.assertTrue(model2.base_model.encoder.encoder.layer[0].attention.self.query.active_adapters == ['lora'])\\n\\n def test_swift_multiple_adapters_switching(self):\\n from swift.tuners.lora import Linear\\n from swift.tuners.adapter import AdapterModule\\n\\n def reset_lora_parameters(self, adapter_name, init_lora_weights):\\n if init_lora_weights is False:\\n return\\n\\n if adapter_name in self.lora_A.keys():\\n if init_lora_weights is True:\\n # initialize A the same way as the default for nn.Linear and B to zero\\n # https://github.com/microsoft/LoRA/blob/a0a92e0f26c067cf94747bdbf1ce73793fa44d19/loralib/layers.py#L124\\n nn.init.ones_(self.lora_A[adapter_name].weight)\\n elif init_lora_weights.lower() == 'gaussian':\\n nn.init.normal_(self.lora_A[adapter_name].weight, std=1 / self.r[adapter_name])\\n else:\\n raise ValueError(f'Unknown initialization {init_lora_weights=}')\\n nn.init.ones_(self.lora_B[adapter_name].weight)\\n if adapter_name in self.lora_embedding_A.keys():\\n # initialize a the same way as the default for nn.linear and b to zero\\n nn.init.ones_(self.lora_embedding_A[adapter_name])\\n nn.init.normal_(self.lora_embedding_B[adapter_name])\\n\\n Linear.reset_lora_parameters = reset_lora_parameters\\n\\n def init_weights(self):\\n\\n def _init_weights(m):\\n if isinstance(m, nn.Linear):\\n nn.init.ones_(m.weight)\\n nn.init.ones_(m.bias)\\n\\n self.apply(_init_weights)\\n\\n AdapterModule.init_weights = init_weights\\n\\n model = Model.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n preprocessor = Preprocessor.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n inputs = preprocessor('how are you')\\n model1 = copy.deepcopy(model)\\n model2 = copy.deepcopy(model)\\n model1 = Swift.prepare_model(\\n model1,\\n config={\\n 'lora1':\\n LoRAConfig(target_modules=['query', 'key', 'value']),\\n 'adapter1':\\n AdapterConfig(\\n dim=model.config.hidden_size,\\n target_modules=r'.*layer\\\\.\\\\d+$',\\n method_name='feed_forward_chunk',\\n hidden_pos=0)\\n })\\n model2 = Swift.prepare_model(\\n model2,\\n config={\\n 'lora2':\\n LoRAConfig(target_modules=['query', 'key', 'value']),\\n 'adapter2':\\n AdapterConfig(\\n dim=model.config.hidden_size,\\n target_modules=r'.*layer\\\\.\\\\d+$',\\n method_name='feed_forward_chunk',\\n hidden_pos=0)\\n })\\n model = Swift.prepare_model(\\n model,\\n config={\\n 'lora1': LoRAConfig(target_modules=['query', 'key', 'value']),\\n 'lora2': LoRAConfig(target_modules=['query', 'key', 'value']),\\n })\\n\\n model = Swift.prepare_model(\\n model,\\n config={\\n 'adapter1':\\n AdapterConfig(\\n dim=model.config.hidden_size,\\n target_modules=r'.*layer\\\\.\\\\d+$',\\n method_name='feed_forward_chunk',\\n hidden_pos=0),\\n 'adapter2':\\n AdapterConfig(\\n dim=model.config.hidden_size,\\n target_modules=r'.*layer\\\\.\\\\d+$',\\n method_name='feed_forward_chunk',\\n hidden_pos=0),\\n })\\n\\n model.deactivate_adapter('adapter2', offload='meta')\\n model.deactivate_adapter('lora2', offload='meta')\\n outputs1 = model(**inputs)\\n outputs2 = model1(**inputs)\\n self.assertTrue(torch.allclose(outputs1.logits, outputs2.logits))\\n model.activate_adapter('adapter2')\\n model.activate_adapter('lora2')\\n model.deactivate_adapter('adapter1', offload='meta')\\n model.deactivate_adapter('lora1', offload='meta')\\n outputs1 = model(**inputs)\\n outputs2 = model2(**inputs)\\n self.assertTrue(torch.allclose(outputs1.logits, outputs2.logits))\\n\\n if os.environ.get('USE_UNIQUE_THREAD') == '0':\\n\\n def thread_func1():\\n model1.set_active_adapters(['lora1', 'adapter1'], offload=None)\\n model.set_active_adapters(['lora1', 'adapter1'], offload=None)\\n outputs_single = model1(**inputs)\\n outputs_t1 = model(**inputs)\\n self.assertTrue(torch.allclose(outputs_single.logits, outputs_t1.logits))\\n\\n def thread_func2():\\n model2.set_active_adapters(['lora2', 'adapter2'], offload=None)\\n model.set_active_adapters(['lora2', 'adapter2'], offload=None)\\n outputs_single = model2(**inputs)\\n outputs_t2 = model(**inputs)\\n self.assertTrue(torch.allclose(outputs_single.logits, outputs_t2.logits))\\n\\n with ThreadPoolExecutor(2) as executor:\\n f1 = executor.submit(thread_func1)\\n f2 = executor.submit(thread_func2)\\n e1 = f1.exception()\\n e2 = f2.exception()\\n if e1 is not None:\\n raise e1\\n if e2 is not None:\\n raise e2\\n\\n def test_swift_side_bert(self):\\n model = Model.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n preprocessor = Preprocessor.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n inputs = preprocessor('how are you')\\n model2 = copy.deepcopy(model)\\n result_origin = model(**inputs).logits\\n print(f'test_swift_side_bert result_origin shape: {result_origin.shape}, '\\n f'result_origin sum: {torch.sum(result_origin)}')\\n\\n side_config = SideConfig(\\n dim=model.config.hidden_size,\\n target_modules=r'.*encoder.encoder',\\n side_module_name='mlp',\\n target_hidden_pos='last_hidden_state')\\n\\n model = Swift.prepare_model(model, config=side_config)\\n result_activate = model(**inputs).logits\\n model.deactivate_adapter('default')\\n result_deactivate = model(**inputs).logits\\n model.activate_adapter('default')\\n result_reactivate = model(**inputs).logits\\n self.assertTrue(torch.allclose(result_origin, result_deactivate))\\n self.assertTrue(not torch.allclose(result_origin, result_activate))\\n self.assertTrue(torch.allclose(result_activate, result_reactivate))\\n print(f'test_swift_side_bert result shape: {result_origin.shape}, result sum: {torch.sum(result_origin)}')\\n\\n self.assertTrue(isinstance(model, SwiftModel))\\n model.save_pretrained(self.tmp_dir)\\n self.assertTrue(os.path.exists(os.path.join(self.tmp_dir, 'default')))\\n self.assertTrue(os.path.exists(os.path.join(self.tmp_dir, 'default', WEIGHTS_NAME)))\\n\\n model2 = Swift.from_pretrained(model2, self.tmp_dir)\\n\\n state_dict = model.state_dict()\\n state_dict2 = model2.state_dict()\\n for key in state_dict:\\n self.assertTrue(key in state_dict2)\\n self.assertTrue(all(torch.isclose(state_dict[key], state_dict2[key]).flatten().detach().cpu()))\\n\\n\\nif __name__ == '__main__':\\n unittest.main()\\n\\n\\nimport math\\nimport unittest\\n\\nimport torch\\nfrom modelscope import Model, Preprocessor\\nfrom torch import nn\\n\\nfrom swift import LoRAConfig, Swift\\n\\n\\nclass TestMergedLinear(unittest.TestCase):\\n\\n def test_swift_lora_forward(self):\\n\\n from swift.tuners.lora import MergedLinear\\n\\n def reset_parameters(self):\\n nn.Linear.reset_parameters(self)\\n if hasattr(self, 'lora_A'):\\n # initialize A the same way as the default for nn.Linear and B to zero\\n nn.init.kaiming_uniform_(self.lora_A, a=math.sqrt(5))\\n nn.init.ones_(self.lora_B)\\n\\n MergedLinear.reset_parameters = reset_parameters\\n\\n model = Model.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n preprocessor = Preprocessor.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n inputs = preprocessor('how are you')\\n lora_config = LoRAConfig(\\n target_modules=['query', 'key', 'value'], use_merged_linear=True, enable_lora=[True, True, True])\\n outputs = model(**inputs)\\n model = Swift.prepare_model(model, config=lora_config)\\n model.eval()\\n outputs_lora = model(**inputs)\\n model.deactivate_adapter('default')\\n outputs_deactivate = model(**inputs)\\n model.activate_adapter('default')\\n outputs_reactivate = model(**inputs)\\n Swift.merge_and_unload(model)\\n outputs_merged = model(**inputs)\\n self.assertTrue(torch.allclose(outputs.logits, outputs_deactivate.logits))\\n self.assertTrue(not torch.allclose(outputs.logits, outputs_lora.logits))\\n self.assertTrue(torch.allclose(outputs_lora.logits, outputs_reactivate.logits))\\n self.assertTrue(torch.allclose(outputs_lora.logits, outputs_merged.logits, atol=1e-4))\\n\\n\\nimport os\\nimport shutil\\nimport tempfile\\nimport unittest\\n\\nimport torch\\nfrom modelscope import AutoTokenizer, Model\\n\\nfrom swift import Swift\\nfrom swift.tuners.rome import RomeConfig\\n\\n\\nclass TestRome(unittest.TestCase):\\n\\n def setUp(self):\\n print(('Testing %s.%s' % (type(self).__name__, self._testMethodName)))\\n self.tmp_dir = tempfile.TemporaryDirectory().name\\n if not os.path.exists(self.tmp_dir):\\n os.makedirs(self.tmp_dir)\\n\\n def tearDown(self):\\n shutil.rmtree(self.tmp_dir)\\n super().tearDown()\\n\\n @unittest.skip('Rome test is skipped because the test image do not have flash-attn2')\\n def test_rome(self):\\n model = Model.from_pretrained('modelscope/Llama-2-7b-ms', device_map='auto', trust_remote_code=True)\\n tokenizer = AutoTokenizer.from_pretrained('modelscope/Llama-2-7b-ms', trust_remote_code=True)\\n request = [{\\n 'prompt': '{} was the founder of',\\n 'subject': 'Steve Jobs',\\n 'target': 'Microsoft',\\n }]\\n config = RomeConfig(\\n model_type='llama-7b',\\n knowledge=request,\\n tokenizer=tokenizer,\\n )\\n\\n model = Swift.prepare_model(model, config)\\n prompt = 'Steve Jobs was the founder of'\\n inp_tok = tokenizer(prompt, return_token_type_ids=False, return_tensors='pt')\\n for key, value in inp_tok.items():\\n inp_tok[key] = value.to('cuda')\\n with torch.no_grad():\\n generated_ids = model.generate(**inp_tok, temperature=0.1, top_k=50, max_length=128, do_sample=True)\\n\\n responses = tokenizer.batch_decode(\\n generated_ids[:, inp_tok['input_ids'].size(1):],\\n skip_special_tokens=True,\\n clean_up_tokenization_spaces=True)\\n self.assertTrue('Microsoft' in responses[0])\\n\\n\\nimport os\\nimport shutil\\nimport tempfile\\nimport unittest\\n\\nimport torch\\nfrom modelscope import AutoModel, Preprocessor\\nfrom peft.utils import WEIGHTS_NAME\\nfrom transformers import PreTrainedModel\\n\\nfrom swift import LoRAConfig, Swift\\nfrom swift.tuners import NEFTuneConfig\\n\\n\\nclass TestNEFT(unittest.TestCase):\\n\\n def setUp(self):\\n print(('Testing %s.%s' % (type(self).__name__, self._testMethodName)))\\n self.tmp_dir = tempfile.TemporaryDirectory().name\\n if not os.path.exists(self.tmp_dir):\\n os.makedirs(self.tmp_dir)\\n\\n def tearDown(self):\\n shutil.rmtree(self.tmp_dir)\\n super().tearDown()\\n\\n def test_neft(self):\\n model = AutoModel.from_pretrained('AI-ModelScope/bert-base-uncased')\\n preprocessor = Preprocessor.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n inputs = preprocessor('how are you')\\n config = NEFTuneConfig()\\n\\n t1 = model.embeddings.word_embeddings(inputs['input_ids'])\\n model = Swift.prepare_model(model, config)\\n model.train()\\n t2 = model.embeddings.word_embeddings(inputs['input_ids'])\\n model.deactivate_adapter('default')\\n t3 = model.embeddings.word_embeddings(inputs['input_ids'])\\n self.assertTrue(torch.allclose(t1, t3))\\n self.assertFalse(torch.allclose(t1, t2))\\n model.save_pretrained(self.tmp_dir)\\n bin_file = os.path.join(self.tmp_dir, 'pytorch_model.bin')\\n self.assertTrue(os.path.isfile(bin_file))\\n model2 = AutoModel.from_pretrained(self.tmp_dir)\\n\\n state_dict = model.state_dict()\\n state_dict2 = model2.state_dict()\\n self.assertTrue(len(state_dict) > 0)\\n for key in state_dict:\\n self.assertTrue(key in state_dict2)\\n self.assertTrue(all(torch.isclose(state_dict[key], state_dict2[key]).flatten().detach().cpu()))\\n\\n shutil.rmtree(self.tmp_dir)\\n PreTrainedModel.origin_save_pretrained = PreTrainedModel.save_pretrained\\n delattr(PreTrainedModel, 'save_pretrained')\\n model.save_pretrained(self.tmp_dir)\\n bin_file = os.path.join(self.tmp_dir, WEIGHTS_NAME)\\n self.assertTrue(os.path.isfile(bin_file))\\n model_new = AutoModel.from_pretrained('AI-ModelScope/bert-base-uncased')\\n model_new_2 = Swift.from_pretrained(model_new, self.tmp_dir)\\n\\n state_dict = model.state_dict()\\n state_dict2 = model_new_2.state_dict()\\n self.assertTrue(len(state_dict) > 0)\\n for key in state_dict:\\n self.assertTrue(key in state_dict2)\\n self.assertTrue(all(torch.isclose(state_dict[key], state_dict2[key]).flatten().detach().cpu()))\\n PreTrainedModel.save_pretrained = PreTrainedModel.origin_save_pretrained\\n\\n def test_neft_lora(self):\\n model = AutoModel.from_pretrained('AI-ModelScope/bert-base-uncased')\\n preprocessor = Preprocessor.from_pretrained('damo/nlp_structbert_sentence-similarity_chinese-base')\\n inputs = preprocessor('how are you')\\n config = NEFTuneConfig()\\n config2 = LoRAConfig(target_modules=['query', 'key', 'value'])\\n\\n t1 = model.embeddings.word_embeddings(inputs['input_ids'])\\n model = Swift.prepare_model(model, {'c1': config, 'c2': config2})\\n model.train()\\n t2 = model.embeddings.word_embeddings(inputs['input_ids'])\\n model.deactivate_adapter('c1')\\n t3 = model.embeddings.word_embeddings(inputs['input_ids'])\\n self.assertTrue(torch.allclose(t1, t3))\\n self.assertFalse(torch.allclose(t1, t2))\\n model.save_pretrained(self.tmp_dir)\\n bin_file = os.path.join(self.tmp_dir, 'c2', WEIGHTS_NAME)\\n self.assertTrue(os.path.isfile(bin_file))\\n bin_file = os.path.join(self.tmp_dir, 'c1', WEIGHTS_NAME)\\n self.assertTrue(not os.path.isfile(bin_file))\\n model_new = AutoModel.from_pretrained('AI-ModelScope/bert-base-uncased')\\n t1 = model_new.embeddings.word_embeddings(inputs['input_ids'])\\n model_new = Swift.from_pretrained(model_new, self.tmp_dir)\\n model_new.train()\\n t2 = model_new.embeddings.word_embeddings(inputs['input_ids'])\\n model_new.eval()\\n t4 = model_new.embeddings.word_embeddings(inputs['input_ids'])\\n model_new.train()\\n model_new.deactivate_adapter('c1')\\n t3 = model_new.embeddings.word_embeddings(inputs['input_ids'])\\n self.assertTrue(torch.allclose(t1, t3))\\n self.assertTrue(torch.allclose(t1, t4))\\n self.assertFalse(torch.allclose(t1, t2))\\n\\n state_dict = model.state_dict()\\n state_dict2 = model_new.state_dict()\\n self.assertTrue(len(state_dict) > 0 and all(['lora' in key for key in state_dict.keys()]))\\n for key in state_dict:\\n self.assertTrue(key in state_dict2)\\n self.assertTrue(all(torch.isclose(state_dict[key], state_dict2[key]).flatten().detach().cpu()))\\n\\n\\nimport os\\nimport shutil\\nimport tempfile\\nimport unittest\\n\\nimport torch\\nfrom modelscope import Model\\nfrom peft.utils import WEIGHTS_NAME\\n\\nfrom swift import LoRAConfig, SwiftModel\\n\\n\\n@unittest.skip\\nclass TestSwift(unittest.TestCase):\\n\\n def setUp(self):\\n print(('Testing %s.%s' % (type(self).__name__, self._testMethodName)))\\n self.tmp_dir = tempfile.TemporaryDirectory().name\\n if not os.path.exists(self.tmp_dir):\\n os.makedirs(self.tmp_dir)\\n\\n def tearDown(self):\\n shutil.rmtree(self.tmp_dir)\\n super().tearDown()\\n\\n def test_swift_multiple_adapters(self):\\n model = Model.from_pretrained('modelscope/Llama-2-7b-ms', device_map='auto')\\n lora_config = LoRAConfig(target_modules=['q_proj', 'k_proj', 'v_proj'])\\n model: SwiftModel = SwiftModel(model, config={'lora': lora_config})\\n self.assertTrue(isinstance(model, SwiftModel))\\n model.save_pretrained(self.tmp_dir, adapter_name=['lora'])\\n state_dict = model.state_dict()\\n with open(os.path.join(self.tmp_dir, 'configuration.json'), 'w') as f:\\n f.write('{}')\\n self.assertTrue(os.path.exists(os.path.join(self.tmp_dir, 'lora')))\\n self.assertTrue(os.path.exists(os.path.join(self.tmp_dir, 'lora', WEIGHTS_NAME)))\\n model = Model.from_pretrained('modelscope/Llama-2-7b-ms', device_map='auto')\\n model = SwiftModel.from_pretrained(model, self.tmp_dir, adapter_name=['lora'], device_map='auto')\\n\\n state_dict2 = model.state_dict()\\n for key in state_dict:\\n self.assertTrue(key in state_dict2)\\n self.assertTrue(all(torch.isclose(state_dict[key], state_dict2[key]).flatten().detach().cpu()))\\n\\n self.assertTrue(len(set(model.hf_device_map.values())) == torch.cuda.device_count())\\n\\n\\n\\n\\ndef test_vllm():\\n import os\\n os.environ['CUDA_VISIBLE_DEVICES'] = '0'\\n\\n from swift.llm import (ModelType, get_vllm_engine, get_default_template_type, get_template, inference_vllm,\\n inference_stream_vllm)\\n\\n model_type = ModelType.qwen_7b_chat\\n llm_engine = get_vllm_engine(model_type)\\n template_type = get_default_template_type(model_type)\\n template = get_template(template_type, llm_engine.hf_tokenizer)\\n # 与`transformers.GenerationConfig`类似的接口\\n llm_engine.generation_config.max_new_tokens = 256\\n generation_info = {}\\n\\n request_list = [{'query': '你好!'}, {'query': '浙江的省会在哪？'}]\\n resp_list = inference_vllm(llm_engine, template, request_list, generation_info=generation_info)\\n for request, resp in zip(request_list, resp_list):\\n print(f\\\"query: {request['query']}\\\")\\n print(f\\\"response: {resp['response']}\\\")\\n print(generation_info)\\n\\n # stream\\n history1 = resp_list[1]['history']\\n request_list = [{'query': '这有什么好吃的', 'history': history1}]\\n gen = inference_stream_vllm(llm_engine, template, request_list, generation_info=generation_info)\\n query = request_list[0]['query']\\n print_idx = 0\\n print(f'query: {query}\\\\nresponse: ', end='')\\n for resp_list in gen:\\n resp = resp_list[0]\\n response = resp['response']\\n delta = response[print_idx:]\\n print(delta, end='', flush=True)\\n print_idx = len(response)\\n print()\\n\\n history = resp_list[0]['history']\\n print(f'history: {history}')\\n print(generation_info)\\n\\n # batched\\n n_batched = 1000\\n request_list = [{'query': '晚上睡不着觉怎么办?'} for i in range(n_batched)]\\n resp_list = inference_vllm(llm_engine, template, request_list, generation_info=generation_info, use_tqdm=True)\\n assert len(resp_list) == n_batched\\n print(resp_list[0]['history'])\\n print(generation_info)\\n\\n request_list = [{'query': '晚上睡不着觉怎么办?'} for i in range(n_batched)]\\n gen = inference_stream_vllm(llm_engine, template, request_list, generation_info=generation_info, use_tqdm=True)\\n for resp_list in gen:\\n pass\\n assert len(resp_list) == n_batched\\n print(resp_list[0]['history'])\\n print(generation_info)\\n\\n\\nif __name__ == '__main__':\\n test_vllm()\\n\\n\\ndef test_lmdeploy_vlm():\\n import os\\n os.environ['CUDA_VISIBLE_DEVICES'] = '1'\\n\\n from swift.llm import (ModelType, get_lmdeploy_engine, get_default_template_type, get_template, inference_lmdeploy,\\n inference_stream_lmdeploy)\\n\\n model_type = ModelType.deepseek_vl_1_3b_chat\\n lmdeploy_engine = get_lmdeploy_engine(model_type)\\n template_type = get_default_template_type(model_type)\\n template = get_template(template_type, lmdeploy_engine.hf_tokenizer)\\n # 与`transformers.GenerationConfig`类似的接口\\n lmdeploy_engine.generation_config.max_new_tokens = 256\\n lmdeploy_engine.generation_config.logprobs = 2\\n generation_info = {}\\n\\n request_list = [{\\n 'query':\\n '这两张图片有什么区别：'\\n 'http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png'\\n 'https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png'\\n }, {\\n 'query': '你好'\\n }]\\n resp_list = inference_lmdeploy(lmdeploy_engine, template, request_list, generation_info=generation_info)\\n for request, resp in zip(request_list, resp_list):\\n print(f\\\"query: {request['query']}\\\")\\n print(f\\\"response: {resp['response']}\\\")\\n print(f\\\"len(logprobs): {len(resp['logprobs'])}\\\")\\n print(generation_info)\\n\\n # stream\\n history0 = resp_list[0]['history']\\n request_list = [{'query': '有几只羊', 'history': history0}]\\n gen = inference_stream_lmdeploy(lmdeploy_engine, template, request_list, generation_info=generation_info)\\n query = request_list[0]['query']\\n print_idx = 0\\n print(f'query: {query}\\\\nresponse: ', end='')\\n for resp_list in gen:\\n resp = resp_list[0]\\n response = resp['response']\\n delta = response[print_idx:]\\n print(delta, end='', flush=True)\\n print_idx = len(response)\\n print()\\n\\n history = resp_list[0]['history']\\n print(f'history: {history}')\\n print(generation_info)\\n print(f\\\"len(logprobs): {len(resp_list[0]['logprobs'])}\\\")\\n\\n # batched\\n n_batched = 1000\\n request_list = [{\\n 'query':\\n '这两张图片有什么区别：'\\n 'http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png'\\n 'https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png'\\n } for i in range(n_batched)]\\n resp_list = inference_lmdeploy(\\n lmdeploy_engine, template, request_list, generation_info=generation_info, use_tqdm=True)\\n assert len(resp_list) == n_batched\\n print(resp_list[0]['history'])\\n print(generation_info)\\n\\n request_list = [{\\n 'query':\\n '这两张图片有什么区别：'\\n 'http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png'\\n 'https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png'\\n } for i in range(n_batched)]\\n gen = inference_stream_lmdeploy(\\n lmdeploy_engine, template, request_list, generation_info=generation_info, use_tqdm=True)\\n for resp_list in gen:\\n pass\\n assert len(resp_list) == n_batched\\n print(resp_list[0]['history'])\\n print(generation_info)\\n\\n\\nif __name__ == '__main__':\\n test_lmdeploy_vlm()\\n\\n\\ndef test_vllm_vlm():\\n import os\\n import vllm\\n from packaging import version\\n os.environ['CUDA_VISIBLE_DEVICES'] = '0'\\n assert version.parse(vllm.__version__) >= version.parse('0.5.1')\\n from swift.llm import (ModelType, get_vllm_engine, get_default_template_type, get_template, inference_vllm,\\n inference_stream_vllm)\\n\\n model_type = ModelType.llava1_6_mistral_7b_instruct\\n llm_engine = get_vllm_engine(model_type)\\n template_type = get_default_template_type(model_type)\\n template = get_template(template_type, llm_engine.hf_tokenizer)\\n # 与`transformers.GenerationConfig`类似的接口\\n llm_engine.generation_config.max_new_tokens = 256\\n llm_engine.generation_config.logprobs = 2\\n generation_info = {}\\n\\n images = ['http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png']\\n request_list = [{'query': 'who are you'}, {'query': 'Describe this image.', 'images': images}]\\n resp_list = inference_vllm(llm_engine, template, request_list, generation_info=generation_info)\\n for request, resp in zip(request_list, resp_list):\\n print(f\\\"query: {request['query']}\\\")\\n print(f\\\"response: {resp['response']}\\\")\\n print(f\\\"len(logprobs): {len(resp['logprobs'])}\\\")\\n print(generation_info)\\n\\n # stream\\n history1 = resp_list[1]['history']\\n request_list = [{'query': '有几只羊', 'history': history1, 'images': images}]\\n gen = inference_stream_vllm(llm_engine, template, request_list, generation_info=generation_info)\\n query = request_list[0]['query']\\n print_idx = 0\\n print(f'query: {query}\\\\nresponse: ', end='')\\n for resp_list in gen:\\n resp = resp_list[0]\\n response = resp['response']\\n delta = response[print_idx:]\\n print(delta, end='', flush=True)\\n print_idx = len(response)\\n print()\\n\\n history = resp_list[0]['history']\\n print(f'history: {history}')\\n print(generation_info)\\n print(f\\\"len(logprobs): {len(resp_list[0]['logprobs'])}\\\")\\n\\n # batched\\n n_batched = 1000\\n images = ['http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png']\\n request_list = [{'query': 'Describe this image.', 'images': images} for i in range(n_batched)]\\n resp_list = inference_vllm(llm_engine, template, request_list, generation_info=generation_info, use_tqdm=True)\\n assert len(resp_list) == n_batched\\n print(resp_list[0]['history'])\\n print(generation_info)\\n\\n images = ['http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png']\\n request_list = [{'query': 'Describe this image.', 'images': images} for i in range(n_batched)]\\n gen = inference_stream_vllm(llm_engine, template, request_list, generation_info=generation_info, use_tqdm=True)\\n for resp_list in gen:\\n pass\\n assert len(resp_list) == n_batched\\n print(resp_list[0]['history'])\\n print(generation_info)\\n\\n\\nif __name__ == '__main__':\\n test_vllm_vlm()\\n\\n\\ndef test_lmdeploy():\\n import os\\n os.environ['CUDA_VISIBLE_DEVICES'] = '1'\\n\\n from swift.llm import (ModelType, get_lmdeploy_engine, get_default_template_type, get_template, inference_lmdeploy,\\n inference_stream_lmdeploy)\\n\\n model_type = ModelType.qwen_7b_chat\\n lmdeploy_engine = get_lmdeploy_engine(model_type)\\n template_type = get_default_template_type(model_type)\\n template = get_template(template_type, lmdeploy_engine.hf_tokenizer)\\n # 与`transformers.GenerationConfig`类似的接口\\n lmdeploy_engine.generation_config.max_new_tokens = 256\\n generation_info = {}\\n\\n request_list = [{'query': '你好!'}, {'query': '浙江的省会在哪？'}]\\n resp_list = inference_lmdeploy(lmdeploy_engine, template, request_list, generation_info=generation_info)\\n for request, resp in zip(request_list, resp_list):\\n print(f\\\"query: {request['query']}\\\")\\n print(f\\\"response: {resp['response']}\\\")\\n print(generation_info)\\n\\n # stream\\n history1 = resp_list[1]['history']\\n request_list = [{'query': '这有什么好吃的', 'history': history1}]\\n gen = inference_stream_lmdeploy(lmdeploy_engine, template, request_list, generation_info=generation_info)\\n query = request_list[0]['query']\\n print_idx = 0\\n print(f'query: {query}\\\\nresponse: ', end='')\\n for resp_list in gen:\\n resp = resp_list[0]\\n response = resp['response']\\n delta = response[print_idx:]\\n print(delta, end='', flush=True)\\n print_idx = len(response)\\n print()\\n\\n history = resp_list[0]['history']\\n print(f'history: {history}')\\n print(generation_info)\\n\\n # batched\\n n_batched = 1000\\n request_list = [{'query': '晚上睡不着觉怎么办?'} for i in range(n_batched)]\\n resp_list = inference_lmdeploy(\\n lmdeploy_engine, template, request_list, generation_info=generation_info, use_tqdm=True)\\n assert len(resp_list) == n_batched\\n print(resp_list[0]['history'])\\n print(generation_info)\\n\\n request_list = [{'query': '晚上睡不着觉怎么办?'} for i in range(n_batched)]\\n gen = inference_stream_lmdeploy(\\n lmdeploy_engine, template, request_list, generation_info=generation_info, use_tqdm=True)\\n for resp_list in gen:\\n pass\\n assert len(resp_list) == n_batched\\n print(resp_list[0]['history'])\\n print(generation_info)\\n\\n\\nif __name__ == '__main__':\\n test_lmdeploy()\\n\\n\\ndef test_pt_vlm():\\n import os\\n os.environ['CUDA_VISIBLE_DEVICES'] = '0'\\n from swift.llm import (ModelType, get_model_tokenizer, get_default_template_type, get_template, inference,\\n inference_stream)\\n\\n model_type = ModelType.internvl2_2b\\n model, tokenizer = get_model_tokenizer(model_type)\\n template_type = get_default_template_type(model_type)\\n template = get_template(template_type, tokenizer)\\n # 与`transformers.GenerationConfig`类似的接口\\n model.generation_config.max_new_tokens = 256\\n model.generation_config.return_dict_in_generate = True\\n generation_info = {}\\n query = 'who are you?'\\n resp = inference(model, template, query=query, generation_info=generation_info)\\n print(f'query: {query}')\\n print(f\\\"response: {resp['response']}\\\")\\n print(generation_info)\\n print(resp.keys())\\n\\n # stream\\n images = ['http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png']\\n history = resp['history']\\n query = '有几只羊'\\n gen = inference_stream(\\n model, template, query=query, history=history, images=images, generation_info=generation_info)\\n print_idx = 0\\n print(f'query: {query}\\\\nresponse: ', end='')\\n for resp in gen:\\n response = resp['response']\\n delta = response[print_idx:]\\n print(delta, end='', flush=True)\\n print_idx = len(response)\\n print()\\n\\n history = resp['history']\\n print(f'history: {history}')\\n print(generation_info)\\n print(resp.keys())\\n\\n\\nif __name__ == '__main__':\\n test_pt_vlm()\\n\\n\\ndef _test_client(print_logprobs: bool = False):\\n import requests\\n import time\\n from pprint import pprint\\n from swift.llm import get_model_list_client, XRequestConfig, inference_client\\n query = '123*234=?'\\n\\n while True:\\n try:\\n model_list = get_model_list_client()\\n except requests.exceptions.ConnectionError:\\n time.sleep(5)\\n continue\\n break\\n model_type = model_list.data[0].id\\n is_chat = model_list.data[0].is_chat\\n is_multimodal = model_list.data[0].is_multimodal\\n print(f'model_type: {model_type}')\\n\\n request_config = XRequestConfig(seed=42, max_tokens=256, temperature=0.8, logprobs=True, top_logprobs=5)\\n resp = inference_client(\\n model_type, query, request_config=request_config, is_chat=is_chat, is_multimodal=is_multimodal)\\n response = resp.choices[0].message.content\\n print(f'query: {query}')\\n print(f'response: {response}')\\n if print_logprobs:\\n pprint(resp.choices[0].logprobs)\\n\\n request_config = XRequestConfig(\\n stream=True, seed=42, max_tokens=256, temperature=0.8, top_k=20, top_p=0.8, logprobs=True, top_logprobs=5)\\n stream_resp = inference_client(\\n model_type, query, request_config=request_config, is_chat=is_chat, is_multimodal=is_multimodal)\\n print(f'query: {query}')\\n print('response: ', end='')\\n for chunk in stream_resp:\\n print(chunk.choices[0].delta.content, end='', flush=True)\\n if print_logprobs and chunk.choices[0].logprobs is not None:\\n pprint(chunk.choices[0].logprobs)\\n print()\\n\\n\\ndef _test(infer_backend):\\n import os\\n os.environ['CUDA_VISIBLE_DEVICES'] = '0'\\n os.environ['TIMEOUT'] = '-1'\\n\\n from swift.llm import DeployArguments\\n from swift.llm.deploy import llm_deploy\\n import multiprocessing\\n mp = multiprocessing.get_context('spawn')\\n process = mp.Process(\\n target=llm_deploy,\\n args=(DeployArguments(model_type='qwen2-7b-instruct', infer_backend=infer_backend, verbose=False), ))\\n process.start()\\n _test_client(True)\\n process.terminate()\\n\\n\\ndef test_vllm():\\n _test('vllm')\\n\\n\\ndef test_lmdeploy():\\n _test('lmdeploy')\\n\\n\\ndef test_pt():\\n _test('pt')\\n\\n\\ndef test_vllm_orgin():\\n import os\\n import subprocess\\n import sys\\n from modelscope import snapshot_download\\n model_dir = snapshot_download('qwen/Qwen2-7B-Instruct')\\n args = [sys.executable, '-m', 'vllm.entrypoints.openai.api_server', '--model', model_dir]\\n process = subprocess.Popen(args)\\n _test_client()\\n process.terminate()\\n\\n\\nif __name__ == '__main__':\\n # test_vllm_orgin()\\n # test_vllm()\\n # test_lmdeploy()\\n test_pt()\\n\\n\\nimport os\\n\\n\\ndef test_eval_llm():\\n os.environ['CUDA_VISIBLE_DEVICES'] = '0'\\n from swift.llm import eval_main, EvalArguments\\n eval_main(EvalArguments(model_type='qwen1half-7b-chat', eval_dataset='ARC_c', infer_backend='lmdeploy'))\\n\\n\\ndef test_eval_vlm():\\n os.environ['CUDA_VISIBLE_DEVICES'] = '1'\\n from swift.llm import eval_main, EvalArguments\\n eval_main(EvalArguments(model_type='internvl2-4b', eval_dataset='RealWorldQA', infer_backend='lmdeploy'))\\n\\n\\ndef test_pt():\\n os.environ['CUDA_VISIBLE_DEVICES'] = '0'\\n from swift.llm import pt_main, PtArguments\\n pt_main(PtArguments(model_type='qwen-1_8b-chat', dataset='alpaca-zh#100', sft_type='lora', tuner_backend='swift'))\\n\\n\\ndef test_vlm_sft():\\n # lora full\\n os.environ['CUDA_VISIBLE_DEVICES'] = '0,1,2,3'\\n from swift.llm import sft_main, SftArguments, infer_main, InferArguments\\n model_type = 'phi3_5-vision-instruct'\\n dataset = 'coco-en-mini#100'\\n infer_main(InferArguments(model_type=model_type, val_dataset=dataset, infer_backend='pt', show_dataset_sample=2))\\n\\n output = sft_main(SftArguments(model_type=model_type, dataset=dataset))\\n last_model_checkpoint = output['last_model_checkpoint']\\n infer_main(\\n InferArguments(ckpt_dir=last_model_checkpoint, load_dataset_config=True, merge_lora=True, infer_backend='pt'))\\n\\n\\ndef test_llm_sft():\\n os.environ['CUDA_VISIBLE_DEVICES'] = '0,1,2,3'\\n from swift.llm import sft_main, SftArguments, infer_main, InferArguments\\n model_type = 'phi3_5-moe-instruct'\\n dataset = 'alpaca-zh#100'\\n infer_main(InferArguments(model_type=model_type, val_dataset=dataset, infer_backend='pt', show_dataset_sample=2))\\n\\n output = sft_main(SftArguments(model_type=model_type, dataset=dataset))\\n last_model_checkpoint = output['last_model_checkpoint']\\n infer_main(\\n InferArguments(ckpt_dir=last_model_checkpoint, load_dataset_config=True, merge_lora=True, infer_backend='pt'))\\n\\n\\ndef test_vlm_dpo():\\n # lora, full, stream\\n os.environ['CUDA_VISIBLE_DEVICES'] = '0,1,2,3'\\n from swift.llm import rlhf_main, RLHFArguments, infer_main, InferArguments\\n model_type = 'internvl2-2b'\\n dataset = 'rlaif-v#100'\\n\\n output = rlhf_main(RLHFArguments(model_type=model_type, dataset=dataset, max_length=8192, sft_type='full'))\\n last_model_checkpoint = output['last_model_checkpoint']\\n infer_main(InferArguments(ckpt_dir=last_model_checkpoint, load_dataset_config=True))\\n\\n\\nif __name__ == '__main__':\\n # test_eval_llm()\\n # test_eval_vlm()\\n # test_pt()\\n # test_vlm_sft()\\n # test_llm_sft()\\n test_vlm_dpo()\\n\\n\\nimport os\\n\\nfrom swift.llm import ExportArguments, InferArguments, SftArguments, export_main, infer_main, sft_main\\n\\nmodel_type = 'qwen1half-7b-chat'\\ntp = 2\\n\\n\\ndef convert2megatron():\\n os.environ['CUDA_VISIBLE_DEVICES'] = '0'\\n export_main(ExportArguments(model_type=model_type, to_megatron=True, tp=tp, dtype='bf16'))\\n\\n\\ndef convert2hf():\\n os.environ['CUDA_VISIBLE_DEVICES'] = '0'\\n export_main(ExportArguments(ckpt_dir=f'{model_type}-tp{tp}-pp1', to_hf=True, dtype='bf16'))\\n\\n\\ndef sft():\\n os.environ['CUDA_VISIBLE_DEVICES'] = '0,1'\\n sft_main(\\n SftArguments(resume_from_checkpoint=f'{model_type}-tp{tp}-pp1', dataset='alpaca-zh', train_backend='megatron'))\\n\\n\\ndef infer():\\n os.environ['CUDA_VISIBLE_DEVICES'] = '0'\\n infer_main(InferArguments(model_type=model_type, model_id_or_path=f'{model_type}-tp{tp}-pp1/{model_type}-hf'))\\n\\n\\nif __name__ == '__main__':\\n convert2megatron()\\n # convert2hf()\\n # infer()\\n # sft()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n\\nfrom swift.aigc import train_dreambooth_lora_sdxl\\n\\nif __name__ == '__main__':\\n train_dreambooth_lora_sdxl()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n\\nfrom swift.aigc import infer_dreambooth_lora_sdxl\\n\\nif __name__ == '__main__':\\n infer_dreambooth_lora_sdxl()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n\\nfrom swift.aigc import train_controlnet\\n\\nif __name__ == '__main__':\\n train_controlnet()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\nfrom swift.aigc import infer_text_to_image_lora\\n\\nif __name__ == '__main__':\\n infer_text_to_image_lora()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n\\nfrom swift.aigc import train_text_to_image_sdxl\\n\\nif __name__ == '__main__':\\n train_text_to_image_sdxl()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n\\nfrom swift.aigc import train_text_to_image_lora_sdxl\\n\\nif __name__ == '__main__':\\n train_text_to_image_lora_sdxl()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n\\nfrom swift.aigc import infer_dreambooth_lora\\n\\nif __name__ == '__main__':\\n infer_dreambooth_lora()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n\\nfrom swift.aigc import infer_controlnet\\n\\nif __name__ == '__main__':\\n infer_controlnet()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\nfrom swift.aigc import infer_text_to_image\\n\\nif __name__ == '__main__':\\n infer_text_to_image()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n\\nfrom swift.aigc import infer_dreambooth\\n\\nif __name__ == '__main__':\\n infer_dreambooth()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\nfrom swift.aigc import infer_text_to_image_sdxl\\n\\nif __name__ == '__main__':\\n infer_text_to_image_sdxl()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n\\nfrom swift.aigc import train_text_to_image_lora\\n\\nif __name__ == '__main__':\\n train_text_to_image_lora()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n\\nfrom swift.aigc import train_dreambooth_lora\\n\\nif __name__ == '__main__':\\n train_dreambooth_lora()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n\\nfrom swift.aigc import train_dreambooth\\n\\nif __name__ == '__main__':\\n train_dreambooth()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n\\nfrom swift.aigc import train_controlnet_sdxl\\n\\nif __name__ == '__main__':\\n train_controlnet_sdxl()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\nfrom swift.aigc import infer_text_to_image_lora_sdxl\\n\\nif __name__ == '__main__':\\n infer_text_to_image_lora_sdxl()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n\\nfrom swift.aigc import infer_controlnet_sdxl\\n\\nif __name__ == '__main__':\\n infer_controlnet_sdxl()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n\\nfrom swift.aigc import train_text_to_image\\n\\nif __name__ == '__main__':\\n train_text_to_image()\\n\\n\\n

Stable Diffusion Example

\\n\\n

\\nModelscope Hub\\n
\\n 中文  ｜  English\\n

\\n\\n## Features\\n1. Support Stable Diffusion [LoRA](https://arxiv.org/abs/2106.09685) method.\\n2. Support Stable Diffusion XL [LoRA](https://arxiv.org/abs/2106.09685) method.\\n\\n## Prepare the Environment\\n```bash\\npip install -r requirements.txt\\n```\\n\\n## Train and Inference\\n```bash\\n# Clone the repository and enter the code directory.\\ngit clone https://github.com/modelscope/swift.git\\n\\n# Stable Diffusion LoRA\\nbash examples/pytorch/stable_diffusion/run_train_lora.sh\\n\\n# Stable Diffusion XL LoRA\\nbash examples/pytorch/stable_diffusion/run_train_lora_xl.sh\\n```\\n\\n## Extend Datasets\\nThe [buptwq/lora-stable-diffusion-finetune](https://www.modelscope.cn/datasets/buptwq/lora-stable-diffusion-finetune/summary) dataset used in the example is from [ModelScope Hub](https://www.modelscope.cn/my/overview), you can replace different datasets ID by modifying the `train_dataset_name` parameter.\\nIn addition, you can also use local datasets. Fill in the path of the dataset file in `train_dataset_name` parameter, which needs to include a `train.csv` file to map image files and text prompts. Please organize it into the following format:\\n```\\nText,Target:FILE\\n[prompt], [image dir]\\n......\\n```\\n\\nHere is an example of `train.csv` file:\\n```\\nText,Target:FILE\\na dog,target/00.jpg\\na dog,target/01.jpg\\na dog,target/02.jpg\\na dog,target/03.jpg\\na dog,target/04.jpg\\n```\\n\\n\\nimport os\\nfrom dataclasses import dataclass, field\\n\\nimport cv2\\nimport torch\\nfrom modelscope import get_logger, snapshot_download\\nfrom modelscope.metainfo import Trainers\\nfrom modelscope.models import Model\\nfrom modelscope.msdatasets import MsDataset\\nfrom modelscope.pipelines import pipeline\\nfrom modelscope.trainers import build_trainer\\nfrom modelscope.trainers.training_args import TrainingArgs\\nfrom modelscope.utils.constant import DownloadMode, Tasks\\n\\nfrom swift import LoRAConfig, Swift\\n\\nlogger = get_logger()\\n\\n\\n# Load configuration file and dataset\\n@dataclass(init=False)\\nclass StableDiffusionLoraArguments(TrainingArgs):\\n prompt: str = field(\\n default='dog', metadata={\\n 'help': 'The pipeline prompt.',\\n })\\n\\n lora_rank: int = field(\\n default=4, metadata={\\n 'help': 'The rank size of lora intermediate linear.',\\n })\\n\\n lora_alpha: int = field(\\n default=32, metadata={\\n 'help': 'The factor to add the lora weights',\\n })\\n\\n lora_dropout: float = field(\\n default=0.0, metadata={\\n 'help': 'The dropout rate of the lora module',\\n })\\n\\n bias: str = field(\\n default='none', metadata={\\n 'help': 'Bias type. Values ca be \\\"none\\\", \\\"all\\\" or \\\"lora_only\\\"',\\n })\\n\\n sample_nums: int = field(\\n default=10, metadata={\\n 'help': 'The numbers of sample outputs',\\n })\\n\\n num_inference_steps: int = field(\\n default=50, metadata={\\n 'help': 'The number of denoising steps.',\\n })\\n\\n\\ntraining_args = StableDiffusionLoraArguments(task='text-to-image-synthesis').parse_cli()\\nconfig, args = training_args.to_config()\\n\\nif os.path.exists(args.train_dataset_name):\\n # Load local dataset\\n train_dataset = MsDataset.load(args.train_dataset_name)\\n validation_dataset = MsDataset.load(args.train_dataset_name)\\nelse:\\n # Load online dataset\\n train_dataset = MsDataset.load(args.train_dataset_name, split='train', download_mode=DownloadMode.FORCE_REDOWNLOAD)\\n validation_dataset = MsDataset.load(\\n args.train_dataset_name, split='validation', download_mode=DownloadMode.FORCE_REDOWNLOAD)\\n\\n\\ndef cfg_modify_fn(cfg):\\n if args.use_model_config:\\n cfg.merge_from_dict(config)\\n else:\\n cfg = config\\n cfg.train.lr_scheduler = {'type': 'LambdaLR', 'lr_lambda': lambda _: 1, 'last_epoch': -1}\\n return cfg\\n\\n\\n# build models\\nmodel = Model.from_pretrained(training_args.model, revision=args.model_revision)\\nmodel_dir = snapshot_download(args.model)\\nlora_config = LoRAConfig(\\n r=args.lora_rank,\\n lora_alpha=args.lora_alpha,\\n lora_dropout=args.lora_dropout,\\n bias=args.bias,\\n target_modules=['to_q', 'to_k', 'to_v', 'query', 'key', 'value', 'to_out.0'])\\nmodel.unet = Swift.prepare_model(model.unet, lora_config)\\n\\n# build trainer and training\\nkwargs = dict(\\n model=model,\\n cfg_file=os.path.join(model_dir, 'configuration.json'),\\n work_dir=training_args.work_dir,\\n train_dataset=train_dataset,\\n eval_dataset=validation_dataset,\\n torch_type=torch.float32,\\n use_swift=True,\\n cfg_modify_fn=cfg_modify_fn)\\n\\ntrainer = build_trainer(name=Trainers.stable_diffusion, default_args=kwargs)\\ntrainer.train()\\n\\n# save models\\nmodel.unet.save_pretrained(os.path.join(training_args.work_dir, 'unet'))\\nlogger.info(f'model save pretrained {training_args.work_dir}')\\n\\n# pipeline after training and save result\\npipe = pipeline(\\n task=Tasks.text_to_image_synthesis,\\n model=training_args.model,\\n model_revision=args.model_revision,\\n lora_dir=os.path.join(training_args.work_dir, 'unet'),\\n use_swift=True)\\n\\nfor index in range(args.sample_nums):\\n image = pipe({'text': args.prompt, 'num_inference_steps': args.num_inference_steps})\\n cv2.imwrite(f'./lora_result_{index}.png', image['output_imgs'][0])\\n\\n\\n

微调稳定扩散模型例子

\\n\\n

\\n魔搭社区\\n
\\n 中文  ｜  English\\n

\\n\\n\\n## 特性\\n1. 支持[LoRA](https://arxiv.org/abs/2106.09685)方法微调稳定扩散模型。\\n2. 支持[LoRA](https://arxiv.org/abs/2106.09685)方法微调XL版本的稳定扩散模型。\\n\\n## 环境准备\\n```bash\\npip install -r requirements.txt\\n```\\n\\n## 训练和推理\\n```bash\\n# 克隆代码库并进入代码目录\\ngit clone https://github.com/modelscope/swift.git\\n\\n# LoRA方法微调和推理稳定扩散模型\\nbash examples/pytorch/stable_diffusion/run_train_lora.sh\\n\\n# LoRA方法微调和推理XL版本的稳定扩散模型\\nbash examples/pytorch/stable_diffusion/run_train_lora_xl.sh\\n```\\n\\n## 数据集拓展\\n示例中使用的数据集[buptwq/lora-stable-diffusion-finetune](https://www.modelscope.cn/datasets/buptwq/lora-stable-diffusion-finetune/summary)来自[ModelScope Hub](https://www.modelscope.cn/my/overview)，您可以在ModelScope Hub选择其他数据集，用被选择的数据集ID来修改`train_dataset_name`参数。\\n\\n除此之外，您也可以使用本地数据集。请用本地数据集路径修改`train_dataset_name`参数，请注意在本地数据集路径中应该包含一个`train.csv`文件用来映射图片和文本提示词。`train.csv`文件请参照以下的格式：\\n```\\nText,Target:FILE\\n[提示词], [图片路径]\\n......\\n```\\n下面是一个 `train.csv` 文件的例子:\\n```\\nText,Target:FILE\\na dog,target/00.jpg\\na dog,target/01.jpg\\na dog,target/02.jpg\\na dog,target/03.jpg\\na dog,target/04.jpg\\n```\\n\\n\\ngit+https://github.com/modelscope/modelscope.git\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\nimport custom\\n\\nfrom swift.llm import sft_main\\n\\nif __name__ == '__main__':\\n output = sft_main()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\nfrom typing import Any, Dict\\n\\nimport torch\\nfrom datasets import Dataset as HfDataset\\nfrom transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer\\nfrom transformers.utils.versions import require_version\\n\\nfrom swift.llm import (LoRATM, Template, TemplateType, dataset_map, get_dataset, get_dataset_from_repo,\\n get_model_tokenizer, get_template, print_example, register_dataset, register_model,\\n register_template)\\nfrom swift.utils import get_logger\\n\\nlogger = get_logger()\\n\\n\\nclass CustomModelType:\\n tigerbot_7b = 'tigerbot-7b'\\n tigerbot_13b = 'tigerbot-13b'\\n tigerbot_13b_chat = 'tigerbot-13b-chat'\\n\\n\\nclass CustomTemplateType:\\n tigerbot = 'tigerbot'\\n\\n\\nclass CustomDatasetName:\\n stsb_en = 'stsb-en'\\n\\n\\n@register_model(CustomModelType.tigerbot_7b, 'TigerResearch/tigerbot-7b-base-v3', LoRATM.llama2,\\n TemplateType.default_generation)\\n@register_model(CustomModelType.tigerbot_13b, 'TigerResearch/tigerbot-13b-base-v2', LoRATM.llama2,\\n TemplateType.default_generation)\\n@register_model(CustomModelType.tigerbot_13b_chat, 'TigerResearch/tigerbot-13b-chat-v4', LoRATM.llama2,\\n CustomTemplateType.tigerbot)\\ndef get_tigerbot_model_tokenizer(model_dir: str,\\n torch_dtype: torch.dtype,\\n model_kwargs: Dict[str, Any],\\n load_model: bool = True,\\n **kwargs):\\n use_flash_attn = kwargs.pop('use_flash_attn', False)\\n if use_flash_attn:\\n require_version('transformers>=4.34')\\n logger.info('Setting use_flash_attention_2: True')\\n model_kwargs['use_flash_attention_2'] = True\\n model_config = AutoConfig.from_pretrained(model_dir, trust_remote_code=True)\\n model_config.pretraining_tp = 1\\n model_config.torch_dtype = torch_dtype\\n logger.info(f'model_config: {model_config}')\\n tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)\\n model = None\\n if load_model:\\n model = AutoModelForCausalLM.from_pretrained(\\n model_dir, config=model_config, torch_dtype=torch_dtype, trust_remote_code=True, **model_kwargs)\\n return model, tokenizer\\n\\n\\n# Ref: https://github.com/TigerResearch/TigerBot/blob/main/infer.py\\nregister_template(\\n CustomTemplateType.tigerbot,\\n Template(['{{SYSTEM}}'], ['\\\\n\\\\n### Instruction:\\\\n{{QUERY}}\\\\n\\\\n### Response:\\\\n'], [], [['eos_token_id']]))\\n\\n\\ndef _preprocess_stsb(dataset: HfDataset) -> HfDataset:\\n prompt = \\\"\\\"\\\"Task: Based on the given two sentences, provide a similarity score between 0.0 and 5.0.\\nSentence 1: {text1}\\nSentence 2: {text2}\\nSimilarity score: \\\"\\\"\\\"\\n query = []\\n response = []\\n for d in dataset:\\n query.append(prompt.format(text1=d['text1'], text2=d['text2']))\\n response.append(f\\\"{d['label']:.1f}\\\")\\n return HfDataset.from_dict({'query': query, 'response': response})\\n\\n\\nregister_dataset(CustomDatasetName.stsb_en, 'swift/stsb', None, _preprocess_stsb, get_dataset_from_repo)\\n\\nif __name__ == '__main__':\\n # The Shell script can view `examples/pytorch/llm/scripts/custom`.\\n # test dataset\\n train_dataset, val_dataset = get_dataset([CustomDatasetName.stsb_en], check_dataset_strategy='warning')\\n print(f'train_dataset: {train_dataset}')\\n print(f'val_dataset: {val_dataset}')\\n # test model base\\n model, tokenizer = get_model_tokenizer(CustomModelType.tigerbot_13b, use_flash_attn=False)\\n # test model chat\\n model, tokenizer = get_model_tokenizer(CustomModelType.tigerbot_13b_chat, use_flash_attn=False)\\n # test template\\n template = get_template(CustomTemplateType.tigerbot, tokenizer)\\n train_dataset = dataset_map(train_dataset, template.encode)\\n print_example(train_dataset[0], tokenizer)\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\nimport custom\\n\\nfrom swift.llm import infer_main\\n\\nif __name__ == '__main__':\\n result = infer_main()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n\\\"\\\"\\\"\\nThis script runs the main process of Human Preferences Alignment using different algorithms.\\nSpecify the algorithm using the --rlhf_type argument.\\n\\nOptions:\\n --rlhf_type dpo : Direct Preference Optimization (DPO)\\n --rlhf_type kto : Kahneman-Tversky Optimization (KTO)\\n --rlhf_type cpo : Contrastive Preference Optimization (CPO)\\n --rlhf_type simpo : Simple Preference Optimization (SimPO)\\n --rlhf_type orpo : Odds Ratio Preference Optimization (ORPO)\\n\\\"\\\"\\\"\\nimport custom\\n\\nfrom swift.llm import rlhf_main\\n\\nif __name__ == '__main__':\\n output = rlhf_main()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n# import os\\n# os.environ['CUDA_VISIBLE_DEVICES'] = '0'\\nimport custom\\n\\nfrom swift.llm import AppUIArguments, ModelType, app_ui_main\\n\\nif __name__ == '__main__':\\n # Please refer to the `infer.sh` for setting the parameters.\\n # text-generation\\n # args = AppUIArguments(model_type=ModelType.chatglm3_6b_base)\\n # or chat\\n args = AppUIArguments(model_type=ModelType.qwen_7b_chat_int4)\\n # or load from ckpt dir\\n # args = AppUIArguments(ckpt_dir='xxx/vx-xxx/checkpoint-xxx')\\n app_ui_main(args)\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n\\nfrom swift.llm import rome_main\\n\\nif __name__ == '__main__':\\n rome_main()\\n\\n\\n[\\n {\\n \\\"name\\\": \\\"custom_general_qa\\\",\\n \\\"pattern\\\": \\\"general_qa\\\",\\n \\\"dataset\\\": \\\"eval_example/custom_general_qa\\\",\\n \\\"subset_list\\\": [\\\"default\\\"]\\n },\\n {\\n \\\"name\\\": \\\"custom_ceval\\\",\\n \\\"pattern\\\": \\\"ceval\\\",\\n \\\"dataset\\\": \\\"eval_example/custom_ceval\\\",\\n \\\"subset_list\\\": [\\\"default\\\"]\\n }\\n]\\n\\n\\nid,question,A,B,C,D,answer,explanation\\n1,要把膜蛋白脂完整地从膜上溶解下来，可以用____。,蛋白水解酶,透明质酸酶,去垢剂,糖苷水解酶,C,\\\"1. 首先，我们需要知道如何将膜蛋白脂从膜上溶解下来。膜蛋白脂是由蛋白质和脂质组成的复合物，因此需要使用一种能够破坏脂质的物质来将其溶解。\\n2. 接着，我们分析选项：(A) 蛋白水解酶是一种能够水解蛋白质的酶，会破坏膜蛋白脂的结构；(B) 透明质酸酶是一种能够水解透明质酸的酶，与膜蛋白脂无关；(C) 去垢剂是一种能够破坏脂质的物质，可以将膜蛋白脂完整地从膜上溶解下来；(D) 糖苷水解酶是一种能够水解糖苷的酶，与膜蛋白脂无关。\\\"\\n2,不连续聚丙烯酰胺凝胶电泳比一般电泳的分辨率高，是因为具有____。,浓缩效应,电荷效应,分子筛效应,黏度效应,A,\\\"1. 浓缩效应是指由于凝胶孔径等的不连续性，样本物质被浓缩为一个狭窄的中间层，是不连续聚丙烯酰胺凝胶电泳分辨率高的主要原因。\\n2. 电荷效应和分子筛效应不是不连续聚丙烯酰胺凝胶电泳特有的效应；黏度效应与分辨率提高没有直接关系。\\\"\\n\\n\\nid,question,A,B,C,D,answer,explanation\\n1,通常来说，组成动物蛋白质的氨基酸有____,4种,22种,20种,19种,C,1. 目前已知构成动物蛋白质的的氨基酸有20种。\\n2,血液内存在的下列物质中，不属于代谢终产物的是____。,尿素,尿酸,丙酮酸,二氧化碳,C,\\\"1. 代谢终产物是指在生物体内代谢过程中产生的无法再被利用的物质，需要通过排泄等方式从体内排出。\\n2. 丙酮酸是糖类代谢的产物，可以被进一步代谢为能量或者合成其他物质，并非代谢终产物。\\\"\\n\\n\\n{\\n \\\"model.embed_tokens\\\": \\\"cuda:0\\\",\\n \\\"model.layers.0\\\": \\\"cuda:0\\\",\\n \\\"model.layers.1\\\": \\\"cuda:0\\\",\\n \\\"model.layers.2\\\": \\\"cuda:0\\\",\\n \\\"model.layers.3\\\": \\\"cuda:0\\\",\\n \\\"model.layers.4\\\": \\\"cuda:0\\\",\\n \\\"model.layers.5\\\": \\\"cuda:0\\\",\\n \\\"model.layers.6\\\": \\\"cuda:0\\\",\\n \\\"model.layers.7\\\": \\\"cuda:1\\\",\\n \\\"model.layers.8\\\": \\\"cuda:1\\\",\\n \\\"model.layers.9\\\": \\\"cuda:1\\\",\\n \\\"model.layers.10\\\": \\\"cuda:1\\\",\\n \\\"model.layers.11\\\": \\\"cuda:1\\\",\\n \\\"model.layers.12\\\": \\\"cuda:1\\\",\\n \\\"model.layers.13\\\": \\\"cuda:1\\\",\\n \\\"model.layers.14\\\": \\\"cuda:2\\\",\\n \\\"model.layers.15\\\": \\\"cuda:2\\\",\\n \\\"model.layers.16\\\": \\\"cuda:2\\\",\\n \\\"model.layers.17\\\": \\\"cuda:2\\\",\\n \\\"model.layers.18\\\": \\\"cuda:2\\\",\\n \\\"model.layers.19\\\": \\\"cuda:2\\\",\\n \\\"model.layers.20\\\": \\\"cuda:2\\\",\\n \\\"model.layers.21\\\": \\\"cuda:3\\\",\\n \\\"model.layers.22\\\": \\\"cuda:3\\\",\\n \\\"model.layers.23\\\": \\\"cuda:3\\\",\\n \\\"model.layers.24\\\": \\\"cuda:3\\\",\\n \\\"model.layers.25\\\": \\\"cuda:3\\\",\\n \\\"model.layers.26\\\": \\\"cuda:3\\\",\\n \\\"model.layers.27\\\": \\\"cuda:3\\\",\\n \\\"model.layers.28\\\": \\\"cuda:4\\\",\\n \\\"model.layers.29\\\": \\\"cuda:4\\\",\\n \\\"model.layers.30\\\": \\\"cuda:4\\\",\\n \\\"model.layers.31\\\": \\\"cuda:4\\\",\\n \\\"model.layers.32\\\": \\\"cuda:4\\\",\\n \\\"model.layers.33\\\": \\\"cuda:4\\\",\\n \\\"model.layers.34\\\": \\\"cuda:4\\\",\\n \\\"model.layers.35\\\": \\\"cuda:4\\\",\\n \\\"model.layers.36\\\": \\\"cuda:5\\\",\\n \\\"model.layers.37\\\": \\\"cuda:5\\\",\\n \\\"model.layers.38\\\": \\\"cuda:5\\\",\\n \\\"model.layers.39\\\": \\\"cuda:5\\\",\\n \\\"model.layers.40\\\": \\\"cuda:5\\\",\\n \\\"model.layers.41\\\": \\\"cuda:5\\\",\\n \\\"model.layers.42\\\": \\\"cuda:5\\\",\\n \\\"model.layers.43\\\": \\\"cuda:5\\\",\\n \\\"model.layers.44\\\": \\\"cuda:6\\\",\\n \\\"model.layers.45\\\": \\\"cuda:6\\\",\\n \\\"model.layers.46\\\": \\\"cuda:6\\\",\\n \\\"model.layers.47\\\": \\\"cuda:6\\\",\\n \\\"model.layers.48\\\": \\\"cuda:6\\\",\\n \\\"model.layers.49\\\": \\\"cuda:6\\\",\\n \\\"model.layers.50\\\": \\\"cuda:6\\\",\\n \\\"model.layers.51\\\": \\\"cuda:6\\\",\\n \\\"model.layers.52\\\": \\\"cuda:7\\\",\\n \\\"model.layers.53\\\": \\\"cuda:7\\\",\\n \\\"model.layers.54\\\": \\\"cuda:7\\\",\\n \\\"model.layers.55\\\": \\\"cuda:7\\\",\\n \\\"model.layers.56\\\": \\\"cuda:7\\\",\\n \\\"model.layers.57\\\": \\\"cuda:7\\\",\\n \\\"model.layers.58\\\": \\\"cuda:7\\\",\\n \\\"model.layers.59\\\": \\\"cuda:7\\\",\\n \\\"model.norm\\\": \\\"cuda:7\\\",\\n \\\"lm_head\\\": \\\"cuda:7\\\"\\n}\\n\\n\\n{\\n \\\"compute_environment\\\": \\\"LOCAL_MACHINE\\\",\\n \\\"debug\\\": false,\\n \\\"distributed_type\\\": \\\"FSDP\\\",\\n \\\"downcast_bf16\\\": \\\"no\\\",\\n \\\"fsdp_config\\\": {\\n \\\"fsdp_auto_wrap_policy\\\": \\\"TRANSFORMER_BASED_WRAP\\\",\\n \\\"fsdp_backward_prefetch\\\": \\\"BACKWARD_PRE\\\",\\n \\\"fsdp_cpu_ram_efficient_loading\\\": true,\\n \\\"fsdp_forward_prefetch\\\": false,\\n \\\"fsdp_offload_params\\\": true,\\n \\\"fsdp_sharding_strategy\\\": \\\"FULL_SHARD\\\",\\n \\\"fsdp_state_dict_type\\\": \\\"SHARDED_STATE_DICT\\\",\\n \\\"fsdp_transformer_layer_cls_to_wrap\\\": \\\"Qwen2DecoderLayer\\\",\\n \\\"fsdp_sync_module_states\\\": true,\\n \\\"fsdp_use_orig_params\\\": true\\n },\\n \\\"machine_rank\\\": 0,\\n \\\"main_training_function\\\": \\\"main\\\",\\n \\\"mixed_precision\\\": \\\"bf16\\\",\\n \\\"num_machines\\\": 1,\\n \\\"num_processes\\\": 4,\\n \\\"rdzv_backend\\\": \\\"static\\\",\\n \\\"same_network\\\": true,\\n \\\"tpu_env\\\": [],\\n \\\"tpu_use_cluster\\\": false,\\n \\\"tpu_use_sudo\\\": false,\\n \\\"use_cpu\\\": false\\n}\\n\\n\\n{\\n \\\"compute_environment\\\": \\\"LOCAL_MACHINE\\\",\\n \\\"debug\\\": false,\\n \\\"distributed_type\\\": \\\"FSDP\\\",\\n \\\"downcast_bf16\\\": \\\"no\\\",\\n \\\"fsdp_config\\\": {\\n \\\"fsdp_auto_wrap_policy\\\": \\\"TRANSFORMER_BASED_WRAP\\\",\\n \\\"fsdp_backward_prefetch\\\": \\\"BACKWARD_PRE\\\",\\n \\\"fsdp_cpu_ram_efficient_loading\\\": true,\\n \\\"fsdp_forward_prefetch\\\": false,\\n \\\"fsdp_offload_params\\\": true,\\n \\\"fsdp_sharding_strategy\\\": \\\"FULL_SHARD\\\",\\n \\\"fsdp_state_dict_type\\\": \\\"FULL_STATE_DICT\\\",\\n \\\"fsdp_sync_module_states\\\": true,\\n \\\"fsdp_use_orig_params\\\": false\\n },\\n \\\"machine_rank\\\": 0,\\n \\\"main_training_function\\\": \\\"main\\\",\\n \\\"mixed_precision\\\": \\\"no\\\",\\n \\\"num_machines\\\": 1,\\n \\\"num_processes\\\": 2,\\n \\\"rdzv_backend\\\": \\\"static\\\",\\n \\\"same_network\\\": true,\\n \\\"tpu_env\\\": [],\\n \\\"tpu_use_cluster\\\": false,\\n \\\"tpu_use_sudo\\\": false,\\n \\\"use_cpu\\\": false\\n}\\n\\n\\n[\\n {\\n \\\"prompt\\\": \\\"{} was the founder of\\\",\\n \\\"subject\\\": \\\"Steve Jobs\\\",\\n \\\"target\\\": \\\"Microsoft\\\"\\n },\\n {\\n \\\"prompt\\\": \\\"{} is located in\\\",\\n \\\"subject\\\": \\\"HangZhou\\\",\\n \\\"target\\\": \\\"Africa\\\"\\n }\\n]\\n\\n\\nimport os\\n\\nimport cv2\\nfrom modelscope.models import Model\\nfrom modelscope.msdatasets import MsDataset\\nfrom modelscope.pipelines import pipeline\\nfrom modelscope.trainers import build_trainer\\nfrom modelscope.utils.config import Config\\nfrom modelscope.utils.constant import ModelFile\\n\\nfrom swift import LoRAConfig, SCETuningConfig, Swift, snapshot_download\\n\\n# load dataset\\ntrain_dataset = MsDataset.load(\\n 'style_custom_dataset', namespace='damo', subset_name='3D',\\n split='train_short').remap_columns({'Image:FILE': 'Target:FILE'})\\n\\n# load pretrained model\\nmodel_id = 'damo/multi-modal_efficient-diffusion-tuning-swift-base'\\ntask = 'efficient-diffusion-tuning'\\nrevision = 'v1.0.1'\\nmodel_dir = snapshot_download(model_id)\\ncfg_dict = Config.from_file(os.path.join(model_dir, ModelFile.CONFIGURATION))\\ncfg_dict.model.inference = False\\nmodel = Model.from_pretrained(model_id, cfg_dict=cfg_dict, revision=revision)\\n\\n# init tuner\\ntuner_type = 'scetuning' # \\\"lora\\\"\\n\\nif tuner_type == 'lora':\\n work_dir = 'tmp/multimodal_swift_lora_style'\\n tuner_config = LoRAConfig(r=64, target_modules='.*unet.*.(to_q|to_k|to_v|to_out.0|net.0.proj|net.2)$')\\n model = Swift.prepare_model(model, tuner_config)\\nelif tuner_type == 'scetuning':\\n work_dir = 'tmp/multimodal_swift_scetuning_style'\\n tuner_config = SCETuningConfig(\\n dims=[1280, 1280, 1280, 1280, 1280, 640, 640, 640, 320, 320, 320, 320],\\n target_modules=r'^unet\\\\.up_blocks\\\\.\\\\d+\\\\.resnets\\\\.\\\\d+$')\\n model = Swift.prepare_model(model, tuner_config)\\nelse:\\n raise Exception(f'Error tuner type: {tuner_type}')\\n\\n\\n# training\\ndef cfg_modify_fn(cfg):\\n cfg.preprocessor.resolution = 512\\n cfg.train.lr_scheduler = {'type': 'LambdaLR', 'lr_lambda': lambda _: 1, 'last_epoch': -1}\\n cfg.train.max_epochs = 100\\n cfg.train.optimizer.lr = 1e-4\\n cfg.train.dataloader.batch_size_per_gpu = 10\\n cfg.model.inference = False\\n cfg.model.pretrained_tuner = None\\n trainer_hook = cfg.train.hooks\\n trainer_hook.append({'type': 'SwiftHook'})\\n trainer_hook.append({'type': 'CheckpointHook', 'interval': 50})\\n cfg.train.hooks = trainer_hook\\n return cfg\\n\\n\\nkwargs = dict(\\n model=model,\\n cfg_file=os.path.join(model_dir, 'configuration.json'),\\n work_dir=work_dir,\\n train_dataset=train_dataset,\\n eval_dataset=train_dataset,\\n cfg_modify_fn=cfg_modify_fn)\\ntrainer = build_trainer(name='efficient-diffusion-tuning', default_args=kwargs)\\ntrainer.train()\\n\\n# inference\\nwork_dir = os.path.join(work_dir, 'output_swift')\\nmodel_dir = snapshot_download(model_id)\\ncfg_dict = Config.from_file(os.path.join(model_dir, ModelFile.CONFIGURATION))\\ncfg_dict.model.inference = True\\nmodel = Model.from_pretrained(model_id, cfg_dict=cfg_dict, revision=revision)\\nmodel = Swift.from_pretrained(model, work_dir)\\npipe = pipeline(task='efficient-diffusion-tuning', model=model)\\ntest_prompt = 'A boy in a camouflage jacket with a scarf'\\nimg_out = pipe({'prompt': test_prompt}, num_inference_steps=50, generator_seed=123)['output_imgs'][0]\\ncv2.imwrite(os.path.join(work_dir, 'inference.png'), img_out)\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n\\nfrom swift.aigc import animatediff_main\\n\\nif __name__ == '__main__':\\n animatediff_main()\\n\\n\\n# Copyright (c) Alibaba, Inc. and its affiliates.\\n\\nfrom swift.aigc import animatediff_infer_main\\n\\nif __name__ == '__main__':\\n animatediff_infer_main()\\n\\n\\n## maintain docs\\n1. build docs\\n ```shell\\n # in root directory:\\n make docs\\n ```\\n\\n2. doc string format\\n\\n We adopt the google style docstring format as the standard, please refer to the following documents.\\n 1. Google Python style guide docstring [link](http://google.github.io/styleguide/pyguide.html#381-docstrings)\\n 2. Google docstring example [link](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html)\\n 3. sample：torch.nn.modules.conv [link](https://pytorch.org/docs/stable/_modules/torch/nn/modules/conv.html#Conv1d)\\n 4. load function as an example：\\n\\n ```python\\n def load(file, file_format=None, **kwargs):\\n \\\"\\\"\\\"Load data from json/yaml/pickle files.\\n\\n This method provides a unified api for loading data from serialized files.\\n\\n Args:\\n file (str or :obj:`Path` or file-like object): Filename or a file-like\\n object.\\n file_format (str, optional): If not specified, the file format will be\\n inferred from the file extension, otherwise use the specified one.\\n Currently supported formats include \\\"json\\\", \\\"yaml/yml\\\".\\n\\n Examples:\\n >>> load('/path/of/your/file') # file is stored in disk\\n >>> load('https://path/of/your/file') # file is stored on internet\\n >>> load('oss://path/of/your/file') # file is stored in petrel\\n\\n Returns:\\n The content from the file.\\n \\\"\\\"\\\"\\n ```\\n\\n\\n# Configuration file for the Sphinx documentation builder.\\n#\\n# This file only contains a selection of the most common options. For a full\\n# list see the documentation:\\n# https://www.sphinx-doc.org/en/master/usage/configuration.html\\n\\n# -- Path setup --------------------------------------------------------------\\n\\n# If extensions (or modules to document with autodoc) are in another directory,\\n# add these directories to sys.path here. If the directory is relative to the\\n# documentation root, use os.path.abspath to make it absolute, like shown here.\\n#\\nimport os\\nimport sys\\n\\n# import sphinx_book_theme\\n\\nsys.path.insert(0, os.path.abspath('../../'))\\n# -- Project information -----------------------------------------------------\\n\\nproject = 'swift'\\ncopyright = '2022-2024, Alibaba ModelScope'\\nauthor = 'ModelScope Authors'\\nversion_file = '../../swift/version.py'\\nhtml_theme = 'sphinx_rtd_theme'\\nlanguage = 'en'\\n\\n\\ndef get_version():\\n with open(version_file, 'r', encoding='utf-8') as f:\\n exec(compile(f.read(), version_file, 'exec'))\\n return locals()['__version__']\\n\\n\\n# The full version, including alpha/beta/rc tags\\nversion = get_version()\\nrelease = version\\n\\n# -- General configuration ---------------------------------------------------\\n\\n# Add any Sphinx extension module names here, as strings. They can be\\n# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom\\n# ones.\\nextensions = [\\n 'sphinx.ext.napoleon',\\n 'sphinx.ext.autosummary',\\n 'sphinx.ext.autodoc',\\n 'sphinx.ext.viewcode',\\n 'sphinx_markdown_tables',\\n 'sphinx_copybutton',\\n 'myst_parser',\\n]\\n\\n# build the templated autosummary files\\nautosummary_generate = True\\nnumpydoc_show_class_members = False\\n\\n# Enable overriding of function signatures in the first line of the docstring.\\nautodoc_docstring_signature = True\\n\\n# Disable docstring inheritance\\nautodoc_inherit_docstrings = False\\n\\n# Show type hints in the description\\nautodoc_typehints = 'description'\\n\\n# Add parameter types if the parameter is documented in the docstring\\nautodoc_typehints_description_target = 'documented_params'\\n\\nautodoc_default_options = {\\n 'member-order': 'bysource',\\n}\\n\\n# Add any paths that contain templates here, relative to this directory.\\ntemplates_path = ['_templates']\\n\\n# The suffix(es) of source filenames.\\n# You can specify multiple suffix as a list of string:\\n#\\nsource_suffix = ['.rst', '.md']\\n\\n# The master toctree document.\\nroot_doc = 'index'\\n\\n# List of patterns, relative to source directory, that match files and\\n# directories to ignore when looking for source files.\\n# This pattern also affects html_static_path and html_extra_path.\\nexclude_patterns = ['build', 'source_en/.ipynb_checkpoints', 'source_en/api/generated', 'Thumbs.db', '.DS_Store']\\n# A list of glob-style patterns [1] that are used to find source files.\\n# They are matched against the source file names relative to the source directory,\\n# using slashes as directory separators on all platforms.\\n# The default is **, meaning that all files are recursively included from the source directory.\\n# -- Options for HTML output -------------------------------------------------\\n\\n# The theme to use for HTML and HTML Help pages. See the documentation for\\n# a list of builtin themes.\\n#\\n# html_theme = 'sphinx_book_theme'\\n# html_theme_path = [sphinx_book_theme.get_html_theme_path()]\\n# html_theme_options = {}\\n\\n# Add any paths that contain custom static files (such as style sheets) here,\\n# relative to this directory. They are copied after the builtin static files,\\n# so a file named \\\"default.css\\\" will overwrite the builtin \\\"default.css\\\".\\nhtml_static_path = ['_static']\\n# html_css_files = ['css/readthedocs.css']\\n\\n# -- Options for HTMLHelp output ---------------------------------------------\\n# Output file base name for HTML help builder.\\n\\n# -- Extension configuration -------------------------------------------------\\n# Ignore >>> when copying code\\ncopybutton_prompt_text = r'>>> |\\\\.\\\\.\\\\. '\\ncopybutton_prompt_is_regexp = True\\n\\n# Example configuration for intersphinx: refer to the Python standard library.\\nintersphinx_mapping = {'https://docs.python.org/': None}\\n\\n\\n# Command Line Arguments\\n\\n## Table of Contents\\n\\n- [sft Parameters](#sft-parameters)\\n- [pt Parameters](#pt-parameters)\\n- [rlhf Parameters](#rlhf-parameters)\\n- [infer merge-lora Parameters](#infer-merge-lora-parameters)\\n- [export Parameters](#export-parameters)\\n- [eval Parameters](#eval-parameters)\\n- [app-ui Parameters](#app-ui-parameters)\\n- [deploy Parameters](#deploy-parameters)\\n\\n## sft Parameters\\n- `--🔥model_type`: Represents the selected model type, default is `None`. `model_type` specifies the default `target_modules`, `template_type`, and other information for the corresponding model. You can fine-tune by specifying only `model_type`. The corresponding `model_id_or_path` will use default settings, and the model will be downloaded from ModelScope and use the default cache path. One of model_type and model_id_or_path must be specified. You can see the list of available `model_type` [here](Supported-models-datasets.md#Models). You can set the `USE_HF` environment variable to control downloading models and datasets from the HF Hub, see [HuggingFace Ecosystem Compatibility Documentation](../LLM/Compat-HF.md).\\n- `--🔥model_id_or_path`: Represents the `model_id` in the ModelScope/HuggingFace Hub or a local path for the model, default is `None`. If the provided `model_id_or_path` has already been registered, the `model_type` will be inferred based on the `model_id_or_path`. If it has not been registered, both `model_type` and `model_id_or_path` must be specified, e.g. `--model_type --model_id_or_path `.\\n- `--model_revision`: The version number corresponding to `model_id` on ModelScope Hub, default is `None`. If `model_revision` is `None`, use the revision registered in `MODEL_MAPPING`. Otherwise, force use of the `model_revision` passed from command line.\\n- `--local_repo_path`: Some models rely on a GitHub repo for loading. To avoid network issues during `git clone`, you can directly use the local repo. This parameter requires input of the local repo path, and defaults to `None`. These models include:\\n - mPLUG-Owl model: `https://github.com/X-PLUG/mPLUG-Owl`\\n - DeepSeek-VL model: `https://github.com/deepseek-ai/DeepSeek-VL`\\n - YI-VL model: `https://github.com/01-ai/Yi`\\n - LLAVA model: `https://github.com/haotian-liu/LLaVA.git`\\n- `--🔥sft_type`: Fine-tuning method, default is `'lora'`. Options include: 'lora', 'full', 'longlora', 'adalora', 'ia3', 'llamapro', 'adapter', 'vera', 'boft', 'fourierft', 'reft'. If using qlora, you need to set `--sft_type lora --quantization_bit 4`.\\n- `--packing`: pack the dataset length to `max-length`, default `False`.\\n- `--full_determinism`: Fix all the values in training, default `False`.\\n- `--auto_find_batch_size`: Auto find batch size according to the GPU memory, default `False`.\\n- `--streaming`: Whether to use iterable dataset, Default `False`.\\n- `--freeze_parameters`: When sft_type is specified as 'full', the layers prefixed with freeze_parameters will be frozen. The default value is `[]`. For example: `--freeze_parameters visual`.\\n- `--🔥freeze_vit`: When sft_type is set to 'full' and a multimodal model is being trained, the parameters of vit can be frozen by setting this parameter to True. The default value is `False`.\\n- `--freeze_parameters_ratio`: When sft_type is set to 'full', freeze the bottommost parameters of the model. Range is 0. ~ 1., default is `0.`. This provides a compromise between lora and full fine-tuning.\\n- `--additional_trainable_parameters`: In addition to freeze_parameters, only allowed when sft_type is 'full', default is `[]`. For example, if you want to train embedding layer in addition to 50% of parameters, you can set `--freeze_parameters_ratio 0.5 --additional_trainable_parameters transformer.wte`, all parameters starting with `transformer.wte` will be activated. You can also set `--freeze_parameters_ratio 1 --additional_trainable_parameters xxx` to customize the trainable layers.\\n- `--tuner_backend`: Backend support for lora, qlora, default is `'peft'`. Options include: 'swift', 'peft', 'unsloth'.\\n- `--🔥template_type`: Type of dialogue template used, default is `'AUTO'`, i.e. look up `template` in `MODEL_MAPPING` based on `model_type`. Available `template_type` options can be found in `TEMPLATE_MAPPING.keys()`.\\n- `--🔥output_dir`: Directory to store ckpt, default is `'output'`. We will append `model_type` and fine-tuning version number to this directory, allowing users to do multiple comparative experiments on different models without changing the `output_dir` command line argument. If you don't want to append this content, specify `--add_output_dir_suffix false`.\\n- `--add_output_dir_suffix`: Default is `True`, indicating that a suffix of `model_type` and fine-tuning version number will be appended to the `output_dir` directory. Set to `False` to avoid this behavior.\\n- `--ddp_backend`: Backend support for distributed training, default is `None`. Options include: 'nccl', 'gloo', 'mpi', 'ccl'.\\n- `--ddp_timeout`: DDP timeout. Default `1800` seconds.\\n- `--seed`: Global seed, default is `42`. Used to reproduce training results.\\n- `--🔥resume_from_checkpoint`: Used for resuming training from a checkpoint, default is `None`. You can set it to the path of the checkpoint, for example: `--resume_from_checkpoint output/qwen-7b-chat/vx-xxx/checkpoint-xxx`, to resume training from that point. Supports adjusting `--resume_only_model` to only read the model file during checkpoint continuation.\\n- `--resume_only_model`: Default is `False`, which means strict checkpoint continuation, this will read the weights of the model, optimizer, lr_scheduler, and the random seeds stored on each device, and continue training from the last paused steps. If set to `True`, it will only read the weights of the model.\\n- `--dtype`: torch_dtype when loading base model, default is `'AUTO'`, i.e. intelligently select dtype: if machine does not support bf16, use fp16; if `MODEL_MAPPING` specifies torch_dtype for corresponding model, use its dtype; otherwise use bf16. Options include: 'bf16', 'fp16', 'fp32'.\\n- `--model_kwargs`: Used for passing additional parameters to the multimodal model, for example: `'{\\\"hd_num\\\": 16}'`. You can either pass a JSON string or directly pass a dictionary. The default is `None`. In addition to using this parameter, you can also pass it through environment variables, for example: `HD_NUM=16`.\\n- `--🔥dataset`: Used to select the training dataset, default is `[]`. You can see the list of available datasets [here](Supported-models-datasets.md#Datasets). If you need to train with multiple datasets, you can use ',' or ' ' to separate them, for example: `--dataset alpaca-en,alpaca-zh` or `--dataset alpaca-en alpaca-zh`. It supports Modelscope Hub/HuggingFace Hub/local paths, subset selection, and dataset sampling. The specified format for each dataset is as follows: `[HF or MS::]{dataset_name} or {dataset_id} or {dataset_path}[:subset1/subset2/...][#dataset_sample]`. The simplest case requires specifying only dataset_name, dataset_id, or dataset_path. Customizing datasets can be found in the [Customizing and Extending Datasets document](Customization.md#custom-dataset)\\n - Supports MS and HF hub, as well as dataset_sample. For example, 'MS::alpaca-zh#2000', 'HF::jd-sentiment-zh#2000' (the default hub used is controlled by the `USE_UF` environment variable, default is MS).\\n - More fine-grained control over subsets: It uses the subsets specified during registration by default (if not specified during registration, it uses 'default'). For example, 'sharegpt-gpt4'. If subsets are specified, it uses the corresponding subset of the dataset. For example, 'sharegpt-gpt4:default/V3_format#2000'. Here, the `default` and `V3_format` sub-datasets are used, separated by '/', and 2000 entries are selected.\\n - Support for dataset_id. For example, 'AI-ModelScope/alpaca-gpt4-data-zh#2000', 'HF::llm-wizard/alpaca-gpt4-data-zh#2000', 'hurner/alpaca-gpt4-data-zh#2000', 'HF::shibing624/alpaca-zh#2000'. If the dataset_id has been registered, it will use the preprocessing function, subsets, split, etc. specified during registration. Otherwise, it will use `SmartPreprocessor`, support 5 dataset formats, and use 'default' subsets, with split set to 'train'. The supported dataset formats can be found in the [Customizing and Extending Datasets document](Customization.md#custom-dataset).\\n - Support for dataset_path. For example, '1.jsonl#5000' (if it is a relative path, it is relative to the running directory).\\n- `--val_dataset`: Specify separate validation datasets with the same format of the `dataset` argument, default is `[]`. If using `val_dataset`, the `dataset_test_ratio` will be ignored.\\n- `--dataset_seed`: The seed used to specify the dataset processing is set by default to `None`, meaning it is designated as the global `seed`. The `dataset_seed` exists in the form of `random_state` and does not affect the global seed.\\n- `--dataset_test_ratio`: Used to specify the ratio for splitting the sub-dataset into training and validation sets. The default value is `0.01`. If `--val_dataset` is set, this parameter becomes ineffective.\\n- `--train_dataset_sample`: The number of samples for the training dataset, default is `-1`, which means using the complete training dataset for training. This parameter is deprecated, please use `--dataset {dataset_name}#{dataset_sample}` instead.\\n- `--val_dataset_sample`: Used to sample the validation set, with a default value of `None`, which automatically selects a suitable number of data samples for validation. If you specify `-1`, the complete validation set is used for validation. This parameter is deprecated and the number of samples in the validation set is controlled by `--dataset_test_ratio` or `--val_dataset {dataset_name}#{dataset_sample}`.\\n- `--🔥system`: System used in dialogue template, default is `None`, i.e. use the model's default system. If set to '', no system is used.\\n- `--tools_prompt`: Select the corresponding tools system prompt for the tools field transformation. The options are ['react_en', 'react_zh', 'toolbench'], which correspond to the English version of ReAct format, Chinese version of ReAct format and the toolbench format, respectively. The default is the English version of the ReAct format. For more information, you can refer to the [Agent Deployment Best Practices](Agent-deployment-best-practices.md).\\n- `--🔥max_length`: Maximum token length, default is `2048`. Avoids OOM issues caused by individual overly long samples. When `--truncation_strategy delete` is specified, samples exceeding max_length will be deleted. When `--truncation_strategy truncation_left` is specified, the leftmost tokens will be truncated: `input_ids[-max_length:]`. If set to -1, no limit.\\n- `--truncation_strategy`: Default is `'delete'` which removes sentences exceeding max_length from dataset. `'truncation_left'` will truncate excess text from the left, which may truncate special tokens and affect performance, not recommended.\\n- `--check_dataset_strategy`: Default is `'none'`, i.e. no checking. If training an LLM model, `'warning'` is recommended as data check strategy. If your training target is sentence classification etc., setting to `'none'` is recommended.\\n\\n- `--custom_train_dataset_path`: Default value is `[]`. This parameter has been deprecated, please use `--dataset {dataset_path}`.\\n- `--custom_val_dataset_path`: Default value is `[]`. This parameter is deprecated. Please use `--val_dataset {dataset_path}` instead.\\n- `--self_cognition_sample`: The number of samples for the self-cognition dataset. Default is `0`. If you set this value to >0, you need to specify `--model_name` and `--model_author` at the same time. This parameter has been deprecated, please use `--dataset self-cognition#{self_cognition_sample}` instead.\\n- `--🔥model_name`: Default value is `[None, None]`. If self-cognition dataset sampling is enabled (i.e., specifying `--dataset self-cognition` or self_cognition_sample>0), you need to provide two values, representing the Chinese and English names of the model, respectively. For example: `--model_name 小黄 'Xiao Huang'`. If you want to learn more, you can refer to the [Self-Cognition Fine-tuning Best Practices](../LLM/Self-cognition-best-practice.md).\\n- `--🔥model_author`: Default is `[None, None]`. If self-cognition dataset sampling is enabled, you need to pass two values, representing the author's Chinese and English names respectively. E.g. `--model_author 魔搭 ModelScope`.\\n- `--quant_method`: Quantization method, default is None. You can choose from 'bnb', 'hqq', 'eetq'.\\n- `--quantization_bit`: Specifies whether to quantize and number of quantization bits, default is `0`, i.e. no quantization. To use 4bit qlora, set `--sft_type lora --quantization_bit 4`.Hqq support 1,2,3,4,8bit, bnb support 4,8bit\\n- `--hqq_axis`: Hqq argument. Axis along which grouping is performed. Supported values are 0 or 1. default is `0`\\n- `--hqq_dynamic_config_path`: Parameters for dynamic configuration. The key is the name tag of the layer and the value is a quantization config. If set, each layer specified by its id will use its dedicated quantization configuration.[ref](https://github.com/mobiusml/hqq?tab=readme-ov-file#custom-quantization-configurations-%EF%B8%8F)\\n- `--bnb_4bit_comp_dtype`: When doing 4bit quantization, we need to dequantize during model forward and backward passes. This specifies the torch_dtype after dequantization. Default is `'AUTO'`, i.e. consistent with `dtype`. Options: 'fp16', 'bf16', 'fp32'. Has no effect when quantization_bit is 0.\\n- `--bnb_4bit_quant_type`: Quantization method for 4bit quantization, default is `'nf4'`. Options: 'nf4', 'fp4'. Has no effect when quantization_bit is 0.\\n- `--bnb_4bit_use_double_quant`: Whether to enable double quantization for 4bit quantization, default is `True`. Has no effect when quantization_bit is 0.\\n- `--bnb_4bit_quant_storage`: Default vlaue `None`.This sets the storage type to pack the quanitzed 4-bit prarams. Has no effect when quantization_bit is 0.\\n- `--🔥target_modules`: Specify lora modules, default is `['DEFAULT']`. If target_modules is passed `'DEFAULT'` or `'AUTO'`, look up `target_modules` in `MODEL_MAPPING` based on `model_type` (The LLM is defaulted to qkv, while the MLLM defaults to all lines in the llm and projector.). If passed `'ALL'`, all Linear layers (excluding head) will be specified as lora modules. If passed `'EMBEDDING'`, Embedding layer will be specified as lora module. If memory allows, setting to 'ALL' is recommended. You can also set `['ALL', 'EMBEDDING']` to specify all Linear and embedding layers as lora modules. This parameter only takes effect when `sft_type` is 'lora'. This argument works when sft_type in lora/vera/boft/ia3/adalora/fourierft.\\n- `--target_regex`: The lora target regex in `Optional[str]`. default is `None`. If this argument is specified, the `target_modules` will have no effect. This argument works when sft_type in lora/vera/boft/ia3/adalora/fourierft.\\n- `--🔥lora_rank`: Default is `8`. Only takes effect when `sft_type` is 'lora'.\\n- `--🔥lora_alpha`: Default is `32`. Only takes effect when `sft_type` is 'lora'.\\n- `--lora_dropout`: Default is `0.05`, only takes effect when `sft_type` is 'lora'.\\n- `--init_lora_weights`: Method to initialize LoRA weights, can be specified as `true`, `false`, `gaussian`, `pissa`, or `pissa_niter_[number of iters]`. Default value `true`.\\n- `--lora_bias_trainable`: Default is `'none'`, options: 'none', 'all'. Set to `'all'` to make all biases trainable.\\n- `--modules_to_save`: Default is `[]`. If you want to train embedding, lm_head, or layer_norm, you can set this parameter, e.g. `--modules_to_save EMBEDDING LN lm_head`. If passed `'EMBEDDING'`, Embedding layer will be added to `modules_to_save`. If passed `'LN'`, `RMSNorm` and `LayerNorm` will be added to `modules_to_save`. This argument works when sft_type in lora/vera/boft/ia3/adalora/fourierft.\\n- `--lora_dtype`: Default is `'AUTO'`, specifies dtype for lora modules. If `AUTO`, follow dtype of original module. Options: 'fp16', 'bf16', 'fp32', 'AUTO'.\\n- `--use_dora`: Default is `False`, whether to use `DoRA`.\\n- `--use_rslora`: Default is `False`, whether to use `RS-LoRA`.\\n- `--neftune_noise_alpha`: The noise coefficient added by `NEFTune` can improve performance of instruction fine-tuning, default is `None`. Usually can be set to 5, 10, 15. See [related paper](https://arxiv.org/abs/2310.05914).\\n- `--neftune_backend`: The backend of `NEFTune`, supported values are `transformers`, `swift`, default is `transformers`.\\n- `--🔥gradient_checkpointing`: Whether to enable gradient checkpointing, default is `True`. This can be used to save memory, although it slightly reduces training speed. Has significant effect when max_length and batch_size are large.\\n- `--🔥deepspeed`: Used to specify the path to the deepspeed configuration file or directly pass JSON formatted configuration information. By default, it is set to `None`, which means deepspeed is not enabled. Deepspeed can save GPU memory. We have written default [ZeRO-2 configuration file](https://github.com/modelscope/swift/blob/main/swift/llm/ds_config/zero2_offload.json), [ZeRO-3 configuration file](https://github.com/modelscope/swift/blob/main/swift/llm/ds_config/zero3.json), [ZeRO-2 Offload configuration file](https://github.com/modelscope/swift/blob/main/swift/llm/ds_config/zero2_offload.json ), and [ZeRO-3 Offload configuration file](https://github.com/modelscope/swift/blob/main/swift/llm/ds_config/zero3_offload.json). You only need to specify 'default-zero2', 'default-zero3', 'zero2-offload', 'zero3-offload'.\\n- `--batch_size`: Batch_size during training, default is `1`. Increasing batch_size can improve GPU utilization, but won't necessarily improve training speed, because within a batch, shorter sentences need to be padded to the length of the longest sentence in the batch, introducing invalid computations.\\n- `--eval_batch_size`: Batch_size during evaluation, default is `None`, i.e. set to 1 when `predict_with_generate` is True, set to `batch_size` when False.\\n- `--🔥num_train_epochs`: Number of epochs to train, default is `1`. If `max_steps >= 0`, this overrides `num_train_epochs`. Usually set to 3 ~ 5.\\n- `--max_steps`: Max_steps for training, default is `-1`. If `max_steps >= 0`, this overrides `num_train_epochs`.\\n- `--optim`: Default is `'adamw_torch'`.\\n- `--adam_beta1`: Default is `0.9`.\\n- `--adam_beta2`: Default is `0.95`.\\n- `--adam_epsilon`: Default is `1e-8`.\\n- `--🔥learning_rate`: Default is `None`, i.e. set to 1e-4 if `sft_type` is lora, set to 1e-5 if `sft_type` is full.\\n- `--weight_decay`: Default is `0.01`.\\n- `--🔥gradient_accumulation_steps`: Gradient accumulation, default is `None`, set to `math.ceil(16 / self.batch_size / world_size)`. `total_batch_size = batch_size * gradient_accumulation_steps * world_size`.\\n- `--max_grad_norm`: Gradient clipping, default is `1`.\\n- `--predict_with_generate`: Whether to use generation for evaluation, default is `False`. If set to False, evaluate using `loss`. If set to True, evaluate using `ROUGE-L` and other metrics. Generative evaluation takes a long time, choose carefully.\\n- `--lr_scheduler_type`: Default is `'cosine'`, options: 'linear', 'cosine', 'constant', etc.\\n- `--warmup_ratio`: Proportion of warmup in total training steps, default is `0.05`.\\n- `--warmup_steps`: The number of warmup steps, default is `0`. If warmup_steps > 0 is set, it overrides warmup_ratio.\\n- `--🔥eval_steps`: Evaluate every this many steps, default is `50`.\\n- `--save_steps`: Save every this many steps, default is `None`, i.e. set to `eval_steps`.\\n- `--🔥save_only_model`: Whether to save only model parameters, without saving intermediate states needed for checkpoint resuming, default is `False`.\\n- `--save_total_limit`: Number of checkpoints to save, default is `2`, i.e. save best and last checkpoint. If set to -1, save all checkpoints.\\n- `--logging_steps`: Print training information (e.g. loss, learning_rate, etc.) every this many steps, default is `5`.\\n- `--dataloader_num_workers`: Default value is `None`. If running on a Windows machine, set it to `0`; otherwise, set it to `1`.\\n- `--push_to_hub`: Whether to sync push trained checkpoint to ModelScope Hub, default is `False`.\\n- `--hub_model_id`: Model_id to push to on ModelScope Hub, default is `None`, i.e. set to `f'{model_type}-{sft_type}'`. You can set this to model_id or repo_name. We will infer user_name based on hub_token. If the remote repository to push to does not exist, a new repository will be created, otherwise the previous repository will be reused. This parameter only takes effect when `push_to_hub` is set to True.\\n- `--hub_token`: SDK token needed for pushing. Can be obtained from [https://modelscope.cn/my/myaccesstoken](https://modelscope.cn/my/myaccesstoken), default is `None`, i.e. obtained from environment variable `MODELSCOPE_API_TOKEN`. This parameter only takes effect when `push_to_hub` is set to True.\\n- `--hub_private_repo`: Whether to set the permission of the pushed model repository on ModelScope Hub to private, default is `False`. This parameter only takes effect when `push__to_hub` is set to True.\\n- `--hub_strategy`: Push strategy, default is `'every_save'`. Options include: 'end', 'every_save', 'checkpoint', 'all_checkpoints'. This parameter shares the same meaning from transformers, and only takes effect when `push_to_hub` is set to True.\\n- `--test_oom_error`: Used to detect whether training will cause OOM, default is `False`. If set to True, will sort the training set in descending order by max_length, easy for OOM testing. This parameter is generally used for testing, use carefully.\\n- `--disable_tqdm`: Whether to disable tqdm, useful when launching script with `nohup`. Default is `False`, i.e. enable tqdm.\\n- `--🔥lazy_tokenize`: If set to False, preprocess all text before `trainer.train()`. If set to True, delay encoding text, reducing preprocessing wait and memory usage, useful when processing large datasets. Default is `None`, i.e. we intelligently choose based on template type, usually set to False for LLM models, set to True for multimodal models (to avoid excessive memory usage from loading images and audio).\\n- `--🔥preprocess_num_proc`: Use multiprocessing when preprocessing dataset (tokenizing text). Default is `1`. Same as `lazy_tokenize` command line argument, used to solve slow preprocessing issue. But this strategy cannot reduce memory usage, so if dataset is huge, `lazy_tokenize` is recommended. Recommended values: 4, 8.\\n- `--🔥use_flash_attn`: Whether to use flash attn, default is `None`. Installation steps for flash_attn can be found at [https://github.com/Dao-AILab/flash-attention](https://github.com/Dao-AILab/flash-attention). Models supporting flash_attn can be found in [LLM Supported Models](Supported-models-datasets.md).\\n- `--ignore_args_error`: Whether to ignore Error thrown by command line parameter errors, default is `False`. Set to True if need to copy code to notebook to run.\\n- `--🔥check_model_is_latest`: Check if model is latest, default is `True`. Set this to `False` if you need to train offline.\\n- `--logging_dir`: Default is `None`. I.e. set to `f'{self.output_dir}/runs'`, representing path to store tensorboard files.\\n- `--report_to`: Default is `['tensorboard']`. You can set `--report_to all` to report to all installed integrations.\\n- `--acc_strategy`: Default is `'token'`, options include: 'token', 'sentence'.\\n- `--save_on_each_node`: Takes effect during multi-machine training, default is `False`.\\n- `--save_strategy`: Strategy for saving checkpoint, default is `'steps'`, options include: 'steps', 'epoch', no'.\\n- `--evaluation_strategy`: Strategy for evaluation, default is `'steps'`, options include: 'steps', 'epoch', no'.\\n- `--save_safetensors`: Default is `True`.\\n- `--include_num_input_tokens_seen`: Default is `False`. Tracks the number of input tokens seen throughout training.\\n- `--max_new_tokens`: Default is `2048`. This parameter only takes effect when `predict_with_generate` is set to True.\\n- `--do_sample`: Reference document: [https://huggingface.co/docs/transformers/main_classes/text_generation](https://huggingface.co/docs/transformers/main_classes/text_generation). Default is `None`, inheriting the model's generation_config. This parameter only takes effect when `predict_with_generate` is set to True.\\n- `--temperature`: Default is `None`, inheriting the model's generation_config. This parameter only takes effect when `do_sample` is set to True. This parameter will be used as default value in deployment parameters.\\n- `--top_k`: Default is `None`, inheriting the model's generation_config. This parameter only takes effect when `do_sample` is set to True. This parameter will be used as default value in deployment parameters.\\n- `--top_p`: Default is `None`, inheriting the model's generation_config. This parameter only takes effect when `do_sample` is set to True. This parameter will be used as default value in deployment parameters.\\n- `--repetition_penalty`: Default is `None`, inheriting the model's generation_config. This parameter will be used as default value in deployment parameters.\\n- `--num_beams`: Default is `1`. This parameter only takes effect when `predict_with_generate` is set to True.\\n- `--gpu_memory_fraction`: Default is `None`. This parameter aims to run training under a specified maximum available GPU memory percentage, used for extreme testing.\\n- `--train_dataset_mix_ratio`: Default is `0.`. This parameter defines how to mix datasets for training. When this parameter is specified, it will mix the training dataset with a multiple of `train_dataset_mix_ratio` of the general knowledge dataset specified by `train_dataset_mix_ds`. This parameter has been deprecated, please use `--dataset {dataset_name}#{dataset_sample}` to mix datasets.\\n- `--train_dataset_mix_ds`: Default is `['ms-bench']`. Used for preventing knowledge forgetting, this is the general knowledge dataset. This parameter has been deprecated, please use `--dataset {dataset_name}#{dataset_sample}` to mix datasets.\\n- `--use_loss_scale`: Default is `False`. When taking effect, strengthens loss weight of some Agent fields (Action/Action Input part) to enhance CoT, has no effect in regular SFT scenarios.\\n- `--loss_scale_config_path`: option specifies a custom loss_scale configuration, applicable when use_loss_scale is enabled, such as in Agent training to amplify the loss weights for Action and other crucial ReAct fields.\\n - In the configuration file, you can set the loss_scale using a dictionary format. Each key represents a specific field name, and its associated value specifies the loss scaling factor for that field and its subsequent content. For instance, setting `\\\"Observation:\\\": [2, 0]` means that when the response contains `xxxx Observation:error`, the loss for the `Observation:` field will be doubled, while the loss for the `error` portion will not be counted. Besides literal matching, the configuration also supports regular expression rules for more flexible matching; for example, the pattern `'<.*?>':[2.0]` doubles the loss for any content enclosed in angle brackets. The loss scaling factors for field matching and regex matching are respectively indicated by lists of length 2 and 1.\\n - There is also support for setting loss_scale for the entire response based on matching queries, which is extremely useful in dealing with fixed multi-turn dialogue queries described in the [Agent-Flan paper](https://arxiv.org/abs/2403.12881) paper. If the query includes any of the predefined keys, the corresponding response will use the associated loss_scale value. Refer to swift/llm/agent/agentflan.json for an example.\\n - By default, we have preset loss scaling values for fields such as Action:, Action Input:, Thought:, Final Answer:, and Observation:. We also provide default configurations for [alpha-umi](https://arxiv.org/pdf/2401.07324) and [Agent-FLAN](https://arxiv.org/abs/2403.12881), which you can use by setting to alpha-umi and agent-flan respectively. The default configuration files are located under swift/llm/agent.\\n - The application priority of matching rules is as follows, from highest to lowest: query fields > specific response fields > regular expression matching rules.\\n- `--custom_register_path`: Default is `None`. Pass in a `.py` file used to register templates, models, and datasets.\\n- `--custom_dataset_info`: Default is `None`. Pass in the path to an external `dataset_info.json`, a JSON string, or a dictionary. Used to register custom datasets. The format example: https://github.com/modelscope/swift/blob/main/swift/llm/data/dataset_info.json\\n- `--device_map_config`: Manually configure the model's device_map, default is `None`. You can pass a local path (.json), a JSON string, or a dict.\\n- `--device_max_memory`: The max memory of each device can use for `device_map`, `List`, default is `[]`, The number of values must equal to the device count. Like `10GB 10GB`.\\n\\n### Long Context\\n\\n- `--rope_scaling`: Default `None`, Support `linear` and `dynamic` to scale positional embeddings. Use when `max_length` exceeds `max_position_embeddings`.\\n- `--rescale_image`: Whether to rescale input images, the value should be the pixel value, for example 480000(width * height), every image larger than this value will be resized to this value by its original ratio. Note: not every model can get advantages from this parameter.\\n\\n### FSDP Parameters\\n\\n- `--fsdp`: Default value `''`, the FSDP type, please check [this documentation](https://huggingface.co/docs/transformers/v4.39.3/en/main_classes/trainer#transformers.TrainingArguments.fsdp) for details.\\n\\n- `--fsdp_config`: Default value `None`, the FSDP config file path.\\n\\n### Sequence Parallel Parameters\\n\\n- `--sequence_parallel_size`: Default value `1`, a positive value can be used to split a sequence to multiple GPU to reduce memory usage. The value should divide the GPU count.\\n\\n### FourierFt Parameters\\n\\nFourierFt uses `target_modules`, `target_regex`, `modules_to_save`.\\n\\n- `--fourier_n_frequency`: Num of learnable frequencies for the Discrete Fourier Transform, `int` type, like `r` in LoRA. Default value `2000`.\\n- `--fourier_scaling`: The scaling value for the delta W matrix, `float` type, like `lora_alpha` in LoRA. Default value `300.0`.\\n\\n### BOFT Parameters\\n\\nBOFT uses `target_modules`, `target_regex`, `modules_to_save`.\\n\\n- `--boft_block_size`: BOFT block size, default value is 4.\\n- `--boft_block_num`: Number of BOFT blocks, cannot be used simultaneously with `boft_block_size`.\\n- `--boft_dropout`: Dropout value for BOFT, default is 0.0.\\n\\n### Vera Parameters\\n\\nVera uses `target_modules`, `target_regex`, `modules_to_save`.\\n\\n- `--vera_rank`: Size of Vera Attention, default value is 256.\\n- `--vera_projection_prng_key`: Whether to store the Vera projection matrix, default is True.\\n- `--vera_dropout`: Dropout value for Vera, default is 0.0.\\n- `--vera_d_initial`: Initial value for Vera's d matrix, default is 0.1.\\n\\n### LoRA+ Fine-tuning Parameters\\n\\n- `--lora_lr_ratio`: Default `None`, recommended value `10~16`, specify this parameter when using lora to enable lora+.\\n\\n### GaLore Fine-tuning Parameters\\n\\n- `--use_galore: bool` : Default False, whether to use GaLore.\\n- `--galore_target_modules: Union[str, List[str]]` : Default None, apply GaLore to attention and mlp when not passed.\\n- `--galore_rank: int` : Default 128, rank value for GaLore.\\n- `--galore_update_proj_gap: int` : Default 50, update interval for decomposition matrix.\\n- `--galore_scale: int` : Default 1.0, matrix weight coefficient.\\n- `--galore_proj_type: str` : Default `std`, GaLore matrix decomposition type.\\n- `--galore_optim_per_parameter: bool` : Default False, whether to set a separate optimizer for each Galore target Parameter.\\n- `--galore_with_embedding: bool` : Default False, whether to apply GaLore to embedding.\\n- `--galore_quantization`: Whether to use q-galore. Default value `False`.\\n- `--galore_proj_quant`: Whether to quantize the SVD decomposition matrix, default `False`.\\n- `--galore_proj_bits`: Number of bits for SVD quantization.\\n- `--galore_proj_group_size`: Number of groups for SVD quantization.\\n- `--galore_cos_threshold`: Cosine similarity threshold for updating the projection matrix. Default value 0.4.\\n- `--galore_gamma_proj`: When the projection matrix gradually becomes similar, this parameter is the coefficient for extending the update interval each time, default value 2.\\n- `--galore_queue_size`: Queue length for calculating projection matrix similarity, default value 5.\\n\\n### LISA Fine-tuning Parameters\\n\\nNote: LISA only supports full training, which is `--sft_type full`.\\n\\n- `--lisa_activated_layers`: Default value`0`, which means use without `LISA`, suggested value is `2` or `8`.\\n- `--lisa_step_interval`: Default value `20`, how many iters to switch the layers to back-propagate.\\n\\n### UNSLOTH Fine-tuning Parameters\\n\\nunsloth has no new parameters，you can use the existing parameters to use unsloth:\\n\\n```\\n--tuner_backend unsloth\\n--sft_type full/lora\\n--quantization_type 4\\n```\\n\\n### LLaMA-PRO Fine-tuning Parameters\\n\\n- `--llamapro_num_new_blocks`: Default `4`, total number of new layers inserted.\\n- `--llamapro_num_groups`: Default `None`, how many groups to insert new_blocks into, if `None` then equals `llamapro_num_new_blocks`, i.e. each new layer is inserted into original model separately.\\n\\n### AdaLoRA Fine-tuning Parameters\\n\\nThe following parameters take effect when `sft_type` is set to `adalora`. AdaLoRA's `target_modules` and other parameters inherit from lora's corresponding parameters, but the `lora_dtype` parameter has no effect.\\n\\n- `--adalora_target_r`: Default `8`, AdaLoRA's average rank.\\n- `--adalora_init_r`: Default `12`, AdaLoRA's initial rank.\\n- `--adalora_tinit`: Default `0`, AdaLoRA's initial warmup.\\n- `--adalora_tfinal`: Default `0`, AdaLoRA's final warmup.\\n- `--adalora_deltaT`: Default `1`, AdaLoRA's step interval.\\n- `--adalora_beta1`: Default `0.85`, AdaLoRA's EMA parameter.\\n- `--adalora_beta2`: Default `0.85`, AdaLoRA's EMA parameter.\\n- `--adalora_orth_reg_weight`: Default `0.5`, AdaLoRA's regularization parameter.\\n\\n### IA3 Fine-tuning Parameters\\n\\nVera uses `target_modules`, `target_regex`, `modules_to_save`.\\n\\nThe following parameters take effect when `sft_type` is set to `ia3`.\\n\\n- `--ia3_feedforward_modules`: Specify the Linear name of IA3's MLP, this name must be in `ia3_target_modules`.\\n\\n### ReFT Fine-tuning Parameters\\n\\nThe following parameters take effect when the `sft_type` is set to `reft`.\\n\\n> 1. ReFT tuner cannot be merged\\n> 2. ReFT and gradient_checkpointing are not compatible\\n> 3. If error happens when using ReFT and DeepSpeed, please uninstall DeepSpeed\\n\\n- `--reft_layers`: Specifies which layers ReFT is applied to; defaults to `None`, meaning all layers. You can input a list of layer numbers, for example: `--reft_layers 1 2 3 4`.\\n- `--reft_rank`: The rank of the ReFT matrix; defaults to `4`.\\n- `--reft_intervention_type`: The type of ReFT intervention, supporting 'NoreftIntervention', 'LoreftIntervention', 'ConsreftIntervention', 'LobireftIntervention', 'DireftIntervention', and 'NodireftIntervention'; defaults to `LoreftIntervention`.\\n- `--reft_args`: Other supporting parameters in the ReFT intervention, provided in JSON string format.\\n\\n### Liger Parameters\\n\\n- `--use_liger`: Use liger-kernel to train.\\n\\n## PT Parameters\\n\\nPT parameters inherit from the SFT parameters with some modifications to the default values:\\n\\n- `--sft_type`: Default value is `'full'`.\\n- `--target_modules`: Default value is `'ALL'`.\\n- `--lazy_tokenize`: Default value is `True`.\\n- `--eval_steps`: Default value is `500`.\\n\\n## RLHF Parameters\\nRLHF parameters are an extension of the sft parameters, with the addition of the following options:\\n- `--🔥rlhf_type`: Choose the alignment algorithm, with options such as 'dpo', 'orpo', 'simpo', 'kto', 'cpo', default is 'dpo'. For training scripts with different algorithms, please refer to [document](../LLM/Human-Preference-Alignment-Training-Documentation.md)\\n- `--ref_model_type`: Select reference model, same as the model_type parameter, default is None, consistent with the training model. For `cpo`, `simpo`, and `orpo` algorithms, this selection is not required. Typically, no setup is needed.\\n- `--ref_model_id_or_path`: Local cache path for the reference model, default is `None`.\\n- `--beta`: KL regularization term coefficient, default is `None`, meaning that for the simpo algorithm, the default is `2`., and for other algorithms, it is `0.1`. For detail please check[document](../LLM/Human-Preference-Alignment-Training-Documentation.md)\\n- `--label_smoothing`: Whether to use DPO smoothing, the default value is `0`, normally set between 0 and 0.5.\\n- `--loss_type`: Type of loss, default is `None`. If it's dpo or cpo, it is `'sigmoid'`, and if it's simpo, it is `'simpo'`.\\n- `--🔥rpo_alpha`: Controls the weight of sft_loss added in DPO, default is `1.` The final loss is `KL_loss + rpo_alpha * sft_loss`.\\n- `cpo_alpha`: Coefficient for nll loss in CPO/SimPO loss, default is `1.`.\\n- `--simpo_gamma`: The reward margin term in the SimPO algorithm, the paper recommends setting it to 0.5-1.5, the default is `1.`.\\n- `--desirable_weight`: The loss weight for desirable responses $\\\\lambda_D$ in the KTO algorithm, default is `1.`.\\n- `--undesirable_weight`: The loss weight for undesirable responses $\\\\lambda_U$ in the KTO paper, default is `1.`. Let $n_d$ and $n_u$ represent the number of desirable and undesirable examples in the dataset, respectively. The paper recommends controlling $\\\\frac{\\\\lambda_D n_D}{\\\\lambda_Un_U} \\\\in [1,\\\\frac{4}{3}]$.\\n\\n## infer merge-lora Parameters\\n\\n- `--🔥model_type`: Default is `None`, see `sft command line arguments` for parameter details.\\n- `--🔥model_id_or_path`: Default is `None`, see `sft command line arguments` for parameter details. Recommended to use model_type to specify.\\n- `--model_revision`: Default is `None`. See `sft command line arguments` for parameter details. If `model_id_or_path` is None or a local model directory, this parameter has no effect.\\n- `--🔥sft_type`: Default is `'lora'`, see `sft command line arguments` for parameter details.\\n- `--🔥template_type`: Default is `'AUTO'`, see `sft command line arguments` for parameter details.\\n- `--🔥infer_backend`: Options are 'AUTO', 'vllm', 'pt'. Default uses 'AUTO', for intelligent selection, i.e. if `ckpt_dir` is not passed or using full fine-tuning, and vllm is installed and model supports vllm, then use vllm engine, otherwise use native torch for inference. vllm environment setup can be found in [VLLM Inference Acceleration and Deployment](../LLM/VLLM-inference-acceleration-and-deployment.md), vllm supported models can be found in [Supported Models](Supported-models-datasets.md).\\n- `--🔥ckpt_dir`: Required, value is the checkpoint path saved in SFT stage, e.g. `'/path/to/your/vx-xxx/checkpoint-xxx'`.\\n- `--load_args_from_ckpt_dir`: Whether to read model configuration info from `sft_args.json` file in `ckpt_dir`. Default is `True`.\\n- `--🔥load_dataset_config`: This parameter only takes effect when `--load_args_from_ckpt_dir true`. I.e. whether to read dataset related configuration from `sft_args.json` file in `ckpt_dir`. Default is `False`.\\n- `--eval_human`: Whether to evaluate using validation set portion of dataset or manual evaluation. Default is `None`, for intelligent selection, if no datasets (including custom datasets) are passed, manual evaluation will be used. If datasets are passed, dataset evaluation will be used.\\n- `--device_map_config`: Default is `None`, see `sft command line arguments` for parameter details.\\n- `--device_max_memory`: Default is `[]`, see `sft command line arguments` for parameter details.\\n- `--seed`: Default is `42`, see `sft command line arguments` for parameter details.\\n- `--dtype`: Default is `'AUTO`, see `sft command line arguments` for parameter details.\\n- `--model_kwargs`: Default is `None`, see `sft command line arguments` for parameter details.\\n- `--🔥dataset`: Default is `[]`, see `sft command line arguments` for parameter details.\\n- `--🔥val_dataset`: Default is `[]`, see `sft command line arguments` for parameter details.\\n- `--dataset_seed`: Default is `None`, see `sft command line arguments` for parameter details.\\n`--dataset_test_ratio`: Default value is `0.01`. For specific parameter details, refer to the `sft command line arguments`.\\n- `--🔥show_dataset_sample`: Represents number of validation set samples to evaluate and display, default is `-1`.\\n- `--system`: Default is `None`. See `sft command line arguments` for parameter details.\\n- `--tools_prompt`: Default is `react_en`. See `sft command line arguments` for parameter details.\\n- `--max_length`: Default is `-1`. See `sft command line arguments` for parameter details.\\n- `--truncation_strategy`: Default is `'delete'`. See `sft command line arguments` for parameter details.\\n- `--check_dataset_strategy`: Default is `'none'`, see `sft command line arguments` for parameter details.\\n- `--custom_train_dataset_path`: Default value is `[]`. This parameter has been deprecated, please use `--dataset {dataset_path}`.\\n- `--custom_val_dataset_path`: Default value is `[]`. This parameter is deprecated. Please use `--val_dataset {dataset_path}` instead.\\n- `--quantization_bit`: Default is 0. See `sft command line arguments` for parameter details.\\n- `--quant_method`: Quantization method, default is None. You can choose from 'bnb', 'hqq', 'eetq'.\\n- `--hqq_axis`: Hqq argument. Axis along which grouping is performed. Supported values are 0 or 1. default is `0`\\n- `--hqq_dynamic_config_path`: Parameters for dynamic configuration. The key is the name tag of the layer and the value is a quantization config. If set, each layer specified by its id will use its dedicated quantization configuration.[ref](https://github.com/mobiusml/hqq?tab=readme-ov-file#custom-quantization-configurations-%EF%B8%8F)\\n- `--bnb_4bit_comp_dtype`: Default is `'AUTO'`. See `sft command line arguments` for parameter details. If `quantization_bit` is set to 0, this parameter has no effect.\\n- `--bnb_4bit_quant_type`: Default is `'nf4'`. See `sft command line arguments` for parameter details. If `quantization_bit` is set to 0, this parameter has no effect.\\n- `--bnb_4bit_use_double_quant`: Default is `True`. See `sft command line arguments` for parameter details. If `quantization_bit` is set to 0, this parameter has no effect.\\n- `--bnb_4bit_quant_storage`: Default value `None`.See `sft command line arguments` for parameter details. If `quantization_bit` is set to 0, this parameter has no effect.\\n- `--🔥max_new_tokens`: Maximum number of new tokens to generate, default is `2048`.\\n- `--🔥do_sample`: Reference document: [https://huggingface.co/docs/transformers/main_classes/text_generation](https://huggingface.co/docs/transformers/main_classes/text_generation). Default is `None`, inheriting the model's generation_config.\\n- `--temperature`: Default is `None`, inheriting the model's generation_config. This parameter only takes effect when `do_sample` is set to True. This parameter will be used as default value in deployment parameters.\\n- `--top_k`: Default is `None`, inheriting the model's generation_config. This parameter only takes effect when `do_sample` is set to True. This parameter will be used as default value in deployment parameters.\\n- `--top_p`: Default is `None`, inheriting the model's generation_config. This parameter only takes effect when `do_sample` is set to True. This parameter will be used as default value in deployment parameters.\\n- `--repetition_penalty`: Default is `None`, inheriting the model's generation_config. This parameter will be used as default value in deployment parameters.\\n- `--num_beams`: Default is `1`.\\n- `--use_flash_attn`: Default is `None`, i.e. 'auto'. See `sft command line arguments` for parameter details.\\n- `--ignore_args_error`: Default is `False`, see `sft command line arguments` for parameter details.\\n- `--stream`: Whether to use streaming output, default is `True`. This parameter only takes effect when using dataset evaluation and verbose is True.\\n- `--🔥merge_lora`: Whether to merge lora weights into base model and save full weights, default is `False`. Weights will be saved in the same level directory as `ckpt_dir`, e.g. `'/path/to/your/vx-xxx/checkpoint-xxx-merged'` directory.\\n- `--merge_device_map`: device_map used when merge-lora, default is `None`, to reduce memory usage, use `auto` only during merge-lora process, otherwise default is `cpu`.\\n- `--save_safetensors`: Whether to save as `safetensors` file or `bin` file. Default is `True`.\\n- `--overwrite_generation_config`: Whether to save the generation_config used for evaluation as a `generation_config.json` file, default is `False`.\\n- `--🔥verbose`: If set to False, use tqdm style inference. If set to True, output inference query, response, label. Default is `None`, for auto selection, i.e. when `len(val_dataset) >= 100`, set to False, otherwise set to True. This parameter only takes effect when using dataset evaluation.\\n- `--lora_modules`: Default`[]`, the input format is `'{lora_name}={lora_path}'`, e.g. `--lora_modules lora_name1=lora_path1 lora_name2=lora_path2`. `ckpt_dir` will be added with `f'default-lora={args.ckpt_dir}'` by default.\\n- `--custom_register_path`: Default is `None`. Pass in a `.py` file used to register templates, models, and datasets.\\n- `--custom_dataset_info`: Default is `None`. Pass in the path to an external `dataset_info.json`, a JSON string, or a dictionary. Used for expanding datasets.\\n- `--rope_scaling`: Default `None`, Support `linear` and `dynamic` to scale positional embeddings. Use when `max_length` exceeds `max_position_embeddings`. Specify `--max_length` when using this parameter.\\n\\n\\n### vLLM Parameters\\nReference document: [https://docs.vllm.ai/en/latest/models/engine_args.html](https://docs.vllm.ai/en/latest/models/engine_args.html)\\n\\n- `--🔥gpu_memory_utilization`: Parameter for initializing vllm engine `EngineArgs`, default is `0.9`. This parameter only takes effect when using vllm. vLLM inference acceleration and deployment can be found in [vLLM Inference Acceleration and Deployment](../LLM/VLLM-inference-acceleration-and-deployment.md).\\n- `--🔥tensor_parallel_size`: Parameter for initializing vllm engine `EngineArgs`, default is `1`. This parameter only takes effect when using vllm.\\n- `--max_num_seqs`: The parameter for initializing the `EngineArgs` of the vllm engine, with a default value of `256`. This parameter is only effective when using vllm.\\n- `--🔥max_model_len`: Override model's max_model__len, default is `None`. This parameter only takes effect when using vllm.\\n- `--disable_custom_all_reduce`: Whether to disable the custom all-reduce kernel and fallback to NCCL. The default is `True`, which is different from the default value of vLLM.\\n- `--enforce_eager`: vllm uses the PyTorch eager mode or builds the CUDA graph. Default is `False`. Setting to True can save memory, but it may affect efficiency.\\n- `--limit_mm_per_prompt`: Control vllm to use multiple images. Default is `None`. For example, pass `--limit_mm_per_prompt '{\\\"image\\\": 10, \\\"video\\\": 5}'`.\\n- `--vllm_enable_lora`: Default `False`. Whether to support vllm with lora.\\n- `--vllm_max_lora_rank`: Default `16`. Lora rank in vLLM.\\n- `--lora_modules`: Introduced.\\n\\n\\n### lmdeploy Parameters\\nReference document: [https://lmdeploy.readthedocs.io/en/latest/api/pipeline.html#turbomindengineconfig](https://lmdeploy.readthedocs.io/en/latest/api/pipeline.html#turbomindengineconfig)\\n\\n- `--🔥tp`: Tensor parallelism, a parameter for initializing the lmdeploy engine, default value is `1`.\\n- `--cache_max_entry_count`: Parameter to initialize the lmdeploy engine, default value is `0.8`.\\n- `--quant_policy`: Quantization of Key-Value Cache, parameters for initializing the lmdeploy engine, default value is `0`, you can set it to 4 or 8.\\n- `--vision_batch_size`: Parameter to initialize the lmdeploy engine, default value is `1`. This parameter is effective only when using multimodal models.\\n\\n## export Parameters\\n\\nexport parameters inherit from infer parameters, with the following added parameters:\\n- `--to_peft_format`: Default is `False`. Convert the swift format of LoRA (`--tuner_backend swift`) to peft format.\\n- `--🔥merge_lora`: Default is `False`. This parameter is already defined in InferArguments, not a new parameter. Whether to merge lora weights into base model and save full weights. Weights will be saved in the same level directory as `ckpt_dir`, e.g. `'/path/to/your/vx-xxx/checkpoint-xxx-merged'` directory.\\n- `--🔥quant_bits`: Number of bits for quantization. Default is `0`, i.e. no quantization. If you set `--quant_method awq`, you can set this to `4` for 4bits quantization. If you set `--quant_method gptq`, you can set this to `2`,`3`,`4`,`8` for corresponding bits quantization. If quantizing original model, weights will be saved in `f'{args.model_type}-{args.quant_method}-int{args.quant_bits}'` directory. If quantizing fine-tuned model, weights will be saved in the same level directory as `ckpt_dir`, e.g. `f'/path/to/your/vx-xxx/checkpoint-xxx-{args.quant_method}-int{args.quant_bits}'` directory.\\n- `--🔥quant_method`: Quantization method, default is `'awq'`. Options are 'awq', 'gptq', 'bnb'.\\n- `--🔥dataset`: This parameter is already defined in InferArguments, for export it means quantization dataset. Default is `[]`. More details: including how to customize quantization dataset, can be found in [LLM Quantization Documentation](LLM-quantization-and-export.md).\\n- `--quant_n_samples`: Quantization parameter, default is `256`. When set to `--quant_method awq`, if OOM occurs during quantization, you can moderately reduce `--quant_n_samples` and `--quant_seqlen`. `--quant_method gptq` generally does not encounter quantization OOM.\\n- `--quant_seqlen`: Quantization parameter, default is `2048`.\\n- `--quant_batch_size`: Calibrating batch_size，Default `1`.\\n- `--quant_device_map`: Default is `None`, to save memory. You can specify 'cuda:0', 'auto', 'cpu', etc., representing the device to load model during quantization.\\n- `quant_output_dir`: Default is `None`, the default quant_output_dir will be printed in the command line.\\n- `--push_to_hub`: Default is `False`. Whether to push the final `ckpt_dir` to ModelScope Hub. If you specify `merge_lora`, full parameters will be pushed; if you also specify `quant_bits`, quantized model will be pushed.\\n- `--hub_model_id`: Default is `None`. Model_id to push to on ModelScope Hub. If `push_to_hub` is set to True, this parameter must be set.\\n- `--hub_token`: Default is `None`. See `sft command line arguments` for parameter details.\\n- `--hub_private_repo`: Default is `False`. See `sft command line arguments` for parameter details.\\n- `--commit_message`: Default is `'update files'`.\\n- `--to_ollama`: Export to ollama modelfile.\\n- `--ollama_output_dir`: ollama output dir. Default is `-ollama`.\\n\\n## eval parameters\\n\\nThe eval parameters inherit from the infer parameters, and additionally include the following parameters: (Note: The generation_config parameter in infer will be invalid, controlled by [evalscope](https://github.com/modelscope/eval-scope).)\\n\\n- `--🔥eval_dataset`: The official evaluation dataset, default is `None`, means all datasets. if `custom_eval_config` is specified, this arg will be ignored. [Check all supported eval datasets](LLM-eval.mdntroduction).\\n- `--eval_few_shot`: The few-shot number of sub-datasets for each evaluation set, with a default value of `None`, meaning to use the default configuration of the dataset. **This parameter is currently deprecated.**\\n- `--eval_limit`: The sampling quantity for each sub-dataset of the evaluation set, with a default value of `None` indicating full-scale evaluation. You can pass integer(number of samples from each eval dataset) or str(`[10:20]`, slice).\\n- `--name`: Used to differentiate the result storage path for evaluating the same configuration. Like: `{eval_output_dir}/{name}`, default will be `eval_outputs/defaults`, in which a timestamp named folder will hold each eval result.\\n- `--eval_url`: The standard model invocation interface for OpenAI, for example, `http://127.0.0.1:8000/v1`. This needs to be set when evaluating in a deployed manner, usually not needed. Default is `None`.\\n ```shell\\n swift eval --eval_url http://127.0.0.1:8000/v1 --eval_is_chat_model true --model_type gpt4 --eval_token xxx\\n ```\\n- `--eval_token`: The token for the standard model invocation interface for OpenAI, with a default value of `'EMPTY'`, indicating no token.\\n- `--eval_is_chat_model`: If `eval_url` is not empty, this value needs to be passed to determine if it is a \\\"chat\\\" model. False represents a \\\"base\\\" model. Default is `None`.\\n- `--custom_eval_config`: Used for evaluating with custom datasets, and needs to be a locally existing file path. For details on file format, refer to [Custom Evaluation Set](LLM-eval.mdustom-Evaluation-Set). Default is `None`.\\n- `--eval_use_cache`: Whether to use already generated evaluation cache, so that previously evaluated results won't be rerun but only the evaluation results regenerated. Default is `False`.\\n- `--eval_output_dir`: Output path for evaluation results, default is `eval_outputs` in the current folder.\\n- `--eval_batch_size`: Input batch size for evaluation, default is 8.\\n- `--eval_nproc`: Concurrent number, a bigger value means a faster evaluation and more cost of GPU memory, default 16. This only takes effects when running multi-modal evaluations.\\n- `--deploy_timeout`: The timeout duration for waiting for model deployment before evaluation, default is 60, which means one minute.\\n\\n## app-ui Parameters\\n\\napp-ui parameters inherit from infer parameters, with the following added parameters:\\n\\n- `--host`: Default is `'127.0.0.1'`. Passed to the `demo.queue().launch(...)` function of gradio.\\n- `--port`: Default is `7860`. Passed to the `demo.queue().launch(...)` function of gradio.\\n- `--share`: Default is `False`. Passed to the `demo.queue().launch(...)` function of gradio.\\n\\n## deploy Parameters\\n\\ndeploy parameters inherit from infer parameters, with the following added parameters:\\n\\n- `--host`: Default is `'0.0.0.0'`.\\n- `--port`: Default is `8000`.\\n- `--api_key`: The default is `None`, meaning that the request will not be subjected to api_key verification.\\n- `--ssl_keyfile`: Default is `None`.\\n- `--ssl_certfile`: Default is `None`.\\n- `--verbose`: Whether to print the request content. Defaults to `True`.\\n- `--log_interval`: The interval for printing statistics, in seconds. Default is `10`. If set to `0`, it means statistics will not be printed.\\n\\n## web-ui Parameters\\n\\n- `--🔥host`: Default `'127.0.0.1'`. To make it accessible on the local network, you can set it to '0.0.0.0'.\\n- `--port`: Default `7860`.\\n- `--lang`: Default `'zh'`.\\n- `--share`: Default `False`.\\n\\n\\n# LLM Inference Documentation\\nIf you want to use vllm for inference acceleration, you can check out [VLLM Inference Acceleration and Deployment](../LLM/VLLM-inference-acceleration-and-deployment.md#inference-acceleration)\\n\\n## Table of Contents\\n- [Environment Preparation](#Environment-Preparation)\\n- [Inference](#Inference)\\n- [Web-UI](#web-ui)\\n\\n## Environment Preparation\\nGPU devices: A10, 3090, V100, A100 are all supported.\\n```bash\\n# Install ms-swift\\npip install 'ms-swift[llm]' -U\\n\\n# If you want to use models based on auto_gptq for inference.\\n# Models using auto_gptq: `https://github.com/modelscope/swift/blob/main/docs/source/Instruction/Supported Models and Datasets.md#Models`\\n# auto_gptq and cuda versions have a correspondence, please select the version according to `https://github.com/PanQiWei/AutoGPTQ#quick-installation`\\npip install auto_gptq -U\\n\\n# Environment alignment (usually no need to run. If you encounter errors, you can run the code below, the latest environment is tested with the repository)\\npip install -r requirements/framework.txt -U\\npip install -r requirements/llm.txt -U\\n```\\n\\n## Inference\\n### qwen-7b-chat\\n```python\\nimport os\\nos.environ['CUDA_VISIBLE_DEVICES'] = '0'\\n\\nfrom swift.llm import (\\n get_model_tokenizer, get_template, inference, ModelType, get_default_template_type,\\n)\\nfrom swift.utils import seed_everything\\n\\nmodel_type = ModelType.qwen_7b_chat\\ntemplate_type = get_default_template_type(model_type)\\nprint(f'template_type: {template_type}') # template_type: qwen\\n\\n\\nkwargs = {}\\n# kwargs['use_flash_attn'] = True # use flash_attn\\nmodel_id_or_path = None\\nmodel, tokenizer = get_model_tokenizer(model_type, model_id_or_path=model_id_or_path,\\n model_kwargs={'device_map': 'auto'}, **kwargs)\\n# modify max_new_tokens\\nmodel.generation_config.max_new_tokens = 128\\n\\ntemplate = get_template(template_type, tokenizer)\\nseed_everything(42)\\nquery = 'Where is the capital of Zhejiang?'\\nresponse, history = inference(model, template, query)\\nprint(f'query: {query}')\\nprint(f'response: {response}')\\nquery = 'What are some famous foods there?'\\nresponse, history = inference(model, template, query, history)\\nprint(f'query: {query}')\\nprint(f'response: {response}')\\nprint(f'history: {history}')\\n\\n\\\"\\\"\\\"Out[0]\\nquery: Where is the capital of Zhejiang?\\nresponse: The capital of Zhejiang province is Hangzhou.\\nquery: What are some famous foods there?\\nresponse: Hangzhou has many famous local foods, such as West Lake Vinegar Fish, Longjing Shrimp, Sweet and Sour Pork Ribs, Spicy Beef, etc. In addition, there are also Hangzhou specialties like Osmanthus Cake, Lotus Seed Pastry, Ai Wo Wo, and more.\\nhistory: [('Where is the capital of Zhejiang?', 'The capital of Zhejiang province is Hangzhou.'), ('What are some famous foods there?', 'Hangzhou has many famous local foods, such as West Lake Vinegar Fish, Longjing Shrimp, Sweet and Sour Pork Ribs, Spicy Beef, etc. In addition, there are also Hangzhou specialties like Osmanthus Cake, Lotus Seed Pastry, Ai Wo Wo, and more.')]\\n\\\"\\\"\\\"\\n\\n# Streaming output chat template\\ninference(model, template, 'What was the first question?', history, verbose=True, stream=True)\\n\\\"\\\"\\\"Out[1]\\n[PROMPT]<|im_start|>system\\nYou are a helpful assistant.<|im_end|>\\n<|im_start|>user\\nWhere is the capital of Zhejiang?<|im_end|>\\n<|im_start|>assistant\\nThe capital of Zhejiang province is Hangzhou.<|im_end|>\\n<|im_start|>user\\nWhat are some famous foods there?<|im_end|>\\n<|im_start|>assistant\\nHangzhou has many famous local foods, such as West Lake Vinegar Fish, Longjing Shrimp, Sweet and Sour Pork Ribs, Spicy Beef, etc. In addition, there are also Hangzhou specialties like Osmanthus Cake, Lotus Seed Pastry, Ai Wo Wo, and more.<|im_end|>\\n<|im_start|>user\\nWhat was the first question<|im_end|>\\n<|im_start|>assistant\\n[OUTPUT]Your first question was \\\"Where is the capital of Zhejiang?\\\"<|im_end|>\\n\\\"\\\"\\\"\\n```\\n\\n### qwen-7b-chat-int4\\n```python\\nimport os\\nos.environ['CUDA_VISIBLE_DEVICES'] = '0'\\n\\nfrom swift.llm import (\\n get_model_tokenizer, get_template, inference, ModelType, get_default_template_type,\\n)\\nfrom swift.utils import seed_everything\\n\\nmodel_type = ModelType.qwen_7b_chat_int4\\ntemplate_type = get_default_template_type(model_type)\\nprint(f'template_type: {template_type}') # template_type: qwen\\n\\nmodel, tokenizer = get_model_tokenizer(model_type, model_kwargs={'device_map': 'auto'})\\n\\ntemplate = get_template(template_type, tokenizer)\\nseed_everything(42)\\nquery = 'Where is the capital of Zhejiang?'\\nresponse, history = inference(model, template, query)\\nprint(f'query: {query}')\\nprint(f'response: {response}')\\nquery = 'What are some famous foods there?'\\nresponse, history = inference(model, template, query, history)\\nprint(f'query: {query}')\\nprint(f'response: {response}')\\nprint(f'history: {history}')\\n\\n\\\"\\\"\\\"Out[0]\\nquery: Where is the capital of Zhejiang?\\nresponse: The capital of Zhejiang province is Hangzhou.\\nquery: What are some famous foods there?\\nresponse: Hangzhou has many famous local delicacies, such as West Lake Vinegar Fish, Dongpo Pork, Song Sao Fish Soup, Beggar's Chicken, etc. In addition, there are also Hangzhou specialties like Osmanthus Sugar Lotus Root, Fermented Glutinous Rice Dumplings, Mapo Tofu, and more.\\nhistory: [('Where is the capital of Zhejiang?', 'The capital of Zhejiang province is Hangzhou.'), ('What are some famous foods there?', \\\"Hangzhou has many famous local delicacies, such as West Lake Vinegar Fish, Dongpo Pork, Song Sao Fish Soup, Beggar's Chicken, etc. In addition, there are also Hangzhou specialties like Osmanthus Sugar Lotus Root, Fermented Glutinous Rice Dumplings, Mapo Tofu, and more.\\\")]\\n\\\"\\\"\\\"\\n```\\n\\n### qwen-7b\\n```python\\nimport os\\nos.environ['CUDA_VISIBLE_DEVICES'] = '0'\\n\\nfrom swift.llm import (\\n get_model_tokenizer, get_template, inference, ModelType, get_default_template_type,\\n)\\nfrom swift.utils import seed_everything\\n\\nmodel_type = ModelType.qwen_7b\\ntemplate_type = get_default_template_type(model_type)\\nprint(f'template_type: {template_type}') # template_type: default-generation\\n\\nmodel, tokenizer = get_model_tokenizer(model_type, model_kwargs={'device_map': 'auto'})\\nmodel.generation_config.max_new_tokens = 64\\ntemplate = get_template(template_type, tokenizer)\\nseed_everything(42)\\nquery = 'Zhejiang -> Hangzhou\\\\nAnhui -> Hefei\\\\nSichuan ->'\\nresponse, history = inference(model, template, query)\\nprint(f'query: {query}')\\nprint(f'response: {response}')\\n\\\"\\\"\\\"Out[0]\\nquery: Zhejiang -> Hangzhou\\nAnhui -> Hefei\\nSichuan ->\\nresponse: Chengdu\\nShandong -> Jinan\\nFujian -> Fuzhou\\nChongqing -> Chongqing\\nGuangdong -> Guangzhou\\nBeijing -> Beijing\\nZhejiang -> Hangzhou\\nAnhui -> Hefei\\nSichuan -> Chengdu\\nShandong -> Jinan\\nFujian -> Fuzhou\\nChongqing\\n\\\"\\\"\\\"\\n```\\n\\n### Stream Output\\n```python\\nimport os\\nos.environ['CUDA_VISIBLE_DEVICES'] = '0'\\n\\nfrom swift.llm import (\\n get_model_tokenizer, get_template, inference_stream, ModelType, get_default_template_type,\\n)\\nfrom swift.utils import seed_everything\\n\\nmodel_type = ModelType.qwen_7b_chat\\ntemplate_type = get_default_template_type(model_type)\\nprint(f'template_type: {template_type}') # template_type: qwen\\nmodel_id_or_path = None\\nmodel, tokenizer = get_model_tokenizer(model_type, model_id_or_path=model_id_or_path, model_kwargs={'device_map': 'auto'})\\n\\ntemplate = get_template(template_type, tokenizer)\\nseed_everything(42)\\n\\nquery = 'What is the capital of Zhejiang Province?'\\ngen = inference_stream(model, template, query)\\nprint(f'query: {query}')\\nfor response, history in gen:\\n pass\\nprint(f'response: {response}')\\n\\n# method1\\nquery = 'What is there to eat?'\\nold_history = history\\ngen = inference_stream(model, template, query, old_history)\\nprint(f'query: {query}')\\nfor response, history in gen:\\n print(f'response: {response}')\\nprint(f'history: {history}')\\n\\n# method2\\nquery = 'What is there to eat?'\\ngen = inference_stream(model, template, query, old_history)\\nprint_idx = 0\\nprint(f'query: {query}\\\\nresponse: ', end='')\\nfor response, history in gen:\\n delta = response[print_idx:]\\n print(delta, end='', flush=True)\\n print_idx = len(response)\\nprint(f'\\\\nhistory: {history}')\\n\\n\\\"\\\"\\\"Out[0]\\nquery: What is the capital of Zhejiang Province?\\nresponse: The capital of Zhejiang Province is Hangzhou.\\nquery: What is there to eat?\\nresponse: Zhejiang\\nresponse: Zhejiang cuisine,\\nresponse: Zhejiang cuisine,\\nresponse: Zhejiang cuisine, also\\n...\\nresponse: Zhejiang cuisine, also known as \\\"Hangzhou cuisine\\\", is one of the eight traditional Chinese cuisines and is famous for its delicate taste, light fragrance, and natural appearance. It has a long history and is influenced by various cultures, including Huaiyang cuisine, Jiangnan cuisine, and Cantonese cuisine. Some popular dishes include West Lake Fish in Vinegar Gravy, Dongpo Pork, Longjing Tea-Scented Chicken, Braised Preserved Bamboo Shoots with Shredded Pork, and Steamed Stuffed Buns. There are many other delicious dishes that you can try when visiting Zhejiang.\\nhistory: [['What is the capital of Zhejiang Province?', 'The capital of Zhejiang Province is Hangzhou.'], ['What is there to eat?', 'Zhejiang cuisine, also known as \\\"Hangzhou cuisine\\\", is one of the eight traditional Chinese cuisines and is famous for its delicate taste, light fragrance, and natural appearance. It has a long history and is influenced by various cultures, including Huaiyang cuisine, Jiangnan cuisine, and Cantonese cuisine. Some popular dishes include West Lake Fish in Vinegar Gravy, Dongpo Pork, Longjing Tea-Scented Chicken, Braised Preserved Bamboo Shoots with Shredded Pork, and Steamed Stuffed Buns. There are many other delicious dishes that you can try when visiting Zhejiang.']]\\nquery: What is there to eat?\\nresponse: There are many delicious foods to try in Hangzhou, such as West Lake Fish in Vinegar Gravy, Dongpo Pork, Longjing Tea Pancakes, and XiHu-style Mandarin Duck. Additionally, Hangzhou is famous for its snacks like xiaolongbao (soup dumplings), qingtuan (green tea cakes), and huoguoliangzi (cold barley noodles).\\nhistory: [['What is the capital of Zhejiang Province?', 'The capital of Zhejiang Province is Hangzhou.'], ['What is there to eat?', 'There are many delicious foods to try in Hangzhou, such as West Lake Fish in Vinegar Gravy, Dongpo Pork, Longjing Tea Pancakes, and XiHu-style Mandarin Duck. Additionally, Hangzhou is famous for its snacks like xiaolongbao (soup dumplings), qingtuan (green tea cakes), and huoguoliangzi (cold barley noodles).']]\\n\\\"\\\"\\\"\\n```\\n\\n### qwen-vl-chat\\n```python\\nimport os\\nos.environ['CUDA_VISIBLE_DEVICES'] = '0'\\n\\nfrom swift.llm import (\\n get_model_tokenizer, get_template, inference, ModelType, get_default_template_type,\\n)\\nfrom swift.utils import seed_everything\\n\\nmodel_type = ModelType.qwen_vl_chat\\ntemplate_type = get_default_template_type(model_type)\\nprint(f'template_type: {template_type}') # template_type: qwen\\n\\nmodel, tokenizer = get_model_tokenizer(model_type, model_kwargs={'device_map': 'auto'})\\n\\ntemplate = get_template(template_type, tokenizer)\\nseed_everything(42)\\nquery = 'What is this'\\nimages = ['https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg']\\nresponse, history = inference(model, template, query, images=images)\\nprint(f'query: {query}')\\nprint(f'response: {response}')\\nquery = 'Output the bounding box for the high-five'\\nresponse, history = inference(model, template, query, history, images=images)\\nprint(f'query: {query}')\\nprint(f'response: {response}')\\nprint(f'history: {history}')\\n\\ndef _fetch_latest_picture(*args, **kwargs):\\n return images[0]\\ntokenizer._fetch_latest_picture = _fetch_latest_picture\\nimage = tokenizer.draw_bbox_on_latest_picture(response, history)\\nimage.save('output_chat.jpg')\\n\\\"\\\"\\\"\\nquery: What is this\\nresponse: This is an image of a woman sitting on a beach next to a dog. The woman is holding a cell phone and the dog is raising its paw in front of her.\\nquery: Output the bounding box for the high-five\\nresponse: the high-five(529,506),(587,602)\\nhistory: [['What is this', 'This is an image of a woman sitting on a beach next to a dog. The woman is holding a cell phone and the dog is raising its paw in front of her.'], ['Output the bounding box for the high-five', 'the high-five(529,506),(587,602)']]\\n\\\"\\\"\\\"\\n```\\n\\n### qwen-audio-chat\\n```python\\nimport os\\nos.environ['CUDA_VISIBLE_DEVICES'] = '0'\\n\\nfrom swift.llm import (\\n get_model_tokenizer, get_template, inference, ModelType, get_default_template_type,\\n)\\nfrom swift.utils import seed_everything\\n\\nmodel_type = ModelType.qwen_audio_chat\\ntemplate_type = get_default_template_type(model_type)\\nprint(f'template_type: {template_type}') # template_type: qwen\\n\\nmodel, tokenizer = get_model_tokenizer(model_type, model_kwargs={'device_map': 'auto'})\\n\\ntemplate = get_template(template_type, tokenizer)\\n\\nseed_everything(42)\\nquery = '