{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"longbench-v2","formal_name":"LongBench v2","introduction":"LongBench v2 evaluates deep understanding and reasoning over long contexts through multiple-choice questions. Its official description lists 503 questions spanning tasks such as single-document and multi-document QA and code-repository understanding.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/zai-org/LongBench-v2","indexing_mode":"noindex","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[]}},"task_id":"c6e1c202-ad16-5151-929a-69d7479d79e0","task_key":"train--66fcfb5fbb02136c067c93ae","task_revision_id":"3","upstream_id":"66fcfb5fbb02136c067c93ae","short_description":"The repository \"StoryMaker\" is a personalized solution that can generate story…","config":"","split":"train","body":"{\"choice_A\":\"It uses a face recognition method to extract the facial features of characters, followed by an image recognition model to describe the clothing. The use of the SDXL model and IP-Adapter enables consistency of characters across different scenes and environments.\",\"choice_B\":\"It utilizes multiple attention mechanisms and has been extended for various scenarios: the default attention processor (AttnProcessor) and the attention processor combined with IP-Adapter (IPAttnProcessor). IP-Adapter is an enhanced model that integrates image features with textual prompts. The purpose of this code snippet is to add control of image prompts on the basis of attention mechanisms, allowing the use of additional visual prompts to influence the model's generation process.\",\"choice_C\":\"The approach begins by extracting the face from the portrait, ensuring a clear focus on the subject's features. An image recognition model is then utilized to generate descriptive prompts that capture the essence of the face. Using these prompts, the Flux model generates four distinct portrait images, each showcasing different artistic interpretations of the original face. Next, reactor face-swapping is applied to seamlessly blend the facial features across the generated images, enhancing diversity and creativity. Finally, the SDXL and ControlNet models are employed to apply stylistic enhancements, transforming the final output into a series of visually striking and stylized portraits that convey a rich narrative and artistic flair.\",\"choice_D\":\"StoryMaker merges conditional information based on facial identity and cropped character images (including clothing, hairstyles, and bodies). Specifically, we utilize a Position-Aware Perceiver Resampler (PPR) to integrate facial identity information with cropped character images, enabling the acquisition of diverse character features.\",\"context\":\"from typing import Any, Callable, Dict, List, Optional, Tuple, Union\\n\\nimport cv2\\nimport math\\n\\nimport numpy as np\\nimport PIL.Image\\nfrom PIL import Image\\nimport torch, traceback, pdb\\nimport torch.nn.functional as F\\n\\nfrom diffusers.image_processor import PipelineImageInput\\n\\nfrom diffusers.models import ControlNetModel\\n\\nfrom diffusers.utils import (\\n    deprecate,\\n    logging,\\n    replace_example_docstring,\\n)\\nfrom diffusers.utils.torch_utils import is_compiled_module, is_torch_version\\nfrom diffusers.pipelines.stable_diffusion_xl import StableDiffusionXLPipelineOutput\\n\\nfrom diffusers import StableDiffusionXLPipeline\\nfrom diffusers.utils.import_utils import is_xformers_available\\n\\nfrom transformers import CLIPImageProcessor, CLIPVisionModelWithProjection\\nfrom insightface.utils import face_align\\n\\nfrom ip_adapter.resampler import Resampler\\nfrom ip_adapter.utils import is_torch2_available\\nfrom ip_adapter.ip_adapter_faceid import faceid_plus\\n\\nfrom ip_adapter.attention_processor import IPAttnProcessor2_0 as IPAttnProcessor, AttnProcessor2_0 as AttnProcessor\\nfrom ip_adapter.attention_processor_faceid import LoRAIPAttnProcessor2_0 as LoRAIPAttnProcessor, LoRAAttnProcessor2_0 as LoRAAttnProcessor\\n\\nlogger = logging.get_logger(__name__)  # pylint: disable=invalid-name\\n\\n\\nEXAMPLE_DOC_STRING = \\\"\\\"\\\"\\n    Examples:\\n        ```py\\n        >>> # !pip install opencv-python transformers accelerate insightface\\n        >>> import diffusers\\n        >>> from diffusers.utils import load_image\\n        >>> import cv2\\n        >>> import torch\\n        >>> import numpy as np\\n        >>> from PIL import Image\\n        \\n        >>> from insightface.app import FaceAnalysis\\n        >>> from pipeline_sdxl_storymaker import StableDiffusionXLStoryMakerPipeline\\n\\n        >>> # download 'buffalo_l' under ./models\\n        >>> app = FaceAnalysis(name='buffalo_l', root='./', providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])\\n        >>> app.prepare(ctx_id=0, det_size=(640, 640))\\n        \\n        >>> # download models under ./checkpoints\\n        >>> storymaker_adapter = f'./checkpoints/ip-adapter.bin'\\n        \\n        >>> pipe = StableDiffusionXLStoryMakerPipeline.from_pretrained(\\n        ...     \\\"stabilityai/stable-diffusion-xl-base-1.0\\\", torch_dtype=torch.float16\\n        ... )\\n        >>> pipe.cuda()\\n        \\n        >>> # load adapter\\n        >>> pipe.load_storymaker_adapter(storymaker_adapter)\\n\\n        >>> prompt = \\\"a person is taking a selfie, the person is wearing a red hat, and a volcano is in the distance\\\"\\n        >>> negative_prompt = \\\"bad quality, NSFW, low quality, ugly, disfigured, deformed\\\"\\n\\n        >>> # load an image\\n        >>> image = load_image(\\\"your-example.jpg\\\")\\n        >>> # load the mask image of portrait\\n        >>> mask_image = load_image(\\\"your-mask.jpg\\\")\\n        \\n        >>> face_info = app.get(cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR))[-1]\\n\\n        >>> # generate image\\n        >>> image = pipe(\\n        ...     prompt, image=image, mask_image=mask_image,face_info=face_info, controlnet_conditioning_scale=0.8\\n        ... ).images[0]\\n        ```\\n\\\"\\\"\\\"\\n\\ndef bounding_rectangle(ori_img, mask):\\n    \\\"\\\"\\\"\\n    Calculate the bounding rectangle of multiple rectangles.\\n    Args:\\n        rectangles (list of tuples): List of rectangles, where each rectangle is represented as (x, y, w, h)\\n    Returns:\\n        tuple: The bounding rectangle (x, y, w, h)\\n    \\\"\\\"\\\"\\n    contours, _ = cv2.findContours(mask[:,:,0], cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)\\n    rectangles = [cv2.boundingRect(contour) for contour in contours]\\n                \\n    min_x = float('inf')\\n    min_y = float('inf')\\n    max_x = float('-inf')\\n    max_y = float('-inf')\\n    for x, y, w, h in rectangles:\\n        min_x = min(min_x, x)\\n        min_y = min(min_y, y)\\n        max_x = max(max_x, x + w)\\n        max_y = max(max_y, y + h)\\n    try:\\n        crop = ori_img[min_y:max_y, min_x:max_x]\\n        mask = mask[min_y:max_y, min_x:max_x]\\n    except:\\n        traceback.print_exc()\\n    return crop, mask\\n\\n\\n    \\nclass StableDiffusionXLStoryMakerPipeline(StableDiffusionXLPipeline):\\n    \\n    def cuda(self, dtype=torch.float16, use_xformers=False):\\n        self.to('cuda', dtype)\\n        if hasattr(self, 'image_proj_model'):\\n            self.image_proj_model.to(self.unet.device).to(self.unet.dtype)\\n        \\n    def load_storymaker_adapter(self, image_encoder_path, model_ckpt, image_emb_dim=512, num_tokens=20, scale=0.8, lora_scale=0.8):     \\n        self.clip_image_processor = CLIPImageProcessor()\\n        self.image_encoder = CLIPVisionModelWithProjection.from_pretrained(image_encoder_path).to(self.device, dtype=self.dtype)\\n        self.set_image_proj_model(model_ckpt, image_emb_dim, num_tokens)\\n        self.set_ip_adapter(model_ckpt, num_tokens)\\n        self.set_ip_adapter_scale(scale, lora_scale)\\n        print(f'successful load adapter.')\\n        \\n    def set_image_proj_model(self, model_ckpt, image_emb_dim=512, num_tokens=16):\\n        \\n        image_proj_model = faceid_plus(\\n            cross_attention_dim=self.unet.config.cross_attention_dim,\\n            id_embeddings_dim=512,\\n            clip_embeddings_dim=1280,\\n        )\\n        image_proj_model.eval()\\n        \\n        self.image_proj_model = image_proj_model.to(self.device, dtype=self.dtype)\\n        state_dict = torch.load(model_ckpt, map_location=\\\"cpu\\\")\\n        if 'image_proj_model' in state_dict:\\n            state_dict = state_dict[\\\"image_proj_model\\\"]\\n        self.image_proj_model.load_state_dict(state_dict)\\n        \\n    def set_ip_adapter(self, model_ckpt, num_tokens, lora_rank=128):\\n        \\n        unet = self.unet\\n        attn_procs = {}\\n        for name in unet.attn_processors.keys():\\n            cross_attention_dim = None if name.endswith(\\\"attn1.processor\\\") else unet.config.cross_attention_dim\\n            if name.startswith(\\\"mid_block\\\"):\\n                hidden_size = unet.config.block_out_channels[-1]\\n            elif name.startswith(\\\"up_blocks\\\"):\\n                block_id = int(name[len(\\\"up_blocks.\\\")])\\n                hidden_size = list(reversed(unet.config.block_out_channels))[block_id]\\n            elif name.startswith(\\\"down_blocks\\\"):\\n                block_id = int(name[len(\\\"down_blocks.\\\")])\\n                hidden_size = unet.config.block_out_channels[block_id]\\n            if cross_attention_dim is None:\\n                attn_procs[name] = LoRAAttnProcessor(hidden_size=hidden_size, cross_attention_dim=cross_attention_dim, rank=lora_rank).to(unet.device, dtype=unet.dtype)\\n            else:\\n                attn_procs[name] = LoRAIPAttnProcessor(hidden_size=hidden_size, cross_attention_dim=cross_attention_dim, rank=lora_rank).to(unet.device, dtype=unet.dtype)\\n        unet.set_attn_processor(attn_procs)\\n        \\n        state_dict = torch.load(model_ckpt, map_location=\\\"cpu\\\")\\n        ip_layers = torch.nn.ModuleList(self.unet.attn_processors.values())\\n        if 'ip_adapter' in state_dict:\\n            state_dict = state_dict['ip_adapter']\\n        ip_layers.load_state_dict(state_dict)\\n    \\n    def set_ip_adapter_scale(self, scale, lora_scale=0.8):\\n        unet = getattr(self, self.unet_name) if not hasattr(self, \\\"unet\\\") else self.unet\\n        for attn_processor in unet.attn_processors.values():\\n            if isinstance(attn_processor, LoRAIPAttnProcessor) or isinstance(attn_processor, LoRAAttnProcessor):\\n                attn_processor.scale = scale\\n                attn_processor.lora_scale = lora_scale\\n\\n    def crop_image(self, ori_img, ori_mask, face_info):\\n        ori_img = np.array(ori_img)\\n        ori_mask = np.array(ori_mask)\\n        crop, mask = bounding_rectangle(ori_img, ori_mask)\\n        mask = cv2.GaussianBlur(mask, (5, 5), 0)/255.\\n        crop = (255*np.ones_like(mask)*(1-mask)+mask*crop).astype(np.uint8)\\n        # cv2.imwrite('examples/results/0crop.jpg', crop[:,:,::-1])\\n        # cv2.imwrite('examples/results/0mask.jpg', (mask*255).astype(np.uint8))\\n        \\n        face_kps = face_info['kps']\\n        # face_image = face_align.norm_crop(crop, landmark=face_kps.numpy(), image_size=224)  #  224\\n        face_image = face_align.norm_crop(ori_img, landmark=face_kps, image_size=224)  #  224\\n        clip_face = self.clip_image_processor(images=face_image, return_tensors=\\\"pt\\\").pixel_values\\n        \\n        ref_img = Image.fromarray(crop)\\n        ref_img = ref_img.resize((224, 224))  \\n        clip_img = self.clip_image_processor(images=ref_img, return_tensors=\\\"pt\\\").pixel_values\\n        return clip_img, clip_face, torch.from_numpy(face_info.normed_embedding).unsqueeze(0)\\n\\n    def _encode_prompt_image_emb(self, image, image_2, mask_image, mask_image_2, face_info, face_info_2, cloth, cloth_2, \\\\\\n                                 device, num_images_per_prompt, dtype, do_classifier_free_guidance):\\n        crop_list = []; face_list = [];  id_list = []\\n        if image is not None:\\n            clip_img, clip_face, face_emb = self.crop_image(image, mask_image, face_info)\\n            crop_list.append(clip_img)\\n            face_list.append(clip_face)\\n            id_list.append(face_emb)\\n        if image_2 is not None:\\n            clip_img, clip_face, face_emb = self.crop_image(image_2, mask_image_2, face_info_2)\\n            crop_list.append(clip_img)\\n            face_list.append(clip_face)\\n            id_list.append(face_emb)\\n        if cloth is not None:\\n            crop_list = []\\n            clip_img = self.clip_image_processor(images=cloth.resize((224, 224)), return_tensors=\\\"pt\\\").pixel_values\\n            crop_list.append(clip_img)\\n        if cloth_2 is not None:\\n            clip_img = self.clip_image_processor(images=cloth_2.resize((224, 224)), return_tensors=\\\"pt\\\").pixel_values\\n            crop_list.append(clip_img)\\n        assert len(crop_list)>0, f\\\"input error, images is None\\\"\\n        clip_image = torch.cat(crop_list, dim=0).to(device, dtype=dtype)\\n        clip_image_embeds = self.image_encoder(clip_image, output_hidden_states=True).hidden_states[-2] \\n        clip_face = torch.cat(face_list, dim=0).to(device, dtype=dtype)\\n        clip_face_embeds = self.image_encoder(clip_face, output_hidden_states=True).hidden_states[-2] \\n        id_embeds = torch.cat(id_list, dim=0).to(device, dtype=dtype)\\n        # print(f'clip_image_embeds: {clip_image_embeds.shape}, clip_face_embeds:{clip_face_embeds.shape}, id_embeds:{id_embeds.shape}')\\n        if do_classifier_free_guidance:\\n            prompt_image_emb = self.image_proj_model(id_embeds, clip_image_embeds, clip_face_embeds)\\n            B, C, D = prompt_image_emb.shape\\n            prompt_image_emb = prompt_image_emb.view(1, B*C, D)\\n            neg_emb = self.image_proj_model(torch.zeros_like(id_embeds), torch.zeros_like(clip_image_embeds), torch.zeros_like(clip_face_embeds))\\n            neg_emb = neg_emb.view(1, B*C, D)\\n            prompt_image_emb = torch.cat([neg_emb, prompt_image_emb], dim=0)\\n        else:\\n            prompt_image_emb = torch.cat([prompt_image_emb], dim=0)\\n            B, C, D = prompt_image_emb.shape\\n            prompt_image_emb = prompt_image_emb.view(1, B*C, D)\\n        \\n        # print(f'prompt_image_emb: {prompt_image_emb.shape}')\\n        bs_embed, seq_len, _ = prompt_image_emb.shape\\n        prompt_image_emb = prompt_image_emb.repeat(1, num_images_per_prompt, 1)\\n        prompt_image_emb = prompt_image_emb.view(bs_embed * num_images_per_prompt, seq_len, -1)\\n        \\n        return prompt_image_emb.to(device=device, dtype=dtype)\\n\\n    @torch.no_grad()\\n    @replace_example_docstring(EXAMPLE_DOC_STRING)\\n    def __call__(\\n        self,\\n        prompt: Union[str, List[str]] = None,\\n        prompt_2: Optional[Union[str, List[str]]] = None,\\n        image: PipelineImageInput = None,\\n        mask_image: Union[torch.Tensor, PIL.Image.Image] = None,\\n        image_2: PipelineImageInput = None,\\n        mask_image_2: Union[torch.Tensor, PIL.Image.Image] = None,\\n        height: Optional[int] = None,\\n        width: Optional[int] = None,\\n        num_inference_steps: int = 50,\\n        guidance_scale: float = 5.0,\\n        negative_prompt: Optional[Union[str, List[str]]] = None,\\n        negative_prompt_2: Optional[Union[str, List[str]]] = None,\\n        num_images_per_prompt: Optional[int] = 1,\\n        eta: float = 0.0,\\n        generator: Optional[Union[torch.Generator, List[torch.Generator]]] = None,\\n        latents: Optional[torch.FloatTensor] = None,\\n        prompt_embeds: Optional[torch.FloatTensor] = None,\\n        negative_prompt_embeds: Optional[torch.FloatTensor] = None,\\n        pooled_prompt_embeds: Optional[torch.FloatTensor] = None,\\n        negative_pooled_prompt_embeds: Optional[torch.FloatTensor] = None,\\n        output_type: Optional[str] = \\\"pil\\\",\\n        return_dict: bool = True,\\n        cross_attention_kwargs: Optional[Dict[str, Any]] = None,\\n        controlnet_conditioning_scale: Union[float, List[float]] = 1.0,\\n        guess_mode: bool = False,\\n        control_guidance_start: Union[float, List[float]] = 0.0,\\n        control_guidance_end: Union[float, List[float]] = 1.0,\\n        original_size: Tuple[int, int] = None,\\n        crops_coords_top_left: Tuple[int, int] = (0, 0),\\n        target_size: Tuple[int, int] = None,\\n        negative_original_size: Optional[Tuple[int, int]] = None,\\n        negative_crops_coords_top_left: Tuple[int, int] = (0, 0),\\n        negative_target_size: Optional[Tuple[int, int]] = None,\\n        clip_skip: Optional[int] = None,\\n        callback_on_step_end: Optional[Callable[[int, int, Dict], None]] = None,\\n        callback_on_step_end_tensor_inputs: List[str] = [\\\"latents\\\"],\\n\\n        # IP adapter\\n        ip_adapter_scale=None,\\n        lora_scale=None,\\n        face_info = None,\\n        face_info_2 = None,\\n        cloth = None,\\n        cloth_2 = None,\\n\\n        **kwargs,\\n    ):\\n        r\\\"\\\"\\\"\\n        The call function to the pipeline for generation.\\n\\n        Args:\\n            prompt (`str` or `List[str]`, *optional*):\\n                The prompt or prompts to guide image generation. If not defined, you need to pass `prompt_embeds`.\\n            prompt_2 (`str` or `List[str]`, *optional*):\\n                The prompt or prompts to be sent to `tokenizer_2` and `text_encoder_2`. If not defined, `prompt` is\\n                used in both text-encoders.\\n            image (`torch.FloatTensor`, `PIL.Image.Image`, `np.ndarray`, `List[torch.FloatTensor]`, `List[PIL.Image.Image]`, `List[np.ndarray]`,:\\n                    `List[List[torch.FloatTensor]]`, `List[List[np.ndarray]]` or `List[List[PIL.Image.Image]]`):\\n                The ControlNet input condition to provide guidance to the `unet` for generation. If the type is\\n                specified as `torch.FloatTensor`, it is passed to ControlNet as is. `PIL.Image.Image` can also be\\n                accepted as an image. The dimensions of the output image defaults to `image`'s dimensions. If height\\n                and/or width are passed, `image` is resized accordingly. If multiple ControlNets are specified in\\n                `init`, images must be passed as a list such that each element of the list can be correctly batched for\\n                input to a single ControlNet.\\n            height (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`):\\n                The height in pixels of the generated image. Anything below 512 pixels won't work well for\\n                [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0)\\n                and checkpoints that are not specifically fine-tuned on low resolutions.\\n            width (`int`, *optional*, defaults to `self.unet.config.sample_size * self.vae_scale_factor`):\\n                The width in pixels of the generated image. Anything below 512 pixels won't work well for\\n                [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0)\\n                and checkpoints that are not specifically fine-tuned on low resolutions.\\n            num_inference_steps (`int`, *optional*, defaults to 50):\\n                The number of denoising steps. More denoising steps usually lead to a higher quality image at the\\n                expense of slower inference.\\n            guidance_scale (`float`, *optional*, defaults to 5.0):\\n                A higher guidance scale value encourages the model to generate images closely linked to the text\\n                `prompt` at the expense of lower image quality. Guidance scale is enabled when `guidance_scale > 1`.\\n            negative_prompt (`str` or `List[str]`, *optional*):\\n                The prompt or prompts to guide what to not include in image generation. If not defined, you need to\\n                pass `negative_prompt_embeds` instead. Ignored when not using guidance (`guidance_scale < 1`).\\n            negative_prompt_2 (`str` or `List[str]`, *optional*):\\n                The prompt or prompts to guide what to not include in image generation. This is sent to `tokenizer_2`\\n                and `text_encoder_2`. If not defined, `negative_prompt` is used in both text-encoders.\\n            num_images_per_prompt (`int`, *optional*, defaults to 1):\\n                The number of images to generate per prompt.\\n            eta (`float`, *optional*, defaults to 0.0):\\n                Corresponds to parameter eta (η) from the [DDIM](https://arxiv.org/abs/2010.02502) paper. Only applies\\n                to the [`~schedulers.DDIMScheduler`], and is ignored in other schedulers.\\n            generator (`torch.Generator` or `List[torch.Generator]`, *optional*):\\n                A [`torch.Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make\\n                generation deterministic.\\n            latents (`torch.FloatTensor`, *optional*):\\n                Pre-generated noisy latents sampled from a Gaussian distribution, to be used as inputs for image\\n                generation. Can be used to tweak the same generation with different prompts. If not provided, a latents\\n                tensor is generated by sampling using the supplied random `generator`.\\n            prompt_embeds (`torch.FloatTensor`, *optional*):\\n                Pre-generated text embeddings. Can be used to easily tweak text inputs (prompt weighting). If not\\n                provided, text embeddings are generated from the `prompt` input argument.\\n            negative_prompt_embeds (`torch.FloatTensor`, *optional*):\\n                Pre-generated negative text embeddings. Can be used to easily tweak text inputs (prompt weighting). If\\n                not provided, `negative_prompt_embeds` are generated from the `negative_prompt` input argument.\\n            pooled_prompt_embeds (`torch.FloatTensor`, *optional*):\\n                Pre-generated pooled text embeddings. Can be used to easily tweak text inputs (prompt weighting). If\\n                not provided, pooled text embeddings are generated from `prompt` input argument.\\n            negative_pooled_prompt_embeds (`torch.FloatTensor`, *optional*):\\n                Pre-generated negative pooled text embeddings. Can be used to easily tweak text inputs (prompt\\n                weighting). If not provided, pooled `negative_prompt_embeds` are generated from `negative_prompt` input\\n                argument.\\n            output_type (`str`, *optional*, defaults to `\\\"pil\\\"`):\\n                The output format of the generated image. Choose between `PIL.Image` or `np.array`.\\n            return_dict (`bool`, *optional*, defaults to `True`):\\n                Whether or not to return a [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] instead of a\\n                plain tuple.\\n            cross_attention_kwargs (`dict`, *optional*):\\n                A kwargs dictionary that if specified is passed along to the [`AttentionProcessor`] as defined in\\n                [`self.processor`](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py).\\n            controlnet_conditioning_scale (`float` or `List[float]`, *optional*, defaults to 1.0):\\n                The outputs of the ControlNet are multiplied by `controlnet_conditioning_scale` before they are added\\n                to the residual in the original `unet`. If multiple ControlNets are specified in `init`, you can set\\n                the corresponding scale as a list.\\n            guess_mode (`bool`, *optional*, defaults to `False`):\\n                The ControlNet encoder tries to recognize the content of the input image even if you remove all\\n                prompts. A `guidance_scale` value between 3.0 and 5.0 is recommended.\\n            control_guidance_start (`float` or `List[float]`, *optional*, defaults to 0.0):\\n                The percentage of total steps at which the ControlNet starts applying.\\n            control_guidance_end (`float` or `List[float]`, *optional*, defaults to 1.0):\\n                The percentage of total steps at which the ControlNet stops applying.\\n            original_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)):\\n                If `original_size` is not the same as `target_size` the image will appear to be down- or upsampled.\\n                `original_size` defaults to `(height, width)` if not specified. Part of SDXL's micro-conditioning as\\n                explained in section 2.2 of\\n                [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952).\\n            crops_coords_top_left (`Tuple[int]`, *optional*, defaults to (0, 0)):\\n                `crops_coords_top_left` can be used to generate an image that appears to be \\\"cropped\\\" from the position\\n                `crops_coords_top_left` downwards. Favorable, well-centered images are usually achieved by setting\\n                `crops_coords_top_left` to (0, 0). Part of SDXL's micro-conditioning as explained in section 2.2 of\\n                [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952).\\n            target_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)):\\n                For most cases, `target_size` should be set to the desired height and width of the generated image. If\\n                not specified it will default to `(height, width)`. Part of SDXL's micro-conditioning as explained in\\n                section 2.2 of [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952).\\n            negative_original_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)):\\n                To negatively condition the generation process based on a specific image resolution. Part of SDXL's\\n                micro-conditioning as explained in section 2.2 of\\n                [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). For more\\n                information, refer to this issue thread: https://github.com/huggingface/diffusers/issues/4208.\\n            negative_crops_coords_top_left (`Tuple[int]`, *optional*, defaults to (0, 0)):\\n                To negatively condition the generation process based on a specific crop coordinates. Part of SDXL's\\n                micro-conditioning as explained in section 2.2 of\\n                [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). For more\\n                information, refer to this issue thread: https://github.com/huggingface/diffusers/issues/4208.\\n            negative_target_size (`Tuple[int]`, *optional*, defaults to (1024, 1024)):\\n                To negatively condition the generation process based on a target image resolution. It should be as same\\n                as the `target_size` for most cases. Part of SDXL's micro-conditioning as explained in section 2.2 of\\n                [https://huggingface.co/papers/2307.01952](https://huggingface.co/papers/2307.01952). For more\\n                information, refer to this issue thread: https://github.com/huggingface/diffusers/issues/4208.\\n            clip_skip (`int`, *optional*):\\n                Number of layers to be skipped from CLIP while computing the prompt embeddings. A value of 1 means that\\n                the output of the pre-final layer will be used for computing the prompt embeddings.\\n            callback_on_step_end (`Callable`, *optional*):\\n                A function that calls at the end of each denoising steps during the inference. The function is called\\n                with the following arguments: `callback_on_step_end(self: DiffusionPipeline, step: int, timestep: int,\\n                callback_kwargs: Dict)`. `callback_kwargs` will include a list of all tensors as specified by\\n                `callback_on_step_end_tensor_inputs`.\\n            callback_on_step_end_tensor_inputs (`List`, *optional*):\\n                The list of tensor inputs for the `callback_on_step_end` function. The tensors specified in the list\\n                will be passed as `callback_kwargs` argument. You will only be able to include variables listed in the\\n                `._callback_tensor_inputs` attribute of your pipeline class.\\n\\n        Examples:\\n\\n        Returns:\\n            [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] or `tuple`:\\n                If `return_dict` is `True`, [`~pipelines.stable_diffusion.StableDiffusionPipelineOutput`] is returned,\\n                otherwise a `tuple` is returned containing the output images.\\n        \\\"\\\"\\\"\\n\\n        callback = kwargs.pop(\\\"callback\\\", None)\\n        callback_steps = kwargs.pop(\\\"callback_steps\\\", None)\\n\\n        if callback is not None:\\n            deprecate(\\n                \\\"callback\\\",\\n                \\\"1.0.0\\\",\\n                \\\"Passing `callback` as an input argument to `__call__` is deprecated, consider using `callback_on_step_end`\\\",\\n            )\\n        if callback_steps is not None:\\n            deprecate(\\n                \\\"callback_steps\\\",\\n                \\\"1.0.0\\\",\\n                \\\"Passing `callback_steps` as an input argument to `__call__` is deprecated, consider using `callback_on_step_end`\\\",\\n            )\\n\\n        # 0. set ip_adapter_scale\\n        if ip_adapter_scale is not None and lora_scale is not None:\\n            self.set_ip_adapter_scale(ip_adapter_scale, lora_scale)\\n\\n        # 1. Check inputs. Raise error if not correct\\n        # self.check_inputs(\\n        #     prompt=prompt,\\n        #     prompt_2=prompt_2,\\n        #     height=height, width=width,\\n        #     callback_steps=callback_steps,\\n        #     negative_prompt=negative_prompt,\\n        #     negative_prompt_2=negative_prompt_2,\\n        #     prompt_embeds=prompt_embeds,\\n        #     negative_prompt_embeds=negative_prompt_embeds,\\n        #     pooled_prompt_embeds=pooled_prompt_embeds,\\n        #     negative_pooled_prompt_embeds=negative_pooled_prompt_embeds,\\n        #     callback_on_step_end_tensor_inputs=callback_on_step_end_tensor_inputs,\\n        # )\\n\\n        self._guidance_scale = guidance_scale\\n        self._clip_skip = clip_skip\\n        self._cross_attention_kwargs = cross_attention_kwargs\\n\\n        # 2. Define call parameters\\n        if prompt is not None and isinstance(prompt, str):\\n            batch_size = 1\\n        elif prompt is not None and isinstance(prompt, list):\\n            batch_size = len(prompt)\\n        else:\\n            batch_size = prompt_embeds.shape[0]\\n\\n        device = self.unet.device\\n        # pdb.set_trace()\\n        # 3.1 Encode input prompt\\n        text_encoder_lora_scale = (\\n            self.cross_attention_kwargs.get(\\\"scale\\\", None) if self.cross_attention_kwargs is not None else None\\n        )\\n        (\\n            prompt_embeds,\\n            negative_prompt_embeds,\\n            pooled_prompt_embeds,\\n            negative_pooled_prompt_embeds,\\n        ) = self.encode_prompt(\\n            prompt,\\n            prompt_2,\\n            device,\\n            num_images_per_prompt,\\n            self.do_classifier_free_guidance,\\n            negative_prompt,\\n            negative_prompt_2,\\n            prompt_embeds=prompt_embeds,\\n            negative_prompt_embeds=negative_prompt_embeds,\\n            pooled_prompt_embeds=pooled_prompt_embeds,\\n            negative_pooled_prompt_embeds=negative_pooled_prompt_embeds,\\n            lora_scale=text_encoder_lora_scale,\\n            clip_skip=self.clip_skip,\\n        )\\n        \\n        # 3.2 Encode image prompt\\n        prompt_image_emb = self._encode_prompt_image_emb(image, image_2, mask_image, mask_image_2, face_info, face_info_2, cloth,cloth_2,\\n                                                         device, num_images_per_prompt,\\n                                                         self.unet.dtype, self.do_classifier_free_guidance)\\n        \\n        # 5. Prepare timesteps\\n        self.scheduler.set_timesteps(num_inference_steps, device=device)\\n        timesteps = self.scheduler.timesteps\\n        self._num_timesteps = len(timesteps)\\n\\n        # 6. Prepare latent variables\\n        num_channels_latents = self.unet.config.in_channels\\n        latents = self.prepare_latents(\\n            batch_size * num_images_per_prompt,\\n            num_channels_latents,\\n            height,\\n            width,\\n            prompt_embeds.dtype,\\n            device,\\n            generator,\\n            latents,\\n        )\\n\\n        # 6.5 Optionally get Guidance Scale Embedding\\n        timestep_cond = None\\n        if self.unet.config.time_cond_proj_dim is not None:\\n            guidance_scale_tensor = torch.tensor(self.guidance_scale - 1).repeat(batch_size * num_images_per_prompt)\\n            timestep_cond = self.get_guidance_scale_embedding(\\n                guidance_scale_tensor, embedding_dim=self.unet.config.time_cond_proj_dim\\n            ).to(device=device, dtype=latents.dtype)\\n\\n        # 7. Prepare extra step kwargs. TODO: Logic should ideally just be moved out of the pipeline\\n        extra_step_kwargs = self.prepare_extra_step_kwargs(generator, eta)\\n\\n        # 7.2 Prepare added time ids & embeddings\\n        original_size = original_size or (height, width)\\n        target_size = target_size or (height, width)\\n\\n        add_text_embeds = pooled_prompt_embeds\\n        if self.text_encoder_2 is None:\\n            text_encoder_projection_dim = int(pooled_prompt_embeds.shape[-1])\\n        else:\\n            text_encoder_projection_dim = self.text_encoder_2.config.projection_dim\\n\\n        add_time_ids = self._get_add_time_ids(\\n            original_size,\\n            crops_coords_top_left,\\n            target_size,\\n            dtype=prompt_embeds.dtype,\\n            text_encoder_projection_dim=text_encoder_projection_dim,\\n        )\\n\\n        if negative_original_size is not None and negative_target_size is not None:\\n            negative_add_time_ids = self._get_add_time_ids(\\n                negative_original_size,\\n                negative_crops_coords_top_left,\\n                negative_target_size,\\n                dtype=prompt_embeds.dtype,\\n                text_encoder_projection_dim=text_encoder_projection_dim,\\n            )\\n        else:\\n            negative_add_time_ids = add_time_ids\\n\\n        if self.do_classifier_free_guidance:\\n            prompt_embeds = torch.cat([negative_prompt_embeds, prompt_embeds], dim=0)\\n            add_text_embeds = torch.cat([negative_pooled_prompt_embeds, add_text_embeds], dim=0)\\n            add_time_ids = torch.cat([negative_add_time_ids, add_time_ids], dim=0)\\n\\n        prompt_embeds = prompt_embeds.to(device)\\n        add_text_embeds = add_text_embeds.to(device)\\n        add_time_ids = add_time_ids.to(device).repeat(batch_size * num_images_per_prompt, 1)\\n        encoder_hidden_states = torch.cat([prompt_embeds, prompt_image_emb], dim=1)\\n\\n        # 8. Denoising loop\\n        num_warmup_steps = len(timesteps) - num_inference_steps * self.scheduler.order\\n        is_unet_compiled = is_compiled_module(self.unet)\\n        \\n        with self.progress_bar(total=num_inference_steps) as progress_bar:\\n            for i, t in enumerate(timesteps):\\n                \\n                # expand the latents if we are doing classifier free guidance\\n                latent_model_input = torch.cat([latents] * 2) if self.do_classifier_free_guidance else latents\\n                latent_model_input = self.scheduler.scale_model_input(latent_model_input, t)\\n\\n                added_cond_kwargs = {\\\"text_embeds\\\": add_text_embeds, \\\"time_ids\\\": add_time_ids}\\n\\n                # predict the noise residual\\n                noise_pred = self.unet(\\n                    latent_model_input,\\n                    t,\\n                    encoder_hidden_states=encoder_hidden_states,\\n                    timestep_cond=timestep_cond,\\n                    cross_attention_kwargs=self.cross_attention_kwargs,\\n                    added_cond_kwargs=added_cond_kwargs,\\n                    return_dict=False,\\n                )[0]\\n\\n                # perform guidance\\n                if self.do_classifier_free_guidance:\\n                    noise_pred_uncond, noise_pred_text = noise_pred.chunk(2)\\n                    noise_pred = noise_pred_uncond + guidance_scale * (noise_pred_text - noise_pred_uncond)\\n\\n                # compute the previous noisy sample x_t -> x_t-1\\n                latents = self.scheduler.step(noise_pred, t, latents, **extra_step_kwargs, return_dict=False)[0]\\n\\n                if callback_on_step_end is not None:\\n                    callback_kwargs = {}\\n                    for k in callback_on_step_end_tensor_inputs:\\n                        callback_kwargs[k] = locals()[k]\\n                    callback_outputs = callback_on_step_end(self, i, t, callback_kwargs)\\n\\n                    latents = callback_outputs.pop(\\\"latents\\\", latents)\\n                    prompt_embeds = callback_outputs.pop(\\\"prompt_embeds\\\", prompt_embeds)\\n                    negative_prompt_embeds = callback_outputs.pop(\\\"negative_prompt_embeds\\\", negative_prompt_embeds)\\n\\n                # call the callback, if provided\\n                if i == len(timesteps) - 1 or ((i + 1) > num_warmup_steps and (i + 1) % self.scheduler.order == 0):\\n                    progress_bar.update()\\n                    if callback is not None and i % callback_steps == 0:\\n                        step_idx = i // getattr(self.scheduler, \\\"order\\\", 1)\\n                        callback(step_idx, t, latents)\\n        \\n        if not output_type == \\\"latent\\\":\\n            # make sure the VAE is in float32 mode, as it overflows in float16\\n            needs_upcasting = self.vae.dtype == torch.float16 and self.vae.config.force_upcast\\n\\n            if needs_upcasting:\\n                self.upcast_vae()\\n                latents = latents.to(next(iter(self.vae.post_quant_conv.parameters())).dtype)\\n\\n            # unscale/denormalize the latents\\n            # denormalize with the mean and std if available and not None\\n            has_latents_mean = hasattr(self.vae.config, \\\"latents_mean\\\") and self.vae.config.latents_mean is not None\\n            has_latents_std = hasattr(self.vae.config, \\\"latents_std\\\") and self.vae.config.latents_std is not None\\n            if has_latents_mean and has_latents_std:\\n                latents_mean = (\\n                    torch.tensor(self.vae.config.latents_mean).view(1, 4, 1, 1).to(latents.device, latents.dtype)\\n                )\\n                latents_std = (\\n                    torch.tensor(self.vae.config.latents_std).view(1, 4, 1, 1).to(latents.device, latents.dtype)\\n                )\\n                latents = latents * latents_std / self.vae.config.scaling_factor + latents_mean\\n            else:\\n                latents = latents / self.vae.config.scaling_factor\\n\\n            image = self.vae.decode(latents, return_dict=False)[0]\\n\\n            # cast back to fp16 if needed\\n            if needs_upcasting:\\n                self.vae.to(dtype=torch.float16)\\n        else:\\n            image = latents\\n\\n        if not output_type == \\\"latent\\\":\\n            # apply watermark if available\\n            if self.watermark is not None:\\n                image = self.watermark.apply_watermark(image)\\n\\n            image = self.image_processor.postprocess(image, output_type=output_type)\\n\\n        # Offload all models\\n        self.maybe_free_model_hooks()\\n\\n        if not return_dict:\\n            return (image,)\\n\\n        return StableDiffusionXLPipelineOutput(images=image)\\n\\n\\n<div align=\\\"center\\\">\\n<h1>StoryMaker: Towards consistent characters in text-to-image generation</h1>\\n\\n<a href='https://arxiv.org/pdf/2409.12576'><img src='https://img.shields.io/badge/Technique-Report-red'></a>\\n<a href='https://huggingface.co/RED-AIGC/StoryMaker'><img src='https://img.shields.io/static/v1?label=Paper&message=Huggingface&color=orange'></a> \\n\\n</div>\\nStoryMaker is a personalization solution preserves not only the consistency of faces but also clothing, hairstyles and bodies in the multiple characters scene, enabling the potential to make a story consisting of a series of images.\\n<p align=\\\"center\\\">\\n  <img src=\\\"assets/day1.png\\\">\\n  Visualization of generated images by StoryMaker. First three rows tell a story about a day in the life of a \\\"office worker\\\" and the last two rows tell a story about a movie of \\\"Before Sunrise\\\".\\n</p>\\n\\n## News\\n- [2024/09/20] 🔥 We release the [technical report](https://arxiv.org/pdf/2409.12576).\\n- [2024/09/02] 🔥 We release the [model weights](https://huggingface.co/RED-AIGC/StoryMaker).\\n\\n## Demos\\n\\n### Two Portraits Synthesis\\n\\n<p align=\\\"center\\\">\\n  <img src=\\\"assets/two.png\\\">\\n</p>\\n\\n### Diverse application\\n\\n<p align=\\\"center\\\">\\n  <img src=\\\"assets/diverse.png\\\">\\n</p>\\n\\n## Download\\n\\nYou can directly download the model from [Huggingface](https://huggingface.co/RED-AIGC/StoryMaker).\\n\\nIf you cannot access to Huggingface, you can use [hf-mirror](https://hf-mirror.com/) to download models.\\n```python\\nexport HF_ENDPOINT=https://hf-mirror.com\\nhuggingface-cli download --resume-download RED-AIGC/StoryMaker --local-dir checkpoints --local-dir-use-symlinks False\\n```\\n\\nFor face encoder, you need to manually download via this [URL](https://github.com/deepinsight/insightface/issues/1896#issuecomment-1023867304) to `models/buffalo_l` as the default link is invalid. Once you have prepared all models, the folder tree should be like:\\n\\n```\\n  .\\n  ├── models\\n  ├── checkpoints/mask.bin\\n  ├── pipeline_sdxl_storymaker.py\\n  └── README.md\\n```\\n\\n## Usage\\n\\n```python\\n# !pip install opencv-python transformers accelerate insightface\\nimport diffusers\\n\\nimport cv2\\nimport torch\\nimport numpy as np\\nfrom PIL import Image\\n\\nfrom insightface.app import FaceAnalysis\\nfrom diffusers import UniPCMultistepScheduler\\nfrom pipeline_sdxl_storymaker import StableDiffusionXLStoryMakerPipeline\\n\\n# prepare 'buffalo_l' under ./models\\napp = FaceAnalysis(name='buffalo_l', root='./', providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])\\napp.prepare(ctx_id=0, det_size=(640, 640))\\n\\n# prepare models under ./checkpoints\\nface_adapter = f'./checkpoints/mask.bin'\\nimage_encoder_path = 'laion/CLIP-ViT-H-14-laion2B-s32B-b79K'  #  from https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K\\n\\nbase_model = 'huaquan/YamerMIX_v11'  # from https://huggingface.co/huaquan/YamerMIX_v11\\npipe = StableDiffusionXLStoryMakerPipeline.from_pretrained(\\n    base_model,\\n    torch_dtype=torch.float16\\n)\\npipe.cuda()\\n\\n# load adapter\\npipe.load_storymaker_adapter(image_encoder_path, face_adapter, scale=0.8, lora_scale=0.8)\\npipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)\\n```\\n\\nThen, you can customized your own images\\n\\n```python\\n# load an image and mask\\nface_image = Image.open(\\\"examples/ldh.png\\\").convert('RGB')\\nmask_image = Image.open(\\\"examples/ldh_mask.png\\\").convert('RGB')\\n    \\nface_info = app.get(cv2.cvtColor(np.array(face_image), cv2.COLOR_RGB2BGR))\\nface_info = sorted(face_info, key=lambda x:(x['bbox'][2]-x['bbox'][0])*(x['bbox'][3]-x['bbox'][1]))[-1] # only use the maximum face\\n\\nprompt = \\\"a person is taking a selfie, the person is wearing a red hat, and a volcano is in the distance\\\"\\nn_prompt = \\\"bad quality, NSFW, low quality, ugly, disfigured, deformed\\\"\\n\\ngenerator = torch.Generator(device='cuda').manual_seed(666)\\nfor i in range(4):\\n    output = pipe(\\n        image=face_image, mask_image=mask_image, face_info=face_info,\\n        prompt=prompt,\\n        negative_prompt=n_prompt,\\n        ip_adapter_scale=0.8, lora_scale=0.8,\\n        num_inference_steps=25,\\n        guidance_scale=7.5,\\n        height=1280, width=960,\\n        generator=generator,\\n    ).images[0]\\n    output.save(f'examples/results/ldh666_new_{i}.jpg')\\n```\\n\\n\\n## Acknowledgements\\n- Our work is highly inspired by [IP-Adapter](https://github.com/tencent-ailab/IP-Adapter) and [InstantID](https://github.com/instantX-research/InstantID). Thanks for their great works!\\n- Thanks [Yamer](https://civitai.com/user/Yamer) for developing [YamerMIX](https://civitai.com/models/84040?modelVersionId=309729), we use it as base model in our demo.\\n\\n\\nimport cv2, os\\nimport torch\\nimport numpy as np\\nfrom PIL import Image\\nfrom pillow_heif import register_heif_opener\\nregister_heif_opener()\\nimport pillow_heif\\npillow_heif.register_avif_opener()  \\nfrom diffusers.utils import load_image\\nfrom diffusers import EulerAncestralDiscreteScheduler, UniPCMultistepScheduler\\n\\nfrom insightface.app import FaceAnalysis\\nfrom pipeline_sdxl_storymaker import StableDiffusionXLStoryMakerPipeline\\n\\ndef resize_img(input_image, max_side=1280, min_side=960, size=None, \\n               pad_to_max_side=False, mode=Image.BILINEAR, base_pixel_number=64):\\n\\n    w, h = input_image.size\\n    if size is not None:\\n        w_resize_new, h_resize_new = size\\n    else:\\n        ratio = min_side / min(h, w)\\n        w, h = round(ratio*w), round(ratio*h)\\n        ratio = max_side / max(h, w)\\n        input_image = input_image.resize([round(ratio*w), round(ratio*h)], mode)\\n        w_resize_new = (round(ratio * w) // base_pixel_number) * base_pixel_number\\n        h_resize_new = (round(ratio * h) // base_pixel_number) * base_pixel_number\\n    input_image = input_image.resize([w_resize_new, h_resize_new], mode)\\n\\n    if pad_to_max_side:\\n        res = np.ones([max_side, max_side, 3], dtype=np.uint8) * 255\\n        offset_x = (max_side - w_resize_new) // 2\\n        offset_y = (max_side - h_resize_new) // 2\\n        res[offset_y:offset_y+h_resize_new, offset_x:offset_x+w_resize_new] = np.array(input_image)\\n        input_image = Image.fromarray(res)\\n    return input_image\\n\\n\\n# Load face encoder\\napp = FaceAnalysis(name='buffalo_l', root='./', providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])\\napp.prepare(ctx_id=0, det_size=(640, 640))\\n\\n# Path to models\\nface_adapter = f'checkpoints/mask.bin'\\nimage_encoder_path = 'laion/CLIP-ViT-H-14-laion2B-s32B-b79K'   #  from https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K\\nbase_model_path = 'huaquan/YamerMIX_v11'  # from https://huggingface.co/huaquan/YamerMIX_v11\\n\\npipe = StableDiffusionXLStoryMakerPipeline.from_pretrained(\\n    base_model_path,\\n    torch_dtype=torch.float16,\\n)\\npipe.cuda()\\npipe.load_storymaker_adapter(image_encoder_path, face_adapter, scale=0.8, lora_scale=0.8)\\npipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)\\n\\ndef demo():\\n    prompt = \\\"a person is taking a selfie, the person is wearing a red hat, and a volcano is in the distance\\\"\\n    n_prompt = \\\"bad quality, NSFW, low quality, ugly, disfigured, deformed\\\"\\n\\n    image = Image.open(\\\"examples/ldh.png\\\").convert('RGB')\\n    mask_image = Image.open(\\\"examples/ldh_mask.png\\\").convert('RGB')\\n    \\n    # image = resize_img(image)\\n    face_info = app.get(cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR))\\n    face_info = sorted(face_info, key=lambda x:(x['bbox'][2]-x['bbox'][0])*(x['bbox'][3]-x['bbox'][1]))[-1] # only use the maximum face\\n\\n    generator = torch.Generator(device='cuda').manual_seed(666)\\n    for i in range(4):\\n        output = pipe(\\n            image=image, mask_image=mask_image, face_info=face_info,\\n            prompt=prompt,\\n            negative_prompt=n_prompt,\\n            ip_adapter_scale=0.8, lora_scale=0.8,\\n            num_inference_steps=25,\\n            guidance_scale=7.5,\\n            height=1280, width=960,\\n            generator=generator,\\n        ).images[0]\\n        output.save(f'examples/results/ldh666_{i}.jpg')\\n\\ndef demo_two():\\n    prompt = \\\"A man and a woman are taking a selfie, and a volcano is in the distance\\\"\\n    n_prompt = \\\"bad quality, NSFW, low quality, ugly, disfigured, deformed\\\"\\n\\n    image = Image.open(\\\"examples/ldh.png\\\").convert('RGB')\\n    mask_image = Image.open(\\\"examples/ldh_mask.png\\\").convert('RGB')\\n    image_2 = Image.open(\\\"examples/tsy.png\\\").convert('RGB')\\n    mask_image_2 = Image.open(\\\"examples/tsy_mask.png\\\").convert('RGB')\\n    \\n    face_info = app.get(cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR))\\n    face_info = sorted(face_info, key=lambda x:(x['bbox'][2]-x['bbox'][0])*(x['bbox'][3]-x['bbox'][1]))[-1] # only use the maximum face\\n    face_info_2 = app.get(cv2.cvtColor(np.array(image_2), cv2.COLOR_RGB2BGR))\\n    face_info_2 = sorted(face_info_2, key=lambda x:(x['bbox'][2]-x['bbox'][0])*(x['bbox'][3]-x['bbox'][1]))[-1] # only use the maximum face\\n    \\n    generator = torch.Generator(device='cuda').manual_seed(666)\\n    for i in range(4):\\n        output = pipe(\\n            image=image, mask_image=mask_image,face_info=face_info,  #  first person\\n            image_2=image_2, mask_image_2=mask_image_2,face_info_2=face_info_2,  # second person\\n            prompt=prompt,\\n            negative_prompt=n_prompt,\\n            ip_adapter_scale=0.8, lora_scale=0.8,\\n            num_inference_steps=25,\\n            guidance_scale=7.5,\\n            height=1280, width=960,\\n            generator=generator,\\n        ).images[0]\\n        output.save(f'examples/results/ldh_tsy666_{i}.jpg')\\n\\ndef demo_swapcloth():\\n    prompt = \\\"a person is taking a selfie, and a volcano is in the distance\\\"\\n    n_prompt = \\\"bad quality, NSFW, low quality, ugly, disfigured, deformed\\\"\\n\\n    image = Image.open(\\\"examples/ldh.png\\\").convert('RGB')\\n    mask_image = Image.open(\\\"examples/ldh_mask.png\\\").convert('RGB')\\n    cloth = Image.open(\\\"examples/cloth2.png\\\").convert('RGB')\\n    \\n    face_info = app.get(cv2.cvtColor(np.array(image), cv2.COLOR_RGB2BGR))\\n    face_info = sorted(face_info, key=lambda x:(x['bbox'][2]-x['bbox'][0])*(x['bbox'][3]-x['bbox'][1]))[-1] # only use the maximum face\\n\\n    generator = torch.Generator(device='cuda').manual_seed(666)\\n    for i in range(4):\\n        output = pipe(\\n            image=image, mask_image=mask_image, face_info=face_info, cloth=cloth,\\n            prompt=prompt,\\n            negative_prompt=n_prompt,\\n            ip_adapter_scale=0.8, lora_scale=0.8,\\n            num_inference_steps=25,\\n            guidance_scale=7.5,\\n            height=1280, width=960,\\n            generator=generator,\\n        ).images[0]\\n        output.save(f'examples/results/ldh_cloth_{i}.jpg')\\n\\n\\nif __name__ == \\\"__main__\\\":\\n    # single portrait generation\\n    demo()\\n\\n    # two portrait generation\\n    # demo_two()\\n\\n\\n# modified from https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py\\nimport torch\\nimport torch.nn as nn\\nimport torch.nn.functional as F\\n\\n\\nclass AttnProcessor(nn.Module):\\n    r\\\"\\\"\\\"\\n    Default processor for performing attention-related computations.\\n    \\\"\\\"\\\"\\n    def __init__(\\n        self,\\n        hidden_size=None,\\n        cross_attention_dim=None,\\n    ):\\n        super().__init__()\\n\\n    def __call__(\\n        self,\\n        attn,\\n        hidden_states,\\n        encoder_hidden_states=None,\\n        attention_mask=None,\\n        temb=None,\\n    ):\\n        residual = hidden_states\\n\\n        if attn.spatial_norm is not None:\\n            hidden_states = attn.spatial_norm(hidden_states, temb)\\n\\n        input_ndim = hidden_states.ndim\\n\\n        if input_ndim == 4:\\n            batch_size, channel, height, width = hidden_states.shape\\n            hidden_states = hidden_states.view(batch_size, channel, height * width).transpose(1, 2)\\n\\n        batch_size, sequence_length, _ = (\\n            hidden_states.shape if encoder_hidden_states is None else encoder_hidden_states.shape\\n        )\\n        attention_mask = attn.prepare_attention_mask(attention_mask, sequence_length, batch_size)\\n\\n        if attn.group_norm is not None:\\n            hidden_states = attn.group_norm(hidden_states.transpose(1, 2)).transpose(1, 2)\\n\\n        query = attn.to_q(hidden_states)\\n\\n        if encoder_hidden_states is None:\\n            encoder_hidden_states = hidden_states\\n        elif attn.norm_cross:\\n            encoder_hidden_states = attn.norm_encoder_hidden_states(encoder_hidden_states)\\n\\n        key = attn.to_k(encoder_hidden_states)\\n        value = attn.to_v(encoder_hidden_states)\\n\\n        query = attn.head_to_batch_dim(query)\\n        key = attn.head_to_batch_dim(key)\\n        value = attn.head_to_batch_dim(value)\\n\\n        attention_probs = attn.get_attention_scores(query, key, attention_mask)\\n        hidden_states = torch.bmm(attention_probs, value)\\n        hidden_states = attn.batch_to_head_dim(hidden_states)\\n\\n        # linear proj\\n        hidden_states = attn.to_out[0](hidden_states)\\n        # dropout\\n        hidden_states = attn.to_out[1](hidden_states)\\n\\n        if input_ndim == 4:\\n            hidden_states = hidden_states.transpose(-1, -2).reshape(batch_size, channel, height, width)\\n\\n        if attn.residual_connection:\\n            hidden_states = hidden_states + residual\\n\\n        hidden_states = hidden_states / attn.rescale_output_factor\\n\\n        return hidden_states\\n    \\n    \\nclass IPAttnProcessor(nn.Module):\\n    r\\\"\\\"\\\"\\n    Attention processor for IP-Adapater.\\n    Args:\\n        hidden_size (`int`):\\n            The hidden size of the attention layer.\\n        cross_attention_dim (`int`):\\n            The number of channels in the `encoder_hidden_states`.\\n        scale (`float`, defaults to 1.0):\\n            the weight scale of image prompt.\\n        num_tokens (`int`, defaults to 4 when do ip_adapter_plus it should be 16):\\n            The context length of the image features.\\n    \\\"\\\"\\\"\\n\\n    def __init__(self, hidden_size, cross_attention_dim=None, scale=1.0, num_tokens=4):\\n        super().__init__()\\n\\n        self.hidden_size = hidden_size\\n        self.cross_attention_dim = cross_attention_dim\\n        self.scale = scale\\n        self.num_tokens = num_tokens\\n\\n        self.to_k_ip = nn.Linear(cross_attention_dim or hidden_size, hidden_size, bias=False)\\n        self.to_v_ip = nn.Linear(cross_attention_dim or hidden_size, hidden_size, bias=False)\\n\\n    def __call__(\\n        self,\\n        attn,\\n        hidden_states,\\n        encoder_hidden_states=None,\\n        attention_mask=None,\\n        temb=None,\\n    ):\\n        residual = hidden_states\\n\\n        if attn.spatial_norm is not None:\\n            hidden_states = attn.spatial_norm(hidden_states, temb)\\n\\n        input_ndim = hidden_states.ndim\\n\\n        if input_ndim == 4:\\n            batch_size, channel, height, width = hidden_states.shape\\n            hidden_states = hidden_states.view(batch_size, channel, height * width).transpose(1, 2)\\n\\n        batch_size, sequence_length, _ = (\\n            hidden_states.shape if encoder_hidden_states is None else encoder_hidden_states.shape\\n        )\\n        attention_mask = attn.prepare_attention_mask(attention_mask, sequence_length, batch_size)\\n\\n        if attn.group_norm is not None:\\n            hidden_states = attn.group_norm(hidden_states.transpose(1, 2)).transpose(1, 2)\\n\\n        query = attn.to_q(hidden_states)\\n\\n        if encoder_hidden_states is None:\\n            encoder_hidden_states = hidden_states\\n        else:\\n            # get encoder_hidden_states, ip_hidden_states\\n            # end_pos = encoder_hidden_states.shape[1] - self.num_tokens\\n            end_pos = 77\\n            encoder_hidden_states, ip_hidden_states = encoder_hidden_states[:, :end_pos, :], encoder_hidden_states[:, end_pos:, :]\\n            if attn.norm_cross:\\n                encoder_hidden_states = attn.norm_encoder_hidden_states(encoder_hidden_states)\\n\\n        key = attn.to_k(encoder_hidden_states)\\n        value = attn.to_v(encoder_hidden_states)\\n\\n        query = attn.head_to_batch_dim(query)\\n        key = attn.head_to_batch_dim(key)\\n        value = attn.head_to_batch_dim(value)\\n\\n        attention_probs = attn.get_attention_scores(query, key, attention_mask)\\n        hidden_states = torch.bmm(attention_probs, value)\\n        hidden_states = attn.batch_to_head_dim(hidden_states)\\n        \\n        # for ip-adapter\\n        ip_key = self.to_k_ip(ip_hidden_states)\\n        ip_value = self.to_v_ip(ip_hidden_states)\\n        \\n        ip_key = attn.head_to_batch_dim(ip_key)\\n        ip_value = attn.head_to_batch_dim(ip_value)\\n        \\n        ip_attention_probs = attn.get_attention_scores(query, ip_key, None)\\n        ip_hidden_states = torch.bmm(ip_attention_probs, ip_value)\\n        ip_hidden_states = attn.batch_to_head_dim(ip_hidden_states)\\n        \\n        hidden_states = hidden_states + self.scale * ip_hidden_states\\n        # import pdb; pdb.set_trace()\\n        # linear proj\\n        hidden_states = attn.to_out[0](hidden_states)\\n        # dropout\\n        hidden_states = attn.to_out[1](hidden_states)\\n\\n        if input_ndim == 4:\\n            hidden_states = hidden_states.transpose(-1, -2).reshape(batch_size, channel, height, width)\\n\\n        if attn.residual_connection:\\n            hidden_states = hidden_states + residual\\n\\n        hidden_states = hidden_states / attn.rescale_output_factor\\n\\n        return hidden_states\\n    \\n    \\nclass AttnProcessor2_0(torch.nn.Module):\\n    r\\\"\\\"\\\"\\n    Processor for implementing scaled dot-product attention (enabled by default if you're using PyTorch 2.0).\\n    \\\"\\\"\\\"\\n    def __init__(\\n        self,\\n        hidden_size=None,\\n        cross_attention_dim=None,\\n    ):\\n        super().__init__()\\n        if not hasattr(F, \\\"scaled_dot_product_attention\\\"):\\n            raise ImportError(\\\"AttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.\\\")\\n\\n    def __call__(\\n        self,\\n        attn,\\n        hidden_states,\\n        encoder_hidden_states=None,\\n        attention_mask=None,\\n        temb=None,\\n    ):\\n        residual = hidden_states\\n\\n        if attn.spatial_norm is not None:\\n            hidden_states = attn.spatial_norm(hidden_states, temb)\\n\\n        input_ndim = hidden_states.ndim\\n\\n        if input_ndim == 4:\\n            batch_size, channel, height, width = hidden_states.shape\\n            hidden_states = hidden_states.view(batch_size, channel, height * width).transpose(1, 2)\\n\\n        batch_size, sequence_length, _ = (\\n            hidden_states.shape if encoder_hidden_states is None else encoder_hidden_states.shape\\n        )\\n\\n        if attention_mask is not None:\\n            attention_mask = attn.prepare_attention_mask(attention_mask, sequence_length, batch_size)\\n            # scaled_dot_product_attention expects attention_mask shape to be\\n            # (batch, heads, source_length, target_length)\\n            attention_mask = attention_mask.view(batch_size, attn.heads, -1, attention_mask.shape[-1])\\n\\n        if attn.group_norm is not None:\\n            hidden_states = attn.group_norm(hidden_states.transpose(1, 2)).transpose(1, 2)\\n\\n        query = attn.to_q(hidden_states)\\n\\n        if encoder_hidden_states is None:\\n            encoder_hidden_states = hidden_states\\n        elif attn.norm_cross:\\n            encoder_hidden_states = attn.norm_encoder_hidden_states(encoder_hidden_states)\\n\\n        key = attn.to_k(encoder_hidden_states)\\n        value = attn.to_v(encoder_hidden_states)\\n\\n        inner_dim = key.shape[-1]\\n        head_dim = inner_dim // attn.heads\\n\\n        query = query.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)\\n\\n        key = key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)\\n        value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)\\n\\n        # the output of sdp = (batch, num_heads, seq_len, head_dim)\\n        # TODO: add support for attn.scale when we move to Torch 2.1\\n        hidden_states = F.scaled_dot_product_attention(\\n            query, key, value, attn_mask=attention_mask, dropout_p=0.0, is_causal=False\\n        )\\n\\n        hidden_states = hidden_states.transpose(1, 2).reshape(batch_size, -1, attn.heads * head_dim)\\n        hidden_states = hidden_states.to(query.dtype)\\n\\n        # linear proj\\n        hidden_states = attn.to_out[0](hidden_states)\\n        # dropout\\n        hidden_states = attn.to_out[1](hidden_states)\\n\\n        if input_ndim == 4:\\n            hidden_states = hidden_states.transpose(-1, -2).reshape(batch_size, channel, height, width)\\n\\n        if attn.residual_connection:\\n            hidden_states = hidden_states + residual\\n\\n        hidden_states = hidden_states / attn.rescale_output_factor\\n\\n        return hidden_states\\n    \\n    \\nclass IPAttnProcessor2_0(torch.nn.Module):\\n    r\\\"\\\"\\\"\\n    Attention processor for IP-Adapater for PyTorch 2.0.\\n    Args:\\n        hidden_size (`int`):\\n            The hidden size of the attention layer.\\n        cross_attention_dim (`int`):\\n            The number of channels in the `encoder_hidden_states`.\\n        scale (`float`, defaults to 1.0):\\n            the weight scale of image prompt.\\n        num_tokens (`int`, defaults to 4 when do ip_adapter_plus it should be 16):\\n            The context length of the image features.\\n    \\\"\\\"\\\"\\n\\n    def __init__(self, hidden_size, cross_attention_dim=None, scale=1.0, num_tokens=4, ip_loss=0):\\n        super().__init__()\\n\\n        if not hasattr(F, \\\"scaled_dot_product_attention\\\"):\\n            raise ImportError(\\\"AttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.\\\")\\n\\n        self.hidden_size = hidden_size\\n        self.cross_attention_dim = cross_attention_dim\\n        self.scale = scale\\n        self.num_tokens = num_tokens\\n        self.ip_loss = ip_loss\\n\\n        self.to_k_ip = nn.Linear(cross_attention_dim or hidden_size, hidden_size, bias=False)\\n        self.to_v_ip = nn.Linear(cross_attention_dim or hidden_size, hidden_size, bias=False)\\n\\n    def __call__(\\n        self,\\n        attn,\\n        hidden_states,\\n        encoder_hidden_states=None,\\n        attention_mask=None,\\n        temb=None,\\n    ):\\n        residual = hidden_states\\n\\n        if attn.spatial_norm is not None:\\n            hidden_states = attn.spatial_norm(hidden_states, temb)\\n\\n        input_ndim = hidden_states.ndim\\n\\n        if input_ndim == 4:\\n            batch_size, channel, height, width = hidden_states.shape\\n            hidden_states = hidden_states.view(batch_size, channel, height * width).transpose(1, 2)\\n\\n        batch_size, sequence_length, _ = (\\n            hidden_states.shape if encoder_hidden_states is None else encoder_hidden_states.shape\\n        )\\n\\n        if attention_mask is not None:\\n            attention_mask = attn.prepare_attention_mask(attention_mask, sequence_length, batch_size)\\n            # scaled_dot_product_attention expects attention_mask shape to be\\n            # (batch, heads, source_length, target_length)\\n            attention_mask = attention_mask.view(batch_size, attn.heads, -1, attention_mask.shape[-1])\\n\\n        if attn.group_norm is not None:\\n            hidden_states = attn.group_norm(hidden_states.transpose(1, 2)).transpose(1, 2)\\n\\n        query = attn.to_q(hidden_states)\\n        if self.ip_loss>0:\\n            query2 = attn.head_to_batch_dim(query)\\n\\n        if encoder_hidden_states is None:\\n            encoder_hidden_states = hidden_states\\n        else:\\n            # get encoder_hidden_states, ip_hidden_states\\n            end_pos = encoder_hidden_states.shape[1] - self.num_tokens\\n            end_pos = 77\\n            # print(encoder_hidden_states.shape[1], self.num_tokens, end_pos)\\n            encoder_hidden_states, ip_hidden_states = encoder_hidden_states[:, :end_pos, :], encoder_hidden_states[:, end_pos:, :]\\n            if attn.norm_cross:\\n                encoder_hidden_states = attn.norm_encoder_hidden_states(encoder_hidden_states)\\n\\n        key = attn.to_k(encoder_hidden_states)\\n        value = attn.to_v(encoder_hidden_states)\\n\\n        inner_dim = key.shape[-1]\\n        head_dim = inner_dim // attn.heads\\n\\n        query = query.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)\\n\\n        key = key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)\\n        value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)\\n\\n        # the output of sdp = (batch, num_heads, seq_len, head_dim)\\n        # TODO: add support for attn.scale when we move to Torch 2.1\\n        hidden_states = F.scaled_dot_product_attention(\\n            query, key, value, attn_mask=attention_mask, dropout_p=0.0, is_causal=False\\n        )\\n\\n        hidden_states = hidden_states.transpose(1, 2).reshape(batch_size, -1, attn.heads * head_dim)\\n        hidden_states = hidden_states.to(query.dtype)\\n        \\n        # for ip-adapter\\n        ip_key = self.to_k_ip(ip_hidden_states)\\n        ip_value = self.to_v_ip(ip_hidden_states)\\n        if self.ip_loss>0:\\n            ip_key = attn.head_to_batch_dim(ip_key)\\n            ip_value = attn.head_to_batch_dim(ip_value)\\n            \\n            attention_probs = attn.get_attention_scores(query2, ip_key, attention_mask)\\n\\n            ip_hidden_states = torch.bmm(attention_probs, ip_value)\\n            ip_hidden_states = attn.batch_to_head_dim(ip_hidden_states)\\n            batch_size, seq_len, dim = attention_probs.shape\\n            head_size = attn.heads\\n            \\n            self.attn_probs = attn.batch_to_head_dim(attention_probs).reshape(batch_size // head_size, seq_len, head_size, dim).permute(0, 2, 3, 1)\\n            self.attn_probs = self.attn_probs.float().mean(dim=1)\\n        else:\\n        \\n            ip_key = ip_key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)\\n            ip_value = ip_value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)\\n\\n            # the output of sdp = (batch, num_heads, seq_len, head_dim)\\n            # TODO: add support for attn.scale when we move to Torch 2.1\\n            ip_hidden_states = F.scaled_dot_product_attention(\\n                query, ip_key, ip_value, attn_mask=None, dropout_p=0.0, is_causal=False\\n            )\\n            \\n            ip_hidden_states = ip_hidden_states.transpose(1, 2).reshape(batch_size, -1, attn.heads * head_dim)\\n            ip_hidden_states = ip_hidden_states.to(query.dtype)\\n        \\n        hidden_states = hidden_states + self.scale * ip_hidden_states\\n\\n        # linear proj\\n        hidden_states = attn.to_out[0](hidden_states)\\n        # dropout\\n        hidden_states = attn.to_out[1](hidden_states)\\n\\n        if input_ndim == 4:\\n            hidden_states = hidden_states.transpose(-1, -2).reshape(batch_size, channel, height, width)\\n\\n        if attn.residual_connection:\\n            hidden_states = hidden_states + residual\\n\\n        hidden_states = hidden_states / attn.rescale_output_factor\\n\\n        return hidden_states\\n\\n\\n## for controlnet\\nclass CNAttnProcessor:\\n    r\\\"\\\"\\\"\\n    Default processor for performing attention-related computations.\\n    \\\"\\\"\\\"\\n\\n    def __init__(self, num_tokens=4):\\n        self.num_tokens = num_tokens\\n\\n    def __call__(\\n        self,\\n        attn,\\n        hidden_states,\\n        encoder_hidden_states=None,\\n        attention_mask=None,\\n        temb=None\\n    ):\\n        residual = hidden_states\\n\\n        if attn.spatial_norm is not None:\\n            hidden_states = attn.spatial_norm(hidden_states, temb)\\n\\n        input_ndim = hidden_states.ndim\\n\\n        if input_ndim == 4:\\n            batch_size, channel, height, width = hidden_states.shape\\n            hidden_states = hidden_states.view(batch_size, channel, height * width).transpose(1, 2)\\n\\n        batch_size, sequence_length, _ = (\\n            hidden_states.shape if encoder_hidden_states is None else encoder_hidden_states.shape\\n        )\\n        attention_mask = attn.prepare_attention_mask(attention_mask, sequence_length, batch_size)\\n\\n        if attn.group_norm is not None:\\n            hidden_states = attn.group_norm(hidden_states.transpose(1, 2)).transpose(1, 2)\\n\\n        query = attn.to_q(hidden_states)\\n\\n        if encoder_hidden_states is None:\\n            encoder_hidden_states = hidden_states\\n        else:\\n            end_pos = encoder_hidden_states.shape[1] - self.num_tokens\\n            encoder_hidden_states = encoder_hidden_states[:, :end_pos] # only use text\\n            if attn.norm_cross:\\n                encoder_hidden_states = attn.norm_encoder_hidden_states(encoder_hidden_states)\\n\\n        key = attn.to_k(encoder_hidden_states)\\n        value = attn.to_v(encoder_hidden_states)\\n\\n        query = attn.head_to_batch_dim(query)\\n        key = attn.head_to_batch_dim(key)\\n        value = attn.head_to_batch_dim(value)\\n\\n        attention_probs = attn.get_attention_scores(query, key, attention_mask)\\n        hidden_states = torch.bmm(attention_probs, value)\\n        hidden_states = attn.batch_to_head_dim(hidden_states)\\n\\n        # linear proj\\n        hidden_states = attn.to_out[0](hidden_states)\\n        # dropout\\n        hidden_states = attn.to_out[1](hidden_states)\\n\\n        if input_ndim == 4:\\n            hidden_states = hidden_states.transpose(-1, -2).reshape(batch_size, channel, height, width)\\n\\n        if attn.residual_connection:\\n            hidden_states = hidden_states + residual\\n\\n        hidden_states = hidden_states / attn.rescale_output_factor\\n\\n        return hidden_states\\n\\n\\nclass CNAttnProcessor2_0:\\n    r\\\"\\\"\\\"\\n    Processor for implementing scaled dot-product attention (enabled by default if you're using PyTorch 2.0).\\n    \\\"\\\"\\\"\\n\\n    def __init__(self,  num_tokens=4):\\n        if not hasattr(F, \\\"scaled_dot_product_attention\\\"):\\n            raise ImportError(\\\"AttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.\\\")\\n        self.num_tokens = num_tokens\\n\\n    def __call__(\\n        self,\\n        attn,\\n        hidden_states,\\n        encoder_hidden_states=None,\\n        attention_mask=None,\\n        temb=None,\\n    ):\\n        residual = hidden_states\\n\\n        if attn.spatial_norm is not None:\\n            hidden_states = attn.spatial_norm(hidden_states, temb)\\n\\n        input_ndim = hidden_states.ndim\\n\\n        if input_ndim == 4:\\n            batch_size, channel, height, width = hidden_states.shape\\n            hidden_states = hidden_states.view(batch_size, channel, height * width).transpose(1, 2)\\n\\n        batch_size, sequence_length, _ = (\\n            hidden_states.shape if encoder_hidden_states is None else encoder_hidden_states.shape\\n        )\\n\\n        if attention_mask is not None:\\n            attention_mask = attn.prepare_attention_mask(attention_mask, sequence_length, batch_size)\\n            # scaled_dot_product_attention expects attention_mask shape to be\\n            # (batch, heads, source_length, target_length)\\n            attention_mask = attention_mask.view(batch_size, attn.heads, -1, attention_mask.shape[-1])\\n\\n        if attn.group_norm is not None:\\n            hidden_states = attn.group_norm(hidden_states.transpose(1, 2)).transpose(1, 2)\\n\\n        query = attn.to_q(hidden_states)\\n\\n        if encoder_hidden_states is None:\\n            encoder_hidden_states = hidden_states\\n        else:\\n            end_pos = encoder_hidden_states.shape[1] - self.num_tokens\\n            encoder_hidden_states = encoder_hidden_states[:, :end_pos] # only use text\\n            if attn.norm_cross:\\n                encoder_hidden_states = attn.norm_encoder_hidden_states(encoder_hidden_states)\\n\\n        key = attn.to_k(encoder_hidden_states)\\n        value = attn.to_v(encoder_hidden_states)\\n\\n        inner_dim = key.shape[-1]\\n        head_dim = inner_dim // attn.heads\\n\\n        query = query.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)\\n\\n        key = key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)\\n        value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)\\n\\n        # the output of sdp = (batch, num_heads, seq_len, head_dim)\\n        # TODO: add support for attn.scale when we move to Torch 2.1\\n        hidden_states = F.scaled_dot_product_attention(\\n            query, key, value, attn_mask=attention_mask, dropout_p=0.0, is_causal=False\\n        )\\n\\n        hidden_states = hidden_states.transpose(1, 2).reshape(batch_size, -1, attn.heads * head_dim)\\n        hidden_states = hidden_states.to(query.dtype)\\n\\n        # linear proj\\n        hidden_states = attn.to_out[0](hidden_states)\\n        # dropout\\n        hidden_states = attn.to_out[1](hidden_states)\\n\\n        if input_ndim == 4:\\n            hidden_states = hidden_states.transpose(-1, -2).reshape(batch_size, channel, height, width)\\n\\n        if attn.residual_connection:\\n            hidden_states = hidden_states + residual\\n\\n        hidden_states = hidden_states / attn.rescale_output_factor\\n\\n        return hidden_states\\n\\nimport torch\\nimport torch.nn.functional as F\\nimport numpy as np\\nfrom PIL import Image\\n\\nimport torch.nn.functional as F\\ndef get_generator(seed, device):\\n\\n    if seed is not None:\\n        if isinstance(seed, list):\\n            generator = [torch.Generator(device).manual_seed(seed_item) for seed_item in seed]\\n        else:\\n            generator = torch.Generator(device).manual_seed(seed)\\n    else:\\n        generator = None\\n\\n    return generator\\n\\ndef is_torch2_available():\\n    return hasattr(F, \\\"scaled_dot_product_attention\\\")\\n\\n# https://github.com/tencent-ailab/IP-Adapter/issues/54\\n# import cv2                                                                                                        \\n# import numpy as np\\n# import insightface\\n# from insightface.app import FaceAnalysis\\n# from insightface.data import get_image as ins_get_image\\n# from insightface.utils import face_align\\n\\n# app = FaceAnalysis(providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])\\n# app.prepare(ctx_id=0, det_size=(640, 640))\\n# img = cv2.imread(\\\"person.png\\\")\\n\\n# faces = app.get(img)\\n# norm_face = face_align.norm_crop(img, landmark=faces[0].kps, image_size=224)\\n\\nimport torch\\nimport torch.nn as nn\\nclass MultiHeadAttention(nn.Module):\\n    def __init__(self, d_model, num_heads):\\n        super(MultiHeadAttention, self).__init__()\\n        assert d_model % num_heads == 0, f'd_model={d_model}, numheads={num_heads}'\\n        self.num_heads = num_heads\\n        self.head_dim = d_model // num_heads\\n        self.W_q = nn.Linear(d_model, d_model)\\n        self.W_k = nn.Linear(d_model, d_model)\\n        self.W_v = nn.Linear(d_model, d_model)\\n        self.W_o = nn.Linear(d_model, d_model)\\n\\n    def split_heads(self, x, batch_size):\\n        x = x.view(batch_size, -1, self.num_heads, self.head_dim)\\n        return x.permute(0, 2, 1, 3)\\n\\n    def forward(self, query, key, value):\\n        batch_size = query.shape[0]\\n        Q = self.split_heads(self.W_q(query), batch_size)\\n        K = self.split_heads(self.W_k(key), batch_size)\\n        V = self.split_heads(self.W_v(value), batch_size)\\n\\n        scores = torch.matmul(Q, K.permute(0, 1, 3, 2)) / (self.head_dim**0.5)\\n        attention_weights = torch.nn.functional.softmax(scores, dim=-1)\\n\\n        x = torch.matmul(attention_weights, V)\\n        x = x.permute(0, 2, 1, 3).contiguous().view(batch_size, -1, self.num_heads * self.head_dim)\\n        x = self.W_o(x)\\n        return x\\n\\nclass TransformerLayer(nn.Module):\\n    def __init__(self, d_model, num_heads):\\n        super(TransformerLayer, self).__init__()\\n        self.multi_head_attention = MultiHeadAttention(d_model, num_heads)\\n        self.feed_forward = nn.Sequential(\\n            nn.Linear(d_model, 4*d_model),\\n            nn.ReLU(),\\n            nn.Linear(4*d_model, d_model)\\n        )\\n        self.layer_norm1 = nn.LayerNorm(d_model)\\n        self.layer_norm2 = nn.LayerNorm(d_model)\\n    \\n    def forward(self, x):\\n        attention_output = self.multi_head_attention(x, x, x)\\n        x = self.layer_norm1(x + attention_output)\\n        feed_forward_output = self.feed_forward(x)\\n        x = self.layer_norm2(x + feed_forward_output)\\n        return x\\n\\nclass Transformer(nn.Module):\\n    def __init__(self, d_model, num_heads, num_layers):\\n        super(Transformer, self).__init__()\\n        self.num_layers = num_layers\\n        self.embedding = nn.Linear(d_model, d_model)\\n        self.layers = nn.ModuleList([TransformerLayer(d_model, num_heads) for _ in range(num_layers)])\\n    \\n    def forward(self, x):\\n        x = self.embedding(x)\\n        for _ in range(self.num_layers):\\n            x = self.layers[_](x)\\n        return x\\n\\n# Example usage:\\n# input_dim = 512  # Dimension of the input tensor\\n# num_heads = 8    # Number of attention heads\\n# num_layers = 3   # Number of transformer layers\\n\\n# # Create an instance of the Transformer model\\n# model = Transformer(input_dim, num_heads, num_layers)\\n\\n# # Test the model with a random input tensor (batch_size, sequence_length, d_model)\\n# batch_size, sequence_length = 16, 20\\n# input_tensor = torch.randn(batch_size, sequence_length, input_dim)\\n# output = model(input_tensor)\\n\\n# print(\\\"Input shape:\\\", input_tensor.shape)\\n# print(\\\"Output shape:\\\", output.shape)\\n\\n\\n\\n\\nimport os\\nfrom typing import List\\n\\nimport torch\\nfrom diffusers import StableDiffusionPipeline\\nfrom diffusers.pipelines.controlnet import MultiControlNetModel\\nfrom transformers import CLIPVisionModelWithProjection, CLIPImageProcessor\\nfrom PIL import Image\\n\\nfrom .utils import is_torch2_available\\nif is_torch2_available():\\n    from .attention_processor import IPAttnProcessor2_0 as IPAttnProcessor, AttnProcessor2_0 as AttnProcessor, CNAttnProcessor2_0 as CNAttnProcessor\\nelse:\\n    from .attention_processor import IPAttnProcessor, AttnProcessor, CNAttnProcessor\\nfrom .resampler import Resampler\\n\\n\\nclass ImageProjModel(torch.nn.Module):\\n    \\\"\\\"\\\"Projection Model\\\"\\\"\\\"\\n    def __init__(self, cross_attention_dim=1024, clip_embeddings_dim=1024, clip_extra_context_tokens=4):\\n        super().__init__()\\n        \\n        self.cross_attention_dim = cross_attention_dim\\n        self.clip_extra_context_tokens = clip_extra_context_tokens\\n        self.proj = torch.nn.Linear(clip_embeddings_dim, self.clip_extra_context_tokens * cross_attention_dim)\\n        self.norm = torch.nn.LayerNorm(cross_attention_dim)\\n        \\n    def forward(self, image_embeds):\\n        embeds = image_embeds\\n        clip_extra_context_tokens = self.proj(embeds).reshape(-1, self.clip_extra_context_tokens, self.cross_attention_dim)\\n        clip_extra_context_tokens = self.norm(clip_extra_context_tokens)\\n        return clip_extra_context_tokens\\n\\n\\nclass IPAdapter:\\n    \\n    def __init__(self, sd_pipe, image_encoder_path, ip_ckpt, device, num_tokens=4):\\n        \\n        self.device = device\\n        self.image_encoder_path = image_encoder_path\\n        self.ip_ckpt = ip_ckpt\\n        self.num_tokens = num_tokens\\n        \\n        self.pipe = sd_pipe.to(self.device)\\n        self.set_ip_adapter()\\n        \\n        # load image encoder\\n        self.image_encoder = CLIPVisionModelWithProjection.from_pretrained(self.image_encoder_path).to(self.device, dtype=torch.float16)\\n        self.clip_image_processor = CLIPImageProcessor()\\n        # image proj model\\n        self.image_proj_model = self.init_proj()\\n        \\n        self.load_ip_adapter()\\n        \\n    def init_proj(self):\\n        image_proj_model = ImageProjModel(\\n            cross_attention_dim=self.pipe.unet.config.cross_attention_dim,\\n            clip_embeddings_dim=self.image_encoder.config.projection_dim,\\n            clip_extra_context_tokens=self.num_tokens,\\n        ).to(self.device, dtype=torch.float16)\\n        return image_proj_model\\n        \\n    def set_ip_adapter(self):\\n        unet = self.pipe.unet\\n        attn_procs = {}\\n        for name in unet.attn_processors.keys():\\n            cross_attention_dim = None if name.endswith(\\\"attn1.processor\\\") else unet.config.cross_attention_dim\\n            if name.startswith(\\\"mid_block\\\"):\\n                hidden_size = unet.config.block_out_channels[-1]\\n            elif name.startswith(\\\"up_blocks\\\"):\\n                block_id = int(name[len(\\\"up_blocks.\\\")])\\n                hidden_size = list(reversed(unet.config.block_out_channels))[block_id]\\n            elif name.startswith(\\\"down_blocks\\\"):\\n                block_id = int(name[len(\\\"down_blocks.\\\")])\\n                hidden_size = unet.config.block_out_channels[block_id]\\n            if cross_attention_dim is None:\\n                attn_procs[name] = AttnProcessor()\\n            else:\\n                attn_procs[name] = IPAttnProcessor(hidden_size=hidden_size, cross_attention_dim=cross_attention_dim,\\n                scale=1.0,num_tokens= self.num_tokens).to(self.device, dtype=torch.float16)\\n        unet.set_attn_processor(attn_procs)\\n        # if hasattr(self.pipe, \\\"controlnet\\\"):\\n        #     if isinstance(self.pipe.controlnet, MultiControlNetModel):\\n        #         for controlnet in self.pipe.controlnet.nets:\\n        #             controlnet.set_attn_processor(CNAttnProcessor(num_tokens=self.num_tokens))\\n        #     else:\\n        #         self.pipe.controlnet.set_attn_processor(CNAttnProcessor(num_tokens=self.num_tokens))\\n        \\n    def load_ip_adapter(self):\\n        state_dict = self.ip_ckpt\\n        # state_dict = torch.load(self.ip_ckpt, map_location=\\\"cpu\\\")\\n        # import pdb; pdb.set_trace()\\n        self.image_proj_model.load_state_dict(state_dict[\\\"image_proj_model\\\"])\\n        ip_layers = torch.nn.ModuleList(self.pipe.unet.attn_processors.values())\\n        ip_layers.load_state_dict(state_dict[\\\"ip_adapter\\\"])\\n        \\n    @torch.inference_mode()\\n    def get_image_embeds(self, pil_image):\\n        if isinstance(pil_image, Image.Image):\\n            pil_image = [pil_image]\\n        clip_image = self.clip_image_processor(images=pil_image, return_tensors=\\\"pt\\\").pixel_values\\n        clip_image_embeds = self.image_encoder(clip_image.to(self.device, dtype=torch.float16)).image_embeds\\n        image_prompt_embeds = self.image_proj_model(clip_image_embeds)\\n        uncond_image_prompt_embeds = self.image_proj_model(torch.zeros_like(clip_image_embeds))\\n        return image_prompt_embeds, uncond_image_prompt_embeds\\n    \\n    def set_scale(self, scale):\\n        for attn_processor in self.pipe.unet.attn_processors.values():\\n            if isinstance(attn_processor, IPAttnProcessor):\\n                attn_processor.scale = scale\\n        \\n    def generate(\\n        self,\\n        pil_image,\\n        prompt=None,\\n        negative_prompt=None,\\n        scale=1.0,\\n        num_samples=4,\\n        seed=-1,\\n        guidance_scale=7.5,\\n        num_inference_steps=30,\\n        **kwargs,\\n    ):\\n        self.set_scale(scale)\\n        \\n        if isinstance(pil_image, Image.Image):\\n            num_prompts = 1\\n        else:\\n            num_prompts = len(pil_image)\\n        \\n        if prompt is None:\\n            prompt = \\\"best quality, high quality\\\"\\n        if negative_prompt is None:\\n            negative_prompt = \\\"monochrome, lowres, bad anatomy, worst quality, low quality\\\"\\n            \\n        if not isinstance(prompt, List):\\n            prompt = [prompt] * num_prompts\\n        if not isinstance(negative_prompt, List):\\n            negative_prompt = [negative_prompt] * num_prompts\\n        \\n        image_prompt_embeds, uncond_image_prompt_embeds = self.get_image_embeds(pil_image)\\n        bs_embed, seq_len, _ = image_prompt_embeds.shape\\n        image_prompt_embeds = image_prompt_embeds.repeat(1, num_samples, 1)\\n        image_prompt_embeds = image_prompt_embeds.view(bs_embed * num_samples, seq_len, -1)\\n        uncond_image_prompt_embeds = uncond_image_prompt_embeds.repeat(1, num_samples, 1)\\n        uncond_image_prompt_embeds = uncond_image_prompt_embeds.view(bs_embed * num_samples, seq_len, -1)\\n\\n        with torch.inference_mode():\\n            prompt_embeds = self.pipe._encode_prompt(\\n                prompt, device=self.device, num_images_per_prompt=num_samples, do_classifier_free_guidance=True, negative_prompt=negative_prompt)\\n            negative_prompt_embeds_, prompt_embeds_ = prompt_embeds.chunk(2)\\n            prompt_embeds = torch.cat([prompt_embeds_, image_prompt_embeds], dim=1)\\n            negative_prompt_embeds = torch.cat([negative_prompt_embeds_, uncond_image_prompt_embeds], dim=1)\\n            \\n        generator = torch.Generator(self.device).manual_seed(seed) if seed is not None else None\\n        images = self.pipe(\\n            prompt_embeds=prompt_embeds,\\n            negative_prompt_embeds=negative_prompt_embeds,\\n            guidance_scale=guidance_scale,\\n            num_inference_steps=num_inference_steps,\\n            generator=generator,\\n            **kwargs,\\n        ).images\\n        \\n        return images\\n    \\n    \\nclass IPAdapterXL(IPAdapter):\\n    \\\"\\\"\\\"SDXL\\\"\\\"\\\"\\n    \\n    def generate(\\n        self,\\n        pil_image,\\n        prompt=None,\\n        negative_prompt=None,\\n        scale=1.0,\\n        num_samples=1,\\n        seed=-1,\\n        num_inference_steps=30,\\n        **kwargs,\\n    ):\\n        self.set_scale(scale)\\n        \\n        if isinstance(pil_image, Image.Image):\\n            num_prompts = 1\\n        else:\\n            num_prompts = len(pil_image)\\n        \\n        if prompt is None:\\n            prompt = \\\"best quality, high quality\\\"\\n        if negative_prompt is None:\\n            negative_prompt = \\\"monochrome, lowres, bad anatomy, worst quality, low quality\\\"\\n            \\n        if not isinstance(prompt, List):\\n            prompt = [prompt] * num_prompts\\n        if not isinstance(negative_prompt, List):\\n            negative_prompt = [negative_prompt] * num_prompts\\n        \\n        image_prompt_embeds, uncond_image_prompt_embeds = self.get_image_embeds(pil_image)\\n        bs_embed, seq_len, _ = image_prompt_embeds.shape\\n        image_prompt_embeds = image_prompt_embeds.repeat(1, num_samples, 1)\\n        image_prompt_embeds = image_prompt_embeds.view(bs_embed * num_samples, seq_len, -1)\\n        uncond_image_prompt_embeds = uncond_image_prompt_embeds.repeat(1, num_samples, 1)\\n        uncond_image_prompt_embeds = uncond_image_prompt_embeds.view(bs_embed * num_samples, seq_len, -1)\\n\\n        with torch.inference_mode():\\n            prompt_embeds, negative_prompt_embeds, pooled_prompt_embeds, negative_pooled_prompt_embeds = self.pipe.encode_prompt(\\n                prompt, num_images_per_prompt=num_samples, do_classifier_free_guidance=True, negative_prompt=negative_prompt)\\n            prompt_embeds = torch.cat([prompt_embeds, image_prompt_embeds], dim=1)\\n            negative_prompt_embeds = torch.cat([negative_prompt_embeds, uncond_image_prompt_embeds], dim=1)\\n            \\n        generator = torch.Generator(self.device).manual_seed(seed) if seed is not None else None\\n        images = self.pipe(\\n            prompt_embeds=prompt_embeds,\\n            negative_prompt_embeds=negative_prompt_embeds,\\n            pooled_prompt_embeds=pooled_prompt_embeds,\\n            negative_pooled_prompt_embeds=negative_pooled_prompt_embeds,\\n            num_inference_steps=num_inference_steps,\\n            generator=generator,\\n            **kwargs,\\n        ).images[0]\\n        \\n        return images\\n    \\n    \\nclass IPAdapterPlus(IPAdapter):\\n    \\\"\\\"\\\"IP-Adapter with fine-grained features\\\"\\\"\\\"\\n\\n    def init_proj(self):\\n        image_proj_model = Resampler(\\n            dim=self.pipe.unet.config.cross_attention_dim,\\n            depth=4,\\n            dim_head=64,\\n            heads=12,\\n            num_queries=self.num_tokens,\\n            embedding_dim=self.image_encoder.config.hidden_size,\\n            output_dim=self.pipe.unet.config.cross_attention_dim,\\n            ff_mult=4\\n        ).to(self.device, dtype=torch.float16)\\n        return image_proj_model\\n    \\n    @torch.inference_mode()\\n    def get_image_embeds(self, pil_image):\\n        if isinstance(pil_image, Image.Image):\\n            pil_image = [pil_image]\\n        clip_image = self.clip_image_processor(images=pil_image, return_tensors=\\\"pt\\\").pixel_values\\n        clip_image = clip_image.to(self.device, dtype=torch.float16)\\n        clip_image_embeds = self.image_encoder(clip_image, output_hidden_states=True).hidden_states[-2]\\n        image_prompt_embeds = self.image_proj_model(clip_image_embeds)\\n        uncond_clip_image_embeds = self.image_encoder(torch.zeros_like(clip_image), output_hidden_states=True).hidden_states[-2]\\n        uncond_image_prompt_embeds = self.image_proj_model(uncond_clip_image_embeds)\\n        return image_prompt_embeds, uncond_image_prompt_embeds\\n\\n\\nclass IPAdapterPlusXL(IPAdapter):\\n    \\\"\\\"\\\"SDXL\\\"\\\"\\\"\\n\\n    def init_proj(self):\\n        image_proj_model = Resampler(\\n            dim=self.pipe.unet.config.cross_attention_dim,\\n            depth=4,\\n            dim_head=64,\\n            heads=12,\\n            num_queries=self.num_tokens,\\n            embedding_dim=self.image_encoder.config.hidden_size,\\n            output_dim=self.pipe.unet.config.cross_attention_dim,\\n            ff_mult=4\\n        ).to(self.device, dtype=torch.float16)\\n        return image_proj_model\\n    \\n    @torch.inference_mode()\\n    def get_image_embeds(self, pil_image):\\n        if isinstance(pil_image, Image.Image):\\n            pil_image = [pil_image]\\n        clip_image = self.clip_image_processor(images=pil_image, return_tensors=\\\"pt\\\").pixel_values\\n        clip_image = clip_image.to(self.device, dtype=torch.float16)\\n        clip_image_embeds = self.image_encoder(clip_image, output_hidden_states=True).hidden_states[-2]\\n        image_prompt_embeds = self.image_proj_model(clip_image_embeds)\\n        # uncond_clip_image_embeds = self.image_encoder(torch.zeros_like(clip_image), output_hidden_states=True).hidden_states[-2]\\n        # uncond_image_prompt_embeds = self.image_proj_model(uncond_clip_image_embeds)\\n        uncond_image_prompt_embeds = torch.zeros_like(image_prompt_embeds)\\n        return image_prompt_embeds, uncond_image_prompt_embeds\\n    \\n    def generate(\\n        self,\\n        pil_image,\\n        prompt=None,\\n        negative_prompt=None,\\n        scale=1.0,\\n        num_samples=1,\\n        seed=-1,\\n        num_inference_steps=30,\\n        **kwargs,\\n    ):\\n        self.set_scale(scale)\\n        \\n        if isinstance(pil_image, Image.Image):\\n            num_prompts = 1\\n        else:\\n            num_prompts = len(pil_image)\\n        \\n        if prompt is None:\\n            prompt = \\\"best quality, high quality\\\"\\n        if negative_prompt is None:\\n            negative_prompt = \\\"monochrome, lowres, bad anatomy, worst quality, low quality\\\"\\n            \\n        if not isinstance(prompt, List):\\n            prompt = [prompt] * num_prompts\\n        if not isinstance(negative_prompt, List):\\n            negative_prompt = [negative_prompt] * num_prompts\\n        \\n        image_prompt_embeds, uncond_image_prompt_embeds = self.get_image_embeds(pil_image)\\n        bs_embed, seq_len, _ = image_prompt_embeds.shape\\n        image_prompt_embeds = image_prompt_embeds.repeat(1, num_samples, 1)\\n        image_prompt_embeds = image_prompt_embeds.view(bs_embed * num_samples, seq_len, -1)\\n        uncond_image_prompt_embeds = uncond_image_prompt_embeds.repeat(1, num_samples, 1)\\n        uncond_image_prompt_embeds = uncond_image_prompt_embeds.view(bs_embed * num_samples, seq_len, -1)\\n\\n        with torch.inference_mode():\\n            prompt_embeds, negative_prompt_embeds, pooled_prompt_embeds, negative_pooled_prompt_embeds = self.pipe.encode_prompt(\\n                prompt, num_images_per_prompt=num_samples, do_classifier_free_guidance=True, negative_prompt=negative_prompt)\\n            prompt_embeds = torch.cat([prompt_embeds, image_prompt_embeds], dim=1)\\n            negative_prompt_embeds = torch.cat([negative_prompt_embeds, uncond_image_prompt_embeds], dim=1)\\n            \\n        generator = torch.Generator(self.device).manual_seed(seed) if seed is not None else None\\n        images = self.pipe(\\n            prompt_embeds=prompt_embeds,\\n            negative_prompt_embeds=negative_prompt_embeds,\\n            pooled_prompt_embeds=pooled_prompt_embeds,\\n            negative_pooled_prompt_embeds=negative_pooled_prompt_embeds,\\n            num_inference_steps=num_inference_steps,\\n            generator=generator,\\n            **kwargs,\\n        ).images[0]\\n        \\n        return images\\n\\n\\n# modified from https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py\\nimport torch\\nimport torch.nn as nn\\nimport torch.nn.functional as F\\n\\nfrom diffusers.models.lora import LoRALinearLayer\\n\\n\\nclass LoRAAttnProcessor(nn.Module):\\n    r\\\"\\\"\\\"\\n    Default processor for performing attention-related computations.\\n    \\\"\\\"\\\"\\n\\n    def __init__(\\n        self,\\n        hidden_size=None,\\n        cross_attention_dim=None,\\n        rank=4,\\n        network_alpha=None,\\n        lora_scale=1.0,\\n    ):\\n        super().__init__()\\n\\n        self.rank = rank\\n        self.lora_scale = lora_scale\\n        \\n        self.to_q_lora = LoRALinearLayer(hidden_size, hidden_size, rank, network_alpha)\\n        self.to_k_lora = LoRALinearLayer(cross_attention_dim or hidden_size, hidden_size, rank, network_alpha)\\n        self.to_v_lora = LoRALinearLayer(cross_attention_dim or hidden_size, hidden_size, rank, network_alpha)\\n        self.to_out_lora = LoRALinearLayer(hidden_size, hidden_size, rank, network_alpha)\\n\\n    def __call__(\\n        self,\\n        attn,\\n        hidden_states,\\n        encoder_hidden_states=None,\\n        attention_mask=None,\\n        temb=None,\\n        *args,\\n        **kwargs,\\n    ):\\n        residual = hidden_states\\n\\n        if attn.spatial_norm is not None:\\n            hidden_states = attn.spatial_norm(hidden_states, temb)\\n\\n        input_ndim = hidden_states.ndim\\n\\n        if input_ndim == 4:\\n            batch_size, channel, height, width = hidden_states.shape\\n            hidden_states = hidden_states.view(batch_size, channel, height * width).transpose(1, 2)\\n\\n        batch_size, sequence_length, _ = (\\n            hidden_states.shape if encoder_hidden_states is None else encoder_hidden_states.shape\\n        )\\n        attention_mask = attn.prepare_attention_mask(attention_mask, sequence_length, batch_size)\\n\\n        if attn.group_norm is not None:\\n            hidden_states = attn.group_norm(hidden_states.transpose(1, 2)).transpose(1, 2)\\n\\n        query = attn.to_q(hidden_states) + self.lora_scale * self.to_q_lora(hidden_states)\\n\\n        if encoder_hidden_states is None:\\n            encoder_hidden_states = hidden_states\\n        elif attn.norm_cross:\\n            encoder_hidden_states = attn.norm_encoder_hidden_states(encoder_hidden_states)\\n\\n        key = attn.to_k(encoder_hidden_states) + self.lora_scale * self.to_k_lora(encoder_hidden_states)\\n        value = attn.to_v(encoder_hidden_states) + self.lora_scale * self.to_v_lora(encoder_hidden_states)\\n\\n        query = attn.head_to_batch_dim(query)\\n        key = attn.head_to_batch_dim(key)\\n        value = attn.head_to_batch_dim(value)\\n\\n        attention_probs = attn.get_attention_scores(query, key, attention_mask)\\n        hidden_states = torch.bmm(attention_probs, value)\\n        hidden_states = attn.batch_to_head_dim(hidden_states)\\n\\n        # linear proj\\n        hidden_states = attn.to_out[0](hidden_states) + self.lora_scale * self.to_out_lora(hidden_states)\\n        # dropout\\n        hidden_states = attn.to_out[1](hidden_states)\\n\\n        if input_ndim == 4:\\n            hidden_states = hidden_states.transpose(-1, -2).reshape(batch_size, channel, height, width)\\n\\n        if attn.residual_connection:\\n            hidden_states = hidden_states + residual\\n\\n        hidden_states = hidden_states / attn.rescale_output_factor\\n\\n        return hidden_states\\n\\n\\nclass LoRAIPAttnProcessor(nn.Module):\\n    r\\\"\\\"\\\"\\n    Attention processor for IP-Adapater.\\n    Args:\\n        hidden_size (`int`):\\n            The hidden size of the attention layer.\\n        cross_attention_dim (`int`):\\n            The number of channels in the `encoder_hidden_states`.\\n        scale (`float`, defaults to 1.0):\\n            the weight scale of image prompt.\\n        num_tokens (`int`, defaults to 4 when do ip_adapter_plus it should be 16):\\n            The context length of the image features.\\n    \\\"\\\"\\\"\\n\\n    def __init__(self, hidden_size, cross_attention_dim=None, rank=4, network_alpha=None, lora_scale=1.0, scale=1.0, num_tokens=4):\\n        super().__init__()\\n\\n        self.rank = rank\\n        self.lora_scale = lora_scale\\n        \\n        self.to_q_lora = LoRALinearLayer(hidden_size, hidden_size, rank, network_alpha)\\n        self.to_k_lora = LoRALinearLayer(cross_attention_dim or hidden_size, hidden_size, rank, network_alpha)\\n        self.to_v_lora = LoRALinearLayer(cross_attention_dim or hidden_size, hidden_size, rank, network_alpha)\\n        self.to_out_lora = LoRALinearLayer(hidden_size, hidden_size, rank, network_alpha)\\n\\n        self.hidden_size = hidden_size\\n        self.cross_attention_dim = cross_attention_dim\\n        self.scale = scale\\n        self.num_tokens = num_tokens\\n\\n        self.to_k_ip = nn.Linear(cross_attention_dim or hidden_size, hidden_size, bias=False)\\n        self.to_v_ip = nn.Linear(cross_attention_dim or hidden_size, hidden_size, bias=False)\\n\\n    def __call__(\\n        self,\\n        attn,\\n        hidden_states,\\n        encoder_hidden_states=None,\\n        attention_mask=None,\\n        temb=None,\\n        *args,\\n        **kwargs,\\n    ):\\n        residual = hidden_states\\n\\n        if attn.spatial_norm is not None:\\n            hidden_states = attn.spatial_norm(hidden_states, temb)\\n\\n        input_ndim = hidden_states.ndim\\n\\n        if input_ndim == 4:\\n            batch_size, channel, height, width = hidden_states.shape\\n            hidden_states = hidden_states.view(batch_size, channel, height * width).transpose(1, 2)\\n\\n        batch_size, sequence_length, _ = (\\n            hidden_states.shape if encoder_hidden_states is None else encoder_hidden_states.shape\\n        )\\n        attention_mask = attn.prepare_attention_mask(attention_mask, sequence_length, batch_size)\\n\\n        if attn.group_norm is not None:\\n            hidden_states = attn.group_norm(hidden_states.transpose(1, 2)).transpose(1, 2)\\n\\n        query = attn.to_q(hidden_states) + self.lora_scale * self.to_q_lora(hidden_states)\\n\\n        if encoder_hidden_states is None:\\n            encoder_hidden_states = hidden_states\\n        else:\\n            # get encoder_hidden_states, ip_hidden_states\\n            # end_pos = encoder_hidden_states.shape[1] - self.num_tokens\\n            end_pos = 77\\n            encoder_hidden_states, ip_hidden_states = (\\n                encoder_hidden_states[:, :end_pos, :],\\n                encoder_hidden_states[:, end_pos:, :],\\n            )\\n            if attn.norm_cross:\\n                encoder_hidden_states = attn.norm_encoder_hidden_states(encoder_hidden_states)\\n\\n        key = attn.to_k(encoder_hidden_states) + self.lora_scale * self.to_k_lora(encoder_hidden_states)\\n        value = attn.to_v(encoder_hidden_states) + self.lora_scale * self.to_v_lora(encoder_hidden_states)\\n\\n        query = attn.head_to_batch_dim(query)\\n        key = attn.head_to_batch_dim(key)\\n        value = attn.head_to_batch_dim(value)\\n\\n        attention_probs = attn.get_attention_scores(query, key, attention_mask)\\n        hidden_states = torch.bmm(attention_probs, value)\\n        hidden_states = attn.batch_to_head_dim(hidden_states)\\n\\n        # for ip-adapter\\n        ip_key = self.to_k_ip(ip_hidden_states)\\n        ip_value = self.to_v_ip(ip_hidden_states)\\n\\n        ip_key = attn.head_to_batch_dim(ip_key)\\n        ip_value = attn.head_to_batch_dim(ip_value)\\n\\n        ip_attention_probs = attn.get_attention_scores(query, ip_key, None)\\n        self.attn_map = ip_attention_probs\\n        ip_hidden_states = torch.bmm(ip_attention_probs, ip_value)\\n        ip_hidden_states = attn.batch_to_head_dim(ip_hidden_states)\\n\\n        hidden_states = hidden_states + self.scale * ip_hidden_states\\n\\n        # linear proj\\n        hidden_states = attn.to_out[0](hidden_states) + self.lora_scale * self.to_out_lora(hidden_states)\\n        # dropout\\n        hidden_states = attn.to_out[1](hidden_states)\\n\\n        if input_ndim == 4:\\n            hidden_states = hidden_states.transpose(-1, -2).reshape(batch_size, channel, height, width)\\n\\n        if attn.residual_connection:\\n            hidden_states = hidden_states + residual\\n\\n        hidden_states = hidden_states / attn.rescale_output_factor\\n\\n        return hidden_states\\n\\n\\nclass LoRAAttnProcessor2_0(nn.Module):\\n    \\n    r\\\"\\\"\\\"\\n    Default processor for performing attention-related computations.\\n    \\\"\\\"\\\"\\n    \\n    def __init__(\\n        self,\\n        hidden_size=None,\\n        cross_attention_dim=None,\\n        rank=4,\\n        network_alpha=None,\\n        lora_scale=1.0,\\n    ):\\n        super().__init__()\\n        \\n        self.rank = rank\\n        self.lora_scale = lora_scale\\n        \\n        self.to_q_lora = LoRALinearLayer(hidden_size, hidden_size, rank, network_alpha)\\n        self.to_k_lora = LoRALinearLayer(cross_attention_dim or hidden_size, hidden_size, rank, network_alpha)\\n        self.to_v_lora = LoRALinearLayer(cross_attention_dim or hidden_size, hidden_size, rank, network_alpha)\\n        self.to_out_lora = LoRALinearLayer(hidden_size, hidden_size, rank, network_alpha)\\n\\n    def __call__(\\n        self,\\n        attn,\\n        hidden_states,\\n        encoder_hidden_states=None,\\n        attention_mask=None,\\n        temb=None,\\n        *args,\\n        **kwargs,\\n    ):\\n        residual = hidden_states\\n\\n        if attn.spatial_norm is not None:\\n            hidden_states = attn.spatial_norm(hidden_states, temb)\\n\\n        input_ndim = hidden_states.ndim\\n\\n        if input_ndim == 4:\\n            batch_size, channel, height, width = hidden_states.shape\\n            hidden_states = hidden_states.view(batch_size, channel, height * width).transpose(1, 2)\\n\\n        batch_size, sequence_length, _ = (\\n            hidden_states.shape if encoder_hidden_states is None else encoder_hidden_states.shape\\n        )\\n        attention_mask = attn.prepare_attention_mask(attention_mask, sequence_length, batch_size)\\n\\n        if attn.group_norm is not None:\\n            hidden_states = attn.group_norm(hidden_states.transpose(1, 2)).transpose(1, 2)\\n\\n        query = attn.to_q(hidden_states) + self.lora_scale * self.to_q_lora(hidden_states)\\n\\n        if encoder_hidden_states is None:\\n            encoder_hidden_states = hidden_states\\n        elif attn.norm_cross:\\n            encoder_hidden_states = attn.norm_encoder_hidden_states(encoder_hidden_states)\\n\\n        key = attn.to_k(encoder_hidden_states) + self.lora_scale * self.to_k_lora(encoder_hidden_states)\\n        value = attn.to_v(encoder_hidden_states) + self.lora_scale * self.to_v_lora(encoder_hidden_states)\\n\\n        inner_dim = key.shape[-1]\\n        head_dim = inner_dim // attn.heads\\n\\n        query = query.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)\\n\\n        key = key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)\\n        value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)\\n\\n        # the output of sdp = (batch, num_heads, seq_len, head_dim)\\n        # TODO: add support for attn.scale when we move to Torch 2.1\\n        hidden_states = F.scaled_dot_product_attention(\\n            query, key, value, attn_mask=attention_mask, dropout_p=0.0, is_causal=False\\n        )\\n\\n        hidden_states = hidden_states.transpose(1, 2).reshape(batch_size, -1, attn.heads * head_dim)\\n        hidden_states = hidden_states.to(query.dtype)\\n\\n        # linear proj\\n        hidden_states = attn.to_out[0](hidden_states) + self.lora_scale * self.to_out_lora(hidden_states)\\n        # dropout\\n        hidden_states = attn.to_out[1](hidden_states)\\n\\n        if input_ndim == 4:\\n            hidden_states = hidden_states.transpose(-1, -2).reshape(batch_size, channel, height, width)\\n\\n        if attn.residual_connection:\\n            hidden_states = hidden_states + residual\\n\\n        hidden_states = hidden_states / attn.rescale_output_factor\\n\\n        return hidden_states\\n\\n\\nclass LoRAIPAttnProcessor2_0(nn.Module):\\n    r\\\"\\\"\\\"\\n    Processor for implementing the LoRA attention mechanism.\\n\\n    Args:\\n        hidden_size (`int`, *optional*):\\n            The hidden size of the attention layer.\\n        cross_attention_dim (`int`, *optional*):\\n            The number of channels in the `encoder_hidden_states`.\\n        rank (`int`, defaults to 4):\\n            The dimension of the LoRA update matrices.\\n        network_alpha (`int`, *optional*):\\n            Equivalent to `alpha` but it's usage is specific to Kohya (A1111) style LoRAs.\\n    \\\"\\\"\\\"\\n\\n    def __init__(self, hidden_size, cross_attention_dim=None, rank=4, network_alpha=None, lora_scale=1.0, scale=1.0, num_tokens=4):\\n        super().__init__()\\n        \\n        self.rank = rank\\n        self.lora_scale = lora_scale\\n        self.num_tokens = num_tokens\\n        \\n        self.to_q_lora = LoRALinearLayer(hidden_size, hidden_size, rank, network_alpha)\\n        self.to_k_lora = LoRALinearLayer(cross_attention_dim or hidden_size, hidden_size, rank, network_alpha)\\n        self.to_v_lora = LoRALinearLayer(cross_attention_dim or hidden_size, hidden_size, rank, network_alpha)\\n        self.to_out_lora = LoRALinearLayer(hidden_size, hidden_size, rank, network_alpha)\\n        \\n        \\n        self.hidden_size = hidden_size\\n        self.cross_attention_dim = cross_attention_dim\\n        self.scale = scale\\n\\n        self.to_k_ip = nn.Linear(cross_attention_dim or hidden_size, hidden_size, bias=False)\\n        self.to_v_ip = nn.Linear(cross_attention_dim or hidden_size, hidden_size, bias=False)\\n\\n    def __call__(\\n        self, attn, hidden_states, encoder_hidden_states=None, attention_mask=None, scale=1.0, temb=None, *args, **kwargs,\\n    ):\\n        residual = hidden_states\\n\\n        if attn.spatial_norm is not None:\\n            hidden_states = attn.spatial_norm(hidden_states, temb)\\n\\n        input_ndim = hidden_states.ndim\\n\\n        if input_ndim == 4:\\n            batch_size, channel, height, width = hidden_states.shape\\n            hidden_states = hidden_states.view(batch_size, channel, height * width).transpose(1, 2)\\n\\n        batch_size, sequence_length, _ = (\\n            hidden_states.shape if encoder_hidden_states is None else encoder_hidden_states.shape\\n        )\\n        attention_mask = attn.prepare_attention_mask(attention_mask, sequence_length, batch_size)\\n\\n        if attn.group_norm is not None:\\n            hidden_states = attn.group_norm(hidden_states.transpose(1, 2)).transpose(1, 2)\\n\\n        query = attn.to_q(hidden_states) + self.lora_scale * self.to_q_lora(hidden_states)\\n        \\n        if encoder_hidden_states is None:\\n            encoder_hidden_states = hidden_states\\n        else:\\n            end_pos = 77\\n            encoder_hidden_states, ip_hidden_states = ( encoder_hidden_states[:, :end_pos, :], encoder_hidden_states[:, end_pos:, :], )\\n\\n            if attn.norm_cross:\\n                encoder_hidden_states = attn.norm_encoder_hidden_states(encoder_hidden_states)\\n        # for text\\n        key = attn.to_k(encoder_hidden_states) + self.lora_scale * self.to_k_lora(encoder_hidden_states)\\n        value = attn.to_v(encoder_hidden_states) + self.lora_scale * self.to_v_lora(encoder_hidden_states)\\n\\n        inner_dim = key.shape[-1]\\n        head_dim = inner_dim // attn.heads\\n\\n        query = query.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)\\n\\n        key = key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)\\n        value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)\\n\\n        # the output of sdp = (batch, num_heads, seq_len, head_dim)\\n        # TODO: add support for attn.scale when we move to Torch 2.1\\n        hidden_states = F.scaled_dot_product_attention(\\n            query, key, value, attn_mask=attention_mask, dropout_p=0.0, is_causal=False\\n        )\\n\\n        hidden_states = hidden_states.transpose(1, 2).reshape(batch_size, -1, attn.heads * head_dim)\\n        hidden_states = hidden_states.to(query.dtype)\\n        \\n        # for ip\\n        ip_key = self.to_k_ip(ip_hidden_states)\\n        ip_value = self.to_v_ip(ip_hidden_states)\\n            \\n        ip_key = ip_key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)\\n        ip_value = ip_value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)\\n        # the output of sdp = (batch, num_heads, seq_len, head_dim)\\n        # TODO: add support for attn.scale when we move to Torch 2.1\\n        ip_hidden_states = F.scaled_dot_product_attention(\\n            query, ip_key, ip_value, attn_mask=None, dropout_p=0.0, is_causal=False\\n        )\\n        ip_hidden_states = ip_hidden_states.transpose(1, 2).reshape(batch_size, -1, attn.heads * head_dim)\\n        ip_hidden_states = ip_hidden_states.to(query.dtype)\\n        \\n        hidden_states = hidden_states + self.scale * ip_hidden_states\\n\\n        # linear proj\\n        hidden_states = attn.to_out[0](hidden_states) + self.lora_scale * self.to_out_lora(hidden_states)\\n        # dropout\\n        hidden_states = attn.to_out[1](hidden_states)\\n\\n        if input_ndim == 4:\\n            hidden_states = hidden_states.transpose(-1, -2).reshape(batch_size, channel, height, width)\\n\\n        if attn.residual_connection:\\n            hidden_states = hidden_states + residual\\n\\n        hidden_states = hidden_states / attn.rescale_output_factor\\n\\n        return hidden_states\\n\\n\\n\\n# modified from https://github.com/mlfoundations/open_flamingo/blob/main/open_flamingo/src/helpers.py\\nimport math\\n\\nimport torch\\nimport torch.nn as nn\\n\\n\\n# FFN\\ndef FeedForward(dim, mult=4):\\n    inner_dim = int(dim * mult)\\n    return nn.Sequential(\\n        nn.LayerNorm(dim),\\n        nn.Linear(dim, inner_dim, bias=False),\\n        nn.GELU(),\\n        nn.Linear(inner_dim, dim, bias=False),\\n    )\\n    \\n    \\ndef reshape_tensor(x, heads):\\n    bs, length, width = x.shape\\n    #(bs, length, width) --> (bs, length, n_heads, dim_per_head)\\n    x = x.view(bs, length, heads, -1)\\n    # (bs, length, n_heads, dim_per_head) --> (bs, n_heads, length, dim_per_head)\\n    x = x.transpose(1, 2)\\n    # (bs, n_heads, length, dim_per_head) --> (bs*n_heads, length, dim_per_head)\\n    x = x.reshape(bs, heads, length, -1)\\n    return x\\n\\n\\nclass PerceiverAttention(nn.Module):\\n    def __init__(self, *, dim, dim_head=64, heads=8):\\n        super().__init__()\\n        self.scale = dim_head**-0.5\\n        self.dim_head = dim_head\\n        self.heads = heads\\n        inner_dim = dim_head * heads\\n\\n        self.norm1 = nn.LayerNorm(dim)\\n        self.norm2 = nn.LayerNorm(dim)\\n\\n        self.to_q = nn.Linear(dim, inner_dim, bias=False)\\n        self.to_kv = nn.Linear(dim, inner_dim * 2, bias=False)\\n        self.to_out = nn.Linear(inner_dim, dim, bias=False)\\n\\n\\n    def forward(self, x, latents):\\n        \\\"\\\"\\\"\\n        Args:\\n            x (torch.Tensor): image features\\n                shape (b, n1, D)\\n            latent (torch.Tensor): latent features\\n                shape (b, n2, D)\\n        \\\"\\\"\\\"\\n        x = self.norm1(x)\\n        latents = self.norm2(latents)\\n        \\n        b, l, _ = latents.shape\\n\\n        q = self.to_q(latents)\\n        kv_input = torch.cat((x, latents), dim=-2)\\n        k, v = self.to_kv(kv_input).chunk(2, dim=-1)\\n        \\n        q = reshape_tensor(q, self.heads)\\n        k = reshape_tensor(k, self.heads)\\n        v = reshape_tensor(v, self.heads)\\n\\n        # attention\\n        scale = 1 / math.sqrt(math.sqrt(self.dim_head))\\n        weight = (q * scale) @ (k * scale).transpose(-2, -1) # More stable with f16 than dividing afterwards\\n        weight = torch.softmax(weight.float(), dim=-1).type(weight.dtype)\\n        out = weight @ v\\n        \\n        out = out.permute(0, 2, 1, 3).reshape(b, l, -1)\\n\\n        return self.to_out(out)\\n\\n\\nclass Resampler(nn.Module):\\n    def __init__(\\n        self,\\n        dim=1024,\\n        depth=8,\\n        dim_head=64,\\n        heads=16,\\n        num_queries=8,\\n        embedding_dim=768,\\n        output_dim=1024,\\n        ff_mult=4,\\n    ):\\n        super().__init__()\\n        \\n        self.latents = nn.Parameter(torch.randn(1, num_queries, dim) / dim**0.5)\\n        \\n        self.proj_in = nn.Linear(embedding_dim, dim)\\n\\n        self.proj_out = nn.Linear(dim, output_dim)\\n        self.norm_out = nn.LayerNorm(output_dim)\\n        \\n        self.layers = nn.ModuleList([])\\n        for _ in range(depth):\\n            self.layers.append(\\n                nn.ModuleList(\\n                    [\\n                        PerceiverAttention(dim=dim, dim_head=dim_head, heads=heads),\\n                        FeedForward(dim=dim, mult=ff_mult),\\n                    ]\\n                )\\n            )\\n\\n    def forward(self, x):\\n        \\n        latents = self.latents.repeat(x.size(0), 1, 1)\\n        \\n        x = self.proj_in(x)\\n        \\n        for attn, ff in self.layers:\\n            latents = attn(x, latents) + latents\\n            latents = ff(latents) + latents\\n            \\n        latents = self.proj_out(latents)\\n        return self.norm_out(latents)\\n\\nimport os\\nfrom typing import List\\n\\nimport torch, random, pdb\\nfrom diffusers import StableDiffusionPipeline\\nfrom diffusers.pipelines.controlnet import MultiControlNetModel\\nfrom PIL import Image\\nfrom safetensors import safe_open\\nfrom transformers import CLIPImageProcessor, CLIPVisionModelWithProjection\\n\\nfrom .attention_processor_faceid import LoRAAttnProcessor, LoRAIPAttnProcessor\\nfrom .utils import is_torch2_available, get_generator\\n\\nUSE_DAFAULT_ATTN = False # should be True for visualization_attnmap\\nif is_torch2_available() and (not USE_DAFAULT_ATTN):\\n    from .attention_processor_faceid import (\\n        LoRAAttnProcessor2_0 as LoRAAttnProcessor,\\n    )\\n    from .attention_processor_faceid import (\\n        LoRAIPAttnProcessor2_0 as LoRAIPAttnProcessor,\\n    )\\nelse:\\n    from .attention_processor_faceid import LoRAAttnProcessor, LoRAIPAttnProcessor\\nfrom .resampler import PerceiverAttention, FeedForward, Resampler\\n\\nclass FacePerceiverResampler(torch.nn.Module):\\n    def __init__(\\n        self,\\n        *,\\n        dim=768,\\n        depth=4,\\n        dim_head=64,\\n        heads=16,\\n        embedding_dim=1280,\\n        output_dim=768,\\n        ff_mult=4,\\n    ):\\n        super().__init__()\\n        \\n        self.proj_in = torch.nn.Linear(embedding_dim, dim)\\n        self.proj_out = torch.nn.Linear(dim, output_dim)\\n        self.norm_out = torch.nn.LayerNorm(output_dim)\\n        self.layers = torch.nn.ModuleList([])\\n        for _ in range(depth):\\n            self.layers.append(\\n                torch.nn.ModuleList(\\n                    [\\n                        PerceiverAttention(dim=dim, dim_head=dim_head, heads=heads),\\n                        FeedForward(dim=dim, mult=ff_mult),\\n                    ]\\n                )\\n            )\\n\\n    def forward(self, latents, x):\\n        x = self.proj_in(x)\\n        for attn, ff in self.layers:\\n            latents = attn(x, latents) + latents\\n            latents = ff(latents) + latents\\n        latents = self.proj_out(latents)\\n        return self.norm_out(latents)\\n\\nclass faceid_plus(torch.nn.Module):\\n    def __init__(self, cross_attention_dim=2048, id_embeddings_dim=512, clip_embeddings_dim=1280,):\\n        super().__init__()\\n        self.cross_attention_dim = cross_attention_dim\\n        self.num_tokens = 4\\n        self.proj = torch.nn.Sequential(\\n            torch.nn.Linear(id_embeddings_dim, id_embeddings_dim*2),\\n            torch.nn.GELU(),\\n            torch.nn.Linear(id_embeddings_dim*2, cross_attention_dim*4),\\n        )\\n        self.norm = torch.nn.LayerNorm(cross_attention_dim)\\n        self.pos_embed = torch.nn.Parameter(torch.zeros(3, 4+16, cross_attention_dim))  #  maxperson=3\\n        self.bg_embed = torch.nn.Parameter(torch.zeros(1, 4+16, cross_attention_dim))  #  one bg embedding\\n        \\n        self.proj_out = torch.nn.Linear(cross_attention_dim, cross_attention_dim)\\n        self.norm_out = torch.nn.LayerNorm(cross_attention_dim)\\n        torch.nn.init.zeros_(self.proj_out.weight); torch.nn.init.zeros_(self.proj_out.bias)\\n        self.perceiver_resampler = FacePerceiverResampler(\\n            dim=cross_attention_dim,\\n            depth=4,\\n            dim_head=64,\\n            heads=cross_attention_dim // 64,\\n            embedding_dim=clip_embeddings_dim,\\n            output_dim=cross_attention_dim,\\n            ff_mult=4,\\n        )\\n        self.resample = Resampler(\\n            dim=1280,  depth=4, dim_head=64, heads= 20, num_queries=16,\\n            embedding_dim=clip_embeddings_dim, output_dim=cross_attention_dim, ff_mult=4 )\\n        \\n    def forward(self, id_embeds, clip_embeds, face_embeds):\\n        x = self.proj(id_embeds)\\n        x = x.reshape(-1, 4, self.cross_attention_dim)\\n        x = self.norm(x)\\n        out = self.perceiver_resampler(x, face_embeds)\\n        out = x + out\\n        clip = self.resample(clip_embeds)\\n        \\n        B = clip_embeds.shape[0]\\n        cat = torch.cat([out, clip], dim=1)+self.pos_embed[:B]   #  B, 20, 2048\\n        res = self.norm_out(self.proj_out(cat))+cat\\n        bg_embed = torch.zeros_like(self.bg_embed) if id_embeds.sum().abs()<1e-2 else self.bg_embed\\n        res = torch.cat([self.bg_embed, res], dim=0)  # :20 is bg emb, 20:80 is 3 ip emb\\n        return res\\n\\nclass IPAdapterFaceID:\\n    def __init__(self, sd_pipe, ip_ckpt, device, lora_rank=128, num_tokens=4, torch_dtype=torch.float16):\\n        self.device = device\\n        self.ip_ckpt = ip_ckpt\\n        self.lora_rank = lora_rank\\n        self.num_tokens = num_tokens\\n        self.torch_dtype = torch_dtype\\n\\n        self.pipe = sd_pipe.to(self.device)\\n        self.set_ip_adapter()\\n\\n        # image proj model\\n        self.image_proj_model = self.init_proj()\\n\\n        self.load_ip_adapter()\\n\\n    def init_proj(self):\\n        image_proj_model = MLPProjModel(\\n            cross_attention_dim=self.pipe.unet.config.cross_attention_dim,\\n            id_embeddings_dim=512,\\n            num_tokens=self.num_tokens,\\n        ).to(self.device, dtype=self.torch_dtype)\\n        return image_proj_model\\n\\n    def set_ip_adapter(self):\\n        unet = self.pipe.unet\\n        attn_procs = {}\\n        for name in unet.attn_processors.keys():\\n            cross_attention_dim = None if name.endswith(\\\"attn1.processor\\\") else unet.config.cross_attention_dim\\n            if name.startswith(\\\"mid_block\\\"):\\n                hidden_size = unet.config.block_out_channels[-1]\\n            elif name.startswith(\\\"up_blocks\\\"):\\n                block_id = int(name[len(\\\"up_blocks.\\\")])\\n                hidden_size = list(reversed(unet.config.block_out_channels))[block_id]\\n            elif name.startswith(\\\"down_blocks\\\"):\\n                block_id = int(name[len(\\\"down_blocks.\\\")])\\n                hidden_size = unet.config.block_out_channels[block_id]\\n            if cross_attention_dim is None:\\n                attn_procs[name] = LoRAAttnProcessor(\\n                    hidden_size=hidden_size, cross_attention_dim=cross_attention_dim, rank=self.lora_rank,\\n                ).to(self.device, dtype=self.torch_dtype)\\n            else:\\n                attn_procs[name] = LoRAIPAttnProcessor(\\n                    hidden_size=hidden_size, cross_attention_dim=cross_attention_dim, scale=1.0, rank=self.lora_rank, num_tokens=self.num_tokens,\\n                ).to(self.device, dtype=self.torch_dtype)\\n        unet.set_attn_processor(attn_procs)\\n\\n    def load_ip_adapter(self):\\n        if os.path.splitext(self.ip_ckpt)[-1] == \\\".safetensors\\\":\\n            state_dict = {\\\"image_proj\\\": {}, \\\"ip_adapter\\\": {}}\\n            with safe_open(self.ip_ckpt, framework=\\\"pt\\\", device=\\\"cpu\\\") as f:\\n                for key in f.keys():\\n                    if key.startswith(\\\"image_proj.\\\"):\\n                        state_dict[\\\"image_proj\\\"][key.replace(\\\"image_proj.\\\", \\\"\\\")] = f.get_tensor(key)\\n                    elif key.startswith(\\\"ip_adapter.\\\"):\\n                        state_dict[\\\"ip_adapter\\\"][key.replace(\\\"ip_adapter.\\\", \\\"\\\")] = f.get_tensor(key)\\n        else:\\n            state_dict = torch.load(self.ip_ckpt, map_location=\\\"cpu\\\")\\n        self.image_proj_model.load_state_dict(state_dict[\\\"image_proj\\\"])\\n        ip_layers = torch.nn.ModuleList(self.pipe.unet.attn_processors.values())\\n        ip_layers.load_state_dict(state_dict[\\\"ip_adapter\\\"])\\n\\n    @torch.inference_mode()\\n    def get_image_embeds(self, faceid_embeds):\\n        \\n        faceid_embeds = faceid_embeds.to(self.device, dtype=self.torch_dtype)\\n        image_prompt_embeds = self.image_proj_model(faceid_embeds)\\n        uncond_image_prompt_embeds = self.image_proj_model(torch.zeros_like(faceid_embeds))\\n        return image_prompt_embeds, uncond_image_prompt_embeds\\n\\n    def set_scale(self, scale):\\n        for attn_processor in self.pipe.unet.attn_processors.values():\\n            if isinstance(attn_processor, LoRAIPAttnProcessor):\\n                attn_processor.scale = scale\\n\\n    def generate(\\n        self,\\n        faceid_embeds=None,\\n        prompt=None,\\n        negative_prompt=None,\\n        scale=1.0,\\n        num_samples=4,\\n        seed=None,\\n        guidance_scale=7.5,\\n        num_inference_steps=30,\\n        **kwargs,\\n    ):\\n        self.set_scale(scale)\\n\\n       \\n        num_prompts = faceid_embeds.size(0)\\n\\n        if prompt is None:\\n            prompt = \\\"best quality, high quality\\\"\\n        if negative_prompt is None:\\n            negative_prompt = \\\"monochrome, lowres, bad anatomy, worst quality, low quality\\\"\\n\\n        if not isinstance(prompt, List):\\n            prompt = [prompt] * num_prompts\\n        if not isinstance(negative_prompt, List):\\n            negative_prompt = [negative_prompt] * num_prompts\\n\\n        image_prompt_embeds, uncond_image_prompt_embeds = self.get_image_embeds(faceid_embeds)\\n\\n        bs_embed, seq_len, _ = image_prompt_embeds.shape\\n        image_prompt_embeds = image_prompt_embeds.repeat(1, num_samples, 1)\\n        image_prompt_embeds = image_prompt_embeds.view(bs_embed * num_samples, seq_len, -1)\\n        uncond_image_prompt_embeds = uncond_image_prompt_embeds.repeat(1, num_samples, 1)\\n        uncond_image_prompt_embeds = uncond_image_prompt_embeds.view(bs_embed * num_samples, seq_len, -1)\\n\\n        with torch.inference_mode():\\n            prompt_embeds_, negative_prompt_embeds_ = self.pipe.encode_prompt(\\n                prompt,\\n                device=self.device,\\n                num_images_per_prompt=num_samples,\\n                do_classifier_free_guidance=True,\\n                negative_prompt=negative_prompt,\\n            )\\n            prompt_embeds = torch.cat([prompt_embeds_, image_prompt_embeds], dim=1)\\n            negative_prompt_embeds = torch.cat([negative_prompt_embeds_, uncond_image_prompt_embeds], dim=1)\\n\\n        generator = get_generator(seed, self.device)\\n\\n        images = self.pipe(\\n            prompt_embeds=prompt_embeds,\\n            negative_prompt_embeds=negative_prompt_embeds,\\n            guidance_scale=guidance_scale,\\n            num_inference_steps=num_inference_steps,\\n            generator=generator,\\n            **kwargs,\\n        ).images\\n\\n        return images\\n\\n\\nclass IPAdapterFaceIDPlus:\\n    def __init__(self, sd_pipe, image_encoder_path, ip_ckpt, device, lora_rank=128, num_tokens=4, torch_dtype=torch.float16):\\n        self.device = device\\n        self.image_encoder_path = image_encoder_path\\n        self.ip_ckpt = ip_ckpt\\n        self.lora_rank = lora_rank\\n        self.num_tokens = num_tokens\\n        self.torch_dtype = torch_dtype\\n\\n        self.pipe = sd_pipe.to(self.device)\\n        self.set_ip_adapter()\\n\\n        # load image encoder\\n        self.image_encoder = CLIPVisionModelWithProjection.from_pretrained(self.image_encoder_path).to(\\n            self.device, dtype=self.torch_dtype\\n        )\\n        self.clip_image_processor = CLIPImageProcessor()\\n        # image proj model\\n        self.image_proj_model = self.init_proj()\\n\\n        self.load_ip_adapter()\\n\\n    def init_proj(self):\\n        image_proj_model = ProjPlusModel(\\n            cross_attention_dim=self.pipe.unet.config.cross_attention_dim,\\n            id_embeddings_dim=512,\\n            clip_embeddings_dim=self.image_encoder.config.hidden_size,\\n            num_tokens=self.num_tokens,\\n        ).to(self.device, dtype=self.torch_dtype)\\n        return image_proj_model\\n\\n    def set_ip_adapter(self):\\n        unet = self.pipe.unet\\n        attn_procs = {}\\n        for name in unet.attn_processors.keys():\\n            cross_attention_dim = None if name.endswith(\\\"attn1.processor\\\") else unet.config.cross_attention_dim\\n            if name.startswith(\\\"mid_block\\\"):\\n                hidden_size = unet.config.block_out_channels[-1]\\n            elif name.startswith(\\\"up_blocks\\\"):\\n                block_id = int(name[len(\\\"up_blocks.\\\")])\\n                hidden_size = list(reversed(unet.config.block_out_channels))[block_id]\\n            elif name.startswith(\\\"down_blocks\\\"):\\n                block_id = int(name[len(\\\"down_blocks.\\\")])\\n                hidden_size = unet.config.block_out_channels[block_id]\\n            if cross_attention_dim is None:\\n                attn_procs[name] = LoRAAttnProcessor(\\n                    hidden_size=hidden_size, cross_attention_dim=cross_attention_dim, rank=self.lora_rank,\\n                ).to(self.device, dtype=self.torch_dtype)\\n            else:\\n                attn_procs[name] = LoRAIPAttnProcessor(\\n                    hidden_size=hidden_size, cross_attention_dim=cross_attention_dim, scale=1.0, rank=self.lora_rank, num_tokens=self.num_tokens,\\n                ).to(self.device, dtype=self.torch_dtype)\\n        unet.set_attn_processor(attn_procs)\\n\\n    def load_ip_adapter(self):\\n        if os.path.splitext(self.ip_ckpt)[-1] == \\\".safetensors\\\":\\n            state_dict = {\\\"image_proj\\\": {}, \\\"ip_adapter\\\": {}}\\n            with safe_open(self.ip_ckpt, framework=\\\"pt\\\", device=\\\"cpu\\\") as f:\\n                for key in f.keys():\\n                    if key.startswith(\\\"image_proj.\\\"):\\n                        state_dict[\\\"image_proj\\\"][key.replace(\\\"image_proj.\\\", \\\"\\\")] = f.get_tensor(key)\\n                    elif key.startswith(\\\"ip_adapter.\\\"):\\n                        state_dict[\\\"ip_adapter\\\"][key.replace(\\\"ip_adapter.\\\", \\\"\\\")] = f.get_tensor(key)\\n        else:\\n            state_dict = torch.load(self.ip_ckpt, map_location=\\\"cpu\\\")\\n        self.image_proj_model.load_state_dict(state_dict[\\\"image_proj\\\"])\\n        ip_layers = torch.nn.ModuleList(self.pipe.unet.attn_processors.values())\\n        ip_layers.load_state_dict(state_dict[\\\"ip_adapter\\\"])\\n\\n    @torch.inference_mode()\\n    def get_image_embeds(self, faceid_embeds, face_image, s_scale, shortcut):\\n        if isinstance(face_image, Image.Image):\\n            pil_image = [face_image]\\n        clip_image = self.clip_image_processor(images=face_image, return_tensors=\\\"pt\\\").pixel_values\\n        clip_image = clip_image.to(self.device, dtype=self.torch_dtype)\\n        clip_image_embeds = self.image_encoder(clip_image, output_hidden_states=True).hidden_states[-2]\\n        uncond_clip_image_embeds = self.image_encoder(\\n            torch.zeros_like(clip_image), output_hidden_states=True\\n        ).hidden_states[-2]\\n        \\n        faceid_embeds = faceid_embeds.to(self.device, dtype=self.torch_dtype)\\n        image_prompt_embeds = self.image_proj_model(faceid_embeds, clip_image_embeds, shortcut=shortcut, scale=s_scale)\\n        uncond_image_prompt_embeds = self.image_proj_model(torch.zeros_like(faceid_embeds), uncond_clip_image_embeds, shortcut=shortcut, scale=s_scale)\\n        return image_prompt_embeds, uncond_image_prompt_embeds\\n\\n    def set_scale(self, scale):\\n        for attn_processor in self.pipe.unet.attn_processors.values():\\n            if isinstance(attn_processor, LoRAIPAttnProcessor):\\n                attn_processor.scale = scale\\n\\n    def generate(\\n        self,\\n        face_image=None,\\n        faceid_embeds=None,\\n        prompt=None,\\n        negative_prompt=None,\\n        scale=1.0,\\n        num_samples=4,\\n        seed=None,\\n        guidance_scale=7.5,\\n        num_inference_steps=30,\\n        s_scale=1.0,\\n        shortcut=False,\\n        **kwargs,\\n    ):\\n        self.set_scale(scale)\\n\\n       \\n        num_prompts = faceid_embeds.size(0)\\n\\n        if prompt is None:\\n            prompt = \\\"best quality, high quality\\\"\\n        if negative_prompt is None:\\n            negative_prompt = \\\"monochrome, lowres, bad anatomy, worst quality, low quality\\\"\\n\\n        if not isinstance(prompt, List):\\n            prompt = [prompt] * num_prompts\\n        if not isinstance(negative_prompt, List):\\n            negative_prompt = [negative_prompt] * num_prompts\\n\\n        image_prompt_embeds, uncond_image_prompt_embeds = self.get_image_embeds(faceid_embeds, face_image, s_scale, shortcut)\\n\\n        bs_embed, seq_len, _ = image_prompt_embeds.shape\\n        image_prompt_embeds = image_prompt_embeds.repeat(1, num_samples, 1)\\n        image_prompt_embeds = image_prompt_embeds.view(bs_embed * num_samples, seq_len, -1)\\n        uncond_image_prompt_embeds = uncond_image_prompt_embeds.repeat(1, num_samples, 1)\\n        uncond_image_prompt_embeds = uncond_image_prompt_embeds.view(bs_embed * num_samples, seq_len, -1)\\n\\n        with torch.inference_mode():\\n            prompt_embeds_, negative_prompt_embeds_ = self.pipe.encode_prompt(\\n                prompt,\\n                device=self.device,\\n                num_images_per_prompt=num_samples,\\n                do_classifier_free_guidance=True,\\n                negative_prompt=negative_prompt,\\n            )\\n            prompt_embeds = torch.cat([prompt_embeds_, image_prompt_embeds], dim=1)\\n            negative_prompt_embeds = torch.cat([negative_prompt_embeds_, uncond_image_prompt_embeds], dim=1)\\n\\n        generator = get_generator(seed, self.device)\\n\\n        images = self.pipe(\\n            prompt_embeds=prompt_embeds,\\n            negative_prompt_embeds=negative_prompt_embeds,\\n            guidance_scale=guidance_scale,\\n            num_inference_steps=num_inference_steps,\\n            generator=generator,\\n            **kwargs,\\n        ).images\\n\\n        return images\\n\\n\\nclass IPAdapterFaceIDXL(IPAdapterFaceID):\\n    \\\"\\\"\\\"SDXL\\\"\\\"\\\"\\n\\n    def generate(\\n        self,\\n        faceid_embeds=None,\\n        prompt=None,\\n        negative_prompt=None,\\n        scale=1.0,\\n        num_samples=4,\\n        seed=None,\\n        num_inference_steps=30,\\n        **kwargs,\\n    ):\\n        self.set_scale(scale)\\n\\n        num_prompts = faceid_embeds.size(0)\\n\\n        if prompt is None:\\n            prompt = \\\"best quality, high quality\\\"\\n        if negative_prompt is None:\\n            negative_prompt = \\\"monochrome, lowres, bad anatomy, worst quality, low quality\\\"\\n\\n        if not isinstance(prompt, List):\\n            prompt = [prompt] * num_prompts\\n        if not isinstance(negative_prompt, List):\\n            negative_prompt = [negative_prompt] * num_prompts\\n\\n        image_prompt_embeds, uncond_image_prompt_embeds = self.get_image_embeds(faceid_embeds)\\n\\n        bs_embed, seq_len, _ = image_prompt_embeds.shape\\n        image_prompt_embeds = image_prompt_embeds.repeat(1, num_samples, 1)\\n        image_prompt_embeds = image_prompt_embeds.view(bs_embed * num_samples, seq_len, -1)\\n        uncond_image_prompt_embeds = uncond_image_prompt_embeds.repeat(1, num_samples, 1)\\n        uncond_image_prompt_embeds = uncond_image_prompt_embeds.view(bs_embed * num_samples, seq_len, -1)\\n\\n        with torch.inference_mode():\\n            (\\n                prompt_embeds,\\n                negative_prompt_embeds,\\n                pooled_prompt_embeds,\\n                negative_pooled_prompt_embeds,\\n            ) = self.pipe.encode_prompt(\\n                prompt,\\n                num_images_per_prompt=num_samples,\\n                do_classifier_free_guidance=True,\\n                negative_prompt=negative_prompt,\\n            )\\n            prompt_embeds = torch.cat([prompt_embeds, image_prompt_embeds], dim=1)\\n            negative_prompt_embeds = torch.cat([negative_prompt_embeds, uncond_image_prompt_embeds], dim=1)\\n\\n        generator = get_generator(seed, self.device)\\n\\n        images = self.pipe(\\n            prompt_embeds=prompt_embeds,\\n            negative_prompt_embeds=negative_prompt_embeds,\\n            pooled_prompt_embeds=pooled_prompt_embeds,\\n            negative_pooled_prompt_embeds=negative_pooled_prompt_embeds,\\n            num_inference_steps=num_inference_steps,\\n            generator=generator,\\n            **kwargs,\\n        ).images\\n\\n        return images\\n\\n\\nclass IPAdapterFaceIDPlusXL(IPAdapterFaceIDPlus):\\n    \\\"\\\"\\\"SDXL\\\"\\\"\\\"\\n\\n    def generate(\\n        self,\\n        face_image=None,\\n        faceid_embeds=None,\\n        prompt=None,\\n        negative_prompt=None,\\n        scale=1.0,\\n        num_samples=4,\\n        seed=None,\\n        guidance_scale=7.5,\\n        num_inference_steps=30,\\n        s_scale=1.0,\\n        shortcut=True,\\n        **kwargs,\\n    ):\\n        self.set_scale(scale)\\n\\n        num_prompts = faceid_embeds.size(0)\\n\\n        if prompt is None:\\n            prompt = \\\"best quality, high quality\\\"\\n        if negative_prompt is None:\\n            negative_prompt = \\\"monochrome, lowres, bad anatomy, worst quality, low quality\\\"\\n\\n        if not isinstance(prompt, List):\\n            prompt = [prompt] * num_prompts\\n        if not isinstance(negative_prompt, List):\\n            negative_prompt = [negative_prompt] * num_prompts\\n\\n        image_prompt_embeds, uncond_image_prompt_embeds = self.get_image_embeds(faceid_embeds, face_image, s_scale, shortcut)\\n\\n        bs_embed, seq_len, _ = image_prompt_embeds.shape\\n        image_prompt_embeds = image_prompt_embeds.repeat(1, num_samples, 1)\\n        image_prompt_embeds = image_prompt_embeds.view(bs_embed * num_samples, seq_len, -1)\\n        uncond_image_prompt_embeds = uncond_image_prompt_embeds.repeat(1, num_samples, 1)\\n        uncond_image_prompt_embeds = uncond_image_prompt_embeds.view(bs_embed * num_samples, seq_len, -1)\\n\\n        with torch.inference_mode():\\n            (\\n                prompt_embeds,\\n                negative_prompt_embeds,\\n                pooled_prompt_embeds,\\n                negative_pooled_prompt_embeds,\\n            ) = self.pipe.encode_prompt(\\n                prompt,\\n                num_images_per_prompt=num_samples,\\n                do_classifier_free_guidance=True,\\n                negative_prompt=negative_prompt,\\n            )\\n            prompt_embeds = torch.cat([prompt_embeds, image_prompt_embeds], dim=1)\\n            negative_prompt_embeds = torch.cat([negative_prompt_embeds, uncond_image_prompt_embeds], dim=1)\\n\\n        generator = get_generator(seed, self.device)\\n\\n        images = self.pipe(\\n            prompt_embeds=prompt_embeds,\\n            negative_prompt_embeds=negative_prompt_embeds,\\n            pooled_prompt_embeds=pooled_prompt_embeds,\\n            negative_pooled_prompt_embeds=negative_pooled_prompt_embeds,\\n            num_inference_steps=num_inference_steps,\\n            generator=generator,\\n            guidance_scale=guidance_scale,\\n            **kwargs,\\n        ).images\\n\\n        return images\",\"difficulty\":\"hard\",\"domain\":\"Code Repository Understanding\",\"length\":\"short\",\"question\":\"The repository \\\"StoryMaker\\\" is a personalized solution that can generate story collections with character consistency. There are already many methods for generating photo sets with consistent characters.  Which method this repository uses to achieve this consistency?\",\"sub_domain\":\"Code repo QA\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/zai-org/LongBench-v2","history":"initial import","indexing_mode":"noindex","subproblems":[],"grids":[]}