# LongBench v2 / 66ec1e3f821e116aacb1ae7d

task_id: 51587ae2-81ae-5999-b2eb-8f3f122bd10d
task_key: train--66ec1e3f821e116aacb1ae7d
task_revision_id: 2

{"choice_A":"This article inserts a module into the pre-trained diffusion model, and then trains the parameters of these models to adapt this module to the task and the priori of the diffusion model.","choice_B":"TPB includes two MLP layers with Layer Normalization and LeakyReLU, ensuring that only the most task-specific attributes are retained","choice_C":"Task-specific priors containing guidance information for the task can adequately guide pre-trained diffusion models to handle low-level tasks while maintaining high-fidelity content consistency.","choice_D":"The spatial feature Fs extracted by SCB processing is calculated from SCB, Ft, Fp, F and has no relationship with TPB.","context":"Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks\n“Please help me enhance the lighting of this photo.”\n“Can you remove the rain in this photo?”\n“I want to enhance the face appearance of this image.”\n  “ I need to remove the snow in this photo.”\nInput\nOutput\nInput\nOutput 1\nOutput 2\nOutput 3\nOutput\nInput\nOutput\nInput\n“    clear haze    ”\n... \n... \n“remove snow and haze   ”    \n... \nFigure 1. Real-world applications of Diff-Plugin visualized across distinct single-type and one multi-type low-level vision tasks. Diff-\nPlugin allows users to selectively conduct interested low-level vision tasks via natural languages and can generate high-fidelity results.\nAbstract\nDiffusion models trained on large-scale datasets have\nachieved remarkable progress in image synthesis.\nHow-\never, due to the randomness in the diffusion process, they\noften struggle with handling diverse low-level tasks that\nrequire details preservation. To overcome this limitation,\nwe present a new Diff-Plugin framework to enable a sin-\ngle pre-trained diffusion model to generate high-fidelity re-\nsults across a variety of low-level tasks. Specifically, we\nfirst propose a lightweight Task-Plugin module with a dual\nbranch design to provide task-specific priors, guiding the\ndiffusion process in preserving image content. We then pro-\npose a Plugin-Selector that can automatically select dif-\nferent Task-Plugins based on the text instruction, allowing\nusers to edit images by indicating multiple low-level tasks\nwith natural language. We conduct extensive experiments\non 8 low-level vision tasks. The results demonstrate the\nsuperiority of Diff-Plugin over existing methods, particu-\nlarly in real-world scenarios. Our ablations further vali-\ndate that Diff-Plugin is stable, schedulable, and supports\nrobust training across different dataset sizes. Project page:\nhttps://yuhaoliu7456.github.io/Diff-Plugin\n†Joint corresponding authors. This project is in part supported by a\nGRF grant (Grant No.: 11205620) from the Research Grants Council of\nHong Kong.\n1. Introduction\nOver the past two years, diffusion models [9, 21, 22, 61]\nhave achieved unprecedented success in image generation\nand shown potential to become vision foundation models.\nRecently, many works [4, 25, 28, 31, 46, 91, 96] have\ndemonstrated that diffusion models trained on large-scale\ntext-to-image datasets can already understand various vi-\nsual attributes and provide versatile visual representations\nfor downstream tasks, e.g., image classification [31], seg-\nmentation [25, 96], translation [46, 91], and editing [4, 28].\nHowever, due to the inherent randomness in the dif-\nfusion process, existing diffusion models cannot maintain\nconsistent contents to the input image and thus fail in han-\ndling low-level vision tasks.\nTo this end, some meth-\nods [46, 63] propose to utilize input images as a prior via\nthe DDIM Inversion [61] strategy when editing images, but\nthey are unstable when the scenes are complex. Other meth-\nods [16, 52, 56, 71, 83] attempt to train new diffusion mod-\nels on task-specific datasets from scratch, limiting them to\nsolve only a single task.\nIn this work, we observe that an accurate text prompt\ndescribing the goal of the task can already instruct a pre-\ntrained diffusion model to address many low-level tasks, but\ntypically leads to obvious content distortion, as illustrated\nin Fig. 2. Our insight to this problem is that task-specific\npriors containing both guidance information of the task and\nspatial information of the input image can adequately guide\narXiv:2403.00644v4  [cs.CV]  28 May 2024\n\n\npre-trained diffusion models to handle low-level tasks while\nmaintaining high-fidelity content consistency. To harness\nthis potential, we propose Diff-Plugin, the first framework\nenabling a pre-trained diffusion model, such as stable dif-\nfusion [54], to accommodate a variety of low-level tasks\nwithout compromising its original generative capability.\nDiff-Plugin consists of two main components. First, it\nincludes a lightweight Task-Plugin module to help extract\ntask-specific priors. The Task-Plugin is bifurcated into the\nTask-Prompt Branch (TPB) and the Spatial Complement\nBranch (SCB). While TPB distills the task guidance prior,\norienting the diffusion model towards the specified vision\ntask and minimizing its reliance on complex textual descrip-\ntions, SCB leverages task-specific visual guidance from\nTPB to assist the spatial details capture and complement,\nenhancing the fidelity of the generated content. Second, to\nfacilitate the use of multiple different Task-Plugins, Diff-\nPlugin includes a Plugin-Selector to allow users to choose\ntheir desired Task-Plugins through text inputs (visual illus-\ntrations are depicted in Fig. 1). To train the Plugin-Selector,\nwe employ multi-task contrastive learning [49], using task-\nspecific visual guidance as pseudo-labels. This enables the\nPlugin-Selector to align different visual embeddings with\ntask-specific text inputs, thereby bolstering the robustness\nand user-friendliness of the Plugin-Selector.\nTo thoroughly evaluate our method, we conducted ex-\ntensive experiments on eight diverse low-level vision tasks.\nOur results affirm that Diff-Plugin is not only stable across\ndifferent tasks but also exhibits remarkable schedulability,\nfacilitating text-driven multi-task applications. Addition-\nally, Diff-Plugin showcases its scalability, adapting to vari-\nous tasks across datasets of varying sizes, from less than 500\nto over 50,000 samples, without affecting existing trained\nplugins. Finally, our results also show that the proposed\nframework outperforms existing diffusion-based methods\nboth visually and quantitatively, and achieves competitive\nperformances compared to regression-based methods.\nOur key contributions are summarized as follows:\n• We present Diff-Plugin, the first framework to enable a\npre-trained diffusion model to perform various low-level\ntasks while maintaining the original generative abilities.\n• We propose a Task-Plugin, a lightweight dual-branch\nmodule designed for injecting task-specific priors into the\ndiffusion process, to enhance the fidelity of the results.\n• We propose a Plugin-Selector to select the appropriate\nTask-Plugin based on the text provided by the user. This\nextends to a new application that can allow users to edit\nimages via text instructions for low-level vision tasks.\n• We conduct extensive experiments on eight tasks, demon-\nstrating the competitive performances of Diff-Plugin over\nexisting diffusion and regression-based methods.\n“A photo of a girl wearing a cotton hat, \nclosing her eyes, with falling snow”\n“A blurry photo of a dog running in garden”\n“A car is moving on road on a rainy day”\n“A bowl on the table with a circle of \nsparkling highlights around the rim”\ncloudy\n(1)\n(3)\n(2)\n(4)\nFigure 2. Stable Diffusion (SD) [54] results on four low-level\nvision tasks: desnowing, deblurring, deraining, and highlight re-\nmoval. Each sub-figure illustrates a two-step process: First, we\ngenerate the left image using SD with a full-text description,\nwhere task-critical attributes are highlighted in red. Then, we re-\nmove unwanted attributes (indicated with strikethrough), option-\nally add new attributes (denoted with orange word), and employ\nthe img2img function in SD, using the left image as a condition\nto generate the edited image on the right. We observe that while\nSD can grasp rich attributes of various low-level tasks and create\ncontent consistent with descriptions, its inherent randomness often\nleads to content change in further editing. For instance, in sub-fig\n(1), besides addressing the primary task-related degradation (e.g.,\nsnow), SD also alters unrelated content (e.g., face profile).\n2. Related Works\nDiffusion models [60, 62] have been applied to image\nsynthesis [9, 21, 22, 61] and achieved remarkable suc-\ncess. With extensive text-image data [59] and large-scale\nlanguage models [49, 50], diffusion-based text-guided im-\nage synthesis [2, 42, 51, 54, 57] has become even more\ncompelling. Leveraging the text-guided synthesis diffusion\nmodel, several approaches harness the generative prowess\nfor text-driven editing. Zero-shot approaches [19, 46, 63]\nrely on a correct initial noise [61] and manipulate the at-\ntention map to edit specified content at precise locations.\nTuning-based strategies strive to balance between image\nfidelity and generated diversity through optimized DDIM\ninversion [65], attention tuning [29], text-image coupling\n[28, 55, 93] and prompt tuning [10, 14, 39]. Conversely,\nInstructP2P [4, 89] generates paired data through latent dif-\nfusion [54] and prompt-to-prompt [19] for training and edit-\ning. However, the randomness in the diffusion process and\nthe absence of task-specific priors render them infeasible\nfor low-level vision tasks that require details preservation.\nConditional generative models use various external inputs\nto ensure output consistency with the conditions. Training-\nfree methods [8, 76] can generate new contents at specified\npositions by manipulating attention layers, yet with limited\ncondition types. Fine-tuning-based approaches inject addi-\ntional guidance to the pre-trained diffusion models by train-\ning a new diffusion branch [40, 90, 94] or the whole model\n\n\n[1]. Despite the global structural consistency, these methods\ncannot ensure high-fidelity between output and input image\ndetails due to the randomness and generative nature.\nDiffusion-based low-level methods can be grouped into\nzero-shot and training-based. The former can borrow gener-\native priors from pre-trained denoising diffusion-based gen-\nerative models [22] to solve linear [27, 70] and/or non-linear\n[7, 12] image restoration tasks, but often produce poor re-\nsults on real-world data. The latter usually train or fine-tune\nan individual model for different tasks via task-dependent\ndesigns, such as super-resolution [58, 74], JPEG compres-\nsion [56], deblurring [52, 73], face restoration [71, 95], low-\nlight enhancement [24, 83, 92], and shadow removal [16].\nConcurrent works, StableSR [66] and DiffBIR [34], use a\nlearnable conditional diffusion branch with degraded or re-\nstored images to train diffusion models specifically for blind\nface restoration. In contrast, our framework enables one\npre-trained diffusion model to handle a variety of low-level\ntasks by equipping it with lightweight task-specific plugins.\nMulti-task models can learn complementary information\nacross different tasks, e.g., object detection and segmenta-\ntion [18], rain detection and removal [80], adverse weather\nrestoration [45, 82, 98] and blind image restoration [33, 47].\nHowever, these methods can only handle the pre-defined\ntasks after training. Instead, our Diff-Plugin is flexible and\ncan integrate new tasks through task-specific plugins, as our\nTask-Plugins are trained individually. Hence, when adding\nnew low-level tasks to Diff-Plugin, we only need to add the\npre-trained Task-Plugins to the framework, without the need\nto retrain the existing ones.\n3. Methodologies\nIn this section, we first review the diffusion model formula-\ntions (Sec. 3.1). Then, we introduce our Diff-Plugin frame-\nwork (Sec. 3.2), which developed from our newly proposed\nTask-Plugin (Sec. 3.3) and Plugin-Selector (Sec. 3.4).\n3.1. Preliminaries\nThe diffusion model consists of a forward process and a\nreverse process.\nIn the forward process, given a clean\ninput image x0, the diffusion model progressively adds\nGaussian noise to it to get noisy image xt at time-step\nt ∈{0, 1, ..., T}, as xt = √¯\nαtx0 + √1 −¯\nαtϵt, where ¯\nαt\nis the pre-defined scheduling variable and ϵt ∼N(0, I)\nis the added noise. In the reverse process, the diffusion\nmodel performs iteratively remove noise from a standard\nGaussian noise xT , and finally estimating a clean image\nx0. This is typically employed to train a noise prediction\nnetwork ϵθ, with supervision informed by the noise ϵt, as\nL = Ex0,t,ϵ∼N (0,1)\nh\n∥ϵ −ϵθ (xt, t)∥2\n2\ni\n.\n“… remove blur …”\n“… enhance lighting …”\n“… remove blur and  \nenhance lighting …”\nPre-trained\nDiffusion Model\nI\nI\nI\nPlugin-Selector\nPriors\nPriors\nPriors\nPriors\nFigure 3.\nSchematic illustration of the Diff-Plugin framework.\nDiff-Plugin identifies appropriate Task-Plugin P based on the user\nprompts, extracts task-specific priors, and then injects them into\nthe pre-trained diffusion model to generate the user-desired results.\n3.2. Diff-Plugin\nOur key observation is the inherent zero-shot capability of\npre-trained diffusion models in performing low-level vision\ntasks, enabling them to generate diverse visual content with-\nout explicit task-specific training. However, this capability\nfaces limitations in more nuanced task-specific editing. For\nexample, in the desnowing task, while the model should ide-\nally only remove snow and leave other contents unchanged,\nas shown in Fig. 2, the inherent randomness of the diffusion\nprocess often leads to unintended alterations in the scene\nbeyond just snow removal. This inconsistency arises from\nthe model’s lack of task-specific priors, which are crucial\nfor precise detail preservation in low-level vision tasks.\nInspired by modular extensions in NLP [75, 77] and\nGPT-4 [43], which utilize plug-and-play tools to enhance\nthe capabilities of large language models for downstream\ntasks without compromising their core competencies, we\nintroduce a novel framework, Diff-Plugin, based on a simi-\nlar idea. This framework integrates several lightweight plu-\ngin modules, termed Task-Plugin, into the pre-trained dif-\nfusion models for various low-level tasks.\nTask-Plugins\nare crafted to provide essential task-specific priors, guiding\nthe models to produce high-fidelity and task-consistent con-\ntent. In addition, while diffusion models can generate con-\ntent based on text instructions for targeted scenarios, they\nlack the ability to schedule Task-Plugins for different low-\nlevel tasks. Even existing conditional generation methods\n[48, 90] can only specify different generation tasks through\ninput conditional images. Thus, to facilitate smooth text-\ndriven task scheduling and enable the switching between\ndifferent Task-Plugins for complex workflows, Diff-Plugin\nincludes a Plugin-Selector to allow users to choose and\nschedule appropriate Task-Plugins with textual commands.\nFig. 3 depicts the Diff-Plugin framework. Given an im-\nage, users specify the task through a text prompt, either\nsingular or multiple, and the Plugin-Selector identifies the\nappropriate Task-Plugin for it. The Task-Plugin then pro-\ncesses the image to extract the task-specific priors, guiding\n\n\nthe pre-trained diffusion model to produce user-desired out-\ncomes. For more intricate tasks beyond the scope of a single\nplugin, Diff-Plugin breaks them down into sub-tasks with a\npredefined mapping table. Each sub-task is tackled by a\ndesignated Task-Plugin, showcasing the framework’s capa-\nbility to handle diverse and complex user requirements.\n3.3. Task-Plugin\nAs illustrated in Fig. 4, our Task-Plugin module is com-\nposed of two branches: a Task-Prompt Branch (TPB) and a\nSpatial Complement Branch (SCB). The TPB is crucial for\nproviding task-specific guidance to the pre-trained diffusion\nmodel, akin to using text prompts in text-conditional image\nsynthesis [54]. We employ visual prompts, extracted via the\npre-trained CLIP vision encoder [49], to direct the model’s\nfocus towards task-relevant patterns (e.g., rain streaks for\nderaining and snow flakes for desnowing). Specifically, for\nan input image I, the encoder EncI(·) first extracts general\nvisual features, which are then distilled by the TPB to yield\ndiscriminative visual guidance priors Fp:\n  \\ mathbf {F}^{p} = \\textit {TPB}(\\textit {Enc}_{I}(\\mathbf {I})) \\text {,} \\label {eq:tpb} \n(1)\nwhere TPB, comprising three MLP layers with Layer Nor-\nmalization and LeakyReLU activations (except for the final\nlayer), ensures the retention of only the most task-specific\nattributes. This approach aligns Fp with the textual features\nthe diffusion model typically uses in its text-driven gen-\neration process, thus facilitating better task alignment for\nPlugin-Selector. Furthermore, using visual prompts simpli-\nfies the user’s role by eliminating the need for complex text\nprompt engineering, which is often challenging for specific\nvision tasks and sensitive to minor textual variations [78].\nHowever, the task-specific visual guidance prior Fp,\nwhile crucial for prompting global semantic attributes, is\nnot sufficient for preserving fine-grained details.\nIn this\ncontext, DDIM Inversion plays a pivotal role by providing\ninitial noise that contains information about the image con-\ntent. Without this step, the inference would rely on random\nnoise devoid of image content, resulting in less controllable\nresults in the diffusion process. However, the inversion pro-\ncess is unstable and time-consuming. To alleviate this, we\nintroduce the SCB to extract and enhance spatial details\npreservation effectively. We utilizes the pre-trained VAE\nencoder [11] EncV (·), to capture full content of input image\nI, denoted as F. This comprehensive image detail, when\ncombined with the semantic guidance from Fp, is then pro-\ncessed by our SCB to distill the spatial feature Fs:\n  \\ mathbf  {F }^{ s } = \\texti t {S CB} (\\mathbf {F}\\text {,} \\ \\mathbf {F}^{t}\\text {,} \\ \\mathbf {F}^{p})=\\textit {Att}(\\textit {Res}(\\mathbf {F}\\text {,} \\ \\mathbf {F}^{t})\\text {,} \\ \\mathbf {F}^{t}\\text {,} \\ \\mathbf {F}^{p}) \\text {,} \\label {eq:SCB} \n(2)\nwhere Ft is time embedding used to denote the varied time\nstep in diffusion process. The Res and Att blocks repre-\nsent the standard ResNet and Cross-Attention transformer\nEncV\nI\nTask-Prompt \n    Branch\nt\nMLP\nFp\nFs\n Res. \nBlock\n  Att.\nBlock\n      Spatial Complement Branch\nI\nEncI\nTask-Plugin\nFigure 4. Schematic illustration of task-specific priors extraction\nvia the proposed lightweight Task-Plugin. Task-Plugin processes\nthree inputs: time step t, visual prompt from EncI(·), and image\ncontent from EncV (·). It distills visual guidance Fp via a task-\nprompt branch and extracts spatial features Fs through a spatial\ncomplement branch, jointly for task-specific priors.\nblocks, from the diffusion model [54]. The output from Res\nis utilized as the Query features and Fp acts as both Key\nand Value features in the cross-attention layer.\nWe then introduce the task-specific visual guidance prior\nFp into the cross-attention layers of the diffusion model,\nwhere it serves to direct the model’s generation process to-\nward the specific requirements of the low-level vision task.\nFollowing this, we directly incorporate the distilled spatial\nprior Fs into the final stage of the decoder as a residual.\nThis placement is based on our experimental observations\nin Table 4, which indicated that the fidelity of spatial de-\ntails in the stable diffusion [54] tends to decrease from the\nshallow layers to the deeper ones. By adding Fs at this spe-\ncific stage, we effectively counteract this tendency, thereby\nenhancing the preservation of fine-grained spatial details.\nTo train the Task-Plugin modules, we adopt the denois-\ning loss as defined in [54], introducing the task-specific pri-\nors into the diffusion denoising training process:\n  \\mathcal {L}=\\m athbb\n \n{E } _{ \\bol ds ymb ol {z\n}\n_\n0\\text {,} t \\text {,} \\mathbf {F}^{p} \\text {,} \\mathbf {F}^{s} \\text {,} \\epsilon \\sim \\mathcal {N}(0\\text {,}1)}\\left [\\| \\epsilon -\\epsilon _\\theta \\left (\\boldsymbol {z}_t\\text {,} \\ t \\text {,} \\ \\mathbf {F}^{p} \\text {,} \\ \\mathbf {F}^{s}\\right ) \\|_2^2\\right ] \\text {,} \\label {eq:denois_loss} (3)\nwhere zt = √¯\nαtz0 + √1 −¯\nαtϵt represents the noised ver-\nsion of the latent-space image at time t, and z0, the latent-\nspace representation of the ground truth image ˆ\nI, is obtained\nas z0 = EncV (ˆ\nI). This loss function ensures that the Task-\nPlugin is effectively trained to incorporate the task-specific\npriors in guiding the diffusion process.\n3.4. Plugin-Selector\nWe propose the Plugin-Selector, enabling users to select the\ndesired Task-Plugin using text input. For an input image I\nand a text prompt T, we define the set of Task-Plugins as\nP = {P1, P2, · · · , Pm}, with each Pi corresponding to a\nspecific vision task, transforming I into task-specific priors\n(Fp\ni , Fs\ni). Then, visual guidance Fp\ni of each Task-Plugin\nis then cast to a new textual-visual aligned multi-modality\nlatent space via a shared visual projection head VP(·) and\ndenoted as V = {v1, v2, · · · , vm}. Concurrently, T is en-\n\n\ncoded into a text embedding by EncT (·) [49] and then pro-\njected to q using a textual project head TP(·), aligning the\ntextual and visual embedding. The process is formulated as:\n  \\ bolds\ny mb\no l  {v}_i = \\textit {VP}(\\mathbf {F}_{i}^{p})\\text {;} \\quad \\boldsymbol {q} = \\textit {TP}(\\textit {Enc}_{T}(\\mathbf {T})) \\text {.} \n(4)\nWe then compare the textual embedding q with each vi-\nsual embedding vi ∈V using cosine similarity function\nsuch that si = sim(vi, q), yielding a set of similarity scores\nS = {s1, s2, · · · , sm}. We select the Task-Plugin Pselected\nthat meet a specified similarity threshold, θ:\n  \\mathca l  {P } _{ \\ te xt  {selected}} = \\{\\mathcal {P}_i \\mid \\boldsymbol {s}_i \\geq \\theta \\text {,} \\ \\mathcal {P}_i \\in \\mathcal {P}\\}. \n(5)\nWe adopt the Fp\ni as the pseudo label and pair it with\ntask-specific text to construct training data. We employ con-\ntrastive loss [5, 49] to optimize the vision and text projection\nheads, enhancing their capability to handle multi-task sce-\nnarios. This involves minimizing the distance between the\nanchor image and positive texts while increasing the dis-\ntance from negative texts. For each image I, a positive text\nrelevant to its task (e.g., “I want to remove rain” for derain-\ning task) and N negative texts from other tasks (e.g., “en-\nhance the face” for face restoration) are sampled. The loss\nfunction for a positive pair of example (i, j) is as follows:\n  \\e l l  _{\ni, \nj\n}=-\n\\\nlog  \\\nf\nra\nc\n {\\e\nxp \\left (\\o per ator name  {s im}\n\\left (\\boldsymbol {v}_{i}\\text {,} \\ \\boldsymbol {q}_{j}\\right ) / \\tau \\right )}{\\sum _{k=1}^{N+1} \\mathbbm {1}_{[k_{c} \\neq i_{c}]} \\exp \\left (\\operatorname {sim}\\left (\\boldsymbol {v}_{i}\\text {,} \\ \\boldsymbol {q}_k\\right ) / \\tau \\right )} \\text {,} \\label {eq:scheduler} \n(6)\nwhere c represents the task type for each sample and\n1[kc̸=ic] ∈{0,1} is an indicator function evaluating to 1\niff kc ̸= ic. τ denotes a temperature parameter.\n4. Experiments\nIn this section, we first introduce our experimental setup,\nincluding datasets, implementation, and metrics. We then\ncompare Diff-Plugin with current diffusion- and regression-\nbased methods in Sec. 4.1, and conduct component analysis\nof Diff-Plugin via ablation studies in Sec. 4.2.\nDatasets.\nTo train the Task-Plugins, we utilize specific\ndatasets for each low-level task, desnowing: Snow100K\n[36], dehazing: Reside [32], deblurring: Gopro [41], de-\nraining: merged train [86], face restoration: FFHQ [26],\nlow-light enhancement: LOL [72], demoireing: LCDMoire\n[85], and highlight removal:\nSHIQ [13].\nFor testing,\nwe evaluate on real-world benchmark datasets, desnow-\ning: realistic test [36], dehazing: RTTS [32], deblurring:\nRealBlur-J [53], deraining: real test [68], face restoration:\nLFW [23, 69], low-light enhancement: merged low-light\n[17, 30, 38, 64, 67, 72], demoireing: LCDMoire [85], and\nhighlight removal: SHIQ [13]. To train the Plugin-Selector,\nwe employ GPT [44] to generate text prompts for each task.\nImplementation. During training and testing, we resize\nthe image to 512×512 for a fair comparison. We employ\nthe AdamW optimizer [37] with its default parameters (e.g.,\nbetas, weight decay). The training of our Task-Plugins was\nconducted using a constant learning rate of 1e−5 and a batch\nsize of 64 on four A100 GPUs, each with 80G of memory.\nTo train the Plugin-Selector, we randomly sample 5,000 im-\nages from each task and augment text diversity by randomly\ncombining text inputs from various tasks. We set the batch\nsize to 8 and adopt the same learning rate for Task-Plugins.\nFor negative texts, we set N = 7 by default. During infer-\nence, we set the specified similarity threshold θ = 0.\nMetrics. We follow [54] to employ widely adopted non-\nreference perceptual metrics, FID [20] and KID [3], to eval-\nuate our Diff-Plugin on real data, as GT is not always avail-\nable.\nAs for the Plugin-Selector, we follow multi-label\nobject classification [6] to report the mean average preci-\nsion (mAP), the average per-class precision (CP), F1 (CF1),\nand the average overall precision (OP), recall (OR), and F1\n(OF1). For each class (i.e., task type), the labels are pre-\ndicted as positive if their confidence score is greater than\nθ. We further propose a stringent zero-tolerance evaluation\nmetric (ZTA) that rigorously assesses sentence-level classi-\nfication results from a user-first perspective, making binary\nclassification to ensure utmost accuracy:\n  \\ t e\nx\nt\n \n{ZT\nA}\n = \n\\fra c { 1 }\n{\nQ\n}\n \\s\num _ {i= 1 }\n^{\nQ} \\left ( \\left ( \\min _{j \\in Y_i} S_{ij} > \\theta \\right ) \\land \\left ( \\max _{k \\in H_i} S_{ik} \\leq \\theta \\right ) \\right ) \\text {,} \\label {eq:zta} \n(7)\nwhere Q is the total number of test samples, Si is the set\nof predicted similarity scores for sample i, Yi is the set of\nindices for positive classes (i.e., user interested tasks), Hi is\nthe set of indices for negative classes (i.e., irrelevant tasks).\n4.1. Comparison with State-of-the-Art Methods\nWe compare the proposed Diff-Plugin with the state-of-the-\nart methods from different low-level vision tasks, includ-\ning regression-based specialized models: DDMSNet [88],\nPMNet [81], Restormer [87], NeRCO [79], VQFR [15],\nUHDM [84], SHIQ [13], multi-task models: AirNet [33],\nWGWS-Net [98] and PromptIR [47], and diffusion-based\nmodels: SD [54], PNP [63], P2P [46], InstructP2P [4], Null-\nText [39] and ControlNet [90]. We conduct the experiment\non real-world datasets to compare the generalization ability.\nQualitative Results. Fig. 5 demonstrates the superior per-\nformances of our Diff-Plugin on eight low-level vision tasks\nwith challenging natural images. First, using SD’s img2img\n[54] function does not ensure content accuracy. It often\nleads to major scene changes (column 8). InstructP2P [4],\nwhich lacks task-specific priors, also falls short, producing\npoorer results in tasks like dehazing and low-light enhance-\nment (column 7). The lack of task-specific priors also leads\nP2P [46] and Null-Text [39] into generating inconsistent\ncontents (columns 5 and 6), despite using initial noise from\nDDIM Inversion [61]. ControlNet [90] handles some tasks\n\n\nDesnowing\nDehazing\nDeblurring\nDeraining\nFace Restoration\nLow-light En.\nDemoireing\nHighlight Re.\n(1) Input\n(2) Ours\n(3) PromptIR [47]\n(4) ControlNet [90]\n(5) Null-Text [39]\n(6) PNP [63]\n(7) InstructP2P [4]\n(8) SD [54]\nFigure 5. Qualitative Comparison. Our Diff-Plugin notably surpasses regression-based method (3) and diffusion-based methods (4)-(8)\nin performance. Magnified regions of several tasks are provided for clarity. Refer to Supplemental for further comparisons.\nwell (column 4) by providing condition information via a\ndiffusion branch, but its strong color distortion reduces its\neffectiveness in these tasks. The latest multi-task method,\nPromptIR [47] (column 3), is limited by model scale and\ncan only handle a few tasks. In contrast, our method uses a\nlightweight task-specific plugin for each task, offering flex-\nibility and stable performance across all tasks (column 2).\nQuantitative Results.\nWe also provide the quantitative\ncomparison in Table 1.\nCompared with diffusion-based\nmethods, our Diff-Plugin achieves SOTA results overall.\nWhile PNP [63] and InstructP2P [4] are capable of pro-\nducing high-quality images with low FID & KID, they of-\nten produce significant content alterations (refer to Fig. 5).\nCompared with regression-based multi-task methods, our\napproach delivers competitive performances in most tasks,\nthough it is slightly ineffective in sparse degradation tasks\nlike demoireing and highlight removal.\nWhile special-\nized models may outperform ours in their respective ar-\neas, their task-dependent designs limit their applicability to\nother tasks. Note that the primary goal of this paper is not to\nachieve top performances in all tasks, but to lay groundwork\nfor future advancements. In addition, Diff-Plugin, enables\ntext-driven low-level task processing, a capability absent in\nregression-based models.\nUser Study. We conduct a user study with 46 participants to\nassess various methods through subjective evaluation. Each\nparticipant reviewed 5 image sets from the test set, each\ncomprising an input image and 10 predicted images, for a\ntotal of 8 tasks. The images were ranked based on content\nconsistency, degradation removal (e.g., rain, snow, high-\nlight), and overall quality. Analyzing 1,840 rankings (46\nparticipants × 40 sets), we compute the Average Ranking\n(AR) of each method. Table 2 shows the results. It is obvi-\nous to see a preference for our approach among the users.\n\n\nDesnowing\nDehazing\nDeblurring\nDeraining\nLow-light Enhanc.\nFace Restoration\nDemoireing\nHighlight Removal\nRealistic [36]\nReside [32]\nRealBlur-J [53]\nreal test [68]\nmerged low.\nLFW [69]\nLCDMoire [85]\nSHIQ [13]\nFID ↓\nKID ↓\nFID ↓\nKID ↓\nFID ↓\nKID ↓\nFID ↓\nKID ↓\nFID ↓\nKID ↓\nFID ↓\nKID ↓\nFID ↓\nKID ↓\nFID ↓\nKID ↓\nRegression-based specialized models\nAll\n33.92\n5.39\n36.40\n15.66\n55.64\n15.70\n52.78\n16.28\n48.47\n10.96\n19.28\n6.72\n29.59\n1.45\n33.74\n18.79\nRegression-based multi-task models\nAirNet* [33]\n35.02\n5.52\n39.53\n17.86\n59.38\n20.95\n52.04\n16.20\n59.92\n19.74\n31.03\n13.35\n33.05\n4.27\n10.13\n5.89\nWGWS-Net* [98]\n34.84\n5.71\n36.25\n15.79\n56.80\n16.83\n53.64\n16.55\n53.67\n12.99\n29.89\n12.08\n29.86\n2.28\n8.28\n3.05\nPromptIR* [47]\n34.66\n5.35\n40.88\n17.80\n55.37\n16.42\n53.78\n16.88\n53.42\n13.16\n30.52\n12.80\n29.01\n1.56\n9.01\n5.07\nDiffusion-based models\nSD [54]\n35.24\n7.88\n48.89\n24.47\n59.21\n18.96\n51.78\n17.69\n53.09\n15.38\n30.90\n9.63\n58.20\n17.34\n36.54\n12.06\nPNP [75]\n35.01\n6.52\n42.82\n16.98\n63.16\n23.58\n52.89\n21.02\n54.19\n14.43\n34.08\n13.45\n36.37\n6.18\n33.09\n14.94\nP2P [46]\n34.48\n6.03\n42.17\n17.33\n63.43\n25.15\n44.49\n13.94\n52.06\n13.26\n54.67\n24.66\n36.37\n9.35\n26.96\n13.11\nInstructP2P [4]\n42.01\n8.54\n33.48\n12.76\n57.38\n19.37\n54.12\n17.87\n55.65\n15.25\n24.66\n9.73\n34.29\n4.73\n16.80\n6.81\nNull-Text [39]\n60.49\n16.38\n39.94\n14.88\n60.38\n20.37\n51.49\n15.43\n52.86\n12.79\n33.06\n12.82\n33.72\n4.91\n14.65\n6.52\nControlNet* [90]\n34.36\n5.70\n37.02\n15.45\n52.30\n17.19\n52.55\n15.22\n51.56\n15.51\n21.59\n7.84\n41.97\n8.80\n15.75\n8.17\nDiff-Plugin (ours)\n34.30\n5.20\n34.68\n14.38\n51.81\n14.63\n50.55\n13.84\n48.98\n11.73\n20.07\n6.91\n29.77\n1.75\n12.58\n6.37\nTable 1. Quantitative comparisons to SOTAs (both regression-based and diffusion-based methods) on eight low-level vision tasks that need\nhigh content-preservation. We summarise all the regression-based specialized models in one line, denoted as “All”. They are: DDMSNet\n[88] (desnowing), PMNet [81] (dehazing), Restormer [87] (deblurring and deraining), NeRCO [79] (low-light enhancement), VQFR [15]\n(face restoration), UHDM [84] (demoireing), SHIQ [13] (highlight removal). KID values are scaled by a factor of 100 for readability. *\nmeans that this method is re-trained on eight tasks by us. The best and second-best results are highlighted.\nMethods AirNet [33] WGWS-Net [98] PromptIR [47] SD [54] PNP [63] P2P [46] InstructP2P [4] Null-Text [39] ControlNet [90] Ours\nAR ↓\n5.26\n2.75\n3.04\n9.66\n6.32\n7.39\n7.14\n7.94\n4.33\n1.17\nTable 2. Average Ranking (AR) of different methods in the User Study. The lower the value, the better the human subjective evaluation.\nInput\n➀Inversion+Edit.\n➁TPB\n➂TPB+Inversion\n➃SCB\n➄TPB+SCB (Rec.)\nOurs\nFigure 6. Visual comparison of various Task-Plugin design variants. Row 1 and Row 2 showcase desnowing and dehazing, respectively.\n4.2. Ablation Study\nTask-Plugin. We first evaluate the efficacy of Task-Plugins\nby exploring various ablated designs and comparing their\nperformances on desnowing and dehazing. Unless speci-\nfied otherwise, random noise is used during inference. We\nhave five ablated models. ➀Inversion + Editing: DDIM\nInversion with a task-specific description (e.g., “a photo of\na snowy day”) inverts the input image into an initial noise,\nretaining content. This is followed by editing using a tar-\nget description (e.g., “a photo of a sunny day”). ➁TPB:\nThe SCB is removed, focusing solely on TPB training. ➂\nTPB + Inversion: Only TPB is trained, but DDIM Inver-\nsion is used for initial noise during inference.\n➃SCB:\nThe TPB is removed to train the SCB exclusively. ➄TPB\n+ SCB (Reconstruction): Training begins with SCB us-\ning self-reconstruction denoising loss, and then proceeds to\nTPB training with the fixed SCB. Performance results and\ncomparison are presented in Fig. 6 and Table 3.\nWe have the following observations. ➀Inversion + Edit-\ning captures the global structure of the input image but\nloses detailed content. ➁TPB provides task-specific visual\nguidance but lacks spatial content constraints due to its fo-\ncus on advanced features only. ➂TPB, using inverted ini-\ntial noise, excels in structured scenes (e.g., large buildings)\nbut tends to deepen colors and create random content for\nsmaller objects. ➃SCB maintains content details, but with-\nout task-specific visual guidance, it struggles to effectively\nremove degradations (e.g., snow or haze). ➄TPB, when\ncombined with reconstruction-based SCB, preserves image\ncontent through reconstruction while relying solely on TPB\nto address degradation. However, as SCB reintroduces all\nimage features in each diffusion iteration, including original\ndegradations (e.g., haze in row-2 of Fig. 6), it inadvertently\ncompromises the desired outcomes. Finally, incorporating\nthe task-specific priors from both TPB and SCB in our Task-\nPlugin enables high-fidelity low-level task processing.\nWe also confirm the placement of SPB within the pre-\ntrained SD model on desnowing task and show the results\nin Table 4. Obviously, we can observe that for both the en-\ncoder and decoder of the pre-trained SD [54], the fidelity\n\n\nMethods \\ FID ↓\nDesnowing\nDehazing\n➀\nInversion + Editing\n48.54\n35.05\n➁\nTPB\n36.02\n37.73\n➂\nTPB + Inversion\n34.87\n33.05\n➃\nSCB\n34.71\n36.16\n➄\nTPB + SCB (Reconstruction)\n34.50\n35.94\nTPB + SCB (Ours)\n34.30\n34.68\nTable 3. Ablation studies of variant Task-Plugin designs on two\ntasks: desnowing, dehazing. Note that although some variants\nhave much lower FID scores, they tend to generate random content\n(refer to ➀-➂of Fig. 6). In contrast, our final model guarantees\nboth content fidelity and robust metric performances.\nMetrics\nEncoder\nDecoder\nE-1\nE-2\nE-3\nE-4\nD-4\nD-3\nD-2\nD-1\nFID ↓\n34.33 34.46 36.58 37.41 37.71 34.59 34.20 34.30\nKID ↓\n5.23\n5.52\n7.18\n7.84\n7.57\n5.55\n5.20\n5.20\nParam.(MB) 14.88 48.77\n182.31\n48.77 14.88\nTable 4. Ablation studies on the placement of SCB within the\npre-trained SD’s Encoder/Decoder stages on desnowing. ‘E/D-i’\nrepresents the i-th stage, with higher numbers indicating deeper\nlayers. We modify the feature dimension in SCB to suit various\nstages of the pre-trained SD model, resulting in varied parameters.\ndiminishes and performance progressively decreases from\nthe shallower to the deeper stages (e.g., stages 1 to 4). Thus,\nwe inject the spatial features into the final stage of the de-\ncoder, balancing performance and parameters. Notably, the\nparameters of Task-Plugin module is only 1.67% of the SD.\nPlugin-Selector.\nAs shown in Table 5, we first evalu-\nate the accuracy of Plugin-Selector in both single-task and\nmulti-task scenarios (row-1 and -2), and observe consis-\ntently high accuracy. In addition, in a significantly exten-\nsive test with 120,000 samples (denoted as Multi-task*), it\nachieves an mAP accuracy of 0.936, demonstrating its ef-\nfectiveness. Further, in a robustness test (denoted as Single\n+ Non.) combining task-specific and task-irrelevant texts, it\nstill achieves a notable zero-toleration accuracy of 0.779.\nWe also conduct an ablation study on the Plugin-Selector\nto evaluate the significance of each component, with results\ndetailed in Table 6. ➀We remove the visual and textual pro-\njection heads separately. ➁We assess the impact of vary-\ning the number of negative samples for contrastive training.\nThe results first reveal that both visual and textual projec-\ntion heads are crucial. Omitting the visual head results in\ntraining collapse and NaN output, while removing the tex-\ntual head lowers the ZTA metric by 15.4%. It also shows\nthat increasing the number of negative samples (e.g., from\nN = 1 to 15) consistently enhances selection accuracy.\nDiverse Applications. Fig. 7 demonstrates the versatility\nof Diff-Plugin. Row-1 exemplifies complex, low-level task\nexecution via sub-task integration (e.g., old photo restora-\nThe default batch size is 8, implying 7 neg. samples and 1 pos. sample.\nTasks\nZTA ↑\nCP ↑\nOP ↑\nOR ↑\nCF1 ↑\nOF1 ↑\nmAP ↑\nSingle-task\n0.998\n-\n0.998\n-\n-\n0.998\n0.998\nMulti-task\n0.979\n0.988\n0.988\n0.927\n0.956\n0.956\n0.933\nMulti-task*\n0.969\n0.983\n0.983\n0.936\n0.960\n0.959\n0.936\nSingle + Non.\n0.779\n0.814\n0.808\n0.941\n0.872\n0.870\n0.775\nTable 5. Quantitative evaluation of the proposed Plugin-Selector.\nAsterisks (*) denotes more sample combinations. A dash (-) indi-\ncates metric not applicable. ‘Single + Non’ refers to random com-\nbinations of single-task text inputs with non-existing (i.e., plugin-\nirrelevant) tasks, to test the Plugin-Selector’s robustness.\nSingle + Non.\nRemove\nNumber of Negative Samples\nVP(·)\nTP(·)\n1\n3\n5\n7\n15\nZTA ↑\nNaN\n0.625\n0.559\n0.648\n0.725\n0.779\n0.817\nTable 6. Ablation studies of Plugin-Selector. ‘NaN’ indicates non-\nconvergence of training, resulting in unavailable result.\nInput\nRestoration\nColorization\nRestor. + Colori.\nInput\nSnow Generation\nInput\nRain Generation\nFigure 7. Diverse uses of Diff-Plugin: multi-task combination in\nrow-1 and reversed low-level tasks in row-2.\ntion can be roughly divided into restoration and coloriza-\ntion.). Row-2 highlights its ability to invert low-level tasks,\nenabling the generation of special effects like rain and snow.\n5. Conclusion\nIn this paper, we presented Diff-Plugin, a novel framework\ntailored for enhancing pre-trained diffusion models in han-\ndling various low-level vision tasks that need stringent de-\ntails preservation. Our Task-Plugin module, with its dual-\nbranch design, effectively incorporates task-specific priors\ninto the diffusion process to allow for high-fidelity details-\npreserving visual results without retraining the base model\nfor each task. The Plugin-Selector further adds intuitive\nuser interaction through text inputs, enabling text-driven\nlow-level tasks and enhancing the framework’s practicality.\nExtensive experiments across various vision tasks demon-\nstrate the superiority of our framework over existing meth-\nods, especially in real-world scenarios.\nOne limitation of our current Diff-Plugin framework is\nthe inability in local editing. For example, in Fig. 1, our\nmethod may fail to remove only the snow specifically on the\nriver while keeping those in the sky. One possible solution\nfor this problem is to integrate LLMs [35, 97] to indicate\nthe region in which the task is performed.\n\n\nReferences\n[1] Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta,\nYaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried,\nand Xi Yin. Spatext: Spatio-textual representation for con-\ntrollable image generation. In CVPR, pages 18370–18380,\n2023. 3\n[2] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat,\nJiaming Song, Karsten Kreis, Miika Aittala, Timo Aila,\nSamuli Laine, Bryan Catanzaro, et al.\nediffi:\nText-to-\nimage diffusion models with an ensemble of expert denois-\ners. arXiv, 2022. 2\n[3] Mikołaj Bi´\nnkowski, Danica J Sutherland, Michael Arbel, and\nArthur Gretton. Demystifying mmd gans. In ICLR, 2018. 5\n[4] Tim Brooks, Aleksander Holynski, and Alexei A Efros. In-\nstructpix2pix: Learning to follow image editing instructions.\nIn CVPR, pages 18392–18402, 2023. 1, 2, 5, 6, 7\n[5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge-\noffrey Hinton. A simple framework for contrastive learning\nof visual representations. In ICML, pages 1597–1607, 2020.\n5\n[6] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen\nGuo.\nMulti-label image recognition with graph convolu-\ntional networks. In CVPR, pages 5177–5186, 2019. 5\n[7] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mc-\ncann, Marc Louis Klasky, and Jong Chul Ye. Diffusion pos-\nterior sampling for general noisy inverse problems. In ICLR,\n2023. 3\n[8] Guillaume Couairon,\nMarl`\nene Careil,\nMatthieu Cord,\nSt´\nephane Lathuili`\nere, and Jakob Verbeek. Zero-shot spatial\nlayout conditioning for text-to-image diffusion models. In\nICCV, pages 2174–2183, 2023. 2\n[9] Prafulla Dhariwal and Alexander Nichol. Diffusion models\nbeat gans on image synthesis. In NeurIPS, pages 8780–8794,\n2021. 1, 2\n[10] Wenkai Dong, Song Xue, Xiaoyue Duan, and Shumin Han.\nPrompt tuning inversion for text-driven image editing using\ndiffusion models. In ICCV, pages 7430–7440, 2023. 2\n[11] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming\ntransformers for high-resolution image synthesis. In CVPR,\npages 12873–12883, 2021. 4\n[12] Ben Fei, Zhaoyang Lyu, Liang Pan, Junzhe Zhang, Weidong\nYang, Tianyue Luo, Bo Zhang, and Bo Dai. Generative dif-\nfusion prior for unified image restoration and enhancement.\nIn CVPR, pages 9935–9946, 2023. 3\n[13] Gang Fu, Qing Zhang, Lei Zhu, Ping Li, and Chunxia Xiao.\nA multi-task network for joint specular highlight detection\nand removal. In CVPR, pages 7752–7761, 2021. 5, 7\n[14] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik,\nAmit H Bermano, Gal Chechik, and Daniel Cohen-Or. An\nimage is worth one word: Personalizing text-to-image gen-\neration using textual inversion. In ICLR, 2023. 2\n[15] Yuchao Gu, Xintao Wang, Liangbin Xie, Chao Dong, Gen\nLi, Ying Shan, and Ming-Ming Cheng.\nVqfr: Blind face\nrestoration with vector-quantized dictionary and parallel de-\ncoder. In ECCV, pages 126–143, 2022. 5, 7\n[16] Lanqing Guo, Chong Wang, Wenhan Yang, Siyu Huang,\nYufei Wang, Hanspeter Pfister, and Bihan Wen. Shadowd-\niffusion: When degradation prior meets diffusion model for\nshadow removal. In CVPR, pages 14049–14058, 2023. 1, 3\n[17] Xiaojie Guo, Yu Li, and Haibin Ling. Lime: Low-light im-\nage enhancement via illumination map estimation.\nIEEE\nTIP, 26(2):982–993, 2016. 5\n[18] Kaiming He, Georgia Gkioxari, Piotr Doll´\nar, and Ross Gir-\nshick. Mask r-cnn. In ICCV, pages 2961–2969, 2017. 3\n[19] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman,\nYael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image\nediting with cross attention control. In ICLR, 2022. 2\n[20] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner,\nBernhard Nessler, and Sepp Hochreiter. Gans trained by a\ntwo time-scale update rule converge to a local nash equilib-\nrium. In NeurIPS, 2017. 5\n[21] Jonathan Ho and Tim Salimans.\nClassifier-free diffusion\nguidance. arXiv, 2022. 1, 2\n[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif-\nfusion probabilistic models. In NeurIPS, pages 6840–6851,\n2020. 1, 2, 3\n[23] Gary B Huang, Marwan Mattar, Tamara Berg, and Eric\nLearned-Miller.\nLabeled faces in the wild: A database\nforstudying face recognition in unconstrained environments.\nIn Technical report, University of Massachusetts, Amherst,\n2007. 5\n[24] Hai Jiang, Ao Luo, Haoqiang Fan, Songchen Han, and\nShuaicheng Liu.\nLow-light image enhancement with\nwavelet-based diffusion models. TOG, 42(6):1–14, 2023. 3\n[25] Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian\nRupprecht. Diffusion models for zero-shot open-vocabulary\nsegmentation. arXiv, 2023. 1\n[26] Tero Karras, Samuli Laine, and Timo Aila. A style-based\ngenerator architecture for generative adversarial networks. In\nCVPR, pages 4401–4410, 2019. 5\n[27] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming\nSong. Denoising diffusion restoration models. In NeurIPS,\n2022. 3\n[28] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen\nChang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic:\nText-based real image editing with diffusion models.\nIn\nCVPR, pages 6007–6017, 2023. 1, 2\n[29] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli\nShechtman, and Jun-Yan Zhu.\nMulti-concept customiza-\ntion of text-to-image diffusion. In CVPR, pages 1931–1941,\n2023. 2\n[30] Chulwoo Lee, Chul Lee, and Chang-Su Kim. Contrast en-\nhancement based on layered difference representation of 2d\nhistograms. IEEE TIP, 22(12):5372–5384, 2013. 5\n[31] Alexander C. Li, Mihir Prabhudesai, Shivam Duggal, Ellis\nBrown, and Deepak Pathak. Your diffusion model is secretly\na zero-shot classifier. In ICCV, pages 2206–2217, 2023. 1\n[32] Boyi Li, Wenqi Ren, Dengpan Fu, Dacheng Tao, Dan Feng,\nWenjun Zeng, and Zhangyang Wang. Benchmarking single-\nimage dehazing and beyond.\nIEEE TIP, 28(1):492–505,\n2018. 5, 7\n[33] Boyun Li, Xiao Liu, Peng Hu, Zhongqin Wu, Jiancheng Lv,\nand Xi Peng. All-in-one image restoration for unknown cor-\nruption. In CVPR, pages 17452–17462, 2022. 3, 5, 7\n\n\n[34] Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Ben Fei,\nBo Dai, Wanli Ouyang, Yu Qiao, and Chao Dong. Diffbir:\nTowards blind image restoration with generative diffusion\nprior. arXiv, 2023. 3\n[35] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.\nVisual instruction tuning. In NeurIPS, 2023. 8\n[36] Yun-Fu Liu, Da-Wei Jaw, Shih-Chia Huang, and Jenq-Neng\nHwang. Desnownet: Context-aware deep network for snow\nremoval. IEEE TIP, 27(6):3064–3073, 2018. 5, 7\n[37] Ilya Loshchilov and Frank Hutter. Decoupled weight decay\nregularization. arXiv, 2017. 5\n[38] Kede Ma, Kai Zeng, and Zhou Wang.\nPerceptual quality\nassessment for multi-exposure image fusion. IEEE TIP, 24\n(11):3345–3356, 2015. 5\n[39] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and\nDaniel Cohen-Or. Null-text inversion for editing real images\nusing guided diffusion models. In CVPR, pages 6038–6047,\n2023. 2, 5, 6, 7\n[40] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhon-\ngang Qi, Ying Shan, and Xiaohu Qie. T2i-adapter: Learning\nadapters to dig out more controllable ability for text-to-image\ndiffusion models. arXiv, 2023. 2\n[41] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep\nmulti-scale convolutional neural network for dynamic scene\ndeblurring. In CVPR, pages 3883–3891, 2017. 5\n[42] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav\nShyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and\nMark Chen. Glide: Towards photorealistic image generation\nand editing with text-guided diffusion models. PMLR, 2021.\n2\n[43] OpenAI. Chatgpt plugins: https://openai.com/blog/chatgpt-\nplugins. 2023. 3\n[44] OpenAI. Gpt-4 technical report. arXiv, 2023. 5\n[45] Ozan ¨\nOzdenizci and Robert Legenstein. Restoring vision in\nadverse weather conditions with patch-based denoising dif-\nfusion models. IEEE TPAMI, 2023. 3\n[46] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun\nLi, Jingwan Lu, and Jun-Yan Zhu. Zero-shot image-to-image\ntranslation. In SIGGRAPH, pages 1–11, 2023. 1, 2, 5, 7\n[47] Vaishnav Potlapalli, Syed Waqas Zamir, Salman Khan, and\nFahad Shahbaz Khan. Promptir: Prompting for all-in-one\nblind image restoration. In NeurIPS, 2023. 3, 5, 6, 7\n[48] Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang,\nYingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming\nXiong, Silvio Savarese, et al. Unicontrol: A unified diffu-\nsion model for controllable visual generation in the wild. In\nNeurIPS, 2023. 3\n[49] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya\nRamesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry,\nAmanda Askell, Pamela Mishkin, Jack Clark, et al. Learn-\ning transferable visual models from natural language super-\nvision. In ICML, pages 8748–8763, 2021. 2, 4, 5\n[50] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee,\nSharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and\nPeter J Liu. Exploring the limits of transfer learning with\na unified text-to-text transformer. JMLR, pages 5485–5551,\n2020. 2\n[51] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu,\nand Mark Chen. Hierarchical text-conditional image gener-\nation with clip latents. arXiv, 2022. 2\n[52] Mengwei Ren, Mauricio Delbracio, Hossein Talebi, Guido\nGerig, and Peyman Milanfar.\nMultiscale structure guided\ndiffusion for image deblurring.\nIn ICCV, pages 10721–\n10733, 2023. 1, 3\n[53] Jaesung Rim, Haeyun Lee, Jucheol Won, and Sunghyun Cho.\nReal-world blur dataset for learning and benchmarking de-\nblurring algorithms. In ECCV, pages 184–201, 2020. 5, 7\n[54] Robin Rombach, Andreas Blattmann, Dominik Lorenz,\nPatrick Esser, and Bj¨\norn Ommer. High-resolution image syn-\nthesis with latent diffusion models. In CVPR, pages 10684–\n10695, 2022. 2, 4, 5, 6, 7\n[55] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch,\nMichael Rubinstein, and Kfir Aberman. Dreambooth: Fine\ntuning text-to-image diffusion models for subject-driven\ngeneration. In CVPR, pages 22500–22510, 2023. 2\n[56] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee,\nJonathan Ho, Tim Salimans, David Fleet, and Mohammad\nNorouzi. Palette: Image-to-image diffusion models. In SIG-\nGRAPH, pages 1–10, 2022. 1, 3\n[57] Chitwan Saharia, William Chan, Saurabh Saxena, Lala\nLi, Jay Whang, Emily L Denton, Kamyar Ghasemipour,\nRaphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans,\net al. Photorealistic text-to-image diffusion models with deep\nlanguage understanding. In NeurIPS, pages 36479–36494,\n2022. 2\n[58] Chitwan Saharia, Jonathan Ho, William Chan, Tim Sali-\nmans, David J Fleet, and Mohammad Norouzi. Image super-\nresolution via iterative refinement.\nIEEE TPAMI, 45(4):\n4713–4726, 2022. 3\n[59] Christoph Schuhmann, Romain Beaumont, Richard Vencu,\nCade Gordon,\nRoss Wightman,\nMehdi Cherti,\nTheo\nCoombes, Aarush Katta, Clayton Mullis, Mitchell Worts-\nman, et al. Laion-5b: An open large-scale dataset for train-\ning next generation image-text models. In NeurIPS, pages\n25278–25294, 2022. 2\n[60] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan,\nand Surya Ganguli.\nDeep unsupervised learning using\nnonequilibrium thermodynamics.\nIn ICML, pages 2256–\n2265, 2015. 2\n[61] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denois-\ning diffusion implicit models. In ICLR, 2021. 1, 2, 5\n[62] Yang Song and Stefano Ermon. Generative modeling by es-\ntimating gradients of the data distribution. In NeurIPS, 2019.\n2\n[63] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali\nDekel.\nPlug-and-play diffusion features for text-driven\nimage-to-image translation.\nIn CVPR, pages 1921–1930,\n2023. 1, 2, 5, 6, 7\n[64] Vassilios Vonikakis, Rigas Kouskouridas, and Antonios\nGasteratos. On the evaluation of illumination compensation\nalgorithms. MTA, 77:9211–9231, 2018. 5\n[65] Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact\ndiffusion inversion via coupled transformations. In CVPR,\npages 22532–22541, 2023. 2\n\n\n[66] Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin CK\nChan, and Chen Change Loy. Exploiting diffusion prior for\nreal-world image super-resolution. arXiv, 2023. 3\n[67] Shuhang Wang, Jin Zheng, Hai-Miao Hu, and Bo Li. Nat-\nuralness preserved enhancement algorithm for non-uniform\nillumination images. IEEE TIP, 22(9):3538–3548, 2013. 5\n[68] Tianyu Wang, Xin Yang, Ke Xu, Shaozhe Chen, Qiang\nZhang, and Rynson WH Lau. Spatial attentive single-image\nderaining with a high quality real rain dataset.\nIn CVPR,\npages 12270–12279, 2019. 5, 7\n[69] Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. To-\nwards real-world blind face restoration with generative facial\nprior. In CVPR, pages 9168–9178, 2021. 5, 7\n[70] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image\nrestoration using denoising diffusion null-space model. In\nICLR, 2022. 3\n[71] Zhixin Wang, Ziying Zhang, Xiaoyun Zhang, Huangjie\nZheng, Mingyuan Zhou, Ya Zhang, and Yanfeng Wang. Dr2:\nDiffusion-based robust degradation remover for blind face\nrestoration. In CVPR, pages 1704–1713, 2023. 1, 3\n[72] Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu.\nDeep retinex decomposition for low-light enhancement. In\nBMVC, 2018. 5\n[73] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan\nSaharia, Alexandros G Dimakis, and Peyman Milanfar. De-\nblurring via stochastic refinement. In CVPR, pages 16293–\n16303, 2022. 3\n[74] Bin Xia, Yulun Zhang, Shiyin Wang, Yitong Wang, Xing-\nlong Wu, Yapeng Tian, Wenming Yang, and Luc Van Gool.\nDiffir: Efficient diffusion model for image restoration. In\nICCV, pages 13095–13105, 2023. 3\n[75] Chaojun Xiao, Zhengyan Zhang, Xu Han, Chi-Min Chan,\nYankai Lin, Zhiyuan Liu, Xiangyang Li, Zhonghua Li, Zhao\nCao, and Maosong Sun. Plug-and-play document modules\nfor pre-trained models. In ACL, 2023. 3, 7\n[76] Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wen-\ntian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff:\nText-to-image synthesis with training-free box-constrained\ndiffusion. In ICCV, pages 7452–7461, 2023. 2\n[77] Canwen Xu, Yichong Xu, Shuohang Wang, Yang Liu, Chen-\nguang Zhu, and Julian McAuley. Small models are valuable\nplug-ins for large language models. arXiv, 2023. 3\n[78] Xingqian Xu, Jiayi Guo, Zhangyang Wang, Gao Huang, Ir-\nfan Essa, and Humphrey Shi. Prompt-free diffusion: Taking”\ntext” out of text-to-image diffusion models. arXiv, 2023. 4\n[79] Shuzhou Yang, Moxuan Ding, Yanmin Wu, Zihan Li, and\nJian Zhang.\nImplicit neural representation for coopera-\ntive low-light image enhancement. In ICCV, pages 12918–\n12927, 2023. 5, 7\n[80] Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zong-\nming Guo, and Shuicheng Yan. Deep joint rain detection and\nremoval from a single image. In CVPR, pages 1357–1366,\n2017. 3\n[81] Tian Ye, Yunchen Zhang, Mingchao Jiang, Liang Chen, Yun\nLiu, Sixiang Chen, and Erkang Chen. Perceiving and mod-\neling density for image dehazing. In ECCV, pages 130–145,\n2022. 5, 7\n[82] Tian Ye, Sixiang Chen, Jinbin Bai, Jun Shi, Chenghao Xue,\nJingxia Jiang, Junjie Yin, Erkang Chen, and Yun Liu. Ad-\nverse weather removal with codebook priors. In ICCV, pages\n12653–12664, 2023. 3\n[83] Xunpeng Yi, Han Xu, Hao Zhang, Linfeng Tang, and Jiayi\nMa. Diff-retinex: Rethinking low-light image enhancement\nwith a generative diffusion model. In CVPR, pages 12302–\n12311, 2023. 1, 3\n[84] Xin Yu, Peng Dai, Wenbo Li, Lan Ma, Jiajun Shen, Jia Li,\nand Xiaojuan Qi. Towards efficient and scale-robust ultra-\nhigh-definition image demoir´\neing. In ECCV, pages 646–662,\n2022. 5, 7\n[85] Shanxin Yuan, Radu Timofte, Gregory Slabaugh, Aleˇ\ns\nLeonardis, Bolun Zheng, Xin Ye, Xiang Tian, Yaowu Chen,\nXi Cheng, Zhenyong Fu, et al. Aim 2019 challenge on image\ndemoireing: Methods and results. In ICCVW, pages 3534–\n3545, 2019. 5, 7\n[86] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar\nHayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling\nShao. Multi-stage progressive image restoration. In CVPR,\npages 14821–14831, 2021. 5\n[87] Syed Waqas Zamir, Aditya Arora, Salman Khan, Mu-\nnawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang.\nRestormer: Efficient transformer for high-resolution image\nrestoration. In CVPR, pages 5728–5739, 2022. 5, 7\n[88] Kaihao Zhang, Rongqing Li, Yanjiang Yu, Wenhan Luo, and\nChangsheng Li. Deep dense multi-scale network for snow\nremoval using semantic and depth priors.\nIEEE TIP, 30:\n7419–7431, 2021. 5, 7\n[89] Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su.\nMagicbrush: A manually annotated dataset for instruction-\nguided image editing. In NeurIPS, 2023. 2\n[90] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding\nconditional control to text-to-image diffusion models.\nIn\nICCV, pages 3836–3847, 2023. 2, 3, 5, 6, 7\n[91] Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang,\nChongyang Ma, Weiming Dong, and Changsheng Xu.\nInversion-based style transfer with diffusion models.\nIn\nCVPR, pages 10146–10156, 2023. 1\n[92] Yi Zhang, Xiaoyu Shi, Dasong Li, Xiaogang Wang, Jian\nWang, and Hongsheng Li. A unified conditional framework\nfor diffusion-based image restoration. In NeurIPS, 2023. 3\n[93] Zhixing Zhang, Ligong Han, Arnab Ghosh, Dimitris N\nMetaxas, and Jian Ren.\nSine: Single image editing with\ntext-to-image diffusion models. In CVPR, pages 6027–6037,\n2023. 2\n[94] Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin\nBao, Shaozhe Hao, Lu Yuan, and Kwan-Yee K Wong.\nUni-controlnet: All-in-one control to text-to-image diffusion\nmodels. In NeurIPS, 2023. 2\n[95] Yang Zhao, Tingbo Hou, Yu-Chuan Su, Xuhui Jia, Yan-\ndong Li, and Matthias Grundmann. Towards authentic face\nrestoration with iterative diffusion models and beyond. In\nICCV, pages 7312–7322, 2023. 3\n[96] Yuzhong Zhao, Qixiang Ye, Weijia Wu, Chunhua Shen, and\nFang Wan. Generative prompt model for weakly supervised\nobject localization. In ICCV, pages 6351–6361, 2023. 1\n\n\n[97] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo-\nhamed Elhoseiny.\nMinigpt-4: Enhancing vision-language\nunderstanding with advanced large language models.\nIn\nICLR, 2024. 8\n[98] Yurui Zhu, Tianyu Wang, Xueyang Fu, Xuanyu Yang, Xin\nGuo, Jifeng Dai, Yu Qiao, and Xiaowei Hu.\nLearn-\ning weather-general and weather-specific features for image\nrestoration under multiple adverse weather conditions.\nIn\nCVPR, pages 21747–21758, 2023. 3, 5, 7","difficulty":"easy","domain":"Single-Document QA","length":"short","question":"Which of the following statements is incorrect?","sub_domain":"Academic"}

Source: https://huggingface.co/datasets/zai-org/LongBench-v2

initial import

Posting: /agents

GET /api/v1/write?intent=publish&task_id=51587ae2-81ae-5999-b2eb-8f3f122bd10d&body={url_encoded_text}&agent_name={optional_name}&nonce={optional_random_id}