{"kind":"task","effective_mode":"full","benchmark":{"kind":"benchmark","effective_mode":"full","slug":"longbench-v2","formal_name":"LongBench v2","introduction":"LongBench v2 evaluates deep understanding and reasoning over long contexts through multiple-choice questions. Its official description lists 503 questions spanning tasks such as single-document and multi-document QA and code-repository understanding.","introduction_ja":"","introduction_en":"","category":"Category not supplied","task_count":null,"acquisition_status":"Acquisition status not supplied","official_url":"https://huggingface.co/datasets/zai-org/LongBench-v2","indexing_mode":"noindex","profile":{"resources":[],"task_format":"","scoring":"","metric":"","size":"","answer_access":"","license":"","citation":"","maintainer":"","released":"","why_hard":"","related":[]}},"task_id":"b371f052-dc42-58df-a732-569f0d3396d9","task_key":"train--66ed364d821e116aacb1f47e","task_revision_id":"3","upstream_id":"66ed364d821e116aacb1f47e","short_description":"Which article dynamically models the temporal sequence of single-frame…","config":"","split":"train","body":"{\"choice_A\":\"Point-Supervised Video Temporal Grounding\",\"choice_B\":\"Probability Distribution Based Frame-supervised Language-driven Action Localization\",\"choice_C\":\"Mm - 2024 - Explicit Granularity and Implicit Scale Correspond.pdf\",\"choice_D\":\"D3G: Exploring Gaussian Prior for Temporal Sentence Grounding with Glance Annotation\",\"context\":\"Video Moment Retrieval from Text Queries via\\nSingle Frame Annotation\\nFudan University, China\\nygj@fudan.edu.cn\\nABSTRACT\\nVideo moment retrieval aims at finding the start and end times-\\ntamps of a moment (part of a video) described by a given natural\\nlanguage query. Fully supervised methods need complete temporal\\nboundary annotations to achieve promising results, which is costly\\nsince the annotator needs to watch the whole moment. Weakly\\nsupervised methods only rely on the paired video and query, but\\nthe performance is relatively poor. In this paper, we look closer\\ninto the annotation process and propose a new paradigm called\\n“glance annotation”. This paradigm requires the timestamp of only\\none single random frame, which we refer to as a “glance”, within\\nthe temporal boundary of the fully supervised counterpart. We\\nargue this is beneficial because comparing to weak supervision,\\ntrivial cost is added yet more potential in performance is provided.\\nUnder the glance annotation setting, we propose a method named\\nas Video moment retrieval via Glance Annotation (ViGA)1 based\\non contrastive learning. ViGA cuts the input video into clips and\\ncontrasts between clips and queries, in which glance guided Gauss-\\nian distributed weights are assigned to all clips. Our extensive\\nexperiments indicate that ViGA achieves better results than the\\nstate-of-the-art weakly supervised methods by a large margin, even\\ncomparable to fully supervised methods in some cases.\\nCCS CONCEPTS\\n• Computing methodologies →Visual content-based index-\\ning and retrieval.\\n∗Both authors contributed equally to this research.\\n†Corresponding authors.\\n1We release our codes and glance re-annotated datasets at https://github.com/r-\\ncui/ViGA.\\nPermission to make digital or hard copies of all or part of this work for personal or\\nclassroom use is granted without fee provided that copies are not made or distributed\\nfor profit or commercial advantage and that copies bear this notice and the full citation\\non the first page. Copyrights for components of this work owned by others than the\\nauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or\\nrepublish, to post on servers or to redistribute to lists, requires prior specific permission\\nand/or a fee. Request permissions from permissions@acm.org.\\nSIGIR ’22, July 11–15, 2022, Madrid, Spain.\\n© 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.\\nACM ISBN 978-1-4503-8732-3/22/07...$15.00\\nhttps://doi.org/10.1145/3477495.3532078\\nKEYWORDS\\nvideo moment retrieval; contrastive learning; cross-modal learning\\nACM Reference Format:\\nRan Cui, Tianwen Qian, Pai Peng, Elena Daskalaki, Jingjing Chen, Xiaowei\\nGuo, Huyang Sun, and Yu-Gang Jiang. 2022. Video Moment Retrieval from\\nText Queries via Single Frame Annotation. In Proceedings of the 45th Inter-\\nnational ACM SIGIR Conference on Research and Development in Information\\nRetrieval (SIGIR ’22), July 11–15, 2022, Madrid, Spain. ACM, New York, NY,\\nUSA, 11 pages. https://doi.org/10.1145/3477495.3532078\\n1\\nINTRODUCTION\\nVideo moment retrieval (VMR), initially proposed in [1, 12], is the\\ntask of retrieving the segment described by a given natural lan-\\nguage query from an untrimmed video. This task, also known as\\nnatural language video localization [25, 48, 52] and video temporal\\ngrounding [5, 29], is a fundamental problem in computer vision\\nunderstanding and visual information retrieval. Differing to an ear-\\nlier task video action localization [11, 22], which aims at localizing\\npre-defined categorized actions from the video, VMR is considered\\nas a more difficult task since the query is generalized to free natural\\nlanguage thus involving with more complex cross-modal contents\\nunderstanding. VMR can be widely applied in many scenarios such\\nas video browsing websites and semantics based video search en-\\ngines.\\nTo date, deep learning methods have approached VMR from two\\ndirections differing in the way of data annotation. In building a\\ndataset of fully supervised VMR, given the target video, the annotator\\nis asked to choose a segment in the video and write a short text\\nquery to describe the segment. In the meanwhile, the start and end\\ntimestamps of this segment are noted down. Thus, one example in\\nthe dataset is a quadruplet of video, query, start and end, denoted by\\n(𝑉,𝑄,𝑠𝑡,𝑒𝑑). Though many methods under fully supervised VMR [8,\\n12, 29, 42, 45, 48, 49] have achieved good performance, an obvious\\ndisadvantage of this data annotating paradigm is its high time cost.\\nBesides, the annotation quality varies according to the annotator’s\\nsubjective judgements, especially in determining the start and end:\\nthe annotator is forced to give specific timestamps of the query, but\\nthe video segment is often not separated to its context with clear\\nborder. For example, to annotate the start and end of a query “the\\nman turns the light on”, one might consider the very second that\\nthe switch is toggled should be the temporal interval, but another\\narXiv:2204.09409v3  [cs.CV]  18 Jun 2022\\n\\n\\nQuery: He flips the eggs, making an omelette.\\nWeakly Supervised\\nAnnotation\\nFully Supervised\\nAnnotation\\nGlance Annotation\\n0 s\\n198.1 s\\n114.9 s\\n156.5 s\\n138.3 s\\nFigure 1: Illustration of different VMR training example an-\\nnotation paradigms. The fully supervised setting marks the\\nstart and end timestamps of the moment corresponding to\\nquery. Weak supervision only annotates the video-text pair.\\nOur proposed glance annotation marks a single timestamp\\nin the moment.\\nmight start the annotation from when the man walks towards\\nthe switch. This makes fully supervised VMR prone to subjective\\nannotation noise. To avoid these disadvantages, weakly supervised\\nVMR is proposed [28], in which one example in the dataset is simply\\n(𝑉,𝑄), and no start nor end annotation is available. Though not\\ncomparable to fully supervised VMR in performance, many studies\\n[17, 23, 27, 37, 39, 44] have shown that weak supervision is a feasible\\ncompromise when the annotating resources are limited.\\nIn our study, we argue that weak annotation can be augmented\\nwith trivial cost and propose “glance annotation”, a new paradigm\\nof data annotation in VMR. As illustrated in Figure 1, a training\\nexample under glance annotation is composed of (𝑉,𝑄,𝑔) in which\\n𝑔is any timestamp between 𝑠𝑡and 𝑒𝑑. This paradigm is motivated\\nby the fact that to annotate even a weak example, it is still inevitable\\nfor one to watch the video in order to write the query, and very\\noften it is sufficient to know what the moment is about by watching\\nonly a short snippet of it. Assuming that with properly designed\\ngraphical user interface to support the annotation, one can note\\ndown an instant timestamp 𝑔during “glancing” the video with no\\nmore effort than a mouse click. Glance annotation not only largely\\nsaves the time consumption in watching the video compared to\\nfull supervision, but also provides more information than weak\\nsupervision.\\nTo validate the usability of glance annotation, we re-annotate\\nthree publicly available fully supervised VMR datasets, namely\\nActivityNet Captions [20], Charades-STA [12] and TACoS [33] by\\nsubstituting 𝑠𝑡and 𝑒𝑑with a uniformly sampled timestamp 𝑔in\\nrange [𝑠𝑡,𝑒𝑑]. Under this setting, we propose a contrastive learning\\nbased method named Video moment retrieval via Glance Annotation\\n(ViGA). Due to the lack of 𝑠𝑡and 𝑒𝑑, ViGA follows the multiple-\\ninstance learning (MIL) strategy widely adopted in weakly super-\\nvised VMR, which uses the correspondence between 𝑉and 𝑄as the\\nsupervision signal. In doing the training, the main objective is to\\nobtain a proper cross-modal encoder to project 𝑉and 𝑄to a joint\\nembedding space, which satisfies that the distance between the\\nembeddings of corresponding (𝑉,𝑄)pos is closer and the distance\\nbetween the embeddings of other combinations (𝑉,𝑄)neg is farther.\\nViGA extends this idea by splitting 𝑉into multiple clips 𝐶and\\nlearning in the granularity (𝐶,𝑄) instead, for making use of 𝑔by\\nenabling an assignment of different weights to all clips. Specifically,\\nwe use heuristic Gaussian distribution peaking at the glance po-\\nsition to generate the weights. In doing the inference, we follow\\nthe common proposal-based inference as in many weakly super-\\nvised methods, yet adjust the classical sliding window proposal\\ngeneration to an anchor driven proposal generation to better fit our\\ntraining strategy. To be specific, those sliding window proposals not\\nincluding a first selected anchor frame are pruned out. We enable\\nthe network finding the anchor by adding an additional training\\nobjective of focusing the attention of our multihead-attention [41]\\nbased backbone to the glance position.\\nAs will be shown in the rest of this paper, ViGA significantly\\noutperforms the state of the art of weakly supervised VMR, even\\ncomparable to fully supervised VMR when a high precision of re-\\ntrieved segment is not required. Our contributions are summarized\\nas follows:\\n• We propose glance annotation, a new VMR annotating para-\\ndigm, which requires no more significant annotating effort\\nthan weakly supervised VMR, while provides more potential\\nregarding the retrieval performance.\\n• We propose a contrastive learning based method ViGA for\\nglance annotated VMR, which achieves significantly better\\nperformance than weakly supervised methods.\\n• We release a unified version of glance annotations on top of\\nthree publicly available datasets ActivityNet Captions [20],\\nCharades-STA [12] and TACoS [33], to encourage future\\nresearch on this topic.\\n2\\nRELATED WORK\\nAfter initially proposed by [1, 12], early VMR studies mostly use\\nthe annotated start and end timestamps for the video-text temporal\\nalignment learning, which we term as fully supervised VMR [1, 5, 6,\\n12, 18, 24, 29, 49]. Due to the expensive annotation cost, researchers\\nthen began to exploit on learning under weak annotation with\\nvideo-text pairs only, which we term as weakly supervised VMR\\n[10, 13, 17, 23, 27, 28, 51].\\n2.1\\nFully Supervised VMR\\nExisting fully supervised VMR methods can be categorized into two\\ngroups. Two-stage methods [1, 12, 14, 18, 19, 24, 45] typically gen-\\nerate some pre-segmentation of proposal candidates using a sliding\\nwindow or other proposal networks, then input the generated pro-\\nposals and the text query separately into a cross-modal matching\\nnetwork to predict matching confidence and select the best match-\\ning segment as the output. Hendricks et al. [1] first proposed Mo-\\nment Context Network (MCN), which generated proposals based on\\nsliding window, and then projected the video moment feature and\\ntext query feature into a common representation space. Then they\\nused 𝐿2 distance as a measure to optimize triplet loss to narrow the\\ndistance of positive samples and enlarge the distance of intra-video\\nand inter-video negative samples. Xu et al. [45] proposed Query-\\nguided Segment Proposal Network (QSPN) for alleviating the huge\\ncomputation burden caused by sliding window. Specifically, QSPN\\nintegrated query features into video features to obtain attention\\nweights in time indexing, then combined it with 3D Region of In-\\nterest (ROI) pooling to obtain the sparse proposals. End-to-end\\nmodels [5, 6, 29, 43, 46, 47, 49] can be divided into anchor-based\\n\\n\\nQuery: He flips \\nthe eggs, making \\nan omelette.\\n3DCNN\\nGloVe\\nmax pooling\\nQAG-KL   +   GLS-NCE\\nCross-modal Representation Module\\nGaussian Alignment Module\\nBi-GRU \\nQuery Encoder\\nQuery to Video\\nCross-modal Encoder\\nVideo to Query\\nCross-modal Encoder\\nSelf-attention\\nVideo Encoder\\nglance\\nglance\\nFigure 2: Illustration of our training framework. In the Gaussian Alignment Module, a blue square denotes a word feature and\\na yellow rectangle denotes the feature of one video frame. We use different heights of yellow squares to illustrate the different\\nweights of the frames. The solid and dashed green curves represent the heuristic Gaussian distribution generated with the\\nglance and the attention distribution generated by the model, respectively.\\n[5, 32, 43, 49] methods and anchor free [6, 7, 29, 46, 48] methods,\\nin which they differ in using / not using proposals in prediction,\\nrespectively. As a typical work in the anchor-based category, Zhang\\net al. [49] proposed 2D Temporal Adjacent Networks (2D-TAN) that\\nmodeled the relations between segments of varying durations using\\na two-dimensional feature map. The (𝑖, 𝑗)-th location of the feature\\nmap indicated the start and end timestamps of the proposed seg-\\nments. It then employed a Convolutional Neural Network (CNN) to\\nmodel the contextual interaction between various segments, using\\nground truth labels to optimize the prediction score of each sug-\\ngestion in the feature map. For anchor-free methods, they usually\\npredict the probability of a frame being the start or end, or use a\\nneural network to directly regress the values of start and end. For\\nexample, Lei et al. proposed XML [21] and used the 1D Convolu-\\ntional Start-End detector (ConvSE) to generate the start and end\\nscores on the late fused query-clip similarity matrix.\\n2.2\\nWeakly Supervised VMR\\nAlthough the fully supervised methods achieve good performance,\\nthe expensive cost of annotating the temporal boundary limits\\npractical applications. Therefore, researchers recently began to pay\\nattention to the weakly supervised VMR [10, 13, 17, 23, 27, 28, 51].\\nUnder the weakly supervised setting, we cannot obtain the detailed\\nstart and end annotation of each query, only know whether the\\nquery and video is a positive pair during training stage. Under this\\nconstraint, most methods adopt the MIL framework. In MIL-based\\nVMR, the model learns the video-text alignment at video-level by\\nmaximizing similarity scores of positive examples and suppress-\\ning them on negative examples. Text-Guided Attention (TGA) [28]\\nwas a typical pioneer work under the weak setting, which learned\\ntext-aware video representation and leverages ranking loss to distin-\\nguish positive and negative samples. Ma et al. proposed VLANet [27]\\nwhich attempted to eliminate some irrelevant suggestions in the pro-\\ncess of MIL. Cross-sentence Relations Mining (CRM) [17] presented\\nby Huang et al. explored the temporal information modeling in\\nMIL using combinational associations among sentences. Semantic\\nCompletion Network (SCN) [23] provided another reconstruction-\\nbased idea of restoring the masked keywords in query according to\\nvisual proposal and context information for the alignment learning\\nbetween modalities. Although weakly supervised VMR greatly re-\\nduces the burden of annotation, the performance of weak method\\nhas a significant gap between the fully supervised method on the\\ntest set.\\n3\\nMETHODOLOGY\\nIn this section, we first formally define the problem of glance an-\\nnotated VMR and give an overview of our method ViGA. We then\\nintroduce the two modules which form our training pipeline in\\nSection 3.3 and 3.4, respectively. The inference process is detailed\\nin Section 3.5.\\n3.1\\nGlance Annotation\\nGiven an untrimmed video 𝑉and a text query 𝑄that semantically\\ndescribes a segment of the video, the VMR task aims at finding\\nthe start and end timestamps 𝑠𝑡and 𝑒𝑑, such that moment 𝑉𝑠𝑡:𝑒𝑑\\nbest corresponds to the query description. In fully supervised VMR,\\ncomplete human annotated 𝑠𝑡and 𝑒𝑑information is provided. In\\ncontrast, under the weakly supervised VMR setting, only aligned\\n(𝑉,𝑄) pairs are available, with no fine-grained 𝑠𝑡or 𝑒𝑑information.\\nOur glance annotation scenario lies in between: a single timestamp\\n𝑔, satisfying 𝑠𝑡≤𝑔≤𝑒𝑑, is available at the training stage. We refer\\nto this timestamp 𝑔as a “glance”.\\n3.2\\nAlgorithm Overview\\nSimilar to the weakly supervised setting, it is not possible to let a\\nnetwork learn to directly output 𝑠𝑡and 𝑒𝑑under glance annotation,\\ndue to the lack of complete supervision signals. Instead, our method\\nselects a clip 𝐶from 𝑉that best matches 𝑄from a set of proposals\\nas the output. To learn this visual-textual alignment, many studies\\nin weakly supervised VMR adopt the MIL strategy and turn into\\nexploiting the correspondence of (𝑉,𝑄). Videos and queries that\\n\\n\\nwe know are from the same example are marked as positive cor-\\nrespondence (𝑉,𝑄)pos, while all other combinations in the batch\\nare treated as negative (𝑉,𝑄)neg. Our work extends this idea to\\na finer-grained (𝐶,𝑄) level. Specifically, we build a network that\\nprojects inputs from textual and visual modalities to a joint embed-\\nding space, and train the network with a clip-to-query contrastive\\nobjective, which pulls the distance between (𝐶,𝑄)pos closer and\\npushes the distance between (𝐶,𝑄)neg farther.\\nTraining. The overall structure of our training pipeline is il-\\nlustrated in Figure 2. After an initial feature extraction from pre-\\ntrained models, our Cross-modal Representation Module encodes the\\ntwo input modalities by first applying two independent uni-modal\\nencoders, and then cross-interacting the two uni-modal features\\nto each other. As a result, token-level (words for text and frames\\nfor video) cross-modal features are obtained. The Query to Video\\nCross-modal Encoder additionally outputs an attention distribution\\nacross all video frames. To train the network, we propose a Gauss-\\nian Alignment Module, in which we generate a heuristic Gaussian\\ndistribution peaking on the glance timestamp. All video frame fea-\\ntures are weighted by this heuristic distribution in calculating our\\nGaussian Label-Smoothed Noise Contrastive Estimation loss (GLS-\\nNCE). The same Gaussian heuristic distribution is further used\\nin our Query Attention Guide Kullback–Leibler Divergence loss\\n(QAG-KL) to guide the learning of our network. The total loss of\\nour network is a fusion of the two loss functions.\\nInference. To align with the training design, we propose a cor-\\nresponding Query Attention Guided Inference strategy. After the\\nnetwork forward pass up to the Cross-modal Representation Module,\\nthe frame that gathers the most attention in Query to Video Cross-\\nmodal Encoder is marked as the anchor frame. We sample proposals\\nof different sizes around this anchor frame (i.e., a proposal must\\ncontain the anchor frame) and form a pool of proposals. The pro-\\nposal that gets the highest dot-product similarity to the sentence\\nfeature is selected as the final output.\\n3.3\\nCross-modal Representation Module\\nGiven a video𝑉= [𝑣1, ..., 𝑣𝐿𝑣], and a query 𝑄= [𝑞1, ...,𝑞𝐿𝑞], we en-\\ncode deep features of the two inputs using the proposed Cross-modal\\nRepresentation Module. Specifically, we first use two independent\\nencoders to ensure sufficient understanding of uni-modal semantics\\nfor video and query. Next, to enable the cross-modal learning, we\\nfuse the semantics of the two modalities in the subsequent bidirec-\\ntional cross-modal encoder. As a result, per-token representations\\nv ∈R𝐿𝑣×𝑑model and q ∈R𝐿𝑞×𝑑model are obtained, where 𝑑model is\\nthe dimension of the joint embedding feature (and also the overall\\nhidden dimension of our network).\\nQuery Encoding. A bidirectional Gated Recurrent Unit (GRU)\\nis applied to encode the sequential semantics of all 𝐿𝑞words in 𝑄,\\ntaking word embeddings from the pre-trained GloVe [31] model as\\ninput. A word-level feature q𝑖is the concatenation of the forward\\nand backward hidden states of the final layer of the GRU, given by\\nq𝑖= [\\n→\\nh𝑖;\\n←\\nh𝑖] ∈R𝑑model.\\n(1)\\n·\\n·\\n·\\nSomeone puts onions \\nin the skillet and beats \\neggs into a bowl \\nbefore adding them to \\nthe skillet.\\nThe coach helps the \\nguy in red with the \\nproper body placement \\nand lifting technique.\\nA band plays music on \\nstage.\\n·\\n·\\n·\\n·\\n·\\n·\\nFigure 3: Illustration of clip-level MIL training strategy in\\none batch in the Gaussian Alignment Module. Green solid\\nlines indicate positive correspondences, and gray dashed\\nlines indicate negative matching.\\nVideo Encoding. For an untrimmed video, we first extract fea-\\ntures using a pre-trained CNN, such as C3D [40], I3D [4] and VGG\\n[36], followed by a fully connected layer to map the feature dimen-\\nsion to 𝑑model. To encode the sequential semantics of the extracted\\nvideo feature v, we apply a multihead self-attention module [41]\\nacross all the frames. The encoding at the 𝑖-th frame is given by\\nAttn(𝑄(v𝑖), 𝐾(v),𝑉(v)) = softmax(𝑄(v𝑖)𝐾(v)𝑇\\n√︁\\n𝑑model/ℎ\\n)𝑉(v),\\n(2)\\nwhere 𝑄(·), 𝐾(·) and 𝑉(·) are three independent linear transforma-\\ntions from 𝑑model to 𝑑model, and ℎdenotes the number of heads.\\nCross-modal Encoding. To fuse the information from the two\\nmodalities, we apply cross-modal multihead attention after the\\nindividual uni-modal self encoding, i.e., using one modality as query\\nand the other as key and value. In this way, the cross-encoding of\\nthe 𝑖-th word is given by\\nAttn(𝑄(q𝑖), 𝐾(v),𝑉(v)) = softmax(𝑄(q𝑖)𝐾(v)𝑇\\n√︁\\n𝑑model/ℎ\\n)𝑉(v),\\n(3)\\nand the cross-encoding of the 𝑖-th frame is given by\\nAttn(𝑄(v𝑖), 𝐾(q),𝑉(q)) = softmax(𝑄(v𝑖)𝐾(q)𝑇\\n√︁\\n𝑑model/ℎ\\n)𝑉(q).\\n(4)\\nFor each encoding module in the uni-modal encoding and the\\ncross-modal encoding, the module is subsequently followed by a\\ntwo-layer feed-forward module activated by ReLU [30] to further\\nenhance the encoding capacity. Moreover, we follow the standard\\nconfiguration of multihead attention modules, where layernorm\\n[2], dropout [38], position embedding [9] and residual connection\\n[16] are applied.\\n3.4\\nGaussian Alignment Module\\nIn MIL-based methods under weakly supervised VMR, the general\\nparadigm is to learn proper deep representation 𝑓𝑣(𝑉) ∈R𝑑model\\nand 𝑓𝑞(𝑄) ∈R𝑑model that corresponding pairs align closer to each\\nother via contrastive learning. We extend this idea of video-level\\n\\n\\nMIL and propose a Gaussian Alignment Module that transforms the\\nproblem to a finer-grained clip-level MIL to train the preceding\\nCross-modal Representation Module. Our motivation is that the exis-\\ntence of glance 𝑔makes frames in the video in-equally important\\nin terms of the relevance to the query. For a frame 𝑣𝑖, the relevance\\nis higher when its temporal distance to 𝑔is closer: consider a long\\nvideo including scene change, the frames that are too far away\\nfrom 𝑔might contain totally irrelevant semantics. Mathematically,\\nGaussian distribution has the characteristic that the highest proba-\\nbility value at the mean point and gradually reduces the probability\\nto both sides, which aligns consistently to our motivation. Thus,\\nwe use Gaussian distribution to model this relevance. As illustrated\\nin Figure 3, all video frames are assigned with Gaussian-distributed\\nweights where the peak position of Gaussian is the glance 𝑔. To\\nget the weight of the 𝑖-th frame, we scale the index 𝑖∈{1, 2, ..., 𝐿𝑣}\\ninto the domain [−1, 1] by linear transformation\\n𝑓(𝑖) = (𝑖−1) ·\\n2\\n𝐿𝑣−1 −1,\\n(5)\\nand sample the Gaussian values via the probability density function\\n𝐺(𝑖) = norm(\\n1\\n√\\n2𝜋𝜎\\nexp(−(𝑓(𝑖) −𝑓(𝑔))2\\n2𝜎2\\n)),\\n(6)\\nwhere 𝜎is a hyperparameter, and the normalization scales 𝐺(𝑖)\\nwhere 𝑖∈[−1, 1] into range [0, 1].\\nAfter different weights are assigned across the video frames,\\nwe are able to get video clips with different weights as training\\nexamples. A sliding window of size 𝐿𝑐with stride 𝑠is applied on\\nthe video to get clips. Each clip is then max pooled along the frame\\ndimension to generate the clip-level feature in the joint embedding\\nspace R𝑑model. To this end, the 𝑖-th clip feature c𝑖is given by\\nc𝑖= max_pool([v(𝑖−1) ·𝑠+1, ..., v(𝑖−1) ·𝑠+𝐿𝑐]) ∈R𝑑model.\\n(7)\\nAnd each clip is assigned with a clip-level weight 𝑤𝑖sampled at\\nthe middle point of the clip, given by\\n𝑤𝑖= 𝐺((𝑖−1) · 𝑠+ 𝐿𝑐\\n2 ).\\n(8)\\nSimilarly, for the text modality, sentence-level feature s is ob-\\ntained by max pooling its word-level features, given by\\ns = max_pool([q1, ..., q𝐿𝑞]) ∈R𝑑model.\\n(9)\\nGLS-NCE Loss. In weakly supervised VMR, standard NCE loss\\non video level can be directly applied to train the video and query\\nencoders 𝑓𝑣(·) and 𝑓𝑞(·) by contrasting (𝑉,𝑄)pos against (𝑉,𝑄)neg\\nin one batch. For one video 𝑉in a batch of 𝐵video query pairs,\\nthere is only one positive matching query 𝑄𝑝, and the rest 𝐵−1\\nqueries are negative queries 𝑄𝑛. Therefore, the standard video-level\\nNCE loss is given by\\nLVideo-NCE = −log(\\n𝑒𝑓𝑣(𝑉)⊤𝑓𝑞(𝑄𝑝)\\n𝑒𝑓𝑣(𝑉)⊤𝑓𝑞(𝑄𝑝) +\\n𝐵−1\\nÍ\\n𝑖=1\\n𝑒𝑓𝑣(𝑉)⊤𝑓𝑞(𝑄𝑛\\n𝑖)\\n).\\n(10)\\n·\\n·\\n·\\n·\\n·\\n·\\nVideo Frame\\nAnchor Frame\\nSliding Window\\nSliding window based inference\\nOur proposed QAGI\\nFigure 4: Comparison between sliding window based infer-\\nence and our proposed Query Attention Guided Inference.\\nHowever, in our method, the proposed GLS-NCE loss is built on\\nclip level. Each video is substituted by 𝑁clips as in Equation 7. On\\nthis basis, for a video in the same batch of size 𝐵, clip-level NCE\\nloss is extended to\\nLClip-NCE = −log(\\n𝑁\\nÍ\\n𝑖=1\\n𝑒c⊤\\n𝑖𝑓𝑞(𝑄𝑝)\\n𝑁\\nÍ\\n𝑖=1\\n𝑒c⊤\\n𝑖𝑓𝑞(𝑄𝑝) +\\n𝑁\\nÍ\\n𝑖=1\\n𝐵−1\\nÍ\\n𝑗=1\\n𝑒c⊤\\n𝑖𝑓𝑞(𝑄𝑛\\n𝑗)\\n).\\n(11)\\nAdditionally, the clips also differ in weights given by Equation 8.\\nTo accommodate this, we implement clip-level NCE in the form of\\ncross-entropy following MoCo [15], and enable the weighting via\\nlabel smoothing. In this GLS-NCE loss, the Gaussian weight 𝑤𝑖of\\na clip c𝑖is assigned as the label smoothing amount, i.e., instead of\\nusing a one-hot label across the 𝐵queries in the batch, we assign\\n𝑤𝑖to the label of the positive query, and smooth the rest 𝐵−1\\nnegative labels to 1−𝑤𝑖\\n𝐵−1 . In summary, for a clip c𝑖with weight 𝑤𝑖,\\nits GLS-NCE loss is given by\\nLGLS-NCE = 𝑤𝑖· log(c⊤\\n𝑖𝑄𝑝) +\\n𝐵−1\\n∑︁\\n𝑗=1\\n1 −𝑤𝑖\\n𝐵−1 log(c⊤\\n𝑖𝑄𝑛\\n𝑗).\\n(12)\\nQAG-KL Loss. To further smooth the learning and to align with\\nthe inference strategy to be explained in Section 3.5, we leverage\\nthe nature of attention mechanism [41] in our encoding module\\nand propose the QAG-KL loss. Specifically, we use a KL divergence\\nbetween the attention distribution of the Query to Video Cross-\\nmodal Encoder (Equation 3) and the Gaussian guidance (Equation\\n6), to pull the attention distribution across all video frames closer\\nto the Gaussian guidance. Since the query 𝑄contains 𝐿𝑞words, we\\ntreat them equally and use the mean of their attention distributions\\nas the sentence-level attention distribution. For the 𝑖-th frame in\\nthe video, the QAG-KL loss is given by\\nLQAG-KL = 𝐺(𝑖)(log𝐺(𝑖) −a𝑖)\\n(13)\\nwhere a =\\n𝐿𝑞\\nÍ\\n𝑗=1\\nsoftmax( q𝑗𝐾(v)𝑇\\n𝑑model/ℎ)\\n𝐿𝑞\\n∈R𝐿𝑣.\\n(14)\\n\\n\\nThe complete loss function of a batch is the combination of the\\nGLS-NCE loss across all clips in the batch and the QAG-KL loss\\nacross all frames of all videos in the batch, given by\\nL =\\n∑︁\\nc\\nLGLS-NCE +\\n∑︁\\n𝑣\\n∑︁\\n𝐿𝑣\\nLQAG-KL.\\n(15)\\n3.5\\nQuery Attention Guided Inference\\nDue to the lack of 𝑠𝑡and 𝑒𝑑annotations, weakly supervised VMR\\nmethods often compromise to designing two independent pipelines\\nfor training and inference. Under weakly supervised VMR, the most\\ncommon inference strategy is to select the best proposal from a se-\\nries of proposals generated by methods like sliding window. Despite\\nthat it is still not possible to design a unified pipeline that handles\\ntraining and inference consistently under glance annotation, we\\npropose to use a Query Attention Guided Inference (QAGI) to best\\nalign the inference process to our aforementioned training strategy.\\nAs illustrated in Figure 4, given a video 𝑉and query 𝑄, we first\\nextract the features v ∈R𝐿𝑣×𝑑model and s ∈R𝑑model via the trained\\ncross-modal representation module as described in previous sec-\\ntions. After that, we select an anchor point index 𝑎𝑝∈{1, 2..., 𝐿𝑣}\\nwith the guidance of the query to video attention distribution.\\nSpecifically, the frame where the attention value reaches its maxi-\\nmum is chosen as the anchor frame, given by\\n𝑎𝑝= arg max\\n𝑖\\na𝑖.\\n(16)\\nA series of proposals are then generated around 𝑎𝑝, i.e., we first\\napply a naive sliding window on the 𝐿𝑣frames to generate a pro-\\nposals pool {𝑝𝑖:𝑗}, then prune out all proposals that does not satisfy\\n𝑖≤𝑎𝑝≤𝑗. On this basis, the proposal that maximizes the similarity\\nscore to the query is select as our final output, given by\\narg max\\n𝑖,𝑗\\nmax_pool([v𝑖:𝑗])⊤s.\\n(17)\\n4\\nEXPERIMENTS\\nTo validate our proposed glance annotation and the method ViGA,\\nextensive experiments are conducted on three publicly available\\ndatasets. We also perform ablation studies on different components\\nin ViGA to investigate their influence in details.\\n4.1\\nDatasets\\nWe re-annotate the following datasets to fit in our proposed glance\\nannotation. Specifically, we augment each example with a uniformly\\nsampled timestamp 𝑔in range [𝑠𝑡,𝑒𝑑].\\nActivityNet Captions. Krishna et al. [20] annotated the Activ-\\nityNet v1.3 dataset [3] which was originally designed for video\\ncaptioning, and released the ActivityNet Captions dataset for VMR.\\nIt contains 19,994 YouTube videos from diverse domains. Follow-\\ning previous studies [49, 50], we use the pre-defined split val_1\\nas validation set and test on val_2. As a result, 37,421, 17,505, and\\n17,031 annotations are used for training, validating, and testing,\\nrespectively.\\nVariants\\nR@IoU=\\nmIoU\\n0.3\\n0.5\\n0.7\\nVideo-NCE\\n35.58\\n18.30\\n8.54\\n25.34\\nClip-NCE\\n16.72\\n6.25\\n2.02\\n14.93\\nGLS-NCE\\n59.61\\n35.79\\n16.96\\n40.12\\nTable 1: Ablation comparison among training with different\\nNCE loss functions on ActivityNet Captions dataset.\\nVariants\\nR@IoU=\\nmIoU\\n0.3\\n0.5\\n0.7\\nw/o QAG-KL\\n54.74\\n34.26\\n16.68\\n37.96\\nw/ QAG-KL\\n59.61\\n35.79\\n16.96\\n40.12\\nTable 2: Ablation comparison between model trained with\\nand without QAG-KL loss on ActivityNet Captions dataset.\\nVariants\\nR@IoU=\\nmIoU\\n0.3\\n0.5\\n0.7\\nSliding Window\\n58.13\\n31.23\\n13.62\\n38.24\\nQuery Attention Guided Inference\\n59.61\\n35.79\\n16.96\\n40.12\\nTable 3: Ablation comparison of inference with query atten-\\ntion guided inference / naive sliding window on ActivityNet\\nCaptions dataset.\\nCharades-STA. Gao et al. [12] annotated the Charades dataset\\n[35] using a semi-automatic approach and formed the Charades-\\nSTA dataset. It contains 9,848 videos of daily indoors activities.\\nWe follow the standard split of 12,408 and 3,720 annotations for\\ntraining and testing defined by the annotator.\\nTACoS. Regneri et al. [33] annotated the MPII Cooking Compos-\\nite Activities dataset [34] which was originally designed for activity\\nrecognition, and formed the TACoS dataset. It contains 127 videos\\nof cooking. We follow the standard split provided by [12], and 9790,\\n4436, and 4001 annotations are included in training, validation and\\ntest set, respectively.\\n4.2\\nEvaluation Metric\\nWe evaluate our method using 1) recall of threshold bounded tem-\\nporal intersection over union (R@IoU), which measures the per-\\ncentage of correctly retrieved predictions where only the temporal\\nIoU between the prediction and the ground truth greater than a\\ncertain threshold is accepted, and 2) mean averaged IoU (mIoU)\\nover all predictions.\\n4.3\\nImplementation Details\\nWe fix the 3D CNN modules for extracting visual features for a\\nfair comparison. For all the three datasets, we use C3D as feature\\nextractor. Since Charades-STA lacks a unified standard of feature\\nextractor in previous studies, additional experiments using I3D and\\nVGG features are also conducted for the completeness of compari-\\nson. For the word embedding, we adopt 840B GloVe for building a\\nmost complete vocabulary. To increase the capacity of our encoders,\\nwe stack two layers of our query, video and cross-modal encoders.\\n\\n\\n0.0\\n0.2\\n0.4\\n0.6\\n0.8\\n1.0\\n1.2\\n1.4\\n1.6\\nsigma\\n10\\n20\\n30\\n40\\n50\\n60\\nRecall@1\\nIoU=0.3\\nIoU=0.5\\nIoU=0.7\\nmIoU\\n36\\n37\\n38\\n39\\n40\\nmIoU\\nFigure 5: Influence of the hyperparameter 𝜎of the Gaussian\\ndistribution in Equation 6 on ActivityNet Captions dataset.\\nThe model dimension 𝑑model is set to 512, and the number of atten-\\ntion heads ℎis set to 8 globally. Our model is trained with AdamW\\n[26] with a learning rate of 0.0001 half decaying on plateau. We\\nclip the gradient norm to 1.0 during training. The batch size and\\n𝜎factor of the three datasets are empirically set to (256, 0.4), (256,\\n0.3) and (128, 1.0), respectively. All experiments are conducted on a\\nNvidia Tesla V100 GPU with 32GB memory.\\n4.4\\nAblation Studies\\nTo evaluate the effectiveness of different components in our pro-\\nposed ViGA, we conduct extensive ablation experiments on the\\nActivityNet Captions dataset.\\nEffects of GLS-NCE. In order to verify the effectiveness of our\\nproposed GLS-NCE loss (Equation 12), we compare it with the\\naforementioned variants Video-NCE loss (Equation 10) and Clip-\\nNCE loss (Equation 11). The Video-NCE treats the video as a whole\\nand maximizes the similarity between it and the text query. The\\nClip-NCE cuts a video into many clips, which increases the number\\nof examples in the batch. However, as the distance between the\\nclip and glance increases, its relevance to the query becomes lower.\\nTherefore, our GLS-NCE assigns different weights to different clips\\naccording to their temporal distances to the glance. The results\\nare listed in Table 1. The performance of GLS-NCE is significantly\\nahead of others, thus showing its effectiveness. Besides, it is worth-\\nwhile to note that scores of Clip-NCE are almost half of Video-NCE,\\nindicating that simply increasing the number of samples through\\nclip segmentation is not beneficial, but sharply harms the perfor-\\nmance instead. Comparing the three groups of experiments, we\\nconclude that the performance improvement of GLS-NCE is not\\nbrought by increasing the number of examples by slicing a video\\ninto clips, while the enhancement from Gaussian label smoothing\\nmakes the main contribution.\\nEffects of QAG-KL. The QAG-KL loss (Equation 14) encourages\\nthe model to pay more attention to the glance frame and its near\\nneighbors in the training stage. To validate the its effectiveness, we\\nconduct the ablation study of simply removing the QAG-KL loss.\\nFrom the results in Table 2, we have the following observations.\\nFirst, QAG-KL improves the moment retrieval performance on all\\nevaluation metrics. This shows that in the training stage, QAG-KL\\ncan indeed make use of the prior information of glance annotation\\nand help the model with better cross-modal alignment learning.\\nSecond, the performance with QAG-KL increases more significantly\\nwhen the IoU threshold is 0.3 than other thresholds, reaching around\\n5%. We consider this gap is due to the fact that glance is a relatively\\nweak prior information, so it performs better when the requirement\\nof retrieval precision (reflected by the IoU) is not strict.\\nSliding Window vs. Query Attention Guided Inference. To\\nverify the effectiveness of our proposed QAGI, we evaluate the\\nsame trained model under different testing strategy, i.e., naive slid-\\ning window vs. QAGI. The results in Table 3 show that QAGI has\\nadvantages over the traditional sliding window based evaluation on\\nall metrics. QAGI uses the attention matrix learned in the training\\nstage to obtain the anchor frame for generating proposals in the\\ntest stage, which can filter out irrelevant proposals to a great extent,\\nespecially those with short durations. It is worthwhile to note that\\nthe improvement is more obvious under the metric with larger IoU\\nthreshold, as the performance raises by 4.5% and 3.3% respectively\\nat IoU threshold of 0.5 and 0.7. This suggests that using the anchor\\nis beneficial especially when the retrieval precision requirement is\\nrelatively high.\\nEffects of the Gaussian distribution parameter 𝝈. In this\\nablation study, we focus on the hyperparameter 𝜎in Equation 6.\\nTheoretically, 𝜎describes the dispersion degree of a Gaussian distri-\\nbution: the larger the 𝜎, the flatter the curve. In the context of our\\nGaussian Alignment Module, the value of 𝜎controls to what extent\\nthat the weight at the glance frame which is always 1.0 disperses to\\nother frames, hence affecting the overall positiveness of all the clips\\nin the video. Consider an extreme example, when 𝜎takes a very\\nlarge value, all frames in the video are assigned with weights close\\nto 1.0. This means that we take all clips almost equally positive,\\nwhich reduces the learning to be approximately equivalent to the\\nvideo-level MIL under weak supervision. Therefore, choosing an ap-\\npropriate 𝜎is important. As reported in Figure 5, as 𝜎increases, the\\nperformance of the four metrics first increases and then decreases.\\nSpecifically, when 𝜎is set to 1.2, i.e., we over-assign positiveness\\nto the clips, the performance of the four metrics decreases sharply\\n(e.g., mIoU decreases from 40 to 36). On the other hand, when 𝜎is\\nvery small, i.e., we only take a very narrow range of video clips as\\nimportant positive examples, the performance decreases because of\\nlosing some clips that are in fact informative positive examples (e.g.,\\nwhen 𝜎is set to 0.05, mIoU decreases by 2%). On the ActivityNet\\nCaptions dataset, the performance achieves its best when 𝜎is set to\\na medium value 0.4. This observation coincides with our theoretical\\nanalysis.\\n4.5\\nComparison with State of the Art\\nWe compare the proposed ViGA with both fully and weakly super-\\nvised methods, which are introduced as follows.\\nCompared Methods. As shown in Table 4, we divide the com-\\npared methods into three sections according to the supervision\\n\\n\\nSupervision\\nMethod\\nCharades-STA\\nActivityNet Captions\\nTACoS\\nR@0.3\\nR@0.5\\nR@0.7\\nmIoU\\nR@0.3\\nR@0.5\\nR@0.7\\nmIoU\\nR@0.3\\nR@0.5\\nR@0.7\\nmIoU\\nFull\\nSupervision\\nCTRL [12]\\n-\\n23.63\\n8.89\\n-\\n-\\n-\\n-\\n-\\n18.32\\n13.3\\n-\\n-\\nQSPN [45]\\n54.7\\n35.6\\n15.8\\n-\\n45.3\\n27.7\\n13.6\\n-\\n-\\n-\\n-\\n-\\n2D-TAN [49]\\n-\\n39.7\\n23.31\\n-\\n59.45\\n44.51\\n26.54\\n-\\n37.29\\n25.32\\n-\\n-\\nLGI [29]\\n72.96\\n59.46\\n35.48\\n51.38\\n58.52\\n41.51\\n23.07\\n41.13\\n-\\n-\\n-\\n-\\nVSLNet [48]\\n70.46\\n54.19\\n35.22\\n50.02\\n63.16\\n43.22\\n26.16\\n43.19\\n29.61\\n24.27\\n20.03\\n24.11\\nWeak\\nSupervision\\nTGA [28]\\n32.14\\n19.94\\n8.84\\n-\\n-\\n-\\n-\\n-\\n-\\n-\\n-\\n-\\nSCN [23]\\n42.96\\n23.58\\n9.97\\n-\\n47.23\\n29.22\\n-\\n-\\n-\\n-\\n-\\n-\\nBAR [44]\\n44.97\\n27.04\\n12.23\\n-\\n49.03\\n30.73\\n-\\n-\\n-\\n-\\n-\\n-\\nVLANet [27]\\n45.24\\n31.83\\n14.17\\n-\\n-\\n-\\n-\\n-\\n-\\n-\\n-\\n-\\nMARN [37]\\n48.55\\n31.94\\n14.81\\n-\\n47.01\\n29.95\\n-\\n-\\n-\\n-\\n-\\n-\\nLoGAN [39]\\n51.67\\n34.68\\n14.54\\n-\\n-\\n-\\n-\\n-\\n-\\n-\\n-\\n-\\nCRM [17]\\n53.66\\n34.76\\n16.37\\n-\\n55.26\\n32.19\\n-\\n-\\n-\\n-\\n-\\n-\\nGlance\\nSupervision\\n2D-TAN [49] †\\n-\\n-\\n-\\n-\\n11.26\\n5.28\\n2.34\\n-\\n13.97\\n5.50\\n1.60\\n-\\nLGI [29] †\\n51.94\\n25.67\\n7.98\\n30.83\\n9.34\\n4.11\\n1.31\\n7.82\\n-\\n-\\n-\\n-\\nViGA (C3D)\\n56.85\\n35.11\\n15.11\\n36.35\\n59.61\\n35.79\\n16.96\\n40.12\\n19.62\\n8.85\\n3.22\\n15.47\\nViGA (VGG)\\n60.22\\n36.72\\n17.20\\n38.62\\n-\\n-\\n-\\n-\\n-\\n-\\n-\\n-\\nViGA (I3D)\\n71.21\\n45.05\\n20.27\\n44.57\\n-\\n-\\n-\\n-\\n-\\n-\\n-\\n-\\nTable 4: Performance comparison with the state-of-the-art methods under different supervision settings. “†” denotes our re-\\nimplemented results of fully supervised methods under glance annotations. In order to align with their original design, we\\ngive a relaxed glance condition by shrinking the original annotations to a random 3-seconds duration instead of one instant\\ntimestamp as in our results.\\ntypes, including full supervision, weak supervision and glance su-\\npervision. When selecting methods from the literature to compare,\\nwe follow the rule of diversely selecting representative methods in\\ndifferent categories as introduced in Section 2 for the completeness\\nof the comparison. For two-stage fully supervised methods, CTRL\\n[12] is sliding window based and QSPN [45] is proposal based. In\\nthe end-to-end fully supervised methods, 2D-TAN [49] belongs\\nto anchor based, while LGI [29] and VSLNet [48] are anchor free.\\nFor weak supervision, a dominant number of methods adopt MIL\\nstrategy than query reconstruction. Therefore, we select MIL-based\\nmethods like TGA [28], VLANet [27], LoGAN [39], CRM [17] and\\none representative reconstruction-based method SCN [23]. Cur-\\nrently, CRM is the state of the art in weakly supervised VMR.\\nIn addition to these existing studies, we apply glance annotation\\nto two well-recognized fully supervised methods (i.e., 2D-TAN and\\nLGI) for a more direct comparison to our proposed ViGA. In order\\nto align with their original design, we give a relaxed glance condi-\\ntion by shrinking the original annotations to a random 3-seconds\\nduration instead of one instant timestamp. Practically, we achieve\\nthis by directly changing the annotations in the data and run their\\npublicly available source codes.\\nObservations and Discussions. According to the results in Ta-\\nble 4, we can make a number of observations worthy discussing.\\n1. In terms of all metrics, our proposed approach significantly\\nexceeds the methods under weak supervision on the Charades-STA\\nand ActivityNet Captions dataset. We improve the recall by 7%,\\n11% and 4% on Charades-STA when IoU is 0.3, 0.5 and 0.7, respec-\\ntively. On ActivityNet Captions, the improvement is 5% and 4%\\nwhen IoU is 0.3 and 0.5, respectively. We believe that on one hand,\\nit shows that the setting of glance annotation is reasonable and\\nhas good potential in performance, and on the other hand, it also\\nshows that ViGA succeeds in exploiting the information provided\\nby glance annotation. In addition, in order to make ViGA standard\\nand concise, we did not use some effective tricks in weak super-\\nvision methods, such as surrogate proposal selection in VLANet\\nand temporal semantic consistency in CRM. This may take the\\nperformance of ViGA further, and we leave this as future work.\\n2. When comparing to some fully supervised methods, we are\\nsurprised to find that when IoU is small (e.g., 0.3), our method almost\\nreaches a same performance level. For example, on Charades-STA,\\nour R@1 IoU=0.3 is 71.21%, 1.6% lower than LGI and 0.8% higher\\nthan VSLNet. On ActivityNet Captions, the recall is 59.61%, 1.9%\\nhigher than LGI and 3.6% lower than VSLNet. This suggests that\\nunder the scenario of coarse-grained retrieval requirements, glance\\nannotation might be more advantageous than full annotation with\\nacceptable performance yet significantly lower cost. However, there\\nis still a lot of space for improvement when a high retrieval precision\\nis required. For example, when the R@1 IoU=0.7, the performance\\ngap between ViGA and LGI on Charades-STA reaches 15.21%.\\n3. For the previously mentioned fully supervised method re-\\nimplemented under relaxed glance annotation, we have the follow-\\ning findings. First, although we relax the setting of glance to 3 sec-\\nonds, our approach shows superior performance in all three datasets.\\nSecond, we observe that the performance of re-implemented meth-\\nods on ActivityNet Captions is not sufficiently satisfying. Therefore,\\nit might be inadvisable to transplant fully supervised methods to\\nglance annotation setting by directly changing the annotation to a\\ninstant moment or a short time duration, especially for the dataset\\nlike ActivityNet Captions, which generally has a long video dura-\\ntion and a wide range of moment lengths.\\n4. As can be seen from Table 4, weakly supervised methods are\\noften not tested on TACoS dataset because the videos in TACoS are\\n\\n\\nQuery: A small child is seen standing on a base with an older man pointing.\\nGround Truth\\nPrediction\\nQ2V \\nAttention\\nVideo (1)\\n0.0s - 25.5s\\n0.0s - 24.7s\\nAttn_max=11.2s\\nVideo duration=124.2s\\nQuery: They put soap on their hands and scrub them together.\\nGround Truth\\nPrediction\\nQ2V \\nAttention\\nVideo (2)\\n7.6s - 75.0s\\n18.9s - 81.8s\\nGlance=1.2s,  Attn_max=11.2s\\nVideo duration=89.8s\\nAttn_max=59.3s\\nQuery: He gets down on the ground and flips around.\\nGround Truth\\nPrediction\\nQ2V \\nAttention\\nVideo (3)\\n43.1s - 55.5s\\n9.3s - 66.6s\\nGlance=1.2s,  Attn_max=11.2s\\nVideo duration=73.1s\\nAttn_max=24.0s\\nFigure 6: Some visualized examples from the test split of ActivityNet Captions. The first two examples are successful and the\\nthird is a failing case.\\nvery long and the moments to be retrieved are too short, i.e., the\\nrequirement of retrieval precision is very high. It might be hard for\\nexisting weakly supervised methods to deal with this situation. Our\\nproposed ViGA shows positive in such case with a similar result to\\nearly fully supervised methods, such as CTRL.\\n4.6\\nQualitative Analysis\\nFigure 6 shows some qualitative examples from the test split of\\nActivityNet Captions dataset, in which the green bar is the ground\\ntruth temporal boundary of the language query and the blue bar\\nrepresents the predicted boundary of ViGA. We also visualize the\\nquery-to-video attention (pink curve under the video flow) to il-\\nlustrate our proposed QAG-KL loss and query attention guided\\ninference. Video (1) and Video (2) are successfully retrieved sam-\\nples with high IoU. They show the effectiveness of our method\\nfrom two aspects. For video (1), the video duration is very long\\n(up to 124.2 seconds) and the moment to be retrieved is relatively\\nshort (25.5 seconds), which reveals that our proposed approach\\nbased on glance annotation can locate precisely when the video\\nsemantics is complex. As can be seen from the plot, this is benefited\\nfrom a reasonable query-to-video attention distribution which is\\nprecisely positioned in the correct moment interval. On one hand, it\\nenhances the cross-modal representation learning, and on the other\\nhand, it provides a good anchor frame for inference. For video (2),\\nwe observe that ViGA successfully retrieves this long moment of\\nnearly one minute. Given that we might be able to have good results\\nof retrieving long segments under single frame glance annotation,\\nit is reasonable to conjecture that the length of full annotation could\\nhave been reduced, even not to the extreme of just one single frame.\\nTherefore, our qualitative results are in favor of the great potential\\nof glance annotation. Inevitably, there are also failing cases. For\\nexample, in Video (3), the language query corresponds to a short\\nclip of the man gets down on the ground and flips around, but our\\nmodel recalls a long range segment containing the man, includ-\\ning a large part of the man standing, showing a lack of sufficient\\nunderstanding of the fine-grained textual semantics. We consider\\n\\n\\nthat this is the hard part in the task of retrieving video moments\\nwith free-form text query. There is not sufficiently large amount\\nof training data for learning fine-grained semantics because the\\nfree-form text query has great diversity. The model can be easily\\nconfused about some fine-grained actions, such as “get down on\\nthe ground and flip around” here.\\n5\\nLIMITATIONS\\nDue to our limited resource, we are only able to re-annotate the\\ndatasets in an automatic way by doing random sample in the time in-\\nterval of original annotation instead of manually annotating them.\\nAlthough we achieve good results in the previous experiments,\\nthere are some inevitable problems in this simple re-annotation\\nstrategy. For example, some queries might contain multiple seman-\\ntics, which are not possible to be captured by only one glance. Also,\\nin some rare cases, meaningless frames that would pollute the train-\\ning data such as camera scene changes might be sampled as the\\nglance, which could have been filtered out manually. We hope a\\nmanually annotated dataset in glance annotation could be collected\\nin the future to support follow-up research in this direction.\\n6\\nCONCLUSION\\nIn this paper, we study the problem of VMR. After analysing the\\nadvantages and limitations of the two existing VMR paradigms fully\\nsupervised VMR and weakly supervised VMR, we find that weakly\\nsupervised VMR can be augmented with trivial cost, and propose a\\nnew data annotation paradigm named as glance annotation. Under\\nglance annotation, we propose ViGA, a novel clip-level contrastive\\nlearning framework, as a pioneer method to solve this problem.\\nExtensive experiments are conducted on three publicly available\\ndatasets ActivityNet Captions, Charades-STA and TACoS, and ViGA\\noutperforms existing weakly supervised methods by a large margin.\\nTherefore, we conclude that glance annotation is a promising new\\ndata annotation paradigm for VMR, and ViGA is a feasible method\\nfor glance annotated VMR. Our results support further research\\nand applications of glance annotation in real-life problems.\\nACKNOWLEDGEMENT\\nThis research is conducted within the first two authors’ intern-\\nship in bilibili. We are grateful to bilibili AI for the support and\\ninspiration.\\nREFERENCES\\n[1] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell,\\nand Bryan Russell. 2017. Localizing moments in video with natural language. In\\nProceedings of the IEEE international conference on computer vision. 5803–5812.\\n[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normaliza-\\ntion. arXiv preprint arXiv:1607.06450 (2016).\\n[3] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.\\n2015. Activitynet: A large-scale video benchmark for human activity understand-\\ning. In Proceedings of the ieee conference on computer vision and pattern recognition.\\n961–970.\\n[4] Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new\\nmodel and the kinetics dataset. In proceedings of the IEEE Conference on Computer\\nVision and Pattern Recognition. 6299–6308.\\n[5] Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat-Seng Chua. 2018. Tem-\\nporally grounding natural sentence in video. In Proceedings of the 2018 conference\\non empirical methods in natural language processing. 162–171.\\n[6] Jingyuan Chen, Lin Ma, Xinpeng Chen, Zequn Jie, and Jiebo Luo. 2019. Localizing\\nnatural language in videos. In Proceedings of the AAAI Conference on Artificial\\nIntelligence, Vol. 33. 8175–8182.\\n[7] Shaoxiang Chen, Wenhao Jiang, Wei Liu, and Yu-Gang Jiang. 2020. Learning\\nmodality interaction for temporal sentence localization and event captioning in\\nvideos. In European Conference on Computer Vision. Springer, 333–351.\\n[8] Yi-Wen Chen, Yi-Hsuan Tsai, and Ming-Hsuan Yang. 2021. End-to-end Multi-\\nmodal Video Temporal Grounding. Advances in Neural Information Processing\\nSystems 34 (2021).\\n[9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT:\\nPre-training of Deep Bidirectional Transformers for Language Understanding.\\narXiv preprint arXiv:1810.04805 (2018).\\n[10] Xuguang Duan, Wenbing Huang, Chuang Gan, Jingdong Wang, Wenwu Zhu,\\nand Junzhou Huang. 2018. Weakly supervised dense event captioning in videos.\\narXiv preprint arXiv:1812.03849 (2018).\\n[11] Victor Escorcia, Fabian Caba Heilbron, Juan Carlos Niebles, and Bernard Ghanem.\\n2016. Daps: Deep action proposals for action understanding. In European Confer-\\nence on Computer Vision. Springer, 768–784.\\n[12] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. 2017. Tall: Temporal\\nactivity localization via language query. In Proceedings of the IEEE international\\nconference on computer vision. 5267–5275.\\n[13] Mingfei Gao, Larry S Davis, Richard Socher, and Caiming Xiong. 2019. Wslln:\\nWeakly supervised natural language localization networks.\\narXiv preprint\\narXiv:1909.00239 (2019).\\n[14] Runzhou Ge, Jiyang Gao, Kan Chen, and Ram Nevatia. 2019. Mac: Mining activity\\nconcepts for language-based temporal localization. In 2019 IEEE Winter Conference\\non Applications of Computer Vision (WACV). IEEE, 245–253.\\n[15] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momen-\\ntum contrast for unsupervised visual representation learning. In Proceedings of\\nthe IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9729–9738.\\n[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual\\nlearning for image recognition. In Proceedings of the IEEE conference on computer\\nvision and pattern recognition. 770–778.\\n[17] Jiabo Huang, Yang Liu, Shaogang Gong, and Hailin Jin. 2021. Cross-sentence\\ntemporal and semantic relations in video activity localisation. In Proceedings of\\nthe IEEE/CVF International Conference on Computer Vision. 7199–7208.\\n[18] Bin Jiang, Xin Huang, Chao Yang, and Junsong Yuan. 2019. Cross-modal video\\nmoment retrieval with spatial and language-temporal attention. In Proceedings\\nof the 2019 on international conference on multimedia retrieval. 217–225.\\n[19] Yifan Jiao, Zhetao Li, Shucheng Huang, Xiaoshan Yang, Bin Liu, and Tianzhu\\nZhang. 2018. Three-dimensional attention-based deep ranking model for video\\nhighlight detection. IEEE Transactions on Multimedia 20, 10 (2018), 2693–2705.\\n[20] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017.\\nDense-Captioning Events in Videos. In International Conference on Computer\\nVision (ICCV).\\n[21] Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. 2020. Tvr: A large-scale\\ndataset for video-subtitle moment retrieval. In Computer Vision–ECCV 2020: 16th\\nEuropean Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16.\\nSpringer, 447–463.\\n[22] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang. 2018.\\nBsn: Boundary sensitive network for temporal action proposal generation. In\\nProceedings of the European Conference on Computer Vision (ECCV). 3–19.\\n[23] Zhijie Lin, Zhou Zhao, Zhu Zhang, Qi Wang, and Huasheng Liu. 2020. Weakly-\\nsupervised video moment retrieval via semantic completion network. In Proceed-\\nings of the AAAI Conference on Artificial Intelligence, Vol. 34. 11539–11546.\\n[24] Meng Liu, Xiang Wang, Liqiang Nie, Qi Tian, Baoquan Chen, and Tat-Seng Chua.\\n2018. Cross-modal moment localization in videos. In Proceedings of the 26th ACM\\ninternational conference on Multimedia. 843–851.\\n[25] Xinfang Liu, Xiushan Nie, Zhifang Tan, Jie Guo, and Yilong Yin. 2021. A Survey\\non Natural Language Video Localization. arXiv preprint arXiv:2104.00234 (2021).\\n[26] Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization.\\narXiv preprint arXiv:1711.05101 (2017).\\n[27] Minuk Ma, Sunjae Yoon, Junyeong Kim, Youngjoon Lee, Sunghun Kang, and\\nChang D Yoo. 2020. Vlanet: Video-language alignment network for weakly-\\nsupervised video moment retrieval. In European Conference on Computer Vision.\\nSpringer, 156–171.\\n[28] Niluthpol Chowdhury Mithun, Sujoy Paul, and Amit K Roy-Chowdhury. 2019.\\nWeakly supervised video moment retrieval from text queries. In Proceedings of the\\nIEEE/CVF Conference on Computer Vision and Pattern Recognition. 11592–11601.\\n[29] Jonghwan Mun, Minsu Cho, and Bohyung Han. 2020. Local-global video-text\\ninteractions for temporal grounding. In Proceedings of the IEEE/CVF Conference\\non Computer Vision and Pattern Recognition. 10810–10819.\\n[30] Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve re-\\nstricted boltzmann machines. In Proceedings of the 27th International Conference\\non International Conference on Machine Learning. 807–814.\\n[31] Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove:\\nGlobal vectors for word representation. In Proceedings of the 2014 conference on\\nempirical methods in natural language processing (EMNLP). 1532–1543.\\n[32] Xiaoye Qu, Pengwei Tang, Zhikang Zou, Yu Cheng, Jianfeng Dong, Pan Zhou,\\nand Zichuan Xu. 2020. Fine-grained iterative attention network for temporal\\nlanguage localization in videos. In Proceedings of the 28th ACM International\\n\\n\\nConference on Multimedia. 4280–4288.\\n[33] Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt\\nSchiele, and Manfred Pinkal. 2013. Grounding Action Descriptions in Videos.\\nTransactions of the Association for Computational Linguistics (TACL) 1 (2013),\\n25–36.\\n[34] Marcus Rohrbach, Michaela Regneri, Mykhaylo Andriluka, Sikandar Amin, Man-\\nfred Pinkal, and Bernt Schiele. 2012. Script data for attribute-based recognition\\nof composite activities. In European conference on computer vision. Springer, 144–\\n157.\\n[35] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and\\nAbhinav Gupta. 2016. Hollywood in homes: Crowdsourcing data collection for\\nactivity understanding. In European Conference on Computer Vision. Springer,\\n510–526.\\n[36] Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks\\nfor large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).\\n[37] Yijun Song, Jingwen Wang, Lin Ma, Zhou Yu, and Jun Yu. 2020. Weakly-supervised\\nmulti-level attentional reconstruction network for grounding textual queries in\\nvideos. arXiv preprint arXiv:2003.07048 (2020).\\n[38] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan\\nSalakhutdinov. 2014. Dropout: a simple way to prevent neural networks from\\noverfitting. The journal of machine learning research 15, 1 (2014), 1929–1958.\\n[39] Reuben Tan, Huijuan Xu, Kate Saenko, and Bryan A Plummer. 2021. Logan: Latent\\ngraph co-attention network for weakly-supervised video moment retrieval. In\\nProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.\\n2083–2092.\\n[40] Du Tran, Lubomir D Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar\\nPaluri. 2014. C3D: generic features for video analysis. CoRR, abs/1412.0767 2, 7\\n(2014), 8.\\n[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,\\nAidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all\\nyou need. In Advances in neural information processing systems. 5998–6008.\\n[42] Hao Wang, Zheng-Jun Zha, Liang Li, Dong Liu, and Jiebo Luo. 2021. Structured\\nMulti-Level Interaction Network for Video Moment Localization via Language\\nQuery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern\\nRecognition. 7026–7035.\\n[43] Jingwen Wang, Lin Ma, and Wenhao Jiang. 2020. Temporally grounding language\\nqueries in videos by contextual boundary-aware prediction. In Proceedings of the\\nAAAI Conference on Artificial Intelligence, Vol. 34. 12168–12175.\\n[44] Jie Wu, Guanbin Li, Xiaoguang Han, and Liang Lin. 2020. Reinforcement Learning\\nfor Weakly Supervised Temporal Grounding of Natural Language in Untrimmed\\nVideos. In Proceedings of the 28th ACM International Conference on Multimedia.\\n1283–1291.\\n[45] Huijuan Xu, Kun He, Bryan A Plummer, Leonid Sigal, Stan Sclaroff, and Kate\\nSaenko. 2019. Multilevel language and vision integration for text-to-clip retrieval.\\nIn Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 9062–9069.\\n[46] Yitian Yuan, Tao Mei, and Wenwu Zhu. 2019. To find where you talk: Tempo-\\nral sentence localization in video with attention based location regression. In\\nProceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 9159–9166.\\n[47] Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang, and Larry S Davis. 2019.\\nMan: Moment alignment network for natural language moment retrieval via\\niterative graph adjustment. In Proceedings of the IEEE/CVF Conference on Computer\\nVision and Pattern Recognition. 1247–1257.\\n[48] Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. 2020. Span-based Lo-\\ncalizing Network for Natural Language Video Localization. In Proceedings of the\\n58th Annual Meeting of the Association for Computational Linguistics. Association\\nfor Computational Linguistics, Online, 6543–6554.\\nhttps://www.aclweb.org/\\nanthology/2020.acl-main.585\\n[49] Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo. 2020. Learning 2d\\ntemporal adjacent networks for moment localization with natural language. In\\nProceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 12870–12877.\\n[50] Zhu Zhang, Zhijie Lin, Zhou Zhao, and Zhenxin Xiao. 2019. Cross-modal inter-\\naction networks for query-based moment retrieval in videos. In Proceedings of\\nthe 42nd International ACM SIGIR Conference on Research and Development in\\nInformation Retrieval. 655–664.\\n[51] Zhu Zhang, Zhou Zhao, Zhijie Lin, Xiuqiang He, et al. 2020. Counterfactual\\ncontrastive learning for weakly-supervised vision-language grounding. Advances\\nin Neural Information Processing Systems 33 (2020), 18123–18134.\\n[52] Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua\\nLin. 2017. Temporal action detection with structured segment networks. In\\nProceedings of the IEEE International Conference on Computer Vision. 2914–2923.\\n\\n\\nD3G: Exploring Gaussian Prior for Temporal Sentence Grounding with\\nGlance Annotation\\nHanjun Li1, Xiujun Shu1, Sunan He2, Ruizhi Qiao1, Wei Wen1, Taian Guo1, Bei Gan1, Xing Sun1*\\n1Youtu Lab, Tencent\\n2Hong Kong University of Science and Technology\\n{hanjunli, xiujunshu, ruizhiqiao, jawnrwen, taianguo, stylegan, winfredsun}@tencent.com\\nsunan.he@connect.ust.hk\\nAbstract\\nTemporal sentence grounding (TSG) aims to locate a\\nspecific moment from an untrimmed video with a given nat-\\nural language query. Recently, weakly supervised methods\\nstill have a large performance gap compared to fully super-\\nvised ones, while the latter requires laborious timestamp\\nannotations. In this study, we aim to reduce the annotation\\ncost yet keep competitive performance for TSG task com-\\npared to fully supervised ones. To achieve this goal, we\\ninvestigate a recently proposed glance-supervised temporal\\nsentence grounding task, which requires only single frame\\nannotation (referred to as glance annotation) for each\\nquery. Under this setup, we propose a Dynamic Gaussian\\nprior based Grounding framework with Glance annotation\\n(D3G), which consists of a Semantic Alignment Group Con-\\ntrastive Learning module (SA-GCL) and a Dynamic Gaus-\\nsian prior Adjustment module (DGA). Specifically, SA-GCL\\nsamples reliable positive moments from a 2D temporal map\\nvia jointly leveraging Gaussian prior and semantic consis-\\ntency, which contributes to aligning the positive sentence-\\nmoment pairs in the joint embedding space. Moreover, to al-\\nleviate the annotation bias resulting from glance annotation\\nand model complex queries consisting of multiple events,\\nwe propose the DGA module, which adjusts the distribu-\\ntion dynamically to approximate the ground truth of tar-\\nget moments. Extensive experiments on three challenging\\nbenchmarks verify the effectiveness of the proposed D3G.\\nIt outperforms the state-of-the-art weakly supervised meth-\\nods by a large margin and narrows the performance gap\\ncompared to fully supervised methods. Code is available at\\nhttps://github.com/solicucu/D3G.\\n1. Introduction\\nTemporal sentence grounding is a fundamental prob-\\nlem in computer vision and receives an increasing atten-\\n*Corresponding author\\nQuery: person pours some water into a glass.\\n5.9 s    (8.0 s)   15.4 s\\n𝑔\\npositive pair\\nnegative pair\\nslide window\\nViGA\\nDuration: 21.96 s\\nDGA\\npositive pair\\nnegative pair\\nD3G\\n2d map\\n5.9 s    (8.0 s)   15.4 s\\n𝑔\\nFigure 1. Illustration of glance annotation g (red dashed line) and\\nsimple comparison between ViGA and D3G. The red rectangle\\nindicates the boundary of target moment.\\ntion in recent years.\\nGiven the query sentence and an\\nuntrimmed video, the goal of TSG is to localize the start and\\nend timestamps of specific moment that semantically corre-\\nsponds to the query. In recent years, full supervised tem-\\nporal sentence grounding (FS-TSG) has achieved tremen-\\ndous achievements [9, 1, 41, 43, 34, 29, 33, 42]. However,\\nobtaining accurate timestamps for each sentence is labor-\\nintensive and subjective, which prevents it from scaling to\\nlarge-scale video-sentence pairs and practical applications.\\nWeakly supervised temporal sentence grounding (WS-\\nTSG), which requires only the video and query pairs, re-\\nceives an increasing attention recently. Although great ad-\\nvances [19, 32, 12, 45, 43, 44] have been achieved in recent\\nyears, there still remains a huge performance gap between\\nWS-TSG and FS-TSG. WS-TSG suffers from severe local-\\nization issues due to the large discrepancy between video-\\nlevel annotations and clip-level task.\\nRecently, Cui et al. [6] propose a new annotating\\nparadigm called glance annotation for TSG, requiring the\\ntimestamp of only random single frame within the tempo-\\nral boundary of the target moment. It is noted that such\\nannotation only increases trivial annotating cost compared\\nto WS-TSG. Figure 1 illustrates the details of glance an-\\nnotation. With glance annotation, Cui et al. propose the\\narXiv:2308.04197v1  [cs.CV]  8 Aug 2023\\n\\n\\nViGA based on contrastive learning. ViGA first cuts the in-\\nput video into clips of fixed length, which are assigned with\\nGaussian weights generated according to the glance anno-\\ntation, and contrasts clips with queries. There are two ob-\\nvious disadvantages in this way. First, moments of interest\\nusually have various durations. Therefore, these clips can-\\nnot cover a wide range of target moments, which inevitably\\naligns the sentence with incomplete moment and obtains\\nsub-optimal performance. Second, ViGA utilizes a fixed\\nscale Gaussian distribution centered at the glance frame to\\ndescribe the span of each annotated moment. However, the\\nglance annotations are not guaranteed at the center of target\\nmoments, which results in annotation bias as shown in Fig-\\nure 2. Besides, since some complex query sentences consist\\nof multiple events, a single Gaussian distribution is hard to\\ncover all events at the same time as shown in Figure 3. To\\naddress the aforementioned defects and fully unleash the\\npotential of Gaussian prior knowledge with the low-cost\\nglance annotation, we propose a Dynamic Gaussian prior\\nbased Grounding framework with Glance annotation (D3G)\\nas shown in Figure 4.\\nWe first generate a wide range of candidate moments fol-\\nlowing 2D-TAN [43]. Afterwards, we propose a Semantic\\nAlignment Group Contrastive Learning module (SA-GCL)\\nto align the positive sentence-moment pairs in the joint em-\\nbedding space. Specifically, for each query sentence, we\\nsample a group of positive moments according to calibrated\\nGausssian prior and minimize the distances between these\\nmoments and the query sentence. In this way, it tends to\\ngradually mine the moments which have increasing over-\\nlap with the ground truth. Moreover, we propose a Dy-\\nnamic Gaussian prior Adjustment module (DGA), which\\nfurther alleviates annotation bias and approximates the span\\nof complex moments consisting of multiple events. Specif-\\nically, we adopt multiple Gaussian distributions to describe\\nthe weight distributions of moments. Therefore, the weight\\ndistributions for various moments can be flexibly adjusted\\nand gradually approach to the ground truth. Our contribu-\\ntions are summarized as follows:\\n• We propose a Dynamic Gaussian prior based Ground-\\ning framework with Glance annotation (D3G), which\\nfacilitates the development of temporal sentence\\ngrounding with lower annotated cost.\\n• We propose a Semantic Alignment Group Contrastive\\nLearning module to align the features of the positive\\nsentence-moment pairs and a Dynamic Gaussian prior\\nAdjustment module to ease the annotation bias and\\nmodel the distributions of complex moments.\\n• Extensive experiments demonstrate that D3G obtains\\nconsistent and significant gains compared to method\\nunder the same annotating paradigm and outperforms\\nweakly supervised methods by a large margin.\\n𝑤!\\n𝑤\\\"\\n𝑤#\\n𝑤!\\n,\\n𝑤\\\"\\n,\\n𝑤#\\n,\\ntarget moment\\ntarget moment\\n𝑔\\n𝑔\\nFigure 2. Illustration of annotation bias and Gaussian prior after\\ndynamic adjustment. Top: the target moment is assigned with a\\nlow weight wm due to the bias of glance annotation according to\\nViGA, which we call annotation bias. Bottom: a reasonable Gaus-\\nsian distribution is obtained via DGA described in Section 3.3.\\n2. Related Work\\nFull Supervised Temporal Sentence Grounding. The FS-\\nTSG methods can be categorized into two groups. Two-\\nstage methods [1, 9, 11, 13, 14, 35] first propose candi-\\ndate segments in a video through sliding window or pro-\\nposal generation. A cross-modal matching network is then\\nemployed to find the best matching clip. However, these\\npropose-and-match paradigms are time-consuming due to\\nthe numerous candidates. To reduce the redundant com-\\nputation, some researchers proposed single-stage meth-\\nods [2, 3, 40, 41, 30, 20, 43, 21, 39, 37]. 2D-TAN [43]\\nconstructs 2D feature map to model the temporal relations\\nof video segment. Recently, Wang et al. [33] propose a Mu-\\ntual Matching Network based on 2D-TAN, and further im-\\nprove the performance via exploiting both intra- and inter-\\nvideo negative samples. Although fully supervised meth-\\nods achieve satisfying performance, they are highly depen-\\ndent on accurate timestamp annotations. It is highly time-\\nconsuming and laborious to obtain these annotations for\\nlarge-scale video-sentence pairs.\\nWeakly Supervised Temporal Sentence Grounding.\\nSpecifically,\\nWS-TSG methods can be grouped into\\nreconstruction-based methods [8, 18, 26, 4] and multi-\\ninstance learning (MIL) methods [19, 10, 5, 38, 12, 27].\\nSCN [18] employs a semantic completion network to re-\\ncover the masked words in the query sentence with the gen-\\nerated proposals, which provides feedback for facilitating\\nfinal predictions. To further exploit the negative samples in\\nMIL-based methods, CNM [44] and CPL [45] propose to\\ngenerate proposals with Gaussian functions and introduce\\nintra-video contrastive learning. WS-TSG methods indeed\\nadvance with low annotation cost, however, there still re-\\nmains a large performance gap compared to FS-TSG meth-\\nods due to the discrepancy between video-level annotations\\nand clip-level task.\\nGlance Supervised Temporal Sentence Grounding. Re-\\n\\n\\n…\\n…\\nA person walks on the roof of a climbing equipment\\nthen he goes down the wall \\nand reach the floor and walk forward.\\ng\\nFigure 3. Illustration of complex query consists of multiple events.\\nNote that g indicates the position of glance annotation. The ac-\\ncording Gaussian distribution (red curves) is hard to cover the\\nwhole target moments. We utilize DGA module to mine multiple\\nlatent Gaussian distributions (dashed line) to model such query.\\ncently, ViGA [6] proposes glance supervised TSG (GS-\\nTSG) task with a new annotating paradigm.\\nViGA uti-\\nlizes a Gaussian function to model the relevance of differ-\\nent clips with target moment and contrasts the clips with the\\nqueries. Though ViGA achieves promising performance, it\\nstill suffers from two limitations as mentioned in Introduc-\\ntion. Concurrently, Xu et al. [36] propose the similar task\\ncalled PS-VTG, and generate pseudo segment-level labels\\nbased on language activation sequences. To better explore\\nthe Gaussian prior for TSG task with glance annotation, we\\npropose a simple yet effective D3G, which achieves com-\\npetitive performance compared with both WS-TSG and FS-\\nTSG methods. Concurrent with our work, Ju et al. [15]\\npropose a robust partial-full union framework (PFU) and\\nachieve excellent performance with glance annotation or\\nshort-clip labels.\\n3. Proposed Method\\n3.1. Overview\\nGiven an untrimmed video V and query sentence S, the\\ntemporal sentence grounding task aims to determine the\\nstart timestamp ts and end timestamp te, where the mo-\\nment Vts:te best semantically corresponds to the query. As\\nfor FS-TSG, the exact timestamps (ts,te) of corresponding\\nmoment is provided given a query description. In contrast,\\nCui et al. [6] propose a new low-cost annotating paradigm\\ncalled glance annotation, which requires only single times-\\ntamp g, satisfying g ∈[ts, te]. Following the setting of [6],\\nwe propose a Dynamic Gaussian prior based Grounding\\nframework with Glance annotation (D3G) to fully unleash\\nthe potential of glance annotations.\\nOur D3G adopts the network architecture similar to\\n[43, 33].\\nGiven an untrimmed video, we firstly encode\\nthe video into feature vectors with pre-trained 2D or 3D\\nconvolutional network [25, 28] and segment the video fea-\\ntures into N video clips. Specifically, we apply average\\npooling to each clip to obtain clip-level features V\\n=\\n{f v\\n1 , f v\\n2 , ..., f v\\nN} ∈RN×Dv. These clip features are then\\npassed through an FC layer to reduce their dimension, de-\\nnoted as F 1d ∈RN×dv. Afterwards, we encode them as\\n2D temporal feature map ˆ\\nF ∈RN×N×dv following 2D-\\nTAN [43] with the max pooling. As for language encoder,\\nwe choose DistilBERT [23] to obtain sentence-level feature\\nˆ\\nf s ∈Rds following [33]. Finally, to estimate the matching\\nscores of candidate moments and the query, we utilize a lin-\\near projection layer to project the textual and visual features\\ninto same dimension d, respectively. The final representa-\\ntion of sentence is f s ∈Rd and the features of all moments\\nare F ∈RN×N×d. The final matching scores are given by\\nthe cosine similarity between f s and elements of F.\\n3.2. Semantic Alignment Group Contrastive Learn-\\ning\\nIn this section, we aim to mine the moment which most\\nsemantically corresponds to the query and maximize the\\nsimilarity between them.\\nTo achieve this goal, we have\\ntwo crucial steps. First, we generate abundant candidate\\nmoments following 2D-TAN and assign them with reli-\\nable Gaussian prior weights generated with the guidance of\\nglance annotation. Second, we propose a semantic align-\\nment group contrastive learning to align a group of positive\\nmoments with corresponding query sentence.\\nTo be specific, given the encoded video features F 1d ∈\\nRN×dv and glance annotation g, we also utilize a Gaussian\\nfunction parameterized with (µ, σ) to model the relations\\nbetween frames and target moment, where the µ is deter-\\nmined by the glance g. We first scale the sequence indices\\nI ∈{1, 2, ..., N} into domain [−1, 1] by a linear transfor-\\nmation as follows:\\nh(i) = 2· i −1\\nN −1 −1.\\n(1)\\nGiven the index i, we can obtain corresponding Gaussian\\nweight via Eq. (2).\\nG(i, µ, σ) = Norm(\\n1\\n√\\n2πσ exp(−(h(i) −h(µ))2\\n2σ2\\n)), (2)\\nwhere µ ∈I and σ is a hyperparameter, and Norm(·) is a\\nfunction used to scale values into range [0, 1].\\nDifferent from ViGA [6], we utilize the characteristic\\nof 2D-TAN to generate a wide range of candidate mo-\\nments with various durations.\\nGiven the video features\\nF 1d ∈RN×dv, we encode them into 2D feature map\\nF ∈RN×N×d as shown in Figure 4, where Fij denotes\\nthe feature of moment that starts at position i and ends at\\nposition j. Note that the moment is valid only when i ≤j.\\nWe then propose a triplet-sample strategy to generate more\\nreasonable weights for candidate moments instead of only\\nsampling the weight at middle point as in [6]. Specifically,\\n\\n\\n𝑤!\\\"\\nupdate\\npull\\npush\\nintra-video positive\\nSA-GCL\\nDGA\\nVideo\\nfeature \\nextractor\\nmax pooling\\n& conv & proj\\nQuery: person pours \\nsome water into a glass.\\nDistillBERT\\nintra-video negative\\ninter-video negative\\ntext feature\\n𝐹∈𝑅#×#×%\\n𝑓& ∈𝑅%\\n𝐹'% ∈𝑅#×%!\\n𝑓&\\n𝐹(\\n𝐹)\\nproj\\n𝑤!\\\"\\n𝑠!\\\"\\n𝑔\\nFigure 4. The overview of proposed D3G, which consists of Semantic Alignment Group Contrastive Learning (SA-GCL) and Dynamic\\nGaussain prior Adjustment (DGA). Note that g indicates the position of glance annotation and the grids with dashed line in F are invalid\\ncandidate moments. “proj” denotes the linear projection layer. “intra/inter” indicate the positive or negative moments sampled from\\nsame/different videos.\\nfor each moment with start position i and end position j, we\\ncompute its Gaussian prior weight as follows:\\nwij = 1\\n3· (G(i, g, σ)+G(j, g, σ)+G(⌊i + j\\n2\\n⌋, g, σ)), (3)\\nwhere g is glance annotation for current target moment. In\\nthis way, those moments containing target moment but hav-\\ning longer durations will be penalized with lower weights.\\nTo remedy the annotation bias, we additionally intro-\\nduce semantic consistency prior to calibrate the Gaussian\\nprior weight wij for each candidate moment. Given the\\nquery features f s ∈Rd and the features F ∈RN×N×d of\\ncandidate moments, we compute their semantic consistency\\nscores via Eq. (4).\\nsij =\\nf s · Fij\\n∥f s ∥∥Fij ∥,\\n(4)\\nwhere ∥· ∥is l2-norm. Afterwards, we rectify the Gaussian\\nweight wij with semantic consistency score sij via multi-\\nplication to obtain new prior weight pij = wij· sij.\\nThe objective of Temporal Sentence Grounding is to\\nlearn a cross-modal embedding space, where the query sen-\\ntence feature should be well aligned with the feature of\\ncorresponding moment and far way from those of irrele-\\nvant video moments. Motivated by [31, 17], we propose\\na Semantic Alignment Group Contrastive Learning module\\n(SA-GCL) to gradually mine candidate moments most se-\\nmantically aligned with given query sentence. To be spe-\\ncific, we first sample top-k candidate moments from F as\\npositive keys for query f s according to the new prior pij,\\ndenoted as F p = {Fij|1 ≤i ≤j ≤N} ∈Rk×d. Simulta-\\nneously, we sample Gaussian weights of corresponding mo-\\nments denoted as W p = {wij|1 ≤i ≤j ≤N} ∈Rk. We\\nthen gather other candidate moments which do not contain\\nthe glance g from intra-video and all candidate moments\\nfrom other videos within same batch as negative keys, de-\\nnoted as F n = {Fij|1 ≤i ≤j ≤N} ∈RNn×d, where Nn\\ndenotes the number of negative moments. The objective of\\nSA-GCL can be described as follows:\\nLalign = −1\\nk\\nk\\nX\\nz=0\\nW p\\nz log exp(f s · F p\\nz /τ)\\nSUM\\n,\\nSUM =\\nk\\nX\\nz=0\\nexp(f s · F p\\nz /τ) +\\nNn\\nX\\nz=0\\nexp(f s · F n\\nz /τ),\\n(5)\\nwhere τ is the temperature scaling factor. SA-GCL aims to\\nmaximize the similarity between the query f s and a group\\nof corresponding positive moments F p under the joint em-\\nbedding space while pushing away negative pairs. Note that\\n\\n\\ndifferent positive moments are assigned with corresponding\\nprior weight W p\\nz . In this way, SA-GCL effectively avoids\\nbeing dominated by inaccurate moments with less similar-\\nity and tends to mine the candidate moments having large\\noverlap with the target moment.\\n3.3. Dynamic Gaussian prior Adjustment\\nTo further ease the annotation bias and characterize com-\\nplex target moments, we propose a novel Dynamic Gaus-\\nsian prior Adjustment module (DGA). Specifically, we uti-\\nlize multiple Gaussian functions with different centers to\\nmodel the local distributions of target moment and aggre-\\ngate them to approximate the distribution of target moment.\\nGiven the video features F 1d ∈RN×dv and annotation\\nglance g, we compute the relevance of other position i with\\nposition g via Eq. (6).\\nrgi =\\nF 1d\\ng\\n· F 1d\\ni\\n∥F 1d\\ng\\n∥∥F 1d\\ni\\n∥.\\n(6)\\n¯\\nrgi = (1 −α)¯\\nrgi + αrgi.\\n(7)\\nTo make the relevance scores more stable, we update\\n¯\\nrgi with momentum factor α as shown in Eq. (7), where\\n¯\\nrgi = rgi at first training epoch. According to the relevance\\n{¯\\nrgi}, we can mine latent local centers for target moment.\\nSpecifically, we utilize a specific threshold Tr to filter the\\ncandidate positions and obtain a mask Mg ∈{0, 1}N for\\nglance g as follows:\\nM i\\ng =\\n(\\n1,\\nif ¯\\nrgi ≥Tr\\n0,\\notherwise\\n(8)\\nWith the mask of latent local centers, we then adjust the\\nGaussian prior dynamically via Eq. (9).\\nˆ\\nG(i, g, σ) = 1\\nC\\nN\\nX\\nz=1\\nM z\\ng · ¯\\nrgi · G(i, z, σ),\\n(9)\\nwhere C is the summation of mask Mg. Afterwards, we\\nreplace the G(i, g, σ) in Eq. (3) with ˆ\\nG(i, g, σ), and nat-\\nurally obtain dynamic Gaussian prior weight during train-\\ning. Compared to ViGA, our dynamic Gaussian prior is\\nmore flexible and able to adjust the center of Gaussian dis-\\ntribution adaptively. Therefore, DGA further alleviates the\\nannotation bias and provides more reliable prior weights.\\nBesides, multiple Gaussian distributions are well suited for\\nmodeling complex target moments consisting of multiple\\nevents as shown in Figure 3. DGA tends to widen the re-\\ngion of high Gaussian weight via self-mining neighboring\\nframes based on the feature of glance g and gradually gen-\\nerates the Gaussian prior weight well aligned with target\\nmoment. In this way, SA-GCL will be provided with pos-\\nitive moments of high quality, which eventually promotes\\nthe cross-modal semantic alignment learning and accurate\\nlocalization of target moments.\\nDiscussion. To clearly distinguish the differences between\\nD3G and few similar works, we give some explanations\\nhere. As for MMN, D3G shares the same process of gener-\\nating candidate moments following 2D-TAN, which is not\\nthe key contribution of our method.\\nMMN utilizes nor-\\nmal one-to-one contrastive learning is no longer suitable\\nto glance annotation. However, D3G instead adopts a suit-\\nable sample strategy and corresponding adapted group con-\\ntrastive learning, which is key component to unleash the po-\\ntential of glance annotations. As for CPL, we also know that\\nit utilizes multiple Gaussian distribution to describe positive\\nmoments. However, it actually select one most matched\\npositive moment guided by the loss of masked language\\nreconstruction for contrastive learning, while D3G utilizes\\nmultiple Gaussian functions to adaptively model complex\\nqueries consisting of multiple events and samples a group\\nof positive moments for contrastive learning.\\n4. Experiments\\nIn order to validate the effectiveness of the proposed\\nD3G, we conduct extensive experiments on three publicly\\navailable datasets: Charades-STA [9], TACoS [9] and Ac-\\ntivityNet Captions [16].\\n4.1. Datasets\\nCharades-STA is built on dataset Charades [24] for tem-\\nporal sentence grounding.\\nIt contains 12,408 and 3,720\\nmoment-sentence pairs for training and testing.\\nTACoS consists of 127 videos selected from the MPII\\nCooking Composite Activities video corpus [22]. We fol-\\nlow the standard split from [9], which contains 10,146,\\n4,589 and 4,083 moment-sentence pairs for training, vali-\\ndation and testing, respectively. We report the evaluation\\nresult on the test set for fair comparison.\\nActivityNet Captions is originally designed for video cap-\\ntioning and recently introduced into temporal sentence\\ngrounding. It contains 37,417, 17,505 and 17,031 moment-\\nsentence pairs for training, validation and testing, respec-\\ntively. We report the evaluation result following [43, 33].\\nSpecially, we adopt the glance annotation released by [6]\\nfor training set, where the temporal boundary is replaced\\nwith the timestamp g uniformly sampled within the original\\ntemporal boundary.\\n4.2. Evaluation Metric and Implementation Details\\nEvaluation Metric.\\nFollowing previous works [9, 43],\\nwe evaluate our model with metric ‘R@n,IoU=m’, which\\nmeans the percentage of at least one of the top-n results\\nhaving Intersection over Union (IoU) larger than m. Specif-\\nically, we report the results with m\\n∈\\n{0.5, 0.7} for\\n\\n\\nCharades-STA, m ∈{0.3, 0.5, 0.7} for TACoS and Activi-\\ntyNet Captions, and n ∈{1, 5} for all datasets.\\nImplementation Details. In this work, our main frame-\\nwork is extended from MMN [33] and most of experiment\\nsettings keep the same. For fair comparison, following [33],\\nwe adopt off-the-shelf video features for all datasets (VGG\\nfeature for Charades and C3D feature for TACoS and Ac-\\ntivityNet Captions). Specifically, the dimension of joint fea-\\nture space d is set to 256 and τ is set to 0.1. In SA-GCL, we\\nset the k as 10, 20 and 20 for Charades, TACoS and Activ-\\nityNet Captions, respectively. The σ in Eq. (2) is set to 0.3,\\n0.2 and 0.6 for Charades, TACoS and ActivityNet Captions.\\nIn DGA, Tr and α is set as 0.9 and 0.7, respectively.\\n4.3. Comparisons with the State-Of-The-Art\\nIn order to provide comprehensive analysis, we com-\\npare the proposed D3G with both fully/weakly/glance su-\\npervised methods. As shown in Table 1, Table 2 and Ta-\\nble 3, D3G achieves highly competitive results on three\\ndatasets under glance supervision, and achieves compara-\\nble performance compared with fully supervised methods.\\nNote that we highlight the best value for each setting re-\\nspectively. Based on the experimental results, we can draw\\nthe following conclusions:\\n(1) Glance annotation provides more potential to achieve\\nbetter performance for temporal sentence grounding with\\nlower annotation cost. Although it is not entirely fair to\\ndirectly compare D3G with other weakly supervised meth-\\nods due to introducing extra supervision, D3G significantly\\nexceeds most of weakly supervised methods by a large mar-\\ngin with trivial increment of annotation cost. Since PS-VTG\\nand PFU adopt more robust I3D feature, they obviously out-\\nperform D3G on Charades-STA. However, D3G instead is\\nsuperior to PS-VTG on more challenging TACoS with same\\nfeatures. Besides, weak supervised methods are often not\\ntested on TACoS, where the videos are very long and con-\\ntain a large number of target moments. However, D3G ob-\\ntains promising performance and outperforms ViGA by a\\nlarge margin on TACoS as shown in Table 2.\\n(2) D3G effectively exploits the information provided by\\nglance annotation and mines more moments of high qual-\\nity for training compared with ViGA. Due to the limitations\\nof fixed scale Gaussian function and fixed sliding window,\\nViGA fails to mine accurate candidate moments to learn\\na well-aligned joint embedding space. Instead, D3G gen-\\nerates a wide range of candidate moments and samples a\\ngroup of reliable candidate moments for group contrastive\\nlearning. Compared to ViGA, D3G achieves obvious gains\\n5.08% and 3.5% at R@1 IoU=0.5 and R@1 IoU=0.7 on\\nCharades-STA, respectively. Specially, significant improve-\\nments are obtained at R@5 on three datasets.\\n(3) D3G substantially narrows the performance gap be-\\ntween weakly/glance supervised methods and fully super-\\nMethod\\nR@1\\nR@5\\nIoU=0.5\\nIoU=0.7\\nIoU=0.5\\nIoU=0.7\\nMAN [41]\\n41.21\\n20.54\\n83.21\\n51.85\\n2D-TAN [43]\\n39.70\\n23.31\\n80.32\\n51.26\\nSSCS [7]\\n43.15\\n25.54\\n84.26\\n54.17\\nMMN [33]\\n47.31\\n27.28\\n83.74\\n58.41\\nCRM [12]\\n34.76\\n16.37\\n-\\n-\\nCNM [44]\\n35.43\\n15.45\\n-\\n-\\nLCNet [38]\\n39.19\\n18.87\\n80.56\\n45.24\\nCPL† [45]\\n32.27\\n14.22\\n78.34\\n43.45\\nPS-VTG‡ [36]\\n39.22\\n20.17\\n-\\n-\\nPFU‡ [15]\\n54.66\\n28.34\\n-\\n-\\nVIGA∗[6]\\n36.56\\n16.10\\n48.90\\n25.86\\nD3G\\n41.64\\n19.60\\n79.25\\n49.30\\nTable 1. Performance comparison on Charades-STA under differ-\\nent supervision settings.Top:full supervision, Middle: weak super-\\nvision, Bottom:glance supervision. †we reproduce the results with\\nofficial code and VGG features for fair comparison.\\n∗we repro-\\nduce the results with official code for results at R@5.‡ indicates\\nthe method utilizes I3D features.\\nMethod\\nR@1\\nR@5\\nIoU=0.3\\nIoU=0.5\\nIoU=0.7\\nIoU=0.3\\nIoU=0.5\\nIoU=0.7\\nCTRL [9]\\n18.32\\n13.30\\n-\\n36.69\\n25.42\\n-\\n2D-TAN [43]\\n37.29\\n25.32\\n-\\n57.81\\n24.04\\n-\\nSSCS [7]\\n41.33\\n29.56\\n-\\n60.65\\n48.01\\n-\\nMMN [33]\\n38.57\\n27.24\\n-\\n65.31\\n50.69\\n-\\nMAT [42]\\n48.79\\n37.57\\n-\\n67.63\\n57.91\\n-\\nVIGA∗[6]\\n20.82\\n9.52\\n3.10\\n27.92\\n15.35\\n6.10\\nPS-VTG [36]\\n23.64\\n10.00\\n3.35\\n-\\n-\\n-\\nD3G\\n27.27\\n12.67\\n4.70\\n54.61\\n31.34\\n12.35\\nTable 2. Performance comparison on TACoS under different super-\\nvision settings.Top:full supervision, Bottom:glance supervision.\\n∗we reproduce the results with official code for results at R@5.\\nvised methods. Specifically, D3G already surpasses previ-\\nous method (e.g., CTRL) on both TACoS and ActivityNet\\nCaptions. Undeniably, there are still non-negligible margin\\ncompared to the state-of-the-art fully supervised methods\\n(e.g., MMN). Note that D3G is very concise and not embed-\\nded with auxiliary module (e.g., MLM used in [45]). D3G\\nstill can be enhanced with some complementary modules.\\n4.4. Ablation Study\\nTo validate the effectiveness of different components of\\nthe proposed D3G and investigate the impact of hyper-\\nparameters, we perform ablation studies on Charades-STA.\\nEffectiveness of SA-GCL and DGA. Since Lalign is the\\nonly loss of D3G, to validate the effectiveness of SA-GCL,\\nwe need to simplify the SA-GCL module as a baseline.\\nSpecifically, we only sample the top-1 positive moment to\\n\\n\\n(a) Effect of top-k.\\n(b) Effect of σ.\\n(c) Effect of Tr\\nFigure 5. Effect of different hyper-parameters on Charades-STA dataset.\\nMethod\\nR@1\\nR@5\\nIoU=0.3\\nIoU=0.5\\nIoU=0.7\\nIoU=0.3\\nIoU=0.5\\nIoU=0.7\\nCTRL [9]\\n47.43\\n29.01\\n10.34\\n75.32\\n59.17\\n37.54\\n2D-TAN [43]\\n59.46\\n44.51\\n26.54\\n85.53\\n77.13\\n61.96\\nLGI [20]\\n58.52\\n41.51\\n23.07\\n-\\n-\\n-\\nSSCS [7]\\n61.35\\n46.67\\n27.56\\n86.89\\n78.37\\n63.78\\nMMN [33]\\n65.05\\n48.59\\n29.26\\n87.25\\n79.50\\n64.76\\nMAT [42]\\n-\\n48.02\\n31.78\\n-\\n78.02\\n63.18\\nCRM [12]\\n55.26\\n32.19\\n-\\n-\\n-\\n-\\nCNM [44]\\n55.68\\n33.33\\n-\\n-\\n-\\n-\\nLCNet [38]\\n48.49\\n26.33\\n-\\n82.51\\n62.66\\n-\\nCPL [45]\\n53.67\\n31.24\\n-\\n63.05\\n43.14\\n-\\nVIGA∗[6]\\n59.78\\n35.39\\n16.25\\n72.19\\n53.19\\n32.69\\nPS-VTG [36]\\n59.71\\n39.59\\n21.98\\n-\\n-\\n-\\nPFU [15]\\n59.63\\n36.35\\n16.61\\n-\\n-\\n-\\nD3G\\n58.25\\n36.68\\n18.54\\n87.84\\n74.21\\n52.47\\nTable 3. Performance comparison on ActivityNet Captions under\\ndifferent supervision settings.Top:full supervision, Middle: weak\\nsupervision, Bottom:glance supervision. ∗we reproduce the results\\nwith official code for results at R@5.\\ncompute the normal contrastive loss (degraded to simpli-\\nfied MMN) as shown in the first row of Table 4. However,\\nthe top-1 moment tends to be the shortest moment and has\\nsmall overlap with target moment, which is decided by the\\nintrinsic characteristic of 2D-TAN. Therefore, the perfor-\\nmance of baseline is undoubtedly very poor, which demon-\\nstrates that the main improvement of D3G is not brought\\nby the backbone of MMN. This phenomenon then encour-\\nages us to sample a group of positive moments in SA-GCL.\\nWith full SA-GCL, the model obtains notable performance\\ngains. Moreover, we introduce the DGA to alleviate annota-\\ntion bias and model some complex target moments consist-\\ning of multiple events. After equipped with DGA, D3G and\\nsimplified D3G achieve obvious performance improvement.\\nImpact of Sampling Strategy. In SA-GCL, sampling a\\ngroup of reliable positive moments is of great importance.\\nWe investigate the impacts of two priors: Gaussian weight\\nand semantic consistency, respectively. As shown in the first\\nrow of Table 5, we sample top-k positive moments accord-\\nModule\\nR@1\\nR@5\\nSA-GCL DGA IoU=0.5 IoU=0.7 IoU=0.5 IoU=0.7\\n✓†\\n5.08\\n0.81\\n14.78\\n3.36\\n✓†\\n✓\\n13.92\\n3.31\\n33.55\\n11.77\\n✓\\n40.51\\n16.10\\n74.41\\n43.31\\n✓\\n✓\\n41.64\\n19.60\\n79.25\\n49.30\\nTable 4. Effectiveness of SA-GCL and DGA in D3G on Charades-\\nSTA. ✓† denotes an simplified implementation of SA-GCL.\\nTypes\\nR@1\\nR@5\\nGW\\nSC\\nIoU=0.5\\nIoU=0.7\\nIoU=0.5\\nIoU=0.7\\n✓\\n38.09\\n16.10\\n66.53\\n36.51\\n✓\\n25.67\\n9.57\\n65.43\\n38.52\\n✓\\n✓\\n40.51\\n16.10\\n74.41\\n43.31\\nTable 5. Impact of different strategies used to sample positive mo-\\nments for SA-GCL on Charades-STA. GW: Gaussian weight, SC:\\nsemantic consistency.\\ning to the Gaussian prior weight. An alternative scheme is\\nthat we sample top-k positive moments according to the se-\\nmantic consistency scores between candidate moments and\\nquery sentence. However, both of them obtain sub-optimal\\nperformance. This is because Gaussian prior weight is not\\nalways reliable due to the annotation bias and semantic con-\\nsistence scores are highly dependent on the stability of fea-\\ntures. Therefore, we finally fuse these two priors to obtain\\nrelatively reliable prior. As shown in the third row of Ta-\\nble 5, obvious performance gains are obtained after both of\\nthem are utilized, which demonstrates that these two priors\\nindeed complement each others.\\nEffect of different hyper-parameters. As shown in Fig-\\nure 5, we investigate three critical hyperparameters in D3G.\\nAs verified in Table 4, sampling enough latent positive mo-\\nments is beneficial to mining target moment for training. As\\nshown in Figure 5 (a), the performance gains increase ob-\\nviously as the k increases. However, it begins to decrease\\n\\n\\nMethod\\nR@1\\nR@5\\nIoU=0.5\\nIoU=0.7\\nIoU=0.5\\nIoU=0.7\\nViGA\\n36.56\\n16.10\\n48.90\\n25.86\\nD3G\\n41.64\\n19.60\\n79.25\\n49.30\\nViGA+ 33.66(−2.90) 14.65(−1.45) 47.45(−1.45) 25.51(−0.35)\\nD3G+\\n40.19(−1.45) 19.62(+0.02) 78.90(−0.35) 49.41(+0.11)\\nTable 6. Performance comparison on Charades-STA with extreme\\nglance annotation.\\n+ indicates according method is trained with\\nextreme glance annotations.\\nafter the k reaches a specific value. We argue that selecting\\nexcessive positive moments tends to incorporate some false\\npositive moments and therefore degrades the performance.\\nWe finally set the k to 10 for Charades-STA, which bal-\\nances well the performance and computational cost. As for\\nhyperparameter σ, it essentially decides the width of Gaus-\\nsian distribution. A larger σ can well characterize the target\\nmoment of longer duration and vice versa. We vary the σ\\nfrom 0.1 to 0.5, and observe that value 0.3 is relatively suit-\\nable for the Charades-STA dataset. As for hyperparameter\\nTr in Eq. (8), it controls the degree of dynamic Gaussian\\nprior adjustment. We conduct experiments with relevance\\nthresholds around 0.9. A small threshold tends to introduce\\ninterference while a large threshold fails to find the neigh-\\nbor frames with consistent semantic. As shown in Figure 5\\n(c), the moderate threshold 0.9 relatively balances the afore-\\nmentioned dilemma.\\nTolerance to Extreme Glance Annotation. In order to\\nverify the ability of addressing extreme glance annotation,\\nwe first generate extreme glance annotation, where only the\\npositions near the start/end timestamps will be sampled as\\nglance g. As shown in Table 6, both ViGA+ and D3G+\\nare confronted with the performance degradation at some\\nmetrics(e.g., R@1 IoU=0.5). However, the performances of\\nD3G are relatively stable compared to ViGA, which demon-\\nstrate that D3G indeed is able to alleviate annotation bias.\\n4.5. Qualitative Analysis\\nTo clearly reveal the effectiveness of our method, we\\nvisualize some qualitative examples from the test split of\\nCharades-STA dataset and ActivityNet Captions dataset. As\\nshown in Figure 6, the proposed D3G achieves more ac-\\ncurate localization of target moment compared to ViGA.\\nSpecifically, ViGA cannot well align the visual content\\nand semantic information and tend to be disturbed by ir-\\nrelevant content, which may be caused by the annotation\\nbias. Instead, D3G utilizes SA-GCL and DGA to alleviate\\nthe annotation bias, which enables D3G to well align the\\nquery with the corresponding moment. Moreover, the DGA\\nadopts multiple Gaussian functions to model target mo-\\nment, which is beneficial to representing the complete dis-\\ntribution of complex moments consisting of multiple events.\\nAs shown in Figure 6 (b), D3G still effectively localizes the\\n15.56 s – 29.39 s\\n21.61 s – 34.58 s\\nQuery: person starts playing games on a laptop.                  \\nduration: 34.58 s             \\nGT\\nViGA\\nD3G\\n21.9 s – 34.58 s\\n(a)\\nQuery: The men cook together while mixing in more \\ningredients and presenting it on a plate.\\nduration: 121.84 s       \\nGT\\nViGA\\nD3G\\n77.37 s – 121.84 s\\n17.06 s – 102.35 s\\n68.53 s – 118.03 s\\n(b)\\nFigure 6. Qualitative examples of top-1 predictions. (a) and (b)\\nis from the Charades-STA dataset and the ActivityNet Captions\\ndataset, respectively.\\nGT indicates the ground truth temporal\\nboundary.\\ncomplex moments while ViGA misses the last events “rep-\\nresent it on a plate”. More qualitative examples will be pro-\\nvided in Supplementary Materials.\\n5. Conclusion\\nIn this study, we investigate a recently proposed task,\\nTemporal Sentence Grounding with Glance Annotation.\\nUnder this setting, we propose a Dynamic Gaussian\\nprior based Grounding framework with Glance annota-\\ntion, termed D3G. Specifically, D3G consists of a Semantic\\nAlignment Group Contrastive Learning module (SA-GCL)\\nand a Dynamic Gaussian prior Adjustment module (DGA).\\nSA-GCL aims to mine a wide range of positive moments\\nand align the positive sentence-moment pairs in the joint\\nembedding space. DGA effectively alleviates the annota-\\ntion bias and models complex query consisting of multiple\\nevents via dynamically adjusting the Gaussian prior with\\nmultiple Gaussian functions, promoting the precision of lo-\\ncalization. Extensive experiments show that D3G signifi-\\ncantly narrows the performance gap between fully super-\\nvised methods and glance supervised methods. Without ex-\\ncessive interaction of visual-language, D3G provides a con-\\ncise framework and a fresh insight to the challenging tem-\\nporal sentence grounding under low-cost glance annotation.\\nLimitations. Although D3G achieves promising improve-\\nments with glance annotations, it still has some limitations.\\nIn this paper, the DAG adjusts Gaussian prior via the com-\\nbination of multiple fixed scale Gaussian functions. It fails\\nto scale down the Gaussian distribution to fit the small mo-\\nments. It is expected to explore dynamic learnable Gaussian\\nfunctions to model moment of arbitrary duration in future\\nwork. Besides, the sampling strategy for SA-GCL is still\\nnot enough flexible to sample accurate positive moments.\\n\\n\\nReferences\\n[1] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef\\nSivic, Trevor Darrell, and Bryan Russell. Localizing mo-\\nments in video with natural language.\\nIn Proceedings of\\nthe IEEE international conference on computer vision, pages\\n5803–5812, 2017. 1, 2\\n[2] Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat-\\nSeng Chua. Temporally grounding natural sentence in video.\\nIn Proceedings of the 2018 conference on empirical methods\\nin natural language processing, pages 162–171, 2018. 2\\n[3] Jingyuan Chen, Lin Ma, Xinpeng Chen, Zequn Jie, and Jiebo\\nLuo. Localizing natural language in videos. In Proceedings\\nof the AAAI Conference on Artificial Intelligence, volume 33,\\npages 8175–8182, 2019. 2\\n[4] Shaoxiang Chen and Yu-Gang Jiang. Towards bridging event\\ncaptioner and sentence localizer for weakly supervised dense\\nevent captioning.\\nIn Proceedings of the IEEE/CVF Con-\\nference on Computer Vision and Pattern Recognition, pages\\n8425–8435, 2021. 2\\n[5] Zhenfang Chen, Lin Ma, Wenhan Luo, Peng Tang, and\\nKwan-Yee K Wong. Look closer to ground better: Weakly-\\nsupervised temporal grounding of sentence in video. arXiv\\npreprint arXiv:2001.09308, 2020. 2\\n[6] Ran Cui, Tianwen Qian, Pai Peng, Elena Daskalaki, Jingjing\\nChen, Xiaowei Guo, Huyang Sun, and Yu-Gang Jiang. Video\\nmoment retrieval from text queries via single frame annota-\\ntion. arXiv preprint arXiv:2204.09409, 2022. 1, 3, 5, 6, 7\\n[7] Xinpeng Ding, Nannan Wang, Shiwei Zhang, De Cheng, Xi-\\naomeng Li, Ziyuan Huang, Mingqian Tang, and Xinbo Gao.\\nSupport-set based cross-supervision for video grounding. In\\nProceedings of the IEEE/CVF International Conference on\\nComputer Vision, pages 11573–11582, 2021. 6, 7\\n[8] Xuguang Duan, Wenbing Huang, Chuang Gan, Jingdong\\nWang, Wenwu Zhu, and Junzhou Huang. Weakly supervised\\ndense event captioning in videos. Advances in Neural Infor-\\nmation Processing Systems, 31, 2018. 2\\n[9] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia.\\nTall: Temporal activity localization via language query. In\\nProceedings of the IEEE international conference on com-\\nputer vision, pages 5267–5275, 2017. 1, 2, 5, 6, 7\\n[10] Mingfei Gao, Larry S Davis, Richard Socher, and Caiming\\nXiong. Wslln: Weakly supervised natural language localiza-\\ntion networks. arXiv preprint arXiv:1909.00239, 2019. 2\\n[11] Runzhou Ge, Jiyang Gao, Kan Chen, and Ram Nevatia. Mac:\\nMining activity concepts for language-based temporal local-\\nization. In 2019 IEEE winter conference on applications of\\ncomputer vision (WACV), pages 245–253. IEEE, 2019. 2\\n[12] Jiabo Huang, Yang Liu, Shaogang Gong, and Hailin Jin.\\nCross-sentence temporal and semantic relations in video ac-\\ntivity localisation. In Proceedings of the IEEE/CVF Inter-\\nnational Conference on Computer Vision, pages 7199–7208,\\n2021. 1, 2, 6, 7\\n[13] Bin Jiang, Xin Huang, Chao Yang, and Junsong Yuan. Cross-\\nmodal video moment retrieval with spatial and language-\\ntemporal attention. In Proceedings of the 2019 on interna-\\ntional conference on multimedia retrieval, pages 217–225,\\n2019. 2\\n[14] Yifan Jiao, Zhetao Li, Shucheng Huang, Xiaoshan Yang, Bin\\nLiu, and Tianzhu Zhang. Three-dimensional attention-based\\ndeep ranking model for video highlight detection.\\nIEEE\\nTransactions on Multimedia, 20(10):2693–2705, 2018. 2\\n[15] Chen Ju, Haicheng Wang, Jinxiang Liu, Chaofan Ma, Ya\\nZhang, Peisen Zhao, Jianlong Chang, and Qi Tian.\\nCon-\\nstraint and union for partially-supervised temporal sentence\\ngrounding. arXiv preprint arXiv:2302.09850, 2023. 3, 6, 7\\n[16] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and\\nJuan Carlos Niebles. Dense-captioning events in videos. In\\nProceedings of the IEEE international conference on com-\\nputer vision, pages 706–715, 2017. 5\\n[17] Hanjun Li, Xingjia Pan, Ke Yan, Fan Tang, and Wei-Shi\\nZheng. Siod: Single instance annotated per category per im-\\nage for object detection. In Proceedings of the IEEE/CVF\\nConference on Computer Vision and Pattern Recognition,\\npages 14197–14206, 2022. 4\\n[18] Zhijie Lin, Zhou Zhao, Zhu Zhang, Qi Wang, and Huasheng\\nLiu. Weakly-supervised video moment retrieval via semantic\\ncompletion network. In Proceedings of the AAAI Conference\\non Artificial Intelligence, volume 34, pages 11539–11546,\\n2020. 2\\n[19] Niluthpol Chowdhury Mithun, Sujoy Paul, and Amit K Roy-\\nChowdhury. Weakly supervised video moment retrieval from\\ntext queries. In Proceedings of the IEEE/CVF Conference\\non Computer Vision and Pattern Recognition, pages 11592–\\n11601, 2019. 1, 2\\n[20] Jonghwan Mun, Minsu Cho, and Bohyung Han.\\nLocal-\\nglobal video-text interactions for temporal grounding.\\nIn\\nProceedings of the IEEE/CVF Conference on Computer Vi-\\nsion and Pattern Recognition, pages 10810–10819, 2020. 2,\\n7\\n[21] Guoshun Nan, Rui Qiao, Yao Xiao, Jun Liu, Sicong Leng,\\nHao Zhang, and Wei Lu.\\nInterventional video ground-\\ning with dual contrastive learning.\\nIn Proceedings of\\nthe IEEE/CVF conference on computer vision and pattern\\nrecognition, pages 2765–2775, 2021. 2\\n[22] Marcus Rohrbach, Michaela Regneri, Mykhaylo Andriluka,\\nSikandar Amin, Manfred Pinkal, and Bernt Schiele. Script\\ndata for attribute-based recognition of composite activities.\\nIn European conference on computer vision, pages 144–157.\\nSpringer, 2012. 5\\n[23] Victor Sanh,\\nLysandre Debut,\\nJulien Chaumond,\\nand\\nThomas Wolf. Distilbert, a distilled version of bert: smaller,\\nfaster, cheaper and lighter. arXiv preprint arXiv:1910.01108,\\n2019. 3\\n[24] Gunnar A Sigurdsson, G¨\\nul Varol, Xiaolong Wang, Ali\\nFarhadi, Ivan Laptev, and Abhinav Gupta.\\nHollywood in\\nhomes: Crowdsourcing data collection for activity under-\\nstanding.\\nIn European Conference on Computer Vision,\\npages 510–526. Springer, 2016. 5\\n[25] Karen Simonyan and Andrew Zisserman. Very deep convo-\\nlutional networks for large-scale image recognition. arXiv\\npreprint arXiv:1409.1556, 2014. 3\\n[26] Yijun Song, Jingwen Wang, Lin Ma, Zhou Yu, and Jun\\nYu.\\nWeakly-supervised multi-level attentional reconstruc-\\ntion network for grounding textual queries in videos. arXiv\\npreprint arXiv:2003.07048, 2020. 2\\n\\n\\n[27] Reuben Tan, Huijuan Xu, Kate Saenko, and Bryan A Plum-\\nmer. Logan: Latent graph co-attention network for weakly-\\nsupervised video moment retrieval. In Proceedings of the\\nIEEE/CVF Winter Conference on Applications of Computer\\nVision, pages 2083–2092, 2021. 2\\n[28] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani,\\nand Manohar Paluri. Learning spatiotemporal features with\\n3d convolutional networks. In Proceedings of the IEEE inter-\\nnational conference on computer vision, pages 4489–4497,\\n2015. 3\\n[29] Hao Wang, Zheng-Jun Zha, Liang Li, Dong Liu, and Jiebo\\nLuo.\\nStructured multi-level interaction network for video\\nmoment localization via language query. In Proceedings of\\nthe IEEE/CVF Conference on Computer Vision and Pattern\\nRecognition, pages 7026–7035, 2021. 1\\n[30] Jingwen Wang, Lin Ma, and Wenhao Jiang.\\nTempo-\\nrally grounding language queries in videos by contextual\\nboundary-aware prediction. In Proceedings of the AAAI Con-\\nference on Artificial Intelligence, volume 34, pages 12168–\\n12175, 2020. 2\\n[31] Ximei Wang, Jinghan Gao, Mingsheng Long, and Jianmin\\nWang. Self-tuning for data-efficient deep learning. In In-\\nternational Conference on Machine Learning, pages 10738–\\n10748. PMLR, 2021. 4\\n[32] Yuechen Wang, Wengang Zhou, and Houqiang Li.\\nFine-\\ngrained semantic alignment network for weakly supervised\\ntemporal language grounding. In Findings of the association\\nfor computational linguistics: EMNLP 2021, pages 89–99,\\n2021. 1\\n[33] Zhenzhi Wang, Limin Wang, Tao Wu, Tianhao Li, and Gang-\\nshan Wu. Negative sample matters: A renaissance of met-\\nric learning for temporal grounding. In Proceedings of the\\nAAAI Conference on Artificial Intelligence, volume 36, pages\\n2613–2623, 2022. 1, 2, 3, 5, 6, 7\\n[34] Shaoning Xiao, Long Chen, Songyang Zhang, Wei Ji, Jian\\nShao, Lu Ye, and Jun Xiao.\\nBoundary proposal network\\nfor two-stage natural language video localization. In Pro-\\nceedings of the AAAI Conference on Artificial Intelligence,\\nvolume 35, pages 2986–2994, 2021. 1\\n[35] Huijuan Xu, Kun He, Bryan A Plummer, Leonid Sigal, Stan\\nSclaroff, and Kate Saenko. Multilevel language and vision\\nintegration for text-to-clip retrieval. In Proceedings of the\\nAAAI Conference on Artificial Intelligence, volume 33, pages\\n9062–9069, 2019. 2\\n[36] Zhe Xu, Kun Wei, Xu Yang, and Cheng Deng.\\nPoint-\\nsupervised video temporal grounding.\\nIEEE Transactions\\non Multimedia, 2022. 3, 6, 7\\n[37] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and\\nCordelia Schmid. Tubedetr: Spatio-temporal video ground-\\ning with transformers. In Proceedings of the IEEE/CVF Con-\\nference on Computer Vision and Pattern Recognition, pages\\n16442–16453, 2022. 2\\n[38] Wenfei Yang, Tianzhu Zhang, Yongdong Zhang, and Feng\\nWu. Local correspondence network for weakly supervised\\ntemporal sentence grounding. IEEE Transactions on Image\\nProcessing, 30:3252–3262, 2021. 2, 6, 7\\n[39] Xun Yang, Shanshan Wang, Jian Dong, Jianfeng Dong,\\nMeng Wang, and Tat-Seng Chua. Video moment retrieval\\nwith cross-modal neural architecture search. IEEE Transac-\\ntions on Image Processing, 31:1204–1216, 2022. 2\\n[40] Yitian Yuan, Tao Mei, and Wenwu Zhu. To find where you\\ntalk: Temporal sentence localization in video with attention\\nbased location regression. In Proceedings of the AAAI Con-\\nference on Artificial Intelligence, volume 33, pages 9159–\\n9166, 2019. 2\\n[41] Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang, and\\nLarry S Davis. Man: Moment alignment network for natural\\nlanguage moment retrieval via iterative graph adjustment. In\\nProceedings of the IEEE/CVF Conference on Computer Vi-\\nsion and Pattern Recognition, pages 1247–1257, 2019. 1, 2,\\n6\\n[42] Mingxing Zhang, Yang Yang, Xinghan Chen, Yanli Ji, Xing\\nXu, Jingjing Li, and Heng Tao Shen.\\nMulti-stage aggre-\\ngated transformer network for temporal language localiza-\\ntion in videos. In Proceedings of the IEEE/CVF Conference\\non Computer Vision and Pattern Recognition, pages 12669–\\n12678, 2021. 1, 6, 7\\n[43] Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo\\nLuo. Learning 2d temporal adjacent networks for moment\\nlocalization with natural language.\\nIn Proceedings of the\\nAAAI Conference on Artificial Intelligence, volume 34, pages\\n12870–12877, 2020. 1, 2, 3, 5, 6, 7\\n[44] Minghang Zheng, Yanjie Huang, Qingchao Chen, and Yang\\nLiu. Weakly supervised video moment localization with con-\\ntrastive negative sample mining. In Proceedings of the AAAI\\nConference on Artificial Intelligence, volume 1, page 3,\\n2022. 1, 2, 6, 7\\n[45] Minghang Zheng, Yanjie Huang, Qingchao Chen, Yuxin\\nPeng, and Yang Liu. Weakly supervised temporal sentence\\ngrounding with gaussian-based contrastive proposal learn-\\ning. In Proceedings of the IEEE/CVF Conference on Com-\\nputer Vision and Pattern Recognition, pages 15555–15564,\\n2022. 1, 2, 6, 7\\n\\n\\nAppendix\\nA. Effectiveness of SA-GCL and DGA\\nTo further analyze the effectiveness of SA-GCL and\\nDGA, we provide more detailed experimental results on\\nActivityNet Captions and TACoS datasets as shown in Ta-\\nble 7 and Table 8. Following the main manuscript, we re-\\ngard the simplified implementation of SA-GCL as a base-\\nline.\\nAfter being equipped with the complete SA-GCL,\\nour model achieves significant improvements on both Ac-\\ntivityNet Captions and TACoS. This phenomenon demon-\\nstrates that sampling enough positive moments for con-\\ntrastive learning is of great importance. Additionally, we\\nfurther incorporate the DGA module for alleviating the an-\\nnotation bias and modeling complex target moments. Since\\nthe ActivityNet Captions dataset has a large number of com-\\nplex query sentences consisting of multiple events, D3G\\nobtains notable performance gains on ActivityNet Cap-\\ntions(e.g. 9.03% at R@5 IoU=0.7). However, TACoS is still\\nchallenging for D3G due to the dense distributions of target\\nmoments.\\nModule\\nR@1\\nR@5\\nSA-GCL DGA IoU=0.5 IoU=0.7 IoU=0.5 IoU=0.7\\n✓†\\n0.83\\n0.28\\n1.78\\n0.58\\n✓\\n32.65\\n16.00\\n65.48\\n43.44\\n✓\\n✓\\n36.68\\n18.54\\n74.21\\n52.47\\nTable 7. Effectiveness of SA-GCL and DAG in D3G on Activi-\\ntyNet Captions. ✓† denotes an simplified implementation of SA-\\nGCL.\\nModule\\nR@1\\nR@5\\nSA-GCL DGA IoU=0.5 IoU=0.7 IoU=0.5 IoU=0.7\\n✓†\\n2.97\\n0.37\\n5.40\\n1.10\\n✓\\n11.95\\n4.20\\n29.07\\n10.30\\n✓\\n✓\\n12.67\\n4.70\\n31.34\\n12.35\\nTable 8. Effectiveness of SA-GCL and DAG in D3G on TACoS.\\n✓† denotes an simplified implementation of SA-GCL.\\nB. Effect of different hyper-parameters\\nIn this section, we investigate the effect of two criti-\\ncal hyperparameters on ActivityNet Captions and TACoS\\ndatasets. As shown in Figure 7 and Figure 8, we report\\nthe changes in performance at four metrics. As for top-k,\\nthe performance increases dramatically as the k increases.\\nHowever, the performance gradually achieves saturation af-\\nter the k reaches 15. We finally select k = 20 for both Ac-\\ntivityNet Captions and TACoS. As for σ, the ActivityNet\\nFigure 7. Effect of top-k and σ on ActivityNet Captions dataset.\\nCaptions dataset tends to prefer large values while small\\nvalues are more suitable for the TACoS dataset. This is be-\\ncause the former contains a large number of long target mo-\\nments while the latter contains numerous short target mo-\\nments. As shown in Figure 7 and Figure 8, we eventually\\nselect σ = 0.6 and σ = 0.2 for ActivityNet Captions and\\nTACoS for optimal performance, respectively.\\nC. Qualitative Analysis\\nIn this section, we provide more qualitative examples\\nfrom the test split of the Charades-STA dataset, ActivityNet\\nCaptions dataset, and TACoS dataset. For each video, we\\nselect two queries for analysis. As shown in Figure 9 (a),\\nD3G locates the target moment accurately while ViGA ig-\\nnores the reason at the front of the target moment, given\\nQuery 1. However, D3G is inferior to ViGA in some cases\\nsuch as Query 2. As for complex queries in ActivityNet\\nCaptions, D3G still localizes a moment with a large over-\\nlap with the target moment. Since sentence-level features\\nmay lose some information about specific events, D3G can-\\nnot perceive accurate boundaries for some complex queries,\\nsuch as Figure 9 (b) Query 2. It is expected to explore event-\\n\\n\\nFigure 8. Effect of top-k and σ on TACoS dataset.\\nlevel features for queries consisting of multiple events in\\nthe future. TACoS is the most challenging dataset, where\\nthe videos have long durations and contain a large number\\nof moment-sentence pairs. As shown in Figure 9 (c), we\\nobserve that D3G fails to locate a simple query of short du-\\nration from the long video, given Query 1. However, D3G\\naccurately locates the target moment of long duration given\\nQuery 2. Note that D3G well attends to the number “the\\nlast two” of the query while ViGA fails to attend to such in-\\nformation and locates irrelevant moments. As observed in\\nFigure 9, D3G is superior to ViGA, which is consistent with\\nthe experimental results in the main manuscript. However,\\nD3G still has some limitations and needs to be improved in\\nthe future.\\n\\n\\nQuery 1: person laughing because they see something funny on the television.\\n0 s – 5.2 s\\n4.03 s – 14.1 s\\n0 s – 6.29 s\\nQuery 2: a person in their dining room is running around.\\n1.3 s – 12.8 s\\n1.01 s – 11.07 s\\n1.57 s – 7.87 s\\nQuery 1: A man and a woman are standing outside at a beach in the sand talking while the lady \\nholds a brown paper bag in her hand and a man begins filming them.\\nduration: 213.42 s             \\n0 s – 44.82 s\\n0 s – 64.03 s\\n0 s – 53.35 s\\nQuery 2: The teams begin to get extremely individual and add words and feathers to their   \\nmasterpiece before the man and lady come around to judge them.\\n138.72 – 185.67 s\\n57.62 s – 164.33 s\\n120.05 s – 213.42 s\\n…\\n5.17 s – 10.03 s\\n0 s – 37.91 s\\n8.89 s – 26.66 s\\nQuery 1:The person gets out a cutting board.\\nQuery 2: The person cuts up the last two slices of pineapple.\\nduration: 379.11 s             \\n327.14 s – 372.11 s\\n185.77 s – 242.63 s\\n346.54 s – 376.15 s\\nGT\\nViGA\\nD3G\\nGT\\nViGA\\nD3G\\nGT\\nViGA\\nD3G\\nGT\\nViGA\\nD3G\\nGT\\nViGA\\nD3G\\nGT\\nViGA\\nD3G\\n(a)\\n(b)\\n(c)\\nduration: 25.17 s             \\nFigure 9. Qualitative examples of top-1 predictions. (a), (b) and (c) is from the Charades-STA dataset, the ActivityNet Captions and the\\nTACoS dataset, respectively. GT indicates the ground truth temporal boundary.\\n\\n\\nProbability Distribution Based Frame-supervised\\nLanguage-driven Action Localization\\nShuo Yang\\nshuoyang@bit.edu.cn\\nBeijing Key Laboratory of Intelligent\\nInformation Technology, Beijing\\nInstitute of Technology\\nGuangdong Laboratory of Machine\\nPerception and Intelligent Computing,\\nShenzhen MSU-BIT University\\nZirui Shang\\nziruishang@bit.edu.cn\\nBeijing Key Laboratory of Intelligent\\nInformation Technology, Beijing\\nInstitute of Technology\\nXinxiao Wu∗\\nwuxinxiao@bit.edu.cn\\nBeijing Key Laboratory of Intelligent\\nInformation Technology, Beijing\\nInstitute of Technology\\nGuangdong Laboratory of Machine\\nPerception and Intelligent Computing,\\nShenzhen MSU-BIT University\\nABSTRACT\\nFrame-supervised language-driven action localization aims to lo-\\ncalize action boundaries in untrimmed videos corresponding to the\\ninput natural language query, with only a single frame annotation\\nwithin the target action in training. This task is challenging due to\\nthe absence of complete and accurate annotation of action bound-\\naries, hindering visual-language alignment and action boundary\\nprediction. To address this challenge, we propose a novel method\\nthat introduces distribution functions to model both the probability\\nof action frame and that of boundary frame. Specifically, we assign\\neach video frame the probability of being the action frame based on\\nthe estimated shape parameters of the distribution function, serv-\\ning as a foreground pseudo-label that guides cross-modal feature\\nlearning. Moreover, we model the probabilities of start frame and\\nend frame of the target action using different distribution functions,\\nand then estimate the probability of each action candidate being\\na positive candidate based on its start and end boundaries, which\\nfacilitates predicting action boundaries by exploring more positive\\nterms in training. Experiments on two benchmark datasets demon-\\nstrate that our method outperforms existing methods, achieving a\\ngain of more than 10% of 𝑅1@𝜇≥0.5 on the challenging TACoS\\ndataset. These results emphasize the significance of generating\\npseudo labels with appropriate probabilities via distribution func-\\ntions to address the challenge of frame-supervised language-driven\\naction localization. 1\\nCCS CONCEPTS\\n• Information systems →Novelty in information retrieval;\\nVideo search.\\n∗corresponding author\\n1Codes could be found at github\\nPermission to make digital or hard copies of all or part of this work for personal or\\nclassroom use is granted without fee provided that copies are not made or distributed\\nfor profit or commercial advantage and that copies bear this notice and the full citation\\non the first page. Copyrights for components of this work owned by others than the\\nauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or\\nrepublish, to post on servers or to redistribute to lists, requires prior specific permission\\nand/or a fee. Request permissions from permissions@acm.org.\\nMM ’23, October 29–November 3, 2023, Ottawa, ON, Canada.\\n© 2023 Copyright held by the owner/author(s). Publication rights licensed to ACM.\\nACM ISBN 979-8-4007-0108-5/23/10...$15.00\\nhttps://doi.org/10.1145/3581783.3612512\\nKEYWORDS\\nlanguage-driven action localization, video moment retrieval, distri-\\nbution, frame-supervised.\\nACM Reference Format:\\nShuo Yang, Zirui Shang, and Xinxiao Wu. 2023. Probability Distribution\\nBased Frame-supervised Language-driven Action Localization. In Proceed-\\nings of the 31st ACM International Conference on Multimedia (MM ’23), Octo-\\nber 29–November 3, 2023, Ottawa, ON, Canada. ACM, New York, NY, USA,\\n10 pages. https://doi.org/10.1145/3581783.3612512\\n1\\nINTRODUCTION\\nLanguage-driven action localization has drawn increasing atten-\\ntion in recent years, which aims to locate the action interval in an\\nuntrimmed video that is semantically relevant to a language query.\\nThis task, also known as video moment retrieval [7, 34, 39, 46] or\\ntemporal sentence grounding [16, 18, 44], is a fundamental prob-\\nlem in video understanding and multi-modal information retrieval,\\nwhich involves not only cross-modal alignment but also action\\nboundary localization. It has been widely applied in various sce-\\nnarios, such as content-based video search and automatic video\\nediting.\\nPrevious methods [1, 6, 11, 31, 37, 43] have achieved remarkable\\nsuccess in the fully-supervised setting that requires annotating both\\nthe start and end timestamps of the target action corresponding to a\\ngiven language query, as shown in Figure 1 (a). However, the frame\\nannotation paradigm is time-consuming because annotators need\\nto review videos multiple times to accurately identify action bound-\\naries. Consequently, recent methods [13, 25, 33, 38, 50] explore the\\nweakly-supervised setting where only language-video pair annota-\\ntions are provided, resulting in less annotation burden but lower\\nperformance, as shown in Figure 1 (b). The frame-supervised set-\\nting, first proposed in [4, 42], uses single-frame annotation within\\nthe target action, achieving a good balance between annotation\\ncost and performance, as shown in Figure 1 (c). However, incom-\\nplete annotations still hinder visual-language alignment and action\\nboundary prediction.\\nIn this paper, we propose a new method to model the probabilities\\nof action frames and boundary frames by introducing distribution\\nfunctions. By exploiting the temporal consistency between video\\nframes and properties of probability distribution functions, we ex-\\ntend annotated frames to other frames with different probabilities.\\nThis enables the generation of frame-wise pseudo-labels of action\\nframes, which is useful for learning video-language alignment. We\\n5164\\n\\n\\nMM ’23, October 29–November 3, 2023, Ottawa, ON, Canada.\\nShuo Yang, Zirui Shang, and Xinxiao Wu\\nLanguage query:  person opens a refrigerator\\nLanguage query:  person opens a refrigerator\\nLanguage query:  person opens a refrigerator\\n10.5s\\n18.1s\\n12.6s\\n(b) Weakly-supervised \\n(c) Frame-supervised \\n(a) Fully-supervised\\nFigure 1: Illustration of different settings of language-driven\\naction localization. Given an input language query and an\\ninput video, (a) the fully-supervised setting provides the start-\\ning and ending boundaries of the target action, (b) the weakly-\\nsupervised setting provides no additional labels, and (c) the\\nframe-supervised setting gives a frame annotation within\\nthe target action.\\nalso model the probabilities of a video frame as the start and end of\\nthe target action via different distribution functions. And by multi-\\nplying the start and end probabilities, we can obtain the probability\\nof an action candidate being a positive target action segment. By\\ndoing so, all action candidates can be treated as positive action can-\\ndidates with varying probabilities, enabling the exploration of all\\npossible positive action candidates with appropriate probabilities\\nand thus improving the accuracy of boundary estimation.\\nSpecifically, we assign each video frame a probability of being\\nthe action frame, which is highest at the annotated frame and grad-\\nually decreases to a minimum near the boundary. To model this\\nprobability, we estimate the parameters of a specific distribution\\nbased on the visual similarity and temporal distance between the\\nvideo frames and the annotated frame. In particular, we use the\\nvarious asymmetrical probability curves of Beta distribution to han-\\ndle the situations in which only few annotated frames are located\\nat the center of the action segments. Using the resulting probabil-\\nity as a soft label for each action frame, we can optimize a binary\\ncross-entropy loss that forces the visual feature to be similar to the\\nlanguage query feature with appropriate loss weights.\\nFurthermore, we introduce another distribution function to model\\nthe probability of each video frame being the start or end bound-\\nary of the target action. In our method, the highest probabilities\\nare assigned to the boundaries of an action candidate, or so-called\\nproposal, that is closest to the language query in the feature embed-\\nding space, while the annotated frame is less likely to be an action\\nboundary, as illustrated in Figure 4. We combine the probabilities\\nof the starting and ending boundaries to give each action candidate\\na probability of being the target action segment. This helps us iden-\\ntify more positive action candidates with reasonable probabilities,\\nthereby facilitating the localization of action boundaries.\\nThe main contributions of this paper are as follows:\\n• We propose a novel method that uses distribution functions,\\nsuch as the Beta distribution, to generate a probability for\\neach video frame being the action frame, serving as a pseudo-\\nlabel to enhance the cross-modal feature learning.\\n• We propose to use different distribution functions to model\\nthe probabilities of the start frame and end frame of the\\ntarget action, so as to explore more positive action candidates\\nduring training, thus facilitating the localization of action\\nboundaries.\\n• Experiments on two benchmark datasets demonstrate that\\nour method outperforms existing methods, especially achiev-\\ning a gain of more than 10% of 𝑅1@𝜇≥0.5 on the challeng-\\ning TACoS dataset.\\n2\\nRELATED WORK\\nCurrent language-driven action localization settings can be roughly\\ndivided into three types: fully-supervised, weakly-supervised and\\nframe-supervised.\\nFully-supervised language-driven action localization re-\\nquires the annotation of start and end timestamps for each query\\nduring training. Existing methods of this setting can be broadly\\ncategorized into two groups: proposal-based and proposal-free. In\\nthe proposal-based methods, candidate proposals are first generated\\nusing sliding windows, proposal generation, or anchor-based meth-\\nods, and are then ranked based on queries. For instance, CTRL [6],\\nMCN [1], MARN [19], HVSARN [20] and TSTNet [40] generate pro-\\nposals of varying lengths through sliding windows. 2D-TAN [48],\\nMGPN [32], HLN [5] and VDI [22] generate proposals by using\\na two-dimensional feature map that model the relationships be-\\ntween segments of varying durations. The proposal-free methods\\ndirectly predict the start and end boundaries of the target action\\non sequences of fine-grained video clips. According to the format\\nof moment boundaries, proposal-free methods are categorized into\\nspan-based and regression-based methods. VSLNet [47], SLP [14]\\nand D-TSG [17] directly predict the probability of each video snip-\\npet or frame being the start and end positions of the target action.\\nTVP [49] and MGSL-Net [15] calculates a time pair and compares\\nit with ground truth for model optimization.\\nWeakly-supervised language-driven action localization\\nonly requires the annotation of pairs of video and query instead of\\nthe annotation of start and end times, thus reducing the high an-\\nnotation cost. Existing weakly-supervised language-driven action\\nlocalization methods can be broadly classified into two categories:\\nmulti-instance learning and reconstruction-based methods. For in-\\nstance, TGA [25] regards the video and its corresponding query de-\\nscriptions as positive pairs, while considering the video with other\\nqueries and the query with other videos as negative pairs. This\\nmethod learns video-level visual-text alignment by maximizing the\\nmatching scores of positive samples while minimizing the scores\\nof negative samples. SAN [38] introduces a multi-scale Siamese\\nmodule that progressively narrows the semantic gap between the\\nvisual and textual modalities. RTBPN [50] uses a language-aware\\nfilter to generate an enhanced video stream and a suppressed video\\nstream, which are used to generate positive proposals and negative\\nproposals for sufficient confrontment, separately.\\n5165\\n\\n\\nDistribution Based Frame-supervised Language-driven Action Localization\\nMM ’23, October 29–November 3, 2023, Ottawa, ON, Canada.\\nFrame-supervision is a setting that aims to strike a balance\\nbetween annotation cost and performance, which has been applied\\nto various computer vision tasks. Bearman et al. [2] introduce the\\nconcept of frame supervision for semantic segmentation. Mettes\\net al. [24] extend the use of frame supervision to spatio-temporal\\naction localization in videos. In recent years, Ma et al. [23] pro-\\npose a SF-Net model for video temporal action localization by using\\nsingle-frame supervision. Li et al. [12] develop a temporal action seg-\\nmentation model that requires only timestamp annotations. More\\nrecently, some studies investigated the implementation of single-\\nframe annotation for language-driven action localization. Cui et\\nal. [4] originally introduce the concept of frame supervision for\\nlanguage-driven action localization, which uses the Gaussian distri-\\nbution to model the probability distribution of foreground frames.\\nMeanwhile, Xu et al. [42] employ a combination of Language Acti-\\nvation Sequence (LAS) and given frame supervision to enhance the\\nmodel’s ability in language-driven action localization.\\n3\\nOUR METHOD\\n3.1\\nProblem Definition\\nGiven an untrimmed video and a language query, the task of frame-\\nsupervised language-driven action localization aims to localize the\\ntarget action boundaries (𝜏𝑠,𝜏𝑒) with an additional frame annota-\\ntion 𝑡𝑝in the training stage, where 𝜏𝑠≤𝑡𝑝≤𝜏𝑒, and 𝜏𝑠and 𝜏𝑒\\nrepresent the start and end frames of the action corresponding to\\nthe language query, respectively. Note that in the inference stage,\\nthe frame annotation is not available.\\n3.2\\nBaseline Model\\nDue to the lack of boundary annotation, we propose a baseline\\nmodel of frame-supervised language-driven action localization,\\nwhich follows an multiple-instance learning (MIL) strategy and\\nconsists of three components: a video encoder, a language encoder,\\nand a cross-modal interaction module, as shown in Figure 2.\\nVideo Encoder. We first split the given video into a sequence\\nof non-overlap clips with a fixed length (e.g., 16 frames) and ex-\\ntract visual features of each clip by a pre-trained 3D-CNN [3, 35].\\nThen we uniformly sample 𝑇features and project them into 𝑑-\\ndimensional representations using a fully-connected (FC) layer.\\nFinally, we encode temporal relationships using a standard Trans-\\nformer block [36] that consists of multi-head self-attention, layer\\nnormalization, residual connection, and feed-forward network. The\\nprocess of video encoding is represented by\\n𝑽′ = 𝑇𝑟𝑎𝑛𝑠𝑓𝑜𝑟𝑚𝑒𝑟(𝐹𝐶(Uniform-Sample(3D-CNN(𝑉)))).\\n(1)\\nLanguage Encoder. Given an input language query, we first ini-\\ntialize its word features using the GloVe embedding [28] and then\\nproject their dimension to 𝑑by a fully-connected (FC) layer, fol-\\nlowed by a three-layer bi-directional Gated Recurrent Unit (GRU)\\nto learn the relationships of words:\\n𝑸′ = Bi-GRU(𝐹𝐶(𝐺𝑙𝑜𝑉𝑒(𝑄))).\\n(2)\\nCross-modal Interaction Module. After encoding the video and\\nlanguage query, we adopt two cross-attention modules for the cross-\\nmodal interaction, each by regarding one modality as the query\\nand the other as key and value, followed by a layer normalization,\\na residual connection and a feed-forward network:\\n𝑭𝑣= 𝐹𝐹𝑁(𝐿𝑁(𝑀𝑆𝐴(𝐹𝐶𝑄(𝑽′), 𝐹𝐶𝐾(𝑸′), 𝐹𝐶𝑉(𝑸′)) + 𝑽′)),\\n𝑭𝑞= 𝐹𝐹𝑁(𝐿𝑁(𝑀𝑆𝐴(𝐹𝐶𝑄(𝑸′), 𝐹𝐶𝐾(𝑽′), 𝐹𝐶𝑉(𝑽′)) + 𝑸′)),\\n(3)\\nwhere 𝑀𝑆𝐴is the multi-head self-attention module[36]; 𝐿𝑁de-\\nnotes layer normalization; 𝐹𝐶𝑗(·) denotes fully connected layer,\\n𝑗∈{𝑄, 𝐾,𝑉}; 𝐹𝐹𝑁(·) is a feed-forward network.\\nTraining loss. For the learned video features 𝑭𝑣, we generate ac-\\ntion candidates 𝑃= {𝑃1, 𝑃2, · · · , 𝑃𝑁}, also known as proposals, by\\nsliding windows, where 𝑁is the total number of action candidates,\\nand 𝑃𝑖= max-pooling([𝑭𝑣,𝑠𝑖, · · · , 𝑭𝑣,𝑒𝑖]) ∈R𝑑is the 𝑖-th action\\ncandidate and 𝑠𝑖and 𝑒𝑖are its start and end frame index, respec-\\ntively; And we also compute the sentence features by mean-pooling\\non the learned language features: 𝑭𝑠= mean-pooling(𝑭𝑞) ∈R𝑑.\\nIn the training stage, we use an intra-video loss and an inter-\\nvideo loss to learn the video-language alignment. The intra-video\\nloss treats the action candidates containing the annotated frame\\nas positive candidates 𝑃+ and others as negatives 𝑃−. It enforces\\nthe similarities between the language query and positive action\\ncandidates larger than the similarities between the language query\\nand negative candidates by the InfoNCE [27] loss, given by\\nL𝑖𝑛𝑡𝑟𝑎= −1\\n𝑀\\n∑︁\\n𝑝𝑖∈𝑃+\\nlog\\nexp(𝑆(𝑝𝑖, 𝑭𝑠)/𝜏)\\nexp(𝑆(𝑝𝑖, 𝑭𝑠)/𝜏) +\\n∑︁\\n𝑝𝑗∈𝑃−\\nexp(𝑆(𝑝𝑗, 𝑭𝑠)/𝜏)\\n,\\n(4)\\nwhere 𝑆(·, ·) is the cosine similarity function; 𝑀is the number of\\npositive action candidates; 𝜏is a temperature parameter and set to\\n0.07 as ViGA [4].\\nThe inter-video loss is also an InfoNCE loss and calculated in a\\nmini-batch, where the positive candidates 𝑃+ in the paired video-\\nsentences are positive terms, and all action candidates of unpaired\\nvideo-sentences are negative terms. The inter-video enforces the\\nsimilarities between positive terms larger than the similarities be-\\ntween negative terms in a mini-batch, given by\\nL𝑖𝑛𝑡𝑒𝑟= −\\n1\\n𝑀×𝐵\\n𝐵\\n∑︁\\n𝑏=0\\n∑︁\\n𝑝𝑏,𝑖∈𝑃+\\n𝑏\\nlog\\nexp(𝑆(𝑝𝑏,𝑖, 𝑭𝑠\\n𝑏)/𝜏)\\nexp(𝑆(𝑝𝑏,𝑖, 𝑭𝑠\\n𝑏)/𝜏) + N ,\\nN =\\n∑︁\\n𝑗≠𝑏\\n\\u0010\\nexp(𝑆(𝑝𝑏,𝑖, 𝑭𝑠\\n𝑗)/𝜏) + exp(𝑆(𝑝𝑗, 𝑭𝑠\\n𝑏)/𝜏)\\n\\u0011\\n,\\n(5)\\nwhere 𝑝𝑏,𝑖∈𝑃+\\n𝑏is the 𝑖-th positive candidates of 𝑏-th video in\\na mini-batch; 𝑀is the size of 𝑃+\\n𝑏; 𝐵is batch size; N denotes the\\nnegative terms of none paired video-sentence in the mini-batch.\\n3.3\\nPseudo-label of Action Frame\\nThe baseline model uses frame annotations to distinguish positive\\nand negative action candidates, which ignores the temporal coher-\\nence of videos. Indeed, the annotated frames play a pivotal role in\\nthe frame-supervised language-driven action localization. A com-\\nmon intuition is that the frames adjacent to the annotated frame\\nare more likely to be action frames, while the frames far from the\\nannotated frame are less likely to be action frames. However, how\\nthese possibilities change remains an open problem.\\nIn this study, we propose to use distribution functions to model\\nthe probability changes, taking into account temporal distance and\\nvisual similarity between video frames. The resulting probabilities\\n5166\\n\\n\\nMM ’23, October 29–November 3, 2023, Ottawa, ON, Canada.\\nShuo Yang, Zirui Shang, and Xinxiao Wu\\nFq\\nLanguage query:\\nperson opens a   \\nrefrigerator\\nVideo \\nEncoder\\nLanguage \\nEncoder\\nCross-modal \\nInteraction\\nFv\\nCandidates\\n……\\nA\\nProbability of Action \\nboundary\\ns’\\ne’\\nstart\\nend\\nIntra-loss\\nInter-loss\\nCandidate \\nlabels\\nProbability of \\nAction frame\\nA\\nAction frame labels\\nFore-loss\\nFs\\nDistribution based loss\\nBaseline model\\nFigure 2: Overview of the proposed method. In the distribution based loss, 𝐴denotes the annotated frame; 𝑠\\n′ and 𝑒\\n′ are the\\nboundaries of the candidate with maximum similarity to the language query; the curves represent the probabilities by the\\nestimated distributions, by which the pseudo labels of action frames and action candidates are generated (dark color means\\nhigh probability).\\nProbability of \\nAction frame\\nη\\n1\\n0\\nProbability of \\nAction frame\\nη\\n1\\n0\\nProbability of \\nAction frame\\nA\\nτs\\nτe\\nη\\n1\\n0\\nT\\n(a) Triangular\\n(b) Gaussian\\n(c) Beta\\nA\\nτs\\nτe\\nT\\nA\\nτs\\nτe\\nT\\nFigure 3: Probabilities of target action frames by different\\ndistributions: (a) Triangular Distribution, (b) Gaussian Distri-\\nbution, and (c) Beta Distribution. 𝜏𝑠and 𝜏𝑒denote the ground-\\ntruth boundary of target action; 𝐴is the annotated frame; 𝑇\\nis the video length; 𝜂is the minimum probability.\\ncan be viewed as pseudo-labels for action frames, which extends\\nthe annotated frame to its neighbors with different probabilities.\\nThis extension provides valuable guidance for learning cross-modal\\nalignment, thereby improving the accuracy of action localization.\\nBy leveraging distribution functions in this way, we aim to improve\\nthe performance of action localization in videos. Here we explore\\nthree distributions to model the probability of frames being the\\naction frames.\\nTriangular distribution. We start with a simple distribution, the\\nTriangular distribution, which models the probability changes based\\nonly on the temporal distance between frames. In this distribution,\\nthe probability decreases linearly with the increasing distance from\\nthe annotated frame. As shown in Figure 3(a), we assume that the\\nminimum probability of vidoe frame is 𝜂, i.e., 𝜂= 1\\n𝑇, where 𝑇is\\nthe length of video, and the maximum probability of the annotated\\nframe 𝐴is 1, the probability of the frame 𝑥being the action frame\\nis calculated by\\n𝑃𝑓\\n𝑡(𝑥) =\\n( 1\\n𝐷( 𝑥(1−𝜂)\\n𝐴\\n+ 𝜂),\\n0 ≤𝑥≤𝐴\\n1\\n𝐷( (𝑇−𝑥) (1−𝜂)\\n(𝑇−𝐴)\\n+ 𝜂),\\n𝐴< 𝑥≤𝑇\\n(6)\\nwhere the factor 1\\n𝐷is used as a normalization factor to ensure that\\nthe sum of all probabilities is equal to 1. However, in cases where\\nwe want the maximum probability to be 1, we can discard this\\nnormalization factor by setting 𝐷= 1 without affecting the shape\\nof the distribution.\\nGaussian distribution. The Gaussian distribution can also model\\nthe probability changes of action frames. This more sophisticated\\ndistribution assumes that the probability changes follow a bell curve,\\nwith the highest probability occurring at the annotated frame and\\ngradually decreasing as the distance from the annotated frame in-\\ncreases, as illustrated in Figure 3(b). The mean 𝜇of the Gaussian\\ndistribution is set to the annotated frame 𝐴, and we estimate its vari-\\nance 𝜎2 by considering both the temporal distance and visual simi-\\nlarity between frames. The temporal distance between the frame 𝑡\\nand the annotated frame 𝐴is 𝐷(𝑡,𝐴) = |(𝑡−𝐴)|\\n𝑇\\n. The visual similar-\\nity is calculated by𝑉(𝑡,𝐴) = 0.5×𝑐𝑜𝑠𝑖𝑛𝑒_𝑠𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦(𝑭𝑣𝑡, 𝑭𝑣𝐴)+0.5,\\nand finally, the similarity is a weighted sum of temporal distance\\nand visual similarity:\\n𝑆𝐼𝑀(𝑡,𝐴) = 𝜆1 · 𝑉(𝑡,𝐴) + 𝜆2 · (1 −𝐷(𝑡,𝐴)),\\n(7)\\nwhere 𝜆1 and 𝜆2 are hyper-parameters. The standard deviation 𝜎\\ncan be estimated by 𝜎= (𝐶𝑙+ 𝐶𝑟)/2, where 𝐶𝑙and 𝐶𝑟denote the\\nnumber of frames on the left side (𝑡< 𝐴) and right side (𝑡> 𝐴) of the\\nannotated frame, respectively. These frames satisfy the condition\\n𝑆𝐼𝑀(𝑡,𝐴) ≥𝜃, where 𝜃is a hyper-parameter set to 0.9 times the\\nmaximum value of 𝑆𝐼𝑀(𝑡,𝐴). With 𝜇and 𝜎, the probability of the\\nvideo frames 𝑥being the action frame is calculated by\\n𝑃𝑓\\n𝑔(𝑥) = exp(−(𝑥−𝜇)2\\n2𝜎2\\n).\\n(8)\\nSince the annotated frame is not always located at the center of the\\naction, the probability changes of the frames on either side of the\\nannotated frame may not be symmetrical. However, the Gaussian\\ndistribution assumes that the changes are symmetrical, which limits\\nits accuracy in certain situations.\\nBeta distribution. To overcome the limitation of the Gaussian\\ndistribution, we introduce the Beta distribution. As shown in Fig-\\nure 3(c), the diverse curve and asymmetric probability of the Beta\\ndistribution enables modeling different relative positions of the\\nannotated frame within an action. The probability density function\\n5167\\n\\n\\nDistribution Based Frame-supervised Language-driven Action Localization\\nMM ’23, October 29–November 3, 2023, Ottawa, ON, Canada.\\nProbability of \\nboundary frame\\nA\\ns’\\nτe\\nη\\n1\\n0\\nT\\n(a) Triangular\\nτs\\ne’\\nProbability of \\nboundary frame\\nη\\n1\\n0\\n(c) Beta\\nA\\nτs\\nτe\\nT\\ns’\\ne’\\nProbability of \\nboundary frame\\nA\\ns’\\nτe\\nη\\n1\\n0\\nT\\n(b) Gaussian\\nτs\\ne’\\nFigure 4: Probabilities of start frames (red curves) and end\\nframes (blue curves) of the target action by the (a) Triangular\\nDistribution, (b) Gaussian Distribution, and (c) Beta Distri-\\nbution. 𝑠\\n′ and 𝑒\\n′ denote the start and end boundaries of the\\nselected action candidate, whose similarity with language\\nquery is the largest;\\nof the Beta distribution is an exponential function of the variable 𝑥\\nand its reflection (1 −𝑥) as follows:\\n𝑓(𝑥;𝛼, 𝛽) = 𝑥𝛼−1(1 −𝑥)𝛽−1\\nB(𝛼, 𝛽)\\nfor 0 ≤𝑥≤1,\\n(9)\\nwhere 𝛼and 𝛽are the shape parameters, and B(𝛼, 𝛽) is the beta\\nfunction serving as a normalization factor to ensure the total prob-\\nability is 1. The mean 𝜇and variance 𝜎2 are calculated by\\n𝜇=\\n𝛼\\n𝛼+ 𝛽,\\n𝜎2 =\\n𝛼𝛽\\n(𝛼+ 𝛽)2(𝛼+ 𝛽+ 1) .\\n(10)\\nAs the work [41] says, it is hard to directly estimate the parameters\\n𝛼and 𝛽. Thus, we estimate the parameters of Beta Distribution by\\nits mean and variance. To do so, we again calculate the similarity\\nbetween the video frame and the annotated frame by Eq.(7), and\\nset the similarity value larger than the threshold 𝜃to 1 and others\\nto minimum probability 𝜂, same as Triangular distribution, we set\\n𝜂= 1\\n𝑇, where 𝑇is the length of video:\\n𝑆𝐼𝑀(𝑥,𝐴)\\n′ =\\n(\\n1,\\n𝑆𝐼𝑀(𝑥,𝐴) ≥𝜃\\n𝜂,\\n𝑜𝑡ℎ𝑒𝑟𝑠\\n(11)\\nThen the mean ¯\\n𝜇and variance ¯\\n𝜎2 are calculated by\\n¯\\n𝜇= 1\\n𝑇\\n𝑇\\n∑︁\\n𝑥=1\\n𝑆𝐼𝑀(𝑥,𝐴)\\n′ · 𝑥\\n𝑇,\\n¯\\n𝜎2 = 1\\n𝑇\\n𝑇\\n∑︁\\n𝑥=1\\n𝑆𝐼𝑀(𝑥,𝐴)\\n′ · ( 𝑥\\n𝑇−𝜇)2.\\n(12)\\nFinally, the parameters of the Beta distribution are derived as\\n𝛼= ¯\\n𝜇\\n\\u0012 ¯\\n𝜇(1 −¯\\n𝜇)\\n¯\\n𝜎2\\n−1\\n\\u0013\\n,\\n𝛽= (1 −¯\\n𝜇)\\n\\u0012 ¯\\n𝜇(1 −¯\\n𝜇)\\n¯\\n𝜎2\\n−1\\n\\u0013\\n(13)\\nTherefore, the probability of the video frames 𝑥being the action\\nframe is calculated by\\n𝑃𝑓\\n𝑏(𝑥) =\\n1\\nB(𝛼, 𝛽)\\n\\u0010 𝑥\\n𝑇\\n\\u0011𝛼−1 \\u0010\\n1 −𝑥\\n𝑇\\n\\u0011𝛽−1\\n.\\n(14)\\nIn cases where we want the maximum probability to be 1, we replace\\nthe normalization factor B(𝛼, 𝛽) with a min-max normalization. In\\ngeneral, the Beta distribution is more complex to estimate than other\\ndistributions, which may limit its applications in some scenarios.\\nNonetheless, this distribution is a valuable tool for modeling the\\nprobability of action frames.\\nForeground Loss. With the probabilities of video frames being\\nthe action frames, we introduce a foreground loss to enforce the\\nembedding of the relevant video frame close to the language query,\\nwhich helps to learn cross-modal alignment, given by\\nL𝑓𝑜𝑟𝑒= 𝐵𝐶𝐸(𝑐𝑜𝑠𝑖𝑛𝑒_𝑠𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦(𝑭𝑞, 𝑭𝑣)/𝜏, 𝑃𝑓\\n𝑑),\\n(15)\\nwhere 𝑭𝑞and 𝑭𝑣are the language features and video features,\\nrespectively; 𝐵𝐶𝐸is the binary cross entropy loss; 𝜏is the tempera-\\nture parameter; 𝑃𝑓\\n𝑑with 𝑑∈{𝑡,𝑔,𝑏} are the probabilities of action\\nframes computed by the Triangular, Gaussian, or Beta distributions.\\n3.4\\nPseudo-label of Boundary Frame\\nIn the baseline model of frame-supervised language-driven action\\nlocalization, all action candidates containing the annotated frame\\nare treated as positive, and others are negative, which may intro-\\nduce significant noise. As such, we propose to use distribution\\nfunctions to model the probability of each video frame being a po-\\ntential starting or ending boundary of the target action moment. By\\nmodeling the temporal boundaries in this manner, each action can-\\ndidate is assigned a probability of being the target action moment\\nby multiplying the probabilities of its boundaries.\\nGiven that the annotated frame is inclined to be within the target\\naction, it is expected to have a minimum probability as a boundary.\\nAnd for the positive action candidates 𝑃+ that include the annotated\\nframe, we designate the one with the maximum cosine similarity\\nto the language query as the pseudo target action segment, whose\\nstaring and ending boundaries are denoted as 𝑠′ and 𝑒′ in Figure 4,\\nand assign it a maximum probability of 1. We formulate the starting\\nboundary probability within the interval [0,𝐴] and the ending\\nboundary probability within the interval [𝐴,𝑇]. As discussed in\\nSection 3.3, the starting probability 𝑃𝑠\\n𝑑and the ending probability\\n𝑃𝑒\\n𝑑are modeled using distribution functions, and details are omitted\\nhere. Consequently, the probability of an action candidate being\\npositive (i.e., belonging to the target action) is computed by taking\\nthe multiplication of its boundary probabilities,given by\\n𝑃𝑝\\n𝑑(𝑖) = 𝑃𝑝\\n𝑑(𝑥1,𝑥2) = 𝑃𝑠\\n𝑑(𝑥1) × 𝑃𝑒\\n𝑑(𝑥2),\\n𝑑∈{𝑡,𝑔,𝑏},\\n(16)\\nwhere 𝑥1 and 𝑥2 are the starting and ending frame indexes of the\\n𝑖-th action candidate, and 𝑑denotes the index of distributions.\\nAs we assign each action candidate a probability by Eq.(16), we\\nupdate the loss function in the baseline model. Specifically, we use\\nthe 𝑃𝑝\\n𝑑(𝑖) as the loss weight, and accordingly the intra-video and\\ninter-video losses are re-written as\\nL\\n′\\n𝑖𝑛𝑡𝑟𝑎= −1\\n𝑀\\n∑︁\\n𝑝𝑖∈𝑃+\\n𝑃𝑝\\n𝑑(𝑖) · log\\nexp(𝑆(𝑝𝑖, 𝑭𝑠)/𝜏)\\nexp( 𝑆(𝑝𝑖,𝑭𝑠)\\n𝜏\\n) +\\n∑︁\\n𝑝𝑗∈𝑃−\\nexp( 𝑆(𝑝𝑗, 𝑭𝑠)\\n𝜏\\n)\\n,\\n(17)\\nL\\n′\\n𝑖𝑛𝑡𝑒𝑟= −\\n1\\n𝑀· 𝐵\\n𝐵\\n∑︁\\n𝑏=0\\n∑︁\\n𝑝𝑏,𝑖∈𝑃+\\n𝑏\\n𝑃𝑝\\n𝑑(𝑖) · log\\nexp(𝑆(𝑝𝑏,𝑖, 𝑭𝑠\\n𝑏)/𝜏)\\nexp(𝑆(𝑝𝑏,𝑖, 𝑭𝑠\\n𝑏)/𝜏) + N .\\n(18)\\n3.5\\nInference\\nIn the inference stage, we first compute the cosine similarity be-\\ntween the video features and language features and then filter out\\n5168\\n\\n\\nMM ’23, October 29–November 3, 2023, Ottawa, ON, Canada.\\nShuo Yang, Zirui Shang, and Xinxiao Wu\\nthe action candidates that do not contain the top-k frames based\\non their similarity scores. After filtering, we rank all the remaining\\naction candidates to determine the most likely one. This method\\nallows us to efficiently identify the action in the video correspond-\\ning to the language query by considering both frame and segment\\nfeatures.\\n4\\nEXPERIMENTS\\n4.1\\nDatasets\\nTo evaluate the proposed method, we conduct experiments on two\\nbenchmark datasets, including the TACoS and the Charades-STA\\ndatasets.\\nThe TACoS dataset is built on the MPII Cooking Compositive\\ndataset [29], which consists of 127 videos with an average length\\nof 4.79 minutes. There are around 148 annotated segments per\\nvideo. The dataset contains 18,818 samples, including 10,146 for\\ntraining, 4,589 for validation, and 4,083 for testing. This dataset is\\nmore challenging due to the long videos and short action segments.\\nThe Charades-STA dataset is built on the Charades dataset [30]\\nand contains 6,672 daily life videos. The average duration of the\\nvideos is 29.76 seconds. There are about 2.4 annotated segments per\\nvideo, whose average duration is 8.2 seconds. The whole dataset\\ncontains 16,128 samples (i.e., pairs of query and action segment),\\nand we follow the standard split of 12,408 and 3,720 samples for\\ntraining and testing.\\n4.2\\nEvaluation Metrics\\nWe adopt two metrics for the performance evaluation: (1)𝑅@𝑛; 𝐼𝑜𝑈≥\\n𝜇, which denotes the recall of top-𝑛predictions at various thresh-\\nolds of the temporal Intersection over Union (IoU). It measures the\\npercentage of predictions that have IoU with ground truth larger\\nthan the threshold 𝜇; (2) mean averaged IoU (mIoU), which de-\\nnotes the average IoU over all the test samples. We set 𝑛= 1 and\\n𝜇∈{0.3, 0.5, 0.7}.\\n4.3\\nImplementation Details\\nWe use C3D [35] for the TACoS dataset and I3D [3] for the Charades-\\nSTA dataset to extract video features. Adam [10] is adopted for\\noptimization with an initial learning rate of 1e-4 and half decaying\\non plateau. The intermediate feature dimension 𝑑is set to 512, and\\nthe head number of multi-head self-attention is set to 8. The hyper-\\nparameters 𝜆1 and 𝜆2 in Eq.(7) are set to 0.2 and 1 for Charades-STA,\\nand 0.6 and 0.8 for TACoS. The loss weights for all loss items are\\nset to 1.\\n4.4\\nAblation Studies\\nWe perform in-depth analysis to evaluate each component of our\\nmethod on the TACoS and Charades-STA datasets.\\nEffectiveness of different distributions. We perform an abla-\\ntion study using the Triangular, Gaussian, and Beta distributions\\nto demonstrate the effectiveness of incorporating different distri-\\nbutions into the baseline model. The results on the TACoS and\\nCharades-STA datasets are shown in Table 1 and Table 2, respec-\\ntively. It is obvious that each of these distributions significantly\\nimproves the performance of the baseline model on the TACoS\\nTable 1: Ablation studies of different distributions on the\\nTACoS dataset.\\nMethods\\n𝑅@1; 𝐼𝑜𝑈≥𝜇\\n𝑚𝐼𝑜𝑈\\n0.3\\n0.5\\n0.7\\nBaseline\\n17.05\\n6.45\\n1.87\\n15.47\\nOurs (Triangular)\\n34.64\\n19.02\\n6.47\\n22.23\\nOurs (Gaussian)\\n35.87\\n19.47\\n6.95\\n22.85\\nOurs (Beta)\\n36.14\\n20.17\\n7.30\\n23.09\\nTable 2: Ablation studies of different distributions on the\\nCharades-STA dataset.\\nMethods\\n𝑅@1; 𝐼𝑜𝑈≥𝜇\\n𝑚𝐼𝑜𝑈\\n0.3\\n0.5\\n0.7\\nBaseline\\n70.4\\n45.05\\n20.03\\n44.30\\nOurs (Triangular)\\n66.94\\n42.63\\n19.22\\n42.67\\nOurs (Gaussian)\\n71.10\\n48.15\\n25.65\\n46.75\\nOurs (Beta)\\n71.72\\n50.13\\n26.72\\n47.35\\ndataset, and similar trends can be seen on the Charades-STA dataset\\nexcept for the Triangular distribution, where the Triangular distri-\\nbution gives small probabilities to the frame near the ground-truth\\ndue to the short videos. Specifically, on the TACoS dataset, the\\nTriangular distribution improves the 𝑅@1; 𝐼𝑜𝑈≥0.3 by 17.59%, the\\nGaussian distribution improves the 𝑅@1; 𝐼𝑜𝑈≥0.3 by 18.82%, and\\nthe Beta distribution achieves the greatest improvement with the\\ngain of 19.09% on the 𝑅@1; 𝐼𝑜𝑈≥0.3. These results demonstrate\\nthat the pseudo-labels with appropriate probabilities provide pos-\\nitive guidance for learning cross-modal alignment and boundary\\nestimation.\\nEffectiveness of the Beta distribution. The asymmetric nature\\nof the Beta distribution makes it well-suited to handle the annotated\\nframes that occur near the action boundaries. To evaluate the effec-\\ntiveness of the Beta distribution, we conduct experiments where\\nthe training videos are re-annotated by placing annotated frames\\nat various positions within the video. For example, the annotated\\nframes are at the first ten percent of the action segment, which we\\ndenote as 0.1 in Table 3. We re-implement ViGA [4] by re-training\\nthe model using the new annotated frames via the open-source\\ncodes for comparison with our method. The comparison results are\\nreported in Table 3.\\nOur results show consistent improvements in 𝑅@1; 𝐼𝑜𝑈≥0.5\\nand 𝑅@1; 𝐼𝑜𝑈≥0.7, regardless of whether the annotation is located\\nin the center of the action (represented as 0.5 in Table 3) or near the\\naction boundaries (represented as 0.1 and 0.9 in Table 3). However,\\nwe observed fewer improvements when the annotations were near\\nthe boundaries, such as at ten (0.1) and ninety (0.9) percentages\\nof the action segment. This can be attributed to the difficulty in\\nestimating the distribution parameters accurately in such situations.\\nWe also noticed a slight decrease (less than 1%) in performance on\\n𝑅@1; 𝐼𝑜𝑈≥0.3, which we believe may be due to the neglect of some\\ninformation when the probabilities for certain video frames are\\nlower. Nonetheless, we are encouraged by the overall effectiveness\\n5169\\n\\n\\nDistribution Based Frame-supervised Language-driven Action Localization\\nMM ’23, October 29–November 3, 2023, Ottawa, ON, Canada.\\nTable 3: Ablation studies of different annotation positions 𝑃\\non the Charades dataset. The value format 𝑎/𝑏in the table\\ndenotes that 𝑎is the result of the re-trained ViGA [4] and 𝑏\\nis the result of our method.\\nP\\n𝑅@1; 𝐼𝑜𝑈≥𝜇\\n𝑚𝐼𝑜𝑈\\n0.3\\n0.5\\n0.7\\n0.1\\n65.59/65.05\\n41.42/43.39\\n20.13/21.64\\n42.44/42.80\\n0.3\\n69.25/70.13\\n46.05/48.60\\n20.73/25.38\\n44.19/46.46\\n0.5\\n71.88/71.53\\n44.97/47.98\\n21.26/23.28\\n45.01/45.93\\n0.7\\n68.39/68.09\\n44.41/45.24\\n19.41/21.34\\n43.12/43.77\\n0.9\\n61.77/61.05\\n36.29/36.99\\n15.32/16.75\\n38.75/38.90\\nTable 4: Ablation studies of different components on the\\nCharades-STA dataset. “r1i3” is the short of 𝑅@1; 𝐼𝑜𝑈≥0.3. “v-\\ns” is the short of visual similarity. “intra” and “inte” represent\\nthe intra-video loss and the inter-video loss, respectively.\\nv-s\\nintra\\ninter\\nr1i3\\nr1i5\\nr1i7\\n𝑚𝐼𝑜𝑈\\n1\\nx\\nx\\nx\\n70.4\\n45.05\\n20.03\\n44.30\\n2\\n✓\\nx\\nx\\n71.32\\n47.77\\n22.5\\n45.44\\n3\\nx\\n✓\\nx\\n70.43\\n47.58\\n24.7\\n46.05\\n4\\nx\\nx\\n✓\\n71.34\\n45.11\\n20.65\\n44.85\\n5\\n✓\\nx\\n✓\\n70.97\\n47.93\\n23.47\\n45.85\\n6\\nx\\n✓\\n✓\\n71.53\\n47.98\\n24.73\\n46.58\\n7\\n✓\\n✓\\nx\\n71.13\\n49.14\\n25.91\\n47.06\\n8\\n✓\\n✓\\n✓\\n71.72\\n50.13\\n26.72\\n47.35\\nof our method, as demonstrated by the consistent improvements in\\nhigher IoU thresholds.\\nEffectiveness of visual similarity. We estimate the distribution\\nparameters by considering both visual similarity and temporal\\ndistance, as shown in Eq.(7). To evaluate the effectiveness of the\\nvisual similarity, we set 𝜆1 = 0 and remove it from the calculation\\nof similarity between frames. The results are shown in Table 4,\\nwhere “v-s\\\" represents the visual similarity. Comparing the results\\nin line 1 and 2 where the frame similarity computed by Eq.(7)\\nis directly used as the probability, and comparing the results in\\nline 6 and 8 where the Beta distribution is used for probability\\ncalculation, we observe improvements of more than 2.5% and 2%\\non 𝑅@1; 𝐼𝑜𝑈≥0.5, respectively, which clearly demonstrates the\\nimportance of the visual similarity.\\nEffectiveness of intra-video loss. The intra-video loss, defined\\nin Eq.(17), enforces similarities between language queries and pos-\\nitive action candidates more than negative action candidates. It\\nis computed in a single language-video pair and helps to rank all\\nthe action candidates, thus improving the accuracy of boundary\\nestimation. To evaluate the effectiveness of the intra-video loss, we\\nremove it for comparison, and the results are shown in Table 4. We\\nobserve that compared with the result in line 1 without the intra-\\nvideo loss, the result in line 3 with the intra-video loss achieves\\nsignificant improvements of more than 4% on 𝑅@1; 𝐼𝑜𝑈≥0.7, and\\nabout 2.5% on 𝑅@1; 𝐼𝑜𝑈≥0.5. This demonstrates that the intra-\\nvideo loss effectively improves the localization accuracy, especially\\nin high-precision scenarios. Similar trends of comparing the results\\nin line 5 and line 8, line 2 and line 7, and line 4 and line 6, further\\nverify the effectiveness of the intra-video loss.\\nEffectiveness of inter-video loss. The inter-video loss, defined\\nin Eq.(18), helps to leverage inter-sample information to learn the\\ndiversities of actions and facilitate model training in the early stage.\\nHowever, it mainly focuses on learning the differences between dif-\\nferent action instances rather than the fine-grained details of action\\nboundaries. To evaluate the effectiveness of the inter-video loss, we\\nremove it for comparison, and the results are shown in Table 4. We\\nobserve that using the inter-video loss (lines 4, 5 and 8) improves\\nthe accuracy by about 1% compared with the results without the\\ninter-video loss (lines 1, 2 and 7). However, this improvement is\\nrelatively small compared to that of the intra-video loss, suggesting\\nthat the intra-video loss is more effective in improving accuracy\\nand localization performance, especially in terms of high precision,\\nwhile the inter-video loss plays a complementary role in improving\\nthe diversity of learned action representations.\\n4.5\\nComparison with State-of-the-art Methods\\nWe compare the proposed method with several state-of-the-art\\nmethods at different levels of supervision, including fully-supervised\\nmethods (CTRL [6], 2D-TAN [48], VSLNet [47]), weakly-supervised\\nmethods (TGA [25], SCN [13], LoGAN [33], CRM [9]), and frame-\\nsupervised methods (ViGA [4], LAS [42]).\\nThe comparison results on the TACoS and Charades-STA datasets\\nare shown in Table 5 and Table 6, respectively. From the results,\\nwe have observations as follows: (1) Compared with the frame-\\nsupervised methods, i.e., ViGA [4], and LAS [42], our method\\nachieves more than 10% improvements on the 𝑅@1; 𝐼𝑜𝑈≥0.3\\nand 𝑅@1; 𝐼𝑜𝑈≥0.5 on the challenging TACoS dataset, and more\\nthan 5% improvements on the 𝑅@1; 𝐼𝑜𝑈≥0.5 and 𝑅@1; 𝐼𝑜𝑈≥0.7\\non the Charades-STA dataset, which demonstrates the effective-\\nness of the proposed distribution-based method on modeling the\\nprobabilities of pseudo-labels, especially in more difficult scenar-\\nios; (2) Compared with the fully-supervised methods shown in the\\nupper parts of Table 5 and Table 6, our method achieves compara-\\nble results, demonstrating the huge potential of the performance\\nof frame-supervised language-driven action localization; (3) Com-\\npared with the weakly supervised methods shown in the middle\\npart of Table 6, our method outperforms all other methods in terms\\nof all metrics by a large margin, showing the superiority of our\\nmethod in the scenario that lacks full annotations. These results\\nsuggest that our method achieves satisfying performance on the\\nTACoS and Charades-STA datasets and is a promising direction for\\nlanguage-driven action localization.\\n4.6\\nQualitative Analysis\\nWe show several examples of action localization results of our\\nmethod and the baseline model on the Charades-STA dataset in\\nFigure 5. From the first three examples in Figure 5 (a), (b), and (c),\\nwe observe that the action boundaries predicted by our method\\nare more accurate than the baseline model since the boundary\\nframes participate in training with appropriate probabilities in our\\nmethod. However, as shown in Figure 5 (d), both our method and\\nthe baseline model fail to locate the action boundaries because\\nthe video frames are too similar to distinguish, showing a lack of\\n5170\\n\\n\\nMM ’23, October 29–November 3, 2023, Ottawa, ON, Canada.\\nShuo Yang, Zirui Shang, and Xinxiao Wu\\nLanguage query: a person kneeling on the floor talks on a phone\\nOurs(Beta)\\nGround Truth\\nBaseline\\n0s\\n8.38s\\n0s\\n8.44s\\n1s\\n10.43s\\n3.30s\\nBeta Distribution \\n𝑃\\nୠ\\n௙\\n𝑇\\n(a)\\n0\\nLanguage query: person throw it on the floor\\nOurs(Beta)\\nGround Truth\\nBaseline\\n4.45s\\n9.85s\\n4.95s\\n8.94s\\n7.99s\\n𝑃\\nୠ\\n௙\\n𝑇\\nBeta Distribution \\n4.39s\\n9.80s\\n0\\n(b)\\n(c)\\n0\\nLanguage query: a person puts food into a sandwich maker\\nOurs(Beta)\\nGround Truth\\nBaseline\\n0s\\n9.09s\\n0s\\n9.20s\\n1s\\n10.18s\\n1.48s\\nBeta Distribution \\n𝑃\\nୠ\\n௙\\n𝑇\\nLanguage query: lastly the person takes a drink from a cup\\nOurs(Beta)\\nGround Truth\\nBaseline\\n12.23s\\n20.54s\\n1.96s\\n10.27s\\n14.65s\\n𝑃\\nୠ\\n௙\\n𝑇\\nBeta Distribution \\n9.80s\\n15.70s\\n(d)\\n0\\nFigure 5: Examples of action localization results. “Baseline” denotes the results from the baseline model; “Ours (Beta)” denotes\\nthe results predicted by the model using the Beta distribution; “Beta Distribution” denotes the curve generated by Eq.(14).\\nTable 5: Comparison with the state-of-the-art methods on\\nthe TACoS dataset. Upper part: Fully-supervised methods;\\nLower part: Frame-supervised methods.\\nMethods\\n𝑅@1; 𝐼𝑜𝑈≥𝜇\\n𝑚𝐼𝑜𝑈\\n0.3\\n0.5\\n0.7\\nCTRL [6]\\n18.32\\n13.3\\n-\\n-\\nTripNet [8]\\n23.95\\n19.17\\n-\\n-\\nABLR [45]\\n19.50\\n9.40\\n-\\n-\\nDEBUG [21]\\n23.45\\n11.72\\n-\\n16.03\\nVSLNet [47]\\n29.61\\n24.27\\n20.03\\n24.11\\n2D-TAN [48]\\n37.29\\n25.32\\n-\\n-\\nViGA [4]\\n19.62\\n8.85\\n3.22\\n15.47\\nLAS [42]\\n23.64\\n10.00\\n3.35\\n17.39\\nOurs (Beta)\\n36.14\\n20.17\\n7.30\\n23.09\\nsufficient fine-grained motion understanding. It is worth noting\\nthat in all four examples in Figure 5, the estimated Beta distributions\\nare reasonable and provide guidance information according to the\\nannotated frame, thereby facilitating the cross-model alignment\\nand boundary estimation during training.\\n5\\nCONCLUSION\\nWe have presented a novel probability distribution based method\\nfor frame-supervised language-driven action localization. By using\\ndistribution functions to model the probabilities of the action frame,\\nas well as the starting and ending boundaries of the target action,\\nour method is able to provide more accurate guidance in learning\\ncross-modal alignment and boundary estimation to compensate\\nTable 6: Comparison with the state-of-the-art methods on the\\nCharades-STA dataset. Upper part: Fully-supervised meth-\\nods; Middle part: Weakly-supervised methods; Lower part:\\nFrame-supervised methods.\\nMethods\\n𝑅@1; 𝐼𝑜𝑈≥𝜇\\n𝑚𝐼𝑜𝑈\\n0.3\\n0.5\\n0.7\\nCTRL [6]\\n-\\n23.63\\n8.89\\n-\\n2D-TAN [48]\\n-\\n39.70\\n23.31\\n-\\nLGI [26]\\n72.96\\n59.46\\n35.48\\n51.38\\nVSLNet [47]\\n70.46\\n54.19\\n35.22\\n50.02\\nTGA [25]\\n32.14\\n19.94\\n8.84\\n-\\nSCN [13]\\n42.96\\n23.58\\n9.97\\n-\\nLoGAN [33]\\n51.67\\n34.68\\n14.54\\n-\\nCRM [9]\\n53.66\\n34.76\\n16.37\\n-\\nLAS [42]\\n60.40\\n39.22\\n20.17\\n39.77\\nViGA [4]\\n71.21\\n45.05\\n20.27\\n44.57\\nOurs (Beta)\\n71.72\\n50.13\\n26.72\\n47.35\\nfor the lack of supervision, thus successfully improving the accu-\\nracy of action localization. Extensive experimental results on two\\nbenchmark datasets demonstrate the effectiveness of our method.\\nWe believe that the distribution-based framework will be a promis-\\ning direction for further research in the field of frame-supervised\\nlanguage-driven action localization.\\nACKNOWLEDGMENTS\\nThis work was supported in part by the Natural Science Foundation\\nof China (NSFC) under Grant No 62072041.\\n5171\\n\\n\\nDistribution Based Frame-supervised Language-driven Action Localization\\nMM ’23, October 29–November 3, 2023, Ottawa, ON, Canada.\\nREFERENCES\\n[1] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell,\\nand Bryan Russell. 2017. Localizing moments in video with natural language. In\\nProceedings of the IEEE international conference on computer vision. 5803–5812.\\n[2] Amy Bearman, Olga Russakovsky, Vittorio Ferrari, and Li Fei-Fei. 2016. What’s\\nthe point: Semantic segmentation with point supervision. In Computer Vision–\\nECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14,\\n2016, Proceedings, Part VII 14. Springer, 549–565.\\n[3] Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new\\nmodel and the kinetics dataset. In proceedings of the IEEE Conference on Computer\\nVision and Pattern Recognition. 6299–6308.\\n[4] Ran Cui, Tianwen Qian, Pai Peng, Elena Daskalaki, Jingjing Chen, Xiaowei Guo,\\nHuyang Sun, and Yu-Gang Jiang. 2022. Video Moment Retrieval from Text\\nQueries via Single Frame Annotation. In Proceedings of the 45th International\\nACM SIGIR Conference on Research and Development in Information Retrieval.\\n1033–1043.\\n[5] Xinpeng Ding, Nannan Wang, Shiwei Zhang, Ziyuan Huang, Xiaomeng Li,\\nMingqian Tang, Tongliang Liu, and Xinbo Gao. 2022. Exploring language hi-\\nerarchy for video grounding. IEEE Transactions on Image Processing 31 (2022),\\n4693–4706.\\n[6] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. 2017. Tall: Temporal\\nactivity localization via language query. In Proceedings of the IEEE international\\nconference on computer vision. 5267–5275.\\n[7] Junyu Gao and Changsheng Xu. 2021. Fast video moment retrieval. In Proceedings\\nof the IEEE/CVF International Conference on Computer Vision. 1523–1532.\\n[8] Meera Hahn, Asim Kadav, James M Rehg, and Hans Peter Graf. 2019. Trip-\\nping through time: Efficient localization of activities in videos. arXiv preprint\\narXiv:1904.09936 (2019).\\n[9] Jiabo Huang, Yang Liu, Shaogang Gong, and Hailin Jin. 2021. Cross-sentence\\ntemporal and semantic relations in video activity localisation. In Proceedings of\\nthe IEEE/CVF International Conference on Computer Vision. 7199–7208.\\n[10] Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic opti-\\nmization. arXiv preprint arXiv:1412.6980 (2014).\\n[11] Juncheng Li, Junlin Xie, Long Qian, Linchao Zhu, Siliang Tang, Fei Wu, Yi Yang,\\nYueting Zhuang, and Xin Eric Wang. 2022. Compositional temporal grounding\\nwith structured variational cross-graph correspondence learning. In Proceedings\\nof the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3032–\\n3041.\\n[12] Zhe Li, Yazan Abu Farha, and Jurgen Gall. 2021. Temporal action segmenta-\\ntion from timestamp supervision. In Proceedings of the IEEE/CVF Conference on\\nComputer Vision and Pattern Recognition. 8365–8374.\\n[13] Zhijie Lin, Zhou Zhao, Zhu Zhang, Qi Wang, and Huasheng Liu. 2020. Weakly-\\nsupervised video moment retrieval via semantic completion network. In Proceed-\\nings of the AAAI Conference on Artificial Intelligence, Vol. 34. 11539–11546.\\n[14] Daizong Liu and Wei Hu. 2022. Skimming, locating, then perusing: A human-like\\nframework for natural language video localization. In Proceedings of the 30th\\nACM International Conference on Multimedia. 4536–4545.\\n[15] Daizong Liu, Xiaoye Qu, Xing Di, Yu Cheng, Zichuan Xu, and Pan Zhou. 2022.\\nMemory-guided semantic learning network for temporal sentence grounding. In\\nProceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 1665–1673.\\n[16] Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou, Yu Cheng, Wei Wei, Zichuan\\nXu, and Yulai Xie. 2021. Context-aware biaffine localizing network for temporal\\nsentence grounding. In Proceedings of the IEEE/CVF Conference on Computer\\nVision and Pattern Recognition. 11235–11244.\\n[17] Daizong Liu, Xiaoye Qu, and Wei Hu. 2022. Reducing the vision and language bias\\nfor temporal sentence grounding. In Proceedings of the 30th ACM International\\nConference on Multimedia. 4092–4101.\\n[18] Daizong Liu, Xiaoye Qu, and Pan Zhou. 2021. Progressively Guide to Attend: An\\nIterative Alignment Framework for Temporal Sentence Grounding. In Proceedings\\nof the 2021 Conference on Empirical Methods in Natural Language Processing. 9302–\\n9311.\\n[19] Daizong Liu, Xiaoye Qu, Pan Zhou, and Yang Liu. 2022. Exploring motion and\\nappearance information for temporal sentence grounding. In Proceedings of the\\nAAAI Conference on Artificial Intelligence, Vol. 36. 1674–1682.\\n[20] Daizong Liu and Pan Zhou. 2023. Jointly visual-and semantic-aware graph\\nmemory networks for temporal sentence localization in videos. In ICASSP 2023-\\n2023 IEEE International Conference on Acoustics, Speech and Signal Processing.\\nIEEE, 1–5.\\n[21] Chujie Lu, Long Chen, Chilie Tan, Xiaolin Li, and Jun Xiao. 2019. Debug: A\\ndense bottom-up grounding approach for natural language video localization. In\\nProceedings of the 2019 Conference on Empirical Methods in Natural Language Pro-\\ncessing and the 9th International Joint Conference on Natural Language Processing.\\n5144–5153.\\n[22] Dezhao Luo, Jiabo Huang, Shaogang Gong, Hailin Jin, and Yang Liu. 2023. To-\\nwards Generalisable Video Moment Retrieval: Visual-Dynamic Injection to Image-\\nText Pre-Training. In Proceedings of the IEEE/CVF Conference on Computer Vision\\nand Pattern Recognition. 23045–23055.\\n[23] Fan Ma, Linchao Zhu, Yi Yang, Shengxin Zha, Gourab Kundu, Matt Feiszli, and\\nZheng Shou. 2020. Sf-net: Single-frame supervision for temporal action local-\\nization. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK,\\nAugust 23–28, 2020, Proceedings, Part IV 16. Springer, 420–437.\\n[24] Pascal Mettes, Jan C Van Gemert, and Cees GM Snoek. 2016. Spot on: Action\\nlocalization from pointly-supervised proposals. In Computer Vision–ECCV 2016:\\n14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Pro-\\nceedings, Part V 14. Springer, 437–453.\\n[25] Niluthpol Chowdhury Mithun, Sujoy Paul, and Amit K Roy-Chowdhury. 2019.\\nWeakly supervised video moment retrieval from text queries. In Proceedings of the\\nIEEE/CVF Conference on Computer Vision and Pattern Recognition. 11592–11601.\\n[26] Jonghwan Mun, Minsu Cho, and Bohyung Han. 2020. Local-global video-text\\ninteractions for temporal grounding. In Proceedings of the IEEE/CVF Conference\\non Computer Vision and Pattern Recognition. 10810–10819.\\n[27] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning\\nwith contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018).\\n[28] Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove:\\nGlobal vectors for word representation. In Proceedings of the 2014 conference on\\nempirical methods in natural language processing. 1532–1543.\\n[29] Marcus Rohrbach, Michaela Regneri, Mykhaylo Andriluka, Sikandar Amin, Man-\\nfred Pinkal, and Bernt Schiele. 2012. Script data for attribute-based recognition\\nof composite activities. In European conference on computer vision. Springer, 144–\\n157.\\n[30] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and\\nAbhinav Gupta. 2016. Hollywood in homes: Crowdsourcing data collection for\\nactivity understanding. In European Conference on Computer Vision. Springer,\\n510–526.\\n[31] Mattia Soldan, Mengmeng Xu, Sisi Qu, Jesper Tegner, and Bernard Ghanem.\\n2021. Vlg-net: Video-language graph matching network for video grounding. In\\nProceedings of the IEEE/CVF International Conference on Computer Vision. 3224–\\n3234.\\n[32] Xin Sun, Xuan Wang, Jialin Gao, Qiong Liu, and Xi Zhou. 2022. You Need to Read\\nAgain: Multi-granularity Perception Network for Moment Retrieval in Videos.\\nIn Proceedings of the 45th International ACM SIGIR Conference on Research and\\nDevelopment in Information Retrieval. 1022–1032.\\n[33] Reuben Tan, Huijuan Xu, Kate Saenko, and Bryan A Plummer. 2021. Logan: Latent\\ngraph co-attention network for weakly-supervised video moment retrieval. In\\nProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.\\n2083–2092.\\n[34] Haoyu Tang, Jihua Zhu, Meng Liu, Zan Gao, and Zhiyong Cheng. 2021. Frame-\\nwise cross-modal matching for video moment retrieval. IEEE Transactions on\\nMultimedia 24 (2021), 1338–1349.\\n[35] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.\\n2015. Learning spatiotemporal features with 3d convolutional networks. In\\nProceedings of the IEEE international conference on computer vision. 4489–4497.\\n[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,\\nAidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all\\nyou need. Advances in neural information processing systems 30 (2017).\\n[37] Hao Wang, Zheng-Jun Zha, Liang Li, Dong Liu, and Jiebo Luo. 2021. Structured\\nmulti-level interaction network for video moment localization via language\\nquery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern\\nRecognition. 7026–7035.\\n[38] Yunxiao Wang, Meng Liu, Yinwei Wei, Zhiyong Cheng, Yinglong Wang, and\\nLiqiang Nie. 2022. Siamese alignment network for weakly supervised video\\nmoment retrieval. IEEE Transactions on Multimedia (2022).\\n[39] Ziyue Wu, Junyu Gao, Shucheng Huang, and Changsheng Xu. 2021. Diving\\ninto the relations: Leveraging semantic and visual structures for video moment\\nretrieval. In 2021 IEEE International Conference on Multimedia and Expo. IEEE,\\n1–6.\\n[40] Zeyu Xiong, Daizong Liu, Pan Zhou, and Jiahao Zhu. 2023. Tracking Objects and\\nActivities with Attention for Temporal Sentence Grounding. In ICASSP 2023-2023\\nIEEE International Conference on Acoustics, Speech and Signal Processing. IEEE,\\n1–5.\\n[41] Zixuan Xu, Banghuai Li, Ye Yuan, and Anhong Dang. 2020. Beta r-cnn: Look-\\ning into pedestrian detection from another perspective. Advances in Neural\\nInformation Processing Systems 33 (2020), 19953–19963.\\n[42] Zhe Xu, Kun Wei, Xu Yang, and Cheng Deng. 2022. Point-Supervised Video\\nTemporal Grounding. IEEE Transactions on Multimedia (2022).\\n[43] Shuo Yang and Xinxiao Wu. 2022. Entity-aware and Motion-aware Transform-\\ners for Language-driven Action Localization. In Proceedings of the Thirty-First\\nInternational Joint Conference on Artificial Intelligence, LD Raedt, Ed. 1552–1558.\\n[44] Yitian Yuan, Lin Ma, Jingwen Wang, Wei Liu, and Wenwu Zhu. 2019. Semantic\\nconditioned dynamic modulation for temporal sentence grounding in videos.\\nAdvances in Neural Information Processing Systems 32 (2019).\\n[45] Yitian Yuan, Tao Mei, and Wenwu Zhu. 2019. To find where you talk: Tempo-\\nral sentence localization in video with attention based location regression. In\\nProceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 9159–9166.\\n5172\\n\\n\\nMM ’23, October 29–November 3, 2023, Ottawa, ON, Canada.\\nShuo Yang, Zirui Shang, and Xinxiao Wu\\n[46] Yawen Zeng, Da Cao, Xiaochi Wei, Meng Liu, Zhou Zhao, and Zheng Qin. 2021.\\nMulti-modal relational graph for cross-modal video moment retrieval. In Pro-\\nceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.\\n2215–2224.\\n[47] Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. 2020. Span-based Lo-\\ncalizing Network for Natural Language Video Localization. In Proceedings of the\\n58th Annual Meeting of the Association for Computational Linguistics. 6543–6554.\\n[48] Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo. 2020. Learning 2d\\ntemporal adjacent networks for moment localization with natural language. In\\nProceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 12870–12877.\\n[49] Yimeng Zhang, Xin Chen, Jinghan Jia, Sijia Liu, and Ke Ding. 2023.\\nText-\\nvisual prompting for efficient 2d temporal video grounding. In Proceedings of the\\nIEEE/CVF Conference on Computer Vision and Pattern Recognition. 14794–14804.\\n[50] Zhu Zhang, Zhijie Lin, Zhou Zhao, Jieming Zhu, and Xiuqiang He. 2020. Regu-\\nlarized two-branch proposal networks for weakly-supervised moment retrieval\\nin videos. In Proceedings of the 28th ACM International Conference on Multimedia.\\n4098–4106.\\n5173\\n\\n\\n1\\n2\\n3\\n4\\n5\\n6\\n7\\n8\\n9\\n10\\n11\\n12\\n13\\n14\\n15\\n16\\n17\\n18\\n19\\n20\\n21\\n22\\n23\\n24\\n25\\n26\\n27\\n28\\n29\\n30\\n31\\n32\\n33\\n34\\n35\\n36\\n37\\n38\\n39\\n40\\n41\\n42\\n43\\n44\\n45\\n46\\n47\\n48\\n49\\n50\\n51\\n52\\n53\\n54\\n55\\n56\\n57\\n58\\n59\\n60\\n61\\n62\\n63\\n64\\n65\\n66\\n67\\n68\\n69\\n70\\n71\\n72\\n73\\n74\\n75\\n76\\n77\\n78\\n79\\n80\\n81\\n82\\n83\\n84\\n85\\n86\\n87\\n88\\n89\\n90\\n91\\n92\\n93\\n94\\n95\\n96\\n97\\n98\\n99\\n100\\n101\\n102\\n103\\n104\\n105\\n106\\n107\\n108\\n109\\n110\\n111\\n112\\n113\\n114\\n115\\n116\\nExplicit Granularity and Implicit Scale Correspondence Learning\\nfor Point-Supervised Video Moment Localization\\nAnonymous Authors\\nABSTRACT\\nVideo moment localization (VML) aims to identify the temporal\\nboundary of the target moment semantically matching the given\\nquery. Existing approaches fall into three paradigms: fully-supervised,\\nweakly-supervised, and point-supervised. Compared to other two\\nparadigms, point-supervised VML strikes a balance between lo-\\ncalization accuracy and annotation cost. However, it is still in its\\ninfancy due to the following two challenges: explicit granularity\\nalignment and implicit scale perception, especially when facing\\ncomplex cross-modal correspondences. To this end, we propose a\\nSemantic Granularity and Scale Correspondence Integration (SG-\\nSCI) framework aimed at modeling the semantic alignment between\\nvideo and text, leveraging limited single-frame annotation infor-\\nmation for correspondence learning. It explicitly models semantic\\nrelations of different feature granularities and adaptively mines\\nthe implicit semantic scale, thereby enhancing and utilizing modal\\nfeature representations of varying granularities and scales. SG-SCI\\nemploys a granularity correspondence alignment module to align\\nsemantic information by leveraging latent prior knowledge. Then\\nwe develop a scale correspondence learning strategy to identify and\\naddress semantic scale differences. Extensive comparison experi-\\nments, ablation studies, and necessary hyperparameter analyses on\\nbenchmark datasets have demonstrated the promising performance\\nof our model over several state-of-the-art competitors.\\nCCS CONCEPTS\\n• Information systems →Novelty in information retrieval;\\nMultimedia and multimodal retrieval.\\nKEYWORDS\\nCross-modal Moment Localization; Cross-Modal Retrieval; Corre-\\nspondence Learning\\nACM Reference Format:\\nAnonymous Authors. 2024. Explicit Granularity and Implicit Scale Cor-\\nrespondence Learning for Point-Supervised Video Moment Localization.\\nIn Proceedings of the 32nd ACM International Conference on Multimedia\\n(MM’24), October 28-November 1, 2024, Melbourne, Australia. ACM, New\\nYork, NY, USA, 10 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn\\nUnpublished working draft. Not for distribution.\\nPermission to make digital or hard copies of all or part of this work for personal or\\nclassroom use is granted without fee provided that copies are not made or distributed\\nfor profit or commercial advantage and that copies bear this notice and the full citation\\non the first page. Copyrights for components of this work owned by others than the\\nauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or\\nrepublish, to post on servers or to redistribute to lists, requires prior specific permission\\nand/or a fee. Request permissions from permissions@acm.org.\\nACM MM, 2024, Melbourne, Australia\\n© 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM.\\nACM ISBN 978-x-xxxx-xxxx-x/YY/MM\\nhttps://doi.org/10.1145/nnnnnnn.nnnnnnn\\nQuery : A man poses for a picture in front of them\\n…\\n…\\nVideo\\nFully-supervised\\nAnnotation:[98.31,103.34]\\nPoint-supervised\\nAnnotation:[101.16]\\nWeakly-supervised\\nAnnotation:[-]\\n98.31\\n103.34\\n101.16\\n(a) Video Moment Localization\\n(b) Explicit Granularity Alignment and Implicit Scale Perception\\n…\\n…\\nGT\\nFine-grained \\nIrrelevant\\nQuery\\nCoarse-grained \\nMismatch\\n…\\n…\\nQuery1\\nQuery2\\nSemantic \\nInclusion\\nGT2\\nSF\\nPM\\nGT1\\nThe man adds \\npepper to the egg\\nThe man is \\npreparing to cook\\nA woman is playing music, and a little boy walks by\\nVideo\\nFigure 1: (a) Illustrative examples of Video Moment Localiza-\\ntion (VML). (b) Illustration of explicit granularity alignment\\nand implicit scale perception among VML. GT: Ground Truth;\\nPM: Prediction Moment; SF: Supervised Frame. (Left) Fine-\\ngrained word-frame irrelevance in green and coarse-grained\\nsentence-moment mismatch in blue. (Right) Illustration of\\nthe semantic inclusion relationships between textual and\\nvisual correspondences.\\n1\\nINTRODUCTION\\nVideo moment localization (VML), which refers to localizing a\\nvisual moment corresponding to a given textual query, is a funda-\\nmental task in video understanding. It benefits many important\\napplication scenarios, such as multimedia retrieval [45, 48] and\\nsmart human–computer interaction [37, 43].\\nCurrent efforts [14, 44, 50, 53] focus on addressing both fully\\nand weakly supervised VML paradigms. As shown in Figure 1(a),\\nthe fully-supervised method requires the ground-truth moment\\nannotations for training, which is laborious and time-consuming to\\nobtain [29]. The weakly-supervised method is more flexible because\\nit does not require moment annotations, but results in a significant\\ndrop in performance [10]. To balance accuracy with annotation\\ncost, the point-supervised paradigm is proposed [21, 25]. Point-\\nsupervised methods require a single frame annotation from the\\nground truth, which is more practical and flexible compared to fully\\nsupervised annotation, costing only 1/6 as much [24].\\nPrior research [2, 4, 18, 35], to enhance video-language compre-\\nhension, can be classified into two primary approaches: granularity\\nmodeling and scale modeling. Granularity modeling involves delin-\\neating relationships among various features (such as frame-word\\nand moment-sentence) through the explicit boundaries present in\\nvideo-language, facilitating the differentiation of data across vary-\\ning entity granularities. In contrast, scale modeling derives from\\nthe intricate semantic logics and inherent hierarchical structures\\nwithin textual and visual domains. This approach acknowledges\\n\\n\\n117\\n118\\n119\\n120\\n121\\n122\\n123\\n124\\n125\\n126\\n127\\n128\\n129\\n130\\n131\\n132\\n133\\n134\\n135\\n136\\n137\\n138\\n139\\n140\\n141\\n142\\n143\\n144\\n145\\n146\\n147\\n148\\n149\\n150\\n151\\n152\\n153\\n154\\n155\\n156\\n157\\n158\\n159\\n160\\n161\\n162\\n163\\n164\\n165\\n166\\n167\\n168\\n169\\n170\\n171\\n172\\n173\\n174\\nACM MM, 2024, Melbourne, Australia\\nAnonymous Authors\\n175\\n176\\n177\\n178\\n179\\n180\\n181\\n182\\n183\\n184\\n185\\n186\\n187\\n188\\n189\\n190\\n191\\n192\\n193\\n194\\n195\\n196\\n197\\n198\\n199\\n200\\n201\\n202\\n203\\n204\\n205\\n206\\n207\\n208\\n209\\n210\\n211\\n212\\n213\\n214\\n215\\n216\\n217\\n218\\n219\\n220\\n221\\n222\\n223\\n224\\n225\\n226\\n227\\n228\\n229\\n230\\n231\\n232\\nimplicit semantic scale in video and language interaction that\\nexists within an alternative modal space.\\nTo be specific, as shown in Figure 1(b), existing point-supervised\\nlabeling inevitably introduces two accompanied challenges: 1) Ex-\\nplicit Granularity Alignment. According to [10, 12], on the one\\nhand, single-frame annotations are randomized within intervals,\\ncausing clear mismatches at a fine-grained level (frame-word); on\\nthe other hand, the incomplete information on interval annotations\\nhinders the direct use of correspondences at a coarse-grained level\\n(moment-query). Undoubtedly, this challenge poses a significant\\nobstacle to effective language and visual alignment. 2) Implicit\\nScale Perception. Prior studies [9, 16] note that query sentences\\nfor the same video may vary in semantic scale. This issue is espe-\\ncially noticeable in the point-supervised approach. These methods\\nstruggle to capture temporal relationships within a video because\\nof using only single-frame annotations, leading to a lack of effective\\ncontextual modeling constraints. As a result, the essential semantic\\nscale information remains unmodeled, impeding temporal learning\\nand video comprehension.\\nTo navigate these challenges, we introduce a novel Semantic\\nGranularity and Scale Correspondence Integration (SG-SCI) frame-\\nwork that integrates a Granularity Correspondence Alignment (GCA)\\nmodule and a Scale Correspondence Learning (SCL) strategy. Firstly,\\nthe GCA module is engineered to enhance the interaction of video-\\nlanguage. By employing a fine-grained alignment approach, it es-\\ntablishes a more detailed and comprehensive mapping between\\nvideo frames and textual descriptions. This module not only facili-\\ntates a deeper understanding of the video content but also ensures\\nthat even the less prominent frames find relevance in the corre-\\nsponding textual narrative. Secondly, the SCL strategy addresses\\nthe disparity in temporal scale. It is designed to learn latent seman-\\ntics adaptively across varying temporal scales, enabling the model\\nto assimilate and correlate extensive video sequences with succinct\\ntextual queries. This strategy ensures a more robust and contextu-\\nally aware matching process, thereby enhancing the accuracy and\\nadaptability. Consequently, this framework offers a more nuanced\\nand adaptable solution for point-supervised VML.\\nTo the best of our knowledge, it is the first work on integrating\\nexplicit granularity alignment and implicit scale perception into\\npoint-supervised VML. Our approach effectively mitigates the gran-\\nularity and scale-related challenges by the synergy of the innovative\\ncomponent and strategy. It boosts the precision of aligning video\\nmoments to text queries and enhances the model’s robustness.\\nIn summary, the main contributions are as follows:\\n• Model Contribution. We introduce an innovative Granular\\nCorrespondence Alignment module. Specifically, it is designed\\nto improve the explicit correspondence relation across varying\\ngranularities among different modalities.\\n• Strategy Contribution. Under the framework of point-supervised,\\nwe develop a Scale Correspondence Learning strategy, which is\\npivotal in capturing the implicit semantic scale in correspondence\\nlearning.\\n• Experimental Contribution. Extensive experiments on two\\nbenchmarks, i.e., Charades-STA [12] and TACoS [33], validate the\\neffectiveness and superiority of our model. The codes and settings\\nare released at https://anonymous.4open.science/r/SG-SCI.\\n2\\nRELATED WORK\\n2.1\\nVideo-Language Modeling\\nCurrent research can be divided into two main categories depending\\non the use of a modeling approach: feature granularity modeling\\nand semantic scale modeling. In terms of entities, the former is\\nexplicit while the latter is implicit.\\nEarly explicit modeling approaches [1, 12] mainly used global\\nsentence-level alignment to enhance the semantics of visual fea-\\ntures, but this approach overlooked fine-grained semantics such\\nas words. As a result, several studies [4, 35, 49] have investigated\\nvarious word-frame interactions using attention mechanisms to\\ncapture the relationships between visual cues and textual queries.\\nRecent research [38, 47, 52] has recognized the significance of local\\nphrase patterns or tokens in sentences for video moment retrieval.\\nCompared to explicit modeling, implicit scale modeling [2, 11,\\n18, 19, 39] utilizes multi-scale relations to achieve better grounding\\nresults. These relations mainly exist at the video-level and language-\\nlevel. However, all grounding methods overlook the possibility of\\ndifferent semantic scales within the same level (e.g., sentence level).\\nAccordingly, DualMIL [9] extends multiple instance learning into a\\ntwo-level framework.\\nAlthough the aforementioned modeling strategies are effective,\\nmost methods do not fully utilize their potential when dealing\\nwith limited supervisory information. Therefore, our focus is on\\nextracting both implicit and explicit information to accurately locate\\nboundaries in point-supervised VML.\\n2.2\\nVideo-Language Correspondence Learning\\nIn the domain of video-language learning, the misalignment be-\\ntween textual descriptions and corresponding visual content intro-\\nduces significant challenges, necessitating the adoption of corre-\\nspondence learning as an innovative approach [15, 17, 20]. This\\nparadigm shift is epitomized by the introduction of MIL-NCE [31],\\nwhich pioneers the strategy of aligning video clips with adjacent\\nsentences to diminish the effects of misalignment. In contrast,\\nTang et al. [40] improves video description quality by integrat-\\ning an external image captioning model, emphasizing data-driven\\nenhancement over immediate error correction.\\nOur method differs itself from preceding endeavors through\\ntwo principal innovations. Firstly, we go beyond the traditional\\nfocus on either model architectures or supervisory strategies by\\nenhancing improvements through the synergistic interaction be-\\ntween model and strategy learning. Secondly, we expand the use\\nof correspondence learning beyond segmented multi-modal data.\\nThis pioneering adaptation allows for precise moment localization\\nwith point-supervised for the first time.\\n2.3\\nVideo Moment Localization\\nDifferent from existing fully and weakly-supervised VML stud-\\nies [12, 14, 27, 30, 44, 50, 53], point-supervised VML strikes a deli-\\ncate balance between annotation efforts and model performance by\\nleveraging a single frame from the localization moment [10, 21, 25].\\nCompared to fully-supervised VML, point-supervised significantly\\nreduces the cost of annotation data and it provides more comprehen-\\nsive information than weakly supervised learning. By eliminating\\n\\n\\n233\\n234\\n235\\n236\\n237\\n238\\n239\\n240\\n241\\n242\\n243\\n244\\n245\\n246\\n247\\n248\\n249\\n250\\n251\\n252\\n253\\n254\\n255\\n256\\n257\\n258\\n259\\n260\\n261\\n262\\n263\\n264\\n265\\n266\\n267\\n268\\n269\\n270\\n271\\n272\\n273\\n274\\n275\\n276\\n277\\n278\\n279\\n280\\n281\\n282\\n283\\n284\\n285\\n286\\n287\\n288\\n289\\n290\\nExplicit Granularity and Implicit Scale Correspondence Learning for Point-Supervised Video Moment Localization\\nACM MM, 2024, Melbourne, Australia\\n291\\n292\\n293\\n294\\n295\\n296\\n297\\n298\\n299\\n300\\n301\\n302\\n303\\n304\\n305\\n306\\n307\\n308\\n309\\n310\\n311\\n312\\n313\\n314\\n315\\n316\\n317\\n318\\n319\\n320\\n321\\n322\\n323\\n324\\n325\\n326\\n327\\n328\\n329\\n330\\n331\\n332\\n333\\n334\\n335\\n336\\n337\\n338\\n339\\n340\\n341\\n342\\n343\\n344\\n345\\n346\\n347\\n348\\nA little boy \\nholding a \\nyellow ball \\nwalks by.\\n(a) Feature Extraction\\nVideo\\nEncoder\\nText\\nEncoder\\n(b) Multi-modal Interaction\\nGranularity Correspondence \\nAlignment\\n(c) Moment Localization\\n(d) Granularity Correspondence Alignment\\n(e) Scale Correspondence Learning\\nTraining\\nInferencing\\nMoment to Frame\\nFrame to Moment\\n1\\n2\\n3\\n4\\nT\\n…\\nPositional \\nEmbedding\\nSelf-Attention Block\\nSelf-Attention Block\\nboy\\nyellow\\nball\\nwalks\\nCross-Attention Block\\n…\\nStep 1\\nStep 2\\n…\\nPooling\\nTemporal Weight\\nSemantic Score\\nGaussian\\nDistribution\\n…\\np ∼N\\n!\\nµf, σ2\\nf\\n\\\"\\nGranularity Perception Matrix\\nyellow\\nboy\\nwalks\\nball\\n𝑠!! 𝑠!\\\"\\n𝑠#!\\n𝑠#\\\"\\n𝑠$! 𝑠$\\\"\\n𝑠!%\\n𝑠#%\\n𝑠$%\\n…\\n…\\n…\\n…\\n…\\n…\\n…\\n…\\n…\\n…\\n…\\n1\\n2\\n𝑇\\n…\\nA little boy \\nholding a \\nyellow ball \\nwalks by.\\nyellow\\nboy\\nwalks\\nball\\n𝑇\\nSentence-Frame\\nAdaptive Granularity Balancing\\n…\\nMean Weighted\\n×𝛼\\n×(1 −𝛼)\\nOutput\\nInput\\nGranularity Aggregation\\nGlobal Perception\\n…\\n…\\n…\\n…\\n…\\n𝑚!\\n\\\"\\n𝑚!#$\\n\\\"\\n𝑚%\\n#\\n𝑚%#$\\n#\\n0\\n0\\n0\\n1\\n1\\n…\\n…\\n…\\n𝑙!\\n𝑙\\\"\\n𝑙\\\"#!\\n𝑙$#!\\n𝑙$\\np ∼N\\n!\\nµf, σ2\\nf\\n\\\"\\n0\\n0\\n0\\n  𝜅$\\n  𝜅$#!\\n…\\n…\\n…\\nLabel\\nKL Divergence\\n…\\n…\\n…\\n𝑚!\\n\\\"\\n𝑚!#$\\n\\\"\\n𝑚%\\n#\\n𝑚%#$\\n#\\nKL Divergence\\nelement dot product\\nmatrix multiplication\\nelement addition\\nelement average\\n…\\n…\\n…\\n1\\n2\\n3\\n4\\nT\\n…\\n𝒂𝒗𝒈\\n𝒂𝒗𝒈\\n𝑭\\n⋅\\n⋅\\n⋅\\n…\\n3\\n𝑇−1\\n⋅\\nFigure 2: Schematic illustration of the proposed SG-SCI model. We first perform (a) Feature Extraction on both video and\\nlanguage separately, followed by (b) Multi-modal Interaction, which combines the obtained cross-attention features and\\nsimilarity information, feeding them into the (d) Granularity Correspondence Alignment module. After the interaction, the\\nfeatures are input into the (c) Moment Localization module. During the training phase, we apply a (e) Scale Correspondence\\nLearning strategy to compare semantics at diverse scales.\\nthe need for precise start and end timestamps for the target moment,\\na quick \\\"glance\\\" at the video and the selection of a single frame are\\noften sufficient, making point-supervised methods applicable to sce-\\nnarios with incomplete annotation information [46]. The concept\\nof point-supervised was initially introduced by Bearman et al. [3]\\nin the context of semantic segmentation tasks and Cui et al. [10]\\napplied the concept of point-supervised to VML and introduced the\\nViGA model, which aligns Gaussian distributions generated from\\nsupervised frame positions with cross-modal attention. Subsequent\\nworks have built upon ViGA’s foundation, D3G [25] is proposed\\nto align features of sentence-moment pairs and dynamically miti-\\ngate annotation bias using Gaussian distributions, while CFMR [21]\\ndevelops a concept-driven multi-modal alignment mechanism to\\ncircumvent the need for cross-modal interaction modules during\\nthe inference process.\\nDespite efforts to enhance retrieval performance, existing work\\nstill falls short in effectively modeling the semantic granularity\\nrelationship between two modalities. Furthermore, due to the con-\\nstraints of the supervised information, it struggles to capture the\\noverall action changes within a moment.\\n3\\nMETHODOLOGY\\nThis section commences by defining the problem and outlining the\\ncore pipeline of point-supervised video moment localization. After\\nthis, we will explore the insights behind our innovative Granularity\\nCorrespondence Alignment method, highlighting its main elements.\\nFinally, we conclude with an exposition of our Scale Correspondence\\nLearning strategy, showing its significance in our research. Figure 2\\npresents an overview of our proposed SG-SCI.\\n3.1\\nProblem Definition\\nIn this work, we aim to address the challenge of cross-modal video\\nlocalization under point-supervised. The dataset comprises a series\\nof quadruples and is denoted as D, which is expressed as follows,\\nD = {(V\\n𝑖, Q𝑖,𝜏𝑠\\n𝑖,𝜏𝑒\\n𝑖) | 𝑖= 1 to 𝑁},\\n(1)\\nwhere V\\n𝑖represents an untrimmed video, Q𝑖denotes the associ-\\nated query sentence, and 𝜏𝑠\\n𝑖, 𝜏𝑒\\n𝑖are the start and end timestamps\\nrespectively. The target video moment aligns with the query Q𝑖.\\nContrasting with a fully-supervised paradigm, our training is\\nbased on a collection of triplet annotations, symbolized as A =\\n{(V\\n𝑖, Q𝑖,𝜏𝑚\\n𝑖) | 𝑖= 1 to 𝑁}, where 𝜏𝑚\\n𝑖\\nsignifies a randomly selected\\npoint within the interval defined by 𝜏𝑠\\n𝑖and 𝜏𝑒\\n𝑖. During the infer-\\nence phase, the objective is to precisely localize the relevant video\\nmoment for each query within the dataset D.\\n3.2\\nPipeline of Point-supervised VML\\nTo fulfil the task of Point-supervised VML, we first introduce the\\npipeline which can be categorized into three parts: Featurn Extrac-\\ntion, Multi-modal Interaction, and Moment Localization.\\n3.2.1\\nFeature Extraction. Before multi-modal interaction, we ex-\\ntract the visual and textual features, and then incorporate learnable\\npositional embedding into the extracted features.\\nVisual Representation. Given an untrimmed video V, our\\napproach utilizes a pre-trained CNN, i.e., C3D [41] or I3D [6], to\\nextract visual features to ensure fair comparison. The extracted\\nfeatures are then processed through a fully-connected (FC) layer,\\nobtaining a visual representation denoted as 𝑉= {𝑣1, 𝑣2, . . . , 𝑣𝑇},\\n\\n\\n349\\n350\\n351\\n352\\n353\\n354\\n355\\n356\\n357\\n358\\n359\\n360\\n361\\n362\\n363\\n364\\n365\\n366\\n367\\n368\\n369\\n370\\n371\\n372\\n373\\n374\\n375\\n376\\n377\\n378\\n379\\n380\\n381\\n382\\n383\\n384\\n385\\n386\\n387\\n388\\n389\\n390\\n391\\n392\\n393\\n394\\n395\\n396\\n397\\n398\\n399\\n400\\n401\\n402\\n403\\n404\\n405\\n406\\nACM MM, 2024, Melbourne, Australia\\nAnonymous Authors\\n407\\n408\\n409\\n410\\n411\\n412\\n413\\n414\\n415\\n416\\n417\\n418\\n419\\n420\\n421\\n422\\n423\\n424\\n425\\n426\\n427\\n428\\n429\\n430\\n431\\n432\\n433\\n434\\n435\\n436\\n437\\n438\\n439\\n440\\n441\\n442\\n443\\n444\\n445\\n446\\n447\\n448\\n449\\n450\\n451\\n452\\n453\\n454\\n455\\n456\\n457\\n458\\n459\\n460\\n461\\n462\\n463\\n464\\nwhere 𝑇represents the number of frames sampled at intervals in\\nthe video and 𝑣𝑖denotes the 𝑖-th frame.\\nTo perceive the time of the video, we incorporate learnable Po-\\nsitional Embedding (PE) into the model and get enhanced visual\\nrepresentation ¯\\n𝑉∈R𝑇×𝑑, where 𝑑represents the dimension of the\\nvisual representation.\\nText Representation. For a given query Q, we employ the pre-\\ntrained Glove model [32] to extract textual features. A bi-directional\\ngated recurrent unit (Bi-GRU) is employed to obtain the latent se-\\nquential order of original sentence, and get new textual representa-\\ntion 𝑄= {𝑞1,𝑞2, . . . ,𝑞𝐿}, where 𝐿indicating the number of words\\nin the query. Similar to the visual representation, PE is also added to\\nthe text representation, obtaining the enhanced features ¯\\n𝑄∈R𝐿×𝑑.\\n3.2.2\\nMulti-modal Interaction. To capture the intra-model seman-\\ntics, we utilize ¯\\n𝑉and ¯\\n𝑄through the self-attention layer [42] to\\ngenerate the intra-modal representation ˆ\\n𝑉and ˆ\\n𝑄. Similarly, we use\\nthe cross-modal multi-head attention to obtain the cross-modal\\nrepresentation ˜\\n𝑉= 𝐴𝑡𝑡𝑛( ˆ\\n𝑄, ˆ\\n𝑉) and ˜\\n𝑄= 𝐴𝑡𝑡𝑛( ˆ\\n𝑉, ˆ\\n𝑄), where 𝐴𝑡𝑡𝑛(·)\\nis formulated as follows,\\n𝐴𝑡𝑡𝑛( ˆ\\n𝑄, ˆ\\n𝑉) = softmax(\\nˆ\\n𝑄ˆ\\n𝑉𝑇\\n√\\n𝑑\\n) ˆ\\n𝑉,\\n𝐴𝑡𝑡𝑛( ˆ\\n𝑉, ˆ\\n𝑄) = softmax(\\nˆ\\n𝑉ˆ\\n𝑄𝑇\\n√\\n𝑑\\n) ˆ\\n𝑄.\\n(2)\\nSubsequently, to serve the subsequent modules, we take out\\nthe last layer of cross-modal attention scores as the cross-modal\\nattention scores 𝐴𝑞→𝑣and 𝐴𝑣→𝑞, which will be further illustrated\\nin 3.3. We input the features and attention scores into our proposed\\ngranularity correspondence alignment module to obtain the multi-\\ngranularity representation ˜\\n𝑉𝑓and ˜\\n𝑄𝑤:\\n˜\\n𝑉𝑓= GCA( ˜\\n𝑉, ˜\\n𝑄,𝐴𝑞→𝑣),\\n˜\\n𝑄𝑤= GCA( ˜\\n𝑄, ˜\\n𝑉,𝐴𝑣→𝑞),\\n(3)\\nwhere GCA(·) is the granularity correspondence alignment mod-\\nule. So far, we have obtained the multi-granularity representa-\\ntion ˜\\n𝑉𝑓and ˜\\n𝑄𝑤via multi-modal interaction. The representation is\\ncompound of multi-level granularities information, assisting cross-\\nmodal semantic alignment. In order to integrate the semantic in-\\nformation of the whole sentence, we use the max-pooling function\\nF(·) to obtain the sentence representation ˜\\n𝑄𝑠as follows,\\n˜\\n𝑄𝑠= F( ˜\\n𝑄𝑤).\\n(4)\\n3.2.3\\nMoment Localization. Moment localization aims to locate\\nthe start and end timestamps based on the obtained representation.\\nIn the training stage, unlike fully-supervised scenario, a random\\npoint within the target video moment is accessed while it is absent\\nduring inference. Because of the divergence between the two stages,\\nwe will elaborate on both of them subsequently.\\nTraining Stage. In the training process, we use sliding windows\\nto slice ˜\\n𝑉𝑓, and generate candidate moments. Gaussian distribution\\nis utilized to measure the moments’ temporal features and cosine\\nsimilarity of semantic features. The temporal information is defined\\nas follows,\\n𝐺𝑖=\\n1\\n√\\n2𝜋𝜎\\nexp\\n©\\n­\\n­\\n«\\n−\\n\\u0010\\n(𝑖−𝜏𝑚) ·\\n2\\n𝐿𝑣−1\\n\\u00112\\n2𝜎2\\nª\\n®\\n®\\n¬\\n,\\n(5)\\nwhere 𝑖is the 𝑖-th frame, 𝜏𝑚is the index of the supervised frame, 𝜎\\nis a hyperparameter, and 𝐿𝑣is the length of the video. The semantic\\ninformation 𝑆𝑖is the cosine similarity between ˜\\n𝑉𝑓(𝑖) and ˜\\n𝑄𝑠.\\nThese temporal and semantic analyses are integrated and applied\\nin our scale correspondence learning strategy, detailed in Sec. 3.4.\\nInference Stage. The inference involves two primary steps,\\nconsisting of identifying the key point that best matches query and\\nexpand from this point to get the predicted video moment most\\nsimilar to the query.\\n3.3\\nGranularity Correspondence Alignment\\nIn this section, we introduce a novel granularity correspondence\\nalignment module within the framework of point-supervised VML,\\nwhich is designed to adaptively regulate the granularity relation-\\nship between different modalities. By exploring semantic relation-\\nships across modalities, the module enriches the original modal\\nrepresentation and supplements existing semantic alignment with\\ninformation across various granularities, ensuring symmetry in\\nintermodal interactions.\\nUnlike existing methods that unconsciously interact the small-\\nest granularity units (frames and words) in representations across\\nmodalities, our main insights derive from exploiting the attention-\\nperceived matrix and utilizing it to capture potential cross-modal\\nsemantic granularity information. Specifically, we use the matrix\\nconsisting of cross-modal attention scores as a granularity percep-\\ntion matrix as follows,\\n𝐴𝑞→𝑣=\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n𝑠11\\n𝑠12\\n· · ·\\n𝑠1𝐿\\n· · ·\\n· · ·\\n· · ·\\n· · ·\\n𝑠𝑖1\\n𝑠𝑖2\\n· · ·\\n𝑠𝑖𝐿\\n· · ·\\n· · ·\\n· · ·\\n· · ·\\n𝑠𝑇1\\n𝑠𝑇2\\n· · ·\\n𝑠𝑇𝐿\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n,\\n(6)\\nwhere 𝐴𝑞→𝑣∈R𝑇×𝐿is the cross-modal attention score matrix, 𝑇\\nand 𝐿are the number of frames and words, respectively, and 𝑠𝑖𝑗is\\nthe attention score between the 𝑖-th frame and the 𝑗-th word. The\\ndifferences of scores in each column represent how much attention\\nthe word pays to the video frame. Therefore, we consider using\\nmax function to aggregate the information in each row to obtain\\nthe potential prior distribution ˜\\n𝑉𝑝∈R𝑇of the complete query with\\nrespect to the video frames, formulated as,\\n˜\\n𝑉𝑝= max(𝐴𝑞→𝑣) = [max(𝑠1·), max(𝑠2·), . . . , max(𝑠𝑇·)]𝑇.\\n(7)\\nNext, we calculate the cosine similarity between the query and\\nthe original video frames to obtain the prior distribution of the\\ncomplete query with respect to the video frames,\\n˜\\n𝑉𝑔= 𝑐𝑜𝑠( ˜\\n𝑉, ˜\\n𝑄𝑠),\\n(8)\\nwhere 𝑐𝑜𝑠(·) is the cosine similarity function. Subsequently, we use\\nthe finest granularity information from attentional perception and\\npooled global granularity information, which are averaged and used\\nas adaptive visual granularity perceptual features ˜\\n𝑉𝑐= 1\\n2 ( ˜\\n𝑉𝑔+ ˜\\n𝑉𝑝),\\nIn this way, we can obtain the visual granularity perception vector\\n\\n\\n465\\n466\\n467\\n468\\n469\\n470\\n471\\n472\\n473\\n474\\n475\\n476\\n477\\n478\\n479\\n480\\n481\\n482\\n483\\n484\\n485\\n486\\n487\\n488\\n489\\n490\\n491\\n492\\n493\\n494\\n495\\n496\\n497\\n498\\n499\\n500\\n501\\n502\\n503\\n504\\n505\\n506\\n507\\n508\\n509\\n510\\n511\\n512\\n513\\n514\\n515\\n516\\n517\\n518\\n519\\n520\\n521\\n522\\nExplicit Granularity and Implicit Scale Correspondence Learning for Point-Supervised Video Moment Localization\\nACM MM, 2024, Melbourne, Australia\\n523\\n524\\n525\\n526\\n527\\n528\\n529\\n530\\n531\\n532\\n533\\n534\\n535\\n536\\n537\\n538\\n539\\n540\\n541\\n542\\n543\\n544\\n545\\n546\\n547\\n548\\n549\\n550\\n551\\n552\\n553\\n554\\n555\\n556\\n557\\n558\\n559\\n560\\n561\\n562\\n563\\n564\\n565\\n566\\n567\\n568\\n569\\n570\\n571\\n572\\n573\\n574\\n575\\n576\\n577\\n578\\n579\\n580\\n˜\\n𝑉𝑐, which can be used to enhance the original visual representation.\\nOn this basis, we use the obtained features to remodel the cross-\\nmodal fusion with text feature,\\n˜\\n𝑉𝑎= ˜\\n𝑉𝑐⊙˜\\n𝑉𝑓.\\n(9)\\nAfter reaggregation, we integrate self-attention-based fine-grained\\nfeatures with multi-scale visual perception features. This enhances\\nfine-grained entity alignment and focuses on overall perception,\\nthereby improving representation. We then apply these features for\\ncross-modal semantic interactions, using prior semantic knowledge\\nto refine visual-word correlations.\\nFinally, we use the weighted mean approach in order to control\\nthe original modal feature information, thus preventing noise inter-\\nference due to excessive introduction of cross-modal information\\nas follows,\\n˜\\n𝑉𝑓= 𝛼˜\\n𝑉+ (1 −𝛼) ˜\\n𝑉𝑎,\\n(10)\\nwhere 𝛼is the mean weighted factor. We use the same strategy\\napplied to another modality. Therefore, we can obtain the final cross-\\nmodal representation ˜\\n𝑉𝑓and ˜\\n𝑄𝑤via the multi-modal interaction.\\n3.4\\nScale Correspondence Learning\\nTo improve the model’s understanding ability under different scales\\nof moments, we utilize potential frame-moment correspondence\\nsemantic information, as the single-frame supervised information\\nalone is not sufficient for effective moment semantics. To be specific,\\nthe target of optimization consists of three parts, the first part\\nmodels the semantic and temporal information of the fused features\\nfrom a global perspective. The second part uses point annotation\\nto compare the differences between potential positive and negative\\nsamples in different intervals to effectively capture the information\\nabout the change of actions within the intervals. The third part\\naims to exploit the single point information in the global moment\\nthat is most similar to the query, which in turn enriches the priori\\nknowledge and enhances the guidance of cross-modal semantics.\\n3.4.1\\nGlobal Alignment Loss. We take the cross-entropy loss to\\nforce the representation information we get to be close to the in-\\nformation provided by the supervised frames in terms of temporal\\ndistance and semantic distance,\\n𝐿𝑔= −\\n𝑇\\n∑︁\\n𝑖=1\\n𝐺𝑖log𝑆𝑖,\\n(11)\\nwhere 𝐺𝑖is the Gaussian distribution weight of the 𝑖-th frame, and\\n𝑆𝑖is the semantic similarity score of the 𝑖-th frame.\\n3.4.2\\nFrame-moment Correspondence Loss. Due to lacking of bound-\\nary annotations, a single point’s information inadequately repre-\\nsents the interval matching the query. This makes us to consider\\nusing video moments that encompass point annotation as potential\\npositives. We construct binary labels to facilitate interval percep-\\ntion: moments containing the point annotation are labeled 1, while\\nothers are labeled 0, as follows,\\ny𝑖=\\n(\\n1,\\n𝜏𝑚∈M𝑖,\\n0,\\n𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒,\\n(12)\\nwhere M𝑖is the 𝑖-th moment by sliding window. Due to the latent\\ndifferences in temporal information of these moments, we introduce\\nGaussian distribution weights to further construct the soft labels\\n˜\\ny𝑖, which can be formulated as:\\n˜\\ny𝑖= y𝑖W(M𝑖),\\n(13)\\nwhere W(·) = 𝐺(𝑠) ∗𝐺(𝑒), 𝐺(𝑠) and 𝐺(𝑒) are the Gaussian distri-\\nbution weights at the beginning and end of M𝑖, respectively. Since\\nit is inevitable that there will be multiple positive sample moments\\nbelonging to a query, it is inappropriate to view similarity score\\nlearning as a 1-in-N classification problem with cross-entropy loss.\\nTo this end, we utilize the Kullback-Leibler divergence to construct\\nframe-moment correspondence loss 𝐿𝑓→𝑚\\n𝑐\\n, as follows,\\n𝐿𝑓→𝑚\\n𝑐\\n=\\n𝑁\\n∑︁\\n𝑖=1\\n˜\\ny𝑖log ˜\\ny𝑖\\nS𝑖\\n,\\n(14)\\nwhere S𝑖is the cosine similarity calculated by moment M𝑖and\\nquery feature ˜\\n𝑄𝑠.\\n3.4.3\\nMoment-frame Correspondence Loss. In the beginning of lo-\\ncation process, we focus on identifying a point that align most\\nclosely with the query, enabling our model to discern and empha-\\nsize subtle points’ differences. We separate essential information\\nfrom positive sample moments, concentrating on feature variances.\\nSpecifically, we extract detailed point data reflecting semantic labels\\nacross moments, leveraging this to enrich point annotation with\\ninherent disparities.\\nThis process can be represented as follows,\\nK𝑖= ˜\\n𝑉𝑓(𝜃),𝜃= arg max\\n𝑗∈[𝑠,𝑒] 𝑆𝑗,\\n(15)\\nwhere 𝑠and 𝑒are the start and end index of the moment M𝑖.\\nFor label construction, we use both hard sample labels and soft\\nlabels with Gaussian weights, just as the Eqn. (12) and (13). We\\nthen employ KL divergence to analyze differences between positive\\nand negative samples, using these insights to refine our alignment\\nprocess, as follows,\\n𝐿𝑚→𝑓\\n𝑐\\n=\\n𝑁\\n∑︁\\n𝑖=1\\n˜\\ny𝑖log ˜\\ny𝑖\\nK𝑖\\n.\\n(16)\\nBy combining the global loss 𝐿𝑔and correspondence loss 𝐿𝑐, we\\nobtain the final loss for model optimization, as follows,\\n𝐿= 𝐿𝑔+ 𝛽𝐿𝑓→𝑚\\n𝑐\\n+ 𝛾𝐿𝑚→𝑓\\n𝑐\\n,\\n(17)\\nwhere 𝛽and 𝛾are used to balance the focus between moments and\\nframes, avoiding the model over underscore the sample information.\\n4\\nEXPERIMENTS\\nThis section will begin by presenting the experimental settings.\\nSubsequently, we will conduct comparative experiments, compre-\\nhensive ablation studies, and evaluate the effectiveness of SG-SCI\\nto answer the following three research questions (RQs):\\n• RQ1: Is our proposed SG-SCI able to outperform several state-\\nof-the-art competitors on VML?\\n• RQ2: Is each component of our SG-SCI helpful for boosting the\\nlocalization performance?\\n• RQ3: How do hyperparameters affect model capability?\\n\\n\\n581\\n582\\n583\\n584\\n585\\n586\\n587\\n588\\n589\\n590\\n591\\n592\\n593\\n594\\n595\\n596\\n597\\n598\\n599\\n600\\n601\\n602\\n603\\n604\\n605\\n606\\n607\\n608\\n609\\n610\\n611\\n612\\n613\\n614\\n615\\n616\\n617\\n618\\n619\\n620\\n621\\n622\\n623\\n624\\n625\\n626\\n627\\n628\\n629\\n630\\n631\\n632\\n633\\n634\\n635\\n636\\n637\\n638\\nACM MM, 2024, Melbourne, Australia\\nAnonymous Authors\\n639\\n640\\n641\\n642\\n643\\n644\\n645\\n646\\n647\\n648\\n649\\n650\\n651\\n652\\n653\\n654\\n655\\n656\\n657\\n658\\n659\\n660\\n661\\n662\\n663\\n664\\n665\\n666\\n667\\n668\\n669\\n670\\n671\\n672\\n673\\n674\\n675\\n676\\n677\\n678\\n679\\n680\\n681\\n682\\n683\\n684\\n685\\n686\\n687\\n688\\n689\\n690\\n691\\n692\\n693\\n694\\n695\\n696\\n4.1\\nDatasets\\nThe experiments that follow are conducted on two benchmark\\ndatasets: Charades-STA [12] and TACoS [33].\\nCharades-STA [12]: It is constructed on the Charades dataset [36]\\nand contains 16,128 \\\"moment-query\\\" pairs with an average video\\nduration of 30 seconds. Following the standard split strategy [12],\\nwe divided the dataset into 12,408 and 3,720 \\\"moment-query\\\" pairs\\nfor training and testing, respectively.\\nTACoS [33]: It is built upon MPII Cooking Compositive dataset [34]\\nand only covers cooking activities that contain pairs of queries with\\nvery similar visual information. It consists of 127 untrimmed videos\\nwith the average duration of 320 seconds and 18,818 queries.\\nIn particular, we use point-supervised information instead of\\nboundary supervision information in the original dataset setting,\\njust like existing effort [10], where the point-supervised information\\ncomes from a random point in the boundary.\\n4.2\\nExperimental Settings\\n4.2.1\\nEvaluation Metrics. Following the existing work [10, 21, 25],\\nwe choose 𝑅𝑛@𝑚and mean averaged IoU (mIoU) as protocols to\\nevaluate the performance of moment localization. To be specific,\\n𝑅𝑛@𝑚refers to the percentage of predictions for which the tempo-\\nral Intersection over Union (IoU) surpasses the thresholds 𝑚within\\nthe top-𝑛of the sorted results, and mean averaged IoU (mIoU) rep-\\nresents the average IoU across all test samples. Since our method\\nlocalizes the target moment with the highest coordinate probability,\\nwe set 𝑛= 1 and 𝑚∈{0.3, 0.5, 0.7}. The higher 𝑅𝑛@𝑚and mIoU\\nare better.\\n4.2.2\\nImplementation Details. In our work, we employ the pre-\\ntrained I3D [6] and C3D [41] network to extract visual features\\nfrom Charades-STA [12] and TACoS [33] respectively. Consistent\\nwith prior research [10], we keep the word embedding module\\nfixed and utilize the 840B GloVe [32] to construct a comprehensive\\nvocabulary. The model dimension 𝑑𝑚𝑜𝑑𝑒𝑙is set to 512, and we\\nemploy AdamW [28] with a learning rate of 1𝑒−4, which decays by\\nhalf when reaching a plateau during training. By default, we set\\n𝛼= 0.3, 𝛽= 𝛾= 0.1, and 𝜎= 0.04 in both datasets. The batch sizes\\nfor the Charades-STA and TACoS are empirically set to 512 and 64,\\nrespectively. All experiments are conducted on a NVIDIA GeForce\\nRTX 4090 with 24GB memory.\\n4.3\\nPerformance Comparison (RQ1)\\nIn this subsection, we compare the proposed method with different\\nkinds of state-of-the-art methods, including fully-supervised meth-\\nods (2D-TAN [51], SS [11], FVMR [13], ADPN [7], MS-DETR [22]),\\nweakly-supervised methods (SCN [26], CNM [54], CWG [8], CPL [55],\\nIRON [5], PPS [23]) and point-supervised methods (ViGA [10],\\nPSVTG [46], CFMR [21], D3G [25]). To ensure the validity of the\\ncomparison, the majority of the selected methods are drawn from\\nthe last three years. The comparisons on Charades-STA and TACoS\\nare encapsulated in Table 1 and Table 2 respectively. Based on the\\ndata presented, the following observations can be derived:\\n• Our method mines more correspondence information com-\\npared with existing point-supervised methods. While tra-\\nditional point-supervised methods focus on the interaction of\\nTable 1: Performance comparison on Charades-STA with\\ndifferent supervision methods. Bold means the best result in\\npoint-supervised method and underline means the second\\nbest.\\nType\\nMethod\\nR1@0.3 R1@0.5 R1@0.7\\nmIoU\\nFully-supervised\\n2D-TAN [AAAI20] [51]\\n-\\n50.62\\n28.71\\n-\\nSS [ICCV21] [11]\\n-\\n56.97\\n32.74\\n-\\nFVMR [ICCV21] [13]\\n-\\n55.01\\n33.74\\n-\\nADPN [ACMMM23] [7]\\n70.35\\n55.32\\n37.47\\n51.15\\nWeakly-supervised\\nCWG [AAAI22] [8]\\n43.41\\n31.02\\n16.53\\n-\\nCPL [CVPR22] [55]\\n66.40\\n49.24\\n22.39\\n-\\nIRON [CVPR23] [5]\\n70.28\\n51.33\\n24.31\\n-\\nPPS [AAAI24] [23]\\n69.06\\n51.49\\n26.16\\n-\\nPoint-supervised\\nViGA [SIGIR22] [10]\\n71.21\\n45.05\\n20.27\\n44.57\\nPSVTG [TMM22] [46]\\n60.40\\n39.22\\n20.17\\n39.77\\nCFMR [ACMMM23] [21]\\n-\\n48.14\\n22.58\\n-\\nD3G [CVPR23] [25]\\n-\\n43.82\\n20.46\\n-\\nSG-SCI (Ours)\\n70.30\\n52.07\\n27.23\\n46.77\\nTable 2: Performance comparison on TACoS with different\\nsupervision methods. Bold means the best result in point-\\nsupervised method and underline means the second best.\\nType\\nMethod\\nR1@0.3 R1@0.5 R1@0.7\\nmIoU\\nFully-supervised\\n2D-TAN [AAAI20] [51]\\n37.29\\n25.32\\n-\\n-\\nSS [ICCV21] [11]\\n41.33\\n29.56\\n-\\n-\\nFVMR [ICCV21] [13]\\n41.48\\n29.12\\n-\\n-\\nMS-DETR [ACL23] [22]\\n47.66\\n37.36\\n25.81\\n35.09\\nWeakly-supervised\\nSCN [AAAI20] [26]\\n11.72\\n4.75\\n-\\n-\\nCNM [AAAI22] [54]\\n7.20\\n2.20\\n-\\n-\\nCPL [CVPR22] [55]\\n11.42\\n4.12\\n-\\n-\\nPoint-supervised\\nViGA [SIGIR22] [10]\\n19.62\\n8.85\\n3.22\\n15.47\\nPSVTG [TMM22] [46]\\n23.64\\n10.00\\n3.35\\n17.39\\nCFMR [ACMMM23] [21]\\n25.44\\n12.82\\n-\\n-\\nD3G [CVPR23] [25]\\n27.27\\n12.67\\n4.70\\n-\\nSG-SCI (Ours)\\n37.47\\n20.59\\n8.27\\n23.83\\nsentence-moments [25], our method involves multi-granularity\\ninteractions among complex video, concise textual descriptions,\\nand point annotation. Therefore, the model can consider the in-\\nteraction between sentences and point annotation at multiple\\ngranularities during the training process, resulting in better cross-\\nmodal information interaction. As illustrated in Table 1, in the\\nCharades-STA dataset, SG-SCI achieves 4.65% and 6.77% higher\\nthan CFMR [21] and D3G [25] in 𝑅1@0.7, respectively.\\n• SG-SCI reduces the performance disparity between fully-\\nsupervised methods and point-supervised methods. Lack-\\ning of annotated timestamps, the positioning accuracy of point-\\nsupervised methods is lower compared to existing supervised\\nlearning methods. By learning features across different scales, SG-\\nSCI exploits the information contained in the supervised frames,\\nto some extent compensating for the performance gap caused by\\ndifferent supervised information. In the Charades-STA dataset,\\nour method even outperforms 2D-TAN [51], fully-supervised\\nmethod, by 1.45% in terms of 𝑅1@0.5.\\n• Trivial annotation cost contributes to localization. The ex-\\nistence of frame annotation provides the model with a clear\\ndirection during iterative convergence, and our proposed corre-\\nspondence module enables the model to continuously converge\\nduring gradient backpropagation based on the multi-granularity\\ninformation from query and frame annotation. As evidenced in\\nTable 1 and Table 2, our approach consistently outperforms the\\ncutting-edge weakly-supervised models.\\n\\n\\n697\\n698\\n699\\n700\\n701\\n702\\n703\\n704\\n705\\n706\\n707\\n708\\n709\\n710\\n711\\n712\\n713\\n714\\n715\\n716\\n717\\n718\\n719\\n720\\n721\\n722\\n723\\n724\\n725\\n726\\n727\\n728\\n729\\n730\\n731\\n732\\n733\\n734\\n735\\n736\\n737\\n738\\n739\\n740\\n741\\n742\\n743\\n744\\n745\\n746\\n747\\n748\\n749\\n750\\n751\\n752\\n753\\n754\\nExplicit Granularity and Implicit Scale Correspondence Learning for Point-Supervised Video Moment Localization\\nACM MM, 2024, Melbourne, Australia\\n755\\n756\\n757\\n758\\n759\\n760\\n761\\n762\\n763\\n764\\n765\\n766\\n767\\n768\\n769\\n770\\n771\\n772\\n773\\n774\\n775\\n776\\n777\\n778\\n779\\n780\\n781\\n782\\n783\\n784\\n785\\n786\\n787\\n788\\n789\\n790\\n791\\n792\\n793\\n794\\n795\\n796\\n797\\n798\\n799\\n800\\n801\\n802\\n803\\n804\\n805\\n806\\n807\\n808\\n809\\n810\\n811\\n812\\n(a) Component Ablation.\\n(b) GCA Ablation.\\n(c) Mean Weighted Factor.\\nFigure 3: Impact of (a) GCA and SCL, (b) multiple granularities in GCA, and (c) the mean weighted factor 𝛼in GCA. Evaluation\\nperformed on Charades-STA.\\nTable 3: Comparison of different loss ablation for our frame-\\nwork. \\\"means retaining it and %means removing it.\\nGlobal F-M M-F R1@0.3 R1@0.5 R1@0.7 mIoU\\n\\\"\\n%\\n%\\n66.83\\n37.74\\n14.14\\n40.81\\n%\\n\\\"\\n%\\n50.56\\n27.96\\n10.27\\n31.04\\n%\\n%\\n\\\"\\n41.13\\n23.87\\n9.46\\n26.33\\n%\\n\\\"\\n\\\"\\n47.50\\n25.59\\n9.70\\n29.13\\n\\\"\\n%\\n\\\"\\n70.00\\n44.70\\n19.81\\n44.04\\n\\\"\\n\\\"\\n%\\n69.89\\n42.07\\n17.50\\n43.28\\n\\\"\\n\\\"\\n\\\"\\n70.30\\n52.07\\n27.23\\n46.77\\n• SG-SCI can handle more challenging task. In contrast to\\nCharades-STA, TACoS contains longer videos with shorter re-\\ntrieved moments, and each frame within the video exhibits a high\\ndegree of semantic approximation, demanding a heightened abil-\\nity from the model to perceive boundaries and differentiate simi-\\nlar moments. Consequently, existing weakly-supervised methods\\nmay struggle to address this scenario, resulting in a significant\\nabsence of the 𝑅1@0.7 metric in Table 2. However, compared to\\nexisting point-supervised methods, like ViGA [10] and D3G [25],\\nour approach exhibits great performance in 𝑅1@0.7.\\n4.4\\nIn-depth Analysis (RQ2 & RQ3)\\nTo demonstrate each component of SG-SCI, we conducted extensive\\nablation experiments on Charades-STA.\\n4.4.1\\nAblation Study.\\n• Impact of GCA and SCL. To discuss whether the proposed\\nGCA and SCL can achieve better performance, we simplify the\\npipeline of SG-SCI and set it as the baseline. Specifically, after\\nmulti-modal interaction, we train the model by calculating the\\nKL divergence of the similarity distribution between the query\\nand each frame with a Gaussian distribution. To demonstrate the\\neffectiveness of SCL, we conduct experiments by setting the soft\\nlabels to 1. From the Figure 3(a), it can be seen that both GCA and\\nSCL effectively enhance the localization ability of the baseline,\\nespecially in 𝑅1@0.7.\\n• Effect of multiple granularities in GCA. As described in\\nSec 3.3, GCA not only helps to understand the video contents\\nmore deeply but also ensures that less significant frames have\\nrelevance in the corresponding textual narrative. We validate\\nthe effectiveness of multi-granularity information interaction\\non cross-modal representation, with specific results shown in\\nFigure 3(b). It can be seen that the multi-granularity interaction\\nproposed in GCA has effectively improved the localization effect\\nby enhancing single-modal representation. At the same time,\\nincreasing the multi-granularity interaction of queries can en-\\nhance the model’s ability to perceive fine-grained boundaries,\\ni.e., improve the 𝑅1@0.7 by 6.10%.\\n• Effectiveness of different loss. The final loss contains three\\nparts: global loss, frame-moment correspondence loss and moment-\\nframe correspondence loss. To evaluate the effectiveness of them,\\nwe remove some of them for comparison and the results are\\nshown in Table 3, and we find that scale correspondence learning\\ncan inspire the potential of the model for boundary perception.\\nIn the presence of global loss, the frame-moment correspondence\\nloss and moment-frame correspondence loss alone can enhance\\nperformance. The combination of the two can produce compre-\\nhensive performance improvement, and the 𝑅1@0.7 even gets\\nmore than 10% improvement than only using global loss, showing\\nthat scale correspondence learning can enhance the ability to\\ncapture key information and boundary perception.\\n4.4.2\\nParameter sensitivity.\\n• The mean weighted factor 𝛼in GCA. An important hyperpa-\\nrameter in GCA is 𝛼, which is used when combining the different\\ngranularities. It represents how much information is interacted\\nand 𝛼= 1 represents merely using global feature information\\nwhile 𝛼= 0 denotes only utilizing the fine granularity infor-\\nmation. The specific results are shown in Figure 3(c). It can be\\nobserved that each metric shows a trend of first rising and then\\nfalling. Moreover, different difficulty of tasks represent diverse 𝛼:\\nfor coarse-grained localization task (𝑚= 0.3), 𝛼= 0.5 is suitable\\nwhile for fine-grained localization task, i.e., 𝑚= 0.5 and 𝑚= 0.7,\\n𝛼= 0.3 is better.\\n• The importance of frame-moment correspondence loss\\nand moment-frame correspondence loss. In the Eqn. (17),\\n𝛽and 𝛾represent the importance of frame-moment correspon-\\ndence loss and moment-frame correspondence loss in the final\\nloss. We test different combinations of 𝛽and 𝛾in the range\\n\\n\\n813\\n814\\n815\\n816\\n817\\n818\\n819\\n820\\n821\\n822\\n823\\n824\\n825\\n826\\n827\\n828\\n829\\n830\\n831\\n832\\n833\\n834\\n835\\n836\\n837\\n838\\n839\\n840\\n841\\n842\\n843\\n844\\n845\\n846\\n847\\n848\\n849\\n850\\n851\\n852\\n853\\n854\\n855\\n856\\n857\\n858\\n859\\n860\\n861\\n862\\n863\\n864\\n865\\n866\\n867\\n868\\n869\\n870\\nACM MM, 2024, Melbourne, Australia\\nAnonymous Authors\\n871\\n872\\n873\\n874\\n875\\n876\\n877\\n878\\n879\\n880\\n881\\n882\\n883\\n884\\n885\\n886\\n887\\n888\\n889\\n890\\n891\\n892\\n893\\n894\\n895\\n896\\n897\\n898\\n899\\n900\\n901\\n902\\n903\\n904\\n905\\n906\\n907\\n908\\n909\\n910\\n911\\n912\\n913\\n914\\n915\\n916\\n917\\n918\\n919\\n920\\n921\\n922\\n923\\n924\\n925\\n926\\n927\\n928\\nFigure 4: The results of R1@0.7 from different frame-\\nmoment correspondence loss and moment-frame correspon-\\ndence loss. Evaluation performed on the Charades-STA\\ndataset.\\nQuery: She picks up the rinds and throws them away.\\nGround Truth\\n90.2 s\\n95.2 s\\nVIGA\\nOurs\\n86.4 s\\n98.1 s\\n96.4 s\\n88.7 s\\nFigure 5: Visualization of localization results on TACoS. The\\nyellow bar represents the ground truth temporal boundaries\\nof the language query, the blue bar depicts the predicted\\nboundaries of ViGA, and the green bar signifies the predicted\\nboundaries of SG-SCI.\\n{0.05, 0.1, 0.2, 0.5, 1}, and display the numerical relationship be-\\ntween 𝑅1@0.7, as shown in Figure 4. Similar to the 𝛼in GCA,\\n𝛽and 𝛾also show an overall trend of first increasing and then\\ndecreasing, and the highest value is obtained at 𝛽= 𝛾= 0.1. The\\nexperimental results indicate that frame-moment correspondence\\nloss and moment-frame correspondence loss have a significant\\nimpact on fine-grained boundary perception.\\n• Sliding Window Size. Sliding window is designed to capture\\nmulti-granularity features via multi-scale sliding windows. In\\nfact, monotonically increasing or decreasing the size of the slid-\\ning window would lead to degradation of the representation\\ndiscrimination, thus affecting overall performance. For instance,\\nin extreme cases, the sliding window may encompass information\\nfrom either the entire video or just a frame. In such scenarios the\\nmodel struggles to learn and comprehend these moments holisti-\\ncally, leading to a decline in overall localization performance. We\\nconducted ablation experiments on the Charades-STA dataset,\\nwith a default step size of 4. The specific experimental results are\\nshown in Table 4. The results indicate that the best performance\\nis achieved when the sliding window size is 8, indicating that\\nthe model needs a moderate sliding window size for sufficient\\nlearning during Scale Correspondence Learning.\\nTable 4: Performance comparison on Charades-STA with\\ndifferent sliding window sizes, and ∗means the stride is the\\nhalf of sliding window size.\\nSize\\nR1@0.3 R1@0.5 R1@0.7 mIoU\\n4\\n70.89\\n49.46\\n23.66\\n46.02\\n8\\n70.30\\n52.07\\n27.23\\n46.77\\n16\\n70.78\\n47.18\\n21.99\\n45.23\\n24\\n70.43\\n46.02\\n21.48\\n44.96\\n32\\n69.84\\n45.3\\n19.46\\n44.22\\n4∗\\n70.86\\n49.46\\n23.31\\n45.91\\n16∗\\n70.83\\n47.15\\n22.07\\n45.22\\n24∗\\n70.46\\n45.86\\n21.64\\n44.70\\n32∗\\n69.09\\n44.44\\n20.16\\n44.03\\nTable 5: Speed comparison on Charades-STA and TACoS be-\\ntween ViGA [10] and SG-SCI, time is averaged.\\nCharades-STA\\nTACoS\\nMethod\\nTrain Inference\\nTrain Inference\\nViGA [10]\\n1.0x\\n1.0x\\n1.0x\\n1.0x\\nSG-SCI\\n1.1x\\n1.2x\\n1.0x\\n1.2x\\n4.4.3\\nInference speed. As Table 5 illustrated, during the training\\nand inference phases, our runtime is almost identical to ViGA’s [10].\\nThis is because our proposed GCA and SCL do not introduce sig-\\nnificant computational complexity. As a result, we consider our\\nmethods to be competitive in terms of efficiency.\\n4.4.4\\nQuantitative Results. To more thoroughly examine the contri-\\nbutions of our proposed SG-SCI framework, we present an example\\nillustrating the results of moment retrieval on the TACoS datasets.\\nAs is shown from the Figure 5, SG-SCI demonstrates superior effec-\\ntiveness in terms of boundary perception. This can be attributed\\nto the training process, where our model effectively enhances the\\nmodel’s perceptual ability across different scales.\\n5\\nCONCLUSION AND FUTURE WORK\\nIn this paper, we designed a new framework to understand the cor-\\nrespondence connections between video and text. This framework\\nused a limited amount of data from single frame to learn. Initially,\\nwe looked at the problems existing methods had in linking the mean-\\ning of video and text that varied in detail. To address these issues,\\nour method included two main strategies: (1) creating a model that\\nused underlying knowledge to connect features of different modal-\\nities, and (2) developing a learning strategy that focused on the\\ndifferences in information between similar samples. These strate-\\ngies helped our method to clearly define and find the differences\\nin meaning across various details and modalities. This made our\\nmethod better at representing features of explicit granularity and\\nimplicit scales. Experiments conducted on two benchmark datasets\\ndemonstrated the effectiveness of our proposed framework.\\nIn the future, in order to further improve the performance of\\nVML, we plan to introduce a more advanced visual-language trans-\\nformer backbone. In addition, inspired by the fully-supervised par-\\nadigm, we plan to study the impact of different inference methods\\non localization performance.\\n\\n\\n929\\n930\\n931\\n932\\n933\\n934\\n935\\n936\\n937\\n938\\n939\\n940\\n941\\n942\\n943\\n944\\n945\\n946\\n947\\n948\\n949\\n950\\n951\\n952\\n953\\n954\\n955\\n956\\n957\\n958\\n959\\n960\\n961\\n962\\n963\\n964\\n965\\n966\\n967\\n968\\n969\\n970\\n971\\n972\\n973\\n974\\n975\\n976\\n977\\n978\\n979\\n980\\n981\\n982\\n983\\n984\\n985\\n986\\nExplicit Granularity and Implicit Scale Correspondence Learning for Point-Supervised Video Moment Localization\\nACM MM, 2024, Melbourne, Australia\\n987\\n988\\n989\\n990\\n991\\n992\\n993\\n994\\n995\\n996\\n997\\n998\\n999\\n1000\\n1001\\n1002\\n1003\\n1004\\n1005\\n1006\\n1007\\n1008\\n1009\\n1010\\n1011\\n1012\\n1013\\n1014\\n1015\\n1016\\n1017\\n1018\\n1019\\n1020\\n1021\\n1022\\n1023\\n1024\\n1025\\n1026\\n1027\\n1028\\n1029\\n1030\\n1031\\n1032\\n1033\\n1034\\n1035\\n1036\\n1037\\n1038\\n1039\\n1040\\n1041\\n1042\\n1043\\n1044\\nREFERENCES\\n[1] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell,\\nand Bryan Russell. 2017. Localizing moments in video with natural language. In\\nProceedings of the IEEE international conference on computer vision. 5803–5812.\\n[2] Peijun Bao, Qian Zheng, and Yadong Mu. 2021. Dense events grounding in video.\\nIn Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 920–928.\\n[3] Amy Bearman, Olga Russakovsky, Vittorio Ferrari, and Li Fei-Fei. 2016. What’s\\nthe point: Semantic segmentation with point supervision. In European conference\\non computer vision. Springer, 549–565.\\n[4] Meng Cao, Long Chen, Mike Zheng Shou, Can Zhang, and Yuexian Zou. 2021.\\nOn Pursuit of Designing Multi-modal Transformer for Video Grounding. In\\nProceedings of the 2021 Conference on Empirical Methods in Natural Language\\nProcessing. 9810–9823.\\n[5] Meng Cao, Fangyun Wei, Can Xu, Xiubo Geng, Long Chen, Can Zhang, Yuex-\\nian Zou, Tao Shen, and Daxin Jiang. 2023. Iterative proposal refinement for\\nweakly-supervised video grounding. In Proceedings of the IEEE/CVF Conference\\non Computer Vision and Pattern Recognition. 6524–6534.\\n[6] Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? a new\\nmodel and the kinetics dataset. In proceedings of the IEEE Conference on Computer\\nVision and Pattern Recognition. 6299–6308.\\n[7] Houlun Chen, Xin Wang, Xiaohan Lan, Hong Chen, Xuguang Duan, Jia Jia,\\nand Wenwu Zhu. 2023. Curriculum-listener: Consistency-and complementarity-\\naware audio-enhanced temporal sentence grounding. In Proceedings of the 31st\\nACM International Conference on Multimedia. 3117–3128.\\n[8] Jiaming Chen, Weixin Luo, Wei Zhang, and Lin Ma. 2022. Explore inter-contrast\\nbetween videos via composition for weakly supervised temporal sentence ground-\\ning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 267–\\n275.\\n[9] Long Chen, Yulei Niu, Brian Chen, Xudong Lin, Guangxing Han, Christopher\\nThomas, Hammad Ayyubi, Heng Ji, and Shih-Fu Chang. 2022. Weakly-Supervised\\nTemporal Article Grounding. In Proceedings of the 2022 Conference on Empirical\\nMethods in Natural Language Processing. 9402–9413.\\n[10] Ran Cui, Tianwen Qian, Pai Peng, Elena Daskalaki, Jingjing Chen, Xiaowei Guo,\\nHuyang Sun, and Yu-Gang Jiang. 2022. Video moment retrieval from text queries\\nvia single frame annotation. In Proceedings of the 45th International ACM SIGIR\\nConference on Research and Development in Information Retrieval. 1033–1043.\\n[11] Xinpeng Ding, Nannan Wang, Shiwei Zhang, De Cheng, Xiaomeng Li, Ziyuan\\nHuang, Mingqian Tang, and Xinbo Gao. 2021. Support-set based cross-supervision\\nfor video grounding. In Proceedings of the IEEE/CVF International Conference on\\nComputer Vision. 11573–11582.\\n[12] Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. 2017. Tall: Temporal\\nactivity localization via language query. In Proceedings of the IEEE international\\nconference on computer vision. 5267–5275.\\n[13] Junyu Gao and Changsheng Xu. 2021. Fast video moment retrieval. In Proceedings\\nof the IEEE/CVF International Conference on Computer Vision. 1523–1532.\\n[14] Mingfei Gao, Larry S Davis, Richard Socher, and Caiming Xiong. 2019. Wslln:\\nWeakly supervised natural language localization networks.\\narXiv preprint\\narXiv:1909.00239 (2019).\\n[15] Zijian Gao, Jingyu Liu, Weiqi Sun, Sheng Chen, Dedan Chang, and Lili Zhao.\\n2021. Clip2tv: Align, match and distill for video-text retrieval. arXiv preprint\\narXiv:2111.05610 (2021).\\n[16] Wenjia Geng, Yong Liu, Lei Chen, Sujia Wang, Jie Zhou, and Yansong Tang.\\n2024. Learning Multi-Scale Video-Text Correspondence for Weakly Supervised\\nTemporal Article Gronding. In Proceedings of the AAAI Conference on Artificial\\nIntelligence, Vol. 38. 1896–1904.\\n[17] Tengda Han, Weidi Xie, and Andrew Zisserman. 2022. Temporal alignment\\nnetworks for long-term video. In Proceedings of the IEEE/CVF Conference on\\nComputer Vision and Pattern Recognition. 2906–2916.\\n[18] Zhijian Hou, Wanjun Zhong, Lei Ji, DIFEI GAO, Kun Yan, WK Chan, Chong-Wah\\nNgo, Mike Zheng Shou, and Nan Duan. 2023. CONE: An Efficient COarse-to-fiNE\\nAlignment Framework for Long Video Temporal Grounding. In The 61st Annual\\nMeeting Of The Association For Computational Linguistics.\\n[19] Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. 2020. Movienet:\\nA holistic dataset for movie understanding. In Computer Vision–ECCV 2020: 16th\\nEuropean Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16.\\nSpringer, 709–727.\\n[20] Zhenyu Huang, Guocheng Niu, Xiao Liu, Wenbiao Ding, Xinyan Xiao, Hua Wu,\\nand Xi Peng. 2021. Learning with noisy correspondence for cross-modal matching.\\nAdvances in Neural Information Processing Systems 34 (2021), 29406–29419.\\n[21] Xun Jiang, Zailei Zhou, Xing Xu, Yang Yang, Guoqing Wang, and Heng Tao\\nShen. 2023. Faster Video Moment Retrieval with Point-Level Supervision. arXiv\\npreprint arXiv:2305.14017 (2023).\\n[22] Wang Jing, Aixin Sun, Hao Zhang, and Xiaoli Li. 2023. MS-DETR: Natural\\nLanguage Video Localization with Sampling Moment-Moment Interaction. In\\nProceedings of the 61st Annual Meeting of the Association for Computational Lin-\\nguistics (Volume 1: Long Papers). 1387–1400.\\n[23] Sunoh Kim, Jungchan Cho, Joonsang Yu, YoungJoon Yoo, and Jin Young Choi.\\n2024. Gaussian Mixture Proposals with Pull-Push Learning Scheme to Capture\\nDiverse Events for Weakly Supervised Temporal Video Grounding. In Proceedings\\nof the AAAI Conference on Artificial Intelligence, Vol. 38. 2795–2803.\\n[24] Pilhyeon Lee and Hyeran Byun. 2021. Learning action completeness from points\\nfor weakly-supervised temporal action localization. In Proceedings of the IEEE/CVF\\ninternational conference on computer vision. 13648–13657.\\n[25] Hanjun Li, Xiujun Shu, Sunan He, Ruizhi Qiao, Wei Wen, Taian Guo, Bei Gan,\\nand Xing Sun. 2023. D3G: Exploring Gaussian Prior for Temporal Sentence\\nGrounding with Glance Annotation. In Proceedings of the IEEE/CVF International\\nConference on Computer Vision. 13734–13746.\\n[26] Zhijie Lin, Zhou Zhao, Zhu Zhang, Qi Wang, and Huasheng Liu. 2020. Weakly-\\nsupervised video moment retrieval via semantic completion network. In Proceed-\\nings of the AAAI Conference on Artificial Intelligence, Vol. 34. 11539–11546.\\n[27] Meng Liu, Xiang Wang, Liqiang Nie, Xiangnan He, Baoquan Chen, and Tat-Seng\\nChua. 2018. Attentive moment retrieval in videos. In The 41st international ACM\\nSIGIR conference on research & development in information retrieval. 15–24.\\n[28] Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization.\\narXiv preprint arXiv:1711.05101 (2017).\\n[29] Fan Ma, Linchao Zhu, Yi Yang, Shengxin Zha, Gourab Kundu, Matt Feiszli, and\\nZheng Shou. 2020. Sf-net: Single-frame supervision for temporal action local-\\nization. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK,\\nAugust 23–28, 2020, Proceedings, Part IV 16. Springer, 420–437.\\n[30] Minuk Ma, Sunjae Yoon, Junyeong Kim, Youngjoon Lee, Sunghun Kang, and\\nChang D Yoo. 2020. Vlanet: Video-language alignment network for weakly-\\nsupervised video moment retrieval. In Computer Vision–ECCV 2020: 16th European\\nConference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVIII 16. Springer,\\n156–171.\\n[31] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic,\\nand Andrew Zisserman. 2020. End-to-end learning of visual representations\\nfrom uncurated instructional videos. In Proceedings of the IEEE/CVF conference\\non computer vision and pattern recognition. 9879–9889.\\n[32] Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove:\\nGlobal vectors for word representation. In Proceedings of the 2014 conference on\\nempirical methods in natural language processing (EMNLP). 1532–1543.\\n[33] Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt\\nSchiele, and Manfred Pinkal. 2013. Grounding action descriptions in videos.\\nTransactions of the Association for Computational Linguistics 1 (2013), 25–36.\\n[34] Marcus Rohrbach, Michaela Regneri, Mykhaylo Andriluka, Sikandar Amin, Man-\\nfred Pinkal, and Bernt Schiele. 2012. Script data for attribute-based recognition of\\ncomposite activities. In Computer Vision–ECCV 2012: 12th European Conference on\\nComputer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part I 12. Springer,\\n144–157.\\n[35] Xingyu Shen, Long Lan, Huibin Tan, Xiang Zhang, Xurui Ma, and Zhigang Luo.\\n2022. Joint modality synergy and spatio-temporal cue purification for moment\\nlocalization. In Proceedings of the 2022 International Conference on Multimedia\\nRetrieval. 369–379.\\n[36] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and\\nAbhinav Gupta. 2016. Hollywood in homes: Crowdsourcing data collection for\\nactivity understanding. In Computer Vision–ECCV 2016: 14th European Conference,\\nAmsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14. Springer,\\n510–526.\\n[37] Yiping Song, Rui Yan, Cheng-Te Li, Jian-Yun Nie, Ming Zhang, and Dongyan Zhao.\\n2018. An Ensemble of Retrieval-Based and Generation-Based Human-Computer\\nConversation Systems. (2018).\\n[38] Haoyu Tang, Jihua Zhu, Lin Wang, Qinghai Zheng, and Tianwei Zhang. 2021.\\nMulti-level query interaction for temporal language grounding. IEEE Transactions\\non Intelligent Transportation Systems 23, 12 (2021), 25479–25488.\\n[39] Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao,\\nJiwen Lu, and Jie Zhou. 2019. Coin: A large-scale dataset for comprehensive in-\\nstructional video analysis. In Proceedings of the IEEE/CVF Conference on Computer\\nVision and Pattern Recognition. 1207–1216.\\n[40] Zineng Tang, Jie Lei, and Mohit Bansal. 2021. Decembert: Learning from noisy\\ninstructional videos via dense captions and entropy minimization. In Proceed-\\nings of the 2021 Conference of the North American Chapter of the Association for\\nComputational Linguistics: Human Language Technologies. 2415–2426.\\n[41] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.\\n2015. Learning spatiotemporal features with 3d convolutional networks. In\\nProceedings of the IEEE international conference on computer vision. 4489–4497.\\n[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,\\nAidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all\\nyou need. Advances in neural information processing systems 30 (2017).\\n[43] Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-\\nFang Wang, William Yang Wang, and Lei Zhang. 2019. Reinforced cross-modal\\nmatching and self-supervised imitation learning for vision-language naviga-\\ntion. In Proceedings of the IEEE/CVF conference on computer vision and pattern\\nrecognition. 6629–6638.\\n\\n\\n1045\\n1046\\n1047\\n1048\\n1049\\n1050\\n1051\\n1052\\n1053\\n1054\\n1055\\n1056\\n1057\\n1058\\n1059\\n1060\\n1061\\n1062\\n1063\\n1064\\n1065\\n1066\\n1067\\n1068\\n1069\\n1070\\n1071\\n1072\\n1073\\n1074\\n1075\\n1076\\n1077\\n1078\\n1079\\n1080\\n1081\\n1082\\n1083\\n1084\\n1085\\n1086\\n1087\\n1088\\n1089\\n1090\\n1091\\n1092\\n1093\\n1094\\n1095\\n1096\\n1097\\n1098\\n1099\\n1100\\n1101\\n1102\\nACM MM, 2024, Melbourne, Australia\\nAnonymous Authors\\n1103\\n1104\\n1105\\n1106\\n1107\\n1108\\n1109\\n1110\\n1111\\n1112\\n1113\\n1114\\n1115\\n1116\\n1117\\n1118\\n1119\\n1120\\n1121\\n1122\\n1123\\n1124\\n1125\\n1126\\n1127\\n1128\\n1129\\n1130\\n1131\\n1132\\n1133\\n1134\\n1135\\n1136\\n1137\\n1138\\n1139\\n1140\\n1141\\n1142\\n1143\\n1144\\n1145\\n1146\\n1147\\n1148\\n1149\\n1150\\n1151\\n1152\\n1153\\n1154\\n1155\\n1156\\n1157\\n1158\\n1159\\n1160\\n[44] Yunxiao Wang, Meng Liu, Yinwei Wei, Zhiyong Cheng, Yinglong Wang, and\\nLiqiang Nie. 2022. Siamese alignment network for weakly supervised video\\nmoment retrieval. IEEE Transactions on Multimedia (2022).\\n[45] Wenhao Wu, Haipeng Luo, Bo Fang, Jingdong Wang, and Wanli Ouyang. 2023.\\nCap4video: What can auxiliary captions do for text-video retrieval?. In Proceedings\\nof the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10704–\\n10713.\\n[46] Zhe Xu, Kun Wei, Xu Yang, and Cheng Deng. 2022. Point-supervised video\\ntemporal grounding. IEEE Transactions on Multimedia (2022).\\n[47] Shuo Yang and Xinxiao Wu. 2022. Entity-aware and motion-aware transformers\\nfor language-driven action localization in videos. arXiv preprint arXiv:2205.05854\\n(2022).\\n[48] Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. 2024. Self-chained\\nimage-language model for video localization and question answering. Advances\\nin Neural Information Processing Systems 36 (2024).\\n[49] Yitian Yuan, Tao Mei, and Wenwu Zhu. 2019. To find where you talk: Tempo-\\nral sentence localization in video with attention based location regression. In\\nProceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 9159–9166.\\n[50] Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. 2023. Temporal sentence\\ngrounding in videos: A survey and future directions. IEEE Transactions on Pattern\\nAnalysis and Machine Intelligence (2023).\\n[51] Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo. 2020. Learning 2d\\ntemporal adjacent networks for moment localization with natural language. In\\nProceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 12870–12877.\\n[52] Songyang Zhang, Jinsong Su, and Jiebo Luo. 2019. Exploiting temporal relation-\\nships in video moment localization with natural language. In Proceedings of the\\n27th ACM International Conference on Multimedia. 1230–1238.\\n[53] Zhu Zhang, Zhou Zhao, Zhijie Lin, Xiuqiang He, et al. 2020. Counterfactual\\ncontrastive learning for weakly-supervised vision-language grounding. Advances\\nin Neural Information Processing Systems 33 (2020), 18123–18134.\\n[54] Minghang Zheng, Yanjie Huang, Qingchao Chen, and Yang Liu. 2022. Weakly\\nsupervised video moment localization with contrastive negative sample mining.\\nIn Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 3517–3525.\\n[55] Minghang Zheng, Yanjie Huang, Qingchao Chen, Yuxin Peng, and Yang Liu. 2022.\\nWeakly supervised temporal sentence grounding with gaussian-based contrastive\\nproposal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision\\nand Pattern Recognition. 15555–15564.\\n\\n\\nIEEE TRANSACTIONS ON MULTIMEDIA, VOL. 25, 2023\\n6121\\nPoint-Supervised Video Temporal Grounding\\nZhe Xu\\n, Kun Wei\\n, Xu Yang, Member, IEEE, and Cheng Deng\\n, Senior Member, IEEE\\nAbstract—Given an untrimmed video and a language query,\\nVideo Temporal Grounding (VTG) aims to locate the time interval\\nin the video semantically relevant to the query. Existing fully-\\nsupervised VTG methods require accurate annotations of temporal\\nboundary, which is time-consuming and expensive to obtain. On\\nthe other hand, weakly-supervised VTG methods where only\\npaired videos and queries are available during training lag far\\nbehind the fully-supervised ones. In this paper, we introduce\\npoint supervision to narrow the performance gap with affordable\\nannotating cost and propose a novel method dubbed Point-\\nSupervised Video Temporal Grounding (PS-VTG). Specifically,\\nan attention-based grounding network is first employed to obtain\\na language activation sequence (LAS). Then pseudo segment-\\nlevel label is generated based on the LAS and the given point\\nsupervision to assist the training process. In addition, multi-\\nlevel distribution calibration and cross-modal contrast are framed\\nto obtain discriminative feature representations and precisely\\nhighlight the language-relevant video segments. Experiments on\\nthree benchmarks demonstrate that our method trained with\\npoint supervision can significantly outperform weakly-supervised\\napproaches and achieve comparable performance with fully-\\nsupervised ones.\\nIndex Terms—Cross-modal contrast, multi-level distribution\\ncalibration, point supervision, video temporal grounding.\\nI. INTRODUCTION\\nW\\nITH the explosive growth of video data, video under-\\nstanding has attracted increasing research attention in\\nthe past few years. Traditional video understanding tasks, e.g.,\\nvideo classification [1], [2], object tracking [3], [4], and action\\nlocalization [5], [6] are limited in the video modality and pre-\\ndefined action categories. Recently, Video Temporal Grounding\\n(VTG) [7], [8], [9] was introduced to locate the time interval in\\nan untrimmed video semantically relevant to a language query,\\nwhich involves both the video and language modalities. Due to\\nthe requirement of the comprehensive understanding and pre-\\ncise alignment of the multi-modal information, VTG is a more\\nchallenging and practical task.\\nAccording to the supervision available during training, exist-\\ning VTG methods can be divided into two categories: fully-\\nsupervised VTG [7], [8], [10], [11] and weakly-supervised\\nManuscript received 21 May 2022; revised 7 August 2022 and 1 Septem-\\nber 2022; accepted 3 September 2022. Date of publication 9 September 2022;\\ndate of current version 1 November 2023. This work was supported in part\\nby the National Natural Science Foundation of China under Grants 62132016,\\n62171343, and 62071361, in part by the Key Research and Development Pro-\\ngramofShaanxiunderGrant2021ZDLGY01-03,andinpartbytheFundamental\\nResearch Funds for the Central Universities under Grant ZDRC2102. The As-\\nsociate Editor coordinating the review of this manuscript and approving it for\\npublication was Dr Xinxiao Wu. (Corresponding author: Cheng Deng.)\\nThe authors are with the School of Electronic Engineering, Xidian University,\\nXi’an 710071, China (e-mail: zhexu@stu.xidian.edu.cn; weikunsk@gmail.com;\\nxuyang.xidian@gmail.com; chdeng.xd@gmail.com).\\nDigital Object Identifier 10.1109/TMM.2022.3205404\\nFig. 1.\\nIllustration of different levels of supervision for VTG. (1) For full su-\\npervision, the start and end time in a video corresponding to a language query are\\nrequired to be annotated. (2) For weak supervision, annotators are only asked to\\nidentify whether the activity corresponding to a language query exists in a video\\nor not. (3) For point supervision, only a single temporal point corresponding to\\na language query need to be annotated.\\nVTG [9], [12], [13]. Specifically, fully-supervised VTG requires\\nannotators to label the accurate start and end time in a video cor-\\nresponding to a language query. Despite the remarkable success,\\nit is time-consuming and expensive to obtain such fine-grained\\nannotations. Thus, weakly-supervised VTG is proposed to re-\\nduce the expensive cost of annotating temporal boundary. In\\nthis setting, annotators are only asked to identify whether the\\nactivity corresponding to a language query exists in a video or\\nnot. While weakly-supervised methods can significantly reduce\\nthe annotation cost, their performances are apparently inferior\\nto the fully-supervised ones.\\nIn order to narrow the performance gap with affordable cost,\\nwe are motivated to introduce an intermediate form supervision,\\ni.e., point supervision, for VTG. Concretely, only a single tem-\\nporal point of a video relevant to a language query needs to be an-\\nnotated in the point-supervised setting. Fig. 1 shows an example\\nof different levels of supervision for VTG. Given an untrimmed\\nvideo and a language query the person puts down the bag, full\\nsupervision needs annotators to watch the video repeatedly to\\ndetermine the accurate start and end time corresponding to the\\nquery, which is time-consuming and labor-intensive. For weak\\nsupervision, annotators can only watch the video once to iden-\\ntify whether the activity corresponding to the query exists in the\\nvideo or not. The proposed point supervision requires negligible\\nextra annotating cost compared with the weakly-supervised one,\\nyet provides a coarse-grained location of the query.\\nSince the point annotation is available during training, how\\nto use it becomes an imperative problem for point-supervised\\nVTG. Intuitively, maximizing the matching score between the\\nannotated video point and the language query can be employed\\nto train the model. The extreme sparsity of label, however, is\\nlikely to enforce the model only pay attention to the annotated\\n1520-9210 © 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.\\nSee https://www.ieee.org/publications/rights/index.html for more information.\\nAuthorized licensed use limited to: XIDIAN UNIVERSITY. Downloaded on November 11,2023 at 03:52:33 UTC from IEEE Xplore.  Restrictions apply. \\n\\n\\n6122\\nIEEE TRANSACTIONS ON MULTIMEDIA, VOL. 25, 2023\\nvideo segments and thus leads to unsatisfying grounding per-\\nformance, as shown in our experiments. Recently, pseudo la-\\nbel generation methods [14], [15] have been employed in other\\npoint-supervised video tasks. But these methods are limited in\\nvideo modality with pre-defined categories and require anno-\\ntated points belonging to different classes to determine the ac-\\ntion boundary. In contrast, point-supervised VTG involves both\\nvideo and language modalities with complex multi-modal in-\\nteractions. How to locate the temporal boundary relevant to a\\nlanguage query under the point supervision is a more challeng-\\ning problem.\\nIn this paper, we present a novel method, dubbed PS-VTG, for\\npoint-supervised VTG. A language activation sequence (LAS)\\nis first obtained by an attention-based grounding network. Since\\nthere is no annotated points belonging to other classes avail-\\nable for reference, segment-level pseudo labels are proposed\\nto be generated in a “candidate + selecting” manner. Specifi-\\ncally, a set of candidate labels are generated based on the LAS\\nand pre-defined activity thresholds. Then the final pseudo la-\\nbel is selected after filtering out the irrelevant and overlapping\\nones by the point supervision and Non-Maximum Suppression\\n(NMS). In addition, cross-modal contrast and multi-level distri-\\nbution calibration are introduced to further enforce the model\\nto precisely highlight the language-relevant segments. Our key\\ncontributions can be summarized as follow:\\n1) To the best of our knowledge, we are the first to introduce\\npoint supervision for VTG, which aims to narrow the per-\\nformance gap between weakly- and fully-supervised VTG\\nwith affordable annotation cost.\\n2) We present a novel method for point-supervised VTG,\\nwhere pseudo segment-level labels are generated based\\non the point supervision to effectively train the model.\\n3) We propose cross-modal contrast and multi-level distribu-\\ntion calibration to precisely align the video and language,\\nassisting to highlight the video segments relevant to the\\nquery.\\n4) Extensive experiments on three benchmark datasets\\ndemonstrate that our method can significantly outperform\\nthe weakly-supervised approaches and achieve compara-\\nble performance with fully-supervised ones.\\nII. RELATED WORK\\nFully-Supervised Video Temporal Grounding. Methods of\\nfully-supervised VTG can be roughly divided into two cate-\\ngories: (1) proposal-based methods, and (2) proposal-free meth-\\nods. Proposal-based methods [7], [8] follow the two-stage “pro-\\nposal + matching” paradigm, where a set of candidate proposals\\nare generated for a given video and then the most relevant time\\ninterval is selected. CTRL [7] uses a dense sliding window to\\nproduce activity proposals and proposes Cross-modal Temporal\\nRegression Localizer to generate alignment scores and location\\nregression results for candidate proposals. SAP [8] proposes a\\nframework named Semantic Activity Proposal to integrate se-\\nmantic information into the proposal generation process.\\nSince proposal-based methods need to compare all the pro-\\nposals with the language query, their computational costs are\\nextremely expensive. Therefore, proposal-free methods [11],\\n[16], [17], [18] are proposed to treat VTG as a regression prob-\\nlem and directly predict the time interval. ABLR [16] introduces\\nco-attention mechanism for VTG and proposes Attention Based\\nLocation Regression to regress the temporal coordinates of lan-\\nguage from attention weights or attended features. LGI [11] ex-\\ntractes multiple semantic phrases from a language query and\\npresents Local Global Interactions to fuse multi-modal infor-\\nmation.\\nDespite the remarkable success, it is time-consuming and ex-\\npensive to obtain the temporal boundary annotations for fully-\\nsupervised VTG.\\nWeakly-Supervised Video Temporal Grounding. Weakly-\\nsupervised VTG [9], [12], [19], [20], [21] is proposed to re-\\nduce the expensive cost of annotating temporal boundary for\\nfully-supervised VTG. TGA [9] presents Text-Guided Atten-\\ntion to highlight video segments relevant to a language query\\nand obtain a single text-dependent video feature. The network is\\ntrained by minimizing the distance between the text-dependent\\nvideo feature and the language feature. VLANet [12] proposes\\nVideo-LanguageAlignmentNetworktopruneoutirrelevantpro-\\nposals and consider various attention flows to learn multi-modal\\nalignment. Recently, CRM [19] introduces cross-sentence tem-\\nporal and semantic consistency as constraints to mine the com-\\nplex relations in videos. Strictly speaking, however, it is not a\\nweakly-supervised method since temporal order is employed as\\nan additional supervision.\\nWhile weakly-supervised methods can significantly reduce\\nthe annotation cost, their performances fall largely behind the\\nfully-supervised ones.\\nPoint Supervision. In order to achieve a better trade-off be-\\ntweentest performanceandtrainingannotationcost, point super-\\nvision [14], [15], [22], [23], [24], [25], [26] has been introduced\\nto some computer vision tasks. Bearman et al. [22] first introduce\\npoint supervision for semantic segmentation. Mettes et al. [23]\\napply point supervision for spatio-temporal action localization\\nin video. Recently, SF-Net [14] proposes to use single-frame\\nsupervision for video temporal action localization. Li et al. [15]\\npresent a temporal action segmentation model using only times-\\ntamps annotations.\\nIn this paper, we introduce the point supervision for VTG.\\nDifferent previous tasks which are limited in uni-modality and\\npre-defined categories, VTG involves complex video and lan-\\nguage interactions, determining PS-VTG to be more suitable\\nfor real-world situations.\\nIII. PROPOSED METHOD\\nGiven an untrimmed video V , a language query Q, and a point\\nannotation tp, point-supervised VTG aims to learn a model f\\nthat can predict the time interval I = (ts, te) in the video cor-\\nresponding to the query: f : (V, Q, tp) →I. ts and te represent\\nthe start and end time, respectively. Notably, tp is a timestamp\\nwithin I, i.e., ts ≤tp ≤te, and the only supervision available\\nduring training.\\nFig. 2 shows the framework of our proposed method. Video\\nsegment features and word-level language features are first\\nAuthorized licensed use limited to: XIDIAN UNIVERSITY. Downloaded on November 11,2023 at 03:52:33 UTC from IEEE Xplore.  Restrictions apply. \\n\\n\\nXU et al.: POINT-SUPERVISED VIDEO TEMPORAL GROUNDING\\n6123\\nFig. 2.\\nThe framework of our proposed point-supervised VTG method. Given video segment features and word-level query features extracted from pre-trained\\nmodels, a fine-grained interaction framework is first employed to fuse the multi-modal information. Then we obtain the language activation sequence (LAS) based\\non video segment features and pooled query feature. Pseudo label is generated by the point supervision and predicted LAS. In addition, cross-modal contrast and\\nmulti-level distribution calibration are proposed to precisely align the video and language.\\nFig. 3.\\nIllustration of the proposed pseudo label generator.\\nextracted from pre-trained video and language models, respec-\\ntively. Then we fuse the video and language features with a\\nfine-grained interaction network to obtain the language acti-\\nvation sequence (LAS). The elaborated label generator is em-\\nployed to generate segment-level pseudo label based on the\\npoint supervision and LAS. In addition, cross-modal contrast\\nand multi-level distribution calibration are performed to better\\nalign the video with the language query.\\nIn the following, we elaborate the main components of the\\nproposed method, i.e., (1) grounding network, (2) pseudo label\\ngeneration, (3) cross-modal contrast, (4) multi-level distribution\\ncalibration, and (5) training and inference.\\nA. Grounding Network\\nSuppose\\nXi ∈RdV ×LV\\nand\\nQi ∈RdQ×LQ\\n(i =\\n1, 2, . . . , N) as the video segment features and query word\\nfeatures extracted from pre-trained video and language models,\\nrespectively. dV is the dimension of video segment features, dQ\\nis the dimension of query word features, LV is the number of\\nsegments per video, LQ is the number of words per sentence,\\nand N is the batch-size. We first embed Xi and Qi into a com-\\nmon space with dimension d. Then a fine-grained interaction\\nnetwork are employed to obtain semantic-aware video segment\\nfeatures V i ∈Rd×LV and global query feature qi ∈Rd.\\nSpecifically, segment-wise query representation Si ∈Rd×LV\\nis first learned by attending word-level features based on each\\nvideo segment feature:\\nSi\\nt =\\nLQ\\n\\u0002\\nl=1\\nλtlQi\\nl,\\n(1)\\nwhere λtl is the normalized attention weight computed by\\nthe t-th segment feature, i.e, λtl =\\nexp(rtl)\\n\\u0003LQ\\nk=1 exp(rtk)\\nand rtl =\\nwT\\nr tanh(WQQi\\nl + WXXi\\nt). WQ, WX ∈Rd×d and wT\\nr are\\nthe learnable matrices.\\nThen we fuse Si and Xi to obtain semantic-aware video\\nsegment features V i:\\nV i\\nt = WV((Xi\\nt ⊙Si\\nt)||Si\\nt)),\\n(2)\\nwhere WV ∈Rd×2 d is a learnable embedding matrix. ⊙and ||\\nrepresent the element-wise multiplication and the concatenation\\nof two features, respectively.\\nIn addition, the global query feature qi is obtained by\\nweighted pooling:\\nqi = QiSoftmax(WGQi)⊤,\\n(3)\\nwhere WG ∈R1×d is a learnable matrix.\\nLAS ai is obtained based on V i and qi:\\nai\\nt = Sigmoid(Wa(V i\\nt||qi)),\\n(4)\\nwhere V i\\nt (t = 1, 2, . . . , LV ) is the t-th segment feature of V i\\nand ai\\nt is the predicted activation value of V i\\nt. || represents the\\nconcatenation of two features and Wa ∈R1×2 d is a learnable\\nembedding matrix.\\nAuthorized licensed use limited to: XIDIAN UNIVERSITY. Downloaded on November 11,2023 at 03:52:33 UTC from IEEE Xplore.  Restrictions apply. \\n\\n\\n6124\\nIEEE TRANSACTIONS ON MULTIMEDIA, VOL. 25, 2023\\nB. Pseudo Label Generation\\nSince point-annotation is available in the proposed point-\\nsupervised setting, how to use it to train the grounding model be-\\ncomes a imperative problem. In this paper, we present a pseudo\\nlabel generator, as shown in Fig. 3, to generate segment-level\\npseudo label based on the point supervision and LAS.\\nGiven the predicted LAS ai, we first generate a set of candi-\\ndates according to pre-defined activity thresholds. Specifically,\\na video clip is selected to be a candidate if the activation values\\nwithin it consecutively greater than the threshold. In addition, a\\nmatching score is computed for each candidate. Taking a can-\\ndidate (ts, te) as an example, the matching score m is defined\\nas:\\nm = 1\\nl\\nte\\n\\u0002\\nt=ts\\nai\\nt −\\n1\\nLV −l\\n\\u0004ts−1\\n\\u0002\\nt=1\\nai\\nt +\\nLV\\n\\u0002\\nt=te+1\\nai\\nt\\n\\u0005\\n,\\n(5)\\nwhere ts, te, and l = te −ts + 1 are the start index, end index,\\nand length of the candidate, respectively. The first and second\\nterms represent the mean activation of the candidate and its tem-\\nporal neighbors, respectively.\\nThen the candidates that do not include the point annotation\\nwill be filtered out. Non-Maximum Suppression (NMS) is per-\\nformed to remove overlapping candidates and pseudo label p is\\nobtained from the left candidates with greatest matching score:\\npi\\nt =\\n\\u0006\\n1, ˆ\\nts ≤t ≤ˆ\\nte\\n0, otherwise\\n(t = 1, 2, . . . , LV ) ,\\n(6)\\nwhere ˆ\\nts and ˆ\\nte are the start and end index of the selected can-\\ndidate, respectively.\\nThe binary cross entropy loss is employed to enforce the pre-\\ndicted ai to be close to the generated pseudo label pi:\\nLBCE = 1\\nN\\nN\\n\\u0002\\ni=1\\n\\u0007\\npi log ai +\\n\\u0007\\n1 −pi\\b\\nlog\\n\\u0007\\n1 −ai\\b\\b\\n.\\n(7)\\nSince the generated pseudo label is binary and includes the\\npoint supervision, the model trained with LBCE will gradually\\nhighlight the video segments semantically relevant to the lan-\\nguage query.\\nC. Cross-Modal Contrast\\nIn order to precisely align the video and language, we present\\ncross-modal contrast based on the point supervision. Specifi-\\ncally, given a language query qi as an anchor, we select video\\nsegment vi including the point annotation as a positive sam-\\nple. Negative samples come from both the intra- and inter\\nvideos, i.e., unmatched segments determined by the pseudo label\\nand segments corresponding to other queries in the mini-batch.\\nVideo2query contrastive loss LCON_V 2Q is employed to ob-\\ntain discriminative representation by pulling the anchor and the\\npositive sample together while pushing away the anchor from\\nFig. 4.\\nIllustration of the optimal distribution of LAS and two local optimums.\\nnegative samples, which can be defined as:\\nLCON_V 2Q = −1\\nN\\nN\\n\\u0002\\ni=1\\n× log\\nes(qi,vi)/τ\\nes(qi,vi)/τ +\\u0003N\\nk̸=i es(qi,vk)/τ +\\u0003M\\nj=1 es(qi,vj)/τ ,\\n(8)\\nwhere s(x, y) = x⊤y/||x||||y|| denotes cosine similarity and τ\\nis the temperature parameter. N and M are the batch-size and\\nthe number of negative intra-video samples, respectively.\\nAnalogously, we define the query2video contrastive loss\\nwhere video segment is treated as an anchor and queries as con-\\ntrastive samples:\\nLCON_Q2V = −1\\nN\\nN\\n\\u0002\\ni=1\\nlog\\nes(vi,qi)/τ\\nes(vi,qi)/τ + \\u0003N\\nk̸=i es(vi,qk)/τ.\\n(9)\\nThe total cross-modal contrastive loss is the sum of\\nvideo2query and query2video contrastive loss:\\nLCON = LCON_V 2Q + LCON_Q2V .\\n(10)\\nD. Multi-Level Distribution Calibration\\nIdeally, the distribution of predicted ai along time is plateau-\\nlike, i.e., the activation values of query-relevant segments in\\nthe middle are close to 1 while close to 0 otherwise. To this\\nend, we propose multi-level distribution calibration to restrict\\nthe distribution of predicted language activation sequence and\\nenforce the model to precisely focus on the relevant segments.\\nFor intra-video distribution calibration, the mean activation\\nwithin the time interval of pseudo label is encouraged to be\\nhigher than that outside the interval. The intra-video distribution\\ncalibration loss is thus defined as:\\nLMDC_R =\\n1\\nN\\nN\\n\\u0002\\ni=1\\nmax\\n⎛\\n⎝0,\\n\\u0003 ˆ\\nts−1\\nt=1 ai\\nt + \\u0003LV\\nt= ˆ\\nte+1 ai\\nt\\nLV −ˆ\\nl\\n−\\n\\u0003 ˆ\\nte\\nt= ˆ\\nts ai\\nt\\nˆ\\nl\\n+ δ\\n⎞\\n⎠,\\n(11)\\nwhere ˆ\\nts, ˆ\\nte, and ˆ\\nl = ˆ\\nte −ˆ\\nts + 1 represent the start index, end\\nindex and length of the pseudo label, respectively. LV is the\\nnumber of segments per video and δ is the margin (set to 0.5).\\nDespite the ability of intra-video distribution calibration to\\nmake the activation within and outside the interval distinguish-\\nable, the predicted ai is still likely to be local optimum. Con-\\ncretely, the value of ai may close to 1 only at the annotated point\\nor at every points, i.e., \\u0003LV\\nt=1 ai\\nt = 1 or \\u0003LV\\nt=1 ai\\nt = LV . Fig. 4\\nAuthorized licensed use limited to: XIDIAN UNIVERSITY. Downloaded on November 11,2023 at 03:52:33 UTC from IEEE Xplore.  Restrictions apply. \\n\\n\\nXU et al.: POINT-SUPERVISED VIDEO TEMPORAL GROUNDING\\n6125\\nshows the distribution of the optimal LAS (4a) and two local\\noptimums (4b and 4c).\\nTo avoid the local optimums, we propose inter-video distribu-\\ntion calibration. The mean activation of videos in a mini-batch\\nis forced to subject a Gaussian distribution with middle value\\nmean. The inter-video distribution calibration loss can be de-\\nfined as:\\nLMDC_E =\\n\\n\\n\\n\\n\\n1\\nN\\nN\\n\\u0002\\ni=1\\n\\u0003LV\\nt=1 ai\\nt\\nLV\\n−μ\\n\\n\\n\\n\\n\\n+\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n1\\nN\\nN\\n\\u0002\\ni=1\\n\\u0003LV\\nt=1\\n\\u0004\\nai\\nt −\\n\\u0003LV\\nt=1 ai\\nt\\nLV\\n\\u00052\\nLV\\n−σ\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n,\\n(12)\\nwhere the first and the second terms of the loss function are\\nemployed to restrain the mean and variance of the predicted\\nactivations, respectively. μ and σ are the mean and variance\\nof pre-defined Gaussian distribution, respectively. As verified\\nlater in the experiments, a middle value mean will encourage\\nthe model to precisely focus on the segments relevant to the\\nlanguage query.\\nThe multi-level distribution calibration loss LMDC is the sum\\nof intra- and inter-video distribution calibration loss:\\nLMDC = LMDC_R + LMDC_E.\\n(13)\\nE. Training and Inference\\nIn summary, three loss functions are employed for training,\\nnamely: binary cross entropy loss LBCE, cross-modal con-\\ntrastive loss LCON, and multi-level distribution calibration loss\\nLMDC. The overall objective function can be formulated as:\\nL = LBCE + αLCON + βLMDC,\\n(14)\\nwhere α and β are the weighting coefficients for the cross-modal\\ncontrastive loss and the multi-level distribution calibration loss,\\nrespectively.\\nDuring inference, given an untrimmed video and a language\\nquery, we first obtain the LAS by the grounding network. Then\\nthe final grounding result is determined by a similar process\\nto pseudo label generation without filtering out candidates ir-\\nrelevant to point supervision, i.e., generating candidates then\\nselecting top1 by NMS.\\nIV. EXPERIMENTS\\nIn this section, the datasets and evaluation metrics we used are\\nfirst introduced in detail. Then, we present the implementation\\ndetails and the simulation of point supervision of our method.\\nFinally, the analysis, comparison with state-of-the-art methods,\\nand qualitative results further prove the effectiveness of our pro-\\nposed approach.\\nA. Datasets\\nWe evaluate the proposed method on three public datasets:\\nActivityNet Captions [27], Charades-STA [7], and TACoS [28].\\nActivityNet Captions. The ActivityNet Captions dataset\\ncontains 10,024, 4,926 and 5,044 videos for training, validation,\\nand testing, respectively. The average length of language query\\nand average number of activities per video are 13.48 words and\\n3.65, respectively. We use the validation set for evaluation since\\nthe testing set is not publicly available.\\nCharades-STA. The Charades-STA dataset is built on Cha-\\nrades dataset for video temporal grounding. It contains 12,408\\nand 3,720 clip-language pairs for training and testing, respec-\\ntively. The average length of language query, average duration of\\nvideos and average number of activities per video are 8.6 words,\\n29.8 seconds, and 2.3 respectively.\\nTACoS. The TACoS dataset is built on MPII-Compositive\\ndataset and consists of 10,146, 4,589, and 4,083 instances for\\ntraining, validation, and testing, respectively. The average dura-\\ntion of videos is around 300 s.\\nB. Evaluation Metrics\\nFollowing previous works, we report two metrics to measure\\nthe performance of video temporal grounding: IoU = n and\\nmIoU.\\nRank@1 IoU = n. Rank@1 IoU = n is referred to as the\\npercentage of test samples whose Intersection over Union (IoU)\\nwith ground-truth (GT) is higher than n. n = {0.3, 0.5, 0.7} are\\nreported in our experiments.\\nmIoU. mIoU is the mean IoU over all test samples.\\nC. Implementation Details\\nWe use pre-trained C3D [29], I3D [30], and C3D models to\\nextract video features for the ActivityNet Captions, Charades-\\nSTA, and TACoS datasets, respectively. Each video is uniformly\\nsampled into LV = 128 segments and embedded into a dV ×\\nLV dimensional representation space by an embedding matrix.\\nGlove [31] is employed to extract word features of the language\\nquery. The dimensions of the video segment dV , the language\\nquery dQ, and the common space d are set to 1024, 300, and\\n128, respectively.\\nOur approach is implemented with PyTorch [32] and opti-\\nmized by ADAM [33] optimizer with a learning rate of 0.0001.\\nWe set the batch size N to 64 and training epoch to 200. The\\nloss weights α and β are determined by the grid search and set\\nto 0.1 and 0.1 respectively. In addition, the activity thresholds\\nfor generating candidates and the threshold in NMS for filtering\\nout overlapping are set to (0.4, 0.6, 0.1) and 0.4, respectively.\\nThe temperature parameter τ in cross-modal contrast is set to\\n1. μ and σ for inter-video distribution calibration are set to (0.3,\\n0.1), (0.3, 0.01), and (0.1, 0.01) on the ActivityNet Captions,\\nCharades-STA, and TACoS datasets, respectively.\\nD. Simulation of Point Supervision\\nSince the datasets used for evaluation do not have point an-\\nnotations themselves, we simulate the point supervision based\\nAuthorized licensed use limited to: XIDIAN UNIVERSITY. Downloaded on November 11,2023 at 03:52:33 UTC from IEEE Xplore.  Restrictions apply. \\n\\n\\n6126\\nIEEE TRANSACTIONS ON MULTIMEDIA, VOL. 25, 2023\\nTABLE I\\nPERFORMANCE COMPARISON WITH WEAKLY- AND FULLY-SUPERVISED\\nCOUNTERPARTS ON THE ACTIVITYNET CAPTIONS, CHARADES-STA, AND\\nTACOS DATASETS\\non the existing boundary annotations. Following the strategy\\nin [34], we generate two sets of point annotations, i.e., PS-U and\\nPS-G, from different distributions. Specifically, given s and e as\\nthe start and end time corresponding to a language query, the\\npoint annotation belonging to PS-U is randomly sampled from\\nthe uniform distribution [s, e]. In addition, we randomly sample\\npoints from a Gaussian distribution with mean (s + e)/2 and\\nstandard deviation 1 s to form the set PS-G.\\nE. Analysis\\nGains from the point supervision. To investigate the gains\\nof point supervision, we compare our point-supervised method\\nwith its weakly- and fully-supervised counterparts. Specifically,\\nthe weakly-supervised model is trained only by cross-modal\\ncontrastive loss, where the video segments are pooled by the\\npredicted language activation sequence to obtain a video fea-\\nture. For fully-supervised counterpart, we replace the gener-\\nated pseudo label with ground-truth one to train the model. Our\\npoint-supervised model is trained with the binary cross entropy\\nloss LBCE, cross-modal contrastive loss LCON, and multi-level\\ndistribution calibration loss LMDC.\\nTable I shows the results under different levels of supervi-\\nsion on the ActivityNet Captions, Charades-STA, and TACoS\\ndatasets. Our model trained with point supervision outperform\\nthe weakly-supervised counterpart with a large margin, e.g.,\\n32.17%, 27.20%, 14.47% and 21.26% in terms of IoU = 0.3,\\nIoU = 0.5, IoU = 0.7, and mIoU on the ActivityNet Captions\\ndataset.Inaddition,comparableresultswiththefully-supervised\\ncounterpart are achieved by the point-supervised model. Consid-\\nering that point supervision requires negligible extra annotation\\ncost comparing with weak supervision, the effectiveness of point\\nsupervision is clearly verified in this experiment.\\nThe annotation cost of point supervision. Given an\\nuntrimmed video and the language query, full supervision needs\\nannotators to watch the video repeatedly to determine the ac-\\ncurate start and end time corresponding to the query, which\\nis time-consuming and labor-intensive. For weak supervision,\\nannotators can only watch the video once to identify whether\\nTABLE II\\nTHE ANNOTATION COST OF DIFFERENT SUPERVISIONS ON THE ACTIVITYNET\\nCAPTIONS DATASET\\nTABLE III\\nEFFECTIVENESS OF PSEUDO LABEL GENERATION ON THE ACTIVITYNET\\nCAPTIONS, CHARADES-STA, AND TACOS DATASETS. LCON,\\nLCON + LBCE−P S, AND LCON + LBCE DIFFER IN THE USE OF POINT\\nSUPERVISION\\nthe activity corresponding to the query exists in the video or not.\\nThe proposed point supervision requires only a few extra pauses\\nto annotate a temporal point, which is negligible extra annotating\\ncost compared with the weakly-supervised one. In order to illus-\\ntrate the differences among supervisions, we randomly annotate\\n100 instances of the ActivityNet Captions dataset.\\nTable II shows the average annotating time of three supervi-\\nsions. The average annotating time at full, point, and weak levels\\nare 504 s, 185 s, and 180 s, while the mIoU are 43.73%, 41.49%,\\nand 20.23%, respectively. The results proves our method obtains\\nimpressive improvement with little increase of annotation cost\\ncompared with weakly-supervised methods.\\nEffectivenessofpseudolabelgeneration.Givenpointsuper-\\nvisionavailableduringtraining,howtouseitbecomestheimper-\\native problem for point-supervised VTG. In this paper, we pro-\\nvide a novel solution. i.e., generating segment-level pseudo label\\nbased on the point supervision and predicted LAS. To investi-\\ngate the effectiveness of pseudo label generation, we compare\\nour method with two baseline approaches in this experiment:\\n(1) LCON and (2) LCON + LBCE−P S. Specifically, LCON is\\nthe baseline approach trained only with cross-modal contrastive\\nloss. In this case, point supervision is used to select segment\\nas contrastive samples from a video. The baseline approach\\nLCON + LBCE−P S is trained with cross-modal contrastive loss\\nand binary cross entropy loss only at the annotated point. In con-\\ntrast, our method is trained with cross-modal contrastive loss and\\nbinary cross entropy loss at all points, i.e., LCON and LBCE.\\nAs shown in Table III, our method trained with LCON and\\nLBCE significantly outperform the two baseline approaches on\\nthe ActivityNet Captions, Charades-STA, and TACoS datasets,\\nwhich indicates the effectiveness of the proposed pseudo label\\ngeneration.\\nAuthorized licensed use limited to: XIDIAN UNIVERSITY. Downloaded on November 11,2023 at 03:52:33 UTC from IEEE Xplore.  Restrictions apply. \\n\\n\\nXU et al.: POINT-SUPERVISED VIDEO TEMPORAL GROUNDING\\n6127\\nFig. 5.\\nLoss and mIoU vs. iteration using the binary cross entropy LBCE on\\nthe Charades-STA dataset.\\nTABLE IV\\nINFLUENCE OF POINT SUPERVISION SIMULATION METHODS ON THE\\nACTIVITYNET CAPTIONS, CHARADES-STA, AND TACOS DATASETS. PS-U AND\\nPS-G REPRESENT POINT ANNOTATIONS SIMULATED FROM DIFFERENT\\nDISTRIBUTIONS\\nIn order to investigate the convergence of training LAS using\\nthe generated pseudo label. We train the model with only the bi-\\nnary cross entropy LBCE and present the training loss as well as\\nmIoU during iteration on the Charades-STA dataset. As shown\\nin Fig. 5, the loss decreases and mIoU increases during the train-\\ning process, which verifies the convergence of training LAS us-\\ning the generated pseudo label. We attribute the convergence to\\nelaborated pseudo label generator: (1) The candidates are\\ngenerated according to pre-defined activity thresholds (set to\\n(0.4:0.6:0.1) in the experiments), which guarantees the consec-\\nutiveness of the activation values in a candidate. (2) The candi-\\ndates that do not include the point supervision are filtered out,\\nwhich makes full use of the point information. (3) The pseudo\\nlabel is set to be binary while the LAS is continuous. Minimizing\\nLBCE will enforce the predicted LAS to be close to the gener-\\nated pseudo label and make the model gradually highlight the\\nvideo segments semantically relevant to the language query.\\nInfluence of point supervision simulation methods. As dis-\\ncussed in the last subsection, we simulate two sets of point-\\nannotations, i.e., PS-U and PS-G, from different distributions. In\\nthis experiment, we aim to investigate the influence of the point\\nsupervision simulation methods. The models are trained with the\\nbinary cross entropy loss LBCE, cross-modal contrastive loss\\nLCON, and multi-level distribution calibration loss LMDC on\\nthe ActivityNet Captions, Charades-STA, and TACoS datasets.\\nTable IV presents the experimental results. Similar results are\\nobtainedbasedonthetwosimulationmethods.Pointannotations\\nfrom PS-U are empolyed in rest experiments.\\nContribution of multi-level distribution calibration. In or-\\nder to verify the effectiveness of multi-level distribution calibra-\\ntion, we present an ablation study on the ActivityNet Captions,\\nTABLE V\\nEFFECTIVENESS OF MULTI-LEVEL DISTRIBUTION CALIBRATION ON THE\\nACTIVITYNET CAPTIONS, CHARADES-STA, AND TACOS DATASETS. MDC_R,\\nMDC_E, AND MDC REPRESENT THE INTRA-VIDEO DISTRIBUTION\\nCALIBRATION, INTER-VIDEO DISTRIBUTION CALIBRATION, AND OVERALL\\nMULTI-LEVEL DISTRIBUTION CALIBRATION, RESPECTIVELY\\nTABLE VI\\nCHOICE OF μ AND σ ON THE ACTIVITYNET CAPTIONS DATASET\\nCharades-STA,andTACoSdatasets.Fourvariantsofourmethod\\nare compared in this experiment: (1) PS-VTG, (2) w/o MDC_R,\\n(3) w/o MDC_E, and (4) w/o MDC. Specifically, PS-VTG is\\nour full model trained with the binary cross entropy loss LBCE,\\ncross-modal contrastive loss LCON, and multi-level distribu-\\ntion calibration loss LMDC. w/o MDC_R, w/o MDC_E, and w/o\\nMDC are the full model without intra-video distribution calibra-\\ntion, inter-video distribution calibration, and overall multi-level\\ndistribution calibration, respectively.\\nWe show the experimental results in Table V. The perfor-\\nmance largely drops when the intra-video or inter-video distri-\\nbution calibration is removed, which indicates the effectiveness\\nof multi-level distribution calibration.\\nChoice of μ and σ. Inter-video distribution calibration is pro-\\nposed to force the mean activation of videos in a mini-batch sub-\\nject a Gaussian distribution with middle value mean. We perform\\nexperiments with different Gaussian distribution to determine\\nthe optimal mean μ and variance σ of the Gaussian distribution.\\nTable VI shows the results on the ActivityNet Captions\\ndataset. A middle mean can bring about different improvements\\nandtheoptimalperformanceisobtainedwhenμ = 0.3, σ = 0.1.\\nF. Comparison With State-of-The-Art Methods\\nWe compare the proposed method with several state-of-the-\\nart methods at different levels of supervison. Among these\\nmethods, TGA [9], WSLLN [35], VLANet [12], SCN [20],\\nRTBPN\\n[13], BAR [21], LCN [36], and LoGAN [45] are\\nAuthorized licensed use limited to: XIDIAN UNIVERSITY. Downloaded on November 11,2023 at 03:52:33 UTC from IEEE Xplore.  Restrictions apply. \\n\\n\\n6128\\nIEEE TRANSACTIONS ON MULTIMEDIA, VOL. 25, 2023\\nweakly-supervised methods. CRM [19], as discussed in re-\\nlated works, requires additional temporal order supervision and\\nthus is a special weakly-supervised approach. While CTRL [7],\\nSAP[8],MLVI[37],TripNet[38],RWM[39],DEBUG[40],2D-\\nTAN [10], ABLR [16], GDP [17], VSLNet [41], LGI [11],\\nDRN [42], PMI-LOC [18], IVG-DCL [43] and MMN [44]\\nare fully-supervised ones. To make the presented paper self-\\ncontained,wegiveabriefintroductionofthemethodsmentioned\\nabove.\\nr TGA presents Text-Guided Attention to highlight video\\nsegments relevant to a language query and obtain a single\\ntext-dependent video feature. The network is trained by\\nminimizing the distance between the text-dependent video\\nfeature and the language feature.\\nr WSLLN proposes a two-branch network to measure\\nsegment-text consistency and conduct segment selection\\n(conditioned on the text) simultaneously.\\nr VLANet proposes Video-Language Alignment Network to\\nprune out irrelevant proposals and consider various atten-\\ntion flows to learn multimodal alignment.\\nr SCN proposes a Semantic Completion Network to measure\\nthe semantic similarity between proposals and query.\\nr RTBPN presents a Regularized Two-Branch Proposal Net-\\nwork for weakly-supervised VTG. A language-aware filter\\nis employed to generate the enhanced and suppressed video\\nstreams to generate positive and negative proposals.\\nr BAR proposes Boundary Adaptive Refinement framework\\nthat resorts to reinforcement learning to guide the process\\nof progressively refining the temporal boundary.\\nr LCN proposes a Local Correspondence Network which\\nconsists of hierarchical feature representation, cycle-\\nconsistent local correspondence, and relation modeling\\namong candidate moments to explore the fine-grained cor-\\nrespondences between video and text.\\nr CRM introduces cross-sentence temporal and semantic\\nconsistency as constraints to mine the complex relations\\nin videos, where temporal order is employed as an addi-\\ntional supervision.\\nr CTRL is a pioneering VTG work. It leverages sliding win-\\ndowtoobtaincandidateclipsofvariouslengthsandfusethe\\ncandidate representations with the sentence representation\\nby three operators (i.e.,add, multiply, and fully-connected\\nlayer) to predict the alignment score.\\nr SAP proposes a Semantic Activity Proposal framework\\nthat integrates the semantic information of sentence queries\\ninto the proposal generation process to get discriminative\\nactivity proposals. Visual and semantic information are\\njointly utilized for proposal ranking and refinement.\\nr MLVI introduces a multilevel feature integration model to\\nfuse language and vision earlier and more tightly.\\nr MAN presents a Moment Alignment Network where lan-\\nguage query is integrated as dynamic filters. In addition,\\nan iterative graph adjustment network is devised to model\\nmoment-wise temporal relations.\\nr TripNet introduces an end-to-end reinforcement learning\\nframework that uses a gated-attention mechanism over\\ncross-modal features.\\nTABLE VII\\nPERFORMANCE COMPARISON WITH STATE-OF-THE-ART METHODS AT\\nDIFFERENT LEVELS OF SUPERVISION ON THE ACTIVITYNET CAPTIONS\\nDATASET. * INDICATES ADDITIONAL SUPERVISION IS EMPLOYED\\nr RWM views VTG as controlling an agent to read the de-\\nscription, to watch the video as well as the current localiza-\\ntion, and then to move the temporal grounding boundaries\\niteratively to find the best matching clip.\\nr 2D-TAN proposes to model the temporal relations between\\nvideo candidates by a two-dimensional map, where one\\ndimension indicates the starting time of a moment and the\\nother indicates the end time. The 2D temporal map can\\ncover diverse video moments with different lengths, while\\nrepresenting their adjacent relations.\\nr ABLR proposes a Attention Based Location Regression\\nmodel to directly regress the temporal coordinates from the\\nglobal attention outputs with a multi-modal co-attention\\nmechanism.\\nr GDP designs a novel bottom-up model: Graph-FPN with\\nDense Predictions. It first generates a frame feature pyra-\\nmid to capture multi-level semantics, then utilizes graph\\nconvolution to encode the plentiful scene relationships.\\nr VSLNet proposes a video span localizing network on top of\\nthe standard span-based QA framework with query-guided\\nhighlighting strategy.\\nr LGI introduces a sequential query attention module to\\nextract representations of multiple and distinct semantic\\nphrases from a text query. Then Local-Global video-text\\nInteraction algorithm is employed to model the relationship\\nbetween video segments and semantic phrases in multiple\\nlevels.\\nr DRN proposes a Dense Regression Network for VTG,\\nwhich provides a new perspective to leverage dense su-\\npervision from the sparse annotations.\\nr PMI-LOC proposes pairwise modality interaction in both\\nthe sequence and channel levels to better understand video\\ncontents.\\nAuthorized licensed use limited to: XIDIAN UNIVERSITY. Downloaded on November 11,2023 at 03:52:33 UTC from IEEE Xplore.  Restrictions apply. \\n\\n\\nXU et al.: POINT-SUPERVISED VIDEO TEMPORAL GROUNDING\\n6129\\nFig. 6.\\nThe qualitative results of different models on the Charades-STA and ActivityNet Captions datasets. GT is the ground-truth time interval. Ours, w/o\\nMDC, and Base Model represent the proposed PS-VTG model, PS-VTG without multi-level distribution calibration, and model without pseudo label generation,\\nrespectively.\\nTABLE VIII\\nPERFORMANCE COMPARISON WITH STATE-OF-THE-ART METHODS AT\\nDIFFERENT LEVELS OF SUPERVISION ON THE CHARADES-STA DATASET. *\\nINDICATES ADDITIONAL SUPERVISION IS EMPLOYED\\nr IVG-DCL proposes interventional video grounding to\\neliminate the spurious correlations between query and\\nvideo features based on causal inference. In addition, a\\ndual contrastive learning approach is employed to better\\nalign the text and video.\\nTABLE IX\\nPERFORMANCE COMPARISON WITH STATE-OF-THE-ART METHODS AT\\nDIFFERENT LEVELS OF SUPERVISION ON THE TACOS DATASET\\nr MMN presents a Mutual Matching Network on a perspec-\\ntive of metric-learning to directly model the similarity\\nbetween language queries and video moments in a joint\\nembedding space.\\nTables VII, VIII, and IX show the results on the ActivityNet\\nCaptions, Charades-STA, and TACoS datasets, respectively. Our\\napproach outperforms all the weakly-supervised methods and\\nachieve comparable results with fully-supervised ones. For ex-\\nample, BAR is the best weakly-supervised VTG method on the\\nActivityNet Captions dataset and our approach outperforms it\\nwith a large margin of 10.68% and 8.86% in terms of IoU = 0.3\\nand IoU = 0.5, respectively. In addition, our approach trained\\nwith point-supervision can even outperform some recent fully-\\nsupervised methods, e.g., CTRL, MLVI, TripNet, RWM, ABLR,\\nAuthorized licensed use limited to: XIDIAN UNIVERSITY. Downloaded on November 11,2023 at 03:52:33 UTC from IEEE Xplore.  Restrictions apply. \\n\\n\\n6130\\nIEEE TRANSACTIONS ON MULTIMEDIA, VOL. 25, 2023\\nDEBUG, GDP, LGI, and PMI-LOC on the ActivityNet Captions\\ndataset.\\nG. Qualitative Results\\nIn Fig. 6, we show the qualitative results of different mod-\\nels. Our proposed model PS-VTG achieves more accurate video\\ntemporal grounding than w/o MDC without multi-level distribu-\\ntion calibration and Base Model without pseudo label generation\\n(i.e., trained with cross-model contrastive loss and binary cross\\nentropy loss only at annotated point), demonstrating the effec-\\ntiveness of our method.\\nV. CONCLUSION AND FUTURE WORK\\nIn this paper, we introduce point supervision for VTG, which\\ncan effectively narrow the performance gap between weakly-\\nand fully-supervised VTG with affordable annotation cost. To\\nmake full use of the point supervision, we generate pseudo\\nsegment-level labels based on the point supervision and the pre-\\ndicted language activation sequence. In addition, cross-modal\\ncontrast and multi-level distribution calibration are proposed to\\nprecisely align the video and language. Extensive experiments\\non three benchmark datasets demonstrate that our method can\\nsignificantly outperform the weakly-supervised approaches and\\nachieve comparable performance with fully-supervised ones.\\nExisting VTG methods rely on large-scale labeled data to train\\na model, which is time-consuming and expensive to obtain. In\\nthe future, we will explore to solve the VTG problem under the\\nlimited data [46], [47] and the limited supervision settings [48],\\n[49], e.g., Semi-Supervised VTG, and Unsupervised VTG.\\nREFERENCES\\n[1] L.-Y. Duan, M. Xu, Q. Tian, C.-S. Xu, and J. S. Jin, “A unified framework\\nfor semantic shot classification in sports video,” IEEE Trans. Multimedia,\\nvol. 7, no. 6, pp. 1066–1083, Dec. 2005.\\n[2] J. Wang, W. Wang, and W. Gao, “Multiscale deep alternative neural net-\\nworkforlarge-scalevideoclassification,”IEEETrans.Multimedia,vol.20,\\nno. 10, pp. 2578–2592, Oct. 2018.\\n[3] X. Dong et al., “Occlusion-aware real-time object tracking,” IEEE Trans.\\nMultimedia, vol. 19, no. 4, pp. 763–771, Apr. 2017.\\n[4] W. Ruan et al., “Multi-correlation filters with triangle-structure con-\\nstraints for object tracking,” IEEE Trans. Multimedia, vol. 21, no. 5,\\npp. 1122–1134, May 2019.\\n[5] Z. Zhou, F. Shi, and W. Wu, “Learning spatial and temporal extents of\\nhuman actions for action detection,” IEEE Trans. Multimedia, vol. 17,\\nno. 4, pp. 512–525, Apr. 2015.\\n[6] D. Guo, W. Li, and X. Fang, “Fully convolutional network for multi-\\nscale temporal action proposals,” IEEE Trans. Multimedia, vol. 20, no. 12,\\npp. 3428–3438, Dec. 2018.\\n[7] J. Gao, C. Sun, Z. Yang, and R. Nevatia, “Tall: Temporal activity lo-\\ncalization via language query,” in Proc. Int. Conf. Comput. Vis., 2017,\\npp. 5267–5275.\\n[8] S. Chen and Y.-G. Jiang, “Semantic proposal for activity localization in\\nvideos via sentence query,” in Proc. AAAI Conf. Artif. Intell., 2019, vol. 33,\\npp. 8199–8206.\\n[9] N. C. Mithun, S. Paul, and A. K. Roy-Chowdhury, “Weakly supervised\\nvideo moment retrieval from text queries,” in Proc. Conf. Comput. Vis.\\nPattern Recognit., 2019, pp. 11592–11601.\\n[10] S. Zhang, H. Peng, J. Fu, and J. Luo, “Learning 2D temporal adjacent\\nnetworks for moment localization with natural language,” in Proc. AAAI\\nConf. Artif. Intell., 2020, vol. 34, pp. 12870–12877.\\n[11] J. Mun, M. Cho, and B. Han, “Local-global video-text interactions for\\ntemporal grounding,” in Proc. Conf. Comput. Vis. Pattern Recognit., 2020,\\npp. 10810–10819.\\n[12] M. Ma et al., “Vlanet: Video-language alignment network for weakly-\\nsupervised video moment retrieval,” in Proc. Eur. Conf. Comput. Vis.\\nSpringer, 2020, pp. 156–171.\\n[13] Z. Zhang, Z. Lin, Z. Zhao, J. Zhu, and X. He, “Regularized two-branch\\nproposal networks for weakly-supervised moment retrieval in videos,” in\\nProc. 28th ACM Int. Conf. Multimedia, 2020, pp. 4098–4106.\\n[14] F. Ma et al., “SF-Net: Single-frame supervision for temporal ac-\\ntion localization,” in Proc. Eur. Conf. Comput. Vis. Springer, 2020,\\npp. 420–437.\\n[15] Z. Li, Y. Abu Farha, and J. Gall, “Temporal action segmentation from\\ntimestamp supervision,” in Proc. Conf. Comput. Vis. Pattern Recognit.,\\n2021, pp. 8365–8374.\\n[16] Y. Yuan, T. Mei, and W. Zhu, “To find where you talk: Temporal sentence\\nlocalization in video with attention based location regression,” in Proc.\\nAAAI Conf. Artif. Intell., 2019, vol. 33, pp. 9159–9166.\\n[17] L. Chen et al., “Rethinking the bottom-up framework for query-based\\nvideo localization,” in Proc. AAAI Conf. Artif. Intell., 2020, vol. 34,\\npp. 10551–10558.\\n[18] S. Chen, W. Jiang, W. Liu, and Y.-G. Jiang, “Learning modality interaction\\nfor temporal sentence localization and event captioning in videos,” in Proc.\\nEur. Conf. Comput. Vis. Springer, 2020, pp. 333–351.\\n[19] J. Huang, Y. Liu, S. Gong, and H. Jin, “Cross-sentence temporal and se-\\nmantic relations in video activity localisation,” in Proc. Int. Conf. Comput.\\nVis., 2021, pp. 7199–7208.\\n[20] Z. Lin, Z. Zhao, Z. Zhang, Q. Wang, and H. Liu, “Weakly-supervised video\\nmoment retrieval via semantic completion network,” in Proc. AAAI Conf.\\nArtif. Intell., 2020, vol. 34, pp. 11539–11546.\\n[21] J. Wu, G. Li, X. Han, and L. Lin, “Reinforcement learning for weakly\\nsupervised temporal grounding of natural language in untrimmed videos,”\\nin Proc. 28th ACM Int. Conf. Multimedia, 2020, pp. 1283–1291.\\n[22] A. Bearman, O. Russakovsky, V. Ferrari, and L. Fei-Fei, “What’s the point:\\nSemantic segmentation with point supervision,” in Proc. Eur. Conf. Com-\\nput. Vis. Springer, 2016, pp. 549–565.\\n[23] P. Mettes, J. C. Van Gemert, and C. G. Snoek, “Spot on: Action localiza-\\ntion from pointly-supervised proposals,” in Proc. Eur. Conf. Comput. Vis.\\nSpringer, 2016, pp. 437–453.\\n[24] P. Lee and H. Byun, “Learning action completeness from points for\\nweakly-supervised temporal action localization,” in Proc. Int. Conf. Com-\\nput. Vis., 2021, pp. 13648–13657.\\n[25] K.-K. Maninis, S. Caelles, J. Pont-Tuset, and L. Van Gool, “Deep extreme\\ncut: From extreme points to object segmentation,” in Proc. Conf. Comput.\\nVis. Pattern Recognit., 2018, pp. 616–625.\\n[26] L. Yang et al., “Background-click supervision for temporal action localiza-\\ntion,” IEEE Trans. Pattern Anal. Mach. Intell., early access, Dec. 2, 2021,\\ndoi: 10.1109/TPAMI.2021.3132058 .\\n[27] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. C. Niebles, “Dense-\\ncaptioning events in videos,” in Proc. Int. Conf. Comput. Vis., 2017,\\npp. 706–715.\\n[28] M. Rohrbach et al., “Script data for attribute-based recognition of compos-\\niteactivities,”inProc.Eur.Conf.Comput.Vis.Springer,2012,pp.144–157.\\n[29] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, “Learning\\nspatiotemporal features with 3D convolutional networks,” in Proc. Int.\\nConf. Comput. Vis., 2015, pp. 4489–4497.\\n[30] J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model\\nand the kinetics dataset,” in Proc. Conf. Comput. Vis. Pattern Recognit.,\\n2017, pp. 6299–6308.\\n[31] J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for\\nword representation,” in Proc. Conf. Empirical Methods Natural Lang.\\nProcess., 2014, pp. 1532–1543.\\n[32] A. Paszke et al., “Pytorch: An imperative style, high-performance\\ndeep learning library,” in Proc. Adv. Neural Inf. Process. Syst., 2019,\\npp. 8026–8037.\\n[33] D. P. Kingma and J. L. Ba, “Adam: A method for stochastic optimization,”\\nin Proc. Int. Conf. Learn. Representations, 2015, pp. 1–41.\\n[34] D. Moltisanti, S. Fidler, and D. Damen, “Action recognition from single\\ntimestamp supervision in untrimmed videos,” in Proc. Conf. Comput. Vis.\\nPattern Recognit., 2019, pp. 9915–9924.\\n[35] M. Gao, L. S. Davis, R. Socher, and C. Xiong, “Wslln: Weakly supervised\\nnatural language localization networks,” in Proc. Conf. Empirical Methods\\nNatural Lang. Process. 9th Int. Joint Conf. Natural Lang. Process., 2019,\\npp. 1481–1487.\\nAuthorized licensed use limited to: XIDIAN UNIVERSITY. Downloaded on November 11,2023 at 03:52:33 UTC from IEEE Xplore.  Restrictions apply. \\n\\n\\nXU et al.: POINT-SUPERVISED VIDEO TEMPORAL GROUNDING\\n6131\\n[36] W. Yang, T. Zhang, Y. Zhang, and F. Wu, “Local correspondence network\\nfor weakly supervised temporal sentence grounding,” IEEE Trans. Image\\nProcess., vol. 30, pp. 3252–3262, 2021.\\n[37] H. Xu et al., “Multilevel language and vision integration for text-to-clip\\nretrieval,” in Proc. AAAI Conf. Artif. Intell., 2019, vol. 33, pp. 9062–9069.\\n[38] M. Hahn, A. Kadav, J. M. Rehg, and H. P. Graf, “Tripping through\\ntime: Efficient localization of activities in videos,” 2019. [Online]. Avail-\\nable: http://arxiv.org/abs/1904.09936\\n[39] D. He et al., “Read, watch, and move: Reinforcement learning for tempo-\\nrally grounding natural language descriptions in videos,” in Proc. AAAI\\nConf. Artif. Intell., 2019, vol. 33, pp. 8393–8400.\\n[40] C. Lu, L. Chen, C. Tan, X. Li, and J. Xiao, “Debug: A. dense bottom-\\nup grounding approach for natural language video localization,” in Proc.\\nEmpirical Methods Natural Lang. Process., 2019, pp. 5144–5153.\\n[41] H. Zhang, A. Sun, W. Jing, and J. T. Zhou, “Span-based localizing net-\\nwork for natural language video localization,” in Proc. 58th Annu. Meeting\\nAssoc. Comput. Linguistics, 2020, pp. 6543–6554.\\n[42] R. Zeng et al., “Dense regression network for video grounding,” in Proc.\\nConf. Comput. Vis. Pattern Recognit., 2020, pp. 10287–10296.\\n[43] G. Nan et al., “Interventional video grounding with dual contrastive learn-\\ning,” in Proc. Conf. Comput. Vis. Pattern Recognit., 2021, pp. 2765–2775.\\n[44] Z. Wang, L. Wang, T. Wu, T. Li, and G. Wu, “Negative sample matters:\\nA renaissance of metric learning for temporal grounding,” in Proc. AAAI\\nConf. Artif. Intell., 2022, vol. 36, pp. 2613–2623.\\n[45] R. Tan, H. Xu, K. Saenko, and B. A. Plummer, “Logan: Latent graph\\nco-attention network for weakly-supervised video moment retrieval,” in\\nProc. IEEE/CVF Winter Conf. Appl. Comput. Vis., 2021, pp. 2083–2092.\\n[46] K. Wei, M. Yang, H. Wang, C. Deng, and X. Liu, “Adversarial fine-grained\\ncomposition learning for unseen attribute-object recognition,” in Proc. Int.\\nConf. Comput. Vis., 2019, pp. 3741–3749.\\n[47] X. Li, Z. Xu, K. Wei, and C. Deng, “Generalized zero-shot learning via dis-\\nentangled representation,” in Proc. AAAI Conf. Artif. Intell., 2021, vol. 35,\\npp. 1966–1974.\\n[48] J. Dong, Y. Cong, G. Sun, Z. Fang, and Z. Ding, “Where and how to trans-\\nfer: Knowledge aggregation-induced transferability perception for unsu-\\npervised domain adaptation,” IEEE Trans. Pattern Anal. Mach. Intell.,\\nearly access, Nov. 16, 2021, doi: 10.1109/TPAMI.2021.3128560.\\n[49] J. Dong, Y. Cong, G. Sun, B. Zhong, and X. Xu, “What can be transferred:\\nUnsupervised domain adaptation for endoscopic lesions segmentation,” in\\nProc. Conf. Comput. Vis. Pattern Recognit., 2020, pp. 4022–4031.\\nZhe Xu received the B.E. degree in 2019 from Xi-\\ndian University, Xi’an, China, where he is currently\\nworking toward the Ph.D. degree with the School of\\nElectronic Engineering. His research interests include\\ncomputer vision and machine learning.\\nKun Wei received the B.E. and Ph.D. degrees in\\nelectronic and information engineering from Xidian\\nUniversity, Xi’an, China, in 2017 and 2022, respec-\\ntively. He is currently a Lecturer with the School of\\nElectronic Engineering, Xidian University. His re-\\nsearch interests include computer vision and machine\\nlearning.\\nXu Yang (Member, IEEE) received the B.E. and\\nPh.D. degrees in electronic and information engineer-\\ning from Xidian University, Xi’an, China, in 2021. He\\nis currently a Lecturer with the School of Electronic\\nEngineering, Xidian University. His research inter-\\nests include computer vision and machine learning.\\nCheng Deng (Senior Member, IEEE) received the\\nB.E., M.S., and Ph.D. degrees in signal and informa-\\ntion processing from Xidian University, Xi’an, China.\\nHe is currently a Full Professor with the School of\\nElectronic Engineering, Xidian University. Dr. Deng\\nis author or coauthor of more than 100 scientific arti-\\ncles at top venues, including IEEE TRANSACTIONS ON\\nPATTERN ANALYSIS AND MACHINE INTELLIGENCE,\\nIEEE TRANSACTIONS ON NEURAL NETWORKS AND\\nLEARNING SYSTEMS, IEEE TRANSACTIONS ON IM-\\nAGE PROCESSING, IEEE TRANSACTIONS ON CYBER-\\nNETICS, NeurIPS, ICML, CVPR, ICCV, AAAI, IJCAI, and KDD. His research\\ninterests include computer vision, pattern recognition, and information hiding.\\nAuthorized licensed use limited to: XIDIAN UNIVERSITY. Downloaded on November 11,2023 at 03:52:33 UTC from IEEE Xplore.  Restrictions apply.\",\"difficulty\":\"easy\",\"domain\":\"Multi-Document QA\",\"length\":\"medium\",\"question\":\"Which article dynamically models the temporal sequence of single-frame distribution functions and differs from other articles?\",\"sub_domain\":\"Academic\"}","display_format":"text","language":"","answer_status":"published","assets":[],"source_url":"https://huggingface.co/datasets/zai-org/LongBench-v2","history":"initial import","indexing_mode":"noindex","subproblems":[],"grids":[]}