# LongBench v2 / 66ed1556821e116aacb1ea14

task_id: 4b28b6ba-c5ab-5404-86fa-01d83d0edea8
task_key: train--66ed1556821e116aacb1ea14
task_revision_id: 2

{"choice_A":"there are three kinds of data introduced, unaligned unlabeled ,aligned unlabeled, aligned unlabeled，and the HSSL\n pre-training phase used unaligned unlabeled samples of each party and aligned unlabeled sample of all parties","choice_B":"there are four kinds of data introduced,, unaligned unlabeled ,aligned unlabeled, aligned labeled and unaligned labeled. The HSSL\n pre-training phase used all these four kinds of data","choice_C":"there are four kinds of data introduced , unaligned unlabeled ,aligned unlabeled, aligned labeled and unaligned labeled. The HSSL used  used unaligned unlabeled samples of each party and aligned unlabeled sample of all parties in pre-training phase","choice_D":"there are three kinds of data introduced, unaligned unlabeled ,aligned unlabeled, aligned unlabeled，and the HSSL used unaligned unlabeled samples of each party and aligned unlabeled sample of all parties and aligned labeled samples of all parties in pre-training phase.","context":"JOURNAL OF L\nAT\nEX CLASS FILES, VOL. 14, NO. 8, JUNE 2023\n1\nA Hybrid Self-Supervised Learning Framework for\nVertical Federated Learning\nAbstract—Vertical federated learning (VFL), a variant of\nFederated Learning (FL), has recently drawn increasing attention\nas the VFL matches the enterprises’ demands of leveraging more\nvaluable features to achieve better model performance. However,\nconventional VFL methods may run into data deficiency as they\nexploit only aligned and labeled samples (belonging to different\nparties), leaving often the majority of unaligned and unlabeled\nsamples unused. The data deficiency hampers the effort of the\nfederation.\nIn this work, we propose a Federated Hybrid Self-Supervised\nLearning framework, named FedHSSL, that utilizes cross-party\nviews (i.e., dispersed features) of samples aligned among parties\nand local views (i.e., augmentation) of unaligned samples within\neach party to improve the representation learning capability\nof the VFL joint model. FedHSSL further exploits invariant\nfeatures across parties to boost the performance of the joint\nmodel through partial model aggregation. FedHSSL, as a frame-\nwork, can work with various representative SSL methods. We\nempirically demonstrate that FedHSSL methods outperform\nbaselines by large margins. We provide an in-depth analysis\nof FedHSSL regarding label leakage, which is rarely investi-\ngated in existing self-supervised VFL works. The experimental\nresults show that, with proper protection, FedHSSL achieves\nthe best privacy-utility trade-off against the state-of-the-art label\ninference attack compared with baselines. Code is available at\nhttps://github.com/jorghyq2016/FedHSSL.\nIndex\nTerms—Vertical\nfederated\nlearning,\nself-supervised\nlearning, privacy preservation, neural network.\nI. INTRODUCTION\nFederated learning (FL) enables independent parties to build\nmachine learning models collaboratively without sharing pri-\nvate data [1], [2]. This makes FL a practical solution to tackle\ndata silo issues while complying with increasingly strict legal\nand regulatory constraints enforced on user privacy, such as the\nGeneral Data Protection Regulation (GDPR). [2] categorizes\nFL into Horizontal FL (HFL) and Vertical FL (VFL). HFL\ntypically involves a large number of parties that have different\nsamples but share the same feature space, while VFL involves\nseveral parties that own distinct features of the same set of\nsamples. Recently, VFL has drawn increasing attention as the\nVFL matches the enterprises’ demands of leveraging more\nvaluable features to achieve better model performance without\njeopardizing data privacy. e.g., VFL has been widely deployed\nin industries such as finance [3] and advertisement [4].\nHowever, VFL has two critical limitations. One is the\ndeficiency of labeled samples. For example, positive labels\nare costly in the credit risk assessment because they are\navailable only when customers either complete their repayment\nor default, which may take a few years. Another limitation is\nthe deficiency of aligned samples. When participating parties\nhave quite different customer bases, their aligned samples\nare likely to be very limited. To address these two limi-\ntations, [5] proposed a federated cross-view approach that\nleverages the aligned samples to estimate missing features and\nlabels, which in turn is utilized for training the joint VFL\nmodel. This approach essentially relies on aligned samples and\nis conducted in a supervised learning manner. Recently, self-\nsupervised learning (SSL) has been introduced to HFL, aiming\nto improve the representation learning capability of the global\nmodel on label deficiency scenarios [6], [7], while the research\non integrating SSL into VFL is understudied. Existing SSL\nworks in VFL either solely used local unlabeled data [8], [9]\nwithout considering cross-party views of the aligned samples\nor only focused on aligned unlabeled sample [10], but failed\nto exploit each party’s local data. Besides, although SSL does\nnot involve labels, sample/feature alignment may result in the\nleakage of label information. Existing SSL-based VFL works\nrarely studied the impact of SSL on label leakage.\nTo fill these gaps, we propose FedHSSL, a Federated Hybrid\nSelf-Supervised Learning framework (illustrated in Fig. 4).\nFedHSSL simultaneously exploits (i) cross-party views (i.e.,\ndispersed features) of samples aligned among parties and (ii)\nlocal views (i.e., augmentations) of samples within each party,\nand aggregates (iii) invariant features shared among parties,\naiming to improve the overall performance of the final joint\nmodel. Furthermore, we analyze the label leakage of both the\npretraining and fine-tuning phases of FedHSSL and investigate\nthe protection against the label inference attack on FedHSSL.\nOur contributions are as follows:\n• We propose a federated hybrid SSL framework that takes\nadvantage of all available data through SSL and partial\nmodel aggregation to address the data deficiency issue in\nVFL. Experimental results show that FedHSSL methods\noutperform baselines by large margins on four datasets.\nThe ablation study demonstrates the effectiveness of each\nstep involved in FedHSSL in improving the performance\nof the VFL joint model.\n• We analyze the label leakage issue of FedHSSL. This\nis one of the first attempts to study label leakage of pre-\ntrained models in VFL. Experimental results demonstrate\nthat FedHSSL achieves a better privacy-utility trade-off\nthan baselines.\nII. RELATED WORKS\nA. Vertical Federated Learning (VFL)\nVFL aims to build a joint machine learning model using\nfeatures dispersed among parties while protecting privacy [11].\narXiv:2208.08934v2  [cs.LG]  8 Jun 2023\n\n\nJOURNAL OF L\nAT\nEX CLASS FILES, VOL. 14, NO. 8, JUNE 2023\n2\nTABLE I\nMAIN FL WORKS EMPLOYING SSL METHODS.\nSetting\nWorks\nData setting\nUsage of labeled data\nlabeled\nunlabeled\nHFL\nFedMOON [18], Fed-PCL [31], FedProc [19]\n√\nused in end-to-end training\nFedCA [20], FedU [21], FedEMA [6], FedX [32]\n√\n√\nused in finetuning\naligned labeled\naligned unlabeled\nunaligned unlabeled\nVFL\nFedCVT [5], FedMC [23]\n√\n√\nused in end-to-end training\nVFed-SSD [10]\n√\n√\nused in finetuning\nSS-VFL [8], VFLFS [9]\n√\n√\nused in finetuning\nFedHSSL(ours)\n√\n√\n√\nused in finetuning\nIn recent years, the literature has presented various algorithms\nin the VFL setting.\n[12] proposed vertical logistic regres-\nsion (VLR) using homomorphic encryption (HE) to protect\ndata privacy.\n[13] further enhanced the privacy-preserving\ncapability of VLR by employing a hybrid strategy combining\nHE and secret sharing (SS). [14] proposed the SecureBoost,\na VFL version of XGBoost, that leverages HE to protect\nthe parameters exchanged among parties. To tackle the data\ndeficiency issue of VFL, [15] integrated transfer learning into\nVFL to help the target party predict labels. [5] applied a semi-\nsupervised learning method to estimate missing features and\nlabels for further training.\nB. Self (Semi)-Supervised Learning in VFL\nWith the success of contrastive learning in computer vision,\nit gradually dominates self-supervised learning (SSL) [16],\n[17]. While several works applied SSL to HFL to address\nnon-IID [18], [19] or label deficiency issues\n[6], [20]–[22],\nthe research on integrating SSL into VFL is limited. [8],\n[9] pretrained participating parties’ local models leveraging\ntheir unaligned local samples without considering aligned\nsamples. [10] used aligned samples for learning discriminative\nrepresentations but did not use unlabeled local samples. [5],\n[23] exploited semi-supervised learning techniques to predict\npseudo labels of unaligned samples and estimate missing\nfeatures to boost the performance of VFL joint models. Table\nI briefly summarizes these works.\nSeveral VFL works aim to build a local predictor for one\nparty instead of a VFL joint model. For example, the goal of\n[24]–[26] is to train a local predictor for the active party for\naddressing the efficiency or availability issue in the inference\nphase, while\n[27]–[30] proposed to transfer the knowledge\nfrom the active party to help the passive party build a classifier.\nThese works are out of the scope of this work.\nC. Privacy Attacks and Protections in VFL\nVFL involves two kinds of privacy leakage: feature leakage\nand label leakage. [33] proposed model inversion attack to\ninfer features of the passive party. However, in the practical\nVFL setting, parties typically have black-box knowledge on\nthe model information of each other. Thus, it is challenging\nfor the attacker to infer features of other parties. The literature\nhas proposed two forms of label inference attacks in VFL:\nthe gradient-based [34] and the model-based [35]. The former\noften applies to binary classification, and the latter is difficult\nto be defended against, but it requires auxiliary training\ndata. [34] proposed three protection methods against gradient-\nbased attacks. [36] proposed a data encoding protection mech-\nanism called CoAE, which can thwart model-based attacks\neffectively in some scenarios. Cryptography-based protections\nare seldom applied to VFL that involves deep neural networks\n(DNN) for their high communication and computational cost.\n[3] proposed a HE-protected interactive layer that protects the\noutputs of parties’ local DNN without protecting gradients.\nThus, it can not defend against label inference attacks.\nIII. PRELIMINARIES\nWe review the concepts of vertical federated learning and\nself-supervised learning methods we adopt in this work.\nA. Vertical Federated Learning\nVertical federated learning deals with scenarios where par-\nticipating parties share the same set of samples but each\nholds a distinct portion of features of these samples. More\nspecifically, often one party holds labels but may or may\nnot owns features. This party is called active party because\nit typically is the initiator of VFL training and inferencing,\nwhile other parties hold only features and are called passive\nparties [37].\nFig. 1. The conventional VFL setting illustrated by two parties. Active party\n1 owns a bottom model f1 and a top model g1, while passive party 2 owns\na bottom model f2. We call the joint VFL model composed of f1, f2, and\ng1 FedSplitNN.\nWe take the 2-party VFL setting as an example (see Figure\n1). We assume the two parties collaboratively own a dataset\n(Y 1\nl , X1\nl , X2\nl ), party 1 is the active party who owns features\n\n\nJOURNAL OF L\nAT\nEX CLASS FILES, VOL. 14, NO. 8, JUNE 2023\n3\nFig. 2. Architecture overview of three representative SSL methods. All methods comprise two encoders: an online encoder f, and a target encoder ˜\nf. Gradients\nare not computed for the target encoder. For MoCo and BYOL, ˜\nf is the moving average of f. MoCo has a queue to provide additional negative samples for\ncalculating InfoNCE loss. BYOL and SimSiam has a predictor on the top of online encoder, and use only positive pairs. For SimSiam, ˜\nf = f.\nand labels (X1\nl , Y 1\nl ), and party 2 is the passive party who\nowns features X2\nl . In this work, we use superscripts to identify\nparticipating party, and subscripts for other denotations.\nThe active party 1 and passive party 2 utilize bottom model\nf 1 and f 2, respectively, to extracts high-level features from\nraw input x1 ∈X1\nl and x2 ∈X2\nl . The active party also has a\ntop model g1 that transforms the aggregated (denoted by ⊕)\noutputs z1 = f 1(x1) and z2 = f 2(x2) into predicted labels,\nwhich together with the ground-truth labels y1 are used to\ncompute the loss formulated in Eq.(1). We call the joint VFL\nmodel composed of f 1, f 2, and g1 FedSplitNN.\nLfed = ℓce(g1(z1 ⊕z2), y1)\n(1)\nwhere ℓce is cross entropy, y1 ∈Y 1\nl . Typical aggregation\nmethods include concatenation along the feature axis, max-\npooling and averaging. By minimizing Lfed in Eq. (1), bottom\nmodel f 1 and f 2, as well as top model g1 are updated.\nB. Self-supervised learning\nAmong various self-supervised learning (SSL) methods,\ncontrastive learning [16] has become the state-of-the-art\nmethod. It essentially groups semantically nearby samples\n(positive pairs) in the representation space while pushing apart\nthe dissimilar samples (negative pairs) as far as possible [17],\n[38]. [39], [40] proposed non-contrastive methods, which use\nonly positive pairs in self-supervised learning and demon-\nstrates competitive performance with reduced complexity.\nTABLE II\nVARIATIONS ON THE IMPLEMENTATION OF ALGO. 1 FOR DIFFERENT SSL\nMETHODS. MLP: MULTIPLE LAYER PERCEPTRON, EMA: EXPONENTIAL\nMOVING AVERAGE.\nMethod\nTarget encoder ˜\nf\nPredictor (h)\nLoss\nSimSiam\nequals online encoder f\nMLP\nLSimSiam\nBYOL\nEMA of online encoder f\nMLP\nLBYOL\nMoCo\nEMA of online encoder f\nidentical function\nLMoCo\nIn this section, we provide a brief introduction of three\nrepresentative SSL methods: MoCo [38], BYOL [39], Sim-\nSiam [40]. A schematic illustration of these three methods\nis shown in Fig. 2, and a comparison of their differences are\nlisted in Table II. Given a batch of sample x, its two augmented\nversion are v1 = T (x) and v2 = T (x). T denotes a data\naugmentation strategy. An online encoder f transforms v1 to\nz1, and a target encoder ˜\nf transforms v2 to ˜\nz2. A predictor,\nh, is used to further convert z1 to p1. That is z1 = f(v1),\n˜\nz2 = ˜\nf(v2), and p1 = h(z1). All three methods follow this\ntwo-tower structure, and it should be noted that gradients\nare not computed for the target encoder. Here we omit the\nsymmetrized computation path by swapping v1 and v2 for the\nsimplicity.\nMoCo. Momentum Contrast (MoCo) [38] utilizes the In-\nfoNCE loss and a momentum encoder to ensure a better\nrepresentation consistency and an additional queue to enable\ntraining with small batch size. That means ˜\nf is a momentum\nversion of f, and a sample Q, which maintains a dynamic\npool of feature vectors from previous batches. The predictor\nh is simply an identical function. The training objective is\nLMoCo = −log\nexp(z1 · st(˜\nz2))\nexp(z1 · st(˜\nz2)) + P\n˜\nzq∈Q exp(z1 · st(˜\nzq))\n(2)\nwhere ˜\nzq ∈Q, st(·) meas stop-gradient. By minimizing this\nloss, the positive pairs are pulled closer while negative pairs\nare pushed away in representation space.\nBYOL. Bootstrap Your Own Latent (BYOL) [39] differs from\nthe MoCo method in that it only requires positive pairs, mak-\ning the training procedure much simpler. The target encoder\n˜\nf is a momentum version of f, the same as MoCo. To avoid\na collapse in representation space, a multi-layer perceptron\n(MLP) is used as the predictor h. The training objective is\nformulated as follows.\nLBYOL = ∥\np1\n∥p1∥2\n−\nst(˜\nz2)\n∥st(˜\nz2)∥2\n∥2\n2\n= 2 −2 ·\n⟨p1, st(˜\nz2)⟩\n∥p1∥2 · ∥st(˜\nz2)∥2\n.\n(3)\nSimSiam. The Simple Siamese (SimSiam) [40] method is\nsimilar to BYOL that it also utilizes an asymmetric MLP\npredictor, h, and a similarity-based objective that only needs\npositive pairs. It further removes the momentum encoder and\n\n\nJOURNAL OF L\nAT\nEX CLASS FILES, VOL. 14, NO. 8, JUNE 2023\n4\nuses the same encoder, ˜\nf = f, for converting v1 and v2. The\ntraining objective becomes:\nLSimSiam = −\np1\n||p1||2\n·\nst(˜\nz2)\n||st(˜\nz2)||2\n.\n(4)\nIn this work, we adopt the three representative SSL methods,\nSimSiam [40], BYOL [39], and MoCo [38], as the base\nSSL methods for FedHSSL to investigate the effectiveness of\nFedHSSL as a framework.\nIV. METHODOLOGY\nIn this section, we formulate our VFL setting and problem.\nWe then elaborate on our FedHSSL framework.\nFig. 3. Virtual dataset owned by two parties. The aligned samples (X1\nal, X2\nal)\naccount for a small portion of each party’s total samples. The amount of\nlabeled aligned samples (Y 1\nl , X1\nl , X2\nl ) is even less, while each party has a\nlarge amount of non-aligned local samples (i.e., X1\nnl and X2\nnl).\nA. Problem Formulation\nWe consider a general VFL setting that involves K par-\nties. The ith party owns a dataset Xi = (Xi\nal, Xi\nnl), i ∈\n{1, . . . , K}, where Xi\nal and Xi\nnl denote aligned and non-\naligned samples, respectively. We assume only party 1 has\nlabels and denote party 1’s labeled samples as (Y 1\nl , X1\nl ), where\nX1\nl ⊆X1\nal. Figure 3 depicts the virtual dataset formed by two\nparties (i.e., parties 1 and 2) for illustrative purposes.\nIn conventional VFL, as explained in Section III-A, partic-\nipating parties collaboratively train a joint model only using\naligned and labeled samples (Y 1\nl , X1\nl , X2\nl , . . . , XK\nl ), leaving\neach party i’s aligned but unlabeled samples Xi\nal\\Xi\nl as well\nas unaligned samples Xi\nnl unused.\nWe propose a Federated Hybrid SSL (FedHSSL) framework\nthat pretrains participants’ local models by leveraging all\navailable unlabeled samples of all parties Xi = (Xi\nal, Xi\nnl) for\ni, i ∈{1, . . . , K}. Then, the conventional VFL is conducted\nto fine-tune pretrained models with a classifier g on top of\npretrained models using aligned and labeled samples.\nThe goal of FedHSSL is to enhance the performance of\nthe VFL joint model trained on downstream supervised task\n(see Section 1). Therefore, we evaluate the performance of\nFedHSSL on downstream supervised tasks.\nAlgorithm 1 FedHSSL Pretraining Procedure\nInput:\nDataset Xi = (Xi\nal, Xi\nnl) of party i, i ∈{1, . . . , K};\nCross-party encoder f i\nc and predictor hi\nc, i ∈{1, . . . , K};\nLocal encoder f i\nl =(f i\nlb, f i\nlt) and predictor hi\nl, i ∈{1, . . . , K};\nOutput:\nPretrained encoders f i\nc and f i\nl , i ∈{1, . . . , K}\n1: // Refer to Table II for implementation variations of adopting different\nSSL methods (i.e., SimSiam, BYOL, and MoCo)\n2: for each global iteration do\n3:\n▷Step 1\n⃝: Cross-party SSL\n4:\nfor party i ∈{1, . . . , K} do\n5:\nfor mini-batch xi\nal ∈Xi\nal do\n6:\nCompute zi\nc = f i\nc(xi\nal) and pi\nc = hi\nc(zi\nc)\n7:\nif i == 1 then\n8:\nSend zi\nc to parties {2, . . . , K};\n9:\nelse\n10:\nSend zi\nc to party 1;\n11:\nend if\n12:\nCompute Li\ncross according to Eq. (5)\n13:\nUpdate model f i\nc and hi\nc\n14:\nend for\n15:\nend for\n16:\n▷Step 2\n⃝: Cross party-guided local SSL\n17:\nfor party i ∈{1, . . . , K} do\n18:\nfor mini-batch xi ∈Xi do\n19:\nvi\n1, vi\n2 = T (xi), T (xi)\n20:\npi\n1,l, ˜\nzi\n2,l = hi\nl(f i\nl (vi\n1)), ˜\nf i\nl (vi\n2)\n21:\nCompute pi\n2,l and ˜\nzi\n1,l by swapping vi\n1 and vi\n2\n22:\n// zi\n1,c and zi\n2,c are for cross-party regularization\n23:\nzi\n1,c, zi\n2,c = f i\nc(vi\n1), f i\nc(vi\n2)\n24:\nCompute Li\nlocal according to Eq. (6)\n25:\nUpdate model f i\nl and hi\nl\n26:\nend for\n27:\nend for\n28:\n▷Step 3\n⃝: Partial model aggregation\n29:\nfor party i ∈{1, . . . , K} do\n30:\nSend local model f i\nlt ◦hi\nl to the server\n31:\nend for\n32:\nThe server performs f G\nlt ◦hG\nl = 1\nK\nPK\ni=1 f i\nlt ◦hi\nl\n33:\nThe server sends f G\nlt ◦hG\nl back to all parties\n34: end for\nB. Federated Hybrid Self-Supervised Learning\nThe core idea of FedHSSL is to utilize cross-party views\n(i.e., dispersed features) of samples aligned among parties and\nlocal views (i.e., augmentations) of samples within each party\nto improve the representation learning capability of the joint\nML model through SSL. FedHSSL further utilizes generic\nfeatures shared among parties to boost the joint model through\npartial model aggregation. Specifically, our FedHSSL consists\nof three steps:\n1) Cross-party SSL using aligned samples;\n2) Cross-party-guided local SSL using local samples;\n3) Partial model aggregation.\n\n\nJOURNAL OF L\nAT\nEX CLASS FILES, VOL. 14, NO. 8, JUNE 2023\n5\nFig. 4. Overview of FedHSSL. Each party has a two-tower structured model. FedHSSL involves 3 steps: 1\n⃝cross-party SSL using aligned samples to train\ncross-party encoders f1\nc and f2\nc ; 2\n⃝each party i leverages local SSL with the guidance of fi\nc to train its local encoders fi\nlt and fi\nlb using local samples; 3\n⃝\nthe server aggregates local top encoders f1\nlt and f2\nlt, and sends the aggregated encoder fG\nlt to all parties. We omit predictors in this figure for brevity.\nThese steps combine the VFL-like cross-party SSL and the\nHFL-like model aggregation, and thus we call them Federated\nHybrid SSL (FedHSSL) as a whole. The training procedure\nof FedHSSL is described in Algo. 1 and illustrated in Fig. 4.\n1) Cross-Party SSL: In VFL, each party can be thought\nof as holding one view of each aligned sample. These cross-\nparty views naturally form positive sample pairs to train the\nSSL model (i.e., the cross-party encoder f i\nc and predictor hi\nc)\nof each party i. The cross-party SSL is described in Step 1\n⃝of\nAlgo. 1. Specifically, for each party i, its input xi is converted\nby the cross-party encoder f i\nc to the representations zi\nc, which\nin turn is transformed to pi\nc via a predictor hi\nc. Then, party\n1 (with labels) exchanges its representations z1\nc with other\nparties’ representations zj\nc, j = 2, . . . , K. Upon receiving\ncorresponding representations, each party i optimize its cross-\nparty model via minimizing the cross-party loss Li\ncross:\nLi\ncross =\n\n\n\n1\nK−1\nPK\nj=2 LSSL(p1\nc, zj\nc),\nif i = 1.\nLSSL(pi\nc, z1\nc),\notherwise.\n(5)\nwhere LSSL is a self-supervised loss and its specific form\ndepends on the specific SSL method applies to FedHSSL (see\nTable II).\nFedHSSL adopts the same message-exchanging strategy as\nthe conventional VFL, in which messages are only exchanged\nbetween active party 1 and passive parties, mainly for commu-\nnication efficiency. The difference is that FedHSSL exchanges\nno gradient between parties, which automatically implements\nthe stop-gradient.\n2) Cross-Party-Guided Local SSL: We propose that each\nparty i uses its trained cross-party encoder f i\nc as guidance to\nregularize its SSL training of local encoder f i\nl and predictor\nhi\nl using its local samples. The knowledge from the cross-\nparty encoder helps improve the discriminative capability of\nf i\nl and hi\nl. Besides, it encourages the representations generated\nby local encoders of different parties to be aligned in the\nrepresentation space, which is beneficial for the partial model\naggregation (i.e., the next step).\nThe cross-party-guided local SSL is described in Step 2\n⃝\nof Algo. 1. More specifically, for each party i, two randomly\naugmented views vi\n1 = T (xi) and vi\n2 = T (xi) of an input xi\nare converted by a local online encoder f i\nl and a local target\nencoder ˜\nf i\nl to the representations zi\n1,l and ˜\nzi\n2,l, respectively.\nT denotes a data augmentation strategy. A local predictor hi\nl\nthen transforms zi\n1,l to pi\n1,l. Following [39], we swap vi\n1 and\nvi\n2 to obtain pi\n2,l and ˜\nzi\n1,l. Then, party i conducts the local SSL\nby minimizing the symmetrized loss:\nLi\nlocal =1\n2\nLSSL(pi\n1,l, ˜\nzi\n2,l) + LSSL(pi\n2,l, ˜\nzi\n1,l)\n\u0001\n+\nγ\nLSSL(pi\n1,l, zi\n1,c) + LSSL(pi\n2,l, zi\n2,c)\n\u0001\n,\n(6)\nwhere LSSL(pi\n1,l, zi\n1,c) + LSSL(pi\n2,l, zi\n2,c) is the regularization\nimposed by the cross-party encoder f i\nc on the training of local\nencoder; zi\n1,c = f i\nc(vi\n1) and zi\n2,c = f i\nc(vi\n2); γ controls the\nstrength of the regularization.\nThe effect of cross-party guidance can be visualized in\nthe representation space illustrated in Step 2\n⃝of Figure 4:\n\n\nJOURNAL OF L\nAT\nEX CLASS FILES, VOL. 14, NO. 8, JUNE 2023\n6\nrepresentations independently learned by the local SSL of each\nparty tend to disperse to different locations in the representa-\ntion space; with the guidance of the cross-party encoder, they\nare forced towards the position of cross-party encoders, which\nare trained to share similar behaviors in Step 1\n⃝.\n3) Partial Model Aggregation (PMA): An effective model\naggregation requires that the models to be aggregated have\nsufficiently similar parameter distribution. The cross-party\nguided local SSL (Step 2\n⃝) encourages the local encoders and\ntheir corresponding predictors f i\nl ◦hi\nl, i ∈{1, . . . , K} to learn\nsimilar feature projection in the representation space, making\nf i\nl ◦hi\nl, i ∈{1, . . . , K} potential candidates for partial model\naggregation.\nWe further divide the local encoder f i\nl of each party i into a\nparty-specific local bottom encoder f i\nlb and a local top encoder\nf i\nlt, and share f i\nlt ◦hi\nl with the server for aggregation. The\nrationale behind this design choice is two-fold: First, the local\ntop encoder tends to learn a generic set of features, making it\nsuitable to be shared among parties. Second, keeping the local\nbottom encoder private is beneficial for preventing parties’\ninput features from being attacked (e.g., gradient inversion\nattack) by the server [41]. The model aggregation is described\nin Step 3\n⃝of Algo. 1.\nImplementation Variations for Different SSL Methods.\nWe integrate SimSiam [40], BYOL [39], and MoCo [38],\nrespectively, into FedHSSL to investigate the effectiveness of\nFedHSSL as a framework. The three SSL methods have three\ndesign differences leading to variations in the implementation\nof Algo. 1, which are summarized in Table II.\nV. EXPERIMENTS\nA. Experimental Setup\nIn this section, we elaborate on the experimental setup,\nincluding datasets, models, baselines, and training details.\nDatasets & models. We conduct experiments on 4 datasets:\nNUSWIDE [42], Avazu [43], BHI [44], and Modelnet [45].\nThe former 2 are tabular datasets, while the latter 2 are image\ndatasets. For NUSWIDE, Avazu, and BHI, we split features\nof the same samples into 2 parts to simulate 2-party VFL\nscenario. For Modelnet, we divide samples describing the same\nobjects into 4 groups to simulate 4-party VFL scenario. Table\nIII shows chosen models corresponding to each dataset for all\nparties. All predictors consist of two fully-connected layers\n(FC). (see Appendix A for more detail on datasets)\nTABLE III\nMODELS FOR EVALUATION. EMB: EMBEDDING LAYER.\nDataset\nlocal and cross-party\nencoders (fl and fc)\nlocal top encoder\nfor PMA (flt)\nNUSWIDE\n2 FC\ntop 1 layer of fl\nAvazu\n1 Emb + 2 FC\ntop 1 layer of fl\nBHI\nResNet-18\ntop three blocks of fl\nModelnet\nResNet-18\ntop three blocks of fl\nTraining Details for FedHSSL. In addition to using\nall local samples for local SSL, we experiment with 40%\naligned samples of a dataset to pretrain cross-party encoder\nand predictor (i.e., cross-party SSL) of FedHSSL. We show\nour experiment with 20% aligned samples for pretraining in\nAppendix C-C. γ is set to 0.5 for all datasets (we investigate\nthe sensitivity of γ in Appendix C-A).\nBaselines. To evaluate the performance of FedHSSL, we\nadopt multiple baselines that cover the VFL methods we\nsurveyed in Section II-B (see Table I).\n• Supervised. The first two baselines are LightGBM\n(LGB) [46] and FedSplitNN (see Figure 1), which are\nwidely used supervised VFL models trained on labeled\nand aligned samples.\n• Semi-supervised. We adopt FedCVT [5] as another\nbaseline. FedCVT leverages labeled aligned and local\nunaligned samples to train a joint model consisting of\nparticipating parties’ local encoders and a global classi-\nfier. FedCVT only works on the 2-party scenario.\n• Self-supervised using local data. We implement three\nbaselines leveraging representative SSL methods, Sim-\nSiam, BYOL, and MoCo, respectively, to pretrain par-\nticipating parties’ local encoders and predictors using\nonly local samples. We name them FedLocalSimSiam,\nFedLocalBYOL, and FedLocalMoCo, respectively. The\nthree baselines cover methods used in SS-VFL [8] and\nVFLFS [9].\n• Self-supervised using aligned data. VFed-SSD [10] pre-\ntrains participating parties’ local encoders and predictors\nusing only aligned unlabeled samples, which is covered\nby FedCSSL, a sub-procedure of FedHSSL.\nAll baselines and FedHSSL use the same amount of labeled\nand aligned samples for training or fine-tuning. For each\ndataset, the local encoders of FedHSSL and baselines have\nthe same model architecture.\nWe evaluate FedHSSL methods and SSL baselines by fine-\ntuning its pretrained encoders and a classifier on top with a\nvarying number of labeled samples ranging from 200 to 1000.\nResults are reported as averages over 5 trials (see more training\ndetails in Appendix B-A).\nData Augmentation. For BHI and Modelnet, data are\naugmented following the setting described in [40]. For\nNUWISDE, 30% features are distorted by replacing the origi-\nnal value with a random value as described in [47]. For Avazu,\nthe continuous features are treated the same way as those of\nthe NUSWIDE, while the categorical features are replaced by\nextra untrained embedding vectors as described in [48].\nB. Main Results\nWe compare the performance of our FedHSSL framework\nintegrated with SimSiam, BYOL, and MoCo, respectively,\nwith the performance of baselines on four datasets. Both Table\nIV and Figure 5 show the results.\nFigure 5 illustrates that FedHSSL methods (red) gener-\nally enhance performance compared with baselines by large\nmargins for all datasets. For example, as reported in Table\nIV, with 200 labeled samples, the performance of FedHSSL-\nSimSiam is improved by 0.102 on NUSWIDE, by 0.048 on\nAvazu, by 0.045 on BHI and by 0.085 on Modelnet, respec-\ntively, compared with FedLocalSimSiam. Similarly, FedHSSL-\nBYOL outperforms FedLocalBYOL by 0.084, 0.055, 0.031,\n\n\nJOURNAL OF L\nAT\nEX CLASS FILES, VOL. 14, NO. 8, JUNE 2023\n7\nTABLE IV\nPERFORMANCE COMPARISON OF FEDHSSL (INTEGRATED WITH SIMSIAM, BYOL, MOCO, RESPECTIVELY) AND BASELINES WITH A VARYING NUMBER\nOF LABELED SAMPLES. TOP-1 ACCURACY IS USED AS THE METRIC FOR NUSWIDE AND MODELNET, WHILE AUC AND F1-SCORE ARE METRICS FOR\nAVAZU AND BHI, RESPECTIVELY. % OF LABELED AND ALIGNED SAMPLES APPLIES ONLY TO FEDHSSL.\n# of labeled and aligned samples:\n200\n400\n600\n800\n1000\n# of parties\nNUSWIDE\n(Top1-Acc)\nLGB\n0.425 ± 0.015\n0.465 ± 0.028\n0.526 ± 0.012\n0.556 ± 0.013\n0.587 ± 0.012\n2\nFedSplitNN\n0.495 ± 0.022\n0.535 ± 0.027\n0.560 ± 0.015\n0.573 ± 0.014\n0.591 ± 0.013\nFedCVT\n0.522 ± 0.019\n0.555 ± 0.013\n0.602 ± 0.003\n0.621 ± 0.006\n0.629 ± 0.014\nFedLocalSimSiam\n0.505 ± 0.027\n0.536 ± 0.018\n0.596 ± 0.013\n0.603 ± 0.019\n0.612 ± 0.017\nFedLocalBYOL\n0.514 ± 0.032\n0.527 ± 0.029\n0.585 ± 0.022\n0.599 ± 0.028\n0.606 ± 0.027\nFedLocalMoCo\n0.566 ± 0.033\n0.596 ± 0.022\n0.625 ± 0.017\n0.634 ± 0.017\n0.639 ± 0.019\nFedHSSL-SimSiam\n0.607 ± 0.003\n0.641 ± 0.008\n0.651 ± 0.006\n0.662 ± 0.006\n0.670 ± 0.003\nFedHSSL-BYOL\n0.598 ± 0.025\n0.624 ± 0.034\n0.645 ± 0.012\n0.659 ± 0.007\n0.664 ± 0.004\nFedHSSL-MoCo\n0.615 ± 0.021\n0.642 ± 0.012\n0.658 ± 0.003\n0.668 ± 0.005\n0.670 ± 0.006\nAvazu\n(AUC)\nLGB\n0.563 ± 0.016\n0.568 ± 0.019\n0.595 ± 0.020\n0.621 ± 0.012\n0.620 ± 0.012\n2\nFedSplitNN\n0.588 ± 0.031\n0.581 ± 0.013\n0.599 ± 0.019\n0.595 ± 0.008\n0.615 ± 0.006\nFedCVT\n0.594 ± 0.026\n0.606 ± 0.022\n0.608 ± 0.029\n0.637 ± 0.015\n0.647 ± 0.013\nFedLocalSimSiam\n0.575 ± 0.007\n0.585 ± 0.020\n0.591 ± 0.016\n0.608 ± 0.026\n0.629 ± 0.024\nFedLocalBYOL\n0.560 ± 0.029\n0.597 ± 0.015\n0.600 ± 0.024\n0.601 ± 0.004\n0.605 ± 0.013\nFedLocalMoCo\n0.573 ± 0.024\n0.591 ± 0.017\n0.584 ± 0.027\n0.596 ± 0.004\n0.601 ± 0.011\nFedHSSL-SimSiam\n0.623 ± 0.016\n0.636 ± 0.026\n0.649 ± 0.008\n0.648 ± 0.014\n0.663 ± 0.007\nFedHSSL-BYOL\n0.615 ± 0.031\n0.634 ± 0.028\n0.631 ± 0.016\n0.630 ± 0.013\n0.648 ± 0.010\nFedHSSL-MoCo\n0.616 ± 0.014\n0.632 ± 0.011\n0.638 ± 0.017\n0.641 ± 0.009\n0.658 ± 0.007\nBHI\n(F1-Score)\nFedSplitNN\n0.731 ± 0.003\n0.738 ± 0.002\n0.754 ± 0.002\n0.752 ± 0.002\n0.760 ± 0.005\n2\nFedCVT\n0.742 ± 0.013\n0.747 ± 0.011\n0.755 ± 0.007\n0.758 ± 0.006\n0.782 ± 0.003\nFedLocalSimSiam\n0.760 ± 0.010\n0.764 ± 0.006\n0.788 ± 0.005\n0.785 ± 0.004\n0.798 ± 0.006\nFedLocalBYOL\n0.760 ± 0.007\n0.769 ± 0.008\n0.781 ± 0.005\n0.786 ± 0.005\n0.796 ± 0.003\nFedLocalMoCo\n0.763 ± 0.003\n0.771 ± 0.008\n0.784 ± 0.012\n0.793 ± 0.002\n0.800 ± 0.008\nFedHSSL-SimSiam\n0.805 ± 0.009\n0.816 ± 0.006\n0.822 ± 0.003\n0.823 ± 0.002\n0.830 ± 0.002\nFedHSSL-BYOL\n0.791 ± 0.011\n0.806 ± 0.004\n0.821 ± 0.002\n0.822 ± 0.004\n0.825 ± 0.003\nFedHSSL-MoCo\n0.806 ± 0.007\n0.817 ± 0.002\n0.822 ± 0.004\n0.829 ± 0.004\n0.831 ± 0.002\nModelnet\n(Top1-Acc)\nFedSplitNN\n0.612 ± 0.019\n0.684 ± 0.011\n0.733 ± 0.002\n0.765 ± 0.007\n0.771 ± 0.005\n4\nFedLocalSimSiam\n0.622 ± 0.022\n0.698 ± 0.017\n0.761 ± 0.009\n0.779 ± 0.004\n0.797 ± 0.006\nFedLocalBYOL\n0.635 ± 0.004\n0.707 ± 0.010\n0.760 ± 0.007\n0.775 ± 0.009\n0.794 ± 0.007\nFedLocalMoCo\n0.659 ± 0.022\n0.722 ± 0.012\n0.784 ± 0.008\n0.798 ± 0.007\n0.815 ± 0.007\nFedHSSL-SimSiam\n0.707 ± 0.009\n0.772 ± 0.006\n0.806 ± 0.008\n0.826 ± 0.007\n0.833 ± 0.006\nFedHSSL-BYOL\n0.681 ± 0.005\n0.752 ± 0.002\n0.800 ± 0.008\n0.807 ± 0.007\n0.825 ± 0.009\nFedHSSL-MoCo\n0.705 ± 0.016\n0.764 ± 0.012\n0.804 ± 0.006\n0.822 ± 0.003\n0.830 ± 0.007\nand 0.046, respectively, on the 4 datasets; FedHSSL-MoCo\noutperforms FedLocalMoCo by 0.049, 0.043, 0.043, and\n0.046, respectively, on the 4 datasets. Besides, with 200\nlabeled samples, the best-performing FedHSSL method out-\nperforms FedCVT by 0.093 on NUSWIDE, 0.029 on Avazu,\nand 0.063 on BHI, respectively.\nFig. 5.\nPerformance comparison of FedHSSL (integrated with SimSiam,\nBYOL, and MoCo, respectively) and baselines.\nWith more labeled samples involved in fine-tuning, the\nperformance improvement of FedHSSL is still noticeable.\nFor example, with 1000 labeled samples, the performance of\nFedHSSL-SimSiam is improved by 0.058 on NUSWIDE, by\n0.034 on Avazu, by 0.032 on BHI, and by 0.036 on Modelnet,\nrespectively, compared with FedLocalSimSiam.\nC. Ablation Study\nTo study the effectiveness of each step in FedHSSL, we\nconsider two sub-procedures of FedHSSL: (i). FedCSSL,\nwhich is the cross-party SSL step in Algo. 1 (i.e., Step 1\n⃝). (ii).\nFedGSSL, which is FedCSSL + cross-party-guided local SSL\nstep in Algo. 1 (i.e., Step 1\n⃝+ Step 2\n⃝). We evaluate FedCSSL\nand FedGSSL in the same way as that of FedHSSL: pretrained\nencoders are fine-tuned by minimizing Eq (1) using aligned\nand labeled data.\nThe Effectiveness of Each Step Involved in FedHSSL.\nFigure 6 illustrates that for each SSL method (i.e., SimSiam,\nBYOL, and MoCo on each column), FedCSSL consistently\noutperforms its corresponding FedLocalSSL as the number of\nlabeled samples increases on the four datasets. By integrating\nlocal SSL into FedCSSL, FedGSSL generally enhances the\nperformance over FedCSSL. The enhancement is significant\non NUSWIDE (by ≈0.05 averagely) and noticeable on the\nother three datasets. By additionally conducting partial model\n\n\nJOURNAL OF L\nAT\nEX CLASS FILES, VOL. 14, NO. 8, JUNE 2023\n8\nTABLE V\nSTUDY THE IMPACT OF CROSS-PARTY ENCODERS ON (1) LOCAL SSL AND (2) PARTIAL MODEL AGGREGATION (PMA). THE LOCAL ENCODERS OF\nFEDLOCALSIMSIAM, FEDLOCALBYOL, AND FEDLOCALMOCO ARE PRETRAINED USING LOCAL SIMSIAM, BYOL, AND MOCO, RESPECTIVELY.\nWHILE THE LOCAL ENCODERS OF FEDGSSL-SIMSIAM∗, FEDGSSL-BYOL∗, AND FEDGSSL-MOCO∗ARE PRETRAINED USING cross-party-guided\nSIMSIAM, BYOL, AND MOCO, RESPECTIVELY. ALL METHODS ARE FINETUNED USING 200 LABELED SAMPLES. THE DOWN ARROW ↓INDICATES THE\nPERFORMANCE DECREASES WHEN THE CORRESPONDING METHODS COMBINE WITH PMA. THE UP ARROW ↑INDICATES OTHERWISE.\nNUSWIDE\nAvazu\nBHI\nModelnet\nMethod\n−\nw/ PMA\n−\nw/ PMA\n−\nw/ PMA\n−\nw/ PMA\nFedLocalSimSiam\n0.505\n0.537 ↑\n0.580\n0.582 ↑\n0.760\n0.743 ↓\n0.622\n0.599 ↓\nFedGSSL-SimSiam∗\n0.543\n0.553 ↑\n0.606\n0.609 ↑\n0.783\n0.789 ↑\n0.679\n0.688 ↑\nFedLocalBYOL\n0.514\n0.512 ↓\n0.560\n0.575 ↑\n0.760\n0.756 ↓\n0.635\n0.629 ↓\nFedGSSL-BYOL∗\n0.543\n0.544 ↑\n0.591\n0.606 ↑\n0.778\n0.785 ↑\n0.640\n0.656 ↑\nFedLocalMoCo\n0.566\n0.563 ↓\n0.573\n0.587 ↑\n0.763\n0.760 ↓\n0.659\n0.639 ↓\nFedGSSL-MoCo∗\n0.613\n0.612 ↓\n0.603\n0.611 ↑\n0.787\n0.795 ↑\n0.664\n0.674 ↑\naggregation (PMA), FedHSSL further boosts the performance\non the four datasets. These results demonstrate the effective-\nness of all three steps involved in FedHSSL.\nThe Impact of Cross-Party Encoders’ Guidance on\nLocal SSL and Model Aggregation. For a fair comparison,\nFedLocalSSL and FedGSSL∗all use pretrained local encoders\nduring fine-tuning. The star ∗distinguishes FedGSSL∗from\nFedGSSL, which leverages both cross-party and local encoders\nfor fine-tuning.\nTable V reports that, for each SSL method (i.e., SimSiam,\nBYOL, and MoCo), FedGSSL∗consistently outperforms its\ncorresponding FedLocalSSL on all datasets. For example,\nFedGSSL-SimSiam outperforms FedLocalSimSiam by 0.038,\n0.026, 0.023, and 0.057 on the four datasets, respectively.\nThis demonstrates the effectiveness of the cross-party SSL in\nimproving the representation learning of local SSL.\nWe further analyze the impact of cross-party encoders\non partial model aggregation (PMA). Table V reports that\ndirectly combining FedLocalSSL and PMA may jeopar-\nFig. 6. Ablations on FedHSSL. We compare the performance of FedCSSL\n(blue), FedGSSL (green), and FedHSSL(red) for SimSam, BYOL, and MoCo,\nrespectively. These methods are pretrained with all local samples and 40%\naligned samples and finetuned with a varying number of labeled and aligned\nsamples. FedLocalSimSiam, FedLocalBYOL, and FedLocalMoCo are base-\nlines for comparison.\ndize the overall performance. For example, the performance\nof FedLocalSimSiam+PMA decreases by around 2% com-\npared with that of FedLocalSimSiam on BHI and Model-\nnet. Similar trends can be found on FedLocalBYOL+PMA\nand FedLocalMoCo+PMA. Assisted by the cross-party en-\ncoder, we observe a noticeable performance improvement\non FedGSSL∗+PMA over FedGSSL∗for all SSL methods\ngenerally across all datasets. This manifests that the guidance\nof cross-party encoders mitigates the heterogeneity among\nfeatures of different parties so that it positively impacts PMA.\nD. Communication Efficiency\nThe pretraining of FedHSSL utilizes all aligned samples,\nwhich results in higher communication overhead compared to\nconventional VFL that only uses labeled aligned samples. To\nmitigate this communication overhead, each party in FedHSSL\ncan perform multiple updates in the cross-party SSL step (Step\n1\n⃝of Figure 4) to reduce communication rounds. Specifically,\nafter received feature representations zc from other parties,\neach party conducts multiple local SSL updates by minimizing\ncross-party SSL loss (5) using zc. This strategy is similar to\nFedBCD [49], in which each party uses received gradients to\nupdate local model for multiple local updates.\nFig. 7. Comparison of the performance of FedHSSL under different numbers\nof local updates in the cross-party SSL step. Results are obtained by averaging\nthree rounds of experiments with different random seeds. 20% training\nsamples are aligned for cross-party SSL and 200 labeled samples are used in\nthe fine-tuning. SimSiam is used as the default SSL method.\nWe investigate the impact of multiple local updates in the\ncross-party SSL (Step 1\n⃝of FedHSSL) on the communication\n\n\nJOURNAL OF L\nAT\nEX CLASS FILES, VOL. 14, NO. 8, JUNE 2023\n9\nefficiency by experimenting with various numbers of local\nupdates in the range of 1, 4, 8. We denote e as the number\nof local updates. For these experiments, we adopt SimSiam as\nthe base SSL method for FedHSSL.\nFigure 7 illustrates the results. It shows that, with larger\ne, FedHSSL generally achieves better main task performance\nwith the same global iterations on 4 datasets. However, on\nBHI, FedHSSL with 8 local updates performs worse than\nFedHSSL with 4 local updates, indicating that larger e do\nnot necessarily lead to better performance and an appropriate\ne should be carefully chosen in order to achieve the best\nperformance.\nVI. PRIVACY ANALYSIS ON LABEL INFERENCE ATTACK\nIn this section, we investigate whether FedHSSL, as a self-\nsupervised VFL framework, can achieve a better privacy-utility\ntrade-off against the label inference attack compared with\nbaseline methods. We adopt SimSiam as the base SSL method\nfor FedHSSL. Each party in FedHSSL pretrains its local model\nusing all local samples and 20% aligned samples. Supervised\nVFL training (including fine-tuning) is conducted using 200\naligned and labeled samples.\nA. Threat Model\nWe first discuss the threat model, including the attacker’s\nobjective, capability, knowledge, and attacking methods.\nAdversary’s objective. We assume that party 2 is the\nadversary who wants to infer labels y1 owned by party 1.\nAccording to the nature of dispersed data in our VFL\nsetting, there can be three adversary objectives [50]: (i) labels\nowned by the active party; (ii) features owned by the active\nparty; (iii) features owned by the passive party. We focus\non label inference attack where a passive party (i.e., party\n2) is the adversary and it wants to infer labels y1 owned by\nthe active party (i.e., party 1) for the reasons that: (i) in the\npractical VFL setting, parties have black-box knowledge on\nthe model information of each other, and thus it is highly\nchallenging to party 1 to infer the features x2 of party 2 [33];\n(ii) during model aggregation of FedHSSL, parties only share\ntheir local top encoders with the server while keeping the local\nbottom encoders private, in which case the server is not able to\nreconstruct features of any party [41]; (iii) the labels owned by\nthe active party is an important target for adversaries in VFL\ncompared to HFL. Because in real-world VFL applications\nsuch as finance and advertisement, the labels may contain\nsensitive user information or are valuable assets.\nAdversary’s capability. We assume that the adversary party\n2 is semi-honest such that the adversary faithfully follows the\nvertical federated training protocol but it may mount privacy\nattacks to infer the private data of other parties.\nAdversary’s knowledge. In VFL, participating parties typ-\nically have blackbox knowledge about each other. However,\nadversaries may guess some of the knowledge about others\naccording to the information they have. In this work, we\nassume that the information about the model structure, input\nshape and number of classes supported by the active party’s\ntask is shared among parties. We also assume party 2 has a\nfew auxiliary labeled samples Daux\nB. Privacy attacking and protection mechanism\nPrivacy attacking mechanism. There are mainly two kinds\nof label inference attacks in the VFL setting: the gradient-\nbased attacks [34] and the model-based attacks [35]. The for-\nmer applies only to binary classification and can be thwarted\neffectively by state-of-the-art privacy protections (e.g., Mar-\nvell [34]), while the latter is difficult to be prevented. In this\nwork, we study the model completion (MC) attack [35], the\nrepresentative of the model-based label inference attack. MC\nattack involves three steps:\n1) Party 1 and party 2 conduct federated training, which\ncan be FedHSSL pertaining or fine-tuning phase of\ndownstream tasks. Upon the completion of training,\nparty 2 obtains trained local models f 2;\n2) Party 2 constructs a complete attacking model AFedHSSL\nby training an inference head g2 on top of f 2 using few\nauxiliary labeled data;\n3) Party 2 infers labels of its inference data x2\ninf through\ny2\ninf = AFedHSSL(x2\ninf) during inference phase.\nAdversary party 2 can launch MC during the pretraining\nphase of FedHSSL or fine-tuning after FedHSSL. In this\nsection, we study both scenarios.\nPrivacy protection mechanism. we adopt isotropic Gaus-\nsian noise (ISO) [34] as the protection method. Specifically,\nparty 1 perturbs model information d ∈Rb×m exposed to the\nadversary (i.e., party 2) by applying ISO to d, which can be\nforward embedding and backward gradients:\nISO(d) = d + εiso\n(7)\nwhere εiso ∼N(0, σ2\niso) is the noise added to protect privacy,\nσiso = (λ · ||dmax||2)/√m is the standard deviation, and\n||dmax||2 is the largest value in the batch-wise 2-norms ||d||2\nof d, λ is the noise amplifier and controls the strength of the\nISO protection. We refer interesting readers to [34] for details\non MC attack and ISO protection.\nDefending against Model Completion This experiment\nis conducted on FedHSSL-SimSiam pretraining. On the one\nhand, the adversary party 2 trains an attacking model AFedHSSL\naccording to the procedure described in Section VI-B. On the\nother hand, party 1 applies ISO to the output of its cross-party\nencoder and parameters of its local top encoder to mitigate\nprivacy leakage. After pretraining, party 2 leverages AFedHSSL\nto predict labels of incoming samples.\nFor a fair comparison, we assume the adversary trains\na baseline attacking model ASimSiam, pretrained by normal\nSimSiam, using Daux. Intuitively, ASimSiam can be thought of\nas the adversary’s prior knowledge on labels, while AFedHSSL\nis the posterior knowledge on labels after the MC attacking.\nTable VI compares ASimSiam and AFedHSSL w/o and w/\nISO protection. Both two MC attacks leverage 80 labeled\nauxiliary samples to train attacking models. Table VI reports\nthat AFedHSSL w/o ISO outperforms ASimSiam by 0.072 on\nNUSWIDE and by ≤0.012 on the other 3 datasets, indicating\nthat FedHSSL leaks label privacy. When ISO protection is\napplied with properly chosen λp, the performance of AFedHSSL\ndrops below that of ASimSiam on 3 out of 4 datasets (except\nNUSWIDE), and the losses of main task performance on\n\n\nJOURNAL OF L\nAT\nEX CLASS FILES, VOL. 14, NO. 8, JUNE 2023\n10\nTABLE VI\nCOMPARISON OF ASIMSIAM AND AFEDHSSL W/O AND W/ ISO. THIS TABLE\nALSO REPORTS THE MAIN TASK PERFORMANCE OF FEDHSSL-SIMSIAM\nEVALUATED ON 200 ALIGNED AND LABELED SAMPLES. λp IS THE NOISE\nLEVEL OF ISO APPLIED TO FEDHSSL PRETRAINING. WE USE LABEL\nRECOVERY ACCURACY TO MEASURE THE PERFORMANCE OF ASIMSIAM\nAND AFEDHSSL.\nw/o ISO protection\nw/ ISO protection\nDataset\nASimSiam\nAFedHSSL\nMain\nAFedHSSL\nMain\nλp\nNUSWIDE\n0.439\n0.511\n0.574\n0.465\n0.539\n0.4\nAvazu\n0.545\n0.547\n0.616\n0.524\n0.617\n0.1\nBHI\n0.716\n0.726\n0.803\n0.682\n0.786\n0.1\nModelnet\n0.429\n0.441\n0.678\n0.426\n0.658\n0.1\nthe 3 datasets are small (≤0.02). This manifests that the\nlabel leakage of FedHSSL can be prevented if protection\nmechanisms are properly applied.\nAnalyzing Privacy-Utility Trade-Off. This experiment is\nconducted on the fine-tuning phase after FedHSSL pretraining.\nWe compare FedHSSL-SimSiam with FedSplitNN and FedLo-\ncalSimSiam in terms of their privacy-utility trade-offs coming\nfrom the competition between the MC attack and the ISO\nprotection during fine-tuning. The fine-tuning of FedHSSL-\nSimSiam and FedLocalSimSiam is conducted based on the two\nmethods’ pretrained models, respectively, whereas FedSplitNN\ninvolves no pretraining. For each method, Party 1 applies ISO\nto gradients sent back to the passive party 2 during fine-\ntuning/training for protection. Upon the completion of fine-\ntuning/training, party 2 trains a MC attacking model based on\nits finetuned/trained local model using Daux.\nFrom the 4 figures (in Table VII), we observe that, on\neach dataset, FedHSSL-SimSiam (red) achieves the best main\ntask performance but fails to preserve the most label privacy.\nThus, it is unclear whether FedHSSL-SimSiam has the best\nprivacy-utility trade-off curve. We adopt Calibrated Averaged\nPerformance (CAP) [51] to quantify the privacy-utility trade-\noff curve of a privacy-protected method so that we can com-\npare trade-offs of different methods based on a single metric.\nWe provide the definition of Calibrated Averaged Performance\nas follows.\nDefinition 1 (Calibrated Averaged Performance). For a given\nprotection mechanism Mλ with a protection strength parame-\nter λ and an attacking mechanism A, the Calibrated Averaged\nPerformance (CAP) for a given privacy-utility trade-off curve\nis defined as follows,\nCAP(Mλ∈{λ1,...,λv}, A) = 1\nv\nλv\nX\nλ=λ1\nU( ¯\nGλ) ∗E( ¯\nDλ, D),\n(8)\nwhere ¯\nGλ = Mλ(G) is the VFL model protected by Mλ, ¯\nDλ =\nA( ¯\nGλ, D) is the data recovered by the attacking mechanism\nA from ¯\nGλ given the private data D as input, U(·) measures\nthe main task utility (e.g., accuracy) of a given model, and\nE(·) measures the distance between recovered data ¯\nDλ and\noriginal data D.\nTable VII reports that, on each dataset, FedHSSL-SimSiam\nhas the highest CAP value, and thus it achieves the best trade-\noff between privacy and main task performance. The reason\nTABLE VII\nCOMPARISON OF CALIBRATED AVERAGED PERFORMANCE (CAP) OF\nISO-PROTECTED FEDSPLITNN, FEDLOCALSIMSIAM, AND\nFEDHSSL-SIMSIAM AGAINST THE MC ATTACK ON 4 DATASETS. CAP\nQUANTIFIES THE PRIVACY-UTILITY TRADE-OFF CURVES VISUALIZED IN\nTHE ABOVE 4 FIGURES. The higher the CAP value is, the better the method\nis at preserving privacy without compromising the main task performances.\nNUMBERS ON THE FIGURES ARE VALUES OF ISO PROTECTION STRENGTH\nλf CHOSEN FROM [1, 5, 25]. A better trade-off curve should be more toward\nthe bottom-right corner of each figure. THE HORIZONTAL DASHED LINE\nDENOTES THE PRIOR KNOWLEDGE OF THE ADVERSARY ON THE LABELS\nOF PARTY 1.\nDataset\nFedSplitNN\nFedLocalSimSiam\nFedHSSL-SimSiam\nNUSWIDE\n0.264\n0.258\n0.284\nAvazu\n0.238\n0.262\n0.262\nBHI\n0.242\n0.221\n0.246\nModelnet\n0.342\n0.334\n0.348\nleading to this outcome is that the amount of performance\nenhanced by FedHSSL-SimSiam outweighs the amount of\nlabel leakage worsened by FedHSSL-SimSiam to the ex-\ntent that FedHSSL-SimSiam obtains better CAP values than\nbaselines. With more aligned samples (i.e., 40%) used for\npretraining, FedHSSL-SimSiam generally achieves better main\ntask performance while leaking more label privacy (see Table\nXI in Appendix C-D), leading to similar CAP values (see Table\nXII in Appendix C-D). These experimental results manifest\nthat the number of aligned samples is a crucial factor that\nimpacts the privacy-utility trade-off of FedHSSL, and should\nbe considered when applying FedHSSL.\nVII. CONCLUSION\nWe propose a federated hybrid SSL framework (FedHSSL)\nthat leverages all aligned and unaligned samples through SSL\nand exploits invariant features shared among parties through\npartial model aggregation to improve the overall performance\nof the VFL joint model. FedHSSL works with representative\nSSL methods. The experimental results show that FedHSSL\noutperforms baselines by a large margin. The ablation demon-\nstrates the effectiveness of each step involved in FedHSSL.\nWe analyze the label leakage of FedHSSL under the Model\nCompletion (MC) attack and apply ISO to defend against MC\nattack. Experimental results show that FedHSSL achieves the\nbest privacy-utility trade-off compared with baselines.\n\n\nJOURNAL OF L\nAT\nEX CLASS FILES, VOL. 14, NO. 8, JUNE 2023\n11\nAPPENDIX A\nDATASETS\nNUSWIDE contains 634-dimensional low-level image fea-\ntures extracted from Flickr and 1000-dimensional correspond-\ning text features. To simulate the VFL setting, one party holds\nimage features, and the other holds text features. There are\n81 ground truth labels, and we build datasets with our desired\nsetting by selecting a subset of these labels. Here ten labels are\nfor the multi-class classification task with 10 selected labels.\nAvazu is for predicting click-through rate. It contains 14\ncategorical features and 8 continuous features. We transform\ncategorical features into embeddings with fixed dimensions\n(32 in this work) before feeding them the model. To simulate\nthe VFL setting, we equally divide both kinds of features into\ntwo parts so that each party has a mixture of categorical and\ncontinuous features. To reduce the computational complexity,\nwe randomly select 100000 samples as the training set and\n20000 samples as the test set.\nBHI (Breast Histopathology Images) is used for binary\nclassification task. It contains 277,524 slide-mount images of\nbreast cancer specimens from several patients. A positive label\nindicates Invasive Ductal Carcinoma (IDC) positive, which is\na subtype of breast cancer. The ratio between positive and\nnegative samples is around 1 : 2.5. We randomly select data\nof 80% patients as the training set and the rest as the test\nset. To simulate the VFL setting, we choose two images of a\npatient with the same label to form a VFL sample, and each\nparty is assigned one image.\nModelnet is a multiview dataset with 40 classes. We select\nsamples of the first 10 classes for our experiments. Each class\ncontains several 3D objects. We generate 12 images for each\nobject, following the procedure described in [52]. To simulate\nthe VFL setting, we split 12 views of each object sequentially\ninto 4 groups so that each contains 3 nearby views, and thereby\neach party holds three views of an object. To expand the\ndataset and make the task harder, we randomly select an image\nfrom each party and build a VFL sample for each object. This\nprocedure is the same for both the train and test sets. In the\nend, we have 24630 training samples and 6204 test samples.\nTABLE VIII\nDETAILED INFORMATION OF THE DATASETS AND CORRESPONDING\nMODELS.\nDataset\nData Type\nClasses\n# of Parties\nMetric\nNUSWIDE\nTabular\n10\n2\nTop-1 Acc\nAvazu\nTabular\n2\n2\nAUC\nBHI\nImage\n2\n2\nF1-score\nModelnet\nImage\n10\n4\nTop-1 Acc\nAPPENDIX B\nEXPERIMENTAL SETUP\nA. Training Details\nFor SSL training, cross-party SSL and guided local SSL\nare conducted alternately. Multiple epochs can be executed\nfor both steps to reduce communication costs. In this work,\nwe set 1 epoch for cross-party SSL and guided local SSL\ntraining. Partial model aggregation is performed directly after\nthe guided SSL. The number of global iterations for FedHSSL\nprertraining is set to 10 for NUSWIDE and 40 for other\ndatasets.\nAll encoders include a projector consisting of 3 fully-\nconnected layers (FC), which is only used in the pretraining\nphase. For FedHSSL-MoCo, the dimension of the projector is\n[512, 512, 128]. For FedHSSL-SimSiam and FedHSSL-BYOL,\nthe dimension of the projector is [512, 512, 512], and an ad-\nditional 2-FC predictor with the dimension [128, 512] is used.\nFor FedHSSL-MoCo, the temperature of the InfoNCE loss is\n0.5, the size of the dictionary is 4096, and the momentum is\n0.99. For FedHSSL-BYOL, the momentum is 0.995.\nFor pretraining, the batch size is 512 for all datasets. For\nthe finetuning, the batch size is 512 for NUSWIDE and Avazu\nand 128 for BHI and Modelnet. The learning rate used in the\nfinetuning stage includes [0.005, 0.01, 0.03], and the best result\nis selected. All experiments are repeated with 5 different seeds,\nand the average results are reported.\nAPPENDIX C\nMORE EXPERIMENTAL RESULTS\nA. The Impact of Cross-Party Regularization λ on Local SSL\nand Model Aggregation\nWe use SimSiam as the base SSL method for FedGSSL∗\nand FedHSSL∗to investigate the impact of γ. All local data\nand 20% aligned data are used for the pretraining. 200 labeled\nand aligned samples are used for the finetuning.\nFig. 8.\nMain task performance of FedGSSL∗and FedHSSL∗(use only\nlocal encoder) pretrained by various γ values. γ = 0 means no cross-party\nregularization is applied to local SSL.\nFig. 8 depicts the main task performance of FedGSSL∗and\nFedHSSL∗using pretrained local encoders when γ increases.\nFrom Fig. 8, we observe that: i) the performance of FedGSSL∗\nand FedHSSL∗increase noticeably when λ > 0 than those\nof FedGSSL∗and FedHSSL∗when λ = 0 on four datasets,\ndemonstrating that the cross-party regularization helps en-\nhance the performance. ii) FedHSSL∗constantly outperforms\nFedGSSL∗on four datasets when the λ is chosen from a\nproper range (i.e., 0.5 to 1.5 in this experiment), indicating\nthat the cross-party regularization has a positive impact on the\npartial model aggregation when properly choosing λ. iii) the\nvalue of λ that leads to the best performance is different for\n\n\nJOURNAL OF L\nAT\nEX CLASS FILES, VOL. 14, NO. 8, JUNE 2023\n12\nTABLE IX\nPERFORMANCE COMPARISON OF FEDCSSL-SIMSIAM AND FEDLOCALSIMSIAM USING VARYING PERCENTAGES OF TRAINING SAMPLES (% OF T.S.)\nFOR PRETRAINING AND 200 LABELED SAMPLES FOR FINETUNING.\nDataset\nNUSWIDE\nAvazu\nBHI\nModelnet\n% of T.S.:\n20%\n40%\n100%\n20%\n40%\n100%\n20%\n40%\n100%\n20%\n40%\n100%\nFedLocalSimSiam\n0.523\n0.517\n0.505\n0.565\n0.566\n0.575\n0.748\n0.755\n0.760\n0.598\n0.609\n0.622\nFedCSSL-SimSiam\n0.535\n0.550\n0.562\n0.615\n0.622\n0.627\n0.762\n0.778\n0.805\n0.652\n0.684\n0.686\nEnhancement\n↑0.012\n↑0.033\n↑0.057\n↑0.050\n↑0.056\n↑0.052\n↑0.014\n↑0.023\n↑0.045\n↑0.054\n↑0.075\n↑0.064\nTABLE X\nPERFORMANCE COMPARISON OF FEDHSSL AND BASELINES WITH DIFFERENT NUMBER OF LABELED SAMPLES. FOR FEDHSSL, RESULTS OF USING\n20% AND 40% ALIGNED SAMPLES ARE GIVEN. TOP-1 ACCURACY IS USED AS THE METRIC FOR NUSWIDE AND MODELNET, WHILE AUC AND\nF1-SCORE ARE THE METRICS FOR AVAZU AND BHI, RESPECTIVELY. % OF ALIGNED SAMPLES APPLIES ONLY TO FEDHSSL.\n# of labeled aligned samples:\n200\n400\n600\n800\n1000\n% of aligned samples:\n20%\n40%\n20%\n40%\n20%\n40%\n20%\n40%\n20%\n40%\nNUSWIDE\n(Top1-Acc)\nLR\n0.530\n0.558\n0.580\n0.589\n0.606\nLGB\n0.425\n0.465\n0.526\n0.556\n0.587\nFedSplitNN\n0.495\n0.535\n0.560\n0.573\n0.591\nFedLocalSimSiam\n0.505\n0.536\n0.596\n0.603\n0.612\nFedLocalBYOL\n0.514\n0.527\n0.585\n0.599\n0.606\nFedLocalMoCo\n0.566\n0.596\n0.625\n0.634\n0.639\nFedHSSL-SimSiam\n0.574\n0.607\n0.624\n0.641\n0.636\n0.651\n0.643\n0.662\n0.654\n0.670\nFedHSSL-BYOL\n0.551\n0.598\n0.592\n0.624\n0.617\n0.645\n0.633\n0.659\n0.640\n0.664\nFedHSSL-MoCo\n0.611\n0.615\n0.636\n0.642\n0.653\n0.658\n0.662\n0.668\n0.665\n0.670\nAvazu\n(AUC)\nLR\n0.554\n0.574\n0.596\n0.602\n0.575\nLGB\n0.563\n0.568\n0.595\n0.621\n0.620\nFedSplitNN\n0.588\n0.581\n0.599\n0.595\n0.615\nFedLocalSimSiam\n0.575\n0.585\n0.591\n0.608\n0.629\nFedLocalBYOL\n0.560\n0.597\n0.600\n0.601\n0.605\nFedLocalMoCo\n0.573\n0.591\n0.584\n0.596\n0.601\nFedHSSL-SimSiam\n0.616\n0.623\n0.625\n0.636\n0.631\n0.649\n0.644\n0.648\n0.657\n0.663\nFedHSSL-BYOL\n0.610\n0.615\n0.617\n0.634\n0.626\n0.631\n0.630\n0.630\n0.641\n0.648\nFedHSSL-MoCo\n0.614\n0.616\n0.623\n0.632\n0.635\n0.638\n0.637\n0.641\n0.646\n0.658\nBHI\n(F1-Score)\nFedSplitNN\n0.731\n0.738\n0.754\n0.752\n0.760\nFedLocalSimSiam\n0.760\n0.764\n0.788\n0.785\n0.798\nFedLocalBYOL\n0.760\n0.769\n0.781\n0.786\n0.796\nFedLocalMoCo\n0.763\n0.771\n0.784\n0.793\n0.800\nFedHSSL-SimSiam\n0.803\n0.805\n0.799\n0.816\n0.816\n0.822\n0.824\n0.823\n0.823\n0.830\nFedHSSL-BYOL\n0.788\n0.791\n0.793\n0.806\n0.808\n0.821\n0.811\n0.822\n0.817\n0.825\nFedHSSL-MoCo\n0.797\n0.806\n0.800\n0.817\n0.815\n0.822\n0.817\n0.829\n0.818\n0.831\nModelnet\n(Top1-Acc)\nFedSplitNN\n0.612\n0.684\n0.733\n0.765\n0.771\nFedLocalSimSiam\n0.622\n0.698\n0.761\n0.779\n0.797\nFedLocalBYOL\n0.635\n0.707\n0.760\n0.775\n0.794\nFedLocalMoCo\n0.659\n0.722\n0.784\n0.798\n0.815\nFedHSSL-SimSiam\n0.678\n0.707\n0.763\n0.772\n0.793\n0.806\n0.806\n0.826\n0.826\n0.833\nFedHSSL-BYOL\n0.678\n0.681\n0.740\n0.752\n0.778\n0.800\n0.799\n0.807\n0.812\n0.825\nFedHSSL-MoCo\n0.696\n0.705\n0.760\n0.764\n0.787\n0.804\n0.809\n0.822\n0.826\n0.830\ndifferent datasets, indicating that λ should be carefully tuned\nfor different datasets (and models).\nB. Federated Cross-Party SSL vs. Local SSL in Learning\nRepresentation\nWe compare the performance of FedCSSL-SimSiam and\nFedLocalSimSiam using varying percentages of aligned sam-\nples for SSL (i.e., 20%, 40%, and 100%) and the same amount\n(i.e., 200) of labeled samples for finetuning. Table IX reports\nthat FedCSSL-SimSiam outperforms FedLocalSimSiam on all\nsample percentages across all datasets. With more samples\nused for pretraining (from 20% to 100%), the performance\nimprovement becomes larger, especially on NUSWIDE (by\n0.045) and BHI (by 0.031). This demonstrates that FedCSSL-\nSimSiam is more effective in pretraining representation than\nFedLocalSimSiam, indicating that the features (cross-party\nviews) of aligned samples form better positive pairs for the\nSSL than the local augmentation. These experiments prove\nthe merit of VFL in building better machine learning models.\nC. The Impact of the Amount of Aligned Samples on FedHSSL\nWe compare the performance of FedHSSL using various\namount of aligned samples, 20% and 40% respectively. The\nresults in Table X show that the performance of FedHSSL\nimproves constantly with more aligned samples. This suggests\nthat more aligned samples help FedHSSL generate better\nrepresentations for downstream tasks.\nD. Privacy Analysis Of FedHSSL with Different Aligned Sam-\nples\nWe investigate the privacy-utility trade-off of FedHSSL\nin terms of various amount of aligned samples. We use\nSimSiam as the base SSL method for FedHSSL. As shown\n\n\nJOURNAL OF L\nAT\nEX CLASS FILES, VOL. 14, NO. 8, JUNE 2023\n13\nTABLE XI\nCOMPARISON OF MC ATTACK (PRIVACY LEAKAGE) VS. MAIN TASK (UTILITY) TRADE-OFFS FOR ISO-PROTECTED FEDLOCALSIMSIAM AND\nFEDHSSL-SIMSIAM ON 4 DATASETS WITH 20% AND 40% ALIGNED SAMPLES, RESPECTIVELY. λf INDICATES THE PROTECTION STRENGTH USED IN THE\nFINETUNING PHASE AND λp THE PROTECTION STRENGTH IN THE PRETRAINING PHASE.\nMethod\nFedLocalSimSiam\nFedHSSL-SimSiam (20%)\nFedHSSL-SimSiam (40%)\nDataset\nλf\nASimSiam\nMain\nAFedHSSL\nMain\nλp\nAFedHSSL\nMain\nλp\nNUSWIDE\n1.0\n0.471\n0.494\n0.471\n0.538\n0.4\n0.474\n0.533\n5.0\n5.0\n0.465\n0.487\n0.449\n0.519\n0.4\n0.471\n0.528\n5.0\n25.0\n0.449\n0.458\n0.443\n0.503\n0.4\n0.458\n0.503\n5.0\nAvazu\n1.0\n0.548\n0.582\n0.568\n0.614\n0.1\n0.571\n0.616\n0.1\n5.0\n0.546\n0.577\n0.565\n0.602\n0.1\n0.566\n0.610\n0.1\n25.0\n0.545\n0.576\n0.563\n0.594\n0.1\n0.561\n0.603\n0.1\nBHI\n1.0\n0.710\n0.756\n0.692\n0.783\n0.1\n0.686\n0.788\n0.1\n5.0\n0.699\n0.732\n0.672\n0.764\n0.1\n0.687\n0.773\n0.1\n25.0\n0.685\n0.710\n0.674\n0.758\n0.1\n0.682\n0.764\n0.1\nModelnet\n1.0\n0.438\n0.597\n0.451\n0.652\n0.1\n0.466\n0.658\n0.1\n5.0\n0.415\n0.573\n0.447\n0.613\n0.1\n0.448\n0.631\n0.1\n25.0\n0.408\n0.564\n0.415\n0.594\n0.1\n0.419\n0.598\n0.1\nTABLE XII\nCOMPARISON OF CALIBRATED AVERAGED PERFORMANCE (CAP) OF ISO-PROTECTED FEDSPLITNN, FEDLOCALSIMSIAM AND FEDHSSL-SIMSIAM\nAGAINST THE MC ATTACK ON 4 DATASETS. CAP QUANTIFIES THE PRIVACY-UTILITY TRADE-OFF CURVES VISUALIZED IN ABOVE 4 FIGURES. The higher\nthe CAP value is, the better the method is at preserving privacy without compromising the main task performances. NUMBERS ON THE FIGURES ARE\nVALUES OF ISO PROTECTION STRENGTH λf CHOSEN FROM [1, 5, 25]. A BETTER TRADE-OFF CURVE SHOULD BE MORE TOWARDS THE BOTTOM-RIGHT\nCORNER OF EACH FIGURE.\nDataset\nFedSplitNN\nFedLocalSimSiam\nFedHSSL-SimSiam (20%)\nFedHSSL-SimSiam (40%)\nNUSWIDE\n0.264\n0.258\n0.284\n0.277\nAvazu\n0.238\n0.262\n0.262\n0.264\nBHI\n0.242\n0.221\n0.246\n0.244\nModelnet\n0.342\n0.334\n0.348\n0.349\nin Table XI, with more aligned samples (i.e., from 20% to\n40%) are used for pretraining, the main task performance\nof FedHSSL-SimSiam is generally improved while the label\nrecovery accuracy is also increasing when the same level of\nprotection strength is applied. This trends is also illustrated\nin figures of Table XII, which reports that, while FedHSSL-\nSimSiam gives different privacy-utility trade-off curves when\nleveraging different amount of aligned samples, the two curves\nhave similar CAP values. This result manifests that the number\nof aligned samples is an important factor that impacts the\nprivacy-utility trade-off of FedHSSL, and should be considered\nwhen applying FedHSSL.\nREFERENCES\n[1] H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A.\ny Arcas, “Communication-Efficient Learning of Deep Networks from\nDecentralized Data,” in Artificial intelligence and statistics.\nPMLR,\n2017, pp. 1273–1282.\n[2] Q. Yang, Y. Liu, Y. Cheng, Y. Kang, T. Chen, and H. Yu, “Federated\nLearning,” Synthesis Lectures on Artificial Intelligence and Machine\nLearning, vol. 13, no. 3, pp. 1–207, Dec. 2019.\n[3] Y. Kang, Y. He, J. Luo, T. Fan, Y. Liu, and Q. Yang, “Privacy-\npreserving federated adversarial domain adaptation over feature groups\nfor interpretability,” IEEE Transactions on Big Data, pp. 1–12, 2022.\n[4] B. Tan, B. Liu, V. Zheng, and Q. Yang, A Federated Recommender\nSystem for Online Services.\nNew York, NY, USA: Association\nfor Computing Machinery, 2020, p. 579–581. [Online]. Available:\nhttps://doi.org/10.1145/3383313.3411528\n[5] Y. Kang, Y. Liu, and X. Liang, “FedCVT: Semi-supervised Vertical\nFederated Learning with Cross-view Training,” ACM Transactions on\nIntelligent Systems and Technology (TIST), May 2022.\n[6] W. Zhuang, Y. Wen, and S. Zhang, “Divergence-aware Federated\nSelf-Supervised Learning,” in International Conference on Learning\nRepresentations, 2022.\n[7] K.-F. Chu and L. Zhang, “Privacy-Preserving Self-Taught Federated\nLearning for Heterogeneous Data,” CoRR, vol. abs/2106.15147, 2021.\n[8] T. Castiglia, S. Wang, and S. Patterson, “Self-supervised vertical\n\n\nJOURNAL OF L\nAT\nEX CLASS FILES, VOL. 14, NO. 8, JUNE 2023\n14\nfederated learning,” in Workshop on Federated Learning: Recent\nAdvances\nand\nNew\nChallenges\n(in\nConjunction\nwith\nNeurIPS\n2022),\n2022.\n[Online].\nAvailable:\nhttps://openreview.net/forum?id=\nz2RNsvYZZTf\n[9] S. Feng, “Vertical federated learning-based feature selection with non-\noverlapping sample utilization,” Expert Systems with Applications, vol.\n208, p. 118097, Dec. 2022.\n[10] W. Li, Q. Xia, J. Deng, H. Cheng, J. Liu, K. Xue, Y. Cheng, and\nS.-T. Xia, “Achieving Lightweight Federated Advertising with Self-\nSupervised Split Distillation,” Sep. 2022.\n[11] Q. Yang, Y. Liu, T. Chen, and Y. Tong, “Federated Machine Learning:\nConcept and Applications,” ACM Transactions on Intelligent Systems\nand Technology, vol. 10, no. 2, pp. 12:1–12:19, Jan. 2019.\n[12] S. Hardy, W. Henecka, H. Ivey-Law, R. Nock, G. Patrini, G. Smith, and\nB. Thorne, “Private federated learning on vertically partitioned data via\nentity resolution and additively homomorphic encryption,” CoRR, vol.\nabs/1711.10677, 2017.\n[13] C. Chen, J. Zhou, L. Wang, X. Wu, W. Fang, J. Tan, L. Wang, A. X. Liu,\nH. Wang, and C. Hong, “When homomorphic encryption marries secret\nsharing: Secure large-scale sparse logistic regression and applications\nin risk control,” in Proceedings of the 27th ACM SIGKDD Conference\non Knowledge Discovery and Data Mining, ser. KDD ’21.\nNew York,\nNY, USA: Association for Computing Machinery, 2021, p. 2652–2662.\n[Online]. Available: https://doi.org/10.1145/3447548.3467210\n[14] K. Cheng, T. Fan, Y. Jin, Y. Liu, T. Chen, D. Papadopoulos, and\nQ. Yang, “SecureBoost: A Lossless Federated Learning Framework,”\nIEEE Intelligent Systems, vol. 36, no. 6, pp. 87–98, 2021.\n[15] Y. Liu, Y. Kang, C. Xing, T. Chen, and Q. Yang, “A Secure Federated\nTransfer Learning Framework,” IEEE Intelligent Systems, vol. 35, no. 4,\npp. 70–82, Jul. 2020.\n[16] P.\nBachman,\nR.\nD.\nHjelm,\nand\nW.\nBuchwalter,\n“Learning\nrepresentations by maximizing mutual information across views,” in\nAdvances in Neural Information Processing Systems, H. Wallach,\nH.\nLarochelle,\nA.\nBeygelzimer,\nF.\nd'Alch´\ne-Buc,\nE.\nFox,\nand\nR.\nGarnett,\nEds.,\nvol.\n32.\nCurran\nAssociates,\nInc.,\n2019. [Online]. Available: https://proceedings.neurips.cc/paper/2019/\nfile/ddf354219aac374f1d40b7e760ee5bb7-Paper.pdf\n[17] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A Simple Framework\nfor Contrastive Learning of Visual Representations,” in International\nconference on machine learning.\nPMLR, 2020, pp. 1597–1607.\n[18] Q. Li, B. He, and D. Song, “Model-Contrastive Federated Learning,”\nMar. 2021.\n[19] X. Mu, Y. Shen, K. Cheng, X. Geng, J. Fu, T. Zhang, and Z. Zhang,\n“FedProc: Prototypical Contrastive Federated Learning on Non-IID\ndata,” Sep. 2021.\n[20] F. Zhang, K. Kuang, Z. You, T. Shen, J. Xiao, Y. Zhang, C. Wu,\nY. Zhuang, and X. Li, “Federated Unsupervised Representation Learn-\ning,” CoRR, vol. abs/2010.08982, Oct. 2020.\n[21] W. Zhuang, X. Gan, Y. Wen, S. Zhang, and S. Yi, “Collaborative\nUnsupervised Visual Representation Learning from Decentralized Data,”\nin Proceedings of the IEEE/CVF International Conference on Computer\nVision, 2021, pp. 4912–4921.\n[22] C. He, Z. Yang, E. Mushtaq, S. Lee, M. Soltanolkotabi, and S. Aves-\ntimehr, “SSFL: Tackling Label Deficiency in Federated Learning via\nPersonalized Self-Supervision,” in International Workshop on Trustable,\nVerifiable and Auditable Federated Learning in Conjunction with AAAI\n2022 (FL-AAAI-22), Oct. 2021.\n[23] Y. Yang, X. Ye, and T. Sakurai, “Multi-View Federated Learning\nwith Data Collaboration,” in 2022 14th International Conference on\nMachine Learning and Computing (ICMLC), ser. ICMLC 2022.\nNew\nYork, NY, USA: Association for Computing Machinery, Jun. 2022, pp.\n178–183.\n[24] C.-j. Huang, L. Wang, and X. Han, “Vertical Federated Knowledge\nTransfer via Representation Distillation for Healthcare Collaboration\nNetworks,” in Proceedings of the ACM Web Conference 2023, ser.\nWWW ’23.\nNew York, NY, USA: Association for Computing Ma-\nchinery, Apr. 2023, pp. 4188–4199.\n[25] Z. Ren, L. Yang, and K. Chen, “Improving Availability of Vertical\nFederated Learning: Relaxing Inference on Non-overlapping Data,”\nACM Transactions on Intelligent Systems and Technology, vol. 13,\nno. 4, pp. 58:1–58:20, Jun. 2022.\n[26] W. Li, Q. Xia, H. Cheng, K. Xue, and S.-T. Xia, “Vertical Semi-\nFederated Learning for Efficient Online Advertising,” Sep. 2022.\n[27] Y. Liu, Y. Kang, C. Xing, T. Chen, and Q. Yang, “Secure Federated\nTransfer Learning,” IEEE Intelligent Systems, vol. 35, no. 4, pp. 70–82,\nJul. 2020.\n[28] S. Feng and H. Yu, “Multi-Participant Multi-Class Vertical Federated\nLearning,” Jan. 2020.\n[29] S. Feng, B. Li, H. Yu, Y. Liu, and Q. Yang, “Semi-Supervised Federated\nHeterogeneous Transfer Learning,” Knowledge-Based Systems, vol. 252,\np. 109384, Sep. 2022.\n[30] ——, “Semi-Supervised Federated Heterogeneous Transfer Learning,”\nKnowledge-Based Systems, vol. 252, p. 109384, Sep. 2022.\n[31] Y. Tan, G. Long, J. Ma, L. Liu, T. Zhou, and J. Jiang, “Federated\nLearning from Pre-Trained Models: A Contrastive Learning Approach,”\nSep. 2022.\n[32] S. Han, S. Park, F. Wu, S. Kim, C. Wu, X. Xie, and M. Cha, “FedX:\nUnsupervised Federated Learning with Cross Knowledge Distillation,”\nJul. 2022.\n[33] Z. He, T. Zhang, and R. B. Lee, “Model inversion attacks against\ncollaborative inference,” in Proceedings of the 35th Annual Computer\nSecurity Applications Conference, 2019, pp. 148–162.\n[34] O. Li, J. Sun, X. Yang, W. Gao, H. Zhang, J. Xie, V. Smith, and\nC. Wang, “Label leakage and protection in two-party split learning,” in\nInternational Conference on Learning Representations, 2022. [Online].\nAvailable: https://openreview.net/forum?id=cOtBRgsf2fO\n[35] C. Fu, X. Zhang, S. Ji, J. Chen, J. Wu, S. Guo, J. Zhou, A. X. Liu, and\nT. Wang, “Label inference attacks against vertical federated learning,”\nin 31st USENIX Security Symposium (USENIX Security 22), 2022.\n[36] T. Zou, Y. Liu, Y. Kang, W. Liu, Y. He, Z. Yi, Q. Yang, and Y. Zhang,\n“Defending batch-level label inference and replacement attacks in ver-\ntical federated learning,” IEEE Transactions on Big Data, pp. 1–12, jul\n2022.\n[37] Y. Liu, Y. Kang, T. Zou, Y. Pu, Y. He, X. Ye, Y. Ouyang, Y.-\nQ. Zhang, and Q. Yang, “Vertical federated learning,” arXiv preprint\narXiv:2211.12814, 2022.\n[38] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum Contrast\nfor Unsupervised Visual Representation Learning,” in Proceedings of\nthe IEEE/CVF conference on computer vision and pattern recognition,\n2020, pp. 9729–9738.\n[39] J.-B.\nGrill,\nF.\nStrub,\nF.\nAltch´\ne,\nC.\nTallec,\nP.\nH.\nRichemond,\nE. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar,\nB. Piot, K. Kavukcuoglu, R. Munos, and M. Valko, “Bootstrap your\nown latent: A new approach to self-supervised Learning,” Advances in\nneural information processing systems, vol. 33, pp. 21 271–21 284, 2020.\n[40] X. Chen and K. He, “Exploring simple siamese representation learning,”\nin Proceedings of the IEEE/CVF Conference on Computer Vision and\nPattern Recognition, 2021, pp. 15 750–15 758.\n[41] Y. Wu, Y. Kang, J. Luo, Y. He, and Q. Yang, “Fedcg: Leverage\nconditional gan for protecting privacy and maintaining competitive\nperformance in federated learning,” in Proceedings of the Thirty-First\nInternational Joint Conference on Artificial Intelligence, IJCAI-22.\nIn-\nternational Joint Conferences on Artificial Intelligence Organization,\n2022.\n[42] T.-S. Chua, J. Tang, R. Hong, H. Li, Z. Luo, and Y.-T. Zheng, “NUS-\nWIDE: A real-world web image database from national university of\nsingapore,” in Proc. of ACM Conf. on Image and Video Retrieval\n(CIVR’09), Santorini, Greece., Jul. 2009.\n[43] S. Wang and W. Cukierski, “Click-Through Rate Prediction,” https://\nkaggle.com/competitions/avazu-ctr-prediction, 2014.\n[44] P. Mooney, “Breast histopathology images,” https://www.kaggle.com/\ndatasets/paultimothymooney/breast-histopathology-images, 2016.\n[45] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao,\n“3D ShapeNets: A deep representation for volumetric shapes,” in\n2015 IEEE Conference on Computer Vision and Pattern Recognition\n(CVPR), Jun. 2015, pp. 1912–1920.\n[46] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-\nY. Liu, “Lightgbm: A highly efficient gradient boosting decision tree,”\nAdvances in neural information processing systems, vol. 30, 2017.\n[47] D.\nBahri,\nH.\nJiang,\nY.\nTay,\nand\nD.\nMetzler,\n“SCARF:\nSelf-\nSupervised Contrastive Learning using Random Feature Corruption,” in\nInternational Conference on Learning Representations, Jun. 2022.\n[48] T. Yao, X. Yi, D. Z. Cheng, F. Yu, T. Chen, A. Menon, L. Hong, E. H.\nChi, S. Tjoa, J. Kang, and E. Ettinger, “Self-supervised Learning for\nLarge-scale Item Recommendations,” in Proceedings of the 30th ACM\nInternational Conference on Information & Knowledge Management,\n2021, pp. 4321–4330.\n[49] Y. Liu, X. Zhang, Y. Kang, L. Li, T. Chen, M. Hong, and Q. Yang,\n“FedBCD: A Communication-Efficient Collaborative Learning Frame-\nwork for Distributed Features,” IEEE Transactions on Signal Processing,\n2022.\n\n\nJOURNAL OF L\nAT\nEX CLASS FILES, VOL. 14, NO. 8, JUNE 2023\n15\n[50] Y. Kang, J. Luo, Y. He, X. Zhang, L. Fan, and Q. Yang, “A framework\nfor evaluating privacy-utility trade-off in vertical federated learning,”\narXiv preprint arXiv:2209.03885, 2022.\n[51] L. Fan, K. W. Ng, C. Ju, T. Zhang, C. Liu, C. S. Chan, and Q. Yang,\nRethinking Privacy Preserving Deep Learning: How to Evaluate and\nThwart Privacy Attacks. Cham: Springer International Publishing, 2020,\npp. 32–50.\n[52] Y. Liu, X. Liang, J. Luo, Y. He, T. Chen, Q. Yao, and Q. Yang,\n“Cross-Silo Federated Neural Architecture Search for Heterogeneous\nand Cooperative Systems,” in Federated and Transfer Learning, ser.\nAdaptation, Learning, and Optimization, R. Razavi-Far, B. Wang, M. E.\nTaylor, and Q. Yang, Eds.\nCham: Springer International Publishing,\n2023, pp. 57–86.","difficulty":"hard","domain":"Single-Document QA","length":"short","question":"In terms of data classification, which types of data are introduced in the article and the method FEDHSSL mentioned in the text uses which parts of the data during the pre-training phase?","sub_domain":"Academic"}

Source: https://huggingface.co/datasets/zai-org/LongBench-v2

initial import

Posting: /agents

GET /api/v1/write?intent=publish&task_id=4b28b6ba-c5ab-5404-86fa-01d83d0edea8&body={url_encoded_text}&agent_name={optional_name}&nonce={optional_random_id}