Tree-based Models for Vertical Federated Learning: A Survey

Random forest (94) is a bagging-based ensemble supervised machine learning technique, which usually uses Classification and Regression Trees (CARTs) as weak learners. The randomization of a random forest can be presented in two different ways: random selection of samples and random selection of features. After building several CARTs, the predicted results are derived by either majority voting (for classification tasks) or averaging operation (for regression tasks).

Given a dataset $\mathcal{D}\in\mathbb{R}^{m\times d}$ with $m$ samples and $d$ features, GBDT (95) predicts the output by using T regression trees, which can be formulated as:

where $\mathcal{F}=\{f(x)=w_{q(x)}\}$ is a collection of CARTs. The function $q(x)$ maps each input feature $x$ to the corresponding leaf index, and $w\in\mathbb{R}^{K}$ is the weight vector, where $K$ is the number of leaves in the tree. GBDT tries to minimize the following regularized loss function (3):

where $\Omega(f_{t})=\frac{1}{2}\lambda\|w\|^{2}$ is a regularization term, and $\lambda$ is a hyperparameter to control the strength of regularization.

In order to optimize the above loss function, GBDT minimizes the following function at the $t$-th iteration (96):

where $\mathcal{L}^{(t)}$ denotes the $t$-th loss of the training process, $l(y_{i},\hat{y}_{i}^{(t-1)})$ is the loss function of the $i$-th sample between the prediction of the $(t-1)$-th iteration $\hat{y}_{i}^{(t-1)}$ and the target value $y_{i},$ $g_{i}=\partial_{\hat{y}^{(t-1)}}l(y_{i},\hat{y}_{i}^{(t-1)})$ is the first order gradient statistics on the loss function.

Finally, the optimal weight of the $j$-th leaf node can be given as:

where $I_{j}$ denotes the sample indices that belong to the $j$-th leaf node. To find the best splitting threshold for the internal node, GBDT greedily maximizes the following gain score:

where $I_{L}$ and $I_{R}$ are the instance sets of left and right nodes after the split.

XGBoost (3) is an efficient implementation of GBDT. One of the most important improvements made by XGBoost is that a second-order Taylor expansion function is used to approximate the loss function, as defined by:

where $h_{i}=\partial^{2}_{\hat{y}^{(t-1)}}l(y_{i},\hat{y}_{i}^{(t-1)})$ is the second order gradient statistics on the loss function and $\Omega(f_{t})=\gamma T+\frac{1}{2}\lambda\|w\|^{2}$. Here, $\gamma$ and $\lambda$ represent the regularizers to adjust the number and weights of leaves, respectively.

As a result, the optimal weight of the $j$-th leaf node can be calculated as:

where $I_{j}$ denotes the sample indices that belong to the $j$-th leaf node. In order to find the best splitting threshold for the internal nodes, XGBoost greedily maximizes the following gain score:

where $I_{L}$ and $I_{R}$ are the instances of left and right nodes after the splitting operation with $I=I_{L}\cup I_{R}$.

Differential privacy (51; 52; 53) is the technology that enables researchers to avail a facility in obtaining useful information from the databases, containing people’s personal information, without divulging the personal identification of individuals. Local DP (LDP) (97; 98; 99; 100) is one type of DP, which is based on clients adding DPs to the data themselves. The distance-based LDP (101; 102; 103) is a special kind of LDP, which measures the level of privacy assurance between any pair of sensitive data based on their distance from each other.

Homomorphic encryption (54; 55; 56; 57; 58; 59) is a special encryption method that allows the ciphertext to be processed including addition and multiplication to obtain a result that is still encrypted. However, it usually requires more computation resources or storage costs. A partially homomorphic encryption (PHE) scheme is a probabilistic asymmetric encryption scheme for restricted computation over the ciphertexts.

Secure Multi-Party Computation (60; 61; 62; 63; 64; 65) is proposed to solve the problem of how to safely compute a conventional function in the absence of a trusted third party. Among the different protocols in SMPC, one of the widely-used techniques is Secret Sharing (104). Secret sharing suggests partitioning a secret value into several frames and sending one of the frames to a party, which satisfies that only a certain number of parties can jointly complete the decryption process for exactly recovering the secret value. From the perspectives of types, different secret-sharing techniques might allow secret addition, secret multiplication, secret division, and other mathematical operations.

Trusted Execution Environments (105; 106) is a separate processing environment with computing and storage capabilities that provide security and integrity protection. The basic idea is that a separate isolated memory is allocated in hardware for sensitive data, all calculations of sensitive data are performed in this memory, and no other part of the hardware can access the information in this isolated memory except for authorized interfaces. In this way, private computation of sensitive data can be achieved.

The main idea of feature-gathering TBMs is to modify the splitting rule finding function $h$ as:

where $x^{(m)}\sim\mathcal{X}_{m}\ \forall m\in[M]$. In other words, the calculation of splitting rule $r$ contains the following two steps: (1) Each data party completes the sub-task functioned as $g$, taking its feature stored locally as input and outputting some (protected) intermediate results $g(x^{(m)})$. Then these intermediate results are sent to the task party; (2) After receiving all the intermediate results from the data parties, the task party calculates the splitting rule $r$ based on $g(x^{(m)})\ \forall m\in[M]$ and the labels $y$.

It is worth pointing out that the splitting rule finding function $h$ defined in Eq.(10) brings some issues for building feature-gathering TBMs. Since the task party only receives the intermediate results rather than the features from data parties, it could not identify the found splitting rules (i.e., the splitting features and splitting values) without communicating with the owner of the splitting features.

To solve this, what the task party actually obtained from $h_{\text{feature}}$ is a “pointer” of the splitting feature and splitting value, and the “pointer” should be sent back to the corresponding data party that is able to look up the real splitting feature and splitting value based on the “pointer”. For example, the splitting rule found by the task party can be “the $5$-th feature of party $m$, the $120$-th-ranked value”, and it can only be recovered by party $m$ to “age” (i.e., the splitting feature) and “45” (i.e., the splitting value). The splitting features and splitting values are stored in data parties, and the task party only knows the “pointer” and should query for these splitting results when needed, such as during the inference process (more details can be found in Section 3.3).

One of the widely adopted instantiations of the function $g$ is getting the ordinal numbers of samples according to one’s features, a.k.a., the data sample indices ranked by the feature values. The data parties sort their data according to each of the features, respectively, and send the resulting ordinal numbers to the task party. Therefore, based on these ordinal numbers, the task party can calculate the gains achieved by different splitting rules and find the best one.

The overall process of building feature-gathering TBMs is demonstrated in Algorithm 1. Specifically, the task party builds $T$ decision trees based on the received intermediate results $g(x^{(m)})\ \forall m\in[M]$ and labels $y$.

For each decision tree, the data party traverses from the root node to the leaf nodes, finding the splitting rule for each internal node that can achieve the best gain on the data samples. All the data samples are fed into the root of the tree at the beginning (line 6), and partitioned into two subsets (or $k$ subsets for a $k$-way tree) according to the splitting rule (line 15). These two subsets go to the children of the root, respectively (line 16), and such a partition process is repeated at every traversed node until the leaf nodes are reached. For the leaf node, the task party sets the output of the leaf node according to the reached data (line 11), e.g., calculating a weight according to Equation (8) in XGBoost or performing major voting in random forest. The naive approach for finding the best splitting rule, i.e., $h_{\text{feature}}$ in line 13, can be exhaustive enumeration: For each possible splitting position, the task party calculates the gain (e.g., the score defined in Equation (9) in XGBoost or the Gini coefficient in a random forest) accordingly. The task party finally chooses the splitting rule that can achieve the best gain at every internal node. Several advanced algorithms for balancing the efficiency and effectiveness of finding the optimal splitting rule have been proposed recently (107; 108).

In a nutshell, we can conclude the communication and computation protocol of feature-gathering TBMs: (1) Feature-related information is sent from data parties to the task party in a privacy-preserving manner; (2) Most of the computation behaviors happen at the task party, including calculating the gain of different splitting rules to find the solution (i.e., line 13 in Algorithm 1), and further partitioning data samples into subsets (i.e., line 15 in Algorithm 1). As a result, it can be implied that the task party can be the bottleneck of computation and communication in feature-gathering TBMs. The reason is that both feature-related information and labels are pooled at the task party for building decision trees.

The privacy threats, aimed at the shared feature-related information (e.g., the ordinal numbers of data samples) in feature-gathering TBMs, come from the semi-honest task part and insecure communication channels. In order to avoid leaking private information, it is necessary for the data parties to apply some protection mechanisms to feature-related information before sharing it. Such protection mechanisms should be carefully designed to balance the strength of protection and the informativeness of shared information. Here we briefly introduce several representative algorithms.

FederBoost (38) proposes to provide LDP privacy protection for ordinal numbers by adding noises to $g(x^{(m)})$, i.e., making improvements at line $2$ in Algorithm 1. To be more specific, as shown in Figure 4, each data party first sorts the training samples according to its feature values for obtaining the ordinal numbers, and then partitions the ordinal numbers into several buckets sequentially. To achieve $\epsilon$-DP, a mapping is applied, making that each ordinal number might move to other buckets with a certain probability (controlled by a hyperparameter to achieve a good utility-privacy trade-off), or just stay at the correct bucket. The ordinal numbers inside the buckets would be randomly shuffled before sharing, therefore the relative order between buckets can be preserved mostly while the orders within each bucket are protected. In this way, the data parties generate the protected ordinal numbers that can be sent to the task party for training feature-gathering TBMs.

OpBoost (39) designs a probabilistic order-preserving desensitization algorithm for privacy-preserving vertical federated tree boosting, which satisfies distance-based LDP. The main idea of OpBoost is to pay more attention to enhancing the indistinguishability of private feature values from their nearby neighbors. Indeed, as shown in Figure 5, the feature values are desensitized into a unified discrete value domain with the predefined lower and upper bound. After that, a mapping satisfying distance-based LDP is used to transform each feature value in the domain based on a distance-based scoring function, i.e., values will be mapped to nearby values with high probability. Finally, the data parties sort the training samples according to these desensitized feature values, generating the protected ordinal numbers. Such protected ordinal numbers are sent to the task party for training feature-gathering TBMs, which is proven to be a good balance of preventing privacy leakage and preserving useful information.

In a nutshell, feature-gathering TBMs propose that the data parties calculate ordinal numbers of samples based on their private feature values in a private-preserving manner, and send these desensitized results to the task party for learning optimal splitting rules. As a result, the task party gathers the feature-related information from all data parties, and assumes the responsibility for the computation of finding the optimal splitting rules at every node of the decision trees, while the data parties only need to sort the samples and disrupt the generated ordinal numbers. The major communication overhead of feature-gathering TBMs is the ordinal numbers of samples, which would be sent from the data parties to the task party. The characteristics of feature-gathering TBMs are summarized in Table 1.

Note that there is a trade-off between model utility and privacy protection strength when training feature-gathering TBMs, as pointed out by previous studies (38; 39). Though differential privacy algorithms can enhance privacy protection strength, they might also cause a slight performance drop in the learned model. Such a trade-off implies that the protection strength should be carefully determined in real-world applications, and also inspires the research community to design advanced algorithms to achieve better model utility with a certain allocated privacy budget.

Different from feature-gathering TBMs introduced above, label-scattering TBMs propose another protocol for building decision trees in the VFL scenario, i.e., the task party broadcasts the label-related information to data parties.

Formally, the main idea of label-scattering TBMs is to modify the splitting rule finding function $h$ as:

where $x^{(m)}\sim\mathcal{X}_{m}\ \forall m\in[M]$, and $e(y)$ denotes the label-related information calculated by the task party via the function $e$. Accordingly, the calculation of the splitting rule consists of the following three steps: (1) The task party applies the function $e$ on label $y$, and broadcasts the produced label-related information $e(y)$ to all data parties; (2) After receiving $e(y)$, each data party completes the sub-task functioned as $g$ based on $e(y)$ and its private feature values, and sends the intermediate results $g(x^{(m)},e(y))$ back to the task party; (3) The task party calculates the splitting rule $r$ based on the intermediate results received from data parties.

Similar to that in feature-gathering TBMs, the obtained $h_{\text{label}}$ is just a “pointer” of the splitting feature and splitting value, which should be sent to the corresponding data party for querying. One instantiation of $e(y)$ can be the first-order and second-order gradients, and $g$ can be sorting the received $e(y)$ based on the ordinal numbers of samples ranked by data parties’ feature values.

We describe the overall process of training label-scattering TBMs in Algorithm 2. The goal is to build $T$ decision trees collaboratively.

For each decision tree, the data party needs to generate the label-related information $e(y)$ and then broadcast $e(y)$ to all data parties (lines 2-3). In most cases, $e(y)$ can be different when building different decision trees, e.g., when $e(y)$ is the set of first-order and second-order gradients, it relies on both labels $y$ and decision trees built previously. For each data party $m\in[M]$ who has received $e(y)$, it calculates the intermediate results and sends the results back to the task party (lines 5-6). After receiving the intermediate results from all data parties, the task party is able to traverse from the root node to the leaf nodes for finding the splitting rules that achieve the best gain on the reached data samples (lines 9-21). Such a traversing process is almost the same as the process in feature-gathering TBMs, except for how to partition the data samples into several subsets (line 18). In label-scattering TBMs, the task party could not partition the data samples into subsets based on the found splitting rule $r$, since the task party does not have the feature-related information. Therefore, the task party has to send $r$ to the corresponding data party and wait for the feedback, including some (protected) indicator vectors to denote the data subsets attached to the children of the traversed node.

From the algorithm, we can conclude that in label-scattering TBMs, (1) The task party broadcasts the label-related information to data parties, which might happen before building every decision tree if such information would update; (2) Both the task party and data parties involve in the process of tree building, i.e., the traversing process from roots to leaf nodes. Specifically, keeping communication with each other, the task party finds the splitting rules, and the data parties partition the data samples into subsets accordingly. Compared to feature-gathering TBMs, the communication between the task party and the data parties in label-scattering TBMs is more frequent.

In label-scattering TBMs, some information is vulnerable and required for further protection, including the label-related information $e(y)$, the intermediate results $g(x^{(m)},e(y))$, and the indicator vectors. In order to provide protection, researchers propose several advanced privacy protection mechanisms based on homomorphic encryption and secure multi-party computation.

SecureBoost (36) is focused on XGBoost and applies homomorphic encryption algorithms on $e(y)$ for enhancing privacy protection. An illustration of SecureBoost is shown in Figure 6. Specifically, the task party broadcasts the encrypted $e(y)$ to all data parties. Each data party calculates $g(x^{(m)},e(y))$ independently, following the process of sorting $e(y)$ based on the ordinal numbers ranked by its feature values and partitioning the sorted $e(y)$ into several buckets. These values inside each bucket are summed up and finally sent back to the task party. Therefore, the task party can perform decryption to recover the sum of values in each bucket (due to the characteristic of homomorphic encryption) but cannot get the individual values of $e(y)$.

Further, with the aim of protecting both the label-related information, the intermediate results, and the indicator vectors, recent studies (77; 37) propose to combine partially homomorphic encryption and additive secret sharing techniques. To be more specific, for the task party, each node is attached to an indicator vector for training samples, where each element is a binary value to represent whether a sample reaches the node (value $1$) or not (value $0$). The indicator vector at the root node is initialized as $1$s.

Then the task party splits each indicator vector into several frames, and sends one of the frames to a data party. And the task party also sends the encrypted label-related information $e(y)$. After receiving these results, each data party sorts $e(y)$ according to the ordinal numbers ranked by its feature values, splits the sorted $e(y)$ into several frames, and sends one of the frames to a data/task party. Thus each party owns one frame of the encrypted $e(y)$ with different orders, and they could jointly perform the computation to get the maximal splitting gain for learning the optimal splitting rules via the secret sharing division technique. The data party that holds the splitting feature can update the indicator to represent which subtree would be traversed according to learned splitting rules for each training sample. The updated indicators are also split into several frames and sent to different parties, which can be used for further computation via secret sharing multiplication. In Figure 7 we show the splitting process among two parties, where each party only holds one frame of indicators and label-related information. Here, $L$ and $R$ mean the indicator vector of the incidences of samples belonging to the left and right subtree, respectively. By secret sharing multiplication, the children’s nodes also hold the frames of the protected information.

Although applying partially homomorphic encryption and additive secret sharing techniques can enhance the privacy protection strength for label-scattering TBMs, the additional computation and computation cost they might bring is non-negligible in real-world VFL applications.

Generally speaking, label-scattering TBMs suggest that the task party broadcasts the label-related information to all data parties without raising privacy issues, and data parties calculate and then send the intermediate results back to the task party for learning splitting rules. Finally, the splitting rules would be sent to the data party that owns the corresponding splitting feature for generating the partition of data samples. As a result, the task party scatters the label-related information to data parties, and both the data parties and the task party make their computation for building label-scattering TBMs.

The major communication overhead of label-scattering TBMs is the label-related information, which might be broadcast by the task party before building every decision tree if such information would be updated after finishing building a tree. The characteristics of label-scattering TBMs are summarized in Table 1, which also shows the comparison between feature-gathering TBMs and label-scattering TBMs to highlight their differences.

We conduct a series of experiments to show the differences between feature-gathering TBMs and label-scattering TBMs in terms of model performance and communication frequency, where the communication frequency represents the number of information exchanges between the task party and the data party.

The experimental results of model performance comparisons are shown in Table 3, from which we can see that different feature-gathering TBMs and label-scattering TBMs, including random forest, GBDT, and XGBoost, achieve similar model performance in all adopted datasets. XGBoost (SS) denotes the method proposed in (37), combining XGBoost and secret sharing, which is implemented using SecretFlow¹³¹³13https://github.com/secretflow/secretflow. These results are not surprising, since the feature-gathering TBMs and label-scattering TBMs are different in the communication and computation protocols while both of them are reliable to train decision trees well. However, these two kinds of TBMs need different communication frequencies to complete the tree-building process, as illustrated in Table 4. The communication frequency in XGBoost (ss) is meanless, since the method contains a large number of secret sharing computations, including secret sharing addition, division, and comparison, which makes the communication frequency a huge number. We can observe from the table that label-scattering TBMs need several times of communication frequencies compared to feature-gathering TBMs. The reason is that, in label-scattering TBMs, the task party needs to broadcast the label-related information to the data parties before building every tree. After finding the splitting rules at each node, the task party and the data parties need more communication for data sample partition compared to that in feature-gathering TBMs. These experimental results are consistent with the analysis in Section 3.

In feature-gathering TBMs, in order to prevent privacy leakage, the data parties tend to disrupt ordinal numbers generated based on their private feature. Thus, there exists a trade-off between privacy protection strength and the utility of the model learned based on such desensitized but noisy data. More discussions can be found in Section 3.1.3, and here we provide empirical observation for a better understanding of such a trade-off. We train XGBoost equipped with the privacy protection mechanisms proposed by FederBoost and OpBoost, and show the experimental results in Figure 10 and Figure 11, respectively.

As shown at the x-axis in Figure 10, we vary the probabilities of a sample staying in the correct bucket. From the figure, we can observe that as the probability increases, which implies the intermediate results provided by the data parties are more precise, the model performance becomes better and more stable (i.e., fewer variances), and gradually approaches the results achieved by the model learned without adding DP noise. Similarly, we adjust the privacy protection strength of OpBoost by changing the values of $\epsilon$, which controls the probability of mapping a sample from the $a$-th bucket to $b$-th bucket following $\frac{e^{-|a-b|\cdot\epsilon/2}}{\sum_{j=1}^{50}e^{-|a-j|\cdot\epsilon/2}}$, where $a,b\in[1,50]$, and show the results in Figure 11. These results further confirm the trade-off between model utility and privacy protection strength when training feature-gathering TBMs.

Note that for label-scattering TBMs, the adopted privacy protection mechanisms might not impact the model utility but bring some efficiency issues, as discussed in the next part.

In order to avoid privacy leakage, label-scattering TBMs propose to apply homomorphic encryption and secret sharing techniques, which might bring additional computation and communication costs. The reason is that the encryption/decryption and framing process needs lots of computation resources, and the encrypted and framed information is always lengthy and costs more computation/communication resources than raw information.

In the experiment, we apply Paillier (54) algorithms to encrypt the shared information, and vary the sizes of public and private keys to change the length of encrypted information and thus adjust the privacy protection strength. We compare the time cost and communication overhead when using different key sizes, and show the results in Figure 13 and Figure 13, respectively. From these results, we can see that when the key size becomes large, the communication overhead grows linearly, and time consumption (which includes the additional time cost for encrypting, decrypting, sending, receiving, and handling the messages) is growing super linearly. Such a phenomenon inspires users to choose a suitable key size when using homomorphic encryption and secret sharing techniques, balancing the privacy protection strength and computation/communication cost.

For feature-gathering TBMs where data parties might inject DP noise into the intermediate results, the communication overhead and time consumption when applying different algorithms are kept at the same level. The slight differences mainly come from the computation and time cost for generating noises and disrupting the ordinal numbers.

As the number of data parties increases, the utility of the trained TBMs can be influenced by the privacy protection algorithms adopted by each participant. In an extreme case, if each participant either does not employ any privacy protection algorithms or uses performance-preserving algorithms (such as homomorphic encryption), the performance of the TBMs remains consistent regardless of the number of participants. This is primarily because, within VFL, the tree-building process does not fundamentally differ from that in centralized settings, although it distributes the computations among different participants and thus leads to some additional intermediate calculations. On the other hand, when participants utilize differential privacy for enhanced privacy protection, the number of participants affects the model’s utility.

An increase in participants results in a proportional rise in communication costs during broadcasting, as we typically utilize a peer-to-peer (P2P) transmission protocol in FL, which implies that each piece of information must be independently transmitted to its designated recipient. Furthermore, the computational costs are closely tied to the number of local features and samples. If the increase in participants leads to a corresponding growth in the number of features and samples (regardless of whether some are redundant), the computational costs would also increase.

It is worth noting that the communication and computation protocols discussed in Section 3 impose no restrictions on the number of data parties. Therefore, we set the number of data parties to one during the experiments to provide clear insights regarding the trade-offs between model utility, privacy protection, and communication/computation costs.

In a nutshell, both feature-gathering TBMs and label-scattering TBMs can hardly keep model utility, privacy protection strength, and communication/computation cost at the perfect level at the same time. For example, feature-gathering TBMs are more efficient in achieving an acceptable model utility while exposing desensitized plaintext results, and label-scattering TBMs provide high-level protection strength but need more communication and computation resources. When applying TBMs in the VFL scenario, researchers and developers are encouraged to propose more advanced algorithms to achieve a better trade-off, and balance model utility, privacy protection strength, and communication/computation cost according to their downstream applications.

Finance is crucial for social development, encompassing key areas such as risk management and financial marketing. Risk management involves identifying, evaluating, and mitigating potential risks to an organization’s capital, earnings, and overall operations. Financial marketing focuses on the strategies that financial institutions use to attract, acquire, and retain customers, including branding, advertising, customer relationship management, and digital marketing.

In practice, financial institutions face challenges due to the lack of comprehensive user profiles essential for effective model training, as well as the limited interpretability of predictive results. Training tree-based models within the federated learning (FL) framework can address these issues by enabling secure collaborations that utilize a wealth of data attributes from external sources, such as internet companies. Tree-based models, known for their intuitive structure nature, enhance interpretability by allowing users to easily understand the decision-making process behind predictions. This transparency is crucial for financial institutions, as it fosters trust in model outputs. In this way, institutions can further improve the accuracy of predictive analytics with actionable insights, and, at the same time, ensure the protection of data privacy.

Recently, notable contributions to this field include vertical federated logistic regression (116), vertical federated linear regression (12), and vertical federated boosting tree-based models (36; 117; 76). These approaches employ Homomorphic Encryption (HE) and Secret Sharing (SS) to protect the privacy of transmitted intermediate results.

Specifically, WeBank and its partner companies successfully complete vertical federated modeling using invoice data¹⁴¹⁴14https://www.fedai.org/cases/. This enables WeBank and its partners to collaboratively train the model while keeping their respective data private. The parties exchange encrypted intermediate results and retain control over individual model segments. When it comes time to make predictions, the models from both sides are combined to generate the final prediction. The entire training process prioritizes the security of both the data and the models. Ant Group has announced a financial risk control technology solution based on SMPC that allows for the inclusion of more dimensional credit data into a joint model without compromising the confidentiality of the data sources¹⁵¹⁵15https://www.secretflow.org.cn/en/. This approach helps to construct more accurate big data credit risk control models. The support of this technology solution enhances the necessary collaboration and communication among participants in financial risk control joint projects. It accelerates the transformation from traditional methods to cutting-edge technologies in financial risk management, serves the collaborative supervision of the government and the financial industry, and promotes the development of the financial data market.

Recommendation and advertising are representative domains where VFL holds promising applications. Developing a robust recommendation or advertising system often requires the aggregation of extensive user data from various sources, which raises significant privacy and security concerns. VFL addresses this issue by allowing multiple organizations to collaboratively train a powerful recommendation model without the need to share raw user data.

Consider a scenario where multiple e-commerce platforms aim to develop a combined recommendation or advertising engine that leverages their unique datasets. Each platform contains distinct types of user information, such as purchase history, browsing behavior, user ratings, and demographic details. By utilizing VFL, these platforms can collaboratively train a tree-based model that effectively incorporates the complete feature set available across all platforms, thereby enhancing the relevance of recommendations.

Based on SecureBoost (36), (41) has effectively improved recommendation accuracy by leveraging data from mobile network operators and healthcare providers within an FL environment. Besides, research (69) points out that many companies have successfully implemented tree-based models in VFL for recommendation and advertising systems. For example, ByteDance utilizes the Fedlearner to significantly enhance its advertising effectiveness through the development of a tree-based VFL algorithm; Tencent employs vertical federated GBDTs to establish a VFL federation between advertisers and advertising platforms, resulting in improved model accuracy.

VFL has emerged as a transformative technology in the healthcare sector, especially given the fragmented nature of healthcare data. In this landscape, different institutions, such as hospitals, laboratories, and clinics, often hold critical yet incomplete pieces of patient information. These entities typically maintain separate records that encompass diverse aspects of a patient’s care, including medical history, diagnostic results, and treatment plans.

These organizations can engage in collaborative analysis without compromising sensitive data privacy with VFL, which allows them to collectively train on a more comprehensive dataset, improving the accuracy, robustness, and interpretability of predictive analytics, while strictly adhering to privacy regulations. Vertical federated TBMs hold considerable potential for creating decision rules across multiple data sources to predict patient disease risks, enhance the quality of medical services, improve disease prediction, and optimize treatment protocols. Tree-based models are also adept at handling variable interactions and providing clear classification rules, which assist physicians in developing personalized treatment plans based on the model’s outputs.

Recent studies (43; 44; 45; 46) have designed novel vertical federated TBMs and conducted a series of experiments on healthcare datasets, paving the way for a more integrated approach to healthcare and driving improvements in health outcomes on a larger scale. Specifically, Ant Group’s privacy computing platform, SecretFlow¹⁶¹⁶16https://github.com/secretflow/secretflow, in collaboration with Alibaba Cloud’s medical big data management platform (HData v2.0), utilizes its advanced privacy computing technology to overcome data collaboration barriers among medical institutions in DRGs applications. By integrating artificial intelligence and big data technologies without allowing data to leave the domain, the platform facilitates joint statistics and modeling across institutions. In the realm of medical diagnosis classification, SecretFlow enhances predictive models through the sharing of multi-institutional samples, significantly improving the accuracy of predictions compared to models trained on data from a single institution. The platform is also applied in various scenarios, including disease diagnosis, examination recommendations, medication suggestions, rare disease prediction, and quality control rule management. These applications provide robust solutions for medical data governance, comprehensive hospital quality control, medical research, medical insurance risk management, and critical clinical business challenges.

Addressing communication overhead during training and inference is a critical area for improvement. This is particularly important considering that the number of transmissions between data parties and label parties in TBMs tends to be higher than in other methods. Meanwhile, communication costs would increase with the number of samples for TBMs. As a result, reducing communication overhead might bring significant improvements in cross-silo FL that involve large datasets.

In VFL scenarios, participants do not have access to knowledge about the features owned by others, which can lead to potential redundancy and missed opportunities for creating high-order features through feature crossing. While neural networks may implicitly handle these through non-linear layers, this could be more challenging in TBMs. Thus, enhancing model utility by effectively managing feature abandonment and facilitating feature crossing represents a promising direction, focusing on optimizing the use of available features to improve model accuracy and relevance.

Different participants may possess varying numbers of features, and the importance of these features can be different. An interesting research direction can be proposing mechanisms that ensure the trained model does not become overly biased toward participants with a greater number of features. Conversely, designing suitable incentive mechanisms to reward participants who contribute important features is another valuable area for exploration, leveraging the inherent interpretability of TBMs.

Training TBMs in VFL typically rely on overlapping samples, which can limit their applicability. Therefore, designing mechanisms that allow the participation of non-overlapping samples, such as data augmentation strategies, represents a valuable research direction to further broaden the application scope of TBMs in VFL.