Abstract

In this paper, we propose a new geometric approach for knowledge graph completion via low rank tensor approximation. We augment a pretrained and well-established Euclidean model based on a Tucker tensor decomposition with a novel hyperbolic interaction term. This correction enables more nuanced capturing of distributional properties in data better aligned with real-world knowledge graphs. By combining two geometries together, our approach improves expressivity of the resulting model achieving new state-of-the-art link prediction accuracy with a significantly lower number of parameters compared to the previous Euclidean and hyperbolic models.

1 INTRODUCTION

Most of the information in the world can be expressed in terms of entities and the relationships between them. This information is effectively represented in the form of a knowledge graph (d’Amato, 2021; Peng et al., 2023), which serves as a repository for storing various forms of relational data with their interconnections. Particular examples include storing user profiles on social networking platforms (Xu et al., 2018), organizing Internet resources and the links between them, constructing knowledge bases that capture user preferences to enhance the functionality of recommender systems (Wang et al., 2019a; Guo et al., 2020). With the recent emergence of large language models (LLM), knowledge graphs have become an essential tool for improving the consistency and trustworthiness of linguistic tasks. Among notable examples of their application are fact checking (Pan et al., 2024), hallucinations mitigation (Agrawal et al., 2023), retrieval-augmented generation (Lewis et al., 2020), and generation of corpus for LLM pretraining (Agarwal et al., 2021). This utilization underscores the versatility and utility of knowledge graphs in managing complex datasets and facilitating the manipulation of interconnected information in various domains and downstream tasks.

On the other hand, knowledge graphs may present an incomplete view of the world. Relations can evolve and change over time, be subject to errors, processing limitations, and gaps in available information. Therefore, the tasks of restoring missing links or predicting the new ones are particularly important to maintain the accuracy and relevance of knowledge graphs. Algorithms designed to address this challenge can significantly reduce the manual labor involved in updating information and ensure consistent representation of the most current and up-to-date knowledge. In this work, we aim to design a new efficient graph completion algorithm that will capture inherent structural properties of the underlying data to improve the link prediction quality.

The key challenge of this task lies in the choice of proper architectural components of the solution suitable for handling a non-trivial nature of data. We start by noticing that knowledge graphs exhibit a strictly non-uniform arrangement of interconnections. Some nodes on the graph are present more frequently than others, which renders a hierarchical structure often reproducing the features of a power law-like distribution. It places knowledge graphs into the so called “complex network” category. In the seminal work on the theoretical foundations of complex networks analysis (Krioukov et al., 2010), it was shown that hyperbolic geometries are especially suitable for modelling this type of structures. In these geometries, as one moves further away from the origin, the distances between points and the area grow exponentially in contrast to the Euclidean geometry with its respective linear and quadratic growth. Due to this effect, hyperbolic embeddings have a higher expressive ability in terms of capturing hierarchical relations within data. Consequent studies have proved the practicality of such geometric approach in various domains such as natural language processing (Nickel and Kiela, 2017), computer vision (Ermolov et al., 2022) and recommender systems (Mirvakhabova et al., 2020).

Despite the promising results, in our experiments, we observe that real-world knowledge graphs may not consistently align with the assumption of strict hierarchical internal structure and may only partially follow a power-law distribution (see Figure  1). More specifically, the characteristics of relations in a knowledge graph or even data preprocessing may render connectivity components consisting of vertices with a large number of links almost uniformly distributed between them. We will call these vertices “active”. As we demonstrate in our work, having a large number of active vertices violates hierarchy and disrupts the total compatibility of hyperbolic geometry with real data (see Figure  2). Therefore, in this paper, we aim to find a balanced solution using different geometry types so that the weakness of each geometry is compensated. We introduce a hybrid mixed geometry model that combines the strengths of both Euclidean and hyperbolic geometry within a single architecture. In this model, the Euclidean component serves the goal of properly extracting information from the active vertices, while the hyperbolic component with embeddings from a hyperbolic space captures additional information from vertices in the graph that are distributed according to a power-law.

Overall, our contributions can be summarized as follows:

• We introduce a new mixed-geometry tensor factorisation (MIG-TF) model that combines Tucker decomposition defined in the Euclidean space with a new low-parametric hyperbolic ternary interaction term.

• We highlight intricacies of applying geometric approach to real-world knowledge graphs and demonstrate the associated with it limitations of using single-geometry modelling.

• The proposed combined approach significantly reduces the number of model parameters compared to state-of-the-art methods. It does so without sacrificing expressive power and achieves more accurate results in most of the common benchmarks.

Figure 1: Links distribution on three benchmark knowledge graphs considered in this work. The FB15k-237 dataset violates the power law the most, which is indicated by the longest platue in the leftmost part of the curve. Correspondingly, the top-performing model on this dataset among previous state-of-the-art turns out to be Euclidean. In contrast, our MIG-TF approach outperforms both Euclidean and hyperbolic models, see Table 3.

The rest of the paper is organized as follows. In Section 2, we review most relevant Euclidean and hyperbolic models. We introduce our parameter-efficient hyperbolic interaction model tetrahedron pooling tensor factorization (TPTF) along with the basics of hyperbolic geometry in Section 3.2. Section 3.3 is devoted to mixed geometry modelling MIG-TF, where we present the mixture of the Euclidean TuckER and the hyperbolic TPTF models. Finally, Section  4 contains the results of numerical experiments, where we showcase the performance of our model and also investigate the impact of using a hyperbolic correction. The anonymized code for reproducing our results is publicly available.

Figure 2: The graphs at the top row demonstrate the top 20 vertices in terms of their number of connections in the FB15k-237 knowledge graph. The pink marks indicate vertices that were predicted in more than $50\%$ of cases by both the Euclidean (left), hyperbolic (center) and mixed geometry (right) models. The bar charts illustrate the hit rate (HR@10) of the predictions for links between vertices from each group. Vertices are arranged in descending order based on the number of incoming links. As can be observed, due to the high number of relations among active vertices, the hyperbolic model (TPTF ) significantly underperforms in predicting relations between active vertices compared to the Euclidean model(TuckER). However, the combination of Euclidean and hyperbolic models outperforms both of them.

2 RELATED WORK

Knowledge graphs represent facts in the triplet form $(e_{s},r,e_{o})$, where $e_{s}$ and $e_{o}$ denote a pair of subject and objects entities and $r$ denotes the corresponding relation between them. This representation can be naturally expressed in the third-order binary tensor form where a non-zero entry at position $(i,j,k)$ encodes an existing $(e_{i},r_{k},e_{j})$ triplet of the knowledge graph. To identify connections in knowledge graphs that have been incorrectly specified in the interaction tensor, various tensor factorization-based models are utilized.

For instance, the SimplIE (Kazemi and Poole, 2018) and ComplEX(Trouillon et al., 2016) models employ Canonical decomposition ($CP$) (Hitchcock, 1927) with embeddings in the real or complex Euclidean space. The PITF (Rendle and Schmidt-Thieme, 2010) model utilizes Pairwise Interaction Tensor Factorisation in the Euclidean space, which splits the standard ternary interaction term of $CP$ into a linear sum of independent interactions between entity-entity and entity-relation pairs. Some other models use distances between embeddings of entities and relations in the vector space. For instance, TransE (Bordes et al., 2013b) uses Euclidean vector representations of relations, which are used as connection vectors between Euclidean embeddings of entities. The further development of this approach named Quate (Zhang et al., 2019) represents embeddings as rotations in the hypercomplex space.

Various artificial neural networks-based approaches were also proposed to solve the link prediction task. For instance, the models like ConvE and ConvKB (Dettmers et al., 2018; Nguyen et al., 2017) utilize convolutional neural networks to work with knowledge graphs. The R-GCN (Schlichtkrull et al., 2018) model proposes a modification of graph-convolutional neural networks GCN (Kipf and Welling, 2016) to adapt graphs convolutions to relational data. Models, such as GAAT (Wang et al., 2019b) or KBGAT (Nathani et al., 2019), use the attention mechanisms on graphs.

Some models rely on the shared-factors paradigm, which utilizes a special structure of the problem based on the fact that objects and subjects belong to the same space. While making the problem inherently non-linear, this approach significantly reduces the number of learned parameters of models and is better suited for predicting symmetric links (Hospedales, 2019) in knowledge graphs. For instance, the MEKER (Chekalina et al., 2022)) model employs CP decomposition with low-rank shared factors, and RESCAL (Nickel et al., 2011) utilizes DEDICOM (Harshman et al., 1982) factorisation decomposing each tensor slice along the relation axis. The TuckER (Hospedales, 2019) and R-TuckER (Peshekhonov et al., 2024) models employ shared-factors TuckER decomposition (Tucker, 1966) to represent knowledge graph entities and relations and supports the prediction of symmetric as well as asymmetric links in knowledge graphs.

Recent advances in understanding of hyperbolic geometry applications has lead to the development of hyperbolic models for knowledge graph completion as well. According to (Krioukov et al., 2010), the structural properties of hyperbolic spaces make them plausible for analyzing a special family of non-uniformly distributed data, which knowledge graphs also belong to. For instance, the RotH and RefH (Re, 2020) propose specific rotation and reflection transformations in a hyperbolic space to improve the expressive power of the learned embeddings. Even though these model show remarkable improvements on some knowledge graphs, there are still some cases where Euclidean models outperform hyperbolic ones. We attribute this observation to the violation of some basic distributional properties required for full compatibility with hyperbolic geometry. It motivates us to develop a hybrid approach similarly to (Gu et al., 2019; Seth and Sharaff, 2022; Kong, 2019) that takes the best from both geometries and therefore compensates for their shortcomings in analyzing real-world data. Similar idea was utilized in models M²GNN (Wang et al., 2021) generalizing graph neural network for multi-curvature and GIE (Cao et al., 2022) introducing message passing and attention algorithms on hyperbolic and hyperspherical spaces. In comparison to previous methods, our MIG-TF approach uses less complex and more stable operations (Section 3.4). Additionally, it implies learning only low-parametric hyperbolic addition to a pretrained model instead of learning the whole model as in RotH, M²GNN and GIE. Further in the text, we provide the detailed description of our approach.

3 MIXED GEOMETRY TENSOR FACTORIZATION

In this section, we introduce our new parameter-efficient hyperbolic interaction term TPTF, and the new mixed geometry model MIG-TF, where the hyperbolic term is used as a correction to the Euclidean model.

3.1 Hyperbolic geometry and Lorentz Distance

We start with basic notation and concepts from hyperbolic geometry. There are several models of hyperbolic geometry, among which Poincare Ball, Klein, and Lorentz model are frequently used in various machine learning approaches. All hyperbolic models are isomorphic – they describe the same geometry and can be interchangeably utilized by means of the corresponding transformations (Ungar, 2005). In our work, we use the Lorentz model of hyperbolic geometry. In the following, we only introduce the basics of Lorentz geometry and refer the reader to (Beem, 2017) for more details.

Let us define the Lorentz inner product for vectors $x,y\in\mathbb{R}^{n+1}$, which is used for measuring distance between hyperbolic embeddings:

$$\langle x,y\rangle_{\mathcal{L}}=-x_{0}y_{0}+\sum_{i=1}^{n}x_{i}y_{i},$$

where

$$x_{0}=\sqrt{\beta+\sum_{i=1}^{n}x_{i}^{2}}.$$

The induced norm of a vector $x\in\mathbb{R}^{n+1}$ in Lorentz geometry is defined as $\|x\|_{\mathcal{L}}^{2}=\langle x,x\rangle_{\mathcal{L}}=-\beta$. The corresponding $n$-dimensional space $\mathcal{H}^{n,\beta}\subset\mathbb{R}^{n+1}$ is called Hyperboloid and defined as follows:

$$\mathcal{H}^{n,\beta}=\left\{x\in\mathbb{R}^{n+1}\,\big{|}\ \|x\|_{\mathcal{L}}^{2}=-\beta,\ \beta\geq 0\right\}.$$

The origin vector of the hyperboloid $\mathcal{H}^{n,\beta}$ equals to $\textbf{0}=(\beta,0,...,0)\in\mathbb{R}^{n+1}$. The inner product of 0 and $x$ is, hence, $\langle\textbf{0},x\rangle_{\mathcal{L}}=-\beta x_{0}$.

The associated geodesic distance is defined as

$$d_{l}(x,y)=arccosh(-\langle x,y\rangle_{\mathcal{L}}).$$

Similarly to (Xu and Wu, 2020), we introduce the square Lorentz distance between $x,y\in\mathcal{H}^{n}$ is defined as

$$d^{2}_{\mathcal{L}}(x,y)=\|x-y\|_{\mathcal{L}}^{2}=-2-2\langle x,y\rangle_{\mathcal{L}}.$$ (1)

This distance fulfills all of the Euclidean distance axioms, except for the triangle inequality:

$$d(x,z)\leq d(x,y)+d(y,z).$$ (2)

Nevertheless, it does not hold in general in the hyperbolic setting. The idea to benefit from this fact appears, for instance, in a two-dimensional hyperbolic model LorentzFM (Xu and Wu, 2020). Inequality violation in this model implies that two entities are far away and, hence, become irrelevant to each other. Generalizing the triangle inequality to the three-dimensional case is not a straightforward task, which we address in the following section.

3.2 Tetrahedron Pooling Tensor Factorization

Solving the knowledge graph link prediction problem requires learning ternary interactions in the form of entity-relation-entity triplets. This can be naturally modelled through tensor representation in the Euclidean space. However, unlike the 2D case, generalizing a representation of such third-order interactions to the hyperbolic space is not straightforward. To the best of our knowledge, there is no unified approach.

We propose to modify (2) to capture ternary interactions. In particular, we utilize the so-called tetrahedron inequality (Itoh et al., 2021): for the points $u,v,t,o$ in the Euclidean space, it holds

$$d(u,v)+d(o,t)\leq d(u,t)+d(v,t)+d(o,u)+d(o,v).$$ (3)

In the hyperbolic context, this inequality does not universally apply. However, as we explore further, the regions where this inequality is violated can be advantageous for our objectives. Consequently, we can naturally introduce the following score function:

$$S_{G}= d_{l}(u,v)+d_{l}(\textbf{0},t)-d_{l}(u,t)$$ (4)

$$-d_{l}(v,t)-d_{l}(\textbf{0},u)-d_{l}(\textbf{0},v),$$

where $u,v\in\mathbb{R}^{d_{l}}$ are entity embeddings and $t\in\mathbb{R}^{d_{l}}$ is the relation embedding. Note that this score function is negative when inequality (3) holds and is positive in the opposite case. Unfortunately, the score function (4) involves arccosh, which is not differentiable everywhere and, hence, may lead to difficulties for gradient-based optimization methods.

Similarly to LorentzFM, (4) is replaced with the following “smoothed” score function that mimics the desired behaviour:

$$S_{H}(u,v,t)=$$ (5)

$$\frac{\dfrac{1}{2}\left(\begin{array}[]{c}d_{\mathcal{L}}^{2}(u,v)+d_{\mathcal{L}}^{2}(\textbf{0},t)-d_{\mathcal{L}}^{2}(u,t)\\ \ -d_{\mathcal{L}}^{2}(t,v)-d_{\mathcal{L}}^{2}(\textbf{0},u)-d_{\mathcal{L}}^{2}(\textbf{0},v)\end{array}\right)}{\langle\textbf{0},v\rangle_{\mathcal{L}}\langle\textbf{0},t\rangle_{\mathcal{L}}+\langle\textbf{0},u\rangle_{\mathcal{L}}\langle\textbf{0},v\rangle_{\mathcal{L}}+\langle\textbf{0},t\rangle_{\mathcal{L}}\langle\textbf{0},v\rangle_{\mathcal{L}}},$$

see Figure 3 for comparison of the two score functions.

Note also that we may rewrite $S_{H}$ as:

$$S_{H}=\frac{(2+t_{0}-u_{0}-v_{0})-(\langle u,v\rangle_{\mathcal{L}}-\langle u,t\rangle_{\mathcal{L}}-\langle t,v\rangle_{\mathcal{L}})}{u_{0}v_{0}+u_{0}t_{0}+t_{0}v_{0}}.$$

Figure 3: Left score function landscape corresponds to our score function (5), whilst right one corresponds to (4). As seen from the plots the landscapes of score functions for different distances are similar. For more score function landscapes see Appendix.

The score function yields a positive value when the points corresponding to linked entities are in close proximity and a negative value when they are distant (see Appendix). Additionally, we note that the terms $d_{\mathcal{L}}^{2}(u,t)$ and $d_{\mathcal{L}}^{2}(t,v)$, which account for interactions between relations and entities, effectively draw entities $u$ and $v$ closer to the relation $t$. Consequently, the trained model exhibits a distinct behavior: entities tend to cluster around well-separated relations, as illustrated in Figure 4.

The model’s predictions $p$ for entity $u$ and relation $t$ can be obtained with $p=\{p_{j}\}_{j=1}^{n_{e}}$:

$$p=\sigma(S_{H}(u,V,t)),$$

where $V\in\mathbb{R}^{d_{l}\times n_{e}}$ is the matrix of all entity embeddings and $n_{e}$ – the number of entities. In TPTF model, we use the binary cross-entropy (BCE) loss function:

$$l_{BCE}(y_{i},p_{j})=-y_{i}\log(p_{j})-(1-y_{i})\log(1-p_{j}),$$

where $y=\{y_{i}\}_{i=1}^{n_{e}}$ – binary label vector of the ground truth. Similarly to (Law et al., 2019), we optimize $u$, $v$ and $t$ embeddings in the Euclidean space with the AdamW (Loshchilov and Hutter, 2019) optimizer and then map them to the hyperboloid as follows: $v\in\mathbb{R}^{n}$ maps to $[\sqrt{\beta+\|v\|_{2}^{2}},v]\in\mathbb{R}^{n+1}$.

Figure 4: Each embedding of a relation ($t_{1},...,t_{n}$) defines a cone that encompasses the embeddings of the entities associated with that relation.

3.3 Mixed Geometry Tensor Factorisation

To improve the link prediction in knowledge graphs we introduce a shared-factor mixed geometry model combining Euclidean and hyperbolic models (see Figure 5). Due to the complexity of the knowledge graph, some vertices are better described with hyperbolic embeddings whilst others are with the Euclidean ones. Therefore with a combination of hyperbolic and Euclidean embeddings, significantly more information on the structure of the knowledge graph is saved with Euclidean and hyperbolic vector representations together. For the hyperbolic model, we used TPTF model (5), and for Euclidean one we used TuckER (Hospedales, 2019). The score function of the Euclidean model is defined as:

$$S_{E}(G,u,v,t)=\sum_{\alpha,\beta,\gamma=1}^{d_{1},d_{1},d_{2}}G_{\alpha\beta\gamma}u_{\alpha}v_{\beta}t_{\gamma},$$ (6)

where $u,v\in\mathbb{R}^{d_{1}}$ – embeddings of entities, $t\in\mathbb{R}^{d_{2}}$ – embedding of relations and $G\in\mathbb{R}^{d_{1}\times d_{1}\times d_{2}}$ – tensor of learnable parameters. Here, $d_{1}$ and $d_{2}$ are the ranks of the Tucker decomposition.

The score function of our mixed-geometry model MIG-TF is the sum of score functions of the Lorentzian and Euclidean models:

$$(S_{\textit{MIG-TF}})_{i}=(S_{E})_{i}+(S_{H})_{i},$$ (7)

where $(S_{E})_{i}=S_{E}(G,u,V_{i},t)$ defined in (6) is an Euclidean interaction term, $(S_{H})_{i}=S_{H}(u^{\mathcal{L}},V_{i}^{\mathcal{L}},t^{\mathcal{L}})$ is a hyperbolic interaction term (5). The vectors $u$ and $t$ Euclidean entity and relation embeddings, $u^{\mathcal{L}}$ and $t^{\mathcal{L}}$ – hyperbolic embeddings, $V_{i}$ and $V_{i}^{\mathcal{L}}$ are $i$-th rows of matrices $V$ and $V^{\mathcal{L}}$ of $n_{e}$ Euclidean and hyperbolic entity embeddings, respectively, and $G\in\mathbb{R}^{d_{1}\times d_{1}\times d_{2}}$ – 3-D tensor of learnable parameters. In MIG-TF model, we utilize pretrained TuckER model (6) and optimize the hyperbolic term parameters (5) of the score function (7) to minimize the BCE loss:

$$\mathcal{L}_{\textit{MIG-TF}}=\frac{1}{n_{e}}\sum_{i=1}^{n_{e}}l_{BCE}(a_{i},\,\sigma((S_{\textit{MIG-TF}})_{i})),$$

where $a_{i}$ is an $i$-th element of interaction vector $a$ for entity and relation with embeddings $u$ and $t$, respectively. We optimize hyperbolic embeddings as in Section 3.2. To prevent model overfitting, we employ the dropout layer (Salakhutdinov, 2012) in both Euclidean and hyperbolic interaction terms.

Figure 5: The proposed MIG-TF model architecture.

3.4 Mixed-geometry models comparison

The next subsection is devoted to comparing our approach with other mixed-geometry methods: M²GNN (Wang et al., 2021) and GIE (Cao et al., 2022).

The translational distance (Zhang et al., 2020) model M²GNN generalizes graph neural networks for multi-relational setting on knowledge graphs and modified it for multi-curvature setting capturing graph structures more effectively, especially hierarchical and cyclic ones.

The model GIE introduces the message passing and attention mechanism for the knowledge graph completion by utilizing mapping on hyperbolic and hyperspherical spaces.

Both of the existing approaches suffers from computational complexity and numerical instability due to utilizing exponential and logarithmic mapping, which are even more numerically unstable than the standard Lorentz model operations (Peng et al., 2021). In contrast, MIG-TF proposes using Square Lorentz distance, a convenient replacement for Geodesic distance. Square Lorentz distance reduces hyperbolic operations, such as exponential and logarithmic mappings and arccosh, to mere inner product computations, decreases complexity of operations and increases numerical stability.

Additionally, only the low-parametric addition TPTF is learned in our MIG-TF method instead of learning the whole models in M²GNN and GIE.

Unlike (Wang et al., 2021) and (Cao et al., 2022), we do not utilize spherical geometry and still our mixed geometry approach achieves the best metrics in the majority of cases. Nevertheless, our framework is sufficiently flexible to accommodate the incorporation of spherical geometry, presenting a promising direction for future research.

4 EXPERIMENTS

In experiments, we shows that mixed geometry model, combining Euclidean and hyperbolic embeddings, with low-dimensional hyperbolic ones outperforms Euclidean, Complex and hyperbolic models with significantly higher dimensions of embeddings. Additionally, we have studied the dependence of curvature of the space of hyperbolic embeddings and expressive ability of hyperbolic and mixed geometry models. Moreover, we have shown that hyperbolic addition to TuckER (Hospedales, 2019) model improves the quality of link prediction between hierarchically distributed entities.

4.1 Datasets

We evaluate our mixed geometry model for link prediction task on three standard knowledge graphs: WN18RR (Bordes et al., 2013b; Dettmers et al., 2018), FB15k-237 (Bordes et al., 2013b; Toutanova et al., 2015) and YAGO3-10 (Mahdisoltani et al., 2013). WN18RR is a subset of WordNet (Miller, 1995) – knowledge base containing lexical relations between words. FB15k-237 is a subset of FreeBase (Bollacker et al., 2008) – collaborative knowledge base containing general world knowledge, such as nationalities of famous people, locations of some buildings or genre of music. YAGO3-10 is a subset of YAGO3 (Suchanek et al., 2007) which mostly contains descriptions o people. The statistics of these knowledge graphs are shown in following table:

Table 1: Knowledge graphs statistics.

$\mathsf{Dataset}$	$\mathsf{Entities}$	$\mathsf{Relations}$	$\mathsf{Triplets}$
$WN18RR$	$40,943$	$11$	$93,003$
$FB15k\text{-}237$	$14,541$	$237$	$310,116$
$YAGO3\text{-}10$	$123,182$	$37$	$1,179,040$

For each knowledge graph we follow standard data augmentation by adding inverse relations (Bordes et al., 2013a).

4.2 Baselines

We compare our models with $8$ baselines: $4$ models with euclidean embeddings, $2$ with complex embeddings and $2$ with hyperbolic embeddings:

• Euclidean models: PITF (Pairwise Interaction Tensor Factorisation) – model using PITF tensor decomposition and Euclidean embeddings (Rendle and Schmidt-Thieme, 2010). DistMult (Distance-based Multiplication) – model utilizing canonical tensor decomposition (Yang et al., 2014).

• TuckER-based Euclidean models: TuckER (Tucker decomposition) – model using Tucker tensor decomposition and Euclidean embeddings (Hospedales, 2019). RTuckER – TuckER model with a Riemanian method of optimization (Peshekhonov et al., 2024).

• Complex models: ComplEx-N3 (Complex Embeddings) – model using real part of the canonical tensor decomposition and complex embeddings (Chen et al., 2021). RotatE (Relational Rotation in complex space) – model using rotations in complex space and complex embeddings (Tang, 2019).

• Hyperbolic models: RotH (Hyperbolic Rotations) and RefH (Hyperbolic Reflections) – attention models using rotations and reflections in hyperbolic space and utilizing hyperbolic embeddings from Poincare Ball (Re, 2020).

• Mixed-geometry models: M²GNN (Mixed-Curvature Multi-Relational Graph Neural Network) – model using multi-curvature embeddings for translational distance and multi-relational graph neural networks. GIE (Geometry Interaction Knowledge Graph Embeddings) – model utilizing attention layers and embeddings from Euclidean, hyperbolic and spherical spaces.

To demonstrate the benefits of our mixed geometry approach to link prediction task we compare baselines with our hyperbolic model TPTF (5) and our mixed geometry models MIG-TF (7) and MIG-TF_QR. Here, MIG-TF_QR is mixed geometry model with orthogonalization heuristic. For more details see Appendix. Additionaly, to better understand the role of hyperbolic interaction term we compare TPTF and MIG-TF models with different fixed hyperbolicities.

4.3 Evaluation metrics

At the test mode, we rank correct entities against all entities using score fictions (5) for TPTF and (7) for MIG-TF. We compare models using two rank-base metrics: $MRR$ (Mean Reciprocal Rank) and $HR@k,$ $k\in{1,3,10}$ (Hit Rates). $MRR$ measures mean inverse rank of correct entity, whilst $HR@k$ measures the share of correct triples among $Top-k$ recommendations (Vasilev, 2021). We follow the standard evaluation protocol in the filtered setting: all true triples in the KG are filtered out during evaluation (Lacroix et al., 2018).

5 RESULTS

Our mixed geometry models achieved new state-of-the-art results on FB15k-237, YAGO3-10 and WN18RR in the majority of metrics. Moreover, our models have significantly less parameters then the best models on WN18RR and YAGO3-10. For instance, our models have $8$ times less parameters than RotH or RefH. Additionally, as we expected, Lorentz addition to the TuckER model significantly increases $MRR$ and $HR@k$ metrics, with low - Lorentzian embeddings. We also show, that MIG-TF model is better that TuckER model in prediction of various relations, especially rare ones Figure 6.

Table 2: Approximate number of models’ parameters on knowledge graphs $WN18RR$, $FB15k-237$ and $YAGO3-10$.

Models	$\mathsf{FB15k\text{-}237}$	$\mathsf{WN18RR}$	$\mathsf{YAGO3\text{-}10}$
$PITF$	$4\cdot 10^{6}$	$8\cdot 10^{6}$	$25\cdot 10^{6}$
$DistMult$	$4\cdot 10^{6}$	$8\cdot 10^{6}$	$25\cdot 10^{6}$
$TuckER_{S_{E}+0\cdot S_{H}}$	$4\cdot 10^{6}$	$8\cdot 10^{6}$	$25\cdot 10^{6}$
$RTuckER_{S_{E}+0\cdot S_{H}}$	$4\cdot 10^{6}$	$8\cdot 10^{6}$	$25\cdot 10^{6}$
$ComplEx\text{-}N3$	$4\cdot 10^{6}$	$8\cdot 10^{6}$	$25\cdot 10^{6}$
$RotatE$	$60\cdot 10^{6}$	$60\cdot 10^{6}$	$120\cdot 10^{6}$
$RotH$	$40\cdot 10^{6}$	$80\cdot 10^{6}$	$120\cdot 10^{6}$
$RefH$	$40\cdot 10^{6}$	$80\cdot 10^{6}$	$120\cdot 10^{6}$
$M^{2}GNN$	$12\cdot 10^{6}$	$24\cdot 10^{6}$	$75\cdot 10^{6}$
$GIE$	$4\cdot 10^{6}$	$18\cdot 10^{6}$	$24\cdot 10^{6}$
Our models
$TPTF_{0\cdot S_{E}+S_{H}}$	$2\cdot 10^{6}$	$4\cdot 10^{6}$	$12\cdot 10^{6}$
$MIG\text{-}TF_{S_{E}+S_{H}}$	$5\cdot 10^{6}$	$10\cdot 10^{6}$	$31\cdot 10^{6}$

Figure 6: The top bar chart demonstrates the cumulative $HR@10$ metrics in the $WN18RR$ knowledge graph with respect to all types of relations ordered by their occurrence frequency. It is seen that Mixed Geometry model (red) has higher cumulative metrics in comparison to TuckER (blue). In addition, Mixed Geometry has significant increase in $HR@10$ metric on rare relations. For instance, on “member of domain region” $+140\%$, “member of domain usage” $+48\%$ and “member meronym” $+36\%$. The bottom bar chart illustrates the number of links with each relation in knowledge graph.

Table 3: Metrics on knowledge graphs: $HR@k$ and $MRR$ metrics of models on knowledge graphs WN18RR, FB12k-237 and YAGO3-10. Bold means the best metric on the knowledge graph and underlined – the second best metric. For each knowledge graph we follow standard data augmentation by adding inverse relations (Bordes et al., 2013a).

	$\mathsf{FB15k\text{-}237}$				$\mathsf{WN18RR}$				$\mathsf{YAGO3\text{-}10}$
Models	$\mathsf{HR@10}$	$\mathsf{HR@3}$	$\mathsf{HR@1}$	$\mathsf{MRR}$	$\mathsf{HR@10}$	$\mathsf{HR@3}$	$\mathsf{HR@1}$	$\mathsf{MRR}$	$\mathsf{HR@10}$	$\mathsf{HR@3}$	$\mathsf{HR@1}$	$\mathsf{MRR}$
$PITF$	0.459	0.347	0.186	0.284	0.403	0.384	0.392	0.296	0.610	0.494	0.350	0.453
$DistMult$	0.419	0.263	0.155	0.241	0.490	0.440	0.390	0.430	0.540	0.380	0.240	0.340
$TuckER_{S_{E}+0\cdot S_{H}}$	0.544	0.394	0.266	0.358	0.526	0.482	0.443	0.470	0.681	0.579	0.466	0.544
$RTuckER_{S_{E}+0\cdot S_{H}}$	0.505	0.387	0.237	0.326	0.546	0.495	0.446	0.479	–	–	–	–
$ComplEx\text{-}N3$	0.547	0.392	0.264	0.357	0.572	0.495	0.435	0.480	0.701	0.609	0.498	0.569
$RotatE$	0.533	0.375	0.245	0.338	0.571	0.492	0.428	0.476	0.670	0.550	0.402	0.495
$RotH$	0.535	0.380	0.246	0.344	0.586	0.514	0.449	0.496	0.706	0.612	0.495	0.573
$RefH$	0.536	0.383	0.252	0.346	0.568	0.485	0.404	0.461	0.711	0.619	0.502	0.576
$M^{2}GNN$	0.565	0.398	0.275	0.362	0.572	0.498	0.444	0.485	0.702	0.605	0.478	0.573
$GIE$	0.552	0.401	0.271	0.362	0.575	0.505	0.452	0.491	0.709	0.618	0.505	0.579
Our models
$TPTF_{0\cdot S_{E}+S_{H}}$	0.504	0.321	0.186	0.238	0.489	0.378	0.252	0.314	0.665	0.521	0.383	0.481
$MIG\text{-}TF_{S_{E}+S_{H}}$	0.554	0.402	0.277	0.367	0.561	0.507	0.450	0.496	0.717	0.621	0.502	0.579
$MIG\text{-}TF_{QR,S_{E}+S_{H}}$	0.553	0.402	0.277	0.366	0.574	0.514	0.452	0.499	0.720	0.622	0.502	0.580

5.1 Hyperbolicity

To study the impact of hyperbolic interaction term in hybrid model we trained TPTF and MIG-TF models with various fixed curvatures. Models with low curvature are similar to euclidean ones, therefore they worse predict links between hierarchically distributed entities of knowledge graph. In contrast, models with curvature $c\approx 1.5$ show the highest metrics both in mixed geometry and hyperbolic cases Figure 7.

Euclidean component in MIG-TF model embeds features of knowledge graph structure, whilst hyperbolic ones are not fully represented with embeddings. Therefore, the hyperbolic component of mixed geometry model represents only the hyperbolic features of knowledge graph that left after Euclidean component application (Ni et al., 2015; Gu et al., 2019). In contrast, RotH and RefH architectures (Re, 2020), are based on rotations and reflections in hyperbolic and the Euclidean space, respectively. In contrast, our model utilizes the hyperbolic triangle inequality, as well as uses embeddings in both Euclidean and hyperbolic spaces. Additionally, our model has less stringent requirements on the structure of the input data.

5.2 Link prediction quality

As demonstrated in Table 3, our mixed geometry approach with mixed geometries outperforms all other competitors on the two out of three datasets. It slightly underperforms its main competitor (the hyperbolic RotH model) on the $WN18RR$ dataset on one metric. We attribute this result to the fact that the structure of this dataset is the most plausible for analysis with hyperbolic geometry. Indeed, by inspecting Figure 1, we see that this dataset is the most compatible with the ideal power law distribution, it contains the minimal amount of active nodes and exhibits almost linear dependency over the majority of entities. This result aligns well with our initial design that is targeted towards mixed hierarchical and non-hierarchical structures and therefore is may not take full advantage of more homogeneous data organization. A possible remedy to this effect would be changing Euclidean component of MIG-TF model to more powerful model, such as Euclidean RotE (Re, 2020) or complex ComplEx-N3 (Chen et al., 2021). We leave investigation of this approach for future work.

On the other two datasets our approach dominates its competitors, which aligns well with our assumptions on the structural complexities of these datasets that prevent purely hyperbolic or purely Euclidean model to learn the best representations. Our approach not only outperforms other models but also requires a lesser amount of parameters, which indicates the higher representation capacity of the mixed geometry embeddings.

Figure 7: The graph presents the dependence of $HR@10$ metric from curvature of Hyperbolic space $\beta$ for MIG-TF model on WN18RR dataset.

By consulting to Figure 1, one can see that the most prominent results are obtained on the $FB15k-237$ dataset exhibiting the highest degree of violation of the power law distribution. Remarkably, the otherwise top-performing hyperbolic models underperform the best Euclidean model on this dataset. This signifies the detrimental effect of the structural non-homogeneity on single-geometry models and highlight the advantages of our mixed geometry approach.

6 CONCLUSION

The use of the proposed mixed geometry model MIG-TF combining the Euclidean and hyperbolic embedding techniques can significantly improve the quality of link predictions in knowledge graphs with non-homogeneous structure. It provides a significant advantage over state-of-the-art single-geometry models, such as the Euclidean TuckER and the hyperbolic RotH and RefH. This effect can be attributed to the fact that both Euclidean and Hyperbolic embeddings capture different aspects of relationships between entities in a knowledge graph. By combining these two approaches, mixed geometry models can capture a broader range of relationships, leading to more accurate predictions. We also show that hybrid models are particularly well-suited for addressing the challenge of link prediction in knowledge graphs under limited computing resources. Due to the geometry-driven improved representation capacity, our proposed approach leads to fewer parameters compared to complex or attention-based models such as RotatE and RotH. Additionally, our model requires training only the minority of its parameter in comparison to the other mixed geometry models such as GIE and M²GNN. This facts make MIG-TF approach more efficient and scalable.

Acknowledgments

The work of Viacheslav Yusupov and Maxim Rakhuba was prepared within the framework of the HSE University Basic Research Program. The calculations were performed in part through the computational resources of HPC facilities at HSE University (Kostenetskiy et al., 2021).

7 SQUARE LORENTZ DISTANCE VS. GEODESIC DISTANCE

In this section, we illustrate that the score function based on the utilized Lorentz distance behaves similarly to the geodesic distance for different $t$. In particular, in Figure 8, we present the two-dimensional landscapes of the TPTF score function for $t\in\{-10,0,10\}$.

Figure 8: The landscapes of the TPTF score function for the Lorentz and geodesic distances. As is seen in the plots, the normalized score function with the Lorentz distance mimics the behaviour of ones with geodesic distance.

8 COMPLEXITY

In this section, we estimate the complexity of our MIG-TF model.

Let us introduce the operation $G\times_{i}V$ for a tensor $G\in\mathbb{R}^{d_{1}\times d_{2}\times...\times d_{n}}$ and matrix $V\in\mathbb{R}^{d_{i}\times m}$

$$(G\times_{i}V)_{a_{1},...,a_{i-1},k,a_{i+1},...,a_{n}}=\sum_{a_{i}=1}^{d_{i}}G_{a_{1},a_{2},...,a_{n}}V_{a_{i},k}.$$

Let $n_{e}$ and $n_{r}$ be respectively the numbers of entities and relations in a knowledge graph, $b$ – a batch size, $d_{e}$ and $d_{r}$ – embedding sizes of the TuckER model, $d_{h}$ – the size of hyperbolic embeddings. Let the core of the Tucker decomposition be $G\in\mathbb{R}^{d_{e}\times d_{e}\times d_{r}}$, the matrices of embeddings – the entities embedding matrix $U\in\mathbb{R}^{b\times d_{e}}$, the relation embedding matrix $T\in\mathbb{R}^{b\times d_{r}}$ and another entity embedding matrix $V\in\mathbb{R}^{n_{e}\times d_{e}}$. Similarly, the matrices in the hyperbolic model are $U^{\mathcal{L}}\in\mathbb{R}^{b\times d_{h}}$, $T^{\mathcal{L}}\in\mathbb{R}^{b\times d_{h}}$ and $V^{\mathcal{L}}\in\mathbb{R}^{n_{e}\times d_{h}}$.

Firstly, we find the complexity of the TuckER model. To compute the TuckER score function, we firstly find $G_{1}=G\times_{1}U\in\mathbb{R}^{b\times d_{e}\times d_{r}}$. It takes $O(bd_{e}(d_{e}d_{r}))$ operations. Then we compute $G_{2}=G_{1}\times_{3}T\in\mathbb{R}^{b\times d_{e}\times b}$ with $O(bd_{r}(bd_{e}))$ operations. Finally, the diagonal slice of $G_{2}$ is taken and multiplied with $V$ with $O(bd_{e}n_{e})$ operations. Overall, the total number of operations is:

$$O_{\textit{TuckER}}=O(bd_{e}(d_{e}d_{r})+bd_{r}(bd_{e})+bd_{e}n_{e})=O(bd_{e}(d_{e}d_{r}+bd_{e}+n_{e})).$$

Secondly, we find the complexity of the TPTF model. First of all, we compute Lorentz zero element of embeddings $x$ as

$$x_{0}=\sqrt{\beta+\sum_{i=1}^{d}x_{i}^{2}},$$

which is $O(d_{h}(b+n_{e}))$ operations. Than we compute Lorentz inner products and it takes $O(bd_{h}n_{e}+bd_{h}n_{e}+bd_{h}n_{e})=O(bd_{h}n_{e})$. Then we compute the sum of all inner products $S_{H}$ using $O(bn_{e})$ operations. Finally, we compute the normalization $N_{H}$ with $O(bn_{e})$ operations and make elementwise division ${S_{H}}/{N_{H}}$ with the complexity $O(bn_{e})$. Overall, the TPTF complexity is:

$$O_{\textit{TPTF}}=O(d_{h}(b+n_{e})+bd_{h}n_{e}+bn_{e}+bn_{e}+bn_{e})=O(bn_{e}d_{h}).$$

The complexity of MIG-TF iteration is a sum of TuckER and TPTF complexities and is equal to:

$$O_{\textit{MIG-TF}}=O_{\textit{TuckER}}+O_{\textit{TPTF}}+O(bn_{e})=O(bd_{e}(d_{e}d_{r}+bd_{e}+n_{e})+bn_{e}d_{h}).$$

In terms of time and parameter complexity, our MIG-TF model adds only $30\%$ overhead on training time and $25\%$ overhead on inference time in comparison to TuckER.

9 MIXED GEOMETRY TENSOR FACTORIZATION MODIFICATIONS

In this section, we describe in more detail additional techniques that we use for our MIG-TF model.

9.1 Riemannian MIG-TF

To improve the quality of the TuckER (Hospedales, 2019) model we use the so-called RTuckER model (Peshekhonov et al., 2024). This model has the same architecture as TuckER, but was trained with the methods of Riemannian optimisation. This Riemannian model significant outperforms TuckER on some knowledge graphs. For more details we refer to (Peshekhonov et al., 2024). We utilize RTuckER as a Euclidean term of our MIG-TF model only on WN18RR knowledge graph on which the Riemannian model significantly outperforms TuckER.

9.2 MIG-TF with QR decomposition

Figure 9: Correlation maps for relations of knowledge graphs FB15k-237, WN18RR and YAGO3-10. As can be seen the vast majority of relations in knowledge graphs are uncorrelated. Therefore, the orthoganality of relation embeddings is promising as embeddings of uncorrelated relations do not correlate.

As was shown in Figure 9, most of the relations in knowledge graphs have no correlations, therefore maintaining the orthogonality of relation embeddings creates better vector representation. For this purpose, on each iteration we update the hyperbolic relation embedding matrix $T^{\mathcal{L}}$ with the help of the QR decomposition as follows:

$$T^{\mathcal{L}}=Q^{\mathcal{L}}R.$$ (8)

Then we compute the score function using $Q^{\mathcal{L}}$ instead of $T^{\mathcal{L}}$:

$$(S_{\textit{MIG-TF}})_{i}=S_{E}(G,u,V_{i},t)+S_{H}(u^{\mathcal{L}},V_{i}^{\mathcal{L}},q^{\mathcal{L}}).$$

In these formulas, $G$ is the core of Tucker decomposition, $u$ and $u^{\mathcal{L}}$ are the Euclidean and hyperbolic entity embeddings, $V_{i}$ and $V_{i}^{\mathcal{L}}$ are respectively the $i$-th rows of the Euclidean $V$ and of the hyperbolic $V^{\mathcal{L}}$ entity embedding matrices. The vector $t$ is the Euclidean relation embedding and $q^{\mathcal{L}}$ is a row of $Q^{\mathcal{L}}$ from (8). We call this modified MIG-TF as MIG-TF_QR.

The MIG-TF model with QR decomposition significantly outperform MIG-TF on knowledge graphs WN18RR and YAGO3-10 where the condition of non correlation of relations is fulfilled almost everywhere. However, MIG-TF_QR slightly loses to MIG-TF on FB15k-237 graph, where the condition of non correlation is usually violated (see Figure 9).

10 ROBUSTNESS

To analyze our models’ robustness, we add random links in the training data knowledge graph. If $n_{train}$ is the number of links in train data, then we add $\alpha n$ random links to train data. We train the mixed geometry model on poisoned training data and test it on the test data, see Figure 10. As we see, even with high levels of noise ($10\%$), both MIG-TF and $\mbox{MIG-TF}_{QR}$ models do not significantly degrade.

Figure 10: The HR@10 metric on WN18RR knowledge graph depended on the fraction of poison $\alpha$ in training data for $2$ mixed geometry models.

11 THE IMPACT OF EUCLIDEAN AND HYPERBOLIC TERMS

In this section, we analyze impact of different terms in the MIG-TF score function. We represent the score function $S_{\textit{MIG-TF}}$ as:

$$S_{\textit{MIG-TF}}=\mu S_{E}+(1-\mu)S_{H},\quad\mu\in(0,1),$$

where $S_{E}$ and $S_{H}$ are respectively Euclidean and hyperbolic components of the score function and $\mu$ is a parameter. By changing $\mu$, we measure the effect of the score function components on the HR@10 metric on WN18RR knowledge graph, see Figure 11. It is interesting to note that the best quality is achieved with the equal effect of the Euclidean and hyperbolic terms. This justifies our choice of the score function.

Figure 11: HR@10 against the $\mu$ parameter for $2$ mixed geometry models. The maximum of HR@10 metrics on WN18RR is achieved in $\mu=0.5$, where the contributions of Euclidean and hyperbolic terms of score function are equal.

12 OPTIMAL CURVATURE

This section demonstrates dependence of link prediction quality on WN18RR knowledge graph on the curvature in hyperbolic parts of TPTF and MIG-TF models, see Figure 12. Note that the optimal curvature for hyperbolic and mixed geometry models are different from each other.

Figure 12: The top graph presents the dependence of $HR@10$ metric on WN18RR knowledge graph from curvature of Hyperbolic space $\beta$ for TPTF model. The bottom one presents the dependence of metric from curvature for MIG-TF model.

13 HYPERPARAMETERS

In this section, we present the hyperparameters of both the Euclidean term (Table 4) and the hyperbolic term (Table 6) of our MIG-TF model. In this tables, $d_{e},d_{r}$ are the sizes of entity and relation embeddings, $lr$ is a learning rate, $\mathcal{N}(0,\rho_{e}^{2}I),\mathcal{N}(0,\rho_{r}^{2}I)$ – distributions of initial weights of entities and relations embeddings in the TPTF component of the hybrid model and $\beta$ is a curvature of hyperboloid in the Lorentz term of MIG-TF model. All the experiments we performed were done on one GPU NVIDIA V100.

Table 4: Hyperparameters for the TuckER component.

$\mathsf{Hyperparametr}$	$\mathsf{WN18RR}$	$\mathsf{FB15k\text{-}237}$	$\mathsf{YAGO3\text{-}10}$
$d_{e}$	200	200	200
$d_{r}$	30	200	30
$lr$	0.01	0.001	0.003
$epochs$	500	500	500
$Dropout_{1}$	0.2	0.3	0.2
$Dropout_{2}$	0.2	0.4	0.2
$Dropout_{3}$	0.3	0.5	0.2

Table 5: Hyperparameters for the TPTF component.

$\mathsf{Hyperparametr}$	$\mathsf{WN18RR}$	$\mathsf{FB15k\text{-}237}$	$\mathsf{YAGO3\text{-}10}$
$d_{\mathcal{L}}$	50	50	50
$lr$	0.003	0.002	0.005
$\beta$	1.3	1	1.1
$epochs$	250	150	250
$\rho_{e}$	0.005	0.001	0.001
$\rho_{r}$	0.005	0.001	0.001
$Dropout_{e}$	0.2	0.3	0.2
$Dropout_{r}$	0.2	0.3	0.2

Table 6: Hyperparameters for the MIG-TF component.

$\mathsf{Hyperparametr}$	$\mathsf{WN18RR}$	$\mathsf{FB15k\text{-}237}$	$\mathsf{YAGO3\text{-}10}$
$d_{\mathcal{L}}$	50	50	50
$lr$	0.003	0.001	0.005
$\beta$	1.5	1	1.1
$epochs$	250	100	150
$\rho_{e}$	0.005	0.001	0.001
$\rho_{r}$	0.005	0.001	0.001
$Dropout_{e}$	0.2	0.3	0.2
$Dropout_{r}$	0.2	0.3	0.2