Since the discovery of quantum Hall (QH) effect, the study of the topological phase of matter has attracted great attention in the condensed-matter community. A voltage caused by the deflected motion of charged particles under external electric field and magnetic field leads to the quantized Hall conductance ^[1]. Nevertheless, the quantum Hall effect may be achieved without external magnetic field. In 1988, Haldane^[2] proposed the quantum anomalous Hall (QAH) effect in a two-dimensional (2D) honeycomb lattice with next-nearest-neighboring hopping modulated by staggered flux. Dissipative boundary states exist in the two edges of materials with QAH effect^[3-5]. The anomalous Hall effect has an intrinsic origin due to spin-dependent band structure of conduction electrons, which can be expressed in terms of the Berry phase or Chern number in the momentum space^[6]. This effect originates from the coupling of electron orbital motion to its spin, i.e., spin orbit coupling (SOC), resulting in the opposite motion of electrons with spin-up and spin-down. In a ferromagnetic metal, the magnetization causes an imbalance in the population between the electrons with spin-up and spin-down and consequently leads to the anomalous Hall effect. Considering the time reversal (TR) symmetry, the quantities of electrons moving in the opposite directions with different spins are equal and as a result there is no charge Hall conductance, but a nonzero spin Hall conductance exhibits quantum spin Hall (QSH) effect^[7]. The QAH insulator^[8] was a recently discovered topological electronic phase where strong SOC and ferromagnetic ordering conspire to generate a band gap E_g in the bulk of a two-dimensional (2D) electron system, as well as conducting gapless chiral edge states in its boundaries. The edge states are robust against impurities or disorders because the electron backscattering in the two edge channels is prohibited due to the TR symmetry. Topologically nontrivial band structure is characterized by a nonzero Chern number by counting the number of edge states. The QAH effect not only occurs in out-of-plane magnetization systems^[9-15], but also exists in in-plane magnetization materials^[16-21].

The layered transition-metal trichlorides materials of MCl₃ (M=Ti, V, Cr, Fe, Mo, Ru, Rh, Ir) have been achieved for many years due to relatively weak van der Waals interaction between the interlayers ^[22-25]. These layered materials, which possess honeycomb lattice, partially occupied d-orbitals, and nonnegligible SOC, may have interesting properties, and thus they are worth studying.

1 Computational methods

The first-principle calculations were performed within the Vienna ab initio simulation package (VASP) using the projector augmented wave (PAW) method in the framework of density functional theory (DFT)^[26-28]. The electron exchange correlation functional was described by the generalized gradient approximation (GGA) in the form proposed by Perdew, Burke, and Ernzerhof (PBE)^[29]. The lattice parameters are chosen from Springer Materials^[30]. The structure relaxation considering both the atomic positions and lattice vectors was performed by the conjugate gradient (CG) scheme until the maximum force on each atom was less than 0.01eV/Å(1Å=0.1nm), and the total energy converge threshold was 10^-6eV with Gaussian smearing method. To avoid unnecessary interactions between the monolayer and its periodic images, the vacuum layer was set as 20Å. The energy cut off of the plane waves was chosen as 520eV. The Brillouin zone (BZ) integration was sampled with a 9×9×1 G-centered Monk horst Pack grid. The relaxed lattice parameter (a=6.012Å) was adopted in the calculations. SOC is included by a second variational procedure on a fully self-consistent basis. An effective tight binding Hamiltonian based on the maximally localized Wannier functions (MLWF) was used to investigate the surface states^{[21, 31]}. The iterative Greens function method^[32] was used with the package WannierTools^[33].

2 Results and discussion

The two-dimensional VCl₃ (VBr₃) consists of a V atomic layer sandwiched by two Cl atomic layers. The V atoms form a honeycomb lattice and each V atom is surrounded by six Cl atoms, which forms an octahedral crystal field, as shown in Fig. 1. It takes the same structure as monolayer OsCl₃ that has been shown to be an intrinsic quantum anomalous Hall insulator^[34]. V is a transition metal element with partially filled 3d-orbitals, which may result in magnetism. Based on the first-principle calculations, we consider several magnetic configurations, including ferromagnetic (FM), antiferromagnetic (AFM), and paramagnetic (PM) states. We firstly take VCl₃ as an example and find that VCl₃ takes the FM ground state with in-plane magnetization along zigzag direction as shown in Table 1.

The band structure calculations show that VCl₃ is a fully spin-polarized half metal. Without considering SOC, it is a 2D Weyl half semimetal with a band crossing point in each G-M path, such that there are total 6 Weyl points in the first BZ. After turning on SOC, a band gap of about 3.4meV was opened, and the material becomes a ferromagnetic semiconductor, which is different from PtCl₃ which is a 2D Weyl half semimetal even with SOC due to additional symmetry protection^[35]. To understand these features, we find that under the octahedral crystal field formed by Cl atoms, V-3d orbitals are split into t_2gand e_g orbital groups, and the latter is higher in energy. For each V³⁺ with two valence electrons, there are four valence electrons in one primitive cell, such that V-t_2g orbitals will be partially filled in one spin channel. The other t_2g spin channel and all e_g orbitals are empty. Therefore, the system is half Weyl semimetal without considering SOC. If turning on SOC, the t_2g orbitals will split into j=1/2 doublet state and j=3/2 quartet state with the former energetically higher, such that the system becomes a semiconductor, as shown in Fig. 2. The topologically nontrivial band structure of VCl₃ may come from the combination effect of honeycomb lattice formed by V atoms and the SOC of transition metal V atoms.

Furthermore, because of the ferromagnetic ordering of the magnetic moments in V atoms, the system results in a quantum anomalous Hall insulator. To show the topology, we calculated the gauge-invariant Berry curvature in momentum space. The Berry curvature Ω_z(k) in 2D can be obtained by analyzing the Bloch wave functions from the self-consistent potentials

$ {\Omega _z}\left( k \right) = {\sum _n}f_n^z{\Omega _n}\left( k \right), $

(1)

$ \begin{array}{l} \Omega _n^z\left( k \right) = - 2{\sum _{m \ne n}}{\rm{Im}}\\ {\rm{ }}\frac{{\left\langle {{\psi _n}\left( k \right)\left| {{v_x}} \right|{\psi _m}\left( k \right)} \right\rangle \left\langle {{\psi _m}\left( k \right)\left| {{v_y}} \right|{\psi _n}\left( k \right)} \right\rangle }}{{{{\left( {{\varepsilon _n}\left( k \right) - {\varepsilon _m}\left( k \right)} \right)}^2}}}, \end{array} $

(2)

where f_n is the Fermi-Dirac distribution function, v_x(y) is the velocity operator, ψ_n(k) is the Bloch wave function, ε_n(k) is the eigenvalue, the summation is over all n occupied bands below the Fermi level, m indicates the unoccupied bands above the Fermi level. Based on the first-principle calculations, one observes that, in the absence of SOC, there are totally 6 Weyl nodes (Fig. 3(a)) related with C₃ and inversion symmetry along the G-M(M') high symmetry lines of the hexagonal Brillouin zone. SOC will open a tiny gap, correspondingly. Six Berry curvature peaks appear. The nonzero Berry curvature mainly distributes around the opened band crossings at the Fermi level. Furthermore, a plot of the 6 Berry curvature peaks over the whole BZ (Fig. 3(c)) indicates that all 6 peaks have the same sign. The Chern number can be obtained by integrating the Berry curvature Ω_z(k) over the BZ

$ C = \frac{1}{{2{\rm{ \mathsf{ π} }}}}\int {{\rm{d}}{k^2}{\Omega _z}(k)} . $

(3)

Two Berry curvature peaks contribute to the nonzero Chern number 1 and consequently the total Chern number is 3 by integrating the Berry curvature in the whole BZ. According to the bulk-edge correspondence^[36], the nonzero Chern number is closely related to the number of nontrivial chiral edge states that emerge inside the bulk gap of a semi-infinite system. With an effective concept of principle layers, an iterative procedure to calculate the Greens function for a semi-infinite system is employed. The momentum and energy dependence of the local density of states at the edge can be obtained from the imaginary part of the surface Greens function

$ A\left( {k, \omega } \right) = - \frac{1}{{\rm{ \mathsf{ π} }}}\mathop {{\rm{lim}}}\limits_{\eta \to {0^ + }} {\rm{ImTrGs}}\left( {k, \omega + {\rm{i}}\eta } \right), $

(4)

and the results are shown in Fig. 4(a) and 4(b). It is obvious that there are three gapless chiral edge states that emerge inside the bulk gap connecting the valence and conduction bands corresponding to the Chern number C=3.

As shown in Fig. 5 (Fig. 6), the change of band structure as a function of correlation strength U for monolayer VCl₃ (VBr₃)indicates that there exits a phase transition at U≈0.45eV (0.35eV), and stronger correlation will turn it into a Mott insulator. We also calculated the properties of a monolayer VBr₃. Although the monolayer VBr₃ exhibits some similar properties as in VCl₃, i.e., a Weyl semimetal without SOC (Fig. 2(b)) and an energy gap is opened when including SOC (Fig. 2(d)), the monolayer VBr₃ has the total Chern number C=1 from 6 Berry curvature peaks, of which 4 peaks have positive sign and 2 peaks have negative sign (Fig. 3(d)). The nonzero Chern number 1 corresponds to one gapless chiral edge state (Fig. 4(b) and 4(d)) connecting the valence and conduction bands.

3 Discussion

In summary, based on the first-principle calculations, we propose that the 2D monolayer VCl₃ and VBr₃ are QAH insulators without considering correlation effect, and they transform into Mott insulators after turning on the correlation effect U. At the mean-field theory level, VCl₃ and VBr₃ exhibit QAH insulating states with Chern numbers of 3 and 1, respectively, thus giving rise to chiral gapless edge states. If the correlation U is larger than about 0.45 (0.35)eV for VCl₃ (VBr₃), it will cause a topological phase transition and the monolayer turns into a Mott insulator. These topological properties can be detected by electrical transport experiment, which has been carried out in detecting the QAHE in V-doped (Bi, Sb)₂Te₃ thin film ^[37], Cr/V-codoped (Bi, Sb)₂Te₃ system^[38], and other doped TIs ^[39].

The authors thank YOU Jingyang for helpful discussion.