Jiang, K (reprint author), Boston Coll, Dept Phys, Chestnut Hill, MA 02467 USA.
We study the Hubbard model on the frustrated honeycomb lattice with nearest-neighbor hopping t(1) and second nearest-neighbor hopping t(2), which is isomorphic to the bilayer triangle lattice, using the SU(2)-invariant slave boson theory. We show that the Coulomb interaction U induces antiferromagnetic (AF) chiral spin density wave (chi SDW) order in a wide range of kappa = t(2)/t(1) where both the two-sublattice AF order at small kappa and the decoupled three-sublattice 120 degrees order at large. are strongly frustrated, leading to three distinct phases with different anomalous Hall responses. We find a continuous transition from a chi SDW semimetal with the anomalous Hall effect to a topological chiral Chern insulator exhibiting the quantum anomalous Hall effect, followed by a discontinuous transition to a chi SDW insulator with a zero total Chern number but an anomalous ac Hall effect. The chi SDW is likely a generic phase of strongly correlated and highly frustrated hexagonal lattice electrons.