Liang, L (reprint author), Chinese Acad Sci, Inst Theoret Phys, State Key Lab Theoret Phys, POB 2735, Beijing 100190, Peoples R China.
; Yu, Yue] Chinese Acad Sci, Inst Theoret Phys, State Key Lab Theoret Phys, Beijing 100190, Peoples R China
; [Wang, Ziqiang] Boston Coll, Dept Phys, Chestnut Hill, MA 02467 USA
; [Yu, Yue] Fudan Univ, Ctr Field Theory & Particle Phys, Dept Phys, Shanghai 200433, Peoples R China
; [Yu, Yue] Fudan Univ, State Key Lab Surface Phys, Shanghai 200433, Peoples R China
We report the finding of a series of distinct-symmetry spin-liquid (SL) states and a rich phase diagram in a half-filled honeycomb lattice Hubbard model with spin-dependent hopping amplitude t'. We first study the magnetic instability of the system and find two antiferromagnetic orders beyond a critical Hubbard U which increases with the ratio t'/t. For t' approaching t, the semimetal transforms to a U(1) SL and then to the Kitaev Z(2) SL as U increases. In a wide middle range of t'/t, the latter is replaced by a U(1) SL to SU(2) SL transition. The physical properties of the stable SL phases are discussed.