The baryon spectrum in the Sakai-Sugimoto model in the D4 background with smeared D0 charges is studied. We follow the instanton description of baryons by Hata et al. [Prog. Theor. Phys. 117, 1157 (2007)]. The background corresponds to an excited state with nonzero glue condensate < tr(F-mu nu (F) over tilde (mu nu))> which is proportional to the D0 charge density. The baryon size shrinks when we turn on small D0 charge density. But for larger D0 charge density where massive modes in the gauge theory may also take effect, the size of baryons will grow. The difference between baryon masses will become smaller when D0 charge density increases. There may also be indications that the baryon will become unstable and cannot exist for sufficiently large D0 density.