Fiqh-us-Sunnah Volume 1, Purification and Prayer, Fiqh 1.103B.

It is not allowed to leave the call unanswered or to leave the mosque after it has been made, unless there is some excuse or one has the intention to return for the prayer. Abu Hurairah related that the Prophet, upon whom be peace, told them, “If one of you is in the mosque and the call is made, he should not leave the mosque until he prays.” (Related by Ahmad with a sahih chain.) It is also related that Abu Hurairah said about a man who left the mosque after the call had been made, “That man has disobeyed Abu al-Qasim (the Prophet, upon whom be peace).” This is related by Muslim and others. Mu’azh at-Jahni related that the Prophet said, “It is the utmost apathy and sign of disbelief and hypocrisy that one who hears the call of Allah to salvation does not respond.” (Related by Ahmad and at-Tabarani.)

Commenting upon this, at-Tirmizhi says, “It has been related from more than one of the companions that one who hears the call and does not respond will have no prayer. Some said that this is the maximum imposition, which shows that there is no excuse for one who does not attend the congregational prayer without a valid reason.”