Maliks Muwatta Book 10, Hadith Number 14.

Section : The Coming of the Imam on the Day of the Id and Waiting for the Khutba.

Yahya related to me that Malik said, “The sunna concerning the time of prayer on the ids of Fitr and Adha – and there is no disagreement amongst us about it – is that the imam leaves his house and as soon as he has reached the place of prayer the prayer falls due.”

Yahya said that Malik was asked whether a man who prayed with the imam could leave before the khutba, and he said, “He should not leave until the imam leaves.”