Maliks Muwatta Book 45, Hadith Number 9.
Section : Residing in Madina and Leaving It.
Malik related to me that he had heard that when Umar ibn Abd al-Aziz left Madina, he turned towards it and wept. Then he said, “O Muzahim! Do you fear that we might be among those that Madina casts off?”