Sahih Muslim Book 7, Hadith Number 3078.

Chapter : The demolishing of the ka’ba and its reconstruction.

‘Aisha (Allah be pleased with her) reported: Allah’s Messenger may peace be upon him) said to me: Had your people not been unbelievers in the recent past (had they not quite recently accepted Islam), I would have demolished the Ka’ba and would have rebuilt it on the foundation (laid) by Ibrahim; for when the Quraish had built the Ka’ba, they reduced its (area), and I would also have built (a door) in the rear.