The PM told the BBC that he and Samantha, plus their two children – Nancy, six, and Elwen, four – had their final breakfast in his Notting Hill home on Thursday morning.
"We are moving in today. So we had the last breakfast at home this morning but I have got quite a lot of other meetings and things I've got to do at the same time," he said.
Later that morning a removal van was spotted outside the Camerons' £2.7 million West London home ready to take their belongings to Downing Street.
The family are to first move into the top floor Georgian flat above Number 10, but will later take up residence in the more spacious flat above 11 Downing Street once it has been refurbished.
There have been reports that the work will include a remodelling of Number 11's kitchen, but the Prime Minister's spokesman declined to comment on specifics.
"There is a maintenance budget and any excess will be paid by the Prime Minister," he said.
Although the Chancellor normally lives at Number 11, recent Prime Ministers have preferred to swap the small Number 10 quarters for the four-bedroom space above Number 11.
The arrangement was first struck in 1997 by Tony Blair when he persuaded his Chancellor Gordon Brown, at that time a bachelor, to let his family live at Number 11.
When Gordon Brown became PM in 2007, he also moved his young family into Number 11.
Now the Camerons will do the same, giving Number 10's smaller flat to Chancellor George Osbourne.
The Camerons are expected to rent out their trendy Notting Hill home, which figured frequently in his 'WebCam' videos during the election campaign.
The family will now divide their time between Downing Street, their constituency home in Oxforshire and the PM's official country residence of Chequers in Buckinghamshire.