The TV chef, worth an estimated £150m, has said that most low-income families don't know how to feed themselves properly, and instead choose expensive options such as ready meals.
In an interview with The Radio Times, Jamie, 38, said that he was "not judgmental" of poor families, pointing to his experiences of people on low incomes on his previous TV shows, but said he finds it "hard to talk about modern-day poverty".
"You might remember that scene in Ministry of Food with the mum and the kid eating chips and cheese out of Styrofoam containers, and behind them is a massive TV. It just didn't weigh up," he said.
"Some of the most inspirational food in the world comes from areas where people are financially challenged.
"The flavour comes from a cheap cut of meat, or something that's slow-cooked, or an amazing texture's been made out of leftover stale bread."
The father-of-four, whose new show Jamie's Money Saving Meals is designed to help people save on their food bill, added, "The fascinating thing for me is that seven times out of 10, the poorest families in this country choose the most expensive way to hydrate and feed their families. The ready meals, the convenience foods.
"I meet people who say, 'You don't understand what it's like'. I just want to hug them and teleport them to the Sicilian street cleaner who has 25 mussels, 10 cherry tomatoes and a packet of spaghetti for 60 pence, and knocks out the most amazing pasta. You go to Italy or Spain and they eat well on not much money. We've missed out on that in Britain, somehow."
Jamie's comments have been met with some angry criticism, with many Twitter users pointing to the price of producing his recipes, and how he has promoted ready meals for Sainsburys.
His remarks were also challenged by the Child Poverty Action Group's head of policy, Imran Hussein.
"As official statistics show, parents of poor children are much less likely to be able to afford fresh fruit for their children. We also know from the evidence that as the incomes of poor families rise, they spend more on things like healthy food and children's clothes.
"The huge hits many working and non-working families are taking in their incomes as a result of cuts in tax credits and benefits are very real, as is the resulting huge growth in demand for food banks.
"The Government's child poverty strategy is seriously adrift and urgently needs rethinking."
Mr Hussein added: "Jamie Oliver has made a huge contribution to improving school meals and we're grateful for the support his foundation has given us in our work on free school meals.
"He is right to say that healthy food doesn't always have to be expensive – one of CPAG's ambassadors, the food blogger Jack Monroe, is an excellent example of this – but for many families it's low income which gets in the way of healthy eating.”