Baroness Ros Altmann said the tax was putting older people off moving to smaller properties because it was too expensive.
She said: “Stamp duty is a massive deterrent.
“There’s a lot of talk about building homes for first-time buyers but what about last-time buyers?
“Ultimately we need to be building more homes for our growing ageing population.
“Smarter homes for this generation will relieve the pressure on social housing.”
The Tory peer is a former head of Saga, which has been lobbying the Government for years to impose a stamp duty exemption on downsizing into properties worth up to £250,000.
Estate agents have been calling for the Government to scrap stamp duty for buyers altogether and just levy it at sellers.
Tory MPs are understood to have lobbied the Treasury to include some stamp duty measures in the autumn Budget, believing it to be responsible for slowing house price growth.
Research by retirement housebuilders McCarthy & Stone reveals that 38 per cent of homeowners aged over 65 would consider downsizing, with an additional 10 per cent encouraged to move with a stamp duty exemption.