Figures released yesterday by the Driver and Vehicle Licensing Agency highlight the problem of repeat offenders.
Over 400 drivers were caught three times, 20 were caught four times and three were even caught five times, according to a Freedom of Information request.
Motoring groups warned the figures illustrate how a perceived lack of enforcement means some drivers flout the rules and use their phones behind the wheel.
The data relates to the number of CU80 endorsements held on driving records.
They are handed out to drivers found not in full control of their vehicle, such as by using a phone illegally. The endorsements are only erased after four years.
Almost 174,000 drivers have at least one CU80 endorsement but just 309 were disqualified because of it.
RAC road safety spokesman Pete Williams said responsible motorists will be “shocked” to discover the number of drivers persistently using a hand-held mobile behind the wheel.
He went on: “The threat of a second endorsement and the possibility that they will lose their licence is clearly not enough to deter this hard core of drivers.”
Since March 1, motorists caught using a hand-held phone have faced receiving six points on their licence and a £200 fine – up from the previous penalty of three points and £100.
Drivers can lose their licence if they receive 12 points within three years, or six points in the first two years after passing the test.
Mr Williams added that repeat mobile phone offenders often believe there is “little chance of seeing a traffic police officer”.
The number of road police officers in England and Wales has declined by 27 per cent in the past decade.
Josh Harris, director of campaigns at road safety charity Brake, called for “the menace of distracted driving” to be urgently addressed.
He said: “Six thousand repeat offenders in the past four years is truly staggering and makes a mockery of the points system.
“The laws clearly need tightening alongside stricter enforcement.”