Ferrer will retire from tennis next month at the Madrid Masters after a decorated career which included 27 ATP Tour titles and runner-up at the French Open.
The Spaniard had the misfortune of playing in the same era as all-time greats Federer, Rafael Nadal and Novak Djokovic and he has losing match records against all three players.
But one player stands out most for Ferrer in terms of being his toughest opponent, losing 17 times.
“For me, it’s Roger Federer,” Ferrer told the ATP.
“He would change the pace and drive me crazy.
“Maybe I made him sweat like many players, but I never had the chance to be able to beat him.”
Ferrer plays this week at the Barcelona Open, his penultimate event before retiring.
When asked about leaving the sport, Ferrer said he was proud of everything he accomplished.
He said: “I feel happy with my transition into accepting that I’m going to stop playing and that this was going to be my last year.
“There is always a little bit of fear at the last tournaments. You think about how you’re going to feel.
“But my goal is to be happy. I feel competitive, I’m playing the tournaments that have given me the most affection and that I am most excited about.
“Thanks to the career I’ve had, I can look back and feel proud of everything I’ve achieved.”
But the Spaniard did admit he would miss the adrenaline of competition.
He added: “That’s what I’ll miss the most. It’s going to be impossible to find.
“Competing will be the thing I miss the most. I’ll have to find other things.
“Cycling or something to calm the beast a little.”