Nadal had taken the first set to leave Cilic a 10/1 outsider to come from behind and knock out the world No 1.
But the Croatian rallied to take the second before losing an agonisingly tight third-set tie-break in which both men lost their serve repeatedly.
However, Cilic hung tough and broke early in the fourth, at which point Nadal called for the trainer.
He had three minutes of treatment on his right leg above the knee and was given tablets to deal with the pain – but he still could not stop Cilic forcing a fifth and deciding set.
And after another treatment session, the No 1 seed was broken again and he promptly retired, much to the disappointment of a packed Rod Laver Arena.
Nadal confirmed afterwards that the problem was not with his troublesome knee or with his hip, but that it was a muscular injury that had initially felt “tired” in the third set before giving way in the fourth.
“It is difficult to know exactly what it is now,” Nadal said.
“It is difficult to know exactly the muscle. Just happened minutes ago.
EPA
Marin Cilic is into his second Australian Open semi-final
“This type of injury is difficult to know immediately, no?
“We need to wait a couple of hours.
“Tomorrow I am going to do a test, an MRI here, then we will know.”
Nadal had already said he would not play in Spain’s Davis Cup tie with Britain (February 2-4) and will now prioritise the top clay-court tournaments in Barcelona, Madrid, Monte Carlo and Rome ahead of defending his French Open crown.
GETTY
Rafael Nadal was forced to retire nearly four hours into the match
But he did call on the ATP Tour to reconsider the schedule and surfaces on which they play after joining Novak Djokovic, Andy Murray, Stan Wawrinka and Kei Nishikori in the ranks of men’s tennis’ walking wounded.
“Somebody who is running the tour should think little bit about what’s going on,” Nadal added.
“Too many people getting injured. I don’t know if they have to think a little bit about the health of the players.
“Not for now that we are playing, but there is life after tennis.
“I don’t know if we keep playing in this very, very hard surfaces what’s going to happen in the future with our lives.”
Cilic meanwhile would not be drawn on whether he wanted to see changes, as many professionals have called for in recent months.
“The calendar is there for so many years,” Cilic said.
“Just in this last year, obviously beginning of this one, we see a lot of top guys that are injured.
“In the end it’s on all of us to try to take care of our bodies, to try to pick the right schedule, to listen to our body, how it feels.
“I completely understand there are a lot of tournaments that we play, mandatory tournaments.
“In my own perspective, we are all picking our own schedule.
“It’s tough to say, ‘Okay, we going to take out two months of the season’ cut that many tournaments, because tennis is such a global sport.
“Everywhere we play people enjoy it. I think tennis is getting more and more popular, which we really want also.”
Cilic will play unseeded Kyle Edmund in the semi-finals of the Australian Open, a stage of the Melbourne tournament the Croatian last reached in 2010, when he lost to another Brit – Andy Murray – in four sets.
Cilic and Edmund will clash for a place in the final at 8.30am UK time on Thursday morning.