The vote was held past midnight after opposition parties brought a motion against him, which was upheld by the Supreme Court. Mr Khan had said he would not recognise an opposition government, claiming without evidence there was a US-led conspiracy to remove him from office.
The assembly is now expected to appoint a new Prime Minister.
The Supreme Court ruled on Thursday that Mr Khan, 69, had acted unconstitutionally when he blocked the no-confidence vote and dissolved parliament.
This made several opposition members furious, with some accusing the former sporting legend of treason.
Minutes before the vote started, the speaker of Pakistan’s lower house of parliament, a major ally of Mr Khan, announced his resignation.
Members of Mr Khan’s party (PTI) left the building, insisting he was the victim of an international conspiracy.
Opposition parties were able to secure 174 votes in the 342-member house in support of the no-confidence motion, the house speaker said, making it a majority vote.
In a tweet, opposition leader Shehbaz Sharif said Pakistan and its parliament were “finally freed from a serious crisis”.
Mr Sharif added: “Congratulations to the Pakistani nation on a new dawn.”
READ MORE: Putin detailed plan to ‘exterminate’ Ukrainians in ‘Mein Kampf’
Mr Khan has repeatedly said that Pakistan’s opposition parties are working with foreign powers.
He also claims he is the target of a US-led conspiracy to remove him because of his refusal to stand with Washington on issues against Russia and China.
The US has said there was “no truth” in these allegations, and Mr Khan has never provided any evidence.
He visited Moscow to meet President Vladimir Putin as Russia was launching the invasion of Ukraine and has previously criticised what the Bush administration called the war on terror.