The Social Democrats party (SPD), led by Martin Schulz, won 37.3 per cent of the votes in Lower Saxony in Germany, which is one of the most densely populated states in the country.
Chancellor Merkel’s Christian Democratic Union (CDU) won 33.4 per cent of the vote in a surprise loss in the snap election.
It was the first test of Mrs Merkel’s strength since September’s national election, which gave her conservatives their worst result since 1949.
The gains made by the SPD will make it increasingly difficult for Chancellor Merkel to get legislation through during the next four years.
The SPD’s state leader Stephan Weil said: “For the first time in 19 years, we will be the largest group in the state parliament.
“Merkel’s Christian Democrats are the clear losers of this election.”
Mr Schulz said Sunday’s result was a “wonderful victory” for the SPD.
Although September saw Mrs Merkel winning a fourth term as chancellor, the far-right Alternative Fur Deutschland (AfD) party made huge gains.
The AfD scored 5.5. per cent in the regional elections meaning the party has made gains in 14 of Germany’s 16 states.
Commentators claimed the SPD making gains in Lower Saxony would weaken Chancellor Merkel as she tried to make an alliance with the Greens and the Free Democrats.
The result is a turnaround for the SPD, which saw its worst performance in a national election since 1949 winning just 20.5 per cent of the vote.
The German state of Lower Saxony has banned migrants from moving to one of its cities.
The government of Lower Saxony, in northwest Germany, claimed the imposed immigration restrictions on migrants in Salzgitter is due to the city’s “exceptionally high immigration”.
Lower Saxony was forced to undergo a snap vote after the coalition of the SPD and the Greens collapsed.