The four ministers claimed they quit because the government was not treating them and their party well.
Ibobi Singh, a former chief minister, said his party will move a no-confidence motion against the state government and he will approach Speaker Yumnam Khemchand Singh to convene a special session of the assembly at the earliest.
Deputy Chief Minister Y JoyKumar Singh, Tribal and Hills Area Development Minister N Kayishii, Youth Affairs and Sports Minister Letpao Haokip and Health and Family Welfare Minister L Jayanta Kumar Singh quit their ministerial posts on Wednesday.
The rest who withdrew support are an MLA from Trinamool Congress and one Independent.
The total strength against the chief minister is now 29 -- 20 from Congress, 4 from NPP, 3 from BJP (who quit on Wednesday), one from TMC and an Independent.
The figure in support of Biren Singh stood at 23, including 18 BJP MLAs, 4 of Naga People's Front (NPF) and one MLA from LJP.
The seven Congress MLAs, who had crossed over to the BJP, are facing anti-defection cases under the Tenth Schedule of the Indian Constitution in the Speaker's Tribunal as well as in the High Court of Manipur.
Assembly sources said the Speaker's Tribunal is likely to hear the matter of the seven Congress defectors during the day.
The Speaker is a second-term BJP MLA.
One Congress MLA who joined the BJP-led government was disqualified from the membership of Manipur Legislative Assembly recently, reducing the strength of the house to 59. Assembly election in Manipur was held in 2017.
The four former council of ministers claimed the way the government was treating them and the party forced them to walk out.
Former Deputy Chief Minister Y Joykumar told media persons Wednesday night, "The way this government has treated our party and ministers including me, we deliberated in detail about as what will be the best course of action of the party and for us...and keeping in view particularly the assembly election in early 2022, we decided to tender our resignation."
Asked if they would support Congress, he said, "Obviously, we may be exploring the option and definitely form a government after moving a no-confidence motion."
The AITC MLA T Robindro told media, "We were with the government for some three years now. We feel like we have been neglected and whatever proposals we put up, it will be sidelined."
The hectic political activities are taking place at a time when Manipur will see election for one Rajya Sabha seat on June 19. Both the BJP and the Congress have fielded candidates for the lone seat of the upper house.
BJP has fielded Leisemba Sanajaoba, the titular king of Manipur, while the Congress nominee is T Mangi Babu.