The builder has handed over this project to the panchayat. Now the panchayat is responsible for maintaining security and providing facilities to the residents. For the sake of providing facilities it cannot impose unreasonable or arbitrary restrictions on the movement of residents and ingress of visitors.
If the builder has installed this gate in violation of the sanctioned site plan then you should file a complaint before the administrative officer of the panchayat for its demolition. Common Cause (A Regd. Society) v. Union of India [AIR 2018 SC 1665] the Supreme Court has held that
Word ‘life’ in Art. 21 has to be construed as life with human dignity. Any physical interference with person is, prima facie, tortious. If it interferes with freedom of movement, it may constitute false imprisonment. If it involves physical touching, it may constitute battery. If it puts a person in fear of violence, it may amount to assault. For any of these wrongs, the victim may be able to obtain damages.
Imposing unreasonable and arbitrary restrictions on locomotion or movement is violation of fundamental rights of citizen enshrined in Article 21 of the Constitution of India.
In this situation you should move a writ petition before the High Court for the illegal activities of the panchayat. You should file a writ of mandamus under Article 226 of the Constitution of India and seek direction to the administrative officer of Panchayat to ensure free movement and dignified living of residents.
Any contract signed by residents which empowers the authority to impose unreasonable restrictions on the movement is void under Section 23 of the Contract Act.
You can also file an FIR against the concerned officer of Panchayat for the offence committed under Section 341/342/344 of the Indian Penal Code.