What can be done if society office bearers skip meetings?

The builder delayed the occupancy certificate (OC) after completing the building. A society was formed by some members without consulting others. In the previous AGM, the society decided to meet the builder before filing a lawsuit to resolve this amicably. The office bearers went to his office and returned. Now, a special general body meeting was called to report the status and decide the next steps regarding the OC matter, but the secretary and treasurer skipped the meeting without informing others. When we ask about the status, they respond irresponsibly, saying they are not our servants and that we should deal with the builder ourselves.