Can an agent demand full course fees without a signed agreement?

I am from Odisha. I enrolled in a B.Ed. program at Andhra University through an agent who is a government school teacher and also runs a coaching centre. The total course fee was ₹1.2 lakh (approximately ₹1.5 lakh including additional expenses). I have already paid him ₹60,000—₹30,000 at the time of admission and ₹30,000 for the first semester. The second semester is scheduled to begin on 10 August. However, I have secured good marks in a State Government recruitment examination, completed document verification, and expect to join a government post soon. Since I cannot continue both, I requested to withdraw from the course. The agent became angry and demanded the entire two-year course fee, insisting that I pay an additional ₹50,000. I offered ₹10,000 to settle the matter amicably, but he refused and started threatening me, saying that he would file an FIR, initiate legal action, and jeopardize my government job. He claims that there is an agreement, but I never signed any document. I have recorded all the threatening phone calls. What legal action can I take in this situation?