NRI not willing to pay TDS. Can buyer cancel agreement and ask refund?

I did an agreement to sale with NRI as per which paid 25% amount as down payment to NRI's NRO account and PoA had assured that he will take care of all tax implications. The rest 75% is from bank loan and it shall be credited directly to Buyer's account. But after breaching 45 days; PoA says that they are not interested in paying the TDS at all. I orally requested for a refund and cancellation of agreement. But seller wants to apply for Nil/Low TDS certificate now after 45 days. But I believe that Nil/Low TDS certificate tax rate will be valid only on the transactions performed during the validity period; any transactions performed before or after the validity are still taxed at normal applicable rates. So, I demanded to return the 23% TDS amount from the initial down-payment so that I can pay the TDS amount at least for that transaction. However the seller is reluctant to refund me and is not allowing me to make TDS payment. Can I cancel the agreement to sale now and ask full refund?