Paying back Proportion of VAT saved on purchase

Hi has anyone had experience of repaying a proportion of the VAT saved on purchase after stopping rentals on LMNP basis?
we have taken back our appartment for our own use which we are allowed to do according to the rules of the residence. PIerre et Vacances ended our lease after the initial 9 years but we continued to rent out under LMNP basis with Maeva.com. We have now decided to use our appartment ourselves and have 2 years remaining before the 20 years period is up. We asked our tax office how to go about repaying what we owe and for the amount but they do not seem to understand our request! We are not selling up so have not engaged à notaire.
Has anybody done this themselves and how did you do it please?