Flat rate VAT - purchase refunds

We use the flat rate VAT scheme. When inputting purchase invoices, I still include the VAT so that I am able to track the savings. However, when I have a purchase refund, the VAT return is including these as sales and inflating my VAT payable. I can correct this by going back to the purchase refunds and removing the VAT but then I am no longer to fully track the differences between the two schemes. Am I doing something wrong?

Thanks Viv