dear nicky did you realise that there would be an economic cost in changing a fixed loan?. the bank manger advised you of the fee. The $5000 is not a fee. not sure why you would think that by fixing a loan there would be further costs if rates want down. So you fixed a rate 1 year ago and now want to take advantage of the lower fixed or variable rates and want to do this at no further costs.
the above is the arguement you will need to be able to answer. Also the loan contracts do include specific references to fixed rates and the additional costs, not fees applicable.
