My impression is that this is a smallish town that can maybe only support one/two optometrists. Do you agree that the OP will now have an unfair advantage with his insider knowledge? Will his opening have an effect on the TO?

I guess we do not have the overall picture an the town might be quite large with numerous optometrists in which case I might think differently, but even then at the end of the day that doesn't change the fact that the OP agreed to the terms of the agreement and is now going back on his word, and know might use lawyers to wangle his way out of it.