Phil, you can and should live wherever you want. You are not greedy. The politicians who lust after your money are greedy. The professional welfare chiselers are greedy.
You are a nice guy and it is not in your character to say "Pound Sand." So don't. Just buy a new house and move. No fanfare. Just do it.
The productive members of society will understand and support you.