The loss of a few wealthy people is not as serious as the loss of jobs related to the companies and service requirements that move with them. When the CEO/Owner of a company decides to move, many of the top officers move also. Non-critical employees will lose their jobs, adding to the government's unemployment costs. Many of the organization's service providers will lose revenue and be forced to downsize. Many of those who provide personal services to the owner and top executives will also be harmed.
The wealthy that stay will adapt. Someone who was netting $500,000 from a $1,000,000 salary will demand a raise up to $2,000,000 to maintain his $500,000 net. Either several non-essential workers must be fired or prices must go up.