We are forced to pay for the benefit of others in other ares of interest to society. ie. Public Education which just about all will agree benefits a society as a whole to have a literate populous.
Ok, bad example but you catch my drift.
Part of the problem which has developed over time is the Legislative and Executive Branches of Government have gotten far too generous with our money pandering to one interest group or another in order to secure their support.
The other part of the problem is we now have a permanent moocher class who feel its perfectly fine to pluck another man's pocket.