First of all, you're generalizing. If there's one thing about libertarians, it's that they don't all agree on all things. It's not a requirement.
You comment and concept of "responsibility toward society" is the beginning of communism, not the other way around. Individuals have no pact with society. They are society. Tom Woods addresses this here.
Being a responsible person is your contribution to society. Of course, many go beyond that. To think you can force someone to do more is the Marxist position.