Arms were traded to the Iranian gov't in an attempt to curry favor with so-called moderate elements. Hostages were not being held by the Iranian gov't, but by a terrorist group. This was a violation of Reagan's stated policy, but not necessarily illegal. Nonetheless he shouldn't have done it.
The funny thing is that the part that was clearly illegal, giving arms to the Nicaraguan Contras, was the still the right thing to do. That's what he got in the biggest trouble for. Violating the Boland Amendment, not for circumventing his own policy.