Sen. Bill Nelson (D-Fla.) is losing patience with a top financial regulator, saying Tuesday that he should either crack down on speculation driving up oil prices or no longer work at his watchdog agency.

In a letter sent to President Obama, Nelson said Gary Gensler, the chairman of the Commodity Futures Trading Commission (CFTC), needs to implement new restrictions on speculation or lose his job.

"Mr. President, if CFTC Chairman Gary Gensler doesn't act soon to implement rules that will cut down on speculation in the oil futures markets, then you should consider not reappointing him," Nelson wrote.