We didn't rinse ours, well we started rinsing it, but too much gravel, a dry summer, a well we protect, even with a good rinse the stuff had dirt, so we just dumped it in, cycled the dirt.
If I pull plants, now, the pea gravel dirt / dirt that accumulates / etc pours into the pond. Within a few hours the water is crystal clear.
When I first turned on the pond and bog my water was totally murky for a few weeks, I did make a pretend skippy type filter, quilt batting, ac filters, stuff to pick up the fine dirt, ran that for a few days, low flow pump going into the barrel. That cleared up the water. I did that a few years back and have not used it since. My guess is the bog was not established enough, i.e. plants roots etc to get the fine dirt out of the water, now it has no issue even if I muck it up enough you can not see a single fish, it clears within hours.