Gravity has no problem with "pollen and fluffy things". Both are acted on by gravity, though they are also acted on by random air currents including wind. There really is no mystery about any of this. We've understood how they work for centuries.
