The "not charging" message doesn't appear when the iPhone 4 is plugged into a lower-powered USB port (in my experience), but is it still sucking the same amount of juice it would from an AC outlet?
The folks over at When Will Apple? took the time to do a pretty decent job of comparing charging times on the iPhone 4, comparing USB to AC outlet. The conclusion: USB charging takes 23% longer to charge an iPhone 4 than charging via an AC outlet. If you're in a rush to charge your phone, that's a pretty significant difference.
The little flaw in this test is not knowing the amount of charge coming from the USB port used, as not all USB ports are created equal. Still, I doubt a better USB port would get you back that lost 23%.