This whole charging / not charging issue with the iPad and iPhone is pretty confusing. At least with the iPad, if it's not plugged into an outlet or a USB port that's delivering enough juice, you get the whole "not charging" message near the battery indicator ... yet, it does charge, but slowly.
The "not charging" message doesn't appear when the iPhone 4 is plugged into a lower-powered USB port (in my experience), but is it still sucking the same amount of juice it would from an AC outlet?
The folks over at When Will Apple? took the time to do a pretty decent job of comparing charging times on the iPhone 4, comparing USB to AC outlet. The conclusion: USB charging takes 23% longer to charge an iPhone 4 than charging via an AC outlet. If you're in a rush to charge your phone, that's a pretty significant difference.
The little flaw in this test is not knowing the amount of charge coming from the USB port used, as not all USB ports are created equal. Still, I doubt a better USB port would get you back that lost 23%.