Let's assume you can figure out what to do with gray items (that would be vendor them ASAP newbie) and concentrate on the white loot. Most white loot can be safely sold to a vendor, however, occasionally some of that white loot will be rather profitable. It could be something needed for a turn in to the Darkmoon Faire, or perhaps it is useful in a repeatable quest. Either way, there is a chance you can get more money out of the item by selling it in the AH rather than simply vendoring it. Head to the auction house and see if anyone else is selling the items, and for how much. If there is a good market, price your loot competitively and move on. If there is no other item like your white item being sold, that could mean it is vendor trash. But it could also mean that it is highly sought after and some rich level 60 just breezed through the AH and bought all of them. How to determine this? Head over to Allakhazam and check out the average auction price for the particular item. If it has been selling on the auction house, you will be able to see how often it has been sold, and for how much.
Now that you know what to do with gray and white items, let's move on to a more tricky variety of loot, green drops. Good News: Green loot will generally sell in the auction house. Bad News: This is not a hard and fast rule. Some green items will have stats on them which are fairly useless for the classes the piece is meant for. For example, If you find cloth armor with +strength, it will not be in high demand. However, a piece of cloth with +int and +stam should be in high demand for a caster, and you will be able to find a buyer a majority of the time. It all depends on the stats. If you just don't know, check out armor in that level range on the auction house to get a gauge on the prices or head over to Allakhazam.
One last note on green items (and less desirable blue gear as well)...If you price your undesirable green item low enough, you may sell it to someone looking to level their enchanting skill. However, this might not be the best use of your time, and selling it to the vendor might just be easier. If the difference between vendoring the green sword and selling it on the AH is only 15 silver, you might realize your time and effort are better spent somewhere else. Whatever you do, don't sell it for less than what the vendor will give you. I have seen many a newbie selling items that could be bought from the auction house and then sold to a vendor for a profit.
Blue drops are hard to come across, and for the most part, will always sell in the AH, a lot of times for a very high price. Server age, usability and a few other factors come into play, but some blue items, especially those useful for twinks, will garner a very high price. Again, check to see if anyone else is selling the same item you are looking to unload, and price your item accordingly. Some people just see the blue gear and want it, whether it will help them or not. So always put your blue drops on the auction house, just make sure you price it right.
Purple items are extremely hard to come by. I know players who have been playing for over a year and have never seen a random purple drop during their grinding sessions. If you get one, I would suggest immediately checking the worth of the item on Allakhazam and in your local server AH. You could either have an amazingly profitable transaction, or you could have the Lei of Lilies. Sure, Lilies sell for about 70 gold, but some purples sell for much, much more.
Remember, auctioning green, blue and purple items only applies if the items are bind on equip (BOE). If an item is bind on pickup (BOP) , you have no choice but to either use the item or vendor it.
Lastly, it might not be a bad idea to check out the prices that certain kinds of animal meat, cloth and trade goods are fetching on your server. Most of the time you can make a tidy profit selling silk to someone trying to level their tailoring skill, or boar meat to someone trying to work on their cooking. As with every other item you are thinking about auctioning, check your server's AH to see what the market is like.
With all of that in mind, you should be losing less money vendoring possibly valuable items, and not wasting your time trying to AH vendor trash. If you don't like Allakhazam for whatever reason, you can try Goblin Workshop, WoWWiki or Thottbot. The reason I am so high on Allakhazam for this particular task is because of the excellent auction house data contained on the site. Also, I wrote this quick guide from the viewpoint of not having enchanting as one of your tradeskills. Obviously, if you are an enchanter, you can toss most of this information out the window. Finally, a lot of the best info in these WoW Rookie posts come from the insightful, helpful comments, so...what do you auction? What do you vendor? Any helpful tips?
[Update: As some helpful comments have suggested, picking up the Auctioneer add on would make your life deciding what to AH and what to vendor a lot easier. You can download the mod here. I probably should have included that from the beginning, but I am trying to keep these posts as simple as possible. Thanks again for the comments]