The first important thing to note is that whether or not you will prospect the ore, smelt it, or sell it raw, will depend heavily on the type of ore.
You should also pay attention to your target buyers. The vast majority of players buying ore are using it to level a crafting profession, namely blacksmithing, jewelcrafting, or engineering.
Currently, aside from treasure chests, fishing and similar means based on chance, there is one way to acquire ore, and two to three ways to acquire gems, generally making low level gems worth less than the ore from which they are prospected.
As of patch 2.1, prospecting has no chance of failure and so you are guaranteed at least one gem, although you are gambling in a sense when you do it. The value of the five ore is a known value, but the gem(s) you'll receive and its value is a mystery.
Tip: When deciding between smelting ore to sell it, or selling raw ore, note that a) if prospecting the metal is profitable, the ore will sell for more than the bars and b) some people pick up crafting professions and not the accompanying gathering profession, making the smelted bars most useful.
Copper, tin and iron
Blacksmithing especially eats up a seemingly impossible amount of metal, and copper, tin and iron can fetch a profit that any low level miner will really appreciate.
Prospecting copper will give you malachite and tigerseye most commonly, and shadowgems occasionally.
Tin will commonly yield shadowgems, as well as moss agate and lesser moonstone. Sometimes it will also produce citrine, jade, or aquamarine.
Iron often produces citrine, lesser moonstone and jade, while occasionally producing aquamarine or star rubies.
When smelting, you'll have to decide whether or not to keep copper and tin separate, or combine them to produce bronze. On my server, I find that the value of the bronze is about equal to selling the copper and tin separately, although prices do vary from day to day. I'd recommend checking the auction house daily, and before you smelt, so you'll be able to make an informed choice.
You'll probably profit more from selling the bars in this case. Most of these low level gems drop readily from chests and mobs, and therefore don't sell for much. The mid-level gems like aquamarine and star rubies, are also common in chests, and I used to farm them in Tanaris.
Other jewelcrafters, who don't get jewelcrafting points for prospecting, are unlikely to purchase the raw ore with the intent of prospecting it, as the gems really are so common. In addition, most level blacksmiths will find that they must farm veins or purchase the metals, because they don't happen upon near enough veins in their travels to supply their crafting needs.
My vote: smelt it and sell it!
Silver and gold
These veins are actually rare spawns of common veins, and therefore the ore is considered to be "rare" and cannot be prospected. Silver is a rare spawn of tin, and gold of iron.
This would seemingly eliminate two of your options, as no one is purchasing the ore for prospecting. In actuality, as of patch 2.4, silver is the best way to level mining in the large gap between the ability to mine tin and iron. Simply smelting it will award skill points.
Similarly, smelting gold can take you from skill level 155 up to 180, five points past what is needed to purchase the ability to mine mithril.
My vote: Sell the raw ore.
Another rare ore, this is a spawn of a mithril vein.
Once you can mine mithril, you'll start finding truesilver veins. As of patch 2.4, smelting this ore will award mining skill points. This can take you from 230 to 250. Truesilver might be worth more smelted because of the limited skill window, so check the auction house to make sure.
My vote: Check the AH before you smelt.
Prospecting mithril will commonly get you citrines, star rubies and aquamarines. Rarely, you'll get azerothian diamonds, blue sapphires, large opals and huge emeralds.
Here is where things start to get murky. Star rubies and aquamarines can sell for a fair profit, as can blue sapphires. The best advice I can give is to figure out the price that the gems fetch, as well as the ore or bars on your server, and then weigh the probabilities.
Ideally, you'd be able to purchase a stack of twenty mithril ore, prospect all of it, sell the gems, and have money left over. Otherwise, you'd get more from simply selling the mithril.
The common gems listed above drop about 30% of the time, while the uncommon gems drop about 2.5% of the time. If you don't want to take the gamble, then sell the raw ore to other jewelcrafters who do!
My vote: It depends on your server's gem market. You can prospect it or, to be on the safe side, sell the ore.
Thorium commonly gives out star rubies, and less commonly, large opals, blue sapphires, azerothian diamonds, and huge emeralds.
If you want to stay safe, I'd sell the bars. Thorium is another metal that blacksmiths consume copious amounts of, and most can never get enough of the stuff.
My vote: Sell the bars.
Prospecting this can be lucrative, because although you have a small chance to earn a blue gem, the green gems are much more profitable than they used to be. This is because of brilliant glass. Using three of each of the six basic green gems, you can make one blue one.
Prospecting it can yield any of the Outland green gems, and rarely, any of the blue ones. Because of the effect of brilliant glass on the Outland gem market, each green gem could sell for 2-4g a piece, and some for more. Blood garnets especially can sell for up to ten gold. Of course, the green cut gems now also sell for a fair chunk of change.
Keep in mind that it takes two fel iron ore to make one fel iron bar. Because of this, when you prospect five fel iron, you are destroying the potential profits yielded from selling five ore, or two and a half bars.
Often, fel iron ore is worth more than the bars, so if you need a bar, don't purchase the ore. A stack of twenty ore should be equal to the price of ten bars, but it is often more expensive. On the other hand, fel steel (three fel iron bars a piece) sells for a lot.
My vote: Prospect it! Either sell the greens, or combine them to make brilliant glass. If you have a popular cut for a green gem, cut it and sell it.
Prospecting adamantite can yield the same gems as fel iron, except you have close to a 20% chance to get a blue gem.
Prospecting adamantite is the only instance where the dust or powder produced is useful instead of vendor trash. Adamantite powder is used in the making of hardened adamantite.
My vote: Don't smelt it! Prospect it, or at the very least sell the ore. As above, you'll be able to make brilliant glass and cut gems for a huge profit. It is worth it to buy the ore, prospect it, and turn a profit that way.
Khorium and eternium and all that jazz
You cannot prospect these. Khorium is a rare spawn vein in place of any vein in Outland, regardless of type, and eternium simply turns up as a bonus from any vein.
Other ore, like dark iron or anything used for questing, also cannot be prospected.
My vote: Sell the ore. If you can smelt dark iron ore, do so!
TIp: Always getting under-cut at the auction house? Choose a flat, realistic price that's a good deal and advertise on trade channel. Players who have been meaning to pick up some ore/bars/gems, and regularly do so, will recognize the deal and snap it up.