Commendation of Kael'Thas
With as much as I hate Kael'Thas, I want to get his commendation. As I said above, it provides 57 stamina, which is the most any trinket in the game provides for a tank. Beyond that, it also gives an amazing 152 dodge when you drop below 35% health. That equates to about an 8.03% dodge increase. When going up against burst damage mobs like Archimonde or Gruul that can drop you below 35% in one shot, the extra dodge can be a wipe saving thing.
Shard of Contempt
Expertise is an often misunderstood stat – it allows your attacks to be parried less and dodged less by the mob you're attacking. There is a rule in raiding that you don't stand infront of the boss with the tank. You do this so you won't cause parries, which can be deadly for the tank (a parry increases the attack speed of a mob by 40%, which means a crushing blow can come in around a tank's shield block rotation). This trinket means that you'll hit more – which is a staple of damage output. Besides that, it has a chance to increase your Attack Power by 230, which occurs at about once a minute (some people may dispute this). This means an average AP increase of 115 over the life of a fight – an amazing number for a trinket by itself; and when combined with the expertise rating, it's nearly godly.
Timbal's Focusing Crystal
It's named after Timbal. Tharfor you should have it. 'nuf said.
Besides it increases damage and healing by 44, which is a nice buff. It also allows your dots to do 285 to 475 more damage. That is just...well... dot-a-licous. A Shadow Priest or Warlock that uses dots will love to add this to their inventory (as long as they can control the extra threat it will produce).
Vial of the Sunwell
First, this provides 15 MP/5. That's great, especially for a trinket slot. Beyond that, Vial of the Sunwell also provides an extra emergency heal. Some healers might not find this the most useful trinket in the game, but in an instance – say for Reliquary of Souls – where mana and thus healing are in short supply, the extra well timed heal could easily make the difference between success or failure.
Of course the down side to all of these trinkets is that they're going to take some work and grinding to get. They all drop off the third boss in heroic Magisters' Terrace – the one where you have to fight five guys at once, just like a five man arena match.
Fun? Yes. Annoying? Yes. Worth it? Definitely yes.