As mentioned, Stealth is a rogue's defining ability. Learned at Level 1, every rogue learns to sneak around and pounce on an opponent. It's their single biggest trump card, the ability to remain unseen. In Arena play, it's critical for the rogue to get the opener, as some of her most powerful attacks require her to be stealthed. Taking a rogue out of stealth removes the element of surprise so it's important to be on your toes. If you see a rogue from a distance entering stealth, always be on the ready. While it is practically impossible to get the jump on a very good rogue, the good news is that there are quite a lot of mediocre rogues who get too close to an opponent assuming their stealth will keep them undetected.
It will help to keep your game sounds on, and any music, whether in-game or maybe iTunes, low to be able to hear the distinctive sound of stealth. Hearing this sound indicates that you will be able to target them, so keep the Tab key handy. If you detect them, you have a very small opening in which to attack or cast a spell. The best abilities to use are those that don't require you to be facing the target -- a quick DoT or AoE should do the trick. These should work fastest and prevent error message delays.
A rogue's first few sequences are predicated on the premise that they get the opener, so taking that away from them breaks their rhythm and forces them to change their battle plan. Rogues hate starting a fight out of stealth, so if you manage to break their stealth at the start of a fight, you've already got a mental edge. Of course, if you can catch them before they even enter stealth, then all the better. Applying a Hunter's Mark or Faerie Fire should help here, or even a DoT if they're within range. Hunter's Mark has a 100 yard range, so it's easy to keep a rogue in check from across the battlefield.
Stuns and incapacitate effects
Another reason why rogues are so fearsome in PvP is because of their vaunted stuns. They can keep an opponent stun-locked or incapacitated in a number of ways so as to effectively neutralize a threat if not kill them outright. This control is why rogues have so much synergy with priests and mages in the Arena. Rogues can stun as an opener from stealth with Cheap Shot, which they'll use mostly against other melee classes. If you can manage it, don't use a cc-break on this because they can easily follow it up with a Kidney Shot as Cheap Shot already awards 2 combo points from the get go.
Basically, expect a rogue to be able to stun you right out from stealth as well as during combat. For the most bang for the buck, rogues will land a few strikes to generate combo points and will wait for one stun to wear off or nearly wear off before applying another one. Because they can apply so much cc, it becomes discretionary when to use your trinket or other stun prevention or removal ability.
In addition to their stuns, rogues also have a pair of incapacitate effects, Sap and Gouge, which take an opponent out of play for a while. Because Sap can only be used on targets that are not in combat and doesn't break their stealth, they can use this to incapacitate one opponent and attack another. Quite often, rogues Sap their opponents to unsettle them and is a very common occurence in world PvP even as a prank when rogues don't necessarily intend to battle their opponents. Sap is a common tool used in the Battlegrounds particularly for capturing the flag against a lone defender. In these cases, it's usually a good idea to break the Sap with a trinket or, in the case of warriors against clueless rogues, Berserker Rage.
Gouge is an interesting ability. It is a quick way for rogues to escape combat and is employed cleverly as an interrupt in a pinch. It is a 4 second incapacitate effect and is usually not worth breaking because it breaks on damage, anyway. Because it requires that their opponent be facing them, rogues often also use this to maneuver behind their opponent. It has a fairly short, 10-second cooldown, making it an extremely potent tool in controlling the fight. The good news is that it is extremely expensive at 45 Energy, so rogues won't always be able to use it even when it's available.
The reason rogues are a caster's bane is because they have so many tools at their disposal to prevent spellcasting. Kick doesn't just interrupt, it also locks casters out of the school for 5 seconds. For a caster, that's about an eternity. You need to remember two things as a caster -- Kick is on a short, 10-second cooldown and it costs 25 Energy. Because of the short cooldown, you should always assume they can use it at any time and it won't matter too much to track it through AddOns. Keep the Energy cost in mind and always assume that a rogue with about a quarter of her Energy left can use it.
The best way to deal with a Kick is to bait it out of the rogue. This means you'll need to keep juking your spells. A /stopcasting macro before every spell allows you to keep canceling spells that are currently being cast instead of stacking it. This should also remove the need to move to break a cast, which is less efficient. One trick is to keep juking with a spell from another, less-important school of magic so that if the rogue bites and uses Kick, you're free to cast spells from another school. Healers have a tougher time here mostly because smart rogues will use Kick on Holy or Nature school spells only.
If a rogue manages to Kick you and you're locked out of your primary (or in the case of paladins, your only) school, you can go two ways -- act as defensively as possible or go all out on offense. When a rogue has locked you out of casting spells, especially heals, they will go all out on damage without even bothering with a stun knowing that you have few options. Fortunately for priests, Psychic Scream is from the Shadow school, so getting Kicked in the middle of a heal means it's probably a good time to cast it. It's also a good idea to go on the offensive by accessing attacks from other schools during this time.