Since your pet is a class-defining component, there's a few things you need to know about the care and feeding of it.
Pets have a stat called Happiness. If the pet is at maximum happiness (green smiley face), it does 125% of normal damage and gains loyalty. The middle state is generally happy (yellow smiley face), but does 100% of damage (so, no bonuses or negatives). The lowest state is unhappy (red frown face) and the pet does 75% of damage and loses loyalty. You raise your pet's happiness level simply by feeding it. So, in addition to the food you need to carry around for your consumption, you'll need to carry pet food as well. The rule of thumb I usually use is to double the amount of food I think I'll need to eat.
Your pet's Happiness level also affects its loyalty rating, but that's really only a factor after you've first tamed it. There are six loyalty levels and pretty much just playing the game for a few play sessions will max the loyalty level. The big thing with loyalty levels is it affects how many skill points your pet has. Pet skill points are used to increase its armor, hit points, special attacks, and resistances. A pet's loyalty rating won't see as much fluctuation as its happiness; once you've got it to level 6, it'll take a dedicated effort on your part to drop it back down to level 5.
Pets can be broken down into two categories: tanking pets and DPS pets. Petopia has a good breakdown of pets here. As you are leveling up, don't worry too much about a DPS pet. A good tanking pet like a boar will be all you need until you hit level 70. At that point, go and train the highest-level DPS pet you can find for instance runs.
Pets also have three states they can be in: aggressive, defensive, passive. Aggressive tells it to attack any aggressive monster that comes close enough. Defensive tells it to attack any target that attacks you or it. Passive tells it to only attack on your command. I don't recommend using aggressive. Defensive is good for when you are our farming or leveling. If you are running an instance, I recommend leaving the pet on passive. That'll avoid any unexpected "oopsies" when the pet runs off and attacks when you'd prefer it doesn't.
Also, all pets have an ability called Growl. Growl lets the pet rise quickly on the threat meter. This is great when you are soloing, but when you are running an instance, you need to turn this off. Otherwise it's very likely your pet will out-aggro the tank, and that's all kinds of bad news.
As you're leveling, questing, and running instances you'll be upgrading your gear. When you're trying to figure out of a piece of gear is good for you, the two ability bonuses you should look for are +agi and +sta. Those will affect your overall attack power and hit points. Since your pet gains most of its power from your stats, you'll also be helping it out. There's a strong argument to be made that during the awkward levels of 40-50, don't worry if if the gear is leather or mail; the + agi/stamina bonuses should win out over any other stat. Once you get to level 70 you'll also want to add in +hit and +crit to that factoring, but don't worry about that until you get to level 70.
There's an argument in the comments section that you should also focus on INT items, but for new players I'm still going to recommend you focus on AGI and STA items, with AGI being the more important. If you want to juggle iun a third stat to think about, I recommend the +Attack Power stat.
There are three talent trees you can spent talent points on: Beast Mastery, Marksmanship, and Survival.
Beast Mastery is currently the most popular talent tree for Hunters. As you'd expect, it's heavily based on increasing your pet's abilities. Two of the big bonuses are Improved Aspect of the Hawk, in which all normal ranged attack speeds are increased by 15% and Ferocious inspiration, when your pet scores a critical, all party members have their damage increased by 1-3%. Since roughly 25% of your damage will come from your pet, keeping him alive is a must.
Marksmanship used to be the the Flavor of the Month talent choice until Burning Crusade launched. It's good for burst damage and is popular in PvP. Right now, the cookie cutter talent template has 41 points spent in Beast Mastery and 20 in Marksmanship, up to the Mortal Shots ability.
Survival isn't as popular, but is great for groups. This talent tree affects the hunter's traps, particularly in how may he or she can have out and the cool downs on it. These bonuses come at the expense of a Hunter's offensive capabilities. It's also very dependent on your agility, so you'll want that to be as high as possible -- over 600. Because of the high AGI requirement, Survival is more effective at the end-game.
Ok, this has gotten a little too high end (but knowing what your role is at level 70 is good to always keep in mind), so I'll bring it back to the newbie level by mentioning what the milestone levels are for hunters:
Level 10: You can train your first pet at this level after you complete a quick quest.
Level 30: Hunters can learn Feign Death. More on that in Threat Management.
Level 40: Hunters can now wear mail armor
Level 62: Hunters gain the Steady Shot skill, which will form the basis for many shot macros
Level 70: Hunters gain the Misdirection ability. Simply put, the aggro from your next three attacks go towards the target you've cast this on. This is great for those pulls when the tank needs to be in a certain spot. Cast MD on the tank and pull. The mob goes to the tank and not you.
Trapping, Crowd Control, and Threat Management
Remember earlier I mentioned in groups they'll want you to do crowd control with your traps? Learning to trap properly is one of those skills that falls in the "I wish I learned how to do this properly before I had to" category. I'd waste some virtual air explaining the basics, but BigRedKitty (not only is he a WoW Insider columnist, but he runs a pretty darn good site of his own) does a great job of explaining chain trapping here.
Now that you're back from reading that, I'll reiterate the point that managing your trap cool downs is the key to success. It'll also be a big source of your frustration. If the tank pulling the instance you're in isn't that familiar with the concept of traps and cool downs, you're likely going to have to take him aside and mention that marking that you want a mob trapped last and then immediately pulling is going to make for a rough go, especially if the trap gets resisted. Naturally, when that happens the tank turns and gives you a "WTF?!" look. Thankfully, those moments will happen less as you become familiar with the instance, and, if you group with guild members often you'll get used to how certain tanks pull.
I can't repeat this enough: learning to trap properly is one of the Keys to Being a Good Hunter. Learn to trap well and you'll get frequent re-invites to groups. One of the downsides to being a Hunter is since it's so easy to solo all the away to level 70, a lot of hunters have anti-social tendencies when he hit that hallowed level - we've never needed groups until that point, and now that the level 70 game is all about instance runs and raids, it's a hellova culture shock. Another Key to Being a Good Hunter is grouping early and often. Even if it means a slower pace to level 70, you'll need those skills. So group often, young grasshopper.
The third Key to Being a Good Hunter is threat management. I mentioned Omen earlier when I talked about making sure you don't out-aggro your pet. You'll also be using that add on to make sure you don't draw aggro from your tank. Too many Hunters turn on the Bow of Chain-gunning and try and make sure they are always at the top of the DPS charts, often pulling the mob off the tank in the process. Said Hunter usually stands there looking at the tank going, "Don't look at me, dude, if you knew how to hold aggro we wouldn't have wiped." Don't be a Bad Hunter -- use an addon like Omen to make sure your name is never the highest on the threat meter. However, occasionally things go bad and you draw aggro. That's where Feign Death comes into play. Assuming it's not resisted, the mob will think you are dead and ignore you. This is great if you are soloing and get in over your head, or are running an instance and need to shake aggro, or when you're the last person standing during a wipe and you want to save the repair cost a resurrection.
In fact, lets talk about the DPS charts. You can use an add on like Recount to see how your DPS compares to the rest of the party. Speaking from experience, it's real easy to invest too much energy into trying to be on top, and getting miffed when you aren't on the top. Don't get too hung up on it. Nowadays, I just use the numbers to see how gear or macro adjustments affect my overall DPS. Also, if you're busy trapping, ignore those numbers altogether; trapping will make a serious impact to your DPS numbers, but drive up your "contributing to the group's welfare" numbers.
The Hunter is a pretty good all-around class. You can solo great, do bring something to groups, even though that skill (trapping) requires practice to learn and maintain. However, like most all-around classes you may have to fight for group spots since there are classes that excel at some of your abilities.
[edits on 5/31: added in feign info, pet stances, survival clarification, slight info on INT[