Gedeon Maheux has an post about how an icon is designed from the aptly-named Iconfactory (specifically the Frenzic app icon), and while I don't have any design skills to speak of at all, it's an interesting look into how those little snippets of art are made.
Icons are increasingly important, especially on the iPhone and OSX, as Apple leverages icons more and more in their interfaces, from the Dock to CoverFlow to the iPhone's home screen. He sounds a little sheepish that they obsess over icons, but why shouldn't they -- every time you decide whether or not to buy or use an app (some might say the most important part of an application's lifeline), you're likely looking right at the icon.
There are a few things to take away here: they designed the icon not by itself, but right alongside all the other icons on an average iPhone screen -- in context, where people would see it. And they walked a thin line: while they wanted it to stand out as something you'd chose even among the apps you've already purchased, they didn't want it to be so bright or flashy that it broke the UI. Truth be told, Frenzic's icon still looks a little bright to me, but the lesson is good: the UI comes first.
Ged closes with a sentiment I've agreed with for a while: while there's a disturbing trend of adding "On Sale" or "New!" to icons in the App Store (not to call anyone out, but ahem) that doesn't serve the developers or their customers. An icon, just like your app, should be subtle and simple and beautiful. After all, isn't that why we're all using Macs in the first place?