Get a decent microphone
You need a good microphone. There's no arguing with this. It doesn't matter if you're starting your own show or planning to guest on someone else's podcast. A great microphone will elevate your voice to help you get the sort of depth and richness you hear on the radio and popular shows like Radiolab. While you could record with your phone or your PC's webcam mic in a pinch, nobody wants to hear that every week.
Your first step should be nabbing a solid USB condenser microphone. They can connect easily to any computer (or even phones and tablets with a dongle), and they'll offer a huge leap in sound quality. I've run through many microphones in my 12 years of podcasting, and few have impressed me as much as the Blue Yeti ($116). It sounds fantastic for the price, and it's very versatile. You can switch between modes for recording on your own, interviewing someone across from you, stereo capture and omnidirectional pickup. You'll only need those first two modes for podcasting, but it's nice to have the option for different scenarios.
There are cheaper USB microphones out there, like Blue's Snowball ($68) and AmazonBasics' Mini Condenser ($45), but generally you'll get far better sound from the Yeti. There's also the slightly stripped down Yeti Nano ($84), but it lags behind its larger sibling when it comes to audio quality. If you're serious about podcasting, it's worth spending a bit more up front: There's a good chance you'll end up chucking a cheaper mic once you hear the difference.
Pro tip: RTFM
You should actually read the instructions and make sure you know what every dial and button does. Most important, make sure you're speaking in the right direction! With most microphones, including the Blue Yeti, you want to aim at the side with the brand label. Some models, especially dynamic mics, need to be addressed from the top. Yes, I know this all sounds basic, but I've run across dozens of people who end up aiming for the wrong part of their mics when they're getting started.
It's also worth picking up a few accessories to make your recordings sound great. Get a pop filter or foam cover to avoid plosives (that annoying titutal pop when you make "p" sounds). If you're going to be recording regularly, it's worth investing in a tabletop arm to hold your mic in an optimal position (and also avoid the extra noise you get from desktop stands).
You could, of course, start exploring more-expensive microphone options, but I'd suggest holding off on those until you're more committed to the podcasting life. The next big level up from USB condensers is the world of XLR microphones, the same interface used for professional audio gear. You'll also need a USB audio interface, like the Tascam US-2x2 ($137) or FocusRite Scarlett 2i2 ($160), to connect those mics to your computer. At that point, you can start looking at options like the Rode Procaster ($228). It sounds noticeably richer than the Yeti, and since it's a dynamic microphone, it's also better at cutting out unwanted noise than a condenser.
Here's some advice: You can save quite a bit by buying all this equipment used or refurbished. I saved $100 on the excellent Shure PG42 USB microphone by going through eBay.