While you can root these files out yourself, it's sometimes far from obvious where they are. To resolve this, a new class of uninstallers have appeared on the scene to address the problem. One of the first was AppZapper ($12.95), but many more have arisen like AppDelete (donations requested), SuperPop ($15), and CleanApp ($10). With most of these you choose the application you want deleted by dropping it on the uninstaller (or choosing it from a list); hit a button and the application plus its support files are sent to the trash. Taking a slightly different approach, Yank ($19.95) actually monitors your system while you are installing applications and creates a "Yank File" that records the location of everything created by the application. You then uninstall by running the Yank File.
Finally, there are uninstallers that integrate with the Trash itself, requiring no separate interface. Into this class fall Hazel ($21.95), which also does other automatic file management tasks, and AppTrap (donations requested). What's great about these last two is that you can just drag an application to the trash and they will automatically look for and offer to delete the support files. Indeed, this seems to me to be the most Mac-like and elegant approach. For my own part I mostly just use Hazel, though I still throw things at AppZapper now and again.