Force quit the front-most application: If you are using OS X 10.5 (Leopard) or later, Apple suggests that you press Option-Shift-Command-Esc and hold down these keys for three seconds.
kill(1) it: if "Force Quit" doesn't work, and you're comfortable using Terminal (the application is in your Utilities folder), you can find the PID (Process ID) and use "kill" to force the app to quit. With a name like "kill" you ought to be reticent to use it, and if you decide to use it, do so with care. If you use kill, don't expect to get a chance to save your data. That said, if the app has been frozen for some time, it's unlikely that you're getting your unsaved data back.
To find a PID for an application, use this line in the Terminal application (Note that I'm using Safari in this example, but you can do the same for any app, just replace "SAFARI.APP" with whatever app you are trying to find):
ps ux | fgrep -i SAFARI.APP | fgrep -v fgrep
you will see a line that looks something like this (note that "tjluoma" is my login name, yours is almost certainly different):
tjluoma 830 0.0 5.7 5219788 239908 ?? S 10:20AM 0:45.82 /Applications/Safari.app/Contents/MacOS/Safari -psn_0_307275
There are only two things you really care about here: the first is the path "/Applications/Safari.app/Contents/MacOS/Safari" which tells you that you have the right process; the second is the number immediately following the login name, which is, in this case "830" which is the "process ID" or "PID" for Safari. Every process on your computer has a PID, and it will be different every time for all but a few very low-level system processes. (I will use "830" in the examples below, but you should be sure to use whichever PID matches the app you want to quit. Failure to do so could lead to Bad Things.)
Next you have to decide which of kill's tricks you want to try. There are many of them, but I generally only use 3:
kill -QUIT [pid] tells the app "Hey, I want you out of here." Unfortunately this probably won't work if the app is "Not Responding" but it doesn't hurt to try. To use the Safari example, I would enter this in Terminal:
kill -QUIT 830
Note: You can also use "kill -3 830" instead, but I find "-QUIT" easier to remember.
After that I try "kill -TERM" as sort of a "No, really, I mean it, get your stuff and get out, now."
kill -TERM 830
Note: you can also use "kill -15 830" instead, but I find "-TERM" easier to remember.
If neither of those work you are left with only two options: reboot the system (which is almost always an option) or "kill -9". Let me make this clear: "kill -9" is the last resort. "kill -9" is putting a gun to the forehead of the app and pulling the trigger. You know how I've mentioned that you can use names or numbers ("kill -15" or "kill -TERM")? The name for "kill -9" is "kill -KILL" just so you know we're not messing around. Think of "kill -9" as your last defense against the zombie apocalypse.
kill -9 830
or (kill -KILL 830) should send your process to process heaven immediately.
"But what if my entire computer is not responding?"
The above will work fine if you are trying to target just one application, but what if you get into a situation where the entire computer won't respond. My old PowerBook would often get into a situation where I could move the mouse, but nothing else. Your choices are fairly limited at this point.
If you don't have another computer available, your best bet is to hold down the power button until the computer shuts off.
If you do have another computer available -- and let me be clear that in this case, "computer" includes and iPad, iPhone, or iPod touch -- you might have another option available. However, to use this option, you have to have a little foresight and setup your Mac to let you login remotely.
Option 1: Screen Sharing or VNC: I mention this first because it's the easier option, but frankly I wouldn't expect it to help in these situations. If the computer is frozen, accessing it via Screen Sharing or VNC is just going to let you see the same problem on a different screen. But, again, it doesn't hurt to try.
Option 2: Remote Login (aka "ssh"): I know this option is likely to work because it has often worked for me. As with Screen Sharing, you have to turn it on via System Preferences » Sharing. Check the box next to "Remote Login" which is what Apple calls "ssh" probably because "Remote Login" is fairly obvious to anyone who can read, whereas "ssh" is only obvious to a few. In Terminal.app, go to Shell » New Remote Connection and click on "Secure Shell (ssh)" and select the computer you want to connect to. Once you are connected, you have two more options:
1) Logout from the commandline:
sudo killall -HUP WindowServer
You will be asked to enter your administrator password. This will have the same effect as logging you out. After that you can either reboot or log back in. Note that this will not save any unsaved work.
2) Reboot from the commandline:
sudo shutdown -r now
You will be asked to enter your administrator password. This will not save any unsaved work. The computer will shutdown whatever processes it can, and then reboot the system.
No one ever wants to find themselves in this situation, but if you do, it's good to know what options you have.