Wacky is fine. Done well, wacky is awesome. I had a Draenei Mage in World of Warcraft who was uniformly wacky, a character clearly meant to be played for laughs rather than pathos. (Think Pinkie Pie with a doctorate and you're most of the way there, although this was years before I gave Pinkie Pie any thought.) But they're two different flavors, and playing a character in the throes of madness as being a font of zany antics is like using a hammer to whittle.
People in the midst of madness will do things that seem random, yes, but the randomness should be disturbing rather than funny. Your madman won't show up at a party and slap someone with a fish, but he might arrive with a fish sewn to his clothes. He might even talk to it, coo at it, ask it if it's all right in a soft voice as the thing desperately flaps and gasps for air, small bloodstains forming where the thread has weaved between creature and garment.
That is the sort of image you're going for with madness. It should provoke a sense of crawling unpleasantness, something that makes you uncomfortable in the pit of your stomach. And there are tricks, I've found, to getting that sense across without resorting to, well, antics.
Processing a broken world: The important distinction to draw here is that a mad character still has a set of rules for how the world works. It's just that those rules are broken, or in some cases, outright wrong. There is a certain logical flow, it's just filtered through a distorted lens and includes a lot of steps that a more sane person wouldn't see as being even remotely related.
The person sewing a fish to a shirt, for example, would think that he was doing something perfectly reasonable. And if you could follow along in his delusions, it would be perfectly reasonable. He's not doing it for shock value, and the fact that it shocks others just leads to his feeling persecuted or beset by enemies or trying to explain through broken logic. But never mistake the mad for people who do things without reasons. They do them for reasons that make no sense to an outside of observer, but there are reasons.
Break taboos in subtle ways: Usually, a mad character who breaks a taboo does so in an obvious way, most often involving murder. But it's equally disturbing to have your character follow others around far too closely and just laugh perpetually. Or sit down and slowly eat his own shoe. Or just burst into tears at the mention of rain.
Normal social interactions are built on a lot of rules, and breaking these little rules says far more about the character than just going out on a roaring rampage of dismemberment. Sure, your characters have never talked about personal space before, but it will become an issue if someone taken by madness walks up and sticks her finger in your mouth without prompting.
Call back: It's never a bad idea to have your maniac do something related to the origin of his madness. If he's from a long line of maniacs, have him wear ancestral clothing, possibly the burial shrouds of departed relatives. If she knows something that broke her mind into pieces, have her try to establish elaborate mathematical formulas for proving that five equals cow minus ambiance.
This is effective because it teases at the idea that maybe something can be done to fix this person, that maybe there's some logic behind everything. There isn't, but it's a red herring that works.
Moments of lucidity: River Tam is an excellent example of a mad character in fiction who worked, being equal parts disturbing, tragic, and dangerous. But part of what sold her character wasn't the fact that she would spout bizarre non sequitors or scream at absolutely nothing. It was the fact that every so often she said something that sounded mad but was, in fact, perfectly reasonable.
If a mad character is meant to disturb others, part of that disturbance comes from the idea that the madwoman is not so far gone as she might seem. And the clearest way to show that is to clarify that every so often she either sees with perfect clarity or with even more clarity than would seem possible. It forces others to ask some uncomfortable questions that don't offer any corresponding answers.
As always, feedback is welcome in the comments below or via mail to firstname.lastname@example.org. Next week, I'm going to talk about the delusions of our characters, and the week after that I want to talk about the insights to be gained from playing one character time and again.
Every Friday, Eliot Lefebvre fills a column up with excellent advice on investing money, writing award-winning novels, and being elected to public office. Then he removes all of that, and you're left with Storyboard, which focuses on roleplaying in MMOs. It won't help you get elected, but it will help you pretend you did.