These instruments can be played in one of six positions: bass, chord, two percussions, harmony, and lead melody. The last three are considered "lead" positions, which set the rhythm for the other three. Some of the positions share instruments (chord and lead melody both have the melodic instruments, for example).
And then everybody shakes their controllers together, doing their best to adhere to a rhythm prescribed by some bouncing things on screen. But it's kind of hard to tell with something like the banjo just what the right rhythm is supposed to be, and in the cacophony of other people swinging their Wiimotes around randomly, it's pretty hard to figure out. Once I've had some more time to mess with it, maybe. But, then, with no penalty for missing rhythm, it basically doesn't matter as long as you don't mind sounding awful.
As it is right now, you try to do what looks like the motion for an instrument, and it sort of plays the song. The rhythm might be accurate, or it might not.
The full drum kit thing with the Balance Board is a separate mode entirely. The Balance Board controls the bass drum and a cymbal, with the other drums controlled by the Wiimote and Nunchuk. In order to play different drums, you have to hold a button on a controller and then drum. It's not really an accurate representation of drums, but it's fun. The crazier you get, the more stuff lights up on screen!