The problem you're describing could be a result of several root-causes. The fellas have already hit on 2 of them, but I'll summarize just for fun:

1. Battery de-charging on its own
2. High resistance short somewhere
3. Charging system not doing its job

Ruling out #1 is easiest by just swapping in another (known, good) battery. I'm concerned that your current battery may already be in a state where it's not holding a charge so it could skew your results by just disconnecting it.

Possibility #2 was covered beautifully by ragtop above. With his excellent instructions, you will be able to do a high-level diagnosis of your various circuits to determine if one of them is the culprit. You'll need to get super-acquainted with your multi-meter to home in on the precise circuit as the fuse-panel trick will only reduce it down to several smaller suspects.

Which leaves option #3. If your charging system isn't working at capacity, this could have the effect of "draining" your battery with use. An easy test is to start the car, and then remove the positive terminal while the engine is running. If you notice any change in operation (like the lights dimming or the engine stopping), your charging system is suspect. You can have your alternator tested easy enough, for example.

Best of luck in your diagnosis.