If your amp is wired directly to the battery without a relay, then yeah, you are most likely killing your own battery. All those charge cycles strip lifespan out of your battery.

Also check the voltage at the output lug of your alternator with your fan and amp powered on to ensure it has enough output to charge your battery under load.

I have a NAPA Ultimate with a Delco topper.