I would say that you certainly could cut the hole without removing the panel but I don't know that you would be best served doing that. I removed one kick panel (passenger) and honestly it wasn't that bad. There are some templates you can find with a little research that show exactly where to cut. I suppose depending on your speaker you can adjust as necessary but you will want to be very close to that. Having the room behind the panel is important.

While I have no personal experience with the aftermarket kick panels designed for speakers I have heard that fitment was poor.