Yes, you can remove the core and leave the fenders attached. I did, no prob. I was told the fenders would sag so I thought about putting some kind of brace across from one side to the other, but in the end just removed it. Like you the exterior had fresh paint and I didn't want to take a single screw out.

I feel for you about the accidental damage. It just turned spring here yesterday, I will sit in the sun and help you drown your sorrows by downing a couple cold ones.