There is a trick to pulling it out. You need to press the button on the side of the switch. It should go back in with no problem. You could try pressing in the button on the side. Maybe you broke something internally if you removed it without pressing the button?
