Is the PVC valve in the valley pan?; Is the PCV system fresh air supply hooked up at the air cleaner?
One needs to have air-in and air-out to ventilate the crankcase positively. With a non-positive system, like we all had in the old days, the crankcase just vented through the oil filler tube and usually had a breather tube running down beside the block to drain any oil, moisture and smoke in the crankcase fumes.
A PCV system needs an air-in supply port and a positive air-out source. On most of our FGFs the air-in was at the air cleaner. There is a pipe sticking out of the air cleaner with a filter inside the cleaner case, a tube is attached to the pipe and goes to one of the rocker covers to feed air into the crankcase. To positively exhaust the air from the crankcase the exhaust tube was connected to a vacuum source which fed the air, fumes and moisture into the intake to be burned along with the air/fuel charge. (see the beautiful attached diagram made at great sacrifice and expense) The PCV valve was usually attached at the valley pan and the connected hose hooked up to the manifold vacuum at the carburetor or the intake manifold. The idle mixture was adjusted to compensate for the extra air entering the intake through the PCV system.
When you added a breather to the rocker cover the idle got rough? That would suggest to me the air-in source at the air cleaner may be plugged and the extra air entering the added breather and exiting into the intake is causing a lean mixture. You would have to adjust the idle mixture and check the air-in source.
If the PCV system worked well before you moved the air-out to the carburetor from the manifold, would suggest to me the vacuum port the system is now connected to is not sufficient to evacuate the crankcase correctly, I would suggest removing the PCV valve hose from the carburetor and teeing it into the manifold vacuum port.
The PCV system doesn't necessarily have to be connected in the manner it was when it came from factory, but does have to have filtered air-in and PCV valve controlled vacuum sourced air-out. I have mine set up with a filtered breather on both rocker covers, the PCV valve in the valley pan with the exhaust hose hooked up to a manifold vacuum port on the carburetor. Don't have a good pic but one that shows the cheapo rocker vents and the line from the valley to the carb. They make filtered vents for the push in and the twist in rocker cover caps.
What is the model of your Holley? Some of them had a pipe thread plug at the back of the base plate that can be removed and a pipe thread nipple be turned in for a vacuum source.
there may also be a pipe thread plug in the intake behind the carburetor, if so it could be removed and a nipple installed to power your brakes.