I feel the same about changing the whole panel I am doing just that and I feel it will be a better job in the long run, for just a daily beater patch panels will work but for those of us that spend days just trying to get the gaps exact why not do it right the first time. I see that as a diamond in the rough but it wouldn't scare me to undertake it, some of us love a challenge.
Good luck and remember when you start to get frustrated just walk away it will still be there the next day.