I'm not really going to speak to the safety of swimming in the surf in cold water wearing waders, but I want to share my experience as I expect a lot of kayak anglers are wearing waders and going through the surf.
First of all I always wear a life jacket with waders. My most serious paddling injury happened when I slipped on a rock, banged my head and ended up in the water. Had I been knocked unconscious I could have easily drowned in knee deep water. So I'll be the nerdy looking guy wearing a life vest whenever I'm on the water. The life vest also serves as a wader belt limiting the amount of water that can scoop into any set of waders.
Secondly, I choose neoprene waders for their extra padding and buoyancy. They also provide needed padding to protect from bruises when you bang into rocks or boats. I want these waders to be tight, about as tight as a wet suit. I can wear socks and thin pants under them and a rash guard but that is all. Waders that are not form fitting inhibit swimming well and have room for more water in them. Some Waders I have can hold about a gallon of water in each foot because they seem to make the foot and ankle area too big on waders. It's a pretty funny sight waking on the beach with this water in my feet!
More thoughts will follow on this topic....