The problem is from a bug called Acanthamoeba that is present in tap and swimming pool water that isnt killed by chlorine instead forming a cyst. Potentially the bug is a threat to everyone whether they wear lenses or not. The risk is however greater for lens wearers as the bug needs a way into the eye through a break in the epithelium and contact lens wearers often have a mild abrasion or sub acute corneal damage that provides this break and entry route.
Add in the fact that if you swim in the lenses the water can get behind the lens and hold the bug in contact with the eye so maximising exposure.