This isn't justifying not wearing one, but I like what my brother does.
He constantly practices, flipping, falling out and getting back in his canoe, spring, summer, fall. I think he normally wears a PFD, but regardless, he's more prepared than most if his canoe should actually flip over. Also he's taken recent swimming lessons.