TMH, I also spend alot of time fishing solo and am as guilty as anyone for not wearing a PFD, but last August I was alone in the boat while casting in some pretty rough water up on the West Arm of Nippissing. There wasn't another boat around for miles and even the cottages are pretty sparce along that north shore.
I realized how stupid I was being and put a jacket on, and it was funny, but I immediately felt much more secure and made a promise to myself right then that I'd start wearing it alot more when I'm solo, specially on those big, relatively empty lakes.