We had that happen to us twice now, and yes, it is scary and disorienting.
First time was on the lower Niagara, we were doing our drift and could not see either side of the shoreline, Canada or US. We figured we were heading back to the ramp and then we almost hit shore and realized we were on the total opposite side, the US side.
The second time was up at Nipissing and we were heading from Lakair to Muskrat Bay. We could not see a thing and went slow as molasses, watching for shorelines and rocks. Since we've done that trip a million times, we made out okay, but had we not done it so many times I think we probably would have aborted the mission.
Glad you made it back to shore safe and sound. It sure is an eerie feeling when the fog is that thick.