Exactly how our neigbour across the channel paid for their place over the last 5 or 6 years. They take about 3 weeks holidays and rent it out the rest of the year at around $800 a week from late May until September.
Personally I said I'd never own a cottage if all I had was a couple week holidays a year. The money it costs for taxes/upkeep etc, let alone the initial cost could put you in some pretty nice lodges/getaways instead.
If you want to approach it as a rental investment... remember you, or someone you will pay, will have to be there EVERY saturday to see the renters off in the morning and bring the new batch in... in the afternoon.. clean the place in between. If you have no issues with being tied to it that way and you figure rental will cover a good part, or all, of the mortgage (if going that route) then why not. If it's on a lake that you're not overly stuck on staying on in future years move in to the place for a year when you are about to sell, so it's now your principal residence, and avoid the captial gain at sale time!