A few things to consider:
It is very difficult to manage a fishery even if you know most/all of the information of a body of water. It is down right impossible from an individual's catch successes and memories.
In very general terms it would be more of a concern if there was a preponderance of small fish in a body of water, because it might be an indicator of a "stunted" population. One factor that can lead to stunted fish is a lack of predation, namely from larger fish.
"Small" fish are not necessarily young fish, and sometimes "big" fish are not the oldest fish.
Over time, lakes will change due to a number of factors. Often these changes are result of humans, but not always. Sometimes we just speed up the aging process of a lake.
Fish populations and the "demographics" of those populations will cycle as well. Without hard data its hard to say whether there is a change that you should be concerned about, or maybe you just had some good fishin'.
The MNR might have some data on your lake.
Come on spring.