It means they're smart and aint falling for that ol' "hook in the tasty treat" gag, they've been caught before and know what you're doing!
Try sneaking up on them, stay low and wear camo... try live worms!!!
You're catching the smaller, stupid, more aggressive fish... but they will soon learn as well!