I mix it up every cast, long pauses, short pauses, steady retrieve, twitching, teasing the cat style. lol.. it all works.
As far as colors, fisherman are to concerned with the looks of the bait when the action of the lure is 95% of attracting a fish, I use natural patterns in clear water, brighter colors in stained/dirty water, that's how I look at it, I try to keep it simple.
Maybe your luck just stinks.