Do video drivers slow down over time?

Whenever I read about performance problems in Windows, someone always recommends that you should upgrade your "Video Drivers".  What confuses me about that statement, is that people usually ask about performance issues when they notice degradation   That means the system used to run faster, and it is now running noticeably slower.  This begs the question, why would updating your video drivers make the system faster?  I get the fact that newer drivers could give you better performance out of your video card, but that won't help if you are suffering degradation.  It won't reclaim the speed that you have lost.  Making this statement seems to imply that drivers degrade in performance over time.  It makes me feel like my video card is a car and upgrading drivers is giving it an oil change.