Maybe you really are not sufficiently productive in your life, and down-time feels bad because you have an excess of it.