Marcus Aurelius says we shouldn't be bored, because we can always retreat into our own mind. But we are not all philosopher-kings; there are certain things that we must listen to even though we don't want to. Consider two case studies:
1) A student who must pay attention in class even though he knows the majority of the material. The trouble with spacing out, he figures, is that the teacher may at any time say something he doesn't know and he wouldn't want to miss it.
2) When politeness dictates that you must listen to a full story even though we can already see where it is going, or experience tells us that the punch line will be a let-down.
I think we can overcome much of our boredom in both of these circumstances. In the second case, remember that politeness is overrated--if you are bored with a story then your fellow interlocutor may in fact be bored in telling it. Don't be afraid to interrupt if you think of something more interesting to discuss. Cutting off the convo early can be mean, but moving it in a different direction can be a net positive.
In the first case, I think we overrate both our own memory and the potentially utility of whatever the upcoming fact may be. If you space out you probably will not miss a "eureka" moment of clarity, as these are uncommon in the real world. Many subjects are simply a collection of small facts that add up to a coherent whole. If you don't care much about the coherent whole, there's no need to trouble yourself with the small facts. Space out more, feel less guilty about it, and you'll end up more productive.
(Full disclosure: I wrote this post word-for-word while sitting through a physics lecture in Sanders late this afternoon.)