I can't even begin to tell you how many times I've tried that. It's really frustrating. Sometimes it keeps us from watching almost anything, but then today, we had one 30 second buffer and then watched 4 episodes of a show in a row with no issue.
Really just seems like a bug somewhere or some network issue, but it persists amongst different players on AppleTV. We're using Infuxe, Plex, & Catch and they all do the same thing.