If you're content with a four- or five-year-old iMac you probably don't need extreme performance, and in that case a Mac mini might be perfectly fine. I have one, and it works very well for me.
That said, an iMac looks far slicker, and is slicker in some ways. For example, when I put my Mac mini to sleep I have to switch off my display manually at the same time as clicking "Sleep" in the Apple menu. If the computer is already sleeping when I switch off the display, the press of the display's power button wakes it up again. And if I switch off the display before sleeping the computer, I can't see where to click "Sleep".
Little things like that make the iMac a more integrated, hassle-free solution.
Whether a TV would work well as a computer display is another matter. I suspect it would be a nightmare to calibrate and colour profile!