Would going into Physics be a good idea?

I like physics. I really like physics. I think I'd enjoy working as a physicist more than I'd enjoy working as anything else. But I wouldn't say that I'm ... pionate about it, as such, and so I can't just throw myself into it without considering anything else. What would my chances of finding a moderately well-paying job in physics (or something related, at least) that wouldn't require my woking really long hours with a BSc be? With a MSc? Most of the job prospects sites I've visited say that the prospects for physicists are either good or average, but in reality it seems to me that finding a job would actually be pretty hard, and too many physicists are either unemployed or forces to find a job unrelated to their area of study. So what's the reality? Anyone here a physicist? Or know any physicists? Any sort of input or advise would be muchly appreciated.