I use Firefox as my default browser, and have been for a while.
As far as security goes Firefox has also had it's share of faults and patches (The Register, Friday), however Mozilla has a better reputation than Microsoft for fixing them. Also (at the moment) people are targeting IE holes, AFAIK any Firefox holes that have been found so far are more "proof-of-concept" than targeted attacks against users.
So even with Firefox you still need to keep everything patched and up-to-date to protect yourself from nasties- and even then be wary of clicking dodgy looking things!
In terms of using secure sites (any beginning with "https://") Firefox uses the same level of encryption as IE, as both adhere to the internet standards on security. It's worth testing with your bank though, the one that I use didn't directly support Firefox initialy, although it does now.
In terms of page-rendering and compliance Firefox is more standards compliant than IE and does render pages faster than IE - although you'd be hard pressed to really notice the difference. For various reasons (IE only optimisations, sloppy coding, non standard HTML (IE is more forgiving of coding mistakes)) some sites won't work with Firefox - although these are getting less and less all the time. I do my browsing in Firefox and open any broken pages directly in IE.
That brings me to one of the great benefits of Firefox - extensions. There's a fantastic growing collection of add-ons that increase the usability of Firefox, so you can tailor it to your needs. One of the ones I use adds a "View this page in IE" option to right-click menus- so if you do come across a site that refuses to work in Firefox you can throw it towards IE to deal with. Another great one is a spell-checker which is useful for posting on MoDaCo!
I would heartily recommend Firefox - it's worth playing with. After all it's free, and you'll still have IE to fall back on if Firefox is not for you.