There are some great posts on this topic already, and more arguments for and against than you can shake a stick at. I’ll link to some external sources in a moment, but here are my own thoughts.
It all depends on two things: first, your own abilities and desire to manage your own process. And, second, how marketable your book is.
“Marketable” isn’t just how well it’s written and produced, although both of those are critical. The other factor is just how much demand there is for this particular product. A book which has 2 million potential purchasers is a very different proposition from one that only 200 people in the world would want to buy.
If your book has a total potential sale volume of only a few hundred copies, but you want to publish it anyway because it will make you feel better, then your only option is to self-publish. Just try to keep the costs down, because you’re not going to get your money back. Most self-published books fall into this category, it seems.
If your book is “just” good enough to attract the attention of agents and publishers, then you have a very tricky choice to make. If you can get a contract, then the publisher will pay an advance and take the financial risk away from you, in exchange for a large chunk of the profits. As a beginning author you’re likely not to get good contractual terms, either. The pros and cons can be debated forever and have been, many times, on the internet, mostly with a great deal of passion. Here is a sane post and a (relatively) civilised discussion:
Remember that as an unknown author you represent a higher risk for the publisher, and they may well come over all cautious and decline a book even if they think it might be good enough. There is obviously a minimum number of copies they need to be sure of selling before it’s worthwhile for them, and if the likely market is below that number (or they think it might be below that number) then you will be out of luck.
In these cases self-publishing could be the right option: you won’t make a fortune, as the market simply isn’t there, but you will make something and if you can do this with enough books then it may add up to an income worth having.
Just remember that if you do self-publish, you are appointing yourself as gatekeeper, project manager, and financial underwriter.
Here are some of my favourite posts on the question of self-publishing:
And, on the slightly broader topic of being a writer at all:
http://www.kboards.com/index.php/topic,150526.0.html (Russell Blake)
http://www.bbc.co.uk/news/entertainment-arts-25366534 (Nick Spalding)
http://kareninglis.wordpress.com/ [collection of advice for self-publishers]