The easiest way to do it: https://addons.mozilla.org/en-US/firefox/addon/app-button-remove/
Hmmm I use FF4 for web development mainly, havent noticed it…
May I suggest Chrome for a better/faster browsing experience 
Good suggestion Sniper! But i’ve used it before as a switch from FF… came back to my baby 
I hope these steps will help you
http://james.greenhalgh.eu/2011/firefox-remove-orange-button/
Here is the solution right up to Firefox 6..
download and install: Movable Firefox Button ADD ON…
press ALT, select VIEW>TOOLBARS>BOOKMARKS TOOLBAR
press ALT, select VIEW>TOOLBARS>CUSTOMISE
Now press ALT, and move the firefox orange button onto the toolbar - then click close.
Finally:
press ALT, select VIEW>TOOLBARS> and uncheck BOOKMARKS TOOLBAR
ALL GONE!