I was glad to find out, after consulting your friend and mine, Google, that I'm not the only person having trouble finding the perfect set of headphones to wear while working out. Unfortunately, though, it seems that not many folks agree on any one style/model.
This is far from an exhaustive study, but here's what I've tried since I started working out again last spring, ranked from worst to best.
Stock iPod earbuds
It doesn't get much worse than these. The sound isn't necessarily awful, but any randomly-chosen pair of $10 headphones would do just as well. The bigger problem is that they're just no good to run in, unless you want to be readjusting them every thirty seconds to keep them from falling out of your ears.
In general, any of the wrap-around-behind-your-head style headphones
I haven't personally done any running with a set of these, but I hear that they do stay put. They're problematic if part of your workout involves laying with your head against a Bowflex bench, so that's a no-go for me.
Philips SHS8000 hook-style headphones
Headphones that hook over the ear seemed like a good option. These Philips ones do stay put a lot better than basic earbuds, but still tend to wiggle loose from your ear every so often. While mine were working, the sound quality was admirable, but then the left headphone died after about three months.
Sennheiser MX75 "twist-to-fit"-style sport headphones
Finally, my current set and all-around favorite. The Sennheisers are a set of earbuds, but with an extra rubber nub on each earpiece so it can be wedged in with the nub pressing against your outer ear and not come loose. This system works great and the sound quality is fantastic for a small set of workout headphones. The only complaint I have so far is that they can make your outer ears slightly sore, but I only really notice it when removing them after having worn them for more than a couple hours, and the soreness goes away after just a minute or so.
The search will continue if the Sennheisers die an early death, but they're the best I've found so far. Results may vary as far as sore ears go, but I'm personally fine living with the very minor discomfort when they're otherwise perfect for what I need.