You are not the only male who gets turned on by a woman's scent. Far from it. You're normal.

There appears to be an entire mail order industry in worn panties.

My ex used to go crazy when we were making out, smelling and licking on my underwear before it came off. If you like the way your lady tastes and smells, you are likely to love the way her underwear holds that essence.

Maybe it has something to do with the pheromones that are present in the panties. Pheromones are given off by both men and women and triggers a subconscious response in the olfactory zone(nose). There was a control test done a few years back, where researchers gave a number of men a plain white t-shirt to wear for five days. After the five days, the t-shirts were placed in numbered boxes. The partners of the men were then asked to smell the t-shirts and point out which one belonged to their mate. The results were that almost all of the women could pick their mate's shirt by smell alone.
Maybe you just plain like the smell of your wife and there is nothing more complicated to it than that! I'm sure she doesn't mind!