Not at all. Not every man wears a wedding band -- my grandfathers didn’t, nor does Prince Phillip, Prince Charles or Prince William.
Don’t be afraid to ask your fiance what his hesitation is to wearing a band, for all you know he’s allergic to gold or the feeling irritates his skin. I wouldn’t assume he is planning on having an affair.
If he won’t wear a wedding band maybe he’ll agree to a tattoo wedding band. Several celebrities have gone the route of having a decorative band inked around their left ring finger. The design can include your wedding date and/or names just like the inside of a metal band.