Is this unreasonable?

You need to tell him to get used to it or get out. If something that small is bothering him, you've probably got a lot of other problems to work on. But don't risk your happiness or health for him ever.