Should I feel this way or am I wrong?

My husband is an lvn at a nursing home and there is not one other male that works there, it is nothing but females. Most of them are married or unattractive(so he says and the ones i have seen i agree), but I sometimes just can't but wonder if he flirts with them or vise versa. I have been with him over 4 years and we are definitely in love. I know I can trust him its just sometimes he will tell me he never really talks to any of the women he works with or they are always arguing because of their drama. But then when we see them in public he tries to avoid them or he will be like oh hi how are you. So it is weird that he tells me he doesn't talk to them at work yet acts like that when we see them in public. Not that I would get mad if he talks to them, but I don't know why he says that he doesn't because its work, you have to talk to somebody. when i ask him about it he says he just doesn't want to make me mad. Any thoughts? Am I just worried for no reason, or is it normal to worry? And im not constantly worried, the thought just pops up every now and then like when we see one of them or when he is 2 hours late or something like that.