I think I meet 4 or 5 people a year who tell me that their horse hates men. Almost always the horse is a mare. They tell me how cranky their horse behaves when handled by men.
I would like to talk to you about how to get a horse happy with other riders. It seems for you to have a good ride the horse had to hear what you are asking then someone else gets on and they don't have that same relationship to get the same willingness and the horse gets confused.
I believe negative reinforcement principles offer the best pathway for providing clarity to a horse. In my opinion, this makes it at the top of the list of kind and gentle training principles.
Many people use ear bonnets or ear plugs to muffle noise while improving focus and reducing stress in horses that are sensitive to noise. They are very common at horse competitions and a lot of people swear they help the performance of their horses.