I think I meet 4 or 5 people a year who tell me that their horse hates men. Almost always the horse is a mare. They tell me how cranky their horse behaves when handled by men.
I would like to talk to you about how to get a horse happy with other riders. It seems for you to have a good ride the horse had to hear what you are asking then someone else gets on and they don't have that same relationship to get the same willingness and the horse gets confused.
They asked if a horse must always experience emotional stress when learning a new task. They said that one of the first things that attracted them to clicker training was the idea of low or no-stress training. At first, this seemed to be the case.