From the man's behaviour, it looks like he knew that what he was doing would piss off the female player, not just make her uncomfortable like anyone would be if someone did that.
As you said, this was not MtG sanctioned play. But you don't need to be a judge to act in this situation. If you feel like someone is doing something they shouldn't do go ahead and try to make the person stop, go talk to the staff, and if the problem persists go and report the store for allowing this behaviour, they are still bound by the WPN Terms and Conditions, including the WPN Code of Conduct.
You may not have the authority of a judge in that context but you don't need it. What you have is the confidence, the charisma and the players' trust of a judge (this last part assuming there is some overlap between MtG and D&D players at your LGS and this is your LGS).
I also feel like this piece of text from the Judge Code of Conduct is relevant here:
Judges should not allow others to create a bad environment by inaction
One could argue that creating a bad environment at a LGS outside MtG play will damage the image of the LGS and make Magic players leave the store for another place, because they don't feel comfortable here anymore.
Let me be clear, I don't blame you for the course of actions you took, I cannot say I would have acted differently in this situation. Or better, I blame you the same way I would expect anyone to blame me if I had done what you did and with the information you gave us. It doesn't mean you're the only one to blame.
Regarding your future course of action. If you have the occasion to go back to that LGS, go talk to the female player, investigate in order to know if the behaviour you witnessed is a recurrent thing, talk to the staff, ask them about that player, if he is known for doing that, if there was a grief with the female player previously. And the next time this kind of thing happens step up, talk to the man (or bring a staff member to talk to him). As Adventure League is Wizards sponsored there could be play consequences for more than the involved persons, set aside complaints for harassment.