The reason for the question is simply it takes a certain personality and trait that I think that Rory is missing because they are lustful in nature and can seduce any men, and Rory is not in the league of Aphrodite and doesn't have her assets.
Aphrodite is also infamous as well because she did create love among mortals but she also created great chaos and conflict and broke up many relationships and was cursed of sorts because she never found true love.
I noticed that most Goddesses of Love rarely find True love look at what I said that is true about this, and when you become a Goddess of Love you can create much chaos and conflict in not a good way because Aphrodite when she found someone that interested her then she pursued this person whether he or she was single or not.
The people that normally refused the Goddesses of Love meet a harsh death because Aphrodite killed her lovers when they refused.