Confronting her about it is what makes you look like a sore loser, if anything. Any woman who is LTR quality is not going to need to be told she shouldn't be talking to her ex's, especially married ones. Telling her you're done just lets her know you have standards and she doesn't meet them.
I say no, don't keep her around. Sex drive is a large part of a guy's motivation to go out and meet women. If you waste it on some loser woman then you're just kneecapping yourself for some fleeting pleasure. Even if you're some ultra masculine super stud who has 100% confidence around women at all times and can stay hard 24\7, the time you waste screwing losers still could have been used to meet a woman you really enjoy. Don't make excuses, just go out and meet a woman you really like. You'll be glad you did.