"Yes, you should be worried. He's not going to divorce you as long as she's still getting all of your huge salary. Soon he'll not only take your place in bed, he'll also take your seat on the couch when they watch Netflix shows in your own home and you have to do it on the floor."