As long as you both are being honest about the situation- and it sounds like you are- then I don't see a problem. Neither of you are after a relationship and just get together when it's convenient for the both of you. Sounds good to me.
You didn't say (or ask) but I can't let this pass without at least mentioning to remember to play safe. He (and you) both have multiple partners so condoms are a necessity.... along with a good spermicidal foam or gel. And if I were the female in this scenario, I'd be on birth control as well.
Have fun and take care.
