Hypothetically, would it be a bad idea to move into a house owned by family even if it's just the two of you?

It was difficult to word the question so let me elaborate.
As some of you know my situation, my boyfriend and I have decided to get an apartment next spring/summer and be out of his family’s house owned by his mom. He currently lives with his sisters and one of the boyfriends still paying mortgage.

My boyfriend told me he finally told them that we were planning a place of our own and we even got house hold items from Black Friday. They said they were going to find a place too (not sure if I believe them because they say this every year but never moved out) and move. I asked what would happen to the house if no one lived in it, rent it out? He said he doesn’t know but he would be open to us living there but he knows I don’t want to live there.

I explained that it wouldn’t feel like my house or my property. It would still be seen as the family house and his mom’s house. I could easily imagine her coming over or if his sisters break up with their boyfriends again they would most likely default back to the “family” house and move in for whoever knows how long.
My boyfriend says we would make rules but I highly doubt it would stick. My gut feeling says we should move on with our plan on our OWN place and he is fine with that and said he is still looking forward to an apartment.

So, hypothetically speaking…if his sisters moved out and we decided to stay in the house, do you all think it’s a bad idea? Discuss. As I said before, we are still planning an apartment soon… I highly doubt they would even be moved out by next year.!