The problem isn't so much his family as him. He has allowed this for years and won't change now. He even pretends he doesn't understand why you are not happy with the current situation. I'd be brutal with him and tell him that you are sick of being treated like his dirty little secret, that you are not his bit on the side, that if he wants a life partner things need to change and FAST. Tell him you don't want your daughter growing up to think that it's fine for daddy to have two families and to put the other one first, to treat her mother as if she were some cheap kept woman who is there only when he needs her. 

If that doesn't get through his thick, selfish, skull, start getting your ducks in a line...

He's letting BM and his kids control his life, because he fears (probably rightly), that if he doesn't, his kids won't speak to him for long periods again.  Sounds like he has a long history of caving in and doing what they want, so as to "not upset them further". 

If you can't live with the status quo, let him know that. Then he has to decide if he wants to be with you, or have his kids in his life, because it appears he can't have both, right now.  If he sets limits and refuses their demands, they will cut him off again, and may or may not ever mature to the point of wanting a relationship with him again. He might not be able to live with that.

i have 2 kids with hubby aged 4.5 & 3. Skids do not make an effort to meet us in our home but rather hubby has to drive 1-2 hours away to get them and its a friggin detour while we have 2 young kids... they are perfectly capable of making an effort but choose not to... 

meets with them are miserable enough when you have 2 little ones and playing chauffeur and them ranting on about bio mum and stepdad to show their relevance and superiority. 

Hubby told the daughters from now on to meet us in our home (that was 2 yrs ago), sd24.5 has been banned by bio mum and stepdad from driving to visit us... she lives in a home hubby bought and not with bio mum. Thats how stupid this all is!!

He lets you down on his birthday, and then tells you that YOU are the selfish one?  no, no, he's employing classic narc tactics of DARVO - (deny, attack & reverse victim & offender).   His 20 somethings sound much younger than the teens you describe them as - more toddler level really.  If your man doesn't mend his ways, I'd leave frankly. 

I'm guessing that the kids see "you" as the reason their parents split and I'm sure their mother has stoked that ember of an idea into a nice cozy fire for them to roast you over.

If I am understanding this.. he had an affair which caused her to kick him out.  His affair ended.. and in that time, he met you and you started a relationship while he was separated.  Then.. sometime later you had a child with him (before/after you married? not sure).  So, while you weren't the person he broke his marriage vows with first.. you do appear to his children to be the reason why their parents didn't reconcile.. and it sounds like his kids are angry at him and have punished him to an extent for this.  He is now afraid if he doesn't bend to their whims.. they will ice him out again.

This is patently unfair to you though.  You and his other child don't deserve to be tossed aside when his other bio kids decide to throw him a crumb of attention.

Obviously this may require him and you to be more proactive.  Did he know you had planned a special dinner for his birthday?  How far in advance had the chinese meal out been planned.

In itself, it really isn't a big deal if he wants to eat out with his children (without his EX)... but obviously you should be aware of the timing so you can plan.. and if they ask to take him out on a night where he already has plans with you?  he should be telling them.. No wednesday doesn't work for me... how about we do lunch on thursday instead.  and "no.. I think it should just be you kids.. I don't want to have your mother there"  You don't necessarily have to be included in his meals with his kids.. as long as it isn't a super frequent thing that is excluding you from major portions of time.

HIs kids don't have to like you.. they don't have to spend time with you.. and vice versa.. but he doesn't get to throw over your pre-existing plans when they say boo.

I'm not going to repeat was ESMOD said because she is exactly right.

He betrayed his family and they are within their rights to still be mad about it. When somebody hurts you like that you don't decide when they get over it. however he is treating them like they're much younger than they are by giving in to these demands and playing happy family with his ex. is he doing this out of guilt or because he likes to still have control over his old family his ex?  

You must end the interference that his failed family and the failed family breeding experiments have on your own young child,  your life, and your marriage.  The adult Skidiots are adults.  They are not a financial burden on him, but being nailed with the better part of two decades of CS, college costs, etc, etc, etc... will make you and your child important to him for pretty much the rest of his life.

Make sure he clearly comprehends what the true priorities are and make sure that pain is associated with the wrong choice.

He is the selfish one. He is making you and your baby the repeated sacrifices to his failed family.

You chose an adulterer.  Bad call.  Now you have to choose yourself and your baby.  

Take care of you.

You deserve more than that!  Its unfair to expect you to be the english muffin and fill in the nooks and crankies of his life instead of being the core of his life.  He hasn't made the commitment to really divorce and separate from his previous 'first family'.  He is trying to play both sides and it is never going to be successful.

I don't get it. I mean totally what we all signed up for here. For our husbands to go spend time with the ex wife and family. Perfect. You might s well book them all a table and say it's on me. 

please dont let this continue!!   i was dating my SO for quite some time until i found out the BM and SS were still all living under the same roof still playing happy families for the sake of the brat.  there was always an excuse and as im a busy independant women , i had no idea this was going on.  after she fianlly left, i found out years kater , he was still taking her on vacations with the kid , just neglecting to tell me. they think if they dont say anything , is not lying .  hes my EX now !  get out of this fast

My experience is very similar to yours. I have 4SK's (SS33, SS30, SD24, and SD20) who are all wrapped around BM's finger.

When I first started dating my partner over four years ago, all events needed to take place at the matrimonial home when BM could play "happy family".  My partner asked if I could attend and BM threw an adult temper tantrum, so I stayed at home alone. 

My partner has attempted to invite all his children to his place or a neutral setting, and they have refused. SD24 and SD20 do not want to leave the matrimonial home where they live with SD24's new husband and BM.

In fact, SD24 threw an adult temper tantrum when my partner brought up meeting me one day in the future. Now, SS33 is afraid of rocking the boat and doesn't want to meet me again for fear of incurring his mother's or his sister's wrath.

My partner and I were both gobsmacked that his children are so entitled, selfish, and enmeshed with BM. They simply cannot be happy for their father. SD24 has already excluded me from her wedding ("you are forbidden from bringing your partner" and my partner almost didn't go) and the others are ready to follow suit.

If my partner and I already had an event planned, we would not change it; however, if SS33 and SS30 want to take my partner out for Father's Day, he would tell me as soon as they invited him so that I could make other plans.

Because his children also currently refuse to either meet me again or for the first time, I have asked him not to attend the matrimonial home this Thanksgiving and Christmas. We shall see how that plays out.  No more Ms. Nice Gal for me!

From reading the comments on this forum, the existence of adult spoilt brats seems to be relatively common. 

You are not alone.

I hope this helps.