RUN! You are setting yourself up for an intolerable situation if you go into it this way. It will only get worse as she get older. I would tell him either you change and you start laying down some rules or enjoy life with your daughter. Let him decide.
Good luck, I hope it works out for the best.

My advice is don't move in with him. It's a real mistake to do it because of reasons of convenience or economy. If you're feeling unhappy now, when you can escape back to your own place, it will only worsen when you're living 24/7 with the issues you are already unhappy about. And then your children will be involved as well.

Only move if and when you feel that you have enough mutual understanding, and respect, and groundrules in place for you to feel comfortable and supported living with his children. I know a lot of posters are going to read my last sentence and snort, because a lot of stepmothers never get to that point and have years of frustration and unhappiness. I had a pretty long struggle myself.

Your post contains some classic symptoms that untold thousands of stepmothers are all too familiar with:
1. A defensive Dad who cannot accept even reasonable, constructive comments on his children, and who is prone to going into attack mode when such comments are made, which can only result in more resentment on your part. Defensive Dads frequently misinterpret frustration, stress and hurt as selfishness
2. A guilty Dad who does not understand that you both need and deserve time together to focus on your relationship, and that ultimately that time will be to the benefit of his children, and yours, because the quality of the relationship between the 2 of you is the bedrock of the family.
3. A Dad with a double standard who consciously or subconsciously wants you to love his children like your own, and be as loving and generous and tolerant as a parent, but without having equal rights to form and express opinions about their behaviour, and without the reward of their loving you back
4. Children who do not have clear, agreed boundaries and who are starting to realise they can manipulate the situation. This does not make them bad children, as I'm sure you know as the mother of two, all small children push to find out where their boundaries are, it's part of understanding their place in the hierarchy and exploring their control over their world. In a good stable home they meet clear, consistent boundaries, in a separated home they often get mixed, flexible or overly negotiable boundaries.

To stand a chance of overcoming these problems you need to find a way of:
1. Sharing and being understanding about each others feelings with regard to the children. He needs to understand how much effort it requires on your part, even if that effort is willingly given, and be understanding and supportive. You need to understand how protective and insecure he is about his children and try not to push his buttons.
2. Discussing and agreeing your role with his children, and his role with yours, particularly with regard to authority to make requests and give instructions, and discipline
3. Discussing and agreeing some fundamental household rules for all 4 children: manners, mealtimes, chores, bedtimes etc. Otherwise these daily routines become a constant source of tension on all sides

Personally I failed to push for enough clarity on 2 and 3, and struggled for years to get my DH to understand how hard it is to have a pseudo parental role with someone elses children. In my experience, and from what I read on this site, many Dad's mistakenly believe that if their partner has a problem with the stepparent role it is 'their' problem, rather than recognising that there are some universal stresses and tensions and problems that 99% of stepfamilies experience. The only way I made any headway with getting DH to understand that my feelings were legitimate was books, and later counselling. When I read to him from some of the books on stepparenting things that absolutely endorsed stuff I had already tried to explain to him, he finally started to get it. A book I particularly liked was The Stepparents Parachute: The Four Cornerstones of Successful Stepparenting.

Oh and see Caregiver1127's post which I just read:
ANOTHER ARTICLE ON HOW TO BE A HAPPY STEP PARENT - These seem to address most of the issues on here - share with your Spouse new
This makes loads of good points