I more or less did what you are thinking about and 10 years later, I'm paying the price. Here's a brief history of what happened:
I was working for a federal law enforcement agency and I was making close to $80K for basically sitting on my a$$ to do nothing. I was bored, getting burned out, wanted a change, etc. My wife was working full time as well, so we were pulling in over $100K combined salaries, no kids, etc. They transferred me to D.C. and that was for a promotion and was supposed to be a good thing. It wasn't - we had our first kid, things were still OK. Had our second kid and things were not OK. We were in D.C. at the time and anyone who has lived out there knows that while that kind of money is pretty good, in D.C. it isn't that great. We ended up in a position where we could either pay for day care for the kids or make our car payments, but we couldn't do both (yes we had other debt, so we weren't in a debt free situation like you are talking about). So the wife opted to stay home with the kids and there went her salary.
Anyway, I ended up taking a job with another law enforcement agency and took a bit of a pay cut. Was there for about 2 1/2 years, realized it was a horrible situation, and went to a different agency (local agency this time). Took another pay hit, but had some OT, but it still made it worse. Well, I ended up getting hurt on that job, ended up being out for 7 months on the injury losing all of my OT, court time, etc. and we got even more behind. Long story short at this point is that we ended up losing our house and my parents agreed to relocate us back to Texas. That was a year ago and I'm now working at a job where I am making less than 50% of what I was making - so we are destroyed financially.
How it relates to what you are talking about is that we basically went from making over $100K a year (80 of it mine) to making around $55K when I changed jobs. That then went down to around $48K with the local department and now I'm not even making $40K at my current job. On an absolutey DAILY basis, I mentally kick myself upside the head for doing what I did. If I had stayed with that federal agency and NOT taken the promotion to D.C., we would be sitting so freaking pretty right now it wouldn't even be funny. Instead I'm living off of my parents (they are paying for the house) and I'm barely putting food on the table. We've been surviving on about $25-$50 between paychecks right now and I just found out today that my freaking health insurance is going up $130/month. That means that I'm now going to be paying $1300/month for health insurance and we are officially screwed. That extra $130/month is now going to be a bill or groceries that I can't pay.
While my situation might not be 100% like yours, I live with regret every day for that one horribly incredibly stupid decision I made. My advice? If you guys are sitting that pretty, stay with it and deal. Those frustrations and burned out feelings that I had over 10 years ago when I made the switch are NOTHING compared to what I'm dealing with now. Take it for what it's worth and good luck either way