I'll write you a dissertation at some point if you like, but the short of it is that it's a lovely place to visit and a terrible place to live, especially if you're still in your earning years.
I still miss the weather, the landscape, and even the food, but I do not miss the people. We left in large part to get our children away from the default attitude of "given up" that 90% of the population lives by. I seem to recall you mentioning having kids. Do not move there and put them in public school if you expect them to know anyone who will ever do anything with their life. The state's horrible policies drive a lot of this. A classic is that property taxes never go up so long as a house stays in the family - sounds like a nice policy until you realize it incentives generations of people to do absolutely nothing because they can live in the same house great grandpa built in 1902 for about $50 a year, so no one is incentivized to do anything beyond the absolute least possible to get by. The end result is New Mexico competes year after year with Mississippi for the poorest state in America.
There are a small handful of people working their assess off to improve the place, and they are met with a state taxing department that treats them like a piggy bank and commits some of the more egregious government overreach I've ever seen. It's straight-up criminal, with no exaggeration, but you will be living in an essentially lawless place, so what is criminal, anyway? If you own a business, bring your own tub of lube. Reason, rationality, and even the law do not factor into any interaction with state tax and rev. If you are deemed to have something beyond a mere subsistence existence, they will try and take it from you until you're broken like everyone else there. This is why the wealthy folks in Santa Fe don't actually reside in New Mexico, they just fly in from somewhere else to their second or third house. They still get dinged at every opportunity by tax and rev, but not nearly as bad as residents.
You see the upside-down attitude of punishing people for trying to do better pop up in lots of places. New Mexico has one of the highest (maybe the highest) rates of uninsured motorists in America. Do they step up enforcement? No, they just charge the people doing the right thing more money. Stealing a car wasn't even a felony until around 2011. Each neighborhood in Albuquerque has a state senator. So whoever has the most relatives in a zip code gets elected. And despite this rather quaint state government, the oil and gas industries really run everything and pay nothing into the system for it. It's a mess of a place with corruption that makes Illinois (or Texas) look like candy apples.
The other aspect that can be really tough there is New Mexico is one of the least hospitable places I've ever been. People keep to themselves and their own group. Almost anywhere I've ever been in the world you can walk into a pub, strike up a conversation, and end your experience with some new friends. That is an extremely rare occurrence in NM. I never really figured out why, but people are not interested in meeting new people in that place. Santa Fe is a bit different as it's filled with tourists and outsiders how spend only part of the year there, but the rest of the state would just as soon you not be there.
So this turned into a poorly written dissertation after all, but in summary, New Mexico is a great place if you're independently wealthy and can shield your assets from a taxing authority that operates more like a Mongol horde than a government department. If you have kids that are still being raised or intend to earn money in the state, run as far away as you possibly can. If you're thinking of retirement, then sure. But know that you'll be surrounded by some of the most corrupt and some of the most sadly beat-down people you'll ever come across. Texas pisses me off all the time, especially the politics, but New Mexico gives me the shivers when I think about it. It's a place to give up. It's not a place to live.