I don't know if Mechanize is just an ambitious startup looking to make splashy headlines and get VC investments, or if there is actually some credibility to their goal of automating all jobs. Even if they fail, the idea of this happening has always been there and someone will always be trying to accomplish this unless society has the proper guardrails in place. Currently, the leadership of our country and China puts that in doubt.
Brisket loves a doom and gloom scenario but I think he's absolutely right to be concerned about this. We can't simply ignore it or write it off as science fiction, then cry about it in 10, 20, 50 years when it actually happens. If a company in the U.S. (or anywhere) has a stated goal of replacing all human jobs, that is obviously quite problematic for the future of the human race. Even if they are only 75% successful in the end, that's an incredibly different world to live in and probably not for the better.
I think with all disruptive technologies, there should be a type of hurricane forecast model that shows the most likely path as well as the extremes on either side. The best case scenario for AI is that it solves some complex problems like developing cures for cancer without too much collateral damage along the way. The worst case scenario for AI is quite terrible, and if we leave it up to the companies developing it to set their own rules, there will eventually be bad actors that will capitalize on the worst of what the technology is capable of. You could argue that there are currently more bad actors than good actors. There are more people motivated by greed, power, and personal glory than there are people motivated to cure cancer and solve the world's problems.
I fear that even in the best case scenario, we will have a future in which the human experience is significantly dulled as a result of pervasive AI. We will lose a lot of our critical thinking capabilities and our creativity. It will be far easier to ask AI to do things for us than to actually attempt to solve our own problems. It will be far easier to ask AI to write a story or compose music or paint a picture than it will be for a human to do it independently. There are already people in this thread saying they use AI to create recipes from their pantry. I'm not judging, but that could become the norm for future generations and people will have a significantly reduced ability to do a lot of the things that makes us human. People will get used to seeing AI generated content on social media that surpasses what they actually experience in real life, which could make the real world less exciting and spectacular.
The internet revolution is probably the best comparison, but there are a ton of negatives for society that people weren't dealing with 30 years ago. Social media is about 90% detrimental to society. Jobs have not been entirely eliminated by the internet, but many sectors were eliminated or significantly reduced. New jobs have been created, so that's a positive. The human experience, IMO, is generally worse by the fact that we are constantly staring at screens instead of doing things with other people.