The fundamental danger isn't human obsolescence, it's that humans will be outcompeted for scarce natural resources. It's possible that AI will grow so valuable that its owners bid up the price of energy astronomically — so high that humans can't afford fuel, electricity, manufactured goods, or even food. At that point, humans would indeed be immiserated en masse.
