Unstructured thoughts on Tech X Risk

Imagine a bag of pebbles. It’s deep. You don’t know how deep. You draw a thousand pebbles. They’re almost all white. Some are grey. Each white pebble you draw makes you stronger, happier, better. Grey pebbles hurt you a little bit but it’s just a twinge. A thousand pebbles in you’re much better off than when you started. That being said, you know that it’s possible that the bag also contains black pebbles. You also know that if you draw a black pebble it will immediately kill you. Should you continue drawing pebbles? If not, at what point should you stop? 0 pebbles? 100? 1000? Does the fact that you’ve already drawn 1000 pebbles and not a single one has been black influence your probability assignment to probability of drawing a future black pebble?

The bag of pebbles analogy, or something like it, is what Bostrom uses to argue that technological progress may well lead to existential risk. I’ve thought about this a bit and I’m not sure what my conclusion is. I think the bag problem raises a bunch of interesting question both philosophically and practically.

Philosophically, there a few interesting problems here. The first one is that we don’t know what the % chance of drawing a black pebble is which makes any EV calculation really tricky to do. It can be tempting to take the naive approach and reason inductively here. “I’ve drawn 1000 pebbles and none have been black. Hence that’s strong evidence that future pebbles won’t be black and/or that the bag contains few black pebbles”. That’s doesn’t really work for a few reasons:

• The boring problem: the distribution isn’t fixed. The further down the tech tree you go the wackier things get. A medieval steel-smith finding a way to forge swords which results in swords that sharper and cheaper may upset a balance of power or disrupt some political equilibria, but it won’t kill everyone, wipe out the biosphere or cause a vacuum collapse which destroys spacetime. Future tech we find (engineered pandemics, vacuum decay bombs/plank worms, etc…) might. To link it back to pebbles, the more pebbles you pick the deeper into the sack you start going and further down layers have a different composition.
• The interesting problem: anthropic shadow. If we had picked a black pebble in the past, or even just a very bad tech, there would be far less people alive today. Hence you probably won’t have exist. Hence the fact that we look around and observe “hey, we’re in a world where no terrible tech was picked which wiped out civilisation” may not be actual evidence as much as it is a selection effect. Trying to do inductive reasoning without taking into account that the environment affects whether you exist and hence what you can observe is akin to concluding that god exists because the universe is fine tuned to support life. (Also, on a meta level I’m really uncertain about anthropic shadow and anthropic reasoning more generally. I plan to do some work at some point to study it and come to a conclusion but until then I’ll leave this here) So if we can’t inductively reason about the distribution of future tech and we don’t have any way of choosing a reasonable prior what then? How do you even approach the problem?

Ok. Putting aside the epistemic confusion there are a bunch of interesting pragmatic considerations:

• Is it possible to control tech progress?
  • It seems practically impossible. We’re in a multi-actor system where every actor benefits hugely from increasing tech. One view is that absent an incredibly repressive global super state, tech will continue to advance. Still we’ve stopped techs dead in the past. Human cloning. Embryo selection. Eugenics. Inadvertently nuclear power.
  • What about avoiding certain specific techs but still making overall progress? Maybe we just blacklist specific technologies or applications, This may work but I’m sceptical for a few reasons. 1: Time. Over the next 1000 years I’m sceptical the genie will stay in the bottle. 2: Ease of access. As the tech level generally rises it becomes easier and easier to research/implement previous techs. What takes a nation state in 2020 may take a team of PHD’s in 2050 and a lone smart teenager in 2200.
• Can we see it coming?
  • It’s really, really hard to predict long term trends in technology. It’s also really hard to predict the social effects of new forms of tech. Who would have thought that access to ultra violet lithography would be an important strategic capabilities for nations in the 21st century? Which Chinese chemist in 900AD would have predicted that gunpowder would contribute to the end of Feudalism and centralisation of power by making castles less and professionalized armies more important. Can we even tell in advance which techs are dangerous?
  • We can certainly do some prediction. If tomorrow there are simultaneous news reports of a new way to synthesise silicone based bacteria + a new way to make mechanical keyboard keycaps I’m pretty confident assessing which has a greater risk. Still, maybe our predictions on average only become accurate enough 5 years out from really bad tech being realised. Maybe some really bad techs aren’t obviously bad and their dangers only become apparent when they become widespread or hit a crucial threshold. (e.g: fast takeoff AI scenarios)
• What’s the tradeoff? What do we give up by slowing tech progress or banning techs?
  • There are natural X-Risks. If we magically stopped tech development at our current level it would just be a matter of time until an asteroid strike, plague or eventually the sun dying would end consciousness in our galaxy.
  • Tech development also increases the number of humans alive at any given time. 10 billion humans for 100 yeas > 100’000 humans for 10’000 years.