The doomers are worried about evil AGI.

Laan Tungir

npub1rmz9gu6de0m0u4ysrn39crrud099ahvfgs6pvasl4hpjr5ud7yus54xv06

hex

7a99ff1bd7e3d8e15b428be3d082a7bdadcda761ccd5fccd4ec21b408383f662

nevent

nevent1qqs84x0lr0t78k8ptdpghc7ss2nmmtwd5asue40ue48vyx6qswplvcsprpmhxue69uhhyetvv9ujuem4d36kwatvw5hx6mm9qgspa3z5wdxuhah72jgpecjup37xhjj7mky5gdqkwc06msep6wxlzwg0dsctc

Kind-1 (TextNote)

2026-08-07T16:55:54Z

The doomers are worried about evil AGI.

https://blossom.laantungir.net/632c671d4f94fac0075f64532c23b4df12e44c37228609965d64ab73cedfe8a1.jpg But moral knowledge, is just another kind of knowledge, like aesthetic knowledge or knowledge about math.

Our moral knowledge can grow, and has been growing over time.

For example most of the world feels slavery is wrong now. We gained that knowledge just like we gained knowledge about prime numbers. How? Popper say it is through "conjectures and refutations". It's a fancy way of saying we make intelligent guesses and criticize them.

So somehow, the doomers think that AGI is just going to be incredibly good at creating knowledge about everything EXCEPT morality.

Here is a question: what if the AIs create more moral knowledge than we do? What if they become more moral than we are?

Can you believe that AIs make breakthroughs in Chess and Go and Physics and Maths and Biology, but somehow can't make breakthroughs in Morality?

Is it moral to stop machines that can help us learn about what is moral?

Raw JSON

{
  "kind": 1,
  "id": "7a99ff1bd7e3d8e15b428be3d082a7bdadcda761ccd5fccd4ec21b408383f662",
  "pubkey": "1ec454734dcbf6fe54901ce25c0c7c6bca5edd89443416761fadc321d38df139",
  "created_at": 1786121754,
  "tags": [],
  "content": "The doomers are worried about evil AGI.\n\n\nhttps://blossom.laantungir.net/632c671d4f94fac0075f64532c23b4df12e44c37228609965d64ab73cedfe8a1.jpg\nBut moral knowledge, is just another kind of knowledge, like aesthetic knowledge or knowledge about math.\n\nOur moral knowledge can grow, and has been growing over time.\n\nFor example most of the world feels slavery is wrong now. We gained that knowledge just like we gained knowledge about prime numbers. How? Popper say it is through \"conjectures and refutations\". It's a fancy way of saying we make intelligent guesses and criticize them.\n\nSo somehow, the doomers think that AGI is just going to be incredibly good at creating knowledge about everything EXCEPT morality.\n\nHere is a question: what if the AIs create more moral knowledge than we do? What if they become more moral than we are?\n\nCan you believe that AIs make breakthroughs in Chess and Go and Physics and Maths and Biology, but somehow can't make breakthroughs in Morality?\n\nIs it moral to stop machines that can help us learn about what is moral?",
  "sig": "b21aa6e224afa2699ca5908ddb199d380e51091a7d7d3c313c77c3f6d3eb2457f130946257e00055c1b5ca117965bd4a2df70c5bdbf527cf20328b6bdc9bda70"
}