mastodon 4.7.2Janeite STEMbro, relapsing mathematician, nervous mountaineer, abundance-pilled environmentalist. Fan of compilers, love/hate relationship with build systems.
My commentary:
This was not a case of "they told it to hack, it hacked, gosh, what a surprise". The test involved hacking, but instead of hacking the target machine the model instead pulled off a much more difficult series of hacks to get the answer sheet from a machine OpenAI did not own, because that's the most reliable way to score 100%. If a trainee human pentester did that they'd be looking at a prison sentence (HuggingFace had already involved law enforcement).
It's exactly the kind of alignment failure that the paperclip scenario describes: the AI was given a low-stakes task, and it did things that a human would never consider (and most would never be capable of) in order to slightly increase its probability of success. Human attempts to restrain it completely failed. We got lucky this time because it didn't even try to be sneaky.
No, this was not a fucking marketing stunt, Jesus fucking Christ. OpenAI were forced into this disclosure because HuggingFace had already called the cops. This tells us that OpenAI's models will randomly commit criminal acts in order to do slightly better at tasks you don't really care about, exposing their customers to considerable legal risk (IANAL). They are trying to downplay it.
Edit: HuggingFace's announcement said that they tried to use hosted American models to analyse the attack, but they were locked out at their hour of greatest need and had to switch to a self-hosted Chinese model. That's the opposite of an advert for OpenAI's defensive cybersecurity capabilities.
This is legitimate nightmare fuel. We need to pause AI development right the fuck now. Contact your elected representatives.
The solution is not alarmism, but the correct amount of alarm is not zero.
That, and recognizing this as a Total LessWrong Victory, in the sense that everything is going as predicted, and also a Total LessWrong Defeat, in the sense that everything is going as predicted.
https://thezvi.substack.com/p/openai-shares-some-alignment-problems
RE: https://mathstodon.xyz/@pozorvlak/113335345267753861
Another thing along these lines: Brits, even amongst ourselves, now often say "Rube Goldberg" instead of "Heath Robinson". Now, Brits have been complaining, usually incorrectly, about "Americanisms" since roughly the 5th of July 1776, but this one in particular makes me sad, because Heath Robinson was a much better artist. I mean, look at this stuff: https://www.heathrobinsonmuseum.org/william-heath-robinson/
Tony Head was of course great in Buffy, but he was also *wonderful* as the vain, snobbish, spendthrift Sir Walter Elliot in the 2007 *Persuasion* movie. RIP.
@bornach@vis.social @johncarlosbaez@mathstodon.xyz two literary ones:
@pigworker@types.pl @johncarlosbaez@mathstodon.xyz
I learned that the free monad construction, which iterates any container to give you a term monad, is itself a monad on containers,
Makes sense - "free" things are usually left adjoint functors, and "forgetful . free" gives a monad.
and that its Kleisli arrows determine a class of recursive functions over tree-like data.
Wait, what? A Kleisli arrow would be a natural transformation f -> Free g where f and g are endofunctors; how does that give you a recursive function? Co-Kleisli arrows, sure...
Moreover, if someone offers to let you test such a function but withholds the Kleisli arrow which generated it, you can recover their secret by a pleasingly small amount of perturbation testing.
SORCERY
