August 07, 2016

"The tools in the CGC [Cyber Grand Challenge] aren't designed to target the same sort of software that runs on everyday laptops or smartphones."

EFF: Instead, DARPA developed a simplified open source operating system extension expressly for the CGC. By Nate Cardozo and Peter Eckersley and Jeremy Gillula

'In part, this was intended to make the work of CGC contestants easier. But it was also done so that any tools designed for use in the CGC would need to be significantly modified for use in the real-world—so they don't really pose much of a danger as is, and no additional safety precautions are likely necessary.

'But what if, a few years from now, the subsequent rounds of the contest target commonplace software? As they move in that direction, the designers of systems capable of automatically finding and exploiting vulnerabilities should take the time to think through the possible risks, and strategies for how to minimize them in advance. That's why we think the people who are experts in this field should come together, discuss the issues we're flagging here (and perhaps raise new ones), and come up with a strategy for handling the safety considerations for any risks they identify.

'In other words, we’d like to encourage the field to fully think through the ramifications of new research as it’s conducted. Much like the genetics community did in 1975, we think researchers working in the intersection of AI, automation, and computer security should come together to hold a virtual “Autonomous Cybersecurity Asilomar Conference.” Such a conference would serve two purposes. It would allow the community to develop internal guidelines or suggestions for performing autonomous cybersecurity research safely, and it would reassure the public that the field isn't proceeding blindly forward, but instead proceeding in a thoughtful way with an eye toward bettering computer security for all of us.'

No comments: