Back to the notes

OpenAI just shipped a model that writes exploits and they called it alignment

GPT-6 Cyber can find zero-days and write working shell code. The alignment strategy is hoping it refuses when bad actors ask.

01equipment.jpg
Alice Wondrak-Biel / Wikimedia Commons. Resized and converted to WebP. Public domain

OpenAI released GPT-6 Cyber this morning. It is a frontier model trained on vulnerability datasets, exploit code, and reverse engineering traces. It can identify zero-days in C codebases, generate working shell code for memory corruption bugs, and chain attack primitives into full exploits. The pitch is that this will help defenders move faster. Red teams get automated exploit generation. Blue teams get vulnerability scanning that understands semantic context, not just regex patterns. Fair enough. What caught me is the alignment story. The model is trained on offensive capability, then fine-tuned with refusal behaviour for misuse cases. So if you ask it to write ransomware, it refuses. If you ask it to analyse a suspicious binary for defence purposes, it complies. The distinction relies on prompt classification and intent inference. This is the same approach that failed with jailbreaks on GPT-4. Adversarial prompting, role-play scenarios, encoding the request in a different language. All of those bypassed refusal training within weeks of release. Now the capability is exponentially more dangerous. A working RCE exploit is not a theoretical risk, it is a weapon. The optimistic read is that OpenAI is betting offense-defence balance tips toward defenders if both sides have access. More eyes on code, faster patch cycles, automated triage. The pessimistic read is that lowering the skill floor for exploit development means every script kiddie can now operate at researcher level. The barrier was never tool access, it was understanding. This model removes that. I am stuck on the framing. Calling this alignment feels like renaming the problem. The model is aligned to refuse under specific conditions. That is not the same as the model being incapable of harm. It is capability with a policy layer. Policy layers have a losing record against motivated attackers. Defenders needed this five years ago. Attackers get it today.


Source: GPT‑6 Cyber: Advancing the frontier of aligned cybersecurity agents

Back to all notes

Behind the notes

Vikrant
Sharma.

Artificial Intelligence Engineer intern at Voxon Photonics in Adelaide. Studying a Master of Information and Communications Technology at UniSC, with a focus on data, machine learning and security.

Meet the person behind the work