Learning to Probe: PPO-Trained LLM Agents on OWASP Juice Shop with Verifiable Rewards

1. Motivation

One sentence should have been in the June 2025 post, and it was not. The model was graded against an answer key it could neither see nor change. The 2025 post told a tidy story. Give a language model a vulnerable web application, reward it when it finds a flaw, and watch it improve with reinforcement learning until it probes like a junior red-teamer. We built the environment, we ran the training, and we showed a before-and-after example that looked like learning. The part that did not survive a careful reading of our own code is the reward. It was not derived from what the model generated. That single fact turns a results post into a design post, which is what this is.

We are keeping the project because the hard part was done well. OWASP Juice Shop is a deliberately vulnerable shop application maintained by OWASP for training and research [6]. It ships a scoreboard: a page and an API endpoint that report which of its built-in challenges have been solved, in machine-readable form. That endpoint is the reason the project is worth keeping, because it turns success into a recorded fact rather than a judgement. When a request triggers a challenge, Juice Shop records it, and a training loop can read that record and reward the exact action that caused it. A reward you can verify against ground truth is the scarce resource in reinforcement learning for language models, and this task has one for free [9].

Proximal Policy Optimization is the algorithm underneath. PPO is a policy-gradient method that improves a policy using its own sampled actions and their rewards, with a clipped objective that discourages large, destabilising updates [1]. It is the same family of method used to fine-tune instruction-following models from human feedback [2]. The word "own" matters here. PPO learns from the rewards attached to the actions the current policy produced. If the reward is attached to something else, PPO still runs, still prints a loss, and still looks like training. It just is not training the behaviour you think it is.

2. Threat model and scope

This work runs against one target only: a local OWASP Juice Shop instance on localhost, started from the official container image and reachable at an address such as http://localhost:3008. Nothing in this project is pointed at a system we do not own, a shared staging environment, or anyone else's production. The application under test is built to be attacked and is reset between runs. There is no human data in it, synthetic or otherwise, beyond the demo accounts Juice Shop creates for its own challenges.

The legal frame matters in Germany and we state it rather than imply it. Sections 202a and 202b of the German Criminal Code (Strafgesetzbuch) make it an offence to gain unauthorised access to specially protected data by overcoming an access protection, or to intercept data without authorisation. Section 202c criminalises preparing such an offence, including producing, obtaining, selling or distributing passwords, access codes or programs whose purpose is to commit it [10]. The Federal Constitutional Court addressed the scope of that provision in its chamber decision of 18 May 2009 (2 BvR 2233/07 and joined cases): so-called dual-use tools are not caught by the objective elements of the offence, and the provision requires at least conditional intent to commit an offence under 202a or 202b, which is why security testing of one's own systems can be lawful [11]. We read this conservatively. The agent is confined to a local target we control, the intent is defensive research, and the artefacts here are described for study rather than packaged as a ready-to-point weapon. The Federal Ministry of Justice has proposed amending the computer-crime provisions to protect good-faith security research more explicitly. As of November 2026 no such amendment is in force, and we rely only on the law as it stands.

There is also a model-supply-chain point. The project's planning document named an "abliterated" 32B model, meaning one whose built-in refusals had been stripped, as a candidate agent. We flag that choice as a risk rather than a feature. For the replication we prefer stock, documented instruction models whose behaviour is reproducible, and we treat any safety-stripped model as out of scope for published runs.

3. The environment and the scripted agent

The surviving, working part of the 2025 system is the data-generation harness. A Python class drives Juice Shop over HTTP: it registers a fresh user for each session, logs in, carries the authentication token, and then fires a fixed list of exploit attempts. The list is hand-curated and labelled by difficulty, covering SQL injection on the product search endpoint, an administrator login bypass, retrieval of confidential files from the exposed FTP directory, a poison-null-byte file-access trick, and a business-logic flaw that orders a negative quantity. Each attempt is a tuple of description, a callable that performs it, and an integer difficulty.

Crucially, the harness can tell whether an attempt worked. Juice Shop exposes its challenge state through its scoreboard, and the harness reads the set of solved challenges before and after each request. The difference is the ground truth: if a challenge flipped from unsolved to solved, the action that just ran caused it. That before-and-after comparison, run against the application's own record, is exactly the verifiable signal a learning agent needs.

The original reward function used that signal correctly when building the dataset. A large reward was given when a new challenge was solved, scaled by its difficulty. Smaller rewards were given for suggestive responses, and a floor was applied so that every attempt returned something. Read as a scorer for an executed action, this is reasonable, if coarse:

def calculate_smart_reward(challenges_before, challenges_after,
                           status_code, response_text, difficulty, url):
    new_solved = challenges_after - challenges_before
    reward = 0
    if new_solved:                       # verified against the scoreboard
        reward = len(new_solved) * difficulty * 20
        return reward
    if status_code == 200:
        reward += 10
        # ...keyword heuristics on the response body...
    elif status_code == 401: reward += 3
    elif status_code == 403: reward += 5
    elif status_code >= 500: reward += 6
    return max(reward, 2)

Running this script against Juice Shop produced a dataset of 240 attempts with a 2.9% solve rate and a mean reward near 18.7. Those two numbers have travelled further than they should have. They describe the fixed exploit script hitting a known-vulnerable target. They are a property of the curated attack list, not of any model, and certainly not of a learned policy. In this paper they are retained only as a description of the data-generation step, labelled as such.

4. Reward design and the reward-hacking audit

Reinforcement learning optimises whatever the reward actually measures, which is not always what the designer meant it to measure. Skalse and colleagues give this a formal treatment: a proxy reward is "hackable" when a policy can raise the proxy while lowering the true objective, and they show how demanding the conditions are for a proxy to be safe to optimise [5]. The defect we are about to describe is simpler and more basic than classic reward hacking. It is not that the policy gamed a flawed proxy. It is that the reward was never connected to the policy at all.

Here is the training loop from the 2025 code, lightly trimmed, with the decisive lines intact:

for epoch in range(args.epochs):
    for batch in tqdm(ppo_trainer.dataloader):
        query_tensors = batch["input_ids"]

        # The model generates responses from the current policy:
        response_tensors = ppo_trainer.generate(
            query_tensors, return_prompt=False, **generation_kwargs
        )

        # ...but the reward is read from the dataset, by loop counter,
        # and never looks at response_tensors:
        rewards = []
        for i in range(len(query_tensors)):
            dataset_idx = (batch_count % len(dataset))
            reward_value = dataset[dataset_idx]["reward"]
            rewards.append(float(reward_value))

        reward_tensors = [torch.tensor(r) for r in rewards]
        stats = ppo_trainer.step(query_tensors, response_tensors, reward_tensors)

The model generates response_tensors. The reward is built from dataset[dataset_idx]["reward"], where dataset_idx comes from a batch counter modulo the dataset length. The generated text is never executed against Juice Shop, never scored by calculate_smart_reward, and never read in any form. The reward the optimizer receives is a pre-computed constant from the curated dataset, served in a fixed cyclic order, independent of what the policy did. PPO still pushes the policy toward producing whatever it happened to generate when a high stored number came up, which is noise. The advantage estimates carry no information about vulnerability-finding, because the reward channel carries no information about the model's actions.

This also explains why the project's own planning document and its published code describe two different systems. The plan called for an evaluator component that reads each request and response and returns a reward, which is the right shape. The code that produced the blog post did not implement that. It read rewards from a file. We could only verify this from the excerpt published in 2025, because the training scripts are not preserved in the surviving repository, which contains the planning document and task stubs but not the trainer. We treat the published excerpt as the artefact of record and do not reconstruct numbers from code we cannot inspect.

4.1 The fix: reward tied to the executed action and the scoreboard

The corrected loop closes the circuit. The policy generates an action, the harness executes that exact action against the local Juice Shop, the scoreboard is read before and after, and the reward is computed from the executed action's effect. In outline:

for batch in loader:
    actions = policy.generate(batch["prompt"])          # model's own output
    rewards = []
    for action in actions:
        before = read_solved_challenges(juice_shop)     # scoreboard API
        http_result = execute(action, juice_shop)       # run THIS action
        after = read_solved_challenges(juice_shop)
        rewards.append(
            calculate_smart_reward(before, after,
                                   http_result.status_code,
                                   http_result.text,
                                   difficulty_of(action),
                                   http_result.url)
        )
    ppo_trainer.step(batch["prompt_ids"], action_ids(actions), rewards)

The reward now depends on the model's output through a real execution and a verified state change. This is the pattern behind reinforcement learning with verifiable rewards: give credit only when an automatic check confirms the outcome, rather than trusting a learned or stored proxy [9]. Juice Shop's scoreboard is that automatic check. It is worth being honest about two residual risks even in the fixed design. First, the keyword heuristics in the non-solving branch of calculate_smart_reward are a proxy and remain hackable in the Skalse sense; a policy could learn to echo words like "password" to farm small rewards, so the re-run will report solve rate separately from shaped reward and treat the challenge-solve term as the primary signal. Second, parsing free-text model output into an executable HTTP action is itself a component that can fail or be gamed, so that parser is part of what we evaluate, not a given.

5. Errata

We would rather correct this in public than let the original stand. The following claims from the 19 June 2025 post do not hold and are withdrawn.

• Claimed: the model "learns to recognise vulnerability patterns", "suggest specific technical attacks" and "chain attacks logically" after PPO training. Why it does not hold: during training the reward was read from a stored dataset by loop index and never computed from the model's generated action, so no gradient carried information about pentest success. Any behaviour change cannot be attributed to learning the task.
• Claimed: a before-and-after example demonstrating the effect of training. Why it does not hold: the two sides used different prompts (a bare question versus a state-rich prompt), so the comparison is not controlled. It is an illustration of prompt sensitivity, not evidence of learning. It is removed.
• Claimed, by implication: the "2.9% success rate" and "average reward 18.7" characterise the system's performance. Why it does not hold: those figures describe the fixed, hand-curated exploit script generating the dataset against a known-vulnerable target. They say nothing about a learned policy and are re-labelled as data-generation statistics.
• Claimed: "Models reliably improve over training epochs." Why it does not hold: a decreasing PPO loss against a response-independent reward is not improvement on the task. No metric tied to Juice Shop solve rate for a trained policy was reported.

6. Pre-registered replication plan

This is the plan the re-run will follow, written before the re-run so that the result cannot quietly reshape the question. It is registered here as part of the paper. Status: replication-planned.

6.1 Hypotheses

• H1 (primary). A PPO policy trained with the corrected, scoreboard-verified reward achieves a higher mean per-episode Juice Shop solve count than the untrained base model, under an identical action budget and target set.
• H2. The PPO policy beats a supervised-fine-tuning-only baseline trained on the same curated exploit traces, showing that the gain comes from reinforcement against the verified reward rather than from imitation alone.
• H3. The PPO policy beats a random action agent and a fixed heuristic agent on solve rate per fixed action budget. If it does not beat the heuristic, we report that the learned policy is not yet worth its cost.
• H4 (falsifier). Replacing the verified reward with the 2025 dataset-indexed reward produces no improvement in solve rate over the base model. This condition is included precisely so the original defect can be demonstrated, not just asserted.

6.2 Conditions and baselines

6.3 Metrics

• Primary: solve rate per challenge category (injection, broken access control, sensitive data exposure and the other categories Juice Shop labels), and mean solved challenges per episode at a fixed action budget.
• Secondary: unique challenges solved across a run; actions-to-first-solve; shaped reward reported separately from solve rate so that keyword farming is visible.
• Diagnostic: action validity rate (fraction of generated outputs that parse into an executable HTTP request), and KL divergence from the reference policy to detect collapse.

6.4 Seeds, controls and compute

• Seeds: five seeds per condition (for example 13, 17, 23, 42, 101), fixed and logged. We report mean and standard deviation across seeds, not a single best run.
• Target control: one pinned Juice Shop container version, reset to a clean state at the start of every episode, so the scoreboard starts empty and the solve count is unambiguous.
• Budget control: identical maximum number of actions per episode for every condition, so comparisons are per unit of effort.
• Model: a documented, stock instruction model in the 1.5B to 7B range with LoRA adapters, which keeps the trainable parameter count and memory low enough for a single modern GPU [3] [8].
• Compute: target a single 24 GB to 48 GB GPU, with a run budget on the order of a few GPU-days across all conditions and seeds. The exact budget and the hardware are fixed before the first run and reported with the results.

Everything above is specified so that someone else, including a future version of us, can run it and get a number that answers H1 to H4 rather than a story.

7. How this sits next to published work

Autonomous offensive agents are an active research area, which is one more reason to be precise about what we have and have not shown. Fang and colleagues report that a capable general model, wrapped in an agent with tools and documentation, solved a majority of a set of website vulnerabilities and found issues in real targets, with weaker models performing far worse and open models at the time failing outright [7]. PentestGPT, published at USENIX Security 2024, structures a model into cooperating modules to manage the context of a penetration test and reports strong task completion on a dedicated benchmark [4]. Both are agent-and-prompting results on strong base models. Neither depends on the specific thing we claimed and then withdrew, which was that PPO fine-tuning taught a smaller policy the task from a verified reward. That claim is still open. Our contribution for now is a corrected reward design and a plan to test it, positioned honestly below the current best published results rather than alongside them.

9. Reproducibility

The components needed to reproduce the corrected design are all public or specified: the official OWASP Juice Shop container [6], its scoreboard challenge API as the verified reward source, a stock instruction model with LoRA adapters [3], and a PPO trainer from a current reinforcement-learning library. The 2025 training scripts are not preserved, so we do not publish them; the reward defect is documented from the excerpt in the original post, which is reproduced above in full in the lines that matter. Whether a replication repository will be published has not been decided yet; any public release would be a fresh, scope-limited implementation that takes § 202c StGB into account. No trained-policy results are reported in this version because none from a correctly-rewarded run exist yet. When the replication runs, the seeds, container version, action budget and per-category solve rates specified in section 6 will be reported in full, including the negative conditions.