BUG JailBreakV-28K loader reads a transfer_attack_type column that does not exist upstream and drops the real from / transfer_from_llm provenance #3161

Describe the bug

_JailbreakV28KDataset (pyrit/datasets/seed_datasets/remote/jailbreakv_28k_dataset.py) copies per-row provenance into each seed's metadata:

row_metadata: dict[str, str | int] = {
    "source_format": item.get("format", ""),
    "transfer_attack_type": item.get("transfer_attack_type", ""),
    "policy": raw_policy,
    "image_path_raw": image_rel_path,
}

The upstream dataset has no transfer_attack_type column. Its columns, in both the JailBreakV_28K and mini_JailBreakV_28K splits, are:

id, jailbreak_query, redteam_query, format, policy, image_path, from, selected_mini, transfer_from_llm

(from https://datasets-server.huggingface.co/rows?dataset=JailbreakV-28K/JailBreakV-28k&config=JailBreakV_28K&split=mini_JailBreakV_28K; the HF commit history shows no data change since the 2024-06-22 upload). The dataset card documents from as the source of the row and transfer_from_llm as True when the jailbreak query was transferred from a text-only LLM attack, which is what distinguishes the ~20k LLM-transfer rows from the ~8k MLLM-specific rows.

Because .get(..., "") masks the missing key, every one of the 28,000 rows is emitted with metadata["transfer_attack_type"] == "", and the two columns that actually carry provenance are discarded. The sibling _JailbreakVRedTeam2KDataset loader keeps "from": item.get("from", "") correctly.

The unit test passes only because its fixture fabricates a transfer_attack_type column (tests/unit/datasets/test_jailbreakv_28k_dataset.py), so CI never sees the real column set. This is the same masking pattern as #2987.

Steps/Code to Reproduce

No network needed: feed the loader a row with the real upstream columns.

import pathlib
from unittest.mock import AsyncMock, patch

from pyrit.datasets.seed_datasets.remote.jailbreakv_28k_dataset import _JailbreakV28KDataset

tmp = pathlib.Path("/tmp/jbv")
(tmp / "JailBreakV_28k" / "llm_transfer_attack").mkdir(parents=True, exist_ok=True)
(tmp / "JailBreakV_28K.zip").write_bytes(b"fake")
(tmp / "JailBreakV_28k" / "llm_transfer_attack" / "SD_related_566.png").write_bytes(b"\x89PNG")

row = {
    "id": 0, "jailbreak_query": "...", "redteam_query": "...", "format": "Template",
    "policy": "Unethical Behavior", "image_path": "llm_transfer_attack/SD_related_566.png",
    "from": "GPT Generate", "selected_mini": True, "transfer_from_llm": True,
}
loader = _JailbreakV28KDataset(zip_dir=str(tmp))
with patch.object(loader, "_fetch_from_huggingface_async", new=AsyncMock(return_value=[row])):
    import asyncio
    dataset = asyncio.run(loader.fetch_dataset_async())
print(dataset.seeds[0].metadata)

Expected Results

{'source_format': 'Template', 'from': 'GPT Generate', 'policy': 'Unethical Behavior', 'image_path_raw': 'llm_transfer_attack/SD_related_566.png', 'row_id': '0', 'transfer_from_llm': True}

Actual Results

{'source_format': 'Template', 'transfer_attack_type': '', 'policy': 'Unethical Behavior', 'image_path_raw': 'llm_transfer_attack/SD_related_566.png', 'row_id': '0'}

Suggested fix

Replace the phantom key with "from": item.get("from", ""), mirroring the RedTeam-2K loader, and keep transfer_from_llm as a boolean when the column is present. Update the test fixture to use the real column set and assert on from / transfer_from_llm, plus a regression assertion that transfer_attack_type is no longer emitted.

I have the fix with regression tests ready and can open the PR.

Versions

• OS: macOS 26.5
• Python version: 3.14.8
• PyRIT version: 1.2.0.dev0, installed from main (93bf08e4) in editable mode