Repository navigation
FIX: read the real provenance columns in the JailBreakV-28K loader - #3162
Merged
Roman Lutz (romanlutz) merged 3 commits intoOct 13, 2026
Merged
Roman Lutz (romanlutz) merged 3 commits into
Roman Lutz (romanlutz) merged 3 commits into
Conversation
The loader copied a transfer_attack_type column that does not exist upstream, so every row carried an empty value while the real from and transfer_from_llm columns were dropped. Read those instead, mirroring the RedTeam-2K loader, and make the test fixture use the real column set. Closes microsoft#3161
Roman Lutz (romanlutz)
approved these changes
Oct 13, 2026
Roman Lutz (romanlutz)
left a comment
Roman Lutz (romanlutz)
left a comment
Contributor
There was a problem hiding this comment.
Nice find!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Closes #3161.
_JailbreakV28KDatasetcopied atransfer_attack_typecolumn into every seed's metadata, but the upstream dataset has never had that column. Its columns areid, jailbreak_query, redteam_query, format, policy, image_path, from, selected_mini, transfer_from_llm. Because the lookup used.get(..., ""), all 28,000 rows were emitted withmetadata["transfer_attack_type"] == "", and the two columns that actually carry provenance (from, and thetransfer_from_llmflag that separates the LLM-transfer rows from the MLLM-specific rows) were discarded. The sibling RedTeam-2K loader already keepsfrom.This replaces the phantom key with
"from": item.get("from", ""), mirroring the RedTeam-2K loader, and keepstransfer_from_llmas a boolean when the column is present. No other behaviour changes.Tests and Documentation
tests/unit/datasets/test_jailbreakv_28k_dataset.py:_rowfixture now mirrors the real upstream column set instead of fabricatingtransfer_attack_type, which is what let CI pass against a column that does not exist.test_fetch_dataset_happy_pathassertsfromandtransfer_from_llmreach all three seeds and thattransfer_attack_typeis no longer emitted.test_fetch_dataset_metadata_keeps_mllm_row_provenancecovers an MLLM row with the flagFalse.test_fetch_dataset_metadata_omits_missing_transfer_flagcovers a row without the column.With only the source change reverted, 3 tests fail; with it, the file is 20 passed.
ruff check,ruff format --checkandty checkare clean. No doc page lists this loader's metadata keys, so no JupyText run was needed.