Conversation
unescape called String#tr on the caller's still-tagged string and only
converted to binary on the next line, so it raised on bytes that are
invalid in that encoding:
CGI.unescape("\x80&".dup.force_encoding("UTF-8"))
# ArgumentError: invalid byte sequence in UTF-8
The C extension returns the string unchanged, and the three siblings in
this file already take a binary copy first: escape and
escapeURIComponent take string.b before transforming, and escape even
defers its own tr! until after the conversion. unescape was the only one
that transformed first.
Take the binary copy first and use tr! on it, matching escape.
This affects the truffleruby and jruby rows, which both resolve
CGI.unescape to this file; the CRuby rows use the C extension and
already behave this way.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The inconsistency
unescapecallsString#tron the caller's still-encoding-tagged string and onlyconverts to binary on the next line:
The other three methods in this file take the binary copy first:
unescapeURIComponentat:65(str = string.b),escapeURIComponentat:52,and
escapeat:23— which even defers its owntr!to:27so it runs on thebinary buffer.
unescapeis the only one of the four that transforms first and converts second,so it is the only one that can raise on bytes that are invalid in the string's
declared encoding.
What a caller sees
The C extension returns
"\x80&"unchanged, and so does the siblingCGI.unescapeURIComponenton the same input.None of the shipped callers rescues it —
CGI#parse(lib/cgi/core.rb:575),cookie parsing (
lib/cgi/cookie.rb:193), the multipart filename path(
lib/cgi/core.rb:736) andlib/cgi/session.rb:446all call it bare.To be precise about scope: I checked whether
CGI.parsereaches this and itdoes not —
String#splitraises first, on both implementations. The divergenceis in
CGI.unescapeitself, which is public and documented.Which CI rows this touches
This repo ships three implementations of
unescape, and I verified each ratherthan assuming:
ext/cgi/escape/escape.coptimized_unescapeRUBY_ENGINE == 'truffleruby'gate atlib/cgi/escape.rb:167The JRuby row is the one I expected to be unaffected, and it is not. Even with
the repo's own extension built (
rake compile,lib/cgi/escape.jar),CGI.method(:unescape).ownerisCGI::Escape— the Ruby definition — not theJava one, although
CGIEscape.java:475does defineunescape. So JRuby raises onmaster too, and the same one-line change fixes it. Verified on JRuby 9.4.15.0 with
the built jar loaded.
Verification
Full suite, run twice per variant — once with the C extension, once forcing the
pure-Ruby path — with the loaded file's md5 printed each time:
The C-mode rows are the important control: the new test passes on master under
the C extension, so it cannot red a CRuby row. The 2 remaining ruby-mode
failures are an artifact of my own harness, not of this change — they are the two
assert_separatelytests, whose subprocess does not inherit the forcedRUBY_ENGINEand therefore loads the C extension while the parent expects pureRuby. They are identical before and after.
Mutation rows, each killed by a different assertion count: dropping the
tr!entirely fails the
a+brow; moving thetr!to after the percent-decode failsthe
%2Brow (it would decode to+and then be wrongly turned into a space).So the ordering is pinned from both sides.
Performance
CGI.unescapeis a query-string hot path, so I measured rather than assumed: twofile variants, order alternated every rep, 14 reps each, 400k calls per rep on a
realistic query string, minimum reported.
About 5% faster, because
string.bplus an in-placetr!allocates oneString where
string.trplus.ballocated two.One behaviour note
string.tr('+', ' ')was character-oriented;string.bthentr!isbyte-oriented. For any ASCII-compatible encoding these are identical, since 0x2B
cannot occur inside a multibyte character. For a non-ASCII-compatible encoding
they could differ — but the C
optimized_unescapeis byte-oriented too, so thismoves the fallback toward the C behaviour rather than away from it.
Disclosure
I used an AI assistant to help find and prepare this change. I reviewed and tested
it myself, and the numbers above are from runs I performed.