Skip to content

Decode on a binary copy in the pure-Ruby unescape - #136

Open
youdie006 wants to merge 1 commit into
ruby:masterfrom
youdie006:fix-unescape-invalid-byte-sequence
Open

youdie006 wants to merge 1 commit into
ruby:masterfrom
youdie006:fix-unescape-invalid-byte-sequence

Conversation

@youdie006

Copy link
Copy Markdown

The inconsistency

unescape calls String#tr on the caller's still-encoding-tagged string and only
converts to binary on the next line:

str = string.tr('+', ' ')   # lib/cgi/escape.rb:36
str = str.b                 # lib/cgi/escape.rb:37

The other three methods in this file take the binary copy first:
unescapeURIComponent at :65 (str = string.b), escapeURIComponent at :52,
and escape at :23 — which even defers its own tr! to :27 so it runs on the
binary buffer.

unescape is the only one of the four that transforms first and converts second,
so it is the only one that can raise on bytes that are invalid in the string's
declared encoding.

What a caller sees

CGI.unescape("\x80&".dup.force_encoding("UTF-8"))
# ArgumentError: invalid byte sequence in UTF-8

The C extension returns "\x80&" unchanged, and so does the sibling
CGI.unescapeURIComponent on the same input.

None of the shipped callers rescues it — CGI#parse (lib/cgi/core.rb:575),
cookie parsing (lib/cgi/cookie.rb:193), the multipart filename path
(lib/cgi/core.rb:736) and lib/cgi/session.rb:446 all call it bare.

To be precise about scope: I checked whether CGI.parse reaches this and it
does not
String#split raises first, on both implementations. The divergence
is in CGI.unescape itself, which is public and documented.

Which CI rows this touches

This repo ships three implementations of unescape, and I verified each rather
than assuming:

rows implementation pristine with this patch
CRuby (ubuntu/macos/windows/mingw/mswin) ext/cgi/escape/escape.c optimized_unescape already correct — it is the oracle unchanged
truffleruby, truffleruby-head this file, via the RUBY_ENGINE == 'truffleruby' gate at lib/cgi/escape.rb:167 raises fixed
jruby, jruby-head also this file raises fixed

The JRuby row is the one I expected to be unaffected, and it is not. Even with
the repo's own extension built (rake compile, lib/cgi/escape.jar),
CGI.method(:unescape).owner is CGI::Escape — the Ruby definition — not the
Java one, although CGIEscape.java:475 does define unescape. So JRuby raises on
master too, and the same one-line change fixes it. Verified on JRuby 9.4.15.0 with
the built jar loaded.

Verification

Full suite, run twice per variant — once with the C extension, once forcing the
pure-Ruby path — with the loaded file's md5 printed each time:

pristine  C-mode    : 505 tests, 1153 assertions, 0 failures, 0 errors
pristine  ruby-mode : 505 tests, 1143 assertions, 2 failures, 1 errors
fixed     C-mode    : 505 tests, 1153 assertions, 0 failures, 0 errors
fixed     ruby-mode : 505 tests, 1148 assertions, 2 failures, 0 errors

The C-mode rows are the important control: the new test passes on master under
the C extension
, so it cannot red a CRuby row. The 2 remaining ruby-mode
failures are an artifact of my own harness, not of this change — they are the two
assert_separately tests, whose subprocess does not inherit the forced
RUBY_ENGINE and therefore loads the C extension while the parent expects pure
Ruby. They are identical before and after.

Mutation rows, each killed by a different assertion count: dropping the tr!
entirely fails the a+b row; moving the tr! to after the percent-decode fails
the %2B row (it would decode to + and then be wrongly turned into a space).
So the ordering is pinned from both sides.

Performance

CGI.unescape is a query-string hot path, so I measured rather than assumed: two
file variants, order alternated every rep, 14 reps each, 400k calls per rep on a
realistic query string, minimum reported.

min (ms/400k) p10 median
master 1479.9 1486.7 1553.3
this PR 1398.7 1453.9 1491.7

About 5% faster, because string.b plus an in-place tr! allocates one
String where string.tr plus .b allocated two.

One behaviour note

string.tr('+', ' ') was character-oriented; string.b then tr! is
byte-oriented. For any ASCII-compatible encoding these are identical, since 0x2B
cannot occur inside a multibyte character. For a non-ASCII-compatible encoding
they could differ — but the C optimized_unescape is byte-oriented too, so this
moves the fallback toward the C behaviour rather than away from it.

Disclosure

I used an AI assistant to help find and prepare this change. I reviewed and tested
it myself, and the numbers above are from runs I performed.

unescape called String#tr on the caller's still-tagged string and only
converted to binary on the next line, so it raised on bytes that are
invalid in that encoding:

  CGI.unescape("\x80&".dup.force_encoding("UTF-8"))
  # ArgumentError: invalid byte sequence in UTF-8

The C extension returns the string unchanged, and the three siblings in
this file already take a binary copy first: escape and
escapeURIComponent take string.b before transforming, and escape even
defers its own tr! until after the conversion. unescape was the only one
that transformed first.

Take the binary copy first and use tr! on it, matching escape.

This affects the truffleruby and jruby rows, which both resolve
CGI.unescape to this file; the CRuby rows use the C extension and
already behave this way.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant