JSON encoder corrupts data, adds random memory contents to end of string

[mniip]

Bug Report

Describe the bug

When JSON with a long string gets re-encoded by fluent-bit, several bytes of random garbage gets added to the end of the string. The garbage is 1-15 or 1-4 bytes (depending on the SIMD supported in the host CPU) of fluent-bit's memory -- usually snippets of other log streams flowing through the process.

To Reproduce

fluent-bit.conf:

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    info
    Parsers_File parsers.conf

[INPUT]
    Name              tail
    Tag               mwe
    Path              input.log
    Parser            mwe-json
    Read_from_Head    On
    Skip_Empty_Lines  On
    Buffer_Chunk_Size 256KB
    Buffer_Max_Size   20MB
    Mem_Buf_Limit     512MB

[OUTPUT]
    Name        loki
    Match       mwe
    Host        127.0.0.1
    Port        3100
    Line_Format json
    Labels      job=mwe
    Remove_keys seq
    Retry_Limit no_limits

parsers.conf:

[PARSER]
    Name        mwe-json
    Format      json
    Time_Key    at
    Time_Format %Y-%m-%dT%H:%M:%S.%L
    Time_Keep   On

Logs, ingesting which reproduces the corruption: long_unicode_payloads.json
Observe that each logline has the shape {"seq":123,"msg":"<TEST ®®®®®®®®®®®®®®®>123,123,...,123,123-------</TEST>"} with .msg being ~40KB ending in </TEST>:

jq < long_unicode_payloads.json '.msg[-7:]'

A simple script that pretends to be Loki and dumps all pushed payloads to stdout: loki_http_to_stdout.py

The multibyte characters in logs are required for reproduction, and so is a Loki [OUTPUT].

Steps to reproduce the problem:

PORT=3100 python3 loki_http_to_stdout.py > loki.jsonl &
touch input.log
docker run --rm --network host -v "$PWD:/mwe" -w /mwe fluent/fluent-bit:5.1.0 /fluent-bit/bin/fluent-bit -c fluent-bit.conf &
cat long_unicode_payloads.json > input.log

Wait a few seconds for it to settle.

kill -INT %1  # python3
kill -INT %2  # docker

Now inspect the contents of loki.jsonl:

jq < loki.jsonl '.streams[].values[][1] | fromjson | .msg[-32:]'

The JSON received by Loki is different: .msg now doesn't end in </TEST> but rather has a few bytes added at the end, with infixes of other loglines.
On my machine it looks like this:

"0,0,0,0,0,0,0,0,0,0,0,0-</TEST>6"
"1,1,1,1,1,1,1,1,1,1,1,1</TEST>16"
"2,2,2,2---------------</TEST>,16"
"3,3,3,3--------------</TEST>2,2,"
"4,4,4,4-------------</TEST>,3,3,"
"5,5,5,5------------</TEST>2,2,2,"
...
"1,191,191----</TEST>0,190,190,19"
"2,192,192---</TEST>90,190,190,19"
"3,193,193--</TEST>192,192,192,19"
"4,194,194-</TEST>,193,193,193,19"
"5,195,195---------------</TEST>9"
"6,196,196--------------</TEST>18"
"7,197,197-------------</TEST>,18"
"8,198,198------------</TEST>>,18"
"9,199,199-----------</TEST>98,19"

Some additional pointers for reproducing:

• Only happens for long lines, the shortest corruption I found in our production logs was 24KB.
• The long value has to be the last key inside the JSON object. Lua filters tend to randomize the ordering of keys.
• There have to be multibyte characters inside the string.
• I believe the root case to be flb_utils_write_str_escaped incorrectly deciding whether a SIMD batch of bytes can be grabbed before the string ends, and grabbing some bytes past the end of the buffer. Whenever those uninitialized memory bytes are all "no escaping needed" bytes (>=0x20, <0x80, not ", not \), the bug reproduces.
  • With a simple file output, the JSON seems to be surrounded by msgpack framing in memory (high bit bytes).
  • Using a Loki output with Remove_keys seems to ensure that the JSON lives in a freshly allocated buffer, potentially reused from a previous log line.

Expected behavior
JSON strings should decode to the same as the orignal.

Your Environment

• Version used: in production we use 5.0.3, but the above MWE works on 5.1.0 also
• Operating System: x86_64 linux

[seshan18]

I like to work on this issue , Would you assign this to me !?