Double free in mariadb #220

Activity

1. quinnj commented on Aug 1, 2026

   

   quinnj

   on Aug 1, 2026

   Member

   More actions

   I hit this same crash and dug into it — I believe I have the root cause, a fast reproducer, and controls that isolate the trigger.

   TL;DR: the API.MYSQL_STMT finalizer calls mysql_stmt_close, which is not a client-side free — it sends COM_STMT_CLOSE over the connection's socket. Finalizers run on whichever thread triggers GC, and GC (plus finalizers) runs while other threads are inside mysql_* ccalls. So an abandoned prepared statement's finalizer does socket I/O on a MYSQL* concurrently with a legitimate call on another thread — no user-level lock can prevent it, because the user's lock isn't (and can't be) held by GC. libmariadb connections are not thread-safe, so this is memory-unsafe.

   The finalizer in question (src/api/apitypes.jl#L56-L61):

   finalizer(stmt) do x
       if x.ptr != C_NULL
           mysql_stmt_close(x.ptr)   # sends COM_STMT_CLOSE over the wire
           x.ptr = C_NULL
       end
   end

   API.MYSQL (mysql_close sends COM_QUIT) and API.MYSQL_RES (mysql_free_result reads any un-fetched rows off the wire) have the same problem in their finalizers.

   Caught in the act

   macOS crash report from the abort, showing two threads simultaneously inside libmariadb on the same connection — thread 3 is a GC finalizer closing a statement, thread 5 is the application thread (which held a ReentrantLock serializing every DB call):

   Thread 3:
     libsystem_c.dylib        abort
     libsystem_malloc.dylib   malloc_vreport / malloc_zone_error / free_tiny_botch   <- double free
     libssl.3.dylib           ossl_ssl_connection_free
     libssl.3.dylib           SSL_free
     libmariadb.3.dylib       ma_tls_close
     libmariadb.3.dylib       ma_pvio_close
     libmariadb.3.dylib       end_server
     libmariadb.3.dylib       mthd_my_send_cmd
     libmariadb.3.dylib       net_stmt_close
     libmariadb.3.dylib       mysql_stmt_close
     libjulia-internal        jl_gc_run_finalizers_in_list      <- GC finalizer
     libjulia-internal        run_finalizers / ijl_gc_collect / maybe_collect / jl_gc_small_alloc_inner
   
   Thread 5 (faulting):
     libsystem_c.dylib        abort
     libsystem_malloc.dylib   malloc_vreport / malloc_zone_error / free_tiny_botch   <- double free
     libssl.3.dylib           SSL_CTX_free
     libssl.3.dylib           SSL_free
     libmariadb.3.dylib       ma_tls_close
     libmariadb.3.dylib       ma_pvio_close
     libmariadb.3.dylib       ma_net_safe_read
     libmariadb.3.dylib       mthd_my_read_query_result
     libmariadb.3.dylib       mysql_commit                      <- application call, lock held
   

   The interleaved use of one SSL* from two threads corrupts OpenSSL state; then both threads' error paths call ma_pvio_close → ma_tls_close → SSL_free on the same SSL — the double free in this issue's title. Across runs I've seen the abort land in SSL_write (under mysql_stmt_execute), in SSL_free/SSL_CTX_free (as above and as in the OP's trace: mysql_stmt_execute → mthd_my_send_cmd → ma_pvio_close → ma_tls_close → SSL_free), and once as a straight SIGSEGV jumping through a freed pointer.

   Why it looks TLS-correlated

   COM_STMT_CLOSE is fire-and-forget (no server response). On a plaintext socket, the racing finalizer write usually just interleaves at packet granularity and nothing visibly breaks — no shared client-side state is being mutated concurrently except the net buffer. Under TLS, both threads mutate the same SSL record layer (cipher sequence, buffers), so you get (2026): TLS/SSL error: ssl/tls alert bad record mac, wedged connections, or heap corruption → SIGABRT/SIGSEGV. The race is present without TLS too — one plaintext run here produced ArgumentError: unsafe_wrap: pointer 0x9 is not properly aligned to 8 bytes from a result fetch, i.e. silent client-side corruption instead of a crash.

   Reproducer

   Crashes within seconds-to-a-minute on every run here (3/3 runs ended in SIGABRT or SIGSEGV; interspersed rounds die with bad record mac). All DB access is behind a single ReentrantLock, as in the OP.

   # julia -t 8 repro.jl   (against e.g.: docker run -d -p 3306:3306 \
   #   -e MYSQL_ALLOW_EMPTY_PASSWORD=yes -e MYSQL_DATABASE=test mysql:8)
   using MySQL, DBInterface
   
   const PORT = 3306
   myconnect() = DBInterface.connect(MySQL.Connection, "127.0.0.1", "root", "";
                                     db="test", port=PORT)
   
   let conn = myconnect()
       DBInterface.execute(conn, "DROP TABLE IF EXISTS finalizer_race")
       DBInterface.execute(conn, "CREATE TABLE finalizer_race (k INT PRIMARY KEY, v INT)")
       DBInterface.execute(conn, "INSERT INTO finalizer_race VALUES (1, 0)")
       DBInterface.close!(conn)
   end
   
   function hammer(conn, secs)
       lk = ReentrantLock()            # serializes ALL user access to conn
       done = Threads.Atomic{Bool}(false)
       firsterr = Ref{Any}(nothing)
       @sync begin
           for _ in 1:6
               Threads.@spawn try
                   while !done[]
                       lock(lk) do
                           # idiomatic prepared-statement use; statements abandoned
                           # (no close!) -> garbage -> finalizer sends COM_STMT_CLOSE
                           # from whatever thread runs GC
                           stmt = DBInterface.prepare(conn,
                               "UPDATE finalizer_race SET v = v + 1 WHERE k = 1")
                           DBInterface.execute(stmt)
                           stmt2 = DBInterface.prepare(conn,
                               "SELECT v FROM finalizer_race WHERE k = 1")
                           DBInterface.execute(stmt2)
                       end
                   end
               catch e
                   firsterr[] === nothing && (firsterr[] = e)
                   done[] = true
               end
           end
           Threads.@spawn while !done[]   # GC pressure: finalizers run on this thread
               GC.gc(false)
           end
           Threads.@spawn (t0 = time(); while !done[] && time() - t0 < secs; sleep(0.2); end; done[] = true)
       end
       return firsterr[]
   end
   
   for r in 1:10
       conn = myconnect()
       err = hammer(conn, 20)
       println("round $r: ", err === nothing ? "clean" : sprint(showerror, err))
       try DBInterface.close!(conn) catch end
   end

   Controls

   configuration	result
   TLS + abandoned statements (above)	SIGABRT/SIGSEGV every run (plus bad record mac rounds)
   TLS + DBInterface.close!(stmt) inside the lock	no crash (4×20s rounds clean)
   plaintext (server with tls_version=) + abandoned statements	no crash, but one round died with unsafe_wrap: pointer 0x9 is not properly aligned

   The explicit-close control is the smoking gun: the only difference is whether mysql_stmt_close runs under the user's lock or in a GC finalizer.

   Single-threaded Julia (-t 1) is safe — finalizers can't run concurrently with a ccall on the same thread. Any -t ≥ 2 workload that ever lets a Statement, un-drained MYSQL_RES, or Connection be GC'd while the connection is in use is exposed, which is why this shows up as rare, unexplainable crashes in long test suites.

   Environment

   • macOS 15.6.1 aarch64 (also reproduces the OP's Linux x86_64 trace shape exactly)
   • Julia 1.12.6, 8 threads
   • MySQL.jl 1.5.1 (bug present on current main, 1.5.2) / MariaDB_Connector_C_jll 3.4.9+1 / OpenSSL_jll 3.5.4
   • Server: mysql:8 (8.4.11) in Docker

   Two aggravating factors worth noting:

   1. With MariaDB_Connector_C_jll 3.4.x, TLS is on by default for every connection that talks to a TLS-capable server, and ssl_mode=SSL_MODE_DISABLED cannot turn it off (that option maps onto a different, unrelated option in libmariadb — filing separately). So essentially all users of current MySQL.jl + mysql:8 are in the crashy configuration.
   2. On macOS the @rpath/libssl.3.dylib reference resolves to Julia's bundled lib/julia/libssl.3.dylib rather than OpenSSL_jll's — didn't affect this bug, but it means the OpenSSL actually in use isn't the one the jll pins.

   Fix sketch

   The invariant the fix needs: finalizers must never touch the socket. One workable shape:

   • Give Connection a small reap queue (spinlock-guarded Vector{Ptr{Cvoid}}). The MYSQL_STMT finalizer pushes the raw ptr and returns — no libmariadb call.
   • Drain the queue (calling mysql_stmt_close on each ptr) at the start of every user-facing operation on that connection (prepare, execute, transaction, close!…). Those sites are already serialized by whatever discipline the user applies, so the close packets go out under the same protection as everything else.
   • DBInterface.close!(conn) drains the queue before mysql_close. If the connection is already closed/dead, drop the ptrs via plain mysql_stmt_close after mysql_close has invalidated them (libmariadb turns that into a local free), or leak them — either is better than a crash.
   • Same treatment for MYSQL_RES (an un-drained result's mysql_free_result reads from the socket) and the MYSQL finalizer's mysql_close.

   Happy to turn this into a PR if the approach sounds right.
2. quinnj

   added a commit that references this issue on Aug 2, 2026

   Never do socket I/O in finalizers; map ssl_mode onto real Connector/C…

   ...

   This issue will close once commit 6144a81 is merged into the 'main' branch.

   6144a81
3. quinnj

   closed this as completedin #241on Aug 2, 2026