Fix data race on AbstractMetaObjectBase's owning-loader list - #236
Open
thomasmoore-torc wants to merge 2 commits into
Open
thomasmoore-torc wants to merge 2 commits into
thomasmoore-torc wants to merge 2 commits into
Conversation
Author
|
For reference, tracking the two related PRs mentioned above, both already merged:
This PR fixes a third, still-unaddressed race in the same area (AbstractMetaObjectBase's owning-loader list). |
AbstractMetaObjectBase::addOwningClassLoader()/removeOwningClassLoader()/ isOwnedBy() (and the other accessors of associated_class_loaders_) took no lock of their own. createInstance() in class_loader_core.hpp released getPluginBaseToFactoryMapMapMutex() before reading factory->isOwnedBy(loader), so a concurrent addOwningClassLoader()/removeOwningClassLoader() on another thread (e.g. from loadLibrary()'s already-loaded branch, or from onPluginDeletion()) could race with that read - and, since it's a std::vector, with any other concurrent read/write of the same metaobject's owner list. Reported by ThreadSanitizer as a data race on std::vector<ClassLoader *>'s push_back()/erase() (via vector's internal reallocation), hit by constructing many independent ClassLoader instances for the same library concurrently - e.g. one rosbag2_cpp::Writer per thread, each internally constructing its own pluginlib::ClassLoader for the serialization format converter. Give AbstractMetaObjectBaseImpl its own recursive_mutex guarding associated_class_loaders_, so every accessor is self-contained and safe regardless of what lock (if any) the caller happens to be holding. Also extend createInstance()'s existing getPluginBaseToFactoryMapMapMutex() scope to cover the isOwnedBy() checks themselves, since between releasing that mutex and reading isOwnedBy(), the factory pointer looked up under it could in principle be invalidated by a concurrent destroyMetaObjectsForLibrary() on another thread. This is separate from ros#234 (loadLibrary()/unloadLibrary() overlap) and ros#231 (has_unmanaged_instance_been_created_): those fixed different unsynchronized globals in this same area. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: thomasmoore-torc <thomas.moore@torc.ai>
The regression test constructed and destroyed STRESS_TEST_NUM_THREADS (500) independent ClassLoaders for the same library with no synchronization between their lifetimes, so the library's refcount could repeatedly bounce to zero under contention, forcing many real dlopen()/dlclose() cycles - and occasionally racing two of those "last owner out" unloads against each other, an existing, separate bug in this library's load/unload refcounting unrelated to the fix under test here. Add an `anchor` ClassLoader kept alive for the whole test so the refcount never reaches zero and that unrelated path is never exercised, isolating the associated_class_loaders_ race this fix actually targets. Verified in a local colcon/ROS 2 rolling Docker environment with ThreadSanitizer: unpatched, this reliably reports the addOwningClassLoader()/removeOwningClassLoader() race (9-10 reports per run); with the fix, zero reports across many runs, and the full `colcon test` suite for class_loader passes 100% (9/9). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: thomasmoore-torc <thomas.moore@torc.ai>
thomasmoore-torc
force-pushed
the
fix/isowned-by-race-and-createinstance-early-unlock
branch
from
September 22, 2026 16:04
80ed16e to
18bc709
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
AbstractMetaObjectBase::addOwningClassLoader()/removeOwningClassLoader()/isOwnedBy()(and the other accessors ofassociated_class_loaders_) take no lock of their own.createInstance()inclass_loader_core.hppreleasesgetPluginBaseToFactoryMapMapMutex()before readingfactory->isOwnedBy(loader), so a concurrentaddOwningClassLoader()/removeOwningClassLoader()on another thread (e.g. fromloadLibrary()'s already-loaded branch) can race with that read — and, since it's astd::vector, with any other concurrent read/write of the same metaobject's owner list.I hit this with ThreadSanitizer as a data race on
std::vector<ClassLoader *>'spush_back()/erase()(via the vector's internal reallocation). It's straightforward to trigger by constructing many independentClassLoaderinstances for the same library concurrently — e.g. onerosbag2_cpp::Writerper thread, each internally constructing its ownpluginlib::ClassLoaderfor the serialization format converter, all for the first time, simultaneously.This is separate from #234 (
loadLibrary()/unloadLibrary()overlap) and #231 (has_unmanaged_instance_been_created_) — both already merged, and both fix different unsynchronized globals in this same area. I checked the currentrollingHEAD before writing this:createInstance()still releases the mutex before theisOwnedBy()checks, andmeta_object.cpp's accessors are still fully unguarded.Fix
AbstractMetaObjectBaseImplits ownrecursive_mutexguardingassociated_class_loaders_, so every accessor (addOwningClassLoader,removeOwningClassLoader,isOwnedBy,isOwnedByAnybody,getAssociatedClassLoadersCount,getAssociatedClassLoader) is self-contained and safe regardless of what lock, if any, the caller happens to be holding.createInstance()'s existinggetPluginBaseToFactoryMapMapMutex()scope to cover theisOwnedBy()checks themselves: between releasing that mutex (previously done right after the map lookup) and readingisOwnedBy(), thefactorypointer looked up under it could in principle be invalidated by a concurrentdestroyMetaObjectsForLibrary()on another thread. The mutex-per-vector fix above closes the vector-corruption race on its own; this closes the separate use-after-free risk on thefactorypointer itself.Testing
Added
ClassLoaderTest.threadSafetyMultipleLoadersPerLibrarytotest/utest.cpp. Unlike the existingClassLoaderTest.threadSafety(oneClassLoadershared by allSTRESS_TEST_NUM_THREADSthreads, soisOwnedBy()is always queried with the loader that already, and permanently, owns the metaobject), this constructs a separateClassLoaderper thread for the same library, so construction/destruction concurrently mutatesassociated_class_loaders_while other threads concurrently read it viaisOwnedBy()increateInstance(). AnanchorClassLoaderis kept alive for the whole test so the library's owner count never reaches zero and the worker threads never trigger the actual unload/dlclose()path — that path has its own pre-existing, unrelated refcounting bug (a library can bedlclose()-d twice when independentClassLoaders for the same library race to be "last owner out"), which isn't what this test is targeting and would otherwise fail it intermittently regardless of this fix.I built and ran this locally against a
ros:rolling-ros-basecontainer (colcon,-DBUILD_TESTING=ON), both with a plain build and with ThreadSanitizer. Unpatched, ThreadSanitizer reliably reports theaddOwningClassLoader()/removeOwningClassLoader()race onassociated_class_loaders_(9-10 reports per run); with the fix, zero reports across many runs. The fullcolcon test --packages-select class_loadersuite passes 100% (9/9) with the fix applied.Generative AI
This fix was investigated, written, and iteratively verified (ThreadSanitizer, stress-testing, and a local colcon/ROS 2 rolling build) by Claude (Anthropic), operating as Claude Code, at the direction of and reviewed by me. The regression test and this PR description were also written by Claude; I reviewed both before submitting.