Skip to content

TSan reports possible sample-recycling race between new_sample and pull_sample_typed #305

Description

@cboulay

Observed behavior

ThreadSanitizer intermittently reports a data race between lsl::factory::new_sample(double, bool) on the inlet's receiving thread and lsl::data_receiver::pull_sample_typed<int>() on the application thread.

This was observed while validating #291 at commit 4a3e8865e776cf9fa8d93b7533edfe433e12ad24. Filing separately for later investigation; this is not the outlet queue producer race addressed by that PR.

Evidence

  • The focused [outlet],[open],[reopen],[sync] suite passed 1,710 assertions across 20 test cases, with no TSan warnings in that run.
  • Repeating have_consumers becomes false after disconnect during push 100 times produced this report in 15 runs, one warning per affected run.
  • All 100 runs passed their Catch2 assertions (5 assertions per run). The 15 affected processes nevertheless exited unsuccessfully with SIGABRT after TSan reported the warning. There were no timeouts.
  • No reports of the outlet queue producer race were observed in these runs.
  • The author of Fix have_consumers staying true after an idle consumer disconnects (#267) #291 also reported the same function pair both before and after the queue fix: earlier report. This suggests it is pre-existing; the local validation described here did not independently rerun the pre-fix commit.

Environment

  • macOS 26.6.2 (25G83), native arm64
  • Xcode 27.0 (27A266a)
  • Apple Clang 21.0.0 (clang-2100.3.34.2)
  • Debug build; library, test executable, and fetched Catch2 built with TSan
  • TSAN_OPTIONS="history_size=7 halt_on_error=0"; no suppressions

Before testing liblsl, a minimal empty program ran successfully under TSan and a deliberately racy program produced the expected data-race warning. These checks also passed with Homebrew LLVM 23.1.1; the liblsl results above used Apple Clang.

Reproduction

Starting from the PR commit above:

cmake -S . -B build-tsan \
  -DCMAKE_BUILD_TYPE=Debug \
  -DLSL_UNITTESTS=ON \
  -DLSL_FRAMEWORK=ON \
  -DLSL_TESTS_PREFER_SYSTEM_CATCH2=OFF \
  -DCMAKE_OSX_ARCHITECTURES=arm64 \
  -DCMAKE_C_COMPILER=/usr/bin/clang \
  -DCMAKE_CXX_COMPILER=/usr/bin/clang++ \
  -DCMAKE_C_FLAGS="-fsanitize=thread -fno-omit-frame-pointer" \
  -DCMAKE_CXX_FLAGS="-fsanitize=thread -fno-omit-frame-pointer" \
  -DCMAKE_EXE_LINKER_FLAGS="-fsanitize=thread" \
  -DCMAKE_SHARED_LINKER_FLAGS="-fsanitize=thread"
cmake --build build-tsan --target lsl_test_exported --parallel 8

Run repeatedly, retaining each report and bounding each run to 120 seconds:

import os
import subprocess
from pathlib import Path

logs = Path("tsan-race-logs")
logs.mkdir(exist_ok=True)
env = dict(os.environ, TSAN_OPTIONS="history_size=7 halt_on_error=0")
for i in range(100):
    with (logs / f"run-{i:03}.log").open("w") as log:
        try:
            result = subprocess.run(
                ["./build-tsan/testing/lsl_test_exported",
                 "have_consumers becomes false after disconnect during push"],
                stdout=log, stderr=subprocess.STDOUT, env=env, timeout=120)
            print(i, result.returncode)
        except subprocess.TimeoutExpired:
            print(i, "timeout")

Representative report

The report identifies an 8-byte write in new_sample() and an 8-byte read in pull_sample_typed<int>() to the same address in a factory-allocated heap block. Full representative warning below (source line symbolization was unavailable in this run):

WARNING: ThreadSanitizer: data race (pid=89863)
  Write of size 8 at 0x00010b600050 by thread T5:
    #0 lsl::factory::new_sample(double, bool) <null> (lsl:arm64+0xd7f80)
    #1 lsl::data_receiver::data_thread() <null> (lsl:arm64+0x1ef4c)
    #2 lsl::data_receiver::data_thread() <null> (lsl:arm64+0x1d8f8)
    #3 void std::__1::__thread_execute[abi:nqe220106]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (lsl::data_receiver::*)(), lsl::data_receiver*, 0ul, 1ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (lsl::data_receiver::*)(), lsl::data_receiver*>&, std::__1::__integer_sequence<unsigned long, 0ul, 1ul>) <null> (lsl:arm64+0x499e8)
    #4 void* std::__1::__thread_proxy[abi:nqe220106]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (lsl::data_receiver::*)(), lsl::data_receiver*>>(void*) <null> (lsl:arm64+0x49340)

  Previous read of size 8 at 0x00010b600050 by main thread:
    #0 double lsl::data_receiver::pull_sample_typed<int>(int*, unsigned int, double) <null> (lsl:arm64+0x1c28c)
    #1 double lsl::stream_inlet_impl::pull_sample_noexcept<int>(int*, int, double, lsl_error_code_t*) <null> (lsl:arm64+0x739c0)
    #2 lsl_pull_sample_i <null> (lsl:arm64+0x73918)
    #3 lsl::stream_inlet::pull_sample(int*, int, double) <null> (lsl_test_exported:arm64+0x10000d4d8)
    #4 (anonymous namespace)::CATCH2_INTERNAL_TEST_2() <null> (lsl_test_exported:arm64+0x1001798ac)
    #5 Catch::(anonymous namespace)::TestInvokerAsFunction::invoke() const <null> (lsl_test_exported:arm64+0x10016189c)
    #6 Catch::TestCaseHandle::invoke() const <null> (lsl_test_exported:arm64+0x100130944)
    #7 Catch::RunContext::invokeActiveTestCase() <null> (lsl_test_exported:arm64+0x100130724)
    #8 Catch::RunContext::runCurrentTest(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&) <null> (lsl_test_exported:arm64+0x10012e160)
    #9 Catch::RunContext::runTest(Catch::TestCaseHandle const&) <null> (lsl_test_exported:arm64+0x10012d9cc)
    #10 Catch::(anonymous namespace)::TestGroup::execute() <null> (lsl_test_exported:arm64+0x1000a283c)
    #11 Catch::Session::runInternal() <null> (lsl_test_exported:arm64+0x1000a1c30)
    #12 Catch::Session::run() <null> (lsl_test_exported:arm64+0x1000a184c)
    #13 main <null> (lsl_test_exported:arm64+0x10004484c)

  Location is heap block of size 6192 at 0x00010b600000 allocated by main thread:
    #0 operator new[](unsigned long) <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x91bc0)
    #1 lsl::factory::factory(lsl_channel_format_t, unsigned int, unsigned int) <null> (lsl:arm64+0xd7b14)
    #2 lsl::factory::factory(lsl_channel_format_t, unsigned int, unsigned int) <null> (lsl:arm64+0xd7e98)
    #3 lsl::data_receiver::data_receiver(lsl::inlet_connection&, int, int) <null> (lsl:arm64+0x1ca8c)
    #4 lsl::data_receiver::data_receiver(lsl::inlet_connection&, int, int) <null> (lsl:arm64+0x1d084)
    #5 lsl::stream_inlet_impl::stream_inlet_impl(lsl::stream_info_impl const&, int, int, bool) <null> (lsl:arm64+0x784e0)
    #6 lsl::stream_inlet_impl::stream_inlet_impl(lsl::stream_info_impl const&, int, int, bool) <null> (lsl:arm64+0x783f0)
    #7 lsl::stream_inlet_impl* create_object_noexcept<lsl::stream_inlet_impl, lsl::stream_info_impl&, int&, int&, bool>(lsl::stream_info_impl&, int&, int&, bool&&) <null> (lsl:arm64+0x71890)
    #8 lsl_create_inlet_ex <null> (lsl:arm64+0x71700)
    #9 lsl::stream_inlet::stream_inlet(lsl::stream_info const&, int, int, bool, lsl_transport_options_t) <null> (lsl_test_exported:arm64+0x100009a34)
    #10 lsl::stream_inlet::stream_inlet(lsl::stream_info const&, int, int, bool, lsl_transport_options_t) <null> (lsl_test_exported:arm64+0x100004cdc)
    #11 (anonymous namespace)::CATCH2_INTERNAL_TEST_2() <null> (lsl_test_exported:arm64+0x1001795c4)
    #12 Catch::(anonymous namespace)::TestInvokerAsFunction::invoke() const <null> (lsl_test_exported:arm64+0x10016189c)
    #13 Catch::TestCaseHandle::invoke() const <null> (lsl_test_exported:arm64+0x100130944)
    #14 Catch::RunContext::invokeActiveTestCase() <null> (lsl_test_exported:arm64+0x100130724)
    #15 Catch::RunContext::runCurrentTest(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&) <null> (lsl_test_exported:arm64+0x10012e160)
    #16 Catch::RunContext::runTest(Catch::TestCaseHandle const&) <null> (lsl_test_exported:arm64+0x10012d9cc)
    #17 Catch::(anonymous namespace)::TestGroup::execute() <null> (lsl_test_exported:arm64+0x1000a283c)
    #18 Catch::Session::runInternal() <null> (lsl_test_exported:arm64+0x1000a1c30)
    #19 Catch::Session::run() <null> (lsl_test_exported:arm64+0x1000a184c)
    #20 main <null> (lsl_test_exported:arm64+0x10004484c)

  Thread T5 (tid=21578428, running) created by main thread at:
    #0 pthread_create <null> (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x33bac)
    #1 std::__1::__libcpp_thread_create[abi:nqe220106](_opaque_pthread_t**, void* (*)(void*), void*) <null> (lsl:arm64+0x492a8)
    #2 std::__1::thread::thread[abi:nqe220106]<void (lsl::data_receiver::*)(), lsl::data_receiver*, 0>(void (lsl::data_receiver::*&&)(), lsl::data_receiver*&&) <null> (lsl:arm64+0x49098)
    #3 std::__1::thread::thread[abi:nqe220106]<void (lsl::data_receiver::*)(), lsl::data_receiver*, 0>(void (lsl::data_receiver::*&&)(), lsl::data_receiver*&&) <null> (lsl:arm64+0x1f5c8)
    #4 lsl::data_receiver::open_stream(double) <null> (lsl:arm64+0x1d41c)
    #5 lsl::stream_inlet_impl::open_stream(double) <null> (lsl:arm64+0x724b4)
    #6 lsl_open_stream <null> (lsl:arm64+0x72130)
    #7 lsl::stream_inlet::open_stream(double) <null> (lsl_test_exported:arm64+0x1000042c8)
    #8 (anonymous namespace)::CATCH2_INTERNAL_TEST_2() <null> (lsl_test_exported:arm64+0x1001795d4)
    #9 Catch::(anonymous namespace)::TestInvokerAsFunction::invoke() const <null> (lsl_test_exported:arm64+0x10016189c)
    #10 Catch::TestCaseHandle::invoke() const <null> (lsl_test_exported:arm64+0x100130944)
    #11 Catch::RunContext::invokeActiveTestCase() <null> (lsl_test_exported:arm64+0x100130724)
    #12 Catch::RunContext::runCurrentTest(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&) <null> (lsl_test_exported:arm64+0x10012e160)
    #13 Catch::RunContext::runTest(Catch::TestCaseHandle const&) <null> (lsl_test_exported:arm64+0x10012d9cc)
    #14 Catch::(anonymous namespace)::TestGroup::execute() <null> (lsl_test_exported:arm64+0x1000a283c)
    #15 Catch::Session::runInternal() <null> (lsl_test_exported:arm64+0x1000a1c30)
    #16 Catch::Session::run() <null> (lsl_test_exported:arm64+0x1000a184c)
    #17 main <null> (lsl_test_exported:arm64+0x10004484c)

SUMMARY: ThreadSanitizer: data race (lsl:arm64+0xd7f80) in lsl::factory::new_sample(double, bool)+0xb4

Interpretation and investigation scope

The source suggests the conflicting field is the sample timestamp: new_sample() resets timestamp_ when reusing an object, while pull_sample_typed() reads the timestamp to return it to the caller. The pull path holds a sample_p; recycling should happen only after the final reference is released.

The root cause is not established. This issue records a reproducible sanitizer report, not a proven premature-reuse bug. Investigation should trace sample ownership, reference-count release/fence synchronization, and publication through the factory freelist, and determine whether this is a real synchronization/lifetime defect or a synchronization pattern TSan does not recognize.

No incorrect timestamps or sample corruption were demonstrated by these tests. Do not infer a fix (or add a suppression) solely from the two reported function names.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions