Hot same-JVM reuse (LSP / resident tooling): failure swallowing, hidden thread-locals, JMM races, cross-run state bleed, and unbounded leaks in the runtime's reuse path #543

Summary

Embedding the XVM in a hot, long-lived JVM — an LSP that compiles and runs user code repeatedly, a watch-mode build daemon, a resident test runner — exposes weaknesses in the runtime's Java embedding API and its shared-plane reuse. This issue is deliberately conservative about evidence: it separates what I reproduced as an observable defect on unmodified master (with a runnable harness) from what is an independently-confirmed bug, from unsafe code shapes I could NOT turn into a failing run, from open questions. I would rather under-claim here than list hazards I cannot stand behind.

Runnable demo of the embedding path (unmodified master):
https://gist.github.com/lagergren/85a31b23179e72578993b1661873e4b5

Runnable reproducer harness for the reproduced items (unmodified master):
https://gist.github.com/lagergren/4540fd52d3ab309ab64406cdba3022f7

The demo does two consecutive compiles and two consecutive runs over one InterpreterConnector in one JVM. It passes — the API appears to support hot reuse. The concern is how it does, over a long-lived, multi-threaded host.

1. Reproduced as observable defects on master (harness + output)

1a. A failed run yields no failure object — only a bare int and a shared log line

InterpreterConnector.join() returns getResult() (an int). When a run throws an unhandled exception the caller does get a non-zero int — so failure is detectable — but gets no exception, no cause chain, no object attributable to this request; the failure text goes only to a shared console/log stream. In a host multiplexing requests, "some run failed, there is a trace somewhere in the process log" is not "this request failed, here is why". Severity: moderate (an observability/ergonomics gap, not data loss). And when a run never goes idle, join() returns nothing at all — an unbounded wait with no deadline (see §3d).

Crasher.run about to throw
... Unhandled exception: IllegalState: deliberate failure the host must learn about
REPRO A1: run() threw; join() returned only int=1; no failure object / cause / request attribution

The machinery to fix this already exists in the runtime — the embedding surface just discards it. ServiceContext.callLater(...) (ServiceContext.java ~line 1033) returns a CompletableFuture<ObjectHandle> that completes normally with the result and exceptionally with a WrapperException carrying the XTC ExceptionHandle. MainContainer.invoke0 calls callLater(...) but throws the returned future away (MainContainer.java ~line 236), and join() then polls idleness and returns a bare int. So this is a small API-surface fix, not new capability.
Fix: capture and expose that completion future (or a wrapper) so a request completes with a result or a failure object carrying cause + request identity — event-driven and deadline-bounded. No new completion mechanism is needed.

1b. join() is a ~500 ms busy-wait poll

join() is a Thread.sleep(500) loop over isIdle(); its own source comment reads "extremely naive; replace". A per-run latency floor an editor pays on every compile-run cycle, with no completion signal.

REPRO A4: a trivial run took 508ms wall-clock; join()'s 500ms sleep-poll dominates

Fix: the callLater completion future from 1a is the event-driven completion signal — awaiting it (with a deadline) replaces the poll outright, no new machinery required.

1c. The shared native pool accumulates novel type constants and never evicts

Each run interns derived constants for the core types it uses into the shared native container's ConstantPool, which lives for the whole process. A session exercising many distinct core generic shapes accumulates them without eviction. (The accumulation is bounded by the number of distinct core-type shapes a session touches — large, but not literally infinite; the point is nothing is ever released.)

REPRO B1: shared native ConstantPool size across runs: 83715 -> 89281 -> 89295 -> 89309 -> 89322 -> 89336
          (only ever grows, never shrinks)

Fix: per-run interning should go to per-run storage that dies with the run; only genuinely shared constants belong in the immortal plane.

2. Independently filed / confirmed

2a. Permanent injector deadlock under concurrency — issue 541

A guest ResourceProvider whose supplier transitively depends on the injection chain never resolves; there is no wait-graph detection, so the symptom is a silent hang (and 1a's join() then never returns). This is issue 541, filed and reproduced independently under heavy concurrent load. It is the one item here I am fully confident is a real, serious bug, because someone else demonstrated it.
Fix: fiber wait-graph capture — on idle-with-waiters, fail one victim with a Deadlock naming the cycle.

3. Unsafe code shapes found by source inspection — NOT reproduced as failures

These are real facts about the master source (cited below), and each is a pattern that is unsafe in principle. I could not turn any of them into a failing run, so I am listing them as hazards to audit, explicitly not as demonstrated bugs. Where I tried and failed to reproduce, I say so.

3a. The connector lifecycle is a plain-field state machine

InterpreterConnector.m_containerMain/m_fStarted are plain fields (no volatile, no monitor); loadModule is an unsynchronized check-then-act; join() reads-polls-nulls. Under concurrent driving this is a JMM data race. Caveat: nothing forces concurrent driving — a single-threaded caller is fine, and current callers are single-threaded — so this is a latent hazard for a multi-threaded embedder, not a bug that bites intended use. Not reproduced.

3b. The connector has no shutdown

InterpreterConnector never calls Runtime.shutdownXVM and is not AutoCloseable (zero such references in the class), so an embedder who does not reach past the API leaks worker threads per engine. A real ergonomics/lifecycle gap; minor. Not a failing reproduction.

3c. Concurrent first-touch of shared lazy state

xTerminalConsole.ensureConsole (and sibling Clock/Random/OSStorage suppliers) are plain lazy if (m_hConsole == null) m_hConsole = createServiceHandle(...) on shared templates, and ConstantPool.f_implicits is a plain HashMap written from service threads. Both are data races under concurrent first use. I hammered f_implicits (3000 rounds × 16 threads, repeated) and it did NOT corrupt or throw — the interned values converge, so the race is real in principle but I have no failing reproduction. Reported as an unsafe shape, not a confirmed bug. (That the injector area does break under concurrency is shown by issue 541; whether these specific caches do is unproven.)

3d. The ambient-pool prerequisite is undocumented (ergonomics, not a bug)

findMethods/invoke0 require the caller to wrap the region in ConstantPool.withPool(...); without it they NPE deep in lazy type building. This is awkward and undocumented, but it works once you know it — so it is an ergonomics wart, not a defect. The genuinely serious angle is that the pool it installs is the shared pool (1c): correct concurrent use points every worker thread at one shared, mutably-growing pool — i.e. it leads you into 1c/3c rather than protecting you from them.

4. Red flags — shapes that look wrong but I could NOT prove are bugs (open questions)

Recorded honestly as open questions, not defects. Each has a reason it might be a bug and the reason I could not confirm it.

RF1. JumpVal (and sibling switch ops) cache owner-derived data on the shared decoded op — wrong for CORE-module switches?

org/xvm/asm/op/JumpVal.java caches m_ahCase (ObjectHandle[]), m_mapJump (Map<ObjectHandle,Integer>), m_algorithm as transient fields on the decoded op (~lines 762–772), not keyed per container. App switches re-decode per run (harmless), but core-library modules are decoded once in the shared native plane and their ops are shared across all containers, so the first container to run a core-library switch caches its handles on the shared op.

• Might be fine: the case values are constant-derived, likely immutable currency that compares equal across containers.
• Might be a bug: m_mapJump is keyed by ObjectHandle; if a second container's equivalent handle is a distinct non-equal object, lookups mis-branch, and if any case handle carries container-specific state, a later container dispatches on foreign data.
• Unconfirmed: could not cheaply construct a core-library switch exercised by two sibling containers to observe it.

RF2. ConstHeap.relocateConst moves handles toward ancestor containers without re-composing them

relocateConst relocates constant handles up to a parent/native heap (so a child can be collected), gating on isShared(parent) but not rebinding the composition when the object is unchanged.

• Might be fine: the handles are constant-backed and immutable; a WeakReference test over trivial consecutive runs showed the previous run was collected.
• Might be a bug: the path plainly moves a creator-composed object to a longer-lived owner without re-owning it; under object-sharing shapes my trivial test did not cover, that is a retention/wrong-owner hazard.
• Unconfirmed: trivial modules did not trigger retention; a heavier module (singletons, reflection, cross-container const sharing) with a leak detector is needed to settle it.

What a hot embedding would need

(1) requests terminate with a result or an attributable failure object, event-driven and deadline-bounded (1a, 1b); (2) a shared-pool growth story with eviction or per-run storage (1c); (3) wait-graph deadlock detection (2a / issue 541); and, if a multi-threaded embedder is to be supported, (4) a real or enforced thread contract on the connector and concurrency-safe shared lazy caches (§3). The through-line worth fixing regardless of which items prove exploitable: plain mutable lifecycle fields, failure paths that hand back an int instead of a cause, an undocumented thread-local prerequisite, and unbounded shared caches — a set of shapes that make a resident, multi-threaded host harder to build safely than the passing demo suggests.