apple_sep: probe() succeeds even when SEPOS never answers the boot handshake

[brentkearney]

apple_sep's probe() returns success as soon as MSG_BOOT_TZ0 is queued, so the driver stays bound with no firmware loaded, no shared memory, and nothing in dmesg when SEPOS never answers. The rest of the boot sequence — MSG_BOOT_TZ0_ACK2 triggering load_fw_and_shmem(), then MSG_BOOT_IMG4 and its ack — runs asynchronously from the mailbox RX callback with no timeout and no completion check.

Impact

• A bound driver with no firmware loaded is indistinguishable from a working one without a kprobe on the mailbox callback, so anyone iterating on this driver through unbind and bind gets silent failure.
• The same path applies on a cold boot. If SEPOS rejects MSG_BOOT_TZ0 for any reason, such as a bad local-policy-manifest or a wrong firmware region, probe() still succeeds and the only symptom is later consumers failing.

Environment

Machine	Apple MacBook Pro (16-inch, 2021), apple,j316s / apple,t6000
Kernel	7.1.6-1-1-ARCH (linux-asahi 7.1.6.asahi1-1), aarch64
Distribution	Arch Linux ARM (Omarchy)
Config	CONFIG_APPLE_SEP=y

t600x does not enable the sep node, so reproducing here requires adding aliases { sep = &sep; } and status = "okay" to t6000-j316s.dtb. The sep node ships enabled on t8103-j293, t8103-j313, t8103-j456, and t8103-j457.

Steps to reproduce

SEPOS boots once per power cycle and ignores a second MSG_BOOT_TZ0, so unbind followed by bind is the easiest way to reach this state.

1. Boot a machine whose sep node is enabled and confirm the driver is bound:

   ls -l /sys/bus/platform/devices/396400000.sep/driver
   

2. Unbind, then bind again:

   echo 396400000.sep | sudo tee /sys/bus/platform/drivers/apple_sep/unbind
   echo 396400000.sep | sudo tee /sys/bus/platform/drivers/apple_sep/bind
   

3. Read dmesg and check the mailbox receive interrupt count:

   grep 396408000.mbox-recv /proc/interrupts
   

Expected behavior

Either the rebind fails with an error, or the driver reports that the SEP did not complete its boot handshake.

Actual behavior

bind succeeds, the driver symlink reappears, and dmesg shows nothing from apple_sep. Evidence that SEPOS never replies:

• With a kprobe on the driver's mailbox RX callback and a control kprobe on apple_mbox_recv_irq, the five seconds after bind recorded 45 mailbox interrupts from other coprocessors and 0 SEP messages.
• The 396408000.mbox-recv interrupt count stayed at 13, the value it reached after the cold bind earlier in the same boot.
• MSG_BOOT_TZ0_ACK2 never arrives, so load_fw_and_shmem() never runs and fw_mapped stays false.

The first unbind also trips the IOMMU WARNs in #591, a separate bug in the unmap path fixed by #592. Unbinding the rebound driver is silent, because remove() skips the unmap when fw_mapped is false.

Possible fixes

• Have process_boot_msg() record handshake progress and warn, or fail a completion that probe() waits on with a timeout, when MSG_BOOT_TZ0_ACK2 or MSG_BOOT_IMG4_ACK does not arrive.
• Alternatively, if unbind and bind are not meant to be supported for a boot-once coprocessor, suppress the bind and unbind attributes so the driver cannot be detached at all.

[brentkearney]

Two more states reach the same observability gap this issue describes: SEPOS stops responding, and nothing in the driver's state says so. Both were found while probing the control endpoints on apple,t6000 / j316s with a research-only debugfs send interface, kernel 7.1.6-1-5-ARCH, sep node enabled.

Two ways SEPOS stops responding

Trigger	State	Sends afterwards
cntl (0x00) type 0x0d, pnon (0x15) type 0x0e	Stops draining the A2I FIFO	Fill the FIFO, 8 deep on this SoC, then fail with -ETIMEDOUT
cntl type 0x18	Drains but never replies	Succeed indefinitely, no reply ever arrives

In both states:

• /sys/bus/platform/devices/396400000.sep/driver is present and the boot-time endpoint list is intact, because probe() succeeded long before.
• Neither state recovers without a power cycle, and a rebind cannot fix it for the reason in this issue: SEPOS ignores a second MSG_BOOT_TZ0.
• The second state is indistinguishable from a healthy idle SEP without sending a message and waiting for a reply.

A completion check on the boot handshake would not catch either state, since both start after a successful handshake. They argue for the same thing from the other direction: the driver needs some notion of whether SEPOS is answering, not only whether probe() finished.

What the probing found

The app set SEPOS starts is fixed before Linux can talk to it. Across 1,025 messages on both advertised control endpoints, no message produced an eighth Got endpoint line:

• cntl answers almost everything within about 1 ms, with type 0x01 and a status byte in param: 0x00 for type 0x00, 0x02 for types 0x02 and 0x03, 0x01 for types 0x01, 0x04–0x0c, 0x0f–0x11, and 0x14–0x17.
• Type 0x0e is a round-trip echo. Param swept 0x00–0xff changes nothing; data swept 0x00–0xff comes back in the reply's data field in 256 of 256 cases.
• Type 0x12 returns a constant type=0x15 param=0x0a data=0x0 for all 512 sends across a param sweep and a data sweep.
• pnon, the boot-policy app, never answers. Types 0x00–0x0d go unanswered while cntl keeps replying to a liveness probe.

Nothing in that surface takes an endpoint number as an argument, so the authentication family absent on this board — SBIO (0x08), SCRD (0x0a), SSE (0x0c), SKS (0x12), skdl (0x17) — cannot be started at runtime from Linux.

Reproducing

Detecting replies from a dmesg line-count delta is not safe here. The kernel ring buffer gets cleared while a run is in flight, so the delta returns empty for replies that did arrive: an early run declared SEPOS dead 0.8 ms after it had answered. Read the journal by cursor instead, with a retry for ingest lag. Write failures are trustworthy either way, because -ETIMEDOUT comes from the write() return rather than from a log line.