drivers/video: add a virtual framebuffer and dirty area reporting - #19911
drivers/video: add a virtual framebuffer and dirty area reporting#19911JorgeGzm wants to merge 4 commits into
Conversation
github-actions
Bot
added
Area: Documentation
|
🔗 Cross-repo PR dependenciesThe read-only Build run reported the following dependent PR(s) and fetched head SHA(s):
CI run: https://github.com/apache/nuttx/actions/runs/32382137080 |
🔗 Cross-repo PR dependenciesThe read-only Build run reported the following dependent PR(s) and fetched head SHA(s):
CI run: https://github.com/apache/nuttx/actions/runs/32418462550 |
|
Please fix the PR title |
JorgeGzm
changed the title
jerpelea
left a comment
jerpelea
left a comment
There was a problem hiding this comment.
please fix the errors from CI
linguini1
left a comment
linguini1
left a comment
There was a problem hiding this comment.
I wonder if we could incorporate the street dirty region stuff into the base FB driver? So that all drivers can benefit?
The dirty-region support is not part of vfb: it lives in drivers/video/fb.c, touching only fb.c and fb.h. It hooks One behaviour change worth flagging: drivers with no updatearea entry, such as an LTDC, used to get -ENOTTY and now get OK. |
|
@JorgeGzm why not chain Vinc (user space->upperhalf->vnc lowerhalf->stm lowerfhalf? |
🔗 Cross-repo PR dependenciesThe read-only Build run reported the following dependent PR(s) and fetched head SHA(s):
CI run: https://github.com/apache/nuttx/actions/runs/32500764202 |
That is what this does, one level up: fb_updatearea already sees A vnc lower half would put the socket back in a kernel thread: 150 KB/s |
An application that draws tells the framebuffer what it changed, through FBIO_UPDATE. Until now only the display driver heard it. A process that enables FBIOC_WATCHAREA on its open file now receives those areas too: queued in a small ring, signalled with POLLPRI, and popped with FBIOC_GETDIRTY. When the ring fills, further areas widen the newest entry's bounding box, so precision degrades but nothing is lost. This is what lets a mirror, a VNC server, a screen recorder, a remote diagnostic view, learn what changed on a display without the drawing application knowing it exists, and without comparing frames. It costs nothing until someone asks: files that never enable it are skipped. A display whose driver has no updatearea, memory-scanned hardware such as an LTDC, no longer answers ENOTTY to FBIO_UPDATE. The notification has reached its watchers by then, which is a service in itself, and the error told applications their update had failed when it had not. Assisted-by: Claude:opus-5 Signed-off-by: Jorge Guzman <jorge.gzm@gmail.com>
Memory, registered as /dev/fbN like any other framebuffer, with the geometry the configuration asks for. Applications draw into it exactly as they would into a panel; whatever wants the pixels reads the same memory and learns what changed through the framebuffer's dirty-area reporting. That gives a board with no display the same graphics stack as one with a panel, the case the VNC server's virtual mode has served alone until now, and it does so for every consumer rather than one: a VNC server, a screen recorder, a test harness comparing renders. It also lets a board whose panel is the wrong size for what is being developed serve a different one. The board registers it with vfb_register(), the interface a driver uses, rather than the up_fbinitialize() an architecture provides. A board that has a panel of its own builds both and gives them different display numbers: the panel on /dev/fb0, a virtual screen of whatever size suits the work on /dev/fb1. Assisted-by: Claude:opus-5 Signed-off-by: Jorge Guzman <jorge.gzm@gmail.com>
Hextile and ZRLE were declared, the encoding between them was not. TRLE is standard RFB (7.7.5) and the natural one for a server with no zlib: it is ZRLE's tiling, palette and run length without the compression stage, which is where most of ZRLE's gain on flat interfaces comes from anyway. Assisted-by: Claude:opus-5 Signed-off-by: Jorge Guzman <jorge.gzm@gmail.com>
Four arrangements of the same two ideas, on two architectures.
linum:vnc the panel, mirrored: the display keeps working and a
viewer sees the same pixels, with its mouse and
keyboard arriving as a second touchscreen
linum:vncfb a display the board does not have: the LTDC is left
out of the build and a virtual framebuffer takes its
place, so the graphics stack runs unchanged and the
only screen is the one on the network
cardputer:lvglvnc the same mirror over Wi-Fi, on Xtensa
cardputer:vncfb the same virtual display, sized to what a board with
no PSRAM can afford: 160x120, because the screen is
paid for twice, LVGL allocates an off-screen buffer
of its own when the driver offers one buffer, and a
larger one leaves a 260 KB heap with nothing left.
At 240x160 the board could not spawn wapi to bring
its own Wi-Fi up
Two board changes come with them. The framebuffer is registered for
either provider now, the panel's or the virtual one, so the graphics
stack above does not care which it got.
And the linum lets the caches work on its external SDRAM. That memory
sits at 0xc0000000, which the Cortex-M7 default map calls External
Device, uncached, every write strongly ordered, and nothing on this
board said otherwise, so everything held there was read and written at
the speed of the SDRAM bus with no cache in front of it. That is most of
a megabyte of framebuffer when the board serves a virtual display, and
the graphics stack pays it on every pixel: comparing one 800x480 frame
against the last took 421 ms as Device memory and 45 ms as Normal,
cacheable. Nothing reads that memory behind the CPU's back, a virtual
framebuffer is drawn and read by software on the same core, and the
data cache is write-through, so a panel scanning it out would see what
was written.
The Ethernet runs at 100 Mbps in the virtual configurations and 10BASE-T
in the mirrors: the link is restricted where the panel is driving,
because its interference with the magnetics costs far more throughput
than the slower link does, and there is no panel in the other two.
The documentation comes with them: the board pages describe both
configurations, and applications/system/fbvnc is the command's own page.
What it is for, how to configure it, and, the part that decides whether
it is the right thing to reach for, how it differs from the server in
drivers/video/vnc: that one allocates a framebuffer and registers a
second, virtual display; this one serves a framebuffer that already
exists and does not care what put the pixels there.
Its limitations are stated where someone deciding will read them: one
client, no authentication, and the LVGL keyboard input that does not work
because LVGL's NuttX port has no keyboard input device.
Three screenshots of it working: the stock lvgldemo of a
linum-stm32h753bi in a viewer, on the command's own page and on the
board's; and the Cardputer's on-screen terminal over Wi-Fi, with the
commands in it typed from the viewer's keyboard.
The board pages called the command vncserver, which is what it was named
before it was renamed to avoid colliding with the CONFIG_VNCSERVER
driver. They say fbvnc now, including the path of the file that the note
about LVGL input points at.
Assisted-by: Claude:opus-5
Signed-off-by: Jorge Guzman <jorge.gzm@gmail.com>
github-actions
Bot
removed
the
Arch: arm
but why not fix the kernel implementation to improve the performance directly? I don't see any technology reason make the kernel performance lower than userspace implementation. |
That was @acassis initial request too, so we tried it first: mirror mode was implemented inside drivers/video/vnc and worked, but was abandoned for being about 70% slower sending the same rectangles. The cause we hit was in the send path. With the server's socket in a kernel thread, enabling CONFIG_NET_TCP_WRITE_BUFFERS asserted in iob_clone_partial, so we ran it without write buffering and sending became stop-and-wait: 123-187 KB/s measured, against 1017 KB/s from user space on the same board and link. We never root-caused that assert, and I am not claiming it is a design limit. If it is a bug worth fixing, that is a change in net/. Two structural points on top of that:
The driver is used by four configurations, all SAMv7 or sim. We have none of that hardware, so reworking its threading model without being able to validate it on a board would be risky. |
🔗 Cross-repo PR dependenciesThe read-only Build run reported the following dependent PR(s) and fetched head SHA(s):
CI run: https://github.com/apache/nuttx/actions/runs/32550150959 |
5 participants
video.webm
Summary
Serve the display a board already has, over VNC, without giving up the panel.
NuttX has a VNC server in
drivers/video/vnc, and it allocates a framebufferof its own to register a second, virtual display. That is the right answer for
a board with no display; it is not the one for a board whose panel is already
showing something and which also needs to be reached from a desk. These
commits add what that case needs, and the userspace server that uses it is in
the companion nuttx-apps PR.
video/fb: dirty areas reported to userspace. A ring per open file fed bythe same
FBIO_UPDATEfunnel every framebuffer user already goes through,drained with
FBIOC_GETDIRTYand signalled byPOLLPRI. Generic videoinfrastructure -- a screen recorder or a test harness comparing renders wants
the same thing -- and it costs nothing when nobody is watching.
video/vfb: a framebuffer that is memory and nothing else, registered as/dev/fbNwith the geometry the configuration asks for. A board with nodisplay runs the same graphics stack as one with a panel, and the pixels are
available to any consumer rather than to one server.
video/rfb: declareRFB_ENCODING_TRLE. Hextile and ZRLE were declared;the standard encoding between them was not.
stm32h7/ethernet: a reply built in the RX path was transmitted withoutchecking that the head TX descriptor had been returned by the DMA, so a busy
link could overwrite a frame in flight -- corruption, or a panic from the RX
work queue with assertions on. The predicate is the one the driver already
applies before every other transmission; the RX reply path was the one that
skipped it. A second commit adds an opt-in way to restrict autonegotiation
to 10BASE-T full duplex, for boards where another subsystem couples noise
into the PHY.
boards: four configurations, two per board, the board changes they need,and the documentation for all of it -- the board pages, and
applications/system/fbvncfor the command itself, alongside the existingvncviewerpage.The linum also gets an MPU region so its external SDRAM is cacheable:
0xc0000000 is External Device in the Cortex-M7 default map, and nothing said
otherwise, so a framebuffer there was read and written with no cache in
front of it.
Impact
New features, all opt-in.
CONFIG_VIDEO_VFBand theFBIOC_*ioctls are newand default off;
drivers/video/vncis untouched and keeps working as itdoes today.
The Ethernet TX-ring commit changes behaviour for every STM32H7 board using
stm32_ethernet.c: a reply that cannot be sent because the ring is full isnow dropped rather than written over a frame the DMA still owns. The reply to
received data is almost always an acknowledgement and the peer retransmits;
overwriting a frame in flight recovers from nothing.
CONFIG_STM32_AUTONEG_10FD_ONLYdefaults off and changes nothing unless aboard asks for it.
The MPU region and the four configurations are board-local.
Testing
Hardware, not simulation:
vncmirrors thepanel while it keeps working;
vncfbserves 1024x600 with the LTDC left outof the build entirely.
lvglvncmirrorsthe panel,
vncfbserves 160x120 with the panel out of the build.Both boards run the stock
lvgldemoandlvglterm; pointer and keyboard fromthe viewer reach them through the uinput devices. A framebuffer application
that is not LVGL and never reports an update (nxdoom) was also served, which is
what the companion PR's
--diffmode exists for.Measured on the linum, comparing one 800x480 frame against the last: 421 ms
with the SDRAM as Device memory, 45 ms as Normal cacheable.
All four configurations build.
tools/nxstyleis clean on every file touched.Known issue, not from this work:
esp32s3-m5-cardputercannot associateto Wi-Fi on current master. Bisected to
a1b9bedbe4("reconnect Wi-Fi STA onAP-side disconnect"): same binary and same procedure, reverting it associates
and keeping it does not. The Cardputer configurations here were validated with
that revert applied locally; they build either way. Reported separately.
Depends-On: apache/nuttx-apps#3739