drivers/video: add a virtual framebuffer and dirty area reporting

Documentation/applications/system/fbvnc/index.rst

@@ -0,0 +1,134 @@
+==================================
+``fbvnc`` Framebuffer VNC Server
+==================================
+
+Serves a framebuffer the board already has over VNC, with no cooperation from
+the application drawing on it.  The panel keeps working;  a viewer somewhere
+else sees the same pixels, and its mouse and keyboard reach the application as
+a touchscreen and a keyboard.
+
+It is a service rather than part of an application, so it can be started before
+or after whatever draws on the screen, and it survives that application being
+restarted.
+
+This is not the server in ``drivers/video/vnc``.  That one allocates a
+framebuffer of its own and registers a second, virtual display, which is the
+answer for a board with no panel.  ``fbvnc`` serves a framebuffer that already
+exists, the panel's, or a virtual one from ``CONFIG_VIDEO_VFB``, and does
+not care what put the pixels there.
+
+Features
+========
+
+- RFB 3.7, with TRLE, Hextile and Raw encodings.  No zlib, no jpeg.
+- 8, 16 and 32 bit true-colour pixel formats, converted from the
+  framebuffer's own when they differ.
+- Dirty areas from the kernel (``FBIOC_WATCHAREA``, ``FBIOC_GETDIRTY``), so
+  only what changed is sent and nothing is scanned.
+- Remote input delivered through the uinput touch and keyboard devices, so it
+  enters the system the way a finger or a key would.
+- Reading and sending on separate threads:  input does not wait behind a frame
+  that is still going out.
+
+Prepare
+=======
+
+.. code-block:: bash
+
+   CONFIG_NET_TCP=y
+   CONFIG_VIDEO_FB=y
+   CONFIG_NETUTILS_FBVNC=y
+   CONFIG_SYSTEM_FBVNC=y
+
+For remote input, the uinput devices the board registers:
+
+.. code-block:: bash
+
+   CONFIG_INPUT_UINPUT=y
+   CONFIG_UINPUT_TOUCH=y
+   CONFIG_UINPUT_KEYBOARD=y
+
+Usage
+=====
+
+.. code-block:: bash
+
+   fbvnc start [<framebuffer>] [--diff | --full]
+   fbvnc stop
+   fbvnc status
+
+``<framebuffer>`` is ``/dev/fb0`` unless another is named.
+
+.. code-block:: bash
+
+   nsh> fbvnc start /dev/fb0
+   fbvnc: listening on port 5900
+   fbvnc: serving /dev/fb0 (1024x600) on port 5900
+   nsh> lvgldemo &
+   nsh> fbvnc status
+   fbvnc: serving 1024x600 on port 5900, client connected
+
+Then, from the host::
+
+   $ vncviewer <board-ip>:5900
+
+.. figure:: fbvnc_lvgl.png
+   :align: center
+
+   The stock ``lvgldemo`` on a linum-stm32h753bi, served from the panel the
+   board is already driving.  The pointer and keyboard of the viewer reach
+   the demo through the uinput devices.
+
+How what changed is decided
+===========================
+
+By default the kernel is asked.  Every application that draws through the
+framebuffer issues ``FBIO_UPDATE`` with the area it redrew, LVGL's fbdev
+driver among them, and those areas are what gets sent. This costs nothing:
+no pixel is compared.
+
+``--diff`` adds a shadow frame and compares against it.  It serves two
+purposes. For an application that draws without reporting anything, a game
+writing straight into its mapping, it is the only way to know what changed.
+And where the kernel does report, each reported area is narrowed to the rows
+inside it that actually differ:  what an application reports is what it
+*redrew*, which is not the same as what changed, and a toolkit that animates
+one element inside a panel invalidates the panel.  It costs a frame of memory
+and a comparison bounded by the reported area.
+
+``--full`` sends a whole frame per update request, for a framebuffer with no
+dirty reporting at all.
+
+Options
+=======
+
+``CONFIG_NETUTILS_FBVNC_PORT``
+   TCP port to listen on.  5900 by default.
+
+``CONFIG_NETUTILS_FBVNC_MIN_UPDATE_MS``
+   A floor between updates.  A client that asks again the instant it has
+   finished parsing would otherwise keep a full screen permanently in flight.
+
+``CONFIG_NETUTILS_FBVNC_ENCODING_TRLE``, ``..._ENCODING_HEXTILE``
+   Encodings offered.  TRLE is preferred where the client takes it:  a tile of
+   few colours costs bits per pixel there against a sub-rectangle each in
+   Hextile, and a widget toolkit draws flat panels and text.
+
+``CONFIG_NETUTILS_FBVNC_TRACE``
+   Report what every update cost, rectangles, pixels, bytes on the wire, and
+   how long the snapshot and the queueing each took.  It is how one tells a
+   slow link from a slow application.  A line per update, so it is off by
+   default.
+
+Limitations
+===========
+
+- The framebuffer must be 16 bits per pixel.  What the client negotiates is
+  converted;  what the board draws is not.
+- One client at a time.
+- No authentication.  Anyone who can reach the port sees the screen.
+- With LVGL applications, a key of the on-screen keyboard clicked from a viewer
+  is typed twice, and a remote keyboard does not reach LVGL widgets at all,
+  LVGL's NuttX port has a touchscreen input device and no keyboard one.  An
+  application that reads the keyboard itself, as ``lvglterm`` does, is
+  unaffected.

Documentation/platforms/arm/stm32h7/boards/linum-stm32h753bi/index.rst

@@ -1298,3 +1298,44 @@ This example demonstrates how to use the CAN-FD peripherals can0 and can1 with t
     can0  051   [8]  00 11 22 33 44 55 66 77
     can0  051  [16]  00 11 22 33 44 55 66 77 88 99 AA BB CC DD EE FF
 
+
+vnc
+---
+
+Mirrors the board's own display over the network.  The panel keeps working;
+a viewer sees the same pixels, and its mouse and keyboard reach the
+application as a second touchscreen and a keyboard::
+
+  nsh> fbvnc start /dev/fb0
+  nsh> lvgldemo &
+
+.. figure:: fbvnc_lvgl.png
+   :align: center
+
+   ``vnc``:  the panel keeps showing the demo and a viewer on the network
+   sees the same pixels, with its mouse and keyboard reaching the
+   application.
+
+vncfb
+-----
+
+Serves a display the board does not have.  The LTDC is left out of the
+build entirely and a virtual framebuffer takes its place, so the graphics
+stack runs unchanged and the only screen is the one on the network::
+
+  nsh> fbvnc start /dev/fb0
+  nsh> lvglterm /dev/ukeyboard &   # a terminal, driven from the viewer
+
+The Ethernet runs at 100 Mbps here, unlike the ``vnc`` configuration:  the
+link is pinned to 10BASE-T there to work around the panel's interference
+with the magnetics, and there is no panel in this one.
+
+.. note::
+   Known limitation with LVGL applications.  A key of LVGL's on-screen
+   keyboard, clicked once from a viewer, is typed twice, and typing into
+   LVGL widgets from the viewer's keyboard does not work at all, LVGL's
+   NuttX port has a touchscreen driver but no keyboard one, so nothing
+   reads ``/dev/ukeyboard`` into an input device.  Pointer input otherwise
+   works:  tabs, buttons and fields all respond.  An application that reads
+   the keyboard itself, as ``lvglterm`` does, is unaffected.  See the TODO
+   in ``apps/system/fbvnc/fbvnc_main.c``.

Documentation/platforms/xtensa/esp32s3/boards/esp32s3-m5-cardputer/index.rst

@@ -148,3 +148,43 @@ lvglterm
     ``.`` scroll the output::
 
         nsh> lvglterm
+
+lvglvnc
+    The panel, mirrored over Wi-Fi.  The ST7789 keeps showing the on-screen
+    terminal and a VNC client sees the same pixels, with its keyboard
+    arriving as a uinput keyboard, so what is typed on a viewer reaches the
+    shell on the panel::
+
+        nsh> wapi mode wlan0 2
+        nsh> wapi essid wlan0 <SSID> 1
+        nsh> wapi psk wlan0 <password> 3
+        nsh> wapi essid wlan0 <SSID> 1
+        nsh> renew wlan0
+        nsh> fbvnc start /dev/fb0
+        nsh> lvglterm /dev/ukeyboard &
+
+.. figure:: fbvnc_lvglterm.png
+   :align: center
+
+   ``lvglvnc``:  the on-screen terminal of the Cardputer, seen from a VNC
+   client over Wi-Fi.  ``uname -a`` and ``free`` were typed on the viewer's
+   keyboard and ran in the shell on the panel.
+
+vncfb
+    A display the board does not have.  The ST7789 is left out of the build
+    entirely and a virtual framebuffer takes its place, so the graphics
+    stack runs unchanged and the only screen is the one on the network.
+    Same commands as ``lvglvnc``.
+
+    The virtual screen is 160x120 because it is paid for twice:  LVGL
+    allocates an off-screen buffer of its own when the framebuffer driver
+    offers a single buffer, and this module has no PSRAM.  A larger one
+    leaves the 260 KB heap with nothing to spare, at 240x160 the board
+    could no longer start ``wapi`` to bring up its own Wi-Fi, which NuttShell
+    reports as ``wapi: command not found``.
+
+.. note::
+   Bring Wi-Fi up before starting the applications.  Both configurations
+   are close to the memory this module has, and a task that cannot be
+   spawned is reported as a missing command rather than as a memory
+   failure.

boards/arm/stm32h7/linum-stm32h753bi/configs/vnc/defconfig

@@ -0,0 +1,126 @@
+#
+# This file is autogenerated: PLEASE DO NOT EDIT IT.
+#
+# You can use "make menuconfig" to make any modifications to the installed .config file.
+# You can then do "make savedefconfig" to generate a new defconfig file that includes your
+# modifications.
+#
+# CONFIG_STANDARD_SERIAL is not set
+# CONFIG_STM32_FB_CMAP is not set
+# CONFIG_STM32_LTDC_L1_CHROMAKEYEN is not set
+# CONFIG_STM32_LTDC_L2 is not set
+CONFIG_ARCH="arm"
+CONFIG_ARCH_BOARD="linum-stm32h753bi"
+CONFIG_ARCH_BOARD_LINUM_STM32H753BI=y
+CONFIG_ARCH_CHIP="stm32h7"
+CONFIG_ARCH_CHIP_STM32=y
+CONFIG_ARCH_CHIP_STM32H753BI=y
+CONFIG_ARCH_CHIP_STM32H7=y
+CONFIG_ARCH_CHIP_STM32H7_CORTEXM7=y
+CONFIG_ARCH_INTERRUPTSTACK=2048
+CONFIG_ARCH_STACKDUMP=y
+CONFIG_ARMV7M_DCACHE=y
+CONFIG_ARMV7M_DCACHE_WRITETHROUGH=y
+CONFIG_ARMV7M_DTCM=y
+CONFIG_ARMV7M_ICACHE=y
+CONFIG_BOARD_LOOPSPERMSEC=43103
+CONFIG_BUILTIN=y
+CONFIG_DEBUG_FEATURES=y
+CONFIG_DEBUG_SYMBOLS=y
+CONFIG_DRIVERS_VIDEO=y
+CONFIG_ETH0_PHY_KSZ8081=y
+CONFIG_EXAMPLES_ALARM=y
+CONFIG_EXAMPLES_FB=y
+CONFIG_EXAMPLES_LVGLDEMO=y
+CONFIG_EXAMPLES_LVGLDEMO_UTOUCH_DEVPATH="/dev/utouch"
+CONFIG_EXAMPLES_TOUCHSCREEN=y
+CONFIG_FB_OVERLAY=y
+CONFIG_FS_PROCFS=y
+CONFIG_FT5X06_SINGLEPOINT=y
+CONFIG_GRAPHICS_LVGL=y
+CONFIG_IDLETHREAD_STACKSIZE=2048
+CONFIG_INIT_ENTRYPOINT="nsh_main"
+CONFIG_INIT_STACKSIZE=4096
+CONFIG_INPUT=y
+CONFIG_INPUT_FT5X06=y
+CONFIG_INTELHEX_BINARY=y
+CONFIG_IOB_BUFSIZE=768
+CONFIG_IOB_NBUFFERS=96
+CONFIG_LIBM=y
+CONFIG_LINE_MAX=64
+CONFIG_LV_FONT_MONTSERRAT_24=y
+CONFIG_LV_MEM_SIZE_KILOBYTES=256
+CONFIG_LV_USE_DEMO_BENCHMARK=y
+CONFIG_LV_USE_DEMO_WIDGETS=y
+CONFIG_LV_USE_NUTTX=y
+CONFIG_LV_USE_NUTTX_TOUCHSCREEN=y
+CONFIG_MM_REGIONS=5
+CONFIG_NET=y
+CONFIG_NETDB_DNSCLIENT=y
+CONFIG_NETDB_DNSSERVER_IPv4ADDR=0x08080808
+CONFIG_NETDEV_PHY_DEBUG=y
+CONFIG_NETINIT_DHCPC=y
+CONFIG_NETINIT_DRIPADDR=0x08080808
+CONFIG_NETINIT_NOMAC=y
+CONFIG_NETINIT_THREAD=y
+CONFIG_NETUTILS_DHCPC=y
+CONFIG_NETUTILS_DISCOVER=y
+CONFIG_NETUTILS_FBVNC=y
+CONFIG_NETUTILS_TELNETD=y
+CONFIG_NET_ARP_IPIN=y
+CONFIG_NET_BROADCAST=y
+CONFIG_NET_ETH_PKTSIZE=1500
+CONFIG_NET_ICMP_SOCKET=y
+CONFIG_NET_IGMP=y
+CONFIG_NET_ROUTE=y
+CONFIG_NET_STATISTICS=y
+CONFIG_NET_TCP=y
+CONFIG_NET_TCP_NWRBCHAINS=16
+CONFIG_NET_TCP_WRITE_BUFFERS=y
+CONFIG_NET_UDP=y
+CONFIG_NSH_BUILTIN_APPS=y
+CONFIG_NSH_FILEIOSIZE=512
+CONFIG_NSH_READLINE=y
+CONFIG_PREALLOC_TIMERS=4
+CONFIG_RAM_SIZE=245760
+CONFIG_RAM_START=0x20010000
+CONFIG_RAW_BINARY=y
+CONFIG_READLINE_CMD_HISTORY=y
+CONFIG_READLINE_TABCOMPLETION=y
+CONFIG_RR_INTERVAL=200
+CONFIG_RTC_ALARM=y
+CONFIG_RTC_DATETIME=y
+CONFIG_RTC_DRIVER=y
+CONFIG_SCHED_HPWORK=y
+CONFIG_SCHED_WAITPID=y
+CONFIG_START_DAY=6
+CONFIG_START_MONTH=12
+CONFIG_START_YEAR=2011
+CONFIG_STM32_ETHMAC=y
+CONFIG_STM32_ETH_NRXDESC=32
+CONFIG_STM32_ETH_NTXDESC=16
+CONFIG_STM32_FMC=y
+CONFIG_STM32_I2C3=y
+CONFIG_STM32_LTDC=y
+CONFIG_STM32_LTDC_FB_BASE=0xC0600000
+CONFIG_STM32_LTDC_FB_SIZE=2097152
+CONFIG_STM32_PHYSR=30
+CONFIG_STM32_PHYSR_100FD=0x6
+CONFIG_STM32_PHYSR_100HD=0x2
+CONFIG_STM32_PHYSR_10FD=0x5
+CONFIG_STM32_PHYSR_10HD=0x1
+CONFIG_STM32_PHYSR_ALTCONFIG=y
+CONFIG_STM32_PHYSR_ALTMODE=0x7
+CONFIG_STM32_PWR=y
+CONFIG_STM32_RMII_MCO1=y
+CONFIG_STM32_RTC=y
+CONFIG_STM32_USART1=y
+CONFIG_SYSTEM_FBVNC=y
+CONFIG_SYSTEM_NSH=y
+CONFIG_SYSTEM_PING=y
+CONFIG_TASK_NAME_SIZE=24
+CONFIG_TESTING_RAMTEST=y
+CONFIG_UINPUT_TOUCH=y
+CONFIG_USART1_SERIAL_CONSOLE=y
+CONFIG_USEC_PER_TICK=1000
+CONFIG_VIDEO_FB=y