Repository navigation
Conversation
karlowich
force-pushed
the
ci/build-and-smoke
branch
13 times, most recently
from
September 22, 2026 14:05
cbefb7c to
a86b507
Compare
karlowich
force-pushed
the
ci/build-and-smoke
branch
from
September 23, 2026 13:07
a86b507 to
ba8209e
Compare
fil is built on top of xNVMe and xal, which are built from source, and nothing in the repository lists the system packages this needs. Add a script that installs them, like xNVMe and xal have, so that a new system, such as a CI runner, can be set up easily. CUDA is not installed by the script. It comes from NVIDIA's repository, and the version is up to whoever sets up the system. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Karl Bonde Torp <k.torp@samsung.com>
The module imports NumPy's C API when it is loaded and returns batches as NumPy arrays, but numpy was only listed as a build requirement. Installing the binding into an environment without numpy therefore succeeds, and the import then fails with "No module named 'numpy'". List numpy under the project dependencies so it is installed along with the binding. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Karl Bonde Torp <k.torp@samsung.com>
The workflows that build and test fil on a target are run with cijoe, from cijoe/, where it writes its output. Ignore those output directories, and the Python caches of the cijoe scripts and tests, in git. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Karl Bonde Torp <k.torp@samsung.com>
Building and testing fil is to happen on a target, such as a QEMU guest or a test machine, so the source must get there first. Add the verify workflow. For now it prints system information and copies the fil checkout to the target. The checkout is copied rather than cloned, so the target gets your local changes, not what was last pushed. Every command and its output is in the cijoe report. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Karl Bonde Torp <k.torp@samsung.com>
Run the package script of the distribution given by the config, from the fil checkout on the target. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Karl Bonde Torp <k.torp@samsung.com>
fil and xNVMe compile and link against CUDA and cuFile. The package script leaves CUDA out, as it comes from NVIDIA's repository. Install the parts of the toolkit that are needed, at the version given by the config, in the verify workflow. On a target without an NVIDIA driver there is no libcuda.so.1 to load. The stub of the toolkit is then made loadable under that name, so that everything not calling CUDA still runs. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Karl Bonde Torp <k.torp@samsung.com>
fil needs the upcie-cuda backend of xNVMe, which is not in a release yet, so xNVMe is built from source. Fetch it at the revision given by the config, then build and install it. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Karl Bonde Torp <k.torp@samsung.com>
Fetch xal at the tag given by the config, then build and install it. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Karl Bonde Torp <k.torp@samsung.com>
Build fil from the checkout on the target in three configurations: debug with warnings as errors, with sub-timers, and release. A change that only breaks one of them is then caught. Install the release build, and run 'filperf --help' to check that it loads. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Karl Bonde Torp <k.torp@samsung.com>
fil needs an NVMe device to read from. CI runners and laptops have none to spare, and testing on a shared machine is slow and can break its setup. Add the config of a QEMU guest to use as the target instead, modelled on the one in aisio. It boots an Ubuntu 26.04 nosi image, pinned by digest, and has no GPU and one emulated NVMe device, given as QEMU arguments in the config. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Karl Bonde Torp <k.torp@samsung.com>
Start the guest with the scripts cijoe ships: pull the nosi image, then initialize and start the guest. What cijoe has no script for is done by commands in the workflow: creating the backing image of the NVMe device, waiting for SSH, and unlocking root SSH, which nosi images lock, through their operator account. The workflow needs cijoe 0.9.59 or later, the first that pulls nosi images. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Karl Bonde Torp <k.torp@samsung.com>
karlowich
force-pushed
the
ci/build-and-smoke
branch
from
October 1, 2026 14:05
ba8209e to
6460458
Compare
The only check run on a pull request is the code format, so a change that breaks the build, or only the debug or sub-timer configuration, is only noticed when someone next builds fil. Start the QEMU guest and run the cijoe verify workflow in it for pull requests and pushes to main, in the nosi container xNVMe and aisio run their guests in. The guest has no GPU, so CUDA is there to compile and link only. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Karl Bonde Torp <k.torp@samsung.com>
A failed run can only be looked into through the log of the job, which does not have the output of each command. Upload the complete cijoe output, the report included, whether the run passes or not. When it fails, also print the end of the serial console of the guest. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Karl Bonde Torp <k.torp@samsung.com>
The binding is built separately from the library, against the installed libfil and its public header, so nothing catches it falling out of step with the C API: a renamed field or changed signature only shows up when someone next builds the module. Build it into a virtual environment against the fil just installed, and import it so that unresolved symbols and missing dependencies fail the workflow as well. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Karl Bonde Torp <k.torp@samsung.com>
fil reads files from an XFS file system on an NVMe device. To test that, the device of the target must have one. Format the device given by the config and mount it, in the verify workflow. A device that is mounted anywhere but at the test mountpoint is refused, as it likely holds data. The default device is the emulated NVMe device of the QEMU guest. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Karl Bonde Torp <k.torp@samsung.com>
Add a tool that writes a dataset in the layout fil expects: a directory per class, holding files. The first files of each class have sizes around the block size, the rest have random sizes. The content is random, from a fixed seed, and no two files are the same. Run it in the verify workflow, then mount the file system again read-only, so that all meta-data is on the device when fil parses it. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Karl Bonde Torp <k.torp@samsung.com>
The files of the dataset are written in one go, so each is a single extent, and reading a file of several extents is not exercised. Grow two more files by appending to them in turn with direct I/O, so that their blocks interleave on the device. A buffered tail leaves each with a partial last block. The step fails if they do not end up fragmented, which is checked with filefrag from e2fsprogs. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Karl Bonde Torp <k.torp@samsung.com>
Run pytest tests from the verify workflow, through the cijoe testrunner. The first test reads the dataset with filperf using the posix backend, with and without O_DIRECT. filperf must succeed and report every file of the dataset. Only posix is tested: cufile needs a GPU, and the aisio backends need the device bound to a user-space driver. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Karl Bonde Torp <k.torp@samsung.com>
filperf succeeding does not show that fil returns the right data. Read the dataset through the Python binding, and compare every buffer and label with the files on disk. Every file must be returned. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Karl Bonde Torp <k.torp@samsung.com>
fil_init() refuses options that do not apply to the chosen backend, such as buffered I/O outside posix or an aisio backend without a max file size to size its heap by, and the device setup refuses an unknown backend. Nothing exercises those paths, so a regression there is only noticed when a mistyped benchmark runs something other than what was asked for. Run filperf with each such combination against the test device and require it to exit with an error; exiting successfully or by a signal fails the test. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Karl Bonde Torp <k.torp@samsung.com>
karlowich
force-pushed
the
ci/build-and-smoke
branch
from
October 2, 2026 10:00
6460458 to
b145eb7
Compare
karlowich
marked this pull request as ready for review
October 2, 2026 10:13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds a simple build, install, verify flow which will be run every PR.
Fil has many features depending on a GPU. These won't be exercised, it is only the posix backend for now.