Is there any potential in Nix?

[ilg-ul]

During the XBB - Second Edition discussions, the following question was raised by @shabble:

is there any potential for using/incorporating something like https://en.wikipedia.org/wiki/Nix_(package_manager)? I don't know all that much about it, and doesn't sound like it supports windows well/at-all, but maybe for this backend stuff, if it would make anything easier/better?

According to wiki, it is cross-platform and advertises more reliable, reproducible, and portable packages, so it might provide some inspiration.

[0Grit]

A very long discussion regarding NIX on windows.
https://discourse.nixos.org/t/nix-on-windows/1113/86

[ilg-ul]

This is quite an old discussion; do you know of any recent results?

Given that Nix is making extensive use of symlinks, I would say that the chances of something fully functional using the Windows so-called links are poor; according to npm/xpm, the only functional replacement of links to files are .cmd shims. However they are not 100% free of problems, since starting applications via such shims require the invoker to adjust its behaviour and use a cmd.exe.

And these shims only solve the problem of symlinks to executables, for libraries and other files, npm/xpm has no functional solution.

[0Grit]

No results. Just exploration and proposals I believe.

[ilg-ul]

Nix seems to be several things under the same umbrella, like a language, a local store, a packages store, an operating system, and several associated projects (like flakes, home manager, etc).

Understanding exactly which is which requires some efforts.

Hello World test

In order to get a better understanding on Nix, I did a small Hello World test.

ilg@berry:~/tmp $ cat hello.cpp
#include <iostream>

int
main(int argc, char* argv[])
{
  std::cout << "Hello World!" << std::endl;

  return 0;
}

When compiled with the system GCC, the resulting binary has references to system libraries:

ilg@berry:~ $ gcc --version
gcc (Debian 10.2.1-6) 10.2.1 20210110

ilg@berry:~ $ g++ hello.cpp -o hello
ilg@berry:~ $ ./hello
Hello World!

ilg@berry:~ $ ldd hello
	linux-vdso.so.1 (0x0000007f80666000)
	libstdc++.so.6 => /lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000007f8043a000)
	libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007f802c5000)
	libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000007f8021a000)
	libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000007f801f6000)
	/lib/ld-linux-aarch64.so.1 (0x0000007f80636000)

When compiled with a Nix GCC, things are a bit different:

ilg@berry:~/tmp $ nix-shell -p gcc
these 41 paths will be fetched (45.84 MiB download, 232.75 MiB unpacked):
  /nix/store/1kyy0w7mdrz8wmpcz437jc6pz0njryki-binutils-wrapper-2.38
  ... [removed]

[nix-shell:~/tmp]$ which gcc
/nix/store/jnnmbvm1zajlwp7s5rs6nzpdxp8d3gg5-gcc-wrapper-9.5.0/bin/gcc

[nix-shell:~/tmp]$ echo $PATH
/nix/store/bmwq0pq3xbzvb3l95zpmwcg05mzisjr2-bash-interactive-5.1-p16/bin:/nix/store/dx7xql0421spl59pb4pgmybqdprpgan4-patchelf-0.15.0/bin:/nix/store/jnnmbvm1zajlwp7s5rs6nzpdxp8d3gg5-gcc-wrapper-9.5.0/bin:/nix/store/d7s25q0iz139ykf8hwd31lvqggr3x7j6-gcc-9.5.0/bin:/nix/store/q5pn1cw1dp7a45is550dcvn3ii8axayk-glibc-2.35-163-bin/bin:/nix/store/xff85ys5shalcnma0xavv21mg4a03jq1-coreutils-9.1/bin:/nix/store/1kyy0w7mdrz8wmpcz437jc6pz0njryki-binutils-wrapper-2.38/bin:/nix/store/700fcakwjxba1yfdqqsbf0d074vi081p-binutils-2.38/bin:/nix/store/xff85ys5shalcnma0xavv21mg4a03jq1-coreutils-9.1/bin:/nix/store/srvyv7nas9irb1s15m8zdpms7rbwbrbp-findutils-4.9.0/bin:/nix/store/5j045494rmbhrkj68xzmnri8dpr2m57n-diffutils-3.8/bin:/nix/store/vn216q2w3rfgf04qniavza0szw88q3fb-gnused-4.8/bin:/nix/store/c29djps1h0qhdd6whmpnr2m73lyjbmzm-gnugrep-3.7/bin:/nix/store/nil9d9sba203ka1ppihwv4s7b33ff0pb-gawk-5.1.1/bin:/nix/store/an7dnb7z83ai2yjdlwmgwr9k43dpw2qy-gnutar-1.34/bin:/nix/store/lzakf3i18dbgkxbrcl1sgql606w71ski-gzip-1.12/bin:/nix/store/xinscp51lhzdj7q55l75f1gv5wfsr70l-bzip2-1.0.8-bin/bin:/nix/store/nw7rq3bky1fnizgr217wjg21drsi4phr-gnumake-4.3/bin:/nix/store/z2v0gvw8jsgplyi8mcrfww0iamdz38k3-bash-5.1-p16/bin:/nix/store/xmjzblksz5ybis0lva5z4w9lkyankckl-patch-2.7.6/bin:/nix/store/rdnqqhza4yspnxw06jx6pb5fmk30pxl5-xz-5.2.5-bin/bin:/nix/store/71n9dq8jb7q2qfdpgs2aln7608dhc9r9-file-5.42/bin:/home/ilg/.nix-profile/bin:/home/ilg/.nvm/versions/node/v16.17.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games

[nix-shell:~/tmp]$ gcc --version
/nix/store/z2v0gvw8jsgplyi8mcrfww0iamdz38k3-bash-5.1-p16/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
gcc (GCC) 9.5.0

[nix-shell:~/tmp]$ g++ hello.cpp -o hello-nix

[nix-shell:~/tmp]$ ldd hello-nix
	linux-vdso.so.1 (0x0000007fa6d2d000)
	libstdc++.so.6 => /nix/store/p24d2c3k0c7rsa2hn205jnkw79f0w5qg-gcc-9.5.0-lib/lib/libstdc++.so.6 (0x0000007fa6b10000)
	libm.so.6 => /nix/store/ydqw172ivd19kydr57hq7vz3qb543cl9-glibc-2.35-163/lib/libm.so.6 (0x0000007fa6a70000)
	libgcc_s.so.1 => /nix/store/ydqw172ivd19kydr57hq7vz3qb543cl9-glibc-2.35-163/lib/libgcc_s.so.1 (0x0000007fa6a40000)
	libc.so.6 => /nix/store/ydqw172ivd19kydr57hq7vz3qb543cl9-glibc-2.35-163/lib/libc.so.6 (0x0000007fa6890000)
	/nix/store/ydqw172ivd19kydr57hq7vz3qb543cl9-glibc-2.35-163/lib/ld-linux-aarch64.so.1 (0x0000007fa6cf6000)

[nix-shell:~/tmp]$ readelf -a hello-nix | grep runpath
 0x000000000000001d (RUNPATH)            Library runpath: [/nix/store/y3ba25fyn5kbdmrbqydhx6ifydm79cfh-shell/lib:/nix/store/ydqw172ivd19kydr57hq7vz3qb543cl9-glibc-2.35-163/lib:/nix/store/p24d2c3k0c7rsa2hn205jnkw79f0w5qg-gcc-9.5.0-lib/lib]

Notes

• Nix identifies the packages by the hash+name, and does not use explicit versions
• Nix uses a fixed location store (/nix/store) where all packages are installed
• all installed packages are uniquely identified in the store by the path
• the linker hard-codes an absolute RUNPATH into the binary, and so it refers to shared libraries in the Nix store, not to the system
• at the base, Nix seems to solve the mess with having multiple versions of toolchains, but the resulting binaries run only in the exact the same environment

Comparisons with npm/xpm

Nix seems to address the reproducibility problem by trying to fix the system, by adding a relatively invasive and complex mechanism; xpm is less invasive/complex and aims to run on any system, without administrator rights.

Nix identifies packages by hash+name, xpm identifies packages by @scope/name@semver.

Lessons learned

Directly integrating Nix into an npm/xpm ecosystem does not seem very likely, they address different problems.

However, the idea of instantiating a shell in the specific context of a set of dependencies seems to have some merit.

xpm shell

A similar functionality can be implemented as the xpm shell command; when issues in an xPack project folder it creates a shell with the xpack/.bin added to the PATH, such that the project dependencies are directly available.