fix(provision): hold the ARK boot chain, and let nvidia-jetpack install (#134)

/boot/Image and /boot/*.dtb* are ordinary package files, not conffiles, and the
image ships NVIDIA's apt source live. NVIDIA's repo has carried a newer stamp
than any image we ship since 36.5.2, so one apt upgrade replaces the ARK
defconfig and DTBs with stock. The DTB half is silent — the board still runs the
ARK tree from the kernel-dtb partition, so only jetson-io notices, dying on the
model mismatch it now sees between /proc/device-tree and /boot/dtb.

The camera pin had a second, unrelated cost: nvidia-jetpack-runtime wants
nvidia-l4t-gstreamer (>> 36.5-0) (<< 36.6-0), so a truthful 36.4.4 stamp made
`apt install nvidia-jetpack` an "impossible situation" no matter what the holds
said. Restamping the repacked debs to 36.5.99 puts them inside that window and
above NVIDIA's newest 36.5.x, so the metapackage resolves and apt still never
wants to upgrade them. Same binaries, same bench-validated stack.

Author: dakejahl

docs/argus_relaunch_regression.md

@@ -45,11 +45,17 @@ Conclusions the data forces:
 
 ## The fix (shipped in this repo)
 
-`--provision` now pins the camera userspace stack to `NV_CAMERA_STACK_VERSION` (versions.env, currently `36.4.4-20250616085344` = JetPack 6.2.1, the newest bench-clean release). Because the 36.4.x debs declare `nvidia-l4t-core (<< 36.5-0)` and exact-stamp deps on cuda/nvsci, provision.sh repacks them (`relax_l4t_deps`): core cap relaxed, out-of-set exact deps unversioned, in-set exact deps retargeted, version suffixed `+ark1` for traceability. They then install as one ordinary `apt-get install --allow-downgrades` transaction — dpkg/apt state stays consistent (`apt-get check` clean) — and are `apt-mark hold` so an on-device upgrade against NVIDIA's repo can't drag them back to the regressed stamp.
+`--provision` now pins the camera userspace stack to `NV_CAMERA_STACK_VERSION` (versions.env, currently `36.4.4-20250616085344` = JetPack 6.2.1, the newest bench-clean release). Because the 36.4.x debs declare `nvidia-l4t-core (<< 36.5-0)` and exact-stamp deps on cuda/nvsci, provision.sh repacks them (`relax_l4t_deps`): core cap relaxed, out-of-set exact deps unversioned, in-set exact deps retargeted, and the version restamped to `NV_CAMERA_PIN_VERSION` (`36.5.99-20250616085344+ark1`). They then install as one ordinary `apt-get install --allow-downgrades` transaction — dpkg/apt state stays consistent (`apt-get check` clean) — and are `apt-mark hold` so an on-device upgrade against NVIDIA's repo can't drag them back to the regressed stamp.
 
 Already-flashed 6.2.2.x devices can be fixed in place with the same four repacked debs: `sudo apt-get install -y --allow-downgrades ./ark1_*.deb && sudo apt-mark hold nvidia-l4t-gstreamer nvidia-l4t-camera nvidia-l4t-multimedia nvidia-l4t-multimedia-utils && sudo systemctl restart nvargus-daemon`.
 
-On each BSP bump, rerun the repro below against the new stock stack; drop the pin (set `NV_CAMERA_STACK_VERSION` back to the BSP stamp) once NVIDIA ships a fixed userspace. Known tradeoff while pinned: the camera stack stops receiving NVIDIA's 36.5.x security/bug updates, and `nvidia-jetpack` metapackage installs that pull exact-version camera components may need the hold lifted.
+On each BSP bump, rerun the repro below against the new stock stack; drop the pin (set `NV_CAMERA_STACK_VERSION` back to the BSP stamp) once NVIDIA ships a fixed userspace. Known tradeoff while pinned: the camera stack stops receiving NVIDIA's 36.5.x security/bug updates.
+
+### Why the version is restamped, not just suffixed
+
+`nvidia-jetpack-runtime` declares `nvidia-l4t-gstreamer (>> 36.5-0), (<< 36.6-0)`, which no truthful 36.4.4 stamp can satisfy — `apt install nvidia-jetpack` failed outright with "you have requested an impossible situation". `36.5.99` sits inside that window and above NVIDIA's newest 36.5.x, so the metapackage resolves and apt still never wants to upgrade the set. Nothing else in that dependency tree is tighter: `nvidia-l4t-jetson-multimedia-api` asks only for `(>> 36.0) (<< 37.0)` on camera/multimedia/multimedia-utils, and the CUDA/TensorRT/VPI side doesn't depend on them at all.
+
+The contents are still `NV_CAMERA_STACK_VERSION`; the `+ark1` suffix and the real `20250616085344` build stamp in the debian revision are what say so. When the pin is dropped, both variables go back to the BSP stamp together.
 
 For Gremsy (#107): their custom IMX586 driver is not the cause — stock sensors reproduce it. They can apply the same pinned userspace on a 36.5.0-based release (best: current kernel + working cameras), or stay on `b2275f3` (R36.4.3) until NVIDIA fixes 36.5.x.
 

provision.sh

@@ -130,9 +130,14 @@ sudo chroot "$ROOTFS_DIR" sh -c 'ls /usr/lib/ark-os/mavsdk/lib/libmavsdk.so.* >/
 # nvidia-l4t-gstreamer, which the BSP set lacks) with the pinned known-good set. The
 # pinned debs' deps assume their own release, so repack each with the out-of-set bounds
 # relaxed (nvidia-l4t-core upper cap, exact-stamp cuda/nvsci) and the in-set exact deps
-# retargeted to the +ark1 version — a clean apt install instead of dpkg --force-depends,
-# so on-device apt stays consistent. Hold the set so an on-device upgrade can't drag it
-# back to the regressed BSP stamp.
+# retargeted to NV_CAMERA_PIN_VERSION — a clean apt install instead of dpkg
+# --force-depends, so on-device apt stays consistent. Hold the set so an on-device
+# upgrade can't drag it back to the regressed BSP stamp.
+#
+# The restamp is also what keeps `apt install nvidia-jetpack` working: the metapackage
+# demands nvidia-l4t-gstreamer (>> 36.5-0) (<< 36.6-0), which a truthful 36.4.4 stamp
+# can never satisfy, so the whole install failed as an "impossible situation". Nothing
+# else in that dependency tree constrains the set any tighter than (>> 36.0, << 37.0).
 relax_l4t_deps() {
     local in="$1" out="$2" work
     work=$(mktemp -d)
@@ -141,13 +146,13 @@ relax_l4t_deps() {
         -e "s/nvidia-l4t-core (<< [0-9.]*-0)/nvidia-l4t-core (<< 37.0-0)/" \
         -e "s/nvidia-l4t-cuda (= [^)]*)/nvidia-l4t-cuda/" \
         -e "s/nvidia-l4t-nvsci (= [^)]*)/nvidia-l4t-nvsci/" \
-        -e "s/(= ${NV_CAMERA_STACK_VERSION})/(= ${NV_CAMERA_STACK_VERSION}+ark1)/g" \
-        -e "s/^Version: .*/&+ark1/" \
+        -e "s/(= ${NV_CAMERA_STACK_VERSION})/(= ${NV_CAMERA_PIN_VERSION})/g" \
+        -e "s/^Version: .*/Version: ${NV_CAMERA_PIN_VERSION}/" \
         "$work/DEBIAN/control"
     dpkg-deb -b --root-owner-group "$work" "$out" >/dev/null
     rm -rf "$work"
 }
-echo "Installing the pinned camera userspace stack (${NV_CAMERA_STACK_VERSION}+ark1)..."
+echo "Installing the pinned camera userspace stack (${NV_CAMERA_STACK_VERSION} as ${NV_CAMERA_PIN_VERSION})..."
 NV_CAMERA_TMP_DEBS=()
 for pkg in "${NV_CAMERA_PKGS[@]}"; do
     deb=$(nv_camera_deb "$pkg")
@@ -163,8 +168,8 @@ sudo chroot "$ROOTFS_DIR" apt-get install -y --allow-downgrades --allow-change-h
 sudo chroot "$ROOTFS_DIR" apt-mark hold "${NV_CAMERA_PKGS[@]}"
 for pkg in "${NV_CAMERA_PKGS[@]}"; do
     v=$(sudo chroot "$ROOTFS_DIR" dpkg-query -W -f='${Version}' "$pkg")
-    [ "$v" = "${NV_CAMERA_STACK_VERSION}+ark1" ] || {
-        echo "ERROR: $pkg is '$v', expected ${NV_CAMERA_STACK_VERSION}+ark1." >&2; exit 1; }
+    [ "$v" = "${NV_CAMERA_PIN_VERSION}" ] || {
+        echo "ERROR: $pkg is '$v', expected ${NV_CAMERA_PIN_VERSION}." >&2; exit 1; }
 done
 # Assert the plugin actually loads and registers nvarguscamerasrc — file existence
 # alone misses unresolvable libraries. Inspect the plugin *file*, not the element:
@@ -177,6 +182,34 @@ sudo chroot "$ROOTFS_DIR" env GST_REGISTRY=/tmp/provision-gst-registry.bin \
 sudo rm -f "$ROOTFS_DIR/tmp/provision-gst-registry.bin"
 for pkg in "${NV_CAMERA_PKGS[@]}"; do sudo rm -f "$ROOTFS_DIR/tmp/ark1_$(nv_camera_deb "$pkg")"; done
 
+### Hold the boot chain we build ourselves
+# /boot/Image and /boot/*.dtb* are ordinary package files owned by nvidia-l4t-kernel
+# and nvidia-l4t-kernel-dtbs, not conffiles, and the image ships NVIDIA's apt source
+# live — so one `apt upgrade` swaps the ARK defconfig and DTBs for stock. The DTB half
+# is the silent one: the board keeps running the ARK tree from the kernel-dtb
+# partition, so only jetson-io notices, dying on the resulting model mismatch.
+# nvidia-l4t-bootloader is in the set because its postinst rewrites the QSPI that
+# carries our MB1 BCT pinmux. The kernel packages hold as one group — vermagic ties
+# the OOT modules to the kernel, so a partial upgrade is worse than either.
+NV_BOOT_CHAIN_PKGS=(
+    nvidia-l4t-kernel
+    nvidia-l4t-kernel-dtbs
+    nvidia-l4t-kernel-headers
+    nvidia-l4t-kernel-oot-headers
+    nvidia-l4t-kernel-oot-modules
+    nvidia-l4t-display-kernel
+    nvidia-l4t-bootloader
+)
+echo "Holding the ARK-built boot chain against NVIDIA's apt repo..."
+sudo chroot "$ROOTFS_DIR" apt-mark hold "${NV_BOOT_CHAIN_PKGS[@]}"
+# apt-mark hold is a no-op on a package that is not installed, so assert the result
+# rather than the command.
+held=$(sudo chroot "$ROOTFS_DIR" apt-mark showhold)
+for pkg in "${NV_BOOT_CHAIN_PKGS[@]}" "${NV_CAMERA_PKGS[@]}"; do
+    printf '%s\n' "$held" | grep -qx "$pkg" || {
+        echo "ERROR: $pkg is not held; an apt upgrade would replace it." >&2; exit 1; }
+done
+
 ### Install pip
 sudo chroot "$ROOTFS_DIR" apt-get install -y python3-pip
 

versions.env

@@ -47,3 +47,10 @@ JETSON_STATS_VERSION="4.3.2"   # jtop daemon + client, installed system-wide via
 # one set: mixed stamps break the NvBufSurface ABI.
 # On each BSP bump: retest with the BSP stamp and drop the pin once NVIDIA fixes it.
 NV_CAMERA_STACK_VERSION="36.4.4-20250616085344"
+
+# Version the repacked debs are restamped to. Must sit inside (36.5-0, 36.6-0) so
+# nvidia-jetpack-runtime's "nvidia-l4t-gstreamer (>> 36.5-0) (<< 36.6-0)" resolves,
+# and above NVIDIA's newest 36.5.x so apt never wants to upgrade the set. Contents
+# are NV_CAMERA_STACK_VERSION; the +ark1 suffix and the real build stamp in the
+# debian revision are what say so.
+NV_CAMERA_PIN_VERSION="36.5.99-20250616085344+ark1"