Setup Guide

This guide provides the instructions required to set up, build, and run HeteroMosaic. It covers system prerequisites, kernel and GTT memory configuration, external library dependencies, environment setup, project compilation, and model execution.

Reference

For more details about HeteroMosaic, please see heteroMosaic:

Please cite our work if you find our code or paper useful to your work.

@inproceedings{jun2026heteromosaic,
  title={HeteroMosaic: Exposing and Exploiting Heterogeneous Opportunities for Energy-Efficient Edge LLM Inference},
  author={Jun, Gregory Hyegang and Pang, Wesley and Richter, Eddie and Saeedi, Mehdi and Amarnath, Aporva and Ferrao, Pallavi and Chen, Deming},
  booktitle={2026 IEEE/ACM International Symposium on Microarchitecture (MICRO)},
  month={October},
  year={2026}
}

Prerequisites

1. ROCm 7.1.1

Ensure ROCm 7.1.1 is installed at /opt/rocm-7.1.1. Reference: ROCm Quick Start Guide

2. AOCL (AMD Optimizing CPU Libraries)

Instructions and downloads can be found here:

• AOCL Archives
• AOCL Building from Source

3. Kernel and GTT Memory (before library setup)

Install the recommended kernel and configure GRUB for large GTT (Graphics Translation Table) memory. Do this before running the library setup steps below.

Install kernel 6.11.0-26-generic (Ubuntu 24.04.4, amd64):

The required 6.11.0-26.26~24.04.1 packages are no longer available through the current Ubuntu package index. Download the unmodified packages directly from Canonical's Launchpad archive into utils/kernel_6.11.0-26:

cd /path/to/heteroMosaic
kernel_dir="utils/kernel_6.11.0-26"
mkdir -p "$kernel_dir"

wget --continue --https-only --directory-prefix="$kernel_dir" \
  https://launchpadlibrarian.net/789269712/linux-image-6.11.0-26-generic_6.11.0-26.26~24.04.1_amd64.deb \
  https://launchpadlibrarian.net/788568255/linux-headers-6.11.0-26-generic_6.11.0-26.26~24.04.1_amd64.deb \
  https://launchpadlibrarian.net/788568258/linux-hwe-6.11-headers-6.11.0-26_6.11.0-26.26~24.04.1_all.deb \
  https://launchpadlibrarian.net/788568280/linux-modules-6.11.0-26-generic_6.11.0-26.26~24.04.1_amd64.deb \
  https://launchpadlibrarian.net/788568281/linux-modules-extra-6.11.0-26-generic_6.11.0-26.26~24.04.1_amd64.deb

Verify the downloads before installation:

(
  cd "$kernel_dir"
  sha256sum --check <<'EOF'
5cde7652fc104f2a218dd519fbf336e005b0fd37537aff974a07e4e9a7c37b2d  linux-headers-6.11.0-26-generic_6.11.0-26.26~24.04.1_amd64.deb
10b29ad0097d17010c1fe5952f7eb663d1864274b498def4d2c49c323ef51f1c  linux-hwe-6.11-headers-6.11.0-26_6.11.0-26.26~24.04.1_all.deb
9077e58bf542b2af12878ed102de97460aa66eea4130dd323384bfda9e33aecf  linux-image-6.11.0-26-generic_6.11.0-26.26~24.04.1_amd64.deb
8ddc0f247f1f9a5570d09899ded2fab7222b60ebe0c8967bd5a2220e9431ff5a  linux-modules-6.11.0-26-generic_6.11.0-26.26~24.04.1_amd64.deb
54aa1a4ab1af7ab140e41222af79721a48ca87ee03f22a27ad49087835e9e02d  linux-modules-extra-6.11.0-26-generic_6.11.0-26.26~24.04.1_amd64.deb
EOF
)

All five files should report OK. Then install the local packages and their supporting packages:

sudo apt update
sudo apt install -y \
  ./utils/kernel_6.11.0-26/*.deb \
  linux-firmware \
  dkms \
  build-essential

The downloads include the image, modules, extra modules, generic headers, and the common HWE headers dependency. Package licensing and corresponding source links are documented in utils/kernel_6.11.0-26/LICENSES.md.

To boot this kernel by default, set in /etc/default/grub:

GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 6.11.0-26-generic"

Configure large GTT memory by editing /etc/default/grub and setting GRUB_CMDLINE_LINUX_DEFAULT. Use one of the following (replace the existing GRUB_CMDLINE_LINUX_DEFAULT line):

# Option A (FOR 64G Systems)
GRUB_CMDLINE_LINUX_DEFAULT="cma=0 transparent_hugepage=madvise zswap.enabled=0 ttm.pages_limit=15728640 ttm.page_pool_size=15728640 amdttm.pages_limit=15728640 amdttm.page_pool_size=15728640"

# Option B (FOR 96G Systems)
GRUB_CMDLINE_LINUX_DEFAULT="cma=0 transparent_hugepage=madvise zswap.enabled=0 ttm.pages_limit=23068672 ttm.page_pool_size=23068672 amdttm.pages_limit=23068672 amdttm.page_pool_size=23068672"

# Option C (FOR 128G Systems)
GRUB_CMDLINE_LINUX_DEFAULT="cma=0 transparent_hugepage=madvise zswap.enabled=0 ttm.pages_limit=29296875 ttm.page_pool_size=2097152 amdttm.pages_limit=29296875 amdttm.page_pool_size=2097152"

Then update GRUB and reboot:

sudo update-grub
sudo reboot

---

Environment Configuration

Before setting up the dependencies, choose a location where you want to install the libraries. You should export the HOME_LIBS environment variable to point to this directory.

Add the following to your ~/.bashrc or equivalent:

export HOME_LIBS="/path/to/your/libraries"  # Replace with your desired path
mkdir -p "$HOME_LIBS"

Then reload your shell or run:

source ~/.bashrc

---

Setup Steps

1. Setup LibTorch

heteroMosaic first tries to discover PyTorch from the active Python environment. If that is unavailable, it falls back to ${HOME_LIBS}/libtorch_7.1.0.

mkdir -p "$HOME_LIBS"
cd "$HOME_LIBS"
wget https://download.pytorch.org/libtorch/rocm7.1/libtorch-shared-with-deps-2.10.0%2Brocm7.1.zip
unzip libtorch-shared-with-deps-2.10.0+rocm7.1.zip
mv libtorch libtorch_7.1.0
rm libtorch-shared-with-deps-2.10.0+rocm7.1.zip

After extraction, the expected layout is:

$HOME_LIBS/libtorch_7.1.0/
  bin/
  include/
  lib/
  share/

If you install LibTorch somewhere else, update the fallback LibTorch path used by the project accordingly.

2. Setup AOCL

Ensure you have the AOCL tarball (e.g., aocl-linux-gcc-5.1.0.tar.gz) available. Select 1 when Prompted.

cd "$HOME_LIBS"
mkdir aocl
# Assuming the tarball is in your current directory or adjust path
tar -xvf /path/to/aocl-linux-gcc-5.1.0.tar.gz
cd aocl-linux-gcc-5.1.0/
./install.sh -t $HOME_LIBS/aocl
cd ..

3. Setup XDNA Driver

Step A — Install .deb packages

From the project root, install the XDNA/XRT packages in utils/xdna_deb_051925:

cd /path/to/heteroMosaic/utils/xdna_deb_051925/ # project root
sudo apt install ./xrt_202520.2.20.0_24.04-amd64-base.deb
sudo apt install ./xrt_202520.2.20.0_24.04-amd64-base-dev.deb
sudo apt install ./xrt_202520.2.20.0_24.04-amd64-npu.deb
sudo apt install ./xrt_plugin.2.20.0_ubuntu24.04-x86_64-amdxdna.deb

If dpkg -i reports missing dependencies, run sudo apt install -f and then re-run the dpkg -i commands as needed.

Step B Clone Source so that the project can link to it

cd "$HOME_LIBS"
git clone https://github.com/amd/xdna-driver.git xdna-driver-05-19-25
cd xdna-driver-05-19-25
git checkout 0e6d303b2cc2b3fe1cf10aba0acbf57a422588fb
cd ..

---

Project Build Instructions

Follow these steps to build the project. Note that a GPU (ROCm) setup and CMake 3.25 or higher are required.

1. Initial Environment Setup

Navigate to the utils directory and run the initialization script. This will set up the virtual environment, install Python dependencies, and install a modern version of CMake (3.25+) inside the environment.

cd utils
source new_setup.sh

This script:

• creates a fresh utils/rocmPytorch virtual environment
• installs ROCm 7.1 Python packages from requirements_rocm7.1.txt, matching the ROCm 7.1.1 system installation
• installs the remaining model and benchmark dependencies used by the project

Important

This script will fail if ROCm is not correctly set up.

2. Build the Project

After the initial setup is complete, close your current terminal and open a new one.

In the new terminal, navigate to the project root and follow these steps:

1. Source the setup script:

   source utils/setup.sh

   This script activates the virtual environment and exports the main runtime variables used by the build:

   • HETEROMOSAIC_ROOT
   • ROCM_PATH=/opt/rocm-7.1.1
   • HSA_OVERRIDE_GFX_VERSION=11.0.0 on non-AMD RYZEN AI MAX+ 395 w/ Radeon 8060S systems

2. Run CMake and Build:

   mkdir -p build
   cd build
   cmake ..
   make -j$(nproc)

---

Running the Model

1. Authenticate

Log in to your Hugging Face account:

hf auth login

2. Run the Python Script

Navigate to the Python directory and run the model script:

cd py/unified_llm_w4a16
python3 llama3_8b_w4a16_model.py

Common Arguments:

• --text "Your prompt here": The input text to process.
• --model-path "meta-llama/Llama-3.1-8B-Instruct": The Hugging Face repository ID or local path to the model.
• --config-path "configs/configs_strixP_llama3.json5": Path to the NPU configuration file. This file controls heterogeneity settings.
• --device "cuda": The device to run on (cuda or cpu).
• --max-new-tokens 16: The maximum number of new tokens to generate.

Example Command:

python3 llama3_8b_w4a16_model.py \
    --text "What is the meaning of life?" \
    --config-path "configs/configs_strixP_llama3.json5" \
    --max-new-tokens 32

3. Configuration

Runtime behavior is controlled by the JSON5 config file at py/unified_llm_w4a16/configs/configs_strixP_llama3.json5. See this file for available options such as heterogeneity mode, warmup, MoE kernel preloading, debug verbosity, and more.

Note

The HIP kernels in this project are primarily optimized for RDNA (gfx11) architectures. They do not work for CDNA architectures with a wave size of 64.

Licensing and Third-Party Software

heteroMosaic is licensed under the Apache License 2.0. The repository vendors BS::thread_pool and JSON for Modern C++, which remain under their respective upstream licenses. Copyright notices, provenance, local modification information, and complete license locations are documented in THIRD_PARTY_NOTICES.md.

Source and binary distributions should include LICENSE, THIRD_PARTY_NOTICES.md, and the license files stored beside the vendored headers under include/third_party/.