Back to the blog

Compiling Custom Kernel Modules on the Jetson Nano

February 3rd, 2021

The Jetson Nano doesn't come with all the standard Linux kernel modules compiled in. You might want one that isn't there by default. For example, TTL mangling doesn't work on the Jetson Nano.

$ sudo iptables -t mangle -I POSTROUTING -j TTL --ttl-set 65
iptables: No chain/target/match by that name.

NVIDIA publishes a guide for compiling the kernel, but it's a bit opaque. Figuring out this process helped me understand a small chunk of the linux kernel module world, so I'd like to share a more didactic guide to compiling the kernel from scratch. If you notice any issues going through this guide, please feel free to email me.

1. Get a VM

First, you'll need to be on an Ubuntu machine. The NVIDIA SDK doesn't allow execution under root, so it won't work with WSL sadly.

2. Install the SDK

Download the SDK Manager and ship that .deb file over to your Ubuntu VM. Install the dependencies, noting that the .deb file specified here needs to match the one you received.

sudo apt install -y libgconf-2-4 libcanberra-gtk-module
sudo dpkg -i sdkmanager_1.4.0-7363_amd64.deb

Run the SDK Manager to query available targets.

sdkmanager --query interactive

Choose the options relevant to you, but skip flashing. My configuration with the Jetson Nano 2GB looks like

$ sdkmanager --query interactive
Authenticating with NVIDIA server...
Login succeeded.
Loading user information...
User information loaded successfully.
Loading server data...
Server data loaded successfully.
- select product: Jetson
- hardware configuration: Host Machine, Target Hardware
- select target: Jetson Nano [2GB developer kit version]
- select target operating system: Linux
- select version: JetPack 4.5
- get detailed options: No
- flash the target board: No

Launch Command:
sdkmanager --cli install  --logintype devzone --product Jetson --host --target P3448-0003 --targetos Linux --version 4.5 --flash skip

Run that launch command that it spits out and go through the interactive process. This will take a while to download/install everything.

3. Install the toolchain

Now, we can follow the NVIDIA guide a little closer. We'll install the compilation toolchain, which includes gcc and all those goodies.

wget http://releases.linaro.org/components/toolchain/binaries/7.3-2018.05/aarch64-linux-gnu/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xz
tar xf gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xz
export CROSS_COMPILE=$(pwd)/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-
export LOCALVERSION=-tegra

4. Grab the kernel source code

You can find the latest version of the source code on the NVIDIA downloads page under "L4T Sources".

wget https://developer.nvidia.com/embedded/L4T/r32_Release_v4.4/r32_Release_v4.4-GMC3/Sources/T210/public_sources.tbz2
tar -xjf public_sources.tbz2
cd Linux_for_Tegra/source/public
tar -xjf kernel_src.tbz2
cd kernel/kernel-4.9

5. Configure and build the kernel

Make sure the proper dependencies are installed.

sudo apt install build-essential bc libncurses5 libncurses5-dev

and specify where the kernel output should go (you can choose a different location, it doesn't matter)

TEGRA_KERNEL_OUT=$HOME/t4l-kernel
mkdir -p $TEGRA_KERNEL_OUT

Initialize the config

make ARCH=arm64 O=$TEGRA_KERNEL_OUT tegra_defconfig

Then configure the build. Choose the options you want. Alternatively, you can specify the specific module by name by editing $TEGRA_KERNEL_OUT/.config.

make menuconfig

If you are specifically looking to enable TTL mangling, enable CONFIG_NETFILTER_XT_TARGET_HL in $TEGRA_KERNEL_OUT/.config.

After configuring, kick off the build, replacing 4 with the number of cores your system has.

make ARCH=arm64 O=$TEGRA_KERNEL_OUT -j4

6. Packaging the kernel

The NVIDIA SDK manager should have created a directory at ~/nvidia/nvidia_sdk/JetPack_4.5_Linux_JETSON_NANO_2GB_DEVKIT/Linux_for_Tegra (or similar, depending on your board choice in the second step). Copy over the Image and dtb files.

cp $TEGRA_KERNEL_OUT/arch/arm64/boot/Image ~/nvidia/nvidia_sdk/JetPack_4.5_Linux_JETSON_NANO_2GB_DEVKIT/Linux_for_Tegra/kernel/Image
cp -r $TEGRA_KERNEL_OUT/arch/arm64/boot/dts/* ~/nvidia/nvidia_sdk/JetPack_4.5_Linux_JETSON_NANO_2GB_DEVKIT/Linux_for_Tegra/kernel/dtb

Then install the modules

sudo make ARCH=arm64 O=$TEGRA_KERNEL_OUT modules_install INSTALL_MOD_PATH=~/nvidia/nvidia_sdk/JetPack_4.5_Linux_JETSON_NANO_2GB_DEVKIT/Linux_for_Tegra/rootfs/

Lastly, change over to this directory and build the sdcard image. Note that the target for jetson-disk-image-creator.sh will change depending on your board. Run ./tools/jetson-disk-image-creator.sh to get the options available.

cd ~/nvidia/nvidia_sdk/JetPack_4.5_Linux_JETSON_NANO_2GB_DEVKIT/Linux_for_Tegra/
sudo ./apply_binaries.sh -r rootfs
sudo ./tools/jetson-disk-image-creator.sh -o sdcard.img -b jetson-nano-2gb-devkit

7. Flash the SD card and we're done!

Take that sdcard.img and flash it with your favorite sd card imager and we're off to the races. For me, that means

sudo iptables -t mangle -I POSTROUTING -j TTL --ttl-set 65

works!


Comment on this post on Twitter