• 时间:
  • 分类:

我转换好的模型: Z841973620/Qwen3.6-35B-A3B-NVFP4-TRT-EDGELLM-ENGINE-Thor

使用 Qwen3.6-35B-A3B

在 x86_64 主机或 Jetson AGX Thor 上量化模型 (NVFP4) 并转为 ONNX 格式:

# docker run -it --rm --runtime=nvidia --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -v /data:/data nvcr.io/nvidia/pytorch:25.11-py3

git clone --recursive --depth=1 -b v0.8.0 https://github.com/NVIDIA/TensorRT-Edge-LLM.git
cd TensorRT-Edge-LLM

sed -i 's/config.intermediate_size/config.moe_intermediate_size/g' tensorrt_edgellm/quantization/models/mtp_draft.py
pip3 install --no-deps .
pip3 install accelerate huggingface_hub==1.5.0 numpy==2.4.6 nvidia-modelopt==0.44.0 onnx==1.19.0 onnxscript==0.7.0 onnx-graphsurgeon==0.6.1 safetensors==0.7.0 transformers==5.9.0

#export HF_ENDPOINT=https://hf-mirror.com
tensorrt-edgellm-quantize llm --model_dir /data/Qwen3.6-35B-A3B --output_dir /data/Qwen3.6-35B-A3B-NVFP4 --quantization nvfp4 --kv_cache_quantization fp8 --lm_head_quantization nvfp4 --visual_quantization fp8
tensorrt-edgellm-export --nvfp4-moe-backend thor --fp8-embedding /data/Qwen3.6-35B-A3B-NVFP4 /data/Qwen3.6-35B-A3B-NVFP4-ONNX

在 Jetson AGX Thor (JetPack 7.1) 上构建 TRT Engine:

git clone --recursive --depth=1 -b v0.8.0 https://github.com/NVIDIA/TensorRT-Edge-LLM.git
mkdir TensorRT-Edge-LLM/build && cd TensorRT-Edge-LLM/build

cmake .. -DCMAKE_BUILD_TYPE=Release -DTRT_PACKAGE_DIR=/usr -DCMAKE_TOOLCHAIN_FILE=cmake/aarch64_linux_toolchain.cmake -DEMBEDDED_TARGET=jetson-thor -DCUDA_CTK_VERSION=13.0 -DENABLE_CUTE_DSL=ALL
make -j$(nproc)

export EDGELLM_PLUGIN_PATH=$(pwd)/libNvInfer_edgellm_plugin.so
./examples/llm/llm_build --onnxDir /data/Qwen3.6-35B-A3B-NVFP4-ONNX/llm --engineDir /data/Qwen3.6-35B-A3B-NVFP4-ENGINE/llm --maxBatchSize 8 --maxInputLen 262144 --maxKVCacheCapacity 262144
./examples/multimodal/visual_build --onnxDir /data/Qwen3.6-35B-A3B-NVFP4-ONNX/visual --engineDir /data/Qwen3.6-35B-A3B-NVFP4-ENGINE

推理:

cat > /tmp/input_llm.json << 'EOF'
{
    "batch_size": 1,
    "temperature": 0.6,
    "top_p": 0.95,
    "top_k": 20,
    "max_generate_length": 8192,
    "requests": [
        {
            "messages": [
                {
                    "role": "user",
                    "content": "若曲线 y=e^x+x 在点 (0,1) 处的切线也是曲线 y=ln(x+1)+a 的切线，求 a 的值"
                }
            ]
        }
    ]
}
EOF
cat > /tmp/input_vlm.json << 'EOF'
{
    "batch_size": 1,
    "temperature": 0.7,
    "top_p": 0.95,
    "top_k": 20,
    "max_generate_length": 1024,
    "requests": [
        {
            "messages": [
                {
                    "role": "user",
                    "content": [
                        {
                            "type": "image",
                            "image": "IMAGE_PATH_PLACEHOLDER"
                        },
                        {
                            "type": "text",
                            "text": "你看到了什么"
                        }
                    ]
                }
            ]
        }
    ]
}
EOF
sed -i "s|IMAGE_PATH_PLACEHOLDER|$(pwd)/../examples/multimodal/pics/red_panda.jpeg|" /tmp/input_vlm.json

export EDGELLM_PLUGIN_PATH=$(pwd)/libNvInfer_edgellm_plugin.so
./examples/llm/llm_stream --engineDir /data/Qwen3.6-35B-A3B-NVFP4-ENGINE/llm --inputFile /tmp/input_llm.json
./examples/llm/llm_stream --engineDir /data/Qwen3.6-35B-A3B-NVFP4-ENGINE/llm --multimodalEngineDir /data/Qwen3.6-35B-A3B-NVFP4-ENGINE --inputFile /tmp/input_vlm.json



参考链接:

添加新评论