New Text

Nemotron 3.5 Lightning

NVIDIA's fast open-weight model for responsive local agents, reasoning, coding, and tool use. It delivers performance comparable to the much larger Nemotron 3 Super.

Command to Run on Jetson Model Details

Parameters 30B total / 3B active

Modalities

Text

Context Length 1M

License OpenMDW 1.1

Precision

NVFP4

Serve the model

Start server

Choose module, then engine and optional parameters on the left, then copy the serve command by clicking the button on the right.

Command

Call the model over Web API

Copy a client command below and paste it into your terminal to make a Web API request to the model you just served.

curl -s http://${JETSON_HOST}:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nemotron35",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

from openai import OpenAI

client = OpenAI(
    base_url="http://${JETSON_HOST}:8000/v1",
    api_key="not-needed",  # vLLM / llama.cpp typically do not enforce a key
)

completion = client.chat.completions.create(
    model="nemotron35",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(completion.choices[0].message.content)

llama.cpp server (OpenAI-compatible API)

After llama-server is running with --network host, call it from another machine on the LAN (set ${JETSON_HOST} or use the field). Default port is often 8080 unless you set --port.

curl -s http://${JETSON_HOST}:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "my_model",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

from openai import OpenAI

client = OpenAI(
    base_url="http://${JETSON_HOST}:8080/v1",
    api_key="not-needed",  # vLLM / llama.cpp typically do not enforce a key
)

completion = client.chat.completions.create(
    model="my_model",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(completion.choices[0].message.content)

Model Details

View on HuggingFace

NVIDIA Nemotron 3.5 Lightning is a fast, open-weight 30B Mixture-of-Experts model that activates only 3B parameters per token. It brings performance close to the much larger Nemotron 3 Super while remaining practical for local coding assistants, research agents, tool-calling workflows, and other always-on applications.

The model supports context lengths of up to one million tokens. Reasoning can be enabled or disabled, and a configurable reasoning budget lets users control how much reasoning the model performs before answering.

The best performance we saw with this model averaged 115 tokens/sec on Jetson AGX Thor and 89 tokens/sec on Jetson AGX Orin across multi-step agentic workloads involving reasoning and tool calls.

Nemotron 3.5 Lightning supports several speculative decoding options, including built-in Multi-Token Prediction, DSpark, and DFlash. We tested the available approaches and found DSpark with vLLM delivered the best performance on both Jetson AGX Thor and Jetson AGX Orin.

Inputs and Outputs

Input: Text

Output: Text

Supported Platforms

Jetson AGX Orin 64GB
Jetson AGX Thor T4000
Jetson AGX Thor T5000 Developer Kit

Why Nemotron 3.5 Lightning on Jetson

Super-class capability: Performance close to Nemotron 3 Super in a model that is much faster and more practical to run locally.
Responsive local agents: Strong performance for multi-step reasoning, tool calls, coding assistants, and research workflows.
Controllable reasoning: Enable or disable reasoning and set a reasoning budget to balance quality, latency, and token usage.
Long-context support: Work with context lengths of up to one million tokens when memory allows.
Fast speculative decoding: Choose from MTP, DSpark, and DFlash, with DSpark providing the best results in our Jetson testing.

Speculative Decoding on Jetson

Nemotron 3.5 Lightning includes Multi-Token Prediction and is released with dedicated DSpark and DFlash checkpoints. The vLLM commands use DSpark with five speculative tokens, which was the fastest configuration in our testing on both supported Jetson platforms. The llama.cpp commands use the DFlash checkpoint.

The vLLM server exposes an OpenAI-compatible API on port 8000 with reasoning parsing, automatic tool selection, and the Qwen3 Coder tool-call parser enabled. The llama.cpp server exposes its API on port 8080.

TensorRT Edge-LLM on Jetson Thor

The TensorRT Edge-LLM command is an experimental, Thor-only NVFP4 path. Its default engine is intentionally conservative: batch size 1, maximum input length 2048 tokens, and KV-cache capacity 2200 tokens. The model’s 1M-token capability is not enabled by this engine configuration.

Standard OpenAI-compatible chat completions have been validated on Jetson AGX Thor. OpenAI tools requests are not supported by this TensorRT Edge-LLM 0.10.0 sample because the shipped tokenizer configuration cannot apply its tool-aware chat template. Use the vLLM command above for validated tool calling, structured reasoning parsing, long-context tuning, or speculative decoding.

The helper serves on port 8000 by default. If that port is already in use, append --port 8001 after --stage serve in the Docker command, then use port 8001 for the OpenAI-compatible API.

Pre-merge staging validation

The public helper URL in the install command is published only after this change is merged and deployed to jetson-ai-lab. It returns 404 before then by design. For staging validation, obtain the helper from the checked-out staging branch instead of using the production URL:

cp public/code-samples/tensorrt_edge_llm/run_nemotron35_lightning.sh "$HOME/run-nemotron35-lightning"
chmod +x "$HOME/run-nemotron35-lightning"

Alternatively, a reviewer with access to the private staging repository can download that same file with gh api and the feat/nemotron-3-5-lightning-edgellm ref. The staging Pages site is access-controlled and is not a reliable unauthenticated curl distribution endpoint.