Our Urdu assistant accepts a question such as “Lahore kahan hai?” and defaults to Pakistani Urdu رسم الخط. If the reader requests Roman Urdu, it should answer in Roman Urdu. Those are two separate requirements: understand the input, then follow the requested output script while preserving the meaning.
In Labverse’s small Qwen3.5-4B pilot on NVIDIA DGX Spark, a 40-step adapter followed all eight explicit requests for Roman Urdu, up from six with the base model. On a separate twelve-example Roman-to-Urdu test, both models used Urdu script every time and only two answers per model exactly matched the reference. That gap between script compliance and exact wording is where the next round of training data starts.
1. Take a testable idea from each paper
Alibaba’s September 25, 2026 IndustryLLM report describes how its team adapted a Qwen model for industrial procurement. It used a combination of continued pretraining and supervised fine-tuning. The idea we carried into the Urdu pilot was to inspect failures and let those failures guide the data. For example, write two registers from the same evidence, correct only the factual span you can verify, and create new questions in areas where the baseline struggles. Alibaba IndustryLLM technical report
DeepSeek announced V4.1 Flash on September 10, 2026. Its technical report was submitted on September 17. The report describes supervised fine-tuning, reinforcement learning and on-policy distillation, with substantial work on task construction and verification. We use that sequence as a design reference: start with supervised examples and hold later objectives for failures we can measure reliably. Our completed run used supervised LoRA on Qwen3.5-4B. The planned curriculum comparison will test the data strategy at a fixed budget. DeepSeek announcement · V4.1 Flash technical report
2. Turn baseline errors into examples
Run the unchanged base model on a diagnostic set. Tag each error separately: wrong output script, misspelled words, awkward register, wrong dates or numbers, unsupported claims, or incorrect reasoning. Keep a second, unseen test set for final scoring. Repeatedly fixing examples from the test set makes it part of development.
The same fact can teach two output formats. Below are two original example rows in the messages format. They illustrate the data contract, rather than quoting records from the pilot.
{"messages":[{"role":"user","content":"Jawab Urdu rasm-ul-khat mein dein. Di gayi maloomat: Daftar subah 9 baje khulta hai. Daftar kab khulta hai?"},{"role":"assistant","content":"دفتر صبح نو بجے کھلتا ہے۔"}]}
{"messages":[{"role":"user","content":"معلومات: دفتر صبح نو بجے کھلتا ہے۔ Roman Urdu mein batayein, daftar kab khulta hai?"},{"role":"assistant","content":"Daftar subah nau baje khulta hai."}]}
Keep the two rows in the same split because they express the same source fact. In a separate dataset manifest, record the source, source group, input script, target script and review status. Group related examples by source document or episode before splitting, and by speaker or recording session for speech. Preserve intentional English names and product terms. Normalize text consistently, then have a native Pakistani Urdu reviewer check spelling, punctuation and register. When using teacher-generated candidates, check every retained answer against its evidence.
3. Run a small supervised LoRA experiment
LoRA trains small adapter matrices while keeping the base weights fixed. Our Spark run used a verified Qwen3.5-4B checkpoint, BF16, ms-swift 4.0.4 and Transformers 5.2.0. The project recorded the input hashes and dependency versions and reloaded the adapter for inference. Verify the model shards against their published digests and run a real BF16 CUDA forward and backward check before launching. Leave memory headroom and use a fresh output directory per run. Qwen3.5-4B model card · SWIFT Qwen3.5 guide
With the compatible Spark runtime already installed, point this command at your verified local checkpoint and UTF-8 JSONL training and development files. This excerpt preserves the pilot’s core trainer settings. Keep the test set out of the trainer.
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 swift sft \
--model ./models/Qwen3.5-4B --dataset ./data/train.jsonl \
--val_dataset ./data/dev.jsonl --tuner_type lora --torch_dtype bfloat16 \
--freeze_vit true --freeze_aligner true --target_modules all-linear \
--lora_rank 16 --lora_alpha 32 --enable_thinking false --attn_impl sdpa \
--max_length 2048 --per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 --gradient_accumulation_steps 4 \
--gradient_checkpointing true --learning_rate 1e-4 --max_steps 40 \
--save_steps 10 --eval_steps 10 --report_to none --seed 42 \
--output_dir ./runs/urdu-script-pilot
Reload the saved adapter against the same base revision. Use the same prompt template and decoding settings for the base and adapter. Freeze those settings when comparing a broadly sampled curriculum with a failure-targeted curriculum. Match the token budget, adapter settings and run conditions. Compare quality at the same budget and track time to a predeclared development target if both runs reach it. NVIDIA DGX Spark fine-tuning playbook
4. Score the behavior you actually care about
The recorded text pilot used 61 training examples, six development examples and 24 held-out examples. The examples were AI-authored. The base-to-adapter comparison measures the behavior of these two checkpoints on that small sample. A matched comparison of curricula and a larger native-reviewed evaluation remain future experiments.
| Held-out check | Base | 40-step adapter |
|---|---|---|
| Requested Roman output used the requested script | 6/8 | 8/8 |
| Requested Roman output matched the reference exactly | 1/8 | 2/8 |
| Roman input produced Urdu script | 12/12 | 12/12 |
| Roman input matched the Urdu reference exactly | 2/12 | 2/12 |
| Dates written in Urdu words matched exactly | 0/2 | 0/2 |
| Short reasoning answers matched exactly | 0/2 | 0/2 |
Use exact matching for constrained tasks such as a date format or a copied number. For open responses, have native reviewers judge factual correctness, grammar, spelling and naturalness. Roman Urdu can have multiple acceptable spellings. A script check identifies the writing system and a reference match identifies literal agreement. Neither metric scores meaning on its own.
5. Evaluate speech and visual tasks separately
A separate Whisper decoder LoRA pilot trained on 85 UrduSpeech clips. Its transcriptions matched the base model on 18 held-out UrduSpeech clips and 299 FLEURS Urdu test recordings. That pilot showed no transcription improvement on either set. Track word and character error rates separately for ASR, and listen to speech synthesis with native reviewers. UrduSpeech dataset · Google FLEURS dataset
For an image or video extension, use source material with terms that cover your intended use. Keep a timestamp and evidence reference for every visual answer and include questions the clip cannot answer. Hold out entire source groups. In the next visual experiment, compare uniform sampling with a mix of scene boundaries and evenly spaced frames under the same visual token budget. Check both the short actions each policy missed and the cost of adding frames. The 40-step text comparison did not evaluate image or video regressions. Qwen3-VL technical report
The next Urdu batch will target dates in words, native spelling and short grounded answers. We’ll compare it with ordinary sampling under the same training budget and use unseen, native-reviewed examples to decide whether the adapter earned the improvement.
