Bug: Vulkan support on Windows does not work consistently

[aittalam]

Contact Details

No response

What happened?

We do have a support for vulkan in, but there are still some configurations where it does not work properly (see e.g. here and here).

Version

0.10.0

What operating system are you seeing the problem on?

Windows

Relevant log output

See here and here.

[aittalam]

#940 addresses part of the issues (on Windows + CUDA13, due to threaded compilation of shaders incompatible with cosmopolitan - fixed by using a single thread).

There are still issues with some Intel integrated CPUs - it'd be great to have some feedback from ppl running Vulkan re: whether they are able or not to run and which is their setup 🙏

[aittalam]

Closing the issue for the time being, if you are having problems with Vulkan on windows please reopen and describe them in a comment below 🙏

[li20034]

@aittalam I cannot re-open this issue (says I don't have permission), but I'm trying to run with Vulkan on Windows 10, using an Intel Iris Xe iGPU on an i7-1180G7. I'm using llamafile 0.10.1 and ggml-vulkan.dll from mozilla.ai's huggingface repo.

On Linux, this setup works perfectly fine (with ggml-vulkan.so from above source), and same official llamafile-0.10.1-thin binary. Also, on both Windows & Linux, llama.cpp built with Vulkan enabled works fine (tested with llama-cli and llama-bench, and monitored GPU usage to confirm its using Vulkan).

Here's the crash that occurs on Windows.

C:\Users\Zonggao\Downloads\llamafile>llamafile-0.10.1-thin.exe -m Qwen3-4B-Q4_K_M.gguf -c 4096 -ngl 99 --flash-attn off --chat --strace
SYS   9960   9960         10'661'005 mmap(0, 10'432, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x6fffffff0000 (27'430'912 bytes total)
SYS   9960    928         19'045'576 mmap(0x6fe000000, 65'536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED, -1, 0) → 0x6fe000000 (27'496'448 bytes total)
SYS   9960    928         19'370'727 getenv("_COSMO_FDS_V2") → NULL
SYS   9960    928         19'519'498 getenv("COSMOPOLITAN_DISABLE_ZIPOS") → NULL
SYS   9960    928         19'669'197 issetugid() → 0
SYS   9960    928         19'819'537 getenv("COSMOPOLITAN_INIT_ZIPOS") → NULL
SYS   9960    928         19'979'731 GetProgramExecutableName() → "/C/Users/Zonggao/Downloads/llamafile/llamafile-0.10.1-thin.exe"
SYS   9960    928         20'313'812 mmap(0, 16, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x6ffffffe0000 (27'500'544 bytes total)
SYS   9960    928         20'516'843 mmap(0, 48, PROT_READ|PROT_WRITE, MAP_ANONYMOUS|MAP_SHARED, -1, 0) → 0x6ffffffd0000 (27'504'640 bytes total)
SYS   9960    928         20'677'345 openat(AT_FDCWD, "/C/Users/Zonggao/Downloads/llamafile/llamafile-0.10.1-thin.exe", O_RDONLY) → 3
SYS   9960    928         21'059'009 fstat(3, [{.st_size=43'265'699, .st_blocks=43'266'048/512, .st_mode=0100775, .st_uid=17094, .st_gid=17094, .st_dev=0x9aaf6974, .st_ino=0x3700000000ce47, .st_flags=20}]) → 0
SYS   9960    928         21'759'165 mmap(0, 43'265'699, PROT_READ, MAP_SHARED, 3, 0) → 0x6ffffd680000 (70'770'688 bytes total)
SYS   9960    928         22'002'401 mmap(0, 1'048, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x6ffffd670000 (70'774'784 bytes total)
SYS   9960    928         22'289'109 munmap(0x6ffffffd0000, 48) → 0 (70'770'688 bytes total)
SYS   9960    928         22'465'550 munmap(0x6ffffffe0000, 16) → 0 (70'766'592 bytes total)
SYS   9960    928         22'642'842 close(3) → 0
SYS   9960    928         22'791'857 __zipos_get("/C/Users/Zonggao/Downloads/llamafile/llamafile-0.10.1-thin.exe") → 0
SYS   9960    928         22'952'745 mmap(0, 3'010'560, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x6ffffd390000 (73'777'152 bytes total)
SYS   9960    928         46'503'465 inflate([u"SYMT☺   ≈f       ≡-      ≡-       @     "...], 3'010'560, u"l▌gTöW╪ε±♂Q░P♠↓ÜH◄◄☺☺◄◄☺)#b7è%÷éè♣c┴^bAú"..., 477'944) → 0
SYS   9960    928         46'644'378 GetSymbolTableFromZip() → 0x6ffffd390000
SYS   9960    928         46'690'923 issetugid() → 0
SYS   9960    928         46'794'951 getenv("COMDBG") → NULL
SYS   9960    928         46'932'885 GetProgramExecutableName() → "/C/Users/Zonggao/Downloads/llamafile/llamafile-0.10.1-thin.exe"
SYS   9960    928         47'225'194 openat(AT_FDCWD, "/C/Users/Zonggao/Downloads/llamafile/llamafile-0.10.1-thin.exe.dbg", O_RDONLY|O_CLOEXEC) → -1 ENOENT
SYS   9960    928         47'502'930 openat(AT_FDCWD, "/C/Users/Zonggao/Downloads/llamafile/llamafile-0.10.1-thin.exe.com.dbg", O_RDONLY|O_CLOEXEC) → -1 ENOENT
SYS   9960    928         47'653'258 getenv("HOME") → "/C/Users/Zonggao"
SYS   9960    928         47'796'197 getenv("TMPDIR") → NULL
SYS   9960    928         47'940'339 getenv("TERM") → "xterm-256color"
SYS   9960    928         48'109'407 getenv("COSMOPOLITAN_HEAP_COUNT") → NULL
SYS   9960    928         48'264'641 mmap(0, 65'536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x6ffffffe0000 (73'842'688 bytes total)
SYS   9960    928         48'426'996 mmap(0, 65'536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x6ffffffd0000 (73'908'224 bytes total)
SYS   9960    928         48'596'376 mmap(0, 65'536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x6ffffd380000 (73'973'760 bytes total)
SYS   9960    928         48'760'461 mmap(0, 65'536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x6ffffd370000 (74'039'296 bytes total)
SYS   9960    928         48'911'648 created 4 dlmalloc arenas for 8 cpus
SYS   9960    928         49'015'678 fstat(0, [{.st_size=0, .st_mode=020666, .st_nlink=0, .st_dev=0x77777777, .st_ino=0x4, .st_blksize=512}]) → 0
SYS   9960    928         49'088'218 getenv("TZ") → NULL
SYS   9960    928         50'211'693 setenv("TZ", "America/Los_Angeles", true) → 0
SYS   9960    928         50'314'936 getenv("_COSMO_PPID") → NULL
SYS   9960    928         50'472'064 getenv("GGML_NO_BACKTRACE") → NULL
SYS   9960    928         50'932'696 mmap(0, 65'536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x6ffffd360000 (74'104'832 bytes total)
SYS   9960    928         51'233'962 mmap(0, 65'536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x6ffffd350000 (74'170'368 bytes total)
SYS   9960    928         51'453'960 getenv("LLAMAFILE_DISABLE_SGEMM") → NULL
SYS   9960    928         51'615'660 openat(AT_FDCWD, "/zip/.args", O_RDONLY) → -1 ENOENT
SYS   9960    928         51'685'515 fstatat(AT_FDCWD, "/zip/ggml-cuda.dll", [n/a], 0) → -1 ENOENT
SYS   9960    928         51'715'498 getenv("HOME") → "/C/Users/Zonggao"
SYS   9960    928         51'833'786 fstatat(AT_FDCWD, "/C/Users/Zonggao/.llamafile/v/0.10.1/ggml-cuda.dll", [n/a], 0) → -1 ENOENT
SYS   9960    928         51'945'808 getenv("HOME") → "/C/Users/Zonggao"
SYS   9960    928         52'165'992 fstatat(AT_FDCWD, "/C/Users/Zonggao/ggml-cuda.dll", [n/a], 0) → -1 ENOENT
SYS   9960    928         52'312'128 fstatat(AT_FDCWD, "/zip/ggml-rocm.dll", [n/a], 0) → -1 ENOENT
SYS   9960    928         52'455'865 getenv("HOME") → "/C/Users/Zonggao"
SYS   9960    928         52'677'745 fstatat(AT_FDCWD, "/C/Users/Zonggao/.llamafile/v/0.10.1/ggml-rocm.dll", [n/a], 0) → -1 ENOENT
SYS   9960    928         52'813'973 getenv("HOME") → "/C/Users/Zonggao"
SYS   9960    928         53'019'619 fstatat(AT_FDCWD, "/C/Users/Zonggao/ggml-rocm.dll", [n/a], 0) → -1 ENOENT
SYS   9960    928         53'164'187 fstatat(AT_FDCWD, "/zip/ggml-vulkan.dll", [n/a], 0) → -1 ENOENT
SYS   9960    928         53'301'657 getenv("HOME") → "/C/Users/Zonggao"
SYS   9960    928         53'579'673 fstatat(AT_FDCWD, "/C/Users/Zonggao/.llamafile/v/0.10.1/ggml-vulkan.dll", [{.st_size=32'253'952, .st_blocks=32'256'000/512, .st_mode=0100775, .st_uid=17094, .st_gid=17094, .st_dev=0x9aaf6974, .st_ino=0x3f000000010ee9, .st_flags=20}], 0) → 0 ENOENT
SYS   9960    928         56'571'410 cosmo_dlopen("/C/Users/Zonggao/.llamafile/v/0.10.1/ggml-vulkan.dll", 1) → 0x7fff83c90000 ENOENT
SYS   9960    928         56'750'429 cosmo_dlsym(0x7fff83c90000, "ggml_backend_vk_init") → 0x7fff83decfb0
SYS   9960    928         56'901'694 cosmo_dlsym(0x7fff83c90000, "ggml_backend_vk_reg") → 0x7fff83ded0a0
SYS   9960    928         57'059'024 cosmo_dlsym(0x7fff83c90000, "ggml_backend_vk_get_device_count") → 0x7fff83dec9a0
SYS   9960    928         57'209'633 cosmo_dlsym(0x7fff83c90000, "ggml_backend_vk_get_device_description") → 0x7fff83dec9c0
SYS   9960    928         57'359'338 cosmo_dlsym(0x7fff83c90000, "ggml_log_set") → 0x7fff83cd67b0
SYS   9960    928         94'162'587 win32 vectored exception 0xE06D7363u raising SIGSEGV cosmoaddr2line /C/Users/Zonggao/Downloads/llamafile/llamafile-0.10.1-thin.exe 7fffed5f5369 0
Terminating on uncaught SIGSEGV. Pass --strace and/or ShowCrashReports() for details.

[aittalam]

Thank you @li20034 , I am reopening the issue so we'll be able to track this 🙏
I am also working on syncing with the latest code from llama.cpp which, as far as I know, introduced quite some vulkan updates... I'll let you know when that part is in so we can (1) verify whether the updates already fixes the issue and (2) fix it ourselves otherwise!

[niutech]

@aittalam See also issue #988, maybe it is the same error.

[li20034]

Vulkan is still not working in 0.10.3 (Windows 10 + Intel iGPU). Am using latest dll from https://huggingface.co/mozilla-ai/llamafile_0.10/blob/main/ggml-vulkan.dll

C:\Users\Zonggao\Downloads\llamafile>llamafile-0.10.3-thin.exe -m Qwen3-1.7B-Q5_K_M-Jinx.gguf --verbose --gpu vulkan
vulkan: INFO: probing library /C/Users/Zonggao/Downloads/llamafile/ggml-vulkan.dll (executable directory)
vulkan: INFO: loaded library /C/Users/Zonggao/Downloads/llamafile/ggml-vulkan.dll from executable directory
vulkan: INFO: Vulkan crashed during device probe; trying next backend
fatal error: support for --gpu vulkan was explicitly requested, but it wasn't available

[aittalam]

Vulkan is still not working in 0.10.3 (Windows 10 + Intel iGPU). Am using latest dll from https://huggingface.co/mozilla-ai/llamafile_0.10/blob/main/ggml-vulkan.dll

Hi @li20034! Can you provide more details regarding your setup (eg which GPU you have installed, which vulkan drivers, etc?). Also, if you call without --gpu does llamafile fallback to CPU?

[li20034]

GPU: Intel Iris Xe iGPU on an i7-1180G7 (it's a laptop with no dGPU)
Driver: Standard Intel driver, version 32.0.101.7026

Fallback doesn't work either, doesn't crash, but doesn't do anything and exits.

C:\Users\Zonggao\Downloads\llamafile>llamafile-0.10.3-thin.exe -m Qwen3-4B-Q4_K_M.gguf --verbose
cuda: INFO: no pre-built GPU library found
cuda: INFO: to enable GPU support, build with:
cuda: INFO:   llamafile/cuda.sh   (for NVIDIA)
cuda: INFO:   llamafile/rocm.sh   (for AMD)
vulkan: INFO: probing library /C/Users/Zonggao/Downloads/llamafile/ggml-vulkan.dll (executable directory)
vulkan: INFO: loaded library /C/Users/Zonggao/Downloads/llamafile/ggml-vulkan.dll from executable directory
vulkan: INFO: Vulkan crashed during device probe; trying next backend
0.00.002.522 I common_params_print_info: build 1780402863 (dbe9c0c8c) with cosmocc for cosmopolitan
0.00.002.528 I log_info: verbosity = 2147483647 (adjust with the `-lv N` CLI arg)
0.00.002.528 I device_info:
0.00.004.238 I   - CPU     : CPU (16087 MiB, 16087 MiB free)
0.00.004.254 I system_info: n_threads = 4 (n_threads_batch = 4) / 8 | CPU : LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.004.254 I srv   server_main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
0.00.004.291 I srv          init: using 8 threads for HTTP server
0.00.004.356 I srv         start: binding port with default address family
0.00.005.987 I srv   server_main: loading model
0.00.005.991 I srv    load_model: loading model 'Qwen3-4B-Q4_K_M.gguf'
0.00.006.028 I common_init_result: fitting params to device memory ...
0.00.006.029 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.00.006.030 I common_params_fit_impl: getting device memory data for initial parameters:
0.00.082.490 I llama_model_loader: loaded meta data with 28 key-value pairs and 398 tensors from Qwen3-4B-Q4_K_M.gguf (version GGUF V3 (latest))
0.00.082.503 I llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
0.00.082.506 I llama_model_loader: - kv   0:                       general.architecture str              = qwen3
0.00.082.506 I llama_model_loader: - kv   1:                               general.type str              = model
0.00.082.507 I llama_model_loader: - kv   2:                               general.name str              = Qwen3 4B Instruct Awq
0.00.082.507 I llama_model_loader: - kv   3:                           general.finetune str              = Instruct-awq
0.00.082.508 I llama_model_loader: - kv   4:                           general.basename str              = Qwen3
0.00.082.508 I llama_model_loader: - kv   5:                         general.size_label str              = 4B
0.00.082.510 I llama_model_loader: - kv   6:                          qwen3.block_count u32              = 36
0.00.082.510 I llama_model_loader: - kv   7:                       qwen3.context_length u32              = 40960
0.00.082.511 I llama_model_loader: - kv   8:                     qwen3.embedding_length u32              = 2560
0.00.082.511 I llama_model_loader: - kv   9:                  qwen3.feed_forward_length u32              = 9728
0.00.082.512 I llama_model_loader: - kv  10:                 qwen3.attention.head_count u32              = 32
0.00.082.512 I llama_model_loader: - kv  11:              qwen3.attention.head_count_kv u32              = 8
0.00.082.515 I llama_model_loader: - kv  12:                       qwen3.rope.freq_base f32              = 1000000.000000
0.00.082.517 I llama_model_loader: - kv  13:     qwen3.attention.layer_norm_rms_epsilon f32              = 0.000001
0.00.082.518 I llama_model_loader: - kv  14:                 qwen3.attention.key_length u32              = 128
0.00.082.518 I llama_model_loader: - kv  15:               qwen3.attention.value_length u32              = 128
0.00.082.519 I llama_model_loader: - kv  16:                       tokenizer.ggml.model str              = gpt2
0.00.082.519 I llama_model_loader: - kv  17:                         tokenizer.ggml.pre str              = qwen2

--gpu disabled does work though. Also, as aforementioned, the same laptop works on Linux with llamafile + Vulkan. llama-cli + Vulkan works too.