Bug: Not clear how to use it with Cuda on Windows (10)

[nor-bee]

Contact Details

nor.bee.hu@gmail.com

What happened?

I'm happily using version 0.9.3 on windows with Cuda (Nvidia driver 596.36) but I can not make any 0.10.x work with cuda.

The error is:
cuda: no pre-built GPU library found
cuda: to enable GPU support, build with:
cuda: llamafile/cuda.sh (for NVIDIA)
cuda: llamafile/rocm.sh (for AMD)
fatal error: support for --gpu nvidia was explicitly requested, but it wasn't available

What I can see is that 0.9.3 created "C:\Users\user.llamafile\v\0.9.3" directory with necessary files (e.g. ggml-cuda.dll) but 0.10.x did not create directory "C:\Users\user.llamafile\v\0.10.x" and it is missing files from there.

Even if I succesfully run 0.10.x on cpu only the C:\Users\user.llamafile\v\0.10.x directory had not been created.

I can see from the --strace output that 0.10.x looked for file "ggml-cuda.dll" in C:\Users\user.llamafile\v\0.10.x and also in C:\Users\user\ and when I just copied this file from 0.9.3 directory to C:\Users\user\ and llamafile again it could locate this file but started to miss something else. (please see logs)

I also installed cuda runtime to have better luck but it did not help.

What I really liked in 0.9.3 is that it worked without any external packages with my Nvidia card (A2000/3060) and nvidia driver only.

I hope you can easily fix it.

Version

llamafile 0.10.0/0.10.1

What operating system are you seeing the problem on?

Windows

Relevant log output

When I run it with --verbose --strace options I can see:

"SYS  10200   3432         53'798'334 openat(AT_FDCWD, "/zip/.args", O_RDONLY) → -1 ENOENT
SYS  10200   3432         54'029'837 fstatat(AT_FDCWD, "/zip/ggml-cuda.dll", [n/a], 0) → -1 ENOENT
SYS  10200   3432         54'133'452 getenv("HOME") → "/C/Users/user"
SYS  10200   3432         54'484'945 fstatat(AT_FDCWD, "/C/Users/user/.llamafile/v/0.10.0/ggml-cuda.dll", [n/a], 0) → -1 ENOENT
SYS  10200   3432         54'711'425 getenv("HOME") → "/C/Users/user"
SYS  10200   3432         54'987'673 fstatat(AT_FDCWD, "/C/Users/user/ggml-cuda.dll", [n/a], 0) → -1 ENOENT
cuda: no pre-built GPU library found
SYS  10200   3432         55'545'433 writev(2, {{, 0}, {u"cuda: no pre-built GPU library found◙", 37}}, 2) → 37 ENOENT
cuda: to enable GPU support, build with:
SYS  10200   3432         60'712'565 writev(2, {{, 0}, {u"cuda: to enable GPU support, build with:"..., 41}}, 2) → 41 ENOENT
cuda:   llamafile/cuda.sh   (for NVIDIA)
SYS  10200   3432         61'050'762 writev(2, {{, 0}, {u"cuda:   llamafile/cuda.sh   (for NVIDIA)"..., 41}}, 2) → 41 ENOENT
cuda:   llamafile/rocm.sh   (for AMD)
SYS  10200   3432         61'625'087 writev(2, {{, 0}, {u"cuda:   llamafile/rocm.sh   (for AMD)◙", 38}}, 2) → 38 ENOENT
fatal error: support for --gpu nvidia was explicitly requested, but it wasn't available




When I copied ggml-cuda.dll from the 0.9.3 directory to /C/Users/user it stopped complaining about it but started to miss something else:

SYS   2104   7172        218'607'547 openat(AT_FDCWD, "/zip/.args", O_RDONLY) → -1 ENOENT
SYS   2104   7172        221'734'389 fstatat(AT_FDCWD, "/zip/ggml-cuda.dll", [n/a], 0) → -1 ENOENT
SYS   2104   7172        223'164'918 getenv("HOME") → "/C/Users/user"
SYS   2104   7172        223'615'111 fstatat(AT_FDCWD, "/C/Users/user/.llamafile/v/0.10.0/ggml-cuda.dll", [n/a], 0) → -1 ENOENT
SYS   2104   7172        223'928'191 getenv("HOME") → "/C/Users/user"
SYS   2104   7172        251'469'719 fstatat(AT_FDCWD, "/C/Users/user/ggml-cuda.dll", [{.st_size=35'871'232, .st_blocks=35'872'768/512, .st_mode=0100775, .st_uid=5570, .st_gid=5570, .st_dev=0x5484031e, .st_ino=0xc000000003483f, .st_flags=20}], 0) → 0 ENOENT
SYS   2104   7172        265'691'153 cosmo_dlopen("/C/Users/user/ggml-cuda.dll", 1) → 0x7ff9bce90000 ENOENT
SYS   2104   7172        265'816'826 cosmo_dlsym(0x7ff9bce90000, "ggml_backend_cuda_init") → 0x7ff9bcf473f0
SYS   2104   7172        266'208'411 cosmo_dlsym(0x7ff9bce90000, "ggml_backend_cuda_reg") → 0
SYS   2104   7172        266'553'914 cosmo_dlsym(0x7ff9bce90000, "ggml_backend_cuda_get_device_count") → 0x7ff9bcf47140
SYS   2104   7172        266'917'867 cosmo_dlsym(0x7ff9bce90000, "ggml_backend_cuda_get_device_description") → 0x7ff9bcf47150
SYS   2104   7172        267'010'001 cosmo_dlsym(0x7ff9bce90000, "ggml_log_set") → 0
SYS   2104   7172        267'078'383 cosmo_dlerror() → "symbol not found: ggml_log_set"
cuda: symbol not found: ggml_log_set: not all symbols could be imported
SYS   2104   7172        267'687'182 writev(2, {{, 0}, {u"cuda: symbol not found: ggml_log_set: no"..., 72}}, 2) → 72 ENOENT
SYS   2104   7172        268'689'859 cosmo_dlclose(0x7ff9bce90000) → 0
cuda: no pre-built GPU library found
SYS   2104   7172        268'805'305 writev(2, {{, 0}, {u"cuda: no pre-built GPU library found◙", 37}}, 2) → 37 ENOENT
cuda: to enable GPU support, build with:
SYS   2104   7172        269'155'747 writev(2, {{, 0}, {u"cuda: to enable GPU support, build with:"..., 41}}, 2) → 41 ENOENT
cuda:   llamafile/cuda.sh   (for NVIDIA)
SYS   2104   7172        269'504'756 writev(2, {{, 0}, {u"cuda:   llamafile/cuda.sh   (for NVIDIA)"..., 41}}, 2) → 41 ENOENT
cuda:   llamafile/rocm.sh   (for AMD)
SYS   2104   7172        269'851'923 writev(2, {{, 0}, {u"cuda:   llamafile/rocm.sh   (for AMD)◙", 38}}, 2) → 38 ENOENT
fatal error: support for --gpu nvidia was explicitly requested, but it wasn't available

[aittalam]

Hi @nor-bee ! Sorry about this, we have changed a few things in the 0.10 builds (namely the windows dll is temporarily not bundled in the llamafiles) and we should have reflected this in our documentation. I'll keep this issue open until I fix this and open a linked PR so you can track our work, stay tuned!

In the meantime, I think following the instructions in this issue might help you. The TL/DR is: you can download the ggml-cuda.dll file from here and add it to your home directory (in the next release or if you want to rebuild from current main, the same directory as the llamafile executable also works)

[nor-bee]

Hello @aittalam, Thank you for your response! I just went through the pages you linked and found the necessary files. I'm happy there is a solution for this. May I suggest to include these files in llamafile versions again? If it is not possible it would be also great to give a suggestion to the user instead of this message:
"cuda: no pre-built GPU library found
cuda: to enable GPU support, build with:
cuda: llamafile/cuda.sh (for NVIDIA)
cuda: llamafile/rocm.sh (for AMD)
fatal error: support for --gpu nvidia was explicitly requested, but it wasn't available"

The beauty of this tool was its compactness and single file operation on any platform, I hope you can keep it somehow. :-)

[aittalam]

Thank you @nor-bee for your candid feedback! I agree with you and the only reason why I have not bundled the CUDA libs in the 0.10.* llamafiles is because they are not compact as before 😅 and I did not want to enlarge each and every llamafile by >800MB. That's all on me as I have started working on a solution already but I haven't had the time to complete it yet (because I prioritised bringing more support + features to optimizations).

I want to assure you this is on the roadmap though and as soon as our cuda DLLs are shrinked back to a decent size 🙃 I'll bundle them back in! Also, keep an eye on the discussions page and on our Discord, as this week I'd like to release my llamafile-builder, a tool that allows you to easily bundle custom llamafiles on your own, picking the binaries, libraries and configurations you prefer.

[aittalam]

Hi @nor-bee , minor update: I just merged #963 today, and CUDA libs now take 1/8 of the space 😅: this means that from the next release we'll bundle them by default into our llamafiles!

[nor-bee]

@aittalam Thank you! Amazing work!