-
Notifications
You must be signed in to change notification settings - Fork 5.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enhanced GPU discovery and multi-gpu support with concurrency #4517
base: main
Are you sure you want to change the base?
Conversation
05ba1ca
to
91be1fa
Compare
ecde7d9
to
d788717
Compare
f02b076
to
076450a
Compare
12b47a0
to
bfbb50e
Compare
The amdgpu drivers free VRAM reporting omits some other apps, so leverage the upstream DRM driver which keeps better tabs on things
Now that we call the GPU discovery routines many times to update memory, this splits initial discovery from free memory updating.
This worked remotely but wound up trying to spawn multiple servers locally which doesn't work
Still not complete, needs some refinement to our prediction to understand the discrete GPUs available space so we can see how many layers fit in each one since we can't split one layer across multiple GPUs we can't treat free space as one logical block
Our default behavior today is to try to fit into a single GPU if possible. Some users would prefer the old behavior of always spreading across multiple GPUs even if the model can fit into one. This exposes that tunable behavior.
adjust timing on some tests so they don't timeout on small/slow GPUs
|
||
switch runtime.GOOS { | ||
case "windows": | ||
oneapiMgmtName = "ze_intel_gpu64.dll" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This DLL gets installed on Windows with Intel iGPUs as part of the OS base install and doesn't always open reliably – it seems to be causing some crashes on both Win10 and Win11 and so we may want to put this behind a flag until we resolve those issues
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What I'm thinking is I'll add a temporary check to see if we have a oneapi runner available, and if not, disable gpu discovery for the oneapi library that way it can still be built from source and theoretically work but be truly a no-op for the official builds until we can test it more fully.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SG!
@@ -232,6 +228,10 @@ func NewLlamaServer(gpus gpu.GpuInfoList, model string, ggml *GGML, adapters, pr | |||
|
|||
params = append(params, "--parallel", fmt.Sprintf("%d", numParallel)) | |||
|
|||
if estimate.TensorSplit != "" { | |||
params = append(params, "--tensor-split", estimate.TensorSplit) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is super cool! Can't wait to try it more on 2x, 4x and 8x gpu systems
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall looks great! Small comment RE some oneapi dll open panics we are seeing on Windows boxes with iGPUs - we'd want to avoid making that part of the critical path until we resolve this
Carries (and obsoletes if we move this one forward first) #4266 and #4441
This refines our GPU discovery to split it into bootstrapping where we discover information about the GPUs once at startup, and then incrementally refresh just free space information, instead of fully rediscovering the GPUs over and over.
Fixes #3158
Fixes #4198
Fixes #3765