peperunas/nixos
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

feat(architect/llm.nix): add OLLAMA_KV_CACHE_TYPE environment variable

Browse Source 
- Added `OLLAMA_KV_CACHE_TYPE` with value `"q8_0"` to the environment variables in `llm.nix`
This commit is contained in:
Giulio De Pasquale 2024-12-08 12:01:01 +00:00
parent 109c738594
commit 29c2526e6a
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
hosts/architect/llm.nix
View File

@ -21,6 +21,7 @@ in
environmentVariables = { environmentVariables = {
OLLAMA_FLASH_ATTENTION = "1"; OLLAMA_FLASH_ATTENTION = "1";
OLLAMA_NUM_PARALLEL = "2"; OLLAMA_NUM_PARALLEL = "2";
OLLAMA_KV_CACHE_TYPE = "q8_0";
}; };
}; };