Comments (4)
sgwhat
commented on August 21, 2024
Step 1.
Start the ollama server as command below:
export ONEAPI_DEVICE_SELECTOR=level_zero:1
./ollama serveIf this step could return libmkl.so related error, please turn to step 2.
Step 2.
set the environment config as below before starting ollama server:
export LD_LIBRARY_PATH=/opt/intel/oneapi/mkl/your_oneapi_version/lib:/opt/intel/oneapi/compiler/your_oneapi_version/libfrom bigdl.
js333031
commented on August 21, 2024
This is Windows 11 but in any case, first set of logs below is setup & initialization of the service. 2nd set of logs is output once the webui is used to write a C++ hello world code using starcoder2 model. No more console output in 2nd set of logs while the webui is waiting for result.
(llm-cpp) D:\vrt>set ONEAPI_DEVICE_SELECTOR=level_zero:1
(llm-cpp) D:\vrt\llama-cpp>set OLLAMA_NUM_GPU=999
(llm-cpp) D:\vrt\llama-cpp>set no_proxy=localhost,127.0.0.1
(llm-cpp) D:\vrt\llama-cpp>set ZES_ENABLE_SYSMAN=1
(llm-cpp) D:\vrt\llama-cpp>set SYCL_CACHE_PERSISTENT=1
(llm-cpp) D:\vrt\llama-cpp>
(llm-cpp) D:\vrt\llama-cpp>ollama serve
2024/06/06 06:48:22 routes.go:999: INFO server config env="map[OLLAMA_DEBUG:false OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MAX_VRAM:0 OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:*] OLLAMA_RUNNERS_DIR:D:\\vrt\\llama-cpp\\dist\\windows-amd64\\ollama_runners OLLAMA_TMPDIR:]"
time=2024-06-06T06:48:22.215-04:00 level=INFO source=images.go:697 msg="total blobs: 27"
time=2024-06-06T06:48:22.255-04:00 level=INFO source=images.go:704 msg="total unused blobs removed: 0"
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)
[GIN-debug] POST /api/pull --> github.com/ollama/ollama/server.(*Server).PullModelHandler-fm (5 handlers)
[GIN-debug] POST /api/generate --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (5 handlers)
[GIN-debug] POST /api/chat --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (5 handlers)
[GIN-debug] POST /api/embeddings --> github.com/ollama/ollama/server.(*Server).EmbeddingsHandler-fm (5 handlers)
[GIN-debug] POST /api/create --> github.com/ollama/ollama/server.(*Server).CreateModelHandler-fm (5 handlers)
[GIN-debug] POST /api/push --> github.com/ollama/ollama/server.(*Server).PushModelHandler-fm (5 handlers)
[GIN-debug] POST /api/copy --> github.com/ollama/ollama/server.(*Server).CopyModelHandler-fm (5 handlers)
[GIN-debug] DELETE /api/delete --> github.com/ollama/ollama/server.(*Server).DeleteModelHandler-fm (5 handlers)
[GIN-debug] POST /api/show --> github.com/ollama/ollama/server.(*Server).ShowModelHandler-fm (5 handlers)
[GIN-debug] POST /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).CreateBlobHandler-fm (5 handlers)
[GIN-debug] HEAD /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).HeadBlobHandler-fm (5 handlers)
[GIN-debug] POST /v1/chat/completions --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (6 handlers)
[GIN-debug] GET / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
[GIN-debug] GET /api/tags --> github.com/ollama/ollama/server.(*Server).ListModelsHandler-fm (5 handlers)
[GIN-debug] GET /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
[GIN-debug] HEAD / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
[GIN-debug] HEAD /api/tags --> github.com/ollama/ollama/server.(*Server).ListModelsHandler-fm (5 handlers)
[GIN-debug] HEAD /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
time=2024-06-06T06:48:22.258-04:00 level=INFO source=routes.go:1044 msg="Listening on 127.0.0.1:11434 (version 0.0.0)"
time=2024-06-06T06:48:22.259-04:00 level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu cpu_avx cpu_avx2]"
time=2024-06-06T06:48:22.259-04:00 level=INFO source=gpu.go:122 msg="Detecting GPUs"
time=2024-06-06T06:48:22.276-04:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
[GIN] 2024/06/06 - 06:51:43 | 200 | 0s | 127.0.0.1 | GET "/api/version"
[GIN] 2024/06/06 - 06:51:56 | 200 | 44.4295ms | 127.0.0.1 | GET "/api/tags"
[GIN] 2024/06/06 - 06:51:58 | 200 | 995.4µs | 127.0.0.1 | GET "/api/version"
[GIN] 2024/06/06 - 06:51:58 | 200 | 4.1466ms | 127.0.0.1 | GET "/api/tags"
time=2024-06-06T06:52:18.117-04:00 level=INFO source=gpu.go:122 msg="Detecting GPUs"
time=2024-06-06T06:52:18.135-04:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-06-06T06:52:18.897-04:00 level=INFO source=cpu_common.go:11 msg="CPU has AVX2"
time=2024-06-06T06:52:18.916-04:00 level=INFO source=server.go:327 msg="starting llama server" cmd="D:\\vrt\\llama-cpp\\dist\\windows-amd64\\ollama_runners\\cpu_avx2\\ollama_llama_server.exe --model C:\\Users\\user\\.ollama\\models\\blobs\\sha256-28bfdfaeba9f51611c00ed322ba684ce6db076756dbc46643f98a8a748c5199e --ctx-size 2048 --batch-size 512 --embedding --log-disable --n-gpu-layers 999 --parallel 1 --port 59588"
time=2024-06-06T06:52:18.924-04:00 level=INFO source=sched.go:326 msg="loaded runners" count=1
time=2024-06-06T06:52:18.924-04:00 level=INFO source=server.go:495 msg="waiting for llama runner to start responding"
{"function":"server_params_parse","level":"INFO","line":2604,"msg":"logging to file is disabled.","tid":"11668","timestamp":1717671138}
{"build":1,"commit":"baa5868","function":"wmain","level":"INFO","line":2821,"msg":"build info","tid":"11668","timestamp":1717671138}
{"function":"wmain","level":"INFO","line":2828,"msg":"system info","n_threads":10,"n_threads_batch":-1,"system_info":"AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | ","tid":"11668","timestamp":1717671138,"total_threads":20}
llama_model_loader: loaded meta data with 19 key-value pairs and 483 tensors from C:\Users\user\.ollama\models\blobs\sha256-28bfdfaeba9f51611c00ed322ba684ce6db076756dbc46643f98a8a748c5199e (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = starcoder2
llama_model_loader: - kv 1: general.name str = starcoder2-3b
llama_model_loader: - kv 2: starcoder2.block_count u32 = 30
llama_model_loader: - kv 3: starcoder2.context_length u32 = 16384
llama_model_loader: - kv 4: starcoder2.embedding_length u32 = 3072
llama_model_loader: - kv 5: starcoder2.feed_forward_length u32 = 12288
llama_model_loader: - kv 6: starcoder2.attention.head_count u32 = 24
llama_model_loader: - kv 7: starcoder2.attention.head_count_kv u32 = 2
llama_model_loader: - kv 8: starcoder2.rope.freq_base f32 = 999999.437500
llama_model_loader: - kv 9: starcoder2.attention.layer_norm_epsilon f32 = 0.000010
llama_model_loader: - kv 10: general.file_type u32 = 2
llama_model_loader: - kv 11: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 12: tokenizer.ggml.tokens arr[str,49152] = ["<|endoftext|>", "<fim_prefix>", "<f...
llama_model_loader: - kv 13: tokenizer.ggml.token_type arr[i32,49152] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv 14: tokenizer.ggml.merges arr[str,48872] = ["Ġ Ġ", "ĠĠ ĠĠ", "ĠĠĠĠ ĠĠ...
llama_model_loader: - kv 15: tokenizer.ggml.bos_token_id u32 = 0
llama_model_loader: - kv 16: tokenizer.ggml.eos_token_id u32 = 0
llama_model_loader: - kv 17: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 18: general.quantization_version u32 = 2
llama_model_loader: - type f32: 302 tensors
llama_model_loader: - type q4_0: 181 tensors
llm_load_vocab: missing pre-tokenizer type, using: 'default'
llm_load_vocab:
llm_load_vocab: ************************************
llm_load_vocab: GENERATION QUALITY WILL BE DEGRADED!
llm_load_vocab: CONSIDER REGENERATING THE MODEL
llm_load_vocab: ************************************
llm_load_vocab:
llm_load_vocab: special tokens definition check successful ( 38/49152 ).
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = starcoder2
llm_load_print_meta: vocab type = BPE
llm_load_print_meta: n_vocab = 49152
llm_load_print_meta: n_merges = 48872
llm_load_print_meta: n_ctx_train = 16384
llm_load_print_meta: n_embd = 3072
llm_load_print_meta: n_head = 24
llm_load_print_meta: n_head_kv = 2
llm_load_print_meta: n_layer = 30
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 12
llm_load_print_meta: n_embd_k_gqa = 256
llm_load_print_meta: n_embd_v_gqa = 256
llm_load_print_meta: f_norm_eps = 1.0e-05
llm_load_print_meta: f_norm_rms_eps = 0.0e+00
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 12288
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 2
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 999999.4
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 16384
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 3B
llm_load_print_meta: model ftype = Q4_0
llm_load_print_meta: model params = 3.03 B
llm_load_print_meta: model size = 1.59 GiB (4.51 BPW)
llm_load_print_meta: general.name = starcoder2-3b
llm_load_print_meta: BOS token = 0 '<|endoftext|>'
llm_load_print_meta: EOS token = 0 '<|endoftext|>'
llm_load_print_meta: UNK token = 0 '<|endoftext|>'
llm_load_print_meta: LF token = 164 'Ä'
[SYCL] call ggml_init_sycl
ggml_init_sycl: GGML_SYCL_DEBUG: 0
ggml_init_sycl: GGML_SYCL_F16: no
found 1 SYCL devices:
| | | | |Max | |Max |Global | |
| | | | |compute|Max work|sub |mem | |
|ID| Device Type| Name|Version|units |group |group|size | Driver version|
|--|-------------------|---------------------------------------|-------|-------|--------|-----|-------|---------------------|
| 0| [level_zero:gpu:0]| Intel Arc A770M Graphics| 1.3| 512| 1024| 32| 16704M| 1.3.28902|
ggml_backend_sycl_set_mul_device_mode: true
detect 1 SYCL GPUs: [0] with top Max compute units:512
llm_load_tensors: ggml ctx size = 0.44 MiB
llm_load_tensors: offloading 30 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 31/31 layers to GPU
llm_load_tensors: SYCL0 buffer size = 1629.01 MiB
llm_load_tensors: CPU buffer size = 81.00 MiB
..............................................................................
llama_new_context_with_model: n_ctx = 2048
llama_new_context_with_model: n_batch = 512
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base = 999999.4
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: SYCL0 KV buffer size = 60.00 MiB
llama_new_context_with_model: KV self size = 60.00 MiB, K (f16): 30.00 MiB, V (f16): 30.00 MiB
llama_new_context_with_model: SYCL_Host output buffer size = 0.20 MiB
llama_new_context_with_model: SYCL0 compute buffer size = 124.00 MiB
[1717671143] warming up the model with an empty run
llama_new_context_with_model: SYCL_Host compute buffer size = 10.01 MiB
llama_new_context_with_model: graph nodes = 1177
llama_new_context_with_model: graph splits = 2
from bigdl.
sgwhat
commented on August 21, 2024
It seems there is an error when running Starcoder2. We are working on resolving this issue.
from bigdl.
Related Issues (20)
- Support for max_loaded_maps and num_parallel variables/parameter HOT 4
- vllm-cpu bug - Qwen2Attention' object has no attribute 'kv_scale' HOT 2
- LLaVA import error on CPU HOT 3
- Chatglm3 model convert to sym_int4 failed HOT 2
- Error: Failed to load the llama dynamic library. Segmentation fault HOT 6
- Add support for StableLM-2-12B on GPUs
- [Windows] Qwen1.5-7B 8K支持 HOT 4
- "can NOT allocate memory block with size larger than 4GB" on Arc A770 GPU when inference HOT 1
- [Windows] Qwen1.5-7B 性能优化
- Is there a way to run ollama with IPEX-LLM on CPU HOT 1
- ubuntu 22.04 MTL 165h benchmark Aborted (core dumped) HOT 2
- IPEX-LLM with Langchain-chatchat runs into httpcore.RemoteProtocolError in MTL with iGPU HOT 2
- IPEX-LLM(llama.cpp) met core dump when run Qwen-7B-Q4_K_M.gguf on Intel ARC770 HOT 4
- Quantized model loading method expects the model should be locally available. HOT 2
- Ollama Linux seg fault with GPU on Ubuntu 22.04 HOT 3
- Ollama on Windows not working HOT 4
- GLM-4-9B-Chat missing 'import math' HOT 1
- Error running llama.cpp with IPEX-LLM on MTL iGPU following quickstart guide (Native API returns: -30 (PI_ERROR_INVALID_VALUE)) HOT 16
- [utils] invalidInputError during RuntimeError HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bigdl.