so I deploy a milvus cluster 2.4 to awk eks, and was hoping to try out gpu indices. however when I try to create gpu index on a tiny collection I got something like
[2024/06/04 12:37:05.472 +00:00] [INFO] [indexnode/task.go:516] ["index params are ready"] [buildID=450232097587185780] ["index params"="{"cache_dataset_on_device":"false","dim":"768","index_type":"GPU_IVF_PQ","m":"16","metric_type":"IP","nbits":"8","nlist":"2048"}"]
E20240604 12:37:05.594095 45 index_factory.cc:49] [KNOWHERE][Create][milvus] failed to find index GPU_IVF_PQ_fp32 in factory
[2024/06/04 12:37:05.595 +00:00] [WARN] [indexcgowrapper/helper.go:71] ["failed to create index, C Runtime Exception: index not supportedn"]
milvus version is milvusdb/milvus:v2.4.3
deployed to an awk eks cluster
I don’t see any gpu specific config except for
gpu:
initMemSize: 0
maxMemSize: 0
the gpu nodes are g5.2xlarge, driver version 550.54.15
nvidia-smi
Wed Jun 5 08:11:43 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 |
| 0% 20C P8 12W / 300W | 0MiB / 23028MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
so wondering what am I missing?
New contributor
rachel song is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.