Preface

I have successfully fine-tuned a LLaMA 2 model. And now I want to load it from the hugging face remote model.

Implementation

1. Here is the snippet of my model before training:

   model_id = "meta-llama/Llama-2-13b-hf"
   
   quantization_config = BitsAndBytesConfig(
     load_in_4bit=True,
     bnb_4bit_compute_dtype=torch.float16,
     bnb_4bit_quant_type="nf4"
    )
   
   model = AutoModelForCausalLM.from_pretrained(
     model_id,
     quantization_config=quantization_config,
   )
   
   tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
   tokenizer.add_special_tokens({'pad_token': '<PAD>'})
   
   lora_config = LoraConfig(
     r=8,
      target_modules=["q_proj", "o_proj", "k_proj", "v_proj", "gate_proj", "up_proj", "down_proj"],
      bias="none",
      task_type="CAUSAL_LM",
    )
   
   model.add_adapter(lora_config)
   

Package version: accelerate==0.24.1, peft==0.6.2, transformers==4.35.2, bitsandbytes==0.40.0, datasets==2.17.1, trl==0.7.4

2. After I fine-tuned the model, I uploaded it on Huggingface. Now, I want to load it:

   finetuned_model_id = "ferguso/llama-2-13b-detect"
   quantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16)
   
   finetuned_model = AutoModelForCausalLM.from_pretrained(
     finetuned_model_id,
     quantization_config=quantization_config,
   )
   

Package version: same as I fine-tuned the model, except the bitsandbytes==0.43.1. Because when I use 0.40.0 it shows this similar issue.

Issue and Question

But it shows error RuntimeError: Failed to import transformers.integrations.bitsandbytes because of the following error (look up to see its traceback): cannot import name 'pack_dict_to_tensor' from 'bitsandbytes.utils' (/usr/local/lib/python3.10/dist-packages/bitsandbytes/utils.py)