I trained a Bert based model and wanted to load it. Hence I came up with a python script-

import torch
from transformers import RobertaForSequenceClassification, AutoTokenizer, AutoModel
RobertaForSequenceClassification
def read_code_file(file_path):
    with open(file_path, 'r') as file:
        code = file.read()
    return code

code_file_path = '/home/hprakash/test/code.c'

input_text = read_code_file(code_file_path)

tokenizer = AutoTokenizer.from_pretrained("/home/hprakash/tokenizer",  local_files_only=True)

model = RobertaForSequenceClassification.from_pretrained("/home/hprakash/Ghidra-O2-Function", local_files_only=True)

input_ids = tokenizer.encode(input_text, return_tensors='pt')

with torch.no_grad():
    output = model(input_ids)

print(output)

After I tried to run this script I ran into this error –

Some weights of the model checkpoint at /home/hprakash/Ghidra-O2-Function were not used when initializing RobertaForSequenceClassification: ['lm_head.dense.weight', 'lm_head2.decoder.weight', 'lm_head3.decoder.bias', 'lm_head3.dense.weight', 'lm_head2.dense.weight', 'lm_head3.decoder.weight', 'lm_head2.layer_norm.bias', 'lm_head2.bias', 'lm_head2.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.bi                                                                                                                              as', 'lm_head3.layer_norm.weight', 'lm_head3.dense.bias', 'lm_head3.bias', 'lm_head.dense.bias', 'lm_head2.decoder.bias', 'lm_head.layer_norm.weight', 'lm_head3.layer_norm.bias', 'lm_head2.layer_norm.weight']

This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). 

This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

Some weights of RobertaForSequenceClassification were not initialized from the model checkpoint at /home/hprakash/Ghidra-O2-Function and are newly initialized:                                                                                                                               ['classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.dense.weight                                                                                                                              ', 'classifier.out_proj.bias'] 

You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. 

SequenceClassifierOutput(loss=None, logits=tensor([[0.2434, 0.1226]]), hidden_states=None, attentions=None)

I have performed the fine-tuning of the model. Still the error log says I need to fine-tune it further.

I tried to debug my script but it dosen’t seem to be working and I keep on getting the same error.