I manipulated the hidden state values obtained from the llama-2 model after feeding it a certain input, let’s call it Input_1. Now, I want to examine the output (causal output) it produces from this. My hypothesis is that it should correspond to a different input, let’s call it Input_2, which would yield a distinct output from the initial input.

I got last layer’s hidden state values in the following manner :

from transformers import LlamaModel, LlamaTokenizer, LlamaForCausalLM
tokenizer = LlamaTokenizer.from_pretrained(path_to_llama2)
model = LlamaModel.from_pretrained(path_to_llama2)
model_ = LlamaForCausalLM.from_pretrained(path_to_llama2)

tokenizer.pad_token = tokenizer.eos_token
inputs = tokenizer(prompt, return_tensors='pt')    

with torch.no_grad():
  outputs = model(**inputs, output_attentions=True, output_hidden_states=True)
  hidden_states = outputs.hidden_states[-1]  # Last layer hidden states

As shown above, I was trying to change hidden_states values which I got from model but now I want to generate a causal output. How can I do it? Are there any suggestions?