I am doing some kind of transfer learning, where I load a dense model and then expand the weight tensor and train only the new values after expanding it and keep the old trained values frozen. in this case I need to set the new weights to requires_grad = True and old weights to requires_grad = False within the same weight tensor. I tried this but it doesnt work:
<code>old_values = weight_mat[0, :, :length[0]]
old_values.requires_grad = False # 1. I tried this and they got optimized
old_values = old_values.unsqueeze(0).detach() # 2. I tried this in addition to 1 and they get optimized
new_values = weight_mat[:, :, length[0]:]
new_values.requires_grad = True
weight_mat = torch.cat((old_values, new_values), dim=-1)
</code>
<code>old_values = weight_mat[0, :, :length[0]]
old_values.requires_grad = False # 1. I tried this and they got optimized
old_values = old_values.unsqueeze(0).detach() # 2. I tried this in addition to 1 and they get optimized
new_values = weight_mat[:, :, length[0]:]
new_values.requires_grad = True
weight_mat = torch.cat((old_values, new_values), dim=-1)
</code>
old_values = weight_mat[0, :, :length[0]]
old_values.requires_grad = False # 1. I tried this and they got optimized
old_values = old_values.unsqueeze(0).detach() # 2. I tried this in addition to 1 and they get optimized
new_values = weight_mat[:, :, length[0]:]
new_values.requires_grad = True
weight_mat = torch.cat((old_values, new_values), dim=-1)
After I print the number of parameters of models that are not trainable I get 0, I also checked the weight tensor values over epochs and found that all values are updated, whereas I am setting old_values to False.