I have a question about using positional encoding in transformer models, particularly regarding the suitability of a normalized simple index in scenarios where the input length is consistently constant.

Typically, positional encoding in transformers involves sinusoidal functions, but this approach doesn’t feel intuitive to me. Given that my task always deals with inputs of the same size to the transformer encoder, I’m considering the potential benefits of adopting a normalized simple index instead. This alternative seems simpler and could potentially streamline computations, given the fixed size of the input.

I’m curious about several aspects:

1. Simplicity and Efficiency: Could the straightforward nature of a normalized simple index lead to more efficient computations and easier implementation without sacrificing performance?

2. Consistency in Encoding: With constant input length, each position would consistently correspond to the same normalized value. Might this consistent mapping enhance the model’s ability to learn position-dependent features effectively?

3. Reduced Overfitting: Could the non-cyclical nature of a normalized index help in reducing the model’s tendency to overfit, especially compared to sinusoidal methods?

4. Direct Proportionality: Since the normalized index scales directly with position, could this provide a more linear and proportional representation of positional information which is beneficial for my specific task?

I would appreciate any insights or experiences regarding the effectiveness of using a normalized simple index for positional encoding in such settings. Thank you!

enter image description here