[x64]

is there some generic algorithm using which I can determine relevant registers for the rex byte? I’m specifically interested in the calculation of REX for instructions with the VEX prefix (so that I can determine ~R ~X ~B)?

I’ve tried using the first and last register operand as base and rx, but that only seems to work for stuff like andn, but fails on bextr.

for example –

andn ecx, r15d, edx 
bextr ecx, r15d, edx

respectively encode to:
c4 e2 00 f2 ca
c4 c2 68 f7 cf
I’m confused about the second VEX byte, which is formed like this

~R          [X_______]
~X          [_X______]
~B          [__X_____]
map_select  [___XXXXX]

e2: 1110 0010 (R = 0, X = 0, B = 0)
c2: 1100 0010 (R = 0, X = 0, B = 1)

why does bextr encode with B = 1 while andn doesn’t?