I’m currently trying to break the XOR encryption of a known custom xor-encryption algorithm.

The Encryption uses 4-Byte XOR keys. The full algorithm has a table with 32768 Keys. The keys are all derived from 2 different seeds. The first half of the key table is created with the first seed (16384 elements) and the remaining half is created with the 2nd seed.

The creation of the key table can be found at github with the “Random” function found here.

func (k *KeyRand) Rand() int64 {
    // some hardcoded function from ep3 sources...
    // probably should be changed with other(better) algorithm
    k.holdRand = k.holdRand*49723125 + 21403831
    return (((((int64)(k.holdRand)>>16)*41894339 + 11741117) >> 16) & 0xFFFF)
}

The Encryption algorithm (taken from here) looks like:

func (e *Encryption) Encrypt(data []byte) ([]byte, error) {
    var iLen = len(data)

    if iLen < S2CHeaderSize {
        // packet is smaller than header...
        return nil, errors.New(
            fmt.Sprintf(
                "Packet length is smaller than required! Detected: %d, required: %d",
                iLen, S2CHeaderSize,
            ),
        )
    }

    var pPayload = make([]byte, iLen)

    var xorKey uint32
    var xorNum uint32

    // header
    xorNum = binary.LittleEndian.Uint32(data) ^ SendXorKey
    binary.LittleEndian.PutUint32(pPayload, xorNum)
    xorKey = e.Key.KeyTable[xorNum&RecvXorKeyNumMask]

    var payloadLen = iLen - (S2CHeaderSize - MainCMDSize)
    var j int32 = 4

    for i := 0; i < payloadLen/4; i++ {
        xorNum = binary.LittleEndian.Uint32(data[j:j+4]) ^ xorKey
        binary.LittleEndian.PutUint32(pPayload[j:j+4], xorNum)
        xorKey = e.Key.KeyTable[xorNum&RecvXorKeyNumMask]
        j += 4
    }

    var remainLen = int32(iLen % 4)
    var remainData = make([]byte, 4)

    copy(remainData, data[j:j+remainLen])
    var r = binary.LittleEndian.Uint32(remainData)
    var m = mask[remainLen]

    xorNum = r ^ xorKey
    r = xorNum ^ (xorKey & m)

    binary.LittleEndian.PutUint32(remainData, r)
    copy(pPayload[j:j+remainLen], remainData[0:remainLen])

    return pPayload, nil
}

The messages (data []byte) are built like:

1. 2 Byte MagicKey
2. 2 Byte Length of the message
3. 2 Byte opcode
4. x Byte depending on the opcode

The secrets are:

• MagicKey
• SendXorKey
• e.Key.KeyTable
• The first seed of the KeyTable
  The second seed of the KeyTable
  And for decryption of client messages:
• RecvXorSeed
• Recv2ndXorSeed

What I know/assume / What I can do

• For some parts/opcodes I can do a known plaintext attack. (Length and opcode, also some data for specific opcodes)
• The plaintext almost always encrypts to the same cipher except another key is taken because the previous 4 bytes were too different.
• The key for the current 4-bytes depends on the data of the previous 4 bytes

What I’m interested in:

• Is it possible to get the secrets to be able to decrypt everything?
• If it is possible how would I do that except KPA for known parts?
• Is it possible to decrypt the parts of messages where a KPA is not possible?
• As a last resort: Could I break the encryption by decompiling the client? I assume that the secrets are stored in the client because it is a symmetric encryption algorithm. If possible I would like to avoid this option.
• I assume brute force can’t be done in a realistic/short time?
• Is it possible to derive the RecvXorSeed and Recv2ndXorSeed knowing the position of a key in the table?