I’m running in circles trying to implement encrypted server communication in a Unity WebGL app. Our server is encrypting all messages that are send to the client using a GO implementation of EC-Diffie Hellman. On Server site everything works fine.

On Client site I generate a ECDH key pair using Bouncy Castle.

        public static void GenerateKeyPair()
        {
            X9ECParameters x9ecParams = NistNamedCurves.GetByName("P-256");
            ECDomainParameters ecParams = new ECDomainParameters(x9ecParams);
            ECKeyPairGenerator generator = new ECKeyPairGenerator();
            generator.Init(new ECKeyGenerationParameters(ecParams, new SecureRandom()));
            _generatedKey = generator.GenerateKeyPair();
        }

I get the servers public key and generate the shared key / symmetric key

        public static void CalculateSharedKey(byte[] publicKeyServer)
        {
            try
            {
                var curve = NistNamedCurves.GetByName("P-256");
                var domainParameters = new ECDomainParameters(curve);
                
                var x = new BigInteger(1, publicKeyServer, 1, 32);
                var y = new BigInteger(1, publicKeyServer, 33, 32);
                var q = curve.Curve.CreatePoint(x, y);
                var ecPublicKeyParameters = new ECPublicKeyParameters(q, domainParameters);
                
                var ecdh = new ECDHBasicAgreement();
                ecdh.Init(_generatedKey.Private);
                var sharedSecretBigInt = ecdh.CalculateAgreement(ecPublicKeyParameters);
                var sharedSecret = sharedSecretBigInt.ToByteArrayUnsigned();
                
                _symmetricKey = SHA256.Create().ComputeHash(sharedSecret);
                
                Debug.Log( "shared key (calculated)" + BitConverter.ToString(_symmetricKey));
            }
            catch (Exception ex)
            {
                Debug.LogError("Error calculating shared key: " + ex.Message);
            }
        }

for debugging purpose, the server is currently sending its calculated shared key, so I can verify both sides calculate the same shared key.

But when it comes to decrypting the messages received from the server, I’m running out of ideas.
Here are the things I already tried:

• System.Security.Cryptography.AesGcm

System.Security.Cryptography.AesGcm since our server uses GCM. This throws errors because Unity’s Mono Runtime is not supporting AesGcm, throwing a Not Implemented Exception.

• Bouncy Castle AesEngine

            var nonce = new byte[12];
            Array.Copy(encryptedDataWithTag, 0, nonce, 0, 12);
            
            var tag = new byte[16];
            Array.Copy(encryptedDataWithTag, encryptedDataWithTag.Length - 16, tag, 0, 16);

            var encryptedData = new byte[encryptedDataWithTag.Length - tag.Length];
            Array.Copy(encryptedDataWithTag, nonce.Length, encryptedData, 0, encryptedData.Length);  
          
            GcmBlockCipher gcm = new GcmBlockCipher(new AesEngine());
            var keyParam = new KeyParameter(_symmetricKey);
            
            gcm.Init(false, new AeadParameters(keyParam, macSize, nonce));
            var plainBytes = new byte[gcm.GetOutputSize(encryptedData.Length)];
            int returnLength = gcm.ProcessBytes(encryptedData, 0, encryptedData.Length, plainBytes, 0);
            
            gcm.DoFinal(plainBytes, returnLength);

the mac size (TagSize * 8) should be 96, because the server uses a tagSize of 12. But this always results in

Decryption failed
Parameter name: mac check in GCM failed

I tried several other approaches, but ran into dead ends on every single one and it starts getting frustrating.
Why is the mac check failing when both, server and client are using 96?

I tried several methods to decrypt the server message, but I need to get around the mac check in GCM failederror