I am trying to encrypt a string using RSA-OAEP-SHA-512 in the WebCrypto API (client side) and then I want to decrypt the string again using Java (server side). However, the server side decryption failes with the Error: Padding error in decryption.

I’m generating the RSA keys in JavaScript using:

async function generateRSAKeyPair() {
    const rsaKeyPair = await window.crypto.subtle.generateKey(
      {
        name: 'RSA-OAEP',
        modulusLength: 4096,
        publicExponent: new Uint8Array([0x01, 0x00, 0x01]), // 65537
        hash: { name: 'SHA-512' },
      },
      true,
      ['encrypt', 'decrypt']
    );
    return rsaKeyPair;
}

I’m encrypting a JSON string using:

sync function encryptWithRSA(jsonString, jsonRSAPublicKey) {
    // Parse JSON keys
    const rsaPublicKey = JSON.parse(jsonRSAPublicKey);

    // Import RSA public key
    const importedRSAPublicKey = await window.crypto.subtle.importKey(
      'jwk',
      rsaPublicKey,
      { name: 'RSA-OAEP', hash: { name: 'SHA-512' } },
      true,
      ['encrypt']
    );

    // Convert JSON String to ArrayBuffer
    const aesKeyBuffer = new TextEncoder().encode(jsonString);

    // Encrypt AES key using RSA public key
    const encryptedAesKeyBuffer = await window.crypto.subtle.encrypt(
      { name: 'RSA-OAEP' },
      importedRSAPublicKey,
      aesKeyBuffer
    );

    // Convert encrypted AES key to base64
      const encryptedAesKeyBase64 = window.btoa(String.fromCharCode(...new Uint8Array(encryptedAesKeyBuffer)));
    return encryptedAesKeyBase64;
}

I am trying to decrypt it in Java using:

import javax.crypto.Cipher;
import java.security.KeyFactory;
import java.security.PrivateKey;
import java.security.spec.PKCS8EncodedKeySpec;
import java.security.interfaces.RSAPrivateKey;
import java.security.spec.RSAPrivateKeySpec;
import java.security.spec.RSAPrivateCrtKeySpec;
import java.math.BigInteger;
import java.util.Base64;
import java.util.Map;
import com.fasterxml.jackson.databind.ObjectMapper;

public void decrypt() {
    try {
        PrivateKey privateKey = loadPrivateKey();

        Cipher cipher = Cipher.getInstance("RSA/ECB/OAEPWithSHA-512AndMGF1Padding");

        cipher.init(Cipher.DECRYPT_MODE, privateKey);

        byte[] encryptedBytes = Base64.getDecoder().decode(cipherText);
        byte[] decryptedBytes = cipher.doFinal(encryptedBytes);

        this.clearText = new String(decryptedBytes);
        System.out.println("Decrypted clear text: " + clearText);
    } catch (Exception e) {
        e.printStackTrace();
        System.err.println("Decryption failed: " + e.getMessage());
    }
}

    private PrivateKey loadPrivateKey() throws Exception {
        // Parse the JWK JSON string
        ObjectMapper mapper = new ObjectMapper();
        Map<String, Object> jwk = mapper.readValue(rsaKey, Map.class);
        System.out.println("JWK parsed: " + jwk);

        // Extract the key components from the JWK
        String nBase64 = (String) jwk.get("n");
        String dBase64 = (String) jwk.get("d");
        String pBase64 = (String) jwk.get("p");
        String qBase64 = (String) jwk.get("q");
        String dpBase64 = (String) jwk.get("dp");
        String dqBase64 = (String) jwk.get("dq");
        String qiBase64 = (String) jwk.get("qi");
        String eBase64 = (String) jwk.get("e");

        // Decode the Base64 URL-encoded components
        BigInteger modulus = new BigInteger(1, Base64.getUrlDecoder().decode(nBase64));
        BigInteger privateExponent = new BigInteger(1, Base64.getUrlDecoder().decode(dBase64));
        BigInteger publicExponent = new BigInteger(1, Base64.getUrlDecoder().decode(eBase64));
        BigInteger primeP = new BigInteger(1, Base64.getUrlDecoder().decode(pBase64));
        BigInteger primeQ = new BigInteger(1, Base64.getUrlDecoder().decode(qBase64));
        BigInteger primeExponentP = new BigInteger(1, Base64.getUrlDecoder().decode(dpBase64));
        BigInteger primeExponentQ = new BigInteger(1, Base64.getUrlDecoder().decode(dqBase64));
        BigInteger crtCoefficient = new BigInteger(1, Base64.getUrlDecoder().decode(qiBase64));

        // Create the RSA private key specification
        RSAPrivateCrtKeySpec keySpec = new RSAPrivateCrtKeySpec(modulus, publicExponent, privateExponent, primeP, primeQ, primeExponentP, primeExponentQ, crtCoefficient);

        // Generate the private key
        KeyFactory keyFactory = KeyFactory.getInstance(AppConfig.CRYPTO_RSA);
        return keyFactory.generatePrivate(keySpec);
    }

This gives the following output:

javax.crypto.BadPaddingException: Padding error in decryption
    at java.base/com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:383)
    at java.base/com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:419)
    at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2244)
    at zkgitclientcli.crypto.RsaEncryptionHandler2.decrypt(RsaEncryptionHandler2.java:48)
    at zkgitclientcli.commands.login.DecryptAesKey.execute(DecryptAesKey.java:23)
    at zkgitclientcli.commands.CommandManager.executeCommand(CommandManager.java:44)
    at zkgitclientcli.ZkGit.main(ZkGit.java:12)
    at org.codehaus.mojo.exec.ExecJavaMojo.doMain(ExecJavaMojo.java:385)
    at org.codehaus.mojo.exec.ExecJavaMojo.doExec(ExecJavaMojo.java:374)
    at org.codehaus.mojo.exec.ExecJavaMojo.lambda$execute$0(ExecJavaMojo.java:296)
    at java.base/java.lang.Thread.run(Thread.java:1583)
Decryption failed: Padding error in decryption

The private key import is most likely working, since strings encrypted in Java using the corresponding public key and be successfully decrypted using this method. It only fails, when using the WebCrypto encrypted string as input.

Also, I can write a JavaScript method that successfully decrypts the encrypted string, so the base64 encoded encrypted string is decryptable, but not in Java…

I have tried the following settings, without success:
SHA-1: RSA/ECB/OAEPWithSHA-1AndMGF1Padding
SHA-224: RSA/ECB/OAEPWithSHA-224AndMGF1Padding
SHA-256: RSA/ECB/OAEPWithSHA-256AndMGF1Padding
SHA-384: RSA/ECB/OAEPWithSHA-384AndMGF1Padding