RSA-OAEP between Python and Kotlin: fix BadPaddingException by matching SHA-512 MGF1 and OAEP parameters

Learn why RSA-OAEP decryption fails between Python cryptography and Kotlin/Java: implicit MGF1 SHA-1 defaults. Fix by pinning SHA-512 for OAEP and MGF1.

Cross-language RSA can be deceptively tricky. A Python FastAPI backend using the cryptography library and an Android client in Kotlin may each encrypt/decrypt fine on their own, yet fail when you try to interoperate. If you see errors like BadPaddingException or “Decryption failed,” the culprit is often the same: hidden defaults in OAEP parameters that don’t match across runtimes.

Minimal repro: works locally, breaks across Python ↔ Kotlin

The server uses RSA-OAEP with SHA-512 in Python. Keys are stored in DER format, the public key as SubjectPublicKeyInfo and the private key as PKCS#8. The snippet below shows the working setup.

import os, base64, re
from cryptography.hazmat.primitives.asymmetric import rsa as rsa_mod, padding as asym_padding
from cryptography.hazmat.primitives import serialization as ser, hashes as digest
from cryptography.hazmat.backends import default_backend as backend
if os.path.exists("private.key") and os.path.exists("public.key"):
    print("Loading existing keys")
    with open("private.key", "rb") as f_priv, open("public.key", "rb") as f_pub:
        priv_obj = ser.load_der_private_key(f_priv.read(), None, backend())
        pub_obj = ser.load_der_public_key(f_pub.read(), backend())
else:
    print("Generating new keys")
    priv_obj = rsa_mod.generate_private_key(public_exponent=65537, key_size=4096, backend=backend())
    priv_bytes = priv_obj.private_bytes(
        ser.Encoding.DER,
        ser.PrivateFormat.PKCS8,
        ser.NoEncryption()
    )
    pub_obj = priv_obj.public_key()
    pub_bytes = pub_obj.public_bytes(
        ser.Encoding.DER,
        ser.PublicFormat.SubjectPublicKeyInfo
    )
    with open("private.key", "wb") as f_priv, open("public.key", "wb") as f_pub:
        f_priv.write(priv_bytes)
        f_pub.write(pub_bytes)
def seal(plain: str) -> str:
    return base64.b64encode(pub_obj.encrypt(
        plain.encode(),
        asym_padding.OAEP(
            mgf=asym_padding.MGF1(digest.SHA512()),
            algorithm=digest.SHA512(),
            label=None
        )
    )).decode()
def open_(ct: str) -> str:
    return priv_obj.decrypt(
        base64.b64decode(ct),
        asym_padding.OAEP(
            mgf=asym_padding.MGF1(digest.SHA512()),
            algorithm=digest.SHA512(),
            label=None
        )
    ).decode()
while (user_in := input("RSA: ")) != "":
    if re.match("^enc", user_in):
        print("Encrypt", seal(user_in[3:].strip()))
    if re.match("^dec", user_in):
        print("Decrypt", open_(user_in[3:].strip()))

On Android/Kotlin, the initial implementation also chose RSA/ECB/OAEPwithSHA-512andMGF1Padding, but left OAEP parameters implicit. It encrypts and decrypts fine locally, yet fails against Python.

import java.io.File
import java.security.KeyFactory
import java.security.spec.PKCS8EncodedKeySpec
import java.security.spec.X509EncodedKeySpec
import java.util.Base64
import javax.crypto.Cipher
fun main() {
    print("Text: ")
    val ct = rsaEnc(File("public.key"), readLine().toString())
    println(ct)
    print("Ciphertext: ")
    println("Decrypted: ${rsaDec(File("private.key"), readLine().toString())}")
}
fun b64e(data: ByteArray) = Base64.getEncoder().encodeToString(data)
fun b64d(txt: String) = Base64.getDecoder().decode(txt)
fun rsaEnc(pubFile: File, plain: String): String {
    val pubK = KeyFactory.getInstance("RSA").generatePublic(X509EncodedKeySpec(pubFile.readBytes()))
    val cipher = Cipher.getInstance("RSA/ECB/OAEPwithSHA-512andMGF1Padding")
    cipher.init(Cipher.ENCRYPT_MODE, pubK)
    return b64e(cipher.doFinal(plain.toByteArray()))
}
fun rsaDec(privFile: File, blob: String): String {
    val privK = KeyFactory.getInstance("RSA").generatePrivate(PKCS8EncodedKeySpec(privFile.readBytes()))
    val cipher = Cipher.getInstance("RSA/ECB/OAEPwithSHA-512andMGF1Padding")
    cipher.init(Cipher.DECRYPT_MODE, privK)
    return cipher.doFinal(b64d(blob)).decodeToString()
}

When decrypting cross-language ciphertexts, the failures look like this:

Exception in thread "main" javax.crypto.BadPaddingException: Padding error in decryption

ValueError: Decryption failed

What’s actually wrong

RSA-OAEP is parameterized. Besides the RSA key and the plaintext, the scheme depends on a hash for the main digest, a hash for MGF1, and a label (PSource). If two sides pick different OAEP parameters, decryption fails with padding errors even if key sizes, encodings, and transformation strings look aligned.

Relying on implicit defaults hides these differences. In this setup, the Python side explicitly uses SHA-512 for both the OAEP digest and MGF1. The Java/Kotlin side, however, did not supply an OAEPParameterSpec, so defaults kicked in. As noted in the fix, the default MGF1 hash on the Java side is SHA-1, which caused the mismatch and broke interoperability.

The fix: pin OAEP parameters on the Kotlin side

Specify all OAEP parameters explicitly with OAEPParameterSpec, matching what the server uses: SHA-512 for the OAEP digest, MGF1 with SHA-512, and PSource.PSpecified.DEFAULT.

import java.io.File
import java.security.KeyFactory
import java.security.spec.PKCS8EncodedKeySpec
import java.security.spec.X509EncodedKeySpec
import java.util.Base64
import javax.crypto.Cipher
import javax.crypto.spec.OAEPParameterSpec
import javax.crypto.spec.MGF1ParameterSpec
import javax.crypto.spec.PSource
fun main() {
    print("Text: ")
    val ct = rsaEnc(File("public.key"), readLine().toString())
    println(ct)
    print("Ciphertext: ")
    println("Decrypted: ${rsaDec(File("private.key"), readLine().toString())}")
}
fun b64e(data: ByteArray) = Base64.getEncoder().encodeToString(data)
fun b64d(txt: String) = Base64.getDecoder().decode(txt)
fun rsaEnc(pubFile: File, plain: String): String {
    val pubK = KeyFactory.getInstance("RSA").generatePublic(X509EncodedKeySpec(pubFile.readBytes()))
    val cipher = Cipher.getInstance("RSA/ECB/OAEPwithSHA-512andMGF1Padding")
    val params = OAEPParameterSpec("SHA-512", "MGF1", MGF1ParameterSpec("SHA-512"), PSource.PSpecified.DEFAULT)
    cipher.init(Cipher.ENCRYPT_MODE, pubK, params)
    return b64e(cipher.doFinal(plain.toByteArray()))
}
fun rsaDec(privFile: File, blob: String): String {
    val privK = KeyFactory.getInstance("RSA").generatePrivate(PKCS8EncodedKeySpec(privFile.readBytes()))
    val cipher = Cipher.getInstance("RSA/ECB/OAEPwithSHA-512andMGF1Padding")
    val params = OAEPParameterSpec("SHA-512", "MGF1", MGF1ParameterSpec("SHA-512"), PSource.PSpecified.DEFAULT)
    cipher.init(Cipher.DECRYPT_MODE, privK, params)
    return cipher.doFinal(b64d(blob)).decodeToString()
}

With matching OAEP parameters, Python-encrypted data decrypts in Kotlin and vice versa.

Why this matters

Interoperability hinges on more than a transformation string. OAEP looks uniform as “RSA/ECB/OAEPwithSHA-512andMGF1Padding,” but that string doesn’t pin every parameter. When one side uses explicit SHA-512 and the other silently falls back to different defaults, the mismatch produces padding errors that mask the real cause. Being precise about OAEP parameters removes ambiguity, keeps crypto handshakes predictable, and prevents time sinks during cross-language integration.

Takeaways

Don’t rely on defaults in cryptographic APIs. Align OAEP parameters explicitly on every platform, including the digest, MGF1 digest, and PSource. Validate both directions of encryption and decryption once parameters are pinned. With that in place, RSA-OAEP between Python cryptography and Kotlin’s Cipher works as intended.