Fix loading pkcs#8 encrypted DER-encoded keys in openssl

When we load DER-encoded keys in the openssl-backend we always turn it
into PEM-encoded keys (essentially we prepend and append a header and
footer and use 'toBase64' on the DER data).

The problem comes from the header and footer which is simply chosen
based on which key algorithm was chosen by the user. Which would be
wrong when the key is a PKCS#8 key. This caused OpenSSL to fail when
trying to read it. Surprisingly it still loads correctly for unencrypted
keys with the wrong header, but not for encrypted keys.

This patch adds a small function which checks if a key is an encrypted
PKCS#8 key and then uses this function to figure out if a PKCS#8 header
and footer should be used (note that I only do this for encrypted PKCS#8
keys since, as previously mentioned, unencrypted keys are read correctly
by openssl).

The passphrase is now also passed to the QSslKeyPrivate::decodeDer
function so DER-encoded files can actually be decrypted.

[ChangeLog][QtNetwork][QSslKey] The openssl backend can now load
encrypted PKCS#8 DER-encoded keys.

Task-number: QTBUG-17718
Change-Id: I52eedf19bde297c9aa7fb050e835b3fc0db724e2
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>

1 files changed, 55 insertions, 0 deletions

diff --git a/tests/auto/network/ssl/qsslkey/keys/genkeys.sh b/tests/auto/network/ssl/qsslkey/keys/genkeys.sh
index 7fb15e91ee..6210b42ab4 100755
--- a/tests/auto/network/ssl/qsslkey/keys/genkeys.sh
+++ b/tests/auto/network/ssl/qsslkey/keys/genkeys.sh
@@ -87,3 +87,58 @@ do
   echo -e "\ngenerating EC public key to DER file ..."
   openssl ec -in ec-pri-$size-$curve.pem -pubout -out ec-pub-$size-$curve.der -outform DER
 done
+
+#--- PKCS#8 ------------------------------------------------------------------------
+# Note: We'll just grab some of the keys generated earlier and convert those
+# https://www.openssl.org/docs/manmaster/man1/pkcs8.html#PKCS-5-v1.5-and-PKCS-12-algorithms
+echo -e "\ngenerating unencrypted PKCS#8-format RSA PEM file ..."
+openssl pkcs8 -topk8 -nocrypt -in rsa-pri-512.pem -out rsa-pri-512-pkcs8.pem
+echo -e "\ngenerating unencrypted PKCS#8-format RSA DER file ..."
+openssl pkcs8 -topk8 -nocrypt -in rsa-pri-512.pem -outform DER -out rsa-pri-512-pkcs8.der
+
+echo -e "\ngenerating unencrypted PKCS#8-format DSA PEM file ..."
+openssl pkcs8 -topk8 -nocrypt -in dsa-pri-512.pem -out dsa-pri-512-pkcs8.pem
+echo -e "\ngenerating unencrypted PKCS#8-format DSA DER file ..."
+openssl pkcs8 -topk8 -nocrypt -in dsa-pri-512.pem -outform DER -out dsa-pri-512-pkcs8.der
+
+echo -e "\ngenerating unencrypted PKCS#8-format EC PEM file ..."
+openssl pkcs8 -topk8 -nocrypt -in ec-pri-224-secp224r1.pem -out ec-pri-224-secp224r1-pkcs8.pem
+echo -e "\ngenerating unencrypted PKCS#8-format EC DER file ..."
+openssl pkcs8 -topk8 -nocrypt -in ec-pri-224-secp224r1.pem -outform DER -out ec-pri-224-secp224r1-pkcs8.der
+
+for pkey in rsa-pri-512 dsa-pri-512 ec-pri-224-secp224r1
+do
+  pkeystem=`echo "$pkey" | cut -d- -f 1`
+  # List: https://www.openssl.org/docs/manmaster/man1/pkcs8.html#PKCS-5-v1.5-and-PKCS-12-algorithms
+  # These are technically supported, but fail to generate. Probably because MD2 is deprecated/removed
+  # PBE-MD2-DES PBE-MD2-RC2-64
+  for algorithm in PBE-MD5-DES PBE-SHA1-RC2-64 PBE-MD5-RC2-64 PBE-SHA1-DES
+  do
+    echo -e "\ngenerating encrypted PKCS#8-format (v1) PEM-encoded $pkeystem key using $algorithm ..."
+    openssl pkcs8 -topk8 -in $pkey.pem -v1 $algorithm -out $pkey-pkcs8-$algorithm.pem -passout pass:1234
+
+    echo -e "\ngenerating encrypted PKCS#8-format (v1) DER-encoded $pkeystem key using $algorithm ..."
+    openssl pkcs8 -topk8 -in $pkey.pem -v1 $algorithm -outform DER -out $pkey-pkcs8-$algorithm.der -passout pass:1234
+  done
+
+  for algorithm in PBE-SHA1-RC4-128 PBE-SHA1-RC4-40 PBE-SHA1-3DES PBE-SHA1-2DES PBE-SHA1-RC2-128 PBE-SHA1-RC2-40
+  do
+    echo -e "\ngenerating encrypted PKCS#8-format (v1 PKCS#12) PEM-encoded $pkeystem key using $algorithm ..."
+    openssl pkcs8 -topk8 -in $pkey.pem -v1 $algorithm -out $pkey-pkcs8-pkcs12-$algorithm.pem -passout pass:1234
+
+    echo -e "\ngenerating encrypted PKCS#8-format (v1 PKCS#12) DER-encoded $pkeystem key using $algorithm ..."
+    openssl pkcs8 -topk8 -in $pkey.pem -v1 $algorithm -outform DER -out $pkey-pkcs8-pkcs12-$algorithm.der -passout pass:1234
+  done
+
+  for algorithm in des3 aes128 aes256 rc2
+  do
+    for prf in hmacWithSHA1 hmacWithSHA256
+    do
+      echo -e "\ngenerating encrypted PKCS#8-format (v2) PEM-encoded $pkeystem key using $algorithm and $prf ..."
+      openssl pkcs8 -topk8 -in $pkey.pem -v2 $algorithm -v2prf $prf -out $pkey-pkcs8-$algorithm-$prf.pem -passout pass:1234
+
+      echo -e "\ngenerating encrypted PKCS#8-format (v2) DER-encoded $pkeystem key using $algorithm and $prf ..."
+      openssl pkcs8 -topk8 -in $pkey.pem -v2 $algorithm -v2prf $prf -outform DER -out $pkey-pkcs8-$algorithm-$prf.der -passout pass:1234
+    done
+  done
+done