Fix loading pkcs#8 encrypted DER-encoded keys in openssl

When we load DER-encoded keys in the openssl-backend we always turn it
into PEM-encoded keys (essentially we prepend and append a header and
footer and use 'toBase64' on the DER data).

The problem comes from the header and footer which is simply chosen
based on which key algorithm was chosen by the user. Which would be
wrong when the key is a PKCS#8 key. This caused OpenSSL to fail when
trying to read it. Surprisingly it still loads correctly for unencrypted
keys with the wrong header, but not for encrypted keys.

This patch adds a small function which checks if a key is an encrypted
PKCS#8 key and then uses this function to figure out if a PKCS#8 header
and footer should be used (note that I only do this for encrypted PKCS#8
keys since, as previously mentioned, unencrypted keys are read correctly
by openssl).

The passphrase is now also passed to the QSslKeyPrivate::decodeDer
function so DER-encoded files can actually be decrypted.

[ChangeLog][QtNetwork][QSslKey] The openssl backend can now load
encrypted PKCS#8 DER-encoded keys.

Task-number: QTBUG-17718
Change-Id: I52eedf19bde297c9aa7fb050e835b3fc0db724e2
Reviewed-by: Edward Welbourne <edward.welbourne@qt.io>

1 files changed, 4 insertions, 3 deletions

diff --git a/src/network/ssl/qsslkey_p.h b/src/network/ssl/qsslkey_p.h
index c93941c198..d6c5af9d47 100644
--- a/src/network/ssl/qsslkey_p.h
+++ b/src/network/ssl/qsslkey_p.h
@@ -81,9 +81,8 @@ public:
 #ifndef QT_NO_OPENSSL
     bool fromEVP_PKEY(EVP_PKEY *pkey);
 #endif
-    void decodeDer(const QByteArray &der, bool deepClear = true);
-    void decodePem(const QByteArray &pem, const QByteArray &passPhrase,
-                   bool deepClear = true);
+    void decodeDer(const QByteArray &der, const QByteArray &passPhrase = {}, bool deepClear = true);
+    void decodePem(const QByteArray &pem, const QByteArray &passPhrase, bool deepClear = true);
     QByteArray pemHeader() const;
     QByteArray pemFooter() const;
     QByteArray pemFromDer(const QByteArray &der, const QMap<QByteArray, QByteArray> &headers) const;
@@ -93,6 +92,8 @@ public:
     QByteArray toPem(const QByteArray &passPhrase) const;
     Qt::HANDLE handle() const;
 
+    bool isEncryptedPkcs8(const QByteArray &der) const;
+
     bool isNull;
     QSsl::KeyType type;
     QSsl::KeyAlgorithm algorithm;