The new large message test in sslapitest needs OPENSSL_NO_DTLS guards

Reviewed-by: Richard Levitte <levitte@openssl.org>

Before loading a key from an engine, it may need to be initialized.
When done loading the key, we must de-initialize the engine.
(if the engine is already initialized somehow, only the reference
counter will be incremented then decremented)

Reviewed-by: Stephen Henson <steve@openssl.org>

This reverts commit 0a720029

.
This fails to call ENGINE_finish; an alternate fix is forthcoming.

Reviewed-by: Dr. Stephen Henson <steve@openssl.org>

When I said before that s_client "used to work in 1.0.2" that was only
partly true. It worked for engines which provided a default generic
method for some key type, because it called ENGINE_set_default() and
that ended up being an implicit initialisation and functional refcount.

But an engine which doesn't provide generic methods doesn't get initialised,
and then when you try to use it you get an error:

cannot load client certificate private key file from engine
140688147056384:error:26096075:engine routines:ENGINE_load_private_key:not initialised:crypto/engine/eng_pkey.c:66:
unable to load client certificate private key file

cf. https://github.com/OpenSC/libp11/issues/107

 (in which we discover
that engine_pkcs11 *used* to provide generic methods that OpenSSL would
try to use for ephemeral DH keys when negotiating ECDHE cipher suites in
TLS, and that didn't work out very well.)

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1639)

This used to work in 1.0.2 but disappeared when the argument parsing was
revamped.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1639)

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add the ability to test both server initiated and client initiated reneg.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add update for testing renegotiation. Also change info on CTLOG_FILE
environment variable - which always seems to be required.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Reviewed-by: Rich Salz <rsalz@openssl.org>

The conversion to WPACKET broke the construction of the renegotiation
extension.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Reviewed-by: Rich Salz <rsalz@openssl.org>

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>

The TLSProxy::Record->new call hard-codes a version, like
70-test_sslrecords.t.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>

Avoid making the CI blow up.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>

This is a regression test for
https://github.com/openssl/openssl/pull/1431

. It tests a
maximally-padded record with each possible invalid offset.

This required fixing a bug in Message.pm where the client sending a
fatal alert followed by close_notify was still treated as success.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>

A mem leak could occur on an error path. Also the mempacket BIO_METHOD
needs to be cleaned up, because of the newly added DTLS test.

Also fixed a double semi-colon in ssltestlib.c

Reviewed-by: Rich Salz <rsalz@openssl.org>

This would have caught 099e2968. This is
a port of the test added in
https://boringssl.googlesource.com/boringssl/+/7c040756178e14a4d181b6d93abb3827c93189c4



Reviewed-by: Emilia Käsper <emilia@openssl.org>
Reviewed-by: Rich Salz <rsalz@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/1496)

commit c536b6be

 introduced a bug that causes a reachable assert. This fixes
it.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Reviewed-by: Richard Levitte <levitte@openssl.org>

Reviewed-by: Emilia Käsper <emilia@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>

The buffer to receive messages is initialised to 16k. If a message is
received that is larger than that then the buffer is "realloc'd". This can
cause the location of the underlying buffer to change. Anything that is
referring to the old location will be referring to free'd data. In the
recent commit c1ef7c97 (master) and 4b390b6c

 (1.1.0) the point in the code
where the message buffer is grown was changed. However s->init_msg was not
updated to point at the new location.

CVE-2016-6309

Reviewed-by: Emilia Käsper <emilia@openssl.org>

If we request more bytes to be allocated than double what we have already
written, then we grow the buffer by the wrong amount.

Reviewed-by: Emilia Käsper <emilia@openssl.org>

Ensure that we send a large message during the test suite.

Reviewed-by: Emilia Käsper <emilia@openssl.org>

We actually construct a HelloVerifyRequest in two places with common code
pulled into a single function. This one commit handles both places.

Reviewed-by: Rich Salz <rsalz@openssl.org>

If the underlying BUF_MEM gets realloc'd then the pointer returned could
become invalid. Therefore we should always ensure that the allocated
memory is filled in prior to any more WPACKET_* calls.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Change code so when switching on an enumeration, have case's for all
enumeration values.

Reviewed-by: Andy Polyakov <appro@openssl.org>

Reviewed-by: Richard Levitte <levitte@openssl.org>

Russian GOST ciphersuites are vulnerable to the KCI attack because they use
long-term keys to establish the connection when ssl client authorization is
on. This change brings the GOST implementation into line with the latest
specs in order to avoid the attack. It should not break backwards
compatibility.

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>

If while calling SSL_peek() we read an empty record then we go into an
infinite loop, continually trying to read data from the empty record and
never making any progress. This could be exploited by a malicious peer in
a Denial Of Service attack.

CVE-2016-6305

GitHub Issue #1563

Reviewed-by: Rich Salz <rsalz@openssl.org>

If a server sent multiple NPN extensions in a single ClientHello then a
mem leak can occur. This will only happen where the client has requested
NPN in the first place. It does not occur during renegotiation. Therefore
the maximum that could be leaked in a single connection with a malicious
server is 64k (the maximum size of the ServerHello extensions section). As
this is client side, only occurs if NPN has been requested and does not
occur during renegotiation this is unlikely to be exploitable.

Issue reported by Shi Lei.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Test that the OCSP callbacks work as expected.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Add a function for testing whether a given OCSP_RESPID matches with a
certificate.

Reviewed-by: Rich Salz <rsalz@openssl.org>

OCSP_RESPID was made opaque in 1.1.0, but no accessors were provided for
setting the name/key value for the OCSP_RESPID.

Reviewed-by: Rich Salz <rsalz@openssl.org>

A malicious client can send an excessively large OCSP Status Request
extension. If that client continually requests renegotiation,
sending a large OCSP Status Request extension each time, then there will
be unbounded memory growth on the server. This will eventually lead to a
Denial Of Service attack through memory exhaustion. Servers with a
default configuration are vulnerable even if they do not support OCSP.
Builds using the "no-ocsp" build time option are not affected.

I have also checked other extensions to see if they suffer from a similar
problem but I could not find any other issues.

CVE-2016-6304

Issue reported by Shi Lei.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Reviewed-by: Matt Caswell <matt@openssl.org>

This issue is very similar to CVE-2016-6307 described in the previous
commit. The underlying defect is different but the security analysis and
impacts are the same except that it impacts DTLS.

A DTLS message includes 3 bytes for its length in the header for the
message.
This would allow for messages up to 16Mb in length. Messages of this length
are excessive and OpenSSL includes a check to ensure that a peer is sending
reasonably sized messages in order to avoid too much memory being consumed
to service a connection. A flaw in the logic of version 1.1.0 means that
memory for the message is allocated too early, prior to the excessive
message length check. Due to way memory is allocated in OpenSSL this could
mean an attacker could force up to 21Mb to be allocated to service a
connection. This could lead to a Denial of Service through memory
exhaustion. However, the excessive message length check still takes place,
and this would cause the connection to immediately fail. Assuming that the
application calls SSL_free() on the failed conneciton in a timely manner
then the 21Mb of allocated memory will then be immediately freed again.
Therefore the excessive memory allocation will be transitory in nature.
This then means that there is only a security impact if:

1) The application does not call SSL_free() in a timely manner in the
event that the connection fails
or
2) The application is working in a constrained environment where there
is very little free memory
or
3) The attacker initiates multiple connection attempts such that there
are multiple connections in a state where memory has been allocated for
the connection; SSL_free() has not yet been called; and there is
insufficient memory to service the multiple requests.

Except in the instance of (1) above any Denial Of Service is likely to
be transitory because as soon as the connection fails the memory is
subsequently freed again in the SSL_free() call. However there is an
increased risk during this period of application crashes due to the lack
of memory - which would then mean a more serious Denial of Service.

This issue does not affect TLS users.

Issue was reported by Shi Lei (Gear Team, Qihoo 360 Inc.).

CVE-2016-6308

Reviewed-by: Richard Levitte <levitte@openssl.org>

A TLS message includes 3 bytes for its length in the header for the message.
This would allow for messages up to 16Mb in length. Messages of this length
are excessive and OpenSSL includes a check to ensure that a peer is sending
reasonably sized messages in order to avoid too much memory being consumed
to service a connection. A flaw in the logic of version 1.1.0 means that
memory for the message is allocated too early, prior to the excessive
message length check. Due to way memory is allocated in OpenSSL this could
mean an attacker could force up to 21Mb to be allocated to service a
connection. This could lead to a Denial of Service through memory
exhaustion. However, the excessive message length check still takes place,
and this would cause the connection to immediately fail. Assuming that the
application calls SSL_free() on the failed conneciton in a timely manner
then the 21Mb of allocated memory will then be immediately freed again.
Therefore the excessive memory allocation will be transitory in nature.
This then means that there is only a security impact if:

1) The application does not call SSL_free() in a timely manner in the
event that the connection fails
or
2) The application is working in a constrained environment where there
is very little free memory
or
3) The attacker initiates multiple connection attempts such that there
are multiple connections in a state where memory has been allocated for
the connection; SSL_free() has not yet been called; and there is
insufficient memory to service the multiple requests.

Except in the instance of (1) above any Denial Of Service is likely to
be transitory because as soon as the connection fails the memory is
subsequently freed again in the SSL_free() call. However there is an
increased risk during this period of application crashes due to the lack
of memory - which would then mean a more serious Denial of Service.

This issue does not affect DTLS users.

Issue was reported by Shi Lei (Gear Team, Qihoo 360 Inc.).

CVE-2016-6307

Reviewed-by: Richard Levitte <levitte@openssl.org>

If OPENSSL_sk_insert() calls OPENSSL_realloc() and it fails, it was leaking
the originally allocated memory.

Reviewed-by: Rich Salz <rsalz@openssl.org>

Certain warning alerts are ignored if they are received. This can mean that
no progress will be made if one peer continually sends those warning alerts.
Implement a count so that we abort the connection if we receive too many.

Issue reported by Shi Lei.

Reviewed-by: Rich Salz <rsalz@openssl.org>