Manual cherry-pick of 0704343f

.

Signed-off-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>

Reviewed-by: Matt Caswell <matt@openssl.org>

If a DTLS client that does not support secure renegotiation connects to an
OpenSSL DTLS server then, by default, renegotiation is disabled. If a
server application attempts to initiate a renegotiation then OpenSSL is
supposed to prevent this. However due to a discrepancy between the TLS and
DTLS code, the server sends a HelloRequest anyway in DTLS.

This is not a security concern because the handshake will still fail later
in the process when the client responds with a ClientHello.

Reviewed-by: Tim Hudson <tjh@openssl.org>

In DTLS if an IO retry occurs during writing of a fragmented ClientHello
then we can end up reseting the finish mac variables on the retry, which
causes a handshake failure. We should only reset on the first attempt not
on retries.

Thanks to BoringSSL for reporting this issue.

RT#4119

Reviewed-by: Tim Hudson <tjh@openssl.org>

The function tls1_get_curvelist() has an explicit check to see if s->cert
is NULL or not. However the check appears *after* calling the tls1_suiteb
macro which derefs s->cert. In reality s->cert can never be NULL because
it is created in SSL_new(). If the malloc fails then the SSL_new call fails
and no SSL object is created.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(cherry picked from commit 6329b609)

Conflicts:
	ssl/t1_lib.c

During work on a larger change in master a number of locations were
identified where return value checks were missing. This backports the
relevant fixes.

Reviewed-by: Richard Levitte <levitte@openssl.org>

./Configure [target] --strict-warnings -Wno-pedantic-ms-format
would not add '-pedantic' because it matches '-Wno-pedantic-ms-format',
which was added first.

Reviewed-by: Matt Caswell <matt@openssl.org>
(cherry picked from commit 6703c4ea)

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>

The various dtls1_get*_methods did not handle the DTLS_ANY_VERSION case,
so this needed to be added.

Reviewed-by: Tim Hudson <tjh@openssl.org>

EVP_SignInit_ex was missing from the NAME section of its man page so
typing "man EVP_SignInit_ex" failed to load the page.

Reviewed-by: Stephen Henson <steve@openssl.org>
(cherry picked from commit 3d866ea6)

Clarify that git format-patch output is preferred for creating patch files.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit f89ee71b)

Close GH Issue 69
Close GH PR 457/RT4113
Some other updates

By Rich Salz, Alessandro Ghedini, Steve Marquess, Collin Anderson

(manual cherry-pick of a2aaf8be and
b06935f4

)

Reviewed-by: Kurt Roeckx <kurt@openssl.org>

Signed-off-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Rich Salz <rsalz@akamai.com>

MR #1276, RT #4107

(cherry picked from commit 94b3664a)

RT#4081

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 070c2332)

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 8cf9d71a)

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 3240e7cf)

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit d900a015)

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit c2319cf9)

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 3f6c7691)

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 8acaabec)

0 is a valid file descriptor.

RT#4068

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 4428c7db)

Previous language was unclear.  New language isn't pretty but I believe
it is more accurate.

Signed-off-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Ben Laurie <ben@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 8cbb048c)

Don't mark a certificate as self signed if keyUsage is present and
certificate signing not asserted.

PR#3979

Reviewed-by: Matt Caswell <matt@openssl.org>
(cherry picked from commit f51e5ed6)

Reviewed-by: Rich Salz <rsalz@openssl.org>

RFC5753 requires that we omit parameters for AES key wrap and set them
to NULL for 3DES wrap. OpenSSL decrypt uses the received algorithm
parameters so can transparently handle either form.

Reviewed-by: Andy Polyakov <appro@openssl.org>
(cherry picked from commit 4ec36aff)

(cherry picked from commit 028bac06

)

Signed-off-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Andy Polyakov <appro@openssl.org>

PR#4079

Reviewed-by: Tim Hudson <tjh@openssl.org>
(cherry picked from commit c69ce935)

The function int_rsa_verify is an internal function used for verifying an
RSA signature. It takes an argument |dtype| which indicates the digest type
that was used. Dependant on that digest type the processing of the
signature data will vary. In particular if |dtype == NID_mdc2| and the
signature data is a bare OCTETSTRING then it is treated differently to the
default case where the signature data is treated as a DigestInfo (X509_SIG).

Due to a missing "else" keyword the logic actually correctly processes the
OCTETSTRING format signature first, and then attempts to continue and
process it as DigestInfo. This will invariably fail because we already know
that it is a bare OCTETSTRING.

This failure doesn't actualy make a real difference because it ends up at
the |err| label regardless and still returns a "success" result. This patch
just cleans things up to make it look a bit more sane.

RT#4076

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit dffe5109)

Reviewed-by: Matt Caswell <matt@openssl.org>
(cherry picked from commit 5850cc75)

Disable -Wshadow error when building with GCC
Add support for linu-x86_64-clang debug; this is needed for Travis CI.
Disable linux-clang and mingw debug builds on Travis CI; not supported.
Fix Travis email notifications config

Signed-off-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Andy Polyakov <appro@openssl.org>

Signed-off-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Rich Salz <rsalz@openssl.org>
MR #1231

(cherry picked from commit 99c20333)

It returns -1 on error, not 0.

Signed-off-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Rich Salz <rsalz@openssl.org>
MR #1231

(cherry picked from commit d6e92c0b)

BN_with_flags() will read the dest->flags to keep the BN_FLG_MALLOCED but
overwrites everything else.

Signed-off-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Rich Salz <rsalz@openssl.org>
MR #1231

(cherry picked from commit f92768e6)

Reviewed-by: Rich Salz <rsalz@openssl.org>
(cherry picked from commit e58c4d3c)

Some extension handling functions were passing in a pointer to the start
of the data, plus the length in order to calculate the end, rather than
just passing in the end to start with. This change makes things a little
more readable.

Reviewed-by: Emilia Käsper <emilia@openssl.org>

RT#4069

Reviewed-by: Emilia Käsper <emilia@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>

Thanks to David Benjamin <davidben@google.com> for pointing them out.

Reviewed-by: Steve Henson <steve@openssl.org>
MR #1198

(cherry picked from commit 605236f6)

PR#4042

Reviewed-by: Tim Hudson <tjh@openssl.org>

Signed-off-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Emilia Käsper <emilia@openssl.org>

RFC 5077 section 3.3 says: If the server determines that it does not
want to include a ticket after it has included the SessionTicket
extension in the ServerHello, then it sends a zero-length ticket in the
NewSessionTicket handshake message.

Previously the client would fail upon attempting to allocate a
zero-length buffer. Now, we have the client ignore the empty ticket and
keep the existing session.

Reviewed-by: Matt Caswell <matt@openssl.org>