In the DTLS ClientHello processing the return value is stored in |ret| which
by default is -1. |ret| is only updated to a positive value once we are past
all points where we could hit an error. We wish to return 1 on success or 2
on success *and* we have validated the DTLS cookie. Previously on successful
validation of the cookie we were setting |ret| to -2, and then once we were
past all error points we set |ret = -ret|. This is non-obvious behaviour and
could be error prone. This commit tries to make this a bit more intuitive.

Reviewed-by: Andy Polyakov <appro@openssl.org>

RT#4142

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 76eba0d9

)

Resolved conflicts:
	crypto/bn/asm/x86_64-mont.pl
	crypto/bn/asm/x86_64-mont5.pl

Reviewed-by: Kurt Roeckx <kurt@openssl.org>

Reviewed-by: Andy Polyakov <appro@openssl.org>

Signed-off-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Rich Salz <rsalz@akamai.com>

GH: #466
(cherry picked from commit 434b5845)

If somewhere in SSL_new() there is a memory allocation failure, ssl3_free() can
get called with s->s3 still being NULL.

Patch also provided by Willy Tarreau <wtarreau@haproxy.com>

Signed-off-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Viktor Dukhovni <openssl-users@dukhovni.org>
(cherry picked from commit 3e7bd2ce0b16f8611298175d6dc7cb35ee06ea6d)

Backport of 0e04674e



Reviewed-by: Steve Henson <steve@openssl.org>

RT #4047, #4110, MR #1356

Reviewed-by: Rich Salz <rsalz@openssl.org>
(cherry picked from commit 63eb10a0)

Thanks to Guido Vranken <guidovranken@gmail.com> for reporting this issue.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(cherry picked from commit 158e5207)

Conflicts:
	crypto/asn1/asn1_par.c

RT#4138

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 193ed2d7)

Strict ISO confirming C compilers only define __sun

Reviewed-by: Viktor Dukhovni <openssl-users@dukhovni.org>

RT #4144, MR #1353

(cherry picked from commit 3d322188)

Original patch by Frank Morgner.

Signed-off-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Rich Salz <rsalz@akamai.com>

GH: #456
(cherry picked from commit 68db80e2)

Though the callers check the function return value and ignore the
size_t output argument on failure, it is still often not ideal to
store -1 in a size_t on error.  That might signal an unduly large
buffer.  Instead set the size_t to 0, to indicate no space.

Reviewed-by: Richard Levitte <levitte@openssl.org>

There are lots of calls to EVP functions from within libssl There were
various places where we should probably check the return value but don't.
This adds these checks.

Reviewed-by: Richard Levitte <levitte@openssl.org>

Reviewed-by: Steve Marquess <marquess@openssl.com>

Reviewed-by: Steve Marquess <marquess@openssl.com>

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit ce24d2ed)

The problem remained unnoticed so far, because it's never called by default.
You have to craft OPENSSL_ppccap environment variable to trigger the problem.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit e4693b4e)

bn/asm/s390x.S: improve performance on z196 and z13 by up to 26%. [even z10 is couple percent faster]. Triggered by RT#4128, but solves the problem by real modulo-scheduling.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(cherry picked from commit 9d0e4dc6)

It was also found that stich performs suboptimally on AMD Jaguar, hence
execution is limited to XOP-capable and Intel processors.

Reviewed-by: Kurt Roeckx <kurt@openssl.org>
(cherry picked from commit a5fd24d1)

Manual cherry-pick of 0704343f

.

Signed-off-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>

Reviewed-by: Matt Caswell <matt@openssl.org>

If a DTLS client that does not support secure renegotiation connects to an
OpenSSL DTLS server then, by default, renegotiation is disabled. If a
server application attempts to initiate a renegotiation then OpenSSL is
supposed to prevent this. However due to a discrepancy between the TLS and
DTLS code, the server sends a HelloRequest anyway in DTLS.

This is not a security concern because the handshake will still fail later
in the process when the client responds with a ClientHello.

Reviewed-by: Tim Hudson <tjh@openssl.org>

In DTLS if an IO retry occurs during writing of a fragmented ClientHello
then we can end up reseting the finish mac variables on the retry, which
causes a handshake failure. We should only reset on the first attempt not
on retries.

Thanks to BoringSSL for reporting this issue.

RT#4119

Reviewed-by: Tim Hudson <tjh@openssl.org>

The function tls1_get_curvelist() has an explicit check to see if s->cert
is NULL or not. However the check appears *after* calling the tls1_suiteb
macro which derefs s->cert. In reality s->cert can never be NULL because
it is created in SSL_new(). If the malloc fails then the SSL_new call fails
and no SSL object is created.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(cherry picked from commit 6329b609)

Conflicts:
	ssl/t1_lib.c

During work on a larger change in master a number of locations were
identified where return value checks were missing. This backports the
relevant fixes.

Reviewed-by: Richard Levitte <levitte@openssl.org>

./Configure [target] --strict-warnings -Wno-pedantic-ms-format
would not add '-pedantic' because it matches '-Wno-pedantic-ms-format',
which was added first.

Reviewed-by: Matt Caswell <matt@openssl.org>
(cherry picked from commit 6703c4ea)

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>

The various dtls1_get*_methods did not handle the DTLS_ANY_VERSION case,
so this needed to be added.

Reviewed-by: Tim Hudson <tjh@openssl.org>

EVP_SignInit_ex was missing from the NAME section of its man page so
typing "man EVP_SignInit_ex" failed to load the page.

Reviewed-by: Stephen Henson <steve@openssl.org>
(cherry picked from commit 3d866ea6)

Clarify that git format-patch output is preferred for creating patch files.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit f89ee71b)

Close GH Issue 69
Close GH PR 457/RT4113
Some other updates

By Rich Salz, Alessandro Ghedini, Steve Marquess, Collin Anderson

(manual cherry-pick of a2aaf8be and
b06935f4

)

Reviewed-by: Kurt Roeckx <kurt@openssl.org>

Signed-off-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Rich Salz <rsalz@akamai.com>

MR #1276, RT #4107

(cherry picked from commit 94b3664a)

RT#4081

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 070c2332)

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 8cf9d71a)

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 3240e7cf)

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit d900a015)

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit c2319cf9)

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 3f6c7691)

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 8acaabec)

0 is a valid file descriptor.

RT#4068

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 4428c7db)