In the DTLS ClientHello processing the return value is stored in |ret| which
by default is -1. We wish to return 1 on success or 2 on success *and* we
have validated the DTLS cookie. Previously on successful validation of the
cookie we were setting |ret| to 2. Unfortunately if we later encounter an
error then we can end up returning a successful (positive) return code from
the function because we already set |ret| to a positive value.

This does not appear to have a security consequence because the handshake
just fails at a later point.

Reviewed-by: Andy Polyakov <appro@openssl.org>

Reviewed-by: Andy Polyakov <appro@openssl.org>
(cherry picked from commit ed024935)

Signed-off-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Rich Salz <rsalz@akamai.com>

GH: #466
(cherry picked from commit 434b5845)

If somewhere in SSL_new() there is a memory allocation failure, ssl3_free() can
get called with s->s3 still being NULL.

Patch also provided by Willy Tarreau <wtarreau@haproxy.com>

Signed-off-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Viktor Dukhovni <openssl-users@dukhovni.org>
(cherry picked from commit 3e7bd2ce0b16f8611298175d6dc7cb35ee06ea6d)

Backport of 0e04674e



Reviewed-by: Steve Henson <steve@openssl.org>

RT #4047, #4110, MR #1356

Reviewed-by: Rich Salz <rsalz@openssl.org>
(cherry picked from commit 63eb10a0)

Thanks to Guido Vranken <guidovranken@gmail.com> for reporting this issue.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(cherry picked from commit 158e5207)

Conflicts:
	crypto/asn1/asn1_par.c

Strict ISO confirming C compilers only define __sun

Reviewed-by: Viktor Dukhovni <openssl-users@dukhovni.org>

RT #4144, MR #1353

(cherry picked from commit 3d322188)

Original patch by Frank Morgner.

Signed-off-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Rich Salz <rsalz@akamai.com>

GH: #456
(cherry picked from commit 68db80e2)

There are lots of calls to EVP functions from within libssl There were
various places where we should probably check the return value but don't.
This adds these checks.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 56d91346)

Conflicts:
	ssl/s3_enc.c
	ssl/s3_srvr.c

Reviewed-by: Steve Marquess <marquess@openssl.com>

Signed-off-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>
(cherry picked from commit f236ef27)

If a DTLS client that does not support secure renegotiation connects to an
OpenSSL DTLS server then, by default, renegotiation is disabled. If a
server application attempts to initiate a renegotiation then OpenSSL is
supposed to prevent this. However due to a discrepancy between the TLS and
DTLS code, the server sends a HelloRequest anyway in DTLS.

This is not a security concern because the handshake will still fail later
in the process when the client responds with a ClientHello.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(cherry picked from commit d40ec4ab)

In DTLS if an IO retry occurs during writing of a fragmented ClientHello
then we can end up reseting the finish mac variables on the retry, which
causes a handshake failure. We should only reset on the first attempt not
on retries.

Thanks to BoringSSL for reporting this issue.

RT#4119

Reviewed-by: Tim Hudson <tjh@openssl.org>
(cherry picked from commit 15a7164e)

During work on a larger change in master a number of locations were
identified where return value checks were missing. This backports the
relevant fixes.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 903738ac)

Conflicts:
	crypto/cms/cms_sd.c

./Configure [target] --strict-warnings -Wno-pedantic-ms-format
would not add '-pedantic' because it matches '-Wno-pedantic-ms-format',
which was added first.

Reviewed-by: Matt Caswell <matt@openssl.org>
(cherry picked from commit 6703c4ea)

EVP_SignInit_ex was missing from the NAME section of its man page so
typing "man EVP_SignInit_ex" failed to load the page.

Reviewed-by: Stephen Henson <steve@openssl.org>
(cherry picked from commit 3d866ea6)

Clarify that git format-patch output is preferred for creating patch files.

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit f89ee71b)

Close GH Issue 69
Close GH PR 457/RT4113
Some other updates

By Rich Salz, Alessandro Ghedini, Steve Marquess, Collin Anderson

(manual cherry-pick of a2aaf8be and
b06935f4

)

Reviewed-by: Kurt Roeckx <kurt@openssl.org>

Signed-off-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Rich Salz <rsalz@akamai.com>

MR #1276, RT #4107

(cherry picked from commit 94b3664a)

RT#4081

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 070c2332)

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 8cf9d71a)

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 3240e7cf)

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit d900a015)

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit c2319cf9)

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 3f6c7691)

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 8acaabec)

0 is a valid file descriptor.

RT#4068

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 4428c7db)

Previous language was unclear.  New language isn't pretty but I believe
it is more accurate.

Signed-off-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Ben Laurie <ben@openssl.org>
Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit 8cbb048c)

PR#4079

Reviewed-by: Tim Hudson <tjh@openssl.org>
(cherry picked from commit c69ce935)

Conflicts:
	crypto/evp/e_des3.c

In X509_cmp, if cert digest is equal, look at DER of the
signed part.  This is what master and 1.0.2 already do.

Reviewed-by: Dr. Stephen Henson <steve@openssl.org>

Add explicit linux-clang targets
Add --strict-warnings support for clang
Disable mingw debug builds

Signed-off-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>

The function int_rsa_verify is an internal function used for verifying an
RSA signature. It takes an argument |dtype| which indicates the digest type
that was used. Dependant on that digest type the processing of the
signature data will vary. In particular if |dtype == NID_mdc2| and the
signature data is a bare OCTETSTRING then it is treated differently to the
default case where the signature data is treated as a DigestInfo (X509_SIG).

Due to a missing "else" keyword the logic actually correctly processes the
OCTETSTRING format signature first, and then attempts to continue and
process it as DigestInfo. This will invariably fail because we already know
that it is a bare OCTETSTRING.

This failure doesn't actualy make a real difference because it ends up at
the |err| label regardless and still returns a "success" result. This patch
just cleans things up to make it look a bit more sane.

RT#4076

Reviewed-by: Richard Levitte <levitte@openssl.org>
(cherry picked from commit dffe5109)

Reviewed-by: Matt Caswell <matt@openssl.org>
(cherry picked from commit 5850cc75)

Signed-off-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Rich Salz <rsalz@openssl.org>
MR #1231

(cherry picked from commit 99c20333)

BN_with_flags() will read the dest->flags to keep the BN_FLG_MALLOCED but
overwrites everything else.

Signed-off-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Rich Salz <rsalz@openssl.org>
MR #1231

(cherry picked from commit f92768e6)

Reviewed-by: Rich Salz <rsalz@openssl.org>
(cherry picked from commit e58c4d3c)

Some extension handling functions were passing in a pointer to the start
of the data, plus the length in order to calculate the end, rather than
just passing in the end to start with. This change makes things a little
more readable.

Reviewed-by: Emilia Käsper <emilia@openssl.org>

Conflicts:
	ssl/s3_srvr.c
	ssl/ssl_locl.h
	ssl/t1_lib.c

RT#4069

Reviewed-by: Emilia Käsper <emilia@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>

Thanks to David Benjamin <davidben@google.com> for pointing them out.

Reviewed-by: Steve Henson <steve@openssl.org>
MR #1198

(cherry picked from commit 605236f6)