Remove an OPENSSL_assert() and replace with a soft assert and check

Following on from CVE-2017-3733, this removes the OPENSSL_assert() check
that failed and replaces it with a soft assert, and an explicit check of
value with an error return if it fails.

Reviewed-by: Richard Levitte <levitte@openssl.org>