Add missing return value checks

The function DH_check_pub_key() was missing some return value checks in
some calls to BN functions.

RT#4278

Reviewed-by: Andy Polyakov <appro@openssl.org>
(cherry picked from commit f5a12207)