check bn_new return value

Slightly modified from the original PR.
Signed-off-by: Rich Salz <rsalz@akamai.com>
Reviewed-by: Richard Levitte <levitte@openssl.org>

(cherry picked from commit a7e974c7)