Fix dh_pub_encode

The return value from ASN1_STRING_new() was not being checked which could
lead to a NULL deref in the event of a malloc failure. Also fixed a mem
leak in the error path.

Reviewed-by: Rich Salz <rsalz@openssl.org>
(cherry picked from commit 6aa8dab2)