add malloc fail check & fix memory leak

Signed-off-by: Hongze Zhu <hongze.zhu@gmail.com>

Reviewed-by: Rich Salz <rsalz@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>