Move malloc fail checks closer to malloc

Move memory allocation failure checks closer to the site of the malloc in
dgst app. Only a problem if the debug flag is set...but still should be
fixed.

Reviewed-by: Tim Hudson <tjh@openssl.org>
(cherry picked from commit be1477ad)