Good hygiene with size_t output argument.

Though the callers check the function return value and ignore the
size_t output argument on failure, it is still often not ideal to
store (-1) in size_t on error.  That might signal an unduly large
buffer.  Instead set the size_t to 0, to indicate no space.

Reviewed-by: Richard Levitte <levitte@openssl.org>