tool: Fixed incorrect return code if password prompting runs out of memory

Due to the changes in commit 3c929ff9 and lack of subsequent
updates, curl could return a CURLE_FTP_ACCEPT_FAILED error if
checkpasswd() ran out of memory in versions 7.33.0 and 7.34.0.

Updated the function declaration and return code to return
CURLE_OUT_OF_MEMORY and CURLE_OK where appropriate.