ftp: fix incorrect out-of-memory code in Curl_pretransfer

- Return value type must match function type.

s/CURLM_OUT_OF_MEMORY/CURLE_OUT_OF_MEMORY/

Caught by Travis CI