The code is more or less copied from the stdout comparison code, maybe
some better reuse is possible.

test 1457 is adjusted to make the output actually match (by using --silent)
test 506 used <stderr> without actually needing it, so that <stderr> block is removed

Closes #3536

Option -F generates an intermediate representation of the mime structure
that is used later to create the libcurl mime structure and generate
the --libcurl statements.

Reported-by: Daniel Stenberg
Fixes #3532
Closes #3546

and bump the version in progress to 7.64.1. If we merge any "change"
before the cut-off date, we update again.

Commit 3f16990e followed-up a bug in b49652ac

 but was
inadvertently introducing a new bug in the ternary expression.

Close #3555
Reviewed-by: Daniel Stenberg <daniel@haxx.se>

There is no benefit to holding the data sharelock when freeing the
addrinfo in case it fails, so ensure releaseing it as soon as we can
rather than holding on to it. This also aligns the code with other
consumers of sharelocks.

Closes #3516
Reviewed-by: Daniel Stenberg <daniel@haxx.se>

On FreeBSD, return non-zero on error otherwise zero.

Reported-by: Marcel Raad

... and added braces to two function calls which fixes warnings if they
are replace by empty macros at build-time.

Closes #3550

The http2 code for connection checking needs a transfer to use. Make
sure a working one is set before handler->connection_check() is called.

Reported-by: jnbr on github
Fixes #3541
Closes #3547

Closes #3544

That is functions that are declared global but are not used from outside
of the file in which it is declared. Such functions should be made
static or even at times be removed.

It also verifies that all used curl_ prefixed functions are "blessed"

Closes #3538

urlapi: turn three local-only functions into statics

conncache: make conncache_find_first_connection static

multi: make detach_connnection static

connect: make getaddressinfo static

curl_ntlm_core: make hmac_md5 static

http2: make two functions static

http: make http_setup_conn static

connect: make tcpnodelay static

tests: make UNITTEST a thing to mark functions with, so they can be static for
normal builds and non-static for unit test builds

... and mark Curl_shuffle_addr accordingly.

url: make up_free static

setopt: make vsetopt static

curl_endian: make write32_le static

rtsp: make rtsp_connisdead static

warnless: remove unused functions

memdebug: remove one unused function, made another static

The build logs will be at https://cirrus-ci.com/github/curl/curl

Some tests are currently failing and so disabled for now. The SSH server
isn't starting for the SSH tests due to unsupported options used in its
config file. The DICT server also is failing on startup.

The check was added back in fa939220 before the URL parser would catch
these problems and therefore these will never trigger now.

Closes #3539

Both nits pointed out by codacy.com

Closes #3540

':' is interpreted as separator by zsh, so if used as part of the argument
or option's description it needs to be escaped.

The problem can be reproduced as follows:

 % curl --reso<TAB>
 % curl -E <TAB>

Bug: https://bugs.debian.org/921452

The current regex fails to match '<...>' arguments properly (e.g. those
with spaces in them), which causes an completion script with wrong
descriptions for some options.

Here's a diff of the generated completion script, comparing the previous
version to the one with this fix:

--- /usr/share/zsh/vendor-completions/_curl	2019-01-15 20:47:40.000000000 +0000
+++ _curl	2019-02-05 20:57:29.453349040 +0000
@@ -9,48 +9,48 @@

 _arguments -C -S \
   --happy-eyeballs-timeout-ms'[How long to wait in milliseconds for IPv6 before trying IPv4]':'<milliseconds>' \
+  --resolve'[Resolve the host+port to this address]':'<host:port:address[,address]...>' \
   {-c,--cookie-jar}'[Write cookies to <filename> after operation]':'<filename>':_files \
   {-D,--dump-header}'[Write the received headers to <filename>]':'<filename>':_files \
   {-y,--speed-time}'[Trigger '\''speed-limit'\'' abort after this time]':'<seconds>' \
   --proxy-cacert'[CA certificate to verify peer against for proxy]':'<file>':_files \
-  --tls13-ciphers'[of TLS 1.3 ciphersuites> TLS 1.3 cipher suites to use]':'<list' \
+  --tls13-ciphers'[TLS 1.3 cipher suites to use]':'<list of TLS 1.3 ciphersuites>' \
   {-E,--cert}'[Client certificate file and password]':'<certificate[:password]>' \
   --libcurl'[Dump libcurl equivalent code of this command line]':'<file>':_files \
   --proxy-capath'[CA directory to verify peer against for proxy]':'<dir>':_files \
-  --proxy-negotiate'[HTTP Negotiate (SPNEGO) authentication on the proxy]':'Use' \
   --proxy-pinnedpubkey'[FILE/HASHES public key to verify proxy with]':'<hashes>' \
   --crlfile'[Get a CRL list in PEM format from the given file]':'<file>':_files \
-  --proxy-insecure'[HTTPS proxy connections without verifying the proxy]':'Do' \
-  --proxy-ssl-allow-beast'[security flaw for interop for HTTPS proxy]':'Allow' \
+  --proxy-negotiate'[Use HTTP Negotiate (SPNEGO) authentication on the proxy]' \
   --abstract-unix-socket'[Connect via abstract Unix domain socket]':'<path>' \
   --pinnedpubkey'[FILE/HASHES Public key to verify peer against]':'<hashes>' \
+  --proxy-insecure'[Do HTTPS proxy connections without verifying the proxy]' \
   --proxy-pass'[Pass phrase for the private key for HTTPS proxy]':'<phrase>' \
+  --proxy-ssl-allow-beast'[Allow security flaw for interop for HTTPS proxy]' \
   {-p,--proxytunnel}'[Operate through an HTTP proxy tunnel (using CONNECT)]' \
   --socks5-hostname'[SOCKS5 proxy, pass host name to proxy]':'<host[:port]>' \
   --proto-default'[Use PROTOCOL for any URL missing a scheme]':'<protocol>' \
-  --proxy-tls13-ciphers'[list> TLS 1.3 proxy cipher suites]':'<ciphersuite' \
+  --proxy-tls13-ciphers'[TLS 1.3 proxy cipher suites]':'<ciphersuite list>' \
   --socks5-gssapi-service'[SOCKS5 proxy service name for GSS-API]':'<name>' \
   --ftp-alternative-to-user'[String to replace USER \[name\]]':'<command>' \
-  --ftp-ssl-control'[SSL/TLS for FTP login, clear for transfer]':'Require' \
   {-T,--upload-file}'[Transfer local FILE to destination]':'<file>':_files \
   --local-port'[Force use of RANGE for local port numbers]':'<num/range>' \
   --proxy-tlsauthtype'[TLS authentication type for HTTPS proxy]':'<type>' \
   {-R,--remote-time}'[Set the remote file'\''s time on the local output]' \
-  --retry-connrefused'[on connection refused (use with --retry)]':'Retry' \
-  --suppress-connect-headers'[proxy CONNECT response headers]':'Suppress' \
-  {-j,--junk-session-cookies}'[session cookies read from file]':'Ignore' \
-  --location-trusted'[--location, and send auth to other hosts]':'Like' \
+  --ftp-ssl-control'[Require SSL/TLS for FTP login, clear for transfer]' \
   --proxy-cert-type'[Client certificate type for HTTPS proxy]':'<type>' \
   {-O,--remote-name}'[Write output to a file named as the remote file]' \
+  --retry-connrefused'[Retry on connection refused (use with --retry)]' \
+  --suppress-connect-headers'[Suppress proxy CONNECT response headers]' \
   --trace-ascii'[Like --trace, but without hex output]':'<file>':_files \
   --connect-timeout'[Maximum time allowed for connection]':'<seconds>' \
   --expect100-timeout'[How long to wait for 100-continue]':'<seconds>' \
   {-g,--globoff}'[Disable URL sequences and ranges using {} and \[\]]' \
+  {-j,--junk-session-cookies}'[Ignore session cookies read from file]' \
   {-m,--max-time}'[Maximum time allowed for the transfer]':'<seconds>' \
   --dns-ipv4-addr'[IPv4 address to use for DNS requests]':'<address>' \
   --dns-ipv6-addr'[IPv6 address to use for DNS requests]':'<address>' \
-  --ignore-content-length'[the size of the remote resource]':'Ignore' \
   {-k,--insecure}'[Allow insecure server connections when using SSL]' \
+  --location-trusted'[Like --location, and send auth to other hosts]' \
   --mail-auth'[Originator address of the original email]':'<address>' \
   --noproxy'[List of hosts which do not use proxy]':'<no-proxy-list>' \
   --proto-redir'[Enable/disable PROTOCOLS on redirect]':'<protocols>' \
@@ -62,18 +62,19 @@
   --socks5-basic'[Enable username/password auth for SOCKS5 proxies]' \
   --cacert'[CA certificate to verify peer against]':'<file>':_files \
   {-H,--header}'[Pass custom header(s) to server]':'<header/@file>' \
+  --ignore-content-length'[Ignore the size of the remote resource]' \
   {-i,--include}'[Include protocol response headers in the output]' \
   --proxy-header'[Pass custom header(s) to proxy]':'<header/@file>' \
   --unix-socket'[Connect through this Unix domain socket]':'<path>' \
   {-w,--write-out}'[Use output FORMAT after completion]':'<format>' \
-  --http2-prior-knowledge'[HTTP 2 without HTTP/1.1 Upgrade]':'Use' \
   {-o,--output}'[Write to file instead of stdout]':'<file>':_files \
-  {-J,--remote-header-name}'[the header-provided filename]':'Use' \
+  --preproxy'[\[protocol://\]host\[:port\] Use this proxy first]' \
   --socks4a'[SOCKS4a proxy on given host + port]':'<host[:port]>' \
   {-Y,--speed-limit}'[Stop transfers slower than this]':'<speed>' \
   {-z,--time-cond}'[Transfer based on a time condition]':'<time>' \
   --capath'[CA directory to verify peer against]':'<dir>':_files \
   {-f,--fail}'[Fail silently (no output at all) on HTTP errors]' \
+  --http2-prior-knowledge'[Use HTTP 2 without HTTP/1.1 Upgrade]' \
   --proxy-tlspassword'[TLS password for HTTPS proxy]':'<string>' \
   {-U,--proxy-user}'[Proxy user and password]':'<user:password>' \
   --proxy1.0'[Use HTTP/1.0 proxy on given port]':'<host[:port]>' \
@@ -81,52 +82,49 @@
   {-A,--user-agent}'[Send User-Agent <name> to server]':'<name>' \
   --egd-file'[EGD socket path for random data]':'<file>':_files \
   --fail-early'[Fail on first transfer error, do not continue]' \
-  --haproxy-protocol'[HAProxy PROXY protocol v1 header]':'Send' \
-  --preproxy'[Use this proxy first]':'[protocol://]host[:port]' \
+  {-J,--remote-header-name}'[Use the header-provided filename]' \
   --retry-max-time'[Retry only within this period]':'<seconds>' \
   --socks4'[SOCKS4 proxy on given host + port]':'<host[:port]>' \
   --socks5'[SOCKS5 proxy on given host + port]':'<host[:port]>' \
-  --socks5-gssapi-nec'[with NEC SOCKS5 server]':'Compatibility' \
-  --ssl-allow-beast'[security flaw to improve interop]':'Allow' \
   --cert-status'[Verify the status of the server certificate]' \
-  --ftp-create-dirs'[the remote dirs if not present]':'Create' \
   {-:,--next}'[Make next URL use its separate set of options]' \
   --proxy-key-type'[Private key file type for proxy]':'<type>' \
-  --remote-name-all'[the remote file name for all URLs]':'Use' \
   {-X,--request}'[Specify request command to use]':'<command>' \
   --retry'[Retry request if transient problems occur]':'<num>' \
-  --ssl-no-revoke'[cert revocation checks (WinSSL)]':'Disable' \
   --cert-type'[Certificate file type (DER/PEM/ENG)]':'<type>' \
   --connect-to'[Connect to host]':'<HOST1:PORT1:HOST2:PORT2>' \
   --create-dirs'[Create necessary local directory hierarchy]' \
+  --haproxy-protocol'[Send HAProxy PROXY protocol v1 header]' \
   --max-redirs'[Maximum number of redirects allowed]':'<num>' \
   {-n,--netrc}'[Must read .netrc for user name and password]' \
+  {-x,--proxy}'[\[protocol://\]host\[:port\] Use this proxy]' \
   --proxy-crlfile'[Set a CRL list for proxy]':'<file>':_files \
   --sasl-ir'[Enable initial response in SASL authentication]' \
-  --socks5-gssapi'[GSS-API auth for SOCKS5 proxies]':'Enable' \
+  --socks5-gssapi-nec'[Compatibility with NEC SOCKS5 server]' \
+  --ssl-allow-beast'[Allow security flaw to improve interop]' \
+  --ftp-create-dirs'[Create the remote dirs if not present]' \
   --interface'[Use network INTERFACE (or address)]':'<name>' \
   --key-type'[Private key file type (DER/PEM/ENG)]':'<type>' \
   --netrc-file'[Specify FILE for netrc]':'<filename>':_files \
   {-N,--no-buffer}'[Disable buffering of the output stream]' \
   --proxy-service-name'[SPNEGO proxy service name]':'<name>' \
-  --styled-output'[styled output for HTTP headers]':'Enable' \
+  --remote-name-all'[Use the remote file name for all URLs]' \
+  --ssl-no-revoke'[Disable cert revocation checks (WinSSL)]' \
   --max-filesize'[Maximum file size to download]':'<bytes>' \
   --negotiate'[Use HTTP Negotiate (SPNEGO) authentication]' \
   --no-keepalive'[Disable TCP keepalive on the connection]' \
   {-#,--progress-bar}'[Display transfer progress as a bar]' \
-  {-x,--proxy}'[Use this proxy]':'[protocol://]host[:port]' \
-  --proxy-anyauth'[any proxy authentication method]':'Pick' \
   {-Q,--quote}'[Send command(s) to server before transfer]' \
-  --request-target'[the target for this request]':'Specify' \
+  --socks5-gssapi'[Enable GSS-API auth for SOCKS5 proxies]' \
   {-u,--user}'[Server user and password]':'<user:password>' \
   {-K,--config}'[Read config from a file]':'<file>':_files \
   {-C,--continue-at}'[Resumed transfer offset]':'<offset>' \
   --data-raw'[HTTP POST data, '\''@'\'' allowed]':'<data>' \
-  --disallow-username-in-url'[username in url]':'Disallow' \
   --krb'[Enable Kerberos with security <level>]':'<level>' \
   --proxy-ciphers'[SSL ciphers to use for proxy]':'<list>' \
   --proxy-digest'[Use Digest authentication on the proxy]' \
   --proxy-tlsuser'[TLS username for HTTPS proxy]':'<name>' \
+  --styled-output'[Enable styled output for HTTP headers]' \
   {-b,--cookie}'[Send cookies from string/file]':'<data>' \
   --data-urlencode'[HTTP POST data url encoded]':'<data>' \
   --delegation'[GSS-API delegation permission]':'<LEVEL>' \
@@ -134,7 +132,10 @@
   --post301'[Do not switch to GET after following a 301]' \
   --post302'[Do not switch to GET after following a 302]' \
   --post303'[Do not switch to GET after following a 303]' \
+  --proxy-anyauth'[Pick any proxy authentication method]' \
+  --request-target'[Specify the target for this request]' \
   --trace-time'[Add time stamps to trace/verbose output]' \
+  --disallow-username-in-url'[Disallow username in url]' \
   --dns-servers'[DNS server addrs to use]':'<addresses>' \
   {-G,--get}'[Put the post data in the URL and use GET]' \
   --limit-rate'[Limit transfer speed to RATE]':'<speed>' \
@@ -148,21 +149,21 @@
   --metalink'[Process given URLs as metalink XML file]' \
   --tr-encoding'[Request compressed transfer encoding]' \
   --xattr'[Store metadata in extended file attributes]' \
-  --ftp-skip-pasv-ip'[the IP address for PASV]':'Skip' \
   --pass'[Pass phrase for the private key]':'<phrase>' \
   --proxy-ntlm'[Use NTLM authentication on the proxy]' \
   {-S,--show-error}'[Show error even when -s is used]' \
-  --ciphers'[of ciphers> SSL ciphers to use]':'<list' \
+  --ciphers'[SSL ciphers to use]':'<list of ciphers>' \
   --form-string'[Specify multipart MIME data]':'<name=string>' \
   --login-options'[Server login options]':'<options>' \
   --tftp-blksize'[Set TFTP BLKSIZE option]':'<value>' \
-  --tftp-no-options'[not send any TFTP options]':'Do' \
   {-v,--verbose}'[Make the operation more talkative]' \
+  --ftp-skip-pasv-ip'[Skip the IP address for PASV]' \
   --proxy-key'[Private key for HTTPS proxy]':'<key>' \
   {-F,--form}'[Specify multipart MIME data]':'<name=content>' \
   --mail-from'[Mail from this address]':'<address>' \
   --oauth2-bearer'[OAuth 2 Bearer Token]':'<token>' \
   --proto'[Enable/disable PROTOCOLS]':'<protocols>' \
+  --tftp-no-options'[Do not send any TFTP options]' \
   --tlsauthtype'[TLS authentication type]':'<type>' \
   --doh-url'[Resolve host names over DOH]':'<URL>' \
   --no-sessionid'[Disable SSL session-ID reusing]' \
@@ -173,14 +174,13 @@
   --ftp-ssl-ccc'[Send CCC after authenticating]' \
   {-4,--ipv4}'[Resolve names to IPv4 addresses]' \
   {-6,--ipv6}'[Resolve names to IPv6 addresses]' \
-  --netrc-optional'[either .netrc or URL]':'Use' \
   --service-name'[SPNEGO service name]':'<name>' \
   {-V,--version}'[Show version number and quit]' \
   --data-ascii'[HTTP POST ASCII data]':'<data>' \
   --ftp-account'[Account data string]':'<data>' \
-  --compressed-ssh'[SSH compression]':'Enable' \
   --disable-eprt'[Inhibit using EPRT or LPRT]' \
   --ftp-method'[Control CWD usage]':'<method>' \
+  --netrc-optional'[Use either .netrc or URL]' \
   --pubkey'[SSH Public key file name]':'<key>' \
   --raw'[Do HTTP "raw"; no transfer decoding]' \
   --anyauth'[Pick any authentication method]' \
@@ -189,6 +189,7 @@
   --no-alpn'[Disable the ALPN TLS extension]' \
   --tcp-nodelay'[Use the TCP_NODELAY option]' \
   {-B,--use-ascii}'[Use ASCII/text transfer]' \
+  --compressed-ssh'[Enable SSH compression]' \
   --digest'[Use HTTP Digest Authentication]' \
   --proxy-tlsv1'[Use TLSv1 for HTTPS proxy]' \
   --engine'[Crypto engine to use]':'<name>' \

Use long for CURLOPT_HTTP09_ALLOWED to fix the following warning:
tool_operate.c: In function 'operate_do':
../include/curl/typecheck-gcc.h:47:9: error: call to
'_curl_easy_setopt_err_long' declared with attribute warning:
curl_easy_setopt expects a long argument for this option [-Werror]

Closes https://github.com/curl/curl/pull/3534

- Fix potential crashes in schannel shutdown.

Ensure any TLS shutdown messages are sent before removing the
association between the connection and the easy handle. Reverts
@bagder's previous partial fix for #3412.

Fixes https://github.com/curl/curl/issues/3412
Fixes https://github.com/curl/curl/issues/3505
Closes https://github.com/curl/curl/pull/3531

The Kerberos subsection was mistakenly a subsubsection under FTP, and
the curlx subsection was missing an anchor for the TOC link.

Closes #3529
Reviewed-by: Daniel Stenberg <daniel@haxx.se>

Commit 7a09b52c

 introduced support
for the draft-ietf-httpbis-cookie-alone-01 cookie draft, and while
the entry was removed from the TODO it was mistakenly left here.
Fix by removing and rewording the entry slightly.

Closes #3530
Reviewed-by: Daniel Stenberg <daniel@haxx.se>

Fix grammatical errors making the document read better. Also fixes
a typo.

Closes #3525
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>

Fixes #3518
Closes #3522

- Use explicit include opt for perl calls.

Prior to this change some scripts couldn't find their dependencies.

At the top, perl is called using with the "-Isrcdir" option, and it
works:

https://github.com/curl/curl/blob/curl-7_63_0/tests/runtests.pl#L183

But on line 3868, that option is omitted. This caused problems for me,
as the symbol-scan.pl script in particular couldn't find its
dependencies properly:

https://github.com/curl/curl/blob/curl-7_63_0/tests/runtests.pl#L3868

This patch fixes that oversight by making calls to perl sub-shells
uniform.

Closes https://github.com/curl/curl/pull/3496

If the incoming len 5, but the buffer does not have a termination
after 5 bytes, the strtol() call may keep reading through the line
buffer until is exceeds its boundary. Fix by ensuring that we are
using a bounded read with a temporary buffer on the stack.

Bug: https://curl.haxx.se/docs/CVE-2019-3823.html
Reported-by: Brian Carpenter (Geeknik Labs)
CVE-2019-3823

Bug: https://curl.haxx.se/docs/CVE-2019-3822.html
Reported-by: Wenxiang Qian
CVE-2019-3822

Bug: https://curl.haxx.se/docs/CVE-2018-16890.html
Reported-by: Wenxiang Qian
CVE-2018-16890

Attempt to add support for Secure Channel binding when negotiate
authentication is used. The problem to solve is that by default IIS
accepts channel binding and curl doesn't utilise them. The result was a
401 response. Scope affects only the Schannel(winssl)-SSPI combination.

Fixes https://github.com/curl/curl/issues/3503
Closes https://github.com/curl/curl/pull/3509

Stick to "Schannel" everywhere. The configure option --with-winssl is
kept to allow existing builds to work but --with-schannel is added as an
alias.

Closes #3504

To make sure Curl_timeleft() also thinks the timeout has been reached
when one of the EXPIRE_*TIMEOUTs expires.

Bug: https://curl.haxx.se/mail/lib-2019-01/0073.html
Reported-by: Zhao Yisha
Closes #3501

Use an ephemeral port number here; previously the example had 8080
which could be confusing as the common web server port number might
be misinterpreted as suggesting this option affects the remote port.

URL: https://curl.haxx.se/mail/lib-2019-01/0084.html
Closes #3513

A backslash should be escaped in Roff / Troff.

By default WinSSL selects and send a client certificate automatically,
but for privacy and consistency we should offer an option to disable the
default auto-send behavior.

Reported-by: Jeroen Ooms

Closes https://github.com/curl/curl/issues/2262

mbedTLS doesn't have a sigpipe management. If a write/read occurs when
the remote closes the socket, the signal is raised and kills the
application.  Use the curl mecanisms fix this behavior.

Signed-off-by: Jeremie Rapin <j.rapin@overkiz.com>

Closes #3502