ispell-buffer

 This document will refer to 'the user' as the person writing the source code

 that uses libcurl. That would probably be you or someone in your position.

 What will be generally refered to as 'the program' will be the collected

 What will be generally referred to as 'the program' will be the collected

 source code that you write that is using libcurl for transfers. The program

 is outside libcurl and libcurl is outside of the program.

    When having compiled the program, you need to link your object files to

    create a single executable. For that to succeed, you need to link with

    libcurl and possibly also with other libraries that libcurl itself depends

    on. Like OpenSSL librararies, but even some standard OS libraries may be

    on. Like OpenSSL libraries, but even some standard OS libraries may be

    needed on the command line. To figure out which flags to use, once again

    the 'curl-config' tool comes to the rescue:

    curl_global_init()

 and it takes one parameter which is a bit pattern that tells libcurl what to

 intialize. Using CURL_GLOBAL_ALL will make it initialize all known internal

 initialize. Using CURL_GLOBAL_ALL will make it initialize all known internal

 sub modules, and might be a good default option. The current two bits that

 are specified are:

  CURL_GLOBAL_WIN32 which only does anything on Windows machines. When used on

  a Windows machine, it'll make libcurl intialize the win32 socket

  a Windows machine, it'll make libcurl initialize the win32 socket

  stuff. Without having that initialized properly, your program cannot use

  sockets properly. You should only do this once for each application, so if

  your program already does this or of another library in use does it, you

  should not tell libcurl to do this as well.

  CURL_GLOBAL_SSL which only does anything on libcurls compiled and built

  SSL-enabled. On these systems, this will make libcurl init OpenSSL properly

  for this application. This is only needed to do once for each application so

  if your program or another library already does this, this bit should not be

  needed.

  SSL-enabled. On these systems, this will make libcurl initialize OpenSSL

  properly for this application. This is only needed to do once for each

  application so if your program or another library already does this, this

  bit should not be needed.

 libcurl has a default protection mechanism that detects if curl_global_init()

 hasn't been called by the time curl_easy_perform() is called and if that is

Features libcurl Provides

 It is considered best-practise to determine libcurl features run-time rather

 It is considered best-practice to determine libcurl features run-time rather

 than build-time (if possible of course). By calling curl_version_info() and

 checking tout he details of the returned struct, your program can figure out

 exactly what the currently running libcurl supports.

 interface first, so please continue reading for better understanding.

 To use the easy interface, you must first create yourself an easy handle. You

 need one handle for each easy session you want to perform. Basicly, you

 need one handle for each easy session you want to perform. Basically, you

 should use one handle for every thread you plan to use for transferring. You

 must never share the same handle in multiple threads.

 set in the handle until set again to something different. Alas, multiple

 requests using the same handle will use the same options.

 Many of the informationals you set in libcurl are "strings", pointers to data

 Many of the options you set in libcurl are "strings", pointers to data

 terminated with a zero byte. Keep in mind that when you set strings with

 curl_easy_setopt(), libcurl will not copy the data. It will merely point to

 the data. You MUST make sure that the data remains available for libcurl to

    curl_easy_setopt(easyhandle, CURLOPT_URL, "http://curl.haxx.se/");

 Let's assume for a while that you want to receive data as the URL indentifies

 Let's assume for a while that you want to receive data as the URL identifies

 a remote resource you want to get here. Since you write a sort of application

 that needs this transfer, I assume that you would like to get the data passed

 to you directly instead of simply getting it passed to stdout. So, you write

Multi-threading issues

 libcurl is completely thread safe, except for two issues: signals and alarm

 handlers. Signals are needed for a SIGPIPE handler, and the alarm() syscall

 is used to catch timeouts (mostly during DNS lookup).

 handlers. Signals are needed for a SIGPIPE handler, and the alarm() Bacall

 is used to catch timeouts (mostly during ENS lookup).

 If you are accessing HTTPS or FTPS URLs in a multi-threaded manner, you are

 then of course using OpenSSL multi-threaded and it has itself a few

 requirements on this. Basicly, you need to provide one or two functions to

 requirements on this. Basilio, you need to provide one or two functions to

 allow it to function properly. For all details, see this:

   http://www.openssl.org/docs/crypto/threads.html#DESCRIPTION

 When using multiple threads you should first ignore SIGPIPE in your main

 thread and set the CURLOPT_NOSIGNAL option to TRUE for all handles.

 Everything will work fine except that timeouts are not honored during the DNS

 lookup - which you can work around by building libcurl with ares-support.

 Ares is a library that provides asynchronous name resolves. Unfortunately,

 ares does not yet support IPv6.

 For SIGPIPE info see the UNIX Socket FAQ at

 http://www.unixguide.net/network/socketfaq/2.22.shtml

 When using multiple threads you should set the CURLOPT_NOSIGNAL option to

 TRUE for all handles. Everything will work fine except that timeouts are not

 honored during the DNS lookup - which you can work around by building libcurl

 with c-ares support. c-ares is a library that provides asynchronous name

 resolves. Unfortunately, c-ares does not yet support IPv6.

 Also, note that CURLOPT_DNS_USE_GLOBAL_CACHE is not thread-safe.

 There's one golden rule when these things occur: set the CURLOPT_VERBOSE

 option to TRUE. It'll cause the library to spew out the entire protocol

 details it sends, some internal info and some received protcol data as well

 details it sends, some internal info and some received protocol data as well

 (especially when using FTP). If you're using HTTP, adding the headers in the

 received output to study is also a clever way to get a better understanding

 wht the server behaves the way it does. Include headers in the normal body

 why the server behaves the way it does. Include headers in the normal body

 output with CURLOPT_HEADER set TRUE.

 Of course there are bugs left. We need to get to know about them to be able

        curl_easy_setopt(easyhandle, CURLOPT_USERPWD, "myname:thesecret");

 Another case where name and password might be needed at times, is for those

 users who need to athenticate themselves to a proxy they use. libcurl offers

 users who need to authenticate themselves to a proxy they use. libcurl offers

 another option for this, the CURLOPT_PROXYUSERPWD. It is used quite similar

 to the CURLOPT_USERPWD option like this:

 many different ways a client can provide those credentials to the server and

 you can control what way libcurl will (attempt to) use. The default HTTP

 authentication method is called 'Basic', which is sending the name and

 password in clear-text in the HTTP request, base64-encoded. This is unsecure.

 password in clear-text in the HTTP request, base64-encoded. This is insecure.

 At the time of this writing libcurl can be built to use: Basic, Digest, NTLM,

 Negotiate, GSS-Negotiate and SPNEGO. You can tell libcurl which one to use

    curl_easy_perform(easyhandle); /* post away! */

 Simple enough, huh? Since you set the POST options with the

 CURLOPT_POSTFIELDS, this automaticly switches the handle to use POST in the

 CURLOPT_POSTFIELDS, this automatically switches the handle to use POST in the

 upcoming request.

 Ok, so what if you want to post binary data that also requires you to set the

    curl_slist_free_all(headers); /* free the header list */

 While the simple examples above cover the majority of all cases where HTTP

 POST operations are required, they don't do multipart formposts. Multipart

 POST operations are required, they don't do multi-part formposts. Multi-part

 formposts were introduced as a better way to post (possibly large) binary

 data and was first documented in the RFC1867. They're called multipart

 data and was first documented in the RFC1867. They're called multi-part

 because they're built by a chain of parts, each being a single unit. Each

 part has its own name and contents. You can in fact create and post a

 multipart formpost with the regular libcurl POST support described above, but

 multi-part formpost with the regular libcurl POST support described above, but

 that would require that you build a formpost yourself and provide to

 libcurl. To make that easier, libcurl provides curl_formadd(). Using this

 function, you add parts to the form. When you're done adding parts, you post

libcurl with C++

 There's basicly only one thing to keep in mind when using C++ instead of C

 There's basically only one thing to keep in mind when using C++ instead of C

 when interfacing libcurl:

    "The Callbacks Must Be Plain C"

 as a substitute for another".

 Proxies are exceedingly common these days. Companies often only offer

 internet access to employees through their HTTP proxies. Network clients or

 user-agents ask the proxy for docuements, the proxy does the actual request

 Internet access to employees through their HTTP proxies. Network clients or

 user-agents ask the proxy for documents, the proxy does the actual request

 and then it returns them.

 libcurl has full support for HTTP proxies, so when a given URL is wanted,

 The fact that the proxy is a HTTP proxy puts certain restrictions on what can

 actually happen. A requested URL that might not be a HTTP URL will be still

 be passed to the HTTP proxy to deliver back to libcurl. This happens

 transparantly, and an application may not need to know. I say "may", because

 transparently, and an application may not need to know. I say "may", because

 at times it is very important to understand that all operations over a HTTP

 proxy is using the HTTP protocol. For example, you can't invoke your own

 custom FTP commands or even proper FTP directory listings.

  Environment Variables

    libcurl automaticly checks and uses a set of environment variables to know

    what proxies to use for certain protocols. The names of the variables are

    following an ancient de facto standard and are built up as

    libcurl automatically checks and uses a set of environment variables to

    know what proxies to use for certain protocols. The names of the variables

    are following an ancient de facto standard and are built up as

    "[protocol]_proxy" (note the lower casing). Which makes the variable

    'http_proxy' checked for a name of a proxy to use when the input URL is

    HTTP. Following the same rule, the variable named 'ftp_proxy' is checked

    names of the variables simply allows different HTTP proxies to be used.

    The proxy environment variable contents should be in the format

    "[protocol://]machine[:port]". Where the protocol:// part is simply

    ignored if present (so http://proxy and bluerk://proxy will do the same)

    and the optional port number specifies on which port the proxy operates on

    the host. If not specified, the internal default port number will be used

    and that is most likely *not* the one you would like it to be.

    "[protocol://][user:password]machine[:port]". Where the protocol:// part

    is simply ignored if present (so http://proxy and bluerk://proxy will do

    the same) and the optional port number specifies on which port the proxy

    operates on the host. If not specified, the internal default port number

    will be used and that is most likely *not* the one you would like it to

    be.

    There are two special environment variables. 'all_proxy' is what sets

    proxy for any URL in case the protocol specific variable wasn't set, and

  SSL and Proxies

    SSL is for secure point-to-point connections. This involves strong

    encryption and similar things, which effectivly makes it impossible for a

    encryption and similar things, which effectively makes it impossible for a

    proxy to operate as a "man in between" which the proxy's task is, as

    previously discussed. Instead, the only way to have SSL work over a HTTP

    proxy is to ask the proxy to tunnel trough everything without being able

    operations over a HTTP proxy. You can in fact use things such as FTP

    upload or FTP custom commands this way.

    Again, this is often prevented by the adminstrators of proxies and is

    Again, this is often prevented by the administrators of proxies and is

    rarely allowed.

    Tell libcurl to use proxy tunneling like this:

  Proxy Auto-Config

    Netscape first came up with this. It is basicly a web page (usually using

    a .pac extension) with a javascript that when executed by the browser with

    the requested URL as input, returns information to the browser on how to

    connect to the URL. The returned information might be "DIRECT" (which

    means no proxy should be used), "PROXY host:port" (to tell the browser

    where the proxy for this particular URL is) or "SOCKS host:port" (to

    direct the brower to a SOCKS proxy).

    Netscape first came up with this. It is basically a web page (usually

    using a .pac extension) with a javascript that when executed by the

    browser with the requested URL as input, returns information to the

    browser on how to connect to the URL. The returned information might be

    "DIRECT" (which means no proxy should be used), "PROXY host:port" (to tell

    the browser where the proxy for this particular URL is) or "SOCKS

    host:port" (to direct the browser to a SOCKS proxy).

    libcurl has no means to interpret or evaluate javascript and thus it

    doesn't support this. If you get yourself in a position where you face

    - Ask your admins to stop this, for a static proxy setup or similar.

Persistancy Is The Way to Happiness

Persistence Is The Way to Happiness

 Re-cycling the same easy handle several times when doing multiple requests is

 the way to go.

 reduces network impact a lot.

 Even if the connection is dropped, all connections involving SSL to the same

 host again, will benefit from libcurl's session ID cache that drasticly

 host again, will benefit from libcurl's session ID cache that drastically

 reduces re-connection time.

 FTP connections that are kept alive saves a lot of time, as the command-

 response roundtrips are skipped, and also you don't risk getting blocked

 response round-trips are skipped, and also you don't risk getting blocked

 without permission to login again like on many FTP servers only allowing N

 persons to be logged in at the same time.

  used for the longest time. This is the default behavior.

  CURLCLOSEPOLICY_OLDEST closes the oldest connection, the one that was

  createst the longest time ago.

  created the longest time ago.

 There are, or at least were, plans to support a close policy that would call

 a user-specified callback to let the user be able to decide which connection

 to dump when this is necessary and therefor is the CURLOPT_CLOSEFUNCTION an

 existing option still today. Nothing ever uses this though and this will not

 be used within the forseeable future either.

 be used within the foreseeable future either.

 To force your upcoming request to not use an already existing connection (it

 will even close one first if there happens to be one alive to the same host

HTTP Headers Used by libcurl

 When you use libcurl to do HTTP requeests, it'll pass along a series of

 headers automaticly. It might be good for you to know and understand these

 When you use libcurl to do HTTP requests, it'll pass along a series of

 headers automatically. It might be good for you to know and understand these

 ones.

  Host

  Pragma

    "no-cache". Tells a possible proxy to not grap a copy from the cache but

    "no-cache". Tells a possible proxy to not grab a copy from the cache but

    to fetch a fresh one.

  Accept:

       headers = curl_slist_append(headers, "Accept:");

    Both replacing and cancelling internal headers should be done with careful

    Both replacing and canceling internal headers should be done with careful

    consideration and you should be aware that you may violate the HTTP

    protocol when doing so.

    chunked" when doing a non-GET HTTP operation, libcurl will switch over to

    "chunked" upload, even though the size of the data to upload might be

    known. By default, libcurl usually switches over to chunked upload

    automaticly if the upload data size is unknown.

    automatically if the upload data size is unknown.

  HTTP Version

    you want to make for example your FTP transfers to behave differently.

    Sending custom commands to a FTP server means that you need to send the

    comands exactly as the FTP server expects them (RFC959 is a good guide

    commands exactly as the FTP server expects them (RFC959 is a good guide

    here), and you can only use commands that work on the control-connection

    alone. All kinds of commands that requires data interchange and thus needs

    a data-connection must be left to libcurl's own judgement. Also be aware

    a data-connection must be left to libcurl's own judgment. Also be aware

    that libcurl will do its very best to change directory to the target

    directory before doing any transfer, so if you change directory (with CWD

    or similar) you might confuse libcurl and then it might not attempt to

    curl_easy_setopt(easyhandle, CURLOPT_COOKIE, "name1=var1; name2=var2;");

 In many cases, that is not enough. You might want to dynamicly save whatever

 cookies the remote server passes to you, and make sure those cookies are then

 use accordingly on later requests.

 In many cases, that is not enough. You might want to dynamically save

 whatever cookies the remote server passes to you, and make sure those cookies

 are then use accordingly on later requests.

 One way to do this, is to save all headers you receive in a plain file and

 when you make a request, you tell libcurl to read the previous headers to

 figure out which cookies to use. Set header file to read cookies from with

 CURLOPT_COOKIEFILE.

 The CURLOPT_COOKIEFILE option also automaticly enables the cookie parser in

 The CURLOPT_COOKIEFILE option also automatically enables the cookie parser in

 libcurl. Until the cookie parser is enabled, libcurl will not parse or

 understand incoming cookies and they will just be ignored. However, when the

 parser is enabled the cookies will be understood and the cookies will be kept

 If you rather use existing cookies that you've previously received with your

 Netscape or Mozilla browsers, you can make libcurl use that cookie file as

 input. The CURLOPT_COOKIEFILE is used for that too, as libcurl will

 automaticly find out what kind of file it is and act accordingly.

 automatically find out what kind of file it is and act accordingly.

 The perhaps most advanced cookie operation libcurl offers, is saving the

 entire internal cookie state back into a Netscape/Mozilla formatted cookie

 libcurl can either connect to the server a second time or tell the server to

 connect back to it. The first option is the default and it is also what works

 best for all the people behind firewalls, NATs or IP-masquarading setups.

 best for all the people behind firewalls, NATs or IP-masquerading setups.

 libcurl then tells the server to open up a new port and wait for a second

 connection. This is by default attempted with EPSV first, and if that doesn't

 work it tries PASV instead. (EPSV is an extension to the original FTP spec

  .netrc

    .netrc is a pretty handy file/feature that allows you to login quickly and

    automaticly to frequently visited sites. The file contains passwords in

    automatically to frequently visited sites. The file contains passwords in

    clear text and is a real security risk. In some cases, your .netrc is also

    stored in a home directory that is NFS mounted or used on another network

    based file system, so the clear text password will fly through your

    Many of the protocols libcurl supports send name and password unencrypted

    as clear text (HTTP Basic authentication, FTP, TELNET etc). It is very

    easy for anyone on your network or a network nearby yours, to just fire up

    a network analyzer tool and evesdrop on your passwords. Don't let the fact

    that HTTP uses base64 encoded passwords fool you. They may not look

    a network analyzer tool and eavesdrop on your passwords. Don't let the

    fact that HTTP uses base64 encoded passwords fool you. They may not look

    readable at a first glance, but they very easily "deciphered" by anyone

    within seconds.

  Showing What You Do

    On a related issue, be aware that even in situations like when you have

    problems with libcurl and ask somone for help, everything you reveal in

    problems with libcurl and ask someone for help, everything you reveal in

    order to get best possible help might also impose certain security related

    risks. Host names, user names, paths, operating system specifics etc (not

    to mention passwords of course) may in fact be used by intruders to gain

    additional information of a potential target.

    To avoid this problem, you must of course use your common sense. Often,

    you can just edit out the senstive data or just rearch/replace your true

    you can just edit out the sensitive data or just search/replace your true

    information with faked data.

 wants to do. Take note that libcurl does also feature some time-out code so

 we advice you to never use very long timeouts on select() before you call

 curl_multi_perform(), which thus should be called unconditionally every now

 and then even if none of its file descriptors have signalled ready. Another

 precation you should use: always call curl_multi_fdset() immediately before

 and then even if none of its file descriptors have signaled ready. Another

 precaution you should use: always call curl_multi_fdset() immediately before

 the select() call since the current set of file descriptors may change when

 calling a curl function.

Footnotes:

[1] = libcurl 7.10.3 and later have the ability to switch over to chunked

      Tranfer-Encoding in cases were HTTP uploads are done with data of an

      Transfer-Encoding in cases were HTTP uploads are done with data of an

      unknown size.

[2] = This happens on Windows machines when libcurl is built and used as a