Added two chapters: Custom Request Elements and Debug.

Online:  http://curl.haxx.se/docs/httpscripting.shtml

Date:    December 7, 2004

Date:    December 9, 2004

                The Art Of Scripting HTTP Requests Using Curl

                =============================================

  you need to replace that space with %20 etc. Failing to comply with this

  will most likely cause your data to be received wrongly and messed up.

 4.3 FILE UPLOAD POST

 4.3 File Upload POST

  Back in late 1995 they defined a new way to post data over HTTP. It was

  documented in the RFC 1867, why this method sometimes is referred to as

  a RFC1867-posting.

  Back in late 1995 they defined an additional way to post data over HTTP. It

  is documented in the RFC 1867, why this method sometimes is referred to as

  RFC1867-posting.

  This method is mainly designed to better support file uploads. A form that

  allows a user to upload a file could be written like this in HTML:

        curl -F upload=@localfilename -F press=OK [URL]

 4.4 HIDDEN FIELDS

 4.4 Hidden Fields

  A very common way for HTML based application to pass state information

  between pages is to add hidden fields to the forms. Hidden fields are

        curl -d "birthyear=1905&press=OK&person=daniel" [URL]

 4.5 FIGURE OUT WHAT A POST LOOKS LIKE

 4.5 Figure Out What A POST Looks Like

  When you're about fill in a form and send to a server by using curl instead

  of a browser, you're of course very interested in sending a POST exactly the

        curl -T uploadfile www.uploadhttp.com/receive.cgi

6. AUTHENTICATION

6. Authentication

 Authentication is the ability to tell the server your username and password

 so that it can verify that you're allowed to do the request you're doing. The

 able to watch your passwords if you pass them as plain command line

 options. There are ways to circumvent this.

7. REFERER

7. Referer

 A HTTP request may include a 'referer' field (yes it is misspelled), which

 can be used to tell from which URL the client got to this particular

        curl -e http://curl.haxx.se daniel.haxx.se

8. USER AGENT

8. User Agent

 Very similar to the referer field, all HTTP requests may set the User-Agent

 field. It names what user agent (client) that is being used. Many

        curl -A "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL]

9. REDIRECTS

9. Redirects

 When a resource is requested from a server, the reply from the server may

 include a hint about where the browser should go next to find this page, or a

 page, you can safely use -L and -d/-F together. Curl will only use POST in

 the first request, and then revert to GET in the following operations.

10. COOKIES

10. Cookies

 The way the web browsers do "client side state control" is by using

 cookies. Cookies are just names with associated contents. The cookies are

        curl https://that.secure.server.com

 11.1 CERTIFICATES

 11.1 Certificates

  In the HTTPS world, you use certificates to validate that you are the one

  you you claim to be, as an addition to normal passwords. Curl supports

        http://curl.haxx.se/docs/sslcerts.html

12. REFERENCES

12. Custom Request Elements

 Doing fancy stuff, you may need to add or change elements of a single curl

 request.

 For example, you can change the POST request to a PROPFIND and send the data

 as "Content-Type: text/xml" (instead of the default Content-Type) like this:

        curl -d "<xml>" -H "Content-Type: text/xml" -X PROPFIND url.com

 You can delete a default header by providing one without content. Like you

 can ruin the request by chopping off the Host: header:

        curl -H "Host:" http://mysite.com

 You can add headers the same way. Your server may want a "Destination:"

 header, and you can add it:

        curl -H "Destination: http://moo.com/nowhere" http://url.com

13. Debug

 Many times when you run curl on a site, you'll notice that the site doesn't

 seem to respond the same way to your curl requests as it does to your

 browser's.

 Then you need to start making your curl requests more similar to your

 browser's requests:

 * Use the --trace-ascii option to store fully detailed logs of the requests

   for easier analyzing and better understanding

 * Make sure you check for and use cookies when needed (both reading with -b

   and writing with -c)

 * Set user-agent to one like a recent popular browser does

 * Set referer like it is set by the browser

 * If you use POST, make sure you send all the fields and in the same order as

   the browser does it. (See chapter 4.5 above)

 A very good helper to make sure you do this right, is the LiveHTTPHeader tool

 that lets you view all headers you send and receive with Mozilla/Firefox

 (even when using HTTPS).

 A more raw approach is to capture the HTTP traffic on the network with tools

 such as ethereal or tcpdump and check what headers that were sent and

 received by the browser. (HTTPS makes this technique inefficient.)

14. References

 RFC 2616 is a must to read if you want in-depth understanding of the HTTP

 protocol.