TheArtOfHttpScripting: grammar

 address and that's what curl will communicate with. Alternatively you specify

 address and that's what curl will communicate with. Alternatively you specify

 the IP address directly in the URL instead of a name.

 the IP address directly in the URL instead of a name.

 For development and other trying out situation, you can point out a different

 For development and other trying out situations, you can point to a different

 IP address for a host name than what would otherwise be used, by using curl's

 IP address for a host name than what would otherwise be used, by using curl's

 --resolve option:

 --resolve option:

 2.3 Port number

 2.3 Port number

 Each protocol curl supports operate on a default port number, be it over TCP

 Each protocol curl supports operates on a default port number, be it over TCP

 or in some cases UDP. Normally you don't have to take that into

 or in some cases UDP. Normally you don't have to take that into

 consideration, but at times you run test servers on other ports or

 consideration, but at times you run test servers on other ports or

 similar. Then you can specify the port number in the URL with a colon and a

 similar. Then you can specify the port number in the URL with a colon and a

 The port number you specify in the URL is the number that the server uses to

 The port number you specify in the URL is the number that the server uses to

 offer its services. Sometimes you may use a local proxy, and then you may

 offer its services. Sometimes you may use a local proxy, and then you may

 need to specify that proxy's port number separate on what curl needs to

 need to specify that proxy's port number separately for what curl needs to

 connect to locally. Like when using a HTTP proxy on port 4321:

 connect to locally. Like when using a HTTP proxy on port 4321:

      curl --proxy http://proxy.example.org:4321 http://remote.example.org/

      curl --proxy http://proxy.example.org:4321 http://remote.example.org/

 2.4 User name and password

 2.4 User name and password

 Some services are setup to require HTTP authentication and then you need to

 Some services are setup to require HTTP authentication and then you need to

 provide name and password which then is transferred to the remote site in

 provide name and password which is then transferred to the remote site in

 various ways depending on the exact authentication protocol used.

 various ways depending on the exact authentication protocol used.

 You can opt to either insert the user and password in the URL or you can

 You can opt to either insert the user and password in the URL or you can

 3.1 GET

 3.1 GET

 The simplest and most common request/operation made using HTTP is to get a

 The simplest and most common request/operation made using HTTP is to GET a

 URL. The URL could itself refer to a web page, an image or a file. The client

 URL. The URL could itself refer to a web page, an image or a file. The client

 issues a GET request to the server and receives the document it asked for.

 issues a GET request to the server and receives the document it asked for.

 If you issue the command line

 If you issue the command line

 4.1 Forms explained

 4.1 Forms explained

 Forms are the general way a web site can present a HTML page with fields for

 Forms are the general way a web site can present a HTML page with fields for

 the user to enter data in, and then press some kind of 'OK' or 'submit'

 the user to enter data in, and then press some kind of 'OK' or 'Submit'

 button to get that data sent to the server. The server then typically uses

 button to get that data sent to the server. The server then typically uses

 the posted data to decide how to act. Like using the entered words to search

 the posted data to decide how to act. Like using the entered words to search

 in a database, or to add the info in a bug track system, display the entered

 in a database, or to add the info in a bug tracking system, display the entered

 address on a map or using the info as a login-prompt verifying that the user

 address on a map or using the info as a login-prompt verifying that the user

 is allowed to see what it is about to see.

 is allowed to see what it is about to see.

 Of course there has to be some kind of program in the server end to receive

 Of course there has to be some kind of program on the server end to receive

 the data you send. You cannot just invent something out of the air.

 the data you send. You cannot just invent something out of the air.

 4.2 GET

 4.2 GET

 4.5 Hidden Fields

 4.5 Hidden Fields

  A very common way for HTML based application to pass state information

  A very common way for HTML based applications to pass state information

  between pages is to add hidden fields to the forms. Hidden fields are

  between pages is to add hidden fields to the forms. Hidden fields are

  already filled in, they aren't displayed to the user and they get passed

  already filled in, they aren't displayed to the user and they get passed

  along just as all the other fields.

  along just as all the other fields.

      <input type=submit name="press" value="OK">

      <input type=submit name="press" value="OK">

    </form>

    </form>

  To post this with curl, you won't have to think about if the fields are

  To POST this with curl, you won't have to think about if the fields are

  hidden or not. To curl they're all the same:

  hidden or not. To curl they're all the same:

        curl --data "birthyear=1905&press=OK&person=daniel" [URL]

        curl --data "birthyear=1905&press=OK&person=daniel" [URL]

 5.1 PUT

 5.1 PUT

 The perhaps best way to upload data to a HTTP server is to use PUT. Then

 Perhaps the best way to upload data to a HTTP server is to use PUT. Then

 again, this of course requires that someone put a program or script on the

 again, this of course requires that someone put a program or script on the

 server end that knows how to receive a HTTP PUT stream.

 server end that knows how to receive a HTTP PUT stream.

 If your proxy requires the authentication to be done using the NTLM method,

 If your proxy requires the authentication to be done using the NTLM method,

 use --proxy-ntlm, if it requires Digest use --proxy-digest.

 use --proxy-ntlm, if it requires Digest use --proxy-digest.

 If you use any one these user+password options but leave out the password

 If you use any one of these user+password options but leave out the password

 part, curl will prompt for the password interactively.

 part, curl will prompt for the password interactively.

 6.4 Hiding credentials

 6.4 Hiding credentials

 to redirect is Location:.

 to redirect is Location:.

 Curl does not follow Location: headers by default, but will simply display

 Curl does not follow Location: headers by default, but will simply display

 such pages in the same manner it display all HTTP replies. It does however

 such pages in the same manner it displays all HTTP replies. It does however

 feature an option that will make it attempt to follow the Location: pointers.

 feature an option that will make it attempt to follow the Location: pointers.

 To tell curl to follow a Location:

 To tell curl to follow a Location:

 (Take note that the --cookie-jar option described below is a better way to

 (Take note that the --cookie-jar option described below is a better way to

 store cookies.)

 store cookies.)

 Curl has a full blown cookie parsing engine built-in that comes to use if you

 Curl has a full blown cookie parsing engine built-in that comes in use if you

 want to reconnect to a server and use cookies that were stored from a

 want to reconnect to a server and use cookies that were stored from a

 previous connection (or hand-crafted manually to fool the server into

 previous connection (or hand-crafted manually to fool the server into

 believing you had a previous connection). To use previously stored cookies,

 believing you had a previous connection). To use previously stored cookies,

 10.1 HTTPS is HTTP secure

 10.1 HTTPS is HTTP secure

 There are a few ways to do secure HTTP transfers. The by far most common

 There are a few ways to do secure HTTP transfers. By far the most common

 protocol for doing this is what is generally known as HTTPS, HTTP over

 protocol for doing this is what is generally known as HTTPS, HTTP over

 SSL. SSL encrypts all the data that is sent and received over the network and

 SSL. SSL encrypts all the data that is sent and received over the network and

 thus makes it harder for attackers to spy on sensitive information.

 thus makes it harder for attackers to spy on sensitive information.

 12.1 Some login tricks

 12.1 Some login tricks

 While not strictly just HTTP related, it still cause a lot of people problems

 While not strictly just HTTP related, it still causes a lot of people problems

 so here's the executive run-down of how the vast majority of all login forms

 so here's the executive run-down of how the vast majority of all login forms

 work and how to login to them using curl.

 work and how to login to them using curl.

 make sure you got there through their login page) so you should make a habit

 make sure you got there through their login page) so you should make a habit

 of first getting the login-form page to capture the cookies set there.

 of first getting the login-form page to capture the cookies set there.

 Some web-based login systems features various amounts of javascript, and

 Some web-based login systems feature various amounts of javascript, and

 sometimes they use such code to set or modify cookie contents. Possibly they

 sometimes they use such code to set or modify cookie contents. Possibly they

 do that to prevent programmed logins, like this manual describes how to...

 do that to prevent programmed logins, like this manual describes how to...

 Anyway, if reading the code isn't enough to let you repeat the behavior

 Anyway, if reading the code isn't enough to let you repeat the behavior