May 5, 2021

9 min read

CURL — using notes

Intro

I think so curl is fun, I had many opportunities to play with curl by POSTing, or for example reading session from cookies, so would like to summarized it.

Test Environment

Ubuntu 18.04 + docker and docker-compose
Curl
jq
Jenkins for docker

Let’s start

Simple docker-compose for Jenkins

version: '3.7'
services:
  jenkins:
    image: jenkins/jenkins:lts
    privileged: true
    user: root
    ports:
      - 8081:8080
      - 50000:50000
    container_name: jenkins
    volumes:
      - /tmpy:/var/jenkins_home
      - /var/run/docker.sock:/var/run/docker.sock
      - /usr/local/bin/docker:/usr/local/bin/docker

Run command docker-compose up -d and after starting the container, go to the address http: //<IP-ADDRESS>: 8081. We will need password from container we can get it with this command docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword. After complete the configuration of Jenkins, we can start play with curl.

GET

If you just want to make a GET request, you don’t need any options.

root@vagrant:/home/vagrant/jenkins# curl http://192.168.123.123:8081/api/json | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   605  100   605    0     0  61489      0 --:--:-- --:--:-- --:--:-- 75625
{
  "_class": "hudson.model.Hudson",
  "assignedLabels": [
    {
      "name": "master"
    }
  ],
  "mode": "NORMAL",
  "nodeDescription": "the master Jenkins node",
  "nodeName": "",
  "numExecutors": 2,
  "description": null,
  "jobs": [],
  "overallLoad": {},
  "primaryView": {
    "_class": "hudson.model.AllView",
    "name": "all",
    "url": "http://192.168.123.123:8081/"
  },
  "quietDownReason": null,
  "quietingDown": false,
  "slaveAgentPort": 50000,
  "unlabeledLoad": {
    "_class": "jenkins.model.UnlabeledLoadStatistics"
  },
  "url": "http://192.168.123.123:8081/",
  "useCrumbs": true,
  "useSecurity": true,
  "views": [
    {
      "_class": "hudson.model.AllView",
      "name": "all",
      "url": "http://192.168.123.123:8081/"
    }
  ]
}
root@vagrant:/home/vagrant/jenkins#

Output to file

We use the same command as above but we add optionally output to a file with -o

root@vagrant:/home/vagrant/jenkins# curl http://192.168.123.123:8081/api/json -o result
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   605  100   605    0     0  53653      0 --:--:-- --:--:-- --:--:-- 60500
root@vagrant:/home/vagrant/jenkins# cat result | jq .
{
  "_class": "hudson.model.Hudson",
  "assignedLabels": [
    {
      "name": "master"
    }
  ],
  "mode": "NORMAL",
  "nodeDescription": "the master Jenkins node",
  "nodeName": "",
  "numExecutors": 2,
  "description": null,
  "jobs": [],
  "overallLoad": {},
  "primaryView": {
    "_class": "hudson.model.AllView",
    "name": "all",
    "url": "http://192.168.123.123:8081/"
  },
  "quietDownReason": null,
  "quietingDown": false,
  "slaveAgentPort": 50000,
  "unlabeledLoad": {
    "_class": "jenkins.model.UnlabeledLoadStatistics"
  },
  "url": "http://192.168.123.123:8081/",
  "useCrumbs": true,
  "useSecurity": true,
  "views": [
    {
      "_class": "hudson.model.AllView",
      "name": "all",
      "url": "http://192.168.123.123:8081/"
    }
  ]
}

Progress Meter

In curl by default curl outputs transfer information such as to the console like below:

But if wee need saw progress bar you can use the option to change the notation to something like below.

root@vagrant:/home/vagrant/jenkins# curl http://192.168.123.123:8081/api/json -o result -#
######################################################################## 100.0%

I think the progress information is not crucial information for us, Since it is output to stderr, it is a good idea to dig into /dev/null, but it is obediently -s controlled by options

#Without -s
root@vagrant:/home/vagrant/jenkins# curl http://192.168.123.123:8089/api/json -0
curl: (7) Failed to connect to 192.168.123.123 port 8089: Connection refused
root@vagrant:/home/vagrant/jenkins##With -s
root@vagrant:/home/vagrant/jenkins# curl -s http://192.168.123.123:8089/api/json -0
root@vagrant:/home/vagrant/jenkins#

However, at this time, even the error message disappears. It think is safer -S to specify together because you will not notice if you make a mistake in the URL or Port .

root@vagrant:/home/vagrant/jenkins# curl -Ss http://192.168.123.123:8089/api/json -0
curl: (7) Failed to connect to 192.168.123.123 port 8089: Connection refused
root@vagrant:/home/vagrant/jenkins#

Check the HTTP Headers

With -I only Header can be acquired and output.

root@vagrant:/home/vagrant/jenkins# curl -I -s  http://192.168.123.123:8081/api/json
HTTP/1.1 200 OK
Date: Thu, 29 Apr 2021 18:53:59 GMT
X-Content-Type-Options: nosniff
X-Jenkins: 2.277.3
X-Jenkins-Session: 7679bd14
X-Frame-Options: deny
Content-Type: application/json;charset=utf-8
Content-Length: 605
Server: Jetty(9.4.39.v20210325)

With -i we have both Response Header and Body can be output.

root@vagrant:/home/vagrant/jenkins# curl -i -s  http://192.168.123.123:8081/api/json?
HTTP/1.1 200 OK
Date: Thu, 29 Apr 2021 18:55:13 GMT
X-Content-Type-Options: nosniff
X-Jenkins: 2.277.3
X-Jenkins-Session: 7679bd14
X-Frame-Options: deny
Content-Type: application/json;charset=utf-8
Content-Length: 605
Server: Jetty(9.4.39.v20210325){"_class":"hudson.model.Hudson","assignedLabels":[{"name":"master"}],"mode":"NORMAL","nodeDescription":"the master Jenkins node","nodeName":"","numExecutors":2,"description":null,"jobs":[],"overallLoad":{},"primaryView":{"_class":"hudson.model.AllView","name":"all","url":"http://192.168.123.123:8081/"},"quietDownReason":null,"quietingDown":false,"slaveAgentPort":50000,"unlabeledLoad":{"_class":"jenkins.model.UnlabeledLoadStatistics"},"url":"http://192.168.123.123:8081/","useCrumbs":true,"useSecurity":true,"views":[{"_class":"hudson.model.AllView","name":"all","url":"http://192.168.123.123:8081/"}]}root@vagrant:/home/vagrant/jenkins#
root@vagrant:/home/vagrant/jenkins#

This is just the response Http Header, but if you want to see the Header when you request it with curl, you can use -v

root@vagrant:/home/vagrant/jenkins# curl -v -s  http://192.168.123.123:8081/api/json?
*   Trying 192.168.123.123...
* Connected to 192.168.123.123 (192.168.123.123) port 8081 (#0)
> GET /api/json? HTTP/1.1
> Host: 192.168.123.123:8081
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 29 Apr 2021 18:56:59 GMT
< X-Content-Type-Options: nosniff
< X-Jenkins: 2.277.3
< X-Jenkins-Session: 7679bd14
< X-Frame-Options: deny
< Content-Type: application/json;charset=utf-8
< Content-Length: 605
< Server: Jetty(9.4.39.v20210325)
<
* Connection #0 to host 192.168.123.123 left intact
{"_class":"hudson.model.Hudson","assignedLabels":[{"name":"master"}],"mode":"NORMAL","nodeDescription":"the master Jenkins node","nodeName":"","numExecutors":2,"description":null,"jobs":[],"overallLoad":{},"primaryView":{"_class":"hudson.model.AllView","name":"all","url":"http://192.168.123.123:8081/"},"quietDownReason":null,"quietingDown":false,"slaveAgentPort":50000,"unlabeledLoad":{"_class":"jenkins.model.UnlabeledLoadStatistics"},"url":"http://192.168.123.123:8081/","useCrumbs":true,"useSecurity":true,"views":[{"_class":"hudson.model.AllView","name":"all","url":"http://192.168.123.123:8081/"}]}
root@vagrant:/home/vagrant/jenkins#

Check more HTTP packet data

With --trace or --trace-ascii we can dump all HTTP request/response data.

Trace

root@vagrant:/home/vagrant# curl -sS  http://192.168.123.123:8081 -X POST -F "test=test" --trace trace.log -o /dev/null
root@vagrant:/home/vagrant# cat trace.log  | head
== Info: Rebuilt URL to: http://192.168.123.123:8081/
== Info:   Trying 192.168.123.123...
== Info: Connected to 192.168.123.123 (192.168.123.123) port 8081 (#0)
=> Send header, 214 bytes (0xd6)
0000: 50 4f 53 54 20 2f 20 48 54 54 50 2f 31 2e 31 0d POST / HTTP/1.1.
0010: 0a 48 6f 73 74 3a 20 31 39 32 2e 31 36 38 2e 31 .Host: 192.168.1
0020: 32 33 2e 31 32 33 3a 38 30 38 31 0d 0a 55 73 65 23.123:8081..Use
0030: 72 2d 41 67 65 6e 74 3a 20 63 75 72 6c 2f 37 2e r-Agent: curl/7.
0040: 34 37 2e 30 0d 0a 41 63 63 65 70 74 3a 20 2a 2f 47.0..Accept: */
0050: 2a 0d 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 *..Content-Lengt
root@vagrant:/home/vagrant#

Output doesn’t look right and reminds us of some binary editor

Trace-ascii

root@vagrant:/home/vagrant# curl -sS  http://192.168.123.123:8081 -X POST -F "test=test" --trace-ascii trace-ascii.log -o /dev/null
root@vagrant:/home/vagrant# cat trace-ascii.log  | head
== Info: Rebuilt URL to: http://192.168.123.123:8081/
== Info:   Trying 192.168.123.123...
== Info: Connected to 192.168.123.123 (192.168.123.123) port 8081 (#0)
=> Send header, 214 bytes (0xd6)
0000: POST / HTTP/1.1
0011: Host: 192.168.123.123:8081
002d: User-Agent: curl/7.47.0
0046: Accept: */*
0053: Content-Length: 143
0068: Expect: 100-continue
root@vagrant:/home/vagrant#

This output seems to be more eye-friendly 😀

If we need for example add date and time to our output, it’s not a problem, just use --trace-time in command.

POST

If we need send POST request, we need to use -X POST grant.

⚠️CSFR Protection is turned on by default in jenkins, for tests we can turn it off using Jenkins Script Console.

Script for disable CSFR Protection

import jenkins.model.Jenkins
def instance = Jenkins.instance
instance.setCrumbIssuer(null)

root@vagrant:/home/vagrant# curl -sS -w '\n' -X POST '192.168.123.123:8081'

POST with parameters

When we POSTing, I think that the majority of requests are made with parameters or data --data or its abbreviation, -d you can describe the data to be sent by POST.

As an example we will try to create an job in Jenkins with POST

root@vagrant:/home/vagrant# curl -w '\n' 'http://192.168.123.123:8081/createItem' --data 'name=Example_FreeStyle_JOb&mode=hudson.model.FreeStyleProject&Submit=OK' -XPOST

Result in jenkins:

Now check with curl if this job exist

root@vagrant:/home/vagrant# curl -sS 'http://192.168.123.123:8081/api/json' | jq .jobs
[
  {
    "_class": "hudson.model.FreeStyleProject",
    "name": "Example_FreeStyle_JOb",
    "url": "http://192.168.123.123:8081/job/Example_FreeStyle_JOb/",
    "color": "notbuilt"
  }
]
root@vagrant:/home/vagrant#

URL encoding and parameterized POST

BY default parameter --data does not URL-encode, so we need to encode it yourself in advance. As a test, we add the file parameter settings to the job Example_FreeStyle_JOb created above.

root@vagrant:/home/vagrant# curl -w '\n' 'http://192.168.123.123:8081/job/Example_FreeStyle_JOb/configSubmit' --data 'json=%7b%22properties%22%3a+%7b%22hudson-model-ParametersDefinitionProperty%22%3a+%7b%22parameterized%22%3a+%7b%22parameter%22%3a+%7b%22name%22%3a+%22FileParameter%22%2c+%22description%22%3a+%22%22%2c+%22stapler-class%22%3a+%22hudson.model.FileParameterDefinition%22%2c+%22%24class%22%3a+%22hudson.model.FileParameterDefinition%22%7d%7d%7d%7d%7d%0d%0a&Submit=Save' -XPOSTroot@vagrant:/home/vagrant#

Test if settings was apply

In Jenkins

With curl

root@vagrant:/home/vagrant# curl -sS 'http://192.168.123.123:8081/job/Example_FreeStyle_JOb/api/json' | jq .actions
[
  {
    "_class": "hudson.model.ParametersDefinitionProperty",
    "parameterDefinitions": [
      {
        "_class": "hudson.model.FileParameterDefinition",
        "defaultParameterValue": null,
        "description": "",
        "name": "",
        "type": "FileParameterDefinition"
      }
    ]
  },
  {},
  {},
  {
    "_class": "org.jenkinsci.plugins.displayurlapi.actions.JobDisplayAction"
  },
  {
    "_class": "com.cloudbees.plugins.credentials.ViewCredentialsAction"
  }
]
root@vagrant:/home/vagrant#

It’s a hassle to URL encode on your own, will be hard to understand at a glance. If you use, curl will URL encode it, so you can write it as it is.
As a test, we add a some description to the parameters added above.

curl -w '\n' 'http://192.168.123.123:8081/job/Example_FreeStyle_JOb/configSubmit' --data-urlencode 'json={"properties": {"hudson-model-ParametersDefinitionProperty": {"parameterized": {"parameter": {"name": "FileParameter", "description": "Test description in Jenkins", "stapler-class": "hudson.model.FileParameterDefinition", "$class": "hudson.model.FileParameterDefinition"}}}}}' -d 'Submit=Save' -XPOST

Test if settings was apply

In Jenkins

Test with curl

root@vagrant:/home/vagrant# curl -sS 'http://192.168.123.123:8081/job/Example_FreeStyle_JOb/api/json' | jq .actions
[
  {
    "_class": "hudson.model.ParametersDefinitionProperty",
    "parameterDefinitions": [
      {
        "_class": "hudson.model.FileParameterDefinition",
        "defaultParameterValue": null,
        "description": "Test description in Jenkins",
        "name": "",
        "type": "FileParameterDefinition"
      }
    ]
  },
  {},
  {},
  {
    "_class": "org.jenkinsci.plugins.displayurlapi.actions.JobDisplayAction"
  },
  {
    "_class": "com.cloudbees.plugins.credentials.ViewCredentialsAction"
  }
]
root@vagrant:/home/vagrant#

Upload file with POST

In this case we run Jenkins build and uploads the file with POST.

#Prepare file
root@vagrant:/home/vagrant# echo "Some test file" > test.txt#Run Job
root@vagrant:/home/vagrant# curl -sS 'http://192.168.123.123:8081/job/Example_FreeStyle_JOb/build' -X POST -F "file=@test.txt" -F 'json={"parameter": [{"name":"FileParameter", "file":"file"}]}'#Check if job are running 
root@vagrant:/home/vagrant# curl -sS 'http://192.168.123.123:8081/job/Example_FreeStyle_JOb/api/json' | jq .builds
[
  {
    "_class": "hudson.model.FreeStyleBuild",
    "number": 1,
    "url": "http://192.168.123.123:8081/job/Example_FreeStyle_JOb/1/"
  }
]#Check parameters file
root@vagrant:/home/vagrant# curl -sS 'http://192.168.123.123:8081/job/Example_FreeStyle_JOb/1/parameters/parameter/test.txt/test.txt'
Some test file
root@vagrant:/home/vagrant#
root@vagrant:/home/vagrant#

CURL and Authentication

For this we will need turn on Matrix-based security in Jenkins’ security settings and create test user.

New User

Matrix Based security

If we configure this try now hit Jenkins from curl.

root@vagrant:/home/vagrant# curl -sS 'http://192.168.123.123:8081/api/json' -I
HTTP/1.1 403 Forbidden
Date: Sat, 01 May 2021 15:03:40 GMT
X-Content-Type-Options: nosniff
Set-Cookie: JSESSIONID.d9cf535a=node01gq5wts64izzk1b1uuc4l048uj18.node0; Path=/; HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/html;charset=utf-8
X-Hudson: 1.395
X-Jenkins: 2.277.3
X-Jenkins-Session: 7679bd14
X-Hudson-CLI-Port: 50000
X-Jenkins-CLI-Port: 50000
X-Jenkins-CLI2-Port: 50000
Content-Length: 561
Server: Jetty(9.4.39.v20210325)root@vagrant:/home/vagrant#

We should receive HTTP 403 and as expected, account restrictions are working.😀

BASIC authentication

We will try run the same command with provide user and password

root@vagrant:/home/vagrant# curl -sS 'http://192.168.123.123:8081/api/json' -I -I -u testuser:P@ssw0rd_123
HTTP/1.1 200 OK
Date: Sat, 01 May 2021 15:07:33 GMT
X-Content-Type-Options: nosniff
X-Jenkins: 2.277.3
X-Jenkins-Session: 7679bd14
X-Frame-Options: deny
Content-Type: application/json;charset=utf-8
Content-Length: 882
Server: Jetty(9.4.39.v20210325)root@vagrant:/home/vagrant#

As you can see, the password remains on the console, which is not entirely good because you can later read it from e.g. bash_history. If we don’t provide a password in the command, you’ll be prompted for it when you run curl.

root@vagrant:/home/vagrant# curl -sS 'http://192.168.123.123:8081/api/json' -I -I -u testuser
Enter host password for user 'testuser':
HTTP/1.1 200 OK
Date: Sat, 01 May 2021 15:10:14 GMT
X-Content-Type-Options: nosniff
X-Jenkins: 2.277.3
X-Jenkins-Session: 7679bd14
X-Frame-Options: deny
Content-Type: application/json;charset=utf-8
Content-Length: 882
Server: Jetty(9.4.39.v20210325)root@vagrant:/home/vagrant#

In addition to specifying a username and password with the -u parameter, we can also do it in another way. An example below

curl -sS 'http://user:password@localhost:8081/api/json' -I

Other frequently used options

Ignore the SSL certificate. Mainly when accessing a web server that uses an not trusted certificate: -k
Specify Proxy : -x
Add Request Header. For example, when specifying Content-Type :-H "Content-Type: application/json"
Access to the redirect destination: -L

Summary

I think Curl is a cool and very powerful tool with which we can do quite a lot of things. However, if we are looking for more useful http commands i would recommend httpie which is considered as more easy and, user-friendly command-line HTTP client for the API era.

httpie/httpie

HTTPie (pronounced aitch-tee-tee-pie) is a command-line HTTP client. Its goal is to make CLI interaction with web…

github.com