CURL — using notes

Photo by AbsolutVision on Unsplash

Intro

I think so curl is fun, I had many opportunities to play with curl by POSTing, or for example reading session from cookies, so would like to summarized it.

Test Environment

Let’s start

  • Simple docker-compose for Jenkins
version: '3.7'
services:
jenkins:
image: jenkins/jenkins:lts
privileged: true
user: root
ports:
- 8081:8080
- 50000:50000
container_name: jenkins
volumes:
- /tmpy:/var/jenkins_home
- /var/run/docker.sock:/var/run/docker.sock
- /usr/local/bin/docker:/usr/local/bin/docker
  • Run command docker-compose up -d and after starting the container, go to the address http: //<IP-ADDRESS>: 8081. We will need password from container we can get it with this command docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword. After complete the configuration of Jenkins, we can start play with curl.

GET

If you just want to make a GET request, you don’t need any options.

root@vagrant:/home/vagrant/jenkins# curl http://192.168.123.123:8081/api/json | jq .
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 605 100 605 0 0 61489 0 --:--:-- --:--:-- --:--:-- 75625
{
"_class": "hudson.model.Hudson",
"assignedLabels": [
{
"name": "master"
}
],
"mode": "NORMAL",
"nodeDescription": "the master Jenkins node",
"nodeName": "",
"numExecutors": 2,
"description": null,
"jobs": [],
"overallLoad": {},
"primaryView": {
"_class": "hudson.model.AllView",
"name": "all",
"url": "http://192.168.123.123:8081/"
},
"quietDownReason": null,
"quietingDown": false,
"slaveAgentPort": 50000,
"unlabeledLoad": {
"_class": "jenkins.model.UnlabeledLoadStatistics"
},
"url": "http://192.168.123.123:8081/",
"useCrumbs": true,
"useSecurity": true,
"views": [
{
"_class": "hudson.model.AllView",
"name": "all",
"url": "http://192.168.123.123:8081/"
}
]
}
root@vagrant:/home/vagrant/jenkins#

Output to file

We use the same command as above but we add optionally output to a file with -o

root@vagrant:/home/vagrant/jenkins# curl http://192.168.123.123:8081/api/json -o result
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 605 100 605 0 0 53653 0 --:--:-- --:--:-- --:--:-- 60500
root@vagrant:/home/vagrant/jenkins# cat result | jq .
{
"_class": "hudson.model.Hudson",
"assignedLabels": [
{
"name": "master"
}
],
"mode": "NORMAL",
"nodeDescription": "the master Jenkins node",
"nodeName": "",
"numExecutors": 2,
"description": null,
"jobs": [],
"overallLoad": {},
"primaryView": {
"_class": "hudson.model.AllView",
"name": "all",
"url": "http://192.168.123.123:8081/"
},
"quietDownReason": null,
"quietingDown": false,
"slaveAgentPort": 50000,
"unlabeledLoad": {
"_class": "jenkins.model.UnlabeledLoadStatistics"
},
"url": "http://192.168.123.123:8081/",
"useCrumbs": true,
"useSecurity": true,
"views": [
{
"_class": "hudson.model.AllView",
"name": "all",
"url": "http://192.168.123.123:8081/"
}
]
}

Progress Meter

In curl by default curl outputs transfer information such as to the console like below:

But if wee need saw progress bar you can use the option to change the notation to something like below.

root@vagrant:/home/vagrant/jenkins# curl http://192.168.123.123:8081/api/json -o result -#
######################################################################## 100.0%

I think the progress information is not crucial information for us, Since it is output to stderr, it is a good idea to dig into /dev/null, but it is obediently -s controlled by options

#Without -s
root@vagrant:/home/vagrant/jenkins# curl http://192.168.123.123:8089/api/json -0
curl: (7) Failed to connect to 192.168.123.123 port 8089: Connection refused
root@vagrant:/home/vagrant/jenkins#
#With -s
root@vagrant:/home/vagrant/jenkins# curl -s http://192.168.123.123:8089/api/json -0
root@vagrant:/home/vagrant/jenkins#

However, at this time, even the error message disappears. It think is safer -S to specify together because you will not notice if you make a mistake in the URL or Port .

root@vagrant:/home/vagrant/jenkins# curl -Ss http://192.168.123.123:8089/api/json -0
curl: (7) Failed to connect to 192.168.123.123 port 8089: Connection refused
root@vagrant:/home/vagrant/jenkins#

Check the HTTP Headers

  • With -I only Header can be acquired and output.
root@vagrant:/home/vagrant/jenkins# curl -I -s  http://192.168.123.123:8081/api/json
HTTP/1.1 200 OK
Date: Thu, 29 Apr 2021 18:53:59 GMT
X-Content-Type-Options: nosniff
X-Jenkins: 2.277.3
X-Jenkins-Session: 7679bd14
X-Frame-Options: deny
Content-Type: application/json;charset=utf-8
Content-Length: 605
Server: Jetty(9.4.39.v20210325)
  • With -i we have both Response Header and Body can be output.
root@vagrant:/home/vagrant/jenkins# curl -i -s  http://192.168.123.123:8081/api/json?
HTTP/1.1 200 OK
Date: Thu, 29 Apr 2021 18:55:13 GMT
X-Content-Type-Options: nosniff
X-Jenkins: 2.277.3
X-Jenkins-Session: 7679bd14
X-Frame-Options: deny
Content-Type: application/json;charset=utf-8
Content-Length: 605
Server: Jetty(9.4.39.v20210325)
{"_class":"hudson.model.Hudson","assignedLabels":[{"name":"master"}],"mode":"NORMAL","nodeDescription":"the master Jenkins node","nodeName":"","numExecutors":2,"description":null,"jobs":[],"overallLoad":{},"primaryView":{"_class":"hudson.model.AllView","name":"all","url":"http://192.168.123.123:8081/"},"quietDownReason":null,"quietingDown":false,"slaveAgentPort":50000,"unlabeledLoad":{"_class":"jenkins.model.UnlabeledLoadStatistics"},"url":"http://192.168.123.123:8081/","useCrumbs":true,"useSecurity":true,"views":[{"_class":"hudson.model.AllView","name":"all","url":"http://192.168.123.123:8081/"}]}root@vagrant:/home/vagrant/jenkins#
root@vagrant:/home/vagrant/jenkins#

This is just the response Http Header, but if you want to see the Header when you request it with curl, you can use -v

root@vagrant:/home/vagrant/jenkins# curl -v -s  http://192.168.123.123:8081/api/json?
* Trying 192.168.123.123...
* Connected to 192.168.123.123 (192.168.123.123) port 8081 (#0)
> GET /api/json? HTTP/1.1
> Host: 192.168.123.123:8081
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 29 Apr 2021 18:56:59 GMT
< X-Content-Type-Options: nosniff
< X-Jenkins: 2.277.3
< X-Jenkins-Session: 7679bd14
< X-Frame-Options: deny
< Content-Type: application/json;charset=utf-8
< Content-Length: 605
< Server: Jetty(9.4.39.v20210325)
<
* Connection #0 to host 192.168.123.123 left intact
{"_class":"hudson.model.Hudson","assignedLabels":[{"name":"master"}],"mode":"NORMAL","nodeDescription":"the master Jenkins node","nodeName":"","numExecutors":2,"description":null,"jobs":[],"overallLoad":{},"primaryView":{"_class":"hudson.model.AllView","name":"all","url":"http://192.168.123.123:8081/"},"quietDownReason":null,"quietingDown":false,"slaveAgentPort":50000,"unlabeledLoad":{"_class":"jenkins.model.UnlabeledLoadStatistics"},"url":"http://192.168.123.123:8081/","useCrumbs":true,"useSecurity":true,"views":[{"_class":"hudson.model.AllView","name":"all","url":"http://192.168.123.123:8081/"}]}
root@vagrant:/home/vagrant/jenkins#

Check more HTTP packet data

With --trace or --trace-ascii we can dump all HTTP request/response data.

Trace

root@vagrant:/home/vagrant# curl -sS  http://192.168.123.123:8081 -X POST -F "test=test" --trace trace.log -o /dev/null
root@vagrant:/home/vagrant# cat trace.log | head
== Info: Rebuilt URL to: http://192.168.123.123:8081/
== Info: Trying 192.168.123.123...
== Info: Connected to 192.168.123.123 (192.168.123.123) port 8081 (#0)
=> Send header, 214 bytes (0xd6)
0000: 50 4f 53 54 20 2f 20 48 54 54 50 2f 31 2e 31 0d POST / HTTP/1.1.
0010: 0a 48 6f 73 74 3a 20 31 39 32 2e 31 36 38 2e 31 .Host: 192.168.1
0020: 32 33 2e 31 32 33 3a 38 30 38 31 0d 0a 55 73 65 23.123:8081..Use
0030: 72 2d 41 67 65 6e 74 3a 20 63 75 72 6c 2f 37 2e r-Agent: curl/7.
0040: 34 37 2e 30 0d 0a 41 63 63 65 70 74 3a 20 2a 2f 47.0..Accept: */
0050: 2a 0d 0a 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 *..Content-Lengt
root@vagrant:/home/vagrant#

Output doesn’t look right and reminds us of some binary editor

Trace-ascii

root@vagrant:/home/vagrant# curl -sS  http://192.168.123.123:8081 -X POST -F "test=test" --trace-ascii trace-ascii.log -o /dev/null
root@vagrant:/home/vagrant# cat trace-ascii.log | head
== Info: Rebuilt URL to: http://192.168.123.123:8081/
== Info: Trying 192.168.123.123...
== Info: Connected to 192.168.123.123 (192.168.123.123) port 8081 (#0)
=> Send header, 214 bytes (0xd6)
0000: POST / HTTP/1.1
0011: Host: 192.168.123.123:8081
002d: User-Agent: curl/7.47.0
0046: Accept: */*
0053: Content-Length: 143
0068: Expect: 100-continue
root@vagrant:/home/vagrant#

This output seems to be more eye-friendly 😀

If we need for example add date and time to our output, it’s not a problem, just use --trace-time in command.

POST

If we need send POST request, we need to use -X POST grant.

⚠️CSFR Protection is turned on by default in jenkins, for tests we can turn it off using Jenkins Script Console.

  • Script for disable CSFR Protection
import jenkins.model.Jenkins
def instance = Jenkins.instance
instance.setCrumbIssuer(null)
root@vagrant:/home/vagrant# curl -sS -w '\n' -X POST '192.168.123.123:8081'

POST with parameters

When we POSTing, I think that the majority of requests are made with parameters or data --data or its abbreviation, -d you can describe the data to be sent by POST.

As an example we will try to create an job in Jenkins with POST

root@vagrant:/home/vagrant# curl -w '\n' 'http://192.168.123.123:8081/createItem' --data 'name=Example_FreeStyle_JOb&mode=hudson.model.FreeStyleProject&Submit=OK' -XPOST

Result in jenkins:

Now check with curl if this job exist

root@vagrant:/home/vagrant# curl -sS 'http://192.168.123.123:8081/api/json' | jq .jobs
[
{
"_class": "hudson.model.FreeStyleProject",
"name": "Example_FreeStyle_JOb",
"url": "http://192.168.123.123:8081/job/Example_FreeStyle_JOb/",
"color": "notbuilt"
}
]
root@vagrant:/home/vagrant#

URL encoding and parameterized POST

BY default parameter --data does not URL-encode, so we need to encode it yourself in advance. As a test, we add the file parameter settings to the job Example_FreeStyle_JOb created above.

root@vagrant:/home/vagrant# curl -w '\n' 'http://192.168.123.123:8081/job/Example_FreeStyle_JOb/configSubmit' --data 'json=%7b%22properties%22%3a+%7b%22hudson-model-ParametersDefinitionProperty%22%3a+%7b%22parameterized%22%3a+%7b%22parameter%22%3a+%7b%22name%22%3a+%22FileParameter%22%2c+%22description%22%3a+%22%22%2c+%22stapler-class%22%3a+%22hudson.model.FileParameterDefinition%22%2c+%22%24class%22%3a+%22hudson.model.FileParameterDefinition%22%7d%7d%7d%7d%7d%0d%0a&Submit=Save' -XPOSTroot@vagrant:/home/vagrant#

Test if settings was apply

  • In Jenkins
  • With curl
root@vagrant:/home/vagrant# curl -sS 'http://192.168.123.123:8081/job/Example_FreeStyle_JOb/api/json' | jq .actions
[
{
"_class": "hudson.model.ParametersDefinitionProperty",
"parameterDefinitions": [
{
"_class": "hudson.model.FileParameterDefinition",
"defaultParameterValue": null,
"description": "",
"name": "",
"type": "FileParameterDefinition"
}

]
},
{},
{},
{
"_class": "org.jenkinsci.plugins.displayurlapi.actions.JobDisplayAction"
},
{
"_class": "com.cloudbees.plugins.credentials.ViewCredentialsAction"
}
]
root@vagrant:/home/vagrant#

It’s a hassle to URL encode on your own, will be hard to understand at a glance. If you use, curl will URL encode it, so you can write it as it is.
As a test, we add a some description to the parameters added above.

curl -w '\n' 'http://192.168.123.123:8081/job/Example_FreeStyle_JOb/configSubmit' --data-urlencode 'json={"properties": {"hudson-model-ParametersDefinitionProperty": {"parameterized": {"parameter": {"name": "FileParameter", "description": "Test description in Jenkins", "stapler-class": "hudson.model.FileParameterDefinition", "$class": "hudson.model.FileParameterDefinition"}}}}}' -d 'Submit=Save' -XPOST

Test if settings was apply

  • In Jenkins
  • Test with curl
root@vagrant:/home/vagrant# curl -sS 'http://192.168.123.123:8081/job/Example_FreeStyle_JOb/api/json' | jq .actions
[
{
"_class": "hudson.model.ParametersDefinitionProperty",
"parameterDefinitions": [
{
"_class": "hudson.model.FileParameterDefinition",
"defaultParameterValue": null,
"description": "Test description in Jenkins",
"name": "",
"type": "FileParameterDefinition"
}
]
},
{},
{},
{
"_class": "org.jenkinsci.plugins.displayurlapi.actions.JobDisplayAction"
},
{
"_class": "com.cloudbees.plugins.credentials.ViewCredentialsAction"
}
]
root@vagrant:/home/vagrant#

Upload file with POST

In this case we run Jenkins build and uploads the file with POST.

#Prepare file
root@vagrant:/home/vagrant# echo "Some test file" > test.txt
#Run Job
root@vagrant:/home/vagrant# curl -sS 'http://192.168.123.123:8081/job/Example_FreeStyle_JOb/build' -X POST -F "file=@test.txt" -F 'json={"parameter": [{"name":"FileParameter", "file":"file"}]}'
#Check if job are running
root@vagrant:/home/vagrant# curl -sS 'http://192.168.123.123:8081/job/Example_FreeStyle_JOb/api/json' | jq .builds
[
{
"_class": "hudson.model.FreeStyleBuild",
"number": 1,
"url": "http://192.168.123.123:8081/job/Example_FreeStyle_JOb/1/"
}
]
#Check parameters file
root@vagrant:/home/vagrant# curl -sS 'http://192.168.123.123:8081/job/Example_FreeStyle_JOb/1/parameters/parameter/test.txt/test.txt'
Some test file
root@vagrant:/home/vagrant#
root@vagrant:/home/vagrant#

CURL and Authentication

For this we will need turn on Matrix-based security in Jenkins’ security settings and create test user.

  • New User
  • Matrix Based security

If we configure this try now hit Jenkins from curl.

root@vagrant:/home/vagrant# curl -sS 'http://192.168.123.123:8081/api/json' -I
HTTP/1.1 403 Forbidden
Date: Sat, 01 May 2021 15:03:40 GMT
X-Content-Type-Options: nosniff
Set-Cookie: JSESSIONID.d9cf535a=node01gq5wts64izzk1b1uuc4l048uj18.node0; Path=/; HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/html;charset=utf-8
X-Hudson: 1.395
X-Jenkins: 2.277.3
X-Jenkins-Session: 7679bd14
X-Hudson-CLI-Port: 50000
X-Jenkins-CLI-Port: 50000
X-Jenkins-CLI2-Port: 50000
Content-Length: 561
Server: Jetty(9.4.39.v20210325)
root@vagrant:/home/vagrant#

We should receive HTTP 403 and as expected, account restrictions are working.😀

BASIC authentication

We will try run the same command with provide user and password

root@vagrant:/home/vagrant# curl -sS 'http://192.168.123.123:8081/api/json' -I -I -u testuser:P@ssw0rd_123
HTTP/1.1 200 OK
Date: Sat, 01 May 2021 15:07:33 GMT
X-Content-Type-Options: nosniff
X-Jenkins: 2.277.3
X-Jenkins-Session: 7679bd14
X-Frame-Options: deny
Content-Type: application/json;charset=utf-8
Content-Length: 882
Server: Jetty(9.4.39.v20210325)
root@vagrant:/home/vagrant#

As you can see, the password remains on the console, which is not entirely good because you can later read it from e.g. bash_history. If we don’t provide a password in the command, you’ll be prompted for it when you run curl.

root@vagrant:/home/vagrant# curl -sS 'http://192.168.123.123:8081/api/json' -I -I -u testuser
Enter host password for user 'testuser':
HTTP/1.1 200 OK
Date: Sat, 01 May 2021 15:10:14 GMT
X-Content-Type-Options: nosniff
X-Jenkins: 2.277.3
X-Jenkins-Session: 7679bd14
X-Frame-Options: deny
Content-Type: application/json;charset=utf-8
Content-Length: 882
Server: Jetty(9.4.39.v20210325)
root@vagrant:/home/vagrant#

In addition to specifying a username and password with the -u parameter, we can also do it in another way. An example below

curl -sS 'http://user:password@localhost:8081/api/json' -I

Other frequently used options

  • Ignore the SSL certificate. Mainly when accessing a web server that uses an not trusted certificate: -k
  • Specify Proxy : -x
  • Add Request Header. For example, when specifying Content-Type :-H "Content-Type: application/json"
  • Access to the redirect destination: -L

Summary

I think Curl is a cool and very powerful tool with which we can do quite a lot of things. However, if we are looking for more useful http commands i would recommend httpie which is considered as more easy and, user-friendly command-line HTTP client for the API era.

 by the author.

--

--

--

DevOps Consultant. I’m strongly focused on automation, security, and reliability.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Fleet 4.9.0 brings performance updates, paginated live query results, and policy YAML doc support.

3SS Insights: Building a Set-Top-Box RDK vs. AOSP vs. Android TV

From PHPUnit to Pest

The Negative Idea of Joy

Hello x Cool Swift 3 Transition Framework

Refining the Refinement. Backlog refinement basics.

What kind of database is best for your app? An overview of noSQL.

Java 8:Best practices

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store