Stripping http response and getting only contents in CURL - bash

I am using the following command to fetch a file from a server:
curl -i -L --user user:pass -o $s.po -F file=#$s -X GET http://address
However, the result is like this:
HTTP/1.1 100 Continue
HTTP/1.1 200 OK
Server: nginx/1.2.1
Date: Wed, 04 Jun 2014 18:00:08 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Content-Language: en
Expires: Wed, 04 Jun 2014 18:00:08 GMT
Vary: Authorization, Host, Accept-Language, Cookie
Last-Modified: Wed, 04 Jun 2014 18:00:08 GMT
Cache-Control: max-age=0
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=15768000; includeSubDomains
X-UA-Compatible: IE=edge,chrome=1
{
"content": "foo\nbar",
"mimetype": "text/x-po"
}
I would like to write only the foo\nbar part of the response to store exactly the same file into the disk. How can I do that?
After removing -i option and adding > output.json gives the following output in the file $s.po:
{
"content": "foo\nbar",
"mimetype": "text/x-po"
}

Remove -i option and pipe output to an awk command and save in a file:
curl -L --user user:pass -o $s.po -F file=#$s -X GET http://address |
awk -F'[:,]' '$1~/content/{gsub(/[" ]/, "", $2); print $2}' > output.json
As per man curl:
-i, --include
(HTTP) Include the HTTP-header in the output. The HTTP-header includes things like
server-name, date of the document, HTTP-version and more...

Related

How to compare dates formatted as `Tue Aug 30 12:01:37 GMT 2022` in BusyBox/Alpine

In shell (no bash because of Alpine) using BusyBox, how can I compare two dates both formatted as Tue Aug 30 12:01:37 GMT 2022?
I want to know which one comes first. date doesn't support this input format. I'm only interested in whole days. The time isn't interesting for me. So two dates on the same day but a different time are equal to me.
Of course I could put all the names of the months in a lookup table and use the index of the month as its integer value (to be able to compare) but I have the feeling I shouldn't be the one programming that out...
Update:
/opt/scripts $ a="Tue Aug 30 12:01:37 GMT 2022"
/opt/scripts $ date -d "$a" +%s
date: invalid date 'Tue Aug 30 12:01:37 GMT 2022'
/opt/scripts $ date --help
BusyBox v1.34.1 (2022-04-04 10:19:27 UTC) multi-call binary.
Usage: date [OPTIONS] [+FMT] [[-s] TIME]
Display time (using +FMT), or set time
-u Work in UTC (don't convert to local time)
[-s] TIME Set time to TIME
-d TIME Display TIME, not 'now'
-D FMT FMT (strptime format) for -s/-d TIME conversion
-r FILE Display last modification time of FILE
-R Output RFC-2822 date
-I[SPEC] Output ISO-8601 date
SPEC=date (default), hours, minutes, seconds or ns
Recognized TIME formats:
#seconds_since_1970
hh:mm[:ss]
[YYYY.]MM.DD-hh:mm[:ss]
YYYY-MM-DD hh:mm[:ss]
[[[[[YY]YY]MM]DD]hh]mm[.ss]
'date TIME' form accepts MMDDhhmm[[YY]YY][.ss] instead
/opt/scripts $
Install dateutils https://pkgs.alpinelinux.org/package/edge/community/x86/dateutils . Use strptime to convert the date to seconds. Compare seconds.
apk add dateutils
a=$(strptime -f %s -i "%a %b %d %T %Z %Y" "Tue Aug 30 12:01:37 GMT 2022")
b=$(strptime -f %s -i "%a %b %d %T %Z %Y" "Tue Aug 30 12:01:38 GMT 2022")
[ "$a" -lt "$b")
You may have to rely on awk:
/ # cat /etc/alpine-release
3.16.0
/ # echo $a
Tue Aug 30 12:01:37 GMT 2022
/ # TZ=GMT awk -v a="$a" 'BEGIN {
> split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec", months)
> split(a, date)
> gsub(/:/, " ", date[4])
>
> for (i=1; i<=12; i++) {
> if (date[2] == months[i]) {
> timestamp = date[6] " " i " " date[3] " " date[4]
> print mktime(timestamp)
> exit
> }
> }
>
> print "hmm, " date[2] " is an unknown month"
> exit 1
> }'
1661860897
Ok, my alpine busybox copy of date doesn't recognize strings as month either.
You want "slick", stick with Glenn's awk solution, so long as the time functions work for you. I hacked out the least-slick kluge using just echo, date, read, if's, and a lot of tempfiles - it's an ugly mess, but it works, and it was a fun exercise in using only the most basic stuff.
/tmp $ ./script
#! /bin/sh
cat "$0"
cd /tmp
echo "01">Jan
echo "02">Feb
echo "03">Mar
echo "04">Apr
echo "05">May
echo "06">Jun
echo "07">Jul
echo "08">Aug
echo "09">Sep
echo "10">Oct
echo "11">Nov
echo "12">Dec
echo "Tue Aug 30 12:01:37 GMT 2022">a_raw
read -r a_raw<a_raw
echo "Fri Jun 3 09:26:55 CDT 2022">b_raw
read -r b_raw<b_raw
read -r _ Mon DD tim z YYYY<a_raw
read -r MM<"$Mon"
date -d "$YYYY-$MM-$DD" +"%s">a_epoch
read -r a_epoch<a_epoch
read -r _ Mon DD tim z YYYY<b_raw
read -r MM<"$Mon"
date -d "$YYYY-$MM-$DD" +"%s">b_epoch
read -r b_epoch<b_epoch
if [ "$a_epoch" -lt "$b_epoch" ]
then echo "$a_raw ($a_epoch) is before $b_raw ($b_epoch)"
else if [ "$a_epoch" -gt "$b_epoch" ]
then echo "$a_raw ($a_epoch) is after $b_raw ($b_epoch)"
else if [ "$a_epoch" -eq "$b_epoch" ]
then echo "$a_raw ($a_epoch) is same as $b_raw ($b_epoch)"
fi
fi
fi
rm Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec a_raw b_raw a_epoch b_epoch
Tue Aug 30 12:01:37 GMT 2022 (1661817600) is after Fri Jun 3 09:26:55 CDT 2022 (1654214400)
original
What do you mean "date doesn't support this input format"?
Something like this ought to work in sh, though I confess I don't have an alpine handy...
a="Tue Aug 30 12:01:37 GMT 2022"
b="Fri Jun 3 09:26:55 CDT 2022"
a_epoch=`date -d "$a" +%s`
b_epoch=`date -d "$b" +%s`
echo "A: [$a] ($a_epoch)"
echo "B: [$b] ($b_epoch)"
if [ "$a_epoch" -lt "$b_epoch" ]; then echo "$a is before $b"; fi
if [ "$a_epoch" -gt "$b_epoch" ]; then echo "$a is after $b"; fi
if [ "$a_epoch" -eq "$b_epoch" ]; then echo "$a is same as $b"; fi
Should say something like
A: [Tue Aug 30 12:01:37 GMT 2022] (1661860897)
B: [Fri Jun 3 09:26:55 CDT 2022] (1654266415)
Tue Aug 30 12:01:37 GMT 2022 is after Fri Jun 3 09:26:55 CDT 2022
There are cleaner ways, but this should get you started.
Lemme spin up a container and try there, brb...

Is there any reason why my AWK functions work only on a shortened version of my file

I have a simple AWK function:
awk '
BEGIN { FS=" "; RS="\n\n" ; OFS="\n"; ORS="\n" }
/ms Response/ { print $0 }
' $FILE
The FILE is a large log that holds sections like this:
2021-10-13 12:15:12 CDT 526ms Request
POST / HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: xxxxxxxxxxxxxxxxxxx
Content-Length: 279
<query xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><product><name>drill</name><price>99</price><stock>5</stock></product>/query>
2021-10-13 12:15:12 CDT 880ms Received
2021-10-13 12:15:12 CDT 896ms Response
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 472
<?xml version="1.0"?>
<query type="c" xmlns="xxxxxxxxxxxxxx">
<product>
<name>screwdriver</name>
<price>5</price>
<stock>51</stock>
</product>
</query>
2021-10-13 12:15:12 CDT 947ms Request
POST / HTTP/1.1
Content-Type: application/x-www-form-urlencoded
Host: xxxxxxxxxxxxxxx
Content-Length: 515
Expect: 100-continue
The above is just a snippet, the file continues for over 14000 lines, repeating the same pattern.
Now when I run my AWK function on the whole file, it just returns the whole file back. But when I run it on a file that was created with (cat $FILE | head -200), It works as expected by returning:
2021-10-13 12:15:12 CDT 896ms Response
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 472
2021-10-13 12:15:13 CDT 075ms Response
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 3207
2021-10-13 12:15:13 CDT 208ms Response
HTTP/1.1 200 OK
Content-Type: application/xml
Content-Length: 4220
Why can I run this on a shortened file but when I run it on a longer version, it does not work? Even though its the same data in the file?
I am working on Ubuntu 18.04 LTS in Bash.
Thank you!
#markp-fuso's comment helped me. My input file had Windows line endings and I just needed to run the below command prior to executing the AWK:
tr -d '\15\32' < OGfile.txt > unixFile.txt
Then it ran as expected.
I received additional syntax help from the following question: Convert line endings
You can use this:
awk -v RS= -v ORS='\n\n' '/ms Response/'
Or this, to avoid a trailing blank line:
awk -v RS= '/ms Response/ && c++ {printf "\n"} /ms Response/'
If RS is an empty string, the record separator is becomes two or more contiguous new lines.

Download the files that are bigger than a certain size

I have Files.txt, which consists of a list of URLs of Excel files:
http://www.bcheadset.ca/applicant.xlsx
http://www.bcheadset.ca/fte.xlsx
http://www.iutox.org/TRTF_Matrix2012_Oct.xlsx
http://www.journalprices.com/2013_FinalSummaryForWeb.xlsx
http://www.camts.org/7__2013_Aircraft_Checklist.xlsx
http://www.nanotr11.org/poster_program.xlsx
http://www.vliruos.be/media/6352100/nss2015_annex_3_budget.xlsx
...
What I want to do is to download (by wget) the files that are bigger than 10Mo.
To verify if a file is bigger than 10Mo before dowloading it, we could use curl --head. For instance, curl --head http://www.bcheadset.ca/fte.xlsx returns
HTTP/1.1 200 OK
Content-Length: 3158847
Content-Type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
Last-Modified: Mon, 27 Jul 2015 22:16:45 GMT
Accept-Ranges: bytes
ETag: "ffb49fecb9c8d01:c05"
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Date: Tue, 06 Oct 2015 15:14:27 GMT
And from Content-Length, we can verify that it is bigger than 10Mo.
My question is how to automate this by a piece of Bash or Perl. Especially I don't know how to extract the Content-Length and does a comparison...
Could anyone help?
You can use LWP and perl and send a HEAD request, fetching response headers. Something like this:
#!/usr/bin/env perl
use strict;
use warnings;
use LWP;
use Data::Dumper;
my $request = HTTP::Request -> new ( HEAD => 'http://www.google.com' );
my $agent = LWP::UserAgent -> new;
my $response = $agent -> request ( $request );
print $response -> header ( 'Content-Length');
And then use LWP to do a GET instead of a HEAD you want that file.
In order to adapt to various HTTP forms and their content some type of pattern matching may be prudent. Here is a rough example of how to use glob matching in shell to accomplish this:
#!/bin/sh
BUFFER=$(curl --head http://www.bcheadset.ca/fte.xlsx )
HOLD=""
TAKENEXT=0
for i in $BUFFER
do case "$i" in
Content-Length:)
TAKENEXT=1
;;
*) if [ "$TAKENEXT" -eq 1 ]
then HOLD="$i"
break
fi
;;
esac
done
printf "SIZE was: %s\n" "$HOLD"
Here is a simpler method using grep:
SIZE=$(curl --head http://www.bcheadset.ca/fte.xlsx | grep 'Content-Length:' | awk '{print $2}')
printf "%s\n" "$SIZE"
.

how to pass variable value in POST JSON data in CURL command?

There are two fields which are unique , so i wrote a command to generate random value everytime the POST method is build and stored those values into the variables and pass those variables in the CURL command line. The script is below.
rcontactno=$(cat /dev/urandom | tr -dc '0-9' | fold -w 10 | head -n 1)
rfirstname=$(cat /dev/urandom | tr -dc 'a-zA-Z' | fold -w 10 | head -n
echo $rcontactno and $rfirstname
STATUS=$(curl -v -X POST -d '{"userName":"$rfirstname","contactNo":$rcontactno}' $1/restaurants/53/managers --header "Content-Type:application/json" --header "Accept:application/json" | grep HTTP | cut -d ' ' -f2 )
#Passing the URL using command-line argument
if [[ STATUS -eq 201 ]]; then
echo “Success”
exit 0
else
echo “Failed”
exit 127
fi
then i execute the script by
bash manager-post.sh
i get this type of error
> POST /restaurants/53/managers HTTP/1.1
> User-Agent: curl/7.37.1
> Host: my-url
> Content-Type:application/json
> Accept:application/json
> Content-Length: 54
>
} [data not shown]
* upload completely sent off: 54 out of 54 bytes
< HTTP/1.1 400 Bad Request
* Server Apache-Coyote/1.1 is not blacklisted
< Server: Apache-Coyote/1.1
< Content-Type: application/json;charset=UTF-8
< Transfer-Encoding: chunked
< Date: Fri, 16 Jan 2015 08:59:01 GMT
< Connection: close
<
{ [data not shown]
* Closing connection 0
“Failed”
but when i run the curl command without the bash script and explicitly mention the values of userName and contactNo , then it will execute successfully.
Where am i making mistake?
The single quotes don't allow variable expansion in the shell, so you need to use double quotes instead. Then you need to subsequently escape the double-quotes you want to send as-is.
A useful debugging technique is to add --trace-ascii dump.txt to your command line and inspect dump.txt after invoking curl to see that it matches what you intended to send.

How to concatenate lines from the same paragraph in bash?

I have a log file with the following lines:
vi test.log
23 Jan 01:29:33.498/GLOBAL/ser: RECEIVED message from 91.x.x.x:33583:
INVITE sip:39329172xxxx#sip.x SIP/2.0^M
Supported: ^M
Allow: INVITE, ACK, OPTIONS, CANCEL, BYE^M
Contact: sip:131400xxxx#91.x.x.x:33583^M
Via: SIP/2.0/UDP 91.x.x.x:33583;branch=z9hG4bKe65d47e555749b753faaf095c3256ec569bde77d37de66f62ff18bc40d492496^M
Call-id: ac755ea7e10821aa8174b2e5cd51d9e6^M
Cseq: 1 INVITE^M
From: sip:131400xxxx#sip.x;tag=5a541f1b2fd279cd0b8af3be3f67c7cf^M
ax-forwards: 70^M
To: sip:39329172xxxx#sip.x^M
Content-type: application/sdp^M
Content-length: 127^M
^M
v=0^M
o=anonymous 1327282173 1327282173 IN IP4 91.x.x.x^M
s=session^M
c=IN IP4 91.x.x.x^M
t=0 0^M
m=audio 5856 RTP/AVP 0^M
23 Jan 01:29:33.499/GLOBAL/ser: SENDING message to 91.x.x.x:33583:
SIP/2.0 100 trying -- your call is important to us^M
Via: SIP/2.0/UDP 91.x.x.x:33583;branch=z9hG4bKe65d47e555749b753faaf095c3256ec569bde77d37de66f62ff18bc40d492496^M
Call-id: ac755ea7e10821aa8174b2e5cd51d9e6^M
Cseq: 1 INVITE^M
From: sip:131400xxxx#sip.x;tag=5a541f1b2fd279cd0b8af3be3f67c7cf^M
To: sip:39329172xxxx#sip.x^M
Server: SSP v2.0.84^M
Content-Length: 0^M
^M
What I'd like to achieve is :
23 Jan 01:29:33.498/GLOBAL/ser: RECEIVED message from 91.x.x.x:33583:|INVITE sip:39329172xxxx#sip.x SIP/2.0|Supported:|Allow: INVITE, ACK, OPTIONS, CANCEL, BYE|Contact: sip:1314007008#91.x.x.x:33583|Via: SIP/2.0/UDP 91.x.x.x:33583;branch=z9hG4bKe65d47e555749b753faaf095c3256ec569bde77d37de66f62ff18bc40d492496|Call-id: ac755ea7e10821aa8174b2e5cd51d9e6|Cseq: 1 INVITE|From: sip:131400xxxx#sip.x;tag=5a541f1b2fd279cd0b8af3be3f67c7cf|Max-forwards: 70|To: sip:39329172xxxx#sip.x|Content-type: application/sdp|Content-length: 127|v=0|o=anonymous 1327282173 1327282173 IN IP4 91.x.x.x|s=session|c=IN IP4 91.x.x.x|t=0 0|m=audio 5856 RTP/AVP 0
23 Jan 01:29:33.499/GLOBAL/ser: SENDING message to 91.x.x.x:33583:|SIP/2.0 100 trying -- your call is important to us|Via: SIP/2.0/UDP 91.x.x.x:33583;branch=z9hG4bKe65d47e555749b753faaf095c3256ec569bde77d37de66f62ff18bc40d492496|Call-id: ac755ea7e10821aa8174b2e5cd51d9e6|Cseq: 1 INVITE|From: sip:131400xxxx#sip.x;tag=5a541f1b2fd279cd0b8af3be3f67c7cf|To: sip:39329172xxxx#sip.x|Server: SSP v2.0.84|Content-Length: 0
Basically all lines from the same paragraph (session) should be concatenated with "|" . A carriage return should then be added and next paragraph concatenated and so on. note that every new lines start with date & time.
So far I was only able to concatenate all the lines but unable to add the carriage return.. Any help would be much appreciated. Thank you.
You can use following awk script to do that:
awk '{if ($0 ~ /^\s*$/) {print line; line="";} else line=line $0 "|"}' file.txt
This is assuming that after end of para always a blank line appears same as your example.
Explanation:
$0 ~ /^\s*$/ - to check if line is completely blank or only has white spaces
if block executes when blank line appears. It prints line var and resets line to ""
else block is concatenating line variable with the current line of file and a pipe
This might work for you (although you data is not clear):
sed '1{h;d};/^[123]\?[0-9] [JFMASOND].. ..:..:..\..../{:a;x;s/\s*\n\+/|/g;s/.$//p;d};H;$ba;d' file
23 Jan 01:29:33.498/GLOBAL/ser: RECEIVED message from 91.x.x.x:33583:|INVITE sip:39329172xxxx#sip.x SIP/2.0|Supported:|Allow: INVITE, ACK, OPTIONS, CANCEL, BYE|Contact: sip:131400xxxx#91.x.x.x:33583|Via: SIP/2.0/UDP 91.x.x.x:33583;branch=z9hG4bKe65d47e555749b753faaf095c3256ec569bde77d37de66f62ff18bc40d492496|Call-id: ac755ea7e10821aa8174b2e5cd51d9e6|Cseq: 1 INVITE|From: sip:131400xxxx#sip.x;tag=5a541f1b2fd279cd0b8af3be3f67c7cf|ax-forwards: 70|To: sip:39329172xxxx#sip.x|Content-type: application/sdp|Content-length: 127|v=0|o=anonymous 1327282173 1327282173 IN IP4 91.x.x.x|s=session|c=IN IP4 91.x.x.x|t=0 0|m=audio 5856 RTP/AVP 0
23 Jan 01:29:33.499/GLOBAL/ser: SENDING message to 91.x.x.x:33583:|SIP/2.0 100 trying -- your call is important to us|Via: SIP/2.0/UDP 91.x.x.x:33583;branch=z9hG4bKe65d47e555749b753faaf095c3256ec569bde77d37de66f62ff18bc40d492496|Call-id: ac755ea7e10821aa8174b2e5cd51d9e6|Cseq: 1 INVITE|From: sip:131400xxxx#sip.x;tag=5a541f1b2fd279cd0b8af3be3f67c7cf|To: sip:39329172xxxx#sip.x|Server: SSP v2.0.84|Content-Length: 0

Resources