Is anyone of you familiar with Dropbox Uploader (https://github.com/andreafabrizi/Dropbox-Uploader) or UNIX sed command.
I would like to get this script to work so it would return an sorted list from dropbox based on modified date.
Command ./droptobox list Test outputs:
Listing "/Test"... DONE
[F] config.bmp
[F] igs.bin
If to do some modifications to script (echoing out $DIR_CONTENT), I can see from the output that there certainly is information I need.
{"revision": 37, "rev": "2514cf1330", "thumb_exists": true, "bytes": 824, "modified": "Thu, 07 Nov 2013 16:14:59 +0000", "client_mtime": "Thu, 07 Nov 2013 16:14:59 +0000", "path": "/Test/config.bmp", "is_dir": false, "icon": "page_white_picture", "root": "dropbox", "mime_type": "image/x-ms-bmp", "size": "824 bytes"}
{"revision": 38, "rev": "2614cf1330", "thumb_exists": false, "bytes": 86, "modified": "Thu, 07 Nov 2013 16:15:18 +0000", "client_mtime": "Thu, 07 Nov 2013 16:15:18 +0000", "path": "/Test/igs.bin", "is_dir": false, "icon": "page_white", "root": "dropbox", "mime_type": "application/octet-stream", "size": "86 bytes"}], "size": "0 bytes"}
I suppose this is the line that responsible for it:
echo "$DIR_CONTENT" | sed -n 's/.*"path": *"\([^"]*\)",.*"is_dir": *\([^"]*\),.*/\1:\2/p' > $RESPONSE_FILE
The problem is that I ain't that familiar with sed, if I try to do some modifications I won't get any output.
Modified line (have no idea if I'm moving in right direction):
echo "$DIR_CONTENT" | sed -n 's/.*"path": *"\([^"]*\)",.*"is_dir": *\([^"]*\),.*"modified": *\([^"]*\),.*/\1:\2\3/p' > $RESPONSE_FILE
But ideally would it be possible to get it to display modified information and sort the results based on it?
Also other part of the db_list function:
#For each line...
while read -r line; do
local FILE=${line%:*}
FILE=${FILE##*/}
local TYPE=${line#*:}
if [[ $TYPE == "false" ]]; then
echo -ne " [F] $FILE\n"
else
echo -ne " [D] $FILE\n"
fi
done < $RESPONSE_FILE
Two things I see:
The "is_dir" value is a boolean and does not have quotes, so change
"is_dir": *\([^"]*\),
to
"is_dir": *\([^,]*\),
"modified" occurs before "path", so you need to write (accounting for quotes as well):
echo "$DIR_CONTENT" | sed -n 's/.*"modified": *"\([^"]*\)",.*"path": *"\([^"]*\)",.*"is_dir": *\([^,]*\),.*/\2:\3 \1/p' > $RESPONSE_FILE
Also, bash has a handy "here-string" feature, so you don't need to "echo"
sed -n '...' <<< "$DIR_CONTENT" > "$RESPONSE_FILE"
Related
I have the following json format, basically it is a huge file with several of such entries.
[
{
"id": "kslhe6em",
"version": "R7.8.0.00_BNK",
"hostname": "abacus-ap-hf-test-001:8080",
"status": "RUNNING",
},
{
"id": "2bkaiupm",
"version": "R7.8.0.00_BNK",
"hostname": "abacus-ap-hotfix-001:8080",
"status": "RUNNING",
},
{
"id": "rz5savbi",
"version": "R7.8.0.00_BNK",
"hostname": "abacus-ap-hf-test-005:8080",
"status": "RUNNING",
},
]
I wanted to fetch all the hostname values that starts with "abacus-ap-hf-test" and without ":8080" into a variable and then wanted to use those values for further commands over a for loop something like below. But, am bit confused how can I extract such informaion.
HOSTAME="abacus-ap-hf-test-001 abacus-ap-hf-test-005"
for HOSTANAME in $HOSTNAME
do
sh ./trigger.sh
done
The first line command update to this:
HOSTAME=$(grep -oP 'hostname": "\K(abacus-ap-hf-test[\w\d-]+)' json.file)
or if you sure that the hostname end with :8080", try this:
HOSTAME=$(grep -oP '(?<="hostname": ")abacus-ap-hf-test[\w\d-]+(?=:8080")' json.file)
you will find that abacus-ap-hf-test[\w\d-]+ is the regex, and other strings are the head or the end of the regex content which for finding result accuracy.
Assuming you have valid JSON, you can get the hostname values using jq:
while read -r hname ; do printf "%s\n" "$hname" ; done < <(jq -r .[].hostname j.json)
Output:
abacus-ap-hf-test-001:8080
abacus-ap-hotfix-001:8080
abacus-ap-hf-test-005:8080
I have the following JSON file:
{
"1":
{
"media_content":"test3.xspf"
},
"2":
{
"media_content":"test3.xspf"
}
}
In the terminal, using bash as shell, I can execute the following commands:
export schedules="1"
echo $(jq '.[env.schedules]["media_content"]' json_file.json)
Which results in outputing this:
test3.xspf
So it works as expected, but when I place that jq command in a script and run it, it just returns null.
I did echo the values of schedules to make sure the value is non-null inside the script, and it is ok:
echo $schedules
But I did not manage to find the reason, why this command works when run directly in shell and does not work when run in script.
I run the script in the following ways:
bash script.sh
./script.sh
PS: yes, I did offer execute permission: chmod +x script.sh
HINT: env.schedules represents the environment variable 'schedules', and I did make sure that it is assigned in the script before calling jq.
EDIT: I am posting now a whole script, specifying the files tree.
There is one directory containing:
script.sh
json_file.json
static.json
script.sh:
export zone=$(cat static.json | jq '.["1"]');
echo "json block: "$zone
export schedules="$(echo $zone | jq '.schedules')"
echo "environment variable: "$schedules
export media_content=$(jq '.[env.schedules]["media_content"]' json_file.json)
echo "What I want to get: \"test3.xspf\""
echo "What I get: "$media_content
json_file.json:
{
"1":
{
"media_content":"test3.xspf"
},
"2":
{
"media_content":"test3.xspf"
}
}
static.json:
{
"1":
{
"x": "0",
"y": "0",
"width": "960",
"height": "540",
"schedules":"1"
}
}
If I run the script, it displays:
json block: { "x": "0", "y": "0", "width": "960", "height": "540", "schedules": "1" }
environment variable: "1"
What I want to get: "test3.xspf"
What I get: null
If I hardcode the variable:
export schedules="1"
The problem no longer occurs
The problem is simple.
It's not jq's fault.
It the unproper way the schedule's value is piped to the next command.
You have to remove the "s that surround the variable's value, add the second command that uses sed to do that:
export schedules="$(echo $zone | jq '.schedules')"
schedules=$( echo $schedules | sed s/\"//g )
Long answer
Let's see:
here schedules is a string and echo shows its value as being 1:
export schedules="1" ; echo $schedules
here even though double quotes are not mentioned:
export schedules=1 ; echo $schedules
But the result from this also generates additional "s:
export schedules=$(echo $zone | jq '.schedules')
If you print it now you will see additional "s:
echo $schedules # "1"
So just remove the "s from the value:
schedules=$( echo $schedules | sed s/\"//g )
I make a script what will check the asda home delivery slots from the api
Here it is, I call it get_slots.sh
You have to start tor or if you don't then you have to get rid of the line about sock5 hostname (you can see tor port number in command line with ps) but if you don't use tor they might cancel your account if they get narky about you polling their website
obviously u have to change the vars at the top
Query parameters and api url was kind of found out with inspector in chrome while using their normal java script thingy for joe public, top secret NOT
#!/bin/bash
my_postcode="SW1A1AA" # CHANGEME
account_id=18972357834 # JUST INVENT A NUMBER
order_id=22985263473 # LIKEWISE
ua='user_agent_I_want_to_fake'
my_tor_port=9150
#----------------
#ftype="POPUP"
#ftype="INSTORE_PICKUP"
ftype="DELIVERY"
format="%Y-%m-%dT00:00:00+01:00"
start_date=$(date "+$format")
end_date=$(date -d "+16 days" "+$format")
read -r -d '' data <<EOF
{
"data": {
"customer_info": {
"account_id": "$account_id"
},
"end_date": "$end_date",
"order_info": {
"line_item_count": 0,
"order_id": "$order_id",
"restricted_item_types": [],
"sub_total_amount": 0,
"total_quantity": 0,
"volume": 0,
"weight": 0
},
"reserved_slot_id": "",
"service_address": {"postcode":"$my_postcode"},
"service_info": {
"enable_express": false,
"fulfillment_type": "$ftype"
},
"start_date": "$start_date"
},
"requestorigin": "gi"
}
EOF
data=$(echo $data | tr -d ' ')
url='https://groceries.asda.com/api/v3/slot/view'
referer='https://groceries.asda.com/checkout/book-slot?origin=/account/orders'
curl -s \
--socks5-hostname localhost:$my_tor_port \
-H "Content-type: application/json; charset=utf-8" \
-H "Referer: $referer" \
-A "$ua" \
-d "$data" \
$url \
| python -m json.tool
anyway now i make another script to keep running it and mail me if any slots are available,
more vars u need 2 change at the top of this one
#!/bin/sh
me="my#email.address"
my_smtp_server="smtp.myisp.net:25"
#------------------------------------
mailed=0
ftmp=/tmp/slots.$$
while true
do
date
f=slots/`date +%Y%m%d/%H/%Y-%m-%d_%H%M%S`.json
d=`dirname $f`
[ -d $d ] || mkdir -p $d
./get_slots.sh > $f
if egrep -B1 'status.*"AVAILABLE"' $f > $ftmp
then
echo "found"
if [ $mailed -eq 0 ]
then
dates=`perl -nle '/start_time.*2020-(..-..T..):/ && print $1' $ftmp`
mailx \
-r "$me" -s "asda on $dates lol" \
-S smtp="$my_smtp_server" "$me" < $ftmp
echo "mailed"
mailed=1
fi
fi
sleep 120
done
so i kind of naughty here cos i need the timestamp for slots with status available to put in the email ... and i really cba to parse the json properly so i just rely on its in the line before the status
like the pretty printed json puts the stuff in alphfabetical order and comes out with something like
"slot_info": {
STUFF
"slot_type": null,
"start_time": "2020-06-10T19:00:00Z",
"status": "AVAILABLE",
"total_discount": 0.0,
"total_premium": 0.0,
MORE STUFF
so yeah all i do is egrep -B1
oh yeah i also naughty hard coded 2020 not do proper regex for the year, cos if this is all still going on after 2020 i might as well just starve anyway so dont want to over engineer it
anyway as you can see once it already mailed me it still keeps running cos i want to store the json files and may be analise them laters , it just dont mail me again after that unless i re start it
anyway my question is my script only check every two minutes and i want it to check more often so i can beat people.
okay sorted it the sleep 120 is 2 minutes i thought it was 1.2 minutes sorry forgot a minute is 60 seconds not 100 lol
oh yeah dont worry im not going to do this every 5 seconds like....!
just now i know the sleep is working properly i can change it to 60, still no more often than a lot of the people sat there re loading it manually believe me ......
I am trying to count the occurence of a particular line in a file using shell script. The content of the file is:
Mon Dec 23 06:21:00 2019 [pid 3294] [ftpuser] FTP response: Client "127.0.0.1", "230 Login successfull."
Mon Dec 23 06:21:00 2019 [pid 3294] [ftpuser] FTP response: Client "127.0.0.1", "230 Login successfull."
Mon Dec 23 06:21:00 2019 [pid 3294] [ftpuser] FTP response: Client "127.0.0.1", "230 Login successfull."
Mon Dec 23 06:21:00 2019 [pid 3294] [ftpuser] FTP response: Client "127.0.0.1", "230 Login successfull."
The shell script code is:
file="vsftpd1.log"
count=0
while read str; do
echo $str
if [[ $str == *"230 Login successfull."* ]];
then
count=$((count+1))
fi
done < $file
echo $count
The output of the last line should be four. I have noticed that if condition is executing only one time. But it should run four times. Please help me to find mistake in the code, if any.
Thankxx in advance!
You could simplify that with grep (match occurences) and its -c (count occurences).
#!/bin/sh
file=vsftpd1.log
grep -c '230 Login successfull.' "$file"
If you need to capture the number into a variable:
count=$(grep -c '230 Login successfull.' "$file")
echo $count
That entire (rather inefficient) snippet of code can be replaced with the much simpler:
count=$(grep -c '230 Login successfull.' vsftpd1.log)
echo ${count}
As more explanation, grep -c returns the line count of the lines that match your search pattern. There is no need for a bash-based loop which reads one line at a time and checks the pattern - this is likely to be much faster in the grep executable.
The following transcript shows this in action:
pax:~$ grep '230 Login successfull.' qq.in
Mon Dec 23 06:21:00 2019 [pid 3294] [ftpuser] FTP response: Client "127.0.0.1", "230 Login successfull."
Mon Dec 23 06:21:00 2019 [pid 3294] [ftpuser] FTP response: Client "127.0.0.1", "230 Login successfull."
Mon Dec 23 06:21:00 2019 [pid 3294] [ftpuser] FTP response: Client "127.0.0.1", "230 Login successfull."
Mon Dec 23 06:21:00 2019 [pid 3294] [ftpuser] FTP response: Client "127.0.0.1", "230 Login successfull."
pax:~$ count=$(grep -c '230 Login successfull.' qq.in) ; echo ${count}
4
I've got this script that runs every time my Ubuntu machine wakes up and goes to sleep.
#!/bin/sh
if [ "${1}" == "pre" ]; then
# Do the thing you want before suspend here, e.g.:
echo "we are suspending at $(date)..." > /home/albin/stuff/suspend_test
elif [ "${1}" == "post" ]; then
# Do the thing you want after resume here, e.g.:
echo "...and we are back from $(date)" >> /home/albin/stuff/suspend_test
else
echo ".2..neither pre or post . $1 . $(date)" >> /home/albin/stuff/suspend_test
fi
This is found in the output file:
.2..neither pre or post . pre . tis 11 sep 2018 20:44:42 CEST
.2..neither pre or post . post . tis 11 sep 2018 20:45:06 CEST
.2..neither pre or post . pre . tis 11 sep 2018 22:12:52 CEST
.2..neither pre or post . post . ons 12 sep 2018 06:55:21 CEST
.2..neither pre or post . pre . ons 12 sep 2018 06:55:22 CEST
.2..neither pre or post . post . ons 12 sep 2018 06:55:43 CEST
.2..neither pre or post . pre . ons 12 sep 2018 07:13:28 CEST
.2..neither pre or post . post . ons 12 sep 2018 07:14:00 CEST
I've checked multiple guides on how to write bash conditionals without finding any issues. The $1 variable is available but ignored in the if statements
You are using == instead of = inside [ ], but that's a bashism. Recent versions of Ubuntu use dash as /bin/sh, and it does not support ==; it gets an error, which is interpreted as the test failing:
$ if [ foo == foo ]; then echo "The strings match"; else echo "The strings do *not* match"; fi
dash: 1: [: foo: unexpected operator
The strings do *not* match
Solution: switch to =
$ if [ foo = foo ]; then echo "The strings match"; else echo "The strings do *not* match"; fi
The strings match
If you're going to use /bin/sh instead of /bin/bash, you really need to watch out for bashisms. There's a good page on this in the Ubuntu wiki. Either that, or switch to /bin/bash.
P.s. If you're going to use bash extensions, I'd recommend using [[ ]] instead of [ ] for conditionals -- it fixes most of the syntactic oddities of [ ], and adds some useful new capabilities like pattern and regular expression matching.
Mostly likely, I think you got your redirection output wrong on line 4.
Must be :
echo "we are suspending at $(date)..." >> /home/albin/stuff/suspend_test
instead of
echo "we are suspending at $(date)..." > /home/albin/stuff/suspend_test
Also, I think $1 is neither equal to 'pre' or 'post'. From looking at the output you posted, I can't understand from why these strings "tis" and "ons" have come from.
Probably caller of this script passing you other arguments than post or pre.
Change log writing to use >> instead of > (to prevent overwriting log all time) and try add echo “${1}” >> log at first line, then you can see what arguments you got in your script