I want to pass file content as quoted programm argument with xargs to skip temporary file creation.
With temp file I can do like this:
myprogram > /tmp/lld.json
zabbix_sender -z 127.0.0.1 -s testhost -k llditem -o "`cat /tmp/lld.json`"
rm /tmp/lld.json
But I don't want this extra actions with /tmp/lld.json.
So I try to use xargs like this:
myprogram |
xargs -e -I'{}' zabbix_sender -z 127.0.0.1 -s testhost -k llditem -o "'{}'"
guiding with xargs manpage:
-I replace-str
-e[eof-str] ... If eof-str is omitted, there is no end of file string..
http://man7.org/linux/man-pages/man1/xargs.1.html
But xargz executes zabbix-sender many times with each of the lines.
I guess that -I and -e options are mutually exclusive options. But I also assume that I misinterpret the xargs manual..
Would this work?
zabbix_sender -z 127.0.0.1 -s testhost -k llditem -o "`myprogram`"
If you insist on using xargs to do exactly that, then use -0:
myprogram | xargs -0 -I{} zabbix_sender -z 127.0.0.1 -s testhost -k llditem -o "'{}'"
Related
I want to check some file on the remote website.
Here is bash command to generate commands that calculate the file md5
[root]# head -n 3 zrcpathAll | awk '{print $3}' | xargs -I {} echo wget -q -O - -i {}e \| md5sum\;
wget -q -O - -i https://example.com/zrc/3d2f0e76e04444f4ec456ef9f11289ec.zrce | md5sum;
wget -q -O - -i https://example.com/zrc/e1bd7171263adb95fb6f732864ceb556.zrce | md5sum;
wget -q -O - -i https://example.com/zrc/5300b80d194f677226c4dc6e17ba3b85.zrce | md5sum;
Then I pipe the outputed commands to bash, but only the first command was executed.
[root]# head -n 3 zrcpathAll | awk '{print $3}' | xargs -I {} echo wget -q -O - -i {}e \| md5sum\; | bash -v
wget -q -O - -i https://example.com/zrc/3d2f0e76e04444f4ec456ef9f11289ec.zrce | md5sum;
3d2f0e76e04444f4ec456ef9f11289ec -
[root]#
Would you please try the following instead:
while read -r _ _ url _; do
wget -q -O - "$url"e | md5sum
done < <(head -n 3 zrcpathAll)
we should not put -i in front of "$url" here.
[Explanation about -i option]
Manpage of wget says:
-i file
--input-file=file
Read URLs from a local or external file. [snip]
If this function is used, no URLs need be present on the command line. [snip]
If the file is an external one, the document will be automatically treated as html if the Content-Type matches text/html.
Furthermore, the file's location will be implicitly used as base
href if none was specified.
where the file will contain line(s) of url such as:
https://example.com/zrc/3d2f0e76e04444f4ec456ef9f11289ec.zrce
https://example.com/zrc/e1bd7171263adb95fb6f732864ceb556.zrce
https://example.com/zrc/5300b80d194f677226c4dc6e17ba3b85.zrce
Whereas if we use the option as -i url, wget first
downloads the url as a file which contains the lines of urls
as above. In our case, the url is the target to download itself,
not the list of urls, wget causes an error: No URLs found in url.
Even if the wget fails, why the command outputs just one line, not
three lines as the result of md5sum?
This seems to be because the head command immediately flushes the remaining
lines when the piped subprocess fails.
I am trying to automate a procedure where the system will fetch the contents of a file (1 Url per line), use wget to grab the files from the site (https folder) and then remove the line from the file.
I have made several tries but the sed part (at the end) cannot understand the string (I tried escaping characters) and remove it from that file!
cat File
https://something.net/xxx/data/Folder1/
https://something.net/xxx/data/Folder2/
https://something.net/xxx/data/Folder3/
My line of code is:
cat File | xargs -n1 -I # bash -c 'wget -r -nd -l 1 -c -A rar,zip,7z,txt,jpg,iso,sfv,md5,pdf --no-parent --restrict-file-names=nocontrol --user=test --password=pass --no-check-certificate "#" -P /mnt/USB/ && sed -e 's|#||g' File'
It works up until the sed -e 's|#||g' File part..
Thanks in advance!
Dont use cat if it's posible. It's bad practice and can be problem with big files... You can change
cat File | xargs -n1 -I # bash -c
to
for siteUrl in $( < "File" ); do
It's be more correct and be simpler to use sed with double quotes... My variant:
scriptDir=$( dirname -- "$0" )
for siteUrl in $( < "$scriptDir/File.txt" )
do
if [[ -z "$siteUrl" ]]; then break; fi # break line if him empty
wget -r -nd -l 1 -c -A rar,zip,7z,txt,jpg,iso,sfv,md5,pdf --no-parent --restrict-file-names=nocontrol --user=test --password=pass --no-check-certificate "$siteUrl" -P /mnt/USB/ && sed -i "s|$siteUrl||g" "$scriptDir/File.txt"
done
#beliy answers looks good!
If you want a one-liner, you can do:
while read -r line; do \
wget -r -nd -l 1 -c -A rar,zip,7z,txt,jpg,iso,sfv,md5,pdf \
--no-parent --restrict-file-names=nocontrol --user=test \
--password=pass --no-check-certificate "$line" -P /mnt/USB/ \
&& sed -i -e '\|'"$line"'|d' "File.txt"; \
done < File.txt
EDIT:
You need to add a \ in front of the first pipe
I believe you just need to use double quotes after sed -e. Instead of:
'...&& sed -e 's|#||g' File'
you would need
'...&& sed -e '"'s|#||g'"' File'
I see what you trying to do, but I dont understand the sed command including pipes. Maybe some fancy format that I dont understand.
Anyway, I think the sed command should look like this...
sed -e 's/#//g'
This command will remove all # from the stream.
I hope this helps!
Here's the command I'm currently running:
curl 'http://test.com/?id=12345' | grep -o -P '(?<=content="2;url=).*?(?=")'
The response from this command is a URL, like this:
$ curl 'http://test.com/?id=12345' | grep -o -P '(?<=content="2;url=).*?(?=")'
http://google.com
I want to use whatever that URL is to essentially do this:
curl 'http://test.com/?id=12345' | grep -o -P '(?<=content="2;url=).*?(?=")' | curl 'http://google.com'
Is there any simple way to do this all in one line?
Use xargs with a place holder for the output from stdin with the -I{} flag as below. The -r flag is to ensure the curl command is not invoked on a empty output from previous grep output.
curl 'http://test.com/?id=12345' | grep -o -P '(?<=content="2;url=).*?(?=")' | xargs -r -I{} curl {}
A small description about the flags, -I and -r from the GNU xargs man page,
-I replace-str
Replace occurrences of replace-str in the initial-arguments with
names read from standard input.
-r, --no-run-if-empty
If the standard input does not contain any nonblanks, do not run
the command. Normally, the command is run once even if there is
no input. This option is a GNU extension
(or) if you are looking for a bash approach without other tools,
curl 'http://test.com/?id=12345' | grep -o -P '(?<=content="2;url=).*?(?=")' | while read line; do [ ! -z "$line" ] && curl "$line"; done
I have a script that logs in to a remote host to pull a directory listing to later present options to the user. It was all working perfectly, until some of the directories started having spaces in them. I have tried several syntaxes and googled the life out of this and I am now at the end of my tether. The original command was this:
SERVERDIRS=($(sshpass -p $PASS ssh -oStrictHostKeyChecking=no $USER#$SERVER ls -l --time-style="long-iso" $FROMFOLDER | egrep '^d' | awk '{print $8}'))
I first off changed this code to be able to read the spaces like this:
SERVERDIRS=($(sshpass -p $PASS ssh -oStrictHostKeyChecking=no $USER#$SERVER ls -l --time-style="long-iso" $FROMFOLDER | egrep '^d' | cut -d' ' -f8-))
However This resulted in each word being recognised as a variable. I have tried many ways to try to solve this, two of which were:
SERVERDIRS=($(sshpass -p $PASS ssh -oStrictHostKeyChecking=no $USER#$SERVER ls -d $FROMFOLDER* |rev| cut -d'/' -f1|rev|sed s/^/\"/g|sed s/$/\"/g))
SERVERDIRS=($(sshpass -p $PASS ssh -oStrictHostKeyChecking=no $USER#$SERVER ls -d $FROMFOLDER* |rev| cut -d'/' -f1|rev|sed 's/ /\\ /g'))
SERVERDIRS=(`sshpass -p $PASS ssh -oStrictHostKeyChecking=no $USER#$SERVER ls -d $FROMFOLDER* |rev| cut -d'/' -f1|rev|sed 's/ /\\ /g'`)
How can I resolve these directories in to separate elements correctly?
If you're trying to read one array value per line instead of space-separated, then $() syntax won't help. Try readarray (Bash 4):
readarray SERVERDIRS < <(sshpass -p $PASS ssh -oStrictHostKeyChecking=no $USER#$SERVER ls -l --time-style="long-iso" $FROMFOLDER | egrep '^d' | cut -d' ' -f8-)
or assign IFS and read with -d, -r, and -a set:
IFS=$'\n' read -d '' -r -a SERVERDIRS < <(sshpass -p $PASS ssh -oStrictHostKeyChecking=no $USER#$SERVER ls -l --time-style="long-iso" $FROMFOLDER | egrep '^d' | cut -d' ' -f8-)
or, really, any other answer to this SO question.
If you're unfamiliar with <() syntax, it's known as process substitution and will allow your variable to be set in your current environment rather than the instantly-discarded subshell that a pipe would create.
Bear in mind that this process is a little dangerous; filenames can also contain newlines, so it's usually much preferred to use find ... -print0.
If you only need to list directories, try this
ls -d /usr/local/src/*/
or
ls -d /path/to/your/directory/*/
You can then loop through all directories
#!/bin/bash
aa=`ls -d /usr/local/src/*/`
for dir in "${aa}[#]"
do
echo "$dir"
done
This works if dir names contain spaces.
I need an output for multiple grep commands.
patterns: ([^"#]+)
wget -q -O - http://www.site1.com | grep -o -E -m 1 'site1content = "([^"#]+)"'
wget -q -O - http://www.site2.com | grep -o -E -m 1 'site2content"([^"#]+)"
.........
Output file:
http://www.site1.com***pattern
http://www.site2.com***pattern
Just redirect the output of your commands to a file.
wget -q -O - http://www.site1.com | grep -o -E -m 1 'site1content = "([^"#]+)"' > output.txt
wget -q -O - http://www.site2.com | grep -o -E -m 1 'site2content"([^"#]+)"' >> output.txt
> overwrites old content and >> appends to the end of the file.
Edit:
Not pretty but a quick and dirty solution might be
echo 'http://www.site1.com***'`wget -q -O - http://www.site1.com | grep -o -E -m 1 'site1content = "([^"#]+)"'` > output.txt
(untested)
As is the output you got from the above commends consists only of the pattern found because of the -o parameter:
http://explainshell.com/explain?cmd=grep+-o
I suggest using the above site for an explanation.