I want to use curl to make a bunch of http requests in parallel for some performance testing of a server.
The request requires a MAC address, which I would like to count upwards in the following way.
00:00:00:00:AA:BB
^ ^
| \___ count from 00 to ff (hex)
\______ count from 00 to ff (hex)
BB shall overflow to AA when it is getting larger than 0xFF
What I got so far is, incrementing BB and send the request in a parallel way.
printf "%02x\n" {0..255} | xargs -I{} -P4 -- \
curl -s \
http://nginx/api/onboard/01:02:03:04:00:{}
This works fine, but I need to request more than 256 times.
xargs is BusyBox v1.31.1
I found a solution. It is actually not xargs related as I thought first.
printf "%04x\n" {0..1000} | \
sed -r 's/([[:xdigit:]]{2})([[:xdigit:]]{2})/\1:\2/g' | \
xargs -I{} -P4 -- \
curl -k -s \
https://nginx/api/onboard/21:12:12:12:{}
I'm creating the needed part of the MAC with a combination of printf and sed. There is probably a better way but it solves my problem.
Related
This question already has answers here:
How can I parse a YAML file from a Linux shell script?
(23 answers)
Closed 26 days ago.
Here is our yaml:
network:
ethernets:
ens160:
addresses:
- 10.200.2.11/22
gateway4: 10.200.0.1
nameservers:
addresses:
- 8.8.8.8
- 4.4.4.4
search:
- cybertax.live
version: 2
I want to change the dns severs only.
From:
- 8.8.8.8
- 4.4.4.4
to:
- 10.10.10.10
- 10.10.10.11
How can I do this? Note: we cannot use or install yq so this needs to be done through sed or awk. Also, yes I know, this is not recommended, but its what needs to be done right now.
What I have tried so far:
sed -i '/ addresses:/,/ search:/ s/^/# /' $netplan_yaml
sed -i '/ nameservers:/a\ \ \ \ \ \ \ \ addresses:' $netplan_yaml
for i in ${!asar_dns[#]}; do
sed -i "/ addresses:/a\ \ \ \ \ \ \ \ - ${asar_dns[$i]}" $netplan_yaml
done
But this does three things wrong (that I can see).
It matches between addresses and search including the line wiht addresses and search. I only want what is AFTER addresses, and BEFORE search.
It puts the DNS addresses in the associative array between the older addresses that is commented out anywhere there is an "addresses". I dont want to do that on the commented out line.
i dont like how I have to use \ \ \ \ \ \ would much rather use a .* if possible but also need to use the addresses in the associative array.
I found a very hacky solution but it works. Open for feedback.
netplan_yaml=/etc/netplan/00-installer-config.yaml
baddns="$(sed -n '/.*nameservers:/,/.*search:/p' /etc/netplan/00-installer-config.yaml | grep -v 'nameservers\|addresses' | grep -v 'search' | grep -v 10.200 | awk '{print$2}')"
mapfile -t arr_baddns <<<$baddns
for i in ${arr_baddns[#]}; do
sed -i "/ - $i/s/^/#/g" $netplan_yaml
done
for i in ${!asar_dns[#]}; do
sed -i "/ addresses:/a\ \ \ \ \ \ \ \ - ${asar_dns[$i]}" $netplan_yaml
done
If ed is available/acceptable, with your given input.
Something like.
ed -s file.yaml <<-'EOF'
g/nameservers:/;/addresses:/;/search:/-s/[[:digit:].]\{1,\}/10.10.10.11/\
-s/[[:digit:].]\{1,\}/10.10.10.10/
,p
Q
EOF
Using variables for the values, something like:
#!/bin/sh
dns1=10.10.10.10
dns2=10.10.10.11
ed -s file.yaml <<-EOF
g/nameservers:/;/addresses:/;/search:/-s/[[:digit:].]\{1,\}/$dns2/\\
-s/[[:digit:].]\{1,\}/$dns1/
,p
Q
EOF
Change Q to w if in-place editing is required, ala sed -i
Remove the ,p to silence the output.
I want to generate System.map from vmlinuz,cause most of machines don't have the file System.map.In fact,vmlinuz are compressed to vmlinuz or bzImage.
It's any tool or script can do this?
I tried:
dd if=/boot/vmlinuz skip=`grep -a -b -o -m 1 -e $'\x1f\x8b\x08\x00' /boot/vmlinuz | cut -d: -f 1` bs=1 | zcat > /tmp/vmlinux
It was failed:
zcat: stdin: not in gzip format
32769+0 records in
32768+0 records out
To extract the uncompressed kernel from the kernel image, you can use the extract-vmlinux script from the scripts directory in the kernel tree (available at least in kernel version 3.5) (if you get an error like
mktemp: Cannot create temp file /tmp/vmlinux-XXX: Invalid argument
you need to replace $(mktemp /tmp/vmlinux-XXX) by $(mktemp /tmp/vmlinux-XXXXXX) in the script). The command is /path/to/kernel/tree/scripts/extract-vmlinux <kernel image> >vmlinux.
If the extracted kernel binary contains symbol information, you should¹ be able to create the System.map file using the mksysmap script from the same subdirectory. The command here is NM=nm /path/to/kernel/tree/scripts/mksysmap vmlinux System.map.
¹ The kernel images shipped with my distribution seem to be stripped, so the script was not able to get the symbols.
As Abrixas2 wrote, you will need a kernel image with symbol information in order to create System.map files and a packed vmlinuz image is not likely to have symbols in it. I can, however, verify that the script in your original post works with '-e' replaced with '-P' and '$' dropped, i.e.,
$ dd if=vmlinuz-3.8.0-19-generic skip=`grep -a -b -o -m 1 -P '\x1f\x8b\x08\x00' vmlinuz-3.8.0-19-generic | cut -d: -f 1` bs=1 | zcat > /tmp/vmlinux
gzip: stdin: decompression OK, trailing garbage ignored
I'm on ubuntu linux.
you can change $'\037\213\010\000' to "$(echo '\037\213\010\000')" in sh
bash$ N=$(grep -abo -m1 $'\037\213\010\000' vmlinuz-4.13.0-37-generic | awk -F: '{print $1+1}') &&
tail -c +$N vmlinuz-4.13.0-37-generic | gzip -d > /tmp/vmlinuz
try this :
dd if=vmlinuz bs=1 skip=24584 | zcat > vmlinux
with
24584 = 24576 + 8
when
od -A d -t x1 vmlinuz | grep '1f 8b 08 00'
gives
....... 0 1 2 3 . . . . 8
0024576 24 26 27 00 ae 21 16 00 1f 8b 08 00 7f 2f 6b 45
enjoy !
Is it possible to Wget a page's title from the command line?
input:
$ wget http://bit.ly/rQyhG5 <<code>>
output:
If it’s broke, fix it right - Keeping it Real Estate. Home
This script would give you what you need:
wget --quiet -O - http://bit.ly/rQyhG5 \
| sed -n -e 's!.*<title>\(.*\)</title>.*!\1!p'
But there are lots of situations where it breaks, including if there is a <title>...</title> in the body of the page, or if the title is on more than one line.
This might be a little better:
wget --quiet -O - http://bit.ly/rQyhG5 \
| paste -s -d " " \
| sed -e 's!.*<head>\(.*\)</head>.*!\1!' \
| sed -e 's!.*<title>\(.*\)</title>.*!\1!'
but it does not fit your case as your page contains the following head opening:
<head profile="http://gmpg.org/xfn/11">
Again, this might be better:
wget --quiet -O - http://bit.ly/rQyhG5 \
| paste -s -d " " \
| sed -e 's!.*<head[^>]*>\(.*\)</head>.*!\1!' \
| sed -e 's!.*<title>\(.*\)</title>.*!\1!'
but there is still ways to break it, including no head/title in the page.
Again, a better solution might be:
wget --quiet -O - http://bit.ly/rQyhG5 \
| paste -s -d " " \
| sed -n -e 's!.*<head[^>]*>\(.*\)</head>.*!\1!p' \
| sed -n -e 's!.*<title>\(.*\)</title>.*!\1!p'
but I am sure we can find a way to break it. This is why a true xml parser is the right solution, but as your question is tagged shell, the above it the best I can come with.
The paste and the 2 sed can be merged in a single sed, but is less readable. However, this version has the advantage of working on multi-line titles:
wget --quiet -O - http://bit.ly/rQyhG5 \
| sed -n -e 'H;${x;s!.*<head[^>]*>\(.*\)</head>.*!\1!;T;s!.*<title>\(.*\)</title>.*!\1!p}'
Update:
As explain in the comments, the last sed above uses the T command which is a GNU extension. If you do not have a compatible version, you can use:
wget --quiet -O - http://bit.ly/rQyhG5 \
| sed -n -e 'H;${x;s!.*<head[^>]*>\(.*\)</head>.*!\1!;tnext;b;:next;s!.*<title>\(.*\)</title>.*!\1!p}'
Update 2:
As above still not working on Mac, try:
wget --quiet -O - http://bit.ly/rQyhG5 \
| sed -n -e 'H;${x;s!.*<head[^>]*>\(.*\)</head>.*!\1!;tnext};b;:next;s!.*<title>\(.*\)</title>.*!\1!p'
and/or
cat << EOF > script
H
\$x
\$s!.*<head[^>]*>\(.*\)</head>.*!\1!
\$tnext
b
:next
s!.*<title>\(.*\)</title>.*!\1!p
EOF
wget --quiet -O - http://bit.ly/rQyhG5 \
| sed -n -f script
(Note the \ before the $ to avoid variable expansion.)
It seams that the :next does not like to be prefixed by a $, which could be a problem in some sed version.
The following will pull whatever lynx thinks the title of the page is, saving you from all of the regex nonsense. Assuming the page you are retrieving is standards compliant enough for lynx, this should not break.
lynx -dump example.com | sed '2q;d'
So I have the equivalent of a list of files being output by another command, and it looks something like this:
http://somewhere.com/foo1.xml.gz
http://somewhere.com/foo2.xml.gz
...
I need to run the XML in each file through xmlstarlet, so I'm doing ... | xargs gzip -d | xmlstarlet ..., except I want xmlstarlet to be called once for each line going into gzip, not on all of the xml documents appended to each other. Is it possible to compose 'gzip -d' 'xmlstarlet ...', so that xargs will supply one argument to each of their composite functions?
Why not read your file and process each line separately in the shell? i.e.
fileList=/path/to/my/xmlFileList.txt
cat ${fileList} \
| while read fName ; do
gzip -d ${fName} | xmlstartlet > ${fName}.new
done
I hope this helps.
Although the right answer is the one suggested by shelter (+1), here is a one-liner "divertimento" providing that the input is the proposed by Andrey (a command that generates the list of urls) :-)
~$ eval $(command | awk '{a=a "wget -O - "$0" | gzip -d | xmlstartlet > $(basename "$0" .gz ).new; " } END {print a}')
It just generates a multi command line that does wget http://foo.xml.gz | gzip -d | xmlstartlet > $(basenname foo.xml.gz .gz).new for each of the urls in the input; after the resulting command is evaluated
Use GNU Parallel:
cat filelist | parallel 'zcat {} | xmlstarlet >{.}.out'
or if you want to include the fetching of urls:
cat urls | parallel 'wget -O - {} | zcat | xmlstarlet >{.}.out'
It is easy to read and you get the added benefit of having on job per CPU run in parallel. Watch the intro video to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ
If xmlstarlet can operate on stdin instead of having to pass it a filename, then:
some command | xargs -i -n1 sh -c 'zcat "{}" | xmlstarlet options ...'
The xargs option -i means you can use the "{}" placeholder to indicate where the filename should go. Use -n 1 to indicate xargs should only one line at a time from its input.
I've found that it's quite powerful to create long pipelines in bash scripts, but the main drawback that I see is that there doesn't seem to be a way to insert comments.
As an example, is there a good way to add comments to this script?
#find all my VNC sessions
ls -t $HOME/.vnc/*.pid \
| xargs -n1 \
| sed 's|\.pid$||; s|^.*\.vnc/||g' \
| xargs -P50 --replace vncconfig -display {} -get desktop \
| grep "($USER)" \
| awk '{print $1}' \
| xargs -n1 xdpyinfo -display \
| egrep "^name|dimensions|depths"
Let the pipe be the last character of each line and use # instead of \, like this:
ls -t $HOME/.vnc/*.pid | #comment here
xargs -n1 | #another comment
...
This works too:
# comment here
ls -t $HOME/.vnc/*.pid |
#comment here
xargs -n1 |
#another comment
...
based on https://stackoverflow.com/a/5100821/1019205.
it comes down to s/|//;s!\!|!.
Unless they're spectacularly long pipelines, you don't have to comment inline, just comment at the top:
# Find all my VNC sessions.
# xargs does something.
# sed does something else
# the second xargs destroys the universe.
# :
# and so on.
ls -t $HOME/.vnc/*.pid \
| xargs -n1 \
| sed 's|\.pid$||; s|^.*\.vnc/||g' \
| xargs -P50 --replace /opt/tools/bin/restrict_resources -T1 \
-- vncconfig -display {} -get desktop 2>/dev/null \
| grep "($USER)" \
| awk '{print $1}' \
| xargs -n1 xdpyinfo -display \
| egrep "^name|dimensions|depths"
As long as comments are relatively localised, it's fine. So I wouldn't put them at the top of the file (unless your pipeline was the first thing in the file, of course) or scribbled down on toilet paper and locked in your desk at work.
But the first thing I do when looking at a block is to look for comments immediately preceding the block. Even in C code, I don't comment every line, since the intent of comments is to mostly show the why and a high-level how.
#!/bin/bash
for pid in $HOME/.vnc/*.pid; do
tmp=${pid##*/}
disp=${tmp%.*}
xdpyinfo -display "$disp" | # commment here
egrep "^name|dimensions|depths"
done
I don't understand the need for vncconfig if all it does is append '(user)' which you subsequently remove for the call to xdpyinfo. Also, all those pipes take quite a bit of overhead, if you time your script vs mine I think you'll find the performance comparable if not faster.