How to determine the latest major and full kernel version string as compactly as possible - bash

So what I'm intending to do here is to determine both the latest major and the full kernel version string as compactly as possible (without a zillion pipes to grep).
I'm already quite content with the result but if anybody has any ideas how to squash the first line even the slightest it'd be very awesome (it has to work when there are no minor patches as well).
The index of kernel.org is only 36kB compared to the 136kB of that of http://www.kernel.org/pub/linux/kernel/v3.x/ so that's why I'm using it:
_major=$(curl -s http://www.kernel.org/ -o /tmp/kernel && cat /tmp/kernel | grep -A1 mainline | tail -1 | cut -d ">" -f3 | cut -d "<" -f1)
pkgver=${_major}.$(cat /tmp/kernel | grep ${_major} | head -1 | cut -d "." -f6)

It's just a thought exercise at this stage as the real answer is in the comments above, but here are some possible improvements.
Original:
_major=$(curl -s http://www.kernel.org/ -o /tmp/kernel && cat /tmp/kernel | grep -A1 mainline | tail -1 | cut -d ">" -f3 | cut -d "<" -f1)
Use tee instead of cat:
_major=$(curl -s http://www.kernel.org/ | tee /tmp/kernel | grep -A1 mainline | tail -1 | cut -d ">" -f3 | cut -d "<" -f1)
Use sed to minimise the number of pipes, and to make the command unreadable
_major=$(curl -s http://www.kernel.org/ | tee /tmp/kernel | sed -n '/ainl/,/<\/s/ s|.*>\([0-9\.]*\)</st.*|\1|p')
Cheap tricks: shorten the URL
_major=$(curl -s kernel.org | tee /tmp/kernel | sed -n '/ainl/,/<\/s/ s|.*>\([0-9\.]*\)</st.*|\1|p')

kernel.org provides a plaintext listing of all the current versions at https://www.kernel.org/finger_banner
For mainline:
curl -s https://www.kernel.org/finger_banner | grep mainline | awk '{print $NF}'
For latest stable:
curl -s https://www.kernel.org/finger_banner | grep -m1 stable | awk '{print $NF}'
The mainline and latest stable versions will never be EOL, but other versions often are, so the above awk commands will not work correctly for all versions. A general solution as a bash function:
latest_kernel() {
curl -s https://www.kernel.org/finger_banner | grep -m1 $1 | sed -r 's/^.+: +([^ ]+)( .+)?$/\1/'
}
Examples:
$ latest_kernel mainline
4.18-rc2
$ latest_kernel stable
4.17.3
$ latest_kernel 4.16
4.16.18

You've got a useless use of cat. You can replace:
cat /tmp/kernel | grep -A1 mainline
with simply:
grep -A1 mainline /tmp/kernel
In your case, you don't even need the file at all. Curl by default will emit to standard output, so you can just do:
curl -s http://www.kernel.org/ | grep -A1 mainline

Expanding on #Justin Brewer's answer, you probably want to know when a kernel is EOL since this is useful information... the following single awk command preserves all this information for you.
latest_kernel() {
curl -s https://www.kernel.org/finger_banner |awk -F ':' -v search="$1" '{if ($1 ~ search) {gsub(/^[ ]+/, "", $2); print $2}}'
}
-F ':' -- field separator because everything after the : is the version string.
-v search="$1" -- pass search string as an awk internal variable
if statement -- check if field $1 matches the search string
gsub -- in-place modify of field $2 to strip leading spaces
Then just print field $2 for any matching records (I presume your search string will only match the left-hand side of one line... if it is important to exit after the first match, use print $2; exit)
Search string can include spaces, etc. Use of awk variables and matching with ~ variable instead of pattern-matching '.../'"$1"'/...' avoids the need to exit single-quote mode and avoids syntax errors where the search string contains "/".

Related

How to grep only matching string from this result?

I am just simply trying to grab the commit ID, but not quite sure what I'm missing:
➜ ~ curl https://github.com/microsoft/vscode/releases -s | grep -oE 'microsoft/vscode/commit/(.*?)/hovercard'
microsoft/vscode/commit/ccbaa2d27e38e5afa3e5c21c1c7bef4657064247/hovercard
The only thing I need back from this is ccbaa2d27e38e5afa3e5c21c1c7bef4657064247.
This works just fine on regex101.com and in ruby/python. What am I missing?
If supported, you can use grep -oP
echo "microsoft/vscode/commit/ccbaa2d27e38e5afa3e5c21c1c7bef4657064247/hovercard" | grep -oP "microsoft/vscode/commit/\K.*?(?=/hovercard)"
Output
ccbaa2d27e38e5afa3e5c21c1c7bef4657064247
Another option is to use sed with a capture group
echo "microsoft/vscode/commit/ccbaa2d27e38e5afa3e5c21c1c7bef4657064247/hovercard" | sed -E 's/microsoft\/vscode\/commit\/([^\/]+)\/hovercard/\1/'
Output
ccbaa2d27e38e5afa3e5c21c1c7bef4657064247
The point is that grep does not support extracting capturing group submatches. If you install pcregrep you could do that with
curl https://github.com/microsoft/vscode/releases -s | \
pcregrep -o1 'microsoft/vscode/commit/(.*?)/hovercard' | head -1
The | head -1 part is to fetch the first occurrence only.
I would suggest using awk here:
awk 'match($0,/microsoft\/vscode\/commit\/[^\/]*\/hovercard/){print substr($0,RSTART+24,RLENGTH-34);exit}'
The regex will match a line containing
microsoft\/vscode\/commit\/ - microsoft/vscode/commit/ fixed string
[^\/]* - zero or more chars other than /
\/hovercard - a /hovercard string.
The substr($0,RSTART+24,RLENGTH-34) will print the part of the line starting at the RSTART+24 (24 is the length of microsoft/vscode/commit/) index and the RLENGTH is the length of microsoft/vscode/commit/ + the length of the /hovercard.
The exit command will fetch you the first occurrence. Remove it if you need all occurrences.
You can use sed:
curl -s https://github.com/microsoft/vscode/releases |
sed -En 's=.*microsoft/vscode/commit/([^/]+)/hovercard.*=\1=p' |
head -n 1
head -n 1 is to print the first match (there are 10)grep -o will print (only) everything that matches, including microsoft/ etc.
Your task can not be achieved with Mac's grep. grep -o prints all matching text (compared to default behaviour of printing matching lines), including microsoft/ etc. A grep which implemented perl regex (like GNU grep on Linux) could make use of look ahead/behind (grep -Po '(?<=microsoft/vscode/commit/)[^/]+(?=/hovercard)'). But it's just not available on Mac's grep.
On MacOS you don't have gnu utilities available by default. You can just pipe your output to a simple awk like this:
curl https://github.com/microsoft/vscode/releases -s |
grep -oE 'microsoft/vscode/commit/[^/]+/hovercard' |
awk -F/ '{print $(NF-1)}'
ccbaa2d27e38e5afa3e5c21c1c7bef4657064247
3a6960b964327f0e3882ce18fcebd07ed191b316
f4af3cbf5a99787542e2a30fe1fd37cd644cc31f
b3318bc0524af3d74034b8bb8a64df0ccf35549a
6cba118ac49a1b88332f312a8f67186f7f3c1643
c13f1abb110fc756f9b3a6f16670df9cd9d4cf63
ee8c7def80afc00dd6e593ef12f37756d8f504ea
7f6ab5485bbc008386c4386d08766667e155244e
83bd43bc519d15e50c4272c6cf5c1479df196a4d
e7d7e9a9348e6a8cc8c03f877d39cb72e5dfb1ff

Curl and xargs in piped commands

I want to process an old database where password are plain text (comma separated ; passwd is the 5th field in the csv file where the database has been exported) to crypt them for further use by dokuwiki. Here is my bash command (grep and sed are there to extract the crypted passwd from curl output) :
cat users.csv | awk 'FS="," { print $4 }' | xargs -l bash -c 'curl -s --data-binary "pass1=$0&pass2=$0" "https://sprhost.com/tools/SMD5.php" -o - ' | xargs | grep -o '<tt.*tt>' | sed -e 's/tt//g' | sed -e 's/<[^>]*>//g'
I get the following comment from xargs
xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option
And only the first line of the file is processed, and nothing appends then.
Using the -0 option, and playing around with quotes, doesn't solve anything. Where am I wrong in the command line ? May be a more advanced language will be more adequate to do this.
Thank for help, LM
In general, if you have such a long pipe of commands, it is better to split them if things go wrong. Going through your pipe:
cat users.csv |
Nothing unexpected there.
awk 'FS="," { print $4 }' |
You probably wanted to do awk 'BEGIN {FS=","} { print $4 }'. Try the first two commands in the pipe and see if they produce the correct answer.
xargs -l bash -c 'curl -s --data-binary "pass1=$0&pass2=$0" "https://sprhost.com/tools/SMD5.php" -o - ' |
Nothing wrong there, although there might be better ways to do an MD5 hash.
xargs |
What is this xargs doing in the pipe? It should be removed.
grep -o '<tt.*tt>' |
Note that this will produce two lines:
<tt>$1$17ab075e$0VQMuM3cr5CtElvMxrPcE0</tt>
<tt><your_docuwiki_root>/conf/users.auth.php</tt>
which is probably not what you expected.
sed -e 's/tt//g' |
sed -e 's/<[^>]*>//g'
which will remove the html-tags, though
sed 's/<tt>//;s/<.tt>//'
will do the same.
So I'd say a wrong awk and an xargs too many.

GREP by result of awk

Output of awk '{print $4}' is
b05808aa-c6ad-4d30-a334-198ff5726f7c
59996d37-9008-4b3b-ab22-340955cb6019
2b41f358-ff6d-418c-a0d3-ac7151c03b78
7ac4995c-ff2c-4717-a2ac-e6870a5670f0
I need to grep file st.log by these records. Something like awk '{print $4}' |xargs -i grep -w "pattern from awk" st.log I dont know how to pass pattern correctly?
What about
awk '{print $4}' | grep -F -f - st.log
Credits to Eric Renouf, who noticed that -f - can be used for standard input instead -f <(cat), Note: -f /dev/stdin also works and avoids launching a new process.
or closer to the question to have the output ordered
awk '{print $4}' | xargs -i grep -F {} st.log
maybe -w was not the option OP needed but -F
grep --help
-F, --fixed-strings PATTERN is a set of newline-separated strings
-w, --word-regexp force PATTERN to match only whole words
-w will match only line that contain exactly pattern
examples
grep -w . <<<a # matches
grep -w . <<<ab # doesn't match
grep -F . <<<a # doesn't match
grep -F . <<<a.b # matches
May be something along these lines be helpful
How to process each line received as a result of grep command
awk '{print $4} | while read -r line; do
grep $line st.log
done

Display interface + ip list nice way

I have to display net interface and IP's attached to it.
I came up with this code:
if [ -f intf ]; then
rm -I intf
fi &&
if [ -f ipl ]; then
rm -I ipl
fi &&
ip ntable | grep dev | sort | uniq | sed -e 's/^.*dev //;/^lo/d' >> intf &&
ip a | grep -oP "inet\s+\K[\w./]+" | grep -v 127 >> ipl &&
paste <(cat intf) <(cat ipl)
It does the job but I believe it's ugly :), created files, IMHO a total mess :)
any one can suggest the nice way to get exact the same result but short and efficient way ?
If there are a few interfaces, right now I'm thinking about looping, but that will make this code even bigger and probably uglier :) What would you suggest?
As the first thing, you can eliminate the need for temporary files with process substitution:
paste <(ip ntable | grep dev | sort -u | sed -e 's/^.*dev //;/^lo/d') <(ip a | grep -oP "inet\s+\K[\w./]+" | grep -v 127)
sort -u does the same thing as sort | uniq
This oneliner outputs the interface name and its ip address:
ifconfig |\
grep -e 'Link' -A 1 |\
paste -d" " - - - |\
grep ' addr' |\
sed -e 's/ */ /g' -e 's/Link.*addr://' |\
cut -d" " -f1,2
Here an explanation of the commands:
Shows network configuration
Filters lines containing Link and the next line to it.
Joins three lines
Filters lines having an assigned address
Trim whitespaces and remove not relevant information
Splits remaining data and keeps only interface name and ip address.
Example output:
br-2065e5d2fc59 172.18.0.1
docker0 172.17.0.1
lo 127.0.0.1
wlp3s0

Solaris: find files not containing a string (alternative to grep -L)

I want to search files that does not contain a specific string.
I used -lv but this was a huge mistake because it was returning all the files that contain any line not containing my string.
I knew what I need exactly is grep -L, however, Solaris grep does not implement this feature.
What is the alternative, if any?
You can exploit grep -c and do the following (thanks #Scrutinizer for the /dev/null hint):
grep -c foo /dev/null * 2>/dev/null | awk -F: 'NR>1&&!$2{print $1}'
This will unfortunately also print directories (if * expands to any) which might not be desired in which case a simple loop, albeit slower, might be your best bet:
for file in *; do
[ -f "${file}" ] || continue
grep -q foo "${file}" 2>/dev/null || echo "${file}"
done
However, if you have GNU awk 4 on your system you can do:
awk 'BEGINFILE{f=0} /foo/{f=1} ENDFILE{if(!f)print FILENAME}' *
after using grep -c u can use grep again to find your desired filenames:
grep -c 'pattern' * | grep ':0$'
and to see just filnames :
grep -c 'pattern' * | grep ':0$' | cut -d":" -f1
You can use awk like this:
awk '!/not this/' file
To do multiple not:
awk '!/jan|feb|mars/' file

Resources