I am writing a shell script to install Arch Linux, currently, I have a very basic disk selection but it does not give enough information, how can I add extra information such as the disk name and size but keep the $DISK variable the same (ie: /dev/nvme0n1)
select_disk () {
count=0
for device in `lsblk -d | tail -n+2 | cut -d" " -f1`; do
count=$((count+1))
dev[$count]=$device
printf '%s: %s\n' "$count" "$device"
done
read -rp "Select disk (numbers 1-$count): " selection
DISK="/dev/${dev[$selection]}"
echo "$DISK" > /tmp/disk
echo "Installing on $DISK."
}
You could use the columns format lsblk offers using the -o flag. Also, you can avoid the tail (which I believe is to remove the header) by using the lsblk flag -n which does exactly that. Something like this:
select_disk () {
count=0
# by default the 'for' loop splits by spaces, change that
# to split by breaklines by redefining IFS
IFS=$'\n'
for device_info in `lsblk -d -n -o NAME,TYPE,SIZE`; do
count=$((count+1))
device_name=$(echo $device_info | cut -d" " -f1)
dev[$count]=$device_name
printf '%s: %s\n' "$count" "$device_info"
done
read -rp "Select disk (numbers 1-$count): " selection
DISK="/dev/${dev[$selection]}"
echo "$DISK" > /tmp/disk
echo "Installing on $DISK."
}
Here is only showed the NAME, TYPE and SIZE, but you can add all the columns you want (separated by a comma). See lsblk -h (the help) for all possible options.
I'm not sure about what you are calling 'disk name' but...
Let's consider, in your 'for;do..done' loop, that $device is your mapped device name (i.e: sda).
You may get the following information:
1)The device label (if exist):
lsblk "/dev/${device}" -dn -o LABEL
2)The device vendor:
lsblk "/dev/${device}" -dn -o VENDOR
3)The device size (in human readable format)
lsblk "/dev/${device}" -dn -o SIZE
and multiple other informations like:
4)device type:
lsblk "/dev/${device}" -dn -o TYPE
5)Hardware block sizes
lsblk "/dev/${device}" -dn -o PHY-SEC,LOG-SEC
6)Linux owners and group assigned to this object
lsblk "/dev/${device}" -dn -o OWNER,GROUP
Or anything in the below list:
NAME KNAME MAJ:MIN FSTYPE MOUNTPOINT LABEL UUID PARTTYPE PARTLABEL PARTUUID PARTFLAGS RA RO RM HOTPLUG MODEL SERIAL SIZE STATE OWNER GROUP MODE ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE TYPE DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO WSAME WWN RAND PKNAME HCTL TRAN SUBSYSTEMS REV VENDOR ZONED
hope it helps.
Related
I have a two column tab delimited file the contains input for a command.
The input file looks like this:
2795.bam 2865.bam
2825.bam 2865.bam
2794.bam 2864.bam
the command line is:
macs2 callpeak -t trt.bam -c ctrl.bam -n Macs.name.bam --gsize hs --nomodel
where trt.bam are the names of files in column 1 and ctrl.bam are the names of files in col2.
what I trying is to read these values from input file and run them.
To do achieve this I am doing following:
cat temp | awk '{print $1 "\t" $2 }' | macs2 callpeak -t $1 -c $2 -n Macs.$1 --gsize hs --nomodel
This is failing. The error that I get is:
usage: macs2 callpeak [-h] -t TFILE [TFILE ...] [-c [CFILE [CFILE ...]]]
[-f {AUTO,BAM,SAM,BED,ELAND,ELANDMULTI,ELANDEXPORT,BOWTIE,BAMPE,BEDPE}]
[-g GSIZE] [--keep-dup KEEPDUPLICATES]
[--buffer-size BUFFER_SIZE] [--outdir OUTDIR] [-n NAME]
[-B] [--verbose VERBOSE] [--trackline] [--SPMR]
[-s TSIZE] [--bw BW] [-m MFOLD MFOLD] [--fix-bimodal]
[--nomodel] [--shift SHIFT] [--extsize EXTSIZE]
[-q QVALUE | -p PVALUE] [--to-large] [--ratio RATIO]
[--down-sample] [--seed SEED] [--tempdir TEMPDIR]
[--nolambda] [--slocal SMALLLOCAL] [--llocal LARGELOCAL]
[--broad] [--broad-cutoff BROADCUTOFF]
[--cutoff-analysis] [--call-summits]
[--fe-cutoff FECUTOFF]
macs2 callpeak: error: argument -t/--treatment: expected at least one argument
In an ideal situation this should be taking inputs like this:
macs2 callpeak -t 2795.bam -c 2865.bam -n Macs.2795 --gsize hs --nomodel
Where Macs is a standalone software that runs on linux. In the present situation, the software is failing to read the input from the file.
Any inputs are deeply appreciated.
I believe what you want to achieve is a loop over all lines in your input file. In bash, you can achieve this as :
while read -r tfile cfile; do
macs2 callpeak -t "$tfile" -c "$cfile" -n "Macs.$tfile" --gsize hs --nomodel
done < "input_file.txt"
See: https://mywiki.wooledge.org/BashFAQ/001 (cfr. Sundeep's comment)
original answer:
while read -a a; do
macs2 callpeak -t "${a[0]}" -c "${a[1]}" -n "Macs.${a[0]}" --gsize hs --nomodel
done < "input_file.txt"
This will read the input file input_file.txt line by line and store it in a bash array named a using read -a a. From that point forward, you process your command with the variables ${a[0]} and ${a[1]}.
I used crc32 to calculate checksums from strings a long time ago, but I cannot remember how I did it.
echo -n "LongString" | crc32 # no output
I found a solution [1] to calculate them with Python, but is there not a direct way to calculate that from a string?
# signed
python -c 'import binascii; print binascii.crc32("LongString")'
python -c 'import zlib; print zlib.crc32("LongString")'
# unsigned
python -c 'import binascii; print binascii.crc32("LongString") % (1<<32)'
python -c 'import zlib; print zlib.crc32("LongString") % (1<<32)'
[1] How to calculate CRC32 with Python to match online results?
I came up against this problem myself and I didn't want to go to the "hassle" of installing crc32. I came up with this, and although it's a little nasty it should work on most platforms, or most modern linux anyway ...
echo -n "LongString" | gzip -1 -c | tail -c8 | hexdump -n4 -e '"%u"'
Just to provide some technical details, gzip uses crc32 in the last 8 bytes and the -c option causes it to output to standard output and tail strips out the last 8 bytes. (-1 as suggested by #MarkAdler so we don't waste time actually doing the compression).
hexdump was a little trickier and I had to futz about with it for a while before I came up with something satisfactory, but the format here seems to correctly parse the gzip crc32 as a single 32-bit number:
-n4 takes only the relevant first 4 bytes of the gzip footer.
'"%u"' is your standard fprintf format string that formats the bytes as a single unsigned 32-bit integer. Notice that there are double quotes nested within single quotes here.
If you want a hexadecimal checksum you can change the format string to '"%08x"' (or '"%08X"' for upper case hex) which will format the checksum as 8 character (0 padded) hexadecimal.
Like I say, not the most elegant solution, and perhaps not an approach you'd want to use in a performance-sensitive scenario but an approach that might appeal given the near universality of the commands used.
The weak point here for cross-platform usability is probably the hexdump configuration, since I have seen variations on it from platform to platform and it's a bit fiddly. I'd suggest if you're using this you should try some test values and compare with the results of an online tool.
EDIT As suggested by #PedroGimeno in the comments, you can pipe the output into od instead of hexdump for identical results without the fiddly options. ... | od -t x4 -N 4 -A n for hex ... | od -t d4 -N 4 -A n for decimal.
Or just use the process substitution:
crc32 <(echo "LongString")
Your question already has most of the answer.
echo -n 123456789 | python -c 'import sys;import zlib;print(zlib.crc32(sys.stdin.read())%(1<<32))'
correctly gives 3421780262
I prefer hex:
echo -n 123456789 | python -c 'import sys;import zlib;print("%08x"%(zlib.crc32(sys.stdin.read())%(1<<32)))'
cbf43926
Be aware that there are several CRC-32 algorithms:
http://reveng.sourceforge.net/crc-catalogue/all.htm#crc.cat-bits.32
On Ubuntu, at least, /usr/bin/crc32 is a short Perl script, and you can see quite clearly from its source that all it can do is open files. It has no facility to read from stdin -- it doesn't have special handling for - as a filename, or a -c parameter or anything like that.
So your easiest approach is to live with it, and make a temporary file.
tmpfile=$(mktemp)
echo -n "LongString" > "$tmpfile"
crc32 "$tmpfile"
rm -f "$tmpfile"
If you really don't want to write a file (e.g. it's more data than your filesystem can take -- unlikely if it's really a "long string", but for the sake for argument...) you could use a named pipe. To a simple non-random-access reader this is indistinguishable from a file:
fifo=$(mktemp -u)
mkfifo "$fifo"
echo -n "LongString" > "$fifo" &
crc32 "$fifo"
rm -f "$fifo"
Note the & to background the process which writes to fifo, because it will block until the next command reads it.
To be more fastidious about temporary file creation, see: https://unix.stackexchange.com/questions/181937/how-create-a-temporary-file-in-shell-script
Alternatively, use what's in the script as an example from which to write your own Perl one-liner (the presence of crc32 on your system indicates that Perl and the necessary module are installed), or use the Python one-liner you've already found.
Here is a pure Bash implementation:
#!/usr/bin/env bash
declare -i -a CRC32_LOOKUP_TABLE
__generate_crc_lookup_table() {
local -i -r LSB_CRC32_POLY=0xEDB88320 # The CRC32 polynomal LSB order
local -i index byte lsb
for index in {0..255}; do
((byte = 255 - index))
for _ in {0..7}; do # 8-bit lsb shift
((lsb = byte & 0x01, byte = ((byte >> 1) & 0x7FFFFFFF) ^ (lsb == 0 ? LSB_CRC32_POLY : 0)))
done
((CRC32_LOOKUP_TABLE[index] = byte))
done
}
__generate_crc_lookup_table
typeset -r CRC32_LOOKUP_TABLE
crc32_string() {
[[ ${#} -eq 1 ]] || return
local -i i byte crc=0xFFFFFFFF index
for ((i = 0; i < ${#1}; i++)); do
byte=$(printf '%d' "'${1:i:1}") # Get byte value of character at i
((index = (crc ^ byte) & 0xFF, crc = (CRC32_LOOKUP_TABLE[index] ^ (crc >> 8)) & 0xFFFFFFFF))
done
echo $((crc ^ 0xFFFFFFFF))
}
printf 'The CRC32 of: %s\nis: %08x\n' "${1}" "$(crc32_string "${1}")"
# crc32_string "The quick brown fox jumps over the lazy dog"
# yields 414fa339
Testing:
bash ./crc32.sh "The quick brown fox jumps over the lazy dog"
The CRC32 of: The quick brown fox jumps over the lazy dog
is: 414fa339
I use cksum and convert to hex using the shell builtin printf:
$ echo -n "LongString" | cksum | cut -d\ -f1 | xargs echo printf '%0X\\n' | sh
5751BDB2
The cksum command first appeared on 4.4BSD UNIX and should be present in all modern systems.
You can try to use rhash.
http://rhash.sourceforge.net/
https://github.com/rhash/RHash
http://manpages.ubuntu.com/manpages/bionic/man1/rhash.1.html
Testing:
## install 'rhash'...
$ sudo apt-get install rhash
## test CRC32...
$ echo -n 123456789 | rhash --simple -
cbf43926 (stdin)
In the sections below, you'll see the shell script I am trying to run on a UNIX machine, along with a transcript.
When I run this program, it gives the expected output but it also gives an error shown in the transcript. What could be the problem and how can I fix it?
First, the script:
#!/usr/bin/bash
while read A B C D E F
do
E=`echo $E | cut -f 1 -d "%"`
if test $# -eq 2
then
I=`echo $2`
else
I=90
fi
if test $E -ge $I
then
echo $F
fi
done
And the transcript of running it:
$ df -k | ./filter.sh -c 50
./filter.sh: line 12: test: capacity: integer expression expected
/etc/svc/volatile
/var/run
/home/ug
/home/pg
/home/staff/t
/packages/turnin
$ _
Before the line that says:
if test $E -ge $I
temporarily place the line:
echo "[$E]"
and you'll find something very much non-numeric, and that's because the output of df -k looks like this:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdb1 954316620 212723892 693109608 24% /
udev 10240 0 10240 0% /dev
: :
The offending line there is the first, which will have its fifth field Use% turned into Use, which is definitely not an integer.
A quick fix may be to change your usage to something like:
df -k | sed -n '2,$p' | ./filter -c 50
or:
df -k | tail -n+2 | ./filter -c 50
Either of those extra filters (sed or tail) will print only from line 2 onwards.
If you're open to not needing a special script at all, you could probably just get away with something like:
df -k | awk -vlimit=40 '$5+0>=limit&&NR>1{print $5" "$6}'
The way it works is to only operate on lines where both:
the fifth field, converted to a number, is at least equal to the limit passed in with -v; and
the record number (line) is two or greater.
Then it simply outputs the relevant information for those matching lines.
This particular example outputs the file system and usage (as a percentage like 42%) but, if you just want the file system as per your script, just change the print to output $6 on its own: {print $6}.
Alternatively, if you do the percentage but without the %, you can use the same method I used in the conditional: {print $5+0" "$6}.
The situation, where I have unknown number of volume groups and their names with unknown number of disks assigned to them.
Example :
pvs -o pv_name,vg_name
PV VG
/dev/vdd appvg01
/dev/vdb appvg01
/dev/vdf3 vg00
/dev/vdh testvg
vgs --noheadings | awk '{print $1}'| while read line ; do echo $line;vgs --noheadings -o pv_name $line; done
appvg01
/dev/vdd
/dev/vdb
testvg
/dev/vdh
vg00
/dev/vdf3
At the final stage I'd like to mirror each volume with new disk that I'll add manually :
for i in `/sbin/lvs| /bin/awk '{if ($2 ~ /appvg01/) print $1}'`; do
/sbin/lvconvert -b -m0 appvg01/$i /dev/vde
done
but, I don't know what volume name should I correlate with, as it might be any other name.
what is the best approach for this structure.
Thanks
The correct data structure to store this kind of information in bash is associative arrays:
declare -A pvs
{
read # skip the header
while read -r pv vg; do
pvs[$pv]=$vg
done
} < <(pvs -o pv_name,vg_name)
Thereafter, you can iterate and do lookups:
for pv in "${!pvs[#]}"; do
vg="${pvs[$pv]}"
echo "vg $vg is backed by pv $pv"
done
I have a config
[LogicalUnit1] UnitInquiry "NFSN00Y5IP51ZL" LUN0 /mnt/extent0 64MB
[LogicalUnit2] UnitInquiry "NFSN00N49CQL28" LUN0 /mnt/extent1 64MB
[LogicalUnit3] UnitInquiry "NFSNBRGQOCXK" LUN0 /mnt/extent4 10MB
[LogicalUnit4] UnitInquiry "NFSNE7IXADFJ" LUN0 /mnt/extent5 25MB
which is read via a bash script, using awk i parse the file and get variables
awk '/UnitInquiry/ {print $1, $3, $5, $6}' $ctld_config | while read a b c d ; do
if [ -f $a ]
then
ctladm create -b block -o file=$c -S $b -d $a
ctladm devlist -v > $lun_config
else
truncate -s $d $c ; ctladm create -b block -o file=$c -S $b -d $a
fi
this will initialize the luns properly on bootup, however if i add a lun then it will recreate them all again, how can i compare whats running, to whats configured and only reinitialize the ones not already live, there is a command to list the devices
ctladm devlist -v
LUN Backend Size (Blocks) BS Serial Number Device ID
0 block 131072 512 "NFSN00Y5IP51ZL [LogicalUnit1]
lun_type=0
num_threads=14
file=/mnt/extent0
1 block 131072 512 "NFSN00N49CQL28 [LogicalUnit2]
lun_type=0
num_threads=14
file=/mnt/extent1
2 block 20480 512 "NFSNBRGQOCXK" [LogicalUnit3]
lun_type=0
num_threads=14
file=/mnt/extent4
3 block 51200 512 "NFSNE7IXADFJ" [LogicalUnit4]
lun_type=0
num_threads=14
file=/mnt/extent5
Why not add the following after the then:
ctladm devlist -v | grep -q "$a" && continue
This will
run the command that show the currently active devices
check if the LogicalUnit name you want to register is already listed, and if yes...
skip the rest of the loop.
If $a (logical unit name) is not unique enough, you can also grep for another, more unique identifier, e.g. the serial number.