Using sed to print content of files in columns - shell

I have an file , my file name is lookup.txt. Sometimes this file content is 100 lines , so everything scroll while printing. Thats why i like to print it on screen in 2 or more columns.
my bash doesnt support commands like pr or columns.Need to be in old fashion shell.
Content of lookup file is:
1-Top Scores
2-Last Scores
3-Username
4-Birthday
5-Last Meal today
6-Coronavirus in Italy
7-Lets Stay Home
8-Winner
9-Germany and Belgium
10-Alfa
11-Bravo
12-Charlie and delta
For printing in columns i use this command found here on stackoverflow:
cat lookup.txt | xargs printf '%-24s\n' | sed '$p;N;s/\n//;$p;N;s/\n//'
and this is the print in columns:
1-Top Scores 2-Last
Scores 3-Username 4-Birthday
5-Last Meal today
6-Coronavirus in Italy
7-Lets Stay Home
8-Winner 9-Germany and
Belgium 10-Alfa 11-Bravo
12-Charlie and delta
What i need is to let sed not split the words into different columns but just the line.
1-Top Scores 6-Coronavirus in Italy 11-Bravo
2-Last Scores 7-Lets Stay Home 12-Charlie and delta
3-Username 8-Winner
4-Birthday 9-Germany and Belgium
5-Last Meal today 10-Alfa
if the 3 columns are to width then x chars then print in 2 columns (this last is optional)
thanks in advance
Available tools:
/bin:
ash gunzip ping
base64 gzip ping6
bash hostname ps
bash.bash ionice ps.procps
busybox kill pwd
busybox.nosuid kill.procps rm
busybox.suid kmod rmdir
cat ln run-parts
chattr login sed
chgrp login.shadow setserial
chmod ls sh
chown lsattr sleep
cp lsmod stat
cttyhack lsmod.kmod stty
date mkdir su
dd mknod su.shadow
df mktemp sync
dmesg more tar
dnsdomainname mount touch
echo mount.util-linux true
editor mountpoint true.coreutils
egrep mountpoint.sysvinit umount
fake-hwclock mpstat uname
false mv usleep
false.coreutils netstat vi
fatattr nice vi.sh
fgrep pidof watch
getopt pidof.procps watch.procps
grep pidof.sysvinit zcat
/sbin:
arp ifup mkfs.ext4 runlevel.sysvinit
blkid init mkswap setconsole
blkid.util-linux init.sysvinit modinfo shutdown
bootlogd inotifyd modinfo.kmod shutdown.sysvinit
bridge insmod modprobe start-stop-daemon
depmod insmod.kmod modprobe.kmod stb-hwclock
depmod.kmod ip mount.cifs sulogin
devmem ip.iproute2 mount.fuse sulogin.util-linux
e2fsck iwconfig mount.ntfs swapoff
e2label iwgetid mount.ntfs-3g swapon
flock iwlist nologin switch_root
flock.util-linux iwpriv nologin.shadow sysctl
fsck.ext2 iwspy poweroff sysctl.procps
fsck.ext3 killall5 poweroff.sysvinit syslogd
fsck.ext4 klogd reboot telinit
fstab-decode ldconfig reboot.sysvinit tunctl
fstrim loadkmap rmmod tune2fs
getty logread rmmod.kmod udhcpc
halt losetup route uevent
halt.sysvinit lsmod routef vigr
hdparm mdev routel vigr.shadow
ifcfg mke2fs rtmon vipw
ifconfig mkfs.ext2 rtpr vipw.shadow
ifdown mkfs.ext3 runlevel
/usr/bin:
[ realpath
[[ renice
alsamixer reset
ar resize
awk rpcclient
basename rtmpdump
bdpoll scp
bsdcat scsi_logging_level
bunzip2 scsi_mandat
bzcat scsi_readcap
c_rehash scsi_ready
chage scsi_satl
cheetah scsi_start
cheetah-analyze scsi_stop
cheetah-compile scsi_temperature
chfn sdparm
chfn.shadow seq
chrt setsid
chsh sexp-conv
chsh.shadow sg
cifsdd sg_compare_and_write
clear sg_copy_results
cmp sg_dd
crontab sg_decode_sense
curl sg_emc_trespass
cut sg_format
dbclient sg_get_config
dbus-cleanup-sockets sg_get_lba_status
dbus-daemon sg_ident
dbus-launch sg_inq
dbus-monitor sg_logs
dbus-run-session sg_luns
dbus-send sg_map
dbus-update-activation-environment sg_map26
dbus-uuidgen sg_modes
dc sg_opcodes
diff sg_persist
dirname sg_prevent
dlist_test sg_raw
dos2unix sg_rbuf
dotlockfile sg_rdac
du sg_read
easy_install sg_read_attr
easy_install-2.7 sg_read_block_limits
eject sg_read_buffer
enigma2 sg_read_long
enigma2.sh sg_readcap
env sg_reassign
expiry sg_referrals
expr sg_rep_zones
faillog sg_requests
find sg_reset
flip sg_reset_wp
free sg_rmsn
free.procps sg_rtpg
fuser sg_safte
fusermount sg_sanitize
get_device sg_sat_identify
get_driver sg_sat_phy_event
get_module sg_sat_read_gplog
gpasswd sg_sat_set_features
grab sg_scan
groups sg_senddiag
groups.shadow sg_ses
gst-inspect-1.0 sg_ses_microcode
gst-launch-1.0 sg_start
gst-stats-1.0 sg_stpg
gst-typefind-1.0 sg_sync
head sg_test_rwbuf
hotplug_e2_helper sg_timestamp
id sg_turs
killall sg_unmap
last sg_verify
last.sysvinit sg_vpd
lastb sg_wr_mode
lastb.sysvinit sg_write_buffer
lastlog sg_write_long
ldbadd sg_write_same
ldbdel sg_write_verify
ldbedit sg_xcopy
ldbmodify sg_zone
ldbrename sginfo
ldbsearch sgm_dd
less sgp_dd
llmnrd sha1sum
lockfile-check sha256sum
lockfile-create sha3sum
lockfile-remove sha512sum
lockfile-touch shellinaboxd
logger showiframe
logname shuf
lsof skill
lspci skill.procps
lsusb slabtop
mail-lock smbcacls
mail-touchlock smbclient
mail-unlock smbcquotas
md5sum smbget
mesg smbspool
mesg.sysvinit smbtar
mid3cp smbtree
mid3iconv snice
mid3v2 snice.procps
mkfifo sort
moggsplit ssh
mutagen-inspect strings
mutagen-pony systool
nc tail
ndg_httpclient taskset
nettle-hash tee
nettle-lfib-stream telnet
nettle-pbkdf2 test
newgidmap time
newgrp timeout
newgrp.shadow tload
newuidmap toix
nmap toms
nohup top
nslookup top.procps
ntfs-3g tr
ntfs-3g.probe traceroute
ntpdate-sync traceroute6
od truncate
odhcp6c tty
ofgwrite ulockmgr_server
ofgwrite_bin uniq
ofgwrite_test unix2dos
openssl unlink
opkg unxz
opkg-check-config unzip
passwd update-alternatives
passwd.shadow uptime
patch uptime.procps
pgrep users
pgrep.procps utmpdump
pilconvert.py utmpdump.sysvinit
pildriver.py vlock
pilfile.py vmstat
pilfont.py volname
pilprint.py w
pkcs1-conv wall
pkill wall.sysvinit
pkill.procps wc
pmap wget
pmap.procps which
printf who
pwdx whoami
pwdx.procps whois
python wpa_passphrase
python-config xargs
python2 xmlcatalog
python2-config xmllint
python2.7 xzcat
python2.7-config yes
readlink
/usr/sbin:
addgroup grpconv pwck
adduser grpunconv pwconv
alsactl hddtemp pwunconv
automount i2cdetect rdate
avahi-daemon i2cdump rdev
chgpasswd i2cget readprofile
chpasswd i2cset rfkill
chpasswd.shadow ifplugd run-postinsts
chroot inetd setlogcons
crond logoutd sfdisk
delgroup mkfs.ubifs telnetd
deluser mtd_debug ubiattach
dropbear mtdinfo ubiblock
dropbearconvert mtdpart ubicrc32
dropbearkey nanddump ubidetach
dropbearmulti nandtest ubiformat
ether-wake nandwrite ubimkvol
ethtool newusers ubinfo
fbset nl-class-add ubinize
flash_erase nl-class-delete ubirename
flash_eraseall nl-class-list ubirmvol
flash_lock nl-classid-lookup ubirsvol
flash_otp_dump nl-cls-add ubiupdatevol
flash_otp_info nl-cls-delete unix_chkpwd
flash_otp_lock nl-cls-list unix_update
flash_otp_write nl-link-list update-ca-certificates
flash_unlock nl-pktloc-lookup update-rc.d
flashcp nl-qdisc-add useradd
ftpd nl-qdisc-delete userdel
genl-ctrl-list nl-qdisc-list usermod
groupadd ntpd vsftpd
groupdel ntpdate wpa_action
groupmems odhcp6c-update wpa_cli
groupmod parted wpa_supplicant
grpck partprobe
root#xp1000max:~#

Not claiming complete credit, you can modify this andlrc's answer on In bash how can I split a column in several column of fixed dimension to your requirement.
Instead of appending the columns with a whitespace, you can use sprintf() to custom print your column strings. Tested on GNU awk
awk '
BEGIN {
# Numbers of rows to print
n=5;
}
{
# Add to array with key = 0, 1, 2, 3, 4, 0, 1, 2, ..
l[(NR-1)%n] = sprintf("%s%-24s", l[(NR-1)%n], $0)
};
END {
# print the array
for (i = 0; i < length(l); i++) {
print l[i];
}
}
' file

Try this:
cat lookup.txt | tr '\n' '\0' | xargs -0 -n 2 printf '%-24s %-24s\n'
The tr command will convert newlines to nul chars.
Then we tell xargs to take nul-separated args and pass two at a time to the command.

Related

Bash concurrency when writing to file

I want to run a script agains a long subset of items, and each of them run concurrently, only when every iteration finishes, write it to a file.
For some reason, it writes to the file without finishing the function:
#!/bin/bash
function print_not_semver_line() {
echo -n "$repo_name,"
git tag -l | while read -r tag_name;do
semver $tag_name > /dev/null || echo -n "$tag_name "
done
echo ""
}
csv_name=~/Scripts/all_repos/not_semver.csv
echo "Repo Name,Not Semver Versions" > $csv_name
while read -r repo_name;do
cd $repo_dir
print_not_semver_line >> $csv_name &
done < ~/Scripts/all_repos/all_repos.txt
of course without &, it does what it supposed to do, but with it, it gets all messed up.
Ideas?
Here's an alternative that uses xargs for its natural parallelization, and a quick script that determines all of the non-semver tags and outputs at the end of the repo.
The premise is that this script does nothing fancy, it just loops over its provided directories and does one at a time, where you can parallelize outside of the script.
#!/bin/bash
log() {
now=$(date -Isec --utc)
echo "${now} $$ ${*}" > /dev/stderr
}
# I don't have semver otherwise available, so a knockoff replacement
function is_semver() {
echo "$*" | egrep -q "^v?[0-9]+\.[0-9]+\.[0-9]+$"
}
log "Called with: ${#}"
for repo_dir in ${#} ; do
log "Starting '${repo_dir}'"
bad=$(
git -C "${repo_dir}" tag -l | \
while read tag_name ; do
is_semver "${tag_name}" || echo -n "${tag_name} "
done
)
log "Done '${repo_dir}'"
echo "${repo_dir},${bad}"
done
log "exiting"
I have a project directory with various cloned github repos, I'll run it using xargs here. Notice a few things:
I am demonstrating calling the script with -L2 two directories per call (not parallelized) but -P4 four of these scripts running simultaneously
everything left of xargs in the pipe should be your method of determining what dirs/repos to iterate over
the first batch of processes starts with PIDs 17438, 17439, 17440, and 17442, and only when one of those quits (17442 then 17439) are new processes started
if you are not concerned with too many things running at once, you might use xargs -L1 -P9999 or something equally ridiculous :-)
$ find . -maxdepth 2 -iname .git | sed -e 's,/\.git,,g' | head -n 12 | \
xargs -L2 -P4 ~/StackOverflow/5783481/62283574_2.sh > not_semver.csv
2020-06-09T17:51:39+00:00 17438 Called with: ./calendar ./callr
2020-06-09T17:51:39+00:00 17439 Called with: ./docker-self-service-password ./ggnomics
2020-06-09T17:51:39+00:00 17438 Starting './calendar'
2020-06-09T17:51:39+00:00 17440 Called with: ./ggplot2 ./grid
2020-06-09T17:51:39+00:00 17439 Starting './docker-self-service-password'
2020-06-09T17:51:39+00:00 17442 Called with: ./gt ./keyring
2020-06-09T17:51:39+00:00 17440 Starting './ggplot2'
2020-06-09T17:51:39+00:00 17442 Starting './gt'
2020-06-09T17:51:39+00:00 17442 Done './gt'
2020-06-09T17:51:40+00:00 17442 Starting './keyring'
2020-06-09T17:51:40+00:00 17438 Done './calendar'
2020-06-09T17:51:40+00:00 17438 Starting './callr'
2020-06-09T17:51:40+00:00 17439 Done './docker-self-service-password'
2020-06-09T17:51:40+00:00 17439 Starting './ggnomics'
2020-06-09T17:51:40+00:00 17442 Done './keyring'
2020-06-09T17:51:40+00:00 17439 Done './ggnomics'
2020-06-09T17:51:40+00:00 17442 exiting
2020-06-09T17:51:40+00:00 17439 exiting
2020-06-09T17:51:40+00:00 17515 Called with: ./knitr ./ksql
2020-06-09T17:51:40+00:00 17518 Called with: ./nanodbc ./nostalgy
2020-06-09T17:51:40+00:00 17515 Starting './knitr'
2020-06-09T17:51:40+00:00 17518 Starting './nanodbc'
2020-06-09T17:51:41+00:00 17438 Done './callr'
2020-06-09T17:51:41+00:00 17438 exiting
2020-06-09T17:51:42+00:00 17440 Done './ggplot2'
2020-06-09T17:51:42+00:00 17440 Starting './grid'
2020-06-09T17:51:43+00:00 17518 Done './nanodbc'
2020-06-09T17:51:43+00:00 17518 Starting './nostalgy'
2020-06-09T17:51:43+00:00 17518 Done './nostalgy'
2020-06-09T17:51:43+00:00 17518 exiting
2020-06-09T17:51:43+00:00 17440 Done './grid'
2020-06-09T17:51:43+00:00 17440 exiting
2020-06-09T17:51:44+00:00 17515 Done './knitr'
2020-06-09T17:51:44+00:00 17515 Starting './ksql'
2020-06-09T17:51:55+00:00 17515 Done './ksql'
2020-06-09T17:51:55+00:00 17515 exiting
The output, in not_semver.csv:
./gt,
./calendar,
./docker-self-service-password,2.7 2.8 3.0
./keyring,
./ggnomics,
./callr,
./ggplot2,ggplot2-0.7 ggplot2-0.8 ggplot2-0.8.1 ggplot2-0.8.2 ggplot2-0.8.3 ggplot2-0.8.5 ggplot2-0.8.6 ggplot2-0.8.7 ggplot2-0.8.8 ggplot2-0.8.9 ggplot2-0.9.0 ggplot2-0.9.1 ggplot2-0.9.2 ggplot2-0.9.2.1 ggplot2-0.9.3 ggplot2-0.9.3.1 show
./nanodbc,
./nostalgy,
./grid,0.1 0.2 0.5 0.5-1 0.6 0.6-1 0.7-1 0.7-2 0.7-3 0.7-4
./knitr,doc v0.1 v0.2 v0.3 v0.4 v0.5 v0.6 v0.7 v0.8 v0.9 v1.0 v1.1 v1.10 v1.11 v1.12 v1.13 v1.14 v1.15 v1.16 v1.17 v1.18 v1.19 v1.2 v1.20 v1.3 v1.4 v1.5 v1.6 v1.7 v1.8 v1.9
./ksql,0.1-pre1 0.1-pre10 0.1-pre2 0.1-pre4 0.1-pre5 0.1-pre6 0.1-pre7 0.1-pre8 0.1-pre9 0.3 v0.2 v0.2-rc0 v0.2-rc1 v0.3 v0.3-rc0 v0.3-rc1 v0.3-rc2 v0.3-rc3 v0.3-temp v0.4 v0.4-rc0 v0.4-rc1 v0.5 v0.5-rc0 v0.5-rc1 v4.1.0-rc1 v4.1.0-rc2 v4.1.0-rc3 v4.1.0-rc4 v4.1.1-rc1 v4.1.1-rc2 v4.1.1-rc3 v4.1.2-beta180719000536 v4.1.2-beta3 v4.1.2-rc1 v4.1.3-beta180814192459 v4.1.3-beta180828173526 v5.0.0-beta1 v5.0.0-beta10 v5.0.0-beta11 v5.0.0-beta12 v5.0.0-beta14 v5.0.0-beta15 v5.0.0-beta16 v5.0.0-beta17 v5.0.0-beta18 v5.0.0-beta180622225242 v5.0.0-beta180626015140 v5.0.0-beta180627203620 v5.0.0-beta180628184550 v5.0.0-beta180628221539 v5.0.0-beta180629053850 v5.0.0-beta180630224559 v5.0.0-beta180701010229 v5.0.0-beta180701053749 v5.0.0-beta180701175910 v5.0.0-beta180701205239 v5.0.0-beta180702185100 v5.0.0-beta180702222458 v5.0.0-beta180706202823 v5.0.0-beta180707005130 v5.0.0-beta180707072142 v5.0.0-beta180718203558 v5.0.0-beta180722214927 v5.0.0-beta180723195256 v5.0.0-beta180726003306 v5.0.0-beta180730183336 v5.0.0-beta19 v5.0.0-beta2 v5.0.0-beta20 v5.0.0-beta21 v5.0.0-beta22 v5.0.0-beta23 v5.0.0-beta24 v5.0.0-beta25 v5.0.0-beta26 v5.0.0-beta27 v5.0.0-beta28 v5.0.0-beta29 v5.0.0-beta3 v5.0.0-beta30 v5.0.0-beta31 v5.0.0-beta32 v5.0.0-beta33 v5.0.0-beta5 v5.0.0-beta6 v5.0.0-beta7 v5.0.0-beta8 v5.0.0-beta9 v5.0.0-rc1 v5.0.0-rc3 v5.0.0-rc4 v5.0.1-beta180802235906 v5.0.1-beta180812233236 v5.0.1-beta180824214627 v5.0.1-beta180826190446 v5.0.1-beta180828173436 v5.0.1-beta180830182727 v5.0.1-beta180902210116 v5.0.1-beta180905054336 v5.0.1-beta180909000146 v5.0.1-beta180909000436 v5.0.1-beta180911213156 v5.0.1-beta180913003126 v5.0.1-beta180914024526 v5.0.1-beta181008233543 v5.0.1-beta181018200736 v5.0.1-rc1 v5.0.1-rc2 v5.0.1-rc3 v5.0.2-beta181116204629 v5.0.2-beta181116204811 v5.0.2-beta181116205152 v5.0.2-beta181117022246 v5.0.2-beta181118024524 v5.0.2-beta181119063215 v5.0.2-beta181119185816 v5.0.2-beta181126211008 v5.1.0-beta180611231144 v5.1.0-beta180612043613 v5.1.0-beta180612224009 v5.1.0-beta180613013021 v5.1.0-beta180614233101 v5.1.0-beta180615005408 v5.1.0-beta180618191747 v5.1.0-beta180618214711 v5.1.0-beta180618223247 v5.1.0-beta180618225004 v5.1.0-beta180619025141 v5.1.0-beta180620180431 v5.1.0-beta180620180739 v5.1.0-beta180620183559 v5.1.0-beta180622181348 v5.1.0-beta180626014959 v5.1.0-beta180627203509 v5.1.0-beta180628064520 v5.1.0-beta180628184841 v5.1.0-beta180630224439 v5.1.0-beta180701010040 v5.1.0-beta180701175749 v5.1.0-beta180702063039 v5.1.0-beta180702063440 v5.1.0-beta180702214311 v5.1.0-beta180702220040 v5.1.0-beta180703024529 v5.1.0-beta180706202701 v5.1.0-beta180707004950 v5.1.0-beta180718203536 v5.1.0-beta180722215127 v5.1.0-beta180723023347 v5.1.0-beta180723173636 v5.1.0-beta180724024536 v5.1.0-beta180730185716 v5.1.0-beta180812233046 v5.1.0-beta180820223106 v5.1.0-beta180824214446 v5.1.0-beta180828022857 v5.1.0-beta180828173516 v5.1.0-beta180829024526 v5.1.0-beta180905054157 v5.1.0-beta180911213206 v5.1.0-beta180912202326 v5.1.0-beta180917172706 v5.1.0-beta180919183606 v5.1.0-beta180928000756 v5.1.0-beta180929024526 v5.1.0-beta201806191956 v5.1.0-beta201806200051 v5.1.0-beta34 v5.1.0-beta35 v5.1.0-beta36 v5.1.0-beta37 v5.1.0-beta38 v5.1.0-beta39 v5.1.0-rc1 v6.0.0-beta181009070836 v6.0.0-beta181009071126 v6.0.0-beta181009071136 v6.0.0-beta181011024526
To reduce verbosity, you could remove logging and such, most of this output was intended to demonstrate the timing and running.
As another alternative, consider something like this:
log() {
now=$(date -Isec --utc)
echo "${now} ${*}" > /dev/stderr
}
# I don't have semver otherwise available, so a knockoff replacement
function is_semver() {
echo "$*" | egrep -q "^v?[0-9]+\.[0-9]+\.[0-9]+$"
}
function print_something() {
local repo_name=$1 tag_name=
bad=$(
git tag -l | while read tag_name ; do
is_semver "${tag_name}" || echo -n "${tag_name} "
done
)
echo "${repo_name},${bad}"
}
csvdir=$(mktemp -d not_semver_tempdir.XXXXXX)
csvdir=$(realpath "${csvdir}")/
log "Temp Directory: ${csvdir}"
while read -r repo_dir ; do
log "Starting '${repo_dir}'"
(
if [ -d "${repo_dir}" ]; then
repo_name=$(basename "${repo_dir}")
tmpfile=$(mktemp -p "${csvdir}")
tmpfile=$(realpath "${tmpfile}")
cd "${repo_dir}"
print_something "${repo_name}" > "${tmpfile}" 2> /dev/null
fi
) &
done
wait
outfile=$(mktemp not_semver_XXXXXX.csv)
cat ${csvdir}* > "${outfile}"
# rm -rf "${csvdir}" # uncomment when you're comfortable/confident
log "Output: ${outfile}"
I don't like it as much, admittedly, but its premise is that it creates a temporary directory in which each repo process will write its own file. Once all backgrounded jobs are complete (i.e., the wait near the end), all files are concatenated into an output.
Running it (without xargs):
$ find . -maxdepth 2 -iname .git | sed -e 's,/\.git,,g' | head -n 12 | \
~/StackOverflow/5783481/62283574.sh
2020-06-10T14:48:18+00:00 Temp Directory: /c/Users/r2/Projects/github/not_semver_tempdir.YeyaNY/
2020-06-10T14:48:18+00:00 Starting './calendar'
2020-06-10T14:48:18+00:00 Starting './callr'
2020-06-10T14:48:18+00:00 Starting './docker-self-service-password'
2020-06-10T14:48:18+00:00 Starting './ggnomics'
2020-06-10T14:48:18+00:00 Starting './ggplot2'
2020-06-10T14:48:19+00:00 Starting './grid'
2020-06-10T14:48:19+00:00 Starting './gt'
2020-06-10T14:48:19+00:00 Starting './keyring'
2020-06-10T14:48:19+00:00 Starting './knitr'
2020-06-10T14:48:19+00:00 Starting './ksql'
2020-06-10T14:48:19+00:00 Starting './nanodbc'
2020-06-10T14:48:19+00:00 Starting './nostalgy'
2020-06-10T14:48:38+00:00 Output: not_semver_CLy098.csv
r2#d2sb2 MINGW64 ~/Projects/github
$ cat not_semver_CLy098.csv
keyring,
ksql,0.1-pre1 0.1-pre10 0.1-pre2 0.1-pre4 0.1-pre5 0.1-pre6 0.1-pre7 0.1-pre8 0.1-pre9 0.3 v0.2 v0.2-rc0 v0.2-rc1 v0.3 v0.3-rc0 v0.3-rc1 v0.3-rc2 v0.3-rc3 v0.3-temp v0.4 v0.4-rc0 v0.4-rc1 v0.5 v0.5-rc0 v0.5-rc1 v4.1.0-rc1 v4.1.0-rc2 v4.1.0-rc3 v4.1.0-rc4 v4.1.1-rc1 v4.1.1-rc2 v4.1.1-rc3 v4.1.2-beta180719000536 v4.1.2-beta3 v4.1.2-rc1 v4.1.3-beta180814192459 v4.1.3-beta180828173526 v5.0.0-beta1 v5.0.0-beta10 v5.0.0-beta11 v5.0.0-beta12 v5.0.0-beta14 v5.0.0-beta15 v5.0.0-beta16 v5.0.0-beta17 v5.0.0-beta18 v5.0.0-beta180622225242 v5.0.0-beta180626015140 v5.0.0-beta180627203620 v5.0.0-beta180628184550 v5.0.0-beta180628221539 v5.0.0-beta180629053850 v5.0.0-beta180630224559 v5.0.0-beta180701010229 v5.0.0-beta180701053749 v5.0.0-beta180701175910 v5.0.0-beta180701205239 v5.0.0-beta180702185100 v5.0.0-beta180702222458 v5.0.0-beta180706202823 v5.0.0-beta180707005130 v5.0.0-beta180707072142 v5.0.0-beta180718203558 v5.0.0-beta180722214927 v5.0.0-beta180723195256 v5.0.0-beta180726003306 v5.0.0-beta180730183336 v5.0.0-beta19 v5.0.0-beta2 v5.0.0-beta20 v5.0.0-beta21 v5.0.0-beta22 v5.0.0-beta23 v5.0.0-beta24 v5.0.0-beta25 v5.0.0-beta26 v5.0.0-beta27 v5.0.0-beta28 v5.0.0-beta29 v5.0.0-beta3 v5.0.0-beta30 v5.0.0-beta31 v5.0.0-beta32 v5.0.0-beta33 v5.0.0-beta5 v5.0.0-beta6 v5.0.0-beta7 v5.0.0-beta8 v5.0.0-beta9 v5.0.0-rc1 v5.0.0-rc3 v5.0.0-rc4 v5.0.1-beta180802235906 v5.0.1-beta180812233236 v5.0.1-beta180824214627 v5.0.1-beta180826190446 v5.0.1-beta180828173436 v5.0.1-beta180830182727 v5.0.1-beta180902210116 v5.0.1-beta180905054336 v5.0.1-beta180909000146 v5.0.1-beta180909000436 v5.0.1-beta180911213156 v5.0.1-beta180913003126 v5.0.1-beta180914024526 v5.0.1-beta181008233543 v5.0.1-beta181018200736 v5.0.1-rc1 v5.0.1-rc2 v5.0.1-rc3 v5.0.2-beta181116204629 v5.0.2-beta181116204811 v5.0.2-beta181116205152 v5.0.2-beta181117022246 v5.0.2-beta181118024524 v5.0.2-beta181119063215 v5.0.2-beta181119185816 v5.0.2-beta181126211008 v5.1.0-beta180611231144 v5.1.0-beta180612043613 v5.1.0-beta180612224009 v5.1.0-beta180613013021 v5.1.0-beta180614233101 v5.1.0-beta180615005408 v5.1.0-beta180618191747 v5.1.0-beta180618214711 v5.1.0-beta180618223247 v5.1.0-beta180618225004 v5.1.0-beta180619025141 v5.1.0-beta180620180431 v5.1.0-beta180620180739 v5.1.0-beta180620183559 v5.1.0-beta180622181348 v5.1.0-beta180626014959 v5.1.0-beta180627203509 v5.1.0-beta180628064520 v5.1.0-beta180628184841 v5.1.0-beta180630224439 v5.1.0-beta180701010040 v5.1.0-beta180701175749 v5.1.0-beta180702063039 v5.1.0-beta180702063440 v5.1.0-beta180702214311 v5.1.0-beta180702220040 v5.1.0-beta180703024529 v5.1.0-beta180706202701 v5.1.0-beta180707004950 v5.1.0-beta180718203536 v5.1.0-beta180722215127 v5.1.0-beta180723023347 v5.1.0-beta180723173636 v5.1.0-beta180724024536 v5.1.0-beta180730185716 v5.1.0-beta180812233046 v5.1.0-beta180820223106 v5.1.0-beta180824214446 v5.1.0-beta180828022857 v5.1.0-beta180828173516 v5.1.0-beta180829024526 v5.1.0-beta180905054157 v5.1.0-beta180911213206 v5.1.0-beta180912202326 v5.1.0-beta180917172706 v5.1.0-beta180919183606 v5.1.0-beta180928000756 v5.1.0-beta180929024526 v5.1.0-beta201806191956 v5.1.0-beta201806200051 v5.1.0-beta34 v5.1.0-beta35 v5.1.0-beta36 v5.1.0-beta37 v5.1.0-beta38 v5.1.0-beta39 v5.1.0-rc1 v6.0.0-beta181009070836 v6.0.0-beta181009071126 v6.0.0-beta181009071136 v6.0.0-beta181011024526
knitr,doc v0.1 v0.2 v0.3 v0.4 v0.5 v0.6 v0.7 v0.8 v0.9 v1.0 v1.1 v1.10 v1.11 v1.12 v1.13 v1.14 v1.15 v1.16 v1.17 v1.18 v1.19 v1.2 v1.20 v1.3 v1.4 v1.5 v1.6 v1.7 v1.8 v1.9
calendar,
ggplot2,ggplot2-0.7 ggplot2-0.8 ggplot2-0.8.1 ggplot2-0.8.2 ggplot2-0.8.3 ggplot2-0.8.5 ggplot2-0.8.6 ggplot2-0.8.7 ggplot2-0.8.8 ggplot2-0.8.9 ggplot2-0.9.0 ggplot2-0.9.1 ggplot2-0.9.2 ggplot2-0.9.2.1 ggplot2-0.9.3 ggplot2-0.9.3.1 show
nostalgy,
callr,
docker-self-service-password,2.7 2.8 3.0
grid,0.1 0.2 0.5 0.5-1 0.6 0.6-1 0.7-1 0.7-2 0.7-3 0.7-4
ggnomics,
nanodbc,
gt,
use a variable or temp file for buffering lines. random file name is used
($0 = script name, $! = most recently background PID)
make sure you have write permissions. if you are worried about eMMC Flash Memory wear-out or write speeds you can also use shared-memory /run/shm
#!/bin/bash
print_not_semver_line() {
# random file name for line buffering
local tmpfile="${0%.*}${!:-0}.tmp~"
touch "$tmpfile" || return 1
# redirect stdout into different tmp file
echo -n "$repo_name," > "$tmpfile"
git tag -l | while read -r tag_name;do
semver $tag_name > /dev/null || echo -n "$tag_name " >> "$tmpfile"
done
echo "" >> "$tmpfile"
# print the whole line from one single ride
cat "$tmpfile" && rm "$tmpfile" && return 0
}
however, it is recommended to limit the maximum number of background processes. for the above example you can count open files with lsof
this function is waiting for given file name. it will check for similar file names and wait until number of open files is below allowed maximum. use it in your loop
first argument is mandatory file name
second argument is optional limit (default 4)
third argument is optional frequency for lsof
usage: wait_of <file> [<limit>] [<freq>]
# wait for open files (of)
wait_of() {
local pattern="$1" limit=${2:-4} time=${3:-1} path of
# check path
path="${pattern%/*}"
pattern="${pattern##*/}"
[ "$path" = "$pattern" ] && path=.
[ -e "$path" ] && [ -d "$(realpath "$path")" ] || return 1
# convert file name into regex
pattern="${pattern//[0-9]/0}"
while [[ "$pattern" =~ "00" ]]
do
pattern="${pattern//00/0}"
done
pattern="${pattern//0/[0-9]*}"
pattern="${pattern//[[:space:]]/[[:space:]]}"
# check path with regex for open files > 4 and wait
of=$(lsof -t "$path"/$pattern 2> /dev/null | wc -l)
while (( ${of:-0} > $limit ))
do
of=$(lsof -t "$path"/$pattern 2> /dev/null | wc -l)
sleep $time
done
return 0
}
# make sure only give one single tmp file name
wait_of "${0%.*}${!:-0}.tmp~" || exit 2
print_not_semver_line >> $csv_name &

How to run a script with correct permissions from custom udev rule?

I set up a custom udev rule triggered when my bluetooth mouse connects:
ACTION=="add" \
, ATTRS{idProduct}=="XXX" \
, ATTRS{idVendor}=="XXX" \
, ENV{DISPLAY}=":0" \
, ENV{XAUTHORITY}="/run/user/1000/gdm/Xauthority" \
, RUN+="/home/XXX/.scripts/mouse_connected.sh"
udevadm test succeeds, everything seems fine.
Then i created my script that should be triggered:
#!/bin/bash
if (xinput list | grep Maus | grep -o id=[0-9]* | grep -o [0-9]*$);
then xinput set-button-map "$(xinput list | grep Maus | grep -o id=[0-9]* | grep -o [0-9]*$)" 1 1 3 && logger "success";
else logger "fail" && sleep 3;
fi
when I manually run the script in terminal using "bash '/home/XXX/.scripts/mouse_connected.sh'" it works as expected. But when it gets successfully triggered via my custom udev rule it fails. - But why?

Execute a code for 5 seconds and then stop it

The goal of code is that, I want to make a random tcp traffic using iperf and capture it over 5, 10, 15, 20 seconds using tcpdump. In addition, capturing the throughput is also important for me. My problem is that, I would like to execute code1, code2, code3 and code4 for 5, 10, 15 and 20 seconds in bash. However I don't know how to put the mentioned condition for it. Here is my code:
for Test_duration in 5 10 15 20
do
echo “Test performing with $Test_duration duration”
sudo tcpdump -G 10 -W 2 -w /tmp/scripttest_$Test_duration -i h1-eth0 &
while true; do
#code1
time2=$(($RANDOM%20+1))&
pksize2=$(($RANDOM%1000+200))&
iperf -c 10.0.0.2 -t $time2 -r -l $pksize2 >> /media/sf_sharedsaeed/throughtput/iperthroughput_host2_$Test_duration.txt &\
#code2
time3=$(($RANDOM%20+1))&
pksize3=$(($RANDOM%1000+200))&
iperf -c 10.0.0.3 -t $time3 -r -l $pksize3 >> /media/sf_sharedsaeed/throughtput/iperthroughput_host3_$Test_duration.txt &\
#code3
time4=$(($RANDOM%20+1))&
pksize4=$(($RANDOM%1000+200))&
iperf -c 10.0.0.4 -t $time4 -r -l $pksize4 >> /media/sf_sharedsaeed/throughtput/iperthroughput_host4_$Test_duration.txt &\
#code4
time5=$(($RANDOM%20+1))&
pksize5=$(($RANDOM%1000+200))&
iperf -c 10.0.0.5 -t $time5 -r -l $pksize5 >> /media/sf_sharedsaeed/throughtput/iperthroughput_host5_$Test_duration.txt &\
done
done
Another constraint is that, code1, code2, code3 and code4 should be executed at the same time so, I used &.
Please help me what should I replace instead of while true; to have periodic execution of codes. Can any body help me?
You could do that by using a background subshell that creates a simple file lock on expiration that you detect from your while loops. Here is an example based on a simplified version of your code:
for Test_duration in 5 10 15 20
do
# TIMEOUT_LOCK will be your file lock
rm -f TIMEOUT_LOCK
# next command will run in a parallel subshell at the background
(sleep $Test_duration; touch TIMEOUT_LOCK) &
echo “Test performing with $Test_duration duration”
while true; do
# check whether current timeout (5 or 10 or 15 ...) has occured
if [ -f TIMEOUT_LOCK ]; then rm -f TIMEOUT_LOCK; break; fi
# do your stuff here - I'm just outputing dots and sleeping
echo -n "."
sleep 1
done
echo ""
done
The output of this code is:
“Test performing with 5 duration”
.....
“Test performing with 10 duration”
..........
“Test performing with 15 duration”
...............
“Test performing with 20 duration”
....................

I want to use parallel-ssh to run a bash script on multiple servers, but it simple prints the echo statements

I have a bash script called sr_run_batch.sh which does super resolution of images. Now I want to do testing on different servers in parallel at the same time. ie. 1 Virtual machine at one given point of time. then 2 virtual machines at one point of time , 3 and then 4.
I tried writing into it the commands
for host in $(cat hosts.txt); do ssh "$host" "$command" >"output.$host"; done
ssh-keygen && for host in $(cat hosts.txt); do ssh-copy-id $host; done
where the file hosts.txt contains the list of servers: username#ip(format) but when I run this, it gives me substitution error
Hence, I tried pssh (parallel-ssh)
pssh -h hosts-file -l username -P $command
command being ./sr_run_batch.sh
but it didn't run, so I modified this to
pssh -h hosts-file -l ben -P -I<./sr_run_batch.sh
But, for some unknown reason, it just prints the echo statements in the code.
here is the code :
NList=(5)
VList=(1)
FList=("input/flower1.jpg" "input/flower2.jpg" "input/flower3.jpg" "input/flower4.jpg")
IList=("320X240" "640X480" "1280X960" "1920X1200")
SList=(2 3)
for VM in ${VList[#]}; do
for ((index=0; index < ${#FList};)) do
file=$FList[$index]
image_size=$IList[$index]
width=`echo $image_size|cut -d "X" -f1`
height=`echo $image_size|cut -d "X" -f2`
for scale_factor in ${SList[#]}; do
for users in ${NList[#]}; do
echo "V: $VM, " "F: $file, " "S: $scale_factor, " "I: $width $height , " "N: $users"
for i in `seq 1 $users` ; do
./sr_run_once.sh $file $width $height $scale_factor &
done
wait
done # for users
done # for scale_factor
done # for index
done # for VM
exit 0
Have you also tried to use pssh with a simple bash-script so see if the communication is set up ok?
$ pssh -h hosts.txt -A -l ben -P -I<./uptime.sh
Warning: do not enter your password if anyone else has superuser
privileges or access to your account.
Password:
10.0.0.67: 11:06:50 up 28 min, 2 users, load average: 0.00, 0.00, 0.00
[1] 11:06:50 [SUCCESS] 10.0.0.67
10.0.0.218: 11:06:50 up 24 min, 2 users, load average: 0.00, 0.05, 0.20
[2] 11:06:50 [SUCCESS] 10.0.0.218

BASH/shell script, blkid fails with newlines in drive labels

I have a BASH/shell script, running on Linux, but it sometimes has a problem getting the drive labels, using blkid.. blkid gets the drive labels nad UUIDs of the given drive.
Example of blkid output:
# blkid /dev/sda1
/dev/sda1: LABEL="Home" UUID="f1e5e82b-1c75-4fd7-8841-6ad766152dcf" TYPE="ext2"
The problem occurs when the drive label has a newline character... I have included the relevant function. The problem seems to be with the eval command.. Can someone help me?
blkid_name_generator() {
#Gather blkid output and filter out required device
if [ ! -f /tmp/blkid.txt ]; then
blkid -c /dev/null > /tmp/blkid.txt
fi
blkid_cleaner &
LABEL=""
LABEL1=""
eval $( cat /tmp/blkid.txt | grep "$#:" | cut --delimiter=" " -f 2- | sed -e 's/ /;/g')
LABEL1=${LABEL//;/ } # akita beta4 fix: remove newlines below
LABEL=`echo $LABEL1 | tr -d '\n'` # akita beta5 fix, remove newlines from drive label
#Generate drive label
[ "$LABEL" = "" ] && ONEDRVLABEL="$#\n \n " || ONEDRVLABEL="${LABEL}\n($#)\n "
echo "${LABEL}" and "${ONEDRVLABEL}"
}
blkid_name_generator /dev/sda1
Here's my bash snippet:
while read curline; do
if [[ ${curline:0:1} == "/" ]]; then
[[ $prevline ]] && echo "$prevline"
prevline="$curline"
else
prevline+="$curline"
fi
done < $INPUTFILE
[[ $prevline ]] && echo "$prevline"
Here's my test file:
/dev/sda1: LABEL="Boot" UUID="fdc5e51d-3da6-4edf-bf07-6397b1765797" TYPE="ext2"
/dev/sda2: LABEL="Root" UUID="f883df24-1b93-46fb-8990-840774c380c4" TYPE="ext4"
/dev/sda3: LABEL="Usr" UUID="ba759de3-e0e4-4603-a324-f11dc25fa784" TYPE="reiserfs"
/dev/sdb1: LABEL="Persis
tents" UUID="50b81ef0-a38b-4677-b9d9-9548b29ce2bb" TYPE="ext4"
/dev/sdc1: LABEL="Tempo
raries" UUID="a649c535-1a8a-4b2b-b0a0-4afbdc60a3bc" TYPE="reiserfs"
/dev/sdd1: LABEL="Usr
Portage" UUID="a228148e-6405-4bbe-990f-df6eaebb1b1d" TYPE="reiserfs"
/dev/sda1: LABEL="Boot" UUID="fdc5e51d-3da6-4edf-bf07-6397b1765797" TYPE="ext2"
/dev/sda2: LABEL="Root" UUID="f883df24-1b93-46fb-8990-840774c380c4" TYPE="ext4"
Here's my test file after processed by the bash snippet:
/dev/sda1: LABEL="Boot" UUID="fdc5e51d-3da6-4edf-bf07-6397b1765797" TYPE="ext2"
/dev/sda2: LABEL="Root" UUID="f883df24-1b93-46fb-8990-840774c380c4" TYPE="ext4"
/dev/sda3: LABEL="Usr" UUID="ba759de3-e0e4-4603-a324-f11dc25fa784" TYPE="reiserfs"
/dev/sdb1: LABEL="Persistents" UUID="50b81ef0-a38b-4677-b9d9-9548b29ce2bb" TYPE="ext4"
/dev/sdc1: LABEL="Temporaries" UUID="a649c535-1a8a-4b2b-b0a0-4afbdc60a3bc" TYPE="reiserfs"
/dev/sdd1: LABEL="UsrPortage" UUID="a228148e-6405-4bbe-990f-df6eaebb1b1d" TYPE="reiserfs"
/dev/sda1: LABEL="Boot" UUID="fdc5e51d-3da6-4edf-bf07-6397b1765797" TYPE="ext2"
/dev/sda2: LABEL="Root" UUID="f883df24-1b93-46fb-8990-840774c380c4" TYPE="ext4"
Hope this helps!
I don't see exactly what your script is intending to do (apart from the missing bits).
Here's what I came up with:
blkid | perl -ne 'print "$1\n" if m/LABEL="(.*?)"/o'
It will show this (on my system):
test^J123
WIN7VIRT
TEMP
Note how the newline in the label is depicted as ^J ? Viewing through a hex encoder shows that it is indeed a two-character combination 0x5e 0x4a
Hope you can get some further with that.
PS: 'proof' that the newline is in fact correctly in the volume label
tune2fs /dev/sdc1 -l
tune2fs 1.41.14 (22-Dec-2010)
Filesystem volume name: test
123
Last mounted on: /media/9868d90c-aede-4e7a-b105-d9312f8b17ab
Filesystem UUID: 9868d90c-aede-4e7a-b105-d9312f8b17ab
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
...
and on block level:
dd if=/dev/sdc1 count=10 | xxd | grep 123
0000470: b105 d931 2f8b 17ab 7465 7374 0a31 3233 ...1/...test.123
Thanks to this question... I could find a solution for another problem. Example (sda2 = ntfs partition):
blkid /dev/sda2
Segmentation Fault (core dump)
Hard to find troubleshooting guide for this issue (ubuntu based OS or any other)
The reason behind is a malformed LABEL in NTFS partition.
(of course, mal-formation we cant see, as newlines ;-)
Solution: Clear LABEL with GParted; then rewrite it or set a new LABEL.

Resources