I need to run the following command in a bash script.
The command needs to be run inside a GNU screen so I can see the progress. So the command needs to be in quotes, but because of that I am having problems with the syntax and the code isn't running properly.
I have a file in a remote server called textfile.txt. It looks like this.
The command gawk command runs fine on its own.
test-server-name 1
test-server-name 2
test-server-name 3
...
test-server-name 23
test-server-name 24
...
I run a screen command together with an ssh command that runs a gawk command to modify a line in the text file, in this case, it should look for test-server-name-1 and add a 0 next to it like this.
test-server-name 1 0
test-server-name 2
test-server-name 3
...
test-server-name 23
test-server-name 24
...
This is what my script looks like in my local server.
localhostname='test-server-name-1'
counter=1
function='textfile'
screen -dmS $counter "ssh -i ~/.ssh/ssh-key username#masteripaddress 'gawk -i inplace -v n='0' -v s='${localhostname}-${function}' '$1 == s { $2 = n } 1' /home/master/Documents/${function}.txt';exec bash;"
But when I run it, the script runs, and I get this error in the attached screen,
gawk: cmd. line:1: ==
gawk: cmd. line:1: ^ syntax error
How do I fix it? What characters need to be escaped
(Don't do this. It'll be much easier if you put a script on remote side.)
Just give you an example (with \-style escaping):
[STEP 101] $ # to run an awk command locally
[STEP 102] $ title='THE SUM: '
[STEP 103] $ printf '%d\n' {1..10} | awk -v title="$title" '{ sum += $1 } END { print title sum }'
THE SUM: 55
[STEP 104] $
[STEP 105] $ # to run the awk command thru screen + ssh
[STEP 106] $ # added 'sleep 1' for easy watching
[STEP 107] $ title='THE SUM: '
[STEP 108] $ screen -c /dev/null -m ssh 127.0.0.1 printf\ \'%d\\n\'\ \{1..10\}\ \|\ awk\ -v\ title=\'"$title"\'\ \'\{\ sum\ +=\ \$1\ \}\ END\ \{\ print\ title\ sum\ \}\'\;\ sleep\ 1
THE SUM: 55
[screen is terminating]
[STEP 109] $
Related
This question already has answers here:
How to read variables from file, with multiple variables per line?
(2 answers)
Closed last month.
I am trying to assign variables obtained by awk, from a 2 columned txt file.
To a command, which includes every two value as two variables in it.
For example, the file I use is;
foo.txt
10 20
33 40
65 78
my command is aiming to print ;
end=20 start=10
end=40 start=33
end=78 start=65
Basically, I want to iterate the code for every line, and for output, there will be two variables from the two columns of the input file.
I am not an awk expert (I am trying my best), what I could have done so far is this fusion;
while read -r line ; do awk '{ second_variable=$2 ; first_variable=$1 ; }' ; echo "end=$first_name start=$second_name"; done <foo.txt
but it only gives this output;
end= start=
only one time without any variable. I would appreciate any suggestion. Thank you.
In bash you only need while, read and printf:
while read -r start end
do printf 'end=%d start=%d\n' "$end" "$start"
done < foo.txt
end=20 start=10
end=40 start=33
end=78 start=65
With awk, you could do:
awk '{print "end=" $2, "start=" $1}' foo.txt
end=20 start=10
end=40 start=33
end=78 start=65
With sed you'd use regular expressions:
sed -E 's/([0-9]+) ([0-9]+)/end=\2 start=\1/' foo.txt
end=20 start=10
end=40 start=33
end=78 start=65
Just in Bash:
while read -r end start; do echo "end=$end start=$start"; done <foo.txt
What about using xargs?
xargs -n2 sh -c 'echo end=$1 start=$2' sh < file.txt
Demo
xargs -n2 sh -c 'echo end=$1 start=$2' sh <<INPUT
10 20
33 40
65 78
INPUT
Output
end=10 start=20
end=33 start=40
end=65 start=78
Hello I need help with some bash script.
I need to insert the below line into a file at a specific line number
su - test -c '$HOME/scripts/auto start' >> /home/test/auto_start.`date +%h_%d_%y`.log 2>&1
In the bigger problem statement, the line to be inserted at a specified line number & into the file, all are dynamic & need to be fetched from a csv file.
The csv file looks like:
$ cat test_file.csv
TARGET_NODE,TARGET_FILE,TARGET_LINE,NEW_LINE
10.10.10.10,test_file.csv,2,su - test -c '$HOME/scripts/auto start' >> /home/test/auto_start.`date +%h_%d_%y`.log 2>&1
Have tried using sed in many ways, but failed. We can't use python due to system limitations.
The simple way to test it:
seq 4|sed 's/<LINE_NUMBER>/<NEW_TEXT>/'
So we tried using:
seq 4|sed 's/2/'su - test -c '$HOME/scripts/auto start' >> /home/test/auto_start.`date +%h_%d_%y`.log 2>&1'/'
You didn't say if you wanted the new line printed before or after or instead of the existing line so here's all 3:
$ seq 4 | awk -F, 'NR==FNR{line[$3]=$4; next} FNR in line{print line[FNR]} 1' test_file.csv -
1
su - test -c '$HOME/scripts/auto start' >> /home/test/auto_start.`date +%h_%d_%y`.log 2>&1
2
3
4
.
$ seq 4 | awk -F, 'NR==FNR{line[$3]=$4; next} 1; FNR in line{print line[FNR]}' test_file.csv -
1
2
su - test -c '$HOME/scripts/auto start' >> /home/test/auto_start.`date +%h_%d_%y`.log 2>&1
3
4
.
$ seq 4 | awk -F, 'NR==FNR{line[$3]=$4; next} FNR in line{$0=line[FNR]} 1' test_file.csv -
1
su - test -c '$HOME/scripts/auto start' >> /home/test/auto_start.`date +%h_%d_%y`.log 2>&1
3
4
The above will work using any awk in any shell and for any string from that CSV on every UNIX box
If you have no system limitations for using perl :
#!/usr/bin/env bash
seq 4 | perl -pe '
BEGIN {
open CSV, shift; # Open test_file.csv
<CSV>; # Ignore first line
# Get line number and command
($n, $c) = <CSV> =~ /^(?:[^,]+,){2}([^,]+),(.*)/;
}
{ s/.*/$c/ if ($. == $n); } # Replace line with number $n
' test_file.csv
Just properly quote the line and use different s command separator because / is used in replacement string... escape special regex characters in replacement string, like & or \.
$ seq 4 | sed 's~2~su - test -c '\''$HOME/scripts/auto start'\'' >> /home/test/auto_start.`date +%h_%d_%y`.log 2>\&1~'
1
su - test -c '$HOME/scripts/auto start' >> /home/test/auto_start.`date +%h_%d_%y`.log 2>&1
3
4
with awk:
seq 4 | awk -vvar='su - test -c '\''$HOME/scripts/auto start'\'' >> /home/test/auto_start.`date +%h_%d_%y`.log 2>&1' 'NR==2{$0=var}1'
I discourage the usage of backticks `. Do not use them. Use $(...) instead.
I want to use a python3 script to generate a 'self-extracting' bash script which has an embedded .tar.gz archive as payload.
In bash I would simply do something like this:
printf "#!/bin/bash
PAYLOAD_LINE=\`awk '/^__PAYLOAD_BELOW__/ {print NR + 1; exit 0; }' \$0\`
tail -n+\$PAYLOAD_LINE \$0 | tar -xvz
#script that does something with the unpacked payload
exit 0
__PAYLOAD_BELOW__\n" > "tmpfile"
cat "tmpfile" "payload.tar.gz" > "myscript.sh"
What I tried in python is this:
with open('myscript.sh','wb') as script:
for line in open(payload.tar.gz, 'rb'):
script.write(line)
I can untar the resulting file manually with cat myscript.sh | tar -xvz
To add the bash script part (PAYLOAD_LINE= .... __PAYLOAD_BELOW__\n"), is there a more elegant way than to open myscript.sh a second time, but not in binary ('wb') mode?
Figured it out.
I just have to convert the string to bytes, before writing it to the file, so I don't have to open it twice.
script_header = """#!/bin/bash
PAYLOAD_LINE=\`awk '/^__PAYLOAD_BELOW__/ {print NR + 1; exit 0; }' \$0\`
tail -n+\$PAYLOAD_LINE \$0 | tar -xvz"""
with open('myscript.sh','wb') as script:
script.write(bytes(script_header + '\n', 'utf-8'))
for line in open(payload.tar.gz, 'rb'):
script.write(line)
The following is output of "cat /proc/softirqs " :
CPU0 CPU1 CPU2 CPU3
HI: 24 13 7 54
TIMER: 344095632 253285150 121234786 108207697
NET_TX: 2366955 319 695 316044
NET_RX: 16337920 16030558 250497436 117201444
BLOCK: 19631 2747 2353 5067051
BLOCK_IOPOLL: 0 0 0 0
TASKLET: 298 93 157 20965
SCHED: 74354472 28133393 30646119 26217748
HRTIMER: 4123645358 2409060621 2466360502 401470590
RCU: 26083738 17708780 15330534 16857905
My another machine has 24 cpu cores and the output is hard to read ,
I like the output to be only cpu0 , cpu2 , cpu4 , cpu6, ....
I know cut or awk might be ued to do that ,
but no idea how to use it to get even output columns .
Edit :
awk -F" " '{printf("%10s\t%s\n", $2,$4) }'
will get
CPU1 CPU3
24 7
344095632 121234786
2366955 695
16337920 250497436
19631 2353
0 0
298 157
74354472 30646119
4123645358 2466360502
26083738 15330534
unfortunately , CPU1 should be CPU0 , CPU3 should be CPU2 ,
the first line has only 4 columns , may I skip the first line
in this shell ?!
Edit2 :
watch -d "cat /proc/softirqs | awk -F" " '{printf("%10s\t%s\n",$2,$4)}' "
encounter errors like the following :
Every 2.0s: cat /proc/softirqs | awk -F '{print }' Tue Jun 21 10:23:22 2016
Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
Usage: awk [POSIX or GNU style options] [--] 'program' file ...
POSIX options: GNU long options: (standard)
-f progfile --file=progfile
-F fs --field-separator=fs
-v var=val --assign=var=val
Short options: GNU long options: (extensions)
-b --characters-as-bytes
-c --traditional
-C --copyright
-d[file] --dump-variables[=file]
-e 'program-text' --source='program-text'
-E file --exec=file
-g --gen-pot
-h --help
-L [fatal] --lint[=fatal]
-n --non-decimal-data
-N --use-lc-numeric
-O --optimize
-p[file] --profile[=file]
-P --posix
-r --re-interval
-S --sandbox
-t --lint-old
-V --version
To report bugs, see node `Bugs' in `gawk.info', which is
section `Reporting Problems and Bugs' in the printed version.
gawk is a pattern scanning and processing language.
By default it reads standard input and writes standard output.
Examples:
gawk '{ sum += $1 }; END { print sum }' file
gawk -F: '{ print $1 }' /etc/passwd
what else should I try ?!
Edit3 :
The final workable shell would like :
# define function encapsulating code; this prevents any need for extra layers of quoting
# or escaping.
run() {
awk 'NR>1{printf("%20s\t%10s\t%s\n",$1,$2,$4)}' </proc/softirqs|egrep 'TIMER|RX'
}
# export function
export -f run
# run function in subshell of watch, ensuring that that shell is bash
# (other shells may not honor exported functions)
watch -d "bash -c run"
One easy way to communicate code to a subprocess of watch that avoids escaping errors is to use an exported function:
# define function encapsulating code; this prevents any need for extra layers of quoting
# or escaping.
run() {
awk -F" " '{printf("%10s\t%s\n",$2,$4)}' </proc/softirqs
}
# export function
export -f run
# run function in subshell of watch, ensuring that that shell is bash
# (other shells may not honor exported functions)
watch "bash -c run"
To avoid the dependency on exported functions, one can also use declare -f to retrieve the function's source in an evalable form, and printf %q to escape it to survive processing by the outer shell invoked by watch:
run() {
awk -F" " '{printf("%10s\t%s\n",$2,$4)}' </proc/softirqs
}
printf -v run_str '%q' "$(declare -f run); run"
watch "bash -c $run_str"
To skip the first line, do:
awk -F" " 'NR>1{printf("%10s\t%s\n", $2,$4) }'
Why do you need -F" ", is a mystery to me. You can as well write:
awk 'NR>1{printf("%10s\t%s\n", $2,$4) }'
(As for the watch part, see other answer/s.)
I have a command dumpsys power with this output:
POWER MANAGER (dumpsys power)
Power Manager State: mDirty=0x0
mWakefulness=Awake #
mWakefulnessChanging=false
mIsPowered=false
mPlugType=0
mBatteryLevel=67 #
mBatteryLevelWhenDreamStarted=0
mDockState=0
mStayOn=false #
mProximityPositive=false
mBootCompleted=true #
mSystemReady=true #
mHalAutoSuspendModeEnabled=false
mHalInteractiveModeEnabled=true
mWakeLockSummary=0x0
mUserActivitySummary=0x1
mRequestWaitForNegativeProximity=false
mSandmanScheduled=false
mSandmanSummoned=false
mLowPowerModeEnabled=false #
mBatteryLevelLow=false #
mLastWakeTime=134887327 (59454 ms ago) #
mLastSleepTime=134881809 (64972 ms ago) #
mLastUserActivityTime=134946670 (111 ms ago)
mLastUserActivityTimeNoChangeLights=134794061 (152720 ms ago)
mLastInteractivePowerHintTime=134946670 (111 ms ago)
mLastScreenBrightnessBoostTime=0 (134946781 ms ago)
mScreenBrightnessBoostInProgress=false
mDisplayReady=true #
mHoldingWakeLockSuspendBlocker=false
mHoldingDisplaySuspendBlocker=true
Settings and Configuration:
mDecoupleHalAutoSuspendModeFromDisplayConfig=false
mDecoupleHalInteractiveModeFromDisplayConfig=true
mWakeUpWhenPluggedOrUnpluggedConfig=true
mWakeUpWhenPluggedOrUnpluggedInTheaterModeConfig=false
mTheaterModeEnabled=false
mSuspendWhenScreenOffDueToProximityConfig=false
mDreamsSupportedConfig=true
mDreamsEnabledByDefaultConfig=true
mDreamsActivatedOnSleepByDefaultConfig=false
mDreamsActivatedOnDockByDefaultConfig=true
mDreamsEnabledOnBatteryConfig=false
mDreamsBatteryLevelMinimumWhenPoweredConfig=-1
mDreamsBatteryLevelMinimumWhenNotPoweredConfig=15
mDreamsBatteryLevelDrainCutoffConfig=5
mDreamsEnabledSetting=false
mDreamsActivateOnSleepSetting=false
mDreamsActivateOnDockSetting=true
mDozeAfterScreenOffConfig=true
mLowPowerModeSetting=false
mAutoLowPowerModeConfigured=false
mAutoLowPowerModeSnoozing=false
mMinimumScreenOffTimeoutConfig=10000
mMaximumScreenDimDurationConfig=7000
mMaximumScreenDimRatioConfig=0.20000005
mScreenOffTimeoutSetting=60000 #
mSleepTimeoutSetting=-1
mMaximumScreenOffTimeoutFromDeviceAdmin=2147483647 (enforced=false)
mStayOnWhilePluggedInSetting=0
mScreenBrightnessSetting=102
mScreenAutoBrightnessAdjustmentSetting=-1.0
mScreenBrightnessModeSetting=1
mScreenBrightnessOverrideFromWindowManager=-1
mUserActivityTimeoutOverrideFromWindowManager=-1
mTemporaryScreenBrightnessSettingOverride=-1
mTemporaryScreenAutoBrightnessAdjustmentSettingOverride=NaN
mDozeScreenStateOverrideFromDreamManager=0
mDozeScreenBrightnessOverrideFromDreamManager=-1
mScreenBrightnessSettingMinimum=10
mScreenBrightnessSettingMaximum=255
mScreenBrightnessSettingDefault=102
Sleep timeout: -1 ms
Screen off timeout: 60000 ms
Screen dim duration: 7000 ms
Wake Locks: size=0 Suspend Blockers: size=4
PowerManagerService.WakeLocks: ref count=0
PowerManagerService.Display: ref count=1
PowerManagerService.Broadcasts: ref count=0
PowerManagerService.WirelessChargerDetector: ref count=0
Display Power: state=ON #
I want to get the lines marked with # in a format of:
mScreenOffTimeoutSetting=60000
mDisplayReady=true
***
ScreenOfftimeoutSetting = 60000
DisplayReady = true
The commands output can vary from device to device and some of the lines might not be there or are in a different place. Thus if the searched line isn't there no errors should be generated.
It's not clear what you want. Aou can use sed to extract variables form the file and do whatever you want with them. Here's an example:
sed -n -e 's/^mSomeName=\(.*\)/newVariable=\1/p' -e 's/^mOtherName=.*+\(.*\)/newVariable2=\1/p' myFile
Explanation:
-n don't output anything per default
-e an expression follows. It's required since we have multiple expressions in place
s/^mSomeName=\(.*\)/newVariable=\1/p if file starts (^) with mSomeName= capture what follows (\(.*\)), replace the line with newVariable=\1, where \1 is what got captured, and print it out (p)
's/^mOtherName=.+(.)/newVariable2=\1/p' similar to the previous expression but will capture whatere comes after a + sign and print it behind newVariable2
This does something like:
$ sed -n -e 's/^mSomeName=\(.*\)/newVariable=\1/p' -e 's/^mOtherName=.*+\(.*\)/newVariable2=\1/p' <<<$'mSomeName=SomeValue\nmOtherName=OtherValue+Somethingelse'
newVariable=SomeValue
newVariable2=Somethingelse
<<<$'...' is a way of passing a string with linebreaks \n directly to the command in bash. You can replace it with a file. This command just outputs a string, nothing will get changed.
If you need them in bash variables use eval:
$ eval $(sed -n -e 's/^mSomeName=\(.*\)/newVariable=\1/p' -e 's/^mOtherName=.*+\(.*\)/newVariable2=\1/p' <<<$'mSomeName=SomeValue\nmOtherName=OtherValue+Somethingelse')
$ echo newVariable=$newVariable - newVariable2=$newVariable2
newVariable=SomeValue - newVariable2=Somethingelse
eval will execute the string which in this case set the variable values:
$ eval a=1
$ echo $a
1
If you want to just use Grep command, you can use -A (After) and -B (Before) options and pipes.
This is a exemple with 2 lines.
File test.txt :
test
aieauieaui
test
caieaieaipe
mSomeName=SomeValue
mOtherName=OtherValue+Somethingelse
nothing
blabla
mSomeName=SomeValue2
mOtherName=OtherValue+Somethingelse2
The command to use :
grep -A 1 'mSomeName' test.txt |grep -B 1 'mOtherName'
The output :
mSomeName=SomeValue
mOtherName=OtherValue+Somethingelse
--
mSomeName=SomeValue2
mOtherName=OtherValue+Somethingelse2