In my project, addgpg-apt (https://launchpad.net/addgpg-apt), I'd like to be able to have input sent into the program. From that input, say...
Unable to verify signatures in PPA. Check that this is fixed: NO_PUBKEY <PGPkeyID>
... how can I, using basic Bash, grep, etc. get that <PGPkeyID> from the string and ignore everything else in the string? (Note that these errors are generated by apt-get/apt, and as such the end of the string is always NO_PUBKEY <PGPkeyID>)
In Java, this could be done with substring, and grab only that PGPkeyID based on the location of the phrase NO_PUBKEY, but I want this to be done in Bash only, so any solutions would be appreciated.
Piping to grep -o "NO_PUBKEY <.*>" | sed -e 's/.*<\(.*\)>.*/\1/'
will yield you:
PGPkeyID
Update
Assuming your input is like this:
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY CAFE0123DEADBEEF
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0123DEADBEEFCAFE
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY DEADBEEFCAFE0123
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY BEEFCAFE0123DEAD
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY CAFE0123DEADBEEF
The following command will extract the keys:
grep -o 'PUBKEY [A-F0-9]\{16\}' | cut -f2 -d" " | sort -u
Like this:
0123DEADBEEFCAFE
BEEFCAFE0123DEAD
CAFE0123DEADBEEF
DEADBEEFCAFE0123
You can do that in pure shell without spawning a process:
cat << EOF > file
...
Unable to verify signatures in PPA. Check that this is fixed: NO_PUBKEY <PGPkeyID1>
...
Unable to verify signatures in PPA. Check that this is fixed: NO_PUBKEY <PGPkeyID2>
...
EOF
cat file | while read line; do
if [[ $line == *\ NO_PUBKEY\ * ]]
then
echo ${line#* NO_PUBKEY }
fi
done
Related
I am trying to create a string with a query that will be save / send to another location, this string contains different variables.
The issue that I am having is that the echo of the variables are completely upside down and mix.
See code below:
tokenID=$(docker exec -ti $dockerContainerID /bin/sh -c "cat /tempdir/tokenfile.txt")
serverName="asdasd"
attQuery="$tokenID $serverName"
agentRegQuery="$./opt/mule/bin/amc_setup -H $attQuery"
echo TOKEN ID $tokenID
echo SERVER NAME $serverName
echo $attQuery
echo $agentRegQuery
Find below the output I am receiving:
TOKEN ID 29a6966f-fa0e-4f08-87eb-418722872d80---46407
SERVER NAME asdasd
asdasdf-fa0e-4f08-87eb-418722872d80---46407
asdasdmule/bin/amc_setup -H 29a6966f-fa0e-4f08-87eb-418722872d80---46407
There's a carriage return character at the end of the tokenID variable, probably because /tempdir/tokenfile.txt is in DOS/Windows format (lines end with carriage return+linefeed), not unix (lines end with just linefeed). When you print tokenID by itself, it looks ok, but if you print something else after that on the same line, it winds up overwriting the first part of the line. So when you print $attQuery, it prints this:
29a6966f-fa0e-4f08-87eb-418722872d80---46407[carriage return]
asdasd
...but with the second line printed on top of the first, so it comes out as:
asdasdf-fa0e-4f08-87eb-418722872d80---46407
The solution is to either convert the file to unix format (dos2unix will do this if you have it), or remove the carriage return in your script. You can do it like this:
tokenID=$(docker exec -ti $dockerContainerID /bin/sh -c "cat /tempdir/tokenfile.txt" | tr -d '\r')
I think everything works as it should
echo TOKEN ID $tokenID -> TOKEN ID 29a6966f-fa0e-4f08-87eb-418722872d80---46407
echo SERVER NAME $serverName -> SERVER NAME asdasd
echo $attQuery -> asdasdf-fa0e-4f08-87eb-418722872d80---46407
echo $agentRegQuery -> asdasdmule/bin/amc_setup -H 29a6966f-fa0e-4f08-87eb-418722872d80---46407
Why do you think something is wron here?
Best regards, Georg
I have an interesting problem that I can't seem to find the answer for. I am creating a simple app that will help my dev department auto launch docker containers with NginX and config files. My problem is, for some reason I can't get the bash script to store the name of a folder, while scanning the directory. Here is an extremely simple example of what I am talking about....
#!/bin/bash
getFolder() {
local __myResultFolder=$1
local folder
for d in */ ; do
$folder=$d
done
__myResultFolder=$folder
return $folder
}
getFolder FOLDER
echo "Using folder: $FOLDER"
I then save that simple script as folder_test.sh and put it in a folder where there is only one folder, change owner to me, and give it correct permissions. However, when I run the script I keep getting the error...
./folder_test.sh: 8 ./folder_test.sh: =test_folder/: not found
I have tried putting the $folder=$d part in different types of quotes, but nothing works. I have tried $folder="'"$d"'", $folder=`$d`, $folder="$d" but none of it works. Driving me insane, any help would be greatly appreciated. Thank you.
If you want to save your result into a named variable, what you're doing is called "indirect assignment"; it's covered in BashFAQ #6.
One way is the following:
#!/bin/bash
# ^^^^ not /bin/sh; bash is needed for printf -v
getFolder() {
local __myResultFolder=$1
local folder d
for d in */ ; do
folder=$d
done
printf -v "$__myResultFolder" %s "$folder"
}
getFolder folderName
echo "$folderName"
Other approaches include:
Using read:
IFS= read -r -d '' "$__myResultFolder" < <(printf '%s\0' "$folder")
Using eval (very, very carefully):
# note \$folder -- we're only trusting the destination variable name
# ...not trusting the content.
eval "$__myResultFolder=\$folder"
Using namevars (only if using new versions of bash):
getFolder() {
local -n __myResultFolder=$1
# ...your other logic here...
__myResultFolder=$folder
}
The culprit is the line
$folder=$d
which is treating the folder names to stored with a = sign before and tried to expand it in that name i.e. literally treats the name =test_folder/ as an executable to be run under shell but does not find a file of that name. Change it to
folder=$d
Also, bash functions' return value is only restricted to integer types and you cannot send a string to the calling function. If you wanted to send a non-zero return code to the calling function on $folder being empty you could add a line
if [ -z "$folder" ]; then return 1; else return 0; fi
(or) if you want to return a string value from the function, do not use return, just do echo of the name and use command-substitution with the function name, i.e.
getFolder() {
local __myResultFolder=$1
local folder
for d in */ ; do
folder=$d
done
__myResultFolder=$folder
echo "$folder"
}
folderName=$(getFolder FOLDER)
echo "$folderName"
I have 100s of config files, each 10,000 to 20,000 lines long. These are config files for hardware. I need to search through all the config files to find the "profile" associated with a given cert name. There are several different versions of hardware software so the configs files are somewhat different. However the profile name is always above the cert. The profile name does not necessarily contain the cert name.
Example of Profile Names:
clientssl_www.profile-cert
clientssl_www.example.com-cert
Example of Cert Name:
www.example.com.crt
Example sections of config:
profile clientssl clientssl_www.profile-cert {
defaults from clientssl
key "www.example.com.key"
cert "www.example.com.crt"
chain "Intermediate-bundle.crt"
options {
cipher server preference
dont insert empty fragments
no sslv2
}
}
ltm profile client-ssl /Common/clientssl_www.example.com-cert {
app-service none
cert /Common/www.example.com.crt
cert-key-chain {
www.example.com_www.example.com {
cert /Common/www.example.com.crt
chain /Common/Intermediate-bundle.crt
key /Common/www.example.com.key
}
}
chain /Common/Intermediate-bundle.crt
ciphers
key /Common/www.example.com.key
options { dont-insert-empty-fragments cipher-server-preference no-sslv2 }
}
I cannot read the config files line by line as there are millions of lines and it simply takes too long.
I can find the cert name with grep using something like this:
$ grep www.example.com *file.conf | egrep 'cert "|cert /Common'
Which gives me something like this:
cert "www.example.com.crt"
cert /Common/www.example.com.crt
cert /Common/www.example.com.crt
I need to find the 'profile name' that is above my search for a given cert name.
Any suggestions?
Thanks!
You can use -B option of grep which comes handy in such cases. From the man pages for grep:
-B NUM, --before-context=NUM
Print NUM lines of leading context before matching lines. Places a line containing a group separator (--) between contiguous groups
of matches. With the -o or --only-matching option, this has no effect and a warning is given.
So, the pattern match will now be:
$ grep www.example.com *file.conf | egrep -B3 'cert "|cert /Common'
Output:
profile clientssl clientssl_www.profile-cert {
defaults from clientssl
key "www.example.com.key"
cert "www.example.com.crt"
--
ltm profile client-ssl /Common/clientssl_www.example.com-cert {
app-service none
cert /Common/www.example.com.crt
cert-key-chain {
www.example.com_www.example.com {
cert /Common/www.example.com.crt
However, you will still need to figure out some common pattern in the line containing profile name to single them out. It becomes difficult in your example to filter it further because in the first case, the profile name is three lines before the cert " pattern whereas in the second example, it is two lines before cert / pattern.
Another approach which i find better is to find some pattern in the profile name itself. If all profile names contain the string profile or if they have a pattern such as clientssl.*-cert, then the following pattern match will do what you need:
$ grep www.example.com *file.conf | egrep 'profile|clientssl.*-cert'
Output:
profile clientssl clientssl_www.profile-cert {
ltm profile client-ssl /Common/clientssl_www.example.com-cert {
Even better, if you know that the profile name starts with clientssl_ and ends with -cert, then
$ grep www.example.com *file.conf | grep -o clientssl_.*-cert
Output:
clientssl_www.profile-cert
clientssl_www.example.com-cert
This may be madness, but whenever I see sample data that fits Tcl's syntax rules, I look to produce a Tcl solution:
#!/usr/bin/env tclsh
proc unknown {cmdname args} {
set data [lindex $args end]
if {[set idx [lsearch -exact $data "cert"]] != -1 && [string match $::cert_pattern [lindex $data [incr idx]]]} {
set idx [expr {$cmdname eq "profile" ? 1 : [lsearch -exact $args "profile"] + 2}]
puts [lindex [split [lindex $args $idx] /] end]
}
}
set cert_pattern "*[lindex $argv 0]*"
foreach file [lrange $argv 1 end] {
source $file
}
Then
$ ./cert.tcl www.example.com file.conf
file.conf
clientssl_www.profile-cert
clientssl_www.example.com-cert
I won't bother to explain how it works unless there's a hue and cry.
I have the following code in my ~/.bashrc:
date=$(which date)
date() {
if [[ $1 == -R || $1 == --rfc-822 ]]; then
# Output RFC-822 compliant date string.
# e.g. Wed, 16 Dec 2009 15:18:11 +0100
$date | sed "s/[^ ][^ ]*$/$($date +%z)/"
else
$date "$#"
fi
}
This works fine, as far as I can tell. Is there a reason to avoid having a variable and a function with the same name?
It's alright apart from being confusing. Besides, they are not the same:
$ date=/bin/ls
$ type date
date is hashed (/bin/date)
$ type $date
/bin/ls is /bin/ls
$ moo=foo
$ type $moo
-bash: type: foo: not found
$ function date() { true; }
$ type date
date is a function
date ()
{
true*emphasized text*
}
$ which true
/bin/true
$ type true
true is a shell builtin
Whenever you type a command, bash looks in three different places to find that command. The priority is as follows:
shell builtins (help)
shell aliases (help alias)
shell functions (help function)
hashed binaries files from $PATH ('leftmost' folders scanned first)
Variables are prefixed with a dollar sign, which makes them different from all of the above. To compare to your example: $date and date are not the same thing. So It's not really possible to have the same name for a variable and a function because they have different "namespaces".
You may find this somewhat confusing, but many scripts define "method variables" at the top of the file. e.g.
SED=/bin/sed
AWK=/usr/bin/awk
GREP/usr/local/gnu/bin/grep
The common thing to do is type the variable names in capitals. This is useful for two purposes (apart from being less confusing):
There is no $PATH
Checking that all "dependencies" are runnable
You can't really check like this:
if [ "`which binary`" ]; then echo it\'s ok to continue.. ;fi
Because which will give you an error if binary has not yet been hashed (found in a path folder).
Since you always have to use $ to dereference a variable in Bash, you're free to use any name you like.
Beware of overriding a global, though.
See also:
http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_02.html
An alternative to using a variable: use bash's command keyword (see the manual or run help command from a prompt):
date() {
case $1 in
-R|--rfc-2822) command date ... ;;
*) command date "$#" ;;
esac
}
I have a Bash script that builds a string to run as a command
Script:
#! /bin/bash
matchdir="/home/joao/robocup/runner_workdir/matches/testmatch/"
teamAComm="`pwd`/a.sh"
teamBComm="`pwd`/b.sh"
include="`pwd`/server_official.conf"
serverbin='/usr/local/bin/rcssserver'
cd $matchdir
illcommando="$serverbin include='$include' server::team_l_start = '${teamAComm}' server::team_r_start = '${teamBComm}' CSVSaver::save='true' CSVSaver::filename = 'out.csv'"
echo "running: $illcommando"
# $illcommando > server-output.log 2> server-error.log
$illcommando
which does not seem to supply the arguments correctly to the $serverbin.
Script output:
running: /usr/local/bin/rcssserver include='/home/joao/robocup/runner_workdir/server_official.conf' server::team_l_start = '/home/joao/robocup/runner_workdir/a.sh' server::team_r_start = '/home/joao/robocup/runner_workdir/b.sh' CSVSaver::save='true' CSVSaver::filename = 'out.csv'
rcssserver-14.0.1
Copyright (C) 1995, 1996, 1997, 1998, 1999 Electrotechnical Laboratory.
2000 - 2009 RoboCup Soccer Simulator Maintenance Group.
Usage: /usr/local/bin/rcssserver [[-[-]]namespace::option=value]
[[-[-]][namespace::]help]
[[-[-]]include=file]
Options:
help
display generic help
include=file
parse the specified configuration file. Configuration files
have the same format as the command line options. The
configuration file specified will be parsed before all
subsequent options.
server::help
display detailed help for the "server" module
player::help
display detailed help for the "player" module
CSVSaver::help
display detailed help for the "CSVSaver" module
CSVSaver Options:
CSVSaver::save=<on|off|true|false|1|0|>
If save is on/true, then the saver will attempt to save the
results to the database. Otherwise it will do nothing.
current value: false
CSVSaver::filename='<STRING>'
The file to save the results to. If this file does not
exist it will be created. If the file does exist, the results
will be appended to the end.
current value: 'out.csv'
if I just paste the command /usr/local/bin/rcssserver include='/home/joao/robocup/runner_workdir/server_official.conf' server::team_l_start = '/home/joao/robocup/runner_workdir/a.sh' server::team_r_start = '/home/joao/robocup/runner_workdir/b.sh' CSVSaver::save='true' CSVSaver::filename = 'out.csv' (in the output after "runnning: ") it works fine.
You can use eval to execute a string:
eval $illcommando
your_command_string="..."
output=$(eval "$your_command_string")
echo "$output"
I usually place commands in parentheses $(commandStr), if that doesn't help I find bash debug mode great, run the script as bash -x script
don't put your commands in variables, just run it
matchdir="/home/joao/robocup/runner_workdir/matches/testmatch/"
PWD=$(pwd)
teamAComm="$PWD/a.sh"
teamBComm="$PWD/b.sh"
include="$PWD/server_official.conf"
serverbin='/usr/local/bin/rcssserver'
cd $matchdir
$serverbin include=$include server::team_l_start = ${teamAComm} server::team_r_start=${teamBComm} CSVSaver::save='true' CSVSaver::filename = 'out.csv'
./me casts raise_dead()
I was looking for something like this, but I also needed to reuse the same string minus two parameters so I ended up with something like:
my_exe ()
{
mysql -sN -e "select $1 from heat.stack where heat.stack.name=\"$2\";"
}
This is something I use to monitor openstack heat stack creation. In this case I expect two conditions, an action 'CREATE' and a status 'COMPLETE' on a stack named "Somestack"
To get those variables I can do something like:
ACTION=$(my_exe action Somestack)
STATUS=$(my_exe status Somestack)
if [[ "$ACTION" == "CREATE" ]] && [[ "$STATUS" == "COMPLETE" ]]
...
Here is my gradle build script that executes strings stored in heredocs:
current_directory=$( realpath "." )
GENERATED=${current_directory}/"GENERATED"
build_gradle=$( realpath build.gradle )
## touch because .gitignore ignores this folder:
touch $GENERATED
COPY_BUILD_FILE=$( cat <<COPY_BUILD_FILE_HEREDOC
cp
$build_gradle
$GENERATED/build.gradle
COPY_BUILD_FILE_HEREDOC
)
$COPY_BUILD_FILE
GRADLE_COMMAND=$( cat <<GRADLE_COMMAND_HEREDOC
gradle run
--build-file
$GENERATED/build.gradle
--gradle-user-home
$GENERATED
--no-daemon
GRADLE_COMMAND_HEREDOC
)
$GRADLE_COMMAND
The lone ")" are kind of ugly. But I have no clue how to fix that asthetic aspect.
To see all commands that are being executed by the script, add the -x flag to your shabang line, and execute the command normally:
#! /bin/bash -x
matchdir="/home/joao/robocup/runner_workdir/matches/testmatch/"
teamAComm="`pwd`/a.sh"
teamBComm="`pwd`/b.sh"
include="`pwd`/server_official.conf"
serverbin='/usr/local/bin/rcssserver'
cd $matchdir
$serverbin include="$include" server::team_l_start="${teamAComm}" server::team_r_start="${teamBComm}" CSVSaver::save='true' CSVSaver::filename='out.csv'
Then if you sometimes want to ignore the debug output, redirect stderr somewhere.
For me echo XYZ_20200824.zip | grep -Eo '[[:digit:]]{4}[[:digit:]]{2}[[:digit:]]{2}'
was working fine but unable to store output of command into variable.
I had same issue I tried eval but didn't got output.
Here is answer for my problem:
cmd=$(echo XYZ_20200824.zip | grep -Eo '[[:digit:]]{4}[[:digit:]]{2}[[:digit:]]{2}')
echo $cmd
My output is now 20200824