I am trying to left align with printf in bash with unknown width.
How do I do so?
So sort of like this, but this does not work
max=<enter random number>
printf "%-${max}s|", "Instance"
The point of this is the instance below will be an unknown length that can is dynamic in length.
Example Input
max=10
Example Output
Instance |
Example Input2
max=12
Example Output2
Instance |
You can use an * for the length:
for ((max=8;max<15;max++)); do
printf "%-*s|\n" ${max} "Instance"
done
Result:
Instance|
Instance |
Instance |
Instance |
Instance |
Instance |
Instance |
I suppose you're trying to print a table. I think you won't be able to do this with printf alone. This would basically require your line outputting command to predict the future output.
If you can tolerate post-processing though, you can simply do it using the column command. Just pick a character you'd like to replace with padding, and do as in the following example (I've picked the backslash \):
printf "%s\\|\n" "Instance" "Linstance" "Mintinginstance" | column -ts'\'
Output:
Instance |
Linstance |
Mintinginstance |
Related
I strongly doubt about the grep best use in my code and would like to find a better and cleaner coding style for extracting the session ID and security level from my cookie file :
cat mycookie
# Netscape HTTP Cookie File
# https://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.
#HttpOnly_127.0.0.1 FALSE / FALSE 0 PHPSESSID 1hjs18icittvqvpa4tm2lv9b12
#HttpOnly_127.0.0.1 FALSE /mydir/ FALSE 0 security medium
The expected output is the SSID hash :
1hjs18icittvqvpa4tm2lv9b12
Piping grep with tr '\n' '\0' works like a charm in the command line, but generates warnings (warning: command substitution: ignored null byte in input”) at the bash code execution. Here is the related code (with warnings):
ssid=$(grep -Po 'PHPSESSID.*' path/sessionFile | grep -Po '[a-z]|[0-9]' | tr '\n' '\0')
I am using bash 4.4.12 (x86_64-pc-linux-gnu) and could read here this crystal clear explanation :
Bash variables are stored as C strings. C strings are NUL-terminated.
They thus cannot store NULs by definition.
I could see here and there in both cases a coding solution using read:
# read content from stdin into array variable and a scalar variable "suffix"
array=( )
while IFS= read -r -d '' line; do
array+=( "$line" )
done < <(process that generates NUL stream here)
suffix=$line # content after last NUL, if any
# emit recorded content
printf '%s\0' "${array[#]}"; printf '%s' "$suffix"
I don't want to use arrays nor a while loop for this specific case, or others. I found this workaround using sed:
ssid=$(grep -Po 'PHPSESSID.*' path/sessionFile | grep -Po '[a-z]|[0-9]' | tr '\n' '_' | sed -e 's/_//g')
My two questions are :
1) Would it be a better way to substitute tr '\n' '\0', without using read into a while loop ?
2) Would it be a better way to extract properly the SSID and security level ?
Thx
It looks like you're trying to get rid of the newlines in the output from grep, but turning them into nulls doesn't do this. Nulls aren't visible in your terminal, but are still there and (like many other nonprinting characters) will wreak havoc if they get treated as part of your actual data. If you want to get rid of the newlines, just tell tr to delete them for you with ... | tr -d '\n'. But if you're trying to get the PHPSESSID value from a Netscape-format cookie file, there's a much much better way:
ssid=$(awk '($6 == "PHPSESSID") {print $7}' path/sessionFile)
This looks for "PHPSESSID" only in the sixth field (not in e.g. the path or cookie values -- both places it could legally appear), and specifically prints the seventh field of matching lines (not just anything after "PHPSESSID" that happens to be a digit or lowercase letter).
You could also try this, if you don't want to use awk:
ssid=$(grep -P '\bPHPSESSID\b' you_cookies_file)
echo $ssid # for debug only
which outputs something like
#HttpOnly_127.0.0.1 FALSE / FALSE 0 PHPSESSID 1hjs18icittvqvpa4tm2lv9b12
Then with cut(1) extract the relevant field:
echo $ssid |cut -d" " -f7
which outputs
1hjs18icittvqvpa4tm2lv9b12
Of course you should capture the last echo.
UPDATE
If you don't want to use cut, it is possible to emulate it with:
echo $ssid | (read a1 b2 c3 d4 e5 f6 g7; echo $g7)
Demonstration to capture in a variable:
$ field=$(echo $ssid | (read a1 b2 c3 d4 e5 f6 g7; echo $g7))
$ echo $field
1hjs18icittvqvpa4tm2lv9b12
$
Another way is to use positional parameters passing the string to a function which then refers to $7. Perhaps cleaner. Otherwise, you can use an array:
array=($(echo $ssid))
echo ${array[6]} # outputs the 7th field
It should also be possible to use regular expressions and/or string manipulation is bash, but they seem a little more difficult to me.
hokay, I am trying to write a script that takes information from the yum - repolist all and puts it into pretty JSON for me to use in some data collecting.. Right now I have my output from the yum command looking like this.
All I have for code right now is just the yum repolist command.
#!/bin/bash -x
yum -v repolist all | grep -B2 -A6 "enabled" | sed 's/[[:space:]]//g' , 's/--//g' , 's/name=name=/name=/g'
the output from that command looks like:
Repo-id: wazuh_repo
Repo-name: Wazuhrepository
Repo-status: enabled
Repo-revision: 1536348945
Repo-updated: FriSep712:35:512018
Repo-pkgs: 73
Repo-size: 920M
Repo-baseurl: https://packages.wazuh.com/3.x/yum/
Repo-expire: 21,600second(s)(last:WedOct3108:59:002018)
There are about 8 entries and the titles are always the same... Can someone explain like I am five how to convert this into json, I've read the jq man page, I've read about hash's. nothing seems to make sense. I know I need to have a "key"/"value" how to I designate these?
I just want to take the output and make it look like pretty JSON, this is part of a larger script I am writing to help keep ontop of the repos we use at work. I am just totally not getting JSON though.
edit: I would prefer not to use a wrapper function and do/learn the proper way
So, first, so people who don't have yum can test this, let's make a wrapper function:
write_output() { cat <<EOF
Repo-id: wazuh_repo
Repo-name: Wazuhrepository
Repo-status: enabled
Repo-revision: 1536348945
Repo-updated: FriSep712:35:512018
Repo-pkgs: 73
Repo-size: 920M
Repo-baseurl: https://packages.wazuh.com/3.x/yum/
Repo-expire: 21,600second(s)(last:WedOct3108:59:002018)
EOF
}
Notably, all your keys come before the string :, and the values come after them -- so we want to read line-by-line, split based on colon-space sequences, treat what was in front as a key, and treat what's in back as a value.
Given that:
jq -Rn '[inputs | split(": ")] | reduce .[] as $kv ({}; .[$kv[0]] = $kv[1])' < <(write_output)
...properly emits:
{
"Repo-id": "wazuh_repo",
"Repo-name": "Wazuhrepository",
"Repo-status": "enabled",
"Repo-revision": "1536348945",
"Repo-updated": "FriSep712:35:512018",
"Repo-pkgs": "73",
"Repo-size": "920M",
"Repo-baseurl": "https://packages.wazuh.com/3.x/yum/",
"Repo-expire": "21,600second(s)(last:WedOct3108:59:002018)"
}
...so, how does that work?
jq -R turns on raw input mode; input is parsed as a sequence of raw strings, not as a sequence of JSON documents.
jq -n treats null as the only direct input, so one can then use input and inputs primitives inside the script where needed.
[ inputs ] reads all your lines of input, and puts them into a single array.
[ inputs | split(": ")] changes that from an array of strings to an array of lists -- with content both before and after the ": " sequence.
reduce .[] as $kv ( {}; ... ) starts a reducer, with an initial value of {}, and then feeds each value that .[] evaluates to (which is to say, each item in your list) into that reducer (the ... code) as the $kv variable, replacing the . value each time.
To run this with your yum command as the real input, change < <(write_output) to < <(yum -v repolist all | grep -B2 -A6 "enabled" | sed 's/[[:space:]]//g' , 's/--//g' , 's/name=name=/name=/g').
Here is a slightly more robust variation of #CharlesDuffy's answer. Since the latter provides excellent explanatory notes, further explanations are not given here.
jq -nR '
[inputs | index(": ") as $ix | {(.[:$ix]): .[$ix+2:]}]
| add'
This avoids using split in case the "value" contains ": ". It might, however, be still better not to assume that a space follows the first relevant ":".
Notice also that add is used here instead of reduce, solely for compactness and simplicity.
For these sorts of problems, I would prefer to use a regular expression to match keys and values. Otherwise, I would take an approach similar to Charles's.
$ ... | jq -Rn 'reduce (inputs | capture("(?<k>[^:]+):\\s*(?<v>.+)")) as {$k, $v} ({}; .[$k] = $v)'
Playing with sed - the commands below do what's required but one liners would be better. I have tried combining the first two commands (with a separating ';') to remove the trailing ':' without success. Otherwise, resorted to removing the last ':' and writing to a new file to perform the next operation.
File 'sys' with a single line containing a variable number characters and ':' separator. For example;
Input - 'sys' first line 3.000000:50: desired output, two variables thd=3 mem=50
thd=$(echo | sed 's/.......:.*//' < sys)
sed 's/:$//' < sys > sys1
mem=$(echo | sed 's|........:||' < sys1)
Is there a way to combine the first two sed commands to avoid writing a second file? I have tried this various ways
Something like this - EDIT: this is the wrong order to remove the trailing ':'
thd=$(echo | sed 's/:$//;s/.......:.*//' < sys)
mem=$(echo | sed 's|........:||' < sys1)
Output 3 50: with the separator attached.
EDIT: This is the correct order and produces the desired output. Bash does not save the result of the first operation in the file sys. Which I should have picked up in the 3 liner.
thd=$(echo | sed 's/.......:.*//' < sys)
mem=$(echo | sed 's|........:||;s/:$//' < sys)
If you need two variables to be assigned values independently, the first containing the number before the point and the second the number between the colons, you can use an approach like
thd=$(cut -f1 -d. < sys)
mem=$(cut -f2 -d: < sys)
Assigning both at the same time is also possible:
read thd mem < <(tr "." ":" < sys | cut -f1,3 -d: --output-delimiter=" ")
Try this:
$ echo '3.000000:50:' | { IFS='.:' read thd x mem; echo "'$thd' '$mem'"; }
'3' '50'
Or this:
$ sys='3.000000:50:'; IFS='.:' read thd x mem <<< "$sys"; echo "'$thd' '$mem'"
'3' '50'
The above sets the "dont care" variable x. If you do not like that, you can assign mem twice.
$ sys='3.000000:50:'; IFS='.:' read thd mem mem <<< "$sys"; echo "'$thd' '$mem'"
'3' '50'
I want to stick all the variables into one variable
A=('blah')
AA=('blah2')
AAA=('blah3')
AAB=('blah4')
AAC=('blah5')
#^^lets pretend theres 100 more of these ^^
#Variable composition
#after AAA, is AAB then AAC then AAD etc etc, does that 100 times
I want them all placed into this MASTER variable
#MASTER=${A}${AA}${AAA} (<-- insert AAB, AAC and 100 more variables here)
I obviously don't want to type 100 variables in this expression because there's probably an easier way to do this. Plus I'm gonna be doing more of these therefore I need it automated.
I'm relatively new to sed, awk, is there a way to append those 100 variables into the master variable?
For this specific purpose I DO NOT want an array.
You can use a simple one-liner, quite straightforward, though more expensive:
master=$(set | grep -E '^(A|AA|A[A-D][A-D])=' | sort | cut -f2- -d= | tr -d '\n')
set lists all the variables in var=name format
grep filters out the variables we need
sort puts them in the right order (probably optional since set gives a sorted output)
cut extracts the values, removing the variable names
tr removes the newlines
Let's test it.
A=1
AA=2
AAA=3
AAB=4
AAC=5
AAD=6
AAAA=99 # just to make sure we don't pick this one up
master=$(set | grep -E '^(A|AA|A[A-D][A-D])=' | sort | cut -f2- -d= | tr -d '\n')
echo "$master"
Output:
123456
With my best guess, how about:
#!/bin/bash
A=('blah')
AA=('blah2')
AAA=('blah3')
AAB=('blah4')
AAC=('blah5')
# to be continued ..
for varname in A AA A{A..D}{A..Z}; do
value=${!varname}
if [ -n "$value" ]; then
MASTER+=$value
fi
done
echo $MASTER
which yields:
blahblah2blah3blah4blah5...
Although I'm not sure whether this is what the OP wants.
echo {a..z}{a..z}{a..z} | tr ' ' '\n' | head -n 100 | tail -n 3
adt
adu
adv
tells us, that it would go from AAA to ADV to reach 100, or for ADY for 103.
echo A{A..D}{A..Z} | sed 's/ /}${/g'
AAA}${AAB}${AAC}${AAD}${AAE}${AAF}${AAG}${AAH}${AAI}${AAJ}${AAK}${AAL}${AAM}${AAN}${AAO}${AAP}${AAQ}${AAR}${AAS}${AAT}${AAU}${AAV}${AAW}${AAX}${AAY}${AAZ}${ABA}${ABB}${ABC}${ABD}${ABE}${ABF}${ABG}${ABH}${ABI}${ABJ}${ABK}${ABL}${ABM}${ABN}${ABO}${ABP}${ABQ}${ABR}${ABS}${ABT}${ABU}${ABV}${ABW}${ABX}${ABY}${ABZ}${ACA}${ACB}${ACC}${ACD}${ACE}${ACF}${ACG}${ACH}${ACI}${ACJ}${ACK}${ACL}${ACM}${ACN}${ACO}${ACP}${ACQ}${ACR}${ACS}${ACT}${ACU}${ACV}${ACW}${ACX}${ACY}${ACZ}${ADA}${ADB}${ADC}${ADD}${ADE}${ADF}${ADG}${ADH}${ADI}${ADJ}${ADK}${ADL}${ADM}${ADN}${ADO}${ADP}${ADQ}${ADR}${ADS}${ADT}${ADU}${ADV}${ADW}${ADX}${ADY}${ADZ
The final cosmetics is easily made by hand.
One-liner using a for loop:
for n in A AA A{A..D}{A..Z}; do str+="${!n}"; done; echo ${str}
Output:
blahblah2blah3blah4blah5
Say you have the input file inputfile.txt with arbitrary variable names and values:
name="Joe"
last="Doe"
A="blah"
AA="blah2
then do:
master=$(eval echo $(grep -o "^[^=]\+" inputfile.txt | sed 's/^/\$/;:a;N;$!ba;s/\n/$/g'))
This will concatenate the values of all variables in inputfile.txt into master variable. So you will have:
>echo $master
JoeDoeblahblah2
I have a large log file, from which I need to extract some specific data, to be more precise, the values of distinct fields that appear repeatedly, i.e. I need to get some information from many CDRs, such as call type, origination number, etc.
The original text formatting is as per below:
Reason Code:"XXX", Result Code:XXX, Desc: "XXX"
..
A_NUMBER.ADDRESS = XXX
..
Using egrep I have managed to get the required lines, which appear to be like:
Reason Code:"XXX", Result Code:XXX, Desc: "XXX"
RECORD_IDENTIFICATION.FILE_ID: XXX
A_NUMBER.ADDRESS = XXX
Call is from XXXX, VDATE=XXXX.
but I am not being able to format them in a tabular style, grouped by Reason, File_ID, A_Num and Call Date, acting as column heads,
like
Reason Code | File_ID | A_Number | Date
xxxx | xxxx | xxxx | xxxx |
I am not really interested in the appearance, I just want the elements to be consecutive, in order to belong to the same call.
I have messed with different variants of awk, sed and printf, but nothing seems to work.
I have tried to put the total characters value as a parameter in printf
printf "%-205s\n" $(grep -E 'Reason Code|RECORD_IDENTIFICATION.FILE_ID|A_NUMBER.ADDRESS|Call is from' file.err)
or
printf "%-65s | %-65s | %-65s | %-65s" $(grep -E 'Reason Code|RECORD_IDENTIFICATION.FILE_ID|A_NUMBER.ADDRESS' file.err | awk 'FS = "\n" {print $1}')
but the values in output are scrambled and unusable.
In my opinion the solution may lay in some sort of loop, which awk seems to support, but I am not being able to sort it out.
Any help would be very appreciated.
Thank You
You can transform the output of your grep command with sed :
sed 'N;N;N;s/Reason Code:"\([^"]*\).*FILE_ID: \([^\n]*\).*A_NUMBER.ADDRESS = \([^\n]*\).*VDATE=\([^.]*\).*/\1 \2 \3 \4/'
$ echo ''' Reason Code:"XXX", Result Code:XXX, Desc: "XXX"
RECORD_IDENTIFICATION.FILE_ID: XXX
A_NUMBER.ADDRESS = XXX
Call is from XXXX, VDATE=XXXX.''' | sed 'N;N;N;s/Reason Code:"\([^"]*\).*FILE_ID: \([^\n]*\).*A_NUMBER.ADDRESS = \([^\n]*\).*VDATE=\([^.]*\).*/\1 \2 \3 \4/'
XXX XXX XXX XXXX
However, it would be best to avoid using grep and let sed also do the filtering. I can't propose such a solution since you didn't post the format of your unfiltered data.