I am trying to gather number from file using bash. My file looks like this:
number = 123;
I use following command to get it:
grep number file.txt | tr -dc 0-9
And what I get is:
0131123
I tried different number and different methods sed 's/[^0-9]//g' but it always gives me result 0131+my_number.
Any ideas why?
UPDATE
Giving answers to your questions
user#domain:/tmp$ grep number file.txt
number = 123;
user#domain:/tmp$ grep number file.txt | tr -dc 0-9
0131123user#domain:/tmp$
user#domain:/tmp$ grep number file.txt | sed 's/[^0-9]//g'
0131123
user#domain:/tmp$ cat file.txt
number = 123;
As you can see file doesn't contain anything except one line.
UPDATE 2
Results of commands given by antak and Gordon Davisson
user#domain:/tmp$ grep number file.txt | cat -v
^[[01;31m^[[Knumber^[[m^[[K = 123;
user#domain:/tmp$ hexdump -C file.txt
00000000 6e 75 6d 62 65 72 20 3d 20 31 32 33 3b 0a |number = 123;.|
0000000e
After cat -v I can see magic 0131, but I have no idea where it comes from.
You can use sed:
sed -n 's/number = \(.*\);/\1/p' file.txt
or with grep and friends:
grep number file.txt | cut -d' ' -f3 | tr -d ';'
or (as you suggested) :
grep number file.txt | tr -dc 0-9
Update: I didn't tried your example before giving the original answer. Just wanted to say that your example worked for me.. (you see above)
It's from your grep --colour=always setting. When it's set, it will color the matching substring with red color by default (see detail here), which will save as ^[[01;31m^[[K....^[[m^[[K tag, where the ... is the matched substring. Turn off the --colour option or change to use --colour=auto, you will solve the problem.
btw, I think I deserve an up-vote for the answer :)
Here is a more general solution which works with numbers and other data as well:
#!/bin/bash
FILENAME='file.txt'
SEARCH_PATTERN='number'
foundLine=$(grep -m 1 "$SEARCH_PATTERN" "$FILENAME")
if [[ $foundLine =~ =.*\; ]]; then
value=$BASH_REMATCH
value=$(echo "${value:1:-1}" | xargs) # remove = and ; and then trim
echo "Your value is: _${value}_"
else
echo 'Line not found :('
fi
This is very useful because on a file with this content:
number = 'complex thing :3' ; # my comment with numbers 123 512
number = 'sadfhsah';
it must return
Your value is: _complex thing :3_
So it does not depend on numbers in comments and other garbage.
P.S. Please note that it will not work with variables that have newlines in them and with comments containing ';' characters
Related
I need to extract from the file the words that contain certain letters in a certain amount.
I apologize if this question has been resolved in the past, I just did not find anything that fits what I am looking for.
File:
wab 12aaabbb abababx ab ttttt baaabb zabcabc
baab baaabb cbaab ab ccabab zzz
For example
1. If I chose the letters a and the number is 1 the output should be:
wab
ab
ab
//only the words that contains a and the char appear in the word 1 time
2. If I chose the letters a,b and the number is 3, the output should be:
12aaabbb
abababx
baaabb
//only the word contains a,b, and both chars appear in the word 3 times
3. If I chose the letters a,b,c and the number 2, the output should be:
ccabab
zabcabc
//only the words that contains a,b,c and the chars appear in the word 3 times
Is it possible to find 2 letters in the same script?
I was able to find in a single letter but I get only the words where the letters appear in sequence and I do not want to find only these words, that's what I did:
egrep '([a])\1{N-1}' file
And another problem I can not get only the specific words, I get all file and the letter I am looking for "a" in red.
I tried using -w but it does not display anything.
::: EDIT :::
try to edit what you did to a for
i=$1
fileName=$2
letters=${#: 3}
tr -s '[:space:]' '\n' < $fileName* |
for letter in $letters; do
grep -E "^[^$letter]*($letter[^$letter]*){$i}$"
done | uniq
There are various ways to split input so that grep sees a single word per line. tr is most common. For example:
tr -s '[:space:]' '\n' file | ...
We can build a function to find a specific number of a particular letter:
NofL(){
num=$1
letter=$2
regex="^[^$letter]*($letter[^$letter]*){$num}$"
grep -E "$regex"
}
Then:
# letter=a number=1
tr -s '[:space:]' '\n' file | NofL 1 a
# letters=a,b number=3
tr -s '[:space:]' '\n' file | NofL 3 a | NofL 3 b
# letters=a,b,c number=2
tr -s '[:space:]' '\n' file | NofL 2 a | NofL 2 b | NofL 2 c
Regexes are not really suited for that job as there are more efficient ways, but it is possible using repeated matching. We first select all words, from those we select words with n as, and from those we select words with n bs and so on.
Example for n=3 and a, b:
grep -Eo '[[:alnum:]]+' |
grep -Ex '[^a]*a[^a]*a[^a]*a[^a]*' |
grep -Ex '[^b]*b[^b]*b[^b]*b[^b]*'
To auto-generate such a command from an input like 3 a b, you need to dynamically create a pipeline, which is possible, but also a hassle:
exactly_n_times_char() {
(( $# >= 2 )) || { cat; return; }
local n="$1" char="$2" regex
regex="[^$char]*($char[^$char]*){$n}"
shift 2
grep -Ex "$regex" | exactly_n_times_char "$n" "$#"
}
grep -Eo '[[:alnum:]]+' file.txt | exactly_n_times_char 3 a b
With PCREs (requires GNU grep or pcregrep) the check can be done in a single regex:
exactly_n_times_char() {
local n="$1" regex=""
shift
for char; do # could be done without a loop using sed on $*
regex+="(?=[^$char\\W]*($char[^$char\\W]*){$n})"
done
regex+='\w+'
grep -Pow "$regex"
}
exactly_n_times_char 3 a b < file.txt
If a matching word appears multiple times (like baaabb in your example) it is printed multiple times too. You can filter out duplicates by piping through sort -u but that will change the order.
A method using sed and bash would be:
#!/bin/bash
file=$1
n=$2
chars=$3
for ((i = 0; i < ${#chars}; ++i)); do
c=${chars:i:1}
args+=(-e)
args+=("/^\([^$c]*[$c]\)\{$n\}[^$c]*\$/!d")
done
sed "${args[#]}" <(tr -s '[:blank:]' '\n' < "$file")
Notice that filename, count, and characters are parameterized. Use it as
./script filename 2 abc
which should print out
zabcabc
ccabab
given the file content in the question.
An implementation in pure bash, without calling an external program, could be:
#!/bin/bash
readonly file=$1
readonly n=$2
readonly chars=$3
while read -ra words; do
for word in "${words[#]}"; do
for ((i = 0; i < ${#chars}; ++i)); do
c=${word//[^${chars:i:1}]}
(( ${#c} == n )) || continue 2
done
printf '%s\n' "$word"
done
done < "$file"
You can match a string containing exactly N occurrences of character X with the (POSIX-extended) regexp [^X]*(X[^X]*){N}. To do this for multiple characters you could chain them, and the traditional way to process one 'word' at a time, simplistically defined as a sequence of non-whitespace chars, is like this
<infile tr -s ' \t\n' ' ' | grep -Ex '[^a]*(a[^a]*){3}' | \grep -Ex '[^b]*(b[^b]*){3}'
# may need to add \r on Windows-ish systems or for Windows-derived data
If you get colorized output from egrep and grep and maybe some other utilities it's usually because in a GNU-ish environment you -- often via a profile that was automatically provided and you didn't look at or modify -- set aliases to turn them into e.g. egrep --color=auto or possibly/rarely =always; using \grep or command grep or the pathname such as /usr/bin/grep disables the alias, or you could just un-set it/them. Another possibility is you may have envvar(s) set in which case you need to remove or suppress it/them, or explicitly say --color=never, or (somewhat hackily) pipe the output through ... | cat which has the effect of making [e]grep's stdout a pipe not a tty and thus turning off =auto.
However, GNU awk (not necessarily others) can also do this more directly:
<infile awk -vRS='[ \t\n]+' -F '' '{delete f;for(i=1;i<=NF;i++)f[$i]++}
f["a"]==3&&f["b"]==3'
or to parameterize the criteria:
<infile awk -vRS='[ \t\n]+' -F '' 'BEGIN{split("ab",w,//);n=3}
{delete f;for(i=1;i<=NF;i++)f[$i]++;s=1;for(t in w)if(f[w[t]]!=occur)s=0} s'
perl can do pretty much everything awk can do, and so can some other general-purpose tools, but I leave those as exercises.
I strongly doubt about the grep best use in my code and would like to find a better and cleaner coding style for extracting the session ID and security level from my cookie file :
cat mycookie
# Netscape HTTP Cookie File
# https://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.
#HttpOnly_127.0.0.1 FALSE / FALSE 0 PHPSESSID 1hjs18icittvqvpa4tm2lv9b12
#HttpOnly_127.0.0.1 FALSE /mydir/ FALSE 0 security medium
The expected output is the SSID hash :
1hjs18icittvqvpa4tm2lv9b12
Piping grep with tr '\n' '\0' works like a charm in the command line, but generates warnings (warning: command substitution: ignored null byte in input”) at the bash code execution. Here is the related code (with warnings):
ssid=$(grep -Po 'PHPSESSID.*' path/sessionFile | grep -Po '[a-z]|[0-9]' | tr '\n' '\0')
I am using bash 4.4.12 (x86_64-pc-linux-gnu) and could read here this crystal clear explanation :
Bash variables are stored as C strings. C strings are NUL-terminated.
They thus cannot store NULs by definition.
I could see here and there in both cases a coding solution using read:
# read content from stdin into array variable and a scalar variable "suffix"
array=( )
while IFS= read -r -d '' line; do
array+=( "$line" )
done < <(process that generates NUL stream here)
suffix=$line # content after last NUL, if any
# emit recorded content
printf '%s\0' "${array[#]}"; printf '%s' "$suffix"
I don't want to use arrays nor a while loop for this specific case, or others. I found this workaround using sed:
ssid=$(grep -Po 'PHPSESSID.*' path/sessionFile | grep -Po '[a-z]|[0-9]' | tr '\n' '_' | sed -e 's/_//g')
My two questions are :
1) Would it be a better way to substitute tr '\n' '\0', without using read into a while loop ?
2) Would it be a better way to extract properly the SSID and security level ?
Thx
It looks like you're trying to get rid of the newlines in the output from grep, but turning them into nulls doesn't do this. Nulls aren't visible in your terminal, but are still there and (like many other nonprinting characters) will wreak havoc if they get treated as part of your actual data. If you want to get rid of the newlines, just tell tr to delete them for you with ... | tr -d '\n'. But if you're trying to get the PHPSESSID value from a Netscape-format cookie file, there's a much much better way:
ssid=$(awk '($6 == "PHPSESSID") {print $7}' path/sessionFile)
This looks for "PHPSESSID" only in the sixth field (not in e.g. the path or cookie values -- both places it could legally appear), and specifically prints the seventh field of matching lines (not just anything after "PHPSESSID" that happens to be a digit or lowercase letter).
You could also try this, if you don't want to use awk:
ssid=$(grep -P '\bPHPSESSID\b' you_cookies_file)
echo $ssid # for debug only
which outputs something like
#HttpOnly_127.0.0.1 FALSE / FALSE 0 PHPSESSID 1hjs18icittvqvpa4tm2lv9b12
Then with cut(1) extract the relevant field:
echo $ssid |cut -d" " -f7
which outputs
1hjs18icittvqvpa4tm2lv9b12
Of course you should capture the last echo.
UPDATE
If you don't want to use cut, it is possible to emulate it with:
echo $ssid | (read a1 b2 c3 d4 e5 f6 g7; echo $g7)
Demonstration to capture in a variable:
$ field=$(echo $ssid | (read a1 b2 c3 d4 e5 f6 g7; echo $g7))
$ echo $field
1hjs18icittvqvpa4tm2lv9b12
$
Another way is to use positional parameters passing the string to a function which then refers to $7. Perhaps cleaner. Otherwise, you can use an array:
array=($(echo $ssid))
echo ${array[6]} # outputs the 7th field
It should also be possible to use regular expressions and/or string manipulation is bash, but they seem a little more difficult to me.
I want to stick all the variables into one variable
A=('blah')
AA=('blah2')
AAA=('blah3')
AAB=('blah4')
AAC=('blah5')
#^^lets pretend theres 100 more of these ^^
#Variable composition
#after AAA, is AAB then AAC then AAD etc etc, does that 100 times
I want them all placed into this MASTER variable
#MASTER=${A}${AA}${AAA} (<-- insert AAB, AAC and 100 more variables here)
I obviously don't want to type 100 variables in this expression because there's probably an easier way to do this. Plus I'm gonna be doing more of these therefore I need it automated.
I'm relatively new to sed, awk, is there a way to append those 100 variables into the master variable?
For this specific purpose I DO NOT want an array.
You can use a simple one-liner, quite straightforward, though more expensive:
master=$(set | grep -E '^(A|AA|A[A-D][A-D])=' | sort | cut -f2- -d= | tr -d '\n')
set lists all the variables in var=name format
grep filters out the variables we need
sort puts them in the right order (probably optional since set gives a sorted output)
cut extracts the values, removing the variable names
tr removes the newlines
Let's test it.
A=1
AA=2
AAA=3
AAB=4
AAC=5
AAD=6
AAAA=99 # just to make sure we don't pick this one up
master=$(set | grep -E '^(A|AA|A[A-D][A-D])=' | sort | cut -f2- -d= | tr -d '\n')
echo "$master"
Output:
123456
With my best guess, how about:
#!/bin/bash
A=('blah')
AA=('blah2')
AAA=('blah3')
AAB=('blah4')
AAC=('blah5')
# to be continued ..
for varname in A AA A{A..D}{A..Z}; do
value=${!varname}
if [ -n "$value" ]; then
MASTER+=$value
fi
done
echo $MASTER
which yields:
blahblah2blah3blah4blah5...
Although I'm not sure whether this is what the OP wants.
echo {a..z}{a..z}{a..z} | tr ' ' '\n' | head -n 100 | tail -n 3
adt
adu
adv
tells us, that it would go from AAA to ADV to reach 100, or for ADY for 103.
echo A{A..D}{A..Z} | sed 's/ /}${/g'
AAA}${AAB}${AAC}${AAD}${AAE}${AAF}${AAG}${AAH}${AAI}${AAJ}${AAK}${AAL}${AAM}${AAN}${AAO}${AAP}${AAQ}${AAR}${AAS}${AAT}${AAU}${AAV}${AAW}${AAX}${AAY}${AAZ}${ABA}${ABB}${ABC}${ABD}${ABE}${ABF}${ABG}${ABH}${ABI}${ABJ}${ABK}${ABL}${ABM}${ABN}${ABO}${ABP}${ABQ}${ABR}${ABS}${ABT}${ABU}${ABV}${ABW}${ABX}${ABY}${ABZ}${ACA}${ACB}${ACC}${ACD}${ACE}${ACF}${ACG}${ACH}${ACI}${ACJ}${ACK}${ACL}${ACM}${ACN}${ACO}${ACP}${ACQ}${ACR}${ACS}${ACT}${ACU}${ACV}${ACW}${ACX}${ACY}${ACZ}${ADA}${ADB}${ADC}${ADD}${ADE}${ADF}${ADG}${ADH}${ADI}${ADJ}${ADK}${ADL}${ADM}${ADN}${ADO}${ADP}${ADQ}${ADR}${ADS}${ADT}${ADU}${ADV}${ADW}${ADX}${ADY}${ADZ
The final cosmetics is easily made by hand.
One-liner using a for loop:
for n in A AA A{A..D}{A..Z}; do str+="${!n}"; done; echo ${str}
Output:
blahblah2blah3blah4blah5
Say you have the input file inputfile.txt with arbitrary variable names and values:
name="Joe"
last="Doe"
A="blah"
AA="blah2
then do:
master=$(eval echo $(grep -o "^[^=]\+" inputfile.txt | sed 's/^/\$/;:a;N;$!ba;s/\n/$/g'))
This will concatenate the values of all variables in inputfile.txt into master variable. So you will have:
>echo $master
JoeDoeblahblah2
I need to convert a string in chinese to its appropriate HEX format. I can do it using sed in the following way
echo -n 欢迎 | xxd -p -u | sed 's/.\{2\}/&\\x/g' | sed 's/^\(.\{0\}\)/\1\\x/' | sed -r 's/(.*)\\x/\1 /'
which gives me output as:
\xE6\xAC\xA2\xE8\xBF\x8E
This is correct answer that I am looking for. Please suggest me making using of sed more efficiently in above command. The above command is being run on ubuntu 16.04 terminal
You can chain sed-commands with ";":
echo -n 欢迎 | xxd -p -u | sed 's/.\{2\}/&\\x/g;s/^\(.\{0\}\)/\1\\x/' | sed -r 's/(.*)\\x/\1 /'
\xE6\xAC\xA2\xE8\xBF\x8E
Since you use sed and sed -r interchangingly, you have to modify the second, remaining sed call, to combine the remaining ones:
echo -n 欢迎 | xxd -p -u | sed 's/.\{2\}/&\\x/g;s/^\(.\{0\}\)/\1\\x/;s/\(.*\)\\x/\1 /'
Having a second look at it, what the output of xxd is without sed, I observed, the solution is much more easy:
echo -n 欢迎 | xxd -p -u | sed -r 's/(..)/\\x\1/g'
Your initial approach appended \x to 2 characters, but you can preceed it your pairs. However chaining multiple sed commands might still be a useful thing to know.
From an efficiency standpoint, about the best option I could come up with would be to replace xdd, 3-pipes, and 3 calls to sed with od and 2 bash parameter expansions. (there may be more efficient ways, but this was what came to mind)
For example, you could assign the result of command substitution $(printf "欢迎" | od -A none -t x1) to a variable which would contain ' e6 ac a2 e8 bf 8e'. Then it is simply a matter of converting to upper-case and then using a substring replacement of 'space' to '\x' (both provided by bash parameter expansions, e.g.
a=$(printf "欢迎" | od -A none -t x1); \
a=${a^^}; \
a=${a// /\\x}; \
echo $a
\xE6\xAC\xA2\xE8\xBF\x8E
(shown with line-continuations above, you can just copy/paste into your terminal to test)
From Your Request in Comment for C
The code in C to output the upper-case hex bytes contained in your string is trivial, e.g.
#include <stdio.h>
int main (void) {
char *s = "欢迎";
while (*s) /* output each byte in upper-case hex */
printf ("\\x%hhX", ((unsigned char)*s++));
putchar ('\n');
return 0;
}
Example Use/Output
$ ./bin/str2hexbytes
\xE6\xAC\xA2\xE8\xBF\x8E
(note: you could use the exact-width types in stdint.h and the exact-width format specifiers provided in inttypes.h for a more formal solution, but it would accomplish the same thing. Similarly, you could use wide-character types, but virtually all modern compilers have no problem handling multibyte characters in an ordinary string or array of char)
I want to make a script that is looking for special numbers.
numbers like this 153 = 1^3+5^3+3^3
bash script 153 3
153
In my script I have this kinda thing
echo "$1" | tr -d " " | sed -e 's/\([[:digit:]]\)/\1+/g' | tr '+' '^"$2"+'
That last command doesn't work, it does change something, it changes 1+5+3+ to 1^+5^+3^+
So my question is: how can I use variables in tr?
tr replaces one character with another one. It can't replace one character with a longer string. That's sed's job:
set -- 153 3
echo "$1" | \
tr -d " " | \
sed -e 's/\([[:digit:]]\)/\1^'"$2"'+/g; s/\+$//'
The answer by choroba is correct. Here is a python based one-liner:
$ set -- 153 3
$ python -c "print '+'.join([x+'^$2' for x in list('$1')])"
1^3+5^3+3^3
Explanation:
list will convert the string "153" to ['1', '5', '3']
[ x+'^$2' for x in <list> ] is called list comprehension. Effectively it returns another list: ['1^3', '5^3', '3^3']
Then join them with '+'
NOTE: Only reason I added this answer was because, this does not require to adjust the completed string after processing by build-in functions.
Below are the other common approaches:
$ python -c "print '^$2+'.join(list('$1')) + '^$2'" # Add "^3" after join returns "1^3+5^3+3"
$ echo $1 | sed "s/./&^$2+/g; s/+$//" # Remove last '+' sign from "1^3+5^3+3^3+"