I'm trying to split a tab delimitted field in bash.
I am aware of this answer: how to split a string in shell and get the last field
But that does not answer for a tab character.
I want to do get the part of a string before the tab character, so I'm doing this:
x=`head -1 my-file.txt`
echo ${x%\t*}
But the \t is matching on the letter 't' and not on a tab. What is the best way to do this?
Thanks
If your file look something like this (with tab as separator):
1st-field 2nd-field
you can use cut to extract the first field (operates on tab by default):
$ cut -f1 input
1st-field
If you're using awk, there is no need to use tail to get the last line, changing the input to:
1:1st-field 2nd-field
2:1st-field 2nd-field
3:1st-field 2nd-field
4:1st-field 2nd-field
5:1st-field 2nd-field
6:1st-field 2nd-field
7:1st-field 2nd-field
8:1st-field 2nd-field
9:1st-field 2nd-field
10:1st-field 2nd-field
Solution using awk:
$ awk 'END {print $1}' input
10:1st-field
Pure bash-solution:
#!/bin/bash
while read a b;do last=$a; done < input
echo $last
outputs:
$ ./tab.sh
10:1st-field
Lastly, a solution using sed
$ sed '$s/\(^[^\t]*\).*$/\1/' input
10:1st-field
here, $ is the range operator; i.e. operate on the last line only.
For your original question, use a literal tab, i.e.
x="1st-field 2nd-field"
echo ${x% *}
outputs:
1st-field
Use $'ANSI-C' strings in the parameter expansion:
$ x=$'abc\tdef\tghi'
$ echo "$s"
abc def ghi
$ echo ">>${x%%$'\t'*}<<"
>>abc<<
read field1 field2 <<< ${tabDelimitedField}
or
read field1 field2 <<< $(command_producing_tab_delimited_output)
Use awk.
echo $yourfield | awk '{print $1}'
or, in your case, for the first field from the the last line of a file
tail yourfile | awk '{x=$1}END{print x}'
There is an easy way for a tab separated string : convert it to an array.
Create a string with tabs ($ added before for '\t' interpretation) :
AAA=$'ABC\tDEF\tGHI'
Split the string as an array using parenthesis :
BBB=($AAA)
Get access to any element :
echo ${BBB[0]}
ABC
echo ${BBB[1]}
DEF
echo ${BBB[2]}
GHI
x=first$'\t'second
echo "${x%$'\t'*}"
See QUOTING in man bash
The answer from https://stackoverflow.com/users/1815797/gniourf-gniourf hints at the use of built in field parsing in bash, but does not really complete the answer. The use of the IFS shell parameter to set the input field separate will complete the picture and give the ability to parse files which are tab-delimited, of a fixed number of fields, in pure bash.
echo -e "a\tb\tc\nd\te\tf" > myfile
while IFS='<literaltab>' read f1 f2 f3;do echo "$f1 = $f2 + $f3"; done < myfile
a = b + c
d = e + f
Where, of course, is replaced by a real tab, not \t. Often, Control-V Tab does this in a terminal.
Related
I have a file which looks like this (file.txt)
{"key":"AJGUIGIDH568","rule":squid:111-some_random_text_here
{"key":"TJHJHJHDH568","rule":squid:111-some_random_text_here
{"key":"YUUUIGIDH566","rule":squid:111-some_random_text_here
{"key":"HJHHIGIDH568","rule":squid:111-some_random_text_here
{"key":"ATYUGUIDH556","rule":squid:111-some_random_text_here
{"key":"QfgUIGIDH568","rule":squid:111-some_random_text_here
I want to loop trough this line by line an extract the key values.
so the result should be like ,
AJGUIGIDH568
AJGUIGIDH568
YUUUIGIDH566
HJHHIGIDH568
ATYUGUIDH556
QfgUIGIDH568
So I wrote a code like this to loop line by line and extract the value between {"key":" and ","rule": because key values is in between these 2 patterns.
while read p; do
echo $p | sed -n "/{"key":"/,/","rule":,/p"
done < file.txt
But this is not working. can someone help me to figure out me this. Thanks in advance.
Your sample input is almost valid json. You could tweak it to make it valid and then extract the values with jq with something like:
sed -e 's/squid/"squid/' -e 's/$/"}/' file.txt | jq -r .key
Or, if your actual input really is valid json, then just use jq:
jq -r .key file.txt
If the "random-txt" may include double quotes, making it difficult to massage the input to make it valid json, perhaps you want something like:
awk '{print $4}' FS='"' file.txt
or
sed -n '/{"key":"\([^"]*\).*/s//\1/p' file.txt
or
while IFS=\" read open_brace key colon val _; do echo "$val"; done < file.txt
For the shown data, you can try this awk:
awk -F '"[:,]"' '{print $2}' file
AJGUIGIDH568
TJHJHJHDH568
YUUUIGIDH566
HJHHIGIDH568
ATYUGUIDH556
QfgUIGIDH568
With the give example you can simple use
cut -d'"' -f4 file.txt
Assumptions:
there may be other lines in the file so we need to focus on just the lines with "key" and "rule"
the only text between "key" and "rule" is the desired string (eg, squid never shows up between the two patterns of interest)
Adding some additional lines:
$ cat file.txt
{"key":"AJGUIGIDH568","rule":squid:111-some_random_text_here
ignore this line}
{"key":"TJHJHJHDH568","rule":squid:111-some_random_text_here
ignore this line}
{"key":"YUUUIGIDH566","rule":squid:111-some_random_text_here
ignore this line}
{"key":"HJHHIGIDH568","rule":squid:111-some_random_text_here
ignore this line}
{"key":"ATYUGUIDH556","rule":squid:111-some_random_text_here
ignore this line}
{"key":"QfgUIGIDH568","rule":squid:111-some_random_text_here
ignore this line}
One sed idea:
$ sed -nE 's/^(.*"key":")([^"]*)(","rule".*)$/\2/p' file.txt
AJGUIGIDH568
TJHJHJHDH568
YUUUIGIDH566
HJHHIGIDH568
ATYUGUIDH556
QfgUIGIDH568
Where:
-E - enable extended regex support (and capture groups without need to escape sequences)
-n - suppress printing of pattern space
^(.*"key":") - [1st capture group] everything from start of line up to and including "key":"
([^"]*) - [2nd capture group] everything that is not a double quote (")
(","rule".*)$ - [3rd capture group] everything from ",rule" to end of line
\2/p - replace the line with the contents of the 2nd capture group and print
I have to format the width of a substring within a string using a bash script, but without using tokens or loops. A single character between two colons should be prepended by a 0 in order to match the standard width of 2 for each field.
For e.g
from:
6:0:36:35:30:30:72:6c:73:0:c:52:4c:30:31:30:31:30:30:30:31:36:39:0:1:3
to
06:00:36:35:30:30:72:6c:73:00:0c:52:4c:30:31:30:31:30:30:30:31:36:39:00:01:03
How can I do this?
sed -r 's/\<([0-9a-f])\>/0\1/g'
Search and replace with a regex. Use \< and \> to match word boundaries so [0-9a-f] only matches single digits.
$ sed -r 's/\<([0-9a-f])\>/0\1/g' <<< "6:0:36:35:30:30:72:6c:73:0:c:52:4c:30:31:30:31:30:30:30:31:36:39:0:1:3"
06:00:36:35:30:30:72:6c:73:00:0c:52:4c:30:31:30:31:30:30:30:31:36:39:00:01:03
awk -F: -v OFS=: '{for(i=1;i<=NF;i++) if(length($i)==1)gsub($i,"0&",$i)}1' file
Output:
06:00:36:35:30:30:72:6c:73:00:0c:52:4c:30:31:30:31:30:30:30:31:36:39:00:01:03
This will divide the whole line into fields separated by : , if the length of any of the field is == 1. then it will replace that field with 0field.
Bash solution:
IFS=:; for i in $string; do echo -n 0$i: | tail -c 3; done
With
str="06:00:36:35:30:30:72:6c:73:00:0c:52:4c:30:31:30:31:30:30:30:31:36:39:00:01:03"
you can add a '0' to all tokens and remove those that are unwanted:
sed -r 's/0([0-9a-f]{2})/\1/g' <<< "0${str//:/:0}"
That doesn't feel right, making errors and repairing them.
A better alternative is
echo $(IFS=:; printf "%2s:" ${str} | tr " " "0")
I have a string:
Log for: squid.log.2017.11.13
I need to cut out squid.log. so that I see:
Log for: 2017.11.13
I tried to cut
echo "Log for: squid.log.2017.11.13" | cut -d'.' -f3-5
But I ended up with:
2017.11.13
How can I get the result I want?
You can use sed to cut the unwanted part:
echo "Log for: squid.log.2017.11.13" | sed 's/squid\.log\.//'
Use sed to remove the part you don't want:
echo "Log for: squid.log.2017.11.13" | sed 's/squid\.log\.//'
awk to the rescue! a non-standard approach to break the monotony...
define the to be removed text as field separator and parse and print the input line.
$ echo Log for: squid.log.2017.11.13 | awk -F' squid\\.log\\.' '{$1=$1}1'
Log for: 2017.11.13
This solution is a bit more reusable than the previous ones offered:
awk '/^Log/{ split($3,x,"."); print $1" "$2" "x[length(x)-2]"."x[length(x)-1]"."x[length(x)] };'
This looks for all lines starting with Log, then grabs the 3rd column which contains squid.log.2017.11.13 and utilizes the the split built-in to break up the string in to array x using the . as the delimiter. Once we have our array x, we know that the last 3 values will always be the date, and this will work regardless of the rest of the string, (even if squid.log was something different) - we can use the length built-in to make sure we only get the last three elements.
Then we just print our reformatted string print $1" "$2" "x[length(x)-2]"."x[length(x)-1]"."x[length(x)] - reinserting the .'s in the appropriate places since they were stripped by using them as the split delimiter.
Output:
Log for: 2017.11.13
I want to write a small script to generate the location of a file in an NGINX cache directory.
The format of the path is:
/path/to/nginx/cache/d8/40/32/13febd65d65112badd0aa90a15d84032
Note the last 6 characters: d8 40 32, are represented in the path.
As an input I give the md5 hash (13febd65d65112badd0aa90a15d84032) and I want to generate the output: d8/40/32/13febd65d65112badd0aa90a15d84032
I'm sure sed or awk will be handy, but I don't know yet how...
This awk can make it:
awk 'BEGIN{FS=""; OFS="/"}{print $(NF-5)$(NF-4), $(NF-3)$(NF-2), $(NF-1)$NF, $0}'
Explanation
BEGIN{FS=""; OFS="/"}. FS="" sets the input field separator to be "", so that every char will be a different field. OFS="/" sets the output field separator as /, for print matters.
print ... $(NF-1)$NF, $0 prints the penultimate field and the last one all together; then, the whole string. The comma is "filled" with the OFS, which is /.
Test
$ awk 'BEGIN{FS=""; OFS="/"}{print $(NF-5)$(NF-4), $(NF-3)$(NF-2), $(NF-1)$NF, $0}' <<< "13febd65d65112badd0aa90a15d84032"
d8/40/32/13febd65d65112badd0aa90a15d84032
Or with a file:
$ cat a
13febd65d65112badd0aa90a15d84032
13febd65d65112badd0aa90a15f1f2f3
$ awk 'BEGIN{FS=""; OFS="/"}{print $(NF-5)$(NF-4), $(NF-3)$(NF-2), $(NF-1)$NF, $0}' a
d8/40/32/13febd65d65112badd0aa90a15d84032
f1/f2/f3/13febd65d65112badd0aa90a15f1f2f3
With sed:
echo '13febd65d65112badd0aa90a15d84032' | \
sed -n 's/\(.*\([0-9a-f]\{2\}\)\([0-9a-f]\{2\}\)\([0-9a-f]\{2\}\)\)$/\2\/\3\/\4\/\1/p;'
Having GNU sed you can even simplify the pattern using the -r option. Now you won't need to escape {} and () any more. Using ~ as the regex delimiter allows to use the path separator / without need to escape it:
sed -nr 's~(.*([0-9a-f]{2})([0-9a-f]{2})([0-9a-f]{2}))$~\2/\3/\4/\1~p;'
Output:
d8/40/32/13febd65d65112badd0aa90a15d84032
Explained simple the pattern does the following: It matches:
(all (n-5 - n-4) (n-3 - n-2) (n-1 - n-0))
and replaces it by
/$1/$2/$3/$0
You can use a regular expression to separate each of the last 3 bytes from the rest of the hash.
hash=13febd65d65112badd0aa90a15d84032
[[ $hash =~ (..)(..)(..)$ ]]
new_path="/path/to/nginx/cache/${BASH_REMATCH[1]}/${BASH_REMATCH[2]}/${BASH_REMATCH[3]}/$hash"
Base="/path/to/nginx/cache/"
echo '13febd65d65112badd0aa90a15d84032' | \
sed "s|\(.*\(..\)\(..\)\(..\)\)|${Base}\2/\3/\4/\1|"
# or
# sed sed 's|.*\(..\)\(..\)\(..\)$|${Base}\1/\2/\3/&|'
Assuming info is a correct MD5 (and only) string
First of all - thanks to all of the responders - this was extremely quick!
I also did my own scripting meantime, and came up with this solution:
Run this script with a parameter of the URL you're looking for (www.example.com/article/76232?q=hello for example)
#!/bin/bash
path=$1
md5=$(echo -n "$path" | md5sum | cut -f1 -d' ')
p3=$(echo "${md5:0-2:2}")
p2=$(echo "${md5:0-4:2}")
p1=$(echo "${md5:0-6:2}")
echo "/path/to/nginx/cache/$p1/$p2/$p3/$md5"
This assumes the NGINX cache has a key structure of 2:2:2.
I run a script with the param -A AA/BB . To get an array with AA and BB, i can do this.
INPUT_PARAM=(${AIRLINE_OPTION//-A / }) #get rid of the '-A ' in the begining
LIST=(${AIRLINES_PARAM//\// }) # split by '/'
Can we achieve this in a single line?
Thanks in advance.
One way
IFS=/ read -r -a LIST <<< "${AIRLINE_OPTION//-A /}"
This places the output from the parameter substitution ${AIRLINE_OPTION//-A /} into a "here-string" and uses the bash read built-in to parse this into an array. Splitting by / is achieved by setting the value of IFS to / for the read command.
LIST=( $(IFS=/; for x in ${AIRLINE_OPTION#-A }; do printf "$x "; done) )
This is a portable solution, but if your read supports -a and you don't mind portability then you should go for #1_CR's solution.
With awk, for example, you can create an array and store it in LIST variable:
$ LIST=($(awk -F"[\/ ]" '{print $2,$3}' <<< "-A AA/BB"))
Result:
$ echo ${LIST[0]}
AA
$ echo ${LIST[1]}
BB
Explanation
-F"[\/ ]" defines two possible field separators: a space or a slash /.
'{print $2$3}' prints the 2nd and 3rd fields based on those separators.