For some reason, I can't, for the life of me, get zsh to produce an array containing one line from the entire shell history per element. (i.e. hist_arr[1] == $(history 1 1 | tr -s " " | cut -d ' ' -f 3-), hist_arr[2] == $(history 2 2 | tr -s " " | cut -d ' ' -f 3-), ... <for ten thousand lines>). I'd like to compute the whole array in a single step, so it's more efficient.
hist_arr[1]=$(history 1 1) works fine, but contains redundant history number.
If that is your problem then simple remove it, eg. this way:
hist_arr[1]=$(history 1 1 | tr -s " " | cut -d ' ' -f 3-)
Edit:
If you want to assign to table all element from history file then
IFS=$'
'
hist_arr=( $(awk 'BEGIN{FS=OFS=";"} {$1=""; sub(/\;/, "")}'1 .zsh_history) )
should work.
Related
I am a novice learner of Unix and shell scripting.
Can anyone explain the meaning of this line and how it works:
Record_count=$(wc -l ${table_dir} "/" $table_file_name | cut -d' ' f1)
I am not sure of what "/" does here.
Let's go step by step.
First step. This wc -l ${table_dir} "/" $table_file_name doesn't work as it's written but I understand it means return the number of lines (wc -l) of the file ${table_dir}/${table_file_name}. It returns something that looks like this (imagining that your_table_file_name.txt has 5 lines):
$ wc -l wc -l "${table_dir}/${table_file_name}"
5 your_table_dir/your_table_file_name.txt
Second step. I think this cut -d' ' f1 has a typo and is actually cut -d ' ' -f1. What this does is splitting a line by the space character (cut -d ' ') and only returns the first item of the sequence (-f1).
So, when you apply it to your line 5 your_table_dir/your_table_file_name.txt, it returns 5.
Third step. So what wc -l "${table_dir}/${table_file_name}" | cut -d ' ' -f1 does is returning the number of lines that ${table_dir}/${table_file_name} has.
Final step. In shell script, foo=$(some_command) means: assign to the variable called foo, the result of the command some_command.
So, what your whole line Record_count=$(wc -l "${table_dir}/${table_file_name}" | cut -d ' ' -f1) does is assigning to the variable Record_count, the count of the lines of the file ${table_dir}/${table_file_name}.
I have a big txt file which I want to edit in pipeline. But on same place in pipeline I want to set number of lines in variable $nol. I just want to see sintax how could I set variable in pipeline like:
cat ${!#} | tr ' ' '\n'| grep . ; $nol=wc -l | sort | uniq -c ...
That after second pipe is very wrong, but how can I do it in bash?
One of solutions is:
nol=$(cat ${!#} | tr ' ' '\n'| grep . | wc -l)
pipeline all from the start again
but I don't want to do script the same thing twice, bec I have more pipes then here.
I musn't use awk or sed...
You can use a tee and then write it to a file which you use later:
tempfile="xyz"
tr ' ' '\n' < "${!#}" | grep '.' | tee > "$tempfile" | sort | uniq -c ...
nol=$(wc -l "$tempfile")
Or you can use it the other way around:
nol=$(tr ' ' '\n' < "${!#}" | grep '.' \
| tee >(sort | uniq -c ... > /dev/tty) | wc -l
You can set a variable in a particular link of a pipeline, but that's not very useful since only that particular link will be affected by it.
I recommend simply using a temporary file.
set -e
trap 'rm -f "$tmpf"' EXIT
tmpf=`mktemp`
cat ${!#} | tr ' ' '\n'| grep . | sort > "$tmpf"
nol="$(wc "$tmpf")"
< "$tmpf" uniq -c ...
You can avoid the temporary file with tee and a named pipe, but it probably won't perform much better (it may even perform worse).
UPDATE:
Took a minute but I got it...
cat ${!#} | tr ' ' '\n'| tee >(nol=$(wc -l)) | sort | uniq -c ...
PREVIOUS:
The only way I can think to do this is storing in variables and calling back. You would not execute the command more than one time. You would just store the output in variables along the way.
aCommand=($(cat ${!#} | tr ' ' '\n'));sLineCount=$(echo ${#aCommand[#]});echo ${aCommand[#]} | sort | uniq -c ...
aCommand will store the results of the first set of commands in an array
sLineCount will count the elements (lines) in the array
;... echo the array elements and continue the commands from there.
Looks to me like you're asking how to avoid stepping through your file twice, just to get both word and line count.
Bash lets you read variables, and wc can produce all the numbers you need at once.
NAME
wc -- word, line, character, and byte count
So to start...
read words line chars < <( wc < ${!#} )
This populates the three variables based on input generated from process substitution.
But your question includes another partial command line which I think you intend as:
nol=$( sort -u ${!#} | wc -l )
This is markedly different from the word count of your first command line, so you can't use a single wc instance to generate both. Instead, one option might be to put your functionality into a script that does both functions at once:
read words uniques < <(
awk '
{
words += NF
for (i=1; i<=NF; i++) { unique[$i] }
}
END {
print words,length(unique)
}
' ${!#}
)
I have a script that loops over a curl command, which pulls in data from an API.
LIST_OF_ID=$(curl -s -X POST -d "username=$USER&password=$PASS&action=action" http://link.to/api.php)
for PHONE_NUMBER in $(echo $LIST_OF_ID | tr '_' ' ' | awk '{print $2}');
do
$VOIP_ID = $(echo $LIST_OF_ID | tr '_' ' ' | awk '{print $1}')
done
I also have a variable of 16 numbers in the range of "447856321455"
NUMBERS=$(cat << EOF
441111111111
441111111112
441111111113
... etc
)
The output on the API call is:
652364_441111111112
As you may notice I have taken the output and cut it into 2 parts and put it in a variable.
What I need is to match the 6 digit code from the output where the number in the output, matches with the number in the variable.
I've attempted it using if statements but I can't work my head around the correct way of doing it.
Any help would be appreciated.
Thank you.
I would do it using join rather than a loop in bash. Like this:
curl -s -X POST -d "$PARAMS" "$URL" | sort \
| join -t _ -2 2 -o 2.1 <(sort numbers.txt) -
What this does is take the sorted output from curl and join it with the sorted contents of numbers.txt (you could use $NUMBERS too), using _ as the separator, using column 2 of file 2 which is - meaning stdin (from curl). Then output field 2.1 which is the six-digit ID.
Read why-is-using-a-shell-loop-to-process-text-considered-bad-practice and then do something like this:
curl ... |
awk -v numbers="$NUMBERS" -F'_' '
BEGIN { split(numbers,tmp,/[[:space:]]+/); for (i in tmp) nums[tmp[i]] }
$2 in nums
'
but to be honest I cant really tell what it is you are trying to do as the numbers in your sample input don't seem to match each other (what does in the range of "447856321455" mean and how does it relate to $NUMBERS containing 441111111111 through 441111111113 and how does any of that relate to match the 6 digit code) and the expected output is missing.
I have this command :
id=$(xl list|egrep $Name| tr -s ' ' | cut -d ' ' -f 2)
which xl list output something like this:
Name ID Mem VCPUs State Time(s)
Domain-0 0 5923 8 r----- 4266.0
new_redhat9-clone 3 1027 1 r----- 1019.6
new_redhat9 4 1027 1 -b---- 40.1
Actually I want to get the ID of a given Name. This works when Name=new_redhat9-clone (it returns 3) but doesnt work when Name=new_redhat9 (it returns: 3 4!!!!).
what is wrong?!!!
grep searches the string pattern match. egrep new_redhat9 match with "new_redhat9" and "new_redhat9-clone". Try add whiteespace (or \t) after pattern, rewrite like this
id=$(xl list|egrep 'new_redhat9 '| tr -s ' ' | cut -d ' ' -f 2)
You could use awk instead of egrep,tr and cut commands,
id=$(xl list | awk '$1=="new_redhat9" {print $2}')
Awk command searches for the exact string new_redhat9 in the first column of xl list output . If it finds any then then value of column2 on the corresponding record is stored to the variable id.
You could check the output through echo $id command.
If the name is stored in a variable, then give a try to the below command
id=$(xl list | awk -v var=$Name '$1==var {print $2}')
Starting from Awk replace a column with its hash value, I tried to hash(md5) a list of numbers:
$ cat -n file
1 40755462755
2 40751685373
3 40730094339
4 40722740446
5 40722740446
6 40743802204
7 40730094339
8 40745188886
9 40740593352
10 40745561530
If I run:
cat file | awk '{cmd="echo -n " $1 " | md5sum|cut -d\" \" -f1"; cmd|getline md5; $1=md5;print;}' | cat -n
1 29ece26ce4633b6e9480255db194cc40
2 120148eca0891d0fc645413d0f26b66b
3 cafc48d392a004f75b669f9d1d7bf894
4 7b4367e8f58835c0827dd6a2f61b7258
5 7b4367e8f58835c0827dd6a2f61b7258
6 49b12d1f3305ab93b33b330e8b1d3165
7 49b12d1f3305ab93b33b330e8b1d3165
8 bee44c89ac9d4e8e4e1f1c5c63088c71
9 f07262ac8f53755232c5abbf062364d0
10 2ac7c22170c00a3527eb99a2bfde2c2c
I don't know why the line 7 get the same md5 as line 6 because if I run them separately they are different:
$ echo -n 40743802204 | md5sum|cut -d" " -f1
49b12d1f3305ab93b33b330e8b1d3165
$ echo -n 40730094339 | md5sum|cut -d" " -f1
cafc48d392a004f75b669f9d1d7bf894
I tried some prints:
cat file| awk '{print $0,NF,NR;cmd="echo -n " $1 " | md5sum|cut -d\" \" -f1"; cmd|getline md5; $1=md5"---"cmd"---"$1;print;}' | cat -n
but with no success to find what's going wrong.
EDIT: As the title says, I try to replace a column in a file(a file with hundred fields). So, $1 would be $24 and NF would be 120 for a file and 233 for another file.
I wouldn't use getline in awk like that. You can do:
while read -r num; do
echo -n $num | md5sum | cut -d ' ' -f1;
done < file
29ece26ce4633b6e9480255db194cc40
120148eca0891d0fc645413d0f26b66b
cafc48d392a004f75b669f9d1d7bf894
7b4367e8f58835c0827dd6a2f61b7258
7b4367e8f58835c0827dd6a2f61b7258
49b12d1f3305ab93b33b330e8b1d3165
cafc48d392a004f75b669f9d1d7bf894
bee44c89ac9d4e8e4e1f1c5c63088c71
f07262ac8f53755232c5abbf062364d0
2ac7c22170c00a3527eb99a2bfde2c2c
Ok, I found the issue. The pipes in awk should be closed.
So, I needed a close(cmd);
I found the solution here
I would GUESS, but can't tell since you aren't testing it's return code, that it's because your getline is failing at line 7 so md5 has the same value it did for the previous line. Use of getline is fraught with caveats and not for use by beginners, see http://awk.info/?tip/getline.
What value are you getting out of using awk for this anyway as opposed to just staying in shell?
It's a bit awkward with all the quoting - I'm not sure why would it fail to be honest. But here's something that uses less awk and works just fine:
< tmp | while read num ; do echo -n $num | md5sum | cut -f1 -d' '; done | cat -n