So I have a variable TRAILER which contains about 50 character. This variable is defined earlier in my shell session. As you can probably tell, it's a trailer to a file we'll be sending. I need to insert the record count of that file into the trailer. This record count is going to be 9 digits long (left padded with zeros if need be) and will start at index 2 of that string TRAILER. I want to retain all other characters in the TRAILER string just insert the RECORD_COUNT variable value into the TRAILER variable starting at index 2 (3rd character)
So the trailer variable is defined like this:
#Trailer details
TRAILER_RECORD_IDENTIFER="T"
LIFE_CYCLE="${LIFE_CYCLE_ENV}"
RECORD_COUNT="" #This will be calculated in the wrapper during the creation step
FILE_NUMBER="1111"
FILE_COUNT="1111"
CONTROL_TOTAL_1=" "
CONTROL_TOTAL_2=" "
CONTROL_TOTAL_3=" "
CONTROL_TOTAL_4=" "
CONTROL_TOTAL_5=" "
TRAILER="${TRAILER_RECORD_IDENTIFER}"\
"${LIFE_CYCLE}"\
"${RECORD_COUNT}"\
"${FILE_NUMBER}"\
"${FILE_COUNT}"\
"${CONTROL_TOTAL_1}"\
"${CONTROL_TOTAL_2}"\
"${CONTROL_TOTAL_3}"\
"${CONTROL_TOTAL_4}"\
"${CONTROL_TOTAL_5}"
Which then prints TRAILER as
TRAILER="TD11111111......" that would be 75 blank spaces for all of the white characters defined by the CONTROL_TOTAL variables.
These variables ALL get defined in the beginning of the shell. REcord count is defined but left blank ebcause we won't know the specific file until later int he shell.
Later in the shell i know the file that i want to use, i get the record coun:
cat ${ADE_DATA_FL_PATH_TMP} | wc -l | read ADE_DATA_FL_PATH_TMP_REC_COUNT >> ${LOG_FILE} 2>&1
Now I want to take ADE_DATA_FL_PATH_TMP_REC_COUNT and write that value into the TRAILER variable starting at the 2nd index, padded with zero's to be 9 characters long. So if my record count is 2700 records the new trailer would look like...
TRAILER="TD00000270011111111......"
You can use printf for padding.
I use TD as fixed first two characters, you can change this the way you want.
printf -v TRAILER "TD%.9d%s" "${ADE_DATA_FL_PATH_TMP_REC_COUNT}" "$(cut -c 12- <<< "${TRAILER}")"
Perhaps this is a good time switching to writing variable names in lowercase.
Related
I have a file - abc, which has the below content -
Bob 23
Jack 44
Rahul 36
I also have a shell script that do the addition of all the numbers here.
The specific line that picks up these numbers is -
while read line
do
num=echo ${line#* }
sum=`expr $sum + $num`
count=`expr $count + 1`
done< "$readfile"
I assumed that the code is just picking up the last field from file, but it's not. If i modify the file like
Bob 23 12
Jack 44 23
Rahul 36 34
The same script fails with syntax error.
NOTE: I know there are other ways to pick up the field value, but i would like to know how this works.
The syntax ${line#* } will skip the shortest string from the beginning till it finds a space and returns the rest. It worked fine when you had just 2 columns. But the same will not work when 3 columns are present as it will return you the last 2 column values which when you use it in the sum operator will throw you an error. To explain that, just imagine
str='foo bar'
printf '%s\n' "${str#* }"
bar
but imagine the same for 3 fields
str='foo bar foobar'
printf '%s\n' "${str#* }"
bar foobar
To fix that use the parameter expansion syntax of "${str##* }" to skip the longest sub-string from beginning. To fix your script for the example with 3 columns, I would use a script as below.
This does a simple input redirection on the file and uses the read command with the default IFS value which is a single white space. So I'm getting only the 3rd field on each line (even if it has multiple fields), the _ mark the fields I'm skipping. You could also have some variables as place-holders and use their value in the scripts also.
declare -i sum
while read -r _ _ value _ ; do
((sum+=value)
done < file
printf '%d\n' "$sum"
See Bash - Parameter Expansion (Substring removal) to understand more.
You could also use the PE syntax ${line##* } as below,
while read -r line ; do
((sum+=${line##* }))
done < file
[Not relevant to the current question]
If you just want the sum to be computed and not specifically worried about using bash script for this. You can use a simple Awk command to sum up values in 3rd column as
awk '{sum+=$3}END{print sum}' inputfile
i have multiple files in a directory that i need to reformat and put the output in one file, the file structure is:
========================================================
Daily KPIs - DATE: 24/04/2013
========================================================
--------------------------------------------------------
Number of des = 5270
--------------------------------------------------------
Number of users = 210
--------------------------------------------------------
Number of active = 520
--------------------------------------------------------
Total non = 713
--------------------------------------------------------
========================================================
I need the output format to be:
Date,Numberofdes,Numberofusers,Numberofactive,Totalnon
24042013,5270,210,520,713
The directory has around 1500 files with the same format and im using Centos 7.
Thanks
First we need a method to join the elements of an array into a string (cf. Join elements of an array?):
function join_array()
{
local IFS=$1
shift
echo "$*"
}
Then we can cycle over each of the files and convert each one into a comma-separated list (assuming that the original file have a name ending in *.txt).
for f in *.txt
do
sed -n 's/[^:=]\+[:=] *\(.*\)/\1/p' < $f | {
mapfile -t fields
join_array , "${fields[#]}"
}
done
Here, the sed command looks inside each input file for lines that:
begin with a substring that contains neither a : nor a = character (the [^:=]\+ part);
then follow a : or a = and an arbitrary number of spaces (the [:=] * part);
finally, end with an arbitrary substring (the *\(.*\) part).
The last substring is then captured and printed instead of the original string. Any other line in the input files is discared.
After that, the output of sed is read by mapfile into the indexed array variable fields (the -t ensures that trailing newlines from each line read are discarded) and finally the lines are joined thanks to our previously-defined join_array method.
The reason whereby we need to wrap mapfile inside a subshell is explained here: readarray (or pipe) issue.
Let's say I have a command my_command which I am sure outputs three lines like
line 1
line 2
line 3
and I want to store the three lines into three variables $x, $y, $z. How can I accomplish this in bash?
for name in x y z; do
read $name
done < <(my_command)
This uses process substitution so that the read commands are not executed in a subshell and the resulting variables are available after the loop completes. The variable name is used to hold the names of the variables to set; $name expands to the name of the variable that read sets, so after the loop you have three variables x, y, and z that hold each of the three lines of your output.
You can also use a here document instead of process substitution:
for name in x y z; do
read $name
done <<EOF
$(my_command)
EOF
And here's another:
IFS=$'\n' read -d '' x y z __ < <(my_command)
With -d '' it sets the delimiter to '\0' and makes read read the whole input in one instance, and not just a single line. IFS=$'\n' sets newline (\n) as the separator for each value. __ is optional and gathers any extra input besides the first 3 lines.
From help read:
Reads a single line from the standard input, or from file descriptor
FD if the -u option is supplied. The line is split into fields as with
word splitting, and the first word is assigned to the first NAME, the
second word to the second NAME, and so on, with any leftover words
assigned to the last NAME. Only the characters found in $IFS are
recognized as word delimiters.
-d delim continue until the first character of DELIM is read, rather than newline
I have a fixed length file in which some records have different special characters like Еӏєпа
I'm able to select those records containing special characters/.
I want to read 2 columns from those records and update it with '*' padded with blanks
Sample Data :
1234562013-09-01 01:05:30Еӏєпа Нцвѡі A other
5657812011-05-05 02:34:56abu jaya B other
Specifically, the 3rd and 4th column containing special characters, should be replaced with a single '*' padded with blanks to fill the length
I need result like below
1234562013-09-01 01:05:30* * A2013-09-01 02:03:40other
5657812011-05-05 02:34:56abu jaya B2013-09-01 07:06:10other
Tried the following commands :
sed -r "s/^(.{56}).{510}/\1$PAD/g;s/^(.{511}).{1023}/\1$PAD/g" errorline.txt
cut -c 57-568
Could someone help me out with this?
I would go with awk, something like:
awk '/[LIST__OF_SPECIAL_CHARS]/ {
l=$0
# for 3rd col
# NOTE the * must be padded if you have a fixed length file
# This can be done with spaces and/or (s)printf, read the docs
if (substr($0,FROM,NUM_OF_CHARS) ~ /[LIST__OF_SPECIAL_CHARS]/) {
l=substr(l,1,START_OF_3RD_COL_MINUS_1) "*" substr(l,START_OF_4TH_COL)
}
# for 4th col
# NOTE the * must be padded if you have a fixed length file
# This can be done with spaces and/or (s)printf, read the docs
if (substr($0,START_OF_4TH_COL,NUM_OF_CHARS) ~ /[LIST__OF_SPECIAL_CHARS]/) {
l=substr(l,1,START_OF_4TH_COL_MINUS_1) "*" substr(l,END_OF_4TH_COL_PLUS_1)
}
# after printing this line, skip to next record.
print l
next
}
{ # prints every other record
print }' INPUTFILE
sed "/.\{56\}.*[^a-zA-Z0-9 ].*.\{7\}/ s/\(.\{56\}\).\{20\}\(.\{7\}\)/\1* * \2/"errorline.txt
where:
56 is the first part of your line that don't contain special char
20 is the second part taht contain maybe special char
7 is the last part, end of your string.
"* * " is the string that will replace your special char section.
Adapt those values to your string structure
This sed read all the file and replace only the lines with special char.
I'm having trouble storing column values into shell script variables when these include white spaces, since all the results are split on whitespaces instead of actual column values.
For example, this is what I got now:
set -A SQL_RESULTS_ARRAY `sqlplus -s un/pass#database << EOF
SET ECHO OFF
SET FEED OFF
SET HEAD OFF
SET SPACE 0
SELECT EMAIL_SUBJECT, MAIL_TO FROM EMAIL_TABLE;
EOF`
echo "${SQL_RESULTS_ARRAY[0]}"
echo "${SQL_RESULTS_ARRAY[1]}"
This doesn't work because the value of EMAIL_SUBJECT is an entire sentence, ie "Message subject test", so those echos just end up printing
Message
subject
Instead of
Message subject test
email1#email.com email2#email.com
Basically, how do I end up with only two items in the array (one per column), instead of five items (one per word)? Is this at all possible with a single connection? (I'd rather not start a new connection per column)
EDIT: Another thing, another one of my CLOB columns is EMAIL_BODY, which can basically be any text-- thus I'd rather not have a preset separator, since EMAIL_BODY can have all sorts of commas, pipes, new lines, etc...
The key you're missing is to set the shell's IFS (internal field separator) to be the same as your query results. Here's a ksh session:
$ results="Message subject test,email1#email.com email2#email.com"
$ set -A ary $results
$ for i in 0 1 2 3 4; do print "$i. ${ary[$i]}"; done
0. Message
1. subject
2. test,email1#email.com
3. email2#email.com
4.
$ IFS=,
$ set -A ary $results
$ for i in 0 1 2 3 4; do print "$i. ${ary[$i]}"; done
0. Message subject test
1. email1#email.com email2#email.com
2.
3.
4.
You'll probably want to do something like this:
results=`sqlplus ...`
old_IFS="$IFS"
IFS=,
set -A SQL_RESULTS_ARRAY $results
IFS="$old_IFS
print "${SQL_RESULTS_ARRAY[0]}"
print "${SQL_RESULTS_ARRAY[1]}"
You may try to set COLSEP and separate by its value.
Try adding double quotes using string concatenation in the select statement. Array elements that are quoted permit white space (at least in bash).
read up about the bash's "Internal Field Separator" $IFS
it is set to whitespace by default, which may be causing your problem.