I'm trying to modify an existing script I have to take up to three text files and transform them. Currently the script will only transform the text from one file. Here's the existing script I have:
if [ $# -eq 1 ]
then
if [ -f $1 ]
then
name="My Name"
echo $name
date
starting_data=$1
sed '/^id/ d' $starting_data > raw_data3
sed 's/-//g' raw_data3 > raw_data4
cut -f1 -d, raw_data4 > cutfile1.col1
cut -f2 -d, raw_data4 > cutfile1.col2
cut -f3 -d, raw_data4 > cutfile1.col3
sed 's/$/:/' cutfile1.col2 > last
sed 's/^ //' last > last2
sed 's/^ //' cutfile1.col3 > first
paste -d\ first last2 cutfile1.col1 > final
cat final
else
echo "$1 cannot be found."
fi
else
echo "Please enter a filename."
fi
All those temp files are unnecessary. awk can do all of what sed and cut can do, so this should be what you want (pending the output field separator question)
if [ $# -eq 0 ]; then
echo "usage: $0 file ..."
exit 1
fi
for file in "$#"; do
if ! [ -f "$file" ]; then
echo "file not found: $file"
continue
fi
name="My Name"
echo "$name"
date
awk -F, -v OFS=" " '
/^id/ {next}
{
gsub(/-/, "")
sub(/^ /, "", $2)
sub(/^ /, "", $3)
print $3, $2 ":", $1
}
' "$file" > final
cat final
done
Note all my double quotes: those are required.
Related
Here is my shell code,My question is I don't want the ',',at the end of json file.
#!/bin/bash
PATCH_VERSION_FILE=/root/workspace/patch_version.json
filepath=/root/workspace/txtdir
for file in "${filepath}"/*.txt; do
echo " {" >> ${PATCH_VERSION_FILE}
filename=`echo ${file} | awk -F'/' '{ print $(NF) }'`
filemd5=`md5sum "${file}" | awk '{ print $1 }'`
echo " \"${filename}\"":"\"$filemd5\"">>${PATCH_VERSION_FILE}
echo " },">>${PATCH_VERSION_FILE}
done
Output:
{
"2001.txt":"d41d8cd98f00b204e9800998ecf8427e"
},
{
"2002.txt":"d41d8cd98f00b204e9800998ecf8427e"
},
{
"2003.txt":"d41d8cd98f00b204e9800998ecf8427e"
},
{
"2004.txt":"d41d8cd98f00b204e9800998ecf8427e"
},
{
"2005.txt":"d41d8cd98f00b204e9800998ecf8427e"
},
I found a soulution,but it looks ugly,the code below:
n=0
for file in "${filepath}"/*.txt; do
if [ $n -ne 0 ];then
echo " ," >> ${PATCH_VERSION_FILE}
fi
echo " {" >> ${PATCH_VERSION_FILE}
filename=`echo ${file} | awk -F'/' '{ print $(NF) }'`
filemd5=`md5sum "${file}" | awk '{ print $1 }'`
echo " \"${filename}\"":"\"$filemd5\"">>${PATCH_VERSION_FILE}
echo " }">>${PATCH_VERSION_FILE}
n=$(( $n + 1 ))
done
but the ',' not the same line with '}',is there any ways to deal with this ?
You could add this at the end of your script (after your for loop). It will simply remove (actually will replace with empty string) the last character of the file :
sed -i '$ s/.$//' $PATCH_VERSION_FILE
In order to have valid JSON data, you can use a JSON aware tool like jq:
md5sum "$filepath"/*.txt | jq -R 'split(" ")|{(.[1]):.[0]}' >> ${PATCH_VERSION_FILE}
-R option allows jq to read strings from md5sum.
The string is splitted in 2 and then assigned to the key/value object.
This is a part of my shell script.
for line in `cat $1`
do
startNum=`echo $line | awk -F "," '{print $1}'`
endNum=`echo $line | awk -F "," '{print $2}'`
operator=`echo $line | awk -F "," '{print $3}'`
termPrefix=`echo $line | awk -F "," '{print $4}'`
if [[ "$endNum" == 81* ]] || [[ "$endNum" == 33* ]] || [[ "$endNum" == 55* ]]
then
areaCode="${endNum:0:2}"
series="${endNum:2:4}"
startCLI="${startNum:6:4}"
endCLI="${endNum:6:4}"
else
areaCode="${endNum:0:3}"
series="${endNum:3:3}"
startCLI="${startNum:6:4}"
endCLI="${endNum:6:4}"
fi
echo "Add,${areaCode},${series},${startCLI},${endCLI},${termPrefix},"
#>> ${File}
done
input is csv contains below many rows :
5557017101,5557017101,102,1694
5515585614,5515585614,102,084
Output od shell script :
,dd,55,5701,7101,7101,1694
,dd,55,1558,5614,5614,0848
Not sure why comma is coming in startign of output, instead as per shell script it should come in the end.
please help
Here is a suggested awk command that should replace all of your shell+awk code. This awk also takes care of trailing \r:
awk -v RS=$'\r' 'BEGIN{FS=OFS=","} NF>3{
startNum=$1; endNum=$2; termPrefix=$4;
if (endNum ~ /^(81|33|55)/) {
areaCode=substr(endNum,1,2); series=substr(endNum,3,4)
}
else {
areaCode=substr(endNum,1,3); series=substr(endNum,4,3)
}
startCLI=substr(startNum,7,4); endCLI=substr(endNum,7,4);
print "Add", areaCode, series, startCLI, endCLI, termPrefix
}' file
Add,55,5701,7101,7101,1694
Add,55,1558,8561,5614,084
Consider the below file.
HEAD~XXXX
XXX~XXX~XXX~XXX~XXX~XXX~~WIN~SCRIPT~~~
XXX~XXX~XXX~XXX~XXX~XXX~~WIN~TPSCRI~~~
XXX~XXX~XXX~XXX~XXX~XXX~~WIN~RSCPIT~~~
TAIL~20
wish the Output to be like below for the above:
HEAD~XXXX
XXX~XXX~XXX~XXX~XXX~XXX~~WIN~SCRIPT~~~
XXX~XXX~XXX~XXX~XXX~XXX~~~~~~
XXX~XXX~XXX~XXX~XXX~XXX~~~~~~
TAIL~20
If the 9th field is SCRIPT, I want both 8th & 9th fields to be empty like the 10th & if the line contains words HEAD/TAIL those have to ignored from our above condition, i.e., NF!=13 - will need the header & footer as it is in the input.
I have tried the below, but there should be a smarter way.
awk -F'~' -v OFS='~' '($9 != "Working line takeover with change of CP" {$9 = ""}) && ($9 != "Working line takeover with change of CP" {$8 = ""}) {NF=13; print}' file
the above doesn't work
head -1 file > head
tail -1 file > tail
sed -i '/HDR/d' file
sed -i '/TLR/d' file
sed -i '/^\s*$/d' file
awk -F'~' -v OFS='~' '$9 != "Working line takeover with change of CP" {$9,$8 = ""} {NF=13; print}' file >> file.tmp //syntax error
cat file.tmp >> head
cat tail >> head
echo "" >> head
mv head file1
I'm trying an UNIX shell script with the below requirements.
Consider a file like this..
XXX~XXX~XXX~XXX~XXX~XXX~~XXX~~SCRIPT~~~
XXX~XXX~XXX~XXX~XXX~XXX~~XXX~~OTHERS~~~~
XXX~XXX~XXX~XXX~XXX~XXX~~XXX~~OTHERS~~~
Each file should have 12 fields(~ as delimiter), if not a ~ has to removed.
If anything OTHER than SCRIPT string present in the 10th field, the field has to be removed.
I tried the below in /bin/bash, I know I'm not doing it so well. I'm feeding line to sed & awk commands.
while read readline
echo "entered while"
do
fieldcount=`echo $readline | awk -F '~' '{print NF}'`
echo "Field count printed"
if [ $fieldcount -eq 13 ] && [ $fieldcount -ne 12 ]
then
echo "entering IF & before deletion"
#remove delimiter at the end of line
#echo "$readline~" >> $S_DIR/$1.tmp
#sed -i '/^\s*$/d' $readline
sed -i s'/.$//' $readline
echo "after deletion"
if [ awk '/SCRIPT/' $readline -ne "SCRIPT"]
then
#sed -i 's/SCRIPT//' $readline
replace_what="OTHERS"
#awk -F '~' -v OFS=~ '{$'$replace_what'=''; print }'
sed -i 's/[^,]*//' $replace_what
echo "$readline" >> $S_DIR/$1.tmp
fi
else
echo "$readline" >> $S_DIR/$1.tmp
fi
done < $S_DIR/$1
awk -F'~' -v OFS='~' '$10 != "SCRIPT" {$10 = ""} {NF=12; print}' file
XXX~XXX~XXX~XXX~XXX~XXX~~XXX~~SCRIPT~~
XXX~XXX~XXX~XXX~XXX~XXX~~XXX~~~~
XXX~XXX~XXX~XXX~XXX~XXX~~XXX~~~~
In bash, I would write:
(
# execute in a subshell, so the IFS setting is localized
IFS='~'
while read -ra fields; do
[[ ${fields[9]} != "SCRIPT" ]] && fields[9]=''
echo "${fields[*]:0:12}"
done < file
)
Your followup question:
awk -F'~' -v OFS='~' '
$1 == "HEAD" || $1 == "TAIL" {print; next}
$9 != "SCRIPT" {$8 = $9 = ""}
{NF=13; print}
' file
If you have further questions, please create a new question instead of editing this one.
I'm looking for the easiest and most readable way to remove a field from a path. So for example, I have /this/is/my/complicated/path/here, and I would like to remove the 5th field ("/complicated") from the string, using bash commands, so that it becomes /this/is/my/path.
I could do this with
echo "/this/is/my/complicated/path/here" | cut -d/ -f-4
echo "/"
echo "/this/is/my/complicated/path/here" | cut -d/ -f6-
but I would like this done in just one easy command, something that would like
echo "/this/is/my/complicated/path" | tee >(cut -d/ -f-4) >(cut -d/ -f6-)
except that this doesn't work.
With cut, you can specify a comma separated list of fields to print:
$ echo "/this/is/my/complicated/path/here" | cut -d/ -f-4,6-
/this/is/my/path/here
So, it's not really necessary to use two commands.
How about using sed?
$ echo "/this/is/my/complicated/path/here" | sed -e "s%complicated/%%"
/this/is/my/path/here
This removes the 5th path element
echo "/this/is/my/complicated/path/here" |
perl -F/ -lane 'splice #F,4,1; print join("/", #F)'
just bash
IFS=/ read -a dirs <<< "/this/is/my/complicated/path/here"
newpath=$(IFS=/; echo "${dirs[*]:0:4} ${dirs[*]:5}")
Anything wrong with a bash script?
#!/bin/bash
if [ -z "$1" ]; then
us=$(echo $0 | sed "s/^\.\///") # Get rid of a starting ./
echo " "Usage: $us StringToParse [delimiterChar] [start] [end]
echo StringToParse: string to remove something from. Required
echo delimiterChar: Character to mark the columns "(default '/')"
echo " "start: starting column to cut "(default 5)"
echo " "end: last column to cut "(default 5)"
exit
fi
# Parse the parameters
theString=$1
if [ -z "$2" ]; then
delim=/
start=4
end=6
else
delim=$2
if [ -z "$3" ]; then
start=4
end=6
else
start=`expr $3 - 1`
if [ -z "$4" ]; then
end=6
else
end=`expr $4 + 1`
fi
fi
fi
result=`echo $theString | cut -d$delim -f-$start`
result=$result$delim
final=`echo $theString | cut -d$delim -f$end-`
result=$result$final
echo $result
What is the right way to cut columns and print some of them multiple times in the output:
cut -d " " -f1,3,3,4 in > out
does not work.
awk '{print $1" "$3" "$3" "$4}' in > out
Bash
OLDIFS="$IFS"
while read -r line
do
set -- $line
echo "$1 $3 $3 $5"
done <file
IFS="$OLDIFS"