Losing newlines when reading from stdin - bash

I have been beating my head up about this.
I wanted to loop over a multiline string character by character in bash but was loosing all newlines. First thing I did when i didn't find any obvious error was to run shellcheck on it, it seemed fine with the program.
script.sh:
#!/usr/bin/env bash
transform_single() {
if [[ $# -ne 1 ]]; then
echo 'Error: illegal number of args' 1>&2
fi
equation=''
delim0=0
delim="$1"
while IFS= read -rn1 c; do
if [[ $delim0 -eq 0 ]] && [[ "$c" == "$delim" ]]; then
delim0=1
equation=''
elif [[ $delim0 -ne 0 ]] && [[ "$c" == "$delim" ]]; then
delim0=0
echo -n "$equation" | texmath
elif [[ $delim0 -ne 0 ]]; then
equation="$equation$c"
else
echo -n "$c"
fi
done
}
transform_single '$'
input.txt:
<newlines>
<newlines>
# Hello world!
<newlines>
This is a test string.
<newlines>
invocation:
bash script.sh < input.txt
output:
# Hello world!This is a test string.
excepted output:
The same as in the input file.

Working script
#!/bin/bash
transform_single() {
if (($# != 1)); then echo 'Error: illegal number of args' 1>&2; fi
equation=''
delim0=0
delim="$1"
while IFS= read -r -d $'\0' -n 1 c; do
if ((delim0 == 0)) && [[ "$c" == "$delim" ]]; then
delim0=1
equation=''
elif ((delim0 != 0)) && [[ "$c" == "$delim" ]]; then
delim0=0
echo -n "$equation" | texmath
elif ((delim0 != 0)); then
equation="$equation$c"
else
echo -n "$c"
fi
done
}
transform_single '$'
The issue is that you must set a delimiter to read, a null character, to preserve line feed.

Related

checking a string in bash to see if its a domain

I am working on a project in Bash that takes a live xlsx file, converts it into a csv file, and checks the file to make sure that the data inside it are urls. This is part of a larger progragam that will eventually test each url for domain squatting.
I am having problems with the verification of the string data. I am having to teach myself bash as i go along since this is a self study class. Thanks for the Help!
INPUT=domain3.csv
while IFS= read -r line
do
if [[ "$line" == *".com"*] || [ "$line" == *".net"*] || [ "$line" == *".org"*] || [ "$line" == *".biz"*]];
then echo "$line"
else echo "$line is not an URL"
fi
echo "Finished!"
done
Use the =~ to perform regular expression match:
if [[ $INPUT =~ \.(com|net|org)$ ]]
then
echo $INPUT is a domain
else
echo $INPUT is not a domain
fi
The expression reads that if $INPUT matches a dot (\.), then one of "com", "net", or "org", then end of string ($), then it is a domain.
[[ ... ]] (since bash 4.1) temporarily enables the extglob option, so you can write
if [[ "$line" == *.#(com|net|org|biz)* ]];
You probably don't actually want the trailing *, which would let you match things like foo.comzzz.
A case statement.
#!/bin/sh
while IFS= read -r line; do
case $line in
*.com|*.net|*.org|*.biz)
echo "$line";;
*) printf >&2 '%s is not a url!\n' "$line" ;;
esac
done
Please execute the below code once and then compare it with your's to find out the error.
while IFS= read -r line
do
if [[ "$line" == *".com"* ]] || [[ "$line" == *".net"* ]] || [[ "$line" == *".org"* ]] || [[ "$line" == *".biz"* ]]
then
echo "$line"
else
echo "$line is not an URL"
fi
done < $INPUT
echo "Finished!"

unable to read a file in while loop

I've the below code:
#!/bin/bash
. ~/.bash_profile
dt=$(date +"%Y.%m.%d")
if [[ "$#" -eq '0' ]] || [[ "$#" -eq '1' ]]
then
echo -e "Not correct usage"
exit 1
fi
tar zvxf $SHARE/*.gz -C $SHARE/landing/
sleep 3
ops () {
//
some sets of command
//
}
while read -r line
do
if [[ "$1" == "A" ]] || [[ "$1" == "B" ]] || [[ "$1" == "C" ]] && [[ "$2" =~ "$line" ]] || [[ "$2" == "ALL" ]]
then
ops
fi
done < $SHARE/landing/dir/ApprovedList."$1"
When i run the script, i can see that it's able to untar the .gz file and place it into the folder :$SHARE/landing/
But i guess it's not able to read the file and perform the operations inside the ops function, below is the last line of output i'm getting after running the script in interactive mode:
+ read -r line
Any help is most welcome!!!
Please show the input format, the argument format, and some explanation of those. In the meantime, tweaked for readability and error handling - edit to taste. YMMV.
#!/bin/bash
. ~/.bash_profile
ops () { # do some stuff
: some sets of commands ...
}
die() {
printf "%s\n\n Use: $0 x y z ...\n\n" "$1" >&2
kill -term $$ # exit in function behaves like return
}
# dt=$(date +"%Y.%m.%d") # unused
case "$#" in
0|1) die "Insufficient arguments" ;;
esac
tar zvxf "$SHARE"/*.gz -C "$SHARE/landing/" || die "tar failed"
sleep 3
infile="$SHARE/landing/dir/ApprovedList.$1"
[[ -e "$infile" ]] || die "$infile does not exist"
[[ -r "$infile" ]] || die "$infile is nor readable by $0 as $LOGNAME"
while read -r line
do case "$1" in
A|B|C) case "$2" in
*"$line"*|ALL) ops ;;
*) : data does not match, skipping ;;
esac ;;
*) die "Invalid arg1 '$1'";;
esac
done < "$infile"
Notes:
If you require exactly 2 arguments, then let's check exactly that:
die() {
printf "%s\n\n Use: $0 x y\n\n" "$1" >&2
kill -term $$ # exit in function behaves like return
}
and
case "$#" in
2) ;;
*) die "Incorrect # of arguments" ;;
esac
Also, better than the kill - add a trap at the top:
trap 'echo >&2 "ERROR in $0($BASH_SOURCE) at line $LINENO: [$BASH_COMMAND]"; exit $LINENO;' ERR
and use a literal error return in the function.
die() {
printf "%s\n\n Use: $0 x y\n\n" "$1" >&2
return 1
}
The check (except for [[ "$2" =~ "$line" ]])
if [[ "$1" == "A" ]] || [[ "$1" == "B" ]] || [[ "$1" == "C" ]] && [[ "$2" =~ "$line" ]] || [[ "$2" == "ALL" ]]
doesn't change inside the while-loop. So better check once before entering the while-loop.
So now you want to perform ops for each line of $SHARE/landing/dir/ApprovedList."$1".
You can use
xargs -a "$SHARE/landing/dir/ApprovedList.$1" -L 1 ops
EDIT:
When you have a simple check to perform for each line, you can move this check into the ops function.
First save $2 before entering the function:
to_be_checked="$2"
...
ops() {
[[ "$0" =~ "${to_be_checked}" ]] && return

Using sed to replace single line in while read loop

I'm needing help setting up a bash script for initializing some BC's in a file. Ideally, my program would iterate through each line and:
1) Read in BC type - (wall, outlet, inlet).
2) Change "type" field based on appropriate BC type.
Unfortunately, my program seems to replace all type fields in Step 2 instead of only the type field associated with the correct BC.
I think this has something to do with the sed command operating over the whole file instead of just the $line variable.
while IFS= read -r line || [[ -n "$line" ]]; do #go through each line of T2 file
if [[ $line == *wall_* ]] #if wall_*
then
echo "attempted to assign wall_*"
var=1 #wall #Go down 2 lines
elif [[ $line == *velocity-inlet* ]]
then
echo "attempted to assign outflow"
var=2 #inlet
elif [[ $line == *outflow* ]]
then
var=3 #outlet
fi
echo $var
if [[ $line == *type* && $var == 1 ]]
then
sed -i 's/.*type.*/type zeroGradient/' 0/T3
echo "Attempted wall zeroGradient"
elif [[ $line == *type* && $var == 2 ]]
then
sed -i 's/.*type.*/type fixedValue\nvalue uniform (3 0 0)/' 0/T3
elif [[ $line == type* && $var == 3 ]]
then
sed -i 's/.*type.*/type zeroGradient/' 0/T3
fi
sed -i '/nFaces*/d' 0/T3 #Deletes irrelevant stuff from boundary file copy
sed -i '/startFace*/d' 0/T3
done <0/T3.
For example, it is supposed to change:
velocity-inlet_1
{
type patch;
nFaces 336;
startFace 75515;
}
outflow_2
{
type patch;
nFaces 136;
startFace 75851;
}
To:
velocity-inlet_1
{
type fixedValue;
value uniform (3 0 0);
}
outflow_2
{
type zeroGradient;
}
But instead changes it wrongly changes it to:
velocity-inlet_1
{
type fixedValue;
value uniform (3 0 0);
}
outflow_2
{
type fixedValue;
value uniform (3 0 0);
}
Help me stack overflow, you're my only hope.
You have a few issues. sed will effect a whole line by default, and you're not telling it which line to modify in the first place. You're also modifying a file as you read it. I might go with something like this:
var="none"
while IFS= read -r line; do
if [[ "$line" =~ "wall" ]]; then
var=wall
elif [[ "$line" =~ "velocity-inlet" ]]; then
var=inlet
fi
if [[ "$line" =~ "type" && "$var" == "wall" ]]; then
echo "$line" | sed 's|\(type *\).*|\1zeroGradient|'
elif [[ "$line" =~ "type" && "$var" == "inlet" ]]; then
echo "$line" | sed 's|\(type *\).*|\1uniform (3 0 0)|'
else
echo "$line"
fi
done
And then do
script.sh < 0/T3 > 0/T3.mod
You can of course modify this to read/write from particular files as well, and you can avoid sed (see here...)

Bash For Loop With If & Elif Logic Broken

I've been playing with bash scripting for 40'ish days with 0 experience so forgive me if my code looks like crap. I have a script that will take the configured NTP servers out of the /etc/ntp.conf file (/root/ntp.conf for testing)
NTPSRVCounter=1
echo "--- NTP Configuration ---"
echo " "
while read -r line; do
if [ $NTPSRVCounter == 1 ] ; then
echo "Primary NTP: $line"
SEDConfiguredNTP1="$(echo $line | sed 's/\./\\./g')"
((NTPSRVCounter++))
echo " "
else
SEDConfiguredNTP2="$(echo $line | sed 's/\./\\./g')"
echo "Secondary NTP: $line"
echo ""
fi
done < <(grep -o -P '(?<=server ).*(?= iburst)' /root/ntp.conf)
And asks you if you want to change it with a case statement:
echo "Do you wish to change it? [Y/n]"
NTPSRVCounter2=1
read opt
case $opt in
Y|y) read -p "Enter in your primary NTP server: " -e -i '0.debian.pool.ntp.org' UserInputNTP1
read -p "Enter in your secondary NTP serer: " -e -i '1.debian.pool.ntp.org' UserInputNTP2
for NTP in "$UserInputNTP1" "$UserInputNTP2" ; do
is_fqdn "$NTP"
if [[ $? == 0 && $NTPSRVCounter2 == 1 ]] ; then
SEDUserInput1=$(echo $UserInputNTP1 | sed 's/\./\\./g')
((NTPSRVCounter2++))
elif [[ $? == 0 && $NTPSRVCounter2 == 2 ]] ; then
SEDUserInput2=$(echo $UserInputNTP2 | sed 's/\./\\./g')
sudo sed -i "s/$SEDConfiguredNTP1/$SEDUserInput1/g" /root/ntp.conf
sudo sed -i "s/$SEDConfiguredNTP2/$SEDUserInput2/g" /root/ntp.conf
else
echo "Fail!!! :-( "
fi
done
;;
N|n) return 0
;;
*) echo "I don't know what happened, but, eh, you're not supposed to be here."
;;
esac
The problem is with the "elif" statement and the function "is_fqdn" on the second run of the function. If I put "bash -x" on the script and run it, I see "is_fqdn" returning 0 on both runs of the function, but the elif statement "$?" is coming up as 1 instead of 0.
The two functions used are below. Have to validate NTP addresses as either valid domain names or I.P. addresses, right? :)
is_fqdn() {
hostname=$1
if [[ "$hostname" =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]; then
valid_ip "$hostname"
elif [[ "$hostname" == *"."* && "$hostname" != "localhost." && "$hostname" != "localhost" ]] ; then
return 0
else
return 1
fi
host $hostname > /dev/null 2>&1 || return 1
}
valid_ip(){
local stat=1
local ip=$1
if [[ $ip =~ ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]; then
OIFS=$IFS
IFS="."
ip=($ip)
IFS=$OIFS
[[ ${ip[0]} -le 255 && ${ip[1]} -le 255 && ${ip[2]} -le 255 && ${ip[3]} -le 255 ]]
stat=$?
fi
return "$stat"
}
The condition in your if sets the value of $?, and that is what's used by the condition in the elif part, not the return value of is_fqdn. You need to save the value if you want to use it in multiple places:
is_fqdn "$NTP"
is_fqdn_rv=$?
if [[ $is_fqdn_rv == 0 && $NTPSRVCounter2 == 1 ]] ; then
SEDUserInput1=$(echo $UserInputNTP1 | sed 's/\./\\./g')
((NTPSRVCounter2++))
elif [[ $is_fqdn_rv == 0 && $NTPSRVCounter2 == 2 ]] ; then
SEDUserInput2=$(echo $UserInputNTP2 | sed 's/\./\\./g')
sudo sed -i "s/$SEDConfiguredNTP1/$SEDUserInput1/g" /root/ntp.conf
sudo sed -i "s/$SEDConfiguredNTP2/$SEDUserInput2/g" /root/ntp.conf
else
echo "Fail!!! :-( "
fi

Remove Newline Character between two specific strings

I am using Unix Shell. How to remove newline character between two specific strings.
For example, input is:
CASE when a in ('abcd','bdcdf') then
Shng,
END as xyz
Output should be:
CASE when a in ('abcd','bdcdf') then Shng END as xyz,
Parse the file line for line and remember when you see a CASE or END.
The code beneath uses a short syntax for an if-statement and an echo that suppresses \n by the -n parameter.
incase=0
cat x.sql | while read -r line; do
[[ ${line} = CASE* ]] && incase=1;
[[ ${line} = END* ]] && incase=0
[[ ${incase} = 0 ]] && echo "${line}"
[[ ${incase} = 1 ]] && echo -n "${line} "
done
EDIT:
When you have nested CASEs (like CASE ... CASE ... END ... END) and all
CASEs start on different lines you can count how deep your nested.
incase=0
cat x.sql | while read -r line; do
[[ ${line} = CASE* ]] && (( incase = incase + 1)) ;
[[ ${line} = END* ]] && (( incase = incase - 1))
[[ ${incase} = 0 ]] && echo "${line}"
[[ ${incase} > 0 ]] && echo -n "${line} "
done
# You might want an extra echo here so your last line will finish with a \n
echo
EDIT 2: Often you can avoid cat (look for UUOC). Here the code is better as
incase=0
cat x.sql | while read -r line; do
[[ ${line} = CASE* ]] && incase=1;
[[ ${line} = END* ]] && incase=0
[[ ${incase} = 0 ]] && echo "${line}"
[[ ${incase} = 1 ]] && echo -n "${line} "
done
EDIT:
When you have nested CASEs (like CASE ... CASE ... END ... END) and all
CASEs start on different lines you can count how deep your nested.
incase=0
while read -r line; do
[[ ${line} = CASE* ]] && (( incase = incase + 1)) ;
[[ ${line} = END* ]] && (( incase = incase - 1))
[[ ${incase} = 0 ]] && echo "${line}"
[[ ${incase} > 0 ]] && echo -n "${line} "
done < x.sql
# You might want an extra echo here so your last line will finish with a \n
echo

Resources