I wanna replace(delete) the last line of the last control character newline form ls command stdout result.
Which I had try sed command is :
ls | sed '$ s/[[:cntrl:]]$//'
ls | sed '$ s/[[:graph:]]$//'
ls | sed '$ s/[[:print:]]$//'
The result I want like this result ls | head -c -1
Any wrong with my sed command?
Thanks !
This might work for you (GNU sed):
sed -z 's/\n$//' file
sed appears to automatically add a newline if the current line was terminated with a newline, and I don't see how to turn that behaviour off.
You might try a different approach: capture the output of the command (the command substitution syntax strips trailing newlines); and then print it without a newline
printf '%s' "$(ls)"
Related
I made a small bash script which gets the current air pressure from a website and writes it into a variable. After the air pressure I would like to add the current date and write everything into a text file. The target should be a kind of CSV file.
My problem. I always get a line break between the air pressure and the date. Attempts to remove the line break by sed or tr '\n' have failed.
2nd guess from me: wget is done "too late" and echo is already done.
So I tried it with && between all commands. Same result.
Operating system is Linux. Where is my thinking error?
I can't get any further right now. Thanks in advance.
Sven
PS.: These are my first attempts with sed. This can be written certainly nicer ;)
#!/bin/bash
luftdruck=$(wget 'https://www.fg-wetter.de/aktuelle-messwerte/' -O aktuell.html && cat aktuell.html | grep -A 0 hPa | sed -e 's/<[^>]*>//g' | sed -e '/^-/d' | sed -e '/title/d' | sed -e 's/ hPa//g')
datum=$(date)
echo -e "${luftdruck} ${datum}" >> ausgabe.txt
Replace sed -e 's/ hPa//g') with sed -e 's/ hPa//g' | dos2unix) to replace trailing carriage return (DOS/Windows) with line feed (Unix/Linux).
The html file you download is using Windows line endings (Carriage Return \r + Line Feed \n). I assume your bash script only removes \ns, but the editor you are using to view the file is showing the \r as a linebreak.
Therefore, you could pipe everything through tr -d \\r\\n which would remove all line breaks.
But there is a better alternative: Extract only the important part instead of whole lines.
luftdruck=$(
wget 'https://www.fg-wetter.de/aktuelle-messwerte/' -O - |
grep -o '[^>]*hPa' | tr -dc 0-9.
)
echo "$luftdruck $(date)" >> ausgabe.txt
I am trying to create a string eg (file1.txt,file2.txt,file3.txt).
All these 3 files names are within a file.
ls file*.txt > lstfiles.txt
while read filename; do
filename+=$line","
done <lstfiles.txt
This returns me with output:
file1.txt,file2.txt,file3.txt,
How can I find the last iteration of the loop so I dont add another comma at the end.
Required output:
file1.txt,file2.txt,file3.txt
For your use case I would rather get rid of the while loop and combine sedand tr commands like so:
sed -e '$ ! s/$/,/g' lstfiles.txt | tr -d '\n'
Where sed command replace each line endings execept the last one with a comma and tr command remove the linebreaks.
Probably avoid using ls in scripts though.
printf '%s,' file*.txt |
sed 's/,$/\n/'
assuming your sed recognizes \n to be a newline, and copes with input which doesn't have a final newline.
I am trying to write a shell script that will replace whatever characters/strings I choose using sed. My first attempt worked with the exception of special characters. I have been trying to use sed to fix the special characters so that they too will be searched for or replaced. I decided to simplify the script for testing purposed, and just deal with a single offending character. However, I am still having problems.
Edited Script
#! /bin/sh
oldString=$1
newString=$2
file=$3
oldStringFixed=$(echo "$oldString" | sed 's/\\/\\\\/g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\[/\\\[/g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\]/\\\]/g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\^/\\\^/g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\*/\\\*/g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\+/\\\+/g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\./\\\./g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\$/\\\$/g')
oldStringFixed=$(echo "$oldStringFixed" | sed 's/\-/\\\-/g')
sed -e "s/$oldStringFixed/$newString/g" "$file" > newfile.updated
mv newfile.updated "$file"#! /bin/sh
In case it is not clear, I am trying to search through oldString for the [ character, and replace it with an escaped version and assign the results to oldStringFixed (do I need the backticks for this?). The bottom two lines are slightly modified versions of my original script that I believe works correctly.
When I echo the fixed string, nothing is displayed, and sed outputs an error
sed: can't read [: No such file or directory
Can anyone explain what Is wrong with my first sed line?
EDIT:
Thanks to Jite, the script is working better. However, I am still having a problem with replacing single quoted characters with spaces, i.e. ' *'. The new version is above.
I suggest two improvements:
Do not stack calls to sed as you do, instead pack all of them in a single function, as escape_string below.
You can use a fancy delimiter for the sed substitute command to avoid issues linked to / being part of the strings involved.
With these changes, your script looks like:
#! /bin/sh
oldString="$1"
newString="$2"
file="$3"
escape_string()
{
printf '%s' "$1" | sed -e 's/[][\\^*+.$-]/\\\1/g'
}
fancyDelim=$(printf '\001')
oldStringFixed=$(escape_string "$oldString")
sed -e "s$fancyDelim$oldStringFixed$fancyDelim$newString${fancyDelim}g" "$file" \
> newfile.updated
mv newfile.updated "$file"
To replace values containing special characters try using sed with "|" instead of "/"
Eg: sed -i 's|'$original_value'|'$new_value'|g'
where original_value="comprising_special_char_/"
new_value="comprising_new_special_char:"
Change:
oldStringFixed= `sed 's/\[/\[/g' "$oldString"\`
to:
oldStringFixed=$(echo "$oldString" | sed 's/\[/\\\[/g')
Problem 1: Space after =, it's not allowed when assigning shell variables.
Problem 2: sed expects a file as input, not a string. You may pipe it as my solution does though.
Problem 3: You need to escape the backslash first \\, then you need to escape your char \[, totalling \\\[ :)
Side note: I changed `` to $() since the latter is the recommended praxis (due to nesting, another topic).
For me it was just a nightmare trying to get sed to do this for the general case. I gave up and wrote a short Python code to replace sed:
#!/usr/bin/python
# replace.py
import sys
# Replace string in a file (in place)
match=sys.argv[1]
replace=sys.argv[2]
filename=sys.argv[3]
print "Replacing strings in",filename
with open(filename,"r") as f:
data = f.read().replace(match,replace)
with open(filename,"w") as f:
f.write(data)
Which can then be used like:
#!/bin/bash
orig='<somethinghorrible>'
out='<replacement>'
python replace.py "$orig" "$out" myfile.txt
you can use this for replacing " with \" sed 's/\"/\\\"/g' filename
How can I join multiple lines into one line, with a separator where the new-line characters were, and avoiding a trailing separator and, optionally, ignoring empty lines?
Example. Consider a text file, foo.txt, with three lines:
foo
bar
baz
The desired output is:
foo,bar,baz
The command I'm using now:
tr '\n' ',' <foo.txt |sed 's/,$//g'
Ideally it would be something like this:
cat foo.txt |join ,
What's:
the most portable, concise, readable way.
the most concise way using non-standard unix tools.
Of course I could write something, or just use an alias. But I'm interested to know the options.
Perhaps a little surprisingly, paste is a good way to do this:
paste -s -d","
This won't deal with the empty lines you mentioned. For that, pipe your text through grep, first:
grep -v '^$' | paste -s -d"," -
This sed one-line should work -
sed -e :a -e 'N;s/\n/,/;ba' file
Test:
[jaypal:~/Temp] cat file
foo
bar
baz
[jaypal:~/Temp] sed -e :a -e 'N;s/\n/,/;ba' file
foo,bar,baz
To handle empty lines, you can remove the empty lines and pipe it to the above one-liner.
sed -e '/^$/d' file | sed -e :a -e 'N;s/\n/,/;ba'
How about to use xargs?
for your case
$ cat foo.txt | sed 's/$/, /' | xargs
Be careful about the limit length of input of xargs command. (This means very long input file cannot be handled by this.)
Perl:
cat data.txt | perl -pe 'if(!eof){chomp;$_.=","}'
or yet shorter and faster, surprisingly:
cat data.txt | perl -pe 'if(!eof){s/\n/,/}'
or, if you want:
cat data.txt | perl -pe 's/\n/,/ unless eof'
Just for fun, here's an all-builtins solution
IFS=$'\n' read -r -d '' -a data < foo.txt ; ( IFS=, ; echo "${data[*]}" ; )
You can use printf instead of echo if the trailing newline is a problem.
This works by setting IFS, the delimiters that read will split on, to just newline and not other whitespace, then telling read to not stop reading until it reaches a nul, instead of the newline it usually uses, and to add each item read into the array (-a) data. Then, in a subshell so as not to clobber the IFS of the interactive shell, we set IFS to , and expand the array with *, which delimits each item in the array with the first character in IFS
I needed to accomplish something similar, printing a comma-separated list of fields from a file, and was happy with piping STDOUT to xargs and ruby, like so:
cat data.txt | cut -f 16 -d ' ' | grep -o "\d\+" | xargs ruby -e "puts ARGV.join(', ')"
I had a log file where some data was broken into multiple lines. When this occurred, the last character of the first line was the semi-colon (;). I joined these lines by using the following commands:
for LINE in 'cat $FILE | tr -s " " "|"'
do
if [ $(echo $LINE | egrep ";$") ]
then
echo "$LINE\c" | tr -s "|" " " >> $MYFILE
else
echo "$LINE" | tr -s "|" " " >> $MYFILE
fi
done
The result is a file where lines that were split in the log file were one line in my new file.
Simple way to join the lines with space in-place using ex (also ignoring blank lines), use:
ex +%j -cwq foo.txt
If you want to print the results to the standard output, try:
ex +%j +%p -scq! foo.txt
To join lines without spaces, use +%j! instead of +%j.
To use different delimiter, it's a bit more tricky:
ex +"g/^$/d" +"%s/\n/_/e" +%p -scq! foo.txt
where g/^$/d (or v/\S/d) removes blank lines and s/\n/_/ is substitution which basically works the same as using sed, but for all lines (%). When parsing is done, print the buffer (%p). And finally -cq! executing vi q! command, which basically quits without saving (-s is to silence the output).
Please note that ex is equivalent to vi -e.
This method is quite portable as most of the Linux/Unix are shipped with ex/vi by default. And it's more compatible than using sed where in-place parameter (-i) is not standard extension and utility it-self is more stream oriented, therefore it's not so portable.
POSIX shell:
( set -- $(cat foo.txt) ; IFS=+ ; printf '%s\n' "$*" )
My answer is:
awk '{printf "%s", ","$0}' foo.txt
printf is enough. We don't need -F"\n" to change field separator.
Here's my problem, I have written the following line of code to format properly a list of files found recursively in a directory.
find * | sed -e '/\(.*\..*\)/ !d' | sed -e "s/^.*/\${File} \${INST\_FILES} &/" | sed -e "s/\( \)\([a-zA-Z0-9]*\/\)/\/\2/" | sed -e "s/\(\/\)\([a-zA-Z0-9\_\-\(\)\{\}\$]*\.[a-zA-Z0-9]*\)/ \2/"
The second step is to write the output of this command in a script. While the code above has the expected behavior, the problem occurs when I try to store its output to a variable, I get a bad substitution error from the first sed command in the line.
#!/bin/bash
nsisscript=myscript.sh
FILES=*
for f in $(find $FILES); do
v=`echo $f | sed -e '/\(.*\..*\)/ !d' | sed -e "s/^.*/\${File} \${INST\_FILES} &/" | sed -e "s/\( \)\([a-zA-Z0-9]*\/\)/\/\2/" | sed -e "s/\(\/\)\([a-zA-Z0-9\_\-\(\)\{\}\$]*\.[a-zA-Z0-9]*\)/ \2/"`
sed -i.backup -e "s/\;Insert files here/$v\\n&/" $nsisscript
done
Could you please help me understand what the difference is between the two cases and why I get this error ?
Thanks in advance!
Well my guess was that your escaping of underscore in INST_FILES is strange as underscore is not a special character in shell nor in sed. The error disappear when you delete the '\' before '_'
my 2 cents
Parsing inside of backquote-style command substitution is a bit weird -- it requires an extra level of escaping (i.e. backslashes) to control when expansions take place. Ugly solution: add more backslashes. Better solution: use $() instead of backquotes -- it does the same thing, but without the weird parsing and escaping issues.
BTW, your script seems to have some other issues. First, I don't know about the sed on your system, but the versions I'm familiar with don't interpret \n in the substitution as a newline (which I presume you want), but as a literal n character. One solution is to include a literal newline in the substitution (preceded by a backslash).
Also, the loop executes for each found file, but for files that don't have a period in the name, the first sed command removes them, $v is empty, and you add a blank line to myscript.sh. You should either put the filtering sed call in the for statement, or add it as a filter to the find command.
#!/bin/bash
nsisscript=myscript.sh
nl=$'\n'
FILES=*
for f in $(find $FILES -name "*.*"); do
v=$(echo $f | sed -e "s/^.*/\${File} \${INST\_FILES} &/" | sed -e "s/\( \)\([a-zA-Z0-9]*\/\)/\/\2/" | sed -e "s/\(\/\)\([a-zA-Z0-9\_\-\(\)\{\}\$]*\.[a-zA-Z0-9]*\)/ \2/")
sed -i.backup -e "s/\;Insert files here/$v\\$nl&/" $nsisscript
done