I am trying to remove the last newline added to the file using bash script.
I have got this -
truncate -s $(($(stat -c '%s' foo.txt)-1)) foo.txt
here foo.txt the file name.
but I want to parametrize the file name, I will pas the file name to the script and it should this remove the newline at last from that file.
Request your help on this. I do not have linux in my machine and tried using cygwin but it is giving error while running the script.
Thanks
To remove last line if it is newline use this sedL
sed -i.bak '/^[[:blank:]]*$/{$d;}' foo.txt
Related
I have a file with lot of a strings and one line starts with LIBXML2_INCLUDE
and the file is generated by another program to be specific by ./configure, this line wrongly gives two path and the first path is not correct and i need to remove it. This is how the line appears in file
LIBXML2_INCLUDE=-I/home/gan/Music/wvm/build/level/ast/deliveryx/libxml2//home/gan/Music/wvm/build/level/ast/deliveryx/libxml2/include/libxml2
i need to remove first /home/gan/Music/wvm/build/level/ast/deliveryx/libxml2/
and expected output is
LIBXML2_INCLUDE=-I/home/gan/Music/wvm/build/level/ast/deliveryx/libxml2/include/libxml2
How can i create a bash one liner to accomplish this?
Try like this:
# cat file
SOMEVAR=-I/some/path//some/path
# sed -i -e '/^SOMEVAR=/s,=-I.*//,=-I/,' file
# cat file
SOMEVAR=-I/some/path
#
To be a bit more fancy --
$ cat file
SOMEVAR=-I/some/path//some/path
$ sed -i -e '/^SOMEVAR=/s,=-I\(.*\)/\1$,=-I\1/,' file
$ cat file
SOMEVAR=-I/some/path/
$
I'm trying to rename a batch of files using a bash script or just in the command line but can't seem to find anything on how to remove characters before the first occurrence of a character.
Right now my files are named:
author1_-_year_-_title_name.txt
author2_-_year_-_title_name.txt
And I want them to look like
_-_year_-_title_name.text
or even
year_-_title_name.text
I've tried sed in the command line:
sed 's/^[^_-_]* _-_ //' *
but this only tried to edit the text files, not the file name
You can't change filenames using sed. Try this simple loop instead:
for fp in ./*_-_*; do
echo mv "$fp" "${fp#*_-_}"
done
If the output looks good, remove echo.
Could you please try rename command as follows.
rename -n s/[^-]*-_// *.txt
Output will be as follows.
rename(author1_-_year_-_title_name.txt, year_-_title_name.txt)
rename(author2_-_year_-_title_name.txt, year_-_title_name.txt)
Once you are Happy with above results(which will print only on terminal) remove -n option in above command and it should rename the files.
I am facing a strange problem. An answer to what I want to do already exists Here. I am trying to remove trailing commas from each line of a file containing thousands of lines. Like this -
This is my command -
sed -i 's/,*$//g' file_name.csv
However, the output I get is exactly the same as the image above and the trailing commas are not removed.
I think SED is not matching the pattern and thus failing to replace the commas. To check if there are any hidden characters in the file, I used VIM's :set list option -
There are only $ at the end of each line which is just what is expected.
I can't understand why the command is failing.
I can suggest you two options:
First One is my favorite.
dos2unix file
#####will work for Huge File also
then try to run the command.
Other way to do this:
cat file | tr -d '\r' > file
###may not work for huge file
then run the command.
tr -d '\r' < file > file.tmp ; mv file.tmp file
##will work for Huge File also
Thanks to #Nahuel for suggesting last command.
I have a Unix shell script which does the following:
creates a backup of a file
appends some text to a file
Now in #2 if I insert a text, ^M gets appended on all the lines of the file.
For example:
echo " a" >> /cust/vivek.txt
echo " b" >> /cust/vivek.txt
vi vivek.txt
abc^M
bcd^M
a^M
b^M
Any way to avoid this?
I'm not sure how echo could be producing ^M characters but you can remove them by running dos2unix on your file, like this:
dos2unix /cust/vivek.txt
Only
sed -e "s/\r//g" file
worked for me
^M is a carriage return, and is commonly seen when files are copied from Windows. Use:
od -xc filename
that should give a low-level list of what your file looks like. If you file does not come from Windows then another possibility is that your terminal setting are not translating correctly. Check that the TERM environment variable is correct.
If the file has come from Windows, then use dos2unix or sed 's/\r//' file > file.new
I suspect this may be an artifact of your vi settings, rather than the concatenation.
What does
cat -v -e filename
show ? This command will dump out your file and mark the control characters so it's clear what's really in your file. See also this Superuser question/answer set.
^M are the meta characters which entered your file when it was used in windows.
the dos2unix command can fix this.
dos2unix <filename>
If you are using windows, you can use notepad++ and to try Edit > EOL Conversion > Unix/OSX Format, before load file to server.
I'm a Java developer and I'm using Ubuntu to develop. The project was created in Windows with Eclipse and it's using the Windows-1252 encoding.
To convert to UTF-8 I've used the recode program:
find Web -iname \*.java | xargs recode CP1252...UTF-8
This command gives this error:
recode: Web/src/br/cits/projeto/geral/presentation/GravacaoMessageHelper.java failed: Ambiguous output in step `CR-LF..data
I've searched about it and get the solution in Bash and Windows, Recode: Ambiguous output in step `data..CR-LF' and it says:
Convert line endings from CR/LF to a
single LF: Edit the file with Vim,
give the command :set ff=unix and save
the file. Recode now should run
without errors.
Nice, but I've many files to remove the CR/LF character from, and I can't open each to do it. Vi doesn't provide any option to command line for Bash operations.
Can sed be used to do this? How?
There should be a program called dos2unix that will fix line endings for you. If it's not already on your Linux box, it should be available via the package manager.
sed cannot match \n because the trailing newline is removed before the line is put into the pattern space, but it can match \r, so you can convert \r\n (DOS) to \n (Unix) by removing \r:
sed -i 's/\r//g' file
Warning: this will change the original file
However, you cannot change from Unix EOL to DOS or old Mac (\r) by this. More readings here:
How can I replace a newline (\n) using sed?
Actually, Vim does allow what you're looking for. Enter Vim, and type the following commands:
:args **/*.java
:argdo set ff=unix | update | next
The first of these commands sets the argument list to every file matching **/*.java, which is all Java files, recursively. The second of these commands does the following to each file in the argument list, in turn:
Sets the line-endings to Unix style (you already know this)
Writes the file out iff it's been changed
Proceeds to the next file
I'll take a little exception to jichao's answer. You can actually do everything he just talked about fairly easily. Instead of looking for a \n, just look for carriage return at the end of the line.
sed -i 's/\r$//' "${FILE_NAME}"
To change from Unix back to DOS, simply look for the last character on the line and add a form feed to it. (I'll add -r to make this easier with grep regular expressions.)
sed -ri 's/(.)$/\1\r/' "${FILE_NAME}"
Theoretically, the file could be changed to Mac style by adding code to the last example that also appends the next line of input to the first line until all lines have been processed. I won't try to make that example here, though.
Warning: -i changes the actual file. If you want a backup to be made, add a string of characters after -i. This will move the existing file to a file with the same name with your characters added to the end.
Update: The Unix to DOS conversion can be simplified and made more efficient by not bothering to look for the last character. This also allows us to not require using -r for it to work:
sed -i 's/$/\r/' "${FILE_NAME}"
The tr command can also do this:
tr -d '\15\32' < winfile.txt > unixfile.txt
and should be available to you.
You'll need to run tr from within a script, since it cannot work with file names. For example, create a file myscript.sh:
#!/bin/bash
for f in `find -iname \*.java`; do
echo "$f"
tr -d '\15\32' < "$f" > "$f.tr"
mv "$f.tr" "$f"
recode CP1252...UTF-8 "$f"
done
Running myscript.sh would process all the java files in the current directory and its subdirectories.
In order to overcome
Ambiguous output in step `CR-LF..data'
the simple solution might be to add the -f flag to force the conversion.
Try the Python script by Bryan Maupin found here (I've modified it a little bit to be more generic):
#!/usr/bin/env python
import sys
input_file_name = sys.argv[1]
output_file_name = sys.argv[2]
input_file = open(input_file_name)
output_file = open(output_file_name, 'w')
line_number = 0
for input_line in input_file:
line_number += 1
try: # first try to decode it using cp1252 (Windows, Western Europe)
output_line = input_line.decode('cp1252').encode('utf8')
except UnicodeDecodeError, error: # if there's an error
sys.stderr.write('ERROR (line %s):\t%s\n' % (line_number, error)) # write to stderr
try: # then if that fails, try to decode using latin1 (ISO 8859-1)
output_line = input_line.decode('latin1').encode('utf8')
except UnicodeDecodeError, error: # if there's an error
sys.stderr.write('ERROR (line %s):\t%s\n' % (line_number, error)) # write to stderr
sys.exit(1) # and just keep going
output_file.write(output_line)
input_file.close()
output_file.close()
You can use that script with
$ ./cp1252_utf8.py file_cp1252.sql file_utf8.sql
Go back to Windows, tell Eclipse to change the encoding to UTF-8, then back to Unix and run d2u on the files.