I am trying to pre-pend a text to a file using a Makefile. The following bash command works in terminal:
echo -e "DATA-Line-1\n$(cat input)" > input
But when I put the above command in a Makefile, it does not work:
copyHeader:
#echo -e "DO NOT EDIT THIS FILE \n$(cat input)" > input
I guess $(cat input) does not work as expected in the Makefile.
I'd recommend sed for prepending a line of text to a file. The i command is kind of a pain; some clever use of the hold space does the same thing in a more complicated but less troublesome way:
copyHeader:
sed -i "" '1{h;s/.*/NEW FIRST LINE/;G;}' input
But if you want to do it your way, I think an extra '$' will do the trick:
copyHeader:
#echo -e "DO NOT EDIT THIS FILE \n$$(cat input)" > input
EDIT: Thanks to MadScientist for pointing out that this method (using $(cat input)) is unreliable.
A file function was added to make in 4.0, which as of 4.2 can also read from files
The newline is a little bit hacky but this can be accomplished with make alone:
define n
endef
copyHeader:
$(file > input,DATA-Line-1$n$(file < input))
After messing around with this for a while (accepted answer does not work with the sed that comes with OSX 10.12, make was too old for the file manipulation options), I settled on the following (ugly) solution:
echo "DATA-Line-1" > line.tmp
mv input input.tmp
cat line.tmp input.tmp > input
rm input.tmp line.tmp
This works for me:
$ cat test
this
is
a
test
$ sed -i "1i new first line" test
$ cat test
new first line
this
is
a
test
Related
I need to create a file that lists all the files in a folder into a text file, along with a comma and the number 15 after. For example
My folder has video.mp4, video2.mp4, picture1.jpg, picture2.jpg, picture3.png
I need the text file to read as follows:
video.mp4,15
video2.mp4,15
picture1.jpg,15
picture2.jpg,15
picture3.png,15
No spaces, just filename.ext,15 on each line. I am using a raspberry pi. I am aware that the command ls > filename.txt would put all the file names into a folder, but how would I get a ,15 after every line?
Thanks
bash one-liner:
for f in *; do echo "$f,15" >> filename.txt; done
To avoid opening the output file on each iteration you may redirect the entire output with > filename.txt:
for f in *; do echo "$f,15"; done > filename.txt
$ printf '%s,15\n' *
picture1.jpg,15
picture2.jpg,15
picture3.png,15
video.mp4,15
video2.mp4,15
This will work if those are the only files in the directory. The format specifier %s,15\n will be applied to each of printf's arguments (the names in the current directory) and they will be outputted with ,15 appended (and a newline).
If there are other files, then the following would work too, regardless of whether there are files called like this or not:
$ printf '%s,15\n' video.mp4 video2.mp4 picture1.jpg picture2.jpg "whatever this is"
video.mp4,15
video2.mp4,15
picture1.jpg,15
picture2.jpg,15
whatever this is,15
Or, on all MP4, PNG and JPEG files:
$ printf '%s,15\n' *.mp4 *.jpg *.png
video.mp4,15
video2.mp4,15
picture1.jpg,15
picture2.jpg,15
picture3.png,15
Then redirect this to a file with printf ...as above... >output.txt.
If you're using Bash, then this will not make use of any external utility, as printf is built into the shell.
You need to do something like this:
#!/bin/bash
for i in $(ls folder_name); do
echo $i",15" >> filename.txt;
done
It's possible to do this in one line, however, if you want to create a script, consider code readability in the long run.
Edit 1: better solution
As #CristianRamon-Cortes suggested in the comments below, you should not rely on the output of ls because of the problems explained in this discussion: why not parse ls. As such, here's how you should write the script instead:
#!/bin/bash
cd folder_name
for i in *; do
echo $i",15" >> filename.txt;
done
You can skip the part cd folder_name if you are already in the folder.
Edit 2: Enhanced solution:
As suggested by #kusalananda, you'd better do the redirection after done to avoid opening the file in each iteration of the for loop, so the script will look like this:
#!/bin/bash
cd folder_name
for i in *; do
echo $i",15";
done > filename.txt
Just 1 command line using 2 msr commands recusively (-r) search specific files:
msr -rp your-dir1,dir2,dirN -l -f "\.(mp4|jpg|png)$" -PAC | msr -t .+ -o '$0,15' -PIC > save-file.txt
If you want to sort by time, add --wt to first command like: msr --wt -l -rp your-dirs
Sort by size? Add --sz but only the prior one is effective if use both --sz and --wt.
If you want to exclude some directory, add like: --nd "^(test|garbage)$"
remove tail \r\n in save-file.txt : msr -p save-file.txt -S -t "\s+$" -o "" -R
See msr.exe / msr.gcc48 etc in my open project https://github.com/qualiu/msr tools directory.
A solution without a loop:
ls | xargs -i echo {},15 > filename.txt
I would like to delete filenames from a textfile to have as output only the folder.
Example:
Creature\FrostwolfPup\FrostWolfPup_Shadow.m2
Creature\FrostwolfPup\FrostWolfPup_Fire.m2
To
Creature\FrostwolfPup\
To match only the Filenames i use [^\\]*$
Now i put it together with sed while /d should delete it
D:\filetype\core\sed.exe -n -e "/^[^\\]*$/d" D:\filetype\listfile\archive\tmp\all.txt > D:\filetype\module\model_bruteforce\tmp\folders_tmp1.txt
But instead of a textfile with my folders i got only a empty textfile as output, and so something must be wrong.
Tested on linux, not cygwin
sed -r 's/[^\\]*$//g' /path/to/original/file > /path/to/new/file
Try:
sed.exe -e "s/[^\\]*$//" path/to/folders.txt
The command s/[^\\]*$// asks sed to remove everything after the last \ on a line to the end of the line.
Caveat: since I don't have a windows machine handy for testing, I am unsure if the backslashes need to be doubled as shown above.
Discussion
-n tells sed not print anything unless we explicitly ask it to. The following command never asks sed to print:
sed.exe -n -e "/^[^\\]*$/d"
Consequently, it produces no output.
Specifically, I'm using a combination of >> and tee in a custom alias to store new Homebrew updates in a text file, as well as output on screen:
alias bu="echo `date "+%Y-%m-%d at %H:%M"` \
>> ~/Documents/Homebrew\ Updates.txt && \
brew update | tee -a ~/Documents/Homebrew\ Updates.txt"
Question: What if I wish to prepend this output in my textfile, i.e. placed at the beginning of the file as opposed to appending it to the end?
Edit1: As someone reported in the answers below, the use of temp files might be a good approach, which at least helped me partially:
targetLog="~/Documents/Homebrew\ Updates.txt"
alias bu="(brew update | cat - $targetLog \
> /tmp/out1 && mv /tmp/out1 $targetLog \
&& echo `date "+%Y-%m-%d at %H:%M":%S` | \
cat - $targetLog > /tmp/out2 \
&& mv /tmp/out2 $targetLog)"
But the problem is the output to STDOUT (previously made possible by tee), which I'm not sure can be incorporated in this tempfile approach …?
sed will happily do that for you, using -i to edit in place, eg.
sed -i -e "1i `date "+%Y-%m-%d at %H:%M"`" some_file
This works by creating an output file:
Let's say we have the initial contents on file.txt
echo "first line" > file.txt
echo "second line" >> file.txt
So, file.txt is our 'bottom' text file. Now prepend into a new 'output' file
echo "add new first line" | cat - file.txt > output.txt # <--- Just this command
Now, output has the contents the way we want. If you need your old name:
mv output.txt file.txt
cat file.txt
The only simple and safe way to modify an input file using bash tools, is to use a temp file, eg. sed -i uses a temp file behind the scenes (but to be robust sed needs more).
Some of the methods used have a subtle "can break things" trap, when, rather than running your command on the real data file, you run it on a symbolic link (to the file you intend to modify). Unless catered for correctly, this can break the link and convert it into a real file which receives the mods and leaves the original real file without the intended mods and without the symlink (no error exit-code results)
To avoid this with sed, you need to use the --follow-symlinks option.
For other methods, just be aware that it needs to follow symlinks (when you act on such a link)
Using a temp file, then rm temp file works only if "file" is not a symlink.
One safe way is to use sponge from package moreutils
Unlike a shell redirect, sponge soaks up all its input before
opening
the output file. This allows for constructing pipelines that read from
and write to the same file.
sponge is a good general way to handle this type of situation.
Here is an example, using sponge
hbu=~/'Documents/Homebrew Updates.txt'
{ date "+%Y-%m-%d at %H:%M"; cat "$hbu"; } | sponge "$hbu"
Simplest way IMO would be to use echo and cat:
echo "Prepend" | cat - inputfile > outputfile
Or for your example basically replace the tee -a ~/Documents/Homebrew\ Updates.txt with cat - ~/Documents/Homebrew\ Updates.txt > ~/Documents/Homebrew\ Updates.txt
Edit: As stated by hasturkun this won't work, try:
echo "Prepend" | cat - file | tee file
But this isn't the most efficient way of doing it any more...
Similar to the accepted answer, however if you are coming here because you want to prepend to the first line - rather than prepend an entirely new line - then use this command.
sed -i "1 s/^/string_replacement/" some_file
The -i flag will do a replacement within the file (rather than creating a new file).
Then the 1 will only do the replacement on line 1.
Finally, the s command is used which has the following syntax s/find/replacement/flags.
In our case we don't need any flags. The ^ is called a caret and it is used to represent the very start of a string.
Try this http://www.unix.com/shell-programming-scripting/42200-add-text-beginning-file.html
There is no direct operator or command AFAIK.You use echo, cat, and mv to get the effect.
{ date; brew update |tee /dev/tty; cat updates.txt; } >updates.txt.new
mv updates.txt.new updates.txt
I've no idea why you want to do this. It's pretty standard that logs like this have later entries appearing, well, later in the file.
I'm re-writing an ancient and pretty broken build and ran across a rule with something in it I've never seen before. It looks like this:
%_ui.cc:
${SOME_UTILITY} ${*}
sed '/\#include "${*}.h"/d' > tempstubs.cc ${*}_stubs.cc
/bin/csh -c 'if (-w ${*}_stubs.cc ) cp -f tempstubs.cc ${*}_stubs.cc'
-rm -f tempstubs.cc
The sed line is the one I'm referring to. I've never seen a redirection like that with two files after the >.
Nevermind, figured it out. The sed line could've been re-written as:
sed 'do whatever' ${*}_stubs.cc > tempstubs.cc
... and appears to be semantically identical.
I've got a makefile (developed for gmake on Linux) that I'm attempting to port to MacOS, but it seems like sed doesn't want to cooperate. What I do is use GCC to autogenerate dependency files, and then tweak them a bit using sed. The relevant portion of the makefile:
$(OBJ_DIR)/%.d: $(SRC_DIR)/%.cpp
$(CPPC) -MM -MD $< -o $#
sed -i 's|\(.*\)\.o:|$(OBJ_DIR)/\1.o $(OBJ_DIR)/\1.d $(TEST_OBJ_DIR)/\1_utest.o:|' $#
While this runs with no trouble under GNU/Linux, I get errors like the following when attempting to build on MacOS:
sed: 1: "test/obj/equipmentConta ...": undefined label 'est/obj/equipmentContainer_utest.d'
sed: 1: "test/obj/dice_utest.d": undefined label 'est/obj/dice_utest.d'
sed: 1: "test/obj/color-string_u ...": undefined label 'est/obj/color-string_utest.d'
It would seem like sed is chopping off a character, but I can't see the solution.
OS X sed handles the -i argument differently to the Linux version.
You can generate a command that might "work" for both by adding -e in this way:
# vv
sed -i -e 's|\(.*\)\.o:|$(OBJ_DIR)/\1.o $(OBJ_DIR)/\1.d $(TEST_OBJ_DIR)/\1_utest.o:|' $#
OS X sed -i interprets the next thing after the -i as a file extension for a backup copy of the in-place edit. (The Linux version only does this if there is no space between the -i and the extension.) Obviously a side affect of using this is that you will get a backup file with -e as an extension, which you may not want. Please refer to other answers to this question for more details, and cleaner approaches that can be used instead.
The behaviour you see is because OS X sed consumes the s||| as the extension (!) then interprets the next argument as a command - in this case it begins with t, which sed recognizes as a branch-to-label command expecting the target label as an argument - hence the error you see.
If you create a file test you can reproduce the error:
$ sed -i 's|x|y|' test
sed: 1: "test": undefined label 'est'
Actually, doing
sed -i -e "s/blah/blah/" files
doesn't do what you expect in MacOS either. Instead it creates backup files with -e extension.
The proper command for MacOS is
sed -i "" -e "s/blah/blah/" files
On Linux, remove the space between -i and "" (see related answer)
sed -i"" -e "s/blah/blah/" files
The currently accepted answer is flawed in two very important ways.
With BSD sed (the OSX version), the -e option is interpreted as
a file extension and therefore creates a backup file with a -e
extension.
Testing for the darwin kernel as suggested is not a reliable
approach to a cross platform solution since GNU or BSD sed could
be present on any number of systems.
A much more reliable test would be to simply test for the --version option which is only found in the GNU version of sed.
sed --version >/dev/null 2>&1
Once the correct version of sed is determined, we can then execute the command in its proper syntax.
GNU sed syntax for -i option:
sed -i -- "$#"
BSD sed syntax for -i option:
sed -i "" "$#"
Finally put it all together in a cross platform function to execute an in place edit sed commend:
sedi () {
sed --version >/dev/null 2>&1 && sed -i -- "$#" || sed -i "" "$#"
}
Example usage:
sedi 's/old/new/g' 'some_file.txt'
This solution has been tested on OSX, Ubuntu, Freebsd, Cygwin, CentOS, Red Hat Enterprise, & Msys.
martin clayton's helpful answer provides a good explanation of the problem[1], but his solution - as he states - has a potentially unwanted side effect.
Here are side-effect-free solutions:
Caveat: Solving the -i syntax problem alone, as below, may not be enough, because there are many other differences between GNU sed and BSD/macOS sed (for a comprehensive discussion, see this answer of mine).
Workaround with -i: Create a backup file temporarily, then clean it up:
With a non-empty suffix (backup-file filename extension) option-argument (a value that is not the empty string), you can use -i in a way that works with both BSD/macOS sed and GNU sed, by directly appending the suffix to the -i option.
This can be utilized to create a backup file temporarily that you can clean up right away:
sed -i.bak 's/foo/bar/' file && rm file.bak
Obviously, if you do want to keep the backup, simply omit the && rm file.bak part.
Workaround that is POSIX-compliant, using a temporary file and mv:
If only a single file is to be edited in-place, the -i option can be bypassed to avoid the incompatibility.
If you restrict your sed script and other options to POSIX-compliant features, the following is a fully portable solution (note that -i is not POSIX-compliant).
sed 's/foo/bar' file > /tmp/file.$$ && mv /tmp/file.$$ file
This command simply writes the modifications to a temporary file and, if the sed command succeeds (&&), replaces the original file with the temporary one.
If you do want to keep the original file as a backup, add another mv command that renames the original first.
Caveat: Fundamentally, this is what -i does too, except that it tries to preserve permissions and extended attributes (macOS) of the original file; however, if the original file is a symlink, both this solution and -i will replace the symlink with a regular file.
See the bottom half of this answer of mine for details on how -i works.
[1] For a more in-depth explanation, see this answer of mine.
This isn't quite an answer to the question, but one can get linux-equivalent behavior through
brew install gnu-sed
# Add to .bashrc / .zshrc
export PATH="/usr/local/opt/gnu-sed/libexec/gnubin:$PATH"
(previously there was a --with-default-names option to brew install gnu-sed but that has recently been removed)
I came across this issue as well and thought of the following solution:
darwin=false;
case "`uname`" in
Darwin*) darwin=true ;;
esac
if $darwin; then
sedi="/usr/bin/sed -i ''"
else
sedi="sed -i"
fi
$sedi 's/foo/bar/' /home/foobar/bar
Works for me ;-), YMMV
I work in a multi-OS team where ppl build on Windows, Linux and OS X. Some OS X users complained because they got another error - they had the GNU port of sed installed so I had to specify the full path.
I've corrected the solution posted by #thecarpy:
Here's a proper cross-platform solution for sed -i:
sedi() {
case $(uname) in
Darwin*) sedi=('-i' '') ;;
*) sedi='-i' ;;
esac
LC_ALL=C sed "${sedi[#]}" "$#"
}
I avoid using sed -i when writing scripts and i came up with simple solution:
printf '%s' "$(sed 's/foo/bar' file)" > file
much compatible and is POSIX-compliant. It is doing pretty much the same as sed -i, but this one does not create temp files, it directly redirect the changes to file.
As a noob idk what's the cons of doing this, the only matters is "It works"