Using wget, a webpage is downloaded as a .txt file. This file saved is named using part of the url of the webpage, eg. wget http://www.example.com/page/12345/ -O 12345.txt, for convenience.
I am running commands from a shell script .sh file, as it can execute multiple commands, one line at time, eg.
After a file is downloaded, I use sed to parse for text / characters I want to keep. Part of the text I want includes blah blah Product ID a5678.
What I want is to use sed to find a5678 and use this to rename the file 12345.txt to a5678.txt.
# script.sh
wget http://www.example.com/page/12345/ -O 12345.txt
sed -i '' 's/pattern/replace/g' 12345.txt
sed command to find a5678 # in line blah blah Product ID a5678
some more sed commands
mv 12345.txt a5678.txt (or use a variable $var.txt)?
How do I do this?
I may also want to use this same ID a5678 and create a folder with the same name a5678. Hence the .txt file is inside the folder like so /a5678/a5678.txt.
mkdir a5678 (or mkdir $var)? && cd a5678
I've searched for answers for half a day, but can't find any. The closest I found is
Find instance of word in files and change it to the filename but it is the exact opposite of what I want. I've also thought about using variables eg. https://askubuntu.com/questions/76808/how-do-i-use-variables-in-a-sed-command but I don't know how to save the found characters as a variable.
Very much look forward to some help! Thank you! I am on a Mac running Sierra.
Trying to minimize, so fit this into your logic.
in=12345.txt
out=$( grep ' Product ID ' $in | sed 's/.* Product ID \([^ ]*\) .*/\1/' )
mkdir -p $out
mv $in $out/$out.txt
Thank you all! With your inspiration, I solved my problem by (without using grep):
in=12345
out=$(sed -n '/pattern/ s/.*ID *//p' $in.txt)
mv $in.txt $out.txt
cd ..
mv $in $out
I need to create a file that lists all the files in a folder into a text file, along with a comma and the number 15 after. For example
My folder has video.mp4, video2.mp4, picture1.jpg, picture2.jpg, picture3.png
I need the text file to read as follows:
video.mp4,15
video2.mp4,15
picture1.jpg,15
picture2.jpg,15
picture3.png,15
No spaces, just filename.ext,15 on each line. I am using a raspberry pi. I am aware that the command ls > filename.txt would put all the file names into a folder, but how would I get a ,15 after every line?
Thanks
bash one-liner:
for f in *; do echo "$f,15" >> filename.txt; done
To avoid opening the output file on each iteration you may redirect the entire output with > filename.txt:
for f in *; do echo "$f,15"; done > filename.txt
$ printf '%s,15\n' *
picture1.jpg,15
picture2.jpg,15
picture3.png,15
video.mp4,15
video2.mp4,15
This will work if those are the only files in the directory. The format specifier %s,15\n will be applied to each of printf's arguments (the names in the current directory) and they will be outputted with ,15 appended (and a newline).
If there are other files, then the following would work too, regardless of whether there are files called like this or not:
$ printf '%s,15\n' video.mp4 video2.mp4 picture1.jpg picture2.jpg "whatever this is"
video.mp4,15
video2.mp4,15
picture1.jpg,15
picture2.jpg,15
whatever this is,15
Or, on all MP4, PNG and JPEG files:
$ printf '%s,15\n' *.mp4 *.jpg *.png
video.mp4,15
video2.mp4,15
picture1.jpg,15
picture2.jpg,15
picture3.png,15
Then redirect this to a file with printf ...as above... >output.txt.
If you're using Bash, then this will not make use of any external utility, as printf is built into the shell.
You need to do something like this:
#!/bin/bash
for i in $(ls folder_name); do
echo $i",15" >> filename.txt;
done
It's possible to do this in one line, however, if you want to create a script, consider code readability in the long run.
Edit 1: better solution
As #CristianRamon-Cortes suggested in the comments below, you should not rely on the output of ls because of the problems explained in this discussion: why not parse ls. As such, here's how you should write the script instead:
#!/bin/bash
cd folder_name
for i in *; do
echo $i",15" >> filename.txt;
done
You can skip the part cd folder_name if you are already in the folder.
Edit 2: Enhanced solution:
As suggested by #kusalananda, you'd better do the redirection after done to avoid opening the file in each iteration of the for loop, so the script will look like this:
#!/bin/bash
cd folder_name
for i in *; do
echo $i",15";
done > filename.txt
Just 1 command line using 2 msr commands recusively (-r) search specific files:
msr -rp your-dir1,dir2,dirN -l -f "\.(mp4|jpg|png)$" -PAC | msr -t .+ -o '$0,15' -PIC > save-file.txt
If you want to sort by time, add --wt to first command like: msr --wt -l -rp your-dirs
Sort by size? Add --sz but only the prior one is effective if use both --sz and --wt.
If you want to exclude some directory, add like: --nd "^(test|garbage)$"
remove tail \r\n in save-file.txt : msr -p save-file.txt -S -t "\s+$" -o "" -R
See msr.exe / msr.gcc48 etc in my open project https://github.com/qualiu/msr tools directory.
A solution without a loop:
ls | xargs -i echo {},15 > filename.txt
Let's say I have a file abc.pdf
I want to rename it to Endesa + the output of this command:
pdftotext -raw abc.pdf - | grep 'Periodo de consumo' | sed -E 's_.*: ([0-9]{2})/([0-9]{2})/([0-9]{4}) a ([0-9]{2})/([0-9]{2})/([0-9]{4})_\3-\2-\1 \6-\5-\4_g'
I.e., Endesa 2016-07-12 2016-08-09.pdf
Can I do this with a one-liner, without having to type the name of the file twice?
I tried
mv abd.pdf < "Endesa "$(pdftotext....)
but that doesn't work.
Just use execute the command with $() without any need to use process substitution. This will be executed in a way that the output of the command will be placed just after "Endesa":
mv abc.pdf "Endesa""$(your command)"
For example I just created a file Endesa1471530429.pdf in my computer by running touch "Endesa"$(date "+%s")".pdf".
Since the parameter of pdftotext is the file itself you can save some time by storing the value in a variable and using it twice:
mv "$your_file" "Endesa""$(pdftotext -raw "$your_file" - | grep ...)"
Simple as:
mv abd.pdf "Endesa $(pdftotext....)"
try like this; using a variable
NEWNAME=$(pdftotext....) ; mv abc.pdf "Endesa $NEWNAME"
I'm looking for a programmatic way to add a file's date to the filename. I'm on a Mac, Yosemite (10.10).
Using Bash, I have put a fair amount of effort into this, but just too new to get there so far. Here's what I have so far:
#!/bin/bash
#!/bin/bash
(IFS='
'
for x in `ls -l | awk '{print$9" "$7"-"$6"-"$9}'`
do
currentfilename=$(expr "$x" : '\($substring\)')
filenamewithdate=$(expr "$x" : '.*\($substring\)')
echo "$currentfilename"
echo "$filenamewithdate"
done)
The idea here is to capture detailed ls output, use awk to capture the strings for the columns with the filename ($9), and also date fields ($7 and $6), then loop that output to capture the previous filename and new filename with the date to mv the file from old filename to new. The awk statement adds a space to separate current filename from new. The echo statement is there now to test if I am able to parse the awk ouput. I just don't know what to add for $substring to get the parts of the string that are needed.
I have much more to learn about Bash scripting, so I hope you'll bear with me as I learn. Any guidance?
Thanks.
Looking at the stat man page, you'd want:
for file in *; do
filedate=$(stat -t '%Y-%m-%dT%H:%M:%S' -f '%m' "$file")
newfile="$file-$filedate"
echo "current: $file -> new: $newfile"
done
Adjust your preferred datetime format to your taste.
You could save a line with
for file in *; do
newfile=$(stat -t '%Y-%m-%dT%H:%M:%S' -f '%N-%m' "$file")
I am trying to write a bash script that goes through a file line by line (ignoring the header), extracts a file name from the beginning of each line, and then finds a file by this name in one directory and moves it to another directory. I will be processing hundreds of these files in a loop and moving over a million individual files. A sample of the file is:
ImageFileName Left_Edge_Longitude Right_Edge_Longitude Top_Edge_Latitude Bottom_Edge_Latitude
21088_82092.jpg: -122.08007812500000 -122.07733154296875 41.33763821961143 41.33557596965434
21088_82093.jpg: -122.08007812500000 -122.07733154296875 41.33970040427444 41.33763821961143
21088_82094.jpg: -122.08007812500000 -122.07733154296875 41.34176252364274 41.33970040427444
I would like to ignore the first line and then grab 21088_82092.jpg as a variable. File names may not always be the same length, but they will always have the format digits_digits.jpg
Any help for an efficient approach is much appreciated.
This should get you started:
$ tail -n +2 input | cut -f 1 -d: | while read file; do test -f $dir/$file && mv -v $dir/$file $destination; done
You can construct a script that will do something like this, then simply run the script. The following command will give you a script which will copy the files from one place to another, but you can make the script generation more complex simply by changing the awk output:
pax:~$ cat qq.in
ImageFileName Left_Edge_Longitude Right_Edge_Longitude
21088_82092.jpg: -122.08007812500000 -122.07733154296875
21088_82093.jpg: -122.08007812500000 -122.07733154296875
21088_82094.jpg: -122.08007812500000 -122.07733154296875
pax:~$ awk -F: '/^[0-9]+_[0-9]+.jpg:/ {
printf "cp /srcdir/%s /dstdir\n",$1
} {}' qq.in
cp /srcdir/21088_82092.jpg /dstdir
cp /srcdir/21088_82093.jpg /dstdir
cp /srcdir/21088_82094.jpg /dstdir
You capture the output of that script (the last three lines) to another file then that file is your script for doing the actual copies.