I got a folder with the following zip files :
13162.zip 14864.zip 19573.zip 20198.zip
In console, when i run :
cd my_folder; echo `ls *{.zip,.ZIP}`
I got the following output (which is perfect) :
ls: cannot access *.ZIP: No such file or directory
13162.zip 14864.zip 19573.zip 20198.zip
Now when in ruby i try the same :
cmd= "cd my_folder; echo `ls {*.zip,*.ZIP}`";
puts `#{cmd}`
It only display :
ls: cannot access {*.zip,*.ZIP}: No such file or directory
=> nil
I try this solution :
Getting output of system() calls in Ruby
But it seem not work in my case.
How can i get the same output in ruby and in shell ?
Ruby Only
You can use Dir.glob with File::FNM_CASEFOLD for case-insensitive search :
Dir.chdir 'my_folder' do
Dir.glob('*.zip', File::FNM_CASEFOLD).each do |zip_file|
puts zip_file
end
end
#=>
# 19573.zip
# 13162.zip
# 14864.zip
# 20198.zip
# 12345.zIp
Ruby + bash
You can use find for case-insensitive search :
paths = `find my_folder -maxdepth 1 -iname '*.zip'`.split
#=> ["my_folder/19573.zip", "my_folder/13162.zip", "my_folder/14864.zip", "my_folder/20198.zip", "my_folder/12345.zIp"]
-printf '%P' can also be used to only display the filenames :
files = `find my_folder -maxdepth 1 -iname '*.zip' -printf '%P\n'`.split
#=> ["19573.zip", "13162.zip", "14864.zip", "20198.zip", "12345.zIp"]
I think this should work directly on the terminal:
echo 'system("ls *ZIP,*zip")' | ruby
or create a ruby file with the following contents
system("cd my_folder; ls {*.zip,*.ZIP}")
and then execute it. Once you write ls, you don't need echo!
Related
I have a FTP folder receiving files from a remote camera. The camera stores the video file name always as ./rec_YYYY-MM-DD_HH-MM.mkv. The video files are stored all in the same folder, the root folder from the FTP server.
I need to move these files to another folder, with this new scheme:
Remove rec_ from the file name.
Change date format to DD-MM-YY.
Remove date from the file name and make it a folder instead, where that same file and all the others in the same date will be stored in.
Final file path would be: ./DD-MM-YYYY/HH-MM.mkv.
The process would continue to all the files, putting them in the folder corresponding to the day it was created.
Summing up: ./rec_YYYY-MM-DD_HH-MM.mkv >> ./DD-MM-YYYY/HH-MM.mkv. The same should apply to all files that are in the same folder.
As I can't make it happen directly from the camera, this needs to be done with Bash on the server that is receiving the files.
So far, what I got is script, which would get the file's creation date and use it to make a folder, and then get creation time to move the file with the new name, based on it's creation time.:
for f in *.mp4
do
mkdir "$f" "$(date -r "$f" +"%d-%m-%Y")"
mv -n "$f" "$(date -r "$f" +"%d-%m-%Y/%H-%M-%S").mp4"
done
I'm getting this output (with testfile 1.mp4):
It creates the folder based on the file's creation date;
it renames the file to it's creation time;
Then, it returns mkdir: cannot create directory ‘1.mp4’: File exists
If two or more files, only one gets renamed and moved as described. The others stay the same and terminal returns:
mkdir: cannot create directory ‘1.mp4’: File exists
mkdir: cannot create directory ‘2.mp4’: File exists
mkdir: cannot create directory ‘12-12-2018’: File exists
Could someone help me out? Better suggestions? Thanks!
Honestly I would just use Perl or Python for this. Here's how to embed either in a shell script.
Here's a perl script that doesn't use any libraries, even ones that ship with Perl (so it'll work without extra packages on distributions like CentOS that don't ship with the entire Perl library). The perl script launches one new process per file in order to perform the copy.
perl -e '
while (<"*.m{p4,kv}">) {
my $path = $_;
my ($prefix, $year, $month, $day, $hour, $minute, $ext) =
split /[.-_]/, $path;
my $sec = q[00];
die "unexpected prefix ($prefix) in $path"
unless $prefix eq q[rec];
die "unexpected extension ($ext) in $path"
unless $ext eq q[mp4] or $ext eq q[mkv];
my $dir = "$day-$month-$year";
my $name = "$hour-$min-$sec" . q[.] . $ext;
my $destpath = $dir . q[/] . $name;
die "$dir . $name is unexpectedly a directory" if (-d $dir);
system("cp", "--", $path, $destpath);
}
'
Here's a Python example, it's compatible with either Python 2 or Python 3 but does use the standard library. The Python script does not spawn any additional processes.
python3 -c '
import os.path as path
import re
from glob import iglob
from itertools import chain
from os import mkdir
from shutil import copyfile
for p in chain(iglob("*.mp4"), iglob("*.mkv")):
fields = re.split("[-]|[._]", p)
prefix = fields[0]
year = fields[1]
month = fields[2]
day = fields[3]
hour = fields[4]
minute = fields[5]
ext = fields[6]
sec = "00"
assert prefix == "rec"
assert ext in ["mp4", "mkv"]
directory = "".join([day, "-", month, "-", year])
name = "".join([hour, "-", minute, "-", sec, ".", ext])
destpath = "".join([directory, "/", name])
assert not path.isdir(destpath)
try:
mkdir(directory)
except FileExistsError:
pass
copyfile(src=p, dst=destpath)
'
Finally, here's a bash solution. It splits paths using -, ., and _ and then extracts various subfields by indexing into $# inside a function. The indexing trick is portable, although regex substitution on variables is a bash extension.
#!/bin/bash
# $1 $2 $3 $4 $5 $6 $7 $8
# path rec YY MM DD HH MM ext
process_file() {
mkdir "$5-$4-$3" &> /dev/null
cp -- "$1" "$5-$4-$3"/"$6-$7-00.$8"
}
for path in *.m{p4,kv}; do
[ -e "$path" ] || continue
# NOTE: two slashes are needed in the substitution to replace everything
# read -a ARRAYVAR <<< ... reads the words of a string into an array
IFS=' ' read -a f <<< "${path//[-_.]/ }"
process_file "$path" "${f[#]}"
done
If you cd /to/some/directory/containing_your_files then you could use the following script
#!/usr/bin/env bash
for f in rec_????-??-??_??-??.m{p4,kv} ; do
dir=${f:4:10} # skip 4 chars ('rec_') take 10 chars ('YYYY_MM_DD')
fnm=${f:15} # skip 15 chars, take the remainder
test -d "$dir" || mkdir "$dir"
mv "$f" "$dir"/"$fnm"
done
note ① that I have not exchanged the years and the days, if you absolutely need to do the swap you can extract the year like this, year=${dir::4} etc and ② that this method of parameter substitution is a Bash-ism, e.g., it doesn't work in dash.
your problem is: mkdir creates folder but you are giving filename for folder creation.
if you want to use fileName for folder creation then use it without extension.
the thing is you are trying to create folder with the already existing fileName
Inside a folder I have a file structure, the files have all a name whose format
name-randomstring.extension
eg
./dir1/aaa-5h34jk5hk.js
./dir2/bbb-5yh45uh9h.css
./dir3/ccc-uiero6tio.js
./dir3/ddd-7y8fygfre.css
. . .
with a bash script I would like to rename them recursively; so to eliminate the -randomstring from every file
./dir1/aaa.js
./dir2/bbb.css
./dir3/ccc.js
./dir3/ddd.css
. . .
You can do this with a native bash loop:
shopt -s globstar # enable ** for recursive expansion
for file in **/*{j,cs}; do
# remove everything up to the last . to get extension
ext=${file##*.}
# remove everything after the last - and concat with extension
new_name=${file%-*}$ext
# -- prevents weird filenames being interpreted as options
mv -- "$file" "$new_name"
done
shopt -u globstar # disable ** if you don't want it anymore e.g. in a script
if you do not mind using Perl then you can do anything you like.
for example for renaming all files with this structure:
dir > find .
.
./00_file.txt
./01_file.txt
./02_file.txt
./dir1
./dir1/00_file.txt
./dir1/01_file.txt
./dir1/02_file.txt
./dir2
./dir2/00_file.txt
./dir2/01_file.txt
./dir2/02_file.txt
./dir3
./dir3/00_file.txt
./dir3/01_file.txt
./dir3/02_file.txt
./find.log
then save your list in a file like: find.log now you can use Perl
dir > perl -lne '-f && ($old=$_) && s/file/ABCD/g && print "$old => $_"' find.log
./00_file.txt => ./00_ABCD.txt
./01_file.txt => ./01_ABCD.txt
./02_file.txt => ./02_ABCD.txt
./dir1/00_file.txt => ./dir1/00_ABCD.txt
./dir1/01_file.txt => ./dir1/01_ABCD.txt
./dir1/02_file.txt => ./dir1/02_ABCD.txt
./dir2/00_file.txt => ./dir2/00_ABCD.txt
./dir2/01_file.txt => ./dir2/01_ABCD.txt
./dir2/02_file.txt => ./dir2/02_ABCD.txt
./dir3/00_file.txt => ./dir3/00_ABCD.txt
./dir3/01_file.txt => ./dir3/01_ABCD.txt
./dir3/02_file.txt => ./dir3/02_ABCD.txt
how it works?
read the file
let only file; not directory or other stuffs with: -f
then save the old name: ($old=$_)
then doing substitution with: s///g operator
finally rename the old file to new one that is $_
NOTE I used print and you should use rename like this:
dir > perl -lne '-f && ($old=$_) && s/file/ABCD/g && rename $old, $_' find.log
dir > find .
.
./00_ABCD.txt
./01_ABCD.txt
./02_ABCD.txt
./dir1
./dir1/00_ABCD.txt
./dir1/01_ABCD.txt
./dir1/02_ABCD.txt
./dir2
./dir2/00_ABCD.txt
./dir2/01_ABCD.txt
./dir2/02_ABCD.txt
./dir3
./dir3/00_ABCD.txt
./dir3/01_ABCD.txt
./dir3/02_ABCD.txt
./find.log
NOTE Since the find returns list of all files and sub-directories, with this technique Perl renames all the files! Thus first use print then rename. And your pattern can be: -.*(?=\.). In fact:
s/-.*(?=\.)//g
I'm using cURL API to submit files on a API service and It returns back with something called taks_id for the files submitted.
#submitter.sh
creation_date=$(date +"%m_%d_%Y")
task_id_file="/home/results/$creation_date"_task_ids".csv"
for i in $(find $1 -type f);do
task_id="$(curl -s -F file=#$i http://X.X.X.X:XXXX/api/tiscale/v1/upload)"
final_task_id=$(echo $task_id | grep -o 'http\?://[^"]\+')
echo "$final_task_id" >> $task_id_file
done
#Result ( 10_13_2016_task_ids.csv )
http://192.168.122.24:8080/api/tiscale/v1/task/17
http://192.168.122.24:8080/api/tiscale/v1/task/18
http://192.168.122.24:8080/api/tiscale/v1/task/19
Run Method :
$./submitter.sh /home/files/pdf/
Now, Using[find $1 -type f] logic will get the full path with file name mentioned.
#find /home/files/pdf -type f
/home/files/pdf/country.pdf
/home/files/pdf/summary.pdf
/home/files/pdf/age.pdf
How can i add the file names along with cURL API response result. For example , When submitting "/home/files/country.pdf", The API might give the task_id with http://192.168.122.24:8080/api/tiscale/v1/task/17'`.
Expecting Result :
country.pdf,http://192.168.122.24:8080/api/tiscale/v1/task/17
summary.pdf,http://192.168.122.24:8080/api/tiscale/v1/task/18
age.pdf,http://192.168.122.24:8080/api/tiscale/v1/task/19
I'm beginner in Bash, Any suggestions on how to achieve this ?
I have a folder structure like follows.
-FOO
-BAG
Rose.TXT
-BAR
JaCk.txt
I need the following output.
-foo
-bag
rose.txt
-bar
jack.txt
I realize you want ruby code, but I present to you a one liner to run in your shell:
for i in `find * -depth`; do (mv $i `echo $i|tr [:upper:] [:lower:]`); done
as found here: http://ubuntuforums.org/showthread.php?t=244738
Run it once, and it should do the trick.
Update
Ruby Code:
Dir.glob("./**/*").each do |file|
File.rename(file, file.downcase) #or upcase if you want to convert to uppercase
end
Dir["**/*"].each {|f| File.rename(f, f.downcase)}
The accepted answer does not work: when it tries to convert a directory first and then a file in that directory.
Here is the code that does work:
Dir.glob("./**/*").sort{|x| x.size}.each do |name|
x = name.split('/')
newname = (x[0..-2] + [x[-1].downcase]).join('/')
File.rename(name, newname)
end
(it sorts the list by length, so the direcotry will be converted after the file in it)
Recursive directory listing :
http://www.mustap.com/rubyzone_post_162_recursive-directory-listing
Uppercase/lowercase conversion :
http://www.programmingforums.org/thread5455.html
Enjoy :)
If you want to rename your files recursively, you can use **/*
folder = "/home/prince"
Dir["#{folder}/**/*"].each {|file| File.rename(file, file.downcase)}
If you just want the file array output in lowercase
Dir["#{folder}/**/*"].map(&:downcase)
Just a bit of Find.find is all you need:
require 'find'
Find.find(directory) do |path|
if(path != '.' && path != '..')
File.rename(path, path.downcase)
end
end
I'm trying to write some ruby that would recursively search a given directory for all empty child directories and remove them.
Thoughts?
Note: I'd like a script version if possible. This is both a practical need and a something to help me learn.
In ruby:
Dir['**/*'] \
.select { |d| File.directory? d } \
.select { |d| (Dir.entries(d) - %w[ . .. ]).empty? } \
.each { |d| Dir.rmdir d }
Looking at the examples from kch, dB. and Vishnu above, I've put together a one-liner that I think is a more elegant solution:
Dir['**/'].reverse_each { |d| Dir.rmdir d if Dir.entries(d).size == 2 }
I use '**/' instead of '/**/*' for the glob, which only returns directories, so I don't have to test whether it's a directory later. I'm using reverse_each instead of sort.reverse.each as it's shorter and supposedly more efficient, according to this post. I prefer Dir.entries(d).size == 2 to (Dir.entries(d) - %w[ . .. ]).empty? because it's a little easier to read and understand, although (Dir.entries(d) - %w[ . .. ]).empty? would probably work better if you had to run your script on Windows.
I've tested this quite a bit on Mac OS X and it works well, even with recursive empty directories.
You have to delete in reverse order, otherwise if you have an empty directory foo with a subdirectory bar you will delete bar, but not foo.
Dir.glob(dir + "/**/*").select { |d|
File.directory?(d)
}.reverse_each { |d|
if ((Dir.entries(d) - %w[ . .. ]).empty?)
Dir.rmdir(d)
end
}
Why not just use shell?
find . -type d -empty -exec rmdir '{}' \;
Does exactly what you want.
Dir['/Users/path/Movies/incompleteAnime/foom/**/*']. \
select { |d| File.directory? d }. \
sort.reverse. \
each {|d| Dir.rmdir(d) if Dir.entries(d).size == 2}
just like the first example, but the first example doesn't seem to handle the recursive bit. The sort and reverse ensures we deal with the most nested directories first.
I suppose sort.reverse could be written as sort {|a,b| b <=> a} for efficiency
I've tested this script on OS X, but if you are on Windows, you'll need to make changes.
You can find the files in a directory, including hidden files, with Dir#entries.
This code will delete directories which become empty once you've deleted any sub-directories.
def entries(dir)
Dir.entries(dir) - [".", ".."]
end
def recursively_delete_empty(dir)
subdirs = entries(dir).map { |f| File.join(dir, f) }.select { |f| File.directory? f }
subdirs.each do |subdir|
recursively_delete_empty subdir
end
if entries(dir).empty?
puts "deleting #{dir}"
Dir.rmdir dir
end
end
Dir.glob('**/*').each do |dir|
begin
Dir.rmdir dir if File.directory?(dir)
# rescue # this can be dangereous unless used cautiously
rescue Errno::ENOTEMPTY
end
end
module MyExtensions
module FileUtils
# Gracefully delete dirs that are empty (or contain empty children).
def rmdir_empty(*dirs)
dirs.each do |dir|
begin
ndel = Dir.glob("#{dir}/**/", File::FNM_DOTMATCH).count do |d|
begin; Dir.rmdir d; rescue SystemCallError; end
end
end while ndel > 0
end
end
end
module ::FileUtils
extend FileUtils
end
end
For a pure shell solution, I found this very useful
find "$dir" -depth -type d |
while read sub; do
[ "`cd "$sub"; echo .* * ?`" = ". .. * ?" ] || continue
echo rmdir "$sub"
#rmdir "$sub"
done
But if you have gnu-find installed (not universal yet)...
find . -depth -type d -empty -printf "rmdir %p\n"
this uses find with xargs...
find . -depth -type d -print0 |
xargs -0n1 sh -c '[ "`cd "$0"; echo .* * ?`" = ". .. * ?" ] &&
echo "rmdir $0";'