I have a relatively complex directory structure in my build, from which I would like to create multiple ZIP files. The structure is like this:
+ project root
+ -- build.gradle
+ -- + foo
+ -- bar-1
| + -- bar-1.a.json
| + -- bar-1.a.xml
| + -- bar-1.b.json
| + -- bar-1.b.xml
|
+ -- bar-2
| + -- bar-2.a.json
| + -- bar-2.a.xml
| + -- bar-2.b.json
| + -- bar-2.b.xml
|
+ ...
+ -- bar-n
+ -- bar-n.a.json
+ -- bar-n.a.xml
+ -- bar-n.b.json
+ -- bar-n.b.xml
From this structure, I would like to create ZIP files for each bar-x directory, e.g. from foo/bar-1 directory, a ZIP file should be created in out/foo/bar-1.zip, from foo/bar-2 a ZIP file should be created in out/foo/bar-2.zip and so on
It is important, that new directories might appear later on, so I cannot hard-code the names, Gradle should list the directories in foo on each build.
Can someone give me a sample on how to achieve this?
Something like the following should do:
task zipAll
file('foo').eachDirMatch(~/bar-.*/) { barDir ->
def taskName = "zip${barDir.name.capitalize()}"
task "$taskName"(type: Zip) {
from barDir
destinationDir file('out/foo/')
baseName barDir.name
}
zipAll.dependsOn taskName
}
Related
The problem
Suppose that I have written a lengthy script
in some language "lang",
and now want to convert this single-file script
into a directory tree with a project consisting of many files. I want to insert some kind of separators and file-paths into this file, and process it in some way so that in the end I obtain:
a proper project directory layout (sth. like this),
build-definition file,
readme's,
separate subdirectories for main/src and test/src etc.
For example, given the following script (pseudocode):
// required dependencies, should be moved
// into the build definition build.foo
require "org.foo" % "foo-core" % "1.2.3"
require "org.bar" % "bar-gui" % "3.2.1"
// A longer comment that should be converted
// into a text file and moved into a 'notes'
// subdirectory
/*
#README
Another lengthy comment that should go into
a readme.md
*/
/** A class that should
* go to src/main/lang/proj/A.lang
*/
class A {
def a = "foo"
}
/** Another class
* that should go to src/main/lang/proj/B.lang
*/
class B {
def b = "bar"
}
/** Some tests,
* should end up in
* src/test/lang/proj/MyTest.lang
#Test def testFoo() {
assert(2 + 2 == 5)
}
and assuming that I can insert arbitrary separators, commands, escape-sequences and file paths into this file, I would like to obtain the following project:
project/
|-- build.txt
|-- notes
| `-- note_01.txt
|-- readme.md
`-- src
|-- main
| `-- lang
| `-- proj
| |-- A.lang
| `-- B.lang
`-- test
`-- lang
`-- proj
`-- MySpec.lang
Edit:
What follows is a less-sophisticated version of my own answer below
What I've tried
Here is one naive way to do it:
Convert the original script into a bash script by prepending #!/bin/bash
split the source code into HEREDOCS
insert package declarations where necessary
add bunch of mkdir -p and cd between the HEREDOC-pieces
cat the HEREDOC pieces into appropriately named files
test the script on empty directories until it works as expected
For the above script, it might look somehow like this:
#!/bin/bash
mkdir project
cd project
cat <<'EOF' > build.txt
// required dependencies, should be moved
// into the build definition build.foo
require "org.foo" % "foo-core" % "1.2.3"
require "org.bar" % "bar-gui" % "3.2.1"
EOF
mkdir notes
cd notes
cat <<'EOF' > note_01.txt
// A longer comment that should be converted
// into a text file and moved into a 'notes'
// subdirectory
EOF
cd ..
cat <<'EOF' > readme.md
/*
#README
Another lengthy comment that should go into
a readme.md
*/
EOF
mkdir -p src/main/lang/proj
cd src/main/lang/proj
cat <<'EOF' > A.lang
package proj
/** A class
* that should go to src/main/lang/proj/A.lang
*/
class A {
def a = "foo"
}
EOF
cat <<'EOF' > B.lang
package proj
/** Another class
* that should go to src/main/lang/proj/B.lang
*/
class B {
def b = "bar"
}
EOF
cd ../../..
mkdir -p test/lang/proj
cd test/lang/proj
cat <<'EOF' > MySpec.lang
package proj
/** Some tests,
* should end up in
* src/test/lang/proj/MyTest.lang
#Test def testFoo() {
// this should end up in test
assert(2 + 2 == 5)
}
EOF
cd ../../..
What's wrong with this approach
It does generate the correct tree, but this approach seems rather error-prone:
it's too easy to cd ../../.. to the wrong nesting level
too easy to mkdir with a wrong name, and then fail to cd into it.
There is no way to handle the entire tree construction as a single
transaction, that is, if something fails later in the script,
there is no simple way to clean up the mess generated before
the error occurred.
I certainly could try to make it a bit less brittle by defining
special functions that mkdir and cd in one go, and
then wrap invocations of those functions together with cats into
(mkdirAndCd d ; cat) etc.
But it just doesn't feel quite right. Isn't there a much simpler
way to do it? Could one somehow combine the standard bash/linux utilities
into a tiny & very restricted domain specific language for
generating directory trees with text files? Maybe some newer version of split where one could specify where to split and where to put the pieces?
Related questions:
mkdir and touch in single command
The reverse of tree - reconstruct file and directory structure from text file contents?
Other interesting proposals that don't seem to work:
Use tar. That would mean that one would have to convert the text file manually into a valid tar-archive. While a tar archive indeed is a single plain-text file, its internal format does not look like the most comfortable DSL for such a simple task. It was never intended to be used by humans directly in that way.
Similar argument holds for shar. Since shar uses the bash itself to extract the archive, my above proposal is, in principle, a manually generated shar-archive in a very uncommon format, therefore shar seems to share all the drawbacks with the above proposal. I'd rather prefer something more restricted, that allows to do fewer things, but provides more guarantees about the quality of the outcome.
Maybe I should emphasize again that I don't have a tree to begin with, so there is nothing to compress. I have only the single script file and a rough idea of what the tree should look like in the end.
Seems to me that you are trying to write a custom parser. Provided that all blocks mentioned by you are ended by double line endings, this could help you
#!/bin/bash
gawk 'BEGIN{RS="\n\n([/][*]|[/]{2,2})"}
{
if ($0 ~ /#README/){
system("echo -e \"\nThis is a Readme.md\n--------\n" $0 "\"")
}else if ($0 ~ /class /){
system("echo -e \"\nThis is a class\n---------\n/*" $0 "\"")
}else if ($0 ~ /require /){
system("echo -e \"\nthis is a conf\n-----------\n" $0 "\"")
}else if($0 ~ /[/]{2,2}.*\n[/]{2,2}/){
system("echo -e \"\nthis is a note\n-----------\n" $0 "\"")
}
}' your_script.lang
The key part is the record separator RS that splits block of code that start with '\n\n//' or '\n\n/*'.
Instead of echo -e you could write custom scripts for each type of block.
Please note that the record separator will not be present on $0 so you have to add the missing characters, as in the /class / example above.
The output of the above code is
this is a conf
-----------
// required dependencies, should be moved
// into the build definition build.foo
require org.foo % foo-core % 1.2.3
require org.bar % bar-gui % 3.2.1
this is a note
-----------
A longer comment that should be converted
// into a text file and moved into a 'notes'
// subdirectory
This is a Readme.md
--------
#README
Another lengthy comment that should go into
a readme.md
*/
This is a class
---------
/** A class that should
* go to src/main/lang/proj/A.lang
*/
class A {
def a = foo
}
This is a class
---------
/** Another class
* that should go to src/main/lang/proj/B.lang
*/
class B {
def b = bar
}
About your concerns:
it's too easy to cd ../../.. to the wrong nesting level
-> define a variable with root path and cd to it.
too easy to mkdir with a wrong name, and then fail to cd into it.
-> define variables with directory names and check if they already exists.
path1=src/main/lang/some
if [ -d $path1 ]; then
do_something
fi
There is no way to handle the entire tree construction as a single transaction ...
-> write to file paths of every NEW directory/file that you create and use it to revert if necessary.
(my own answer)
Consider the following definition of a tiny embedded domain specific language for defining directory trees with text files:
#!/bin/bash
enter() {
local subdir="$1"
if [ ! -d "$subdir" ]
then
mkdir -p "$subdir"
fi
pushd "$subdir" > /dev/null
}
leave() {
popd > /dev/null
}
save() {
local fileName="$1"
cut -d'|' -f2- > "$fileName"
}
The enter command creates a directory if necessary, and cds into this directory, it works with arbitrary relative paths. The save command saves the text content of a here-document to file. The leave command changes to previous directory.
When a file is saved, the margin (empty space followed by '|') is stripped from each line. This is to ensure that the indentation of the script does not interfere with the indentation of the written files.
If these definitions are sourced, then the tree-generation script can be written as follows:
#!/bin/bash
source explode.sh
mkdir project
cd project
enter "src"
enter "main/lang/proj"
save "A.lang" <<'____EOF'
|package proj
|
|/** A totally useful class
| * that should go to src/main/lang/proj/A.lang
| */
|class A {
| def a = "foo"
|}
____EOF
save "B.lang" <<'____EOF'
|package proj
|/** Another very useful class
| * that should go to src/main/lang/proj/B.lang
| */
|class B {
| def b = "bar"
|}
____EOF
leave
enter "test/lang/proj"
save "MyTest.lang" <<'____EOF'
|package proj
|
|/** A test that should end up in
| * src/test/lang/proj/MyTest.lang
|#Test def testFoo() {
| assert(2 + 2 == 5)
|}
____EOF
leave
leave
save "build.txt" <<'EOF'
|require "org.foo" % "foo-core" % "1.2.3"
|require "org.bar" % "bar-gui" % "3.2.1"
EOF
enter "notes"
save "note_01.txt" <<'__EOF'
|A longer comment that should be converted
|into a text file and moved into a 'notes'
|subdirectory. This is a very long comment
|about the purpose of the project. Blah
|blah blah.
__EOF
leave
save "README.md" <<'EOF'
|#README
|
|This is a readme file for my awesome project.
|It ends with this line. Bye.
EOF
When executed, the script generates the following directory tree:
project/
├── build.txt
├── notes
│ └── note_01.txt
├── README.md
└── src
├── main
│ └── lang
│ └── proj
│ ├── A.lang
│ └── B.lang
└── test
└── lang
└── proj
└── MyTest.lang
The bash-script mirrors the tree structure very closely, and it's impossible to mess up the cd ../../../../../.. commands. It still lacks various desirable properties though (not transactional, no dry-run capability).
Using the following test tree folder for example:
- test1
- folder2
- test1 # This is the file rubyzip will break on.
- test2
And copied this code from here:
path = File.expand_path path
archive = File.join(__dir__, File.basename(path)) + '.zip'
FileUtils.rm archive, force: true
Zip::File.open(archive, Zip::File::CREATE) do | zipfile |
Dir["#{path}/**/**"].reject{|f|f==archive}.each do | item |
basename = File.basename(item)
zipfile.add(basename, item)
end
end
It fails because there is two files having the same name even if their are not in the same directory (test1 in my example).
Is there something I am missing ?
Thanks to #simonoff (here), I shouldn't use the basename but the full relative path so Rubyzip could make the difference between test1 and folder2/test1.
Here is a code fixing it:
basename = File.basename path
dirname = File.dirname path
internal_path = path.sub %r[^#{__dir__}/], ''
archive = File.join(dirname, basename) + '.zip'
FileUtils.rm archive, force: true
Zip::File.open(archive, Zip::File::CREATE) do | zipfile |
Dir["#{internal_path}/**/**"].map{|e|e.sub %r[^#{internal_path}/],''}.reject{|f|f==archive}.each do | item |
zipfile.add(item, File.join(internal_path, item))
end
end
There is certainly a much cleaner way to do it.
So, I know that mv tends to work better for moving files around, so while in Stata I wanted to move all the files which contained a certain year in the name to their own directories. However, when I include /* in the ! terminal command, it comments out the rest of my work in the do file. Is there any way to work around this?
Here is the section in question:
forvalues i = 2010/2013 {
!mkdir ~/Documents/Thesis/Data/EIA_AMI/Test/`i'
!mv ~/Documents/Thesis/Data/EIA_AMI/Test/*`i'* ~/Documents/Thesis/Data/EIA_AMI/Test/`i'/
}
Try ASCII codes with the char() function if surrounding with quotes doesn't work.
A working example on my machine is:
clear all
set more off
forvalues i = 2010/2010 {
!mkdir ~/Desktop/test/`i'
!mv /home/roberto/Desktop/test/`=char(42)'`i'`=char(42)'.txt /home/roberto/Desktop/test/`i'/
}
with
The result is:
An example run:
. local i = 2010
. // initial state
. !ls ~/Desktop/test
test2010test.txt
. // move
. !mkdir ~/Desktop/test/`i'
. !mv ~/Desktop/test/`=char(42)'`i'`=char(42)'.txt ~/Desktop/test/`i'/
. // final state
. !ls ~/Desktop/test
2010
. !ls ~/Desktop/test/2010
test2010test.txt
You could probably find some luck storing the file path in a local in quotes and then evaluating that local in the mv call:
. do move.do
. ls
move.do
testDir/
testFile1.txt
testFile2.md
. local cmd "./*.txt testDir/"
. !mv `cmd'
. ls
move.do
testDir/
testFile2.md
. ls testDir/
testFile1.txt
What's interesting is that the /* gets interpreted as a comment if you don't go through a local. I'm sure people with better understanding of Stata can explain why, but I can't.
. !mv "./*.md testDir/"
mv: missing destination file operand after `./*.md testDir/'
Try `mv --help' for more information.
. ls
move.do
testDir/
testFile2.md
. ls testDir/
testFile1.txt
end of do-file
Another approach is to use filelist (from SSC) to create a dataset of files to manage and do all the file maintenance in Stata. The following example assumes that in Stata's current directory there a directory called "from" that contains a bunch of files with a year in the name.
. filelist, dir(from) norecur
Number of files found = 6
. list
+--------------------------------------+
| dirname filename fsize |
|--------------------------------------|
1. | from test2010test 3.txt 8,540 |
2. | from test2010test.txt 17 |
3. | from test2010test2.txt 9 |
4. | from test2011test 3.txt 5 |
5. | from test2011test.txt 3 |
|--------------------------------------|
6. | from test2011test2.txt 3 |
+--------------------------------------+
The code below will create a new directory called "years" in the current directory and subdirectories named after the years. By using only relative directories and only Stata commands, you make your code more portable. Of course, the following could be modified to use shell commands to process each file.
* filelist if from SSC
filelist, dir(from) norecur
* extract the year from the file name
gen year = regexs(1) if regexm(filename,"([0-9][0-9][0-9][0-9])")
assert !mi(year)
* automatically discover years
levelsof year, clean
local years `r(levels)'
* create destination directories if they don't alread exist
cap mkdir "years"
foreach y of local years {
cap mkdir "years/`y'"
}
* use Stata commands to copy the files one by one.
local obs = _N
forvalues i = 1/`obs' {
local fname = filename[`i']
local dname = dirname[`i']
local from = "`dname'/`fname'"
local year = year[`i']
local to = "years/`year'/`fname'"
copy "`from'" "`to'", replace
rm "`from'"
}
After running the above example, I get
. filelist , dir(years)
Number of files found = 6
. list
+-----------------------------------------+
| dirname filename fsize |
|-----------------------------------------|
1. | years/2010 test2010test 3.txt 8,540 |
2. | years/2010 test2010test.txt 17 |
3. | years/2010 test2010test2.txt 9 |
4. | years/2011 test2011test 3.txt 5 |
5. | years/2011 test2011test.txt 3 |
|-----------------------------------------|
6. | years/2011 test2011test2.txt 3 |
+-----------------------------------------+
I have a series of subfolders, each with images on them.
Structure looks like
/stuff/
IMG_012.JPG
IMG_013.JPG
IMG_014.JPG
/morestuff/
IMG_022.JPG
IMG_023.JPG
IMG_024.JPG
I would like to write a script on my mac terminal to loop through each folder and rename the images sequentially including the folder name. So, the above structure would look like:
/stuff/
stuff_1.JPG
stuff_2.JPG
stuff_3.JPG
/morestuff/
morestuff_1.JPG
morestuff_1.JPG
morestuff_1.JPG
I orignally tried creating a Automator workflow and using variables, but had difficulty assigning the folder name as the variable and getting the loop to work.
I'm hoping there is an elegant solution with the terminal.
Any ideas?
This should work nicely for you. Save it in your HOME directory as "RenameImages", then make it executable like this:
cd
chmod +x RenameImages
Then you can run it (-exec) it on every directory (-type d) like this:
find . -type d -exec ./RenameImages {} \;
Here is the script:
#!/bin/bash
# Ignore case, i.e. process *.JPG and *.jpg
shopt -s nocaseglob
shopt -s nullglob
# Go to where the user says
echo Processing directory $1
cd "$1" || exit 1
# Get last part of directory name
here=$(pwd)
dir=${here/*\//}
i=1
for image in *.JPG
do
echo mv "$image" "${dir}${i}.jpg"
((i++))
done
At the moment it does nothing except show you what it would do. If you like what it is doing, simply remove the word echo from the 3rd to last line and run it again.
I would just like to throw out another way to do this since you mentioned you tried using Automator. This file search and process software: http://www.softpedia.com/get/File-managers/JFileProcessor.shtml https://github.com/stant/jfileprocessor
Will let you search for files with glob or regex, in subfolders to X or all depth, by name, size, date. You can save to a List window or file. Then you can run a groovy (think java) script to do whatever you want to the list of files; zip or tar them, modify the list strings like sed, delete, move, copy files, grep or ls -l them, whatever. In your case you can modify an existing groovy example to do something like:
int numItems = defaultComboBoxModel.getSize();
System.out.println( "defaultComboBoxModel.getSize() num of items =" + numItems + "=" );
String str = "";
for( int i = 0; i < numItems; i++ )
{
str = defaultComboBoxModel.getElementAt( i ).toString();
System.out.println( "file of index =" + i + " str =" + str + "=" );
String cmd = "mv " + str + " " + i + ".jpg"; // or whatever
def list = cmd.execute().text // this stuff just captures output and write to a log file
list.eachLine{
outFile << it;
}
outFile << System.getProperty("line.separator") + "-------------------------------------" + System.getProperty("line.separator");
}
It will also let you massage your list like add to, delete from, subtract one list from another.
How do you use variables to rename files in Ruby?
File.rename("text1.txt", "text2.txt")
The above example is fine when using irb, but I writing a script where both var1 and var2 are unknown to me.
for example:
script_dir = File.expand_path File.dirname(__FILE__)
Dir.chdir(script_dir)
Dir.glob('Cancer1-1.pencast').each do |pencast|
pencast_title = File.basename(File.basename(pencast), '.*')
i = 1
audio_title = File.basename(`unzip -l #{pencast} | grep .aac | awk '{print $4;}' | awk 'NR=='#{i}''`)
audio_path = `unzip -l #{pencast} | grep .aac | awk '{print $4;}' | awk 'NR=='#{i}''`
audio_extension = File.extname(File.basename(audio_path))
new_name = "#{pencast_title}-#{i}#{audio_extension}"
File.rename(audio_title, new_name)
does not work...
but if i use puts var1 I see the file name I want.
The error I get is:
prog_test.rb:12:in `rename': No such file or directory - audio-0.aac (Errno::ENOENT)
or Cancer1-1-1.aac
from prog_test.rb:12
from prog_test.rb:5:in `each'
from prog_test.rb:5
but the file audio-0.aac is there... I'm looking at it.
I am certain I have located the problem:
it seems to be adding a variable to another variable. This is a simplified example that produces the same output:
audio_title = "audio-0.aac"
fullPath = File::SEPARATOR + "Users" + File::SEPARATOR + "name" + File::SEPARATOR + "Desktop" + File::SEPARATOR + audio_title
newname = File::SEPARATOR + "Users" + File::SEPARATOR + "name" + File::SEPARATOR + "Desktop" + File::SEPARATOR + "audio1.aac"
puts fullPath
puts newname
File.rename(fullPath, newname)
OUTPUT :
/Users/name/Desktop/audio-0.aac
/Users/name/Desktop/audio1.aac
prog_test.rb:22:in `rename': No such file or directory - /Users/name/Desktop/audio-0.aac or /Users/name/Desktop/audio1.aac (Errno::ENOENT)
from prog_test.rb:22
You should be passing the full file path to File.rename, not just the basename
I am not sure what is going on in your example inside File.basename() , but imagine the following:
fullPath = "C:" + File::SEPARATOR + "Folder" + File::SEPARATOR + "File.txt" # C:\Folder\File.txt
basename = File.basename(fullPath) # File
newFileName = "File.bak"
File.rename(basename, newFileName)
# How can Ruby possibly know which directory to find the above file in, or where to put it? - It will just look in the current working directory
So instead, you need to pass the full path to File.rename, like so:
fullPath = "C:" + File::SEPARATOR + "Folder" + File::SEPARATOR + "File.txt" # C:\Folder\File.txt
directory = File.dirname(fullPath) # C:\Folder
newFileName = "File.bak"
File.rename(fullPath, directory + File::SEPARATOR + newFileName)