I have a folder on my desktop called "images".
Inside are a bunch of sub-folders called various things "Flower_Shape", "Smile_Shape", "Dog_Shape", etc. and each folder contains 3 images "Variant1.PNG", "Variant2.PNG", and "Variant3.PNG" and then some other file formats that are called various things but they do not end in .png, they end in things like ".psd"...
How can I take all of the contents of these hundreds of folders, and move them all into 1 folder while changing the names from "Variant1.png" to whatever the name of the folder it was in is called (for example the "Variant1.png" inside the folder "Flower_Shape" would be re-named "Flower_Shape-Variant1.png") and then delete every file that doesn't end in ".png".
My first main question is what language would I use to do something like this? Would it be PERL?
After that is established, does anyone have any tips for how to go about doing this... I'm assuming just a for loop with some if statements should be all it really takes but I know nothing of perl (or any other languages that deal with changing files on my own computer for that matter)
Thanks!
Mac OS X comes with a whole assortment of scripting languages. It comes with Perl, Python, PHP, Ruby, and BASH.
Mac OS X also comes with Automator.
What should you choose?
If you're really, really interested in learning to program, I would suggest you start with Bash. BASH is what is called the command shell. If you type a command in at a prompt, it will be executed. The Unix shell is a powerful tool, and the BASH shell includes a lot of control structures that can be used. In BASH, you could use the find command and the mv command and write a loop to do what you want.
.
Shell scripts are limited in their power. They aren't suppose to be full service languages and make no apologies for it. If you need more than a few dozen lines or more complex control, you should switch to a scripting language.
I am a Perl programmer, but I would recommend, if you're really interested in learning how to program, to learn Python. Python is probably the most popular language and its use is certainly growing.
Perl is a nice language because much of its syntax structure comes from shell scripting languages. Despite what you've heard, Perl is still very much a popular language. I've found that people pick up Perl faster than Python because of it's looser syntax rules and the fact it doesn't have to be object oriented. That is also a major problem with Perl. Most Perl programmers never get past the basic hacking stage and Perl has a reputation of being a difficult language to maintain because of the sheer number of awful Perl scripts out there.
Ruby hasn't been as popular as Perl or Python, but it has its fans. The big power of Ruby is something called Ruby on Rails. Rails is a programming framework that allows you to quickly build web-based applications. You first should learn Ruby, then learn the Rails component.
PHP use to be the glue that held the web together. I believe its popularity has been dropping in recent years as newer web development platforms have come up. PHP is the mob rule of programming. Unlike all of the other languages, it doesn't have a single champion shaping the language and has developed a lot of detritus. It is sloppy and can be hard to maintain.
.
Still, it is the basis of a lot of web based forums and content management systems like Joomla. After all, PHP was designed to live inside of webpages.
If you don't want to learn script programming, you should try Automator. It comes with a Graphical User Interface and can allow you to build programs to do all sorts of tasks. People who have gotten into Automator have done some amazing stuff with it.
Whatever you choose, you can find some tutorials and information on the web, but if you really want to learn, you should get some good manuals. O'Reilly and Associates has a long storied history with computing and the Internet. Tim O'Reilly (the brains behind the company) has been producing computer manuals since 1978. Almost all of my books are O'Reilly books. Manning is another one, and there are dozens more. Go to a real bookstore before Amazon drives them all out of business and peruse their shelves for something you like.
You have to combine File::Find with File::Copy to reach those requirements
If you use perl, you should look into Path::Class and File:::Copy. That'll handle your path parsing. You'd want to iterate over the filesystem (Path::Class::Dir->next will be your friend here), build a destination file with something like:
my $curr_file = file('whatever_path...');
my $dest_dir = dir('some other path...');
my $dest_file = file($dest_dir, $curr_file->dir->basename . '-' . $curr_file->basename);
then use File::Copy to move:
move($curr_file, $dest_file);
Start by looking up those methods/subroutines and their usage; then you just need to code up the iteration.
I was able to accomplish this using this code (and changing the value inside the "glob" function to different levels of /// manually)
#!/usr/bin/perl
foreach my $files (glob ('*/*/*')) {
use strict;
use File::Basename;
use Cwd 'abs_path';
my $f = "$files";
my $d = basename(dirname(abs_path($f)));
my($file, $dir, $ext) = fileparse($files);
my $initialName = "$dir$file";
my $finalName = "$dir$d$file";
print("$initialName --> $finalName \n");
rename($initialName, $finalName);
}
Then it was as simple as moving the files with a move function into a new folder, like so:
#!/usr/bin/perl
foreach my $files (glob ('*/*/*')) {
use strict;
use File::Basename;
use Cwd 'abs_path';
use File::Copy;
my $f = "$files";
my $d = basename(dirname(abs_path($f)));
my($file, $dir, $ext) = fileparse($files);
my $initialLocation = "$dir$file";
my $finalLocation = "AllFiles/$file";
print("$finalLocation\n");
move($initialLocation,$finalLocation);
}
Related
I both love and hate writing Bash. I love that it's so streamlined for operating on files and working with processes (I agree with this popular question that it's way better in this regard than Python, Ruby, etc.), but I hate the syntax, particularly around conditionals, loops, etc.
(This is subjective, but I find it both confusing and annoying. E.g. $var when reading, but var when writing; writes silently fail if there are spaces around =; the double brackets in ifs when using regexp; double semicolons sometimes and single semicolons others; etc.)
As a huge fan of CoffeeScript, which compiles to JS, I've been wondering: are there any languages that have the aesthetic/syntax of languages like Python/Ruby/CoffeeScript but which compile and run as Bash instead of one of those other runtimes?
E.g. I'd love to be able to write mostly-Bash with just a bit simpler syntax:
$AGGREGATE_FILENAME = 'allfiles.txt'
if not exists $AGGREGATE_FILENAME
touch $AGGREGATE_FILENAME
for $file in files/*
cat $file >> $AGGREGATE_FILENAME
switch $1
case 'test'
run-tests
echo 'Tests finished!'
case 'deploy'
echo 'Packaging...'
mv foo bar/
deploy-bar
This is a super contrived example, and the syntax is a strawman (mostly inspired from CoffeeScript but keeping the essential Bash notions of first-class commands, separated from variables, and loose typing).
Anyway, just a question and food for thought. I'd love to be able to write my scripts in something nicer than Bash. =) Thanks!
You could also try Batsh, which is a DSL (Domain-Specific Language) that compiles a C-syntax language to Bash (and Windows Batch).
Project
Online demo
Since I originally asked this question, two projects have been released which attack this problem and do a pretty good job. Both reimplement many/most Unix tools in more programming-friendly runtimes.
Plumbum is implemented in Python and looks pretty solid:
http://plumbum.readthedocs.org/en/latest/index.html
ShellJS is implemented on Node.js and also looks pretty good:
https://github.com/arturadib/shelljs
Exciting developments! I'm looking forward to trying them out. If you already have, it'd be great to hear your experiences in the comments. Thanks!
I tried all of the above (results) and started powscript.
Differences powscript vs the tools above
extremely portable preprocessor (100% bash)
balances between coffeescript and bash
hasslefree portable all-in-one-file compiler/runtime, written in bash
loose transpiler: inline bash always possible
Bish is another option:
https://github.com/tdenniston/bish
Shell scripting with a modern feel.
Bish is a lightweight language created to bring shell scripting into the 21st century. It gives programmers the comfort of modern syntax but compiles to Bash, resulting in good portability (in as much as Bash is portable).
The problem is that the whole strings-based semantics of Bash is so horribly broken, it'd be pretty difficult to do something like CoffeeScript for Bash.
Since you probably don't need function-level interoperability to call functions that are written in Bash, you're better off using something entirely different. Perl is close to Bash in being nasty and full of shortcuts and weird syntax, but its semantics are mostly sound. Python is less comfortable for things such as launching processes but is far better for general systems programming, clean and easy to maintain. Python has great libraries and modules for everything; Perl even better.
I recently developed a language called BashClass which is Object Oriented, has type checking and allow multi-dimensional arrays. The language syntax is inspired by different programming languages.
Here's an example on how a List class is implemented (Full example here):
class List extends Object {
var Object[] data = new Object[];
var int size = 0;
constructor List(){
super_constructor();
}
function void add(var Object object) {
data[size] = object;
size = size + 1;
}
function void pop() {
if(size == 0) {
exception("Cannot remove element from an empty list");
}
size = size - 1;
data[size] = null;
}
function int size() {
return size;
}
function Object get(var int index) {
if(index < 0 || index >= size) {
exception("Cannot access element out of bound");
}
return data[index];
}
}
Classes and multi-dimensional arrays in BashClass are converted to Bash 4.4 associative arrays. The language is at its first release and is open source on Github. Feel free to contirbute and suggest features.
You might want to give ZSh a try, it has a lot of improvements to make your shell script more readable.
http://www.zsh.org
You might want to take a look into nscript, in which you can write shell scripts using javascript. All the common bash constructions are in there, like exit codes, pipes, stream redirects, argument expansion, globbing, prompt etc.
I am new to mac, and am in the process of getting my computer setup with all the programs I need, one of them being Haskell.
To my surprise, the Haskell platform for OS X is not like in Windows (where there is an GUI editor built on the platform installation - winGHiC). After looking a lot around, I found this editor TextMate which is supposed to be compatible with Haskell but am finding it quite complex to setup the Haskell Bundle for it.
I have already downloaded and installed the following:
Haskell Platform for Mac OS X
Xcode 3.4 (Haskell Prerequisite)
TextMate
haskell.tmbundle files (mentioned above)
Is anyone familiar on how to get it working? It will mean a lot to me a detailed stepwise explanation, like I said, I has been only 1 day since I used OS X for the first time.
Thanks a bunch.
Well it's good work that you have found that the tmbundle is on Github these days.
You should be able to find the inbuilt options by clicking Bundles, then Haskell. There are fairly few in the standard Haskell.tmbundle. The most obviously useful one is ... load file command-shift-r or command-R. If you have written a module with ending .hs or .lhs, it ... opens it in ghci.
(There was some talk of an integrated terminal in TextMate II, but who knows? One annoying feature of a non-integrated terminal is that one is tempted to 'reload' by clicking command-R rather than by doing :r inside ghci; in certain frenzies it will develop that I have 15 copies of Terminal open.)
Many of the nice features are just general TextMate things, determined by the language description, so it might be good to read a general description of TextMate niceties. For example, if several lines are highlighed, then command-/ comments them out with --s; or, if they are already commented, uncomments them. I had hacked together something to do this, long before I realized it was already there, not having studied the manual closely enough.
Everything has keybindings, of course, and it is very easy to add your own to run little scripts and insert little snippets, much more so than in Emacs, say.
Under Bundles click Bundle Editor and examine the text for different kinds of things.
So, for example, to make a tab trigger to start a language extension pragma {-#LANGUAGE ... #-} where the cursor is in the space ...make a copy of a Snippet and substitute
{-#LANGUAGE ${1}#-}
selecting Activation: Tab Trigger, and (say) LANG as the trigger.
One nice thing is that they are all shell scripts, or else (like this one) partial shell scripts with some TextMate variables around, and you can pretty much write them in your own preferred language. (For the Haskell bundle I don't have any Haskell ones to speak of any more, but for other bundles I do.)
The syntax highlighting is surprisingly sound, but trips over a few fancy extensions, e.g. "PackageImports" , GADT syntax, markup for the Haddock documentation system, and some other oddities. I have hacked my own, but I find the format of the syntax files pretty unintelligible, so it's no use sending you a copy. The Haskell.tmbundle has been emended by some knowledgeable Haskellers in the last two or three years. The person who first made it was just learning Haskell, and hadn't e.g. composed Haddocked modules, but on the other hand, he seems fortunately to have been very skilled and to have had an intimate knowledge of the TextMate machinery.
Notice by the way that TextMate stores the emendations you make in the Bundle Editor in a rather strange way. The bundles that come with TextMate, e.g. C, Ruby, HTML, LaTeX, etc. are in /Applications/Textmate.app/.../Bundles. The ones you install are in /Library/Application\ Support/TextMate/Bundles. When you make emendations through the Bundle Editor, they are stored in your local ~/Library/Application\ Support/TextMate/Bundles. It sort of makes sense but it's a little complicated, and impedes public improvement of the Haskell bundle. (The one bundle I share with people, not the Haskell one, I keep in the lattermost, outermost directory under git revision, so the original and my emendations are together.)
The "Lookup on Hoogle" keybinding/option acts on highlighted terms; here's a replica for the hayoo website, which can search for uses of a type-signature
echo "<meta http-equiv=\"refresh\" content=\"0;
http://holumbus.fh-wedel.de/hayoo/hayoo.html?query=${TM_SELECTED_TEXT:=$TM_CURRENT_WORD}\">"
If you cabal install hoogle, then you can make a local call to hoogle with a script like so:
hoogle --w --n=100 '${TM_SELECTED_TEXT:=$TM_CURRENT_WORD}'
choosing Save: Nothing, Input: Selected Text, Or: Word, Output: Show as HTML
Other emendations I have made are mostly trivial, like a tab trigger snippet for `{-#LANGUAGE ... #-} or else eccentricities of my own.
One thing worth mentioning that I managed to integrate is the typeof executable from Hackage, cabal install typeof, which runs to ghci for an inferred type signature. I have a key binding to show the inferred type as displayed bit of html, but also to insert it. It's a bit delicate but here is the text for the displayer of types
#!/bin/bash
word=${TM_SELECTED_TEXT:-$TM_CURRENT_WORD}
module="${TM_FILEPATH}"
echo $word | typeof $module
choosing Input: Selected Text, Or: Line ; Output: Show as Tool Tip, Activation : Key Equivalent (then choose a keybinding , mine is control-option-command-j) Similarly, for type insertion via typeof make a new C (command file) heading and include this:
#!/bin/bash
word=${TM_SELECTED_TEXT:-$TM_CURRENT_WORD}
module="${TM_FILEPATH}"
echo $word | typeof $module | typeof_wordorder
# typeof_wordorder is the following hack compiled
# in my local ~/bin
# module Main where
# main = interact reword where
# reword :: String -> String
# reword xs =
# xs ++ (head . words . concat . lines $ xs)
Here typeof and typeof_worderorder are Haskell executables, the first cabal-installed, the second is the above commented idiocy, compiled in my local ~\bin to get around some escaping nonsense. Here you should choose Output : Insert as Snippet
Sorry, I'm just thinking of random things. You should keep posting questions under this heading, as I think it would be worthwhile to see how one might trip up, but also what hacks our cleverer Haskeller friends may have thought of. I keep meaning to put a 'fork' of my tmbundle on github, but it's not too thrilling, and the organization of Bundle directories forever defeats me.
I often use Sweave to produce LaTeX documents where certain chunks are produced dynamically by executing R code. This works well - but is it also possible to have code chunks that are executed in different ways, e.g. by executing the code in the shell, or by running Perl, and so on? It would be helpful to be able to mix things up, so I could do things like run some shell commands to fetch some data, run some perl commands to pre-process it, and then run R commands to analyze it.
Of course I could use all R chunks and use system() as a poor-man's substitute, but that doesn't make for very pleasant reading in the document.
The new new thing (for multi-language, multi-format) docs may be dexy.it which for example these guys at opengamma.org use as the backend.
Ana, who is behind dexy, is also giving a lot of talks about it so also look at the dexy blog.
It's not directly related to Sweave, but org-babel, which is part of Emacs org-mode, allows to mix code chunks of different languages in one file, pass data from one chunk to another, execute them, and generate LaTeX or HTML export from the output.
You can find more informations about org-mode here :
http://www.orgmode.org/
And to see how org-babel works :
http://orgmode.org/worg/org-contrib/babel/
There is certainly no easy way to do this other than through either foreign language interfaces from R (maybe through inline if it's supported), or system(). For what it's worth, I would just use system(); that should be easy enough.
You can see this previous question about having a Sweave equivalent for Python, where one of the respondents actually creates a separate interface. This can give you a sense what what it would take to embed other languages which may not already be supported. At a minimum, you have to do major hacking on the Sweave driver.
Do you know emacs" org-mode and, more specifically, Babel? If you already know Emacs or are willing to switch to Emacs, then org-mode and Babel are the answer to your question(s).
For instance, I am currently working on a document which contains some shell-scripts, does computations with R and creates flow charts with dot (graphviz). Org-mode can export a variety of formats, e.g. LaTeX (that's what I use).
There is the StatWeave project which uses java rather than R to do the weaving, but will run multiple programs instead of just R. I don't know how hard it would be to get it to do Perl or other programs like that, but the homepage indicates that it already works with R, SAS, Stata, and others:
http://www.cs.uiowa.edu/~rlenth/StatWeave/
I know the question is very subjective. But I cannot form the question in a much more better manner. I would appreciate some guidance.
I often as a developer feel how easier it would have been for me if I could have some tools for doing some reasonably complex task on a log file , or a set of source files, or some data set etc.
Clearly, when the same type of task needs to be done repetitively and when speed is critical, I can think of writing it in C++/Java.
But most of the times, it is some kind of text processing or file searching activity that I want to do only once just to perform a quick check or to do some preliminary analysis etc. In such cases, I would be better off doing the task manually rather than writing in C++/Java. But I could surely doing it in seconds if I knew some language like Bash/Python/Perl/Ruby/Sed/Awk.
I know this whole question is subjective and there is no objective definite answer, but in general what does the developer community feel as a whole? What subset of these languages should I know so that I can do all these kinds of tasks easily and improve my productivity.
Would Perl be a good choice?
It is a super set of Sed/Awk, plus it allows to write terse code. I can get done with fewer lines of code. It is neither readable nor easily maintainable, but I never wanted those features anyway.
The only thing that bothers me is the negative publiciity that Perl has got lately and it has been criticized by the Ruby/Python community a lot. Also, I am not sure if it can replace bash scripting totally.
If not, then is Perl+Bash a good combination for these kind of tasks?
I tend to do a lot of processing with ruby. It has all the functionality of perl, but I find it to be a little more readable. Both perl and ruby support the -n, -e, and -p options.
-e 'command' one line of script. Several -e's allowed. Omit [programfile]
-n assume 'while gets(); ... end' loop around your script
-p assume loop like -n but print line also like sed
For example in ruby
seq 1 4 | ruby -ne 'BEGIN{ $product = 1 }; $product *= $_.to_i; END { puts $product }'
24
Which is very similar to perl
seq 1 4 | perl -ne 'BEGIN{ $product = 1 }; $product *= $_; END { print $product }'
24
In Python, the same would look like this:
seq 1 4 | python -c 'import sys; print reduce(lambda x,y : int(x)*int(y), sys.stdin.read().splitlines(True))'
24
While it's possible to do the above in bash/awk/sed, you'll be limited by their lack of more advanced features.
Python is more expressive and readable than bash, but requires more setup: import os and so on. For simple tasks, bash is quicker -- which is the most important for this. And don't underestimate the power of input/output redirection in bash!
I would use Perl over a bash/sed/awk combination. Why ?
You only have the one executable, rather than spawning off multiple executables to do work.
You can make use of a wide range of Perl modules to do most anything (see CPAN for the modules available)
In fact I would recommend any scripting language over the shell/awk/sed combination, for the same reasons. I don't have a problem with sed/awk per se, but as your required solutions become more complex/lengthy, I find the more powerful scripting languages more scalable, and (to some degree) refactorable for re-use.
I find Python+Bash a very nice combo.
I usually use Python because it's very readable and maintainable. And because there are lots of online documentation available.
Btw, I suggest you to read http://www.ibm.com/developerworks/aix/library/au-python/
With shell scripting, all you ever need to know is a bit bash/sh and a lot of awk. Bash for calling your commands, and awk for processing. Some of the unix tools below, contrary to the fact that many people use them, are not necessary because awk can do their functions.
1) cut
2) sed
3) wc
4) (e)grep
5) cat
6) head
7) etc..
and a few others whose functions overlap. In the end, your script will not cluttered with redundant tools and slow down your script.
Perl/Python are very useful sysadmin tools as well. Both of them do similar things and have libraries that help in your sysadmin tasks. The only significant difference is, aesthetically speaking, the appearance of your code written in them.
You can learn about Ruby if you want, but in terms of sysadmin, I would say go for Perl/Python instead.
In the time it took you to write those few paragraphs, you could have already learned enough Python to make your life significantly better.
Anyone who already knows C++ or Java can become productive in Python in about 4 hours. Just read the tutorial.
My first port of call is bash with sed to provide regular expression processing. You can do a lot with a bash for loop, grep and some regular expressions.
It's worth learning regular expressions if you don't already know them. An editor which lets you use them (like vi) is extremely useful when manipulating files (e.g. you have a set of data extracted from a logfile, and you need to turn it into a set of SQL statements for example).
If it takes me more than a few minutes to figure out how to do whatever parsing task I'm trying to do in bash/sed, I usually end up using perl instead. As suggested by ikkebr, python is probably as good as (or better than) perl; I just had the misfortune to learn perl first, so am much more familiar with it - if I was to start again, I'd learn python instead I think.
Do you know if there's any tool for compiling bash scripts?
It doesn't matter if that tool is just a translator (for example, something that converts a bash script to a C program), as long as the translated result can be compiled.
I'm looking for something like shc (it's just an example -- I know that shc doesn't work as a compiler). Are there any other similar tools?
A Google search brings up CCsh, but it will set you back $50 per machine for a license.
The documentation says that CCsh compiles Bourne Shell (not bash ...) scripts to C code and that it understands how to replicate the functionality of 50 odd standard commands avoiding the need to fork them.
But CCsh is not open source, so if it doesn't do what you need (or expect) you won't be able to look at the source code to figure out why.
I don't think you're going to find anything, because you can't really "compile" a shell script. You could write a simple script that converts all lines to calls to system(3), then "compile" that as a C program, but this wouldn't have a major performance boost over anything you're currently using, and might not handle variables correctly. Don't do this.
The problem with "compiling" a shell script is that shell scripts just call external programs.
In theory you could actually get a good performance boost.
Think of all the
if [ x"$MYVAR" == x"TheResult" ]; then echo "TheResult Happened" fi
(note invocation of test, then echo, as well as the interpreting needed to be done.)
which could be replaced by
if ( !strcmp(myvar, "TheResult") ) printf("TheResult Happened");
In C: no process launching, no having to do path searching. Lots of goodness.