I want to learn Sed. Can you please point me to good references so that I can fully utilize it.
I want to learn it to perform more of the do-once-then-forget type administrative or dev-tools like tasks. So, I don't really care about performance or modularity or object orientedness etc when writing this type of code. Do you think it would be a good idea to learn Sed? Or should I learn Ruby? My main concern is the conciseness of scripts, but not to the point of making it totally obscure looking piece of code.
Thanks
Ajay
I can't think of anything that sed is better at than ruby. Sed's syntax for a lot of things is very similar to Ruby sub and gsub string methods. I guess I would point you to Ruby unless you really have no other use for a scripting language other than text processing. You could probably get a reasonable understanding of how to use sed a little quicker than how to do the same stuff in ruby.
Sed - an Introduction and Tutorial
O'Reilly Sed & Awk book
I would encourage you to take a look at Perl for tasks like this. Many of Perl's features are designed expressly with the goal of "processing text with concise syntax". It can of course jump over into cryptic, but so can Ruby.
Perl also has extensive support for one line programs typed into the shell. echo "hello world" | perl -lpe 's/\b(.)/\U$1/g' prints Hello World
Forget sed, anything more than a simple, line-oriented search and replace is just unreadable. Learn Awk. And then learn Ruby too: it's good for your soul.
Related
I'm a user of the Ruby language, and while the idea of using the Crystal language as a one-liner may be silly, the Crystal language is so fast that even when you add up the compile time, it can sometimes run faster than writing one-liners in Ruby. Is there a way to do one-liners in Crystal?
It can, but its definitely not written with that in mind like Perl and to some degree Ruby.
This is mostly down to Crystal intentionally not inheriting many of the Perlisms that makes this so convenient in Ruby, such as the -p, -n etc command line flags and globals like $_, $' etc.
Other than that, nothing stops you from running crystal eval 'some code' to your heart's content.
Yes, it definitely is not only possible but very easy. I have never used Crystal before in my life, and I was able to create a one-liner in less than 5 seconds:
puts "Hello, World!"
I would like to make a DSL in Ruby, where I will can using simple commands do something. For example "GO PRINT 10 *" will print 10 stars
or "GO PRINT 5 &" will print 5 ampersand. Anyone know good tutorials ?
I heard about the gem docile It is worthy of attention?
Here are some links relevant to what you want to do:
https://robots.thoughtbot.com/writing-a-domain-specific-language-in-ruby
Tutorials for writing DSL in Ruby
https://www.leighhalliday.com/creating-ruby-dsl
If you want a very flexible DSL (i.e., one in which you can't tell that it's actually Ruby, which sounds like what you want), I'd suggest learning a language such as Racket which is very well-geared towards creating all sorts of languages, e.g., a brainf*ck clone, a stack-based calculator, these, and these.
I have a folder on my desktop called "images".
Inside are a bunch of sub-folders called various things "Flower_Shape", "Smile_Shape", "Dog_Shape", etc. and each folder contains 3 images "Variant1.PNG", "Variant2.PNG", and "Variant3.PNG" and then some other file formats that are called various things but they do not end in .png, they end in things like ".psd"...
How can I take all of the contents of these hundreds of folders, and move them all into 1 folder while changing the names from "Variant1.png" to whatever the name of the folder it was in is called (for example the "Variant1.png" inside the folder "Flower_Shape" would be re-named "Flower_Shape-Variant1.png") and then delete every file that doesn't end in ".png".
My first main question is what language would I use to do something like this? Would it be PERL?
After that is established, does anyone have any tips for how to go about doing this... I'm assuming just a for loop with some if statements should be all it really takes but I know nothing of perl (or any other languages that deal with changing files on my own computer for that matter)
Thanks!
Mac OS X comes with a whole assortment of scripting languages. It comes with Perl, Python, PHP, Ruby, and BASH.
Mac OS X also comes with Automator.
What should you choose?
If you're really, really interested in learning to program, I would suggest you start with Bash. BASH is what is called the command shell. If you type a command in at a prompt, it will be executed. The Unix shell is a powerful tool, and the BASH shell includes a lot of control structures that can be used. In BASH, you could use the find command and the mv command and write a loop to do what you want.
.
Shell scripts are limited in their power. They aren't suppose to be full service languages and make no apologies for it. If you need more than a few dozen lines or more complex control, you should switch to a scripting language.
I am a Perl programmer, but I would recommend, if you're really interested in learning how to program, to learn Python. Python is probably the most popular language and its use is certainly growing.
Perl is a nice language because much of its syntax structure comes from shell scripting languages. Despite what you've heard, Perl is still very much a popular language. I've found that people pick up Perl faster than Python because of it's looser syntax rules and the fact it doesn't have to be object oriented. That is also a major problem with Perl. Most Perl programmers never get past the basic hacking stage and Perl has a reputation of being a difficult language to maintain because of the sheer number of awful Perl scripts out there.
Ruby hasn't been as popular as Perl or Python, but it has its fans. The big power of Ruby is something called Ruby on Rails. Rails is a programming framework that allows you to quickly build web-based applications. You first should learn Ruby, then learn the Rails component.
PHP use to be the glue that held the web together. I believe its popularity has been dropping in recent years as newer web development platforms have come up. PHP is the mob rule of programming. Unlike all of the other languages, it doesn't have a single champion shaping the language and has developed a lot of detritus. It is sloppy and can be hard to maintain.
.
Still, it is the basis of a lot of web based forums and content management systems like Joomla. After all, PHP was designed to live inside of webpages.
If you don't want to learn script programming, you should try Automator. It comes with a Graphical User Interface and can allow you to build programs to do all sorts of tasks. People who have gotten into Automator have done some amazing stuff with it.
Whatever you choose, you can find some tutorials and information on the web, but if you really want to learn, you should get some good manuals. O'Reilly and Associates has a long storied history with computing and the Internet. Tim O'Reilly (the brains behind the company) has been producing computer manuals since 1978. Almost all of my books are O'Reilly books. Manning is another one, and there are dozens more. Go to a real bookstore before Amazon drives them all out of business and peruse their shelves for something you like.
You have to combine File::Find with File::Copy to reach those requirements
If you use perl, you should look into Path::Class and File:::Copy. That'll handle your path parsing. You'd want to iterate over the filesystem (Path::Class::Dir->next will be your friend here), build a destination file with something like:
my $curr_file = file('whatever_path...');
my $dest_dir = dir('some other path...');
my $dest_file = file($dest_dir, $curr_file->dir->basename . '-' . $curr_file->basename);
then use File::Copy to move:
move($curr_file, $dest_file);
Start by looking up those methods/subroutines and their usage; then you just need to code up the iteration.
I was able to accomplish this using this code (and changing the value inside the "glob" function to different levels of /// manually)
#!/usr/bin/perl
foreach my $files (glob ('*/*/*')) {
use strict;
use File::Basename;
use Cwd 'abs_path';
my $f = "$files";
my $d = basename(dirname(abs_path($f)));
my($file, $dir, $ext) = fileparse($files);
my $initialName = "$dir$file";
my $finalName = "$dir$d$file";
print("$initialName --> $finalName \n");
rename($initialName, $finalName);
}
Then it was as simple as moving the files with a move function into a new folder, like so:
#!/usr/bin/perl
foreach my $files (glob ('*/*/*')) {
use strict;
use File::Basename;
use Cwd 'abs_path';
use File::Copy;
my $f = "$files";
my $d = basename(dirname(abs_path($f)));
my($file, $dir, $ext) = fileparse($files);
my $initialLocation = "$dir$file";
my $finalLocation = "AllFiles/$file";
print("$finalLocation\n");
move($initialLocation,$finalLocation);
}
Looking for a command line code formatter that can be used for bash code. It must be configurable and preferably usable from command line.
I have a big project in bash, which I need to use Q in mind for. So far I am happy with a program written in python by Paul Lutus (a remake of his previous version in Ruby).
See http://arachnoid.com/python/beautify_bash_program.html (also cloned here https://github.com/ewiger/beautify_bash).
but I would like to learn any serious alternative to this tool if it exists. Requirements: it should provide robust enough performance and behavior of treating/parsing rather complicated code.
PS I believe full parsing of bash code is generally complicated because there exists no official language grammar (but please correct me if I am wrong about it).
You could give shfmt a try. It implements its own shell parser including Bash support, so it's more robust than plaintext-based tools.
And both the parser and printer are available as Go packages, so it should be easy to write a 20-line Go program to manipulate or play with shell code.
Please note that I'm the author, so the advice may be a bit biased :)
you can script vim to do: "gg=G" that means "indent all the file"
I discovered that the type builtin will print functions in a formatted manner.
#/usr/bin/env bash
source <(cat <(echo 'wrapper() {') - <(echo '}'));
type wrapper | tail -n +4 | head -n -1 | sed 's/^ //g'
https://github.com/bas080/flush
On the contrary the shell does have a rigorous grammar.
It is described both in English in the ISO standard and documentation for Bash and other shells, and in formal terms in the shell.y file in the Bash source tree.
What makes it "hard" is that where one normally thinks of, say, a quoted string as single lexical token, In the shell every meta character is a separate lexical token, so the meaning of a character can change depending on its grammatical context.
So the parsing tokens do not align with the "shell words" that a user thinks of, and a simple quoted string is at least 3 tokens.
The implementations typically take some shortcuts involving using multiple lexical analysers chosen by whether the grammar is inside quotes, inside numeric context, or outside both.
I know the question is very subjective. But I cannot form the question in a much more better manner. I would appreciate some guidance.
I often as a developer feel how easier it would have been for me if I could have some tools for doing some reasonably complex task on a log file , or a set of source files, or some data set etc.
Clearly, when the same type of task needs to be done repetitively and when speed is critical, I can think of writing it in C++/Java.
But most of the times, it is some kind of text processing or file searching activity that I want to do only once just to perform a quick check or to do some preliminary analysis etc. In such cases, I would be better off doing the task manually rather than writing in C++/Java. But I could surely doing it in seconds if I knew some language like Bash/Python/Perl/Ruby/Sed/Awk.
I know this whole question is subjective and there is no objective definite answer, but in general what does the developer community feel as a whole? What subset of these languages should I know so that I can do all these kinds of tasks easily and improve my productivity.
Would Perl be a good choice?
It is a super set of Sed/Awk, plus it allows to write terse code. I can get done with fewer lines of code. It is neither readable nor easily maintainable, but I never wanted those features anyway.
The only thing that bothers me is the negative publiciity that Perl has got lately and it has been criticized by the Ruby/Python community a lot. Also, I am not sure if it can replace bash scripting totally.
If not, then is Perl+Bash a good combination for these kind of tasks?
I tend to do a lot of processing with ruby. It has all the functionality of perl, but I find it to be a little more readable. Both perl and ruby support the -n, -e, and -p options.
-e 'command' one line of script. Several -e's allowed. Omit [programfile]
-n assume 'while gets(); ... end' loop around your script
-p assume loop like -n but print line also like sed
For example in ruby
seq 1 4 | ruby -ne 'BEGIN{ $product = 1 }; $product *= $_.to_i; END { puts $product }'
24
Which is very similar to perl
seq 1 4 | perl -ne 'BEGIN{ $product = 1 }; $product *= $_; END { print $product }'
24
In Python, the same would look like this:
seq 1 4 | python -c 'import sys; print reduce(lambda x,y : int(x)*int(y), sys.stdin.read().splitlines(True))'
24
While it's possible to do the above in bash/awk/sed, you'll be limited by their lack of more advanced features.
Python is more expressive and readable than bash, but requires more setup: import os and so on. For simple tasks, bash is quicker -- which is the most important for this. And don't underestimate the power of input/output redirection in bash!
I would use Perl over a bash/sed/awk combination. Why ?
You only have the one executable, rather than spawning off multiple executables to do work.
You can make use of a wide range of Perl modules to do most anything (see CPAN for the modules available)
In fact I would recommend any scripting language over the shell/awk/sed combination, for the same reasons. I don't have a problem with sed/awk per se, but as your required solutions become more complex/lengthy, I find the more powerful scripting languages more scalable, and (to some degree) refactorable for re-use.
I find Python+Bash a very nice combo.
I usually use Python because it's very readable and maintainable. And because there are lots of online documentation available.
Btw, I suggest you to read http://www.ibm.com/developerworks/aix/library/au-python/
With shell scripting, all you ever need to know is a bit bash/sh and a lot of awk. Bash for calling your commands, and awk for processing. Some of the unix tools below, contrary to the fact that many people use them, are not necessary because awk can do their functions.
1) cut
2) sed
3) wc
4) (e)grep
5) cat
6) head
7) etc..
and a few others whose functions overlap. In the end, your script will not cluttered with redundant tools and slow down your script.
Perl/Python are very useful sysadmin tools as well. Both of them do similar things and have libraries that help in your sysadmin tasks. The only significant difference is, aesthetically speaking, the appearance of your code written in them.
You can learn about Ruby if you want, but in terms of sysadmin, I would say go for Perl/Python instead.
In the time it took you to write those few paragraphs, you could have already learned enough Python to make your life significantly better.
Anyone who already knows C++ or Java can become productive in Python in about 4 hours. Just read the tutorial.
My first port of call is bash with sed to provide regular expression processing. You can do a lot with a bash for loop, grep and some regular expressions.
It's worth learning regular expressions if you don't already know them. An editor which lets you use them (like vi) is extremely useful when manipulating files (e.g. you have a set of data extracted from a logfile, and you need to turn it into a set of SQL statements for example).
If it takes me more than a few minutes to figure out how to do whatever parsing task I'm trying to do in bash/sed, I usually end up using perl instead. As suggested by ikkebr, python is probably as good as (or better than) perl; I just had the misfortune to learn perl first, so am much more familiar with it - if I was to start again, I'd learn python instead I think.