I'd like to get short Scala scripts running as fast as python scripts do, particularly in terms of script startup time.
Can anyone recommend some ways of doing this, that doesn't involve compiling with GCJ, for instance?
One way I can think of is to run the script using the JVM's client mode, but I can't seem to get this working. An example (known-good) shebang for this would be great.
UPDATE: I'm aware of the other questions, but don't think any workable answers have been found so far, as I'm looking for solutions that work on STANDARD installs, without additional requirements. This is what I was trying to get at with "doesn't involve compiling with GCJ, for instance".
It seems that -client mode is designed for this express purpose, but it's just awkward to activate from scala scripts for some reason.
As many other questions have gone before, if one could only know how to look for them, use Nailgun.
Other ways to improve on script performance is to start fsc at system boot, so it will be available for scripts, and use -savecompiled, to avoid repeated compilation of scripts.
EDIT
You mention -client mode, but I think that's really not a good choice. The will give you a slower Scala compiler, and do little to improve the startup time of the compiler itself, if not Java. Much better to have fsc as daemon, running as -server, and/or save compiled scripts with -savecompiled.
Now, I don't know what problems you are having with -client, but I have read that it doesn't work with 64 bits JVM. Might that be your case?
PS: Looking up similar questions, I noticed JRuby has builtin Nailgun support!
I haven't tried this yet, but scala-native promises near-instant startup because it compiles into a native binary. So one solution is to provide it as a number of binary downloads.
http://www.scala-native.org/en/latest/
I just tried to pass the '-client' parameter through Scala to the JVM this way:
#!/bin/sh
exec scala -J-client "$0" "$#"
!#
args.foreach(println)
It seems to work. Daniel C. Sobral wrote that he read that it doesn't work with 64 bits JVM. I don't know, maybe this is outdated. Anyway it seem to drop the startup time a little bit.
Running:
:~$ time /tmp/testScalasScript arg1
arg1
real 0m2,170s
user 0m2,228s
sys 0m0,217s
This was the fastest run of just a couple of tests. Without it it takes up to 0.5s longer. But this was really a quick test and should be done more systematically to come to meaningful results.
Wasn't there a way to make Scala compile and save the compilation result at the first run of the script for faster reuse? But I don't no for sure at this point of time.
UPDATE:
I just saw that on 'java -help' the option '-client' is not documented (anymore?). Anyway there is no error thrown (which is done on usage of unexisting options). So I am not sure if the '-client' option really has consequences.
Related
Instead of recording my own keystrokes to create an asciinema tutorial I do want to script them, so I can easily improve my tutorial over time without being being stressed about redoing it each time I want to make a change, or spending a lot of time trying to correct typos in the recording.
Mainly I want to be able to do somethign like:
type: "df"
wait: 2s
type: "echo foo"
This would enable me to easily rebuild the recording from scratch when I change the script.
How can I do this?
I've searched for a lot of solutions for my own projects, and this is the solution that I've come up with:
My solution:
What I'm planning on doing for a project I'm in is to use both asciinema as well as demo-magic.sh.
Specifically, I'll be using asciinema for the recording with the -c flag (see here), and demo-magic.sh for the automated typing, since it supports waiting for commands to finish executing (or not) and custom wait times on top of that. The command would look something like asciinema rec -c "./mydemoscript.sh" myrecording. See the projects for proper usage.
Other things I've come across:
doitlive -- I'm not using this because it's more of a fake-typing automated thing where you actually have to "type like a madman", which could be useful if you're doing a talk in front of an audience but you don't want to mess up your keystrokes, or if you don't want to forget to do certain commands.
asciiscript -- it's written in Go and it works, but you need to compile it yourself, and it also doesn't support waiting for the previous command to finish.
spielbash -- a Ruby project designed specifically to automate asciinema recordings using tmux. I'm not using this, though, because it's not as portable since you need an existing Ruby installation, and more importantly, it keeps corrupting the active console I'm in, and there's also unpredictable corruption in the recordings. Also, the project hasn't been committed to since 2019.
termscript -- it just doesn't work for me, and it freezes my console. It runs on python 2.
UPDATE: I have recently come across terminalizer. It seems to provide support for an window-like border around the actual terminal, and it has a built-in GIF renderer, but I have not tried it.
There is a project svg-term which allows to record a screencast from a command, and I've seen a script that simulates commands being typed letter after letter.
For the moment I ended up using AppleScript to automate iTerm but this approach is buggy and locks you to MacOS and iTerm... and if you happen do to something during recording, you are doomed.
I am still looking for a better approach.
TL;DR: asciinema-automation should do what OP wants. Its dependencies are asciinema and pexpect.
(disclaimer: I developed it for my own usage)
Some time ago, I was also looking for a way to automate asciinema recordings and I saw the very nice answer of a-la-linuques. I chose asciiscript because it used asciinema, but it is not maintained anymore. I tried to keep on using it via a fork adding new small features, but being not very familiar with go, I finally decided to rewrite everything in python in this repo.
It reads bash files, where comments can give special instruction like adding time between command or key stroke, or wait for an expected output (see examples). Of course, this is very much in the spirit of asciiscript.
I have inherited a sprawling crontab that I need to maintain and update. I don't have much experience with it or bash scripting (I think I've got a decent grip on the basics) and I want to do a good job.
Short request: Any guidelines for 'refactoring' a messy crontab and set of bash scripts
Long request: I've run into a number of issues, but are so many people using cron files etc that I feel like I must be missing some large repository of information, best practices and tools - or is this just a stylistic difference for this kind of programming? (My bias: why do something manually if I can use a tool to do it faster, consistently and well?).
Examples of issues so far:
Due to an external event, the crontab didn't run for a couple of days. Along with someone else, we manually went through the list, trying to figure out what didn't run, what we needed to rerun, and what scripts we needed to edit and run with earlier dates etc.
What I can't find:
There are plenty of (slightly pointless) 'cron generators' online. Where are the reverse? Something I can feed in a long crontab, two dates, and have it output which processes should have run when, or just how many times total?
This seems within my meager scripting capabilities, so shouldn't it exist already? ;)
Alternatively, if I ever have to do that again, is there some way of calling a bashscript so that any instances of date() are pre-set to an earlier time, rather than changing every date call within the script? (e.g. for all the missed reports and billing invoices)
It turns out a particular report hadn't been running for two years. It was just requested again, and lo, there it was in the crontab! The bash script just had broken path references to the relevant files.
What I can't find: some kind of path checker for bash files? Like a website link checker. Yes I'll be going through these all manually eventually, but it'd show up some at least some of the problem areas.
It sounds like some times, there has either been too long or short a gap between dependent processes, so updates have happened after the first has been run, or the first hasn't finished running before the second has been called. I've seen a few possible options for this (eg anacron runs in sequential order), but what would you recommend?
There are also a large number of essentially meaningless emails generated from the crontab (scripts throwing errors but running 'correctly', failing mostly silently, or just printing everystep of non-essential scripts). I'll be manually going through scripts and trying to get them to provide more useful data, or 'succeed quietly', but y'know - any guidelines?
If my understanding or layout of the issue is confused, then I apologize, but hey - you see my problem then! I need to go from newbie, to knowing what to do to get this right, and not screw up a touchy system further. Thanks!
Not a full answer, but more resources that have been helpful:
http://blog.endpoint.com/2008/12/best-practices-for-cron.html
I am slowly going through this, and trying to implement each of the points. I hadn't thought to google 'best practices cron' til after my post. :P
For version control, I'm just going to use RCS in the meantime, as I edit scripts on a file-by-file basis, but I've been advised to get Git set up (or Mercurial if I was on a Windows system).
This actually sounds great:
http://everythingsysadmin.com/2010/09/xed-202-released.html
"xed is a perl script that locks a file, runs $EDITOR on the file, then unlocks it."...and puts it in RCS if it wasn't already.
Completely brainless version control. If I get my head around bash, I'd like to create an editing shortcut that automatically commits to whichever version control system I use.
Other tips I received from an System Admin,
Dates: Rather than using say, date, or --date="last monday", use a fixed date and add a day/week etc to it each time it runs (if not more than current day obviously), because then if the script doesn't run, I can just re-run the script repeatedly until it catches up. Ah!
(And, this might sound obvious, but heaps of the reports I'll be eventually edit, don't say prominently what dates the report is running for. Will fix.)
And was reassured I should try and get the cron emails as quiet as possible, so that I actually notice if there's an error email.
There are wrappers for better cron error reporting that I have not yet investigated, linked here: http://habilis.net/cronic/
Herculean task ahead of you, best of luck. :)
I'd suggest finding all the tasks that run daily and shove them into their own scripts in /etc/cron.daily/. Same for weekly into /etc/cron.weekly, hourly, and monthly.
You might want to investigate use of anacron(8) for scheduling your jobs, if the machine won't always be online, but you still need some level of control over when the jobs are run. It's been the default cron-helper-tool for multiple distributions for a few years, so hopefully it's stable enough to rely on for your own tasks; but I could easily imagine that it might not perfectly meet your needs.
Faking the dates to scripts can be done with at least two packages on Ubuntu: datefudge and faketime. I have no experience with either, but both sound like they should be able to help. I hope you won't need it in the future. :)
Sorry, I know of no path-checker for bash scripts. It seems unlikely, since simple scripts are simple and easy to check by eye :) and complex scripts will be generating their pathnames at runtime anyhow. Maybe you could keep a database of pathnames used by each script and write a new script to verify that database regularly.
You could disable the cron email by setting MAILTO="". I'm not sure I like this. Maybe setting MAILTO to a logging-only account would help the deluge. Another option is getting really good at your procmail(1) rules so you can stuff them in another mailbox completely.
Getting good at mutt color or score controls can help you spot the wheat amongst the chaff. (color index red black ERROR or similar commands might help you spot the problems more quickly.)
I'm looking for writing a GUI client for a existing application in my job, this application is CLI and because this is not widely used.
This is the first time I'm writing something similar, the I ask you for recommendations, books, techniques, methodologies, advices. My first approach is to create the interface and to make calls to the original CLI client, is this a congruent approach?
Though it's not ideal, I don't think it's a bad approach, creating a GUI shell for your CLI app. In this design, the GUI acts as the CLI program's user. You have to consider things like:
Can the GUI anticipate or understand
all possible CLI program output? How about errors? How
complex will that be? Consider
parsing Unix "ls" output. Simple enough. How
about Windows command prompt "dir" output? A
bit more funky.
The CLI program may take time to
execute, this must be presented in
the GUI. The GUI may have to prevent
the user from running another instance of the CLI.
You might want to consider tcl/tk. I've written several successful commercial GUIs that work in exactly this manner.
I'll admit that it maybe takes a little skill to craft a stunning GUI but it's not impossible, and not even that hard. You won't be able to reproduce the eye candy of flash or silverlight, and if that is important this might not be the right solution for you.
If, on the other hand, you are more concerned with Getting The Job Done, tcl is a very viable choice. It's easy to learn and easy to I integrate with command line tools.
For someone who has been using bash for years, what's the quickest way to get productive with Microsoft's PowerShell? I've noticed my old friends ls and ps work, but sadly grep didn't. I could just man up and read through the piles of documentation, but I'm guessing there is a quicker path to productivity for someone with a long history of bash usage.
I don't know if it matters or not, but the reason I'm looking to get acquainted is because it seems like there is finally a proper shell on Windows instead of needing to install cygwin, and I'd like to start using it exclusively instead of writing bat files for automated activities like builds and deployments and such.
I'm not sure your Bash history is going to help you much with PowerShell. It is quite a bit different since the idea is to pipeline rich .NET objects as opposed to text or byte streams. For a leg up, check out my Effective PowerShell free eBook. It is about 60 pages but I think does a decent job of outlining the mental model you need to get the most out of PowerShell. For a full, free book check out Master PowerShell. At the point you think you're willing to part with some $$, Windows PowerShell in Action is the best book out there IMO.
Note: there are some "Unix" oriented aliases to get you by for a while (ls, ps, etc) but you'll run out of steam pretty quick because the parameters are quite different and as I said the pipeline is quite different than in Bash.
I would also recommended looking over some of the Q&A on PowerShell on StackOverflow especially for those questions that you think - "yeah, how would you do that in PowerShell". I think you would learn a good bit just doing that.
A quick Cheat sheet
I'd suggest just man up and read through the "piles" of documentation. It's actually not that difficult once you get the basics down. You'll need the knowledge anyway if you intend to do anything even remotely productive with PowerShell.
there GNU utilities such as your favourite grep that is ported to windows. see GNU packages
on the other hand, if you loathe the idea of downloading stuff, you can use vbscript. it comes natively installed.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 4 years ago.
Improve this question
I was looking at http://tldp.org/LDP/abs/html/why-shell.html and was struck by:
When not to use shell scripts
...
Mission-critical applications upon which you are betting the future of the company
Why not?
Using shell scripts is fine when you're using their strengths. My company has some class 5 soft switches and the call processing code and the provisioning interface is written in java. Everything else is written in KSH - DB dumps for backups, pruning, log file rotation, and all the automated reporting. I would argue that all those support functions, though not directly related to call-path, are mission critical. Especially the DB interaction. If something went wrong with the DB-interaction code and dumped the call routing tables it could put us out of business.
But nothing ever does go wrong, because shell scripts are the perfect language for stuff like this. They're small, they're well understood, manipulating files is their strength, and they're stable. It's not like KSH09 is going to be a complete rewrite because someone thinks it should compile to byte code, so it's a stable interface. Frankly, the provisioning interface written in Java goes wonky fairly often and the shell scripts have never messed up that I can remember.
I kind of think the article points out a really good list of the reasons when not to use shell scripts - with the single mission critical bullet you point out being more of a conclusion based on all the other bullets.
With that said, I think the reason you do not want to build a mission critical application on a shell script is because even if none of the other bullet points apply today, any application that is going to be maintained over a period of time will evolve to the point of being bit by at least one of the those potential pitfalls at some point.....and there isn't anything you are really going to be able to do about it without it being a complete do over to come up with a fix....wishing you used something more industrial strength from the beginning.
Scripts are nothing more or less than computer programs. Some would argue that scripts are less sophisticated. These same folks will usually admit that you can write sophisticated code in scripting languages, but that these scripts are really not scripts any more, but full-fledged programs, by definition.
Whatever.
The correct answer, in my opinion, is "it depends". Which, by the way, is the same answer to the converse question of whether you should place your trust in compiled executables for mission critical applications.
Good code is good, and bad code is bad - whether it is written as a Bash script, a Windows CMD file, in Python, Ruby, Perl, Basic, Forth, Ada, Pascal, Common Lisp, Cobol, or compiled C.
Which is not to say that choice of language doesn't matter. There are very good reasons, sometimes, for choosing a particular language or for compiling vs. interpreting (performance, scalability, capability, security, etc). But, all things being equal, I would trust a shell script written by a great programmer over an equivalent C++ program written by a doofus any day of the week.
Obviously, this is a bit of a straw man for me to knock down. I really am interested in why people believe shell scripts should be avoided in "mission-critical applications", but I can't think of a compelling reason.
For instance, I've seen (and written) some ksh scripts that interact with an Oracle database using SQL*Plus. Sadly, the system couldn't scale properly because the queries didn't use bind variables. Strike one against shell scripts, right? Wrong. The issue wasn't with the shell scripts but with SQL*Plus. in fact, the performance problem went away when I replaced SQL*Plus with a Perl script that connected to the database and used bind variables.
I can easily imagine putting shell scripts in spacecraft flight software would be a bad idea. But Java or C++ may be an equally poor choices. The best choice would be whatever language (assembly?) is usually used for that purpose.
The fact is, if you use any flavor of Unix, you are using shell scripts in mission-critical situations assuming you think booting up is mission critical. When a script needs to do something that shell isn't particularly good at, you put that portion into a sub-program. You don't throw out the script wholesale.
It is probably shell scripts that help take a company into the future. I know just from a programming standpoint that I would waste a lot of time doing repetitive tasks that I have delegated to shell scripts. For example, I know most of the subversion commands for the command line but if I can lump all those commands into one script I can fire at will I save time and mental energy.
Like a few other people have said language is a factor. For my short don't-want-to-remember steps and glue programs I completely trust my shell scripts and rely upon them. That doesn't mean I'm going to build a website that runs bash on the backend but I will surely use bash/ksh/python/whatever to help me generate the skeleton project and manage my packaging and deployment.
When I read thise quote I focus on the "applications" part rather than the "mission critical" part.
I read it as saying bash isn't for writing applications it's for, well, scripting. So sure, your application might have some housekeeping scripts but don't go writing critical-business-logic.sh because another language is probably better for stuff like that.
I would wager the author is showing they are uncomfortable with certain aspects of qualtiy wrt shell scripting. Who unit tests BASH scripts for example.
Also scripts are rather heavily coupled with the underlying operating system, which could be something of a negative thing.
No matter we all need a flexible tool to interact with os. It is human readable interaction with an os that we use; it's like using a screwdriver with the screws. The command line will always be a tool we need either admin, programmer, or network. Look at the window they even expanded on their Powershell.
Scripts are inappropriate for implementing certain mission-critical functions, since they must have both +r and +x permissions to function. Executables need only have +x.
The fact that a script has +r means users might be able to make a copy of the script, edit/subvert it, and execute their edited Cuckoo's-Egg version.