I have inherited a sprawling crontab that I need to maintain and update. I don't have much experience with it or bash scripting (I think I've got a decent grip on the basics) and I want to do a good job.
Short request: Any guidelines for 'refactoring' a messy crontab and set of bash scripts
Long request: I've run into a number of issues, but are so many people using cron files etc that I feel like I must be missing some large repository of information, best practices and tools - or is this just a stylistic difference for this kind of programming? (My bias: why do something manually if I can use a tool to do it faster, consistently and well?).
Examples of issues so far:
Due to an external event, the crontab didn't run for a couple of days. Along with someone else, we manually went through the list, trying to figure out what didn't run, what we needed to rerun, and what scripts we needed to edit and run with earlier dates etc.
What I can't find:
There are plenty of (slightly pointless) 'cron generators' online. Where are the reverse? Something I can feed in a long crontab, two dates, and have it output which processes should have run when, or just how many times total?
This seems within my meager scripting capabilities, so shouldn't it exist already? ;)
Alternatively, if I ever have to do that again, is there some way of calling a bashscript so that any instances of date() are pre-set to an earlier time, rather than changing every date call within the script? (e.g. for all the missed reports and billing invoices)
It turns out a particular report hadn't been running for two years. It was just requested again, and lo, there it was in the crontab! The bash script just had broken path references to the relevant files.
What I can't find: some kind of path checker for bash files? Like a website link checker. Yes I'll be going through these all manually eventually, but it'd show up some at least some of the problem areas.
It sounds like some times, there has either been too long or short a gap between dependent processes, so updates have happened after the first has been run, or the first hasn't finished running before the second has been called. I've seen a few possible options for this (eg anacron runs in sequential order), but what would you recommend?
There are also a large number of essentially meaningless emails generated from the crontab (scripts throwing errors but running 'correctly', failing mostly silently, or just printing everystep of non-essential scripts). I'll be manually going through scripts and trying to get them to provide more useful data, or 'succeed quietly', but y'know - any guidelines?
If my understanding or layout of the issue is confused, then I apologize, but hey - you see my problem then! I need to go from newbie, to knowing what to do to get this right, and not screw up a touchy system further. Thanks!
Not a full answer, but more resources that have been helpful:
http://blog.endpoint.com/2008/12/best-practices-for-cron.html
I am slowly going through this, and trying to implement each of the points. I hadn't thought to google 'best practices cron' til after my post. :P
For version control, I'm just going to use RCS in the meantime, as I edit scripts on a file-by-file basis, but I've been advised to get Git set up (or Mercurial if I was on a Windows system).
This actually sounds great:
http://everythingsysadmin.com/2010/09/xed-202-released.html
"xed is a perl script that locks a file, runs $EDITOR on the file, then unlocks it."...and puts it in RCS if it wasn't already.
Completely brainless version control. If I get my head around bash, I'd like to create an editing shortcut that automatically commits to whichever version control system I use.
Other tips I received from an System Admin,
Dates: Rather than using say, date, or --date="last monday", use a fixed date and add a day/week etc to it each time it runs (if not more than current day obviously), because then if the script doesn't run, I can just re-run the script repeatedly until it catches up. Ah!
(And, this might sound obvious, but heaps of the reports I'll be eventually edit, don't say prominently what dates the report is running for. Will fix.)
And was reassured I should try and get the cron emails as quiet as possible, so that I actually notice if there's an error email.
There are wrappers for better cron error reporting that I have not yet investigated, linked here: http://habilis.net/cronic/
Herculean task ahead of you, best of luck. :)
I'd suggest finding all the tasks that run daily and shove them into their own scripts in /etc/cron.daily/. Same for weekly into /etc/cron.weekly, hourly, and monthly.
You might want to investigate use of anacron(8) for scheduling your jobs, if the machine won't always be online, but you still need some level of control over when the jobs are run. It's been the default cron-helper-tool for multiple distributions for a few years, so hopefully it's stable enough to rely on for your own tasks; but I could easily imagine that it might not perfectly meet your needs.
Faking the dates to scripts can be done with at least two packages on Ubuntu: datefudge and faketime. I have no experience with either, but both sound like they should be able to help. I hope you won't need it in the future. :)
Sorry, I know of no path-checker for bash scripts. It seems unlikely, since simple scripts are simple and easy to check by eye :) and complex scripts will be generating their pathnames at runtime anyhow. Maybe you could keep a database of pathnames used by each script and write a new script to verify that database regularly.
You could disable the cron email by setting MAILTO="". I'm not sure I like this. Maybe setting MAILTO to a logging-only account would help the deluge. Another option is getting really good at your procmail(1) rules so you can stuff them in another mailbox completely.
Getting good at mutt color or score controls can help you spot the wheat amongst the chaff. (color index red black ERROR or similar commands might help you spot the problems more quickly.)
Related
Instead of recording my own keystrokes to create an asciinema tutorial I do want to script them, so I can easily improve my tutorial over time without being being stressed about redoing it each time I want to make a change, or spending a lot of time trying to correct typos in the recording.
Mainly I want to be able to do somethign like:
type: "df"
wait: 2s
type: "echo foo"
This would enable me to easily rebuild the recording from scratch when I change the script.
How can I do this?
I've searched for a lot of solutions for my own projects, and this is the solution that I've come up with:
My solution:
What I'm planning on doing for a project I'm in is to use both asciinema as well as demo-magic.sh.
Specifically, I'll be using asciinema for the recording with the -c flag (see here), and demo-magic.sh for the automated typing, since it supports waiting for commands to finish executing (or not) and custom wait times on top of that. The command would look something like asciinema rec -c "./mydemoscript.sh" myrecording. See the projects for proper usage.
Other things I've come across:
doitlive -- I'm not using this because it's more of a fake-typing automated thing where you actually have to "type like a madman", which could be useful if you're doing a talk in front of an audience but you don't want to mess up your keystrokes, or if you don't want to forget to do certain commands.
asciiscript -- it's written in Go and it works, but you need to compile it yourself, and it also doesn't support waiting for the previous command to finish.
spielbash -- a Ruby project designed specifically to automate asciinema recordings using tmux. I'm not using this, though, because it's not as portable since you need an existing Ruby installation, and more importantly, it keeps corrupting the active console I'm in, and there's also unpredictable corruption in the recordings. Also, the project hasn't been committed to since 2019.
termscript -- it just doesn't work for me, and it freezes my console. It runs on python 2.
UPDATE: I have recently come across terminalizer. It seems to provide support for an window-like border around the actual terminal, and it has a built-in GIF renderer, but I have not tried it.
There is a project svg-term which allows to record a screencast from a command, and I've seen a script that simulates commands being typed letter after letter.
For the moment I ended up using AppleScript to automate iTerm but this approach is buggy and locks you to MacOS and iTerm... and if you happen do to something during recording, you are doomed.
I am still looking for a better approach.
TL;DR: asciinema-automation should do what OP wants. Its dependencies are asciinema and pexpect.
(disclaimer: I developed it for my own usage)
Some time ago, I was also looking for a way to automate asciinema recordings and I saw the very nice answer of a-la-linuques. I chose asciiscript because it used asciinema, but it is not maintained anymore. I tried to keep on using it via a fork adding new small features, but being not very familiar with go, I finally decided to rewrite everything in python in this repo.
It reads bash files, where comments can give special instruction like adding time between command or key stroke, or wait for an expected output (see examples). Of course, this is very much in the spirit of asciiscript.
I need to build a file/folder tree with associated file icons and special locations like network computers.
Currently I'm using Shell API to achieve it: SHGetFileInfo, IShellFolder.EnumObjects and other functions.
It works fine most of the time, but occasionally, on customer's machines it causes various errors like random access violations deep in system libraries. Analyzing bug reports, some of those seem to be a result of 3rd party shell extensions which are loaded to my app's address space when the Shell API is used.
I'm thinking to somehow avoid using Shell API and do the job another way. What are the other good approaches to build a folder tree?
If the problem really is due to faulty shell extensions then the only sensible approach, in my view, is to remove those shell extensions. Trying to work with the shell, but avoid using the shell API won't lead anywhere useful. In fact I think that the likely outcome is that your alternative code will be less functional. All for the sake of one user that won't fix their broken machine. That's a terrible trade off.
If explorer is also crashing then that is a clear indication that the problem is indeed due to shell extensions.
Having said all of that, you post makes me suspect that you have had bug reports from multiple clients. That makes your diagnosis much less plausible. The shell API is a complex beast and it is very plausible that your code is defective in some way. I suspect that you may be guilty of a case of diagnosis by wishful thinking. It's very easy, when facing a fault that is hard to reproduce and diagnose, to believe that your code is not to blame. If multiple clients are reporting problems then my bet is that the defect can be found in your code.
By 'challenging development environment' I don't mean you're on a small boat that's rocking up and down and someone is holding a gun to your head. I mean, are the tools at your disposal making the problem difficult?
Development is typically a cycle of code, run, observe the result, repeat. In some environments this is a very quick and painless process, but in others it's very difficult. We end up using little tricks to help us observe the result and run the code faster.
I was thinking of this because I just started using SSIS (an ETL tool included with SQL Server 2005/8). It's been quite challenging for me to make progress, mainly because there's no guidance on what all the dialogs mean and also because the errors are very cryptic and most of the time don't really tell you what the problem is.
I think the easiest environment I've had to work in was VB6 because there you can edit code while the application is running and it will continue running with your new code! You don't even have to run it again. This can save you a lot of time. Surprisingly, in Netbeans with Java code, you can do the same. It steps out of the method and re-runs the method with the new code.
In SQL Server 2000 when there is an error in a trigger you get no stack trace, which can make it really tricky to locate where the problem occurred since an insert can have a cascading effect and trigger many triggers. In Oracle you get a very nice little stack trace with line numbers so resolving the problem is very easy.
Some of the things that I see really help in locating problems:
Good error messages when things go wrong.
Providing a stack trace when a problem occurs.
Debug environment where you can pause, then output the value of variables and step to follow the execution path.
A graphical debug environment that shows the code as it's running.
A screen that can show the current values of variables so you can print to them.
Ability to turn on debug logging on a production system.
What is the worst you've seen and what can be done to get around these limitations?
EDIT
I really didn't intend for this to be flame bait. I'm really just looking for ideas to improve systems so that if I'm creating something I'll think about these things and not contribute to people's problems. I'm also looking for creative ways around these limitations that I can use if I find myself in this position.
I was working on making modifications to Magento for a client. There is very little information on how the Magento system is organized. There are hundreds of folders and files, and there are at least a thousand view files. There was little support available from Magento forums, and I suspect the main reason for this lack of information is because the creators of Magento want you to pay them to become a certified Magento developer. Also, at that time last year there was no StackOverflow :)
My first task was to figure out how the database schema worked and which table stored some attributes I was looking for. There are over 300 tables in Magento, and I couldn't find out how the SQL queries were being done. So I had just one option...
I exported the entire database (300+ tables, and at least 20,000 lines of SQL code) into a .sql file using PhpMyAdmin, and I 'committed' this file into the subversion repositry. Then, I made some changes to the database using the Magento administration panel, and redownloaded the .sql file. Then, I ran a DIFF using TortioseSvn, and scrolled through the 20k+ lines file to find which lines had changed, LOL. As crazy as it sounds, it did work, and I was able to figure out which tables I needed to access.
My 2nd problem was, because of the crazy directory structure, I had to ftp to about 3 folders at the same time for trivial changes. So I had to keep 3 windows of my ftp program open, switch between them and ftp each time.
The 3rd problem was figuring out how the url mapping worked and where some of the code I wanted was being stored. Here, by sheer luck, I managed to find the Model class I was looking for.
Mostly by sheer luck and other similar crazy adventures I managed to work my way through and complete the project. Since then, StackOverflow was started and by a helpful answer to this bounty question I was able to finally get enough information about Magento that I can do future projects in a less crazy manner (hopefully).
Try keypunching your card deck in Fortran, complete with IBM JCL (Job Control Language), handing it in at the data center window, coming back the next morning and getting an inch-thick stack of printer paper with the hex dump of your crash, and a list of the charges to your account.
Grows hair on your fingernails.
I guess that was an improvement on the prior method of sitting at the console, toggling switches and reading the lights.
Occam on a 400x transputer network. As there was only one transputer that could output to console debugging was a nightmare. Had to build a test harness on a Sun network.
I took a class once, that was loosely based on SICP, except it was taught in Dylan rather than Scheme. Actually, it was in the old Dylan syntax, the prefix one that was based on Scheme. But because there were no interpreters for that old version of Dylan, the professor wrote one. In Java. As an applet. Which meant that it had no access to the filesystem; you had to write all of your code in a separate text editor, and then paste it into the Dylan interpreter. Oh, and it had no debugging facilities, of course. And being a Dylan interpreter written in Java, and this was back in 2000, it was ungodly slow.
Print statement debugging, lots of copying and pasting, and an awful lot of cursing at the interpreter were involved.
Back in the 90's, I was developing applications in Clipper, a compilable dBase-like language. I don't remember if it came with a debugger, we often used a 3rd-party debugger called 'Mr Debug' (really!). Although Clipper was fast, some of our more intensive routines were written in C. If you prayed to the correct gods and uttered the necessary incantations, you could use Microsoft's CodeView debugger to debug the C code. But usually not for more than a few minutes, as the program usually didn't like to spend much time running with CodeView (usually memory problems).
I had a series of makefile switches that I used to stub out the sections of code that I didn't need to debug at the time. My debugging environment was very sparse so there was as much free memory as possible for the program. I also think I drank a lot more back then...
Some years ago I reverse engineered game copy protections. Because the protections was written in C or C++ they were fairly easy to disassemble and understand what was going on. But in some cases it got hairy when the copy protection took a detour into the kernel to obfuscate what was happening. A few of them also started to use of custom made virtual machines to make the problem less understandable. I spent hours writing hooks and debuggers to be able to trace into them. The environment really offered a competetive and innovative mind. I had everything at my disposal save time. Misstakes caused reboots and very little feedback what went wrong. I realized thinking before acting is often a better solution.
Today I dispise debuggers. If the problem is in code visible to me I find it easiest to use verbose logging. (Sometimes the error is not understanding the interface/environment then debuggers are good.) I have also realized time is of an essance. You need to have a good working environment with possibility to test your code instantly. If you compiler takes 15 sec, your environment takes 20 sec to update or your caches takes 5 minutes to clear find another way to test your code. Progress keeps me motivated and without a good working environment I get bored, angry and frustrated.
The last job I had I was a Sitecore Developer. Bugfixing can be very painful if the bug only occurs on the client's system, and they do not have Visual Studio installed on the system, with the remote debugging off, and the problem only happens on the production server (not the staging server).
The worst in recent memory was developing SSRS reports using Dundas controls. We were doing quite a bit with the grids which required coding. The pain was the bugginess of the controls, and the lack of debugging support.
I never got around the limitations, but just worked through them.
I write many scripts at home and on the job. Most of the time the scripts get used only a few times to accomplish their chosen task and then are never used again. However, sometimes I write a script to do something more complicated, something that requires user input. It is at this point that I usually agonize over whether to implement a GUI or stick with a y/n, press 1-10, etc. command-line interface. This type of interface can become tedious to use and difficult to maintain.
I know some things lend themselves to a GUI more than others, such as selecting things in a giant list. However, the time it takes to switch a command-line application to use a GUI is prohibitive. For me, it takes a good amount of time to add a GUI with even the most simple framework I can find.
I am curious if any developers have a method of determining at what point their script has grown enough to need a GUI. Or am I going about this the wrong way, should I always be writing my scripts assuming I might later add a GUI?
This doesn't answer your question but FWIW an intermediate step, between UI and command-line, is to have a configuration file instead of a UI:
Edit the configuration file
Run the program
A configuration file format can, if necessary, be complicated and well-commented.
As with many questions of this type, the answer is that it depends.
If your program/script does just one single thing by receiving a number of inputs from the user, it is better to stick with the non-GUI mode.
If the application is doing more than one thing and if you think that the user will use the application to do a lot of stuff, you may consider using a GUI.
Are you planning to distribute this program to others? Then it is better to provide a GUI.
If the users are non-technical, a GUI is a must!
Thats it.
When you want to hand your stuff over to someone else in a discoverable way. Command-line scripts are awesome because they are simple and elegant, but they are not very discoverable. That is, if you were to hand your scripts over to someone else with no documentation, would they be able to figure out what they are and how to use them? If your tasks are so simple that myscript /? will explain what you need to do fully, then you don't need a GUI.
If on the other hand, you are handing your scripts over to someone who isn't so technical, or needs some more visual guidance about the task to be done, than by all means, a GUI is a good way to go. You might even want to keep your scripts as they are and just create a separate GUI that runs them for maximum flexibility.
I think this decission also depends on the audience who will be using your script: If it is people who are comfortable working with the command line, then there is not pressing need to add a GUI as long as your script has a good /help which explains all the parameters it accepts. But if you want the "average user" to be able to use your program, I'd rather add a GUI because otherwise your program might not be intuitive enough for that user group.
If you only need some "Dialogs" to improve your scripts, you can use KDE Kdialog or Gnome Zenity.
I can't count the number of times I've written what I thought would be a 'one-off' and it became more useful than I thought and ended up writing a GUI for it, or I've need to come back to use a program months later. The advantage of the GUI is it makes it easier to remember what would otherwise likely be command line arguments. I.e. for flags and options you can simply use check boxes, combo boxes, radio buttons, and file selectors filenames. I use Borland C++ RAD so it is quite quick and easy to throw together a simple (or even not so simple) dialog box. I now often start with creating the GUI.
If you use Linux, try Zenity. It's an easy to use tool to make a GUI for command-line programs.
I work in a very small shop (2 people), and since I started a few months back we have been relying on Windows Scheduled tasks. Finally, I've decided I've had enough grief with some of its inabilities such as
No logs that I can find except on a domain level (inaccessible to machine admins who aren't domain admins)
No alerting mechanism (e-mail, for one) when the job fails.
Once again, we are a small shop. I'm looking to do the analogous scheduling system upgrade than I'm doing with source control (VSS --> Subversion). I'm looking for suggestions of systems that
Are able to do the two things outlined above
Have been community-tested. I'd love to be a guinae pig for exciting software, but job scheduling is not my day job.
Ability to remotely manage jobs a plus
Free a plus. Cheap is okay, but I have very little interest in going through a full blown sales pitch with 7 power point presentations.
Built-in ability to run common tasks besides .EXE's a (minor) plus (run an assembly by name, run an Excel macro by name a plus, run a database stored procedure, etc.).
I think you can look at :
http://www.visualcron.com/
Consider Cygwin and its version of "cron". It meets requirements #1 thru 4 (though without a nice UI for #3.)
Apologize for kicking up the dust here on a very old thread. But I couldn't disagree more with what's been presented here.
Scheduled tasks in Windows are AWESOME (a %^#% load better than writing services I might add). Yes, not without limitations. But still extremely powerful. I rely on them in earnest for a variety of different things.
If you even have a slight grasp on c# you can write as custom "task" (essentially a console application) to do, well, virtually anything. If persistent/accessible logging is what you're after, why not something like Serilog or NLog? Even at the time of writing, it had a very robust feature set. This tool in and of itself, in conjunction with some c#, could've solved both your problems very easily.
Perhaps I'm missing the point, but it seems to me that this isn't really a problem. At least not anymore...
If you're looking for a free tool there is plenty of implementations for the popular Cron tool for Windows, for example CRONw. It's pretty easy to configure and maintain. You could easily write add custom WSH scripts to send your emails and add log entries.
If you're going commercial way BMC Control-M is arguably one of the best but I don't believe that it is particularly cheap.
You may also consider some upcoming packages like JobScheduler
Pretty old question, but we use Jenkins. Yes its main purpose is for CI\CD, but its also a really nice UI for CRON with a ton of plugins and integrations.