I'm trying to write a script that handles application deployment, based off new builds. Unfortunately, our software (java/weblogic) doesn't have a very reliable "shutdown" command, so we're more or less forced to just kill the running processes before deploying the new code to the environment location.
What I would like to be able to do is write a script that can either (a) determine what PIDs exist for a specific directory, or (b) list all processes and parse the directories from that output.
So far, I've only made slight progress with (b), in that I can wmic process list and get everything in the world printed. Is there a way to narrow this list down? I searched but couldn't find flags that would allow me to request just the PIDs and directories they exist in. I could use the current output, since it looks like directories are printed everywhere, but this seems super inefficient.
Related
I would like to improve an existing Makefile, so it prints out the number of warnings and/or errors that were encountered during the build process.
My basic idea is that there must be a way to pipe the output to grep and have the number of occurences of a certain string in either stderr or stdout stream (i.e. "Warning:") stored into a variable that can then simply be echo'ed out at the end make command.
Requirements / Challenges:
Current console output and exit code must remain exactly the same
That means also without even changing control characters. Dev's using the MakeFile must not recognize any difference to what the output was prior to my change (except for a nice, additional Warning count output at the end of the make process). Any approaches with tee i tried so far were not successful, as the color coding of stderr messages in the console is lost, changing them to all black & white.
Must be system-independent
The project is currently being built by Win/OSX/Linux devs and thus needs to work with standard tools available out-of-the-box in most *nix / CygWin shells. Introducing another dependency such as unbuffer is not an option.
It must be stable and free of side-effects (see also 5.)
If the make process is interrupted (i.e. by the user pressing CTRL+C or for any other reason), there should be no side-effects (such as having an orphaned log file of the output left behind on disk)
(Bonus) It should be efficient
The amount of output may get >1MB and more, so just piping to a file and greping it will be a small performance hit and also there will be additional the disk I/O (thus unnecessarily slowing down the build). I'm simply wondering if this cannot be done w/o a temp file as i understand pipes as sort of "streams" that just need to be analysed as the flow through.
(Bonus) Make it locale-independent w/o changing the current output
Depending on the current locale, the string to grep and count is localized differently, i.e. "Warning:" (en_US.utf8) or "Warnung:" (de_DE.utf8). Surely i could have locale switch to en_US in the Makefile, but that would change console output for users (Hence breaking requirement 1.), so i'd like to know if there's any (efficient) approach you could think of for this.
At the end of the day, i'd be able to do with a solid solution that just fullfills requirement 1. + 2.
If 3. to 5. are not possible to be done then i'd have to convince the project maintainers to have some changes to .gitignore, have the build process slightly take up more time and resources, and/or fix the make output to english only but i assume they will agree that would be worth it.
Current solution
The best i have so far is:
script -eqc "make" make.log && WARNING_COUNT=$(grep -i --count "Warning:" make.log)" && rm make.log || rm make.log
That fulfills my requirements 1, 2 and almost no. 3: still, if the machine has a power-outage while running the command, make.log will remain as an unwanted artifact. Also the repetition of rm make.log looks ugly.
So i'm open on alternative approaches and improvements by anybody. Thanks in advance.
At some step of my software building automatization, which I attempt to implement using GNU make Makefiles, I run into the case of not only having targets a requirement being source files, but as a sort of different type of requirement I would like the target to depend on another software is started and hence exist as an operation system process.
Such a program could be background process but also a foreground process such as a Webbrowser which running a HTML5 application, which might play a role in a building process by for instance interacting with files it is fed through the building process.
I would hence like to write a rule somewhat like this:
.PHONY: firefoxprocess
Html5DataResultFile: HTML5DataSourceFile firefoxprocess
cp HTML5DataSourceFile folder/checked/by/html5app/
waitforHtml5DataResultFile
firefoxprocess:
/usr/bin/firefox file://url/to/html5app &
As seen I have taken the idea that .PHONY targets are somewhat non-file targets and hence would allow for requirering a process to be started?
Yet I a unsure if that is right. The documentation of GNU make is excellent and quite large and I am unsure understood it completely. To the best of my knowledge the documentation did not really report on the use of processes being used in rules, which motivates the question here.
My feeling has been that pidfiles are somewhat a link between processes and files, but they come with several problems (i.e. race conditions, uniqueness etc)
Sometimes a Makefile dependency tree includes elements that aren't naturally or necessarily time-dependent files. There are two answers:
create a file to represent the step, or
just do the work "in line" as part of the step.
The second option is usually easiest. For instance, if a target file is to be created in a directory that might not exist yet, you don't want to make the directory name itself a dependency, because that would cause the file to be out of date whenever the directory changed. Instead, I do:
d/foo:
#test -d d || mkdir -p d
...
In your case, you could something similar; you just need a way to test for a running instance of firefox, and to be able to start it. Something like this might do:
Html5DataResultFile: HTML5DataSourceFile
pgrep firefox || { /usr/bin/firefox && sleep 5; }
cp HTML5DataSourceFile folder/checked/by/html5app/
waitforHtml5DataResultFile
The sleep call just lets FF initialize, because it might not be ready to do anything the instant it returns.
The problem with option #1 in your case is that it's undependable and a little circular. Firefox won't reliably remove the pidfile if the process dies. If it does successfully remove the file when it exits, and re-creates it when it restarts, you have a new problem: the timestamp on the file spuriously defines any dependencies as out of date, when in fact the restarted process hasn't invalidated them.
I can't think of a way of doing what I am trying to do and hoping for a little advice. I am working with data on a computing cluster, and would like to process individual files on separate computing nodes. The workflow I have right now is something like the following:
**file1.py**
Get files, parameters, other info from user
Then Call: file2.sh
**file2.sh**
Submit file3.py to computing node
**file3.py**
Process input file with parameters given
What I am trying to do is call file2.sh and pass it each input data file one at a time so that there are multiple instances of file3.py running, one per file. Is there a good way to do this?
I suppose that the root of the problem is that if i were to iterate through a list of input files in file1.py I don't know how to then pass that information to file2.sh and then on to file3.py.
From this description, I'd say the the straightforward way is to call file2.sh directly from Python.
status, result = commands.getstatusoutput("file2.sh" + arg_string)
Is that enough of a start to get you moving? Are the nodes conversant enough for one to launch a command directly on another? If not, you may want to consider looking up "interprocess communication" on Linux. If they're not even on the same Internet node, you'll likely need REST commands (post and get operations), from whence things grow more overhead.
I need to run a ruby script for one week and check whether it is running for every hour.
Could you please suggest me some way? I need to check this in windows machine.
For ex:- I have script called one_week_script.rb which will run for one week, in between i want to check whether the script is running or not? if it is not running, then running that script from another script
A typical solution is to use a "heartbeat" strategy. The "to be monitored" notifies a "watchdog" process on a regular interval. A simple way of doing this might be to update the contents of some file every so often, and the watchdog simply checks that same file to see if it's got recent data.
The alternative, simply checking if the process is still 'loaded' has some weaknesses, The program could be locked up, even though it's still apparently 'running'. Using the heartbeat/watchdog style means you know that the watched process is operating normally, because you're getting feedback from it.
In a typical scenario, you might just write the current time, and some arbitrary diagnostic data, say the number of bytes processed (whatever that might mean for you).
Want to upgrade my file management productivity by replacing 2 panel file manager with command line (bash or cygwin). Can commandline give same speed? Please advise a guru way of how to do e.g. copy of some file in directory A to the directory B. Is it heavy use of pushd/popd? Or creation of links to most often used directories? What are the best practices and a day-to-day routine to manage files of a command line master?
Can commandline give same speed?
My experience is that commandline copying is significantly faster (especially in the Windows environment). Of course the basic laws of physics still apply, a file that is 1000 times bigger than a file that copies in 1 second will still take 1000 seconds to copy.
..(howto) copy of some file in directory A to the directory B.
Because I often have 5-10 projects that use similar directory structures, I set up variables for each subdir using a naming convention :
project=NewMatch
NM_scripts=${project}/scripts
NM_data=${project}/data
NM_logs=${project}/logs
NM_cfg=${project}/cfg
proj2=AlternateMatch
altM_scripts=${proj2}/scripts
altM_data=${proj2}/data
altM_logs=${proj2}/logs
altM_cfg=${proj2}/cfg
You can make this sort of thing as spartan or baroque as needed to match your theory of living/programming.
Then you can easily copy the cfg from 1 project to another
cp -p $NM_cfg/*.cfg ${altM_cfg}
Is it heavy use of pushd/popd?
Some people seem to really like that. You can try it and see what you thing.
Or creation of links to most often used directories?
Links to dirs are, in my experience used more for software development where a source code is expecting a certain set of dir names, and your installation has different names. Then making links to supply the dir paths expected is helpful. For production data, is just one more thing that can get messed up, or blow up. That's not always true, maybe you'll have a really good reason to have links, but I wouldn't start out that way, just because it is possible to do.
What are the best practices and a day-to-day routine to manage files of a command line master?
( Per above, use standardized directory structure for all projects.
Have scripts save any small files to a directory your dept keeps in the /tmp dir, .
i.e /tmp/MyDeptsTmpFile (named to fit your local conventions) )
It depends. If you're talking about data and logfiles, dated fileNames can save you a lot of time. I recommend dateFmts like YYYYMMDD(_HHMMSS) if you need the extra resolution.
Dated logfiles are very handy, when a current process seems like it is taking a long time, you can look at the log file from a week ago and quantify exactly how long this process took, a week, month, 6 months (up to how much space you can afford). LogFiles should also capture all STDERR messages, so you never have to re-run a bombed program just to see what the error message was.
This is Linux/Unix you're using, right? Read the man page for the cp cmd installed on your machine. I recommend using an alias like alias CP='/bin/cp -pi' so you always copy a file with the same permissions and with the original files' time stamp. Then it is easy to use /bin/ls -ltr to see a sorted list of files with the most recent files showing up at the bottom of the list. (No need to scroll back to the top, when you sort by time,reverse). Also the '-i' option will warn you that you are going to overwrite a file, and this has saved me more than a couple of times.
I hope this helps.
P.S. as you appear to be a new user, if you get an answer that helps you please remember to mark it as accepted, and/or give it a + (or -) as a useful answer.