I'm using Google Cloud Platform and have a virtual machine. I'm also messing around with webscrapers.
I'm currently trying to do a simple scrape of reddit using a ruby script. That part works pretty well. It essentially continues down and down (to the end of reddit!) scrape the articles, though this, obviously takes some time.
Right now, in order to scrape (I'm running ruby scrape.rb > reddit.txt) I have to keep the google virtual machine ssh browser window open on my computer or the process will exit (which makes enough sense). However, what I'd like to do is have the process persist even if I close the window.
Is there a way to somehow have this process continue to run? Then I can periodically log in and check reddit.txt which will continue to grow even I'm not ssh'ed in.
Thanks!
This seems to be the answer.
The command nohup is exactly what I was looking for!
Related
Every now and then, my sleeping disk wakes up, does what sounds like a single read, and then sits idle until it falls asleep again. Sometimes a program that I am using completely freezes for about 10 seconds while the disk spins up, even though that program doesn't seem to need to read from that drive.
Is there an api for listening to file accesses as they happen, or similar, so I can figure out what is read from that drive, so I can move it? If not on Windows, can I do this on Linux?
This is also applicable for figuring out what files/folders a program is accessing in general, so I wouldn't say it only applies to my very narrow problem.
There's a simple tool called What's My Computer Doing? that you can use to get a quick idea of what's causing activity on your computer.
Install and run it, and leave it running in the background. Once you use this tool to narrow down which process is causing the disk activity, you'll want a more comprehensive tool. I use Process Monitor from Sysinternals/Microsoft.
It can be a bit daunting at first, but that's mainly because it is so powerful. It can also alter the behavior of the computer. When it's running, it backs up the huge quantity of data it collects to the disk. So that's why I suggest using the 'What's My Computer Doing?' tool first. Once you know which process is generating the disk access you can add a new filter rule (keep all the defaults, as they mask out a bunch of normal system processes) and select "Process Name" "is" "process_name", or select "PID" "is" "actual_PID".
There are plenty of tutorials like this one that can help you get started with Process Monitor.
I built an application using openframeworks that is live 24/7 on a kiosk. Every now and then (every few weeks) it will randomly go unresponsive and I still can't get to the bottom of it because it's so random and infrequent it is hard to debug.
I wrote a ruby script that looks for the application running and if it doesn't exist it will start it up. This works in all cases where the application name doesn't show up in activity monitor. Meaning if the app crashes and completely force quits itself or something. It works just fine.
However, if the app just freezes and goes unresponsive (red highlight in activity monitor) the app doesn't quit out completely unless I force quit manually. Is there some kind of script I can write to look for all "unresponsive apps/processes" every few seconds and force quits them automatically? That way my app launcher script will be able to detect that the app isn't running and boot it up again.
I suggest you look at Monit because it's solid, well tested, well documented, and easy to learn.
If you still want to write your own monitoring script, Monit is a good example to follow.
The most reliable way to detect an unresponsive app is to have a "vital sign" which is a generic term for a signal that an app emit to prove it's healthy. Some people call this a "pulse" or "heartbeat" or "brainwave". Your external script watches the vital sign. If your external script sees the vital sign flatline, then the script takes action to cure the app or restart it.
An alternate way is to have a "ping" which is a generic term for your external script sending a signal to the app or the system, then listening for a reply. You can use tools such as the Unix ps command for processes, or AppleScript Activity Monitor. As far as I know, these do a pretty good job of catching common cases, but have trouble catching apps that are soaking up resources, such as being caught in endless loops.
I want to make a windows application whose GUI will be streamed to another device (allowing remote control). The point is that I'm not willing to rely on creating Windows Sessions to isolate the GUI I/O's (1)
To achieve this, I started observing some existing solutions that are able to enable remote access using this premise to see if I could get a clue about where to start.
One of these solution is Winflector (BTW: it is free up to 2 connections).
I got interested in this solution because it seems (I'm guessing) it detects only the repainted regions. What I took from my observations are that:
While the streamed application is "invisibly" running locally in the same session I'm logged in (it shows the application process in my task manager), the application window seems not to be created anyhow - at least Inspect can't get any window information/handle of the application process - It looks like sort of a "GUI StdOut Redirection".
Apparently, no additional Desktop is created;
Also apparently, no Mirror Driver is installed;
Using Process Explorer, I found out Winflector adds some thread's to the original application process. I suspect it is about the GUI redirection (by the thread's names);
The application is started by the Winflector server - that is, it has control about the CreateProcess arguments.
What is the most likely technique to be used in this case?
Windows Hook?
Windows Messages interception?
Special Display Driver?
Sort of Memory Device Context?
Where should I start researching to get a similar approach? Any open source project regarding this subject would also be very welcomed.
PS: By my programming experience, this is sort of a whole "new world" - sorry if my questions are redundant/obvious/non-sense.
(1) That is, this application could be spawned, streamed and interact
with the remote client using the same session which a local user is
already logged in, without conflicting the IO (like a regular VNC
would do, for example). PS: At this moment, I'm focusing only at the
output.
I am doing some analysis work on some software we are running where I work. The software seems to have memory issues some where along the line which are proving difficult to track down. We have decided to use Sysinternals VMMap to track the memory being used by the software.
We have VMMap exporting the usage every 20 seconds using Windows scheduler to launch a batch script which pulls back the target process PID and launches VMMap with it. The process runs for a while, output appearing the out directory but after a while it stops. Windows scheduler reports the job ran fine and will start another instance when the trigger is meant, once again with no output.
After a bit of investigation it looks like VMMap is failing to open the process and is trying to report an error through its GUI. Since we are running in batch, we cannot see this error to dismiss it. This is causing numerous process' to be spawned but not actually doing anything.
Has anyone come across this issue when using VMMap, or know of anything that may help? I am thinking there may be some flag I can pass which suppresses messages or maybe some way I can handle it in the batch but Google hasn't helped nor has the Sysinternals forum. Any help would be really appreciated.
VMMap is a GUI tool, so trying to capture its output in an automated way will be difficult. Instead, try using another SysInternals tool, Handle, that captures a lot of the same information, but exports/reports on it in command line, where it can be captured much easier. Alternatively, don't run the output in an auto-repeating way when using VMMap, but instead have your script somehow detect the error or missing expected results/data and stop so the GUI output can be examined.
All Sysinternals tools do pop up a consent dialog for the first time they are started on a new machine to accept their license. I think you did deploy the tool to a production machine and it was trying to show the consent dialog but nobody did press ok.
They do basically create a registry key on the machine which you can fake if you need a fully automated deployement or you can start in once on the target machine for the user in question.
I need to find out exactly what files/directories a Lua program uses so I can try to only pack what it needs into a ZIP file, and come up with a simple way to deploy this script.
I used SysInternals' Process Monitor, but I'm surprised by the small amount of information it returned while it watched the program (For Lua users out there, it's wsapi.exe, which is the launcher for the light-weight Xavante web server).
Does someone know of a good Windows application that can completely monitor what a program does, eg. something like a live version of the venerable PCMag's InCtrl5.
Thank you.
Process monitor will catch everything. If it's not catching the action then it must be happening in a different process. Try filtering based on the files you expect to be used rather than the process you expect it to happen in.