I did the following:
nohup find / &
rm nohup.out
Oddly, the nohup -command continued to run. I awaited for a new file to be created. For my surprise there was no such file. Where did the stdout of the command go?
Removing a file in UNIX does two things:
it removes the directory entry for it.
if no processes have it open and no other directory entries point to it (hard links), it releases the space.
Your nohupped process will gladly continue to write to the file that used to be called nohup.out, but is now known as nothing but a file descriptor within that process.
You can even have another process create a nohup.out, it won't interfere with the first.
When all hard links are gone, and all processes have closed it, the disk space will be recovered.
if you will delete the nohup.out file, the handle will be lost and it will write only to the file descriptor but if you want to clean out the nohup.out file then just run this
true > nohup.out
it will delete the contents of the file but not the file.
That's standard behaviour on Unix.
When you removed the file, the last link in the file system was removed, but the file was still open and therefore the output of find (in this case) was written to disk blocks in the kernel buffer pool, and possibly even to disk. But the file had no name. When find exited, no process or file (inode) referenced the file, so the space was released. This is one way that temporary files that will vanish when a program exits are created - by opening the file and then removing it. (This presumes you do not need a name for the file; clearly, if you need a name for the temporary, this technique won't work.)
cat /dev/null > nohup.out
from here
Related
I am using fswatch to monitor a directory and run a script when video files are copied into that directory:
fswatch -o /Path/To/Directory/Directory | xargs -n 1 sh /Path/To/Script/Script.sh
The problem is that the file is often not completed its copy before the script is actioned. The files are video files of varying size. Small files are OK, larger files are not.
How can I delay the fswatch notification until the file has completed its copy?
First of all, the behaviour of the fswatch "monitors" is OS-specific: when asking question about fswatch you'd better specify the OS you use.
However, there's no way to do that using fswatch alone. A process may open a file for writing and keep it open for an amount of time sufficiently long for the OS to send multiple events. I'm afraid there is nothing fswatch can do about it.
An alternate approach may be using another tool to check whether the modified file is currently open: if it is not, then run your script, otherwise skip it and wait for its next event. Such tools are OS-specific: in OS X and Linux you may use lsof. Beware this approach does not protect you from another process opening that file while your script is running.
I am new to bash scripting and I have to create a script that will run on all computers within my group at work (so it's not just checking one computer). We have a spreadsheet that keeps certain file information, and I am working to automate the updating of that spreadsheet. I already have an existing python script that gathers the information needed and writes to the spreadsheet.
What I need is a bash script (cron job, maybe?) that is activated anytime a user deletes a file that matches a certain extension within the specified file path. The script should hold on to the file name before it is completely deleted. I don't need any other information besides the name.
Does anyone have any suggestions for where I should begin with this? I've searched a bit but not found anything useful yet.
It would be something like:
for folders and files in path:
if file ends in .txt and is being deleted:
save file name
To save the name of every file .txt deleted in some directory path or any of its subdirectories, run:
inotifywait -m -e delete --format "%w%f" -r "path" 2>stderr.log | grep '\.txt$' >>logfile
Explanation:
-m tells inotifywait to keep running. The default is to exit after the first event
-e delete tells inotifywait to only report on file delete events.
--format "%w%f" tells inotifywait to print only the name of the deleted file
path is the target directory to watch.
-r tells inotifywait to monitor subdirectories of path recursively.
2>stderr.log tells the shell to save stderr output to a file named stderr.log. As long as things are working properly, you may ignore this file.
>>logfile tells the shell to redirect all output to the file logfile. If you leave this part off, output will be directed to stdout and you can watch in real time as files are deleted.
grep '\.txt$' limits the output to files with .txt extensions.
Mac OSX
Similar programs are available for OSX. See "Is there a command like “watch” or “inotifywait” on the Mac?".
I have a background process that I do not want to restart. Its output is actively being logged to a file.
nohup mycommand 1> myoutputfile.log 2>&1 &
I want to "archive" the file the process is currently writing its output to, and make it start writing to a blank file at the same file name. I must be able to do this without having to kill the process and start it again.
I tried simply renaming the existing file (to myoutputfile_.log), hoping that the shell now finding that the file is no longer there, will create a new file with the original file name (myoutputfile.log). But this does not work as the shell holds a reference to the file's location and keeps appending to it.
I looked here. On executing ls, I see that the streams are now marked as (deleted) but I'm quite confused what to do next. In the gdb command, do I have to specify the process executable in addition to the process ID? What happens if I don't specify it or I get it wrong? Once in gdb, how do I force the stream to re-create a file in the deleted file's same location (same path and filename)?
How can I use the commands in shell to signal it to start a new file for an existing process's output redirection?
PS: I can't do a trial-and-error because it's rather important I get this right. If it is relevant to know, this is a java process.
I resolved this issue by doing the following:
cp myoutputfile.log myoutputfile_.log; echo > myoutputfile.log
This essentially reset the log file after copying the original contents to a new file.
#!/bin/bash
python /home/sites/myapp/Main.py &> /home/sites/myapp/logs/init.log &
This script produces a log of approximately 1G/week.
When i manually delete the init.log during runtime and not restarted the script it still save the data in a missing init.log. init.log will be visible again when the script is restarted.
Does restarting the script the only way to see the log?
On unix system When init.log is created then an inode is created. There is a counter in every inodes which counts all the references to that file. A reference means a hard link or when a file is opened. The file is only deleted when this counter goes back to zero.
So when stdout is redirected to init.log its inode has the counter value 2 (referenced by the directory entry, and counted because of open. When rm (uses unlink function) deletes the file, this counter become 1, so file is not referenced by any directory entry, but the inode still exists. When the script finishes, the counter become 0 and the inode is deleted.
There is no easy way to read an inode, which not referenced by any directory entry.
A file is not actually deleted until nothing references it. In this case, you have deleted all directory entries, but your program still has an open file descriptor so the data is not completely removed until the program exits. Note that it will also continue to hog disk space.
In Linux, you can still view the file's contents in /proc/PID/fd/FD where PID is the process's id and FD is the file descriptor that your are interested in. Once the program exits, the data is toast and the disk space can be reclaimed... so get your data while you can ;)
You shouldn't remove a log file if you know that a program still has open file descriptors on it. Instead, truncate the file with cat /dev/null > log.file or in bash just use > log.file.
I have created a script which pick the files from a directory inbox. I have to include a handling in the script whether files are coming from another process or not. if files are coming through another process then my script should wait until files are copied.
For this i have create a flag
CHECK_COPY_PROCESS=$(ps -ef|grep -E 'cp|mv|scp'|grep inbox)
if flag CHECK_COPY_PROCESS contains some value then process will go to waiting state. but problem is that suppose some files are coming from sub directory of that directory inbox
then process will generate like this cp file_name .. above logic is not working.
You could use the tip of Basile, using lsof in conjunction with awk (grepping only on the first column, which is the command name)
Example:
lsof +D /path/to/inbox|awk '$1~/mv|cp|scp/'
This is not tested as I currently don't have big files that take a while to copy on my machine.