We need to see the output (stdout) that is generated by the jupyter notebook in google colab. Doing some investigation it seems that the output is piped up all the way to the main process:
root 1 0 0 Jun27 ? 00:00:00 /bin/bash -e /datalab/run.sh
The pipeline that the output runs through seems to be as follows:
/usr/bin/python2 /usr/local/bin/jupyter-notebook .....
/tools/node/bin/node /datalab/web/app.js
node /tools/node/bin/forever ..... /datalab/web/app.js
/bin/bash -e /datalab/run.sh
Any ideas on how I could access it?
I just discovered that forever doesn't forward the output from app.js. forever list suggests that the output is going to /content/.forever/BQBW.log which doesn't exist. I still don't understand why nor if this is really where stdout ends up.
Use wurlitzer. Here's a full example:
https://colab.research.google.com/drive/1jpAOdWJDCh_YzmqidGnlYHHCFODNKQkB
This notebook:
Saves a C file that prints to stdout.
Compiles as a shared library.
Loads the shared library into the running Python backend.
Uses wurlitzer to capture the output when invoking the library.
(I realize you're unblocked, but leaving this answer here to hopefully help future travelers)
Wurlitzer uses a thread to flush its pipes, and I guess your C++ code was crashing before its pipe was flushed.
https://colab.research.google.com/drive/1i6x882Dn6E5PwaptVQ4ADGyEvBZAHm7i shows an example where TF C++ code emits dev placement to stderr and then is killed before execution completes. Flushing quickly results in all the output showing up before the kernel is killed, but leaving it at the default (0.2s) results in partial or no output showing up.
If the output you want is an assert/FATAL message right before process death, wurlitzer's approach is unlikely to work for you, and running as a subprocess is likely to be a faster iteration path, e.g. write the code you'd have in the cell out to a file e.g. using %%writefile and then run a subprocess python like:
!python3 file.py
Any stdout/stderr the subprocess emits (whether from python code writing to sys.std{out,err} or C++ code writing to fd={1,2}) should show up in the cell's output.
Is the output you're looking for coming from your code or from Jupyter itself?
If it's jupyter, it takes a little work to enable logging -- here's a full example: https://colab.research.google.com/drive/1q2mhsj4bwwdQK-KZIxrIIKed8O11MQl0
I ended up writing a c++ wrapper for cout and py::print that can be used to enable or disable python printing. Pretty disgusting given the fact that I needed to change my entire c++ source to use the wrapper instead of std::cout.
Related
I am starting my service running on a Raspberry Pi 2 (Raspbian) using a command in rc.local which looks like this:
python3.4 /home/pi/SwitchService/ServiceStart.py >/home/pi/SwitchService/log &
python3.4 /home/pi/test.py >/home/pi/log2 &
For some reason I don't see any text in the log file of my service although the script prints to stdout.
the two scripts look like this:
test.py
print("Test")
ServiceStart.py
from Server import Server
print("Test")
if __name__ == "__main__":
server = Server()
Because I couldn't get the bash solution to work I tried this other solution whether that works for me. It behaves exactly the same like the bash based method. So my service writes nothing to the log file although the empty file is created.
First, make sure that your script is actually running. Many schedulers and startup routines don't have PATH set, so it may not be finding python3.4. Try modifying the command to include the full path (e.g. /full/path/python3.4).
Secondly, it's not recommended to include long running scripts in rc.local without running them in the background (the documentation even states this). The Raspberry Pi waits for the commands to finish before continuing to boot, so if it runs forever, your Raspberry Pi may never finish booting.
Lastly, assuming the previous two issues have been taken care of, make sure that your program isn't buffering output too aggressively. You can try flushing stdout to see if that helps.
I need to start up a Golang web server and leave it running in the background from a bash script. If the script in question in syntactically correct (as it will be most of the time) this is simply a matter of issuing a
go run /path/to/index.go &
However, I have to allow for the possibility that index.go is somehow erroneous. I should explain that in Golang this for something as "trival" as importing a module that you then fail to use. In this case the go run /path/to/index.go bit will return an error message. In the terminal this would be something along the lines of
index.go:4:10: expected...
What I need to be able to do is to somehow change that command above so I can funnel any error messages into a file for examination at a later stage. I tried variants on go run /path/to/index.go >> errors.txt with the terminating & in different positions but to no avail.
I suspect that there is a bash way to do this by altering the priority of evaluation of the command via some judiciously used braces/brackets etc. However, that is way beyond my bash capabilities. I would be most obliged to anyone who might be able to help.
Update
A few minutes later... After a few more experiments I have found that this works
go run /path/to/index.go &> errors.txt &
Quite apart from the fact that I don't in fact understand why it works there remains the issue that it produces a 0 byte errors.txt file when the command goes to completion without Golang throwing up any error messages. Can someone shed light on what is going on and how it might be improved?
Taken from man bash.
Redirecting Standard Output and Standard Error
This construct allows both the standard output (file descriptor 1) and the standard error output (file descriptor 2) to be redirected to the file whose name is the expansion of word.
There are two formats for redirecting standard output and standard error:
&>word
and
>&word
Of the two forms, the first is preferred. This is semantically equivalent to
>word 2>&1
Appending Standard Output and Standard Error
This construct allows both the standard output (file descriptor 1) and the standard error output (file descriptor 2) to be appended to the file whose name is the expansion of word.
The format for appending standard output and standard error is:
&>>word
This is semantically equivalent to
>>word 2>&1
Narūnas K's answer covers why the &> redirection works.
The reason why the file is created anyway is because the shell creates the file before it even runs the command in question.
You can see this by trying no-such-command > file.out and seeing that even though the shell errors because no-such-command doesn't exist the file gets created (using &> on that test will get the shell's error in the file).
This is why you can't do things like sed 'pattern' file > file to edit a file in place.
If I create a process from a cmd prompt using the start command (opening a new cmd) is it possible to redirect the stdout and stderr from that process back to the calling cmd?
If you want the output of the STARTed process to appear in the parent command console, then simply use the START /B option.
If you want to process the output of your command, then you should use FOR /F ... in ('someCommand') DO ... instead.
OK. I have yet to find a straightforward answer to this question. I didn't want to bog down my question with what I thought unnecessary detail but seeing as I'm being criticized for the lack of this I'll expand a bit here.
I want to automate the updating of FWs on our production line, so I've a python app that gets the FWs from ftp and then uses the processors flash tool via python subprocess command to upload it to the board's flash. OK for all but one of the tools.
The one tool seems to have a problem when it's not running in its own terminal, so I provide a start to the subprocess command string which allows it to run OK in its own terminal. However, I need the output from this other terminal for logging reasons.
A possible solution was to log stdout and stderr to file using >> or wintee and then poll this file via a thread in my original app (perhaps a rather convoluted solution to the question). However the tool also has a separate problem where it doesn't like any std redirection, so this doesn't work for me.
I want to put my program's output into a file. I keyed in the following :
./prog > log 2>&1
But there is nothing in the file "log". I am using the Ubuntu 11.10 and the default shell is bash.
Anybody know the cause of this AND how I can debug this?
There are many possible causes:
The program reads the input from log file while you try to redirect into it with truncation (see Why doesn't "sort file1 > file1" work?)
The output is buffered so that you don't see data in the file until the output buffer is flushed. You can manually call fflush or output std::flush if using C++ I/O stream etc.
The program is smart enough and disables output if the output stream is not a terminal.
You look at the wrong file (i.e. in another directory).
You try to dump file's contents incorrectly.
Your program outputs '\0' as the first character so the output appears to be empty, even though there is some data.
Name your own.
Your best bet is to run this application under a debugger (like gdb) or use strace or ptrace (or both) and see what the program is doing. I mean, really, output redirection works for the last like 40 years, so the problem must be somewhere else.
I have a FORTRAN program output I want to redirect to file. I've done this before and use
$myprog.out>>out.txt 2>&1
and for some reason this is not working. I test it with a another simple test program
$myprog.test>>out.txt 2>&1
and it works
I run myprog.out and the output goes to screen as usual but redirecting it seems to fail! It was working and now seems to have stopped working. It's very strange. I commented out a few print statements that I no longer wanted, recompiled and then band redirect does not work.
There is clearly something different going on with my output but how to diagnose where it is going?
You probably need to flush your output. See for example this SO topic. How to do that depends on your compiler I guess. Because only Fortran 2003 Standard includes flush() statement and the ability to determine numbers that corresponds to stdout/stderr units.
However in gfortran (for example) you can use flush() intrinsic procedure with the equivalents of Unix file descriptors: UNIT=5 for stdin, UNIT=6 for stdout and UNIT=0 for stderr.
PROGRAM main
PRINT *, "Hello!"
CALL flush(6)
CALL flush(0)
END PROGRAM main
With >> you are appending the output of your program to out.txt every time you run it.
Can you just try scrolling to the end of out.txt and see if your output is there?