I'm trying to automate IPython scripts using batch files and the Windows Task Manager. After much digging I've found that you can use Task Manager to trigger a BAT and through that execute the nbconvert package in order to run the Jupyter Notebook without the interactive bits.
In the BAT my code first "turns the Windows cmd into an Anaconda Prompt" with the code: <anaconda3 path>\Scripts\activate.bat <anaconda3 path>
(No formal CS education, so bear with me on this.)
That line allows me to run conda and jupyter commands out of the Windows cmd. I then run
jupyter nbconvert --execute --ExecutePreprocessor.enabled=True --ExecutePreprocessor.timeout=60 <Notebook>
I keep receiving a timeout error when this occurs. Yes, I could adjust the timer, but before setting the timer to 10 hours and coming back in the morning I'm hoping for some understanding of what is going on behind the scenes in nbconvert, and maybe some insights into how to get this working. The .ipynb file works fine when run through Jupyter Notebook. There is however a point at which the script is attempting to load several gb of data, which has taken ~20 minutes each time I've run it.
To that end - when nbconvert engages the Pre-processors, are they running the entire script preemptively? I'm struggling to find a reason as to why nothing opens, pings, runs, updates, etc... when executing the nbconvert command. I would expect at least an uptick in RAM usage, but as far as I can tell that isn't happening either.
How can I work around this, and what is going on behind the scenes when nbconvert is executed?
Traceback:
C:\Users\User>jupyter nbconvert --execute "C:\Users\User\Desktop\_for_Python\CBSA.ipynb"
[NbConvertApp] Converting notebook C:\Users\User\Desktop\_for_Python\CBSA.ipynb to html
[NbConvertApp] Executing notebook with kernel: python3
[NbConvertApp] ERROR | Timeout waiting for execute reply (30s).
Traceback (most recent call last):
File "C:\Users\User\AppData\Local\Continuum\anaconda3\lib\site-packages\nbconvert\preprocessors\execute.py", line 423, in _wait_for_reply
msg = self.kc.shell_channel.get_msg(timeout=timeout)
File "C:\Users\User\AppData\Local\Continuum\anaconda3\lib\site-packages\jupyter_client\blocking\channels.py", line 57, in get_msg
raise Empty
_queue.Empty
For anyone else facing this error with a correct path, I was able to extend the timeout using the flag --ExecutePreprocessor.timeout=300
According to the documentation:
The timeout traitlet defines the maximum time (in seconds) each
notebook cell is allowed to run, if the execution takes longer an
exception will be raised. The default is 30 s, so in cases of
long-running cells you may want to specify an higher value.
Fixed by adding the Anaconda3/Library/Bin path to c/windows/system32 PATH. Timeout was occurring due to DLL error, being returned as Timeout in command line.
According to the documentation: "The timeout option can also be set to None or -1 to remove any restriction on execution time.". So that would be: --ExecutePreprocessor.timeout=-1
Related
A while a go we set up a post_commit hook script in TortoiseSVN, this is a simple batch file, and this worked fine. We are running Windows 10.
Recently (around the beginning of 2023) this stopped working. TortoiseSVN now reports:
The hook script returned an error:
The process cannot access the file because it is being used by another process.
The script executes fine when run manually.
Changing the script to execute to another batch file or a program, results in the same message.
Does anyone have any suggestions?
Thanks,
Chris
Edit:
The hook script is client side and is configured as:
Working Copy Path:
C:\Temp\Repo
Command Line To Execute:
C:\Users\chris\SVNFeeder\testrepo_sync.bat
Wait for script to finish: True
Hide the script while running: False
Always execute the script: True
The batch file contains the following:
java -jar %USERPROFILE%/SVNFeeder/SvnFeeder.jar -c %USERPROFILE%/SVNFeeder/testrepo_configFile
The java file updates our Jira server with commit messages.
Changing the command line to execute the java file directly results in the same error, even a simple batch file that only creates a folder doesn't execute.
There is nothing wrong with the batch file or the java file as both execute fine outside of TSVN.
I have just found out that Sophos was blocking the execution of the script.
There was no mention of this in the events page, which didn't help track it down.
I have read somewhere, that the commands entered in the run dialog box are executed as if they are executed using the start command in CMD. Which I believed was the case (as i could executed applications registered at app paths key just by their name using run).
But i noticed a little bit of deviation from this behavior when I tried to run a command on run without extensions. To illustrate this behavior:-
I added .msc and .MSC (i am unaware of extension case sensitivity/insensitivity on windows) to the PATHEXT environment variable, such that it became like this:-
PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.msc
After this I executed the following command in CMD, to start the Device Manager:-
start devmgmt
Which lead to execution of Device Manager.
But when i typed in the following command in run dialog box:-
devmgmt
it prompted me with an error stating:-
Well, If my statement in the first paragraph is correct, then why did I ended up with an error in the second case?
P.S.:- Upon googling about this issue before, i read several article which stated run prompt inherits PATHEXT values of CMD, for extension resolution. And if it doesn't, then there exists an policy (in gpedit or something) which makes the run prompt only possess the default PATHEXT values!! (correct me if i was wrong here)
Sometimes when I'm working on RStudio the command prompt > doesn't appear and I can't even save the script.
This time it happened after I was trying to a load a package.
How can I get over this?
I'm trying to schedule in task scheduler:
Run a batch file
Activate a conda environment
Run a python program
Exit command prompt
Everything works fine, except the python.exe window will remain open while the command prompt closes.
My batch file: (the sleep is for the python code to run. It takes a few seconds)
call activate python2
start C:\Users\Chris\Anaconda3\envs\python2\python.exe testtest.py
sleep 30
exit
My python script:
driver = webdriver.Chrome(executable_path="C:\path\to\chromedriver")
driver.get('http://website.com')
# Find email and pw fields and then fill them in
email = driver.find_element_by_id("user_email")
email.send_keys('myemail#email.com')
pw = driver.find_element_by_id("user_password")
pw.send_keys('password')
# Click on sign-in button
driver.find_element_by_class_name("button").click()
time.sleep(5)
# Click on save button to update
driver.find_element_by_class_name("button").click()
# Close driver
driver.close()
Last thing, the program/script is the batch file, no arguments, and the start in is in the directory that the batch file is in.
Any help would be appreciated!
put you python codes in a main() function.
and give:
if __name__ == '__main__':
main()
at the end.
Just tested works for me.
#pk2019 's answer really helped me.
One improvement is to use
drv = webdriver.Chrome()
# Do your things.
...
drv.close()
drv.quit()
No need to do the dirty work of killing task.
I'm not a python expert, but I think you just need to call sys.exit() or quit().
Instead of using sleep and waiting too long or possibly not long enough, call start with the wait option:
call activate python2
start /WAIT C:\Users\Chris\Anaconda3\envs\python2\python.exe testtest.py
exit
If you don't need the batch file to do anything else, you can just start the python script and exit.
I had a similar problem with a Python selenium web scraper running geckodriver.exe Firefox web driver on Windows (executed via Task Scheduler using a .bat file). The problem is that my geckodriver.exe process is still running after the Python script is done ... and I think that running process is preventing the Windows command prompt from closing.
In order to test this hunch, I inserted the tasklist command into the .bat file, both before and after the Python script. It prints out a list of running Windows tasks to the console ... I noticed the geckodriver.exe file was still running the Python script was finished.
The way to kill a Windows process (using taskkill) is described in these two different Stackoverflow responses:
Safely killing a python script on Windows
Batch script to close all open Command Prompt windows
How to close Command Prompt window after Batch file execution in python?
Here is the Windows documentation for taskkill:
https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/taskkill
In my case, I added this as the 2nd to last line: taskkill /im geckodriver.exe /f, and the last line was exit. That worked.
I use ebook-convert.exe from a batch file, and on one of my machines (old Windows vista laptop), any calls ebook-convert.exe prevent further commands in a batch file from running (in that .bat file, or in that cmd shell instance for instance). The conversion to .mobi actually succeeds, prints no errors even in -v -v full verbose mode. I tried versions 1.20.0 and 0.9.6 of calibre portable. It seems to be related to the executable itself, and not to the specific conversion job (just running it with -h flag has same effect). I tried running cmd in administrator mode, wrapping it with 'call' command, nothing helps. Assuming the calibre issue is a black hole that can't be solved, is there a DOS trick I could possibly try to keep batch commands running no matter what odd thing happens with ebook-convert.exe on that machine?
Using call in a batch file should work on normal console programs but it is possible for programs to "escape" if they really want to.
Using Start instead might work but it does not wait by default and its syntax is not sane:
Start /B /WAIT c:\path\without\spaces.exe optionalParameter
or
Start "" /B /WAIT "c:\path\w i t h\spaces.exe" optionalParameter
Remove /B to run the application in a new console.
Another option is to call ping localhost and For (to get the file size with %%~zA) in a loop until the size of the destination file stops increasing. This is a bit of a hack and might not work, it depends on how the converter opens the file and how often it flushes the writes.