Rshell on windows freezes randomly while sending files to ESP32 running micropython via rsync or cp - windows

I'm having trouble copying files to ESP32 device running micropython via rshell (remote micropython shell). The problem seems to occur randomly. Sometimes I manage to successfuly send all files (like 20 of them) with rsync and sometimes I cannot even copy one file with cp. There is no error message and script doesn't crash, it just stops working and freezes the console. I've tried with and without -a parameter (I'm sending only .py files). The last thing printed with debug is code to be run on microcontroller below and it just stops there. I didn't find any pattern. I've tried using some other esp32 device and other windows PC. Same results. I even tried lowering default 32 buffer to 16, no improvement. The worst thing is that is sometimes works fine and I cannot get constant results (even bad). It stops on random files, not always the same one.
Rsync command (with --mirror parameter) is very helpful and I could not just copy all files by hand.
EDIT: just tested on mac and it works fine. I guess it's just problem on windows...
Adding /pyboard/protocol/parser.py
----- About to send 2269 bytes of code to the pyboard -----
def recv_file_from_host(src_file, dst_filename, filesize, dst_mode='wb'):
"""Function which runs on the pyboard. Matches up with send_file_to_remote."""
import sys
import ubinascii
import os
if False:
try:
import pyb
usb = pyb.USB_VCP()
except:
try:
import machine
usb = machine.USB_VCP()
except:
usb = None
if usb and usb.isconnected():
# We don't want 0x03 bytes in the data to be interpreted as a Control-C
# This gets reset each time the REPL runs a line, so we don't need to
# worry about resetting it ourselves
usb.setinterrupt(-1)
try:
with open(dst_filename, dst_mode) as dst_file:
bytes_remaining = filesize
if not False:
bytes_remaining *= 2 # hexlify makes each byte into 2
buf_size = 32
write_buf = bytearray(buf_size)
read_buf = bytearray(buf_size)
while bytes_remaining > 0:
# Send back an ack as a form of flow control
sys.stdout.write('\x06')
read_size = min(bytes_remaining, buf_size)
buf_remaining = read_size
buf_index = 0
while buf_remaining > 0:
if False:
bytes_read = sys.stdin.buffer.readinto(read_buf, read_size)
else:
bytes_read = sys.stdin.readinto(read_buf, read_size)
if bytes_read > 0:
write_buf[buf_index:bytes_read] = read_buf[0:bytes_read]
buf_index += bytes_read
buf_remaining -= bytes_read
if False:
dst_file.write(write_buf[0:read_size])
else:
dst_file.write(ubinascii.unhexlify(write_buf[0:read_size]))
if hasattr(os, 'sync'):
os.sync()
bytes_remaining -= read_size
return True
except:
return False
output = recv_file_from_host(None, '/protocol/parser.py', 1467)
if output is None:
print("None")
else:
print(output)
-----

I had the same problem ,
when trying
C:\> cp src\main.py /pyboard/
the cmd freezes.
When I copied using the following
C:\> cp src/main.py /pyboard/
There where no issues, so maybe rshell has some problems when there are "\" in the path

Related

Commands not accepted by instrument over RS232 w/ ruby serialport script

MY ENVIRONMENT
I'm sending commands to a camera simulator device made by Vivid Engineering (Model CLS-211) over RS-232 from my laptop which is running CentOS 7.
ISSUE
I have installed two different serial monitors (minicom, gtkterm) and can successfully send successive commands over and over to the device. I can send a command to dump the memory contents as well. There are several configuration commands I have to send to the CLS-211 to set it up per a specific test. I want to automate this process and have written a ruby script to write a list of commands to the CLS-211 to make this process easier. However, it appears that I am not terminating each command correctly or the CLS-211 requires a specific terminator/signal that I am not giving to it in my ruby script. I'm confused why I can successfully accomplish this task with a serial monitor but not a ruby script. I've configured the serial port settings correctly per their manual so I know this is not the issue. You'll see those settings below defined in my scirpts. Their manual points out they use HyperTerminal which I can't use because I'm on a Linux system. Manufacture mentioned that other serial terminals should work just fine but they have only chosen to test one out being HyperTerminal. I've asked for their feedback on the issue and they simply say "We don't use Linux but it shouldn't be much different, good luck".
TROUBLESHOOTING
I have verified that my "send.rb" script is working to the best of my knowledge by writing a "read.rb" script to read back what I sent. I essentially connected pin 2 "rx" to pin 3 "tx" on the RS-232 cable for a loopback test. Below is my two scripts and the resulting output from running this test.
## Filename send.rb
require 'serialport'
ser = SerialPort.new("/dev/ttyS0", 9600, 8, 1, SerialPort::NONE)
ser.write "LVAL_LO 5\r\n"
ser.write "LVAL_HI 6\r\n"
ser.write "FVAL_LO 7\r\n"
ser.write "FVAL_HI 8\r\n"
ser.close
## Filename read.rb
require 'serialport'
ser = SerialPort.new("/dev/ttyS0", 9600, 8, 1, SerialPort::NONE)
while true do
printf("%c", ser.getc)
end
ser.close
Just found out that I cannot post more than 2 links since my reputation is so low. Anyways the output is just the following...
username#hostname $ ruby read.rb
LVAL_LO 5
LVAL_HI 6
FVAL_LO 7
FVAL_HI 8
I have hooked up the CLS-211 and dumped the memory contents by issuing the DUMP command using GtkTerm and this works fine. The following image shows the memory contents of the first four parameters being LVAL_LO, LVAL_HI, FVAL_LO, and FVAL_HI. I'm just choosing to show four values in the memory dump for the sake of keeping this thread short versus listing all of them. Since I cannot include more than 2 links because my reputation is low being a new guy I'm typing what the output looks like in GtkTerm instead...
CLS-211 initializing, please wait
............
ready
CLS211 Camera Link Simulator CLI
Vivid Engineering
Rev 3.0
DUMP
LVAL_LO = 0x0020 / 32
LVAL_HI = 0x0100 / 256
FVAL_LO = 0x0002 / 2
FVAL_HI = 0x0100 / 256
In the above image you can clearly see that the system boots as expected. After I typed in the command "DUMP" it printed out the memory contents successfully. You see that LVAL_LO = 32, LVAL_HI = 256, FVAL_LO = 2, and FVAL_HI = 256. As I mentioned before I can successfully type in a command to GtkTerm to change a specific parameter as well. The below images shows that after typing in the command LVAL_LO 5 to GtkTerm and then issuing a DUMP command the value 5 was read correctly and LVAL_LO was changed as expected. I can replicate this successful behavior with every command using GtkTerm.
Again, I can't post more than 2 links so I'm writing the output below...
LVAL_LO 5
DUMP
LVAL_LO = 0x0005 / 5
LVAL_HI = 0x0100 / 256
FVAL_LO = 0x0002 / 2
FVAL_HI = 0x0100 / 256
At this point I was like ok everything is working as expected. Lets see if I can execute my ruby script and replicate the same successfully. I then ran the script I typed up above titled "send.rb". Then I opened up GtkTerm and issued a DUMP command afterwards to see if those values were taken. Let it be known that before I ran "send.rb" the values that existed in memory on the CLS-211 were LVAL_LO = 32, LVAL_HI = 256, FVAL_LO = 2, and FVAL_HI = 256. You can see that after running "send.rb", opening GtkTerm back up and issuing the DUMP command the CLS-211 replied w/ "invalid entry". After issuing it again you'll see that it dumped the contents of memory and showed LVAL_LO was changed correctly but the other three values were not.
Almost Successful
At this point I assumed that the first value was being received and written to the memory contents of the CLS-211 correctly but the other commands were not being received correctly. I assumed this was most likely because of the lack of any delay. Therefore, I placed a 1 second delay between each ser.write command. I changed the script "send.rb" to the following.
## Filename send.rb
require 'serialport'
ser = SerialPort.new("/dev/ttyS0", 9600, 8, 1, SerialPort::NONE)
ser.write "LVAL_LO 9\r\n"
sleep(1)
ser.write "LVAL_HI 10\r\n"
sleep(1)
ser.write "FVAL_LO 11\r\n"
sleep(1)
ser.write "FVAL_HI 12\r\n"
sleep(1)
ser.close
The following is the result of running "send.rb" again with the above changes, opening up GtkTerm, and executing the DUMP command to verify memory.
Added sleep(1)
Nothing really changed. I was able to tell that the script took longer to execute and the first value did change but like before the last three values I sent did not get received and saved to memory on the CLS-211.
CONCLUSION
How can I continue troubleshooting this issue? What sort of terminations are happening to each command that I send through GtkTerm and is that different from what I have sent in my ruby script "send.rb" being "...\r\n". Totally lost and out of options on what I can do next.
[EDIT/UPDATE 10/09/17]
I'm so stupid. The one termination character I forgot to try out "by itself" was carriage return "\r". Using a carriage return by itself after each command fixed the issue. I'm curious what requirements drive a manufacturer to define how a serial packet should be constructed in terms of a termination character(s). I would think there would be a predefined standard to what termination character(s) should be used. For completeness I have included my code below to what it should be in the case of communicating correctly with the CLS-211 device. Basically, i took out the '\n' and kept the '\r' that was it.
## Filename send.rb
require 'serialport'
ser = SerialPort.new("/dev/ttyS0", 9600, 8, 1, SerialPort::NONE)
ser.write "LVAL_LO 9\r"
sleep(1)
ser.write "LVAL_HI 10\r"
sleep(1)
ser.write "FVAL_LO 11\r"
sleep(1)
ser.write "FVAL_HI 12\r"
sleep(1)
ser.close

How to completely skip writing files to disk with libtorrent downloads?

I have the following code to download a torrent off of a magnet URI.
#python
#lt.storage_mode_t(0) ## tried this, didnt work
ses = lt.session()
params = { 'save_path': "/save/here"}
ses.listen_on(6881,6891)
ses.add_dht_router("router.utorrent.com", 6881)
#ses = lt.session()
link = "magnet:?xt=urn:btih:395603fa..hash..."
handle = lt.add_magnet_uri(ses, link, params)
while (not handle.has_metadata()):
time.sleep(1)
handle.pause () # got meta data paused, and set priority
handle.file_priority(0, 1)
handle.file_priority(1,0)
handle.file_priority(2,0)
print handle.file_priorities()
#output is [1,0,0]
#i checked no files written into disk yet.
handle.resume()
while (not handle.is_finished()):
time.sleep(1) #wait until download
It works, However in this specific torrent, there are 3 files, file 0 - 2 kb, file 1 - 300mb, file 3 - 2kb.
As can be seen from the code, file 0 has a priority of 1, while the rest has priority 0 (i.e. don't download).
The problem is that when the 0 file finishes downloading, i want to it to stop and not download anymore. but it will sometimes download 1 file -partially, sometimes 100mb, or 200mb, sometimes couple kb and sometimes the entire file.
So my question is: How can i make sure only file 0 is downloaded, and not 1 and 2.
EDIT: I added a check for whether i got metadata, then set priority and then resume it, however this still downloads the second file partially.
The reason this happens is because of the race between adding the torrent (which starts the download) and you setting the file priorities.
To avoid this you can set the file priorities along with adding the torrent, something like this:
p = parse_magnet_uri(link)
p['file_priorities'] = [1, 0, 0]
handle = ses.add_torrent(p)
UPDATE:
You don't need to know the number of files, it's OK to provide file priorities for more files than ends up being in the torrent file. The remaining ones will just be ignored. However, if you don't want to download anything (except for the metadata/.torrent) from the swarm, a better way is to set the flag_upload_mode flag. See documentation.
p = parse_magnet_uri(link)
p['flags'] |= add_torrent_params_flags_t.flag_upload_mode
handle = ses.add_torrent(p)

Python PermissionError on some file operations

I created the following Python 3.5 script:
import sys
from pathlib import Path
def test_unlink():
log = Path('log.csv')
fails = 0
num_tries = int(sys.argv[1])
for i in range(num_tries):
try:
log.write_text('asdfasdfasdfasdfasdfasdfasdfasdf')
with log.open('r') as logfile:
lines = logfile.readlines()
# Check the second line to account for the log file header
assert len(lines) == 1
log.unlink()
not log.exists()
except PermissionError:
sys.stdout.write('! ')
sys.stdout.flush()
fails += 1
assert fails == 0, '{:%}'.format(fails / num_tries)
test_unlink()
and run it like this: python test.py 10000. On Windows 7 Pro 64 with Python 3.5.2, the failure rate is not 0: it is small, but non-zero. Sometimes, it is not even that small: 5%! If you printout the exception, it will be this:
PermissionError: [WinError 5] Access is denied: 'C:\\...\\log.csv'
but it will sometimes occur at the exists(), other times at the write_text(), and I wouldn't be surprised if it happens at the unlink() and the open() too.
Note that the same script, same Python (3.5.2), but on linux (through http://repl.it/), does not have this issue: failure rate is 0.
I realize that a possible workaround could be:
while True:
try: log.unlink()
except PermissionError: pass
else: break
but this is tedious and error prone (several methods on Path instance would need this, easy to forget), and should (IMHO) not be necessary, so I don't think it is practical solution.
So, does anyone have an explanation for this, and a practical workaround, maybe a mode flag somewhere that can be set when Python starts?

looking for more info on EIO error returned by readdir_r () on Mac OS X

On Mac OS X 10.6.8, some C code that calls readdir_r() sometimes gets an I/O error 5 (EIO) returned. I've seen this just a few times, always on external USB drives. Each time I've seen it, if I immediately cd to the parent direction and do an ls, I appear to see all the files. And, if I re-run the C program that saw the error, it will run fine.
On some platforms, when one gets an error like EIO, there's something else you can call that gives you much more detailed error information (e.g., hperrmsg on MPE/iX, which accesses the per-process error stack). Is there something similar on OS X?
I guess I'm assuming that the error is false
i = readdir_r (dirp, &dire, &dire_ptr);
save_errno = errno;
if (i)
{
if (i == EACCES)
...
else
{
my_perror (save_errno, "readdir_r failed: ");
espout4 ("readdir_r returned %s; errno = %s, dire_ptr = %s; parent = %s\n",
num64 (i), num64 (save_errno), fmt_p (dire_ptr), parent_dir);
espout1 (" (ERROR: readdir_r failed here, errno %s)\n", num64 (save_errno));
}
}
Update, a week later...
I've determined that the combo of the Macally case (model G-S350SU)
and the Western Digital drive was causing sporadic problems. I noticed that the WD SATA connector didn't mate tightly with the Macally socket. Using a Seagate drive with the case was fine, and using the WD drive in a different case was fine.
Of course, I'd still like to be able to get better / more detailed error information in my programs :)

cygwin rsync all log locations

I'm trying to secure an rsync file transfers between two machines through cygwin on windows.
Where are the location of all log files that rsync creates? I'm currently using
rsync -e ssh -avzP --delete "/cygdrive/C/secure/data/" "/cygdrive/D/data" --log-file=/cygdrive/C/secure/log/c.log
When rsync starts it says 'building file list' is this done in memory or is it writing a temp file somewhere?
You asked if "building file list..." happens in memory, or is stored somewhere. Let's take a look at rsync's sources, namely, flist.c:
2089 rprintf(FLOG, "building file list\n");
2090 if (show_filelist_p())
2091 start_filelist_progress("building file list");
2092 else if (inc_recurse && INFO_GTE(FLIST, 1) && !am_server)
2093 rprintf(FCLIENT, "sending incremental file list\n");
2094
2095 start_write = stats.total_written;
2096 gettimeofday(&start_tv, NULL);
2097
2098 if (relative_paths && protocol_version >= 30)
2099 implied_dirs = 1; /* We send flagged implied dirs */
2100
2101 #ifdef SUPPORT_HARD_LINKS
2102 if (preserve_hard_links && protocol_version >= 30 && !cur_flist)
2103 init_hard_links();
2104 #endif
2105
2106 flist = cur_flist = flist_new(0, "send_file_list");
2107 if (inc_recurse) {
2108 dir_flist = flist_new(FLIST_TEMP, "send_file_list");
2109 flags |= FLAG_DIVERT_DIRS;
2110 } else
2111 dir_flist = cur_flist;
flist_new just allocates a new file list pool, without doing anything. The string parameter is there to inform about out of memory errors.
Then there are several hundred lines whose purpose is to send the file list. There are no rprintf calls whatsoever in there, except for possible error messages. The most interesting function calls in that loop are calls to send_file_name, but these don't call rprintf too.
So yes, it builds the files in memory, and you can't log it without patching the sources yourself. Furthermore, not everything in rsync is logged to log file, some messages are printed directly to your screen. In fact, you might be better off using simple shell redirection, depending on your progress settings. (Or use tee to print output to a file and your stdout simultaneously.)

Resources