I have the following code to download a torrent off of a magnet URI.
#python
#lt.storage_mode_t(0) ## tried this, didnt work
ses = lt.session()
params = { 'save_path': "/save/here"}
ses.listen_on(6881,6891)
ses.add_dht_router("router.utorrent.com", 6881)
#ses = lt.session()
link = "magnet:?xt=urn:btih:395603fa..hash..."
handle = lt.add_magnet_uri(ses, link, params)
while (not handle.has_metadata()):
time.sleep(1)
handle.pause () # got meta data paused, and set priority
handle.file_priority(0, 1)
handle.file_priority(1,0)
handle.file_priority(2,0)
print handle.file_priorities()
#output is [1,0,0]
#i checked no files written into disk yet.
handle.resume()
while (not handle.is_finished()):
time.sleep(1) #wait until download
It works, However in this specific torrent, there are 3 files, file 0 - 2 kb, file 1 - 300mb, file 3 - 2kb.
As can be seen from the code, file 0 has a priority of 1, while the rest has priority 0 (i.e. don't download).
The problem is that when the 0 file finishes downloading, i want to it to stop and not download anymore. but it will sometimes download 1 file -partially, sometimes 100mb, or 200mb, sometimes couple kb and sometimes the entire file.
So my question is: How can i make sure only file 0 is downloaded, and not 1 and 2.
EDIT: I added a check for whether i got metadata, then set priority and then resume it, however this still downloads the second file partially.
The reason this happens is because of the race between adding the torrent (which starts the download) and you setting the file priorities.
To avoid this you can set the file priorities along with adding the torrent, something like this:
p = parse_magnet_uri(link)
p['file_priorities'] = [1, 0, 0]
handle = ses.add_torrent(p)
UPDATE:
You don't need to know the number of files, it's OK to provide file priorities for more files than ends up being in the torrent file. The remaining ones will just be ignored. However, if you don't want to download anything (except for the metadata/.torrent) from the swarm, a better way is to set the flag_upload_mode flag. See documentation.
p = parse_magnet_uri(link)
p['flags'] |= add_torrent_params_flags_t.flag_upload_mode
handle = ses.add_torrent(p)
Related
MY ENVIRONMENT
I'm sending commands to a camera simulator device made by Vivid Engineering (Model CLS-211) over RS-232 from my laptop which is running CentOS 7.
ISSUE
I have installed two different serial monitors (minicom, gtkterm) and can successfully send successive commands over and over to the device. I can send a command to dump the memory contents as well. There are several configuration commands I have to send to the CLS-211 to set it up per a specific test. I want to automate this process and have written a ruby script to write a list of commands to the CLS-211 to make this process easier. However, it appears that I am not terminating each command correctly or the CLS-211 requires a specific terminator/signal that I am not giving to it in my ruby script. I'm confused why I can successfully accomplish this task with a serial monitor but not a ruby script. I've configured the serial port settings correctly per their manual so I know this is not the issue. You'll see those settings below defined in my scirpts. Their manual points out they use HyperTerminal which I can't use because I'm on a Linux system. Manufacture mentioned that other serial terminals should work just fine but they have only chosen to test one out being HyperTerminal. I've asked for their feedback on the issue and they simply say "We don't use Linux but it shouldn't be much different, good luck".
TROUBLESHOOTING
I have verified that my "send.rb" script is working to the best of my knowledge by writing a "read.rb" script to read back what I sent. I essentially connected pin 2 "rx" to pin 3 "tx" on the RS-232 cable for a loopback test. Below is my two scripts and the resulting output from running this test.
## Filename send.rb
require 'serialport'
ser = SerialPort.new("/dev/ttyS0", 9600, 8, 1, SerialPort::NONE)
ser.write "LVAL_LO 5\r\n"
ser.write "LVAL_HI 6\r\n"
ser.write "FVAL_LO 7\r\n"
ser.write "FVAL_HI 8\r\n"
ser.close
## Filename read.rb
require 'serialport'
ser = SerialPort.new("/dev/ttyS0", 9600, 8, 1, SerialPort::NONE)
while true do
printf("%c", ser.getc)
end
ser.close
Just found out that I cannot post more than 2 links since my reputation is so low. Anyways the output is just the following...
username#hostname $ ruby read.rb
LVAL_LO 5
LVAL_HI 6
FVAL_LO 7
FVAL_HI 8
I have hooked up the CLS-211 and dumped the memory contents by issuing the DUMP command using GtkTerm and this works fine. The following image shows the memory contents of the first four parameters being LVAL_LO, LVAL_HI, FVAL_LO, and FVAL_HI. I'm just choosing to show four values in the memory dump for the sake of keeping this thread short versus listing all of them. Since I cannot include more than 2 links because my reputation is low being a new guy I'm typing what the output looks like in GtkTerm instead...
CLS-211 initializing, please wait
............
ready
CLS211 Camera Link Simulator CLI
Vivid Engineering
Rev 3.0
DUMP
LVAL_LO = 0x0020 / 32
LVAL_HI = 0x0100 / 256
FVAL_LO = 0x0002 / 2
FVAL_HI = 0x0100 / 256
In the above image you can clearly see that the system boots as expected. After I typed in the command "DUMP" it printed out the memory contents successfully. You see that LVAL_LO = 32, LVAL_HI = 256, FVAL_LO = 2, and FVAL_HI = 256. As I mentioned before I can successfully type in a command to GtkTerm to change a specific parameter as well. The below images shows that after typing in the command LVAL_LO 5 to GtkTerm and then issuing a DUMP command the value 5 was read correctly and LVAL_LO was changed as expected. I can replicate this successful behavior with every command using GtkTerm.
Again, I can't post more than 2 links so I'm writing the output below...
LVAL_LO 5
DUMP
LVAL_LO = 0x0005 / 5
LVAL_HI = 0x0100 / 256
FVAL_LO = 0x0002 / 2
FVAL_HI = 0x0100 / 256
At this point I was like ok everything is working as expected. Lets see if I can execute my ruby script and replicate the same successfully. I then ran the script I typed up above titled "send.rb". Then I opened up GtkTerm and issued a DUMP command afterwards to see if those values were taken. Let it be known that before I ran "send.rb" the values that existed in memory on the CLS-211 were LVAL_LO = 32, LVAL_HI = 256, FVAL_LO = 2, and FVAL_HI = 256. You can see that after running "send.rb", opening GtkTerm back up and issuing the DUMP command the CLS-211 replied w/ "invalid entry". After issuing it again you'll see that it dumped the contents of memory and showed LVAL_LO was changed correctly but the other three values were not.
Almost Successful
At this point I assumed that the first value was being received and written to the memory contents of the CLS-211 correctly but the other commands were not being received correctly. I assumed this was most likely because of the lack of any delay. Therefore, I placed a 1 second delay between each ser.write command. I changed the script "send.rb" to the following.
## Filename send.rb
require 'serialport'
ser = SerialPort.new("/dev/ttyS0", 9600, 8, 1, SerialPort::NONE)
ser.write "LVAL_LO 9\r\n"
sleep(1)
ser.write "LVAL_HI 10\r\n"
sleep(1)
ser.write "FVAL_LO 11\r\n"
sleep(1)
ser.write "FVAL_HI 12\r\n"
sleep(1)
ser.close
The following is the result of running "send.rb" again with the above changes, opening up GtkTerm, and executing the DUMP command to verify memory.
Added sleep(1)
Nothing really changed. I was able to tell that the script took longer to execute and the first value did change but like before the last three values I sent did not get received and saved to memory on the CLS-211.
CONCLUSION
How can I continue troubleshooting this issue? What sort of terminations are happening to each command that I send through GtkTerm and is that different from what I have sent in my ruby script "send.rb" being "...\r\n". Totally lost and out of options on what I can do next.
[EDIT/UPDATE 10/09/17]
I'm so stupid. The one termination character I forgot to try out "by itself" was carriage return "\r". Using a carriage return by itself after each command fixed the issue. I'm curious what requirements drive a manufacturer to define how a serial packet should be constructed in terms of a termination character(s). I would think there would be a predefined standard to what termination character(s) should be used. For completeness I have included my code below to what it should be in the case of communicating correctly with the CLS-211 device. Basically, i took out the '\n' and kept the '\r' that was it.
## Filename send.rb
require 'serialport'
ser = SerialPort.new("/dev/ttyS0", 9600, 8, 1, SerialPort::NONE)
ser.write "LVAL_LO 9\r"
sleep(1)
ser.write "LVAL_HI 10\r"
sleep(1)
ser.write "FVAL_LO 11\r"
sleep(1)
ser.write "FVAL_HI 12\r"
sleep(1)
ser.close
I'm synchronizing my Sites folder on two Macs in the local network. Some Wordpress sites in this folder contain readme.txt files that are different because of different plugin or theme versions on the two Macs. They differ in file size and timestamp.
My default.prf has a setting for merging .txt files:
merge = Name *.txt -> diff3 CURRENT1 CURRENTARCHOPT CURRENT2 > NEW
which works fine for most text files, but not for these readme.txt files. When I start syncing, Unison gives me merge errors for these files. Here's a typical output:
Contacting server...
Connected [//mac1.local//Users/timm -> //mac2.local//Users/timm]
Looking for changes
Waiting for changes from server
Reconciling changes
local mac1.local
new file <-M-> new file Sites/wp-sites/example.dev/wp-content/plugins/kocuj-sitemap/readme.txt
Proceed with propagating updates? [] y
Propagating updates
UNISON 2.48.4 started propagating changes at 15:21:22.86 on 16 May 2017
Merge command: diff3 '/Users/timm/Sites/wp-sites/example.dev/wp-content/plugins/kocuj-sitemap/.unison.merge1-readme.txt' '/Users/timm/Sites/wp-sites/example.dev/wp-content/plugins/kocuj-sitemap/.unison.merge2-readme.txt' > '/Users/timm/Sites/wp-sites/example.dev/wp-content/plugins/kocuj-sitemap/.unison.mergenew1-readme.txt'
Merge result (exited (2)):
diff3: missing operand after `/Users/timm/Sites/wp-sites/example.dev/wp-content/plugins/kocuj-sitemap/.unison.merge2-readme.txt'
diff3: Try `diff3 --help' for more information.
Saving synchronizer state
Synchronization incomplete at 15:21:24 (0 items transferred, 0 skipped, 1 failed)
failed: Sites/wp-sites/example.dev/wp-content/plugins/kocuj-sitemap/readme.txt
I realise that merge must fail on new files, but I'd think that the program should copy them over rather than trying to merge. I even tried ignoring the (pretty much useless) files:
ignore = Name {Sites/*/readme.txt}
but for some reason Unison doesn't ignore them. The thing is, I need text file merging because I rely on simple text files for my documentation. The default.prf looks like this:
# default profile
servercmd=/opt/local/bin/unison
root = /Users/timm/
root = ssh://mac2.local//Users/timm/
path = Sites
auto = true
times = true
# ignore permissions
perms = 0
rsrc = false
ignore = Name {.DS_Store}
ignore = Name {.localized}
ignore = Name {*/*.app/*}
ignore = Name {*/temp/*}
ignore = Name {*/cache/*}
ignore = Name {Sites/*/wp-content/languages/*}
ignore = Name {Sites/*/wp-content/plugins/*}
ignore = Name {Sites/*/wp-content/themes/twenty*}
ignore = Name {Sites/*/wp-sites/wordpress/*}
# ???
ignore = Name {Sites/*/readme.txt}
# diff and merge
diff = diff
merge = Name *.txt -> diff3 CURRENT1 CURRENTARCHOPT CURRENT2 > NEW
backup = Name *.txt
backupcurrent = Name *.txt
maxbackups = 10
log = true
logfile = /Users/timm/.unison/unison.log
I'm running Unison 2.48.4 on Mac El Capitan 10.11.6.
Did I overlook a setting? Is there any other way to make Unison copy and not merge new files?
I have some code, where i check some directories on ftp server and download new files on my server. There are above 3 million files on server (zip archives). I am doing many not optimize things in this code, but all of them works fast, except part with downloading. Here is this part:
lf = open(local_filename, "wb") //here i create blank file
print ("opened")
try:
ftp.retrbinary("RETR "+name, lf.write) //here i write data
print ("wrote")
except ftplib.error_perm:
pass
lf.close() //here i close file with data
print ("closed")
my problem in the part between print ("opened") and print ("wrote"). My python console (2.7) keep silence for 10-20 second on this fase, but size of downloading files is very tiny. Its below 2-3 Kb.
Strange thing in next: when i start script from my own PC (windows 7), it works great and fast, but when i start it on windows server 2012 R2 (VDS), i got this sadly pause. Guys, i need your help. What should i do for configuration my server and fast downloading?
i got the answer. just need to run next command:
netsh int tcp set global ecncapability=disabled
and everything will be excellent!
We have bash script (job wrapper) that writes to a file, launches a job, then at job completion it appends to the file information about the job. The wrapper is run on one of several thousand batch nodes, but has only cropped up with several batch machines (I believe RHEL6) accessing one NFS server, and at least one known instance of a different batch job on a different batch node using a different NFS server. In all cases, only one client host is writing to the files in question. Some jobs take hours to run, others take minutes.
In the same time period that this has occurred, there seems to be 10-50 issues out of 100,000+ jobs.
Here is what I believe to effectively be the (distilled) version of the job wrapper:
#!/bin/bash
## cwd is /nfs/path/to/jobwd
## This file is /nfs/path/to/jobwd/job_wrapper
gotEXIT()
{
## end of script, however gotEXIT is called because we trap EXIT
END="EndTime: `date`\nStatus: Ended”
echo -e "${END}" >> job_info
cat job_info | sendmail jobtracker#example.com
}
trap gotEXIT EXIT
function jobSetVar { echo "job.$1: $2" >> job_info; }
export -f jobSetVar
MSG=“${email_metadata}\n${job_metadata}”
echo -e "${MSG}\nStatus: Started" | sendmail jobtracker#example.com
echo -e "${MSG}" > job_info
## At the job’s end, the output from `time` command is the first non-corrupt data in job_info
/usr/bin/time -f "Elapsed: %e\nUser: %U\nSystem: %S" -a -o job_info job_command
## 10-360 minutes later…
RC=$?
echo -e "ExitCode: ${RC}" >> job_info
So I think there are two possibilities:
echo -e "${MSG}" > job_info
This command throws out corrupt data.
/usr/bin/time -f "Elapsed: %e\nUser: %U\nSystem: %S" -a -o job_info job_command
This corrupts the existing data, then outputs it’s data correctly.
However, some job, but not all, call jobSetVar, which doesn't end up being corrupt.
So, I dig into time.c (from GNU time 1.7) to see when the file is open. To summarize, time.c is effectively this:
FILE *outfp;
void main (int argc, char** argv) {
const char **command_line;
RESUSE res;
/* internally, getargs opens “job_info”, so outfp = fopen ("job_info", "a”) */
command_line = getargs (argc, argv);
/* run_command doesn't care about outfp */
run_command (command_line, &res);
/* internally, summarize calls fprintf and putc on outfp FILE pointer */
summarize (outfp, output_format, command_line, &res); /
fflush (outfp);
}
So, time has FILE *outfp (job_info handle) open the entire time of the job. It then writes the summary at the end of the job, and then doesn’t actually appear to close the file (not sure if this is necessary with fflush?) I've no clue if bash also has the file handle open concurrently as well.
EDIT:
A corrupted file will typically end consist of the corrupted part, followed with the non-corrupted part, which may look like this:
The corrupted section, which would occur before the non-corrupted section, is typically largely a bunch of 0x0000, with maybe some cyclic garbage mixed in:
Here's an example hexdump:
40000000 00000000 00000000 00000000
00000000 00000000 C8B450AC 772B0000
01000000 00000000 C8B450AC 772B0000
[ 361 x 0x00]
Then, at the 409th byte, it continues with the non-corrupted section:
Elapsed: 879.07
User: 0.71
System: 31.49
ExitCode: 0
EndTime: Fri Dec 6 15:29:27 PST 2013
Status: Ended
Another file looks like this:
01000000 04000000 805443FC 9D2B0000 E04144FC 9D2B0000 E04144FC 9D2B0000
[96 x 0x00]
[Repeat above 3 times ]
01000000 04000000 805443FC 9D2B0000 E04144FC 9D2B0000 E04144FC 9D2B0000
Followed by the non corrupted section:
Elapsed: 12621.27
User: 12472.32
System: 40.37
ExitCode: 0
EndTime: Thu Nov 14 08:01:14 PST 2013
Status: Ended
There are other files that have much more random corruption sections, but more than a few were cyclical similar to above.
EDIT 2: The first email, sent from the echo -e statement goes through fine. The last email is never sent due to no email metadata from corruption. So MSG isn't corrupted at that point. It's assumed that job_info probably isn't corrupt at that point either, but we haven't been able to verify that yet. This is a production system which hasn't had major code modifications and I have verified through audit that no jobs have been ran concurrently which would touch this file. The problem seems to be somewhat recent (last 2 months), but it's possible it's happened before and slipped through. This error does prevent reporting which means jobs are considered failed, so they are typically resubmitted, but one user in specific has ~9 hour jobs in which this error is particularly frustrating. I wish I could come up with more info or a way of reproducing this at will, but I was hoping somebody has maybe seen a similar problem, especially recently. I don't manage the NFS servers, but I'll try to talk to the admins to see what updates the NFS servers at the time of these issues (RHEL6 I believe) were running.
Well, the emails corresponding to the corrupt job_info files should tell you what was in MSG (which will probably be business as usual). You may want to check how NFS is being run: there's a remote possibility that you are running NFS over UDP without checksums. That could explain some corrupt data. I also hear that UDP/TCP checksums are not strong enough and the data can still end up corrupt -- maybe you are hitting such a problem (I have seen corrupt packets slipping through a network stack at least once before, and I'm quite sure some checksumming was going on). Presumably the MSG goes out as a single packet and there might be something about it that makes checksum conflicts with the garbage you see more likely. Of course it could also be an NFS bug (client or server), a server-side filesystem bug, busted piece of RAM... possibilities are almost endless here (although I see how the fact that it's always MSG that gets corrupted makes some of those quite unlikely). The problem might be related to seeking (which happens during the append). You could also have a bug somewhere else in the system, causing multiple clients to open the same job_info file, making it a jumble.
You can also try using different file for 'time' output and then merge them together with job_info at the end of script. That may help to isolate problem further.
Shell opens 'job_info' file for writing, outputs MSG and then shall close its file descriptor before launching main job. 'time' program opens same file for append as stream and I suspect the seek over NFS is not done correctly which may cause that garbage. Can't explain why, but normally this shall not happen (and is not happening). Such rare occasions may point to some race condition somewhere, can be caused by out of sequence packet delivery (due to network latency spike) or retransmits which causes duplicate data, or a bug somewhere. At first look I would suspect some bug, but that bug may be triggered by some network behavior, e.g. unusually large delay or spike of packet loss.
File access between different processes are serialized by kernel, but for additional safeguard may be worth adding some artificial delays - sleep timers between outputs for example.
Network is not transparent, especially a large one. There can be WAN optimization devices which are known to cause application issues sometimes. CIFS and NFS are good candidates for optimization over WAN with local caching of filesystem operations. Might be worth looking for recent changes with network admins..
Another thing to try, although can be difficult due to rare occurrences is capture of interesting NFS sessions via tcpdump or wireshark. In really tough cases we do simultaneous capturing on both client and server side and then compare the protocol logic to prove that network is or is not working correctly. That's a whole topic in itself, requires thorough preparation and luck but usually a last resort of desperate troubleshooting :)
It turns out this was actually another issue altogether, apparently to do with an out-of-date page being written to disk.
A bug fix was supplied to the linux-nfs implementation:
http://www.spinics.net/lists/linux-nfs/msg41357.html
NOTE: This is a complete re-write of this question. I'd previously conflated some ACL issues with the problem I'm hunting, which is probably why there were no answers.
I have a windows service that uses the standard open/close/write routines to write a log file (it reads stuff from a pipe and stuffs it into the log). A new log file is opened each day at midnight. The system is Windows XP Embedded.
The service runs as the Local System service (CreateService with NULL for the user).
When the service initially starts up, it creates a log file and writes to it with no problems. At this point everything is OK, and you can restart the service (or the computer) with no issues.
However, at midnight (when the day changes), the service creates a new log file and writes to it. The funny thing is, this new log file has the 'read only' flag set. That's a problem because if the service (or the computer) restarts, the service can no longer open the file for writing.
Here's the relevant information from the system with the problem having already happened:
Directory of C:\bbbaudit
09/16/2009 12:00 AM <DIR> .
09/16/2009 12:00 AM <DIR> ..
09/16/2009 12:00 AM 437 AU090915.ADX
09/16/2009 12:00 AM 62 AU090916.ADX
attrib c:\bbbaudit\*
A C:\bbbaudit\AU090915.ADX <-- old log file (before midnight)
A R C:\bbbaudit\AU090916.ADX <-- new log file (after midnight)
cacls output:
C:\ BUILTIN\Administrators:(OI)(CI)F
NT AUTHORITY\SYSTEM:(OI)(CI)F
CREATOR OWNER:(OI)(CI)(IO)F
BUILTIN\Users:(OI)(CI)R
BUILTIN\Users:(CI)(special access:)
FILE_APPEND_DATA
BUILTIN\Users:(CI)(IO)(special access:)
FILE_WRITE_DATA
Everyone:R
C:\bbbaudit BUILTIN\Administrators:(OI)(CI)F
NT AUTHORITY\SYSTEM:(OI)(CI)F
CFN3\Administrator:F
CREATOR OWNER:(OI)(CI)(IO)F
Here's the code I use to open/create the log files:
static int open_or_create_file(char *fname, bool &alreadyExists)
{
int fdes;
// try to create new file, fail if it already exists
alreadyExists = false;
fdes = open(fname, O_WRONLY | O_APPEND | O_CREAT | O_EXCL);
if (fdes < 0)
{
// try to open existing, don't create new file
alreadyExists = true;
fdes = open(fname, O_WRONLY | O_APPEND);
}
return fdes;
}
I'm having real trouble figuring out how the file is getting that read-only flag on it. Anyone who can give me a clue or a direction, I'd greatly appreciate it.
Compiler is VC 6 (Yea, I know, it's so far out of date it isn't funny. Until you realize that we're just now upgraded to XPE from NT 3.51).
The Microsoft implementation of open() has an optional third argument 'pmode', which is required to be present when the second argument 'oflag' includes the O_CREAT flag. The pmode argument specifies the file permission settings, which are set when the new file is closed for the first time. Typically you would pass S_IREAD | S_IWRITE for pmode, resulting in an ordinary read/write file.
In your case you have specified O_CREAT but omitted the third argument, so open() has used whatever value happened to be on the stack at the third argument position. The value of S_IWRITE is 0x0080, so if the value in the third argument position happened to have bit 7 clear, it would result in a read-only file. The fact that you got a read-only file only some of the time, is consistent with stack junk being passed as the third argument.
Below is the link for the Visual Studio 2010 documentation for open(). This aspect of the function's behaviour has not changed since VC 6.
http://msdn.microsoft.com/en-us/library/z0kc8e3z.aspx
Well, I have no idea what the underlying problem is with the 'open' APIs in this case. In order to 'fix' the problem, I ended up switching to using the Win32 APIs for file management (CreateFile, WriteFile, CloseHandle).