I want to debug in vscode for following codes. But it didnot run as expected!
def main():
args = parse_args()
if args.mode == "single":
train_cmd = "python lib/train/run_training.py --script %s --config %s --save_dir %s --use_lmdb %d " \
"--script_prv %s --config_prv %s --distill %d --script_teacher %s --config_teacher %s --use_wandb %d"\
% (args.script, args.config, args.save_dir, args.use_lmdb, args.script_prv, args.config_prv,
args.distill, args.script_teacher, args.config_teacher, args.use_wandb)
elif args.mode == "multiple":
train_cmd = "python -m torch.distributed.launch --nproc_per_node %d --master_port %d lib/train/run_training.py " \
"--script %s --config %s --save_dir %s --use_lmdb %d --script_prv %s --config_prv %s --use_wandb %d " \
"--distill %d --script_teacher %s --config_teacher %s" \
% (args.nproc_per_node, random.randint(10000, 50000), args.script, args.config, args.save_dir, args.use_lmdb, args.script_prv, args.config_prv, args.use_wandb,
args.distill, args.script_teacher, args.config_teacher)
elif args.mode == "multi_node":
train_cmd = "python -m torch.distributed.launch --nproc_per_node %d --master_addr %s --master_port %d --nnodes %d --node_rank %d lib/train/run_training.py " \
"--script %s --config %s --save_dir %s --use_lmdb %d --script_prv %s --config_prv %s --use_wandb %d " \
"--distill %d --script_teacher %s --config_teacher %s" \
% (args.nproc_per_node, args.ip, args.port, args.world_size, args.rank, args.script, args.config, args.save_dir, args.use_lmdb, args.script_prv, args.config_prv, args.use_wandb,
args.distill, args.script_teacher, args.config_teacher)
else:
raise ValueError("mode should be 'single' or 'multiple'.")
os.system(train_cmd) # here
When step debuger executes to os.system(train_cmd), vscode debuger cannot run as expected. Instead, it run the train_cmd in debug console. Although program can run normally, I cannot debug. Who can help me, thanks a lot!!!
Related
I am looking to run multiple instances of a command line script at the same time. I am new to this concept of "multi-threading" so am at bit of a loss as to why I am seeing the things that I am seeing.
I have tried to execute the sub-processing in two different ways:
1 - Using multiple calls of Popen without a communicate until the end:
command = 'raster2pgsql -I -C -e -s 26911 %s -t 100x100 -F p839.%s_image_sum_sum1 | psql -U david -d projects -h pg3' % (workspace + '\\r_sumsum1{}'.format(i), str(i))
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
command = 'raster2pgsql -I -C -e -s 26911 %s -t 100x100 -F p839.%s_image_sum_sum2 | psql -U david -d projects -h pg3' % (workspace + '\\r_sumsum2{}'.format(i), str(i))
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
command = 'raster2pgsql -I -C -e -s 26911 %s -t 100x100 -F p839.%s_image_sum_sum3 | psql -U david -d projects -h pg3' % (workspace + '\\r_sumsum3{}'.format(i), str(i))
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
(stdoutdata, stderrdata) = process.communicate()
this starts up each of the command line item but only completes the last entry leaving the other 2 hanging.
2 - Attempting to implement an example from Python threading multiple bash subprocesses? but nothing happens except for a printout of the commands (program hangs with no command line arguments running as observed in windows task manager:
import threading
import Queue
import commands
import time
workspace = r'F:\Processing\SM'
image = 't08r_e'
image_name = (image.split('.'))[0]
i = 0
process_image_tif = workspace + '\\{}{}.tif'.format((image.split('r'))[0], str(i))
# thread class to run a command
class ExampleThread(threading.Thread):
def __init__(self, cmd, queue):
threading.Thread.__init__(self)
self.cmd = cmd
self.queue = queue
def run(self):
# execute the command, queue the result
(status, output) = commands.getstatusoutput(self.cmd)
self.queue.put((self.cmd, output, status))
# queue where results are placed
result_queue = Queue.Queue()
# define the commands to be run in parallel, run them
cmds = ['raster2pgsql -I -C -e -s 26911 %s -t 100x100 -F p839.%s_image_sum_sum1 | psql -U david -d projects -h pg3' % (workspace + '\\r_sumsum1{}'.format(i), str(i)),
'raster2pgsql -I -C -e -s 26911 %s -t 100x100 -F p839.%s_image_sum_sum2 | psql -U david -d projects -h pg3' % (workspace + '\\r_sumsum2{}'.format(i), str(i)),
'raster2pgsql -I -C -e -s 26911 %s -t 100x100 -F p839.%s_image_sum_sum3 | psql -U david -d projects -h pg3' % (workspace + '\\r_sumsum3{}'.format(i), str(i)),
]
for cmd in cmds:
thread = ExampleThread(cmd, result_queue)
thread.start()
# print results as we get them
while threading.active_count() > 1 or not result_queue.empty():
while not result_queue.empty():
(cmd, output, status) = result_queue.get()
print(cmd)
print(output)
How can I run all of these commands at the same time achieving a result at the end? I am running in windows, pyhton 2.7.
My first try didn't work because of the repeated definitions of stdout and sterror. Removing these definitions causes expected behavior.
I am trying to invoke a git log from go and redirect the output to a given file.
cmdArgs = []string{"log", "--numstat", "--reverse", fmt.Sprintf("%s..HEAD", "89c98f5ec48c8ac383ea9e27d792c3dc77fa6240"), `--pretty="format:=%P %H %an %ae %ad %at %s %b"`}
cmdArgs = append(cmdArgs, ">> "+workingDir+"/logs/"+repoName+".log && cat "+workingDir+"/logs/"+repoName+".log")
cmd := exec.Command("git", cmdArgs...)
cmd.Dir = workingDir + repoName
var out bytes.Buffer
var stderr bytes.Buffer
cmd.Stdout = &out
cmd.Stderr = &stderr
err := cmd.Run()
if err != nil {
fmt.Println("git", strings.Join(cmdArgs, " "), "in", workingDir+repoName)
fmt.Println(fmt.Sprint(err) + ": " + stderr.String())
panic(err)
}
fails with
git log --numstat --reverse --pretty="format:=%P %H %an %ae %ad %at %s %b" 89c98f5ec48c8ac383ea9e27d792c3dc77fa6240..HEAD > /someplace/xx-tmp.log && cat /someplace/xx-tmp.log in /someplace
exit status 128: fatal: ambiguous argument ' > /someplace/xx-tmp.log && cat /someplace/xx-tmp.log: unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Executing the command in bash directly poses no problem.
Go's cmd.Run() acts similarly to the C fork() and exec() process for starting a new program. This doesn't implicitly invoke a shell -- which is, from a security perspective, an extremely good thing; unnecessary shell invocation often leads to command injection vulnerabilities.
If you want capabilities that a shell can add -- here, redirection and compound command syntax -- but want to avoid the security risks, pass your data out-of-band from code:
cmdArgs = []string{
"-c", // tells interpreter that script is next argument
`outfile=$1; shift; "$#" >"$outfile" && cat "$outfile"`, // script to execute
"_", // this is $0
workingDir+"/logs/"+repoName+".log", // $1, becomes outfile
"git", "log", "--numstat", "--reverse", // remaining args are in "$#"
fmt.Sprintf("%s..HEAD", "89c98f5ec48c8ac383ea9e27d792c3dc77fa6240"),
"--pretty=format:=%P %H %an %ae %ad %at %s %b"
}
cmd := exec.Command("sh", cmdArgs...)
The above is equivalent to the following shell script:
#!/bin/sh
# ^^- not /bin/bash; this only guarantees support for POSIX syntax
outfile=$1 # assign first positional argument to variable "$outfile"
shift # rename $2 to $1, $3 to $2, etc
if "$#" >"$outfile"; then # run remaining arguments as a single command, stdout to outfile
cat "$outfile" # if that succeeded, then cat our "$outfile" to stdout
fi
Note that I removed the literal quotes from inside --pretty=. That's because when you run the command in a shell, those quotes are treated as syntax by the shell -- an instruction not to split on the spaces within the format string. Here, there is no shell parsing that string as code; if we left the quotes in, they would become a literal part of your format string.
The above being said, there's no good reason to use a shell for any of this. You can do the redirection in native Go trivially:
cmdArgs = []string{
"log", "--numstat", "--reverse",
fmt.Sprintf("%s..HEAD", "89c98f5ec48c8ac383ea9e27d792c3dc77fa6240"),
`--pretty=format:=%P %H %an %ae %ad %at %s %b`}
cmd := exec.Command("git", cmdArgs...)
cmd.Dir = workingDir + repoName
outputFile, err = os.OpenFile(workingDir+"/logs/"+repoName+".log",
os.O_APPEND | os.O_CREATE | os.O_WRONLY, 0644)
if err != nil { panic(err) }
defer outputFile.Close()
cmd.Stdout = outputFile
...and then run the cat as a separate exec.Command instance (if you have a good reason not to just implement it in native Go).
Here is my script. I want to replace text in multiple dest. How I can use wildcard in (dest=/home/*/conf/server.xml).
- hosts: 192.168.8.11
user: mohitmehral
sudo: yes
tasks:
- replace:
dest=/home/5/conf/server.xml
#dest=/home/1/conf/server.xml
#dest=/home/2/conf/server.xml
#dest=/home/3/conf/server.xml
#dest=/home/4/conf/server.xml
#dest=/home/5/conf/server.xml
regexp='pattern="%{X-Forwarded-For}i %h %t %a %p %v %q "%{Referer}i" %m "%U" "%S" "%{User-agent}i" %b %s %D"/>'
replace='pattern="%{X-Forwarded-For}i %h %t %a %p %v %q"%{Referer}i" %m "%U" "%{User-agent}i" "%b" "%s" "%D""/>'
backup=yes
If the regex and replace pattern are same, then you can do like this:
- hosts: 192.168.8.11
user: mohitmehral
sudo: yes
tasks:
- replace:
dest="/home/{{ item }}/conf/server.xml"
regexp='pattern="%{X-Forwarded-For}i %h %t %a %p %v %q "%{Referer}i" %m "%U" "%S" "%{User-agent}i" %b %s %D"/>'
replace='pattern="%{X-Forwarded-For}i %h %t %a %p %v %q"%{Referer}i" %m "%U" "%{User-agent}i" "%b" "%s" "%D""/>'
backup=yes
with_items: [1,2,3,4,5]
I need to see my program's printf output in sync with the dtrace output.
I like to build my own version of dtrace command that produce the equivalent output of the "sudo dtruss -t write_nocancel ls" command.
This is the "correct dtruss command/output":
sudo dtruss -t write_nocancel ls
Chap1 Chap10 Chap11 Chap12 Chap2 Chap3 Chap4 Chap5 Chap6 Chap7 Chap8 Chap9 README
SYSCALL(args) = return
write_nocancel(0x1, "Chap1\tChap10\tChap11\tChap12\tChap2\tChap3\tChap4\tChap5\tChap6\tChap7\tChap8\tChap9\tREADME\n\0", 0x52) = 82 0
Base on looking at the dtruss script source code, I tried this dtrace command, but it failed.
sudo dtrace -q \
-n '*:*:write_nocancel:entry {self->arg0=arg0; self->arg1 =arg1; \
self->arg2 =arg2; self->code=0; } ' \
-n '*:*:write_nocancel:return { \
printf("return %s(0x%X, \"%S\", 0x%X) = %d %d", \
probefunc,self->arg0, arg0 == -1 ? "" : stringof(copyin(self->arg1,arg0)),self->arg2,(int)arg0, \
(int)errno); }' \
-c ls 2>&1
Chap1
Chap10
Chap11
Chap12
Chap2
Chap3
Chap4
Chap5
Chap6
Chap7
Chap8
Chap9
README
dtrace: error on enabled probe ID 3 (ID 209288: fbt:mach_kernel:write_nocancel:return): invalid address (0xffffff80218dfc40) in action #3 at DIF offset 92
dtrace: error on enabled probe ID 4 (ID 958: syscall::write_nocancel:return): invalid address (0xffffff80218dfc40) in action #3 at DIF offset 92
dtrace: error on enabled probe ID 3 (ID 209288: fbt:mach_kernel:write_nocancel:return): invalid address (0xffffff801a7c0010) in action #3 at DIF offset 92
Any dtrace experts out there might have a clue on how to fixe this?
Find the answer: (The issue of two -n options).
sudo dtrace -q -n \
'syscall::write_nocancel:entry{self->start = 1; \
self->vstart = 1; self->arg0 = arg0; \
self->arg1 = arg1; self->arg2 = arg2;} \
*:*:write_nocancel:return /self->start/ \
{ printf("return %s(0x%X, \"%S\", 0x%X) = %d %d" \
,probefunc,self->arg0, \
arg0 == -1 ? "" : stringof(copyin(self->arg1,arg0)),\
self->arg2,(int)arg0, (int)errno); }' \
-c ls 2>&1
Code typically speaks better than thousand bytes. So did this demo code that does not obey my logic. Typically I trust compilers and programs, so I think I have a typo or something somewhere.
Please, could you point it out to me?
copy & paste code below into mydemo.sh file, and run it using command sh ./mydemo.sh
in my ref system this prints out:
mytux:~# ./mydemo.sh
This should spit out:
{USER1NAME} {Firstname Surname} {user1name}
{USER2NAME} {Secondname Surname} {user2name}
{USER3NAME} {Thirdname Surname} {user3name}
---- but it spits out:
{USER1NAME} {somestring: user1name}, {user1name}
{USER2NAME} {somestring: user2name}, {user2name}
{USER3NAME} {somestring: user3name}, {user3name}
----
Why so and how to fix it?
and here's the code:
#!/bin/sh
echo "# Firstname Surname, Company">usernames.txt
echo "somestring: user1name">>usernames.txt
echo "# Secondname Surname, Company">>usernames.txt
echo "somestring: user2name">>usernames.txt
echo "# Thirdname Surname, Company">>usernames.txt
echo "somestring: user3name">>usernames.txt
echo "This should spit out:"
echo "{USER1NAME} {Firstname Surname}, {user1name}"
echo "{USER2NAME} {Secondname Surname}, {user2name}"
echo "{USER3NAME} {Thirdname Surname}, {user3name}"
echo "---- but it spits out:"
echo "{USER1NAME} {somestring: user1name}, {user1name}"
echo "{USER2NAME} {somestring: user2name}, {user2name}"
echo "{USER3NAME} {somestring: user3name}, {user3name}"
echo "---- See:"
cat usernames.txt|awk \
'BEGIN { $mylink="";} \
{ \
if(match($0,"^#")!=0) \
{ \
split($0, a, ","); \
$mylink=$a[1]; \
} \
else \
{ \
if(length($mylink)>0) \
{ \
print "{" toupper($2) "} {" $mylink "}, {" $2 "}"; \
} \
$mylink=""; \
} \
}'
echo "----"
echo "Why so and how to fix it?"
Is this what you're trying to achieve? It would be easier to post the input text as is.
$ awk -F"[, ]" -v OFS="}, {" '
/^#/{n=$2" "$3;next}
{print "{" toupper($2), n, $2"}"}
' usernames.txt
{USER1NAME}, {Firstname Surname}, {user1name}
{USER2NAME}, {Secondname Surname}, {user2name}
{USER3NAME}, {Thirdname Surname}, {user3name}