For MacOS (Mavericks), I am making a shell script to gather transfer stats over time for command dd.
The manual page says:
If dd receives a SIGINFO (see the status argument for stty(1)) signal,
the current input and output block counts will be written to the
standard error output in the same format as the standard completion
message.
Therefore, just like in Linux, I tried:
kill -INFO <pid_of_dd>
The command completes successfully with status 0, however the terminal in which dd process connected to, there is no stats information in standard output/standard error.
So what is the correct way to get dd to print stats in its output?
You can also press Ctrl+T in the Terminal tab to get the same behavior:
MacBook-Pro:~ $ dd if=~/source_image.dmg of=/dev/disk1
load: 0.87 cmd: dd 7229 uninterruptible 0.21u 3.91s
265809+0 records in
265808+0 records out
136093696 bytes transferred in 131.170628 secs (1037532 bytes/sec)
load: 0.99 cmd: dd 7229 uninterruptible 0.32u 5.89s
415769+0 records in
415768+0 records out
212873216 bytes transferred in 203.357068 secs (1046795 bytes/sec)
It seems to work for me:
$ dd if=/dev/zero of=/dev/null bs=1k &
[1] 33990
$ kill -INFO 33990
4787784+0 records in
4787784+0 records out
4902690816 bytes transferred in 4.260769 secs (1150658706 bytes/sec)
$ kill -INFO 33990
8357846+0 records in
8357846+0 records out
8558434304 bytes transferred in 7.428820 secs (1152058392 bytes/sec)
$ kill 33990
$ ps
PID TTY TIME CMD
1342 ttys000 0:00.02 -bash
2290 ttys001 0:00.17 -bash
[1]+ Terminated: 15 dd if=/dev/zero of=/dev/null bs=1k
$
I also found via commandlinefu that you can also do:
killall -INFO dd
If you had to run sudo dd to start dd you might try:
sudo killall -INFO dd
Also, I started dd in the background and with nohup so when I ran sudo killall -INFO dd and got nothing back for output I had to remember to go and look at the nohup.out file because that is where the response was logged to.
Worked great on OS X Mavericks.
You can press Ctrl+T while the dd command is running or, to have a nice progress bar, you can install pv (pipe viewer) with homebrew:
brew install pv
and then place pv in between
dd if=diskimage.img | pv | dd of=/dev/disk2
example output 1
18MB 0:00:11 [1.70MiB/s] [ <=> ]
(with size of transferred data, elapsed time and speed)
Progress bar and ETA
you can also input the size of the image (16GB in this example), to have :
dd if=diskimage.img | pv -s 16G | dd of=/dev/disk2
example output 2 (with also progress bar and estimated time):
1.61GiB 0:12:19 [2.82MiB/s] [===> ] 10% ETA 1:50:25
Related
I've huge size(few million) job contain list and wants to run java written tool to perform the features comparison. This tool completes the calculation in
real 0m0.179s
user 0m0.005s
sys 0m0.000s sec
Running 5 nodes(each have 72 cpus) with pbs torque scheduler in the GNU parallel, tool runs fine and produces the results but as I set 72 jobs per node, it should run 72 x 5 jobs at a time but I can see only it runs 25-35 jobs!
Checking of cpu utilization on each node also shows low utilization.
I desire to run 72 X 5 jobs or more at a time and produce the results by utilizing all the available source (72 X 5 cpus).
As I mentioned have ~200 millions of job to run, I desire to complete it faster(1-2 hours) by using/increasing the number of nodes/cpus.
Current code, input and job state:
example.lst (it has ~300 million lines)
ZNF512-xxxx_2_N-THRA-xxtx_2_N
ZNF512-xxxx_2_N-THRA-xxtx_3_N
ZNF512-xxxx_2_N-THRA-xxtx_4_N
.......
cat job_script.sh
#!/bin/bash
#PBS -l nodes=5:ppn=72
#PBS -N job01
#PBS -j oe
#work dir
export WDIR=/shared/data/work_dir
cd $WDIR;
# use available 72 cpu in each node
export JOBS_PER_NODE=72
#gnu parallel command
parallelrun="parallel -j $JOBS_PER_NODE --slf $PBS_NODEFILE --wd $WDIR --joblog process.log --resume"
$parallelrun -a example.lst sh run_script.sh {}
cat run_script.sh
#!/bin/bash
# parallel command options
i=$1
data=/shared/TF_data
# create tmp dir and work in
TMP_DIR=/shared/data/work_dir/$i
mkdir -p $TMP_DIR
cd $TMP_DIR/
# get file name
mk=$(echo "$i" | cut -d- -f1-2)
nk=$(echo "$i" | cut -d- -f3-6)
#run a tool to compare the features of pair files
/shared/software/tool_v2.1/tool -s1 $data/inf_tf/$mk -s1cf $data/features/$mk-cf -s1ss $data/features/$mk-ss -s2 $data/inf_tf/$nk.pdb -s2cf $data/features/$nk-cf.pdb -s2ss $data/features/$nk-ss.pdb > $data/$i.out
# move output files
mv matrix.txt $data/glosa_tf/matrix/$mk"_"$nk.txt
mv ali_struct.pdb $data/glosa_tf/aligned/$nk"_"$mk.pdb
# move back and remove tmp dir
cd $TMP_DIR/../
rm -rf $TMP_DIR
exit 0
PBS submission
qsub job_script.sh
Login to one of the node : ssh ip-172-31-9-208
top - 09:28:03 up 15 min, 1 user, load average: 14.77, 13.44, 8.08
Tasks: 928 total, 1 running, 434 sleeping, 0 stopped, 166 zombie
Cpu(s): 0.1%us, 0.1%sy, 0.0%ni, 98.4%id, 1.4%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 193694612k total, 1811200k used, 191883412k free, 94680k buffers
Swap: 0k total, 0k used, 0k free, 707960k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15348 ec2-user 20 0 16028 2820 1820 R 0.3 0.0 0:00.10 top
15621 ec2-user 20 0 169m 7584 6684 S 0.3 0.0 0:00.01 ssh
15625 ec2-user 20 0 171m 7472 6552 S 0.3 0.0 0:00.01 ssh
15626 ec2-user 20 0 126m 3924 3492 S 0.3 0.0 0:00.01 perl
.....
All of the nodes top shows the similar state and produces the results by running only ~26 at a time!
I've aws-parallelcluster contains 5 nodes(each have 72 cpus) with torque scheduler and GNU Parallel 2018, Mar 2018
Update
By introducing the new function that takes input on stdin and running the script in parallel works great and utilizes all the CPU in local machine.
However, when its runs over remote machines it produces a
parallel: Error: test.lst is neither a file nor a block device
MCVE:
A simple code that echoing list gives the same error while running it in remote machines but works great in local machine:
cat test.lst # contains list
DNMT3L-5yx2B_1_N-DNMT3L-5yx2B_2_N
DNMT3L-5yx2B_1_N-DNMT3L-6brrC_3_N
DNMT3L-5yx2B_1_N-DNMT3L-6f57B_2_N
DNMT3L-5yx2B_1_N-DNMT3L-6f57C_2_N
DNMT3L-5yx2B_1_N-DUX4-6e8cA_4_N
DNMT3L-5yx2B_1_N-E2F8-4yo2A_3_P
DNMT3L-5yx2B_1_N-E2F8-4yo2A_6_N
DNMT3L-5yx2B_1_N-EBF3-3n50A_2_N
DNMT3L-5yx2B_1_N-ELK4-1k6oA_3_N
DNMT3L-5yx2B_1_N-EPAS1-1p97A_1_N
cat test_job.sh # GNU parallel submission script
#!/bin/bash
#PBS -l nodes=1:ppn=72
#PBS -N test
#PBS -k oe
# introduce new function and Run from ~/
dowork() {
parallel sh test_work.sh {}
}
export -f dowork
parallel -a test.lst --env dowork --pipepart --slf $PBS_NODEFILE --block -10 dowork
cat test_work.sh # run/work script
#!/bin/bash
i=$1
data=pwd
#create temporary folder in current dir
TMP_DIR=$data/$i
mkdir -p $TMP_DIR
cd $TMP_DIR/
# split list
mk=$(echo "$i" | cut -d- -f1-2)
nk=$(echo "$i" | cut -d- -f3-6)
# echo list and save in echo_test.out
echo $mk, $nk >> $data/echo_test.out
cd $TMP_DIR/../
rm -rf $TMP_DIR
From your timing:
real 0m0.179s
user 0m0.005s
sys 0m0.000s sec
it seems the tool uses very little CPU power. When GNU Parallel runs local jobs it has an overhead of 10 ms CPU time per job. Your jobs use 179 ms time, and 5 ms CPU time. So GNU Parallel will be using quite a bit of the time spent.
The overhead is much worse when running jobs remotely. Here we are talking 10 ms + running an ssh command. This can easily be in the order of 100 ms.
So how can we minimize the number of ssh commands and how can spread the overhead over multiple cores?
First let us make a function that can take input on stdin and run the script - one job per CPU thread in parallel:
dowork() {
[...set variables here. that becomes particularly important we when run remotely...]
parallel sh run_script.sh {}
}
export -f dowork
Test that this actually works by running:
head -n 1000 example.lst | dowork
Then let us look at running jobs locally. This can be done similar to described here: https://www.gnu.org/software/parallel/man.html#EXAMPLE:-Running-more-than-250-jobs-workaround
parallel -a example.lst --pipepart --block -10 dowork
This will split example.lst into 10 blocks per CPU thread. So on a machine with 72 CPU threads this will make 720 blocks. It will the start 72 doworks and when one is done it will get another of the 720 blocks. The reason I choose 10 instead of 1 is if one of the jobs "get stuck" for a while, then you are unlikely to notice this.
This should make sure 100% of the CPUs on the local machine is busy.
If that works, we need to distribute this work to remote machines:
parallel -j1 -a example.lst --env dowork --pipepart --slf $PBS_NODEFILE --block -10 dowork
This should in total start 10 ssh per CPU thread (i.e. 5*72*10) - namely one for each block. With 1 running per server listed in $PBS_NODEFILE in parallel.
Unfortunately this means that --joblog and --resume will not work. There is currently no way to make that work, but if it is valuable to you contact me via parallel#gnu.org.
I am not sure what tool does. But if the copying takes most of the time and if tool only reads the files, then you might just be able symlink the files into $TMP_DIR instead of copying.
A good indication of whether you can do it faster is to look at top of the 5 machines in the cluster. If they are all using all cores at >90% then you cannot expect to get it faster.
I need to log some data from a serial device. So I would like to first call minicom with the & parameter:
minicom -D /dev/ttyXYZ -b 115200 -C logFile &
But taking a look to ps aux shows, that minicom had become a terminated zombie process:
edeviser 8835 0.0 0.0 19696 2628 pts/0 T 15:29 0:00 minicom -D /dev/ttyXYZ -b 115200 -C logFile
How to send minicom to background?
Further information:
I would like to send it to background because I'll need to trigger some actions after minicom has started to log the serial data to the logFile. Using cat /dev/ttyXYZ > logFile is no option, because I must specify the baudrate. Using a second terminal is also no option, because this work will be done by a bash script.
Have you tried:
nohup minicom -D /dev/ttyXYZ -b 115200 -C logFile &
I ran into the same issue when I was trying to convert movies with handbrake-cli.
nohup will background the task and push the output into a txt file called nohup.txt.
Oftentimes I want to post something to a github bug like
$ ping google.com
PING google.com (216.58.195.238): 56 data bytes
Request timeout for icmp_seq 0
64 bytes from 216.58.195.238: icmp_seq=0 ttl=53 time=1064.747 ms
Right now I run the command, use screen's C-a C-[ to highlight the area, enter to copy it to that buffer, paste it into vim, write it to a file and then cat that into pbcopy. There has to be a better way.
Is there a command I can run which will tee the command I type prefixed with a $ and all the output to pbcopy? Or anything close? I envision
$ demo ping google.com
PING google.com (216.58.195.238): 56 data bytes
Request timeout for icmp_seq 0
64 bytes from 216.58.195.238: icmp_seq=0 ttl=53 time=1064.747 ms
^C
$
and now the original thing I pasted is in my mac clipboard.
You can do
script log.txt
ping www.google.com
exit
And you'll have your command and output saved in log.txt
Edit
Based on your comment, what you want is
command="whatever command you want to run"
echo \$ $command > log.txt
$command >> log.txt
I don't think you'll find a single command that does exactly this.
I'm trying to copy a .img of Ubuntu 14.04.1 to my bootable usb using the command sudo dd if=~/Documents/targetUbuntu.img of=/dev/rdisk1 bs=1m but it's taking long and I can't see the progress. So i'm trying to use pv using this command sudo dd if=~/Documents/targetUbuntu.img | pv | dd of=/dev/rdisk1 bs=1m but im getting this error: dd: /dev/rdisk1: Permission denied. If I do do ctrl-C in the first scenario where its taking too long it tells me that it copied over X amount of bytes in X secs and that's it. When I try to boot the usb, it says "isolinux.iso is missing or corrupt". So I want to make sure that file is copying over properly and I want to do that by using pv to check the progress, but I keep getting that error. Any solutions?
Add sudo to the second "dd" command instead.
dd if=~/Documents/targetUbuntu.img | pv | sudo dd of=/dev/rdisk1 bs=1m
You'll also want to provide "size" to pv command (replace 123456 with file size.)
dd if=~/Documents/targetUbuntu.img | pv -s 123456 | sudo dd of=/dev/rdisk1 bs=1m
I suspect its because the sudo is only being used for the first command in your pipe, where you are using dd to read the image file.
See this answer over on the unix SE
Or you can try instead something like:
sudo "pv -tpreb myubuntu.img | dd of=/dev/sdc
per post by Greg Kroah-Hartman on g+
I've read this and several other articles but I can't figure this out. This is what I've put together:
sudo nohup or nohup sudo (I've tried both) the command without the background & (so you can enter your password before detaching it)
Enter the password
^Z then bg
disown the pid (doesn't work for some reason)
Log out
Here's the output:
[user#localhost ~]$ nohup sudo dd if=/dev/zero of=/dev/sda bs=1M
nohup: ignoring input and appending output to ‘nohup.out’
[sudo] password for user:
^Z
[1]+ Stopped nohup sudo dd if=/dev/zero of=/dev/sda bs=1M
[user#localhost ~]$ bg
[1]+ nohup sudo dd if=/dev/zero of=/dev/sda bs=1M &
[user#localhost ~]$ sudo ps | grep dd
2458 pts/32 00:00:00 dd
[user#localhost ~]$ disown 2458
-bash: disown: 2458: no such job
[user#localhost ~]$ logout
Connection to [server] closed.
$ remote
Last login: Mon Feb 3 11:32:59 2014 from [...]
[user#localhost ~]$ sudo ps | grep dd
[sudo] password for user:
[user#localhost ~]$
So the process is gone. I've tried a few other combinations with no success. Any ideas to make this work?
Use the -b option to sudo to instruct it to run the given command in the background. Job control doesn't work because the nohup process is not a child of the current shell, but of the sudo process.
sudo -b nohup dd if=/dev/zero of=/dev/sda bs=1M
You are not using disown correctly, the right argument should be the jobspec ('%' + job number).
In your example you should have disown %1 instead of disown 2458.
For listing your current shell jobs list, you can use the bash builtin jobs