I'm trying to set up monitoring with logman, something like
logman create counter test_counter -f bin -si 05 -o output -cf counter.txt
In counter.txt, I'm specifying something like
\Process(*)\% User Time
\Process(*)\% Privileged Time
\Process(*)\Page Faults/sec
\Process(*)\Thread Count
\Process(*)\Handle Count
Is there any way to configure logman such that it only writes into output when value of a counter is larger than a threshold? for example when % User Time is bigger than 0.
This is because when machine has hundreds of processes, output size grows pretty fast and most of them are actually having 0 counter most of the time. We want to minimize this and only interested when processes are actually consuming resources.
Note, I see for alert we can do something like:
logman create alert new_alert -th \Processor(_Total)\% Processor time>50
Seems like the syntax is not applicable for Counter.
Related
Best case would be, if I had a (debug)-tool which runs in the background and tells me the name of the process or driver that breaks my latency requirement to my system. Which tool is suitable? Do you have a short example of its usage for the following case?
Test case:
The oscilloscope measures the time between the trigger of a GPIO input and the response on a GPIO output. Usually the response time is 150µs. I trigger every 25ms.
My linux user test program uses poll() and read()+write() to mirror the detected signal of the input as response back to an output.
The Linux kernel is patched with the Preempt_rt patch.
In the dimension of hours I can see response time peaks of up to 20ms.
The best real chance is to
switch on tracing in the kernel configuration and build such Linux kernel:
CONFIG_FTRACE=y
CONFIG_FUNCTION_TRACER=y
CONFIG_FUNCTION_GRAPH_TRACER=y
CONFIG_SCHED_TRACER=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_STACK_TRACER=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_FUNCTION_PROFILER=y
CONFIG_DEBUG_FS=y
then run your application until weird things happen by using a tool trace-cmd
trace-cmd start -b 10000 -e 'sched_wakeup*' -e sched_switch -e gpio_value -e irq_handler_entry -e irq_handler_exit /tmp/myUserApplication
and get a trace.dat file.
trace-cmd stop
trace-cmd extract
Load that trace.dat file in KernelShark and analyse the CPUs, threads, interrupts, kworker threads and user space threads. It's great to see which blocks the system.
I'm experimenting with 2 Gluster 3.7 servers in 1x2 configuration. Servers are connected over 1 Gbit network. I'm using Debian Jessie.
My use case is as follows: open file -> append 64 bytes -> close file and do this in a loop for about 5000 different files. Execution time for such loop is roughly 10 seconds if I access files through mounted glusterfs drive. If I use libgfsapi directly, execution time is about 5 seconds (2 times faster).
However, the same loop executes in 50ms on plain ext4 disk.
There is huge performance difference between Gluster 3.7 end earlier versions which is, I believe, due to the cluster.eager-lock setting.
My target is to execute the loop in less than 1 second.
I've tried to experiment with lots of Gluster settings but without success. dd tests with various bsize values behave like that TCP no-delay option is not set, although from Gluster source code it seems that no-delay is default.
Any idea how to improve the performance?
Edit:
I've found a solution that works in my case so I'd like to share it in case anyone else faces the same issue.
The root cause of the problem is the number of roundtrips between client and Gluster server during execution of open/write/close sequence. I don't know exactly what is happening behind but timing measurements shows exactly that pattern. Now, the obvious idea would be to "pack" open/write/close sequence into a single write function. Roughly, the C prototype of such function would be:
int write(const char* fname, const void *buf, size_t nbyte, off_t offset)
But, there is already such API function glfs_h_anonymous_write in libgfapi (thanks goes to Suomya from Gluster mailing group). Kind of hidden thing there is the file identifier which is not plain file name, but something of type struct glfs_object. Clients obtain an instance of such object through API calls glfs_h_lookupat/glfs_h_creat. The point here is that glfs_object representing filename is "stateless" in a sense that corresponding inode is left intact (not ref counted). One should think of glfs_object as plain filename identifier and use it as you would use filename (actually, glfs_object stores plain pointer to corresponding inode without ref counting it).
Finally, we should use glfs_h_lookupat/glfs_h_creat once and write many times to the file using glfs_h_anonymous_write.
That way I was able to append 64 bytes to 5000 files in 0.5 seconds, which is 20 times faster than using mounted volume and open//write/close sequence.
I was thinking of ways to have my laptop HDD backed up safely, and still being able to put the backup rapidly in use if needed.
My method would be the following: I would buy an 2.5" HDD of the same size with USB to SATA cable and clone the internal to it, when disaster strikes, I would just have to swap the HDD in the laptop for the other one and I would be good to go again.
However, I would like to avoid writing 500GB each time I want to backup my HDD, especially when I know that a fair part of it (+/- 80GB) is rarely written to, this is where the following md5sum/dd script comes to the rescue, I hope:
#!/bin/bash
block="1M"
end=50000
count=10
input="/dev/sda"
output="/dev/sdb"
output="/path/to/imagefile"
function md5compute()
{
dd if=$1 skip=$2 bs=$block count=$count | md5sum - | awk '{ print $1 }'
}
for i in {0..$end}
do
start=$(($i*$count))
md5source=$(md5compute $input $start)
md5destination=$(md5compute $output $start)
if [ "$md5source" != "$md5destination" ]
then
dd if=$input of=$output skip=$start seek=$start count=$count conv=sync,noerror,notrunc
fi
done
Now, the question part:
A) With running this, would I miss some part of the disk? Do you see some flaws?
B) Could I win some time compared to the 500GB read/write?
C) Obviously I potentially write less to the target disk. Will I improve the lifetime of that disk?
D) I was thinking of leaving count to 1, and increasing the block size.Is this good idea/bad idea?
E) Would this same script work with an image file as output?
Not being very fluent in programming, there should be plenty of room for improvement, any tips?
Thank you all...
Point by point answer:
With running this, would I miss some part of the disk?
no.
Do you see some flaws?
While units are differents. this implie double read at source and full read before write at destination. This will mostly improve backup time.
there is a little probability of having MD5 matching while differences exists. This probability is reducted by using SHA1 or MD256 or other harder checksum algo. but this implie more resource on both ends. (See Birthday problem on wikipedia)
Could I win some time compared to the 500GB read/write?
In case both units are already same, yes, because reading is generaly quicker than writting. (depending on processor, for checksum computation: this could be significant on very poor processors)
Obviously I potentially write less to the target disk. Will I improve the lifetime of that disk?
In this case, yes, but if you write only diff, this will go a lot faster and improve really you disks lifetime.
When disk are different, you re-write whole disk, this is not efficient!
I was thinking of leaving count to 1, and increasing the block size.Is this good idea/bad idea?
I find this globally a bad idea. Why re-inventing wheel?
Would this same script work with an image file as output?
Yes.
Functionnality answer.
For jobs like this, you may use rsync! With this tools you may
Compress data during transfer
copy over network
tunneling with SSH (or not)
transfer (an write) only modified blocks
Using bash, dd and md5sum
There is a kind of command I run sometime:
ssh $USER#$SOURCE "dd if=$PATH/$SRCDEV |tee >(sha1sum >/dev/stderr);sleep 1" |
tee >(sha1sum >/dev/tty) | dd of=$LOCALPATH/$LOCALDEV
This will do a full read on souce host, than a sha1sum before sending to localhost (destination), than a sha1sum to ensure transfer before writting to local device.
This may render something like:
2998920+0 records in
2998920+0 records out
1535447040 bytes (1.4gB) copied, 81.42039 s, 18.3 MB/s
d61c645ab2c561eb10eb31f12fbd6a7e6f42bf11 -
d61c645ab2c561eb10eb31f12fbd6a7e6f42bf11 -
2998920+0 records in
2998920+0 records out
1535447040 bytes (1.4gB) copied, 81.42039 s, 18.3 MB/s
I'm making an alarm clock and wanted to start at basically no volume and increase the sound per 2 seconds up one 'notch'/value until a certain predefined point where it won't increase anymore.
As of right now I'm using mplayer (it's a radio station, so I'm running mplayer http://66.225.205.192:80), but I don't care what I use (VLC, etc.))
My full code is
while true; do
mplayer http://66.225.205.192:80
sleep 1
done
Googling for 'mplayer alarm clock' actually yields a lot of pages dealing with this problem and solutions that you can actually use right away, but let's give it a try anyway.
#!/bin/bash
{
for ((volume = 0; volume <= 100; volume += 5)); do
/usr/bin/aumix -v${volume} -w100 >/dev/null
sleep 2
done
} &
mplayer http://66.225.205.192:80
echo "good morning! :-)"
You need to install aumix, which is used here to change the volume (but you could use something else, of course). The block between { } gets run in the background. The aumix command sets the PCM volume to 100% and gradually adjusts the main volume in 5 percent increments every two seconds, once it hits 100% the loop finishes and the background job exits.
I have never used aumix, and you might want to read its man page in case it does not work as expected (this is untested).
mplayer runs in the foreground until you quit it, upon which it makes coffee for you greets you with a warm welcome.
Does that get you started?
I'm trying use perf tool from the linux kernel package to measure several raw PMU events. In the manpage of perf-record there is an "-l" option (Scale counter values), which is useful for my case because I want to know the total counter value, not just sample count. However it seems the -l is not recognized, is this expected? How can I get a total count?
Another question is that how can I specify multiple events and use different sample-after value for each of them? like perf record -c 10000,2000000,2000000 -e r2d4,r03c,r0c0
thank you
in the example, I am showing I have already installed libpfm4 so that perf knows the user-friendly event names. There is a rather clumsy default syntax that can be used that allows the sampling period to be set per event
levinth#ubuntu18-2:~$ perf record -e cpu/inst_retired.any_p,period=2000000/,cpu/cpu_clk_unhalted.thread_any,period=3000000/ -- sleep 5
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.017 MB perf.data ]