I am searching all the Hadoop completed(100s of) jobs between a time interval. This time interval is in milliseconds.
Following is the format:
JobId State StartTime UserName Queue Priority UsedContainers RsvdContainers UsedMem RsvdMem NeededMem AM info
job_xxxxxxx SUCCEEDED 1458844667431 default default NORMAL N/A N/A N/A N/A N/A http://xxxxxxxx:8088/proxy/application_xxxxxxxxxx/jobhistory/job/job_xxxxxxxx
job_xxxxxxx SUCCEEDED 1459449718363 default default NORMAL N/A N/A N/A N/A N/A http://xxxxxx.xxxxx.com:8088/proxy/application_xxxxxxxxx/jobhistory/job/job_xxxxx
Following is my format:
STARTTIME="Tue Apr 12 10:24:29 EDT 2016"
ENDTIME="Tue Apr 12 15:24:29 EDT 2016"
CONVERTTIME_1=`date --date="$STARTTIME" +%s%3N`
CONVERTTIME_2=`date --date="$ENDTIME" +%s%3N`
echo $CONVERTTIME_1, $CONVERTTIME_2
mapred job -list all | sed -n '/$CONVERTTIME_1/,/$CONVERTTIME_2/p' > out
Output: all the jobs like above within that timerange.
Can anyone please help how to get these?
OUTPUT OF mapred job -list all
mapred job -list all
job_1457613852865_5163 SUCCEEDED 1459199337140 zzzzzzzzzz uuuuuuuu_critical NORMAL N/A N/A N/A N/A N/A http://xxxxx254.yyyyyy.com:8088/proxy/application_1457613852865_5163/jobhistory/job/job_1457613852865_5163
job_1457613852865_4633 SUCCEEDED 1458992402216 zzzyyyy yyyyyy_default NORMAL N/A N/A N/A N/A N/A http://xxxxx254.yyyyyy.com:8088/proxy/application_1457613852865_4633/jobhistory/job/job_1457613852865_4633
job_1457613852865_4821 SUCCEEDED 1459078845580 zzzyyyy yyyyyy_default NORMAL N/A N/A N/A N/A N/A http://xxxxx254.yyyyyy.com:8088/proxy/application_1457613852865_4821/jobhistory/job/job_1457613852865_4821
job_1457613852865_0322 SUCCEEDED 1457717313217 zzzddd uuuuuuuu_critical NORMAL N/A N/A N/A N/A N/A http://xxxxx254.yyyyyy.com:8088/proxy/application_1457613852865_0322/jobhistory/job/job_1457613852865_0322
job_1457613852865_5304 SUCCEEDED 1459254375921 zzzyyyy yyyyyy_default NORMAL N/A N/A N/A N/A N/A http://xxxxx254.yyyyyy.com:8088/proxy/application_1457613852865_5304/jobhistory/job/job_1457613852865_5304
job_1457613852865_8744 SUCCEEDED 1460195126188 zzzyyyy yyyyyy_default NORMAL N/A N/A N/A N/A N/A http://xxxxx254.yyyyyy.com:8088/proxy/application_1457613852865_8744/jobhistory/job/job_1457613852865_8744
job_1457613852865_3384 SUCCEEDED 1458649020794 zzzyyyy yyyyyy_default NORMAL N/A N/A N/A N/A N/A http://xxxxx254.yyyyyy.com:8088/proxy/application_1457613852865_3384/jobhistory/job/job_1457613852865_3384
job_1457613852865_9038 SUCCEEDED 1460291694279 zzzyyyy yyyyyy_default NORMAL N/A N/A N/A N/A N/A http://xxxxx254.yyyyyy.com:8088/proxy/application_1457613852865_9038/jobhistory/job/job_1457613852865_9038
job_1457613852865_8487 SUCCEEDED 1460115319590 zzzyyyy yyyyyy_default NORMAL N/A N/A N/A N/A N/A http://xxxxx254.yyyyyy.com:8088/proxy/application_1457613852865_8487/jobhistory/job/job_1457613852865_8487
job_1457613852865_8321 SUCCEEDED 1460038991587 dddyyy uuuuuuuu_critical NORMAL N/A N/A N/A N/A N/A http://xxxxx254.yyyyyy.com:8088/proxy/application_1457613852865_8321/jobhistory/job/job_1457613852865_8321
job_1457613852865_4661 SUCCEEDED 1458994901619 zzzyyyy yyyyyy_default NORMAL N/A N/A N/A N/A N/A http://xxxxx254.yyyyyy.com:8088/proxy/application_1457613852865_4661/jobhistory/job/job_1457613852865_4661
job_1457613852865_1975 SUCCEEDED 1458216683800 zzzyyyy yyyyyy_default NORMAL N/A N/A N/A N/A N/A http://xxxxx254.yyyyyy.com:8088/proxy/application_1457613852865_1975/jobhistory/job/job_1457613852865_1975
I used this:
#!/bin/bash
STARTTIME="Tue Apr 12 10:13:01 EDT 2016"
ENDTIME="Tue Apr 12 10:13:59 EDT 2016"
start=$(date -d "$STARTTIME" '+%s%3N')
end=$(date -d "$ENDTIME" '+%s%3N')
echo "start=$start :: end=$end"
mapred job -list all | awk -v start="$start" -v end="$end" '$3>=start && $3<=end'
Got one extra job:
job_1457613852865_9785 SUCCEEDED 1460470436726 yyyyyyyyyy nnnnnnnnnn NORMAL N/A N/A N/A N/A N/A http://888888.xxxxxxxxxx.com:8088/proxy/application_1457613852865_9785/jobhistory/job/job_1457613852865_9785
You can convert your dates into millisec value and then use awk to filter your data:
STARTTIME="Tue Apr 12 10:24:29 EDT 2016"
ENDTIME="Tue Apr 12 15:24:29 EDT 2016"
start=$(date -d "$STARTTIME" '+%s%3N')
end=$(date -d "$ENDTIME" '+%s%3N')"
echo "start=$start :: end=$end"
mapred job -list all | awk -v start="$start" -v end="$end" '$3>=start && $3<=end'
Related
I am trying to use Python-Libtorrent to download torrents. I have an issue where pausing the torrent doesn't work as expected:
import libtorrent as lt
import time
import sys
def get_libtorrent_session_and_start_dht(start_port, end_port):
ses = lt.session()
ses.listen_on(start_port, end_port)
ses.add_dht_router('dht.transmissionbt.com', start_port)
ses.add_dht_router('router.bittorrent.com', start_port)
ses.add_dht_router('router.utorrent.com', start_port)
ses.start_dht()
return ses
const_state_str = ['queued', 'checking', 'downloading metadata', \
'downloading', 'finished', 'seeding', 'allocating', 'checking fastresume']
const_pause_torrent = "pause_torrent"
const_resume_torrent = "resume_torrent"
const_kill_torrent = "kill_torrent"
const_user_logged_in = "user_logged_in"
const_user_logged_out = "user_logged_out"
const_quit_seeding = "quit_seeding"
ses = get_libtorrent_session_and_start_dht(6881, 6891)
torrent_info = None
torrent_handle = None
try:
torrent_info = lt.torrent_info("/home/horvste/Downloads/ubuntu-14.04.4-desktop-amd64.iso.torrent")
torrent_handle = ses.add_torrent(
{'ti': torrent_info, 'save_path': "/home/horvste/Downloads"})
except Exception as exception:
raise exception
while not torrent_handle.has_metadata():
print "Getting meta data"
time.sleep(1)
current_iteration = 0
while not torrent_handle.is_seed():
torrent_status = torrent_handle.status()
print "current_iteration: " + str(current_iteration)
if current_iteration == 10:
print "Calling torrent_handle.pause()...Pausing Torrent"
sys.stdout.flush()
torrent_handle.pause()
while not torrent_handle.status().paused:
print "Torrent Is Not Paused Yet"
time.sleep(1)
sys.stdout.flush()
while torrent_handle.status().paused:
print "Torrent Is Paused!"
torrent_handle.pause()
time.sleep(1)
sys.stdout.flush()
if const_state_str[torrent_status.state] == "checking":
print 'Checking Torrent....'
continue
print '\r%.2f%% complete (down: %.1f kb/s up: %.1f kB/s peers: %d) %s' % \
(torrent_status.progress * 100, torrent_status.download_rate / 1000, torrent_status.upload_rate / 1000, \
torrent_status.num_peers, const_state_str[torrent_status.state]), \
sys.stdout.flush()
current_iteration += 1
if torrent_status.paused:
print "Is Paused"
else:
print "Is Not Paused"
time.sleep(1)
Here is this scripts output:
...Previous Output Checking Torrent
Checking Torrent....
current_iteration: 0
4.17% complete (down: 0.0 kb/s up: 0.0 kB/s peers: 0) downloading None
Is Not Paused
current_iteration: 1
4.17% complete (down: 0.0 kb/s up: 0.0 kB/s peers: 0) downloading None
Is Not Paused
current_iteration: 2
4.17% complete (down: 0.0 kb/s up: 0.0 kB/s peers: 4) downloading None
Is Not Paused
current_iteration: 3
4.17% complete (down: 0.0 kb/s up: 0.0 kB/s peers: 7) downloading None
Is Not Paused
current_iteration: 4
4.17% complete (down: 1.0 kb/s up: 1.0 kB/s peers: 11) downloading None
Is Not Paused
current_iteration: 5
4.17% complete (down: 4.0 kb/s up: 2.0 kB/s peers: 16) downloading None
Is Not Paused
current_iteration: 6
4.17% complete (down: 52.0 kb/s up: 5.0 kB/s peers: 21) downloading None
Is Not Paused
current_iteration: 7
4.17% complete (down: 132.0 kb/s up: 8.0 kB/s peers: 28) downloading None
Is Not Paused
current_iteration: 8
4.26% complete (down: 234.0 kb/s up: 12.0 kB/s peers: 33) downloading None
Is Not Paused
current_iteration: 9
4.31% complete (down: 317.0 kb/s up: 15.0 kB/s peers: 38) downloading None
Is Not Paused
current_iteration: 10
Calling torrent_handle.pause()...Pausing Torrent
Torrent Is Paused!
Torrent Is Paused!
Torrent Is Paused!
4.31% complete (down: 418.0 kb/s up: 19.0 kB/s peers: 46) downloading None
Is Not Paused
current_iteration: 11
4.44% complete (down: 0.0 kb/s up: 0.0 kB/s peers: 10) downloading None
Is Not Paused
current_iteration: 12
4.44% complete (down: 0.0 kb/s up: 1.0 kB/s peers: 12) downloading None
Is Not Paused
current_iteration: 13
4.44% complete (down: 1.0 kb/s up: 2.0 kB/s peers: 21) downloading None
...Torrent keeps downloading
As you can see from the above script, I call pause on the torrent handle and it pauses for 3 iterations of the loop and starts downloading again. It is important to note that the previous script did not have the torrent_handle.pause() call in this while loop:
while torrent_handle.status().paused:
print "Torrent Is Paused!"
torrent_handle.pause()
time.sleep(1)
sys.stdout.flush()
I am still getting the same output; the torrent doesn't pause as expected. I am running Ubuntu 14.04.4 and everything is installed via apt-get. In the libtorrent init.py my version number is version = '0.16.13.0'. Am I missing something or misusing the library?
Torrents are auto managed by default and that could bring them out of pause, check flags in add_torrent_params
Hi and thanks in advance.
What is the difference between the memory in the tasklist ( which you run in the cmd) and that GUI task manager. I noticed for browser processes, that the memory is off by a great deal. Which is more accurate of the process's memory.
Task Manager has lots of memory counters see View menu - Select Columns.
The standard one shown is private working set. This is a/ private - so only bytes in memory specific to this program (so no shell32 common code is counted) and b/ working set - the amount of memory mapped and present in that processes address space.
Even memory not present in address space may be in physical memory as on the standby list or in the file cache or being used by another program. It only requires flicking a bit to make it available to the process. Run two copies of notepad, notepad in now in the file cache (and being small) in two processes. But the code is only in memory once not three times.
If you want to make your own tasklist.
Set objWMIService = GetObject("winmgmts:{impersonationLevel=impersonate}!\\.\root\cimv2")
Set colItems = objWMIService.ExecQuery("Select * From Win32_Process")
For Each objItem in colItems
' If objitem.Name = "mspaint.exe" Then
wscript.echo objitem.name & " PID=" & objItem.ProcessID & " SessionID=" & objitem.sessionid
' objitem.terminate
' End If
Next
Lines starting with a ' are commented out.
To use in a command prompt
cscript //nologo c:\path\script.vbs
These are the properties
Property Type Operation
======== ==== =========
CSName N/A N/A
CommandLine N/A N/A
Description N/A N/A
ExecutablePath N/A N/A
ExecutionState N/A N/A
Handle N/A N/A
HandleCount N/A N/A
InstallDate N/A N/A
KernelModeTime N/A N/A
MaximumWorkingSetSize N/A N/A
MinimumWorkingSetSize N/A N/A
Name N/A N/A
OSName N/A N/A
OtherOperationCount N/A N/A
OtherTransferCount N/A N/A
PageFaults N/A N/A
PageFileUsage N/A N/A
ParentProcessId N/A N/A
PeakPageFileUsage N/A N/A
PeakVirtualSize N/A N/A
PeakWorkingSetSize N/A N/A
Priority N/A N/A
PrivatePageCount N/A N/A
ProcessId N/A N/A
QuotaNonPagedPoolUsage N/A N/A
QuotaPagedPoolUsage N/A N/A
QuotaPeakNonPagedPoolUsage N/A N/A
QuotaPeakPagedPoolUsage N/A N/A
ReadOperationCount N/A N/A
ReadTransferCount N/A N/A
SessionId N/A N/A
Status N/A N/A
TerminationDate N/A N/A
ThreadCount N/A N/A
UserModeTime N/A N/A
VirtualSize N/A N/A
WindowsVersion N/A N/A
WorkingSetSize N/A N/A
WriteOperationCount N/A N/A
WriteTransferCount N/A N/A
And the methods
Call [ In/Out ]Params&type Status
==== ===================== ======
AttachDebugger (null)
Create [IN ]CommandLine(STRING) (null)
[IN ]CurrentDirectory(STRING)
[IN ]ProcessStartupInformation(OBJECT)
[OUT]ProcessId(UINT32)
GetOwner [OUT]Domain(STRING) (null)
[OUT]User(STRING)
GetOwnerSid [OUT]Sid(STRING) (null)
SetPriority [IN ]Priority(SINT32) (null)
Terminate [IN ]Reason(UINT32) (null)
Which is the same as
wmic process where name='notepad.exe' get /format:list
Further reading
https://msdn.microsoft.com/en-us/library/ms810627.aspx
https://www.labri.fr/perso/betrema/winnt/ntvmm.html (this no longer appears on the MSDN)
I fired the following command on elasticsearch
PUT /_cluster/settings
{
"persistent" : {
"threadpool.index.queue_size": -1
}
}
But now my elastuicsearch cluster health is stuck in yellow. Nothing is moving, and the cat recovery api gives following
books 0 613 replica done server1.internal.com server2.internal.com n/a n/a 1 100.0% 79 100.0%
books 0 53479 replica done server2.internal.com server3.internal.com n/a n/a 146 100.0% 435062890 100.0%
books 1 592 replica done server1.internal.com server2.internal.com n/a n/a 1 100.0% 79 100.0%
books 1 5901 replica done server2.internal.com server3.internal.com n/a n/a 198 34.3% 449403096 2.7%
books 2 551 replica done server1.internal.com server2.internal.com n/a n/a 1 100.0% 79 100.0%
books 2 9018 replica done server2.internal.com server3.internal.com n/a n/a 201 28.9% 451881473 4.4%
books 3 519 replica done server1.internal.com server2.internal.com n/a n/a 1 100.0% 79 100.0%
books 3 3869 replica done server2.internal.com server3.internal.com n/a n/a 170 61.8% 434156880 1.2%
books 4 525 replica done server1.internal.com server2.internal.com n/a n/a 1 100.0% 79 100.0%
books 4 33468 replica done server2.internal.com server3.internal.com n/a n/a 136 100.0% 428616146 100.0%
For about 30 minutes there has been no progress.
Can anyone help me how to solve this?
I just figured out this was due to corrupted disk on one of the nodes.
Classification of Elasticsearch Cluster Status:
RED: Some or all of (primary) shards are not ready.
YELLOW: Elasticsearch has allocated all of the primary shards, but some/all of the replicas have not been allocated.
GREEN: Cluster is fully operational. Elasticsearch is able to allocate all shards and replicas to machines within the cluster.
I am writing data in a text file as below by running a script. This data is updated every second.
eth0: Sent Bytes: 1 Kb/s | Received Bytes: 2 Kb/s | Sent Packets: 18 Pkts/s | Received Packets: 13 Pkts/s
eth0: Sent Bytes: 1 Kb/s | Received Bytes: 2 Kb/s | Sent Packets: 18 Pkts/s | Received Packets: 12 Pkts/s
eth0: Sent Bytes: 1 Kb/s | Received Bytes: 3 Kb/s | Sent Packets: 20 Pkts/s | Received Packets: 13 Pkts/s
eth0: Sent Bytes: 15 Kb/s | Received Bytes: 4 Kb/s | Sent Packets: 33 Pkts/s | Received Packets: 25 Pkts/s
eth0: Sent Bytes: 1 Kb/s | Received Bytes: 3 Kb/s | Sent Packets: 19 Pkts/s | Received Packets: 12 Pkts/s
I want to make a graph of the # of bytes sent and the # of bytes received. Same for packets.
You can use https://github.com/holman/spark to create a graph with just shell script (although it only works with bash and not POSIX sh). You can watch it update in real time with watch.
graph.sh
!/bin/sh
field=1
tail "$1" | cut -d '|' -f $field | sed -e 's!.*: \([0-9]\+\) .*!\1!' | ./spark/spark
Interactive console
git clone https://github.com/holman/spark
your-process > logfile &
watch sh graph.sh logfile
Output
Every 2.0s: sh graph.sh logfile Fri Dec 19 22:22:04 2014
▁▁▁█▁
adb shell dumpsys meminfo of my package shows the following and my native allocated size increases and finally causing mobile restart. Is that any memory ? How can i fix that??
native dalvik other total limit bitmap nativeBmp
size: 445456 5955 N/A 451411 32768 N/A N/A
allocated: 445024 3726 N/A 448750 N/A 10948 1912
free: 43 2229 N/A 2272 N/A N/A N/A
(Pss): 132631 870 300292 433793 N/A N/A N/A
(shared dirty): 2532 1656 5552 9740 N/A N/A N/A
(priv dirty): 132396 708 298960 432064 N/A N/A N/A
Objects
Views: 0 ViewRoots: 0
AppContexts: 0 Activities: 0
Assets: 6 AssetManagers: 6
Local Binders: 5 Proxy Binders: 14
Death Recipients: 1
OpenSSL Sockets: 0
SQL
heap: 0 MEMORY_USED: 0
PAGECACHE_OVERFLOW: 0 MALLOC_SIZE: 50
Asset Allocations
zip:/data/app/com.outlook.screens-2.apk:/resources.arsc: 25K
zip:/data/app/com.outlook.screens-2.apk:/assets/font/RobotoCondensedRegular.
ttf: 156K
zip:/data/app/com.outlook.screens-2.apk:/assets/font/RobotoCondensedBold.ttf
: 158K
zip:/data/app/com.outlook.screens-2.apk:/assets/font/RobotoCondensedLight.tt
f: 157K
Uptime: 190161845 Realtime now=432619753