I am a newbie with Varnish. I have successfully installed it and now its working, but I need some guidance from the more knowledgeable people about how the server is performing.
I read this article - http://kristianlyng.wordpress.com/2009/12/08/varnishstat-for-dummies/ but I am still not sure howz the server performance.
The server has been running since last 9 hours. I understand that more content will be cached with time so cache hit ratio will better, but right now my concern is about intermediate help from your side on server performance.
Hitrate ratio: 10 100 613
Hitrate avg: 0.2703 0.3429 0.4513
239479 8.00 7.99 client_conn - Client connections accepted
541129 13.00 18.06 client_req - Client requests received
157594 1.00 5.26 cache_hit - Cache hits
3 0.00 0.00 cache_hitpass - Cache hits for pass
313499 9.00 10.46 cache_miss - Cache misses
67377 4.00 2.25 backend_conn - Backend conn. success
316739 7.00 10.57 backend_reuse - Backend conn. reuses
910 0.00 0.03 backend_toolate - Backend conn. was closed
317652 8.00 10.60 backend_recycle - Backend conn. recycles
584 0.00 0.02 backend_retry - Backend conn. retry
3 0.00 0.00 fetch_head - Fetch head
314040 9.00 10.48 fetch_length - Fetch with Length
4139 0.00 0.14 fetch_chunked - Fetch chunked
5 0.00 0.00 fetch_close - Fetch wanted close
386 . . n_sess_mem - N struct sess_mem
55 . . n_sess - N struct sess
313452 . . n_object - N struct object
313479 . . n_objectcore - N struct objectcore
38474 . . n_objecthead - N struct objecthead
368 . . n_waitinglist - N struct waitinglist
12 . . n_vbc - N struct vbc
61 . . n_wrk - N worker threads
344 0.00 0.01 n_wrk_create - N worker threads created
2935 0.00 0.10 n_wrk_queued - N queued work requests
1 . . n_backend - N backends
47 . . n_expired - N expired objects
149425 . . n_lru_moved - N LRU moved objects
1 0.00 0.00 losthdr - HTTP header overflows
461727 10.00 15.41 n_objwrite - Objects sent with write
239468 8.00 7.99 s_sess - Total Sessions
541129 13.00 18.06 s_req - Total Requests
64678 3.00 2.16 s_pipe - Total pipe
5346 0.00 0.18 s_pass - Total pass
318187 9.00 10.62 s_fetch - Total fetch
193589421 3895.84 6459.66 s_hdrbytes - Total header bytes
4931971067 14137.41 164569.09 s_bodybytes - Total body bytes
117585 3.00 3.92 sess_closed - Session Closed
2283 0.00 0.08 sess_pipeline - Session Pipeline
892 0.00 0.03 sess_readahead - Session Read Ahead
458468 10.00 15.30 sess_linger - Session Linger
414010 9.00 13.81 sess_herd - Session herd
36912073 880.96 1231.68 shm_records - SHM records
What VCL are you using? If the answer is 'none' then you are probably not getting a very good hitrate. On a fresh install, Varnish is quite conservative about what it caches (and rightly so), but you can probably improve matters by reading how to achieve a high hitrate. If it's safe to, you can selectively unset cookies and normalise requests with your VCL, which will result in fewer backend calls.
How much of your website is cacheable? Is your object cache big enough? If you can answer those two questions, you ought to be able to achieve a great hitrate with Varnish.
Related
In an effort to become better with Haskell, I'm rewriting a small CLI that I developed originally in Python. The CLI mostly makes GET requests to an API and allows for filtering/formatting the JSON result.
I'm finding my Haskell version to be a lot slower than my Python one.
To help narrow down the problem, I excluded all parts of my Haskell code except the fetching of data - essentially, it's this:
import Data.Aeson
import qualified Data.ByteString.Char8 as BC
import Data.List (intercalate)
import Network.HTTP.Simple
...
-- For testing purposes
getUsers :: [Handle] -> IO ()
getUsers hs = do
let handles = BC.pack $ intercalate ";" hs
req <- parseRequest (baseUrl ++ "/user.info")
let request = setRequestQueryString [("handles", Just handles)] $ req
response <- httpJSON request
let (usrs :: Maybe (MyApiResponseType [User])) = getResponseBody response
print usrs
And I'm using the following dependencies:
dependencies:
- base >= 4.7 && < 5
- aeson
- bytestring
- http-conduit
To test this, I timed how long it takes for my Haskell program to retrieve data for a particular user (without any particular formatting). I compared it with my Python version (which formats the data), and Curl (which I piped into jq to format the data):
I ran each 5 times and took the average of the 3 middle values, excluding the highest and lowest times:
Haskell Python Curl
real: 1017 ms 568 ms 214 ms
user: 1062 ms 367 ms 26 ms
sys: 210 ms 45 ms 10 ms
Ok, so the Haskell version is definitely slower. Next I tried profiling tools to narrow down the cause of the problem.
I profiled the code using an SCC annotation for the function above:
> stack build --profile
...
> stack exec --profile -- my-cli-exe +RTS -p -sstderr
...
244,904,040 bytes allocated in the heap
27,759,640 bytes copied during GC
5,771,840 bytes maximum residency (6 sample(s))
245,912 bytes maximum slop
28 MiB total memory in use (0 MB lost due to fragmentation)
Tot time (elapsed) Avg pause Max pause
Gen 0 228 colls, 228 par 0.849s 0.212s 0.0009s 0.0185s
Gen 1 6 colls, 5 par 0.090s 0.023s 0.0038s 0.0078s
Parallel GC work balance: 30.54% (serial 0%, perfect 100%)
TASKS: 21 (1 bound, 20 peak workers (20 total), using -N8)
SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)
INIT time 0.004s ( 0.003s elapsed)
MUT time 0.881s ( 0.534s elapsed)
GC time 0.939s ( 0.235s elapsed)
RP time 0.000s ( 0.000s elapsed)
PROF time 0.000s ( 0.000s elapsed)
EXIT time 0.010s ( 0.001s elapsed)
Total time 1.833s ( 0.773s elapsed)
Alloc rate 277,931,867 bytes per MUT second
Productivity 48.1% of total user, 69.1% of total elapsed
Seems like a lot of time is being spent in garbage collection.
I looked at the generated .prof file, which gave this:
COST CENTRE MODULE SRC %time %alloc
>>=.\.ks' Data.ASN1.Get Data/ASN1/Get.hs:104:13-61 10.2 9.8
fromBase64.decode4 Data.Memory.Encoding.Base64 Data/Memory/Encoding/Base64.hs:(299,9)-(309,37) 9.0 12.3
>>=.\ Data.ASN1.Parse Data/ASN1/Parse.hs:(54,9)-(56,43) 5.4 0.7
fromBase64.loop Data.Memory.Encoding.Base64 Data/Memory/Encoding/Base64.hs:(264,9)-(296,45) 4.2 7.4
>>=.\ Data.ASN1.Get Data/ASN1/Get.hs:(104,9)-(105,38) 4.2 3.5
decodeSignedObject.onContainer Data.X509.Signed Data/X509/Signed.hs:(167,9)-(171,30) 3.6 2.9
runParseState.go Data.ASN1.BinaryEncoding.Parse Data/ASN1/BinaryEncoding/Parse.hs:(98,12)-(129,127) 3.0 3.2
getConstructedEndRepr.getEnd Data.ASN1.Stream Data/ASN1/Stream.hs:(37,11)-(41,82) 3.0 12.7
getConstructedEnd Data.ASN1.Stream Data/ASN1/Stream.hs:(23,1)-(28,93) 3.0 7.8
readCertificates Data.X509.CertificateStore Data/X509/CertificateStore.hs:(92,1)-(96,33) 3.0 2.2
fmap.\.ks' Data.ASN1.Get Data/ASN1/Get.hs:88:13-52 1.8 2.2
decodeConstruction Data.ASN1.BinaryEncoding Data/ASN1/BinaryEncoding.hs:(48,1)-(50,66) 1.8 0.0
fmap Data.ASN1.Parse Data/ASN1/Parse.hs:41:5-57 1.8 1.0
concat.loopCopy Data.ByteArray.Methods Data/ByteArray/Methods.hs:(210,5)-(215,28) 1.2 0.4
fromBase64.rset Data.Memory.Encoding.Base64 Data/Memory/Encoding/Base64.hs:(312,9)-(314,53) 1.2 0.0
localTimeParseE.allDigits Data.Hourglass.Format Data/Hourglass/Format.hs:358:9-37 1.2 0.3
getWord8 Data.ASN1.Get Data/ASN1/Get.hs:(200,1)-(204,43) 1.2 0.0
fmap.\ Data.ASN1.Get Data/ASN1/Get.hs:(88,9)-(89,38) 1.2 0.6
runParseState.runGetHeader.\ Data.ASN1.BinaryEncoding.Parse Data/ASN1/BinaryEncoding/Parse.hs:131:44-66 1.2 0.0
mplusEither Data.ASN1.BinaryEncoding.Parse Data/ASN1/BinaryEncoding/Parse.hs:(67,1)-(70,45) 1.2 4.9
getOID.groupOID Data.ASN1.Prim Data/ASN1/Prim.hs:299:9-92 1.2 0.3
getConstructedEndRepr.getEnd.zs Data.ASN1.Stream Data/ASN1/Stream.hs:40:48-73 1.2 0.0
getConstructedEndRepr.getEnd.(...) Data.ASN1.Stream Data/ASN1/Stream.hs:40:48-73 1.2 0.4
getConstructedEnd.(...) Data.ASN1.Stream Data/ASN1/Stream.hs:28:48-80 1.2 0.3
decodeEventASN1Repr.loop Data.ASN1.BinaryEncoding Data/ASN1/BinaryEncoding.hs:(54,11)-(67,69) 1.2 2.5
put Data.ASN1.Parse Data/ASN1/Parse.hs:(72,1)-(74,24) 1.2 0.0
fromASN1 Data.X509.ExtensionRaw Data/X509/ExtensionRaw.hs:(55,5)-(61,71) 1.2 0.0
compare Data.X509.DistinguishedName Data/X509/DistinguishedName.hs:31:23-25 1.2 0.0
putBinaryVersion Network.TLS.Packet Network/TLS/Packet.hs:(109,1)-(110,41) 1.2 0.0
parseLBS.onSuccess Data.ASN1.BinaryEncoding.Parse Data/ASN1/BinaryEncoding/Parse.hs:(147,11)-(149,64) 0.6 1.7
pemParseLBS Data.PEM.Parser Data/PEM/Parser.hs:(92,1)-(97,41) 0.6 1.0
runParseState.terminateAugment Data.ASN1.BinaryEncoding.Parse Data/ASN1/BinaryEncoding/Parse.hs:(87,12)-(93,53) 0.0 1.7
parseOnePEM.getPemContent Data.PEM.Parser Data/PEM/Parser.hs:(56,9)-(64,93) 0.0 1.8
This doesn't seem too bad, and when I scrolled down to functions I had defined and they didn't seem to be taking much time either.
This leads me to believe it's a memory leak problem(?), so I profiled the heap:
stack exec --profile -- my-cli-exe +RTS -h
hp2ps my-cli-exe.hp
open my-cli.exe.ps
So it seems as though lots of space is being allocated on the heap, and then suddenly cleared.
The main issue is, I'm not sure where to go from here. My function is relatively small and is only getting a small JSON response of around 500 bytes. So where could the issue be coming from?
It seemed odd that the performance of a common Haskell library was so slow for me, but somehow this approach solved my concerns:
I found that the performance of my executable was faster when I used stack install to copy the binaries:
stack install
my-cli-exe
instead of using build and run.
Here are the running times again for comparison:
HS (stack install) HS (stack run) Python Curl
real: 373 ms 1017 ms 568 ms 214 ms
user: 222 ms 1062 ms 367 ms 26 ms
sys: 141 ms 210 ms 45 ms 10 ms
When running Varnish 4.0.3 and looking at varnishstat, it reports a cache-hit ratio of about 0.9 - 0.95 which I interpret as a 90-95% cache-hit ratio (incorrect?). I have a very low hit-pass count, so I cannot see that there is a lot of traffic that is not cacheable. The part that bothers me is that the backend servers are working quite a bit despite this. When I look at iftop I see that the server (which isn't running any other applications) is transmitting at about 19Mbit (includes answers to the clients and request headers to backend servers) and receiving about 14Mbit from the backend servers.
I can add that I also have a small grace period in my cache and I see that about 1-2 requests per second get counted as hits but still make a request to the backend, but that is an insignificant number of requests.
I must be missing something that should be obvious. But what?
Edit:
MAIN.uptime 70613 1.00 Child process uptime
MAIN.sess_conn 43313 0.61 Sessions accepted
MAIN.sess_drop 0 0.00 Sessions dropped
MAIN.sess_fail 0 0.00 Session accept failures
MAIN.sess_pipe_overflow 0 0.00 Session pipe overflow
MAIN.client_req_400 0 0.00 Client requests received, subject to 400 errors
MAIN.client_req_411 0 0.00 Client requests received, subject to 411 errors
MAIN.client_req_413 0 0.00 Client requests received, subject to 413 errors
MAIN.client_req_417 0 0.00 Client requests received, subject to 417 errors
MAIN.client_req 6438096 91.17 Good client requests received
MAIN.cache_hit 5231719 74.09 Cache hits
MAIN.cache_hitpass 149819 2.12 Cache hits for pass
MAIN.cache_miss 619678 8.78 Cache misses
MAIN.backend_conn 19491 0.28 Backend conn. success
MAIN.backend_unhealthy 0 0.00 Backend conn. not attempted
MAIN.backend_busy 0 0.00 Backend conn. too many
MAIN.backend_fail 0 0.00 Backend conn. failures
MAIN.backend_reuse 1279156 18.12 Backend conn. reuses
MAIN.backend_toolate 5046 0.07 Backend conn. was closed
MAIN.backend_recycle 1284221 18.19 Backend conn. recycles
MAIN.backend_retry 5 0.00 Backend conn. retry
MAIN.fetch_head 7 0.00 Fetch no body (HEAD)
MAIN.fetch_length 166348 2.36 Fetch with Length
MAIN.fetch_chunked 861097 12.19 Fetch chunked
MAIN.fetch_eof 0 0.00 Fetch EOF
MAIN.fetch_bad 0 0.00 Fetch bad T-E
MAIN.fetch_close 862 0.01 Fetch wanted close
MAIN.fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed
MAIN.fetch_zero 192834 2.73 Fetch zero len body
MAIN.fetch_1xx 0 0.00 Fetch no body (1xx)
MAIN.fetch_204 0 0.00 Fetch no body (204)
MAIN.fetch_304 77295 1.09 Fetch no body (304)
MAIN.fetch_failed 0 0.00 Fetch failed (all causes)
MAIN.fetch_no_thread 0 0.00 Fetch failed (no thread)
MAIN.pools 2 . Number of thread pools
MAIN.threads 400 . Total number of threads
MAIN.threads_limited 0 0.00 Threads hit max
MAIN.threads_created 400 0.01 Threads created
MAIN.threads_destroyed 0 0.00 Threads destroyed
MAIN.threads_failed 0 0.00 Thread creation failed
MAIN.thread_queue_len 0 . Length of session queue
MAIN.busy_sleep 22 0.00 Number of requests sent to sleep on busy objhdr
MAIN.busy_wakeup 22 0.00 Number of requests woken after sleep on busy objhdr
MAIN.sess_queued 0 0.00 Sessions queued for thread
MAIN.sess_dropped 0 0.00 Sessions dropped for thread
MAIN.n_object 61732 . object structs made
MAIN.n_vampireobject 0 . unresurrected objects
MAIN.n_objectcore 61791 . objectcore structs made
MAIN.n_objecthead 59996 . objecthead structs made
MAIN.n_waitinglist 100 . waitinglist structs made
MAIN.n_backend 8 . Number of backends
MAIN.n_expired 412530 . Number of expired objects
MAIN.n_lru_nuked 0 . Number of LRU nuked objects
MAIN.n_lru_moved 1938371 . Number of LRU moved objects
MAIN.losthdr 0 0.00 HTTP header overflows
MAIN.s_sess 43313 0.61 Total sessions seen
MAIN.s_req 6438096 91.17 Total requests seen
MAIN.s_pipe 192 0.00 Total pipe sessions seen
MAIN.s_pass 586507 8.31 Total pass-ed requests seen
MAIN.s_fetch 1206185 17.08 Total backend fetches initiated
MAIN.s_synth 0 0.00 Total synthethic responses made
MAIN.s_req_hdrbytes 4609529442 65278.77 Request header bytes
MAIN.s_req_bodybytes 615275 8.71 Request body bytes
MAIN.s_resp_hdrbytes 2888897421 40911.69 Response header bytes
MAIN.s_resp_bodybytes 95537815896 1352977.72 Response body bytes
MAIN.s_pipe_hdrbytes 57835 0.82 Pipe request header bytes
MAIN.s_pipe_in 45698 0.65 Piped bytes from client
MAIN.s_pipe_out 1305816 18.49 Piped bytes to client
MAIN.sess_closed 24747 0.35 Session Closed
MAIN.sess_pipeline 0 0.00 Session Pipeline
MAIN.sess_readahead 0 0.00 Session Read Ahead
MAIN.sess_herd 813362 11.52 Session herd
MAIN.shm_records 512140676 7252.78 SHM records
MAIN.shm_writes 18119050 256.60 SHM writes
MAIN.shm_flushes 788498 11.17 SHM flushes due to overflow
MAIN.shm_cont 89983 1.27 SHM MTX contention
MAIN.shm_cycles 277 0.00 SHM cycles through buffer
MAIN.sms_nreq 0 0.00 SMS allocator requests
MAIN.sms_nobj 0 . SMS outstanding allocations
MAIN.sms_nbytes 0 . SMS outstanding bytes
MAIN.sms_balloc 0 . SMS bytes allocated
MAIN.sms_bfree 0 . SMS bytes freed
MAIN.backend_req 1298448 18.39 Backend requests made
MAIN.n_vcl 1 0.00 Number of loaded VCLs in total
MAIN.n_vcl_avail 1 0.00 Number of VCLs available
MAIN.n_vcl_discard 0 0.00 Number of discarded VCLs
MAIN.bans 1 . Count of bans
MAIN.bans_completed 1 . Number of bans marked 'completed'
MAIN.bans_obj 0 . Number of bans using obj.*
MAIN.bans_req 0 . Number of bans using req.*
MAIN.bans_added 1 0.00 Bans added
MAIN.bans_deleted 0 0.00 Bans deleted
MAIN.bans_tested 0 0.00 Bans tested against objects (lookup)
MAIN.bans_obj_killed 0 0.00 Objects killed by bans (lookup)
MAIN.bans_lurker_tested 0 0.00 Bans tested against objects (lurker)
MAIN.bans_tests_tested 0 0.00 Ban tests tested against objects (lookup)
MAIN.bans_lurker_tests_tested 0 0.00 Ban tests tested against objects (lurker)
MAIN.bans_lurker_obj_killed 0 0.00 Objects killed by bans (lurker)
MAIN.bans_dups 0 0.00 Bans superseded by other bans
MAIN.bans_lurker_contention 0 0.00 Lurker gave way for lookup
MAIN.bans_persisted_bytes 13 . Bytes used by the persisted ban lists
MAIN.bans_persisted_fragmentation 0 . Extra bytes in persisted ban lists due to fragmentation
MAIN.n_purges 0 . Number of purge operations executed
MAIN.n_obj_purged 0 . Number of purged objects
MAIN.exp_mailed 949607 13.45 Number of objects mailed to expiry thread
MAIN.exp_received 949607 13.45 Number of objects received by expiry thread
MAIN.hcb_nolock 6001341 84.99 HCB Lookups without lock
MAIN.hcb_lock 447556 6.34 HCB Lookups with lock
MAIN.hcb_insert 447556 6.34 HCB Inserts
MAIN.esi_errors 0 0.00 ESI parse errors (unlock)
MAIN.esi_warnings 0 0.00 ESI parse warnings (unlock)
MAIN.vmods 2 . Loaded VMODs
MAIN.n_gzip 558747 7.91 Gzip operations
MAIN.n_gunzip 22866 0.32 Gunzip operations
MAIN.vsm_free 970832 . Free VSM space
MAIN.vsm_used 83963776 . Used VSM space
MAIN.vsm_cooling 0 . Cooling VSM space
MAIN.vsm_overflow 0 . Overflow VSM space
MAIN.vsm_overflowed 0 0.00 Overflowed VSM space
MGT.uptime 70613 1.00 Management process uptime
MGT.child_start 1 0.00 Child process started
MGT.child_exit 0 0.00 Child process normal exit
MGT.child_stop 0 0.00 Child process unexpected exit
MGT.child_died 0 0.00 Child process died (signal)
MGT.child_dump 0 0.00 Child process core dumped
MGT.child_panic 0 0.00 Child process panic
MEMPOOL.vbc.live 24 . In use
MEMPOOL.vbc.pool 10 . In Pool
MEMPOOL.vbc.sz_wanted 88 . Size requested
MEMPOOL.vbc.sz_needed 120 . Size allocated
MEMPOOL.vbc.allocs 19491 0.28 Allocations
MEMPOOL.vbc.frees 19467 0.28 Frees
MEMPOOL.vbc.recycle 19436 0.28 Recycled from pool
MEMPOOL.vbc.timeout 11998 0.17 Timed out from pool
MEMPOOL.vbc.toosmall 0 0.00 Too small to recycle
MEMPOOL.vbc.surplus 0 0.00 Too many for pool
MEMPOOL.vbc.randry 55 0.00 Pool ran dry
MEMPOOL.busyobj.live 6 . In use
MEMPOOL.busyobj.pool 9 . In Pool
MEMPOOL.busyobj.sz_wanted 65536 . Size requested
MEMPOOL.busyobj.sz_needed 65568 . Size allocated
MEMPOOL.busyobj.allocs 1298643 18.39 Allocations
MEMPOOL.busyobj.frees 1298637 18.39 Frees
MEMPOOL.busyobj.recycle 1298436 18.39 Recycled from pool
MEMPOOL.busyobj.timeout 41750 0.59 Timed out from pool
MEMPOOL.busyobj.toosmall 0 0.00 Too small to recycle
MEMPOOL.busyobj.surplus 0 0.00 Too many for pool
MEMPOOL.busyobj.randry 207 0.00 Pool ran dry
MEMPOOL.req0.live 4 . In use
MEMPOOL.req0.pool 9 . In Pool
MEMPOOL.req0.sz_wanted 65536 . Size requested
MEMPOOL.req0.sz_needed 65568 . Size allocated
MEMPOOL.req0.allocs 423162 5.99 Allocations
MEMPOOL.req0.frees 423158 5.99 Frees
MEMPOOL.req0.recycle 423143 5.99 Recycled from pool
MEMPOOL.req0.timeout 36313 0.51 Timed out from pool
MEMPOOL.req0.toosmall 0 0.00 Too small to recycle
MEMPOOL.req0.surplus 0 0.00 Too many for pool
MEMPOOL.req0.randry 19 0.00 Pool ran dry
MEMPOOL.sess0.live 4 . In use
MEMPOOL.sess0.pool 9 . In Pool
MEMPOOL.sess0.sz_wanted 384 . Size requested
MEMPOOL.sess0.sz_needed 416 . Size allocated
MEMPOOL.sess0.allocs 21655 0.31 Allocations
MEMPOOL.sess0.frees 21651 0.31 Frees
MEMPOOL.sess0.recycle 21642 0.31 Recycled from pool
MEMPOOL.sess0.timeout 10076 0.14 Timed out from pool
MEMPOOL.sess0.toosmall 0 0.00 Too small to recycle
MEMPOOL.sess0.surplus 0 0.00 Too many for pool
MEMPOOL.sess0.randry 13 0.00 Pool ran dry
MEMPOOL.req1.live 6 . In use
MEMPOOL.req1.pool 9 . In Pool
MEMPOOL.req1.sz_wanted 65536 . Size requested
MEMPOOL.req1.sz_needed 65568 . Size allocated
MEMPOOL.req1.allocs 418423 5.93 Allocations
MEMPOOL.req1.frees 418417 5.93 Frees
MEMPOOL.req1.recycle 418406 5.93 Recycled from pool
MEMPOOL.req1.timeout 35924 0.51 Timed out from pool
MEMPOOL.req1.toosmall 0 0.00 Too small to recycle
MEMPOOL.req1.surplus 0 0.00 Too many for pool
MEMPOOL.req1.randry 17 0.00 Pool ran dry
MEMPOOL.sess1.live 6 . In use
MEMPOOL.sess1.pool 10 . In Pool
MEMPOOL.sess1.sz_wanted 384 . Size requested
MEMPOOL.sess1.sz_needed 416 . Size allocated
MEMPOOL.sess1.allocs 21659 0.31 Allocations
MEMPOOL.sess1.frees 21653 0.31 Frees
MEMPOOL.sess1.recycle 21645 0.31 Recycled from pool
MEMPOOL.sess1.timeout 10040 0.14 Timed out from pool
MEMPOOL.sess1.toosmall 0 0.00 Too small to recycle
MEMPOOL.sess1.surplus 0 0.00 Too many for pool
MEMPOOL.sess1.randry 14 0.00 Pool ran dry
SMA.s0.c_req 1349644 19.11 Allocator requests
SMA.s0.c_fail 0 0.00 Allocator failures
SMA.s0.c_bytes 22210720975 314541.53 Bytes allocated
SMA.s0.c_freed 19148751042 271178.83 Bytes freed
SMA.s0.g_alloc 105992 . Allocations outstanding
SMA.s0.g_bytes 3061969933 . Bytes outstanding
SMA.s0.g_space 2306739187 . Bytes available
SMA.Transient.c_req 1343403 19.02 Allocator requests
SMA.Transient.c_fail 0 0.00 Allocator failures
SMA.Transient.c_bytes 24207322266 342816.79 Bytes allocated
SMA.Transient.c_freed 24201619786 342736.04 Bytes freed
SMA.Transient.g_alloc 7499 . Allocations outstanding
SMA.Transient.g_bytes 5702480 . Bytes outstanding
SMA.Transient.g_space 0 . Bytes available
VBE.backend1([IP]).vcls 1 . VCL references
VBE.backend1([IP]).happy 18446744073709551615 . Happy health probes
VBE.backend1([IP]).bereq_hdrbytes 173320400 2454.51 Request header bytes
VBE.backend1([IP]).bereq_bodybytes 238971 3.38 Request body bytes
VBE.backend1([IP]).beresp_hdrbytes 124704577 1766.03 Response header bytes
VBE.backend1([IP]).beresp_bodybytes 19209840560 272043.97 Response body bytes
VBE.backend1([IP]).pipe_hdrbytes 14288 0.20 Pipe request header bytes
VBE.backend1([IP]).pipe_out 8069 0.11 Piped bytes to backend
VBE.backend1([IP]).pipe_in 335893 4.76 Piped bytes from backend
VBE.backend2([IP]).vcls 1 . VCL references
VBE.backend2([IP]).happy 18446744073709551615 . Happy health probes
VBE.backend2([IP]).bereq_hdrbytes 173379423 2455.35 Request header bytes
VBE.backend2([IP]).bereq_bodybytes 76962 1.09 Request body bytes
VBE.backend2([IP]).beresp_hdrbytes 124652416 1765.29 Response header bytes
VBE.backend2([IP]).beresp_bodybytes 18690565655 264690.15 Response body bytes
VBE.backend2([IP]).pipe_hdrbytes 14629 0.21 Pipe request header bytes
VBE.backend2([IP]).pipe_out 8558 0.12 Piped bytes to backend
VBE.backend2([IP]).pipe_in 284168 4.02 Piped bytes from backend
VBE.backend3([IP]).vcls 1 . VCL references
VBE.backend3([IP]).happy 18446744073709551615 . Happy health probes
VBE.backend3([IP]).bereq_hdrbytes 173448768 2456.33 Request header bytes
VBE.backend3([IP]).bereq_bodybytes 75294 1.07 Request body bytes
VBE.backend3([IP]).beresp_hdrbytes 124641060 1765.13 Response header bytes
VBE.backend3([IP]).beresp_bodybytes 18738758550 265372.64 Response body bytes
VBE.backend3([IP]).pipe_hdrbytes 17940 0.25 Pipe request header bytes
VBE.backend3([IP]).pipe_out 16704 0.24 Piped bytes to backend
VBE.backend3([IP]).pipe_in 301836 4.27 Piped bytes from backend
VBE.backend4([IP]).vcls 1 . VCL references
VBE.backend4([IP]).happy 18446744073709551615 . Happy health probes
VBE.backend4([IP]).bereq_hdrbytes 173265357 2453.73 Request header bytes
VBE.backend4([IP]).bereq_bodybytes 227023 3.22 Request body bytes
VBE.backend4([IP]).beresp_hdrbytes 124724341 1766.31 Response header bytes
VBE.backend4([IP]).beresp_bodybytes 18642562615 264010.35 Response body bytes
VBE.backend4([IP]).pipe_hdrbytes 22537 0.32 Pipe request header bytes
VBE.backend4([IP]).pipe_out 12367 0.18 Piped bytes to backend
VBE.backend4([IP]).pipe_in 383919 5.44 Piped bytes from backend
LCK.sma.creat 2 0.00 Created locks
LCK.sma.destroy 0 0.00 Destroyed locks
LCK.sma.locks 5272603 74.67 Lock Operations
LCK.hcb.locks 837080 11.85 Lock Operations
LCK.hcl.creat 0 0.00 Created locks
LCK.hcl.destroy 0 0.00 Destroyed locks
LCK.hcl.locks 0 0.00 Lock Operations
LCK.vcl.creat 1 0.00 Created locks
LCK.vcl.destroy 0 0.00 Destroyed locks
LCK.vcl.locks 2604799 36.89 Lock Operations
LCK.sess.creat 43306 0.61 Created locks
LCK.sess.destroy 43304 0.61 Destroyed locks
LCK.sess.locks 4765 0.07 Lock Operations
LCK.wstat.creat 1 0.00 Created locks
LCK.wstat.destroy 0 0.00 Destroyed locks
LCK.wstat.locks 3421304 48.45 Lock Operations
LCK.wq.locks 6449131 91.33 Lock Operations
LCK.objhdr.creat 447638 6.34 Created locks
LCK.objhdr.destroy 387644 5.49 Destroyed locks
LCK.objhdr.locks 42615452 603.51 Lock Operations
LCK.exp.creat 1 0.00 Created locks
LCK.exp.destroy 0 0.00 Destroyed locks
LCK.exp.locks 4462106 63.19 Lock Operations
LCK.lru.creat 2 0.00 Created locks
LCK.lru.destroy 0 0.00 Destroyed locks
LCK.lru.locks 4259219 60.32 Lock Operations
LCK.cli.creat 1 0.00 Created locks
LCK.cli.destroy 0 0.00 Destroyed locks
LCK.cli.locks 23549 0.33 Lock Operations
LCK.ban.creat 1 0.00 Created locks
LCK.ban.destroy 0 0.00 Destroyed locks
LCK.ban.locks 1594008 22.57 Lock Operations
LCK.vbp.creat 1 0.00 Created locks
LCK.vbp.destroy 0 0.00 Destroyed locks
LCK.vbp.locks 112922 1.60 Lock Operations
LCK.backend.locks 2626872 37.20 Lock Operations
LCK.vcapace.creat 1 0.00 Created locks
*Had to remove some (uneventful parts) from the log due to message limit.
In this I calculated "other non-cacheable requests" by client_req - (cache_hit + cache_hitpass + cache_miss) = 436880. That along with cache_hitpass and cache misses (or just client_req-cache-hit) which are not cached brings the total to 586699, or about 18.7%. If they are heavy requests I suppose that could account for the traffic. But I'm a bit unconvinced.
Have you tried to look at the access log of one of the servers? Maybe you have requests coming in not through Varnish?
Also, I know that at least in Varnish 3, the cache hit rate is calculated from hits that can come from cache to begin with. For example, with a standard configuration, POST requests are not counted there.
The access log may give you some insight...
I am using CPLEX in Cpp.
After googling I found out what parameters need to be set to avoid cplex from printing to terminal and I use them like this:
IloCplex cplex(model);
std::ofstream logfile("cplex.log");
cplex.setOut(logfile);
cplex.setWarning(logfile);
cplex.setError(logfile);
cplex.setParam(IloCplex::MIPInterval, 1000);//Controls the frequency of node logging when MIPDISPLAY is set higher than 1.
cplex.setParam(IloCplex::MIPDisplay, 0);//MIP node log display information-No display until optimal solution has been found
cplex.setParam(IloCplex::SimDisplay, 0);//No iteration messages until solution
cplex.setParam(IloCplex::BarDisplay, 0);//No progress information
cplex.setParam(IloCplex::NetDisplay, 0);//Network logging display indicator
if ( !cplex.solve() ) {
....
}
but yet cplex prints such things:
Warning: Bound infeasibility column 'x11'.
Presolve time = 0.00 sec. (0.00 ticks)
Root node processing (before b&c):
Real time = 0.00 sec. (0.01 ticks)
Parallel b&c, 4 threads:
Real time = 0.00 sec. (0.00 ticks)
Sync time (average) = 0.00 sec.
Wait time (average) = 0.00 sec.
------------
Total (root+branch&cut) = 0.00 sec. (0.01 ticks)
Is there any way to avoid printing them?
Use setOut method from IloAlgorithm class (IloCplex inherits from IloAlgorithm). You can set a null output stream as a parameter and prevent logging the message on the screen.
This is what works in C++ according to cplex parameters doc:
cplex.setOut(env.getNullStream());
cplex.setWarning(env.getNullStream());
cplex.setError(env.getNullStream());
I'm working on collecting test measurement data from product in manufacturing environment.
The test measurement result of units under test are generated by the test system. It is in an 2Mb txt file and was keep in share folders separated by products.
The folder structure looks like...
LOGS
|-Product1
| |-log_p1_1.txt
| |-log_p1_2.txt
| |..
|-Product2
| |-log_p2_1.txt
| |-log_p2_2.txt
| |..
|-...
My ruby script can iterate through each Product directory under LOGS and then read each log_px_n.txt file, parse data I need in the file and update it into database.
The thing is that all log_px_n.txt files of must be keep in its current directory, both old file and new files, while I need to keep my database update as soon as the new log_px_n.tx file was generated.
what I did today is to try iterate through each Product directories then read each individual .txt file and after that update file into database if it was not exist.
My script looks like..
Dir['*'].each do |product|
product_dir = File.join(BASE_DIR, product)
Dir.chdir(product_dir)
Dir['*.txt'].each do |log|
if (Time.now - File.mtime(log) < SIX_HOURS_AGO) # take only new files in last six hours
# Here we do..
# - read each 2Mb .txt file
# - extract infomation from txt file
# - update into database
end
end
end
There are upto 30 differents product directories and each product contain around 1000 .txt file (2Mb each), and they are growing !
I don't have issue about disk space to store such .txt file but the time it take to complete this operation.
It takes >45min to complete task each time when run above code block.
Is there any better way to deal with this situation ?
Update:
I tried as Iced's suggest to use profiler, so I run below code and got following result...
require 'profiler'
class MyCollector
def initialize(dir, period, *filetypes)
#dir = dir
#filetypes = filetypes.join(',')
#period = period
end
def collect
Dir.chdir(#dir)
Dir.glob('*').each do |product|
products_dir = File.join(#dir, product)
Dir.chdir(products_dir)
puts "at product #{product}"
Dir.glob("**/*.{#{#filetypes}}").each do |log|
if Time.now - File.mtime(log) < #period
puts Time.new
end
end
end
end
path = '//10.1.2.54/Shares/Talend/PRODFILES/LOGS'
SIX_HOURS_AGO = 21600
Profiler__::start_profile
collector = MyCollector.new(path, SIX_HOURS_AGO, "LOG")
collector.collect
Profiler__::stop_profile
Profiler__::print_profile(STDOUT)
The result shows...
at product ABU43E
..
..
..
at product AXF40J
at product ACZ16C
2014-04-21 17:32:07 +0700
at product ABZ14C
at product AXF90E
at product ABZ14B
at product ABK43E
at product ABK01A
2014-04-21 17:32:24 +0700
2014-04-21 17:32:24 +0700
at product ABU05G
at product ABZABF
2014-04-21 17:32:28 +0700
2014-04-21 17:32:28 +0700
2014-04-21 17:32:28 +0700
2014-04-21 17:32:28 +0700
2014-04-21 17:32:28 +0700
2014-04-21 17:32:28 +0700
% cumulative self self total
time seconds seconds calls ms/call ms/call name
32.54 1.99 1.99 43 46.40 265.60 Array#each
24.17 3.48 1.48 41075 0.04 0.04 File#mtime
13.72 4.32 0.84 43 19.AX 19.AX Dir#glob
9.13 4.88 0.AX 41075 0.01 0.03 Time#-
8.14 5.38 0.50 41075 0.01 0.01 Float#quo
6.65 5.79 0.41 41075 0.01 0.01 Time#now
2.06 5.91 0.13 41084 0.00 0.00 Time#initialize
1.79 6.02 0.11 41075 0.00 0.00 Float#<
1.79 6.13 0.11 41075 0.00 0.00 Float#/
0.00 6.13 0.00 1 0.00 0.00 Array#join
0.00 6.13 0.00 51 0.00 0.00 Kernel.puts
0.00 6.13 0.00 51 0.00 0.00 IO#puts
0.00 6.13 0.00 102 0.00 0.00 IO#write
0.00 6.13 0.00 42 0.00 0.00 File#join
0.00 6.13 0.00 43 0.00 0.00 Dir#chdir
0.00 6.13 0.00 10 0.00 0.00 Class#new
0.00 6.13 0.00 1 0.00 0.00 MyCollector#initialize
0.00 6.13 0.00 9 0.00 0.00 Integer#round
0.00 6.13 0.00 9 0.00 0.00 Time#to_s
0.00 6.13 0.00 1 0.00 6131.00 MyCollector#collect
0.00 6.13 0.00 1 0.00 6131.00 #toplevel
[Finished in 477.5s]
It turn out that it take up to 7 mins to walk over each files in each directories. then call mtime.
Although my .txt file is 2Mb, it should not suppose to take time that long, no ?
Any suggestion, pls ?
Relying on mtime is not robust. In fact, Rails switched from using mtime to hash in naming the versions of asset files.
You should keep a list of file-hash pair. That can be obtained like this:
require "digest"
file_hash_pair =
Dir.glob("LOGS/**/*")
.select{|f| File.file?(f)}
.map{|f| [f, Digest::SHA1.hexdigest(File.read(f))]}
and perhaps you can keep the content of this in a file as YAML. You can run the code above each time, and whenever file_hash_pair is different from the previous value, you can tell that there was a change. If file_hash_pair.transpose[0] changed, then you can tell there was a file manipulation. If for a particular [file, hash] pair, the hash changed, then you can tell that the file file changed.
I am using Rails 4 and postgresql database and I have a question about entering in a CSV dataset into the database.
Date Advertiser Name Impressions Clicks CPM CPA CPC CTR
10/21/13 Advertiser 1 77 0 4.05 0.00 0.00 0.00
10/21/13 Advertiser 2 10732 23 5.18 0.00 2.42 0.21
10/21/13 Advertiser 3 16941 14 4.64 11.23 5.62 0.08
10/22/13 Advertiser 1 59 0 3.67 0.00 0.00 0.00
10/22/13 Advertiser 2 10130 15 5.24 53.05 3.54 0.15
10/22/13 Advertiser 3 18400 22 4.59 10.55 3.84 0.12
10/23/13 Advertiser 1 77 0 4.06 0.00 0.00 0.00
10/23/13 Advertiser 2 9520 22 5.58 26.58 2.42 0.23
Using the data above I need to create a show page for each Advertiser.
Ultimately I need to have a list of Advertiser's that I can click on any one of them and go to their show page and display the informations relevant to each advertiser (impressions, clicks, cpm, etc)
Where I am confused is how to import the CSV data when there are rows with duplicate Advertiser's, but the other columns contain relevant and non duplicate information. How can I set up my database tables so that I will not have duplicate Advertiser's and still import and then display the correct information?
You will want to create two models: Advertiser and Site. (or maybe date).
Advertiser "has many" Sites, and Site "has one" advertiser. This association will allow you to import your data correctly.
See: http://api.rubyonrails.org/classes/ActiveRecord/Associations/ClassMethods.html
Instead of creating two different models I just created 1 advertiser model and inputted the complete dataset into that model.
require 'csv'
desc "Import advertisers from csv file"
task :import => [:environment] do
CSV.foreach('db/MediaMathPerformanceReport.csv', :headers => true) do |row|
Advertiser.create!(row.to_hash)
end
end
After the data was imported by the above rake task, I simply set up the show route as follows:
def show
#advertiser = Advertiser.where(advertiser_name: advertiser_name)
end