I need a stratified random sample that consists in a determined number of addresses. My stratification variable is the zip code.
The sample size is already pre-determined and has different number of observations per zip code. How can I incorporate the desired number ob observations per zip code? Is it possible to do this with PROC SURVEYSELECT? I have the sample size in an external file.
Example:
ZIP CODE NUMBER_OBSERVATIONS
9999 10
8888 15
7777 10
6666 5
Is it possible? How can I do this?
Thanks
Use the sampsize option and specify your dataset. It will limit all of the samples to each size of the strata.
proc surveyselect data = have
out = want
sampsize=external
;
strata zip;
size number_observations;
run;
I have the following script, which works good, locally (Windows 10 IIS, windows 2003 Server), but not on our hosting server (Windows 2003 Server). Anything over 4mb will download really slow and then timeout before it gets to the end of the file. However, locally, it downloads fast and full.
Doing a Direct Download (link to the file itself) downloads a 26.5mb file in 5 seconds from our hosting provider server. So, there is not an issue with a download limit. There is an issue it seems, with the hosting server and this script. Any ideas?
Response.AddHeader "content-disposition","filename=" & strfileName
Response.ContentType = "application/x-zip-compressed" 'here your content -type
Dim strFilePath, lSize, lBlocks
Const CHUNK = 2048
set objStream = CreateObject("ADODB.Stream")
objStream.Open
objStream.Type = 1
objStream.LoadFromfile Server.MapPath("up/"&strfileName&"")
lSize = objStream.Size
Response.AddHeader "Content-Size", lSize
lBlocks = 1
Response.Buffer = False
Do Until objStream.EOS Or Not Response.IsClientConnected
Response.BinaryWrite(objStream.Read(CHUNK))
Loop
objStream.Close
Just looking at the code snippet it appear to be fine and is the very approach I would use for downloading large files (especially like the use of Response.IsClientConnected).
However having said that, it's likely the size of the chunks being read in relation to the size of the file.
Very roughly the formula is something like this...
time to read = ((file size / chunk size) * read time)
So if we use your example of a 4 MB file (4194304 bytes) and say it takes 100 milliseconds to read each chunk then the following applies;
Chunk Size of 2048 bytes (2 KB) will take approx. 3 minutes to read.
Chunk Size of 20480 bytes (20 KB) will take approx. 20 seconds to read.
Classic ASP pages on IIS 7 and above have a default scriptTimeout of 00:01:30 so in the example above a 4 MB file constantly read at 100 milliseconds in 2 KB chunks would timeout before the script could finish.
Now these are just rough statistics your read time won't constantly stay the same and it's likely faster then 100 milliseconds (depending on disk read speeds) but I think you get the point.
So just try increasing the CHUNK.
Const CHUNK = 20480 'Read in chunks of 20 KB
The code I have is bit different, using a For..Next loop instead of Do..Until loop. Not 100% sure this will really work in your case, but worth a try. Here is my version of the code:
For i = 1 To iSz / chunkSize
If Not Response.IsClientConnected Then Exit For
Response.BinaryWrite objStream.Read(chunkSize)
Next
If iSz Mod chunkSize > 0 Then
If Response.IsClientConnected Then
Response.BinaryWrite objStream.Read(iSz Mod chunkSize)
End If
End If
Basically is due the script timeout. I had the same problem with 1GB files in IIS 10 after upgraded to Win 2016 with IIS 10 (default timeout is shorter by default).
I use chunks of 256000 and Server.ScriptTimeout = 600 '10 minutes
So I have a h2 database event table, that I am monitoring for events. There is a thread that fires every 2 secs and checks with select * from eventTable limit 10 offset 0.
I was wondering what is the performance impact of this hammering in an h2 database table. It is B-tree based but the db itself is a file. Does the h2db go to the file and has to read blocks and so to determine if table is empty. Think Oracle Db and High Water Mark problem for querying tables with large rows that get deleted later on without truncate and this causes unnecessary read of blocks to get select * done and is bad for performance.
If at all this is bad, would swapping out the thread part be recommended with the Trigger approach for Insert operations described in this qt here.
Regards
Here are some numbers from io monitoring:
I started my application that has the monitoring thread with the select statement. Then I started "sudo fsusage MYPID". The pattern with 4 reads and 2 writes repeats itself:
16:18:10.809774 pread F=44 B=0x400 O=0x00037000 0.000020 java.11010
16:18:10.809809 pread F=44 B=0x400 O=0x00037000 0.000005 java.11010
16:18:10.809825 pread F=44 B=0x400 O=0x00037000 0.000003 java.11010
16:18:10.809839 pread F=44 B=0x400 O=0x00037000 0.000004 java.11010
16:18:10.810044 pwrite F=44 B=0x1000 O=0x00031000 0.000034 java.11010
16:18:10.810087 pwrite F=44 B=0x2000 O=0x00000000 0.000010 java.11010
FD is file desciptor and is through lsof -p PID confirmed as the database file. B= no of bytes count read or written. O is offset in file.
I see the above pattern of read and writes constantly. The reads are than most likely the selects. There is no other activity on DB. So even for empty tables I am reading something like 1600 bytes consistently and making 2 writes in the range of 3000 to 4000 bytes.
But than I went into more detail, since I am on macosx, strace is not an option but Dtruss works nicely. So just gave dtruss -a -p PID and following is the relevant output for the reads and writes:
632/0x653a: 2229289 37 24 pread(0x2C, "chunk:3157,block:31,len:2,map:9,max:1540,next:4d,pages:6,root:c55c0000027cf,time:18ffdc4,version:3157 \n\0", 0x400, 0x31000) = 1024 0
632/0x652c: 773689 86 2 gettimeofday(0x70000B107C68, 0x0, 0x0) = 0 0
632/0x653a: 2229327 13 5 pread(0x2C, "chunk:3157,block:31,len:2,map:9,max:1540,next:4d,pages:6,root:c55c0000027cf,time:18ffdc4,version:3157 \n\0", 0x400, 0x31000) = 1024 0
632/0x653a: 2229347 10 4 pread(0x2C, "chunk:3157,block:31,len:2,map:9,max:1540,next:4d,pages:6,root:c55c0000027cf,time:18ffdc4,version:3157 \n\0", 0x400, 0x31000) = 1024 0
632/0x653a: 2229373 11 4 pread(0x2C, "chunk:3157,block:31,len:2,map:9,max:1540,next:4d,pages:6,root:c55c0000027cf,time:18ffdc4,version:3157 \n\0", 0x400, 0x31000) = 1024 0
632/0x653a: 2229621 45 34 pwrite(0x2C, "chunk:3159,block:24,len:1,map:9,max:b80,next:35,pages:4,root:c5640000027cf,time:19001ef,version:3159 \n\0", 0x1000, 0x24000) = 4096 0
632/0x653a: 2229686 32 24 pwrite(0x2C, "H:2,block:24,blockSize:1000,chunk:3159,created:1610362b746,format:1,version:3159,fletcher:1d05a51e\n\0", 0x2000, 0x0) = 8192 0
So adding above the return values of pread and pwrite I can see actual read is 1024 x 4 Bytes and writes is 4096 + 8192 bytes. And also one can see what is read and written. The last write sometime appears and sometimes doesn't. The 1st param to fread and fwrite is the file descriptor 0x2c which matches that of the database file. And the 2nd param is the buffer being written. I wonder why we need to write here anything though. But that got explained when I read the following architecture explanation in h2 project page:
The above writes and reads can be explained by h2database.com/html/mvstore.html#fileFormat
Browsing the source code I find that the BackgroundWriterThread class, which i noticed in the profiler as well churning bytes up as time goes by ( but no memory leaks, it cleans up properly), is waking up every second and just blindly commiting the store. That gives the location of the writes and reads above in code.
More googling revelead the problem was discussed here in the google group, though no resolution occured except someone later on posting that the parameter WRITE_DELAY does the trick for him. groups.google.com/forum/#!searchin/h2-database/… Then I wondered if he did not try setting autoCommit on connection to false. I tried and the above pattern of read and writes stopped for me.
So adding ;AUTOCOMMIT=OFF to the connection parameter does the trick and the query is in memory, so overhead of select * from an empty table is quite minimum. This ends this investigation for me. The data is in memory as I am using the Version 1.4.195 and that has the MVStore database as default. So querying an empty table in memory should be a relatively inexpensive operation.
Regards
I have an application that reads from serial port from PC. When i read using my standalone application, all the expected read bytes are received. But when i incorporate the application into HWUT ( Hello World Unit Testing), the .exe output generated in OUT folder contains a portion of the received data and fills the rest will NULL. I use the same receive buffer size for both cases. What could be the reason?
When you run the application on the command line, is the output correct?
Does 'fflush(stdout)' help?
How large is th output? Note, that HWUT has an in-buildt oversize detection. If you need larger output respond to "--hwut-info" with
... printf("SIZE-LIMIT: 4711MB;\n"); ...
Change MB to KB for kilo byte or GB for giga byte. 4711 is your size limit.
I'm experimenting with 2 Gluster 3.7 servers in 1x2 configuration. Servers are connected over 1 Gbit network. I'm using Debian Jessie.
My use case is as follows: open file -> append 64 bytes -> close file and do this in a loop for about 5000 different files. Execution time for such loop is roughly 10 seconds if I access files through mounted glusterfs drive. If I use libgfsapi directly, execution time is about 5 seconds (2 times faster).
However, the same loop executes in 50ms on plain ext4 disk.
There is huge performance difference between Gluster 3.7 end earlier versions which is, I believe, due to the cluster.eager-lock setting.
My target is to execute the loop in less than 1 second.
I've tried to experiment with lots of Gluster settings but without success. dd tests with various bsize values behave like that TCP no-delay option is not set, although from Gluster source code it seems that no-delay is default.
Any idea how to improve the performance?
Edit:
I've found a solution that works in my case so I'd like to share it in case anyone else faces the same issue.
The root cause of the problem is the number of roundtrips between client and Gluster server during execution of open/write/close sequence. I don't know exactly what is happening behind but timing measurements shows exactly that pattern. Now, the obvious idea would be to "pack" open/write/close sequence into a single write function. Roughly, the C prototype of such function would be:
int write(const char* fname, const void *buf, size_t nbyte, off_t offset)
But, there is already such API function glfs_h_anonymous_write in libgfapi (thanks goes to Suomya from Gluster mailing group). Kind of hidden thing there is the file identifier which is not plain file name, but something of type struct glfs_object. Clients obtain an instance of such object through API calls glfs_h_lookupat/glfs_h_creat. The point here is that glfs_object representing filename is "stateless" in a sense that corresponding inode is left intact (not ref counted). One should think of glfs_object as plain filename identifier and use it as you would use filename (actually, glfs_object stores plain pointer to corresponding inode without ref counting it).
Finally, we should use glfs_h_lookupat/glfs_h_creat once and write many times to the file using glfs_h_anonymous_write.
That way I was able to append 64 bytes to 5000 files in 0.5 seconds, which is 20 times faster than using mounted volume and open//write/close sequence.