In system-verilog i have a random number generator like this:
task set_rand_value(output bit [7:0] rand_frms, output bit [13:0] rand_bcnt);
begin
bit [7:0] rand_num;
// Set the random value between 2 and 254
rand_num=$urandom_range(254,2);
rand_bcnt=$urandom_range(2047,1);
// Check to ensure number is even
if ((rand_num/2)*2 != rand_num) begin
rand_num=rand_num+1;
$display("Generated %d frames. 8'h%h", rand_num, rand_num);
$display("Packets generated with 0x%4h bytes.", rand_bcnt);
end
// Set the random frame number
else begin
rand_frms = rand_num;
$display("Generated %d frames. 8'h%h", rand_num, rand_num);
$display("Packets generated with 0x%4h bytes.", rand_bcnt);
end
end
endtask // set_rand_value
For some reason the first time I run this (and the third and fifth... etc) it returns a value of zero even though a number is properly generated.
I am calling it like this:
bit [ 7:0] num_frms;
bit [13:0] size_val;
// Generate random values
set_rand_value(num_frms, size_val); // Run twice to correct initial write error
$display("Error correction for packets sent: 0x%h", num_frms);
set_rand_value(num_frms, size_val);
$display("The number of packets being sent: 0x%h", num_frms);
Which gives me this output:
Generated 214 frames. 8'hd6
Packets generated with 0x05b2 bytes.
Error correction for packets sent: 0x00
Generated 252 frames. 8'hfc
Packets generated with 0x011f bytes.
The number of packets being sent: 0xfc
0x80fc011f // Expected number
I have tried quite a few things to correct it but for some reason the only way i can get it to behave the way i expect it to is to just run it twice every time i need to use it.
You only set num_frms if the random number is even. Why not just do
function void set_rand_value(output bit [7:0] rand_frms, output bit [13:0] rand_bcnt);
begin
// Set the random value between 2 and 254
rand_frms=$urandom*2;
rand_bcnt=$urandom_range(2047,1);
endfunction
P.S Only use tasks for routines that may block.
The reason why the code above did not work was you were not assigning rand_frms = rand_num; in the first half of the if condition . So only for 50% of the times read_frm would have a value in it. [ 50 % just been approximate ] .
Related
The code added fails with a ValueError and i have no idea whats going wrong. Here is what i want to do:
In my fpga i receive data via spi.
Data is a bipolar signal (in the sense of a measurement signal) arriving in a 16 bit register rxdata.
the task is to interpret this signal as being signed and only the upper 12 bits (including sign) are needed. The variable bipolar is therefore 12bit wide and signed i.e. declared like in the code. Then i assign:
bipolar.next=rxdata[:4].signed()
my problem is, that the assignment of the 12bit slice fails as soon as data becomes negative (i.e most significant bit becomes 1).
With data 0x8fff for instance at runtime i get:
'ValueError: intbv value 2303 >= maximum 2048'
I don't expect this, as both sides are declared signed and data fits into the variable bipolar.
Is there another way to do this?
(by the way: bipolar[:].next=rxdata[:4].signed() i get 0 as a result which i would not expect either)
#testcase.py sk 09.12.2020
#assign a slice to a signed signal (bipolar) fails at runtime with negative numbers
from myhdl import *
nspi=16
n=12
tend=1e-6
#block
def testcase():
CLK = Signal(bool(0))
RESET = ResetSignal(1,active = 0, isasync=True)
bipolar=Signal(intbv(0,min=-2**(n-1),max=2**(n-1)))
rxdata = Signal(intbv(0)[nspi:0]) #received data is bipolar, transferred via spi into rxdata
''' Clock driver 16MHz'''
#always(delay(31))
def driver():
CLK.next = not CLK
#instance
def init():
rxdata.next=0x8fff #0x7fff i.e. positive passes, 0x8fff i.e negative fails runtime check
yield delay(100)
#always_seq(CLK.negedge,reset=RESET)
def assign():
#bipolar[:].next=rxdata[:(nspi-n)].signed() #this passes - but result is 0! (unexpected for me)
bipolar.next=rxdata[:(nspi-n)].signed() #this fails with negative numbers (unexpected for me)
print(bipolar, 'bipolar=', int(str(bipolar),16))
return instances()
tc = testcase()
tc.run_sim(tend*1e9)
print('Simulated to tend='+str(tend))
Just found the way to do it: use shadow signals! i.e use round brackets () instead of square ..[]
#testcase.py sk 12.12.2020
#assign a slice to a signed signal (bipolar) fails at runtime with negative numbers
#-> use shadow signals instead! #bipolar[:].next=rxdata[:(nspi-n)].signed() #this passes - but result is 0! (unexpected for me)
from myhdl import *
nspi=16
n=12
tend=1e-6
#block
def testcase():
CLK = Signal(bool(0))
RESET = ResetSignal(1,active = 0, isasync=True)
bipolar=Signal(intbv(0,min=-2**(n-1),max=2**(n-1)))
#rxdata = Signal(intbv(0)[nspi:0]) #received data is bipolar, transferred via spi into rxdata
rxdata = Signal(intbv(0,min=0,max=2**nspi))
''' Clock driver 16MHz'''
#always(delay(31))
def driver():
CLK.next = not CLK
#instance
def init():
rxdata.next=0xffff #0x7fff i.e. positive passes, 0x8fff i.e negative fails runtime check when not using shadow
yield delay(100)
#always_seq(CLK.negedge,reset=RESET)
def assign():
#bipolar.next=rxdata[:(nspi-n)].signed() #this fails in runtime check -> number too big
#bipolar[:].next=rxdata[:(nspi-n)].signed() #this passes - but result is 0! (unexpected for me)
bipolar.next=rxdata(nspi,(nspi-n)).signed() #this is the way: use shadow signal !
print(bipolar, 'bipolar=', int(str(bipolar),16))
return instances()
tc = testcase()
tc.run_sim(tend*1e9)
print('Simulated to tend='+str(tend))
I am new to python so please forgive me if I'm asking a dumb question. In my function I generate a random byte array for a given number of bytes called "input_data", then I add bytewise some bit errors and store the result in another byte array called "output_data". The print function shows that it works exactly as expected, there are different bytes. But if I compare the byte arrays afterwards they seem to be identical!
def simulate_ber(packet_length, ber, verbose=False):
# generate input data
input_data = bytearray(random.getrandbits(8) for _ in xrange(packet_length))
if(verbose):
print(binascii.hexlify(input_data)+" <-- simulated input vector")
output_data = input_data
#add bit errors
num_errors = 0
for byte in range(len(input_data)):
error_mask = 0
for bit in range(0,7,1):
if(random.uniform(0, 1)*100 < ber):
error_mask |= 1 << bit
num_errors += 1
output_data[byte] = input_data[byte] ^ error_mask
if(verbose):
print(binascii.hexlify(output_data)+" <-- output vector")
print("number of simulated bit errors: " + str(num_errors))
if(input_data == output_data):
print ("data identical")
number of packets: 1
bytes per packet: 16
simulated bit error rate: 5
start simulation...
0d3e896d61d50645e4e3fa648346091a <-- simulated input vector
0d3e896f61d51647e4e3fe648346001a <-- output vector
number of simulated bit errors: 6
data identical
Where is the bug? I am sure the problem is somewhere between my ears...
Thank you in advance for your help!
output_data = input_data
Python is a referential language. When you do the above, both variables now refer to the same object in memory. e.g:
>>> y=['Hello']
>>> x=y
>>> x.append('World!')
>>> x
['Hello', 'World!']
>>> y
['Hello', 'World!']
Cast output_data as a new bytearray and you should be good:
output_data = bytearray(input_data)
The following code is a part of a Fortran 90 program that I wrote in Plato IDE:
It is just the Box-Mueller algorithm to generate Gaussian random numbers.
Program brownstep2_single_stage
Integer:: i,j,m,n,countsucc!,a
Real:: dt,D,epsa,r1,r2,w,fptsum,fptdef1,fptdef2
Real,Dimension(0:100002) :: fx !gt
!T=1000.0 and n*dt=T
dt=0.001
m=100000
n=100000
D=1.0
!a=7
w=2
epsa=0.00001
fx(0)=6.0
!gt(0)=0
fptsum=0
countsucc=0
Call random_seed()
Do i=0,m
!Call random_seed(a)
Do j=0,n
Do while (w>=1.0.and.w<0.0)
Call random_number(r1)
Call random_number(r2)
!r=rand()
r1=2.0*r1-1
r2=2.0*r2-1
w=r1*r1+r2*r2
End do
w=sqrt((-2.0*log(w))/w)
r1=r1*w
r2=r2*w
If(mod(j,2)==0) then
w=r1
Else if(mod(j,2)==1) then
w=r2
End if
fx(j+1)=fx(j)+w*sqrt(2.0*D*dt)
If(fx(j+1)<epsa) then
fptsum=fptsum+(j+1)*dt
countsucc=countsucc+1
exit
End if
print *,i,j
End do
End do
fptdef1=fptsum/m
fptdef2=fptsum/countsucc
print *,'The value of fpt by 1st definition is:',fptdef1
print *,'The value of fpt by 2nd definition is:',fptdef2
print *,'The number of successful events is:',countsucc
print *,'The total number of events is:',m
End program brownstep2_single_stage
During compilation, it shows no error, but when run, it shows the following runtime error, claiming a negative or zero argument to the logarithm function.
Runtime error from program:e:\my files\sample2brownstep_gauss.exe
Run-time Error
Error: Negative or zero argument to logarithm routine
BROWNSTEP2_SINGLE_STAGE - in file sample2brownstep_gauss.f90 at line 31 [+02cc]
What should I do to avoid this?
The changed code above still has problems. w is still not set before the do while loop is reached for the first time and w is used in the condition. Use an 'infinite' do loop with an exit statement. This ensures that one attempt at w is always attempted. This would be better:
do
Call random_number(r1)
Call random_number(r2)
r1=2.0*r1-1
r2=2.0*r2-1
w=r1*r1+r2*r2
if (w .lt. 1.0) exit
End do
w=sqrt((-2.0*log(w))/w)
r1=r1*w
r2=r2*w
I would like to use fortran to read ultraviolet radiation data that has been produced by the Japan Aerospace Exploration Agency. This data is at a daily and monthly temporal resolution from 2000-2010 at a ~5 km spatial resolution. This question is worth answering as the data could be useful for a number of environment/health projects and is freely available, with proper acknowledgement of source and sharing of preprint of any subsequent publications, from:
ftp://suzaku.eorc.jaxa.jp/pub/GLI/glical/Global_05km/monthly/uvb/
There is a readme file available, which provides instructions on how to read data using fortran as follows:
Instructions for _le files
Header
Read header (size= pixel size *2byte):
character head*14400
read(10,rec=1) head
read(head,'(2i6,2f8.2,f8.4,2e12.5,a1,a8,a1,a40)')
& npixel,nline,lon_min,lat_max,reso,slope,offset,',',
& para,',',outfile
Read data (e.g., fortran77)
parameter(nl=7200, ml=3601)
... open file by "unformatted", "recl=nl*2(byte)" (,"bytereclen")
integer*2 i2buf(nl,ml)
do m=1,ml
read(10,rec=1+m) (i2buf(n,m), n=1,nl)
do n=1,nl
par=i2buf(n,m)*slope+offset
write(6,*) 'PAR[Ein/m^2/day]=',par
enddo
enddo
slope values
par__le : daily PAR [Ein/m^2/day] = DN * 0.01
dpar_le : direct PAR = DN * 0.01
swr__le : daily mean shortwave radiation [W/m^2] = DN * 0.01
tip__le : transmittance of instantaneous PAR at noon = DN * 0.0001
uva__le : daily mean UVA [W/m^2] = DN * 0.001
uvb__le : daily mean UVB [W/m^2] = DN * 0.0001
rpar_le : PAR-range surface reflectance (TOP of canopy/solid surfaces) = DN * 0.0001 (monthly data only)
error values
-1 as signed short integer (int16)
65535 as unsigned short integer (uint16)
Progress so far
I have downloaded and installed gfortran successfully on mac OSX. I have downloaded a test file (MOD02SSH_A20000224Av6_v601_7200_3601_uvb__le.gz) and decompressed it. I have created a program file:
PROGRAM readuvr
IMPLICIT NONE
!some code
END PROGRAM
I will then type the following into the command line to create an executable and run it to extract the data.
gfortran -o executable
./executable
As a complete beginner to fortran, my question is: how can I use the instructions provided to build a program that can read the data and output it into a text file?
Well, that file expands to 51,868,800 bytes. The comments imply the header is 14,400 bytes, which leaves 51,854,400 bytes of actual data payload.
There seem to be 7200 lines of data, so that means there are 7202 bytes per line. There seem to be 2 bytes (16-bit samples) so if we assume 2 bytes/sample, that means there are 3601 samples per line, which matches the ml=3601.
So basically, you need to read 14,400 bytes of header, then 7200 lines of data, each line consisting of 3601 values, each of those being 2 bytes wide...
Actually, if you are that unfamiliar with FORTRAN, you may like to extract the data with Perl which is already installed and available on OS X anyway. I have started a VERY SIMPLISTIC Perl program that reads the dat and prints the first 2 values on each line:
#!/usr/bin/perl
use strict;
use warnings;
# Read 14,400 bytes of header
my $buffer;
my $nBytes = 14400;
my $bytesRead = read (STDIN, $buffer, $nBytes) ;
my ($npixel,$nline,$lon_min,$lat_max,$reso,$slope,$offset,$junk)=split(' ',$buffer);
print "npixel:$npixel\n";
print "nline:$nline\n";
print "lon_min:$lon_min\n";
print "lat_max:$lat_max\n";
print "reso:$reso\n";
print "slope:$slope\n";
$offset =~ s/,.*//; # strip trailing comma and junk
print "offset:$offset\n";
# Read actual lines of data
my $line;
for(my $m=1;$m<=$nline;$m++){
read(STDIN,$line,$npixel*2);
my $x=$npixel*2;
my #values=unpack("S$x",$line);
printf "Line: %d",$m;
for(my $j=0;$j<2;$j++){
printf ",%f",$values[$j]*$slope+$offset;
}
printf "\n"; # newline
}
Save it as go.pl and then in the Terminal, type the following once to make it executable
chmod +x go.pl
and then run it like this
./go.pl < MOD02SSH_A20000224Av6_v601_7200_3601_uvb__le
Sample output extract:
npixel:7200
nline:3601
lon_min:0.00
lat_max:90.00
reso:0.0500
slope:0.10000E-03
offset:0.00000E+00
...
...
Line: 3306,0.099800,0.099800
Line: 3307,0.099900,0.099900
Line: 3308,0.099400,0.074200
Line: 3309,0.098900,0.098900
Line: 3310,0.098400,0.098400
Line: 3311,0.074300,0.074200
Line: 3312,0.071300,0.071200
fortran (f2003 or so) solution. (The linked instructions are awful by the way )
implicit none
character*80 para,outfile
character(len=:),allocatable::header,infile
integer npixel,nline,blen,i
c note kind=2 is not standard. This needs to be a 2-byte integer.
integer(kind=2),allocatable :: data(:,:)
real lon_min,lat_max,reso,slope,off
c header is plain text, so first open formatted and
c directly read header data
infile='MOD02SSH_A20000224Av6_v601_7200_3601_uvb__le'
open(10,file=infile)
read(10,*)npixel,nline,lon_min,lat_max,reso,slope,off,
$ para,outfile
close(10)
write(*,*)npixel,nline,lon_min,lat_max,reso,slope,off,
$ trim(para),' ',trim(outfile)
blen=2*npixel
allocate(character(len=blen)::header)
allocate(data(npixel,nline))
if( sizeof(data(1,1)).ne.2 )then
write(*,*)'error kind=2 did not give a 2 byte integer'
stop
endif
c now close and reopen for binary read.
c direct access approach:
open(20,file=infile,access='direct',recl=blen/4)
c note the granularity of the recl= specifier is not standard.
c ifort uses 4 bytes. (note this will break if npixel is not even )
read(20,rec=1)header
write(*,*)trim(header)
do i=1,nline
read(20,rec=i+1)data(:,i)
enddo
c note streams if available is simpler: (we don't need to know rec len )
c open(20,file=infile,access='stream')
c read(20)header,data
end
This is not actually validated because I don't have known file content to compare against.
Problem is to read a file of size about 20GB simultaneously by n processes. File contains one string at each line and Length of the strings may or may not be same. String length can be at-most 10 bytes long.
I have a cluster of having 16 nodes. Each node are the uni-processor and having 6GB RAM.I am using MPI to write Parallel codes.
What are the efficient way to partition this big file so that all resources can be utilized ?
Note: The constraints to the partitions is to read file as a chunk of fixed number of lines.
Assume file contains 1600 lines(e.g. 1600 strings). then first process should read from 1st line to 100th line, second process should do from 101th line to 200th line and so on....
As i think that one can't read a file by more than one processes at a time because we have only one file handler that point to somewhere only one string. then how other processes can read parallely from different chunks?
So as you're discovering, text file formats are poor for dealing with large amounts of data; not only are they larger than binary formats, but you run into formatting problems like here (seaching for newlines), and everything is much slower (data must be converted into strings). There can easily be 10x difference in IO speeds between text-based formats and binary formats for numerical data. But we'll assume for now you're stuck with the text file format.
Presumably, you're doing this partitioning for speed. But unless you have a parallel filesystem -- that is, multiple servers serving from multiple disks, and a FS that can keep those coordinated -- it's unlikely you're going to get a significant speedup from having multiple MPI tasks reading from the same file, as ultimately these requests are all going to get serialized anyway at the server/controller/disk level.
Further, reading in large blocks of data is going to be much faster than fseek()ing around and doing small reads looking for newlines.
So my suggestion would be to have one process (perhaps the last) read all the data in as few chunks as it can and send the relevant lines to each task (including, finally, itself). If you know how many lines the file has at the start, this is fairly simple; read in say 2 GB of data, search through memory for the end of the N/Pth line, and send that to task 0, send task 0 a "completed your data" message, and continue.
You don't specify if there are any constraints on the partitions, so I'll assume there are none. I'll also assume that you want the partitions to be as close to equal in size as possible.
The naïve approach would be to split the file into chunks of size 20GB/n. The starting position of chunk i wouild be i*20GB/n for i=0..n-1.
The problem with that is, of course, that there's no guarantee that chunk boundaries would fall between the lines of the input file. In general, they won't.
Fortunately, there's an easy way to correct for this. Having established the boundaries as above, shift them slightly so that each of them (except i=0) is placed after the following newline.
That'll involve reading 15 small fragments of the file, but will result in a very even partition.
In fact, the correction can be done by each node individually, but it's probably not worth complicating the explanation with that.
I think it would be better to write a piece of code that would get line lengths and distribute lines to processes. That distributing function would work not with strings themselves, but only their lengths.
To find an algorythm for even distribution of sources of fixed size is not a problem.
And after that the distributing func will tell other processes what pieces they have to get for work. Process 0 (distributor) will read a line. It already knows, that the line num. 1 should be worked by the process 1. ... P.0 reads line num. N and knows what process has to work with it.
Oh! We needn't optimize the distribution from the start. Simply the distributor process reads a new line from input and gives it to a free process. That's all.
So, you have even two solutions: heavily optimized and easy one.
We could reach even more optimalization if the distributor process will reoptimalize the unread yet strings from time to time.
Here is a function in python using mpi and the pypar extension to read the number of lines in a big file using mpi to split up the duties amongst a number of hosts.
def getFileLineCount( file1 ):
import pypar, mmap, os
"""
uses pypar and mpi to speed up counting lines
parameters:
file1 - the file name to count lines
returns:
(line count)
"""
p1 = open( file1, "r" )
f1 = mmap.mmap( p1.fileno(), 0, None, mmap.ACCESS_READ )
#work out file size
fSize = os.stat( file1 ).st_size
#divide up to farm out line counting
chunk = ( fSize / pypar.size() ) + 1
lines = 0
#set start and end locations
seekStart = chunk * ( pypar.rank() )
seekEnd = chunk * ( pypar.rank() + 1 )
if seekEnd > fSize:
seekEnd = fSize
#find start of next line after chunk
if pypar.rank() > 0:
f1.seek( seekStart )
l1 = f1.readline()
seekStart = f1.tell()
#tell previous rank my seek start to make their seek end
if pypar.rank() > 0:
# logging.info( 'Sending to %d, seek start %d' % ( pypar.rank() - 1, seekStart ) )
pypar.send( seekStart, pypar.rank() - 1 )
if pypar.rank() < pypar.size() - 1:
seekEnd = pypar.receive( pypar.rank() + 1 )
# logging.info( 'Receiving from %d, seek end %d' % ( pypar.rank() + 1, seekEnd ) )
f1.seek( seekStart )
logging.info( 'Calculating line lengths and positions from file byte %d to %d' % ( seekStart, seekEnd ) )
l1 = f1.readline()
prevLine = l1
while len( l1 ) > 0:
lines += 1
l1 = f1.readline()
if f1.tell() > seekEnd or len( l1 ) == 0:
break
prevLine = l1
#while
f1.close()
p1.close()
if pypar.rank() == 0:
logging.info( 'Receiving line info' )
for p in range( 1, pypar.size() ):
lines += pypar.receive( p )
else:
logging.info( 'Sending my line info' )
pypar.send( lines, 0 )
lines = pypar.broadcast( lines )
return ( lines )