convert 1mio Perl bitvector to .5 mio integer array - performance

Question

I have bitvector coming from the database of length 1 Mio with two bits
each representing one integer number for compressed storage:
the bit string : 10110001 from the database
array 2 3 0 1 needed for further processing
The current solution is:
my $bitstring =
$sth->fetchrow_array(); # has 2 bits / snp, need 2 convert to I
my $snp_no = 1000000;
for ( my $i = 0; $i <= $snp_no - 1; $i++ ) {
my $A2 = substr ($bitstring ,$j,2);
$j = $j + 2;
my $vec = Bit::Vector->new_Bin(32, $A2);
#bitArray->[$i] = $vec->to_Dec();
}
This does work but is is waaay too slow: to process one such vector take a second
and with thousands of them the processing will take hours.
does someone have an idea how this can be made faster?

Answer 1

If you start with the data "packed", use the following:
my #decode =
map [
($_ >> 6) & 3,
($_ >> 4) & 3,
($_ >> 2) & 3,
($_ >> 0) & 3,
],
0x00..0xFF;
my #nums = map #{ $decode[$_] }, unpack 'C*', $bytes;
For me, this takes about roughly 1.1s for 1,000,000 bytes, which is to say 1.1 microseconds per byte.
A specialized pure C solution takes about half the time.
use Inline C => <<'__EOI__';
void decode(AV* av, SV* sv) {
STRLEN len;
U8* p = (U8*)SvPVbyte(sv, len);
av_fill(av, len*4);
av_clear(av);
while (len--) {
av_push(av, newSViv(*p >> 6 ));
av_push(av, newSViv(*p >> 4 & 3));
av_push(av, newSViv(*p >> 2 & 3));
av_push(av, newSViv(*(p++) & 3));
}
}
__EOI__
decode(\my #nums, $bytes);
If you start with the binary representation of the bits, use the following first:
my $bytes = packed('B*', $bits);
(This assumes the number of bits is divisible by 8. Left-pad with zeroes if it isn't, and don't forget to remove the extra entries this creates in #decode.)

Answer 2

Is this any faster?
#!/usr/bin/env perl
use warnings;
use strict;
my %bin2dec = (
'0' => 0,
'1' => 1,
'00' => 0,
'01' => 1,
'10' => 2,
'11' => 3
);
#warn "$_ => $bin2dec{$_}\n" for sort keys %bin2dec;
my #results;
while (<>)
{
foreach my $bitstring (/([01]+)/g)
{
my #result;
#warn "bitstring is $bitstring\n";
for ( my $i = 0 ; $i < length($bitstring) ; $i += 2 )
{
#warn "value is ", substr( $bitstring, $i, 2 ), "\n";
push( #result, $bin2dec{ substr( $bitstring, $i, 2 ) } );
}
push( #results, \#result );
}
}
foreach my $result (#results)
{
print join( ' ', #$result ), "\n";
}
saved to file b2dec. Example output:
$ echo 10010101010010010101001111001011010101w00101010 | b2dec
2 1 1 1 1 0 2 1 1 1 0 3 3 0 2 3 1 1 1
0 2 2 2
$ b2dec b2dec
0
0
1
1
0
0
1
1
2
3
1
0