Given a .txt file (DNA-sequence alignment report), in this format:
5463784 reads; of these:
5463784 (100.00%) were paired; of these:
841569 (15.40%) aligned concordantly 0 times
4469608 (81.80%) aligned concordantly exactly 1 time
152607 (2.79%) aligned concordantly >1 times
----
841569 pairs aligned 0 times concordantly or discordantly; of these:
1683138 mates make up the pairs; of these:
1407028 (83.60%) aligned 0 times
226521 (13.46%) aligned exactly 1 time
49589 (2.95%) aligned >1 times
87.12% overall alignment rate
What is the easiest and shortest way to grab subportions of specific lines? For example if I wanted to grab the 'exactly' lines I can use:
awk '/exactly/{print}'
Which would return:
4469608 (81.80%) aligned concordantly exactly 1 time
226521 (13.46%) aligned exactly 1 time
But i'm not sure how to then split what's returned to obtain 4469608 and 226521 within an array (to then eventually sum together) to give a variable set to 4696129.
awk '/exactly/ {sum=sum+$1;}END{print sum}' dna
Take actions on those line where exactly is present, store the value of first column in a awk variable called sum and in the end print.
Related
I decompiled a file in ghidra and I sawed a lot of CONCAT+RandomNumber in decompiled file!
what does they mean ?
Let me cite the Ghidra Help (F1) first:
CONCAT31(x,y) - Concatenation operator - PIECE
The digit '3' indicates the size of the input operand 'x' in bytes.
The digit '1' indicates the size of the input operand 'y' in bytes.
The parameters 'x' and 'y' hold the values being concatenated.
CONCAT31(0xaabbcc,0xdd) = 0xaabbccdd
Concatenate the bytes in 'x' with the bytes in 'y'. 'x' becomes the most significant bytes, and 'y' the least significant bytes, in the result. So all these "functions" prefixed with CONCAT belong to a set of internal decompiler functions used by Ghidra to express things that normally not simply expressed in the C-like high level representation.
CONCAT in particular could be modeled as a left shift of the first argument by the size of the second argument and then logical and-ing the two parameters. But for humans it's much easier to think of it as "put the two things next to each other".
The numbers following CONCAT only matter if the passed arguments are not the expected sizes and are probably mainly there to make things more explicit. Concretely, you shouldn't read CONCAT15 as "concat fifteen" but as "concat one five": The first argument is expected to have a size of one byte while the second has a size of five, totaling to an amount of six bytes: CONCAT15(0x12, 0x3456789012) is the same as 0x123456789012.
P.S.: CONCAT412 almost certainly means concating 4 and 12 bytes, not 41 and 2.
Given the following:
for(condition which changes $z)
aptitude show $z | grep -E 'Uncompressed Size: |x' | sed s/Uncompressed Size: //";
done
That means 3 items are outputting to screen ($Z, Uncompressed Size, x).
I want all of that to fit on one line, and a line I deem is = 100 characters.
So, ($Z, Uncompressed Size, x) must fit on one line. But X is very long and will have to be truncated. So there is a requirement to add "used" characters by $z and Uncompressed Size, so that x can be truncated dynamically. I love scripting and being able to do this I deem an absolute must. Needless to say all 3 items being output to screen change hence the characters of the first two outputs must be calculated to subtract from the allowed characters for x, and sum of all characters between all 3 items cannot exceed 100 characters.
sed 's/.//5g'
Lmao, sometimes I wish I thought in simpler terms; complicated description + simple solution = simple problem over complicated by interpreter.
Thank you, Barmar
That only leaves sed (100 - amount of characters used by $z which is this function: ${#z}
I want to write a module/function, which takes input an unsigned integer(32-bit) and outputs 8-bit value. output values must be in relative order as input.
and the output value once assigned should not be altered.
testing:
input: randomly generated 32-bit integer
output: in range 0-255
repeat the action 256 times.. till the output values are exhausted.
all output values should be relatively in same order as input values.
EDIT: Based on comments, it is impossible to find a perfect solution for this.
However, Can we find best solution possible. i.e without comparing the input(32-bit) values.. generating relative order output of size 8-bit.
Assume the first input number is 123456. You don't know whether the following numbers will be smaller or greater than this. So the best thing to do is to output a number in the middle of the available range: 127.
Store this assignment (123456,127).
Now, if the second number is greater, e.g. 234567, output a number form the upper half of the output range:
(234567,191)
With each following number, output the center of the corresponding output range:
(200000, 159)
You get the idea.
Of course, depending on the input sequence, you normally cannot assign unique output to each input.
Worst case:
Input sequence 2^31, 2^30, ... i.e. 2147483648, 1073741824 yields output 127, 63, 31, 15, 7, 3, 1, 0, 0, ...
So you can only use 8 of the 256 available output numbers.
You can however fine tune this near the limits of the input range. If the input number is smaller than 127, then output == input.
From your comment objecting to the idea of simply taking the top bits, and reading between the lines of "till the output values are exhausted", it sounds like you want an algorithm which:
Takes 256 distinct 32-bit integers as its input
Returns the values 0..255 associated with those integers, in order.
But of course this is simply a sort algorithm. Sort the 256 inputs, and then the output value is the 0-based index in the sort result.
If you're looking for some way to return the output value without having first examined all the inputs, it's pretty clear that that's not possible, given your "output value once assigned should not be altered" requirement.
This is just a sorting problem in disguise.
You can't output anything until you've seen all the values - unless you're under-specifying the problem, and there's a limit on the number of inputs.
If you have the intuitive limit of 256 inputs, whatever output you assign the first input, you can come up with less than 255 inputs such that you won't be able to map them.
In my app I will have several lines of numbers from 4 to 10
characters in length. How would I go around separating the generated
numbers every 3 numbers (at thousand, million, billion).
I will not know the length of the numbers beforehand so the same .text
label can sometimes be longer or shorter.
e.g. from 1234567 12345 1234567890
to 1,234,567 12,345 1,234,567,890
The separators can be a space, a dot or a comma (any is fine). I am
quite familiar with AI2 but can't figure this one out.
EDIT: Seems like I managed to segment the text (yay!)
What would I have to do now if I have let's say 10 labels with all different lengths? It would be extremely excessive to check length of each label using an if and then segmenting it using corresponding length option. Is there a way to shorten the process?
Thanks a lot :)
You can write your own custom procedure, for example see the following App Inventor Classic example, which uses a for range loop:
The algorithm works like this (example=2134256):
result after first loop: ,256
result after second loop: ,134,256
then add the remaining digits of the number, in this case 2
result in the end: 2,134,256
Edit: For App Inventor 2, you also have to use a procedure with result. Also let me recommend to use a local variable result instead of a global variable.
Can anyone please explain arithmetic encoding for data compression with implementation details ? I have surfed through internet and found mark nelson's post but the implementation's technique is indeed unclear to me after trying for many hours.
Mark nelson's explanation on arithmetic coding can be located at
http://marknelson.us/1991/02/01/arithmetic-coding-statistical-modeling-data-compression/
The main idea with arithmetic compression is its the capability to code a probability using the exact amount of data length required.
This amount of data is known, proven by Shannon, and can be calculated simply by using the following formula : -log2(p)
For example, if p=50%, then you need 1 bit.
And if p=25%, you need 2 bits.
That's simple enough for probabilities which are power of 2 (and in this special case, huffman coding could be enough). But what if the probability is 63% ? Then you need -log2(0.63) = 0.67 bits. Sounds tricky...
This property is especially important if your probability is high. If you can predict something with a 95% accuracy, then you only need 0.074 bits to represent a good guess. Which means you are going to compress a lot.
Now, how to do that ?
Well, it's simpler than it sounds. You will divide your range depending on probabilities. For example, if you have a range of 100, 2 possible events, and a probability of 95% for the 1st one, then the first 95 values will say "Event 1", and the last 5 remaining values will say "Event 2".
OK, but on computers, we are accustomed to use powers of 2. For example, with 16 bits, you have a range of 65536 possible values. Just do the same : take the 1st 95% of the range (which is 62259) to say "Event 1", and the rest to say "Event 2". You obviously have a problem of "rounding" (precision), but as long as you have enough values to distribute, it does not matter too much. Furthermore, you are not constrained to 2 events, you could have a myriad of events. All that matters is that values are allocated depending on the probabilities of each event.
OK, but now i have 62259 possible values to say "Event 1", and 3277 to say "Event 2". Which one should i choose ?
Well, any of them will do. Wether it is 1, 30, 5500 or 62256, it still means "Event 1".
In fact, deciding which value to select will not depend on the current guess, but on the next ones.
Suppose i'm having "Event 1". So now i have to choose any value between 0 and 62256. On next guess, i have the same distribution (95% Event 1, 5% Event 2). I will simply allocate the distribution map with these probabilities. Except that this time, it is distributed over 62256 values. And we continue like this, reducing the range of values with each guess.
So in fact, we are defining "ranges", which narrow with each guess. At some point, however, there is a problem of accuracy, because very little values remain.
The idea, is to simply "inflate" the range again. For example, each time the range goes below 32768 (2^15), you output the highest bit, and multiply the rest by 2 (effectively shifting the values by one bit left). By continuously doing like this, you are outputting bits one by one, as they are being settled by the series of guesses.
Now the relation with compression becomes obvious : when the range are narrowed swiftly (ex : 5%), you output a lot of bits to get the range back above the limit. On the other hand, when the probability is very high, the range narrow very slowly. You can even have a lot of guesses before outputting your first bits. That's how it is possible to compress an event to "a fraction of a bit".
I've intentionally used the terms "probability", "guess", "events" to keep this article generic. But for data compression, you just to replace them with the way you want to model your data. For example, the next event can be the next byte; in this case, you have 256 of them.
Maybe this script could be useful to build a better mental model of arithmetic coder: gen_map.py. Originally it was created to facilitate debugging of arithmetic coder library and simplify generation of unit tests for it. However it creates nice ASCII visualizations that also could be useful in understanding arithmetic coding.
A small example. Imagine we have an alphabet of 3 symbols: 0, 1 and 2 with probabilities 1/10, 2/10 and 7/10 correspondingly. And we want to encode sequence [1, 2]. Script will give the following output (ignore -b N option for now):
$ ./gen_map.py -b 6 -m "1,2,7" -e "1,2"
000000111111|1111|111222222222222222222222222222222222222222222222
------011222|2222|222000011111111122222222222222222222222222222222
---------011|2222|222-------------00011111122222222222222222222222
------------|----|-------------------------00111122222222222222222
------------|----|-------------------------------01111222222222222
------------|----|------------------------------------011222222222
==================================================================
000000000000|0000|000000000000000011111111111111111111111111111111
000000000000|0000|111111111111111100000000000000001111111111111111
000000001111|1111|000000001111111100000000111111110000000011111111
000011110000|1111|000011110000111100001111000011110000111100001111
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
001100110011|0011|001100110011001100110011001100110011001100110011
010101010101|0101|010101010101010101010101010101010101010101010101
First 6 lines (before ==== line) represent a range from 0.0 to 1.0 which is recursively subdivided on intervals proportional to symbol probabilities. Annotated first line:
[1/10][ 2/10 ][ 7/10 ]
000000111111|1111|111222222222222222222222222222222222222222222222
Then we subdivide each interval again:
[ 0.1][ 0.2 ][ 0.7 ]
000000111111|1111|111222222222222222222222222222222222222222222222
[ 0.7 ][.1][ 0.2 ][ 0.7 ]
------011222|2222|222000011111111122222222222222222222222222222222
[.1][ .2][ 0.7 ]
---------011|2222|222-------------00011111122222222222222222222222
Note, that some intervals are not subdivided. That happens when there is not enough space to represent every subinterval within given precision (which is specified by -b option).
Each line corresponds to a symbol from the input (in our case - sequence [1, 2]). By following subintervals for each input symbol we'll get a final interval that we want to encode with minimal amount of bits. In our case it's a first 2 subinterval on a second line:
[ This one ]
------011222|2222|222000011111111122222222222222222222222222222222
Following 7 lines (after ====) represent the same interval 0.0 to 1.0, but subdivided according to binary notation. Each line is a bit of output and by choosing between 0 and 1 you choose left or right half-subinterval. For example bits 01 corresponds to subinterval [0.25, 05) on a second line:
[ This one ]
000000000000|0000|111111111111111100000000000000001111111111111111
The idea of arithmetic coder is to output bits (0 or 1) until the corresponding interval will be entirely inside (or equal to) the interval determined by the input sequence. In our case it's 0011. The ~~~~ line shows where we have enough bits to unambiguously identify the interval we want.
Vertical lines formed by | symbol show the range of bit sequences (rows) that could be used to encode the input sequence.
First of all thanks for introducing me to the concept of arithmetic compression!
I can see that this method has the following steps:
Creating mapping: Calculate the fraction of occurrence for each letter which gives a range size for each alphabet. Then order them and assign actual ranges from 0 to 1
Given a message calculate the range (pretty straightforward IMHO)
Find the optimal code
The third part is a bit tricky. Use the following algorithm.
Let b be the optimal representation. Initialize it to empty string (''). Let x be the minimum value and y the maximum value.
double x and y: x=2*x, y=2*y
If both of them are greater than 1 append 1 to b. Go to step 1.
If both of them are less than 1, append 0 to b. Go to step 1.
If x<1, but y>1, then append 1 to b and stop
b essentially contains the fractional part of the number you are transmitting. Eg. If b=011, then the fraction corresponds to 0.011 in binary.
What part of implementation do you not understand?