Iterative and conditional deleting of lines in a file - macos

Question

Intro
I have a file named data.dat with the following structure:
1: 67: 1 :s
1: 315: 1 :s
1: 648: 1 :ns
1: 799: 1 :s
1: 809: 1 :s
1: 997: 1 :ns
2: 32: 1 :s
Algorithm
The algorithm that I'm looking for is:
Generate a random number between 1 and number of lines in this file.
Delete that line if the fourth column is "s".
Otherwise generate another random number and repeat this until the number of lines reaches to a certain value.
Technical Concepts
Though technical concepts are irrelevant to this algorithm, but I try to explain the problem. The data shows connectivity table of a network. This algorithm allows us to run it over different initial conditions and study general properties of these networks. Especially, because of randomness property of deleting bonds, any common behavior among these networks can be interpreted as a fundamental law.
Update: Another good reason to produce a random number in each step is that after removing each line, it's possible that property of being s/ns of remaining lines can be changed.
Code
Here is the code I have until now:
#!/bin/bash
# bash in OSX
While ((#there is at least 1 s in the fourth column)); do
LEN=$(grep -c "." data.dat) # number of lines
RAND=$((RANDOM%${LEN}+1)) # generating random number
if [[awk -F, "NR==$RAND" 'data.dat' | cut -d ':' -f 4- == "s"]]; then
sed '$RANDd' data.txt
else
#go back and produce another random
done
exit
I try to find the fourth column with awk -F, "NR==$RAND" 'data.dat' | cut -d ':' -f 4- and deleting the line by sed '$RANDd' data.txt.
Questions
How should I check that there is s pairs in my file?
I am not sure if the condition in if is correct.
Also, I don't know how to force loop after else to go back to generate another random number.
Thank you,
I really appreciate your help.

Answer 1

Personally, I would recommend against doing this in bash unless you have absolutely no choice.
Here's another way you could do it in Perl (quite similar in functionality to Alex's answer but a bit simpler):
use strict;
use warnings;
my $filename = shift;
open my $fh, "<", $filename or die "could not open $filename: $!";
chomp (my #lines = <$fh>);
my $sample = 0;
my $max_samples = 10;
while ($sample++ < $max_samples) {
my $line_no = int rand #lines;
my $line = $lines[$line_no];
if ($line =~ /:s\s*$/) {
splice #lines, $line_no, 1;
}
}
print "$_\n" for #lines;
Usage: perl script.pl data.dat
Read the file into the array #lines. Pick a random line from the array and if it ends with :s (followed by any number of spaces), remove it. Print the remaining lines at the end.
This does what you want but I should warn you that relying on built-in random number generators in any language is not a good way to arrive at statistically significant conclusions. If you need high-quality random numbers, you should consider using a module such as Math::Random::MT::Perl to generate them, rather than the built-in rand.

Answer 2

#!/usr/bin/env perl
# usage: $ excise.pl < data.dat > smaller_data.dat
my $sampleLimit = 10; # sample up to ten lines before printing output
my $dataRef;
my $flagRef;
while (<>) {
chomp;
push (#{$dataRef}, $_);
push (#{$flagRef}, 1);
}
my $lineCount = scalar #elems;
my $sampleIndex = 0;
while ($sampleIndex < $sampleLimit) {
my $sampleLineIndex = int(rand($lineCount));
my #sampleElems = split("\t", $dataRef->[$sampleLineIndex];
if ($sampleElems[3] == "s") {
$flagRef->[$sampleLineIndex] = 0;
}
$sampleIndex++;
}
# print data.dat to standard output, minus any sampled lines that had an 's' in them
foreach my $lineIndex (0..(scalar #{$dataRef} - 1)) {
if ($flagRef->[$lineIndex] == 1) {
print STDOUT $dataRef->[$lineIndex]."\n";
}
}

Answer 3

NumLine=$( grep -c "" data.dat )
while [ ${NumLine} -gt ${TargetLine} ]
do
# echo "Line at start: ${NumLine}"
RndLine=$(( ( ${RANDOM} % ${NumLine} ) + 1 ))
RndValue="$( echo " ${RANDOM}" | sed 's/.*\(.\{6\}\)$/\1/' )"
sed "${RndLine} {
s/^\([^:]*:\)[^:]*\(:.*:ns$\)/\1${RndValue}\2/
t
d
}" data.dat > /tmp/data.dat
mv /tmp/data.dat data.dat
NumLine=$( grep -c "" data.dat )
#cat data.dat
#echo "- Next Iteration -------"
done
tested on AIX (so not a GNU sed). Under Linux, use --posix for sed option and you can use a -i in place of temporary file + redirection + move in this case
Dont't forget that RANDOM is NOT a real RANDOM so study on network behavior based on not random value could not reflect a reality bu a specific case