Dissociated press algorithm using ngram - algorithm

Question

I solve that kind of algorithm, and stuck how does algorithm works.
Dissociated press algorithm is https://en.wikipedia.org/wiki/Dissociated_press
N gram - https://en.wikipedia.org/wiki/N-gram
can make possible random string in consecutive string, so It can be implemented.
The dissociated press algorithm starts by printing a random n-gram.
Then it takes the last n-1 words it has printed, and chooses a random
n-gram that starts with these n-1 words. It prints the last word of
this n-gram, and repeats. So every consecutive n words of the output
text are an n-gram of the original text. In can sometimes happen that
the original text contains no n-gram starting with the n-1 words just
printed. In this case, the algorithm simply stops.
actually I don't know. how It is terminated.
ngram(1,2) ngram(2,3) ngram (3,4) ........ T T
is any example for me? I can't understand its text.

Answer 1

Well, first you split the test into n-grams:
The dissociated press algorithm starts by printing a random n-gram.
becomes (for n = 4)
the dissociated press algorithm
dissociated press algorithm starts
press algorithm starts by
algorithm starts by printing
starts by printing a
by printing a random
printing a random n-gram
etc. Then, you start with any n-gram you like, and start adding words that would complete last n-1 words of the text built so far to a known n-gram. As a result, the text you create seems almost readable — the larger n is, the more readable text you'll have.

Answer 2

It is not an awfully complex algorithm. The given version runs quite well:
public class Dissociator {
// Required size of the overlap
int overlapSize = 8;
// Size of the fragment
int fragmentSize = overlapSize;
// The initial sequence to dissociate, characters or words (could also dissociate some other objects).
ArrayList<String> initial;
boolean space;
boolean wordMode;
Random r = new Random(System.currentTimeMillis());
// Dissociate the given string.
public String dissociate(String in) {
ArrayList<String> a;
if (wordMode)
a = wordBased(in);
else
a = charBased(in);
ArrayList<String> out = dissociate(a);
StringBuilder b = new StringBuilder(out.size());
for (String s : out) {
b.append(s);
if (wordMode)
b.append(' ');
}
return b.toString();
}
/**
* Run dissociation algorithm
*
* #param input the initial sequence
* #return the dissociated sequence.
*/
public ArrayList<String> dissociate(ArrayList<String> input) {
initial = input;
ArrayList<String> out = new ArrayList<String>();
while (out.size() < input.size()) {
int size = r.nextInt(overlapSize);
if (size == 0)
size = 1;
ArrayList<String> tail = getTailOf(out, size);
// Find random sequence in the input that matches the tail
int p = r.nextInt(input.size() - 1) + 1; // Avoid zero.
int was = p - 1; // This variable allows to break dissociation if it is not possible to find
// the acceptable continuation.
boolean ok = false;
if (tail.size() > 0)
do {
while (input.get(p).equals(tail.get(0)) && p != was)
p = (p + 1) % input.size();
for (int j = 1; j < tail.size(); j++)
if (j + p < input.size()) {
if (!tail.get(j).equals(input.get(j + p))) {
ok = false;
break;
}
}
ok = true;
} while (!ok && p != was);
for (int j = p; j < Math.min(p + fragmentSize, input.size()); j++)
out.add(input.get(j));
}
return out;
}
// Get the tail of the given size.
private ArrayList<String> getTailOf(ArrayList<String> out, int size) {
if (size >= out.size())
return out;
else {
ArrayList<String> r = new ArrayList<String>(size);
for (int p = out.size() - size; p < out.size(); p++) {
r.add(out.get(p));
}
return r;
}
}
private static ArrayList<String> charBased(String in) {
ArrayList<String> is = new ArrayList<String>();
for (int i = 0; i < in.length(); i++)
is.add(in.substring(i, i + 1));
return is;
}
private static ArrayList<String> wordBased(String in) {
ArrayList<String> is = new ArrayList<String>();
StringTokenizer st = new StringTokenizer(in, " ,:()?!\"'");
while (st.hasMoreTokens())
is.add(st.nextToken());
return is;
}
public static void main(String[] args) throws Exception {
String in;
File f = new File(args[0]);
BufferedReader r = new BufferedReader(new FileReader(f));
String sr;
StringBuilder bb = new StringBuilder((int) f.length());
while ((sr = r.readLine()) != null) {
bb.append(sr);
bb.append(' ');
}
in = bb.toString();
Dissociator d = new Dissociator();
String b = d.dissociate(in);
System.out.println(b);
}
}