Wrapping delimited lines, retaining first column, with minimum final length - ruby

Question

Looking to split up lines of content, retaining a headword.
I do a ton of text processing, and I like to use unix one-liners because they are easy for me to organize over time (vs. tons of scripts), I can easily chain them together, and I like (re)learning how to use classic unix functions. Often I will use a short awk, perl, or ruby one-liner, depending on which is the most elegant.
Here I have lines with X number of comma-delimited items. I want to divide these up, retaining the headword.
INPUT:
animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare, goose, horse, mouse, pig, dog, frog, bug, fish, duck, camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider, deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit, elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth, shark, salmon, shrimp, mosquito, horseshoe crab
OUTPUT:
animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare
animals = goose, horse, mouse, pig, dog, frog, bug, fish, duck
animals = camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider
animals = deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit
animals = elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth
animals = shark, salmon, shrimp, mosquito, horseshoe crab
Algorithm details:
input lines consist of a headword, then equals-sign, then a comma delimited list of at least 1 item.
In this example, most words are singles, but words could contain spaces (e.g. "horseshoe crab" at the end)
Split is at 9 items, UNLESS there are <3, in which case the final split could yield 12 on a line
There are multiple lines. e.g. the next line could be planets.
I had an idea to escape spaces, then use unix fold, and then awk to pull down the first column. This works exactly like the above:
echo "animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare, goose, horse, mouse, pig, dog, frog, bug, fish, duck, camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider, deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit, elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth, shark, salmon, shrimp, mosquito, horseshoe crab" \
| \tr ' ,' '_ ' \
| fold -s \
| perl -pe 's/=/\t/; s/^_/\t_/g;' \
| awk 'BEGIN{FS=OFS="\t"} $1==""{$1=p} {p=$1} 1' \
| tr '\t _' '=, '
But it only considers character length (not item count), and fails to consider my special case that I don't want <3 items hanging on the final line.
I think this is an elegant little puzzle, got ideas?

Answer 1

With Perl, one way
perl -wnE'
($head, #items) = split /\s*[,=]\s*/;
while (#items) {
#elems = splice #items, 0, 9;
if (#elems < 3) { $lines[-1] .= ", " . join ", ", #elems }
else { push #lines, join ", ", #elems }
}
say "$head = $_" for #lines; #lines = ()
' file
or
perl -wnE'
($head, #items) = split /\s*[,=]\s*/;
push #lines, join ", ", splice #items, 0, 9 while #items;
$lines[-2] .= ", " . pop #lines if 2 > $lines[-1] =~ tr/,//;
say "$head = $_" for #lines; #lines = ()
' file
Shown over multiple lines for readability, and can be copy-pasted into a bash terminal as such, but they can also be entered on one line. Tested with an added line of 11 (9+2) items.
Notes
split-ing by either , or = extracts the head-word first, and then the items on a line
splice removes and returns (the first 9) elements, which joined by , generate a line to print, until all elements are processed. The last group is added to the previous line-to-print instead if it has fewer than 3 elements
In the second version all elements are processed and then the last line-to-print checked for whether it need be added to the previous one instead, by counting commas in it

Answer 2

You may consider this awk:
awk 'BEGIN {FS=OFS=" = "} {
s = $2
while (match(s, /([^,]+, ){1,9}(([^,]+, ){2}[^,]+$)?/)) {
v = substr(s, RSTART, RLENGTH)
sub(/, $/, "", v)
print $1, v
s = substr(s, RLENGTH+1)
}
}' file
animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare
animals = goose, horse, mouse, pig, dog, frog, bug, fish, duck
animals = camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider
animals = deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit
animals = elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth
animals = shark, salmon, shrimp, mosquito, horseshoe crab
Pay special attention to regex used here /([^,]+, ){1,9}(([^,]+, ){2}[^,]+$)?/
That matches 1 to 9 words separated with , delimiter. This regex also has an optional part that matches upto 3 words before end of line.

Answer 3

With your shown samples only, please try following awk program. Written and tested in GNU awk should work in any awk.
Where I have created an awk variable named numberOfFields which contains number of fields you want to print(as segregated with new line as per shown samples).
awk -v numberOfFields="9" '
BEGIN{
FS=", ";OFS=", "
}
{
line=$0
sub(/ = .*/,"",line)
sub(/^[^ ]* =[^ ]* /,"")
for(i=1;i<=NF;i++){
printf("%s",(i%numberOfFields==0?OFS $i ORS line" = ":\
(i==1?line " = " $i:(i%numberOfFields>1?OFS $i:$i))))
}
}
END{
print ""
}
' Input_file
OR Above code is having printf statement in 2 lines(for readability purposes) if you want to have that into a single line itself then try following:
awk -v numberOfFields="9" '
BEGIN{
FS=", ";OFS=", "
}
{
line=$0
sub(/ = .*/,"",line)
sub(/^[^ ]* =[^ ]* /,"")
for(i=1;i<=NF;i++){
printf("%s",(i%numberOfFields==0?OFS $i ORS line" = ":(i==1?line " = " $i:(i%numberOfFields>1?OFS $i:$i))))
}
}
END{
print ""
}
' Input_file
Explanation: Adding detailed explanation for above.
awk -v numberOfFields="9" ' ##Starting awk program from here, creating variable named numberOfFields and setting its value to 9 here.
BEGIN{ ##Starting BEGIN section of awk here.
FS=", ";OFS=", " ##Setting FS and OFS to comma space here.
}
{
line=$0 ##Setting value of $0 to line here.
sub(/ = .*/,"",line) ##Substituting space = space everything till last of value in line with NULL.
sub(/^[^ ]* =[^ ]* /,"") ##Substituting from starting till first occurrence of space followed by = followed by again first occurrence of space with NULL in current line.
for(i=1;i<=NF;i++){ ##Running for loop here for all fields.
printf("%s",(i%numberOfFields==0?OFS $i ORS line" = ":\ ##Using printf and its conditions are explained below of code.
(i==1?line " = " $i:(i%numberOfFields>1?OFS $i:$i))))
}
}
END{ ##Starting END block of this program from here.
print "" ##Printing newline here.
}
' Input_file ##Mentioning Input_file name here.
Explanation of printf condition above:
(
i%numberOfFields==0 ##checking if modules value of i%numberOfFields is 0 here, if this is TRUE:
?OFS $i ORS line" = " ##Then printing OFS $i ORS line" = "(comma space field value new line line variable and space = space)
:(i==1 ##If very first condition is FALSE then checking again if i==1
?line " = " $i ##Then print line variable followed by space = space followed by $i
:(i%numberOfFields>1?OFS $i:$i) ##Else if if modules value of i%numberOfFields is greater than 1 then print OFS $i else print $i.
)
)

Answer 4

One awk idea:
awk -F'[=,]' -v min=3 -v max=9 '
{ for (i=2; i<=NF; i++) {
if ( (i-1) % max == 1 && (NF-i+1 > min) ) {
if ( i > max ) print newline
newline=$1 "="
pfx=""
}
newline=newline pfx $i
pfx=","
}
print newline
}
' raw.dat
Sample data:
$ cat raw.dat
animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare, goose, horse, mouse, pig, dog, frog, bug, fish, duck, camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider, deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit, elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth, shark, salmon, shrimp, mosquito, horseshoe crab
planets = mercury, venus, earth, mars, jupiter, saturn, uranus, neptune, pluto, vulcan, arrakis, hoth, naboo
numbers = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
numbers2 = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13
With -v min=3 -v max=9 we get:
animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare
animals = goose, horse, mouse, pig, dog, frog, bug, fish, duck
animals = camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider
animals = deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit
animals = elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth
animals = shark, salmon, shrimp, mosquito, horseshoe crab
planets = mercury, venus, earth, mars, jupiter, saturn, uranus, neptune, pluto
planets = vulcan, arrakis, hoth, naboo
numbers = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
numbers2 = 1, 2, 3, 4, 5, 6, 7, 8, 9
numbers2 = 10, 11, 12, 13
Addressing OP's comment about using one-liners ...
While this awk script can certainly be jammed into a one-liner I'm guessing OP will a) find it hard to edit/maintain and b) too easy to screw up if having to (re)type over and over again.
One (obvious?) idea is to wrap the awk code in a function, eg:
splitme() {
awk -F'[=,]' -v min=$1 -v max=$2 '
{ for (i=2; i<=NF; i++) {
if ( (i-1) % max == 1 && (NF-i+1 > min) ) {
if ( i > max ) print newline
newline=$1 "="
pfx=""
}
newline=newline pfx $i
pfx=","
}
print newline
}' "${3:--}"
}
NOTES:
parameterized the min and max values so as to pull from the command line
parameterized the file reference to pull from either the command line ($3) or stdin (-)
OP can add more logic to verify/validate input parameters as needed
Whether calling as a standalone against a file:
$ splitme 3 9 raw.dat
Or calling in a pipeline:
$ cat raw.dat | splitme 3 9
Both generate:
animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare
animals = goose, horse, mouse, pig, dog, frog, bug, fish, duck
animals = camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider
animals = deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit
animals = elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth
animals = shark, salmon, shrimp, mosquito, horseshoe crab
planets = mercury, venus, earth, mars, jupiter, saturn, uranus, neptune, pluto
planets = vulcan, arrakis, hoth, naboo
numbers = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12
numbers2 = 1, 2, 3, 4, 5, 6, 7, 8, 9
numbers2 = 10, 11, 12, 13

Answer 5

awk -F"[=,]" -v max="9" '{
for(i=2; i<=NF; i+=max){
row = ""
for(j=i; j<=i+max-1; j++){
row=row $(j) ","
}
gsub(/,+$/, "", row)
printf "%s=%s \n", $1, row
}
}' input_file
animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare
animals = goose, horse, mouse, pig, dog, frog, bug, fish, duck
animals = camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider
animals = deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit
animals = elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth
animals = shark, salmon, shrimp, mosquito, horseshoe crab
planets = mercury, venus, earth, mars, jupiter, saturn, uranus, neptune, pluto
planets = vulcan, arrakis, hoth, naboo
numbers = 1, 2, 3, 4, 5, 6, 7, 8, 9
numbers = 10, 11, 12, 13, 14, 15, 16
cars = mercedes benz, bmw, audi, vw, porsche, seat, skoda, opel, renault
cars = mazda, toyota, honda

Answer 6

Here are two Ruby solutions to process one line. The variable str holds one line (the line beginning 'animals = ...' in the example).
#1 Use a regular expression
RGX = \A\w+| *= *|(?:[^,]+, *){0,10}[^,]+\z|(?:[^,]+, *){9}
def break_line(str)
headword, _, *lines = str.scan(RGX)
lines.each { |line| puts "#{headword} = #{line.sub(/, *\z/, '')}" }
end
brake_line(str)
animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare
animals = goose, horse, mouse, pig, dog, frog, bug, fish, duck
animals = camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider
animals = deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit
animals = elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth
animals = shark, salmon, shrimp, mosquito, horseshoe crab
The regular expression can be written in free-spacing mode to make it self-documenting.
RGX =
/
\A # match beginning of string
\w+ # match one or more word chars (e.g., "animals")
| # or
[ ]*=[ ]* # "=" preceded and followed by zero or more spaces
| # or
(?: # begin a non-capture group
[^,]+ # match one or more chars other than a comma
,[ ]* # match a comma and zero or more spaces
){0,10} # end non-capture group and execute 0-10 times
[^,]+ # match one or more chars other than a comma
\z # match end of string
| # or
(?: # begin a non-capture group
[^,]+ # match one or more chars other than a comma
,[ ]* # match a comma and zero or more spaces
){9} # end non-capture group and execute 1-7 times
/x # invoke free-spacing regex definition mode
Demo
When executed for the example str we would find the following.
headword
#=> "animals"
_
#=> "="
lines
#=> ["lizard, bird, bee, snake, whale, eagle, beetle, ",
"mule, hare, goose, horse, mouse, pig, dog, ",
"frog, bug, fish, duck, camel, squirrel, owl, ",
"chicken, pigeon, lion, sheep, bear, spider, deer, ",
"tiger, lobster, dinosaur, cat, goat, rat, cricket, ",
"rabbit, elephant, crow, fox, donkey, monkey, butterfly, ",
"crab, leopard, moth, shark, salmon, shrimp, mosquito, horseshoe crab"]
Ruby has a convention of using the variable _ in situations when its value is not subsequently used in calculations. This is mainly to so-inform the reader.
#2 Extract and group words
def break_line(str)
headword, *words = str.split(/ *[,=] */)
groups = words.each_slice(9).to_a
if groups[-1].size < 3
groups[-2] += groups[-1]
groups.pop
end
groups.each { |group| puts "#{headword} = #{group.join(', ')}" }
end
brake_line(str)
animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare
animals = goose, horse, mouse, pig, dog, frog, bug, fish, duck
animals = camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider
animals = deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit
animals = elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth
animals = shark, salmon, shrimp, mosquito, horseshoe crab
By way of a partial explanation, we would obtain the following for the example:
headword
#=> "animals"
words
#=> ["lizard", "bird",,..."horseshoe crab"]
groups
#=> [["lizard", "bird", "bee", "snake", "whale", "eagle",
"beetle", "mule", "hare"],
["goose", "horse", "mouse", "pig", "dog", "frog",
"bug", "fish", "duck"],
["camel", "squirrel", "owl", "chicken", "pigeon", "lion",
"sheep", "bear", "spider"],
["deer", "tiger", "lobster", "dinosaur", "cat", "goat",
"rat", "cricket", "rabbit"],
["elephant", "crow", "fox", "donkey", "monkey", "butterfly",
"crab", "leopard", "moth"],
["shark", "salmon", "shrimp", "mosquito", "horseshoe crab"]]
As the element of groups contains more than two elements (it contains five), groups is not subsequently modified. Had the last line been permitted to have at most 14 (rather than 11) elements it would have been changed to
["elephant", "crow", "fox", "donkey", "monkey", "butterfly", "crab",
"leopard", "moth", "shark", "salmon", "shrimp", "mosquito", "horseshoe crab"]

Answer 7

Took a while to modify my solution to make it work across both gawk and mawk by performing the equivalent of $1 = $1 towards the end of the regex chain;
$(NF!=NF=NF) expands to NF != (NF=NF) internally, which is always false, so the whole thing just means $0, but embedding $1=$1 within it :
input ::
1 animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare, goose, horse, mouse, pig, dog, frog, bug, fish, duck, camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider, deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit, elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth, shark, salmon, shrimp, mosquito, horseshoe crab
2 planets = mercury, venus, earth, mars, jupiter, saturn, uranus, neptune, pluto-cuz-it-shoudlve-been, planetX
command ::
[mg]awk '
BEGIN {
FS = (OFS = " = ") "*"
_=__ = (___="[^,]+")"[,]"
gsub(".",_,__)
__ = (__)_ "(("_")?("_")?"___"$)?"
_ = ORS } gsub(__,"&"_ $1 OFS)+gsub("[,]"_,_)+sub((_)"+([^,]*)$","", $(NF!=NF=NF))'
output (mawk 1.3.4) ::
1 animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare
2 animals = goose, horse, mouse, pig, dog, frog, bug, fish, duck
3 animals = camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider
4 animals = deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit
5 animals = elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth
6 animals = shark, salmon, shrimp, mosquito, horseshoe crab
7 planets = mercury, venus, earth, mars, jupiter, saturn, uranus, neptune, pluto-cuz-it-shoudlve-been, planetX
output (gawk 5.1.1) ::
1 animals = lizard, bird, bee, snake, whale, eagle, beetle, mule, hare
2 animals = goose, horse, mouse, pig, dog, frog, bug, fish, duck
3 animals = camel, squirrel, owl, chicken, pigeon, lion, sheep, bear, spider
4 animals = deer, tiger, lobster, dinosaur, cat, goat, rat, cricket, rabbit
5 animals = elephant, crow, fox, donkey, monkey, butterfly, crab, leopard, moth
6 animals = shark, salmon, shrimp, mosquito, horseshoe crab
7 planets = mercury, venus, earth, mars, jupiter, saturn, uranus, neptune, pluto-cuz-it-shoudlve-been, planetX