This question already has answers here:
How to select lines between two marker patterns which may occur multiple times with awk/sed
(10 answers)
Closed 6 years ago.
There is some text I need from a web page, a page whose length changes somewhat from day to day. I am looking to download that text periodically. I do not want/need several dozen lines from both the beginning and end of the roughly 250 line page. The total number of lines on the page will be unpredictable, so I will be needing to establish beginning/end points for the deletion I wish to perform based on bits of text that do not change from day to day. I've identified the target text patterns, so I'm looking to parse the content based on those such that the unwanted lines get deleted in the resulting document. I'm wanting to use command line utilites for this since I would like to automate the process and make a cron job out of it.
The download method of choice is to use lynx -dump www.specified.url my-download.txt
That part is working fine. But processing the dump so as to cut off the unwanted beginning and ending lines is so far not working. I found a sed example that, it seems, should do what I need:
sed -n '/Phrase toward the beginning/,/Phrase toward the end/p' file_to_parse.txt >parsed_file.txt
It works partially, meaning it cuts off the file's beginning at the right point (all lines preceding "Phrase toward the beginning"). But I cannot seem to make it cut lines from the end, i.e., lines following the phrase "Phrase toward the end." All my attempts using this formula have so far not touched the end of the file at all. I should probably mention that most of the lines in the dumped file lynx produces begin, for whatever reason, with 3 blank spaces--including the "Phrase toward the end" line I'm trying to specify as the point after which further lines should be deleted.
I assume there may be more than one utility that can do the sort of parsing I'm after--sed and awk are the likely candidates I can think of. I tend to gravitate toward sed since its workings are slightly less mysterious to me than are awk's. But truth be told, I really only have the vaguest of conceptions as to how to use sed. When it comes to using and/or understanding awk, I get lost very, very quickly. Perhaps there are other utilities that can, based on textual patterns, lop off portions of the beginning and ending of a text file?
Input on how I might use sed, awk--or any other similar utility--to accomplish my goal, will be appreciated. This is to be done on an Ubuntu machine, btw.
LATER EDIT: sorry for not having posted and example. The downloaded page will look something like the following
Unwanted line 1
Unwanted line 2
Unwanted line 3
Unwanted line etc
Phrase toward the beginning
Wanted line 1
Wanted line 2
Wanted line 3
Wanted line ca 4-198
Phrase toward the end
Unwanted line 200
Unwanted line 201
Unwanted line 202
Unwanted line . . . (to end of file)
The final output should look, on the other hand, like
Phrase toward the beginning
Wanted line 1
Wanted line 2
Wanted line 3
Wanted line ca 4-198
Phrase toward the end
I hope things will be clearer now. Please do bear in mind, though I've used line numbers to help better illustrate what I'm aiming to do, that I will be unable to do the desired deletions based on line numbers owing to the unpredictable ways in which the page I'm downloading will be changing.
If sed seems too difficult to debug, consider a double grep; for example here we list numbers 1 to 250, then grep for 70, plus up to 1000 lines after that, then grep for 80, plus up to 1000 lines before that:
seq 250 | grep -A 1000 '^70$' | grep -B 1000 '^80$'
Output:
70
71
72
73
74
75
76
77
78
79
80
Since the maximum length of the input files is known, 1000 is a safe number for your data (but overkill for the above example).
Applied to the OP data, the example would become:
grep -A 1000 'Phrase toward the beginning' download_page.txt | \
grep -B 1000 'Phrase toward the end'
The debugging advantage over sed is that the error messages from grep are simpler than those from sed.
Related
I'm just trying to count the occurrences of a word without writing an iteration file by file. I don't mind which kind of file it is. The closest I got is:
COUNT=$(grep -r -n -i "theWordImSearchingFor" .)
echo $COUNT
I thought about splitting that by spaces, but the problem is the output does not contain just the filename and the line but also the content (and that may have tons of spaces). w.g. I got:
./doc1.txt:29: This is the content containing theWordImSearchingFor but also other stuff
./doc1.txt:43: This is another line containing theWordImSearchingFor
./dir123/doc2.txt:339: .This is another...file...theWordImSearchingFor....
Any idea on how to keep it simple? TIA
To count the number of occurrences of a specific word, you need to use the same layout of code, but simpler. There are many ways to do this, but there are two much simpler versions of the word count that you have listed here.
The much two simpler versions,
1st way
2nd way
They both should work, unless problem with package installation.
I want to put serialized numbers on defined positions in a text file.
My idea is to use character patterns in the file, count up a variable and put them by using sed in the file. I tried this:
for number in 1 2 3 4 ; do
sed -ibak "s/var/$number" file.txt > file2.txt
done
(the arguments 1 2 3 ... are not the best solution, but I think, it should work)
With this code and tiny variations of it, I get different results, but no success.
I can cut/paste the pattern in the text, but it is always the last argument inserted (="3"). Why doesn´t sed take the iterated variable? (which is counted up, I tested it with echo).
The first iteration replaces var by 1, the next iteration replaces exactly the same var by 2, etc. - because you operate on the same input every time, and the pattern isn't dynamic.
It's not clear what you want to achieve, so it's hard to provide a working solution.
It might be easier to reach for Perl:
perl -pe 's/picvar/"pic" . ++$i/e'
I need help understanding a weird problem with sed, bash and a while loop.
MY data looks like this:
-File 1- CSV
account,hostnames,status,ipaddress,port,user,pass
-File 2- XML - This is a sample record set for two entries under one account
<accountname="account">
<cname="fqdn or simple name goes here">
<field="hostname">ahostname or ipv4 goes here</field>
<protocol>aprotocol</protocol>
<field="port">aportnumber</field>
<field="username">ausername</field>
<field="password">apassword</field>
</cname>
<cname="fqdn or simple name goes here">
<field="hostname">ahostname or ipv4 goes here</field>
<protocol>aprotocol</protocol>
<field="port">aportnumber</field>
<field="username">ausername</field>
<field="password">apassword</field>
</cname>
</accountname>
So far, I can add records in between the respective account holder from File1 to File2. But, if I need to remove records that no longer exists it does not work efficiently since it wipes other records from different accounts, ie it does not delete between a matched accountname.
I import from File 1 into File 2 with a while loop in my bash program:
-Bash Program excerpts-
//Read File in to F//
cat File 2 | while read F
do
//extract fields from F into variables
_vmname="$(echo $F |grep 'cname'| sed 's/<cname="//g' |sed 's/.\{2\}$//g')"
_account="$(echo $F | grep 'accountname' | sed 's/accountname="//g' |sed 's/.\{2\}$//g')"
// I then compare my File1 and look for stale records that are still in File2
if grep "$_vmname" File1 ;then
continue
else
// if not matched, delete between the respective accountname
sed -i '/'"$_account"'/,/<\/accountname>/ {/'"$_vmname"'/,/<\/cname>/d}' File2
If I manually declare _vmname and _account and run
sed -i '/'"$_account"'/,/<\/accountname>/ {/'"$_vmname"'/,/<\/cname>/d}' File2
It removes the stale records from File2. When I let my bash script run, it does not.
I think I have three problems:
Reading the variables for _vmname and _account name inside a loop makes it read numerous times. Any better way to do is appreciated.
I do not think the sed statement for matching these two patterns and then delete works like I want inside a while loop.
I may have a logic problem with my thought chain.
Any pointers, and please no awk, perl, lxml or python for this one.
Thanks!
and please no awk
I appreciate that you want to keep things simple, and I suppose awk seems more complicated than what you're doing. But I'd like to point out you have so far 3 grep and 4 sed invocations per line in the file, to process another file N times, once per line. That's O(mn) using the slowest method on the planet to read the file (a while loop). And it doesn't work.
I may have a logic problem with my thought chain.
I'm afraid we must allow for that possibility!
The right advice is to tackle XML with an XML parser, because XML is not a regular language and so can't be parsed with regular expressions. And that's really what you need here, because your program processes the whole XML document. You're not just plucking out bits and depending on incidental formatting artifacts; you want to add records that aren't there and remove those that "no longer exist". Apparently there is information in the XML document you need to preserve, else you would just produce it from the CSV. A parser would spoon-feed it to you.
The second-best advice is to use awk. I suppose you might try an approach like:
Process the CSV and produce the XML to be inserted.
In awk, first read the new input XML into an array keyed by cname, Then process the XML target once. For every CNAME, consult your array; if you find a match, insert your pre-constructed XML replacement (or modify the "paragraph" accordingly).
I'm not sure what the delete criteria are, so I don't know if it can be done in the same pass with step #2. If not, extract the salient information somehow. Maybe print a list of keys from each of the two files, and use comm(1) to produce a list of to-be-deleted. Then, similar to step #2, read in that list, and process the XML file one more time. Write anything you delete to stderr so you can keep track of what went missing, from what lines.
Any pointers
Whenever you find yourself processing the same file N times for N inputs, you know you're headed for trouble. One of the two inputs is always smaller, and that one can be put in some kind of array. cat file | while read is another warning signal, telling you use awk or any of a dozen obvious utilities that understand lines of text.
You posted your question on SO two weeks ago. I suspect no one answered it because you warned them away: preemptively saying, in effect, don't tell me to use good tools. I'm only here to suggest that you'll be more comfortable after you take off that straightjacket. Better tools, in this case, are the only right answer.
I know how to remove contiguous duplicates in vi. Either
:%!uniq
or
:g/^\(.*\)$\n\1$/d).
But I have a file which has data in a random order and there are some duplicate lines which are scattered all over the file. How do I remove all these duplicates without disturbing the order of lines? The first unique line should be kept and the next(or rest all) duplicate should be removed?
E.g. cat file1
Here's looking at you, Kid.
Casablanca
Here's looking at you, Kid.
Go ahead, make my day.
Dirty Harry
sleep 5
Go ahead, make my day.
Yippee-ki-yay
Output should be:
Here's looking at you, Kid.
Casablanca
Go ahead, make my day.
Dirty Harry
sleep 5
Yippee-ki-yay
There is one awk liner very handful for this:
$ awk '!a[$0]++' file
Here's looking at you, Kid.
Casablanca
Go ahead, make my day.
Dirty Harry
sleep 5
Yippee-ki-yay
It keeps track of the lines processed in the array a[]. Whenever the line comes again, the counter is already positive so that the condition is false and the line is not printed.
If you want to run it in vim, do:
:%!awk '\!a[$0]++'
^^
you have to escape the ! to be treated properly
Ok,first post..
So I have this assignment to decrypt cryptograms by hand,but I also wanted to automate the process a little if not all at least a few parts,so i browsed around and found some sed and awk one liners to do some things I wanted done,but not all i wanted/needed.
There are some websites that sort of do what I want, but I really want to just do it in bash for some reason,just because I want to understand it better and such :)
The script would take a filename as parameter and output another file such as solution$1 when done.
if [ -e "$PWD/$1" ]; then
echo "$1 exists"
else
echo "$1 doesnt exists"
fi
Would start the script to see if the file in param exists..
Then I found this one liner
sed -e "s/./\0\n/g" $1 | while read c;do echo -n "$c" ; done
Which works fine but I would need to have the number of occurences per letter, I really don't see how to do that.
Here is what I'm trying to achieve more or less http://25yearsofprogramming.com/fun/ciphers.htm for the counting unique letter occurences and such.
I then need to put all letters in lowercase.
After this I see the script doing theses things..
-a subscript that scans a dictionary file for certain pattern and size of words
the bigger words the better.
For example: let's say the solution is the word "apparel" and the crypted word is "zxxzgvk"
is there a regex way to express the pattern that compares those two words and lists the word "apparel" in a dictionnary file because "appa" and "zxxz" are similar patterns and "zxxzgvk" is of similar length with "apparel"
Can this be part done and is it realistic to view the problem like this or is this just far fetched ?
Another subscript who takes the found letters from the previous output word and that swap
letters in the cryptogram.
The swapped letters will be in uppercase to differentiate them over time.
I'll have to figure out then how to proceed to maybe rescan the new found words to see if they're found in a dictionnary file partly or fully as well,then swap more letters or not.
Did anyone see this problem in the past and tried to solve it with the patterns in words
like i described it,or is this just too complex ?
Should I log any of the swaps ?
Maybe just scan through all the crypted words and swap as I go along then do another sweep
with having for constraint in the first sweep to not change uppercase letters(actually to use them as more precise patterns..!)
Anyone did some similar script/program in another langage? If so which one? Maybe I can relate somehow :)
Maybe we can use your insight as to how you thought out your code.
I will happily include the cryptograms I have decoded and the one I have yet to decode :)
Again, the focus of my assignment is not to do this script but just to resolve the cryptograms. But doing scripts or at least trying to see how I would do this script does help me understand a little more how to think in terms of code. Feel free to point me in the right directions!
The cryptogram itself is based on simple alphabetic substitution.
I have done a pastebin here with the code to be :) http://pastebin.com/UEQDsbPk
In pseudocode the way I see it is :
call program with an input filename in param and optionally a second filename(dictionary)
verify the input file exists and isnt empty
read the file's content and echo it on screen
transform to lowercase
scan through the text and count the amount of each letter to do a frequency analysis
ask the user what langage is the text supposed to be (english default)
use the response to specify which letter frequencies to use as a baseline
swap letters corresponding to the frequency analysis in uppercase..
print the changed document on screen
ask the user to swap letters in the crypted text
if user had given a dictionary file as the second argument
then scan the cipher for words and find the bigger words
find words with a similar pattern (some letters repeating letters) in the dictionary file
list on screen the results if any
offer to swap the letters corresponding in the cipher
print modified cipher on screen
ask again to swap letters or find more similar words
More or less it the way I see the script structured.
Do you see anything that I should add,did i miss something?
I hope this revised version is more clear for everyone!
Tl,dr to be frank. To the only question i've found - the answer is yes:) Please split it to smaller tasks and we'll be happy to assist you - if you won't find the answer to these smaller questions before.
If you can put it out in pseudocode, it would be easier. There's all kinds of text-manipulating stuff in unix. The means to employ depend on how big are your texts. I believe they are not so big, or you would have used some compiled language.
For example the easy but costly gawk way to count frequences:
awk -F "" '{for(i=1;i<=NF;i++) freq[$i]++;}END{for(i in freq) printf("%c %d\n", i, freq[i]);}'
As for transliterating, there is tr utility. You can forge and then pass to it the actual strings in each case (that stands true for Caesar-like ciphers).
grep -o . inputfile | sort | uniq -c | sort -rn
Example:
$ echo 'aAAbbbBBBB123AB' | grep -o . | sort | uniq -c | sort -rn
5 B
3 b
3 A
1 a
1 3
1 2
1 1