Is it possible to grab text from a online text file via grep/cat/awk or someting else? (in bash)
The way i currently do this is i download the text file to the drive and grep/cat into the file for it's text.
curl -o "$TMPDIR"/"text.txt" http://www.example.com/text.txt
cat/grep "$TMPDIR"/text.txt
rm -rf "$TMPDIR"/"text.txt"
Is one of the text grabbers (or another one) capable enough to grab something from a text file on the internet?
This would get rid of the whole downloadfile-readfile-deletefile process and just replace it with one command, speeding up things considerably if you have a lot of those strings.
I couldn't find anything via the man pages or googling around, maybe you guys know something.
Use curl -o - http://www.example.com/text.txt | grep "something".
-o - tells curl that it "downloads to stdout", other utils such as wget, lynx and links also have corresponding functionality.
You might try netcat - this is exactly what it was made for.
You could at least pipe your commands to avoid manually creating a temporary file:
curl … | cat/grep …
Related
The following situation:
I am on a different mac (no command history) using the Terminal (bash) remembering only a part of a command e.g. searching for a command with util in it. Did not remember that it was mdutil.
How to fuzzy search for a command in an efficient manner completely in the terminal, without creating new files?
Typical ways I do it now:
To find that command I could google, not always efficient and needs internet connection and browser.
Or Tab Tab, see all commands and scroll through them until I recognize the right one.
Or output all commands to a textfile and search in that.
I guess you could do something like this:
oldIFS="$IFS"
IFS=:
for dir in $PATH; do
ls $dir/*util* 2> /dev/null
done
IFS="$oldIFS"
That would loop through all the directories in your $PATH looking for a command that contains util.
How about starting with man -k and refining, like this:
man -k util | grep -i meta
Moose::Util::MetaRole(3pm) - Apply roles to any metaclass, as well as the object base class
mdutil(1) - manage the metadata stores used by Spotlight
compgen -ca | grep util
did it the best. Instead of util you can search any part of a command.
Like gniourf_gniourf said, a better solution would be
compgen -caX '!*util*'
i've started playing around with curl a few days ago. For any reason i couldn't figure out how to archive the following.
I would like to get the original filename with the output option
-O -J
AND put there some kind of variable, like time stamp, source path or whatever. This would avoid the file overwriting issue and also make it easier for further work with it.
Here are a few specs about my setup
Win7 x64
curl 7.37.0
Admin user
just commandline no PHP or script or so one
no scripting solutions please, need tihs command in a single line for Selenium automation
C:>curl --retry 1 --cert c:\certificate.cer --URL https://blabla.com/pdf-file --user username:password --cookie-jar cookie.txt -v -O -J
I've played around with various things i found online like
-o %(file %H:%s)
-O -J/%date%
-o $(%H) bla#1.pdf
but it always just print out the file as it is named link "%(file.pdf" or some other shitty names. I guess this is something pointing to escaping and quoting issues but cant find it right now.
No scripting solutions please, I need tihs command in a single line for Selenium automation.
Prefered output
originalfilename_date_time_source.pdf
Let me know if you get a solution for this.
Hi I am new to scripting and I do mean a complete noobie. I am working on a script to automatically make a torrent with nemo scripts.
#!/bin/bash
DIR="$NEMO_SCRIPT_SELECTED_FILE_PATHS"
BNAME=$(basename "$DIR")
TFILE="$BNAME.torrent"
TTRACKER="http://tracker.com/announce.php"
USER="USERNAME"
transmission-create -o "/home/$USER/Desktop/$TFILE" -t $TTRACKER "$DIR"
It does not work.
However if I replace
DIR="$NEMO_SCRIPT_SELECTED_FILE_PATHS"
with
DIR="absolutepath"
than it works like a charm. It creates it on the desktop with the tracker I want. I think this would come in handy for many people. I dont really know what to put. Have questions please ask. Again complete noobie.
The $NEMO_SCRIPT_SELECTED_FILE_PATHS is the same as $NAUTILUS_SCRIPT_SELECTED_FILE_PATHS. It's populated by nemo/nautilus when you run the script and contains a newline-delimited (I think) list of the selected files/folders. Assuming you are selecting only one file or folder, I don't really see why it wouldn't work - unless the newline character is in there and causing problems. If that's the case, you may be able to strip it with sed. Not running nemo or nautilus, so I can't test it.
I finally found the solution to yours and my problem [https://askubuntu.com/questions/243105/nautilus-scripts-nautilus-script-selected-file-paths-have-problems-with-spac][1]
The variable $NEMO_SCRIPT_SELECTED_FILE_PATH/$NAUTILUS_SCRIPT_SELECTED_FILE_PATH is a list of paths/filenames seperated by a Newline. This messes up anything that assumes its just one filename, even if it is.
#!/bin/bash
echo "$NEMO_SCRIPT_SELECTED_FILE_PATHS" | while read DIR; do
BNAME=$(basename "$DIR")
TFILE="$BNAME.torrent"
TTRACKER="http://tracker.com/announce.php"
USER="USERNAME"
transmission-create -o "/home/$USER/Desktop/$TFILE" -t $TTRACKER "$DIR"
done
Notice it seems to do an extra pass for the newline. You either need to filter that out or put an if the file/folder exists
I am a newbie to unix scripting, I want to do following and I have little clue how to proceed.
I want to log the input and output of certain set of commands, given on the terminal, to a trace file. I should be able to switch it on and off.
E.g.
switch trace on
user:echo Hello World
user:Hello World
switch trace off
Then the trace log file, e.g. trace.log, it's content should be
echo Hello World
Hello World
One thing that I can think to do is to use set -x, redirecting its output to some file, but couldn't find a way to do that. I did man set, or man -x but I found no entry. Maybe I am being too naive, but some guidance will be very helpful.
I am using bash shell.
See script(1), "make typescript of terminal session". To start a new transcript in file xyz: script xyz. To add on to an existing transcript in file xyz: script -a xyz.
There will be a few overhead lines, like Script started on ... and Script done on ... which you could use awk or sed to filter out on printout. The -t switch allows a realtime playback.
I think there might have been a recent question regarding how to display a transcript in less, and although I can't find it, this question and this one address some of the same issues of viewing a file that contains control characters. (Captured transcripts often contain ANSI control sequences and usually contain Returns as well as Linefeeds.)
Update 1 A Perl program script-declutter is available to remove special characters from script logs.
The program is about 45 lines of code found near the middle of the link. Save those lines of code in a file called script-declutter, in a subdirectory that's on your PATH (for example, $HOME/bin if that's on your search path, else (eg) /usr/local/bin) and make the file executable. After that, a command like
script-declutter typescript > out
will remove most special characters from file typescript,
while directing the result to file out.
I want to take a list of images from any web page(my task at university) using unix script. I am new at unix scripting, so all I managed to find was with command: w3m -dump http://www.myweb.com . But this command doesn't give me any html tags so I could know where the images are and to get them with grep. I also tried -dump_source, but it only gives some unknown symbols.
Can anybody help me with this?
You can download a file using this syntax:
wget -O destination-filename 'http://www.example.com'
If you want to pipe this file through another command, like grep, just specify an output filename of -:
wget -O - 'http://www.example.com'
If your goal is to download all of the images that a page uses, man wget and have a look at the -p option.