I am writing a bash script and trying to change lines in my file. I currently have:
if [[ ! $line == *[+]* ]]
then
[command to change line]
I merely want to change the line by adding on to the text already there. Any suggestions? I have tried:
sed -i 'Ns/.*/replacement-line/' file.txt
and
sed -i '/Text_to_be_replaced/c\This is the new line.' file.txt
among some others found online to no avail.
My full script is:
#!bin/bash
filename="227.dat"
while ((i++)); read -r line; do
sed -i 's/(/ /g' $filename
sed -i 's/)//g' $filename
sed -i 's/,/ /g' $filename
sed -i 's/-x/-1 0 0/g' $filename
sed -i 's/x/ 1 0 0/g' $filename
sed -i 's/-y/ 0 -1 0/g' $filename
sed -i 's/y/ 0 1 0/g' $filename
sed -i 's/-z/ 0 0 -1/g' $filename
sed -i 's/z/ 0 0 1/g' $filename
[*command to add to line*]
done < "$filename"
You can put the test in the sed command:
sed -i '/+/! s/.*/replacement-line/' file.txt
The ! means to do the replacement only on lines that don't match the regular expression.
Related
For the following MVCE:
echo "test_num: 0" > test.txt
test_num=$(grep 'test_num:' test.txt | cut -d ':' -f 2)
new_test_num=$((test_num + 1))
echo $test_num
echo $new_test_num
sed -i "s/test_num: $test_num/test_num: $new_test_num/g" test.txt
cat test.txt
echo "sed -i "s/test_num: $test_num/test_num: $new_test_num/g" test.txt"
sed -i "s/test_num: 0/test_num: 1/g" test.txt
cat test.txt
Which outputs
0 # parsed original number correctly
1 # increment the number
test_num: 0 # sed with interpolated variable, does not work
sed -i s/test_num: 0/test_num: 1/g test.txt # interpolated parameter looks right
test_num: 1 # ???
Why does sed -i "s/test_num: $test_num/test_num: $new_test_num/g" test.txt not produce the expected result when sed -i "s/test_num: 0/test_num: 1/g" test.txt works just fine in the above example?
As mentioned in the comment, there is a white space in ${test_num}. Therefore in your sed there should not be an empty space between the colon and your variable.
Also I guess you should surround your variable with curly bracket {} to increase readability.
sed "s/test_num:${test_num}/test_num: ${new_test_num}/g" test.txt
test_num: 1
If you just want the number in ${test_num}, you can try something like:
grep 'test_num:' test.txt | awk -F ': ' '{print $2}'
awk allows to specify delimiter with more than 1 character.
Instead of grep|cut you can also use sed.
#! /bin/bash
exec <<EOF
test_num: 0
EOF
grep 'test_num:' | cut -d ':' -f 2
exec <<EOF
test_num: 0
EOF
sed -n 's/^test_num: //p'
When using regexp replace in sed there is special meaning to $ .
Suggesting to rebuild your sed command segments as follow:
sed -i 's/test_num: '$test_num'/test_num: '$new_test_num'/g' test.txt
Other option, use echo command to expand variables in sed command.
sed_command=$(echo "s/test_num:${test_num}/test_num: ${new_test_num}/g")
sed -i "$sed_command" test.txt
I want to output a specific line from bash argument using sed, I have tried many ways, but none work:
#!/bin/bash
sed -n '$2p' $1
sed -n '${2}p' $1
sed -n "$2p" $1
sed -n "${2}p" $1
sed -n ''"$2"'p' $1
How on earth do I get the correct result?
Try
sed -n "$2p" $1
Demo:
$seq 10 > file.txt
$cat temp.ksh
#!/bin/bash
set -x
sed -n "$2p" $1
$./temp.ksh file.txt 3
+ sed -n 3p file.txt
3
$
I have the following lines in input.txt:
client_citic_plat_fix44;CITICHK;interbridge_ulnet_se_eqx
client_citic_plat_fix44;CITICHK;interbridge_ulnet_se_eqx;CITICHK;interbridge_hk_eqx
client_dkp_crd;DELIVERTOCOMPID;DESTINATION
client_dkp_crd;NORD;interbridge_fr
client_dkp_crd;NORD;interbridge_fr;broker_nordea_2
client_dkp_crd;AVIA;interbridge_fr
client_dkp_crd;AVIA;interbridge_fr;interbridge_ld
client_dkp_crd;SEBAP;interbridge_fr
client_dkp_crd;SEBAP;interbridge_fr;broker_seb_ss_thl
client_epf_crd;DELIVERTOCOMPID;DESTINATION
I need some bash (awk/sed) script to remove the lines that are partially similar to others. Desired output should be:
client_citic_plat_fix44;CITICHK;interbridge_ulnet_se_eqx;CITICHK;interbridge_hk_eqx
client_dkp_crd;DELIVERTOCOMPID;DESTINATION
client_dkp_crd;NORD;interbridge_fr;broker_nordea_2
client_dkp_crd;AVIA;interbridge_fr;interbridge_ld
client_dkp_crd;SEBAP;interbridge_fr;broker_seb_ss_thl
client_epf_crd;DELIVERTOCOMPID;DESTINATION
Columns 1, 2 and 3 are always similar and I always want to remove the shortest line between the two compared.
Thanks!
Here's a solution using grep and sed:
#!/bin/bash
file="filepath"
while IFS= read -r line;do
(($(grep $line "$file" -c)>1)) && sed -i "/^$line$/d" "$file"
done <"$file"
Note: This will replace your file.
To not replace your file and to put the output to another file, you can do this:
#!/bin/bash
infile="infilepath"
outfile="outfilepath"
cp "$infile" "$outfile"
while IFS= read -r line;do
(($(grep $line "$infile" -c)>1)) && sed -i "/^$line$/d" "$outfile"
done <"$infile"
I am attempting to grep all of the /home directory for certain words. If these words occur it writes a /var/logfile and then from there I would like this to be read line by line and then echo each line with information. This is what I have so far.
grep -r -i "word\|word\|word\|word\|word" /home >>/var/testlog
UNAME=`grep -r -i "word\|word\|word\|word\|word" /var/testlog | cut -f 3 -d "/"`
BLINE=`grep -r -i "word\|word\|word\|word\|word" /var/testlog | cut -f 2 -d ":"`
FILEP=`grep -r -i "word\|word\|word\|word\|word" /var/testlog | cut -f 1 -d ":"`
while [ "$UNAME" == "true" ] && [ "$BLINE" == "true" ] && [ "$FILEP" == "true" ];
do
echo "User is: $UNAME, The line flag with the word is: $BLINE, and the file path for the text is: $FILEP."
done
One solution is:
grep -r -i "word\|word\|word\|word\|word" /home >>/var/testlog
while IFS= read -r line
do
[[ $line =~ (/home)/([^/]+)/([^:]*):(.*) ]] || echo Failed on line=$line
echo "User is: ${BASH_REMATCH[2]}, The line flag with the word is: ${BASH_REMATCH[4]}, and the file path for the text is: ${BASH_REMATCH[1]/${BASH_REMATCH[2]}}/${BASH_REMATCH[3]}."
done <testlog
If the use of sed is allowed, then the while loop is unnecessary:
grep -r -i "word\|word\|word\|word\|word" /home >>/var/testlog
sed -r 's|(/home/(\w+)/[^:]*):(.*)|User is: \2, The line flag with the word is: \3, and the file path for the text is: \1.|' testlog
I'm trying to write a bash script that will download all of the youtube videos from a playlist and save them to a specific file name based on the title of the youtube video itself. So far I have two separate pieces of code that do what I want but I don't know how to combine them together to function as a unit.
This piece of code finds the titles of all of the youtube videos on a given page:
curl -s "$1" | grep '<span class="title video-title "' | cut -d\> -f2 | cut -d\< -f1
And this piece of code downloads the files to a filename given by the youtube video id (e.g. the filename given by youtube.com/watch?v=CsBVaJelurE&feature=relmfu would be CsBVaJelurE.flv)
curl -s "$1" | grep "watch?" | cut -d\" -f4| while read video;
do youtube-dl "http://www.youtube.com$video";
done
I want a script that will output the youtube .flv file to a filename given by the title of the video (in this case BASH lesson 2.flv) rather than simply the video id name. Thanks in advance for all the help.
OK so after further research and updating my version of youtube-dl, it turns out that this functionality is now built directly into the program, negating the need for a shell script to solve the playlist download issue on youtube. The full documentation can be found here: (http://rg3.github.com/youtube-dl/documentation.html) but the simple solution to my original question is as follows:
1) youtube-dl will process a playlist link automatically, there is no need to individually feed it the URLs of the videos that are contained therein (this negates the need to use grep to search for "watch?" to find the unique video id
2) there is now an option included to format the filename with a variety of options including:
id: The sequence will be replaced by the video identifier.
url: The sequence will be replaced by the video URL.
uploader: The sequence will be replaced by the nickname of the person who uploaded the video.
upload_date: The sequence will be replaced by the upload date in YYYYMMDD format.
title: The sequence will be replaced by the literal video title.
ext: The sequence will be replaced by the appropriate extension (like
flv or mp4).
epoch: The sequence will be replaced by the Unix epoch when creating
the file.
autonumber: The sequence will be replaced by a five-digit number that
will be increased with each download, starting at zero.
the syntax for this output option is as follows (where NAME is any of the options shown above):
youtube-dl -o '%(NAME)s' http://www.youtube.com/your_video_or_playlist_url
As an example, to answer my original question, the syntax is as follows:
youtube-dl -o '%(title)s.%(ext)s' http://www.youtube.com/playlist?list=PL2284887FAE36E6D8&feature=plcp
Thanks again to those who responded to my question, your help is greatly appreciated.
If you want to use the title from youtube page as a filename, you could use -t option of youtube-dl. If you want to use the title from your "video list" page and you sure that there is exactly one watch? URL for every <span class="title video-title" title, then you can use something like this:
#!/bin/bash
TMPFILE=/tmp/downloader-$$
onexit() {
rm -f $TMPFILE
}
trap onexit EXIT
curl -s "$1" -o $TMPFILE
i=0
grep '<span class="title video-title "' $TMPFILE | cut -d\> -f2 | cut -d\< -f1 | while read title; do
titles[$i]=$title
((i++))
done
i=0
grep "watch?" $TMPFILE | cut -d\" -f4 | while read url; do
urls[$i]="http://www.youtube.com$url"
((i++))
done
i=0; while (( i < ${#urls[#]} )); do
youtube-dl -o "${titles[$i]}.%(ext)" "${urls[$i]}"
((i++))
done
I did not tested it because I have no "video list" page example.
this following method work and play you titanic from youtube
youtube-downloader.sh
youtube-video-url.sh
#!/bin/bash
decode() {
to_decode='s:%([0-9A-Fa-f][0-9A-Fa-f]):\\x\1:g'
printf "%b" `echo $1 | sed 's:&:\n:g' | grep "^$2" | cut -f2 -d'=' | sed -r $to_decode`
}
data=`wget http://www.youtube.com/get_video_info?video_id=$1\&hl=pt_BR -q -O-`
url_encoded_fmt_stream_map=`decode $data 'url_encoded_fmt_stream_map' | cut -f1 -d','`
signature=`decode $url_encoded_fmt_stream_map 'sig'`
url=`decode $url_encoded_fmt_stream_map 'url'`
test $2 && name=$2 || name=`decode $data 'title' | sed 's:+: :g;s:/:-:g'`
test "$name" = "-" && name=/dev/stdout || name="$name.vid"
wget "${url}&signature=${signature}" -O "$name"
#!/usr/bin/env /bin/bash
function youtube-video-url {
local field=
local data=
local split="s:&:\n:g"
local decode_str='s:%([0-9A-Fa-f][0-9A-Fa-f]):\\x\1:g'
local yt_url="http://www.youtube.com/get_video_info?video_id=$1"
local grabber=`command -v curl`
local args="-sL"
if [ ! "$grabber" ]; then
grabber=`command -v wget`
args="-qO-"
fi
if [ ! "$grabber" ]; then
echo 'No downloader available.' >&2
test x"${BASH_SOURCE[0]}" = x"$0" && exit 1 || return 1
fi
function decode {
data="`echo $1`"
field="$2"
if [ ! "$field" ]; then
field="$1"
data="`cat /dev/stdin`"
fi
data=`echo $data | sed $split | grep "^$field" | cut -f2 -d'=' | sed -r $decode_str`
printf "%b" $data
}
local map=`$grabber $args $yt_url | decode 'url_encoded_fmt_stream_map' | cut -f1 -d','`
echo `decode $map 'url'`\&signature=`decode $map 'sig'`
}
[ $SHLVL != 1 ] && export -f youtube-video-url
bash youtube-player.sh saalGKY7ifU
#!/bin/bash
decode() {
to_decode='s:%([0-9A-Fa-f][0-9A-Fa-f]):\\x\1:g'
printf "%b" `echo $1 | sed 's:&:\n:g' | grep "^$2" | cut -f2 -d'=' | sed -r $to_decode`
}
data=`wget http://www.youtube.com/get_video_info?video_id=$1\&hl=pt_BR -q -O-`
url_encoded_fmt_stream_map=` decode $data 'url_encoded_fmt_stream_map' | cut -f1 -d','`
signature=` decode $url_encoded_fmt_stream_map 'sig'`
url=`decode $url_encoded_fmt_stream_map 'url'`
test $2 && name=$2 || name=`decode $data 'title' | sed 's:+: :g;s:/:-:g'`
test "$name" = "-" && name=/dev/stdout || name="$name.mp4"
# // wget "${url}&signature=${signature}" -O "$name"
mplayer -zoom -fs "${url}&signature=${signature}"
It uses decode and bash, that you may have installed.
I use this bash script to download a given set of songs from a given youtube's playlist
#!/bin/bash
downloadDirectory = <directory where you want your videos to be saved>
playlistURL = <URL of the playlist>
for i in {<keyword 1>,<keyword 2>,...,<keyword n>}; do
youtube-dl -o ${downloadDirectory}"/youtube-dl/%(title)s.%(ext)s" ${playlistURL} --match-title $i
done
Note: "keyword i" is the title (in whole or part; if part, it should be unique to that playlist) of a given video in that playlist.
Edit: You can install youtube-dl by pip install youtube-dl
#!/bin/bash
# Coded by Biki Teron
# String replace command in linux
echo "Enter youtube url:"
read url1
wget -c -O index.html $url1
################################### Linux string replace ##################################################
sed -e 's/%3A%2F%2F/:\/\//g' index.html > youtube.txt
sed -i 's/%2F/\//g' youtube.txt
sed -i 's/%3F/?/g' youtube.txt
sed -i 's/%3D/=/g' youtube.txt
sed -i 's/%26/\&/g' youtube.txt
sed -i 's/%252/%2/g' youtube.txt
sed -i 's/sig/&signature/g' youtube.txt
## command to get filename
nawk '/<title>/,/<\/title>/' youtube.txt > filename.txt ## Print the line between containing <title> and <\/title> .
sed -i 's/.*content="//g' filename.txt
sed -i 's/">.*//g' filename.txt
sed -i 's/.*<title>//g' filename.txt
sed -i 's/<.*//g' filename.txt
######################################## Coding to get all itag list ########################################
nawk '/"fmt_list":/,//' youtube.txt > fmt.html ## Print the line containing "fmt_list": .
sed -i 's/.*"fmt_list"://g' fmt.html
sed -i 's/, "platform":.*//g' fmt.html
sed -i 's/, "title":.*//g' fmt.html
# String replace command in linux to get correct itag format
sed -i 's/\\\/1920x1080\\\/99\\\/0\\\/0//g' fmt.html ## Replace \/1920x1080\/99\/0\/0 by blank .
sed -i 's/\\\/1920x1080\\\/9\\\/0\\\/115//g' fmt.html ## Replace \/1920x1080\/9\/0\/115 by blank.
sed -i 's/\\\/1280x720\\\/99\\\/0\\\/0//g' fmt.html ## Replace \/1280x720\/99\/0\/0 by blank.
sed -i 's/\\\/1280x720\\\/9\\\/0\\\/115//g' fmt.html ## Replace \/1280x720\/9\/0\/115 by blank.
sed -i 's/\\\/854x480\\\/99\\\/0\\\/0//g' fmt.html ## Replace \/854x480\/99\/0\/0 by blank.
sed -i 's/\\\/854x480\\\/9\\\/0\\\/115//g' fmt.html ## Replace \/854x480\/9\/0\/115 by blank.
sed -i 's/\\\/640x360\\\/99\\\/0\\\/0//g' fmt.html ## Replace \/640x360\/99\/0\/0 by blank.
sed -i 's/\\\/640x360\\\/9\\\/0\\\/115//g' fmt.html ## Replace \/640x360\/9\/0\/115 by blank.
sed -i 's/\\\/640x360\\\/9\\\/0\\\/115//g' fmt.html ## Replace \/640x360\/9\/0\/115 by blank.
sed -i 's/\\\/320x240\\\/7\\\/0\\\/0//g' fmt.html ## Replace \/320x240\/7\/0\/0 by blank.
sed -i 's/\\\/320x240\\\/99\\\/0\\\/0//g' fmt.html ## Replace \/320x240\/99\/0\/0 by blank.
sed -i 's/\\\/176x144\\\/99\\\/0\\\/0//g' fmt.html ## Replace \/176x144\/99\/0\/0 by blank.
# Command to cut a part of a file between any two strings
nawk '/"url_encoded_fmt_stream_map":/,//' youtube.txt > url.txt
sed -i 's/.*url_encoded_fmt_stream_map"://g' url.txt
#Display video resolution information
echo ""
echo "Video resolution:"
echo "[46=1080(.webm)]--[37=1080(.mp4)]--[35=480(.flv)]--[36=180(.3gpp)]"
echo "[45=720 (.webm)]--[22=720 (.mp4)]--[34=360(.flv)]--[17=144(.3gpp)]"
echo "[44=480 (.webm)]--[18=360 (.mp4)]--[5=240 (.flv)]"
echo "[43=360 (.webm)]"
echo ""
echo "itag list= "`cat fmt.html`
echo "Enter itag number: "
read fmt
####################################### Coding to get required resolution #################################################
## cut itag=?
sed -e "s/.*,itag=$fmt//g" url.txt > "$fmt"_1.txt
sed -e 's/\u0026quality.*//g' "$fmt"_1.txt > "$fmt".txt
sed -i 's/.*u0026url=//g' "$fmt".txt ## Ignore all lines before \u0026url= but print all lines after \u0026url=.
sed -e 's/\u0026type.*//g' "$fmt".txt > "$fmt"url.txt ## Ignore all lines after \u0026type but print all lines before \u0026type.
sed -i 's/\\/\&/g' "$fmt"url.txt ## replace \ by &
sed -e 's/.*\u0026sig//g' "$fmt".txt > "$fmt"sig.txt ## Ignore all lines before \u0026sig but print all lines after \u0026sig.
sed -i 's/\\/\&ptk=machinima/g' "$fmt"sig.txt ## replace \ by &
echo `cat "$fmt"url.txt``cat "$fmt"sig.txt` > "$fmt"url.txt ## Add string at the end of a line
echo `cat "$fmt"url.txt` > link.txt ## url and signature content to 44url.txt
rm "$fmt"sig.txt
rm "$fmt"_1.txt
rm "$fmt".txt
rm "$fmt"url.txt
rm youtube.txt
########################################### Coding for filename with correct extension #####################################
if [ $fmt -eq 46 ]
then
echo `cat filename.txt`.webm > filename.txt
elif [ $fmt -eq 45 ]
then
echo `cat filename.txt`.webm > filename.txt
elif [ $fmt -eq 44 ]
then
echo `cat filename.txt`.webm > filename.txt
elif [ $fmt -eq 43 ]
then
echo `cat filename.txt`.webm > filename.txt
elif [ $fmt -eq 37 ]
then
echo `cat filename.txt`.mp4 > filename.txt
elif [ $fmt -eq 22 ]
then
echo `cat filename.txt`.mp4 > filename.txt
elif [ $fmt -eq 18 ]
then
echo `cat filename.txt`.mp4 > filename.txt
elif [ $fmt -eq 35 ]
then
echo `cat filename.txt`.flv > filename.txt
elif [ $fmt -eq 34 ]
then
echo `cat filename.txt`.flv > filename.txt
elif [ $fmt -eq 5 ]
then
echo `cat filename.txt`.flv > filename.txt
elif [ $fmt -eq 36 ]
then
echo `cat filename.txt`.3gpp > filename.txt
else
echo `cat filename.txt`.3gpp > filename.txt
fi
rm fmt.html
rm url.txt
filename=`cat filename.txt`
linkdownload=`cat link.txt`
wget -c -O "$filename" $linkdownload
echo "Download Finished!"
read