VIM - Passing colon-commands list via command-line - bash

Good day,
I am writing a simple script within my BASHRC file to accommodate something I couldn't quite resolve in a previous question:
Side-by-side view in Vim of svn-diff for entire directory
I basically generate a list of all files which have a "Modified" SVN status. For each of these files, I want to create a side-by-side visual diff, convert it to HTML, then append it to a running HTML file.
eg:
MODIFIED_FILES="$(svn status | grep "^M" | cut -c9-)"
for i in ${MODIFIED_FILES}; do
# Generate a side-by-side diff in vim via VIMDIFF
# Convert via ToHTML
# Append the HTML file to a file called "overall_diff.html"
done
I can accomplish the vimdiff easily enough by creating a clean copy of the file, and having a copy of the modified file.
vimdiff has an issue at first, ie:
2 files to edit
Error detected while processing /Users/Owner/.vimrc:
line 45:
E474: Invalid argument: listchars=tab:>-,trail:.,extends:>,precedes:«
Press ENTER or type command to continue
So, I am trying to get past this so I don't have to hit ENTER for each file in my list.
Next, I need to have vimdiff call the ToHTML command, and then issue the command to append the HTML buffer to a running file:
:'<,'>w! >>overall_diff.html
In short, how do I:
Get past this issue with listchars when vimdiff is called. This issue doesn't occur when I run vim, so I don't know why it occurs when I run vimdiff.
Pass a list of colon-commands to VIM to have it run them at startup without requiring a change to my .vimrc file.

In the end, I created a separate VIMRC file that gets passed to the vim command at run time, via:
`vim -d file1 fil2 -u my_special_vimrc_file`
function createVimDiff()
{
# Create some buffers
TEMP_FILE="./tmp_file"
VIM_TEMP="./temp.html"
REVISION=""
BUFFER_FILE="./overall_diff.html"
# Get a list of the files that have changed
MODIFIED_FILES="$(svn status | grep '^M' | cut -c9-)"
# Remove buffers
rm "${BUFFER_FILE}"
for i in ${MODIFIED_FILES}; do
# Remove intermediate buffers
rm "${TEMP_FILE}"
rm "${VIM_TEMP}"
# Get the current SVN rev number for the current file
REVISION="$(svn info ${i} | grep Revision)"
# Echo the name of the file to the report
echo "FILE: ${i}" >> "${BUFFER_FILE}"
# Same with the revision number
echo "${REVISION}" >> "${BUFFER_FILE}"
echo "<br>" >> "${BUFFER_FILE}"
# First print a copy of the unmodified file in a temporary buffer
svn cat "${i}" > "${TEMP_FILE}"
# Now print the unmodified file on the left column, and the
# modified file in the right column, so they appear side-by-side
vim -d "${TEMP_FILE}" "${i}" -u ~/.vimdiff_rc
# Write the side-by-side diff to a file
cat "${VIM_TEMP}" >> "${BUFFER_FILE}"
echo "<br>" >> "${BUFFER_FILE}"
done
# Cleanup temporary buffers
rm "${TEMP_FILE}"
rm "${VIM_TEMP}"
}
And the following was put into my VIMRC file:
" Convert the diff to HTML
autocmd VimEnter * silent TOhtml
" Write output to temporary buffer
autocmd VimEnter * w! ./temp.html
" Quit VIM
autocmd VimEnter * qa!

Related

Need help formatting Tshark command string from bash script

I'm attempting to run multiple parallel instances to tshark to comb through a large number of pcap files in a directory and copy the filtered contents to a new file. I'm running into an issue where Tshark is throwing an error on the command I'm feeding it.
It must have something to do with the way the command string is interpreted by tshark as I can copy / paste the formatted command string to the console and it runs just fine. I've tried formatting the command several ways and read threads from others who had similar issues. I believe I'm formatting correctly... but still get the error.
Here's what I'm working with:
Script #1: - filter
#Takes user arguments <directory> and <filter> and runs a filter on all captures for a given directory.
#
#TO DO:
#Add user prompts and data sanitization to avoid running bogus job.
#Add concatenation via mergecap /w .pcap suffix
#Delete filtered, unmerged files
#Add mtime filter for x days of logs
starttime=$(date)
if [$1 = '']; then echo "no directory specified, you must specify a directory (VLAN)"
else if [$2 = '']; then echo "no filter specified, you must specify a valid tshark filter expression"
else
echo $2 > /home/captures-user/filtered/filter-reference
find /home/captures-user/Captures/$1 -type f | xargs -P 5 -L 1 /home/captures-user/tshark-worker
rm /home/captures-user/filtered/filter-reference
fi
fi
echo Start time is $starttime
echo End time is $(date)
Script #2: - tshark-worker
# $1 = path and file name
#takes the output from the 'filter' command stored in a file and loads a local variable with it
filter=$(cat /home/captures-user/filtered/filter-reference)
#strips the directory off the current working file
file=$(sed 's/.*\///' <<< $1 )
echo $1 'is the file to run' $filter 'on.'
#runs the filter and places the filtered results in the /filtered directory
command=$"tshark -r $1 -Y '$filter' -w /home/captures-user/filtered/$file-filtered"
echo $command
$command
When I run ./filter ICE 'ip.addr == 1.1.1.1' I get the following output for each file. Note the the inclusion of == in the filter expression is not the issue, I've tried substituting 'or' and get the same output. Also, tshark is not aliased to anything, and there's no script with that name. It's the raw tshark executable in /usr/sbin.
Output:
/home/captures-user/Captures/ICE/ICE-2019-05-26_00:00:01 is the file to run ip.addr == 1.1.1.1 on.
tshark -r /home/captures-user/Captures/ICE/ICE-2019-05-26_00:00:01 -Y 'ip.addr == 1.1.1.1' -w /home/captures-user/filtered/ICE-2019-05-26_00:00:01-filtered
tshark: Display filters were specified both with "-d" and with additional command-line arguments.
As I mentioned in the comments, I think this is a problem with quoting and how your command is constructed due to spaces in the filter (and possibly in the file name and/or path).
You could try changing your tshark-worker script to something like the following:
# $1 = path and file name
#takes the output from the 'filter' command stored in a file and loads a local variable with it
filter="$(cat /home/captures-user/filtered/filter-reference)"
#strips the directory off the current working file
file="$(sed 's/.*\///' <<< $1 )"
echo $1 'is the file to run' $filter 'on.'
#runs the filter and places the filtered results in the /filtered directory
tshark -r "${1}" -Y "${filter}" -w "${file}"-filtered

Bash insert saved file name to variable

The below script download file using CURL, I'm trying inside the loop to save the file and also to insert the saved file name into a variable and then to print it.
My script downloads the script and saved the file but can't echo the saved file name:
for link in $url2; do
cd /var/script/twitter/html_files/ && file1=$({ curl -O $link ; cd -; })
echo $file1
done
Script explanation:
$url2 contains one or more URLs
curl -O write output to a file named as the remote file
Your code has several problems. Assuming $url2 is a list of valid URLs which do not require shell quoting, you can make curl print the output variable directly.
cd /var/script/twitter/html_files
for link in $url2; do
curl -s -w '%{filename_effective}\n' -O "$link"
done
Without the -w formatstring option, the output of curl does not normally contain the output file name in a conveniently machine-readable format (or actually at all). I also added an -s option to disable the download status output it prints by default.
There is no point in doing cd to the same directory over and over again inside the loop, or capturing the output into a variable which you only use once to print to standard output the string which curl by itself would otherwise print to standard output.
Finally, the cd - does not seem to do anything useful here; even if it did something useful per se, you are doing it in a subshell, which doesn't change the current working directory of the script which contains the $(cd -) command substitution.
If your task is to temporarily switch to that directory, then switch back to where you started, just cd once. You can use cd - in Bash but a slightly more robust and portable solution is to run the fetch in a subshell.
( cd directory;
do things ...
)
# you are now back to where you were before the cd
If you genuinely need the variable, you can trivially use
for link in $url2; do
file1=$(curl -s -w '%{filename_effective}' -O "$link")
echo "$file1"
done
but obviously the variable will only contain the result from the last iteration after the loop (in the code after done). (The format string doesn't need the final \n here because the command substitution will trim off any trailing newline anyway.)

no such file or directory error when using variables (works otherwise)

I am new to programming and just starting in bash.
I'm trying to print a list of directories and files to a txt file, and remove some of the path that gets printed to make it cleaner.
It works with this:
TODAY=$(date +"%Y-%m-%d")
cd
cd Downloads
ls -R ~/Music/iTunes/iTunes\ Media/Music | sed 's/\/Users\/BilPaLo\/Music\/iTunes\/iTunes\ Media\/Music\///g' > music-list-$TODAY.txt
But to clean it up I want to use variables like so,
# Creates a string of the date, format YYYY-MM-DD
TODAY="$(date +"%Y-%m-%d")"
# Where my music folders are
MUSIC="$HOME/Music/iTunes/iTunes\ Media/Music/"
# Where I want it to go
DESTINATION="$HOME/Downloads/music-list-"$TODAY".txt"
# Path name to be removed from text file
REMOVED="\/Users\/BilPaLo\/Music\/iTunes\/iTunes\ Media\/Music\/"
ls -R "$MUSIC" > "$DESTINATION"
sed "s/$REMOVED//g" > "$DESTINATION"
but it gives me a 'no such file or directory' error that I can't seem to get around.
I'm sure there are many other problems with this code but this one I don't understand.
Thank you everyone! I followed the much needed formatting advice and #amo-ej1's answer and now this works:
# Creates a string of the date format YYYY-MM-DD
today="$(date +"%Y-%m-%d")"
# Where my music folders are
music="$HOME/Music/iTunes/iTunes Media/Music/"
# Where I want it to go
destination="$HOME/Downloads/music-list-$today.txt"
# Temporary file
temp="$HOME/Downloads/temp.txt"
# Path name to be removed of text file to only leave artist name and album
remove="\\/Users\\/BilPaLo\\/Music\\/iTunes\\/iTunes\\ Media\\/Music\\/"
# lists all children of music and writes it in temp
ls -R "$music" > "$temp"
# substitutes remove by nothing and writes it in destination
sed "s/$remove//g" "$temp" > "$destination"
rm $temp #deletes temp
First when debugging bash it can be helpful to start bash with the -x flags (bash -x script.sh) or within the script enter set -x, that way bash will print out the commands it is executing (with the variable expansions) and you can more easily spot errors that way.
In this specific snippet our ls output is being redirected to a file called $DESTINATION and and sed will read from standard input and write also to $DESTINATION. So however you wanted to replace the pipe in your oneliner is wrong. As a result this will look as if your program is blocked but sed will simply wait for input arriving on standard input.
As for the 'no such file or directory', try executing with set -x and doublecheck the paths it is trying to access.

Investigating a diff error in a bash script when variables are used instead of hardcoded file names

I have a script that looks for files of specific type in a specified directory and if they are present, generates a file with the basenames before creating a tar.gz. Once compressed, I check to ensure the tarball contains all the files by running a diff check.
I have created a pair of variables that are the pre-compressed file list and those found in the tarball. When I run an if statement including diff of the variables, I receive this error:
diff: missing operand after `/my/original/dir/filelist.txt'
diff: Try `diff --help' for more information.
I worked around this by referencing the files themselves rather than the created variables. If I run the if statement in a separate bash script, it works just fine using the variables so I am entirely lost as to what my error is in my larger script. Below I provide both the snippet from the large script and the diff statement as its own script for reference.
The if diff in its own script:
#!/bin/sh
filelist=(filelist.txt)
tarfiles=(tarfiles.txt)
#differences=$(diff filelist.txt tarfiles.txt) #Uncomment if below fails
differences=$(diff $filelist $tarfiles)
if $differences > /dev/null ; then
echo Same
else
echo Different
fi
The above works just fine.
Now including this at the end my larger script:
TARFILES=$(tar -tzf "$ARCHIVES/tarredfiles.tar.gz" | awk -F/ '{ if($NF != "") print $NF }' > $LOGS/tarfiles.txt)
FILELIST=($LOGS/filelist.txt)
#Check to see if it all worked
DIFF=$(diff $FILELIST $TARFILES)
cd $LOGS #I shouldn't need to do this but I do as a safety mechanism
#if diff filelist.txt tarfiles.txt > /dev/null ; then
if diff $FILELIST $TARFILES > /dev/null ; then
echo "Today's files have been archived and checked."
else
echo "Some or none of today's files have been archived, check the logs to find the error."
echo (diff $TARFILES $FILELIST) > $LOGS/$(date '+%Y%m%d')errors.txt
fi
I have tried enclosing the variables in "" and it didn't seem to make a difference.
The way you populate TARFILES results in it being empty. What is it that you're trying to store in the variable?
This line
TARFILES=$(tar -tzf "$ARCHIVES/tarredfiles.tar.gz" | awk -F/ '{ if($NF != "") print $NF }' > $LOGS/tarfiles.txt)
Does the following steps
Extracts a list of the filenames (-t) from the compressed (-z) tar file (-f) named tarredfiles.tar.gz in the directory referred to by the $ARCHIVES variables
Sends (pipes) that list of filenames into awk where you print the last component of the filename, that is the last field ($NF) of each line when it is split by / (-F/)
Sends (redirects) all of that output into the log file $LOGS/tarfiles.txt
Captures any other output (of which there will be none!) and stores it in the TARFILES variable.
So, the variable TARFILES is always empty, but the file tarfiles.txt has content in it.
It seems that you want the diff to compare the content of tarfiles.txt with the content of filelist.txt, but you're trying to use your variables in a way that isn't really compatible with that.
An expression of the form:
TARFILES=$( command goes here )
captures the output of that command.
And
TARFILES=$( command goes here > some-file.txt )
sends the output of the command into the file, and then captures nothing.
What you want is something like:
TARFILES=some-file.txt
command goes here > $TARFILES
which will set the variable to be the name of your file, and then run a command which put content into that file.
So, specifically:
TARFILES=$LOGS/tarfiles.txt
tar -tzf "$ARCHIVES/tarredfiles.tar.gz" | awk -F/ '{ if($NF != "") print $NF }' > $TARFILES
When working will shell scripts, it is very common to be running commands that produce output that goes into files, etc. One thing you need to be really clear about in the logic of your script is when you want your variables to contain actual content (that is, the output of a command), and when you want them to contain filenames.
In your case you want to run diff on 2 files ("tarfiles" and "filelist") that happen to contain a list of filenames, so that means there's a little bit more to keep track of, but essentially you want to populate "tarfiles" with the output from a command, and then run a diff where you pass in the 2 files names "tarfiles" and "filelist". So you never want to use $( ... ) to populate tarfiles.txt because that is how you capture the output of a command into a variable, and what you're trying to do is store a filename in your variable.

How to redirect and replace the input file with the output (don't erase myfile when doing "cat myfile > myfile")

What is the right way to redirect to a file used as input and replace the file instead of erasing the file?
I want to do something like:
cat myfile > myfile
where the program (i.e., cat) reads the file and then redirects to the file, so in the example above myfile would be unchanged.
I know that the example above will erase the file, as explained here.
What is the proper syntax so the file will be replaced instead of erased? Is stdout and redirection possible, or should the program handle opening and writing the new data to the file? I would like to let my program send output to stdout and then the user can redirect or pipe or whatever.
BTW, I don't want to >> (concatenate). I want the function on the left to write "new" data to the file -- to replace the file contents.
Perhaps a better way to state it is that I want the left side of the redirection to fully occur before the streaming occurs -- is this possible? Do I have a fundamental misunderstanding of bash?
You have to either read all the data to memory first, or write to a temporary file and swap it out.
To read it into memory (like vim, ed and other editors do):
contents=$(<myfile)
cat <<< "$contents" > myfile
To create a temporary file (like sed -i, rsync and other updating tools do):
tmpfile=$(mktemp fooXXXXXX)
cat myfile > "$tmpfile" && mv "$tmpfile" myfile
Presumably you're interested in doing something more complex than cat myfile.
Some commands have command-line arguments that let you do this (typically -i) -- but those options work by writing to a temporary file and then renaming the temporary.
You can do the same thing yourself:
cat myfile > myfile.$$ && mv myfile.$$ myfile
I often use $$, which expands to my shell's process ID, as a suffix for a temporary file; it makes it fairly unlikely that the name will collide with some other file.
Using && means that it won't clobber your input file unless the cat command succeeds.
One possible problem with this approach is that your new file myfile.$$ is a newly created file, so it won't keep the permissions of the original input file. If you have tne GNU Coreutils version of the chmod command, you can avoid that problem:
cat myfile > myfile.$$ && \
chmod --reference=myfile myfile.$$ && \
mv myfile.$$ myfile
I'm taking your question literally here. By introducing a pipe to read the output of cat, the second pipe process reads the output from stdin and redirects it to the original file name, resulting in an unchanged file:
cat myfile | cat - > myfile
DANGER: This is a race condition. There is no guarantee that the first process can read the entire comments of myfile before the second process truncates it with the output redirection.

Resources