I have a pipeline I use to preview csv files:
cat file_name.csv | sed -e 's/,,/, ,/g' | column -t -s ","| less -s
But i want to create an alias viewcsv that will allow to just replace the filename.
I tried viewcsv="cat $1 | sed -e 's/,,/, ,/g' | column -t -s ","| less -s" but that didn't work. Googling turned up that I need to convert this pipeline to a function? How can i convert this to a function so that viewcsv file_name.csv will return same output as cat file_name.csv | sed -e 's/,,/, ,/g' | column -t -s ","| less -s does?
Function syntax looks like this:
viewcsv() {
sed -e 's/,,/, ,/g' "$1" | column -t -s ","| less -s
}
Notice that I have replaced cat "$1" | sed with sed "$1".
csvkit has a CSV previewer, by the way:
$ csvlook <<< $'a,b,c\n10,20,30'
| a | b | c |
| -- | -- | -- |
| 10 | 20 | 30 |
Related
I need to store the output of this command into a variable:
sh "curl -s 'http://nexus-cicd.stgcloud.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.xml' | grep '<version>.*</version>' | sort --version-sort | uniq | tail -n1 | sed -e 's#\\(.*\\)\\(<version>\\)\\(.*\\)\\(</version>\\)\\(.*\\)#\\3#g'"
I tried the following, but echo output null
parentLast = sh ("curl -s 'http://nexus-cicd.stgcloud.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.xml' | grep '<version>.*</version>' | sort --version-sort | uniq | tail -n1 | sed -e 's#\\(.*\\)\\(<version>\\)\\(.*\\)\\(</version>\\)\\(.*\\)#\\3#g'")
echo "$parentLast"
I am trying to write a simple shell script that takes a text file as input and checks all non-punctuated words against a dictionary (english.txt). It should return all non-matching (misspelled) words. I am using grep but it does not seem to successfully match all the lines in english.txt. I have included my code below.
#!/bin/bash
cat $1 |
tr ' \t' '\n\n' |
sed -e "/'/d" |
tr -d '[:punct:]' |
tr -cd '[:alpha:]\n' |
sed -e "/^$/d" |
grep -v -i -w -f english.txt
cat joined.txt | xargs -t -a <(cut --fields=1 | sort -u | grep -E '\S') -I{} --max-args=1 --max-procs=4 echo "mkdir -p imdb/movies/{}; grep '^{}' joined.txt > imdb/movies/{}/movies.txt" | bash
The code above works but substituting the redundant cat at the start of the code with a redirection like below doesn't work and leads to a cut input output error.
< joined.txt xargs -t -a <(cut --fields=1 | sort -u | grep -E '\S') -I{} --max-args=1 --max-procs=4 echo "mkdir -p imdb/movies/{}; grep '^{}' joined.txt > imdb/movies/{}/movies.txt" | bash
In either case, it is the cut command inside the process substitution (and not xargs) that should be reading from joined.txt, so to be completely safe, you should put either the pipe or the input redirection inside the the process substitution. Actually, neither is necessary; cut can just take joined.txt as an argument.
xargs -t -a <( cat joined.txt | cut ... ) ... | bash
or
xargs -t -a <( cut -f1 joined.txt | ... ) ... | bash
However, it would be clearest to skip the process substitution altogether, and pipe the output of that pipeline to xargs:
cut -f joined.txt | sort -u | grep -E '\S' | xargs -t ...
This short script scrapes some log files daily to create a simple extract. It works from the command line and when I echo the $cmd and copy/paste, it also works. But it will breaks when I try to execute from the script itself.
I know this is a nightmare of patterns that I could probably improve, but am I missing something simple to just execute this correctly?
#!/bin/bash
priorday=$(date --date yesterday +"%Y-%m-%d")
outputfile="/home/CCHCS/da14/$priorday""_PROD_message_processing_times.txt"
cmd="grep 'Processed inbound' /home/rules/care/logs/RootLog* | cut -f5,6,12,16,18 -d\" \" | grep '^"$priorday"' | sed 's/\,/\./' | sed 's/ /\t/g' | sed -r 's/([0-9]+\-[0-9]+\-[0-9]+)\t/\1 /' | sed 's/ / /g' | sort >$outputfile"
printf "command to execute:\n"
echo $cmd
printf "\n"
$cmd
ouput:
./make_log_extract.sh command to execute: grep 'Processed inbound' /home/rules/care/logs/RootLog.log /home/rules/care/logs/RootLog.log.1
/home/rules/care/logs/RootLog.log.10
/home/rules/care/logs/RootLog.log.11
/home/rules/care/logs/RootLog.log.12
/home/rules/care/logs/RootLog.log.2
/home/rules/care/logs/RootLog.log.3
/home/rules/care/logs/RootLog.log.4
/home/rules/care/logs/RootLog.log.5
/home/rules/care/logs/RootLog.log.6
/home/rules/care/logs/RootLog.log.7
/home/rules/care/logs/RootLog.log.8
/home/rules/care/logs/RootLog.log.9 | cut -f5,6,12,16,18 -d" " | grep
'^2014-01-30' | sed 's/\,/./' | sed 's/ /\t/g' | sed -r
's/([0-9]+-[0-9]+-[0-9]+)\t/\1 /' | sed 's/ / /g' | sort
/home/CCHCS/da14/2014-01-30_PROD_message_processing_times.txt
grep: 5,6,12,16,18: No such file or directory
As grebneke comments, do not store the command and then execute it.
What you can do is to execute it but firstly print it: Bash: Print each command before executing?
priorday=$(date --date yesterday +"%Y-%m-%d")
outputfile="/home/CCHCS/da14/$priorday""_PROD_message_processing_times.txt"
set -o xtrace # <-- set printing mode "on"
grep 'Processed inbound' /home/rules/care/logs/RootLog* | cut -f5,6,12,16,18 -d\" \" | grep '^"$priorday"' | sed 's/\,/\./' | sed 's/ /\t/g' | sed -r 's/([0-9]+\-[0-9]+\-[0-9]+)\t/\1 /' | sed 's/ / /g' | sort >$outputfile"
set +o xtrace # <-- revert to normal
I have text coming from some command (in example it's echo -e "10 ABC \n5 DEF \n87 GHI"). This text goes through the pipeline and I get wanted output (in example it's GHI). Wanted output is sent to the following pipeline step (in example it's | xargs -I {} grep -w {} FILES |).
My question is:
I want to append a variable to an "inter pipe" output before it's sent to a following step - How can I do this?
Example:
echo -e "10 ABC \n5 DEF \n87 GHI" |
sort -nr -k1 |
head -n1 |
cut -f 2 | # Wanted output comes here. I want to append it to a variable before it goes to `grep`
xargs -I {} grep -w {} FILES |
# FOLLOWING ANALYSIS
You can't set a shell variable in the middle of the pipeline, but you can send the output to a file using the tee command, and then read that file later.
echo -e "10 ABC \n5 DEF \n87 GHI" |
sort -nr -k1 |
head -n1 |
cut -f 2 |
tee intermediate.txt |
xargs -I {} grep -w {} FILES |
# FOLLOWING ANALYSIS
# intermediate.txt now contains 87 GHI
How about something like this
echo -e "10 ABC \n5 DEF \n87 GHI" | sort -nr -k1 | head -n1 | cut -f 2 | while read MYVAR; do echo "intermediate value: $MYVAR"; echo $MYVAR | xargs -I {} grep -w {} FILES; done
Insert it to a stream. so I think your looking just to add the ?contents? of a variable to every 'line' from the stream? This prepends the contents of $example
ie
example="A String"
echo -e "10 ABC \n5 DEF \n87 GHI" |
sort -nr -k1 |
head -n1 |
cut -f 2 |
sed s/^/$example/ |
xargs -I {} grep -w {} FILES |
# FOLLOWING ANALYSIS
sed s/$/$example/ to append
NB I tend to do lot of things this way in bash, but a long pipeline of cuts, seds and heads etc does suggest maybe its time to break out awk or perl.