I'm trying to echo the contents of this link and it exhibits what to me is bizarre behavior.
git#gud:/home/git$ URL="https://raw.githubusercontent.com/fivethirtyeight/data/master/births/US_births_1994-2003_CDC_NCHS.csv"
git#gud:/home/git$ content=$(wget $URL -q -O -)
git#gud:/home/git$ echo $content
2003,12,31,3,12374_month,day_of_week,births
I expected this code to print the contents as I see them when I open the link on a browser. But instead, the output, on its entirety, is 2003,12,31,3,12374_month,day_of_week,births, that's it.
I actually see this behaviour locally as well, after downloading the file. Tried it both using curl and simply copy and pasting into a text editor and saving the file. They all exhibit the same behavior. The same happens with cat, cut, head, tail and even awk.
This doesn't happen with other files and works fine on Python. What am I missing? How do I get it to work?
I realize that the file doesn't end with a new line character, but adding it doesn't fix it.
I'm on Ubuntu 18.04.1 LTS and the CLI I'm using is Bash release 4.4.19(1).
The data file uses Mac-style end-of-line markers (carriage return only). When you echo the content, or just cat the file, the lines are all printing over eachother. If you were to view the file with less or vim, you would see the complete content.
Try this:
$ URL="https://raw.githubusercontent.com/fivethirtyeight/data/master/births/US_births_1994-2003_CDC_NCHS.csv"
$ curl -o data.csv "$URL"
The wc command thinks that the file has zero lines:
$ wc -l data.csv
0 data.csv
Now let's translate those end-of-line markers:
$ tr '\r' '\n' < data.csv > data-modified.csv
wc now sees a more reasonable number of lines:
$ wc -l data-modified.csv
3652 data-modified.csv
And if we were to cat the file:
$ cat data-modified.csv
.
.
.
2003,12,28,7,7645
2003,12,29,1,12823
2003,12,30,2,14438
2003,12,31,3,12374
I am trying to use Unix grep in my ruby code which I am able to run my shell terminal but it is not working when I am calling it through my ruby code. Can someone help me finding the problem?
I am trying to read the lines that match the pattern that starts with /platform/app_name and ends with username or service_name or hostname or port in config.txt file and write them into sub_config.txt file. Below is the piece of code that is blocking me right now.
exec ("cd #{$USER_HOME}; grep -E \'(^/platform/app_name/.*/username) | (^/platform/app_name/.*/port) | (^/platform/app_name/.*/service_name) | (^/platform/app_name/.*/hostname) \' config.txt > .sub_config.txt")
I am trying to remove the last newline added to the file using bash script.
I have got this -
truncate -s $(($(stat -c '%s' foo.txt)-1)) foo.txt
here foo.txt the file name.
but I want to parametrize the file name, I will pas the file name to the script and it should this remove the newline at last from that file.
Request your help on this. I do not have linux in my machine and tried using cygwin but it is giving error while running the script.
Thanks
To remove last line if it is newline use this sedL
sed -i.bak '/^[[:blank:]]*$/{$d;}' foo.txt
I'm working on a text file that contains normal text with LaTeX-style comments (lines starting with a %). To determine the non-comment word count of the file, I was running this command in Bash:
grep -v "^%" filename | wc -w
which returns about the number of words I would expect. However, if from within vim I run this command:
:r! grep -v "^%" filename | wc -w
It outputs the word count which includes the comments, but I cannot figure out why.
For example, with this file:
%This is a comment.
This is not a comment.
Running the command from outside vim returns 5, but opening the file in vim and running the similar command prints 9.
I also was having issues getting vim to prepend a "%" to the command's output, but if the output is wrong anyways, that issue becomes irrelevant.
The % character is special in vi. It gets substituted for the filename of the current file.
Try this:
:r! grep -v "^\%" filename | wc -w
Same as before but backslash-escaping the %. In my testing just now, your example :r! command printed 9 as it did for you, and the above printed 5.
I have earlier used Python for doing pretty output of JSON data like this:
python -mjson.tool input.json
I wanted to get similar output using Ruby. I am doing it like this:
ruby -rrubygems -e 'require "json"; ARGV.each { |f| print JSON.pretty_generate(JSON.load(File.open(f))) }' input.json
This is a lot for a small shell command. Can you suggest a better way?
You can shorten your script:
# ruby 1.9.2
ruby -rjson -e 'ARGF.each(nil) {|f| puts JSON.pretty_generate(JSON.parse(f)) }' file1 file2
# ruby 1.8.7
ruby -rubygems -e 'require "json"; ARGF.each(nil) {|f| puts JSON.pretty_generate(JSON.parse(f)) }' file1 file2
ARGF is a stream designed for use in scripts that process files given
as command-line arguments or passed in via STDIN.
I pass nil to ARGF#each method to split ARGF by files, not by lines (default behavior of #each).
There is a gem colorful_json which does exactly that.
You can also use the awesome_print gem.
Install it with: gem install awesome_print
Create a Bash function which will act as a shortcut
function jcurl { curl -s $* | ruby -rawesome_print -rjson -e 'ap JSON.parse(STDIN.read)'; }; export -f jcurl
Use your function to get pretty JSON in your Linux terminal:
jcurl http://127.0.0.1:3000/persons/1.json