Read JSON file in git repo without checkout - ruby

I have the following method:
def getEndpointContent(url)
return JSON.parse(open(url).read)
end
I want to use this to return the contents of a json file located in a git repo without checking out the repository.
However, if I pass in, for example, https://github.com/MyRep/myFile.json for the url parameter, I get the following error:
`connect': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (OpenSSL::SSL::SSLError)
Is what I'm trying to do possible, and if so, how?

You won't be able to access the file using that URL.
GitHub provides raw file access using a different domain, and you haven't included your user or organization name. Also remember that a Git repository isn't simply a directory; you'll also have to provide a branch name or commit hash or something similar to tell GitHub which version of the file you want to see.
Something like this should work:
https://raw.githubusercontent.com/MyUser/MyRepo/master/myFile.json
You can find the raw link for a file by browsing to it in the GitHub UI and clicking the "Raw" link in the file's header.

Ruby doesn't trust the Github SSL certificate probably because it's too new for your version of Ruby and/or your OS.
Try the following from the command-line (assuming linux/OSX):
wget http://curl.haxx.se/ca/cacert.pem
Now in your Ruby code:
ENV['SSL_CERT_FILE'] = "/path/to/your/download/cacert.pem" # where you downloaded file to
require 'open-uri' # ensure this is after the above line.
def getEndpointContent(url)
return JSON.parse(open(url).read)
end

Related

compute sha-256 using chef

I have a code to compare sha-256 value from the file and artifactory which throws error if the checksum does not match. But I want to compute checksum in my own recipe rather than reading it from artifactory. Below is the code I have so far and trying to find out if there are any functions or methods which I can use in my recipe to compute sha-256 for a file.
Thanks in advance
only_if { node['abc'] }
not_if { ::File.exist?(checksum_file) and ::File.read(checksum_file).strip==coordinates['checksum'].strip }
message 'The previously deployed checksum is not aligned with the actual value'
level :debug
notifies :create, 'remote_file[download file]', :immediately
end```
chef is built on top of ruby, so we can unleash the power of ruby within recipes.
if you would like to compute the sha-256 checksome for a file, the following ruby code might be handy
require 'digest'
Digest::SHA256.file('/path/to/file').hexdigest
the snippet in your post, does not specify which chef resource are you using, although it feels like you are using the the log resource.
since you mentioned that you are interested in downloading the file from artifactory, remote_file resource can do that and it has a checksum property:
checksum: Optional, see use_conditional_get. The SHA-256 checksum of the file. Use to prevent a file from being re-downloaded. When the local file matches the checksum, Chef Infra Client does not download it.

Chef :: remote_file unable to download file from FTP

I am developing a recipe to download some file from FTP server. I am successfully able authenticate the FTP server and fetching the list of the files. But when my chef is executed to download the files I am getting error.
Here is code
fileList.each do |file|
remote_file "C:\\" + file do
source URI::encode("ftp://user:password#server/folder/" + file)
action :create_if_missing
end
end
and here is error's log
PS C:\chef-repo> chef-apply .\cookbooks\visireview-sqlserver-config\recipes\ftp_restore_db.rb
[2016-02-17T17:14:20+05:00] WARN: ftp://user:password#server/folder/file.bak was an invalid URI. Trying to escape invalid characters
[2016-02-17T17:14:20+05:00] FATAL: Stacktrace dumped to C:/chef/cache/chef-stacktrace.out
[2016-02-17T17:14:20+05:00] FATAL: Chef::Exceptions::InvalidRemoteFileURI: "ftp://user:password#server/folder/file.bak" is not a valid `source` parameter for remote_file. `source` must be an absolute URI or an array of URIs.
Please spare some of your previous time and help me to get rid of this issue.
Thanks
I think you replaced too much of the real content. Probably something in the username, password, or hostname is causing a problem with the URI parsing.
> URI.parse("ftp://user:password#server/folder/file.bak").absolute?
true
Get that working with whatever values you actually have and it should be fine.

Images don't download correctly Ruby

I'm trying to make a simple program that downloads a file (image) from the Internet and stores it on my computer using Ruby. I get it to download something but the images look really weird. I run Windows 10 and Ruby 2.2.3. This is my code:
require "open-uri"
require "openSSL"
OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE
File.open("test.jpg", "w+") do |f|
open("https://upload.wikimedia.org/wikipedia/commons/c/c9/Moon.jpg","r") do |file|
f.puts file.read
end
end
These two lines:
require "openSSL"
OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE
are to solve a problem where I get this error if I try to download a file via https:
C:/Ruby22-x64/lib/ruby/2.2.0/net/http.rb:923:in `connect': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (OpenSSL::SSL::SSLError)
Better solutions for this are very welcome!
Here is one example when I tried to download this image: https://upload.wikimedia.org/wikipedia/commons/c/c9/Moon.jpg the downloaded file looked like this: test.jpg
But it only seems to happen to images. HTML files look exactly the same. I know images can look different depending on file type but the URL ends with .jpg and when you download it via for example chrome, it is stored as a .jpg.
All suggestions are appreciated!

How to install ruby locally via rbenv/ruby-build?

I require installing ruby without internet access. As ruby-build docs suggest I can change the mirror URL via specifying the environment variable RUBY_BUILD_MIRROR_URL. I did this and although it looks at my local repo for ruby it still attempts to connect to online repo to install yaml.
env RUBY_BUILD_MIRROR_URL=http://10.10.161.39/platforms/common/ruby-2.0.0-p247.tar.gz#3e71042872c77726409460e8647a2f304083a15ae0defe90d8000a69917e20d3 /opt/rbenv/bin/rbenv install 2.0.0-p247
Downloading yaml-0.1.6.tar.gz...
-> http://10.152.161.39/platforms/proteus/common/ruby-2.0.0-p247.tar.gz#3e71042872c77726409460e8647a2f304083a15ae0defe90d8000a69917e20d3/7da6971b4bd08a986dd2a61353bc422362bd0edcc67d7ebaac68c95f74182749
-> http://pyyaml.org/download/libyaml/yaml-0.1.6.tar.gz
error: failed to download yaml-0.1.6.tar.gz
BUILD FAILED (RedHatEnterpriseServer 5.10 using ruby-build 20150928)
I tried placing the yaml-0.1.6.tar.gz file in my local repo however that makes no difference besides it will fail since the sha2 checksum provided in the URL is for ruby-2.0.0-p247.tar.gz file.
How can install ruby offline with rbenv?
Update 1
I discovered that you can modify the lookup config file to point to a local mirror instead. i.e: /opt/rbenv/plugins/ruby-build/share/ruby-build/2.0.0-p247
install_package "yaml-0.1.6" "http://10.10.161.39/platforms/common/yaml-0.1.6.tar.gz#7da6971b4bd08a986dd2a61353bc422362bd0edcc67d7ebaac68c95f74182749" --if needs_yaml
install_package "openssl-1.0.1p" "ttp://10.10.161.39/platforms/common/openssl-1.0.1p.tar.gz#bd5ee6803165c0fb60bbecbacacf244f1f90d2aa0d71353af610c29121e9b2f1" mac_openssl --if has_broken_mac_openssl
install_package "ruby-2.0.0-p247" "http://10.10.161.39/platforms/common/ruby-2.0.0-p247.tar.gz#3e71042872c77726409460e8647a2f304083a15ae0defe90d8000a69917e20d3"
Is there a better way or is this is the best way forward?
So here's how I got it to work:
Update the contents of the download file in /opt/rbenv/plugins/ruby-build/share/ruby-build/<ruby-version> to point to your local repo.
You will also notice how each file has a long hash valued after the '#' symbol in the URL. For Example:
install_package "yaml-0.1.6" "http://10.10.161.39/platforms/common/yaml-0.1.6.tar.gz#7da6971b4bd08a986dd2a61353bc422362bd0edcc67d7ebaac68c95f74182749" --if needs_yaml
This hash value is the sha256sum the file which rbenv will use to validate if it is the expected file.
So you will need to generate the value by running sha256sum <filename> and appending to each file in the URL path.
Complete example below:
install_package "yaml-0.1.6" "http://10.10.161.39/platforms/common/yaml-0.1.6.tar.gz#7da6971b4bd08a986dd2a61353bc422362bd0edcc67d7ebaac68c95f74182749" --if needs_yaml
install_package "openssl-1.0.1p" "ttp://10.10.161.39/platforms/common/openssl-1.0.1p.tar.gz#bd5ee6803165c0fb60bbecbacacf244f1f90d2aa0d71353af610c29121e9b2f1" mac_openssl --if has_broken_mac_openssl
install_package "ruby-2.0.0-p247" "http://10.10.161.39/platforms/common/ruby-2.0.0-p247.tar.gz#3e71042872c77726409460e8647a2f304083a15ae0defe90d8000a69917e20d3"
In the example above we have a dedicated repository server at http://10.10.161.39/platforms/common. If your packages are locally available, you will need to point to the local path and verify if it works.

How to download files through https in Ruby

I have a webapp where files were uploaded. You can login to the site with a valid account and then download those files. I am currently automating the whole framework using Ruby, Capybara and Selenium Webdriver, but I cannot automate the process of downloading files.
So far I tried using Selenium (which didn't work), also I used the Ruby library open-uri:
def downloadFile(path)
open('testing.docx', 'wb') do |file|
file << open(path).read
end
download = open(path)
IO.copy_stream(download, File.expand_path("resources\\downloads"))
end
Where path is the href of the link to the file, but at first I got the following error:
openssl::ssl::sslerror: ssl_connect returned=1 errno=0 state=sslv3 read server certificate b: certificate verify failed
In order to avoid it I used the following code:
OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE
But in the end, I could not download the file.
At this point I think I should load a certificate or maybe retrieve login token from cookies or else where, but I could not figure it out where exactly.
Is there a way to download files from a page which requires login?
If you use Selenium, you should download files through the browser by clicking corresponding links and buttons.
Here is described how to set up browser downloads.
https://watirwebdriver.com/browser-downloads/
Try to change the uri from "https" to "http" like this:
path = path.sub("https","http")

Resources