I want to get Dropbox directory like "c:\Users\foo\Dropbox" or "/Users/foo/Dropbox" by using ruby.
Is there a simple way to do it?
Newer versions of the Dropbox client write a file that contains the path(s) to the Dropbox folder(s) that you can read for this purpose. You can find more information here:
https://www.dropbox.com/help/4584
In short, you can read that info.json file and parse the contents as JSON to retrieve the Dropbox folder path(s).
From the dropbox API page:
# Install this the SDK with "gem install dropbox-sdk"
require 'dropbox_sdk'
# Get your app key and secret from the Dropbox developer website
APP_KEY = 'INSERT_APP_KEY'
APP_SECRET = 'INSERT_APP_SECRET'
flow = DropboxOAuth2FlowNoRedirect.new(APP_KEY, APP_SECRET)
authorize_url = flow.start()
# Have the user sign in and authorize this app
puts '1. Go to: ' + authorize_url
puts '2. Click "Allow" (you might have to log in first)'
puts '3. Copy the authorization code'
print 'Enter the authorization code here: '
code = gets.strip
# This will fail if the user gave us an invalid authorization code
access_token, user_id = flow.finish(code)
client = DropboxClient.new(access_token)
puts "linked account:", client.account_info().inspect
file = open('working-draft.txt')
response = client.put_file('/magnum-opus.txt', file)
puts "uploaded:", response.inspect
root_metadata = client.metadata('/')
puts "metadata:", root_metadata.inspect
At this stage, you could do worse than invoke client.get_file for every root_metadata['contents'] where root_metadata['contents']['is_dir'] is false. How best to do this, is left as an exercise for OP.
Related
My test suite has a cucumber front end with a ruby backend, running the latest version of watir-webdriver and its dependencies atop the latest version of OSX. My cucumber environment is setup to execute in Firefox.
The export feature of our app creates a zip file but to test the import feature, I need the contents of the zip file.
My actual test needs to unpack that zip file and select the individual files in it for use in testing the import feature of our web application.
Can anyone point me to a reference that can help me figure out how to write that?
Based off my experience, you download this file the same way that a normal user might. So first off, you just click the download button or whatever and then can access the file wherever it is and check out its contents.
Assuming the downloads just go to your Downloads folder by default, there is some simple code you can use to select the most recently downloaded item:
fn = Dir.glob("~/Downloads/*.zip").max { |a,b| File.ctime(a) <=> File.ctime(b)}
Then just use the unzip shell command to unzip the file. No reason to add another gem into the mix when you can just use generic shell commands.
`unzip #{fn}`
Then, you'd use Dir.glob again to get the filenames of everything inside the unzipped files folder. Assuming the file was named "thing.zip", you do this:
files = Dir.glob("~/Downloads/thing/*")
If you want to files to be downloaded directly to your project folder, you can try this. This also prevents the popup from asking you if you really want to save the file which is handy. I think this still works but haven't used it in some time. The above stuff works for sure though.
profile = Selenium::WebDriver::Firefox::Profile.new
download_dir = Dir.pwd + "/test_downloads"
profile['browser.download.dir'] = download_dir
profile['browser.helperApps.neverAsk.saveToDisk'] = "application/zip"
b = Watir::Browser.new. :firefox, :profile => profile
I ended up adding the rubyzip gem at https://github.com/rubyzip/rubyzip
the solution is on that link but i modified mine a little bit. I added the following to my common.rb file. see below:
require 'Zip'
def unpack_zip
test_home='/Users/yournamegoeshere/SRC/watir_testing/project files'
sleep(5) #<--manually making time to hit the save download dialog
zip_file_paths = []
Find.find(test_home) do |path|
zip_file_paths << path if path =~ /.*\.zip$/
end
file_name=zip_file_paths[0]
Zip::File.open(file_name) do |zip_file|
# Handle entries one by one
zip_file.each do |entry|
# Extract to file/directory/symlink
puts "Extracting #{entry.name}"
entry.extract(test_home + "/" + entry.name)
# Read into memory
content = entry.get_input_stream.read
end
# Find specific entry
entry = zip_file.glob('*.csv').first
puts entry.get_input_stream.read
end
end
This solution works great!
I am doing data scraping with Ruby and Nokogiri. Is it possible to download and parse a local file in my computer?
I have:
require 'open-uri'
url = "file:///home/nav/Desktop/Scraping/scrap1.html"
It gives error as:
No such file or directory # rb_sysopen - file:\home/nav/Desktop/Scraping/scrap1.html
If you want to parse a local file with Nokogiri you can do it like this.
file = File.read('/home/nav/Desktop/Scraping/scrap1.html')
doc = Nokogiri::HTML(file)
When you open a local file in a browser, the URL in the address bar is displayed as:
file:///Users/7stud/Desktop/accounts.txt
But that doesn't mean you use that format in a Ruby script. Your Ruby script doesn't send the file name to a browser and then ask the browser to retrieve the file. Your Ruby script searches your file system directly.
The same is true for URLs: your Ruby script doesn't ask your browser to go retrieve a page from the internet, Ruby retrieves the page itself by sending a request using your system's network interface. After all, a browser and a Ruby program are both just computer programs. What your browser can do over a network, a Ruby program can do, too.
This works for me:
require 'open-uri'
text = open('./data.txt').read
puts text
You have to get your path right, though. The only reason I can think of to use open() is if you had an array of filenames and URLs mixed together. If that isn't your situation, see new2code's answer.
This is how I do it as according to the documentation.
f = File.open("//home/nav/Desktop/Scraping/scrap1.html")
doc = Nokogiri::HTML(f)
f.close
I would make use of Mechanize and save the file locally, then parse it with Nokogiri like so:
# Save the file
agent = Mechanize.new
agent.pluggable_parser.default = Mechanize::Download
current_url = 'http://www.example.com'
file = agent.get(current_url)
file.save!("#{Rails.root}/tmp/")
# Read the file
page = Nokogiri::HTML::Reader(File.open(file))
Hope that helps!
require 'net/ftp'
require 'nokogiri'
server = "xxxxxx"
user = "xxxxx"
password = "xxxxx"
ftp = Net::FTP.new(server, user, password)
files = ftp.nlst('File*.xml')
files.each do |file|
ftp.getbinaryfile(file)
doc = Nokogiri::XML(open(file))
# some operations with doc
end
With the code above I'm able to parse/read XML file, because it first downloads a file.
But how can I parse remote XML file without downloading it?
The code above is a part of rake task that loads rails environment when run.
UPDATE:
I'm not going to create any file. I will import info into the mongodb using mongoid.
If you simply want to avoid using a temporary local file, it is possible to to fetch the file contents direct as a String, and process in memory, by supplying nil as the local file name:
files.each do |file|
xml_string = ftp.getbinaryfile( file, nil )
doc = Nokogiri::XML( xml_string )
# some operations with doc
end
This still does an FTP fetch of the contents, and XML parsing happens at the client.
It is not really possible to avoid fetching the data in some form or other, and if FTP is the only protocol you have available, then that means copying data over the network using an FTP get. However, it is possible, but far more complicated, to add capabilities to your FTP (or other net-based) server, and return the data in some other form. That could include Nokogiri parsing done remotely on the server, but you'd still need to serialise the end result, fetch it and deserialise it.
Since Heroku does not allow saving dynamic files to disk, I've run into a dilemma that I am hoping you can help me overcome. I have a text file that I can create in RAM. The problem is that I cannot find a gem or function that would allow me to stream the file to another FTP server. The Net/FTP gem I am using requires that I save the file to disk first. Any suggestions?
ftp = Net::FTP.new(domain)
ftp.passive = true
ftp.login(username, password)
ftp.chdir(path_on_server)
ftp.puttextfile(path_to_web_file)
ftp.close
The ftp.puttextfile function is what is requiring a physical file to exist.
StringIO.new provides an object that acts like an opened file. It's easy to create a method like puttextfile, by using StringIO object instead of file.
require 'net/ftp'
require 'stringio'
class Net::FTP
def puttextcontent(content, remotefile, &block)
f = StringIO.new(content)
begin
storlines("STOR " + remotefile, f, &block)
ensure
f.close
end
end
end
file_content = <<filecontent
<html>
<head><title>Hello!</title></head>
<body>Hello.</body>
</html>
filecontent
ftp = Net::FTP.new(domain)
ftp.passive = true
ftp.login(username, password)
ftp.chdir(path_on_server)
ftp.puttextcontent(file_content, path_to_web_file)
ftp.close
David at Heroku gave a prompt response to a support ticket I entered there.
You can use APP_ROOT/tmp for temporary file output. The existence of files created in this dir is not guaranteed outside the life of a single request, but it should work for your purposes.
Hope this helps,
David
We have an automatic build system that spits out packages, regression-tested & wrapped into a neat installer, ready for end-users to d/l & deploy.
We do tracking of end user support requests/bug reports via redmine. So far we uploaded the packages manually to the resp. 'Files' section of the redmine project, via the web interface.
What I'd like to do is to automate this step.
I imagine this would requires a few lines of Ruby to interface with redmine's db. I have zero knowledge about redmine's internals. :)
Basically I want the equivalent of a
mv package-x.y.z.tbz /usr/local/redmine/files/
as a Ruby (or whatever language suits the need) script that creates the right filename and registers the file in redmine's db so it shows up as if it had been uploaded through the Web interface, manually.
Cheers!
I've been frustrated with Redmine about things like this before. But before I go much further: is there a specific reason why you're using the Files section for this? It seems another tool (such as SSH/SFTP for uploading to someplace accessible to HTTP) might be a better fit for your needs. It would also be easily scriptable. Just point people to some constant URL like http://yourcompany.com/productname-current.zip.
If you really need to use Redmine for managing this, you might check out Mechanize: http://mechanize.rubyforge.org/. They should have a RESTful API also, but I've never used it.
I found this post, hope it can help you
Automating packaging and RedMine
I'm a bit late, but I've wrote a Redmine upload tool in Perl, using the WWW::Mechanize module.
Please find it on http://github.com/subogero/redgit
As already stated, you can use Mechanize for that.
There's a Python script written by Gert van Dijk's: https://github.com/gertvdijk/redmine-file-uploader
To use it you'll have to install Python Mechanize package first:
easy_install mechanize
If you prefer Ruby, you can use:
require 'mechanize'
# Replaces \ with / and removes "
ARGV.map!{|a|a.gsub('\\','/').gsub(/^"(.+)"$/,'\\1')}
filename = ARGV[0] || abort('Filename must be specified')
puts "File: #{filename}"
url = ARGV[1] || abort('Redmine URL must be specified')
puts "Redmine URL: #{url}"
username = ARGV[2] || abort('Redmine username must be specified')
puts "Username: #{username}"
password = ARGV[3] || abort('Redmine password must be specified')
puts "Password: #{'*' * password.length}"
project = ARGV[4] || abort('Redmine project must be specified')
puts "Project: #{project}"
login_page_path = '/login'
files_page_path = '/projects/' + project + '/files'
agent = Mechanize.new
# No certificate verification (I had to use this hack because our server is bound to custom port)
# agent.agent.http.verify_mode = OpenSSL::SSL::VERIFY_NONE
agent.get(URI.join(url, login_page_path)) do |login_page|
login_page.form_with(:action => login_page_path) do |login_form|
login_form.username = username
login_form.password = password
end.submit
end
agent.get(URI.join(url, files_page_path + '/new')) do |upload_page|
upload_page.form_with(:action => files_page_path) do |upload_form|
upload_form.file_uploads.first.file_name = filename
end.submit
end
And don't forget to install gem first:
gem install mechanize