Pasting large text into a Watir Webdriver textfield in Ruby - ruby

I have a text field that I am trying to manipulate with Watir Webdriver in Ruby, which is in this format:
<div class="fieldwrapper" ng-hide="tempPageContent.eulaModal.standardEula">
<label class="ng-binding">Custom License Agreement</label>
<textarea ng-model="tempEula.EULAText" class="med ng-pristine ng-valid"></textarea>
<!-- <span text-area-with-counter="tempEula.EULAText"
text-limit="{{ referenceData.appMetaDataReference.maxEulaChars }}"
text-area-class="med"
text-area-required="tempPageContent.eulaModal.customEula"
text-area-itc-field-server-error="versionInfo.eula.errorKeys"
text-area-itc-field-orig-val="orignalVersionInfo.eula.EULAText"
text-area-itc-field-cur-val="tempEula.EULAText"
text-area-itc-empty-errormsg="Enter the license agreement"
text-area-itc-char-exceed-errormsg="The license agreement can not exceed {{ referenceData.appMetaDataReference.maxEulaChars }} characters"></span> -->
</div>
I need to insert a large String that is extracted from a text file into this text field, but using the standard Watir.textarea.set won't work as it would time-out in 30 seconds. Here is what I am trying to do at the moment:
#browser.execute_script("arguments[0].value = arguments[1]", text_field, eula_text)
which injects the text into the text field, but does not enable the 'Save' button, which is triggered by the native set method, but not by Javascript.
I saw some post by jarib, who proposed the usage of Mac's pbcopy for copying text and then using the send_keys([:command, 'v']), but using the send_keys does not work, although the text is in the IO buffer. I tried both open and popen methods. I also tried using pbpaste on Watir textarea element...
I cannot think of a novel idea to accomplish my task and any pointers in the right direction would be appreciated. I am just not familiar with the way AngularJS text fields work with text input.
I am using the latest watir-webdriver 0.9.1 and chromedriver.

3 possibilities:
1) Increase the client timeout:
client = Selenium::WebDriver::Remote::Http::Default.new
client.timeout = 180
b = Watir::Browser.new :chrome, http_client: client
2) Don't do all the text at once:
File.open('xxx.txt').each do |line|
textfield.append(line)
end
3) Use your javascript code to copy everything in, then use textfield.append(' ') to enable the save button.

Related

Ruby, Watir, and how to fill out text fields inside of a Shadow DOM? (Safari)

EDIT: PARTIAL SOLUTION FOUND: I was using Safari. Running the task through Chrome made turned what was in the Shadow DOM into the Light Dom so that I can now access it!
I am attempting to automate a browser task and automate some form filling with Watir, but I am having the hardest time trying to fill out text boxes in Shadow DOMs.
Image of the text field:
https://i.imgur.com/FWccWea.png
This is the corresponding HTML for the Postal Code text field:
<input class="textbook" id="Fm2_Ctrol19_TextBox" name="Fm2_Ctrol19_TextBox"
onchange="SearchJs.setBackgroundColor(this, SearchJs.checkTextBox_;"
onkeypress="return
SearchJS.onTextBoxKeyPress(event, this);"
onkeyup="SearchJs.setBackgroundColor(this,SearchJs.checkTextBox);"
onpaste="return SearchJs.onTextBoxPaste(event, this);" style="width:100px;
background=color: #ffffff;"
title="The zip code in which the property is located." type="text"> =$0
Shadow Content (User Agent) # The text field is in here, I imagine.
</input>
I am not familiar enough with javascript and Shadow DOM to know if the articles I have googled are relevant.
I have tried using plain' Watir in the past, but it comes out Watir::Exception::UnknownObjectException (timed out after 30 seconds, waiting for #<Watir::TextField: located: false; {:id=>"Fm2_Ctrol19_TextBox", :tag_name=>"input"}> to be located) after a 30 second time out.
b.text_field(id: 'Fm2_Ctrol19_TextBox').set 'POSTAL CODE HERE' # What I've tried
Does anyone have any information that could help me fill out that box?
PARTIAL SOLUTION FOUND: I was using Safari. Running the task through Chrome made turned what was in the Shadow DOM into the Light Dom so that I can now access the data!

How to use send keys when input='file' doesn't exist using Ruby and Selenium

I'm working on a web scraper with Ruby and Selenium that will take a CSV file and import it into Google Sheets.
Everything is fine until I need to target the "select a file from your device" input and upload the file using send keys. Normally I see an input='file' tag and use that, however I'm no longer seeing it.
What I would do is
driver.find_element(xpath: '//*[#type="file"]').send_keys("#{Dir.pwd}/my_file.csv")
but this no longer works as there's no type='file'.
This is what I see when I inspect:
<div role="button" class="a-b-c d-u d-u-F" tabindex="0" style="user-select: none;">Select a file from your device</div>
Is there a different way I could do this without targeting an input with a type of "file"?
You could try this XPath expression :
driver.manage.timeouts.implicit_wait=10
element = driver.find_element(:xpath, "//div[#class][contains(text(),'device')]")
element.click
element.send_keys "#{Dir.pwd}/my_file.csv", :enter

Is there something special about a file type input that makes it not able to find?

I'm looking to add files to an <input type="file">
Here is a snippet of the html
<span class="btn btn-xs btn-primary btn-file"> #found
<span class="blahicon blahicon-upload"></span>
Browse
<input type="file" data-bind="value: fileName, event: { change: uploadImagesOnChange }"
accept="blah/txt" multiple=""> #not found
</span>
and here's the capybara and ruby
within_frame('frame1') do
within_frame('frame2') do
within(:xpath, [containing span xpath]) do # finds this
find(:xpath, './/*[#type="file"]').send_keys('C:\Users\...\blah.txt') #ElementNotFound
end
end
end
I see no hidden block and it's super scoped. Any thoughts?
Instead of using within(:xpath, [containing span xpath]) can you directly check the xpath of input like: has_xpath?(".//span[contains(text(),'Browse')]/input")
and if it is returning true then try with using
find(:xpath, ".//span[contains(text(),'Browse')]/input").send_keys ('C:\Users\...\blah.txt')
If you are aware about the 'pry' gem then you can debug this thing by trying various xpath combinations instead of just running the whole script, So you will get to know the actual problem.
I think there is mistake in your xpath
.//*[#type="file"]
Change to
.//*[#type='file']
as browser can detect attribute value with double quotes(" ") but inside script you need to use it in single quote (' ')
Also can make different combinations of your XPath like
//input[#type='file']
Judging by the class btn-file on the wrapper class, it's probable you're using Bootstrap and one of the "standard" methods to hide the actual file input element so that it can be style the same across multiple browsers. There are a number of ways of hiding the button from just setting display:none on it to the more "modern" method of expanding it to the same size as the replacement button and seeting it's opacity to 0 so it become a transparent overlay on the replacement.
The basic strategy to dealing with this kind of setup in Capybara is to use execute_script to make the element visible first and then use attach_file or set as normal. For instance if your site is using the opacity method of hiding the file element you could do something like
within_frame('frame1') do
within_frame('frame2') do
within(:xpath, [containing span xpath]) do # finds this
file_input = find(:file, visible: :all)
page.driver.browser.execute_script("$(arguments[0]).css('opacity',1)", file_input.native)
file_input.set('C:\Users\...\blah.txt')
end
end
end
Note - this code assumes you're using jQuery in your page and will only work with the selenium driver since it uses seleniums driver specific ability to pass elements from capybara to selenium in the execute_script call. If not using jQuery the JS will need to change and if using another driver you would need to find the element in the JS script using DOM methods and then modify its opacity.

Problems uploading a file using file_field in Watir

My system:
Windows 10 Pro 64-bit
ruby 2.1.9p490 (2016-03-30 revision 54437) [x64-mingw32]
FireFox 47.0.1
To start, here is the code I'm dealing with:
<div class="dz-style col-sm-7" is="null">
<div is="null">You can drag and drop your supporting document files here, or click to select files to upload.</div>
<input style="display: none;" multiple="" is="null" type="file"></div>
Here is my watir testing code:
Identify and confirm file is valid
local_file = '/Users/tom.feodoroff/Desktop/Charlie_Snoopy.jpg'
File.exists? local_file
raise "error" unless File.exists? local_file
Change the style display so I can interact with control
element = BROWSER.input(:type => 'file')
puts element.attribute_value('style') #display: none;
script = "return arguments[0].style = 'display: inline'"
BROWSER.execute_script(script, element)
puts element.attribute_value('style') #display: inline;
Use suggested syntax to add file to application
BROWSER.file_field(:type => 'file').set(local_file)
This doesn't generate any errors, but it also doesn't attach the file so that my Submit button becomes active. Do I need a different version of Ruby (Watir) to make this work or is there something I'm missing?
I don't understand why, but I added sleep(5) just before clicking the submit button, and now it works. My file icon now appears on the page and the submit button is active and successfully submitted the form. Things that make you go 'hmmmm' :)
Thanks for the responses. Hopefully this will help someone else with this problem?

Web scraping from youtube with nokogiri

I want to scrape all the names of the users who commented below a youtube video.
I'm using ruby and nokogiri.
require 'rubygems'
require 'nokogiri'
require 'open-uri'
url = "https://www.youtube.com/watch?v=tntOCGkgt98"
doc = Nokogiri::HTML(open(url))
doc.css(".comment-thread-renderer > .comment-renderer").each do |comment|
name = comment.css("#comment-section-renderer-items .g-hovercard").text
puts name
end
But it's not working, I'm not getting any output, no error either.
I won't be able to give you a solution, but at least I can give you a couple of hints that may help you to move forward.
The code you have is not working because the comments section is loaded via an ajax call after the page is loaded. If you do a hard reload in your browser, you will see that there is a spinner icon and a Loading... text in the sections comment, waiting for the content to be loaded. When Nokogiri gets the page via the http request, it gets the html content that you see before the comments are loaded. As a matter of fact the place where the contents will be later added looks like:
<div id="watch-discussion" class="branded-page-box yt-card">
<div id="comment-section-renderer"
class="comment-section-renderer vve-check"
data-visibility-tracking="CCsQuy8iEwjr3P3u1uzNAhXIepAKHRV9D8Ao-B0=">
<div class="action-panel-loading">
<p class="yt-spinner ">
<span class="yt-spinner-img yt-sprite" title="Loading icon">
</span>
<span class="yt-spinner-message">Loading...</span>
</p>
</div>
</div>
</div>
That is the reason why you won't find the divs you are looking for, because they aren't part of the html you have.
Looking at the network console in the browser, it seems that the ajax request to get the comments data is being sent to https://www.youtube.com/watch_fragments_ajax?v=tntOCGkgt98&tr=time&distiller=1&ctoken=EhYSC3RudE9DR2tndDk4wAEAyAEA4AEBGAY%253D&frags=comments&spf=load. As you can see the v parameter is the video id, however there are a couple of caveats:
There is a ctoken param, which you can get by scraping the original page contents. It is inside a <script> tag, in the form of
'COMMENTS_TOKEN': "<token>".
However, you still need to send a session_token as a form data in the body of the AJAX request (which is a POST). That I don't know where is coming from :(.
I think that you will be pushing the limits of Nokogiri here, as AFAIK it is not intended to follow ajax requests or handling Javascript. Maybe the ruby Selenium driver is better suited for this.
HTH
I think you need name.css("#comment-section..."
The each statement will iterate over the elements, using the variable name.
You may want to use node instead of name:
doc.css(".comment-thread-renderer > .comment-renderer").each do |node|
name = node.css("#comment-section-renderer-items .g-hovercard").text
puts name
end
I wrote this rails app using nokogiri to see all the tags that a page has before any javascript is run in the browser. The source code is here, so you can adjust it if you need to add more info about the node in the view.
That can easily tell you if the particular tag element that you are looking for is something you can retrieve without having to do some JS eval.
Most web crawlers don't support client-side rendering, which gives you an idea that it's not a trivial task to execute JS when scraping content.
YouTube is a dynamically rendered JavaScript website, though it could be parsed with Nokogiri without using Selenium or another package. Try open the Network tab in dev tools, scroll to the comment section, and see what request being send.
You need to make a post request in order to fetch comments data. You can preview the output in the "Preview" tab.
Preview output:
Which is equivalent to this comment:
Note: Since this comment brings very little value, this answer will be updated with the attached code once there will be an available solution.

Resources