Scraping, My Query Parameters Are Wrong - ruby

I'm on a website using Ruby and Mechanize to pass a POST query to a site. The query that gets to the site, based on firebug, looks like this
param.PrdNo=-1&param.Type=Prop&param.RequestType=Normal&param.PropParams%5B0%5D.CrId=Base-MLB+Su+Washington+Na%40Atlanta+Brave
The QUERY I pass in my ruby code is this
QUERY = { "param.PrdNo" => "-1",
"param.Type" => "Prop",
"param.RequestType" => "Normal",
"param.PropParams[0].CrId" => "Base-MLB+Su+Washington+Na#Atlanta+Brave"}
doc.agent.post(url, QUERY, content_type)
The logger prints out the following
D, [2014-08-10T14:46:24.844744 #15801] DEBUG -- : query: "param.PrdNo=-1&param.Type=Prop&param.RequestType=Normal&param.PropParams%5B0%5D.CrId=Base-MLB%2BSu%2BWashington%2BNa%40Atlanta%2BBrave"
How do I get my code to make a query that looks like the query from firebug?

You could post the string:
vars = "param.PrdNo=-1&param.Type=Prop&param.RequestType=Normal&param.PropParams%5B0%5D.CrId=Base-MLB+Su+Washington+Na%40Atlanta+Brave"
doc.agent.post url, vars, content_type
It will get sent as the raw post body.

Related

How to send query parameters for gmail api for list_user_messages?

I am trying to send query parameters to fetch list of messages via gmail api https://developers.google.com/gmail/api/reference/rest/v1/users.messages/list
Here is my code
message_list = gmail.list_user_messages user_id
It returns the message list
response -
"messages": [
{
"id": "16641115eca503dc",
"threadId": "16641115eca503dc"
},
Now I want to pass the query parameters
params = {userId: 'me',
maxResults: 1,
pageToken: pageToken}
message_list = gmail.list_user_messages(params)
But it is not working as expected. Please share the correct way to add query parameters.
Go to documentation. Use 'Try this API' option and there you can learn about query params like 'is:unread' ,'from:some#abc.com' etc.
If you want to know about more about valid query strings like 'is:unread' then you can use gmail search options and generate valid query strings.
Edit: Search operator documentation

Questions with making calls to an API

I've been working on creating a CLI gem for a job board. I've been setting up my API class, but I have been struggling to get it to work correctly in terms of successful calls; I'm using HTTParty to parse. When I have been testing this, it keeps giving me a method error for "[]". I've gone over everything, made sure the syntax is correct but have hit a wall in figuring out what seems to break this. Here is the method I created to list all of the jobs on the specific board:
def all_jobs_call
url = "https://boards-api.greenhouse.io/v1/boards/flatironschoolcareers/jobs"
response = HTTParty.get(url)
response["absolute_url"]["location"]["metadata"]["id"]["title"].each do |job|
absolute_url = job["absolute_url"]
location = job["location"]
metadata = job["metadata"]
id = job["id"]
title = job["title"]
end
end
I would greatly appreciate any insight as to what I could be doing wrong or if I'm missing something glaring. Thanks!
The JSON response you get from https://boards-api.greenhouse.io/v1/boards/flatironschoolcareers/jobs looks like this:
{
"jobs": [
{
"absolute_url": "https://boards.greenhouse.io/flatironschoolcareers/jobs/4460392002",
"internal_job_id": 4375855002,
"location": {
"name": "New York, NY"
},
"metadata": [
{
"id": 4019377002,
"name": "Employment Type",
"value": "Full-time",
"value_type": "single_select"
},
...
HTTParty converts that response to Ruby objects. So just like in that JSON response, response has a top level "jobs" key which contains an array of jobs.
In order to get the 1st job you'd use:
response["jobs"][0]
#=> {"absolute_url"=>"https://boa...", "internal_job_id"=>4375855002, ...}
and to get it's absolute_url:
response["jobs"][0]["absolute_url"]
#=> "https://boards.greenhouse.io/flatironschoolcareers/jobs/4460392002"
And to traverse all jobs you call each on the array, i.e.:
response["jobs"].each do |job|
puts job["absolute_url"]
end
Output:
https://boards.greenhouse.io/flatironschoolcareers/jobs/4460392002
https://boards.greenhouse.io/flatironschoolcareers/jobs/4460383002
https://boards.greenhouse.io/flatironschoolcareers/jobs/4472889002
...

Elasticsearch Multi Get working through curl, but no results are returned through Java API

I am running elasticsearch 2.3.4, but the syntax does not seem to have changed in 5.x.
Multiget over curl is working just fine. Here is what my curl looks like:
curl 'localhost:9200/_mget' -d '{
"docs" : [
{
"_index" : "logs-2017-04-30",
"_id" : "e72927c2-751c-4b33-86de-44a494abf78f"
}
]
}'
And when I want to pull the "message" field off that response, I use this request:
curl 'localhost:9200/_mget' -d '{
"docs" : [
{
"_index" : "logs-2017-04-30",
"_id" : "e72927c2-751c-4b33-86de-44a494abf78f",
"fields" : ["message"]
}
]
}'
Both of the above queries return the log and information that I am looking for.
But when I try to translate it to Java like this:
MultiGetRequestBuilder request = client.prepareMultiGet();
request.add("logs-2017-04-30", null, "e72927c2-751c-4b33-86de-44a494abf78f");
MultiGetResponse mGetResponse = request.get();
for (MultiGetItemResponse itemResponse : mGetResponse.getResponses()) {
GetResponse response = itemResponse.getResponse();
logger.debug("Outputing object: " + ToStringBuilder.reflectionToString(response));
}
I appear to be getting null objects back. When I try to grab the message field off the null-looking GetResponse object, nothing is there:
GetField field = response.getField("message"); <--- returns null
What am I doing wrong? doing a rest call to elasticsearch proves the log exists, but my Java call is wrong somehow.
The documentation page for the Java multi get completely skips over the extra syntax required to retrieve data beyond the _source field. Just like the REST API, doing a multi get with the minimum information required to locate a log gets very limited information about it. In order to get specific fields from a log in a multi get call through the Java API, you must pass in a MultiGetRequest.Item to the builder. This item needs to have the fields you want specified in it before you execute the request.
Here is the code change (broken into multiple lines for clarity) that results in the fields I want being present when I make the query:
MultiGetRequestBuilder request = client.prepareMultiGet();
MultiGetRequest.Item item = new MultiGetRequest.Item("logs-2017-04-30", "null", "e72927c2-751c-4b33-86de-44a494abf78f");
item.fields("message");
request.add(item);
MultiGetResponse mGetResponse = request.get();
Now I can ask for the field I specified earlier:
GetField field = response.getField("message");

Parsing request JSON in Sinatra app

I am having some difficulty with parsing the JSON from a request to my Sinatra application:
response = JSON.pretty_generate(request.env)
reply = response["rack.request.form_hash"]
results in reply just returning:
rack.request.form_hash
as a string rather than just the relevant part of the response:
{...
"rack.request.form_hash": {
"token": "token",
"team_id": "team",
"team_domain": "teamname",
"service_id": "service",
"channel_id": "channel",
"channel_name": "testing-webhooks",
"timestamp": "1424480976.000910",
"user_id": "U029W1WF2",
"user_name": "myusername",
"text": "checkeverything",
"trigger_word": "checkeverything"
},
...}
which is within the JSON request object I'm trying to parse. When I use:
response["rack.request.form_hash"]["user_name"]
there is nothing returned. The following is returned in my log:
App 1662 stdout:
App 1640 stderr: JSON::ParserError - 746: unexpected token at 'No text specified':
So it looks like it's not iterating properly, or perhaps can't access it.
I've looked through other documentation and other posts, but found nothing that worked for me, but I am definitely overlooking something, but I'm not sure what.
What is the best way to parse this nested array in a request to Sinatra?
This should fix it :
res = JSON.parse( JSON.generate(request.env) )
res.class
# => Hash
res["rack.url_scheme"]
# => http
The reason is that the JSON.generate only generates JSON syntax for objects and arrays in a string. Then you need to parse the generated JSON string into a hash in Ruby with JSON.parse.

Parsing Hashie::Mash output to Json in ruby

I'm getting user input in JSON format for an API created using grape. In a particular parameter, I'm giving an array of JSON in the following format.
"personal" : {
"details" : {
"firstname" :"nagalakshmi",
"lastname" : "n"
}
}
When I tried to print "personal" attribute it is showing like below
#<Hashie::Mash details=#<Hashie::Mash firstname="nagalakshmi" lastname="n">>
Is there any way to parse the attribute to json format?
I was able to do it using to_json
I also got this same object as params with Grape.
Hashie::Mash
params: #<Hashie::Mash data=#<Hashie::Mash attributes=#<Hashie::Mash title="Ember Hamster"> type="pictures">>
If you are using Rails We can convert into ruby hash as following:
params.to_hash
gives
{"data"=>{"type"=>"pictures", "attributes"=>{"title"=>"Ember Hamster"}}}

Resources