So I'm trying to add info to my .json file, a first name and a last name,
from just running the script via terminal in kali linux.
I have not have had much progress I have been searching online for the past 5 hours to find a way to fix this but there has not been much progress.
Which leaves me here to post this, anyone know how to fix this error?
Error from Terminal
Traceback (most recent call last):
4: from /root/Desktop/JSON_thingy.rb:8:in `<main>'
3: from /root/Desktop/JSON_thingy.rb:8:in `open'
2: from /root/Desktop/JSON_thingy.rb:9:in `block in <main>'
1: from /usr/lib/ruby/vendor_ruby/json/common.rb:156:in `parse'
/usr/lib/ruby/vendor_ruby/json/pure/parser.rb:118:in `parse': source in not valid JSON! (JSON::ParserError)
Code
require 'json'
require 'json/pure'
json = {"first_name" => "john", "last_name" => "doe"}
initial_json = File.read('/root/Desktop/jsonDatabase.json')
File.open("/root/Desktop/jsonDatabase.json","w") do |f|
f.puts JSON.pretty_generate(JSON.parse(initial_json) << json )
end
JSON
[
{
"first_name": "albert",
"last_name": "einstein"
},
{
"first_name": "edgar",
"last_name": "poe"
}
]
Of course, simply joining two JSON strings together don't work and would result in an invalid JSON. In your example, the hash you try to add would not end up in the array but behind.
The correct way is to read the existing JSON file and parse it into a RUBY data structure and add the new hash in Ruby to the array before writing everything back to JSON.
require 'json'
filename = '/root/Desktop/jsonDatabase.json'
hash = { "first_name" => "john", "last_name" => "doe" }
array = JSON.parse(File.read(filename)
array << hash
File.open(filename, 'w') do |f|
f.write(JSON.pretty_generate(array))
end
The previous answer is cleaner and I would recommend using that.
However, the trace error clearly tells you what the problem is, this line:
/usr/lib/ruby/vendor_ruby/json/pure/parser.rb:118:in `parse':
source in not valid JSON! (JSON::ParserError)
The part of your code which has errored is this:
JSON.parse(initial_json)
So you need to first make sure the data in the file you're loading is actually valid JSON. You can either use the jsonlint ruby gem or other online linters like https://jsonlint.com/ or if you're a duck duck go user this, either of which will "beautify" your json for you.
UPDATE
If you use ruby jsonlint you may get a deprecation warning, you can safely ignore it. But it's because one of its' dependencies trollop is throwing the deprecation warning. I have opened a pull request to fix it. But it looks like it might not be maintained, last commit is almost 2 years ago. *Sigh
Related
TL;DR: How to get the raw input line (not line number) while parsing a csv file?
I'm parsing a delimited file with Ruby's CSV class. I'd like to retrieve the raw line from the file for each row, in addition to the parsed fields from that row.
Here is what I have now:
CSV.foreach(input_file, csv_params) do |row|
add_uploaded_user(row)
end
That works perfectly. Every file is parsed correctly, and add_uploaded_user does what it is supposed to.
We are getting some unusual files from one client, with unexpected user names in the data. The file is valid csv and parses correctly. They claim we are messing up their records, so we want to capture each raw line from the file before it is parsed. We already save the whole CSV file, but it is inconvenient to manually pull the file and find the source record when we get a complaint. We'd like to give them a tool so they can verify exactly what they sent us. Also, we cannot reveal other records from that file the user in question, so we cannot share the entire file.
So, we'd like to capture the raw line of input with each parsed record we create from their file. Something like this:
CSV.foreach(input_file, csv_params) do |row|
add_uploaded_user(row, row.raw_line)
end
...where raw_line is some method/attribute/helper from CSV that reveals the line that was just parsed.
I've gone through the CSV docs, and found https://ruby-doc.org/stdlib-2.6.1/libdoc/csv/rdoc/CSV.html#method-i-line :
line() - The last row read from this file.
But I can't figure out how to call line(). I've tried several invocations, and they all turn out pretty much the same, with NoMethodError: undefined method 'line' for CSV:Class :
irb(main):022:0> CSV.line
NoMethodError: undefined method 'line' for CSV:Class
irb(main):049:0* csv = CSV.new("a,b,c\n1,2,3\n")
=> <#CSV io_type:StringIO encoding:UTF-8 lineno:0 col_sep:"," row_sep:"\n" quote_char:"\"">
irb(main):050:0> csv.each do |row|
irb(main):051:1* puts row
irb(main):052:1> puts csv.line
irb(main):053:1> end
a
b
c
NoMethodError: undefined method 'line' for #<CSV:0x00007feeb25de3c0>
from (irb):52:in 'block in irb_binding'
from (irb):50
irb(main):054:0>
And a simpler example, reading an actual file:
irb(main):055:0> csv = CSV.new(File.open('3_licenses.csv'))
=> <#CSV io_type:File io_path:"3_licenses.csv" encoding:UTF-8 lineno:0 col_sep:"," row_sep:"\r\n" quote_char:"\"">
irb(main):062:0> csv.shift
=> ["first_name", "last_name", "license_number"]
irb(main):063:0> csv.shift
=> ["David ", "Hempy", "1001"]
irb(main):064:0> csv.line
NoMethodError: undefined method 'line' for #<CSV:0x00007feeb2591020>
from (irb):64
irb(main):065:0> csv.shift
=> ["Santa", "Claus", "np.1"]
UPDATE:
The docs I was reading was for 2.6. I'm running ruby 2.4.5, but it looks like it was there then, as well: https://ruby-doc.com/stdlib-2.4.5/libdoc/csv/rdoc/CSV.html#method-i-line . Interestingly, .line is not mentioned in https://docs.ruby-lang.org/en/2.4.0/CSV.html Hmm....
Also, I don't need the line number -- I need the raw line from the input file.
At this point, I'm about ready to just read the lines myself, then call CSV separately for each line. That will certainly work and put me in control...but I'm still confused why I can't call the .line() method described in the docs. If anyone can see why I'm getting "undefined method 'line'", I'd surely appreciate it.
When the documentation refers to CSV#line they mean you have to call it on an instance of CSV:
require 'csv'
csv = CSV.new(File.open('example.csv'))
csv.each do |row|
p csv.line
end
I am trying to convert the following JSON to CSV via Ruby, but am having trouble with my code. I am learning as I go, so any help is appreciated.
require 'json'
require 'net/http'
require 'uri'
require 'csv'
uri = 'https://www.mapquestapi.com/search/v2/radius?key=Imjtd%7Clu6t200zn0,bw=o5-layg1&radius=3000&callback=processPOIs&maxMatches=4000&origin=40.7686973%2C-73.9918181&hostedData=mqap.33882_stores_prod%7Copen_status%20=%20?%20OR%20open_status%20=%20?%20OR%20open_status%20=%20?%7CExisting,Coming%20Soon,New%7C'
response = Net::HTTP.get_response(URI.parse(uri))
struct = JSON.parse(response.body.scan(/processPOIs\((.*)\);/).first.first)
CSV.open("output.csv", "w") do |csv|
JSON.parse(struct).read.each do |hash|
csv << hash.values
end
end
The error I receive is:
from c:/RailsInstaller/Ruby2.2.0/lib/ruby/gems/2.2.0/gems/json-1.8.3/lib/json/common.rb:155:in `new'
from c:/RailsInstaller/Ruby2.2.0/lib/ruby/gems/2.2.0/gems/json-1.8.3/lib/json/common.rb:155:in `parse'
from test.rb:14:in `block in <main>'
from c:/RailsInstaller/Ruby2.2.0/lib/ruby/2.2.0/csv.rb:1273:in `open'
from test.rb:13:in `<main>'
I am trying to get all the data off of the following link and put it into a CSV file that I can analyse later. https://www.mapquestapi.com/search/v2/radius?key=Imjtd%7Clu6t200zn0,bw=o5-layg1&radius=3000&callback=processPOIs&maxMatches=4000&origin=40.7686973%2C-73.9918181&hostedData=mqap.33882_stores_prod%7Copen_status%20=%20?%20OR%20open_status%20=%20?%20OR%20open_status%20=%20?%7CExisting,Coming%20Soon,New%7C
You have several problems here, the most significant of which is that you're calling JSON.parse twice. The second time you call it on struct, which was the result of calling JSON.parse the first time. You're basically doing JSON.parse(JSON.parse(string)). Oops.
There's another problem on the line where you call JSON.parse a second time: You call read on the value it returns. As far as I know JSON.parse does not ordinarily return anything that responds to read.
Fixing those two errors, your code looks something like this:
struct = JSON.parse(response.body.scan(/processPOIs\((.*)\);/).first.first)
CSV.open("output.csv", "w") do |csv|
struct.each do |hash|
csv << hash.values
end
end
This ought to work iif struct is an object that responds to each (like an array) and the values yielded by each all respond to values (like a hash). In other words, this code assumes that JSON.parse will return an array of hashes, or something similar. If it doesn't—well, that's beyond the scope of this question.
As an aside, this is not great:
response.body.scan(/processPOIs\((.*)\);/).first.first
The purpose of String#scan is to find every substring in a string that matches a regular expression. But you're only concerned with the first match, so scan is the wrong choice.
An alternative is to use String#match:
matches = response.body.match(/processPOIs\((.*)\)/)
json = matches[1]
struct = JSON.parse(json)
However, that's overkill. Since this is a JSONP response, we know that it will look like this:
processPOIs(...);
...give or take a trailing semicolon or newline. We don't need a regular expression to find the parts inside the parentheses, because we already know where it is: It starts 13 characters from the start (i.e. index 12) and ends two characters before the end ("index" -3). That makes it easy work with String#slice, a.k.a. String#[]:
json = response.body[12..-3]
struct = JSON.parse(json)
Like I said, "give or take a trailing semicolon or newline," so you might need to tweak that ending index depending on what the API returns. And with that, no more ugly .first.first, and it's faster, too.
Thank you everybody for the help. I was able to get everything into a CSV and then just used some VBA to organize it the way I wanted.
require 'json'
require 'net/http'
require 'uri'
require 'csv'
uri = 'https://www.mapquestapi.com/search/v2/radius?key=Imjtd%7Clu6t200zn0,bw=o5-layg1&radius=3000&callback=processPOIs&maxMatches=4000&origin=40.7686973%2C-73.9918181&hostedData=mqap.33882_stores_prod%7Copen_status%20=%20?%20OR%20open_status%20=%20?%20OR%20open_status%20=%20?%7CExisting,Coming%20Soon,New%7C'
response = Net::HTTP.get_response(URI.parse(uri))
matches = response.body.match(/processPOIs\((.*)\)/)
json = response.body[12..-3]
struct = JSON.parse(json)
CSV.open("output.csv", "w") do |csv|
csv << struct['searchResults'].map { |result| result['fields']}
end
I'd like to check if a string is valid YAML. I'd like to do this from within my Ruby code with a gem or library. I only have this begin/rescue clause, but it doesn't get rescued properly:
def valid_yaml_string?(config_text)
require 'open-uri'
file = open("https://github.com/TheNotary/the_notarys_linux_mint_postinstall_configuration")
hard_failing_bad_yaml = file.read
config_text = hard_failing_bad_yaml
begin
YAML.load config_text
return true
rescue
return false
end
end
I am unfortunately getting the terrible error of:
irb(main):089:0> valid_yaml_string?("b")
Psych::SyntaxError: (<unknown>): mapping values are not allowed in this context at line 6 column 19
from /home/kentos/.rvm/rubies/ruby-1.9.3-p374/lib/ruby/1.9.1/psych.rb:203:in `parse'
from /home/kentos/.rvm/rubies/ruby-1.9.3-p374/lib/ruby/1.9.1/psych.rb:203:in `parse_stream'
from /home/kentos/.rvm/rubies/ruby-1.9.3-p374/lib/ruby/1.9.1/psych.rb:151:in `parse'
from /home/kentos/.rvm/rubies/ruby-1.9.3-p374/lib/ruby/1.9.1/psych.rb:127:in `load'
from (irb):83:in `valid_yaml_string?'
from (irb):89
from /home/kentos/.rvm/rubies/ruby-1.9.3-p374/bin/irb:12:in `<main>'
Using a cleaned-up version of your code:
require 'yaml'
require 'open-uri'
URL = "https://github.com/TheNotary/the_notarys_linux_mint_postinstall_configuration"
def valid_yaml_string?(yaml)
!!YAML.load(yaml)
rescue Exception => e
STDERR.puts e.message
return false
end
puts valid_yaml_string?(open(URL).read)
I get:
(<unknown>): mapping values are not allowed in this context at line 6 column 19
false
when I run it.
The reason is, the data you are getting from that URL isn't YAML at all, it's HTML:
open('https://github.com/TheNotary/the_notarys_linux_mint_postinstall_configuration').read[0, 100]
=> " \n\n\n<!DOCTYPE html>\n<html>\n <head prefix=\"og: http://ogp.me/ns# fb: http://ogp.me/ns/fb# githubog:"
If you only want a true/false response whether it's parsable YAML, remove this line:
STDERR.puts e.message
Unfortunately, going beyond that and determining if the string is a YAML string gets harder. You can do some sniffing, looking for some hints:
yaml[/^---/m]
will search for the YAML "document" marker, but a YAML file doesn't have to use those, nor do they have to be at the start of the file. We can add that in to tighten up the test:
!!YAML.load(yaml) && !!yaml[/^---/m]
But, even that leaves some holes, so adding in a test to see what the parser returns can help even more. YAML could return an Fixnum, String, an Array or a Hash, but if you already know what to expect, you can check to see what YAML wants to return. For instance:
YAML.load(({}).to_yaml).class
=> Hash
YAML.load(({}).to_yaml).instance_of?(Hash)
=> true
So, you could look for a Hash:
parsed_yaml = YAML.load(yaml)
!!yaml[/^---/m] && parsed_yaml.instance_of(Hash)
Replace Hash with whatever type you think you should get.
There might be even better ways to sniff it out, but those are what I'd try first.
I'm brand new to ruby (first day working with ruby) so please forgive any novice questions and lack of understanding.
I'm trying to validate the responses to http callouts.
For example, let's say the endpoint is the following:
https://applicationname-api-sbox02.herokuapp.com
And, I'm trying to authenticate a user by sending a get request like this:
get_response = RestClient.get( "https://applicationname-api-sbox02.herokuapp.com/api/v1/users",
{
"Content-Type" => "application/json",
"Authorization" => "token 4d012314b7e46008f215cdb7d120cdd7",
"Manufacturer-Token" => "8d0693ccfe65104600e2555d5af34213"
}
)
Now, I want to validate the response and do the following:
- parse the response to ensure that it is valid JSON
- do some validation and verify the JSON has the correct data (verify that id == 4 for example)
- if an error is encountered, raise an exception using the 'raise' method.
In my first feeble attempt I tried the following:
puts get_response.body
if get_response.code == 200
puts "********* Get current user successful"
else
puts "Get current user failed!!"
end
Now, this returned that getting the current user was successful, but how do I actually parse the json, verify the correct id, and raise an exception if an error occurred?
Instead of raising an exception, write a test.
A straightforward approach, using the json parser and unit test framework from the std lib:
require 'minitest/autorun'
require 'rest_client'
require 'json'
class APITest < MiniTest::Unit::TestCase
def setup
response = RestClient.get("https://applicationname-api-sbox02.herokuapp.com/api/v1/users",
{
"Content-Type" => "application/json",
"Authorization" => "token 4d012314b7e46008f215cdb7d120cdd7",
"Manufacturer-Token" => "8d0693ccfe65104600e2555d5af34213"
}
)
#data = JSON.parse response.body
end
def test_id_correct
assert_equal 4, #data['id']
end
end
Execute with ruby $filename
JSON.parse parses a JSON string into a ruby hash
Getting started with minitest
If you are using ruby 1.8, you'll need to install the json gem and either install the minitest gem, or switch to the older testunit API. If you choose the latter, then you'll need to change require 'minitest/autorun' -> require 'test/unit' and MiniTest::Unit::TestCase -> Test::Unit::TestCase
I'm a little late to the party, but I recently co-created an rspec driven framework called Airborne for just this purpose. Check it out: https://github.com/brooklynDev/airborne
here is an example from our specs so you can see how we test json api:
it 'returns charge' do
get "/charges/#{charge.id}", '', headers
expect(response.status).to eq(200)
expect(response).to match_response_schema(:charge)
expect(response).to match_json(<<-JSON)
{
"id":"{id}",
"email": "{email}",
"ip": "127.0.0.1",
"amount": 10500,
"state": "captured",
"captured_amount": 10500,
}
JSON
end
Lets look at it closely
match_response_schema(:charge)
This matcher checks that json we get in response is in general valid. We use json-schema (json schema validator) for it. Guys from Thoughtbot have a detailed guide how to use json schema validator and create own matcher in this blog post.
Understanding JSON Schema is where I got a lot of useful information on how to create schemas for JSON documents.
match_json
This is our own matcher and we have released match_json gem recently. Using it you can test structure and values of your json. Here are two great features of this matcher:
if you don't know exact values, you can use patterns like {id}, {uuid} {date_time}, etc. we have predefined patterns but you can add your own too.
you get clear failure message what is wrong with your json e.g. "5" was not found in " > array":[1,2,3]
Parsing json can be done with the json gem: http://flori.github.com/json/
Parsed json is accessed through key/value just like in javascript. You can easily verify the values and conditionally raise errors.
Raising errors is done like so:
raise "the ID was #{id} instead of 4"
And writing unit tests can be done with Test::Unit - http://www.ruby-doc.org/stdlib-1.9.3/libdoc/test/unit/rdoc/Test/Unit.html
I have a working method that opens and parses a json file. Now I'm trying to iterate through a directory of json files and display their contents.
Working method for a single file:
def aperson
File.open("people/Elvis Presley.json") do |f|
parse = JSON.parse(f.read)
end
end
Non-working method to iterate through a directory:
16. def list
17. Dir.glob('people/*').each do |f|
18. parse = JSON.parse(f)
19 end
20. end
My error is:
/Users/ad/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/json/common.rb:148:in `parse': 743: unexpected token at 'people/Elvis Presley.json' (JSON::ParserError)
from /Users/ad/.rbenv/versions/1.9.3-p194/lib/ruby/1.9.1/json/common.rb:148:in `parse'
from app.rb:18:in `block in list'
from app.rb:17:in `each'
from app.rb:17:in `list'
from app.rb:24:in `<main>'
All of the files in the directory have the same content and are valis as per JSONlint.
Any help would be greatly appreciated.
You tried to parse the filename as JSON, which won't work.
Instead, you need to read the file first:
parse = JSON.parse(File.read(f))
not sure, but can you try to parse the content of file instead of file name:
parse = JSON.parse( File.read f )
In your non-working code, f is just string of the expanded file name. So you need to read the file after you've received the filename in the block.
While writing it, #nneonneo already gave you solution. So I'm not giving again.