I know how to import data from the local JSON file, but can the same thing be done for the file that is stored in a remote location? My JSON file is located on a remote server, and it can be accessed via HTTPS protocol.
Here is what the file data.json looks like:
{
"first_name": "James",
"last_name": "Bond",
"pets": ["dog", "cat", "fish"]
}
To load JSON files from another local or remote location, just replace the argument of the procedure with the appropriate path or URL. If you want to create a different kind of graph, you need to change the query accordingly.
To create a node with the label Person and first_name, last_name and pets as properties from the data.json file. You can run the following query:
CALL json_util.load_from_url("https://download.memgraph.com/asset/mage/data.json")
YIELD objects
UNWIND objects AS o
CREATE (:Person {first_name: o.first_name, last_name: o.last_name, pets: o.pets});
Related
I have below sample CSV data coming in multi record format. I want to convert to JSON format like below. I am using Nifi 1.8.
CSV:
id,name,category,status,country
1,XXX,ABC,Active,USA
1,XXX,DEF,Active,HKG
1,XXX,XYZ,Active,USA
Expected JSON:
{
"id":"1",
"status":"Active",
"name":[
"ABC",
"DEF",
"XYZ"
],
"country":[
"USA",
"HKG"
]
}
I tried FetchFile -> ConvertRecord but it is converting every csv record to one JSON object.
Ideal way would be using QueryRecord processor to run Apache calcite SQL query to group by and collect as set to get your desired output.
But i don't know what exactly functions we can use in Apache calcite :(
(or)
You can store the data into HDFS then create a temporary/staging table on top of the hdfs directory.
Use SelectHiveQL processor run the below query:
select to_json(
named_struct(
'id',id,
'status',status,
'category',collect_set(category),
'country',collect_set(country)
)
) as jsn
from <db_name>.<tab_name>
group by id,status
Will result output flowfile as:
+-----------------------------------------------------------------------------------+
|jsn |
+-----------------------------------------------------------------------------------+
|{"id":"1","status":"Active","category":["DEF","ABC","XYZ"],"country":["HKG","USA"]}|
+-----------------------------------------------------------------------------------+
You can Remove header by using csv header to false in case of csv output.
Currently I am caching data from active record to redis by doing following:
redis.rb
$redis = Redis::Namespace.new("bookstore", :redis => Redis.new)
authors_helper.rb
def fetch_authors
authors = $redis.get('authors')
if authors.nil?
authors = Author.all.to_json
$redis.set("authors", authors).to_json
$redis.expire("authors", 5.hour.to_i)
end
JSON.load authors
end
So currently I am using basic set and get to cache and read data from redis.
I want to use hmset instead of just set. The redis way to to this job is as follows:
(Just an example)
HMSET user:1001 name "Mary Jones" password "hidden" email "mjones#example.com"
The authors table in my app consists of following field: id ,name, created_at, updated_at
What is the ruby way to use hmset so that I can cache authors data in a redis hash?
I don't think you can save all authors this way. This is because a hash can store only one value for each key. So name and created_at cannot be keys, because all authors need to store their own values for these keys, but you can use each key only once.
If you're using Ruby on Rails, using Rails.cache is preferred - this way you don't have to worry about the way Rails stores the object in Redis.
However, if you want to use hmset for some reason, I believe the best you can do is something like this:
authors = Author.all.flat_map { |author| [author.id.to_s, author.attributes.to_json] }
$redis.hmset("authors", *authors_data)
The first line will return something like this:
['1', '{"name": "Mary Jones", "email": "m#example.com"}', '2', '{"name": "Another name", "email": "e#example.com"']
hmset command does not accept array, but a flat list of attributes, that's why in the 2nd line you need to pass *authors_data to the function.
Then, internally it will look like this:
{
'1' => '{"name": "Mary Jones", "email": "m#example.com"}',
'2' => '{"name": "Another name", "email": "e#example.com"'
}
Later you can do $redis.hmget("authors", '1') which will return a String '{"name": "Mary Jones", "email": "m#example.com"}'
The examples given by postman team https://www.getpostman.com/docs/v6/postman/collection_runs/working_with_data_files
However, in my situation, i don't have a simple flat file as their example provides. I have a nested data structure. I wanted to know how I can prepare the form data in a nested manner so that the data file is correctly searched and replaced.
Ex:
Name:{{Name}}
Age:{{age}}
addressId:{{addressId}}
addressName:{{addressName}}
addressLine1:{{addressLine1}}
---- following is what I'm unsure about on how to indicate the remaining form data as it is an array of addresses. It will not vary in the number of addresses, but is a child of the above.
my tought was
Addresses:
[
addressId:{{addressId}}
addressName:{{addressName}}
addressLine1:{{addressLine1}}
]
This form data will be used to load multiple individuals with their addresses reading of a data file with 300 records. Sample datafile structure below:
[{
id:1
Name:user1,
Age:34,
Addresses:
[
{addressId:1001
addressName:home,
addressLine1:123 XYZ St}
] },
id:2
Name:user2,
Age:35,
Addresses:
[
{addressId:1002
addressName:home,
addressLine1:124 XYZ St}
]
}, {
id:3
Name:user3,
Age:34,
Addresses:
[
{addressId:1003
addressName:home,
addressLine1:125 XYZ St}
]
}]
All you really need to do is construct the request body in the format you require using the placeholder {{...}} syntax.
In the data file use those same names in either a CSV or a JSON file and assign the values that you require. When the collection is run, it will replace the variables in the request.
[
{
"addressOne": "blah road",
"addressTwo": "blah avenue",
"addressThree": "blah street"
}
]
If those keys were the placeholders in your request body, then those values will be used as the data set when the collection is run.
I am using mailgun for sending email. It has api for add “Unsubscribe me” feature. I am using it in my rails app.
Using this command, i get list of all unsubscribed users i.e. entries in unsubscribes table of mailgun.
RestClient.get "https://api:key-3ax6xnjp29jd6fds4gc373sgvjxteol0" "#api.mailgun.net/v2/samples.mailgun.org/unsubscribes"
I am storing its output in #unsubscriber. So my controller has:
#unsubscribers = RestClient.get "https://api:key-3ax6xnjp29jd6fds4gc373sgvjxteol0" "#api.mailgun.net/v2/samples.mailgun.org/unsubscribes"
When i display the output in view, <%= #unsubscribers %> i get string:
{
"total_count": 1,
"items": [{
"created_at": "Sun, 11 Aug 2013 08:07:22 GMT",
"tag": "*",
"id": "sdfsdfw12423535456",
"address": "xyz#abc.com"
}]
}
As I want to delete unsubscribed emails from my database, I want only emails in #unsubscribers. But it contains whole string.
I am not getting how to extract email from above string so that i can have list of emails in #unsubscribers and i can delete them from my app.
Can anybody help me?
If you are trying to access the address of the first subscriber:
#unsubscriber['items'].first['address']
Safer code:
((#unsubscriber['items'] || []).first || {} )['address']
If you are trying to collect all the addresses:
(#unsubscriber['items'] || []).map{|s| ['address']}
#unsubscriber is a hash. items is an array of hashes. You need to get the first item using [0]. Then you need the address value
#unsubscriber["items"][0]["address"]
Is there a way to take a hash of data (or even better an XML document) could be mapped straight to a mongo document. So this hash:
#hash = {
"doc_id" => 3928,
"header" =>[
{"merchantId"=>["35701"],
"merchantName" =>["Lingerie.com"],
"createdOn" =>["2011-09-23/00:33:35"]}
],
"trailer" =>[
{"numberOfProducts"=>["0"]}
]
}
Would become:
> db.doc.first()
{
_id : ObjectId("4e77bb3b8a3e000000004f7a"),
doc_id : 3928
header : [{
merchantId : "35701",
merchantName : Lingerie.com
}],
trailer : [{
numberOfProducts : 0
}]
}
You could convert the hash to json with to_json:
require 'json'
#hash.to_json
gets you(with formatting):
{
"doc_id":3928,
"header":[{
"merchantId":["35701"],
"merchantName":["Lingerie.com"],
"createdOn":["2011-09-23/00:33:35"]
}],
"trailer":[{
"numberOfProducts":["0"]
}]
}
It should be just a short hop to bson and mongo from there.
Edit:
Okay, I just read the tutorial here:
http://api.mongodb.org/ruby/current/file.TUTORIAL.html
and it looks like you can just put the hash in there and the mongodb driver takes care of it. Maybe I don't understand the question?
It's not clear what you're trying to do, so I'm assuming you want to import XML data into mongo.
I've been importing XML files to mongo. First I convert the xml files to json objects using this command line tool I wrote, xml-to-json. Then, I use mongoimport to import the data into mongo.
Check out the documentation of xml-to-json, including some example input vs. output.