I am looking to import csv file data to a temporary table or even any other ways of dealing with csv data import in API platform.
I do know in MySQL, we can import csv using the LOAD DATA INFILE query to a temporary table, validate and store data, in high speed.
Any help would be appreciated!
Related
Is there anyway to import data directly from Elasticsearch index to h2o through any api.
Thanks
Sarvendra
H2O does not support importing data directly from Elasticsearch. You can write it to CSV and then import it to H2O. The file formats that are supported are: CSV (delimited) files, ORC, SVMLight, ARFF, XLS/XLSX, Avro, Parquet. Data sources that could be used are: Local or Remotes Files, S3, HDFS, JDBC, Hive. You can refer to the documentation on Getting Data Into H2O.
I'm trying to restore some historic backup files that saved in parquet format, and I want to read from them once and write the data into a PostgreSQL database.
I know that backup files saved using spark, but there is a strict restriction for me that I cant install spark in the DB machine or read the parquet file using spark in a remote device and write it to the database using spark_df.write.jdbc. Everything needs to happen on the DB machine and in the absence of spark and Hadoop only using Postgres and Bash scripting.
my files structure is something like:
foo/
foo/part-00000-2a4e207f-4c09-48a6-96c7-de0071f966ab.c000.snappy.parquet
foo/part-00001-2a4e207f-4c09-48a6-96c7-de0071f966ab.c000.snappy.parquet
foo/part-00002-2a4e207f-4c09-48a6-96c7-de0071f966ab.c000.snappy.parquet
..
..
I expect to read data and schema from each parquet folder like foo, create a table using that schema and write the data into the shaped table, only using bash and Postgres CLI.
You can using spark and converting parquet files to csv format, then moving the files to DB machine and import them by any tools.
spark.read.parquet("...").write.csv("...")
import pandas as pd
df = pd.read_csv('mypath.csv')
df.columns = [c.lower() for c in df.columns] #postgres doesn't like capitals or spaces
from sqlalchemy import create_engine
engine = create_engine('postgresql://username:password#localhost:5432/dbname')
df.to_sql("my_table_name", engine)
I made a library to convert from parquet to Postgres’ binary format: https://github.com/adriangb/pgpq
I have a single table in MySql which contains around 24000000 records. I need a way to import this data into a table in MapR DB with multiple column families. I initially chose Sqoop as the tool to import the data but later found that I cannot use Sqoop to directly import the data as Sqoop does not support multiple column family import as yet.
I have populated the data in MapR FS using Sqoop from the MySql database.
What are my choices to import this data from MapR FS to MapR DB table with 3 column families?
It seems for bulk import, I have two choices:
ImportTSV tool: this probably requires the source data to be in TSV format. But the data that I have imported in MapR FS from MySql using Sqoop seems to be in the CSV format. What is the standard solution for this approach?
Write a custom Map Reduce program to translate the data in MapR FS to HFile and load it into MapR DB.
I just wanted to ensure that are these the only two choices available to load the data. This seems to be a bit restrictive given the fact that such a requirement is a very basic one in any system.
If custom Map Reduce is the way to go, an example or working sample would be really helpful.
Create Hive table pointing to MapRDB using HBaseStorageHandler. You can use sqoop to import to hive table.
If you have already downloaded the data MapRFS. Use hive load command to load data to MapRDB.
When trying to load csv file into Oracle table through ODI, ODI is not able to fetch the data from csv file. The csv file format is an issue here with the data in a single line. But when we are opening the csv file through excel and then saving it as csv the format is changing and the data is getting arranged properly and then we are able to import it through ODI.
Problem is we need to import the original csv file whatever format it is. Is there a possibility of doing the same?
SQL Loader will be the first thing that has came to my mind. I use this a lot.
SQL Developer will be a better option if you dont want to work with command line utilities.
Try using External Tables...you can configure how the CSV should be read in the EXTERNAL TABLE configuration
I want to import some specific tables in a user from dmp file which were already in my user. I want to replace these table. Any Help
You just can't beat the information given in the Oracle documentation. This chapter should give you everything you need to import your specific tables.
http://download.oracle.com/docs/cd/B10500_01/server.920/a96652/ch02.htm