sqoop error : Column 'customernumber' in where clause is ambiguous - sqoop

sqoop import
--connect jdbc:mysql://localhost/classicmodels
--username root --password cloudera
--query ' select c.customernumber, c.customername, o.orderdate, o.ordernumber from customers AS c JOIN orders As o ON c.customernumber = o.customernumber WHERE $CONDITIONS '
--boundary-query 'select min(customernumber), max(customernumber) from customers '
--target-dir /data/info/customerdata/join
--split-by customernumber ;
mysql> describe customers ;
+------------------------+---------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------------------+---------------+------+-----+---------+-------+
| customerNumber | int(11) | NO | PRI | NULL | |
| customerName | varchar(50) | NO | | NULL | |
| contactLastName | varchar(50) | NO | | NULL | |
| contactFirstName | varchar(50) | NO | | NULL | |
| phone | varchar(50) | NO | | NULL | |
| addressLine1 | varchar(50) | NO | | NULL | |
| addressLine2 | varchar(50) | YES | | NULL | |
| city | varchar(50) | NO | | NULL | |
| state | varchar(50) | YES | | NULL | |
| postalCode | varchar(15) | YES | | NULL | |
| country | varchar(50) | NO | | NULL | |
| salesRepEmployeeNumber | int(11) | YES | MUL | NULL | |
| creditLimit | decimal(10,2) | YES | | NULL | |
+------------------------+---------------+------+-----+---------+-------+
mysql> describe orders ;
+----------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------------+-------------+------+-----+---------+-------+
| orderNumber | int(11) | NO | PRI | NULL | |
| orderDate | date | NO | | NULL | |
| requiredDate | date | NO | | NULL | |
| shippedDate | date | YES | | NULL | |
| status | varchar(15) | NO | | NULL | |
| comments | text | YES | | NULL | |
| customerNumber | int(11) | NO | MUL | NULL | |
+----------------+-------------+------+-----+---------+-------+

Try using tablename.column name syntax for the sql in your query's where clause.

sqoop import
--connect jdbc:mysql://localhost/classicmodels
--username root --password cloudera
--query 'select customers.customernumber, customers.customername,
orders.orderdate, orders.ordernumber FROM customers, orders WHERE
customers.customernumber = orders.customernumber AND $CONDITIONS'
--boundary-query 'select min(customernumber), max(customernumber) from customers'
--target-dir /data/info/customerdata/join
--split-by customers.customernumber ;
For --boundary-query, make sure customernumber should be numeric columns and should not be null.

Related

Laravel Eloquent A good way to make relationship between country state and city table

I have 3 tables with schema like below
countries (rows: 250)
+------------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+---------------------+------+-----+---------+----------------+
| id | bigint(20) unsigned | NO | PRI | | auto_increment |
| name | varchar(255) | NO | | | |
| code | varchar(255) | NO | UNI | | |country code
| phone_code | varchar(255) | NO | | | |
| region | varchar(255) | NO | | | |
| subregion | varchar(255) | NO | | | |
| created_at | timestamp | YES | | | |
| updated_at | timestamp | YES | | | |
| deleted_at | timestamp | YES | | | |
+------------+---------------------+------+-----+---------+----------------+
states (rows: 4866)
+-------------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+---------------------+------+-----+---------+----------------+
| id | bigint(20) unsigned | NO | PRI | | auto_increment |
| name | varchar(255) | NO | | | |
| country_code| varchar(255) | NO | MUL | | | this is country code
| state_code | varchar(255) | YES | | | |
| lat | varchar(255) | YES | | | |
| lon | varchar(255) | YES | | | |
| created_at | timestamp | YES | | | |
| updated_at | timestamp | YES | | | |
| deleted_at | timestamp | YES | | | |
+-------------+---------------------+------+-----+---------+----------------+
cities (rows: 146068)
+------------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+---------------------+------+-----+---------+----------------+
| id | bigint(20) unsigned | NO | PRI | | auto_increment |
| name | varchar(255) | NO | | | |
| lat | varchar(255) | YES | | | |
| lng | varchar(255) | YES | | | |
| population | varchar(255) | YES | | | |
| state_code | varchar(255) | NO | | | |state code not unique
| created_at | timestamp | YES | | | |
| updated_at | timestamp | YES | | | |
| deleted_at | timestamp | YES | | | |
+------------+---------------------+------+-----+---------+----------------+
I am using quickadminpanel to generate these CRUD but the main issue is i imported these from a csv files
git link for csv and csvimport trait like https://pastebin.com/G9z8Rjf1
is there any way i can build relationship between these three tables using
country:code and state:country_code relationship and state:state_code and city:state_code
relationship because i cannot add states (rows: 4866) and cities (rows: 146068) manually
so how can i form relationship using models or any better way or any better trait for making relationship?
Just Change The Primary Keys of each table to code,state_code,city_code Respectively
NB:change multiple to unique the state_code in states table and city_code in cities table
And In your Models change The Relationship like
/* return $this->hasMany(Model::class, 'foreign_key', 'local_key');*
return $this->hasMany(State::class, 'country_code', 'code');
And
/// return $this->belongsTo(Model::class, 'foreign_key', 'owner_key');///
return $this->belongsTo(Country::class, 'code', 'country_code');
then You can access all data like normal..

NiFi CaptureChangeMySQL converts varchar columns to nulls

I have problem with Apache NiFi 1.12.1. For some unknown for me reason CaptureChangeMySQL returns many nulls. Basically, only columns which are int, return correct values. I'm new in a matter of using NiFi so I might miss some obvious thing in configuration.
I have following table:
create table inventory.abc
(
id int auto_increment
primary key,
first_name varchar(100) not null,
last_name varchar(100) not null,
age int not null
);
Processor config:
Bin logs settings:
mysql> show variables like '%bin%';
+--------------------------------------------+--------------------------------+
| Variable_name | Value |
+--------------------------------------------+--------------------------------+
| bind_address | * |
| binlog_cache_size | 32768 |
| binlog_checksum | CRC32 |
| binlog_direct_non_transactional_updates | OFF |
| binlog_error_action | ABORT_SERVER |
| binlog_format | ROW |
| binlog_group_commit_sync_delay | 0 |
| binlog_group_commit_sync_no_delay_count | 0 |
| binlog_gtid_simple_recovery | ON |
| binlog_max_flush_queue_time | 0 |
| binlog_order_commits | ON |
| binlog_row_image | FULL |
| binlog_rows_query_log_events | OFF |
| binlog_stmt_cache_size | 32768 |
| binlog_transaction_dependency_history_size | 25000 |
| binlog_transaction_dependency_tracking | COMMIT_ORDER |
| innodb_api_enable_binlog | OFF |
| innodb_locks_unsafe_for_binlog | OFF |
| log_bin | ON |
| log_bin_basename | /var/lib/mysql/mysql-bin |
| log_bin_index | /var/lib/mysql/mysql-bin.index |
| log_bin_trust_function_creators | OFF |
| log_bin_use_v1_row_events | OFF |
| log_statements_unsafe_for_binlog | ON |
| max_binlog_cache_size | 18446744073709547520 |
| max_binlog_size | 1073741824 |
| max_binlog_stmt_cache_size | 18446744073709547520 |
| sql_log_bin | ON |
| sync_binlog | 1 |
+--------------------------------------------+--------------------------------+
29 rows in set (0.00 sec)
And I get results like this:
Any idea why I get so many nulls in output? I thought it might be related to Distributed Map Cache Client but since this option is not mandatory I don't think that's a problem.

org.hibernate.util.JDBCExceptionReporter - Invalid collation 'latin1_general_ci'. with MariaDB

Guys I am trying to integrated my application with MariaDb, I have created intended database and Table as per my requirement and also I have inserted demo data to test application.
As I start application I have to read some configuration values from Database to kick-off the system.
While reading data from one of the table, I am getting
org.hibernate.util.JDBCExceptionReporter - Invalid collation 'latin1_general_ci'.
I also have check the collation of database and Table -
For Database -
SELECT ##character_set_database, ##collation_database;
+--------------------------+----------------------+
| ##character_set_database | ##collation_database |
+--------------------------+----------------------+
| utf8 | utf8_general_ci |
+--------------------------+----------------------+
For Table
SHOW FULL COLUMNS FROM USM_CONFIGURATION;
+---------------------+--------------+-----------------+------+-----+---------+----------------+---------------------------------+---------+
| Field | Type | Collation | Null | Key | Default | Extra | Privileges | Comment |
+---------------------+--------------+-----------------+------+-----+---------+----------------+---------------------------------+---------+
| ID | bigint(20) | NULL | NO | PRI | NULL | auto_increment | select,insert,update,references | |
| ELEMENT_TYPE | int(11) | NULL | NO | | NULL | | select,insert,update,references | |
| INTERNAL_NAME | varchar(64) | utf8_general_ci | NO | | NULL | | select,insert,update,references | |
| PARENT_ID | bigint(20) | NULL | YES | MUL | NULL | | select,insert,update,references | |
| CONFIGURATION_ORDER | int(11) | NULL | YES | | NULL | | select,insert,update,references | |
| HIDDEN | smallint(6) | NULL | NO | | NULL | | select,insert,update,references | |
| READ_ONLY | smallint(6) | NULL | NO | | NULL | | select,insert,update,references | |
| REMOVABLE | smallint(6) | NULL | NO | | NULL | | select,insert,update,references | |
| ALLOW_BLANK | smallint(6) | NULL | NO | | NULL | | select,insert,update,references | |
| PREFERENCE | smallint(6) | NULL | NO | | NULL | | select,insert,update,references | |
| TEMPLATE | smallint(6) | NULL | NO | | NULL | | select,insert,update,references | |
| DISPLAY_NAME_KEY | varchar(64) | utf8_general_ci | YES | | NULL | | select,insert,update,references | |
| DISPLAY_NAME | varchar(256) | utf8_general_ci | YES | | NULL | | select,insert,update,references | |
| DISPLAY_WIDTH | int(11) | NULL | YES | | NULL | | select,insert,update,references | |
| DESCRIPTION_KEY | varchar(256) | utf8_general_ci | YES | | NULL | | select,insert,update,references | |
| DEFAULT_KEY | varchar(64) | utf8_general_ci | YES | | NULL | | select,insert,update,references | |
| DEFAULT_VALUE | float | NULL | YES | | NULL | | select,insert,update,references | |
| USAGE_NOTE | varchar(256) | utf8_general_ci | YES | | NULL | | select,insert,update,references | |
| VALIDATION_CLASS | varchar(256) | utf8_general_ci | YES | | NULL | | select,insert,update,references | |
| OWNER | varchar(64) | utf8_general_ci | YES | | NULL | | select,insert,update,references | |
| UPDATE_DATE | datetime | NULL | YES | | NULL | | select,insert,update,references | |
| NS_THREAD | int(11) | NULL | NO | MUL | NULL | | select,insert,update,references | |
| NS_LEFT | int(11) | NULL | NO | | NULL | | select,insert,update,references | |
| NS_RIGHT | int(11) | NULL | NO | | NULL | | select,insert,update,references | |
| VERSION | int(11) | NULL | YES | | 0 | | select,insert,update,references | |
+---------------------+--------------+-----------------+------+-----+---------+----------------+---------------------------------+---------+
In both Database and Table I couldn't see latin1_general_ci collation.
Why i am getting this error, is there anything which I missed while intalling DB or creating DB.
I have check with SO answers and other blogs for this error and couldn't find any specific solution.
spring.jpa.properties.hibernate.connection.characterEncoding=utf-8
spring.jpa.properties.hibernate.connection.CharSet=utf-8
spring.jpa.properties.hibernate.connection.useUnicode=true

Laravel Eloquent relation depending on type of item

I have model Product, it has 2 types: Lamps and Bulbs. A lamp have set of attributes that differs from a bulb, so I have 2 another models Lamp and Bulb which represent set of attributes. What type of relation from Product to attribute model should I implement in this situation?
I've been trying one-to-one relation, but in my situation this second "one" differs from product's type.
I've been thinking of EAV pattern, but I don't want tons of joins in my architecture right now because I don't need more than these two types of item.
What would you suggest?
UPD1: here are the tables (simplified):
mysql> show columns from products;
+-------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| name | varchar(127) | NO | | NULL | |
| price | double | NO | | NULL | |
| old_price | double | NO | | NULL | |
| category_id | int(11) | NO | | NULL | |
+-------------+--------------+------+-----+---------+----------------+
5 rows in set (0.00 sec)
mysql> show columns from lamps;
+------------+---------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+---------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| product_id | int(11) | NO | | NULL | |
| width | int(11) | NO | | NULL | |
| height | int(11) | NO | | NULL | |
| length | int(11) | NO | | NULL | |
| weight | int(11) | NO | | NULL | |
+------------+---------+------+-----+---------+----------------+
7 rows in set (0.00 sec)
mysql> show columns from bulbs;
+------------+---------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+---------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| product_id | int(11) | NO | | NULL | |
| voltage | int(11) | NO | | NULL | |
| power | int(11) | NO | | NULL | |
| base | int(11) | NO | | NULL | |
| type | int(11) | NO | | NULL | |
+------------+---------+------+-----+---------+----------------+
6 rows in set (0.00 sec)
UPD2: I understand that I can use two foreign keys in products table and set up only one of them for each record, but is there some more elegant scheme for this situation?

Laravel 5 Eloquent ORM - Many to Many through Many to Many

I'm working on a rather complex Users system that requires the ability to grant access privileges to either individuals or entire groups. For example, you could grant Jane Doe access to moderate the Message Board, or all users of the Communications Group to send emails, moderate the message board, and post events (all users assigned to the group are granted access to those privileges when added).
Rather than strictly track individual privileges for each user, I want to grant access to a feature first based on whether or not the user belongs to a group containing that privilege. If they don't, I then want to check to see if they are granted individual access (privilege_user pivot table).
Basically, I want something like this:
// 1 = Message Board Moderator
// See if any of the user's groups contain this privilege -
$hasAccess = Auth::user()->groups->privileges->contains(1);
Here is my table structure:
Users
| Field | Type | Null | Key | Default | Extra |
+----------------+------------------+------+-----+---------------------+----------------+
| id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| username | varchar(80) | NO | | NULL | |
| email | varchar(120) | NO | | NULL | |
| first_name | varchar(20) | NO | MUL | NULL | |
| last_name | varchar(45) | NO | MUL | NULL | |
| password | varchar(140) | NO | | NULL | |
| active | tinyint(1) | NO | | NULL | |
| remember_token | varchar(100) | NO | | NULL | |
| last_login | timestamp | YES | | NULL | |
| last_login_ip | varchar(45) | YES | | NULL | |
| updated_at | timestamp | NO | | 0000-00-00 00:00:00 | |
| created_at | timestamp | NO | | 0000-00-00 00:00:00 | |
| deleted_at | timestamp | YES | | NULL | |
+----------------+------------------+------+-----+---------------------+----------------+
Groups (table: user_groups)
+-------------+------------------+------+-----+---------------------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------------+------------------+------+-----+---------------------+----------------+
| id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| name | varchar(255) | NO | | NULL | |
| slug | varchar(255) | NO | | NULL | |
| description | text | NO | | NULL | |
| created_at | timestamp | NO | | 0000-00-00 00:00:00 | |
| updated_at | timestamp | NO | | 0000-00-00 00:00:00 | |
+-------------+------------------+------+-----+---------------------+----------------+
User Groups Pivot (table: user_user_group)
+---------------+------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------+------------------+------+-----+---------+-------+
| user_id | int(10) unsigned | NO | MUL | NULL | |
| user_group_id | int(10) unsigned | NO | MUL | NULL | |
+---------------+------------------+------+-----+---------+-------+
Privileges (table: privleges)
+-------+------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+------------------+------+-----+---------+----------------+
| id | int(10) unsigned | NO | PRI | NULL | auto_increment |
| name | varchar(255) | NO | | NULL | |
| slug | varchar(255) | NO | | NULL | |
+-------+------------------+------+-----+---------+----------------+
Privileges User Group Pivot (table: privilege_user_group)
+---------------+------------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+---------------+------------------+------+-----+---------+-------+
| privilege_id | int(10) unsigned | NO | MUL | NULL | |
| user_group_id | int(10) unsigned | NO | MUL | NULL | |
+---------------+------------------+------+-----+---------+-------+
Was disappointed that nobody chimed in here on SO but fortunately I was able to find some great help at Laracasts.com.
I went with Zizaco/entrust, which does exactly what I was describing. The only part lacking is answering the question "does this User have a Permission" without the user being assigned to a Group, but that is easy to accomplish with Eloquent. What entrust does do is answer the question "Does this User have access to this Permission through a Group?" And that's what I was after.

Resources