monetdb full outer join resulting in varchar type_digits=0 - monetdb

I am using MonetDB v11.29.7 "Mar2018-SP1" on a Windows10 x64 bit operating system. When I perform a full outer join with two tables on respective varchar columns with lengths > 0 (type_digits > 0), the resultant column in the target table yields a varchar column with type_digits=0, although the column data seems to display the proper, non-null varchar records.
I am not sure how to interpret column information of type=varchar and type_digits=0. This state is causing issues in the subsequent handling/extraction of data via Python interfaces (UDFs), as the expected Python dtype for the data of this column is ambiguous for Python numpy conversion.
I have provided a simple example whereby I created two small tables (dummy4 and dummy5) with two columns each and then create a third table (dummy6) using a full outer join command.
For table dummy6 and column "key", I would have expected the type_digits=32 (as per the "key" columns in the two source tables dummy4 & dummy5). Additionally, how should I interpret type=varchar and type_digits=0 state? What would be the proper handling/expectation when accessing/allocating a Python/numpy array for extracting the "key" column of table "dummy6" (via Python UDFs) in this case?
create table dummy4(key varchar(32), val int);
insert into dummy4 values('AAAAAAAA',1);
insert into dummy4 values('BBBBBBBBB',2);
select * from dummy4;
+-----------+------+
| key | val |
+===========+======+
| AAAAAAAA | 1 |
| BBBBBBBBB | 2 |
+-----------+------+
create table dummy5(key varchar(32), val int);
insert into dummy5 values('CCCCCCCC',3);
insert into dummy5 values('DDDDDDDD',4);
select * from dummy5;
+----------+------+
| key | val |
+==========+======+
| CCCCCCCC | 3 |
| DDDDDDDD | 4 |
+----------+------+
create table dummy6 as select key, dummy4.val as "val4", dummy5.val as "val5" from dummy4 full outer join dummy5 using (key);
select * from dummy6;
+-----------+------+------+
| key | val4 | val5 |
+===========+======+======+
| AAAAAAAA | 1 | null |
| BBBBBBBBB | 2 | null |
| CCCCCCCC | null | 3 |
| DDDDDDDD | null | 4 |
+-----------+------+------+
select t.name as "table_name", t.id as "table_id", c.id as "column_id", c.name as "column_name", c.type, c.type_digits from sys.tables t JOIN sys.columns c ON c.table_id = t.id where t.name = 'dummy4';
+------------+----------+-----------+-------------+---------+-------------+
| table_name | table_id | column_id | column_name | type | type_digits |
+============+==========+===========+=============+=========+=============+
| dummy4 | 78445 | 78443 | key | varchar | 32 |
| dummy4 | 78445 | 78444 | val | int | 32 |
+------------+----------+-----------+-------------+---------+-------------+
select t.name as "table_name", t.id as "table_id", c.id as "column_id", c.name as "column_name", c.type, c.type_digits from sys.tables t JOIN sys.columns c ON c.table_id = t.id where t.name = 'dummy5';
+------------+----------+-----------+-------------+---------+-------------+
| table_name | table_id | column_id | column_name | type | type_digits |
+============+==========+===========+=============+=========+=============+
| dummy5 | 78449 | 78447 | key | varchar | 32 |
| dummy5 | 78449 | 78448 | val | int | 32 |
+------------+----------+-----------+-------------+---------+-------------+
select t.name as "table_name", t.id as "table_id", c.id as "column_id", c.name as "column_name", c.type, c.type_digits from sys.tables t JOIN sys.columns c ON c.table_id = t.id where t.name = 'dummy6';
+------------+----------+-----------+-------------+---------+-------------+
| table_name | table_id | column_id | column_name | type | type_digits |
+============+==========+===========+=============+=========+=============+
| dummy6 | 78457 | 78454 | key | varchar | 0 |
| dummy6 | 78457 | 78455 | val4 | int | 32 |
| dummy6 | 78457 | 78456 | val5 | int | 32 |
+------------+----------+-----------+-------------+---------+-------------+

In fact this was a MonetDB's bug and was fixed today. Th fix will be featured on the upcoming Nov2019 release.

Related

Oracle SQL: Display single columns from multiple rows of a single table with Logic

Oracle SQL
I would like to look-up data from 2 rows of the same column from the same table together in an existing long query with multiple joins.
Current table set-up (single table):
Table: ACCOUNT_DETAILS
| TRX_ID | TYPE | FAC_ID | ACC_ID |
| ------ | ---- | ------ | ------ |
| 1234 | CRDR | ABC123 | AB1234 |
| 1234 | DBTR | XYZ222 | XY9800 |
| 9876 | CRDR | EFG999 | EF7659 |
| 9876 | DBTR | ABC123 | AB9900 |
Expected Result:
Table: REPORT
| TRX_ID | Counterparty FAC_ID | Counterparty ACC_ID |
| ------ | ------------------- | ------------------- |
| 1234 | XYZ222 | XY9800 |
| 9876 | EFG999 | EF7659 |
Logic needed:
If FAC_ID NOT LIKE 'ABC%' then refer to the Counterparty FAC_ID (e.g. for TRX_ID = 1234, it will refer to the DBTR FAC_ID and DBTR ACC_ID; TRX_ID = 9876, it will refer to the CRDR FAC_ID AND CRDR ACC_ID)
Example:
SELECT (CASE WHEN TYPE = 'DBTR' AND FAC_ID LIKE 'ABC%' THEN (SELECT FAC_ID FROM ACCOUNT_DETAILS WHERE TYPE = 'CRDR')
ELSE (SELECT FAC_ID FROM ACCOUNT_DETAILS WHERE TYPE = 'DBTR') END)
FROM ACCOUNT_DETAILS
I've tried options such as JOINs, UNIONs and subqueries but it does not work. I would like to have the Counterparty FAC_ID and Counterparty ACC_IDs in separate single lines in the query, as I will include it in a long query that I already have.
We can use CASE in a CTE to flag the rows where FAC_ID does not start with "ABC" and then use the flag in the WHERE clause.
See db<>fiddle here for schema.
with cte as
(
select
TRX_ID,
FAC_ID,
ACC_ID,
CASE WHEN FAC_ID LIKE 'ABC%' THEN 0 ELSE 1 END ordinal
FROM ACCOUNT_DETAILS)
SELECT
TRX_ID,
FAC_ID,
ACC_ID
FROM CTE
WHERE ordinal = 1;
TRX_ID | FAC_ID | ACC_ID
-----: | :----- | :-----
1234 | XYZ222 | XY9800
9876 | EFG999 | EF7659
db<>fiddle here

Querying HIVE Metadata

I need to query the following table and view information from my Apache HIVE cluster:
Each row needs to contain the following:
TABLE SCHEMA
TABLE NAME
TABLE DESCRIPTION
COLUMN NAME
COLUMN DATA TYPE
COLUMN LENGTH
COLUMN PRECISION
COLUMN SCALE
NULL OR NOT NULL
PRIMARY KEY INDICATOR
This can be easily queried from most RDBMS (metadata tables/views), but I am struggling to find much information about the equivalent metadata tables/views in HIVE.
Please help :)
This information is available from the Hive metastore. The below example query is for a MySQL-backed metastore (Hive version 1.2).
SELECT
DBS.NAME AS TABLE_SCHEMA,
TBLS.TBL_NAME AS TABLE_NAME,
TBL_COMMENTS.TBL_COMMENT AS TABLE_DESCRIPTION,
COLUMNS_V2.COLUMN_NAME AS COLUMN_NAME,
COLUMNS_V2.TYPE_NAME AS COLUMN_DATA_TYPE_DETAILS
FROM DBS
JOIN TBLS ON DBS.DB_ID = TBLS.DB_ID
JOIN SDS ON TBLS.SD_ID = SDS.SD_ID
JOIN COLUMNS_V2 ON COLUMNS_V2.CD_ID = SDS.CD_ID
JOIN
(
SELECT DISTINCT TBL_ID, TBL_COMMENT
FROM
(
SELECT TBLS.TBL_ID TBL_ID, TABLE_PARAMS.PARAM_KEY, TABLE_PARAMS.PARAM_VALUE, CASE WHEN TABLE_PARAMS.PARAM_KEY = 'comment' THEN TABLE_PARAMS.PARAM_VALUE ELSE '' END TBL_COMMENT
FROM TBLS JOIN TABLE_PARAMS
ON TBLS.TBL_ID = TABLE_PARAMS.TBL_ID
) TBL_COMMENTS_INTERNAL
) TBL_COMMENTS
ON TBLS.TBL_ID = TBL_COMMENTS.TBL_ID;
Sample output:
+--------------+----------------------+-----------------------+-------------------+------------------------------+
| TABLE_SCHEMA | TABLE_NAME | TABLE_DESCRIPTION | COLUMN_NAME | COLUMN_DATA_TYPE_DETAILS |
+--------------+----------------------+-----------------------+-------------------+------------------------------+
| default | temp003 | This is temp003 table | col1 | string |
| default | temp003 | This is temp003 table | col2 | array<string> |
| default | temp003 | This is temp003 table | col3 | array<string> |
| default | temp003 | This is temp003 table | col4 | int |
| default | temp003 | This is temp003 table | col5 | decimal(10,2) |
| default | temp004 | | col11 | string |
| default | temp004 | | col21 | array<string> |
| default | temp004 | | col31 | array<string> |
| default | temp004 | | col41 | int |
| default | temp004 | | col51 | decimal(10,2) |
+--------------+----------------------+-----------------------+-------------------+------------------------------+
Metastore tables referred in query:
DBS: Details of databases/schemas.
TBLS: Details of tables.
COLUMNS_V2: Details about columns.
SDS: Details about storage.
TABLE_PARAMS: Details about table parameters (key-value pairs)

Update in oracle with joining two table

I have these two tables below, I need to update Table1.Active_flag to Y, where Table2.Reprocess_Flag is N.
Table1
+--------+--------------+--------------+--------------+-------------+
| Source | Subject_area | Source_table | Target_table | Active_flag |
+--------+--------------+--------------+--------------+-------------+
| a | CUSTOMER | ADS_SALES | ADS_SALES | N |
| b | CUSTOMER | ADS_PROD | ADS_PROD | N |
| CDW | SALES | CD_SALES | CD_SALES | N |
| c | PRODUCT | PD_PRODUCT | PD_PRODUCT | N |
| d | PRODUCT | PD_PD1 | PD_PD1 | N |
| e | ad | IR_PLNK | IR_PLNK | N |
+--------+--------------+--------------+--------------+-------------+
Table2
| Source | Subject_area | Source_table | Target_table | Reprocess_Flag |
+--------+--------------+--------------+--------------+----------------+
| a | CUSTOMER | ADS_SALES | ADS_SALES | N |
| b | CUSTOMER | ADS_PROD | ADS_PROD | N |
| CDW | SALES | CD_SALES | CD_SALES | N |
| c | PRODUCT | PD_PRODUCT | PD_PRODUCT | Y |
| d | PRODUCT | PD_PD1 | PD_PD1 | Y |
| e | ad | IR_PLNK | IR_PLNK | N |
+--------+--------------+--------------+--------------+----------------+
Use all three columns in a single select statement.
UPDATE hdfs_cntrl SET active_flag = 'Y'
where (source,subject_area ,source_table ) in ( select source,subject_area ,source_table from proc_cntrl where Reprocess_Flag = 'N');
Updating one table based on data in another table is almost always best done with the MERGE statement.
Assuming source is a unique key in table2:
merge into table1 t1
using table2 t2
on (t1.source = t2.source)
when matched
then update set t1.active_flag = 'Y'
where t2.reprocess_flag = 'N'
;
If you are not familiar with the MERGE statement, read about it - it's just as easy to learn as UPDATE and INSERT and DELETE, it can do all three types of operations in a single statement, it is much more flexible and, in some cases, more efficient (faster).
merge into table1 t1
using table2 t2
on (t1.sorce=t2.source and t1.Subject_area = t2.Subject_area and t1.Source_table = t2.Source_table and t1.Target_table = t2.Target_table and t2.flag_status = 'N')
when matched then update set
t1.flag = 'Y';
UPDATE hdfs_cntrl SET active_flag = 'Y' where source in ( select source from proc_cntrl where Reprocess_Flag = 'N') and subject_area in (select subject_area from proc_cntrl where Reprocess_Flag = 'N') and source_table in (select target_table from proc_cntrl where Reprocess_Flag = 'N')

How to use ResultSet to fetch the ID of the record

I have got a table with name table_listnames whose structure is given below
mysql> desc table_listnames;
+-------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| name | varchar(255) | NO | | NULL | |
+-------+--------------+------+-----+---------+----------------+
2 rows in set (0.04 sec)
It has got sample data as shown
mysql> select * from table_listnames;
+----+------------+
| id | name |
+----+------------+
| 6 | WWW |
| 7 | WWWwww |
| 8 | WWWwwws |
| 9 | WWWwwwsSSS |
| 10 | asdsda |
+----+------------+
5 rows in set (0.00 sec)
I have a requirement where if name not found under the table , i need to insert or else do nothing
I am achieving it this way
String sql = "INSERT INTO table_listnames (name) SELECT name FROM (SELECT ?) AS tmp WHERE NOT EXISTS (SELECT name FROM table_listnames WHERE name = ?) LIMIT 1";
pst = dbConnection.prepareStatement(sql);
pst.setString(1, salesName);
pst.setString(2, salesName);
pst.executeUpdate();
Is it possible to know the id of the record of the given name in this case

How are product attributes and attribute options stored in Magento database?

I am trying to figure out how the linkage between attribute and attribute options, and product and attributes are made in Magento. Is there any reference to how this is working? or anyone give me a hint on this.
Thanks,
Balan
As Alan Storm says: "you do not have to know about how your db works. You have to learn how the models work ". (This is not an exact quote. I gave you the meaning).
But I created own scheme to understand the DB structure. So this screen shows how it works:
Hope, it helps.
Also I recommend you to look through these links:
http://www.magentocommerce.com/wiki/2_-_magento_concepts_and_architecture/magento_database_diagram
http://alanstorm.com/magento_advanced_orm_entity_attribute_value_part_1
1) The attributes are stored in eav_attribute. There you get the attribute_id.
2) The options are stored in eav_attribute_option_value. There yout get the option_id.
3) The options are assigned to the product in catalog_product_entity_varchar. There you need the entity_id of the product, the attribute_id from 1) and the value which are the comma separated option_ids from 2)
I've found these queries to be very helpful for hunting down things like - where does it say the product color is black?, for example.
-- show_product_attr.sql
select
p.entity_id,
p.entity_type_id,
p.attribute_set_id,
p.type_id,
p.sku,
a.attribute_id,
a.frontend_label as attribute,
av.value
from
catalog_product_entity p
left join catalog_product_entity_{datatype} av on
p.entity_id = av.entity_id
left join eav_attribute a on
av.attribute_id = a.attribute_id
where
-- p.entity_id = 28683
-- p.sku = '0452MR'
p.entity_id = {eid}
;
And for attr_options
-- show_product_attr_options.sql
select
p.entity_id,
-- p.entity_type_id,
-- p.attribute_set_id,
p.type_id,
p.sku,
a.attribute_id,
a.frontend_label as attribute,
-- a.attribute_code,
av.value,
ao.*
from
catalog_product_entity p
left join catalog_product_entity_int av on
p.entity_id = av.entity_id
left join eav_attribute a on
av.attribute_id = a.attribute_id
left join eav_attribute_option_value ao on
av.value = ao.option_id
where
-- p.entity_id = 28683
p.entity_id = {eid}
;
You need to replace {datatype} with text, varchar, int, decimal, etc, for the first query, and {eid} with entity_id for both queries. Which you can do on the command like like this:
$ cat show_product_attr_options.sql | sed -e "s/{eid}/30445/" | mysql -uUSER -pPASS DATABASE -t
+-----------+---------+--------------+--------------+---------------------------+-------+----------+-----------+----------+--------------------+-------------+
| entity_id | type_id | sku | attribute_id | attribute | value | value_id | option_id | store_id | value | colorswatch |
+-----------+---------+--------------+--------------+---------------------------+-------+----------+-----------+----------+--------------------+-------------+
| 30445 | simple | 840001179127 | 96 | Status | 1 | 5972 | 1 | 0 | Male | NULL |
| 30445 | simple | 840001179127 | 102 | Visibility | 1 | 5972 | 1 | 0 | Male | NULL |
| 30445 | simple | 840001179127 | 122 | Tax Class | 2 | 5973 | 2 | 0 | Female | NULL |
| 30445 | simple | 840001179127 | 217 | Size | 257 | 17655 | 257 | 0 | XS | NULL |
| 30445 | simple | 840001179127 | 217 | Size | 257 | 17657 | 257 | 1 | XS | NULL |
| 30445 | simple | 840001179127 | 224 | Color | 609 | 18717 | 609 | 0 | Arctic Ice Heather | NULL |
| 30445 | simple | 840001179127 | 260 | Featured | 0 | NULL | NULL | NULL | NULL | NULL |
| 30445 | simple | 840001179127 | 262 | Clearance Product | 0 | NULL | NULL | NULL | NULL | NULL |
| 30445 | simple | 840001179127 | 263 | Skip from Being Submitted | 0 | NULL | NULL | NULL | NULL | NULL |
| 30445 | simple | 840001179127 | 283 | Discontinued | 0 | NULL | NULL | NULL | NULL | NULL |
+-----------+---------+--------------+--------------+---------------------------+-------+----------+-----------+----------+--------------------+-------------+
A similar set of sql scripts can be created for catalog.
Product Attributes are extra values that you can assign to a product and is stored in the main EAV table, by name, and the data is then stored in a few different tables based on the data type, like varchar, decimal, text Integer, date, etc.
if you had multiple values for your Product Attribute, then that will be stored in the Attribute Options tables, again, different tables based on the data type.
the following link explains the relationships better:
http://www.magentocommerce.com/wiki/2_-_magento_concepts_and_architecture/magento_database_diagram
And deeper developer's detail:
http://www.magentocommerce.com/knowledge-base/entry/magento-for-dev-part-7-advanced-orm-entity-attribute-value
And Attribute sets will be the other thing you come across, like the name suggests, a set of attributes grouped together. http://www.magentocommerce.com/knowledge-base/entry/how-do-i-create-an-attribute-set
HTH
Shaun
SELECT pei.value
FROM `catalog_product_entity_int` pei
JOIN `eav_attribute` ea
ON pei.attribute_id = ea .attribute_id
WHERE pei.entity_id = {your product_id}
AND ea.attribute_code = '{your attribute_code}'
Note that there are a number of different tables like catalog_product_entity_int depending on the type of the attribute, so one of those other ones could be appropriate.
You can get all product properties by using this query:
SELECT CPEV.entity_id, CPE.sku, EA.attribute_id, EA.frontend_label, CPEV.value
FROM catalog_product_entity_varchar AS CPEV
INNER JOIN catalog_product_entity AS CPE ON CPE.entity_id = CPEV.entity_id
INNER JOIN eav_attribute AS EA ON(CPEV.attribute_id = EA.attribute_id AND EA.entity_type_id = 4)
INNER JOIN catalog_eav_attribute AS CEA ON(CEA.attribute_id = EA.attribute_id AND CEA.is_visible_on_front = 1 AND CEA.is_visible_in_grid = 1)

Resources