I am trying to run the following query in apache drill. I am querying data stored in parquet files using the following query
select
pan, count(*) as number_of_transactions,
terminal_id,
sum((cast(SETTLE_AMOUNT_IMPACT as double) * -1) / 100) as settle_amount_impact
from
dfs.`/iswdata/storage/products/superswitch/parquet/transactions`
where
pan like '506126%'
and terminal_id like '1%'
and sink_node_name like ('SWTDB%')
and source_node_name not like ('SWTDBLsrc')
and tran_completed = 1
and tran_reversed = 0
and tran_postilion_originated = 1
and tran_type = '01'
--and pan like '506126%0011'
group by
pan, terminal_id
The schema for the data I am querying is as follows
post_tran_id LONG 2
post_tran_cust_id :LONG
settle_entity_id :INTEGER
batch_nr : INTEGER
prev_post_tran_id : LONG
next_post_tran_id : LONG
sink_node_name : STRING
tran_postilion_originated : DECIMAL
tran_completed : DECIMAL
tran_amount_req : DECIMAL
tran_amount_rsp : DECIMAL
settle_amount_impact : DECIMAL
tran_cash_req : DECIMAL
tran_cash_rsp : DECIMAL
tran_currency_code : STRING
tran_tran_fee_req : DECIMAL
tran_tran_fee_rsp : DECIMAL
tran_tran_fee_currency_code : STRING
tran_proc_fee_req : DECIMAL
tran_proc_fee_rsp : DECIMAL
tran_proc_fee_currency_code : STRING
settle_amount_req : DECIMAL
settle_amount_rsp : DECIMAL
settle_cash_req : DECIMAL
settle_cash_rsp : DECIMAL
settle_tran_fee_req : DECIMAL
settle_tran_fee_rsp : DECIMAL
settle_proc_fee_req : DECIMAL
settle_proc_fee_rsp : DECIMAL
settle_currency_code : STRING
However When I run the query against the dataset, I get the following exception
SYSTEM ERROR: ClassCastException: org.apache.drill.exec.vector.NullableDecimal28SparseVector cannot be cast to org.apache.drill.exec.vector.VariableWidthVector
More so, the same error occurs when I include a decimal field in the select clause.
Please, is there something I am missing or doing wrong, Any pointer will be deeply appreciated
Kind Regards
Decimal values in parquet table are stored using BINARY primitive type, but currently, Drill does not support decimals stored as binary.
It will be fixed in DRILL-6094 and it will be available in 1.14 release.
Related
i use Ef core and Oracle
i have a Column in Oracle Database With Data Type NVARCHAR2(20)
but in my Class Model i want Convert It Decimal
this is Config For ValueConverter :
entity.Property(e => e.ColumnName)
.HasMaxLength(20)
.HasColumnType("NVARCHAR2(20)")
.HasColumnName("COLUMN_NAME")
.HasConversion(
v => v.ToString(),
v => v=="" ? 0 : decimal.Parse(v, NumberStyles.Any, CultureInfo.InvariantCulture));
and this my My Model :
public decimal? ColumnName{ get; set; }
but after use this query this error show :
ORA-00932: inconsistent datatypes: expected NUMBER got NCHAR'
I Convert Column Via Oracle Query with check for number combability
(case when REGEXP_LIKE(ColumnName, '^[[:digit:]]+$') then to_number(ColumnName) else 0 end) as ColumnName
My Serial object contains the following values
string Class
int Product
int Code
Together they form a Serial Number. Note that 2 of them are integers.
Class + Product.ToString() + "-" + Code.ToString()
I am attempting to filter based on a string. Example:
serials = _context.Serials.AsQueryable();
serials = serials.Where(serial => serial.Class.Contains(searchText));
I am looking to perform a .Contains() on the concatenated SerialNumber. How would I perform this in EF Core?
Something like this does not work. Throwing an error about query not being translated.
serials = serials.Where(serial => string.Concat(serial.ProductClass, serial.ProductNumber.ToString(), serial.CodeNumber.ToString()).Contains(searchText))
Only string.Concat(string, string) overload is supported.
Use + operator instead:
.Where(serial =>
(serial.ProductClass + serial.ProductNumber.ToString() + serial.CodeNumber.ToString())
.Contains(searchText))
I have a source record that represents a date like this:
20151104
when I used Pig to load the source file, I defined the record like this:
data_raw = LOAD '/user/hue/myfile.csv' USING PigStorage(',') AS
(date:datetime)
Then use the following code to push it to a new format:
data_values = FOREACH data_raw GENERATE ToString(date, 'yyyyMMdd') AS
date
When I dump the variable out, I get:
(201511040101)
Where is the 0101 coming from?
The input is not in ISO date and time formats.
Change input date to 20151104 to 2015-11-04, you will be able to see the expected result.
Ref :
http://www.w3.org/TR/NOTE-datetime
https://pig.apache.org/docs/r0.11.1/func.html#datetime-functions
If you can read the input as String and if its in expected format then you need not do any conversion, if not make use of DateTime Functions to achieve the same.
Update : If you have Date as String in one format and you like to convert it to any other format then if you have to work with ToDate() and ToString() methods.
N.B : Return type of ToDate is DateTime object and that of ToString is String
http://pig.apache.org/docs/r0.12.0/func.html#to-date
http://pig.apache.org/docs/r0.12.0/func.html#to-string
Input :
20151104
PigScript :
A = LOAD 'date_input' USING PigStorage(',') AS (my_date:chararray);
B = FOREACH A GENERATE ToDate(my_date, 'yyyyMMdd') AS my_date;
C = FOREACH B GENERATE ToString(my_date,'yyyy-MM-dd') AS my_date;
Output :
DUMP B :
(2015-11-04T00:00:00.000-08:00)
DUMP C :
(2015-11-04)
I am getting following exception on windows while running the below
ERROR: operator does not exist: numeric = character varying Hint: No operator matches the given name and argument type(s). You might need to add explicit type casts." while executing with query parameter
I am passing the Numeric String for parameter using function as a named parameter to the query
getUIDCount(String id) {
...
select count(UID) as icrd FROM UID_tbl WHERE id = ?
...
}
where id is numeric(5,0)" in table
Everything works well on Ubuntu but getting Error while running the same code on windows. I have to do the explicit casting just for windows. I am using PostgreSQL 9.4.3. I am using "org.hibernate.dialect.PostgreSQLDialec" and grails 2.3.11 with runtime 'org.postgresql:postgresql:9.3-1100-jdbc41'
updated with how it is getting called
def Integer getUIDSetSize(String _id)
{
Integer i = 0;
Sql sql = new Sql(dataSource);
String sqlt = """select count(UID) as icrd FROM UID_tbl WHERE _id = ?""";
log.trace(sqlt);
sql.eachRow(sqlt, [_id], { row -> i = row.icrd; });
return i;
}
This how it get called def _id1 = params._id1; count1 = HelperService.getUIDSetSize(_id1)
The workaround for casting from varchar to numeric is
CREATE CAST(VARCHAR AS NUMERIC) WITH INOUT AS IMPLICIT;
This is not the best solution and would suggest selective casting from the code.
I've to calculate the différence between two Dates : TODAY() and DATE_DEB_VAC.
With Oracle, it's kinda easy : TODAY()-DATE_DEB_VAC -> give the number of day between those 2 date.
But I've to do it with in an ETL (GENIO). I've a column to stock it like that :
NUMBER_DAY_DIFF (NUMBER 10) = TODAY()-DATE_DEB_VAC. But it's impossible to stock it cause it's 2 date.
How can i do this ? :(
You can try the val function of GENIO ETL
VAL(TODAY()-DATE_DEB_VAC)
this is equivalent to to_numbre in Oracle
NUMBER_DAY_DIFF (NUMBER 10) = DATEDIFF (TODAY; DATE_DEB_VAC)
Should give you what you need.