ERROR 1128: Cannot find field dryTemp - hadoop
my pig was run code temperature and me an error, put the code below and the error to facilitate the understanding of my problem occurred.
the error is in line 38 column 15, tried to delete the dryTemp, but also gave another error.
Code:
--Load files into relations
month1 = LOAD 'hdfs:/data/big/data/weather/weather/201201hourly.txt' USING PigStorage(',');
month2 = LOAD 'hdfs:/data/big/data/weather/weather/201202hourly.txt' USING PigStorage(',');
month3 = LOAD 'hdfs:/data/big/data/weather/weather/201203hourly.txt' USING PigStorage(',');
month4 = LOAD 'hdfs:/data/big/data/weather/weather/201204hourly.txt' USING PigStorage(',');
month5 = LOAD 'hdfs:/data/big/data/weather/weather/201205hourly.txt' USING PigStorage(',');
month6 = LOAD 'hdfs:/data/big/data/weather/weather/201206hourly.txt' USING PigStorage(',');
--Combine relations
months = UNION month1, month2, month3, month4, month5, month6;
/* Splitting relations
SPLIT months INTO
splitMonth1 IF SUBSTRING(date, 4, 6) == '01',
splitMonth2 IF SUBSTRING(date, 4, 6) == '02',
splitMonth3 IF SUBSTRING(date, 4, 6) == '03',
splitRest IF (SUBSTRING(date, 4, 6) == '04' OR SUBSTRING(date, 4, 6) == '04');
*/
/* Joining relations
stations = LOAD 'hdfs:/data/big/data/QCLCD201211/stations.txt' USING PigStorage() AS (id:int, name:chararray)
JOIN months BY wban, stations by id;
*/
--filter out unwanted data
clearWeather = FILTER months BY skyCondition == 'CLR';
--Transform and shape relation
shapedWeather = FOREACH clearWeather GENERATE date, SUBSTRING(date, 0, 4) as year, SUBSTRING(date, 4, 6) as month, SUBSTRING(date, 6, 8) as day, skyCondition, dryTemp;
--Group relation specifying number of reducers
groupedByMonthDay = GROUP shapedWeather BY (month, day) PARALLEL 10;
--Aggregate relation
aggedResults = FOREACH groupedByMonthDay GENERATE group as MonthDay, AVG(shapedWeather.dryTemp), MIN(shapedWeather.dryTemp), MAX(shapedWeather.dryTemp), COUNT(shapedWeather.dryTemp) PARALLEL 10;
--Sort relation
sortedResults = ORDER aggedResults BY $1 DESC;
--Store results in HDFS
STORE sortedResults INTO 'hdfs:/data/big/data/weather/pigresults' USING PigStorage(':');
Put down the error, he was kinda big, still do not know much about the pig, I'm still studying, I believe that error has to do with the type of variable that is not recognized but do not know fix it hopefully help me.
Error:
ERROR 1128: Cannot find field dryTemp in :bytearray,year:chararray,month:chararray,day:chararray,:bytearray,:bytearray
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1000: Error during parsing. Cannot find field dryTemp in :bytearray,year:chararray,month:chararray,day:chararray,:bytearray,:bytearray
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1691)
at org.apache.pig.PigServer$Graph.access$000(PigServer.java:1411)
at org.apache.pig.PigServer.parseAndBuild(PigServer.java:344)
at org.apache.pig.PigServer.executeBatch(PigServer.java:369)
at org.apache.pig.PigServer.executeBatch(PigServer.java:355)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:140)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:202)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:607)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
Caused by: Failed to parse: Pig script failed to parse:
<file Documentos/pig/weather.pig, line 38, column 15> pig script failed to validate: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1128: Cannot find field dryTemp in :bytearray,year:chararray,month:chararray,day:chararray,:bytearray,:bytearray
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:196)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1678)
... 15 more
Caused by:
<file Documentos/pig/weather.pig, line 38, column 15> pig script failed to validate: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1128: Cannot find field dryTemp in :bytearray,year:chararray,month:chararray,day:chararray,:bytearray,:bytearray
at org.apache.pig.parser.LogicalPlanBuilder.buildForeachOp(LogicalPlanBuilder.java:1017)
at org.apache.pig.parser.LogicalPlanGenerator.foreach_clause(LogicalPlanGenerator.java:15870)
at org.apache.pig.parser.LogicalPlanGenerator.op_clause(LogicalPlanGenerator.java:1933)
at org.apache.pig.parser.LogicalPlanGenerator.general_statement(LogicalPlanGenerator.java:1102)
at org.apache.pig.parser.LogicalPlanGenerator.statement(LogicalPlanGenerator.java:560)
at org.apache.pig.parser.LogicalPlanGenerator.query(LogicalPlanGenerator.java:421)
at org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:188)
... 16 more
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1128: Cannot find field dryTemp in :bytearray,year:chararray,month:chararray,day:chararray,:bytearray,:bytearray
at org.apache.pig.newplan.logical.expression.DereferenceExpression.translateAliasToPos(DereferenceExpression.java:215)
at org.apache.pig.newplan.logical.expression.DereferenceExpression.getFieldSchema(DereferenceExpression.java:149)
at org.apache.pig.newplan.logical.optimizer.FieldSchemaResetter.execute(SchemaResetter.java:264)
at org.apache.pig.newplan.logical.expression.AllSameExpressionVisitor.visit(AllSameExpressionVisitor.java:148)
at org.apache.pig.newplan.logical.expression.DereferenceExpression.accept(DereferenceExpression.java:84)
at org.apache.pig.newplan.ReverseDependencyOrderWalker.walk(ReverseDependencyOrderWalker.java:70)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visitAll(SchemaResetter.java:67)
at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:122)
at org.apache.pig.newplan.logical.relational.LOGenerate.accept(LOGenerate.java:245)
at org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75)
at org.apache.pig.newplan.logical.optimizer.SchemaResetter.visit(SchemaResetter.java:114)
at org.apache.pig.parser.LogicalPlanBuilder.buildForeachOp(LogicalPlanBuilder.java:1015)
... 22 more
Here are a few lines of the file 201211 hourly.txt:
WBAN,Date,Time,StationType,SkyCondition,SkyConditionFlag,Visibility,VisibilityFlag,WeatherType,WeatherTypeFlag,DryBulbFarenheit,DryBulbFarenheitFlag,DryBulbCelsius,DryBulbCelsiusFlag,WetBulbFarenheit,WetBulbFarenheitFlag,WetBulbCelsius,WetBulbCelsiusFlag,DewPointFarenheit,DewPointFarenheitFlag,DewPointCelsius,DewPointCelsiusFlag,RelativeHumidity,RelativeHumidityFlag,WindSpeed,WindSpeedFlag,WindDirection,WindDirectionFlag,ValueForWindCharacter,ValueForWindCharacterFlag,StationPressure,StationPressureFlag,PressureTendency,PressureTendencyFlag,PressureChange,PressureChangeFlag,SeaLevelPressure,SeaLevelPressureFlag,RecordType,RecordTypeFlag,HourlyPrecip,HourlyPrecipFlag,Altimeter,AltimeterFlag
03011,20120101,0015,0,CLR, ,10.00, , , ,23, ,-5.0, ,15, ,-9.5, ,-9, ,-23.0, , 24, , 5, ,120, , , ,21.70, , , , , ,M, ,AA, , , ,30.43,
03011,20120101,0035,0,CLR, ,10.00, , , ,21, ,-6.0, ,14, ,-10.2, ,-9, ,-23.0, , 26, , 6, ,130, , , ,21.70, , , , , ,M, ,AA, , , ,30.43,
03011,20120101,0055,0,CLR, ,10.00, , , ,21, ,-6.0, ,13, ,-10.5, , -13, ,-25.0, , 21, , 0, ,000, , , ,21.71, , , , , ,M, ,AA, , , ,30.44,
03011,20120101,0115,0,CLR, ,10.00, , , ,21, ,-6.0, ,14, ,-10.1, ,-8, ,-22.0, , 27, , 0, ,000, , , ,21.71, , , , , ,M, ,AA, , , ,30.44,
03011,20120101,0135,0,CLR, ,10.00, , , ,21, ,-6.0, ,13, ,-10.4, , -11, ,-24.0, , 23, , 0, ,000, , , ,21.72, , , , , ,M, ,AA, , , ,30.45,
03011,20120101,0155,0,CLR, ,10.00, , , ,21, ,-6.0, ,13, ,-10.5, , -13, ,-25.0, , 21, , 6, ,130, , , ,21.72, , , , , ,M, ,AA, , , ,30.45,
03011,20120101,0215,0,CLR, ,10.00, , , ,21, ,-6.0, ,14, ,-10.2, ,-9, ,-23.0, , 26, , 5, ,090, , , ,21.73, , , , , ,M, ,AA, , , ,30.46,
03011,20120101,0235,0,CLR, ,10.00, , , ,21, ,-6.0, ,14, ,-10.2, ,-9, ,-23.0, , 26, , 6, ,120, , , ,21.74, , , , , ,M, ,AA, , , ,30.47,
03011,20120101,0255,0,CLR, ,10.00, , , ,21, ,-6.0, ,13, ,-10.4, , -11, ,-24.0, , 23, , 7, ,130, , , ,21.74, , , , , ,M, ,AA, , , ,30.48,
03011,20120101,0315,0,CLR, ,10.00, , , ,23, ,-5.0, ,15, ,-9.4, ,-8, ,-22.0, , 25, , 9, ,120, , , ,21.74, , , , , ,M, ,AA, , , ,30.47,
03011,20120101,0335,0,CLR, ,10.00, , , ,23, ,-5.0, ,15, ,-9.4, ,-8, ,-22.0, , 25, , 8, ,120, , , ,21.74, , , , , ,M, ,AA, , , ,30.47,
03011,20120101,0355,0,CLR, ,10.00, , , ,21, ,-6.0, ,14, ,-10.2, ,-9, ,-23.0, , 26, , 7, ,120, , , ,21.73, , , , , ,M, ,AA, , , ,30.46,
03011,20120101,0415,0,CLR, ,10.00, , , ,23, ,-5.0, ,14, ,-9.7, , -13, ,-25.0, , 19, , 7, ,130, , , ,21.73, , , , , ,M, ,AA, , , ,30.46,
I have done few modification in your script,
1. Load the data with proper schema (you can change the datatype of each field according to your need)
2. Optimized all the 6 loads into 1 load.
3. Removed the commented code
I have tested the below pig script with your input and its working fine, pasted the output also.
PigScript:
--Load all the files into relations
months = LOAD 'hdfs:/data/big/data/weather/weather/20120[1-6]hourly.txt' USING PigStorage(',') AS (WBAN:int,Date:chararray,Time:chararray,StationType:int,SkyCondition:chararray,SkyConditionFlag,Visibility,VisibilityFlag,WeatherType,WeatherTypeFlag,DryBulbFarenheit:int,DryBulbFarenheitFlag,DryBulbCelsius:double,DryBulbCelsiusFlag,WetBulbFarenheit:int,WetBulbFarenheitFlag,WetBulbCelsius:double,WetBulbCelsiusFlag,DewPointFarenheit,DewPointFarenheitFlag,DewPointCelsius,DewPointCelsiusFlag,RelativeHumidity,RelativeHumidityFlag,WindSpeed,WindSpeedFlag,WindDirection,WindDirectionFlag,ValueForWindCharacter,ValueForWindCharacterFlag,StationPressure,StationPressureFlag,PressureTendency,PressureTendencyFlag,PressureChange,PressureChangeFlag,SeaLevelPressure,SeaLevelPressureFlag,RecordType,RecordTypeFlag,HourlyPrecip,HourlyPrecipFlag,Altimeter,AltimeterFlag);
--filter out unwanted data
clearWeather = FILTER months BY SkyCondition == 'CLR';
--Transform and shape relation
shapedWeather = FOREACH clearWeather GENERATE Date,
SUBSTRING(Date,0,4) AS year,
SUBSTRING(Date,4,6) AS month,
SUBSTRING(Date,6,8) AS day,
SkyCondition,
DryBulbFarenheit AS dryTemp;
--Group relation specifying number of reducers
groupedByMonthDay = GROUP shapedWeather BY (month, day) PARALLEL 10;
--Aggregate relation
aggedResults = FOREACH groupedByMonthDay GENERATE group as MonthDay, AVG(shapedWeather.dryTemp), MIN(shapedWeather.dryTemp), MAX(shapedWeather.dryTemp), COUNT(shapedWeather.dryTemp) PARALLEL 10;
--Sort relation
sortedResults = ORDER aggedResults BY $1 DESC;
--Store results in HDFS
STORE sortedResults INTO 'hdfs:/data/big/data/weather/pigresults' USING PigStorage(':');
Output: (based on your above input samples)
(01,01):21.615384615384617:21:23:13
MonthDay:(01,01)
Avg:21.615384615384617
Min:21
Max:23
Count:13
It looks like you are loading 'month1','month2' etc without specifying the schema (where you should specify 'dryTemp'). You may try something like:
month1 = LOAD 'hdfs:/data/big/data/weather/201201hourly.txt' USING PigStorage(',')
AS (wban,year_month_day,time,station_type,maint_indic,
sky_cond,visibility,weather_type,dryTemp);
Similarly for all the other months.
Thanks
Related
Ways to optimize DB2 Query
I have used the below SQL statement to create a DB2 view. Can someone help me on how to improve this view performance as it runs slow sometimes.
The one thing I noted is that it runs slow the first time but then for subsequent runs, it picks up speed. I have analyzed with the visual explain but when it runs slow, I don't see any index advises. So not really sure how to optimize this.
-- Generate SQL
-- Version: V7R4M0 190621
-- Generated on: 02/09/21 03:08:31
-- Relational Database: BIGBLUE
-- Standards Option: Db2 for i
CREATE VIEW IESQAFILE.PSCMPORDVW
AS
WITH INPROGRESS AS
(
SELECT
DIODR#
, DIDISP
, DIUNIT
, DISTST
, DIAPRV
, DIETAD
, DITRLR AS TRAILER_ID
, DIDR1
, DIETAT
FROM
IESQAFILE.LOAD
WHERE
DIETAD <> 0
AND DIETAT <> '0000'
ORDER BY
1
)
, STOPGROUP AS
(
SELECT
SOORD STOPORDER
, COUNT(*) STOPSREMAIN
, MIN(SOSTP#) NEXTSTOP
, MAX(SOAPPR) APPTREQ
FROM
PAVT.STOP
INNER JOIN
INPROGRESS
ON
DIODR# = SOORD
WHERE
SOARDT = 0
GROUP BY
SOORD
ORDER BY
1
)
, STOPAPPTS AS
(
SELECT
SOORD APPTORDER
, SOCUST STOPCUST
, SOEDA ETADATE
, SOETA ETATIME
, SOADT1 EARLYDATE
, SOATM1 EARLYTIME
, SOADT2 LATEDATE
, SOATM2 LATETIME
, SOCTYC NEXTCITY
, SOSTP# APPTSTOP
, SOST NEXTSTATE
FROM
IESQAFILE.STOPOFF
INNER JOIN
STOPGROUP
ON
STOPORDER = SOORD
AND NEXTSTOP = SOSTP#
)
SELECT
ORDER_NUMBER
, SHIPPER_ID
, SHIPPER_NAME
, SHIPPER_ADDRESS_1
, SHIPPER_ADDRESS_2
, SHIPPER_CITY
, SHIPPER_ST
, SHIPPER_ZIP
, SHIPPER_ZIP_EXT
, LOAD_AT_ID
, LOAD_AT_NAME
, LOAD_AT_ADDRESS_1
, LOAD_AT_ADDRESS_2
, LOAD_AT_CITY
, LOAD_AT_ST
, LOAD_AT_ZIP
, LOAD_AT_ZIP_EXT
, LOAD_AT_LATITUDE
, LOAD_AT_LONGITUDE
, EARLY_PU_DATE_TIME
, LATE_PU_DATE_TIME
, EARLY_DELV_DATE_TIME
, EST_REVENUE
, ORDER_DIV
, CONSIGNEE_ID
, CONSIGNEE_NAME
, CONSIGNEE_ADDRESS_1
, CONSIGNEE_ADDRESS_2
, CONSIGNEE_CITY
, CONSIGNEE_ST
, CONSIGNEE_ZIP
, CONSIGNEE_ZIP_EXT
, CONSIGNEE_LATITUDE
, CONSIGNEE_LONGITUDE
, TRAILER_TYPE
, ORDER_MESSAGE
, ADDITIONAL_STOPS
, CMDTY_CODE
, CMDTY_DESCRIPTION
, ORDER_MILES
, ORDER_WGT
, ORIGIN_CITY_CODE
, ORIGIN_CITY
, ORIGIN_ST
, DEST_CITY_CODE
, DEST_CITY_NAME
, DEST_ST
, PICK_UP_AREA
, PLAN_INFO
, NUMBER_LDS
, NUMBER_DISP
, SHIP_DATE_TIME
, NEW_PICKUP_AREA
, EQUIPMENT_NUMBER
, APPT_REQ
, APPT_MADE
, PRE_T_SEQ
, PRE_T_AREA
, LOAD_DISPATCHED
, CUST_SERV_REP
, NEGOTIATIONS
,
(
CASE
WHEN UNUNIT IS NOT NULL
THEN UNUNIT
ELSE ' '
END
)
UNIT_DISPATCHED
,
(
CASE
WHEN UNSUPR IS NOT NULL
THEN UNSUPR
ELSE ' '
END
)
DRIVER_MGR_CODE
, COALESCE(SUPNAM, ' ') DRIVER_MGR_NAME
,
(
CASE
WHEN UNFMGR IS NOT NULL
THEN UNFMGR
ELSE ' '
END
)
FLEET_MGR_CODE
, COALESCE(FLTNAM,' ') FLEET_MGR_NAME
,
(
CASE
WHEN UNTRL1 IS NOT NULL
THEN UNTRL1
ELSE ' '
END
)
TRAILER_ID,
DIDISP DISPATCH_NUMBER,
(COALESCE(BCMCNEW, ' ')) FED_MC_ID,
DIUNIT DISPATCHED_UNIT,
CASE
WHEN UNETAD <> 0
AND UNETAT = ''
THEN SMIS.CVTDATETIM(CHAR(UNETAD),'0000', (
SELECT
SUBSTR(DATA_AREA_VALUE, 1109, 2) AS TIMEZONE
FROM
TABLE(QSYS2.DATA_AREA_INFO('COMPAN', '*LIBL'))
)
)
WHEN UNETAD <> 0
THEN SMIS.CVTDATETIM(CHAR(UNETAD),UNETAT, (
SELECT
SUBSTR(DATA_AREA_VALUE, 1109, 2) AS TIMEZONE
FROM
TABLE(QSYS2.DATA_AREA_INFO('COMPAN', '*LIBL'))
)
)
WHEN UNETAD = 0
THEN '0000-00-00T00:00:00-00:00'
END AS ETA_DATE_TIME,
NEXTSTOP , CASE
WHEN SOARDT <> 0
AND SOARTM = ''
THEN SMIS.CVTDATETIM(CHAR(SOARDT),'0000', (
SELECT
SUBSTR(DATA_AREA_VALUE, 1109, 2) AS TIMEZONE
FROM
TABLE(QSYS2.DATA_AREA_INFO('COMPAN', '*LIBL'))
)
)
WHEN SOARDT <> 0
THEN SMIS.CVTDATETIM(CHAR(SOARDT),SOARTM, (
SELECT
SUBSTR(DATA_AREA_VALUE, 1109, 2) AS TIMEZONE
FROM
TABLE(QSYS2.DATA_AREA_INFO('COMPAN', '*LIBL'))
)
)
WHEN SOARDT = 0
THEN '0000-00-00T00:00:00-00:00'
END AS STOP_ARRIVAL_DATE_TIME
, CASE
WHEN SOLUDT <> 0
AND SOLUTM = ''
THEN SMIS.CVTDATETIM(CHAR(SOLUDT),'0000', (
SELECT
SUBSTR(DATA_AREA_VALUE, 1109, 2) AS TIMEZONE
FROM
TABLE(QSYS2.DATA_AREA_INFO('COMPAN', '*LIBL'))
)
)
WHEN SOLUDT <> 0
THEN SMIS.CVTDATETIM(CHAR(SOLUDT),SOLUTM, (
SELECT
SUBSTR(DATA_AREA_VALUE, 1109, 2) AS TIMEZONE
FROM
TABLE(QSYS2.DATA_AREA_INFO('COMPAN', '*LIBL'))
)
)
WHEN SOLUDT = 0
THEN '0000-00-00T00:00:00-00:00'
END AS STOP_DEPART_DATE_TIME
, ORBAMT ORDER_INV_AMT
, ORARST AR_STATUS_FLAG
, DISTST SETTLEMENT_FLAG
, DIAPRV APPROVED_FOR_PAY
, BCCARR CARRIER_CODE
, BCNAME CARRIER_NAME
, BCADDR CARRIER_ADDRESS_1
, BCADR2 CARRIER_ADDRESS_2
, BCCITY CARRIER_CITY
, BCST CARRIER_ST
, BCZIP CARRIER_ZIP
FROM
INPROGRESS
INNER JOIN
IESQAFILE.PSMAINORVW A
ON
DIODR# = ORDER_NUMBER
AND DIDISP = NUMBER_DISP
AND
(
SUBSTR(ORDER_NUMBER, 1, 2) <> 'DH'
AND SUBSTR(ORDER_NUMBER, 1, 1) <> 'M'
)
LEFT OUTER JOIN
IESQAFILE.STOPOFF
ON
DIODR# = SOORD
AND SOSTP# = 90
LEFT OUTER JOIN
IESQAFILE.LMCARR
ON
DIUNIT = BCCARR
LEFT OUTER JOIN
IESQAFILE.MMILES
ON
MMORD# = DIODR#
AND MMRECTYPE = 'D'
AND MMDSP# = DIDISP
EXCEPTION JOIN
IESQAFILE.ORDBILL B
ON
B.ORODR# = DIODR#
AND B.ORSEQ = ' '
AND ORARST = '1'
LEFT OUTER JOIN
STOPGROUP
ON
STOPORDER = DIODR#
LEFT OUTER JOIN
STOPAPPTS
ON
APPTORDER = STOPORDER
AND APPTSTOP = NEXTSTOP
LEFT OUTER JOIN
IESQAFILE.UNITS
ON
UNUNIT = DIUNIT
AND UNORD# = ORDER_NUMBER
LEFT OUTER JOIN
IESQAFILE.SUPMAST
ON
SUPCDE = UNSUPR
LEFT OUTER JOIN
IESQAFILE.FLTMAST
ON
UNFMGR = FLTCDE
WHERE
DIETAD <> 0
AND DIETAT <> '0000'
RCDFMT PSCMPORDVW ;
Without seeing the Visual Explain (VE) data (and it's not really something easily sharable here) giving you a "magic wand" answer not likely. I'm surprised you don't see any indexes advised. But even so, you should be able to compare the VE data between "fast" and "slow" runs to see where the differences are. Make sure you have "View"-->"Highlighting"-->"Expensive Icons"-->... turned on. Also check "Options"-->"Attribute Detail" and "Options"-->"Graph Detail"... Always a good idea to be using the latest and greatest version (1.1.8.6) of Access Client Solutions (ACS) which includes the Run SQL Script & VE components. A couple of things jump out, an Encoded Vector Index (EVI) that include aggregates(count,min,max) might help here... select soord stoporder ,count(*) stopsremain ,min(sostp#) nextstop ,max(soappr) apptreq from pavt.stop And index (or two) with derived columns substr(order_number,....) might help here: from inprogress inner join iesqafile.psmainorvw a on diodr# = order_number and didisp = number_disp and (substr(order_number, 1, 2) <> 'DH' and substr(order_number, 1, 1) <> 'M') Again VE is your friend, you'll have to dig into what's going on and where the time is being spent. Break the various parts of your statement down and try to optimize individual components when possible. Hopefully, the gains will remain when you put it back together. One thing to remember, a view is performance neutral. It neither helps nor hurts performance. You don't give an example of how you access the view, but have you considered making use of user defined table functions (UDTF)? While the query engine attempts to push down selection criteria, a UDTF's parameters make it easy for you to explicitly include selection criteria. You might even benefit from encapsulating the statement in a UDTF and building the view over the UDTF. You can see an example (via Generate SQL Source) for the message_queue_info view & UDTF in QSYS2 that IBM provided.
I have Oracle Join query that picks data very slow
I have an Oracle join query that picks data very slow. It is like 1000 rows for 7 mins. Please could you help in writing the code in a different way so the data is pulled faster. The next steps for it is using the Select values and dumping the data into MySQL table. I am using Pentaho tool here. Thanks select null id, ss.ILOAN_CODE , ss.INST_NUM , ss.INST_AMT , ss.INST_PRINCIPAL , ss.INST_INTEREST , ss.BALANCE_PRINCIPAL , ss.INST_DUE_DATE , ss.PAID_FLAG , ss.LATE_FEE , ss.PAYMENT_DATE , ss.INST_AMT_PAID , ss.INST_AMT_DUE , ss.REV_CHECK_NUM , ss.REV_CHECK_AMT , ss.CREATED_BY , ss.DATE_CREATED , ss.UPDATED_BY , ss.DATE_UPDATED , ss.INST_DAYS , ss.MATURED_INTEREST , ss.UNPAID_INTEREST , ss.ADJ_INST_PRINCIPAL , ss.ADJ_INST_AMT , ss.ADJ_INST_INTEREST , ss.ADJ_BALANCE_PRINCIPAL , ss.ADJ_MATURED_INTEREST , ss.ADJ_UNPAID_INTEREST , ss.IS_PRINTED , ss.RTN_FEE_AMT , ss.WAIVE_FEE_AMT , ss.LATE_FEE_AMT , ss.APR_BALANCE_PRINCIPAL , ss.ACHDEPOSIT_DATE , ss.ACHRETURN_DATE , ss.ACHCLEAR_DATE , ss.APR_INST_INTEREST , ss.APR_UNPAID_INTEREST , ss.CSO_FEE , ss.MATURED_CSO_FEE , ss.UNPAID_CSO_FEE , ss.CSO_FEE_BALANCE from ST_IL_SCHEDULE ss, ST_IL_MASTER sm, BO_MASTER bm where sm.iloan_code = ss.iloan_code and sm.bo_code = bm.bo_code and ss.ILOAN_CODE in (select distinct loan_Number from SVP_LOAN_MASTER_INVENTORY)
and ss.ILOAN_CODE in (select distinct loan_Number from SVP_LOAN_MASTER_INVENTORY) This is candidate for being slow. You don't need distinct here and also please use explicit join for readability. Try: Select null id, ss.ILOAN_CODE , ss.INST_NUM , ss.INST_AMT , ss.INST_PRINCIPAL , ss.INST_INTEREST , ss.BALANCE_PRINCIPAL , ss.INST_DUE_DATE , ss.PAID_FLAG , ss.LATE_FEE , ss.PAYMENT_DATE , ss.INST_AMT_PAID , ss.INST_AMT_DUE , ss.REV_CHECK_NUM , ss.REV_CHECK_AMT , ss.CREATED_BY , ss.DATE_CREATED , ss.UPDATED_BY , ss.DATE_UPDATED , ss.INST_DAYS , ss.MATURED_INTEREST , ss.UNPAID_INTEREST , ss.ADJ_INST_PRINCIPAL , ss.ADJ_INST_AMT , ss.ADJ_INST_INTEREST , ss.ADJ_BALANCE_PRINCIPAL , ss.ADJ_MATURED_INTEREST , ss.ADJ_UNPAID_INTEREST , ss.IS_PRINTED , ss.RTN_FEE_AMT , ss.WAIVE_FEE_AMT , ss.LATE_FEE_AMT , ss.APR_BALANCE_PRINCIPAL , ss.ACHDEPOSIT_DATE , ss.ACHRETURN_DATE , ss.ACHCLEAR_DATE , ss.APR_INST_INTEREST , ss.APR_UNPAID_INTEREST , ss.CSO_FEE , ss.MATURED_CSO_FEE , ss.UNPAID_CSO_FEE , ss.CSO_FEE_BALANCE from ST_IL_SCHEDULE ss, inner join ST_IL_MASTER sm on (sm.iloan_code = ss.iloan_code) inner join BO_MASTER bm on (sm.bo_code = bm.bo_code) inner join SVP_LOAN_MASTER_INVENTORY slm on (ss.loan_code = slm.loan number) If that not helps please consider creating indexes on columns used in join.
Conversion from Oracle into SQL Server
I am trying to interpret Oracle script into SQL script and I see that I am converting the joins wrong as Oracle is using multiple instances of the same table(see orguserfield_c, orguserfield_e, etc in select statment). Could anyone please help me in converting the Oracle into SQL script. Thanks
ORACLE SCRIPT:
select distinct
wh_acctcommon.effdate
, wh_acctcommon.acctnbr
, wh_acctcommon.acctclosecurrmonthyn
, wh_acctcommon.acctofficer
, wh_acctcommon.acctofficernbr
, wh_acctcommon.acctopencurrmonthyn
, wh_acctcommon.notebal
, wh_acctcommon.branchname
, wh_acctcommon.branchorgnbr
, orguserfield_c.value branch_internal_no
, orguserfield_e.value branch_status
, orguserfield_h.value branch_hub_no
, orguserfield_i.value metro_micro
, orguserfield_j.value division
, userfieldvalue_l.userfieldvaluedesc division_name
, orguserfield.value region
, userfieldvalue.userfieldvaluedesc region_name
, wh_acctcommon.primaryownercity
, wh_acctcommon.closedate
, wh_acctcommon.compoundcalpercd
, wh_acctcommon.contractdate
, wh_acctcommon.datelastmaint
, wh_acctcommon.ownername
, wh_acctcommon.bankorgnbr
, wh_acctcommon.intbase
, wh_acctcommon.intmethcd
, wh_acctcommon.noteintrate
, wh_acctcommon.ownersortname
, wh_acctcommon.loanofficer
, wh_acctcommon.loanofficersnbr
, wh_acctcommon.mjaccttypcd
, wh_acctcommon.managingofficer
, wh_acctcommon.managingofficernbr
, acctacctrolepers.persnbr SBB_Portfolio_Mgr_Nbr
, persview.fullname SBB_Portfolio_Mgr
, acctacctrolepers_c.persnbr Orig_Loan_Officer_Nbr
, persview_d.fullname Orig_Loan_Officer
, acctacctrolepers_e.persnbr Collection_Officer_Nbr
, persview_e.fullname Collection_Officer
, wh_acctcommon.datemat
, wh_acctcommon.intminbalamt
, wh_acctcommon.intmincalcbaltypcd
, wh_acctcommon.monthendyn
, wh_acctcommon.notemtdavgbal
, wh_acctcommon.nameaddr1
, wh_acctcommon.nameaddr2
, wh_acctcommon.nameaddr3
, wh_acctcommon.nameaddr4
, wh_acctcommon.nameaddr5
, wh_acctcommon.notenextratechangedate
, wh_acctcommon.noteopenamt
, wh_acctcommon.originatingperson
, wh_acctcommon.origpersnbr
, wh_acctcommon.bookbalance
, wh_acctcommon.businessphone
, wh_acctcommon.homephone
, wh_acctcommon.currmiaccttypcd
, wh_acctcommon.product
, wh_acctcommon.calcbaltypcd
, wh_acctcommon.noteratechangecalpercd
, wh_acctcommon.daysmethcd
, wh_acctcommon.noteintcalcschednbr
, wh_acctcommon.ratetypcd
, wh_acctcommon.primaryownerstate
, wh_acctcommon.curracctstatcd
, wh_acctcommon.curracctstateffdate
, wh_acctcommon.taxidnbr
, wh_acctcommon.taxrptfororgnbr
, wh_acctcommon.taxrptforpersnbr
, wh_acctcommon.currterm
, wh_acctcommon.primaryownerzipcd
, wh_acctcommon.primaryownerzipcdsuff
from OSIBANK.wh_acctcommon
, OSIBANK.orguserfield
, OSIBANK.orguserfield orguserfield_c
, OSIBANK.orguserfield orguserfield_e
, OSIBANK.orguserfield orguserfield_h
, OSIBANK.orguserfield orguserfield_i
, OSIBANK.orguserfield orguserfield_j
, OSIBANK.userfieldvalue
, OSIBANK.userfieldvalue userfieldvalue_l
, OSIBANK.acctacctrolepers
, OSIBANK.persview
, OSIBANK.acctacctrolepers acctacctrolepers_c
, OSIBANK.persview persview_d
, OSIBANK.acctacctrolepers acctacctrolepers_e
, OSIBANK.persview persview_e
where ( wh_acctcommon.branchorgnbr = orguserfield.orgnbr(+)
and upper(orguserfield.userfieldcd(+)) = 'WREG'
and orguserfield.userfieldcd = userfieldvalue.userfieldcd(+)
and orguserfield.value = userfieldvalue.userfieldvalue(+)
)
and ( wh_acctcommon.branchorgnbr = orguserfield_c.orgnbr(+)
and upper(orguserfield_c.userfieldcd(+)) = 'WBRN'
)
and ( wh_acctcommon.branchorgnbr = orguserfield_e.orgnbr(+)
and upper(orguserfield_e.userfieldcd(+)) = 'WBRS'
)
and ( wh_acctcommon.branchorgnbr = orguserfield_h.orgnbr(+)
and upper(orguserfield_h.userfieldcd(+)) = 'WHBN'
)
and ( wh_acctcommon.branchorgnbr = orguserfield_i.orgnbr(+)
and upper(orguserfield_i.userfieldcd(+)) = 'WSIZ'
)
and ( wh_acctcommon.branchorgnbr = orguserfield_j.orgnbr(+)
and upper(orguserfield_j.userfieldcd(+)) = 'WDIV'
and orguserfield_j.userfieldcd = userfieldvalue_l.userfieldcd(+)
and orguserfield_j.value = userfieldvalue_l.userfieldvalue(+)
)
and ( wh_acctcommon.acctnbr = acctacctrolepers.acctnbr(+)
and upper(acctacctrolepers.acctrolecd(+)) = 'PMGR'
and acctacctrolepers.persnbr = persview.persnbr(+)
)
and ( wh_acctcommon.acctnbr = acctacctrolepers_c.acctnbr(+)
and acctacctrolepers_c.acctrolecd(+) = 'OLOF'
and acctacctrolepers_c.persnbr = persview_d.persnbr(+)
)
and ( wh_acctcommon.acctnbr = acctacctrolepers_e.acctnbr(+)
and acctacctrolepers_e.acctrolecd(+) = 'COFF'
and acctacctrolepers_e.persnbr = persview_e.persnbr(+)
);
SQL SCRIPT:
select
wh_acctcommon.[EFFDATE]
, wh_acctcommon.acctnbr
, wh_acctcommon.acctclosecurrmonthyn
, wh_acctcommon.acctofficer
, wh_acctcommon.acctofficernbr
, wh_acctcommon.acctopencurrmonthyn
, wh_acctcommon.notebal
, wh_acctcommon.branchname
, wh_acctcommon.branchorgnbr
,OUF.value AS 'branch_internal_no'
, OUF.value AS 'branch_status'
, OUF.value AS 'branch_hub_no'
, OUF.value AS 'metro_micro'
, OUF.value AS 'division'
, UFV.[USERFIELDVALUEDESC] AS 'division_name'
, OUF.value AS 'region'
, UFV.userfieldvaluedesc AS 'region_name'
, wh_acctcommon.primaryownercity
, wh_acctcommon.closedate
, wh_acctcommon.compoundcalpercd
, wh_acctcommon.contractdate
, wh_acctcommon.datelastmaint
, wh_acctcommon.ownername
, wh_acctcommon.bankorgnbr
, wh_acctcommon.intbase
, wh_acctcommon.intmethcd
, wh_acctcommon.noteintrate
, wh_acctcommon.ownersortname
, wh_acctcommon.loanofficer
, wh_acctcommon.loanofficersnbr
, wh_acctcommon.mjaccttypcd
, wh_acctcommon.managingofficer
, wh_acctcommon.managingofficernbr
, AARP.persnbr AS 'SBB_Portfolio_Mgr_Nbr'
, PV.fullname as 'SBB_Portfolio_Mgr'
, AARP.persnbr AS 'Orig_Loan_Officer_Nbr'
, PV.fullname AS 'Orig_Loan_Officer'
, AARP.persnbr as 'Collection_Officer_Nbr'
, PV.fullname AS 'Collection_Officer'
, wh_acctcommon.datemat
, wh_acctcommon.intminbalamt
, wh_acctcommon.intmincalcbaltypcd
, wh_acctcommon.monthendyn
, wh_acctcommon.notemtdavgbal
, wh_acctcommon.nameaddr1
, wh_acctcommon.nameaddr2
, wh_acctcommon.nameaddr3
, wh_acctcommon.nameaddr4
, wh_acctcommon.nameaddr5
, wh_acctcommon.notenextratechangedate
, wh_acctcommon.noteopenamt
, wh_acctcommon.originatingperson
, wh_acctcommon.origpersnbr
, wh_acctcommon.bookbalance
, wh_acctcommon.businessphone
, wh_acctcommon.homephone
, wh_acctcommon.currmiaccttypcd
, wh_acctcommon.product
, wh_acctcommon.calcbaltypcd
, wh_acctcommon.noteratechangecalpercd
, wh_acctcommon.daysmethcd
, wh_acctcommon.noteintcalcschednbr
, wh_acctcommon.ratetypcd
, wh_acctcommon.primaryownerstate
, wh_acctcommon.curracctstatcd
, wh_acctcommon.curracctstateffdate
, wh_acctcommon.taxidnbr
, wh_acctcommon.taxrptfororgnbr
, wh_acctcommon.taxrptforpersnbr
, wh_acctcommon.currterm
, wh_acctcommon.primaryownerzipcd
, wh_acctcommon.primaryownerzipcdsuff
FROM
[DNA_Staging].[dbo].[WH_ACCTCOMMON] wh_acctcommon
LEFT OUTER JOIN [DNA_Staging].[dbo].orguserfield OUF
ON wh_acctcommon.branchorgnbr = OUF .orgnbr
and upper(OUF.userfieldcd) IN( 'WREG','WBRN','WBRS','WHBN','WSIZ','WDIV')
LEFT OUTER JOIN [DNA_Staging].[dbo].userfieldvalue UFV
ON OUF.userfieldcd = UFV.userfieldcd
and OUF.value = UFV.userfieldvalue
LEFT OUTER JOIN [DNA_Staging].[dbo].[ACCTACCTROLEPERS] AARP
ON wh_acctcommon.acctnbr = AARP.acctnbr
and upper(AARP.acctrolecd) IN ('PMGR','OLOF','COFF')
LEFT OUTER JOIN [DNA_Cleanup_DM].[dbo].PERSVIEW PV
ON AARP.persnbr = PV.persnbr
This is mostly about converting the pre-SQL99 Oracle proprietary joins into SQL99 joins. Rather than try to convert your actual code, here is some advice: The number of tables in the from clause should remain exactly the same in both queries. If a table is listed 5 times (with aliases) in the original query, you should have the same 5 instances in the revised query. A fatal flaw with your revised query is that you're trying to collapse these relationships using in. You simply can't do that and retain the meaning of the original query. Start with the first table and work down, converting each comma in the original from clause into an on. If a relationship in the original where clause contains the outer join indicator ((+)), then that relationship must go in the outer table's on clause. For instance, a.a = b.a(+) must become left join b on a.a = b.a.
Script to find multi level dependencies of a package
I've a package which references many objects from the same schema and other schemas. I want to find all the dependencies of the package. I can get only first level dependencies from user_dependencies. Also utldtree would give me the objects which are dependent on my current object.utldtree also gives only the referenced objects in the same schema. While I'm trying to find the solution for this on the net, I came across the following link http://rodgersnotes.wordpress.com/2012/01/05/notes-on-deptree/ where he mentioned that, he uses his own script to find the multi level dependencies of an object. Could you please help me out, how to proceed for such a script which will get us the multi-level dependencies of an object,(for example if the package is referencing views, then our script should mention the views and the tables/views upon which our view is build as we get in deptree)
You can use a connect by on user_dependencies for most cases. Determining dependencies Sample which works for any Oracle user since PUBLIC has been granted select access on user_dependencies: select name , type , prior name , prior type from user_dependencies start with name='BUBS#MUNT_EENHEDEN' and type='PACKAGE' connect by nocycle name = prior referenced_name and type = prior referenced_type Sample output Level 1: BUBS#MUNT_EENHEDEN PACKAGE Level 2: BUBS_MUNT_EENHEDEN_V VIEW BUBS#MUNT_EENHEDEN PACKAGE Level 3: BUBS#VERTALINGEN PACKAGE BUBS_MUNT_EENHEDEN_V VIEW Level 4: ITGEN_LANGUAGES_V VIEW BUBS#VERTALINGEN PACKAGE Complex scenarios For complex scenarios I've found it necessary to use an own view directly on the data dictionary. Do this only when you know what you are doing and what RDBMS version you want to support! For instance, datamodel versions introduced major changes in the data dictionary. Sample: create or replace force view itgen_object_tree_changes_r as select o_master.obj# ojt# , o_master.name ojt_name , o.mtime ojt_ref_mtime , o.name ojt_ref_name , o.owner# ojt_ref_owner# , decode ( o.type# , 0, 'NEXT OBJECT' , 1, 'INDEX' , 2, 'TABLE' , 3, 'CLUSTER' , 4, 'VIEW' , 5, 'SYNONYM' , 6, 'SEQUENCE' , 7, 'PROCEDURE' , 8, 'FUNCTION' , 9, 'PACKAGE' , 11, 'PACKAGE BODY' , 12, 'TRIGGER' , 13, 'TYPE' , 14, 'TYPE BODY' , 19, 'TABLE PARTITION' , 20, 'INDEX PARTITION' , 21, 'LOB' , 22, 'LIBRARY' , 23, 'DIRECTORY' , 24, 'QUEUE' , 28, 'JAVA SOURCE' , 29, 'JAVA CLASS' , 30, 'JAVA RESOURCE' , 32, 'INDEXTYPE' , 33, 'OPERATOR' , 34, 'TABLE SUBPARTITION' , 35, 'INDEX SUBPARTITION' , 40, 'LOB PARTITION' , 41, 'LOB SUBPARTITION' , 42, nvl ( ( select 'REWRITE EQUIVALENCE' from sys.sum$ s where s.obj# = o.obj# and bitand ( s.xpflags, 8388608 ) = 8388608 ), 'MATERIALIZED VIEW' ) , 43, 'DIMENSION' , 44, 'CONTEXT' , 46, 'RULE SET' , 47, 'RESOURCE PLAN' , 48, 'CONSUMER GROUP' , 51, 'SUBSCRIPTION' , 52, 'LOCATION' , 55, 'XML SCHEMA' , 56, 'JAVA DATA' , 57, 'EDITION' , 59, 'RULE' , 60, 'CAPTURE' , 61, 'APPLY' , 62, 'EVALUATION CONTEXT' , 66, 'JOB' , 67, 'PROGRAM' , 68, 'JOB CLASS' , 69, 'WINDOW' , 72, 'WINDOW GROUP' , 74, 'SCHEDULE' , 79, 'CHAIN' , 81, 'FILE GROUP' , 82, 'MINING MODEL' , 87, 'ASSEMBLY' , 90, 'CREDENTIAL' , 92, 'CUBE DIMENSION' , 93, 'CUBE' , 94, 'MEASURE FOLDER' , 95, 'CUBE BUILD PROCESS' , 'UNDEFINED' ) ojt_ref_type from sys.obj$ o , ( /* All dependencies from the object if there are any. */ select distinct connect_by_root d_obj# obj#, dep.p_obj# obj_ref# from sys.dependency$ dep connect by nocycle dep.d_obj# = prior dep.p_obj# start with dep.d_obj# in ( select obj.obj# from itgen_schemas_r sma, sys.obj$ obj where obj.owner# = sma.owner# ) union all /* Union all allowed, 'in' ignores duplicates. */ /* The object itself. */ select obj.obj# , obj.obj# from itgen_schemas_r sma , sys.obj$ obj where obj.owner# = sma.owner# ) deps , sys.obj$ o_master where o_master.obj# = deps.obj# and o.obj# = deps.obj_ref# -- -- View: itgen_object_tree_changes_r -- -- Overview of dependencies between a master object and all objects used by it. It can be used to analyze the reason why a project version views must be recalculated. -- -- Code (alias): ote_r -- -- Category: Hardcoded. -- -- Example: -- -- The object 'X' is invalid, since 'Y' is invalid. --
ORA-30928: "Connect by filtering phase runs out of temp tablespace"
i have created a query that is sued to display a data in a label. This particular query will then be stored into a program that we use. The query runs just fine until this morning when it returns the error ORA-30928: "Connect by filtering phase runs out of temp tablespace". I have Googled and found out that I can do any of the following: Include a NO FILTERING hint - but did not work properly Increase the temp tablespace - not applicable to me since this runs in a production server that I don't have any access to. Are there other ways to fix this? By the way, below is the query that I use. SELECT * FROM( SELECT gn.wipdatavalue , gn.containername , gn.l , gn.q , gn.d , gn.l2 , gn.q2 , gn.d2 , gn.l3 , gn.q3 , gn.d3 , gn.old , gn.qtyperbox , gn.productname , gn.slot , gn.dt , gn.ws_green , gn.ws_pnr , gn.ws_pcn , intn.mkt_number dsn , gn.low_number , gn.high_number , gn.msl , gn.baketime , gn.exptime , NVL(gn.q, 0) + NVL(gn.q2, 0) + NVL(gn.q3, 0) AS qtybox , row_number () over (partition by slot order by low_number) as n FROM ( SELECT tr.* , TO_NUMBER(SUBSTR(wipdatavalue, 1, INSTR (wipdatavalue || '-', '-') - 1)) AS low_number , TO_NUMBER(SUBSTR(wipdatavalue, 1 + INSTR ( wipdatavalue, '-'))) AS high_number , pm.msllevel MSL , pm.baketime BAKETIME , pm.expstime EXPTIME FROM trprinting tr JOIN CONTAINER c ON tr.containername = c.containername JOIN a_lotattributes ala ON c.containerid = ala.containerid JOIN product p ON c.productid = p.productid LEFT JOIN otherdb.pkg_main pm ON trim(p.brandname) = trim(pm.pcode) WHERE (c.containername = :lot OR tr.SLOT= :lot) )gn LEFT JOIN otherdb.intnr intn ON TRIM(gn.productname) = TRIM(intn.part_number) connect by level <= HIGH_NUMBER + 1 - LOW_NUMBER and LOW_NUMBER = prior LOW_NUMBER and prior SYS_GUID() is not null ORDER BY low_number,n ) WHERE n LIKE :n AND wipdatavalue LIKE :wip AND ROWNUM <= 300 AND wipdatavalue NOT LIKE 0 I am using Oracle 11g too. Thanks for the help everyone.