I can't find a complete example on how I would be able to load a CSV file directlry with a external table into a Sql Datawarehouse.
The file is on a Storage account https://tstodummy.blob.core.windows.net/
Blob container referencedata-in, folder csv-uploads, file something.csv.
This is my code
CREATE DATABASE SCOPED CREDENTIAL tstodummy_refdata_credential
WITH IDENTITY = 'USER',
SECRET = '....'
GO
CREATE EXTERNAL DATA SOURCE tstodummy_referencedata
WITH ( TYPE = HADOOP,
LOCATION = 'wasb://referencedata-in#tstodummy.blob.core.windows.net',
CREDENTIAL = tstodummy_refdata_credential);
GO
CREATE EXTERNAL FILE FORMAT aps_bma_referencedata_ff
WITH (FORMAT_TYPE = DELIMITEDTEXT,
FORMAT_OPTIONS(
FIELD_TERMINATOR = ';',
STRING_DELIMITER = '"',
FIRST_ROW = 2,
USE_TYPE_DEFAULT = True)
)
CREATE EXTERNAL TABLE [stg_aps_bma_refdata].[PlanDeMaintenance]
( [Version] VARCHAR(255) NULL
, [Description] VARCHAR(255) NULL
, [Date_Start] VARCHAR(255) NULL
, [Date_Stop] VARCHAR(255) NULL
) WITH ( LOCATION = '\referencedata-in\csv-uploads\PlanDeMaintanance'
, DATA_SOURCE = tstodummy_referencedata
, FILE_FORMAT = aps_bma_referencedata_ff
, REJECT_TYPE = VALUE
, REJECT_VALUE = 0
)
I've been playing with all kind of combinations in the Location ... But, Nogo
The error is
Msg 105002, Level 16, State 1, Line 26
EXTERNAL TABLE access failed because the specified path name '/referencedata-in/csv-uploads/PlanDeMaintanance.csv' does not exist. Enter a valid path and try again.
I can't see your storage structure, but I think you'll find that the problem is the inclusion of "/referencedata-in" in the external table location.
One small thing, you might also want to consider a "wasbs" prefix on your storage URL, so that SSL encryption is applied to the transfer.
Finally, this did the trick in case of others encounterig the sale troubles.
I did not yet add, in this code the remark I received, meanwhile done.
CREATE EXTERNAL DATA SOURCE tsto_referencedata
WITH ( TYPE = HADOOP,
LOCATION = 'wasb://referencedata-in#tsto.blob.core.windows.net',
CREDENTIAL = tsto_refdata_credential);
GO
CREATE EXTERNAL FILE FORMAT aps_bma_referencedata_ff
WITH (FORMAT_TYPE = DELIMITEDTEXT,
FORMAT_OPTIONS(
FIELD_TERMINATOR = ',',
STRING_DELIMITER = '"',
FIRST_ROW = 2,
USE_TYPE_DEFAULT = True)
)
CREATE EXTERNAL TABLE [stg_aps_bma_refdata].[PlanDeMaintenance.csv]
( [Version] VARCHAR(255) NULL
, [Description] VARCHAR(255) NULL
, [Date_Start] VARCHAR(255) NULL
, [Date_Stop] VARCHAR(255) NULL
) WITH ( LOCATION = '/csv-uploads/PlanDeMaintenance.csv'
, DATA_SOURCE = tsto_referencedata
, FILE_FORMAT = aps_bma_referencedata_ff
, REJECT_TYPE = VALUE
, REJECT_VALUE = 0
)
Related
Can someone let me know if there is an DECLARE equivalent in Databricks SQL
The SQL Code that I have trying to execute with Databricks SQL is as follows:
DECLARE
#EnrichedViewDatabase sysname,
#EnrichedViewSchema sysname,
#EnrichedColumnSuffix varchar(50),
#LanguageCode varchar(10),
#BaseTableSuffix varchar(50),
#PreviewOnly bit, --Indicate whether to preview the SQL Script (without creating the views) = 1 ; Create views = 0;
#CurrentDatabase sysname,
#CurrentDatabaseSchema sysname
SET #EnrichedViewDatabase = 'mydatabasenr1'
SET #EnrichedViewSchema = 'dbo'
SET #EnrichedColumnSuffix = 'code'
SET #LanguageCode = 1033
SET #BaseTableSuffix = ''
SET #PreviewOnly = 0
SET #CurrentDatabase = 'mydatabasenr2'
SET #CurrentDatabaseSchema = 'dbo'
DECLARE #ColumnMetadata nvarchar(MAX), #ColumnMetadataSQL nvarchar(MAX)
The above SQL gives me the following error:
mismatched input 'DECLARE'
== SQL ==
DECLARE
^^^
#EnrichedViewDatabase sysname,
#EnrichedViewSchema sysname,
#EnrichedColumnSuffix varchar(50),
#LanguageCode varchar(10),
#BaseTableSuffix varchar(50),
#PreviewOnly bit, --Indicate whether to preview the SQL Script (without creating the views) = 1
Any thoughts?
DECLARE is not supported in Databricks SQL. The equivalent code to the one which you are trying to achieve in the above case would be to use SET directly.
%sql
SET EnrichedViewDatabase = 'mydatabasenr1';
SET EnrichedViewSchema = 'dbo';
SET EnrichedColumnSuffix = 'code';
SET LanguageCode = 1033;
SET BaseTableSuffix = '';
SET PreviewOnly = 0 ;
SET CurrentDatabase = 'mydatabasenr2';
SET CurrentDatabaseSchema = 'dbo';
SET returns a key-value pair as result. To access the value using the respective key (assigned in the above procedure), you can do it in the following way:
%sql
select ${hiveconf:CurrentDatabase} as x,${hiveconf:EnrichedViewDatabase} as y,${hiveconf:LanguageCode} as z
I have a two node gg1 and gg2 setup of goldengate. Seem to work. However when I insert a row into my table T1
connect sender/oracle
create table t1 (f1 char, f2 char);
alter table t1 add constraint t1_i1 primary key (f1);
I get an error
2022-01-30T19:16:05.705-0500 WARNING OGG-01154 Oracle GoldenGate Delivery for Oracle, d_rep.prm: SQL error 1 mapping SENDER.T1 to SENDER.T1 OCI Error ORA-00001: unique constraint (SENDER.T1_I1) violated (status = 1), SQL <INSERT INTO "SENDER"."T1" ("F1","F2") VALUES (:a0,:a1)>.
enter image description here
enter image description here
enter image description here
here are prm files for gg1
# cat s_ext.prm
extract s_ext
userid ggs_owner, password Newpassword_2
tranlogoptions excludeuser ggs_owner
exttrail /u01/gg/dirdat/lt
ddl include all
getupdatebefores
sequence sender.*;
table sender.*;
# cat s_pmp.prm
extract s_pmp
userid ggs_owner, password Newpassword_2
rmthost 10.10.0.216, mgrport 7809
rmttrail /u01/gg/dirdat/rt
passthru
sequence sender.*;
table sender.*;
# cat d_rep.prm
replicat d_rep
userid ggs_owner, password Newpassword_2
assumetargetdefs
discardfile /u01/gg/dirrpt/drep1.dsc, append
reperror (default, exception)
map sender.*, target sender.*;
MACRO #exception_handler
BEGIN
, TARGET GGS_OWNER.GGS_EXCEPTIONS
, COLMAP ( rep_name = #GETENV('GGENVIRONMENT', 'GROUPNAME')
, TABLE_NAME = #GETENV ('GGHEADER', 'TABLENAME')
, ERRNO = #GETENV ('LASTERR', 'DBERRNUM')
, DBERRMSG = #GETENV ('LASTERR', 'DBERRMSG')
, OPTYPE = #GETENV ('LASTERR', 'OPTYPE')
, ERRTYPE = #GETENV ('LASTERR', 'ERRTYPE')
, LOGRBA = #GETENV ('GGHEADER', 'LOGRBA')
, LOGPOSITION = #GETENV ('GGHEADER', 'LOGPOSITION')
, COMMITTIMESTAMP = #GETENV ('GGHEADER', 'COMMITTIMESTAMP')
, GGS_FILENAME = #GETENV('GGFILEHEADER', 'FILENAME')
, CDRFAIL = #GETENV('DELTASTATS','CDR_RESOLUTIONS_FAILED')
, CDRSUC = #GETENV('DELTASTATS','CDR_RESOLUTIONS_SUCCEEDED')
, CDRDETECT = #GETENV('DELTASTATS','CDR_CONFLICTS'))
, INSERTALLRECORDS
, EXCEPTIONSONLY;
END;
MAP sender.* #exception_handler();
site gg2 is similar.
All row that i insert get insert with no abends but the warning confuses me.
According to the message you are getting a Key violation. Does the target table already have a record with the same key? Your exceptions mapping is handling it and you should see records in the ggs.ggs_exceptions table. Also, do a STATS on Replicat.
STATS REPLICAT repname TOTAL
and look for discards or collisions.
It seem like key "1" might already be on the target database. can you check that first?
ALso, you might want to remove the reperror and exception to test.
Failed to execute query. Error: File 'https://track2gen2storage.blob.core.windows.net/\sourcedata\sample.csv' cannot be opened because it does not exist or it is used by another process.
we performed these steps:-
create database SalesdataDemo
use salesdataDemo
-----create master key
CREATE MASTER KEY ENCRYPTION BY PASSWORD = <Password>;
SELECT *
FROM sys.symmetric_keys AS SK
WHERE SK.name = '##MS_DatabaseMasterKey##';
CREATE DATABASE SCOPED CREDENTIAL ADL_User
WITH
IDENTITY = '<client_id>#<OAuth_2.0_Token_EndPoint>',Secret = <Key>
CREATE DATABASE SCOPED CREDENTIAL adls_credential
WITH IDENTITY ='SHARED ACCESS SIGNATURE',
SECRET = <azure_storage_account_key>
CREATE EXTERNAL DATA SOURCE adlsdatasource
WITH
( LOCATION = 'https://track2gen2storage.blob.core.windows.net',
CREDENTIAL = adls_credential
) ;
CREATE EXTERNAL FILE FORMAT adls_csv
WITH (
FORMAT_TYPE = DELIMITEDTEXT,
FORMAT_OPTIONS ( FIELD_TERMINATOR = ',', STRING_DELIMITER = '"', FIRST_ROW = 2 )
);
CREATE EXTERNAL TABLE sampledata ( <ColumnName><Datatype>)
WITH (
LOCATION = '/sourcedata/sample.csv',
DATA_SOURCE = adlsdatasource,
FILE_FORMAT = adls_csv
)
select * from sampledata
I think the problem is your external table location is starting with /. Try changing it to:
CREATE EXTERNAL TABLE sampledata ( <ColumnName><Datatype>)
WITH (
LOCATION = 'sourcedata/sample.csv',
DATA_SOURCE = adlsdatasource,
FILE_FORMAT = adls_csv
)
Here is the document you can also take a look for reference:
https://learn.microsoft.com/en-us/azure/synapse-analytics/sql/create-use-external-tables
One more question, why do you need database scoped credential named ADL_User?
I have an eloquent query as
Role::where("(company_id = Auth::user()->company_id or name = 'admin')and id in(2,3)")->pluck('name');
According to my eloquent the sql should be as
select `name` from `roles` where ( company_id = 1 or name = admin ) and id IN (2, 3) and `roles`.`deleted_at` is null
But it executes as
select `name` from `roles` where ( company_id = 1 or name = admin ) and id IN (2, 3) is null and `roles`.`deleted_at` is null
Can anyone help me concluding why extra is null condition is applied in the query?
Note: I am using soft deletes
You should use whereRaw instead of where
Role::whereRaw("(company_id = Auth::user()->company_id or name = 'admin')and id in(2,3)")->pluck('name');
I defined a stored procedure with input and output parameters and am getting the following error when I try to call the stored procedure.
[SQL0312] Variable KUNNR not defined or not usable.
Here is the call:
CALL R3QA6DATA.SP_ADDRESS_CHANGES ('1999-12-31 23:59:59', '2016-06-01 23:59:59', :KUNNR,:KUNN2,:NAME1,:NAME2,:STRAS,:ORT01,:REGIO,:PSTLZ,:LAND1,:TELF1,:TELFX,:DEFPA)
GO
Here is the stored procedure:
CREATE PROCEDURE R3QA6DATA.SP_ADDRESS_CHANGES
(IN STARTDATE TIMESTAMP, IN ENDDATE TIMESTAMP,
OUT KUNNR GRAPHIC(10), OUT KUNN2 GRAPHIC(10), OUT NAME1 GRAPHIC(35), OUT NAME2 GRAPHIC(35), OUT STRAS GRAPHIC(35), OUT ORT01 GRAPHIC(35),
OUT REGIO GRAPHIC(3), OUT PSTLZ GRAPHIC(10), OUT LAND1 GRAPHIC(3), OUT TELF1 GRAPHIC(16), OUT TELFX GRAPHIC(31), OUT DEFPA GRAPHIC(1) )
LANGUAGE SQL
BEGIN
SELECT DISTINCT
knvp.kunnr
, knvp.kunn2
, kna1.name1
, kna1.name2
, kna1.stras
, kna1.ort01
, kna1.regio
, kna1.pstlz
, kna1.land1
, kna1.telf1
, kna1.telfx
, knvp.defpa
INTO KUNNR, KUNN2, NAME1, NAME2, STRAS, ORT01, REGIO, PSTLZ, LAND1, TELF1, TELFX, DEFPA
FROM
R3QA6DATA.KNA1 AS kna1
INNER JOIN
R3QA6DATA.ZMBCM AS zmbcm
ON
kna1.KUNNR = zmbcm.KUNAG
INNER JOIN
R3QA6DATA.KNVV AS knvv
ON
( kna1.KUNNR = knvv.KUNNR )
INNER JOIN
R3QA6DATA.KNVP AS knvp
ON
(
knvv.KUNNR = knvp.KUNNR
AND
knvv.VKORG = knvp.VKORG
AND
knvv.VTWEG = knvp.VTWEG
AND
knvv.SPART = knvp.SPART
)
WHERE
kna1.MANDT = '010'
AND
knvp.PARVW IN ('WE', 'AG')
AND
(
knvv.VKORG = zmbcm.VKORG
AND
knvv.VTWEG = zmbcm.VTWEG
AND
knvv.SPART = zmbcm.SPART
)
AND
kna1.loevm = ' '
AND
knvv.loevm = ' '
AND
knvp.KUNN2 IN
(
SELECT
SUBSTRING(bdcp2.tabkey, 4, 10)
FROM
R3QA6DATA.BDCP2 AS bdcp2
WHERE
bdcp2.mestype = 'DEBMAS'
AND
(
( bdcp2.tabname = 'KNA1'
AND
bdcp2.fldname IN
('NAME1'
, 'NAME2'
, 'STRAS'
, 'ORT01'
, 'REGIO'
, 'LAND1'
, 'PSTLZ'
, 'TELF1'
, 'TELFX') )
OR
( bdcp2.tabname = 'KNVP' )
)
AND
(
bdcp2.cretime > STARTDATE
AND
bdcp2.cretime < ENDDATE
)
)
AND
zmbcm.STOREID = 4 ;
END
Go
In RPGLE or COBOL this issue usually happens if you have the variables defined in an external source and accessed through COPY statement. If this is the case in your problem then try using the source through INCLUDE.