I am creating some new ETL tasks for our data pipeline. We have currently have several hundred loading data from various S3 buckets.
So it would go like this:
create or replace stage ETL_STAGE url='s3://bucketname/'
file_format = csv_etl;
create or replace file format csv_etl
type = 'CSV'
field_delimiter = ','
skip_header = 1
FIELD_OPTIONALLY_ENCLOSED_BY='"'
copy into db.schema.table
from #ETL_STAGE/Usage
pattern='/.*[.]csv'
on_error = 'continue'
However, whenever I use this my file format is not only not not escaping the enclosed double quotes it is not even skipping the header so I get this:
Pretty perplexed by this as I am 99% certain the formatting options are correct here.
+-------------------+----------+----------------+---------------------+-------------------+
| "Usage Task Name" | "Value" | "etl_uuid" | "etl_deviceServer" | "etl_timestamp" |
| "taskname" | "0" | "adfasdfasdf" | "hostserverip" | "2020-04-06 2124" |
+-------------------+----------+----------------+---------------------+-------------------+
Run below command by including file_format. This applied the file format while loading file:
copy into db.schema.table
from #ETL_STAGE/Usage
pattern='/.*[.]csv'
on_error = 'continue'
file_format = csv_etl;
I created an AWS Config rule and lambda operating on resource type AWS::RDS::DBInstance and Trigger Type = 'Configuration changes'. CloudWatch logs verify that the function return is ...
{ "ResultToken": "<Redacted>",
"Evaluations": [
{"ComplianceResourceId": "db-<Redacted>",
"ComplianceResourceType": "AWS::RDS::DBInstance",
"ComplianceType": "COMPLIANT",
"OrderingTimestamp": 1576676501.52}
]
}
And although the rule is successfully invoked, the AWS console claims that the compliance status of the rule is 'No results available'. Additionally, this bit of powershell script using the AWSPowershell module ...
Get-CFGComplianceByConfigRule -configrulename security-group-of-rds | select -expandProperty Compliance
... returns ...
INSUFFICIENT_DATA
Why isn't the reported compliance status COMPLIANT?
My first thought is that I've got the schema for the return object wrong, but based on the example functions that AWS has supplied, it looks correct to me.
The short answer is:
Evaluation results need to be reported via a call to config:Put_Evaluations() rather than the actual lambda return.
The lambda return should just be the evaluations list.
The long answer is, Here is my solution that works:
AWS Lambda Function (language python3.8) for the Config Rule
'''
#####################################
## Gherkin ##
#####################################
Rule Name:
security-group-of-rds
Description:
Checks that all Oracle databases are using the correct security group and only that group.
Trigger:
Configuration Change on AWS::RDS::DbInstance . Scope of changes == Resources.
Reports on:
AWS::RDS::DbInstance
Parameters:
| ----------------------|-----------|-----------------------------------------------|
| Parameter Name | Type | Description |
| ----------------------|-----------|---------------------------------------------- |
| vpcSecurityGroupId | string | Id of the required vpc Security Group. |
| ----------------------|-----------|---------------------------------------------- |
| Assume-Rule-Role | boolean | If true, switch to the config role. |
| | | Defaults to false. |
|-----------------------|-----------|-----------------------------------------------|
| Mode | Enum | Range: Fully-Operational-DeathStar | |
| | | Put-Evaluations-Test | |
| | | Lambda-Console-Test |
| | | Defaults to Fully-Operational-DeathStar . |
| | | Meanings: |
| | | Fully-Operational-DeathStar: |
| | | Normal operation. |
| | | Put-Evaluations-Test: Set TestMode to True, |
| | | when invoking put_evaluations. |
| | | Refer: https://docs.aws.amazon.com/config/latest/APIReference/API_PutEvaluations.html
| | | Lambda-Console-Test: |
| | | Do not call put_evaluations() at all. | |
|-----------------------|-----------|-----------------------------------------------|
Envars:
| ----------------------|-----------|-----------------------------------------------|
| Envar Name | Type | Description |
| ----------------------|-----------|---------------------------------------------- |
| PROXY | string | http(s) proxy. Default to no proxy. |
|-----------------------|-----------|-----------------------------------------------|
| NO_PROXY | comma- | list of exemptions to proxy. |
| | separated-| Defaults to no exemptions |
| | list | |
|-----------------------|-----------|-----------------------------------------------|
| TURN_OFF_SSL | boolean | Turns of SSL verification. Defaults to False |
|-----------------------|-----------|-----------------------------------------------|
| REGION | string | Region for config service. |
| | | Defaults to the lambda region |
|-----------------------|-----------|-----------------------------------------------|
| CONFIG_ENDPOINT | string | Customised end-point for config service |
| | | Defaults to the standard end-point. |
|-----------------------|-----------|-----------------------------------------------|
Feature:
In order to: to protect the data confidentiality for Oracle oracle-ee RDS databases.
As: a Developer
I want: To ensure that all databases have the correct security group attached.
Scenarios:
Scenario 1:
Given: Wrong security group
And: The group is inactive
Then: No conclusion.
Scenario 2:
Given: Wrong security group
And: The group is active
And: type == oracle-ee
Then: return NON_COMPLIANT
Scenario 3:
Given: Right security group
And: The group is active
And: type == oracle-ee
Then: return COMPLIANT
Scenario 4:
Given: No security group
And: type == oracle-ee
Then: return NON_COMPLIANT
Scenario 5:
Given: type != oracle-ee
Then: return NOT_APPLICABLE
Required Role Policy Statements:
If you are not assuming the config rule role, then the lambda role needs all these
actions, except sts:AssumeRole.
If you ARE assuming the config rule role, then the lambda role needs the logs and sts
actions, and the config rule role needs the logs and config actions.
| ----------------------|-------------|-----------------------------------------------|
| Action | Resource | Condition | Why do we need it? |
| ----------------------|-------------|---------------------------------------------- |
| logs:CreateLogGroup | * | Always | For logging. |
| logs:CreateLogStream | | | |
| logs:PutLogEvents | | | |
| ----------------------|-------------|------------|----------------------------------|
| sts:AssumeRole | Your AWS | if Assume-Rule-Role == True | If you want the |
| | config role | | lambda to execute in the main |
| | | | config role. |
| ----------------------|-------------|------------|----------------------------------|
| config:PutEvaluations | * | Always | To put the actual results. |
| ----------------------|-------------|------------|----------------------------------|
Inline Constants Configuration:
| ----------------------|-----------|-----------------------------------------------|
| Identifier | Type | Description |
| ----------------------|-----------|---------------------------------------------- |
| defaultRegion | string | Default region, if we can't get it from the |
| | | Lambda environment. |
| ----------------------|-----------|---------------------------------------------- |
'''
import json
import datetime
import time
import boto3
import botocore
import os
proxy = None
no_proxy = None
configClient = None
defaultRegion = 'ap-southeast-2'
def setEnvar( name, value):
if os.environ.get( name, '') != value:
if value != '':
os.environ[ name] = value
else:
del os.environ[ name]
def setProxyEnvironment():
# Sometimes lamdba's sit in VPC's which require proxy forwards
# in order to access some or all internet services.
global proxy
global noProxy
proxy = os.environ.get( 'PROXY' , None)
noProxy = os.environ.get( 'NO_PROXY', None)
if proxy is not None:
setEnvar( 'http_proxy' , proxy )
setEnvar( 'https_proxy', proxy )
if noProxy is not None:
setEnvar( 'no_proxy' , noProxy)
def jpath( dict1, path, sep = '.', default = None):
# Traverse a hierarchy of dictionaries, as described by a path, and find a value.
ret = dict1
if isinstance( path, str):
particleList = path.split( sep)
else:
particleList = path
for particle in particleList:
if isinstance( ret, dict):
ret = ret.get( particle, None)
elif (isinstance( ret, list) or isinstance( ret, tuple)) and particle.isdigit():
idx = int( particle)
if (idx >= 0) and (idx < len(ret)):
ret = ret[ idx]
else:
ret = None
else:
ret = None
if ret is None:
break
if ret is None:
ret = default
return ret
def coerceToList( val):
# Make it into a list.
if val is None:
return list()
else:
return val
def coerceToBoolean( val):
if isinstance( val, str):
return val.lower() == 'true'
else:
return bool( val)
def get_region():
# Find the region for AWS services.
return os.environ.get( 'REGION', os.environ.get( 'AWS_REGION', defaultRegion))
def get_assume_role_credentials( role_arn):
# Switch to a role. We need sts:AssumeRole for this.
global proxy
if coerceToBoolean( os.environ.get( 'TURN_OFF_SSL', False)):
sts_client = boto3.client('sts', verify=False)
else:
sts_client = boto3.client('sts')
try:
assume_role_response = sts_client.assume_role(RoleArn=role_arn, RoleSessionName="configLambdaExecution")
print( 'Switched role to ' + role_arn)
return assume_role_response['Credentials']
except botocore.exceptions.ClientError as ex:
# Scrub error message for any internal account info leaks
if 'AccessDenied' in ex.response['Error']['Code']:
ex.response['Error']['Message'] = "AWS Config does not have permission to assume the IAM role."
else:
ex.response['Error']['Message'] = "InternalError"
ex.response['Error']['Code'] = "InternalError"
print(str(ex))
raise ex
def get_client(service, event):
# Get the AWS service client for the specified service.
# If specified, switch roles and go through a custom service end-point.
global proxy
region = get_region()
ruleRole = jpath( event, 'executionRoleArn')
doAssumeRuleRole = coerceToBoolean( jpath( event, 'ruleParameters-parsed.Assume-Rule-Role', '.', False)) and (ruleRole is not None)
parms = {}
if coerceToBoolean( os.environ.get( 'TURN_OFF_SSL', False)):
parms['verify'] = False
if region is not None:
parms['region_name'] = region
if doAssumeRuleRole:
credentials = get_assume_role_credentials( ruleRole)
parms['aws_access_key_id' ] = credentials['AccessKeyId' ]
parms['aws_secret_access_key'] = credentials['SecretAccessKey']
parms['aws_session_token' ] = credentials['SessionToken' ]
endPointEnvarName = service.upper() + '_ENDPOINT'
endPointEnvarValue = os.environ.get( endPointEnvarName, '')
if endPointEnvarValue != '':
parms['endpoint_url'] = endPointEnvarValue
return boto3.client(service, **parms)
def get_configClient( event):
# Get the AWS 'config' service, and store it in a global singleton.
global configClient
if configClient is None:
configClient = get_client( 'config', event)
return configClient
def initiate_Globals():
# Mainly setup the proxy forward, if required.
configClient = None
setProxyEnvironment()
def evaluate_compliance( configuration_item, ruleParameters):
# Evaluate the compliance of the given changed resource.
# Return a dictionary in the standard 'evaluation' schema.
referenceVpcSecurityGroupId = ruleParameters.get('vpcSecurityGroupId','')
annotation = 'Ok'
if ((jpath( configuration_item, 'configuration.engine') == 'oracle-ee') and
(configuration_item.get('resourceType','') == 'AWS::RDS::DBInstance')):
ok = False
for vpcSecurityGroup in coerceToList( jpath( configuration_item, 'configuration.vpcSecurityGroups')):
actualId = vpcSecurityGroup.get('vpcSecurityGroupId','')
ok = ((actualId == referenceVpcSecurityGroupId) or
(vpcSecurityGroup.get('status','inactive') != 'active'))
if not ok:
# The security group was active, but was not equal to the prescribed one.
annotation = 'Wrong security group'
break
if ok:
# All active security groups, and at least one, are the prescribed one.
compliance_type = 'COMPLIANT'
else:
if referenceVpcSecurityGroupId == '':
annotation = 'Malformed rule parameter configuration'
if annotation == 'Ok':
annotation = 'No security groups'
compliance_type = 'NON_COMPLIANT'
else:
# This rule only deals with oracle-ee RDS databases.
compliance_type = 'NOT_APPLICABLE'
evaluation = dict()
evaluation['ComplianceResourceType'] = configuration_item['resourceType']
evaluation['ComplianceResourceId' ] = configuration_item['resourceId']
evaluation['OrderingTimestamp' ] = configuration_item['configurationItemCaptureTime']
evaluation['ComplianceType' ] = compliance_type
evaluation['Annotation' ] = annotation
return evaluation
def printEnvars( envarList):
for envarName in envarList.split(','):
envarValue = os.environ.get( envarName, None)
if envarValue is not None:
print( f'Envar {envarName} == {envarValue}')
def lambda_handler(event, context):
global configClient
# Phase 1: Setup and parsing input.
# Uncomment this when debugging:
# print( 'event == ' + json.dumps( event))
printEnvars( 'PROXY,NO_PROXY,TURN_OFF_SSL,REGION,CONFIG_ENDPOINT')
initiate_Globals()
invokingEvent = json.loads( event.get('invokingEvent','{}'))
event['invokingEvent-parsed'] = invokingEvent
ruleParameters = json.loads( event.get('ruleParameters','{}'))
event['ruleParameters-parsed'] = ruleParameters
print( 'Config rule Arn == ' + event.get( 'configRuleArn', ''))
print( 'Rule parameters == ' + json.dumps( ruleParameters))
get_configClient( event)
configuration_item = invokingEvent['configurationItem']
# Phase 2: Evaluation.
evaluation = evaluate_compliance( configuration_item, ruleParameters)
# Phase 3: Reporting.
evaluations = list()
evaluations.append( evaluation)
mode = ruleParameters.get( 'Mode', 'Fully-Operational-DeathStar')
if mode == 'Fully-Operational-DeathStar':
response = configClient.put_evaluations( Evaluations=evaluations, ResultToken=event['resultToken'])
elif mode == 'Put-Evaluations-Test':
response = configClient.put_evaluations( Evaluations=evaluations, ResultToken=event['resultToken'], TestMode=True)
else:
response = {'mode': mode}
# Uncomment this when debugging:
# print( 'response == ' + json.dumps( response))
print( 'evaluations == ' + json.dumps( evaluations))
return evaluations
I’m setting up a new server, and I'm updating some old script (PHP 5+) to PHP 7.
I'm connecting to a DB2 database via PDO ODBC and reading a CHAR field with CCSID 870 and saving it on a MySQL mediumtext field in a table with CHARSET=utf8. But i got wrong characters on MySQL database and event in PHP console.
I tried to switch to odbc_connect() like the old script but the results was the same.
Even saving the field in a txt file the results is the same.
utf8_encode & utf8_decode doesn't help.
Here an example of code:
$as = new PDO("odbc:MYODBC",$user, $psw);
$as->setAttribute(PDO::ATTR_DEFAULT_FETCH_MODE, PDO::FETCH_ASSOC);
$res = $as->query("SELECT FIELD FROM MYTABLE");
$rows = $res->fetchAll();
$mysql = new PDO("mysql:host=srvip;dbname=mydbname;charset=utf8",$user, $psw);
$mysql->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$mysql->setAttribute(PDO::ATTR_EMULATE_PREPARES, false);
$mysql->setAttribute(PDO::ATTR_DEFAULT_FETCH_MODE, PDO::FETCH_ASSOC);
$ins = $mysql->prepare("INSERT INTO my_MySQL_TABLE (FIELD) VALUES (?)");
$ins->execute(array(trim($rows[0]["FIELD"])));
I expect the results on MySQL to be Wąż, but the actual output is W?? or WÈØ.
Edit on 2019-06-06
| Source | String | HEX |
|------------------|--------|------------|
| DB2 | Wąż | E6A0B2 |
| MySQL | W?? | 573F3F |
| MySQL C/P Insert | Wąż | 57C485C5BC |
The last version is a simple copy-paste to MySQL using a GUI
Edit on 2019-06-07
C:\Users\ME\>echo %DB2CODEPAGE%
1208
C:\Users\ME\>acs.exe /PLUGIN=cldownload /system=MYSYS /sql="SELECT FIELD as char,HEX(FIELD) as hex FROM TABLE" /display=1
CHAR HEX
W?? E6A0B2
If I use /clientfile=test.txt instead of /display=1 Notepad++ show me the file as UTF-8
I'm writing a program to test update scripts for Azure sql.
The idea is to
- first clone a database (or fill a clone with the source schema and content)
- then run the update script on the clone
Locally I have this working, but for azure I have the probem that I don't see any file names. If I restore one database to another on the same azure "server", don't I have to rename the data files during restore too?
For local restore I do this:
restore.Devices.AddDevice(settings.BackupFileName, DeviceType.File);
restore.RelocateFiles.Add(new RelocateFile("<db>", Path.Combine(settings.DataFileDirectory, settings.TestDatabaseName + ".mdf")));
restore.RelocateFiles.Add(new RelocateFile("<db>_log", Path.Combine(settings.DataFileDirectory, settings.TestDatabaseName + "_1.ldf")));
restore.SqlRestore(srv);
Is something similar required for cloning a database on azure?
Lots of Greetings!
Volker
You can create a database as a copy of [source]:
CREATE DATABASE database_name [ COLLATE collation_name ]
| AS COPY OF [source_server_name].source_database_name
{
(<edition_options> [, ...n])
}
<edition_options> ::=
{
MAXSIZE = { 100 MB | 500 MB | 1 | 5 | 10 | 20 | 30 … 150…500 } GB
| EDITION = { 'web' | 'business' | 'basic' | 'standard' | 'premium' }
| SERVICE_OBJECTIVE =
{ 'basic' | 'S0' | 'S1' | 'S2' | 'S3'
| 'P1' | 'P2' | 'P3' | 'P4'| 'P6' | 'P11'
| { ELASTIC_POOL(name = <elastic_pool_name>) } }
}
[;]
Problem:
In our application, we have same locators(id) for submit images (link + form submit) in Popup and main window. Main window submit image executes two javascript functions for click event. While trying through selenium, I'm getting following selenium error and submit action not happened. Unable to process, even if I clicked manually on the browser, which was opened by Selenium. But this scenario is working fine if I do manually end to end.
I tried with FormSubmit and ClickAt..No luck... Any trick/solution?
I found similar thread without any solution --> Selenium RC - selenium-browserbot.js error (http://stackoverflow.com/questions/2380543/selenium-rc-selenium-browserbot-js-error)
Environment:
Browser: IE8
Java: Sun Microsystems Inc. 16.0-b13
OS: Windows XP 5.1 x86
Selenium RC- selenium-server-1.0.3
Selenium Error Message
An error has occured in the script on this page
Line: 2120
Char: 9
Error: Permission denied
Code: 0
URL: file:///C:/DOCUME~1/script1/LOCALS~1/Temp/customProfileDira540839f44a5460e8f29cdcb8f3632a7/core/scripts/selenium-browserbot.js
Do you want to continue running scripts on this page?
HTML Source
<A onmouseover="imgOn('cmdSubmitbutton', 'submit');" onmouseout="imgOff('cmdSubmitbutton', 'submit');" onclick="return verifyAttachmentJS();submitOrder();" href="javascript:void(0);"><IMG title="Submit this form." border=0 name=cmdSubmitbutton src="/webtop/images/buttons/submit_off.gif"> </A>
selenium RC command history
type(desc1, asdasdas)
click(cmdAttachbutton)
click(cmdSubmitbutton)
click(cmdFinishbutton)
selectWindow(null)
type(ORD_TrackingNbr, 6666)
isElementPresent(cmdSubmitbutton)
clickAt(cmdSubmitbutton, 20,8)
Selenium Log Console
info(1331131785603): Executing: |type | findCnum | ABCDP60 |
info(1331131789088): Executing: |click | gobutton | |
error(1331131792666): Caught an exception attempting to log location; this should get noticed soon!
error(1331131792666): Unexpected Exception: Permission denied
error(1331131792666): Exception details: name -> Error, number -> -2146828218, description -> Permission denied, message -> Permission denied
info(1331131795494): Executing: |selectFrame | relative=up | |
info(1331131796275): Executing: |selectFrame | relative=up | |
info(1331131797056): Executing: |getLocation | | |
info(1331131797494): Executing: |click | xpath=//a[#id="ABCDP60" or #name="ABCDP60" or #href="ABCDP60" or normalize-space(descendant-or-self::text())="ABCDP60" or #href="http://172.18.70.63/webtop/ABCDP60"] | |
error(1331131806338): Caught an exception attempting to log location; this should get noticed soon!
error(1331131806338): Unexpected Exception: Permission denied
error(1331131806338): Exception details: name -> Error, number -> -2146828218, description -> Permission denied, message -> Permission denied
info(1331131808056): Executing: |click | //form[#id='form1']/table/tbody/tr/td[6]/a[4]/font/b | |
info(1331131815478): Executing: |click | cmdZipbutton | |
error(1331131815712): Caught an exception attempting to log location; this should get noticed soon!
error(1331131815712): Unexpected Exception: Permission denied
error(1331131815712): Exception details: name -> Error, number -> -2146828218, description -> Permission denied, message -> Permission denied
info(1331131835384): Executing: |selectWindow | name=attachdoc | |
info(1331131865634): Executing: |type | desc1 | asdasdas |
info(1331131876431): Executing: |click | cmdAttachbutton | |
info(1331131882290): Executing: |click | cmdSubmitbutton | |
info(1331131888290): Executing: |click | cmdFinishbutton | |
info(1331131894196): Executing: |selectWindow | null | |
info(1331131894306): Executing: |type | ORD_TrackingNbr | 6666 |
info(1331131898399): Executing: |isElementPresent | cmdSubmitbutton | |
info(1331131902290): Executing: |clickAt | cmdSubmitbutton | 20,8 |
info(1331132070023): Done appending missed logging messages
error(1331132070023): Caught an exception attempting to log location; this should get noticed soon!
error(1331132070023): Unexpected Exception: Permission denied
error(1331132070039): Exception details: name -> Error, number -> -2146828218, description -> Permission denied, message -> Permission denied
error(1331132115492): Caught an exception attempting to log location; this should get noticed soon!
error(1331132115492): Unexpected Exception: Permission denied
error(1331132115492): Exception details: name -> Error, number -> -2146828218, description -> Permission denied, message -> Permission denied