Drools - Find minimum Java 8 Local Date - java-8

I am trying to find the minimum date from list of dates (Java 8) using accumulate function in Drools.
This is my rule:
rule "Print minimum Service Date from Bill Lines"
when
accumulate (
$lineItem : LineLevelData($dateOfService : dateOfService) ,
$epochDay : min($dateOfService.toEpochDay())
)
$minServiceDate : LocalDate() from LocalDate.ofEpochDay($epochDay)
then
System.err.println("Min. Service Date used in rules calculation : " + $minServiceDate);
end
This is the exception I get:
Unable to Analyse Expression LocalDate.ofEpochDay($epochDay):
[Error: unable to resolve method using strict-mode: java.time.LocalDate.ofEpochDay(java.lang.Comparable)]
[Near : {... LocalDate.ofEpochDay($epochDay) ....}]
^ : [Rule name='Print minimum Service Date from Bill Lines']
Obviously, I am missing some basics here. Can somebody help me to fix this one?
Drools version: 7.5.0
POJO:
public class LineLevelData{
LocalDate dateofService;
}
Update:
rule "Print minimum Service Date from Bill Lines"
when
accumulate ( $lineItem : LineLevelData ( $dateOfService : dateOfService ) ,
$epochDay : min($dateOfService.toEpochDay()) )
$epochLong : Number (longValue > 0 ) from $epochDay
$minServiceDate : LocalDate( ) from LocalDate.ofEpochDay($epochLong)
then
System.err.println("Min. Service Date used in rules calculation : " + $minServiceDate);
end
After adding the cast to Long, epoch is converted to Local date. Adding this in case if it helps someone looking for this.

I guess the problem is that Drools is not preserving the type of the min function is returning and it is treating it as a Comparable instead of as a long.
There are 3 ways you can solve this problem:
Create your own accumulate function to deal with LocalDate
Use an accumulate with inline code
Use the workaround bellow to force Drools to cast the Comparable back to LocalDate.
Workaround:
rule "Print minimum Service Date from Bill Lines"
when
$c: Comparable() from accumulate (
LineLevelData($dateOfService : dateOfService) ,
min($dateOfService)
)
$minServiceDate: LocalDate() from $c
then
System.err.println("Min. Service Date used in rules calculation : " +
$minServiceDate);
end
Hope it helps,

Related

Google Ads API: How to Send Batch Requests?

I'm using Google Ads API v11 to upload conversions and adjust conversions.
I send hundreds of conversions each day and want to start sending batch requests instead.
I've followed Google's documentation and I upload/ adjust conversions exactly the way they stated.
https://developers.google.com/google-ads/api/docs/conversions/upload-clicks
https://developers.google.com/google-ads/api/docs/conversions/upload-adjustments
I could not find any good explanation or example on how to send batch requests:
https://developers.google.com/google-ads/api/reference/rpc/v11/BatchJobService
Below is my code, an example of how I adjust hundreds of conversions.
An explanation of how to do so with batch requests would be very appreciated.
# Adjust the conversion value of an existing conversion, via Google Ads API
def adjust_offline_conversion(
client,
customer_id,
conversion_action_id,
gclid,
conversion_date_time,
adjustment_date_time,
restatement_value,
adjustment_type='RESTATEMENT'):
# Check that gclid is valid string else exit the function
if type(gclid) is not str:
return None
# Check if datetime or string, if string make as datetime
if type(conversion_date_time) is str:
conversion_date_time = datetime.strptime(conversion_date_time, '%Y-%m-%d %H:%M:%S')
# Add 1 day forward to conversion time to avoid this error (as explained by Google: "The Offline Conversion cannot happen before the ad click. Add 1-2 days to your conversion time in your upload, or check that the time zone is properly set.")
to_datetime_plus_one = conversion_date_time + timedelta(days=1)
# If time is bigger than now, set as now (it will be enough to avoid the original google error, but to avoid a new error since google does not support future dates that are bigger than now)
to_datetime_plus_one = to_datetime_plus_one if to_datetime_plus_one < datetime.utcnow() else datetime.utcnow()
# We must convert datetime back to string + add time zone suffix (+00:00 or -00:00 this is utc) **in order to work with google ads api**
adjusted_string_date = to_datetime_plus_one.strftime('%Y-%m-%d %H:%M:%S') + "+00:00"
conversion_adjustment_type_enum = client.enums.ConversionAdjustmentTypeEnum
# Determine the adjustment type.
conversion_adjustment_type = conversion_adjustment_type_enum[adjustment_type].value
# Associates conversion adjustments with the existing conversion action.
# The GCLID should have been uploaded before with a conversion
conversion_adjustment = client.get_type("ConversionAdjustment")
conversion_action_service = client.get_service("ConversionActionService")
conversion_adjustment.conversion_action = (
conversion_action_service.conversion_action_path(
customer_id, conversion_action_id
)
)
conversion_adjustment.adjustment_type = conversion_adjustment_type
conversion_adjustment.adjustment_date_time = adjustment_date_time.strftime('%Y-%m-%d %H:%M:%S') + "+00:00"
# Set the Gclid Date
conversion_adjustment.gclid_date_time_pair.gclid = gclid
conversion_adjustment.gclid_date_time_pair.conversion_date_time = adjusted_string_date
# Sets adjusted value for adjustment type RESTATEMENT.
if conversion_adjustment_type == conversion_adjustment_type_enum.RESTATEMENT.value:
conversion_adjustment.restatement_value.adjusted_value = float(restatement_value)
conversion_adjustment_upload_service = client.get_service("ConversionAdjustmentUploadService")
request = client.get_type("UploadConversionAdjustmentsRequest")
request.customer_id = customer_id
request.conversion_adjustments = [conversion_adjustment]
request.partial_failure = True
response = (
conversion_adjustment_upload_service.upload_conversion_adjustments(
request=request,
)
)
conversion_adjustment_result = response.results[0]
print(
f"Uploaded conversion that occurred at "
f'"{conversion_adjustment_result.adjustment_date_time}" '
f"from Gclid "
f'"{conversion_adjustment_result.gclid_date_time_pair.gclid}"'
f' to "{conversion_adjustment_result.conversion_action}"'
)
# Iterate every row (subscriber) and call the "adjust conversion" function for it
df.apply(lambda row: adjust_offline_conversion(client=client
, customer_id=customer_id
, conversion_action_id='xxxxxxx'
, gclid=row['click_id']
, conversion_date_time=row['subscription_time']
, adjustment_date_time=datetime.utcnow()
, restatement_value=row['revenue'])
, axis=1)
I managed to solve it in the following way:
The conversion upload and adjustment are not supported in the Batch Processing, as they are not listed here.
However, it is possible to upload multiple conversions in one request since the conversions[] field (list) could be populated with several conversions, not only a single conversion as I mistakenly thought.
So if you're uploading conversions/ adjusting conversions you can simply upload them in batch this way:
Instead of uploading one conversion:
request.conversions = [conversion]
Upload several:
request.conversions = [conversion_1, conversion_2, conversion_3...]
Going the same way for conversions adjustment upload:
request.conversion_adjustments = [conversion_adjustment_1, conversion_adjustment_2, conversion_adjustment_3...]

'Limit of total fields crossed more than 1000 Elasticsearch exception

My Elasticsearch index has more than 1000 fields due to my Sql schema and I get below exception:
{'type': 'illegal_argument_exception', 'reason': 'Limit of total
fields [1000] in index }
And my bulk insert looks like this:
with open('audit1.txt') as file:
for line in file:
columns = line.split(r'||')
dict['TimeStamp']=columns[0].strip('\'')
dict['BusinessTimeStamp']=columns[1].strip('\'')
dict['RuntimeMicroflowID']=columns[2].strip('\'')
dict['MicroflowID']=columns[3].strip('\'')
dict['UserId']=columns[4].strip('\'')
dict['ClientId']=columns[5].strip('\'')
dict['Userlocation']=columns[6].strip('\'')
dict['Transactionid']=columns[7].strip('\'')
dict['Catagorie']=columns[8].strip('\'')
dict['EventType']=columns[9].strip('\'')
dict['Operation']=columns[10].strip('\'')
dict['PrimaryData']=columns[11].strip('\'')
dict['SecondayData']=columns[12].strip('\'')
i=13
while i < len(columns):
tempdict['BFOLDVALUE'] = columns[i+1].strip('\'')
tempdict['BFNEWVALUE'] = columns[i+2].strip('\'')
if columns[i].strip('\'') is not None:
dict[columns[i].strip('\'')] = tempdict.copy()
i+=3
tempdict.clear()
#print(json.dumps(dict,indent = 4))
batch.append(dict)
if counter==BATCHSIZE:
try:
helpers.bulk(es, batch, index='audit-index', doc_type='audit')
insertedrecords+=counter
counter = 0
batch.clear()
print(insertedrecords," - Records Has Been inserted ")
except BulkIndexError:
print("Error Occured -- continuing")
print(json.dumps(dict,indent = 4))
print(BulkIndexError)
batch.clear()
break
counter+=1
dict.clear()
So, I am assuming I am trying to index this wrongly... is there a better way of indexing this kind of formats in elasticsearch? Note than I am using ELK version 7.5.
Here is the sample file I am parsing to elasticsearch:
2018.07.17/15:41:53.735||2018.07.17/15:41:53.735||'0164a8424fbbp84h%2139165'||'BT_TTB_CashDep_PRC'||'eskedarz'||'UXP'||'00001039'||'0164a842e519pJpA'||'Persistence'||''||'CREATE'||'DailyTxns'||'0164a842e4eapJnu'||'CurrentThread'||'WebContainer : 15'||''||'ParentThread'||'system'||''||'TCPWorkerThreadID'||'WebContainer : 15'||''||'f_POSTINGDT'||'2018-07-17'||''||'versionNum'||'0'||''||'f_TXNAMTDR'||'0'||''||'f_ACCOUNTID'||'013XXXXXXXXX0'||''||'f_VALUEDTTM'||'2018-07-17 15:41:53.0'||''||'f_POSTINGDTTM'||'2018-07-17 15:41:53.692'||''||'f_TXNCLBAL'||'25551.610000'||''||'f_TXNREF'||'0000103917071815410685326'||''||'f_PIEVENTTYPE'||'N'||''||'f_TXNAMT'||'5000.00'||''||'f_TRANSACTIONID'||'0164a842e4e9pJng'||''||'f_TYPE'||'N'||''||'f_USERID'||'xxxarz'||''||'f_SRNO'||'1'||''||'f_TXNBASEEQ'||'5000.00'||''||'f_TXNSRCBRANCH'||'0000X039'||''||'f_TXNCODE'||'T08'||''||'f_CHANNELID'||'BranchTeller'||''||'f_TXNAMTCR'||'5000.00'||''||'f_TXNNARRATION'||'SELF '||''||'f_ISACCRUALPENDING'||'false'||''||'f_TXNDTTM'||'2018-07-17 15:41:53.689'||''
if you carefully look at this part of the error message it would be clear.
Limit of total fields [1000] in index
1000 is the default limit of total fields in the Elasticsearch index as shown in their source code.
public static final Setting<Long> INDEX_MAPPING_TOTAL_FIELDS_LIMIT_SETTING =
Setting.longSetting("index.mapping.total_fields.limit", 1000L, 0, Property.Dynamic, Property.IndexScope);
Please note this is a dynamic setting, hence can be changed on a given index, by updating index setting
PUT test_index/_settings
{
"index.mapping.total_fields.limit": 1500. --> changed it to what is suitable for your index.
}
More info on this issue can be found here and here.
better way to handle such exploding index is to normalize as RDBMS that means store some of the key : value combinations in a nested structure
example
{"keyA":"ValueA","keyB":"ValueB","keyC":"ValueC"...} - record to
{"keyA":"ValueA","Keyvalue":{"keyB":"ValueB"
"keyC":"ValueC"}} - record
so searching would look like Keyvalue.Value == KeyB and KeyValue.Value = ValueB

Robust Standard Errors in lm() using stargazer()

I have read a lot about the pain of replicate the easy robust option from STATA to R to use robust standard errors. I replicated following approaches: StackExchange and Economic Theory Blog. They work but the problem I face is, if I want to print my results using the stargazer function (this prints the .tex code for Latex files).
Here is the illustration to my problem:
reg1 <-lm(rev~id + source + listed + country , data=data2_rev)
stargazer(reg1)
This prints the R output as .tex code (non-robust SE) If i want to use robust SE, i can do it with the sandwich package as follow:
vcov <- vcovHC(reg1, "HC1")
if I now use stargazer(vcov) only the output of the vcovHC function is printed and not the regression output itself.
With the package lmtest() it is possible to print at least the estimator, but not the observations, R2, adj. R2, Residual, Residual St.Error and the F-Statistics.
lmtest::coeftest(reg1, vcov. = sandwich::vcovHC(reg1, type = 'HC1'))
This gives the following output:
t test of coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -2.54923 6.85521 -0.3719 0.710611
id 0.39634 0.12376 3.2026 0.001722 **
source 1.48164 4.20183 0.3526 0.724960
country -4.00398 4.00256 -1.0004 0.319041
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
How can I add or get an output with the following parameters as well?
Residual standard error: 17.43 on 127 degrees of freedom
Multiple R-squared: 0.09676, Adjusted R-squared: 0.07543
F-statistic: 4.535 on 3 and 127 DF, p-value: 0.00469
Did anybody face the same problem and can help me out?
How can I use robust standard errors in the lm function and apply the stargazer function?
You already calculated robust standard errors, and there's an easy way to include it in the stargazeroutput:
library("sandwich")
library("plm")
library("stargazer")
data("Produc", package = "plm")
# Regression
model <- plm(log(gsp) ~ log(pcap) + log(pc) + log(emp) + unemp,
data = Produc,
index = c("state","year"),
method="pooling")
# Adjust standard errors
cov1 <- vcovHC(model, type = "HC1")
robust_se <- sqrt(diag(cov1))
# Stargazer output (with and without RSE)
stargazer(model, model, type = "text",
se = list(NULL, robust_se))
Solution found here: https://www.jakeruss.com/cheatsheets/stargazer/#robust-standard-errors-replicating-statas-robust-option
Update I'm not so much into F-Tests. People are discussing those issues, e.g. https://stats.stackexchange.com/questions/93787/f-test-formula-under-robust-standard-error
When you follow http://www3.grips.ac.jp/~yamanota/Lecture_Note_9_Heteroskedasticity
"A heteroskedasticity-robust t statistic can be obtained by dividing an OSL estimator by its robust standard error (for zero null hypotheses). The usual F-statistic, however, is invalid. Instead, we need to use the heteroskedasticity-robust Wald statistic."
and use a Wald statistic here?
This is a fairly simple solution using coeftest:
reg1 <-lm(rev~id + source + listed + country , data=data2_rev)
cl_robust <- coeftest(reg1, vcov = vcovCL, type = "HC1", cluster = ~
country)
se_robust <- cl_robust[, 2]
stargazer(reg1, reg1, cl_robust, se = list(NULL, se_robust, NULL))
Note that I only included cl_robust in the output as a verification that the results are identical.

How to import angle values from a catalog into GalSim

I'm trying to import galaxy values (Sersic index, half light radius, etc) from an external ascii file in to GalSim. I'm having trouble reading in the position angle value 'beta', and would like to know if this is possible using the YAML format.
When I try I get the error message:
galsim.errors.GalSimConfigValueError: Invalid value_type specified for parameter beta with type=Catalog. Value <class 'coord.angle.Angle'> not in (<class 'float'>, <class 'int'>, <class 'bool'>, <class 'str'>)
I realise that I'm getting this error message because I'm unable to append the string 'deg' after the input to specify that the units of this input are in degrees.
I've tried adding 'deg' directly in to the input catalogue (inside "" speech marks), with no success. I've also tried adding 'deg' after the catalogue read statement directly in the code, also to no success.
A minimum working example is below. This relies on a file named 'input.dat' in the same directory with a single number inside (45, for example). Then, save the code below as 'test.yaml' and run on the command line as $ galsim test.yaml:
gal :
type : Sersic
n : 1
half_light_radius : 1
flux : 1
ellip :
type : QBeta
q : 0.5
beta : { type : Catalog , col : 0 }
input :
catalog :
file_name : input.dat
I expect to be able to read in beta position angle arguments from an input ascii catalogue and have them replicated in the output galaxy profiles. The above MWE should produce a small postage stamp image of a moderately elliptical galaxy at a position angle of 45 degrees (or whatever number is placed inside 'input.dat'.
Thank you in advance for any help or advice on this front.
Try this:
gal :
type : Sersic
n : 1
half_light_radius : 1
flux : 1
ellip :
type : QBeta
q : 0.5
beta :
type: Radians
theta: { type : Catalog , col : 0 }
input :
catalog :
file_name : input.dat
There is also a Degrees type that works the same way if your catalog columns list the angle in degrees.

Displaying/Formatting Output in Exponential Form in VBScript

I'm working on some automation project.
The Data is passed in LabVIEW form a third party hardware, and is further passed to WinCC Flexible via a OPC server in float datatype.
The output display field supports string output. The data is displayed on the display field is processed in the VBScript.
The problem being faced is :
WinCC support float tags and has a maximum length of 4.
The output to be displayed on Display Field is of type string.
When the data is passed through LabVIEW following happens:
Example 1:
LabVIEW Data : 1.27e-4 | Output on WinCC Flex : 0.000127 [Wrong Representation]
All the data below e-5 are represented like above.
Example 2:
LabVIEW Data : 1.27e-10 | Output on WinCC Flex : 1.27E-10 [Correct Rep.]
Is there any way in VBScript to format the output data into exponential notation?
Currently am using this VBS for the representation...
If SmartTags("tag_06_1") = 0 Then SmartTags("output_1") = CStr (SmartTags("presseure_test"))
The best (most bang for the buck (bug?)) solution to all formatting problems in VBScript is to harness .NET formatting. Simple POC script:
Dim aNums : aNums = Split("0.123 1.27e-4 1.27e-10")
Dim sNum
For Each sNum in aNums
WScript.Echo sNum, CDbl(sNum), fmtExpNum(CDbl(sNum))
Next
Function fmtExpNum(dblX)
Dim oSB : Set oSB = CreateObject("System.Text.StringBuilder")
oSB.AppendFormat "{0:E2}", dblX
fmtExpNum = oSB.ToString()
End Function
output (german locale):
0.123 0,123 1,23E-001
1.27e-4 0,000127 1,27E-004
1.27e-10 0,000000000127 1,27E-010

Resources