Spark: How do I save an RDD before unpersist it - caching

I want to unpersist myRDD after the data is saved. I have the following code:
val isDone = myRDD.saveAsTextFile("myOutput")
if (isDone != null) {
myRDD.unpersist()
}
but the line:
isDone != null
keeps saying: comparing values of types Unit and Null using `!=' will always yield true
What should be the correct way to solve this problem? Thank you!

This should work fine:
myRDD.saveAsTextFile("myOutput")
myRDD.unpersist()
The data will be saved before the RDD is unpersisted.
Note that the saveAsTextFile method returns Unit. This is a type that is returned by any method (procedure) which does not return anything useful, and there is only one instance, i.e. (). So nothing useful is achieved by testing on the value returned by saveAsTextFile. Also, Unit being a subtype of AnyVal can never be equal to null, that's why you're getting the particular error that you're seeing. The same thing happens for Ints:
def foo(x: Int) = x != null

Related

Golang Mocking a call with some parameters in body with values not fixed

I am mocking a method call as follows:
tctx.someMock.On("addProd",
product.NewAddProductParamsWithContext(ctx).
WithID("someid").
WithCreateRequest(pro.CreateProdBody{
creationDate: "someDate" ,
}), nil).
Return(nil, nil)
which works fine.
Now, here, instead of passing a fixed value for the field creationDate, if I want to generalize it so it will work for any value passed, how can I achieve that? I am pretty new to Go, so not sure how to do this
the values for creationDate could be any value like - 2021-03-19T18:57:16.589Z or 2022-04-23T14:17:56.589Z etc. I just dont want to limit the mock call to work for a fixed value of creationDate, but I would like it to work for any date string passed
Assuming you're using github.com/stretchr/testify/mock, you should be able to use mock.MatchedBy() to match on specific parts of the argument. For example:
tctx.someMock.On("addProd", mock.MatchedBy(func(i interface{}) bool {
p := i.(*product.AddProductParams)
return p.ID() == "someid"
})).Return(nil, nil)
However, I find this to be most useful when needing to take a different action depending on the input. If you're simply verifying addProd was called with a specific argument, consider asserting that instead:
tctx.someMock.On("addProd", mock.Anything).Return(nil, nil)
...
tctx.someMock.AssertCalled(t, "addProd", mock.MatchedBy(func(i interface{}) bool {
p := i.(*product.AddProductParams)
return p.ID() == "someid"
})).Return(nil, nil)

Getting and Setting values in java 8 using optional and map

I am trying to use Optional instead of standard null checks in java
#Data
public class InputObj {
private Double savings;
}
#Data
public class Result {
private String outputSavings;
}
public Result convertInputObjToResult(InputObj inputObj){
Result result = new Result();
Optional<InputObj> optionalInputObj = Optional.ofNullable(inputObj);
optionalInputObj.map(InputObj::getSavings).map(value -> util.convertRoundAndAbs(value,true)).ifPresent(result::setOutputSavings);
return result;
}
which is equivalent of below
public Result convertInputObjToResult(InputObj inputObj){
Result result = new Result();
if(inputObj != null){
if(inputObj.getSavings() != null){
result.setOutputSavings(util.convertRoundAndAbs(inputObj.getSavings(),true));
}
}
return result;
}
I wrote some test cases and I do not get any Null Pointer Exception but I am unable to understand that ifPresent condition is at end and map is before but still I don't get any NPE. Do you see any thing wrong with this code or how it can be improved? This is part of a spring boot application and #Data annotation is used for lombok.
Here's a link to further describe how the map operation works for the Java Optional class.
If a value is present, apply the provided mapping function to it, and if the result is non-null, return an Optional describing the result. Otherwise return an empty Optional.
So in a case where you have a null value from the map method, it will automatically be converted to Optional.empty().
Then, taking a look at the ifPresent method
If a value is present, invoke the specified consumer with the value, otherwise do nothing.
So this is why you aren't getting any NPE's; the map operations are able to map null results or passed values to Optional.empty(), and the ifPresent operation doesn't execute if passed an empty Optional.
There is no issue with the code except for the typo “ optionalMembershipDetails”. Assuming you meant to use “ optionalInputObj”.
You need to read about java streams to understand the full concept. In a nutshell, operations are evaluated in lazy manner. So ifPresent call will trigger operations which appear before it. In this case, the object is wrapped inside optional, so each intermediate operation will pass another optional to the next operation. Having Optional prevents you getting NPE.

Comparator.nullsFirst with null-safe-comparator

I want to have a null-safe comparator, but it does not work:
Comparator<Item> sort_high = (i1, i2)-> Double.compare(i2.getUser().getValue(), i1.getUser().getValue());
items.sort(Comparator.nullsFirst(sort_high));
However, I get a NPE, if item.getUser().getValue() (or item.getUser()) is null.
at Item.lambda$1(Item.java:270)
at java.util.Comparators$NullComparator.compare(Comparators.java:83)
at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355)
at java.util.TimSort.sort(TimSort.java:220)
at java.util.Arrays.sort(Arrays.java:1438)
at java.util.List.sort(List.java:478)
What is wrong?
nullsFirst will take care of null Items ("i1"), but once two valid objects are found, your Comparator is invoked and you need to handle internal null references.
In your case, you could use something like:
items.sort(
Comparator.nullsFirst(
Comparator.comparing(Item::getUser,
Comparator.nullsFirst(Comparator.comparingDouble(User::getValue))
)
)
);
(Assuming getValue() returns a double)
But I hardly recommend such convoluted code.
Comparator.nullsFirst() doesn't mean that any NullPointerException thrown during the compare() method will be caught.
It says rather that a null compared object with always be less than a non null compared object and that if both are null, they are considered equal.
<T> Comparator<T> java.util.Comparator.nullsFirst(Comparator<? super
T> comparator)
Returns a null-friendly comparator that considers null to be less than
non-null. When both are null, they are considered equal. If both are
non-null, the specified Comparator is used to determine the order. If
the specified comparator is null, then the returned comparator
considers all non-null values to be equal.
In your case :
Comparator<Item> sort_high = (i1, i2)-> Double.compare(i2.getUser().getValue(), i1.getUser().getValue());
Compared objects are not null, that is i1 or i2 but fields of them are: i1.getUser() or i2.getUser().
You don't have a good use case for using directly Comparator.nullsFirst().
As workaround, you could use it to compare the User field in your compare() implementation as soon as one of two compared objects present a null User field.
I think that it is a good trade off between handling yourself null case and readability of the code :
private Comparator<User> comparatorUserNullFirst = Comparator.nullsFirst(Comparator.comparingDouble(User::getValue));
Comparator<Item> sortHigh = (i1, i2) -> {
if (i1.getUser() == null || i2.getUser() == null) {
return comparatorUserNullFirst.compare(i1.getUser(), i2.getUser());
}
return Double.compare(i2.getUser().getValue(), i1.getUser().getValue());
};
Another possibility, that is slightly more readable but does not distinguish between an item being null and an item having a user that is null, is the following:
Comparator<Item> sortLow = Comparator.comparing(item -> Optional.ofNullable(item)
.map(Item::getUser)
.map(User::getValue)
.orElse(null),
Comparator.nullsFirst(Comparator.naturalOrder())
);

What's the difference when assigning a value to a field in square brackets LINQ?

I have seen when coding in LINQ that when a value is assigned to a field sometimes is in this way Table["Field"] and any others like this Table.Field but can somebody explain me what's the difference please?
For example when modifying a field:
var ttAbccode_xRow =
(from ttAbccode_Row in ds.ABCCode select ttAbccode_Row).FirstOrDefault();
if (ttAbccode_xRow != null) {
ttAbccode_xRow["PI"] = 3.1416;
}
or
if (ttAbccode_xRow != null) {
ttAbccode_xRow.PI = 3.1416;
}
Accessing field via indexer (square brackets) returns object data type. That means that your compiler cannot detect data types incompatibility. You could assing for example string value (eg. "abcd") and you won't get error at design time, but as late as at runtime.
Second method (if available in your result set) is much more safe. Your property will have proper data type hence compiler will detect data types incompatibility at design time.
If I had both access methods available I would always prefer second one. It is less error prone.

Incorrect Resharper Suggestion - "Merge Conditional Expression"

Given an oracle query which is returning a single string value, to get the value from the query I use the following two lines:
var result = cmd.ExecuteOracleScalar();
return result != null ? ((OracleString)result).Value : null;
The "!= null" in this statement is underlines with the suggestion to "Merge Conditional Expression". If I accept the suggestion it changes it to:
return ((OracleString)result).Value;
Which throws an exception because the value returned will be null for a number of executions.
Is there anyway to use the ternary operator but not have this warning?
Note that if I change the code to:
var result = cmd.ExecuteOracleScalar();
if (result == null)
return null;
return ((OracleString)result).Value;
Resharper then first suggests that I "Convert to Return Statement" which just changes it back to use the ternary operator.
Any Suggestions?
This looks like exactly the bug identified in RSRP-434610:
given original code that checks an object reference for nullity, and accesses a property of the object if the object reference is not null
R# proposes a refactoring that always accesses the property, and will therefore fail when the object reference is null
The issue has a fix version of 9.1, which was released just a few days ago, although watch out trying to upgrade from within VS.

Resources