how to filter a spark dataframe by two boolean column - user-defined-functions

So I have a dataframe table call latest_key_status, it have two boolean column 'a' and 'b'
When I want to filter it by 'a', like this:
latest_key_status.filter(latest_key_status['a'] == False)
It works
However, when I want to do this
latest_key_status.filter((latest_key_status['a'] == False) | (latest_key_status['b'] == False))
It failed
Could anyone tell me how can I do this? Thanks

Related

Excel PowerQuery: how do I add an IsNotNull column

I have a simple function that I'd like to run on the values in a column, resulting in another column.
let
ThisIsNotNull = (input) => if (input = null) then false else true,
Source = ...
eventually there is a text column with Nulls in it, let's call it TextColumn.
I'd like to add another column alongside it with a value of =ThisIsNotNull(TextColumn).
Add column... Custom column with formula
= ThisIsNotNull([NameOfColumnToTest])
But really you can skip the function and just use
= if [NameOfColumnToTest] = null then false else true

linq to entity,How does the where clause use ?: Expression

TAB_XXX and TAB_XXX_details are one-to-many relationships, I need to query the two tables, however, we need to be filtered TAB_XXX_details。
The code is as follows:
var qu = from c in db.TAB_XXX.Where(n => n.DELETE_MARK == false)
let dets = c.TAB_XXX_DETAILS.Where(n => condition.SaleType.HasValue ? n.SALE_TYPE == (decimal)condition.SaleType : 1 == 1)
select new
{
c,
dets
};
Condition.SaleType is number?, if the condition.SaleType is a valid number, such as 1, 2, 3 ... I want to filter the child record based on these numbers; when the condition.SaleType is null, I want to query TAB_XXX and all its child records;
How do I modify the where clause?
Thank you for your answer!
Since 1 == 1 is always true, your condition boils down to this:
let dets = c.TAB_XXX_DETAILS
.Where(n => !condition.SaleType.HasValue || n.SALE_TYPE == condition.SaleType.Value)
Essentially, you want to return all rows when condition.SaleType does not have value; otherwise, you make a comparison to condition.SaleType.Value.

How to create a dynamic where clause using a name value collection?

I have column search filter values sent to my web api and I can't figure how to make the where clause dynamic?
Look below (too much code)!!
Name Value Collection:
public DataTablePager<AccountDTO> Get([FromUri] DataTableParameter param)
{
NameValueCollection nvc = HttpUtility.ParseQueryString(Request.RequestUri.Query);
Casting the search values:
if (!String.IsNullOrEmpty(nvc["sSearch_0"]) && !int.TryParse(nvc["sSearch_0"], out tmpInt) ||
!String.IsNullOrEmpty(nvc["sSearch_1"]) && !int.TryParse(nvc["sSearch_1"], out tmpInt) ||
!String.IsNullOrEmpty(nvc["sSearch_10"]) && !int.TryParse(nvc["sSearch_10"], out tmpInt)
Setup Where Clause:
filteredresults = filteredresults.Where(i => CorrectNumericTypes
&& (Lead_ID == null || i.Lead_ID == Lead_ID)
&& (Account_ID == null || i.Account_ID == Account_ID)
This is a classic case for the Pipes and Filters Architectural pattern.
You can construct a list of filters based on dynamic (runtime) conditions. Or keep a static list of filters, the ones where there is no criteria, the data just passes through. Apply this list of filters on the data set or the Query.
I don't think you can achieve that with LINQ. You can design a query builder (using the same Filters pattern)
Here's what I needed (predicate): http://www.c-sharpcorner.com/UploadFile/c42694/dynamic-query-in-linq-using-predicate-builder/

Entity Framework: Any or All - Unable to create a constant value of type 'System.Collections.Generic.List`1'

I am trying to do something like this:
from t in ent.myEntities
where SelectedProperties == null || SelectedProperties.Any(le => le == t.Entity)
select t
basically trying to cover 2 cases. accepting an empty list, should return all entities, or filter on the list if it is supplied.
above actually does work when i supply the list, however in the case when it is null i get:
Unable to create a constant value of type
'System.Collections.Generic.List`1'. Only primitive types ('such as
Int32, String, and Guid') are supported in this context
also tried using this with a string array:
where arr == null || arr.Contains(t.Entity)
is it possible to have such a condition without having to build a predicate (which is a bigger effort)?
You might want to try using the list in a simpler way:
where SelectedProperties == null || SelectedProperties.Contains(t.Entity)
It may well not work, but it's worth a try. Otherwise, if this is really your whole query, I'd just write it as:
var query = SelectedProperties == null
? ent.myEntities
: ent.myEntities.Where(t => SelectedProperties.Contains(t.Entity));
EDIT: Okay, if you have to use Any, and have lots of these to compose, you can do it like this:
var query = ent.myEntities;
if (SelectedProperties != null)
{
query = query.Where(t => SelectedProperties.Any(x => x == t.Entity));
}
if (SomethingElse)
{
query = query.Where(...);
}
// etc
I'm using EF5, something like this will fix the issue:
ent.myEntities.ToList().Where(t => SelectedProperties == null || SelectedProperties.Contains(t.Entity));

In Operator in Linq

I tried to use the suggestion provided here for using In operator in linq but, i am not able to convert my requirement into LINQ statement.
Below is the SQL query which i need to convert to Linq
select *
from navigator_user_field_property
where user_id = 'albert'
and field_id in (
select field_id
from navigator_entity_field_master
where entity_id = 1
and use_type = 0)
order by field_id
I want this to be converted to a Efficient Linq.
Most of the answers deal with the predetermined list of string array which is not working in my case.
Thanks
Looks like a join to me:
var query = from navigator in db.NavigatorUserFieldProperties
where navigator.UserId == "albert"
join field in db.NavigatorEntityFieldMasters
.Where(f => f.EntityId == 1 && f.UseType == 0)
on navigator.FieldId equals field.FieldId
select navigator;
Note that this will return the same value multiple times if there are multiple fields with the same ID - but I suspect that's not the case.
You could do a more literal translation like this:
var query = from navigator in db.NavigatorUserFieldProperties
where navigator.UserId == "albert" &&
db.NavigatorEntityFieldMasters
.Where(f => f.EntityId == 1 && f.UseType == 0)
.select(f => f.FieldId)
.Contains(navigator.FieldId)
select navigator;
... and that may end up translating to the same SQL... but I'd personally go with the join.
Here is an efficient and readable LINQ query:
var fields =
from field in db.navigator_entity_field_masters
where field.entity_id == 1 && field.user_type == 0
select field;
var properties =
from property in db.navigator_user_field_properties
where property.user_id == "albert"
where fields.Contains(property.field)
select property;
Look mama!! Without joins ;-)

Resources