I am trying to implement the Hive UDF with Parameter and so I am extending GenericUDF class.
The problem is my UDF works find on String Datatype however it throws error if I run on other data types. I want UDF to run regardless of data type.
Would someone please let me know what's wrong with following code.
#Description(name = "Encrypt", value = "Encrypt the Given Column", extended = "SELECT Encrypt('Hello World!', 'Key');")
public class Encrypt extends GenericUDF {
StringObjectInspector key;
StringObjectInspector col;
#Override
public ObjectInspector initialize(ObjectInspector[] arguments) throws UDFArgumentException {
if (arguments.length != 2) {
throw new UDFArgumentLengthException("Encrypt only takes 2 arguments: T, String");
}
ObjectInspector keyObject = arguments[1];
ObjectInspector colObject = arguments[0];
if (!(keyObject instanceof StringObjectInspector)) {
throw new UDFArgumentException("Error: Key Type is Not String");
}
this.key = (StringObjectInspector) keyObject;
this.col = (StringObjectInspector) colObject;
return PrimitiveObjectInspectorFactory.javaStringObjectInspector;
}
#Override
public Object evaluate(DeferredObject[] deferredObjects) throws HiveException {
String keyString = key.getPrimitiveJavaObject(deferredObjects[1].get());
String colString = col.getPrimitiveJavaObject(deferredObjects[0].get());
return AES.encrypt(colString, keyString);
}
#Override
public String getDisplayString(String[] strings) {
return null;
}
}
Error
java.lang.ClassCastException: org.apache.hadoop.hive.serde2.objectinspector.primitive.JavaIntObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.primitive.StringObjectInspector
I would suggest you to replace StringObjectInspector col with PrimitiveObjectInspector col and the corresponding cast this.col = (PrimitiveObjectInspector) colObject. Then there are two ways:
First is to process every possible Primitive type, like this
switch (((PrimitiveTypeInfo) colObject.getTypeInfo()).getPrimitiveCategory()) {
case BYTE:
case SHORT:
case INT:
case LONG:
case TIMESTAMP:
cast_long_type;
case FLOAT:
case DOUBLE:
cast_double_type;
case STRING:
everyting_is_fine;
case DECIMAL:
case BOOLEAN:
throw new UDFArgumentTypeException(0, "Unsupported yet");
default:
throw new UDFArgumentTypeException(0,
"Unsupported type");
}
}
Another way, is to use PrimitiveObjectInspectorUtils.getString method:
Object colObject = col.getPrimitiveJavaObject(deferredObjects[0].get());
String colString = PrimitiveObjectInspectorUtils.getString(colObject, key);
It just pseudocode like examples. Hope it helps.
I'm trying to find how to make this to work with various types.
Thanks in advance for the help!
public static void Main()
{
GetKeySelector("String01"); // OK
GetKeySelector("Date01"); // ArgumentException: Expression of type 'System.Nullable`1[System.DateTime]' cannot be used for return type 'System.String'
GetKeySelector("Integer01"); // ArgumentException: Expression of type 'System.Int32' cannot be used for return type 'System.String'
}
private static Expression<Func<Project,string>> GetKeySelector(string propertyName)
{
var paramExpr = Expression.Parameter(typeof (Project), "p");
var property = Expression.Property(paramExpr, propertyName);
var finalLambda = Expression.Lambda<Func<Project, string>>(property, paramExpr);
return finalLambda;
}
class Project
{
public DateTime? Date01 {get;set;}
public int Integer01 {get;set;}
public string String01 {get;set;}
}
The problem is that you are trying to use properties of types other than string in an expression that produces a string. This is not allowed without a conversion.
One way to solve this issue would be changing the code to produce objects instead of strings, like this:
private static Expression<Func<Project,object>> GetKeySelector(string propertyName) {
var paramExpr = Expression.Parameter(typeof (Project), "p");
var property = Expression.Property(paramExpr, propertyName);
var cast = Expression.Convert(property, typeof(object));
return Expression.Lambda<Func<Project,object>>(cast, paramExpr);
}
Alternatively you could call Convert.ToString through an expression.
I'm just learning visual c# in visual studio
Ok, so I have a ton of data fields in a form, and then I want to write the handlers to all call one main method, update, which then updates a resultsEntry object, which encapsulates a bunch of uint variables with various names.
How do I write an update method to put in either the results object, or the update method that will take the name of the variable in resultsEntry as a string, and the integer to update it with, and then update that field.
Basically, I need to do a
resultsEntry.(inStringHere) = inValueHere;
where resultsEntry is the object being updated, the inStringHere specifies the field to be updated, and the inValueHere represents the integer value to assign to it.
Thanks!
Sam French
You have two challenges,
Setting a field/property in class using a string (the focus of your question). This will be accomplished using reflection.
Converting values to the type in your class (this may not be a problem for you, you may have 'typed' values. I have an ugly solution because this is not the main focus of your question.
Setting a property by name (see comments preceded with '**'):
static class Program
{
// A 'ResultEntry' type
public class ResultEntry
{
public int ID { get; set; }
public string FirstName { get; set; }
public string LastName { get; set; }
}
static void Main()
{
// some tuples Field1 = Property Name; Field2 = Raw Value (string)
List<Tuple<string, string>> rawEntries = new List<Tuple<string, string>>() {
new Tuple<string,string>("ID", "1")
, new Tuple<string, string>("FirstName", "George")
, new Tuple<string, string>("LastName", "Washington")
};
ResultEntry resultEntry = new ResultEntry();
// ** Get MemberInfo's for your ResultEntry. Do this once, not for each instance of ResultEntry!
MemberInfo[] members = resultEntry.GetType().GetMembers(BindingFlags.Public | BindingFlags.Instance);
// Iterate over input
foreach (var raw in rawEntries)
{
// find a MemberInfo (PropertyInfo) that matches your input 'PropertyName'
MemberInfo member = members.FirstOrDefault(m => m.MemberType == MemberTypes.Property && m.Name == raw.Item1);
if (member != null)
{
// if you input is typed you will not have to deal with
// conversion of the string to the actual type of the property
object val = raw.Item2.MyConverter(((PropertyInfo)member).PropertyType);
// ** set the value in 'ResultEntry'
((PropertyInfo)member).SetValue(resultEntry, val, null);
}
}
Console.WriteLine(string.Format("Result Entry: ID = {0}, FirstName = {1}, LastName = {2}", resultEntry.ID, resultEntry.FirstName, resultEntry.LastName));
Console.ReadLine();
}
}
If you need to deal with converting raw string input to type (e.g. string to int), then is just one strategy...
public static class Extensions
{
public static object MyConverter(this string rawValue, Type convertToMe)
{
// ugly, there are other strategies
object val;
if (convertToMe == typeof(Int32))
val = Convert.ToInt32(rawValue);
// ... add other type conversions
else
val = rawValue;
return val;
}
}
Hope this helps.
I'm trying to update definition but facing an exception
SQL Server does not provide comparison of NETXT, XML, Image etc
don't know how to handle it.
My table structure looks like this:
[Table]
public class Dictionary
{
private string _definition;
[Column(IsPrimaryKey = true)]
public string Word
{
get;
set;
}
[Column(CanBeNull = false, Storage="_definition", DbType="NTEXT")]
public string Definition
{
get { return this._definition; }
set { this._definition = value; }
}
}
Here is my update code
using (DictionaryDataContext d8db = new DictionaryDataContext(strConnectionString))
{
IQueryable<Dictionary> DicQuery = from word in d8db.tb_Dictionary where word.Word == w.Word select word;
Dictionary wordShow = DicQuery.FirstOrDefault();
wordShow.Definition = "some string";
d8db.SubmitChanges();
}
Add a rowversion column to your Dictionary class:
[Column(IsVersion = true)]
private Binary _version;
I have pasted my entire test app below. It's fairly compact so I am hoping that it's not a problem. You should be able to simply cut and paste it into a console app and run it.
I need to be able to filter on any one or more of the Person objects' properties, and I don't know which one(s) until runtime. I know that this has beed discussed all over the place and I have looked into and am also using tools such as the PredicateBuilder & Dynamic Linq Library but the discussion aroung them tends to focus more on Sorting and ordering, and each have been struggling with their own issues when confronted with Nullable types. So I thought that I would attempt to build at least a supplemental filter that could address these particular scenarios.
In the example below I am trying to filter out the family members who were born after a certain date. The kick is that the DateOfBirth field on the objects being filterd is a DateTime property.
The latest error I am getting is
No coercion operator is defined between types 'System.String' and 'System.Nullable`1[System.DateTime]'.
Which is the problem. I have attempted several different means of casting and converting but to varying degrees of failure. Ultimately this will be applied against an EF database whcih has also balked at conversion methods such as DateTime.Parse(--).
Any assistence would be greatly appreciated!
using System;
using System.Collections.Generic;
using System.Linq;
using System.Linq.Expressions;
using System.Reflection;
using System.Text;
using System.Threading.Tasks;
namespace ConsoleApplication1
{
class Program
{
static void Main(string[] args)
{
List<Person> people = new List<Person>();
people.Add(new Person { FirstName = "Bob", LastName = "Smith", DateOfBirth = DateTime.Parse("1969/01/21"), Weight=207 });
people.Add(new Person { FirstName = "Lisa", LastName = "Smith", DateOfBirth = DateTime.Parse("1974/05/09") });
people.Add(new Person { FirstName = "Jane", LastName = "Smith", DateOfBirth = DateTime.Parse("1999/05/09") });
people.Add(new Person { FirstName = "Lori", LastName = "Jones", DateOfBirth = DateTime.Parse("2002/10/21") });
people.Add(new Person { FirstName = "Patty", LastName = "Smith", DateOfBirth = DateTime.Parse("2012/03/11") });
people.Add(new Person { FirstName = "George", LastName = "Smith", DateOfBirth = DateTime.Parse("2013/06/18"), Weight=6 });
String filterField = "DateOfBirth";
String filterOper = "<=";
String filterValue = "2000/01/01";
var oldFamily = ApplyFilter<Person>(filterField, filterOper, filterValue);
var query = from p in people.AsQueryable().Where(oldFamily)
select p;
Console.ReadLine();
}
public static Expression<Func<T, bool>> ApplyFilter<T>(String filterField, String filterOper, String filterValue)
{
//
// Get the property that we are attempting to filter on. If it does not exist then throw an exception
System.Reflection.PropertyInfo prop = typeof(T).GetProperty(filterField);
if (prop == null)
throw new MissingMemberException(String.Format("{0} is not a member of {1}", filterField, typeof(T).ToString()));
Expression convertExpression = Expression.Convert(Expression.Constant(filterValue), prop.PropertyType);
ParameterExpression parameter = Expression.Parameter(prop.PropertyType, filterField);
ParameterExpression[] parameters = new ParameterExpression[] { parameter };
BinaryExpression body = Expression.LessThanOrEqual(parameter, convertExpression);
Expression<Func<T, bool>> predicate = Expression.Lambda<Func<T, bool>>(body, parameters);
return predicate;
}
}
public class Person
{
public string FirstName { get; set; }
public string LastName { get; set; }
public DateTime? DateOfBirth { get; set; }
string Nickname { get; set; }
public int? Weight { get; set; }
public Person() { }
public Person(string fName, string lName)
{
FirstName = fName;
LastName = lName;
}
}
}
Update: 2013/02/01
My thought was then to convert the Nullabe type to it's Non-Nullable type version. So in this case we want to convert the <Nullable>DateTime to a simple DateTime type. I added the following code block before the call Expression.Convert call to determine and capture the type of the Nullable value.
//
//
Type propType = prop.PropertyType;
//
// If the property is nullable we need to create the expression using a NON-Nullable version of the type.
// We will get this by parsing the type from the FullName of the type
if (prop.PropertyType.IsGenericType && prop.PropertyType.GetGenericTypeDefinition() == typeof(Nullable<>))
{
String typeName = prop.PropertyType.FullName;
Int32 startIdx = typeName.IndexOf("[[") + 2;
Int32 endIdx = typeName.IndexOf(",", startIdx);
String type = typeName.Substring(startIdx, (endIdx-startIdx));
propType = Type.GetType(type);
}
Expression convertExpression = Expression.Convert(Expression.Constant(filterValue), propType);
This actually worked in removing the Nullable-ness from the DateTime but resulted in the following Coercion error. I remain confused by this as I thought that the purpose of the "Expression.Convert" method was to do just this.
No coercion operator is defined between types 'System.String' and 'System.DateTime'.
Pushing on I explicitly parsed the value to a DateTime and plugged that into the mix ...
DateTime dt = DateTime.Parse(filterValue);
Expression convertExpression = Expression.Convert(Expression.Constant(dt), propType);
... which resulted in an exception that outpaces any knowledge I have of Expressions, Lambdas and their related ilk ...
ParameterExpression of type 'System.DateTime' cannot be used for delegate parameter of type 'ConsoleApplication1.Person'
I am not sure what's left to try.
The problem is that when generating binary expressions, the operands have to be of compatible types. If not, you need to perform a conversion on one (or both) until they are compatible.
Technically, you cannot compare a DateTime with a DateTime?, the compiler implicitly promotes one to the other which allows us to do our comparisons. Since the compiler is not the one generating the expression, we need to perform the conversion ourself.
I've tweaked your example to be more general (and working :D).
public static Expression<Func<TObject, bool>> ApplyFilter<TObject, TValue>(String filterField, FilterOperation filterOper, TValue filterValue)
{
var type = typeof(TObject);
ExpressionType operation;
if (type.GetProperty(filterField) == null && type.GetField(filterField) == null)
throw new MissingMemberException(type.Name, filterField);
if (!operationMap.TryGetValue(filterOper, out operation))
throw new ArgumentOutOfRangeException("filterOper", filterOper, "Invalid filter operation");
var parameter = Expression.Parameter(type);
var fieldAccess = Expression.PropertyOrField(parameter, filterField);
var value = Expression.Constant(filterValue, filterValue.GetType());
// let's perform the conversion only if we really need it
var converted = value.Type != fieldAccess.Type
? (Expression)Expression.Convert(value, fieldAccess.Type)
: (Expression)value;
var body = Expression.MakeBinary(operation, fieldAccess, converted);
var expr = Expression.Lambda<Func<TObject, bool>>(body, parameter);
return expr;
}
// to restrict the allowable range of operations
public enum FilterOperation
{
Equal,
NotEqual,
LessThan,
LessThanOrEqual,
GreaterThan,
GreaterThanOrEqual,
}
// we could have used reflection here instead since they have the same names
static Dictionary<FilterOperation, ExpressionType> operationMap = new Dictionary<FilterOperation, ExpressionType>
{
{ FilterOperation.Equal, ExpressionType.Equal },
{ FilterOperation.NotEqual, ExpressionType.NotEqual },
{ FilterOperation.LessThan, ExpressionType.LessThan },
{ FilterOperation.LessThanOrEqual, ExpressionType.LessThanOrEqual },
{ FilterOperation.GreaterThan, ExpressionType.GreaterThan },
{ FilterOperation.GreaterThanOrEqual, ExpressionType.GreaterThanOrEqual },
};
Then to use it:
var filterField = "DateOfBirth";
var filterOper = FilterOperation.LessThanOrEqual;
var filterValue = DateTime.Parse("2000/01/01"); // note this is an actual DateTime object
var oldFamily = ApplyFilter<Person>(filterField, filterOper, filterValue);
var query = from p in people.AsQueryable().Where(oldFamily)
select p;
I don't know if this will work as-is for all cases but it certainly works for this particular case.
If you interrogate your body variable, you can see that the body of the expression you're creating is essentially DateOfBirth <= '2000/01/01'.
While on the face this may seem correct, you are trying to assign that body to a function that takes a Person (that's what T would be in your example) and returns a bool. You need to alter your logic such that the body reflects the input as a Person object, accesses the DateOfBirth property on that instance of the Person, and then performs the comparison.
In other words, the body of your expression has to take a T, find the right property on it, and then compare.