Errors writing Geopandas Dataframe to Shapefile - geopandas

I'm writing a geopandas polygon file to an Esri Shapefile. I can't write directly because I have date fields that I don't want to convert to text, I want to keep them as date.
I've written a custom schema, but how do I handle the geomtry column in the custom shapefile? It's a WKT field.
This is my custom schema (shortened for length):
schema = {
'geometry':'MultiPolygon',
'properties':{
'oid':'int',
'date_anncd':'date',
'value_mm':'float',
'geometry':??
}
}


Change the geometry type to 'Polygon' (MultiPolygon isn't an Esri type) and drop the geometry value in the properties.

Related

How to query in GraphQL with no fixed input object contents?

I want to query to get a string, for which I have to provide a input object, however the input data can have multiple optional keys and quite grow large as well. So I just wanted something like Object as data type while defining schema.
// example of supposed schema
input SampleInput {
id: ID
data: Object // such thing not exist, wanted something like this.
}
type Query {
myquery(input: SampleInput!): String
}
here data input can be quite large so I do not want to define a type for it. is there a way around this?

How to create Glue table with Parquet format?

In the documentation I found how to create Glue table in JSON format but I cannot find how to create one in Parquet format.
I think I could provide a subtype of glue.DataFormat, but I don't know how to do that https://docs.aws.amazon.com/cdk/api/latest/docs/aws-glue-readme.html

Ok, I found what are the working values on Terraform's website.
https://www.terraform.io/docs/providers/aws/r/glue_catalog_table.html
const glue_DataFormat_Parquet = <glue.DataFormat> {
inputFormat: new glue.InputFormat('org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'),
outputFormat: new glue.OutputFormat('org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'),
serializationLibrary: new glue.SerializationLibrary('org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe')
};

How to choose data type for creating table in hive

I have to ingest data in hive table from hdfs but I don't know how to choose correct data type for the data mentioned below:-
$34740$#$Disrupt Worldwide LLC$#$40425$#$null$#$13$#$6$#$317903$#$null$#$Scott Bodily$#$+$#$null$#$10$#$0$#$1$#$0$#$disruptcentral.com$#$null$#$null$#$1$#$null$#$null$#$null$#$Scott Bodily$#$1220DB56-56D7-E411-80D6-005056A451E3$#$true$
$34741$#$The Top Tipster Leagues Limited$#$35605$#$null$#$13$#$7$#$317902$#$null$#$AM Support Team$#$+447886 027371$#$null$#$1$#$1$#$1$#$0$#$www.toptipsterleagues.com, www.toptipsterleagues.co.uk, http://test.toptipsterleague.com$#$Jamil Johnson$#$Cheng Liem Li$#$1$#$0.70$#$1.50$#$1.30$#$Bono van Nijnatten$#$0B758BF9-F1D6-E411-80D7-005056A44C5C$#$true$

Refer this link for different data types,
Click here
Other than all the numeric and decimal fields you can use STRING data type. For the numeric fields based on the range and precision you can use INT or DECIMAL.
Using string and varchar or any other string data types will read null in your data as string i.e "null" for handling nul you should mention the table properties like below,
ALTER TABLE tablename SET
SERDEPROPERTIES ('serialization.null.format' = 'null');
Let me know if anything needed on this.

Mlib RandomForest (Spark 2.0) predict a single vector

After training a RandomForestRegressor in PipelineModel using mlib and DataFrame (Spark 2.0)
I loaded the saved model into my RT environment in order to predict using the model, each request
is handled and transform through the loaded PipelineModel but in the process I had to convert the
single request vector to a one row DataFrame using spark.createdataframe all of this takes around 700ms!
comparing to 2.5ms if I uses mllib RDD RandomForestRegressor.predict(VECTOR).
Is there any way to use the new mlib to predict a a single vector without converting to DataFrame or do something else to speed things up?

The dataframe based org.apache.spark.ml.regression.RandomForestRegressionModel also takes a Vector as input. I don't think you need to convert a vector to dataframe for every call.
Here is how I think you code should work.
//load the trained RF model
val rfModel = RandomForestRegressionModel.load("path")
val predictionData = //a dataframe containing a column 'feature' of type Vector
predictionData.map { row =>
Vector feature = row.getAs[Vector]("feature")
Double result = rfModel.predict(feature)
(feature, result)
}

What is the correct date format for a Date column in YUI DataTable?

I have produced a data table. All the columns are sortable. It has a date in one column which I formatted dd/mm/yyyy hh:mm:ss . This is different from the default format as defined in the doco, but I should be able to define my own format for non-american formats. (See below)
The DataTable class provides a set of
built-in static functions to format
certain well-known types of data. In
your Column definition, if you set a
Column's formatter to
YAHOO.widget.DataTable.formatDate,
that function will render data of type
Date with the default syntax of
"MM/DD/YYYY". If you would like to
bypass a built-in formatter in favor
of your own, you can point a Column's
formatter to a custom function that
you define.
The table is generated from HTML Markup, so the data is held within "" tags.
This gives me some more clues about compatible string dates for javascript:
In general, the RecordSet expects to
hold data in native JavaScript types.
For instance, a date is expected to be
a JavaScript Date instance, not a
string like "4/26/2005" in order to
sort properly. Converting data types
as data comes into your RecordSet is
enabled through the parser property in
the fields array of your DataSource's
responseSchema
I suspect that the I'm missing something in the date format. So what is an acceptable string date for javascript, that Yui dataTable will recognise, given that I want format it as "dd/mm/yyyy hh:mm:ss" ?

Define your locale
YAHOO.util.DateLocale["pt-BR"] = YAHOO.lang.merge(YAHOO.util.DateLocale, {
x:"%d/%m/%Y"
});
And your column settings as follows
{key:"columnKey", label:"columnLabel",
formatter:function(container, record, column, data) {
container.innerHTML = YAHOO.util.Date.format(data, {format:"%x"}, "pt-BR");
}
}

Javascript is able to construct a Date using a datestring in the following format:
new Date("April 22, 2010 14:15:23");
If you don't have control of the format of the datestring on the server-side (or don't want to change it) write a custom parsing function that takes the datestring and returns a newly constructed Date object.
You could either use this parser function when constructing the data represented in your DataTable.
rows:[{name:"John",born:customParser("[date string here"]},
{name:"Bill",born:customParser("[date string here"]}
]
-OR-
If you are using Yui's DataSource module (as indicated by your second grey box), you can register this parser function on the date field so that it will already be a Date() before the DataTable is constructed.
dataSource.responseSchema = {
. . .
fields: [
...
{ key: "birth_dt", parser: customParser }
...
]
....
}

You can pass dates from server to client in any format you wish, just in DataSource.responseSchema.fields set parser field to a function that will parse it. Look for stringToDate in dynamic data example.
On client side, you can display Dates in any other format, providing function in formatter field in respective ColumnDefs.
In order to set whole DataTable's date format, set its dateOptions option; alternatively, you can set ColumnDef.dateOptions, all same as described in YAHOO.util.Date.format() docs for oConfig parameter.
For current locale and whole DataTable, it should be
var myConfigs = {
initialRequest: ...
...
// http://developer.yahoo.com/yui/docs/YAHOO.util.Date.html
dateOptions: {format: '%c'}
};
You can also set sLocale there.

Resources