I am getting the below error while using sink_parquet on a LazyFrame. Earlier I was using .collect() on the output of the scan_parquet() to convert the result into a DataFrame but unfortunately it is not working with larger than RAM datasets. Here is the error I received -
PanicException: sink_parquet not yet supported in standard engine. Use 'collect().write_parquet()'
I am trying to write the LazyFrame (the output from scan_parquet) into a local file after I added some filter and join conditions on the LazyFrame.
It seems the error is coming from the below location -
https://github.com/pola-rs/polars/blob/master/py-polars/polars/internals/lazyframe/frame.py#L1235 (In Python)
https://github.com/pola-rs/polars/blob/master/polars/polars-lazy/src/physical_plan/planner/lp.rs#L154 (In Rust) .
I have tried updating to the latest version 0.15.16 0.16.1 but this issue still exists .
Sample code :
pl.scan_parquet("path/to/file1.parquet")
.select([
pl.col("col2"),
pl.col("col2").apply( lambda x : ...)
.alias("splited_levels"),
..followed by more columns and .alias()
])
.join(<another lazyframe>,on="some key",how="inner")
.filter(...)
.filter(..)
..followed by some more filters
.sink_parquet("path/to/result2.parquet")
The parquet file should be written in local system. Instead I am getting the below error -
PanicException: sink_parquet not yet supported in standard engine. Use 'collect().write_parquet()'
Here are the details of the installed packages after I used polars.show_versions() -
--- Version info----
Polars : 0.15.16
Index type : UInt32
Platform : Linux-4.15.0-191-generic-x86_64-with-glibc2.28
Python: 3.9.16
[GCC 8.3.0]
--- Optional dependencies---
pyarrow : 11.0.0
pandas : not installed
numpy : 1.24.1
fsspec : 2023.1.0
connectorx : not installed
xlsx2csv : not installed
deltalake: not installed
matplotlib : not installed
Update : I have raised a github issue here for the same and it seems all types of queries are not supported for streaming at this moment . So I am looking for a work around in this case or any alternative way of doing this with polars
https://github.com/pola-rs/polars/issues/6603
FileStream is deprecated in favor of CharStream,
But in the Go runtime using CharStream.FromFileName(...) results in
antlr.CharStream.FromFileName undefined (type "github.com/antlr/antlr4/runtime/Go/antlr".CharStream has no field or method FromFileName)
Checking the source I can see the char_stream.go does not implement/expose the FromFileName method...
While using NewFileStream(...) results in
cannot use file (variable of type *"github.com/antlr/antlr4/runtime/Go/antlr".FileStream) as type "github.com/antlr/antlr4/runtime/Go/antlr/v4".CharStream in argument to NewralfLexer:
What is the correct way to read in a file in this case?
You are using the deprecated nsconfig.json file. Just be aware that NativeScript 7 has an improved nativescript.config.(js|ts) file for when you're ready to upgrade this project.
The "path" argument must be of type string. Received null
Because can't find template path.
ns create ProjectName --template #nativescript/TemplateName
Example :
ns create HelloWorld --template #nativescript/template-blank
ns create HelloWorld --template #nativescript/template-blank-ts
ns create HelloWorld --template #nativescript/template-blank-ng
windows 10
centos 7
python 3.6.5
VScode
I'm trying OpenPose, finally I commanded this code
[vagrant#localhost tf-pose-estimation]$ python3 run.py --model=mobilenet_thin --resize=432x368 --image=images/p2.jpg
but this doesn't show any windows by matplotlib
I changed backend code
\home\vagrant.local\lib\python3.6\site-packages\matplotlib\mpl-data\matplotlibrc
backend : Agg => backend : TKAgg
[vagrant#localhost tf-pose-estimation]$ python3 run.py --model=mobilenet_thin --resize=432x368 --image=images/p2.jpg
/home/vagrant/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1)
or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/vagrant/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1)
or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/vagrant/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1)
---except parts of scripts------
TfPoseEstimator/Openpose/MConv_Stage6_L2_5_pointwise/BatchNorm/FusedBatchNorm
TfPoseEstimator/Openpose/concat_stage7/axis
TfPoseEstimator/Openpose/concat_stage7
WARNING:tensorflow:From /home/vagrant/tf-pose-estimation/tf_pose/estimator.py:341: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
2019-09-10 12:57:56,417 WARNING From /home/vagrant/tf-pose-estimation/tf_pose/estimator.py:341: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
WARNING:tensorflow:From /home/vagrant/tf-pose-estimation/tf_pose/estimator.py:342: The name tf.image.resize_area is deprecated. Please use tf.compat.v1.image.resize_area instead.
2019-09-10 12:57:56,420 WARNING From /home/vagrant/tf-pose-estimation/tf_pose/estimator.py:342: The name tf.image.resize_area is deprecated. Please use tf.compat.v1.image.resize_area instead.
WARNING:tensorflow:From /home/vagrant/tf-pose-estimation/tf_pose/tensblur/smoother.py:96: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.
2019-09-10 12:57:56,442 WARNING From /home/vagrant/tf-pose-estimation/tf_pose/tensblur/smoother.py:96: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.
WARNING:tensorflow:From /home/vagrant/tf-pose-estimation/tf_pose/estimator.py:354: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
2019-09-10 12:57:56,463 WARNING From /home/vagrant/tf-pose-estimation/tf_pose/estimator.py:354: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
2019-09-10 12:57:56.563567: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar,
or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
2019-09-10 12:57:57.584627: W tensorflow/core/framework/allocator.cc:107] Allocation of 24164352 exceeds 10% of system memory.
[2019-09-10 12:58:04,331] [TfPoseEstimatorRun] [INFO] inference image: images/p2.jpg in 1.3923 seconds.
2019-09-10 12:58:04,331 INFO inference image: images/p2.jpg in 1.3923 seconds.
Solved it.
Vbox just not on GUI.
Thanks watching.
This teached me how to install GUI on centos7
https://zero-config.com/centos/gnome-0001.html
Using version 3.7 of equinox I can set :
Constants.FRAMEWORK_STORAGE_CLEAN, "true" and
"osgi.console", consolePortasString()
With equinox version 3.9 setting FRAMEWORK_STORAGE_CLEAN to "true" does not work but setting to "onFirstInit" does.
Has the "true"setting been intentionally dropped from equinox?
Also the console port setting does not work in 3.9.
What is the suggested/intended method for setting these values in 3.9?