I have the following error while changing the version of pandas between runs of Streamlit:
AttributeError: Can't get attribute '_unpickle_block' on <module 'pandas._libs.internals' from '/opt/conda/lib/python3.8/site-packages/pandas/_libs/internals.cpython-38-x86_64-linux-gnu.so'>
Since I am using #st.experimental_memo(show_spinner=False, max_entries=3, persist="disk") , it seems that pickled data persisted on disk is still being used, even after killing and reloading the Streamlit app. That is making the pickle manipulations fail.
How can I get rid of that error (or get rid of the cached data)?
The cached data is stored inside ~/.streamlit/cache:
$ ls -al ~/.streamlit/cache
-rw-rw-r-- 1 vvvvv vvvvv 444K nov. 3 16:43 3b9268941beaac500fe825eae1588edd-10e37a1e5ea7a25c0ce5eb261dc18d4d.memo
-rw-rw-r-- 1 vvvvv vvvvv 4 nov. 10 16:20 3b9268941beaac500fe825eae1588edd-23e3001508dcffed68f049cb063bc262.memo
-rw-rw-r-- 1 vvvvv vvvvv 428K nov. 10 16:20 3b9268941beaac500fe825eae1588edd-663346b0bef4b568b0833efd2d033bcf.memo
-rw-rw-r-- 1 vvvvv vvvvv 428K nov. 3 16:44 3b9268941beaac500fe825eae1588edd-a0b908404721bc015e4b911157e41ad1.memo
-rw-rw-r-- 1 vvvvv vvvvv 4 nov. 3 16:43 3b9268941beaac500fe825eae1588edd-e5ac73f7820ca5608235d6d5b20ee1cf.memo
-rw-rw-r-- 1 vvvvv vvvvv 4 nov. 3 16:43 3b9268941beaac500fe825eae1588edd-f05dc4d32c92e4f1f8ca6f1074e7853e.memo
Removing the files here solved my problem.
I found the solution using snehankekre's answer on https://discuss.streamlit.io/t/where-data-is-cached/2931/4.
Note: this solution worked in my case, but the cache could be used by more than one Streamlit app. Therefore, deleting all the files like this only works if only one app is using the cache.
Related
I am currently using MWAA (Airflow version: 2.0.2) to build the data pipeline. cx_Oracle throws the exception while I am trying to connect to Oracle database (Oracle 10g) with the provider.
Execute after install apache-airflow[oracle]==2.0.2
cx_Oracle.DatabaseError: DPI-1047: Cannot locate a 64-bit Oracle Client library: "libclntsh.so: cannot open shared object file: No such file or directory"
Upload plugins.zip with instant_client_12_1 directory within, and overwrite the get_conn function of OracleHook to execute init_oracle_client function that specify the directory of instant client
cx_Oracle.DatabaseError: DPI-1047: Cannot locate a 64-bit Oracle Client library: "libmql1.so: cannot open shared object file: No such file or directory"
but the file do exist under the folder (as below)
-rw-rw-r-- 1 martin martin 29404 Jun 23 07:30 adrci
-rw-rw-r-- 1 martin martin 450 Jun 23 07:30 BASIC_LITE_README
-rw-rw-r-- 1 martin martin 43944 Jun 23 07:30 genezi
lrwxrwxrwx 1 martin martin 21 Jun 23 07:35 libclntshcore.so -> libclntshcore.so.12.1
-rw-rw-r-- 1 martin martin 6990875 Jun 23 07:30 libclntshcore.so.12.1
lrwxrwxrwx 1 martin martin 17 Jun 23 07:35 libclntsh.so -> libclntsh.so.12.1
-rw-rw-r-- 1 martin martin 58793741 Jun 23 07:30 libclntsh.so.12.1
-rw-rw-r-- 1 martin martin 1768370 Jun 23 07:30 libipc1.so
-rw-rw-r-- 1 martin martin 544150 Jun 23 07:30 libmql1.so
-rw-rw-r-- 1 martin martin 6213011 Jun 23 07:30 libnnz12.so
lrwxrwxrwx 1 martin martin 15 Jun 23 07:35 libocci.so -> libocci.so.12.1
-rw-rw-r-- 1 martin martin 2576030 Jun 23 07:30 libocci.so.12.1
-rw-rw-r-- 1 martin martin 6005681 Jun 23 07:30 libociicus.so
-rw-rw-r-- 1 martin martin 156353 Jun 23 07:30 libocijdbc12.so
-rw-rw-r-- 1 martin martin 337137 Jun 23 07:30 libons.so
-rw-rw-r-- 1 martin martin 118491 Jun 23 07:30 liboramysql12.so
-rw-rw-r-- 1 martin martin 3692096 Jun 23 07:30 ojdbc6.jar
-rw-rw-r-- 1 martin martin 3698857 Jun 23 07:30 ojdbc7.jar
-rw-rw-r-- 1 martin martin 227410 Jun 23 07:30 uidrvci
-rw-rw-r-- 1 martin martin 71202 Jun 23 07:30 xstreams.jar
I don't think it is possible to update environment variables like LD_LIBRARY_PATH on MWAA, any ideas? Thanks.
I have solved the issue by following this instruction
Download Oracle instant client and put it into plugins folder.
Execute following Docker commands (I was using Docker Desktop on Windows)
docker pull amazonlinux
docker run -it amazonlinux:latest /bin/bash
yum -y install libaio
cd /lib64
rm libaio.so.1
ln -s ./libaio.so.1.0.1 libaio.so.1 # This is critical for me to change the symbol link to relative path
Execute the following commands from Powershell (as I was using Docker Desktop on Windows)
docker container ls # copy the container ID that you created on step#2
docker cp {docker ID}:/lib64/libaio.so.1 C:\dir\of\oracle\instant\client\under\plugins
docker cp {docker ID}:/lib64/libaio.so.1.0.0 C:\dir\of\oracle\instant\client\under\plugins
docker cp {docker ID}:/lib64/libaio.so.1.0.1 C:\dir\of\oracle\instant\client\under\plugins
Create __ init __.py under plugins folder, or append following codes if you already have one
from airflow.plugins_manager import AirflowPlugin
import os
# the [instantclient] below depends on your folder structure
os.environ["LD_LIBRARY_PATH"]='/usr/local/airflow/plugins/instantclient'
os.environ["DPI_DEBUG_LEVEL"]="64"
class EnvVarPlugin(AirflowPlugin):
name = 'env_var_plugin'
Zip the plugins to plugins.zip
Upload plugins.zip to S3
Update MWAA to use the updated plugins.zip
DONE~
The problem can be solved by building airflow docker with oracle client. Please check the thread
I maintain some legacy code that runs on RH Linux that sends inserts over the network to a client's Sybase. We were using Sybase 12.5 libraries and have just migrated to use Sybase 15.0 client libraries.
My application logs the time at which it sends the insert over the network and also the time it get the acknowledgment back from the target Sybase. When using 12.5 libraries the time was ~5 ms, now with the 15.5 libraries it's roughly 50 ms.
The only change I've made on the application side is to specify the location of the interfaces file on the command line. Previously the file was located in the default location - the location of the Sybase installation. Now it's located where the application is deployed, hence the need to specify the location explicitly.
Would anyone have any idea what is causing the dramatic change in speed, or have hints on where I could look or ideas on how to trace the root cause?
Please forgive the lack of technical details. I'm not a DB admin but a developer using a compiled library to connect to Sybase and don't have access to the nitty-gritty internals. That being said, I'm using the same internal library in both cases, it's only the Sybase librairies that are different.
My Sybase 12.5 and 15 installations look like this:
$ ls -l /opt/sybase/
total 48
-rw-r--r-- 1 root root 555 Jul 2 2019 ASE150.csh
-rw-r--r-- 1 root root 259 Jul 2 2019 ASE150.env
-rw-r--r-- 1 root root 388 Jul 2 2019 ASE150.sh
drwxr-xr-x 10 root root 4096 Feb 2 2017 OCS-15_0
-rw-r--r-- 1 root root 555 Jul 2 2019 SYBASE.csh
-rw-r--r-- 1 root root 259 Jul 2 2019 SYBASE.env
-rw-r--r-- 1 root root 388 Jul 2 2019 SYBASE.sh
drwxr-xr-x 58 root root 4096 Jul 2 2019 charsets
drwxr-xr-x 3 root root 4096 Jul 2 2019 collate
drwxr-xr-x 2 root root 4096 Nov 23 20:55 config
-rw-r--r-- 1 root root 1239 Jul 2 2019 interfaces
drwxr-xr-x 5 root root 4096 Nov 23 20:55 locales
$ ls -l ~/12_5/sybase/
total 28
drwxrwxr-x 4 oadc oadc 4096 Nov 29 2017 OCS-12_5
drwxrwxr-x 58 oadc oadc 4096 Nov 29 2017 charsets
drwxrwxr-x 2 oadc oadc 4096 Mar 16 09:45 config
drwxrwxr-x 2 oadc oadc 4096 Mar 16 09:45 include
-r-xr-xr-x 1 oadc oadc 1184 Mar 16 09:45 interfaces
drwxrwxr-x 2 oadc oadc 4096 Mar 16 09:45 lib
drwxrwxr-x 5 oadc oadc 4096 Mar 16 09:45 locales
EDIT
After some more digging it looks like the libraries under OCS-12-5 are not actually for 12_5 but for 15_5!
$ strings sybase/OCS-12_5lib/libsybct*.a | grep "Sybase Client-Library"
Sybase Client-Library/15.5/P/DRV.15.5.0/Linux x86_64/Linux 2.6.9-55.ELsmp x86_64/BUILD1550-003/64bit/OPT/Mon Oct 5 23:16:48 2009
Sybase Client-Library/15.5/P/DRV.15.5.0/Linux x86_64/Linux 2.6.9-55.ELsmp x86_64 Native Threads/BUILD1550-003/64bit/OPT/Tue Oct 6 00:06:57 2009
Which means that my assumption that 12.5 was faster than 15.0 is wrong. What is actually happening is that 15.5 is faster than 15.0. Which makes more sense.
I'm not going to go hunt down the idiot that submitted these files into a directory labelled OCS-12-5 ...
After some more digging it looks like the libraries under OCS-12-5 are not actually for 12_5 but for 15_5!
$ strings sybase/OCS-12_5lib/libsybct*.a | grep "Sybase Client-Library" Sybase Client-Library/15.5/P/DRV.15.5.0/Linux x86_64/Linux 2.6.9-55.ELsmp x86_64/BUILD1550-003/64bit/OPT/Mon Oct 5 23:16:48 2009 Sybase Client-Library/15.5/P/DRV.15.5.0/Linux x86_64/Linux 2.6.9-55.ELsmp x86_64 Native Threads/BUILD1550-003/64bit/OPT/Tue Oct 6 00:06:57 2009
Which means that my assumption that 12.5 was faster than 15.0 is wrong. What is actually happening is that 15.5 is faster than 15.0. Which makes more sense.
I've updated the question with this new information.
I am following complete tutorial from the link: https://piotrminkowski.wordpress.com/2018/04/26/quick-guide-to-microservices-with-spring-boot-2-0-eureka-and-spring-cloud/ and in this tutorial I've upgraded spring-boot-starter-parent to 2.1.3.RELEASE and spring-cloud-dependencies version to Greenwich.SR1.
config-service I've started successfully on port 8088, but when I start the discovery-service, it's not Fetching config from server at : http://localhost:8088.
I run the config-service with the
-Dspring.profiles.active=native
File System
$ ls -ltr
total 10
-rw-r--r-- 1 user 1049089 233 Apr 2 09:45 discovery-service.yml
-rw-r--r-- 1 user 1049089 212 Apr 2 09:45 employee-service.yml
-rw-r--r-- 1 user 1049089 206 Apr 2 09:45 employee-service-instance2.yml
-rw-r--r-- 1 user 1049089 925 Apr 2 09:45 gateway-service.yml
-rw-r--r-- 1 user 1049089 212 Apr 2 09:45 organization-service.yml
-rw-r--r-- 1 user 1049089 486 Apr 2 09:45 proxy-service.yml
-rw-r--r-- 1 user 1049089 212 Apr 2 09:45 department-service.yml
I was able to solve this issue by adding correct spring-cloud-starter-config
Here is the dependency that you need to add -
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-config</artifactId>
</dependency>
How do I add the lz4 native libraries for use by Spark workers?
I have tried to add them via both LD_LIBRARY_PATH and ( as shown - but no accepted or even upvoted answer - in Apache Spark Native Libraries ) - in SPARK_LIBRARY_PATH. They are not working: we get:
java.lang.RuntimeException: native lz4 library not available
at org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1201)
at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1094)
at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1444)
at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:277)
at BIDMat.HDFSIO.writeThing(HDFSIO.scala:96)
Here is the LD_LIBRARY_PATH
$echo $LD_LIBRARY_PATH
/usr/local/Cellar/lz4/r131/lib:/usr/local/Cellar/hadoop/2.7.2/libexec/lib:
12:15:35/BIDMach_Spark $ll /usr/local/Cellar/lz4/r131/lib
and the contents of the lz4 related entry:
$ll /usr/local/Cellar/lz4/r131/lib
total 528
-r--r--r-- 1 macuser admin 71144 Sep 21 2015 liblz4.a
drwxr-xr-x 7 macuser admin 238 Sep 21 2015 .
drwxr-xr-x 3 macuser admin 102 Jun 13 10:41 pkgconfig
-r--r--r-- 1 macuser admin 64120 Jun 13 10:41 liblz4.dylib
-r--r--r-- 1 macuser admin 64120 Jun 13 10:41 liblz4.1.dylib
-r--r--r-- 1 macuser admin 64120 Jun 13 10:41 liblz4.1.7.1.dylib
Update your hadoop jars and should work perfectly fine.
I want to build a dmg with a executable with debug symbols so I can take to
another machine and debug it there. If I use macdeployqt with -use-debug-libs
the dmg does seem to have debug symbols and I can use it on the machine it was
built on.
But when I copy the dmg to another mac it fails with:
Library not loaded: #rpath/QtOpenGL.framework/Versions/5/QtOpenGL
Reason: image not found
If I build the dmg without -use-debug-libs and copy the dmg to
another machine it does work. This seems like a bug in macdeployqt.
More info:
They both reference the same lib:
#rpath/QtOpenGL.framework/Versions/5/QtOpenGL (compatibility version 5.5.0, current version 5.5.1)
The non-debug Frameworks dir has this:
lrwxrwxr-x 1 LarryMartell staff 25 Mar 15 11:00 QtOpenGL -> Versions/Current/QtOpenGL
lrwxrwxr-x 1 LarryMartell staff 26 Mar 15 11:00 Resources -> Versions/Current/Resources
drwxrwxr-x# 4 LarryMartell staff 136 Mar 15 11:00 Versions
And the debug one has this:
lrwxrwxr-x 1 LarryMartell staff 31 Mar 15 10:41 QtOpenGL_debug -> Versions/Current/QtOpenGL_debug
lrwxrwxr-x 1 LarryMartell staff 26 Mar 15 10:41 Resources -> Versions/Current/Resources
drwxrwxr-x# 4 LarryMartell staff 136 Mar 15 10:41 Versions