Julia: limit on the number of files open while using readdlm - macos

I have some Julia code where I open a very large number of files using the readdlm function. My code looks like this:
for file in large_list_of_files
open(file) do filehandle
data = readdlm(filehandle)
end
#Do some data processing and store results
end
When I run this I get the error:
LoadError: SystemError: opening file <filename>: Too many open files
while loading In[28], in expression starting on line 28
in open at /Applications/Julia-0.4.0.app/Contents/Resources/juli/lib/julia/sys.dylib
in open at iostream.jl:102
in open at iostream.jl:112
in process_data at In[13]:11
[inlined code] from In[28]:31
in anonymous at no file:30
This doesn't make sense, since the do block should be closing the file every time I go around the loop, therefore the file should only be open in the do block. Also, if anyone knows a function in Julia that can count the number of files that are open, then that would be great :)
System info:
Mac OSX, Yosemite
Julia Version 0.4.0
Commit 0ff703b* (2015-10-08 06:20 UTC)
Platform Info:
System: Darwin (x86_64-apple-darwin13.4.0)
CPU: Intel(R) Core(TM) i7-3720QM CPU # 2.60GHz
WORD_SIZE: 64
BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
LAPACK: libopenblas64_
LIBM: libopenlibm
LLVM: libLLVM-3.3
Also, I am running this code using Jupyter with a Julia 0.4.0 kernel, but I doubt that that would make any difference.
EDIT: I noticed this issue on Github, ViralBShah says that the problem is fixed but I don't think that it is.
https://github.com/JuliaLang/julia/issues/8891
EDIT EDIT: I have tried using the gc() function and that doesn't fix the problem either.

Seems to be solved:
Tested under
MacOS HighSierra 10.13.6
julia version 1.5.1
using DelimitedFiles
# 329571 files:
for file in split(String(read(a)),"\n")[1:end-1]
open(file) do fh
try readdlm(fh)
catch ff
end
end
end

Related

PlatformIO - STM32 - Library not loaded: /opt/local/lib/libusb-1.0.0.dylib [upload] Error 134

I am trying to program my STM32 via PlatformIO. I am using the built in blink example so I 'know; that my code is not what is causing the issue. I am using macOS - M1 Max - Monterey(12.13.1).
Here is the board I am using: https://ubld.it/products/stm32-midi-dev
The program builds just fine, but when trying to upload, I get the following error:
dyld[54244]: Library not loaded: /opt/local/lib/libusb-1.0.0.dylib
Referenced from: /Users/***/.platformio/packages/tool-stm32duino/dfu-util/dfu-util
Reason: tried: '/opt/local/lib/libusb-1.0.0.dylib' (no such file), '/usr/local/lib/libusb-1.0.0.dylib' (no such file), '/usr/lib/libusb-1.0.0.dylib' (no such file)
/Users/***/.platformio/packages/tool-stm32duino/maple_upload: line 53: 54244 Abort trap: 6 ${DFU_UTIL} -d ${usbID} -a ${altID} -D ${binfile} -R ${dfuse_addr} -R
*** [upload] Error 134
My platformio.ini file is as follows:
[env:genericSTM32F103CB]
platform = ststm32
board = genericSTM32F103CB
framework = arduino
board_build.core = maple
upload_protocol = dfu
I have tried the following:
brew install libusb - but it was already installed
brew uninstall libusb - then - brew install libusb
Library not loaded: /opt/local/lib/libssl.1.0.0.dylib (LoadError) - This seemed to be more SSL related, but tried anyways
I have read the error message and understand that it appears that a file does not exist, but unsure of what next steps would be as I tried reinstalling libusb above.
Here is my full upload output if helpful:
CONFIGURATION: https://docs.platformio.org/page/boards/ststm32/genericSTM32F103CB.html
PLATFORM: ST STM32 (15.3.0) > STM32F103CB (20k RAM. 128k Flash)
HARDWARE: STM32F103CBT6 72MHz, 20KB RAM, 128KB Flash
DEBUG: Current (blackmagic) External (blackmagic, cmsis-dap, jlink, stlink)
PACKAGES:
- framework-arduinoststm32-maple # 3.10000.201129 (1.0.0)
- tool-dfuutil # 1.9.211020
- tool-openocd # 2.1100.211028 (11.0)
- tool-stm32duino # 1.0.1
- toolchain-gccarmnoneeabi # 1.70201.0 (7.2.1)
LDF: Library Dependency Finder -> https://docs.platformio.org/en/latest/librarymanager/ldf.html
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 30 compatible libraries
Scanning dependencies...
No dependencies
Building in release mode
Checking size .pio/build/genericSTM32F103CB/firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM: [== ] 21.1% (used 4312 bytes from 20480 bytes)
Flash: [= ] 14.3% (used 18712 bytes from 131072 bytes)
Configuring upload protocol...
AVAILABLE: blackmagic, cmsis-dap, dfu, jlink, serial, stlink
CURRENT: upload_protocol = dfu
Looking for upload port...
Auto-detected: /dev/cu.usbmodem1101
Uploading .pio/build/genericSTM32F103CB/firmware.bin
Failed to open serial device.
dyld[54773]: Library not loaded: /opt/local/lib/libusb-1.0.0.dylib
Referenced from: /Users/***/.platformio/packages/tool-stm32duino/dfu-util/dfu-util
Reason: tried: '/opt/local/lib/libusb-1.0.0.dylib' (no such file), '/usr/local/lib/libusb-1.0.0.dylib' (no such file), '/usr/lib/libusb-1.0.0.dylib' (no such file)
/Users/***/.platformio/packages/tool-stm32duino/maple_upload: line 53: 54773 Abort trap: 6 ${DFU_UTIL} -d ${usbID} -a ${altID} -D ${binfile} -R ${dfuse_addr} -R
*** [upload] Error 134
Thanks in advance for the help. Pretty new to embedded.
=== EDIT ===
As I continue to research, I think it is related to homebrew not installing in the normal spot for libusb. Homebrew Mac M1 can't find installs
Ok. Thanks to Ivan at PlatformIO, he was able to give me the below resolution. It now works great.
There are no new versions of STM32Duino /dfu-binaries for macOS ARM:
https://github.com/stm32duino/Arduino_Tools/tree/main/macosx/dfu-util
Could I ask you to try the next things?
Edit ~/.platformio/packages/tool-stm32duino/maple_upload
Comment line 43 and add a new one below
DFU_UTIL=~/.platformio/packages/tool-dfuutil/bin/dfu-util

Fault in reticulate in rstudio

476/5000
rstudio generates error when running R code with python code. At first the code worked very well but in the second installation it started to generate faults. Here is the link of the code in case you want to play #https: //www.r-bloggers.com/r-and-python-using-reticulate-to-get-the-best-of-both-worlds/
. My laptop is hp 4 gigs of ram and 500 mb of hard disk.
I would greatly appreciate who can explain the failure and how to solve it.
I have reinstalled anaconda last version twice.
library(reticulate)
use_python("/usr/local/bin/python")
use_virtualenv("r-reticulate")
os <- import("os")
Error: lexical error: invalid char in json text.
WARNING: The conda.compat modul
(right here) ------^
os$listdir(".")
Error: object 'os' not found
py_config()
Error: lexical error: invalid char in json text.
WARNING: The conda.compat modul
(right here) ------^
Actually, it comes from a deprecation warning reported a few days ago (fix compat import showing warning #8507).
But the fact is that this warning in "conda.compat" generates some errors, for example in the case of using conda environments in R through "reticulate" functions. This issue has been reported quite recently Conda warnings taken as errors #477.
The solution is to simply install the github version of "reticulate" in order to avoid these errors and ignore the conda warnings which shouldn't necessarily impede these functionalities. You can do it like this way:
remotes::install_github("rstudio/reticulate")
And don't forget to have the latest Rtools installed in your operating system!
This might be useful for you: https://docs.anaconda.com/anaconda/user-guide/tasks/switch-environment/
I have Anaconda, and set up Python 2 environment. Once activated, run 'use_python' with the path to that environment and set the 'required' parameter to TRUE.
I had the same problem. I solved the problem by installing an older version of conda.
In anaconda promt
conda install conda = 4.1.6

Installing Perl and PDL

I am currently trying - unsuccessfully to install perl PDL as I would like to automate some of my data analysis and graphics generation.
Unfortunately I seem unable to even install the necessary modules without issue, let alone create a small script to deal with my data files.
I have some rudementary experience with Perl but Nothing fancy and am using Windows 10 x64.
Initially I went for the x64 strawberry perl v5.24 installation, followed by ppm installations of Astro::FITS::Header, Convert::UU, OpenGL and PDL - as instructed here http://pdl.perl.org/?page=install. However, there were issues with the installation of Inline::C and after trying the example listed on page 5 of the PDL_Book_Latest http://sourceforge.net/projects/pdl/files/PDL/2.4.10/PDL-Book-20120205.pdf/download there wasn't any image appearing, nor image file in any location I could think to search. Back to Google and still no luck finding any information on the source of this issue so I decided to go back to square one and use an older version in the hopes that it would be more well documented.
So, uninstallation of strawberry perl through Windows installed programs dialog, deletion of strawberry folder and registry entries containing "strawberry". Then a new installation of perl, with the Padre IDE, v5.14 - maybe some debugging fonctionality will help pinpoint the errors. Started installing the same modules (Astro::FITS::Header, Convert::UU, OpenGL, PGPLOT). Astro and Convert worked fine, but the OpenGL started throwing errors about ExtUtils::F77 which I am apparently missing after a few more hours searching. I go to install this module and it keeps giving errors about 'gfortan' being an unknown command... It was my understanding that the idea of PDL was to avoid fortran and C, and I have found next to no information about this installation problem so far.
So i decided to try an alternative approach. I installed the ppm module from cpan and repeated the initial installation process through the ppm command. This time i get:
"Unknown element 'PROVIDE' found inside SOFTPKG. at C:/Dwimperl/perl/site/lib/PPM.pm line 1462."
So my question is as follows:
Am I missing some critical step in the installation process? Is my system just handing this really badly, and if so is there any sort of possible work around? Is the fact that I tried two different versions liable to create path issues that could be at the root of this problem?
Could anyone provide an alternative and preferably not too complex route to install PDL with a given perl installation on Windows 10?
EDIT 1:
Here is the code for the error I got trying to install Inline::C, an apparent pre-requisite for PDL
code deleted to make space for updates...
EDIT 3:
I reinstalled GnuWin32 in the directory specified by PATH, rather that changing PATH, and the tried the installation again. Inline::C gave the same 'diff' and 'rm' errors. So i ran "cpanm --notest 'package'" to install Inline::C and then PDL, both said they were successful. However, even using '--notest' cpanm could not install OpenGL and PGPLOT.
The final objectif is to be able to produce graphics using the fonctions from PGPLOT: for example
# use PDL;
# use PDL::Graphics::PGPLOT;
# imag(sin(rvals(200,200)+1));
yet this doesn't work of course if PGPLOT can't install.
EDIT 4:
Tried the fix found by #Dr.Avalange at sourceforge https://sourceforge.net/p/pogl/bugs/26/ and noticed that I seem to have multiple copies of this file.
Here
Is this normal...?
EDIT 5:
So i uninstalled perl, deleted any lingering files I could find in %APPDATA%, %USERPROFILE% and C:/DWIM, and then ran ccleaner just in case I missed anything.
I then downloaded Strawberry perl 5.24.1 x64 for Windows and installed it.
After this, I ran 'cpanm Astro::FITS::Header', cpanm 'Convert-UU' and 'cpanm OpenGL' as per http://pdl.perl.org/?page=install. This all worked fine. Then I ran 'cpanm PDL' which installed Inline, Pegex, Win32::Mutex and Module::Compile dependencies. However, if had the same error with Inline::C that you mentioned previously.
I already had GetGnuWin32 installed before so not sure why this isn't working. The Windows 'Path' environment (not PATH - this doesn't exist according to the advaced system dialogue) is noted as
%USERPROFILE%\AppData\Local\Microsoft\WindowsApps;
In this directory I have the folder GetGnuWin32 which should have installed all the packages from the link you provided, unless there is a supplementary step that I missed to install this correctly? So I redownloaded the DiffUtils setup and ran that, installing to \programfiles(x86) (default option). I then reran 'cpanm PDL'.
I still get the same 'diff' error message. So is DiffUtils, not installing properly? (I did close and re-open the commandline between tries and after the installation)
EDIT 6:
So after installing PDL and PGPLOT with --notest I tried to run the following test and it failed. See below
C:\Windows\system32>perldl
perlDL shell v1.357
PDL comes with ABSOLUTELY NO WARRANTY. For details, see the file
'COPYING' in the PDL distribution. This is free software and you
are welcome to redistribute it under certain conditions, see
the same file for details.
ReadLines, NiceSlice, MultiLines enabled
Reading PDL/default.pdl...
Found docs database C:/Strawberry/perl/site/lib/PDL/pdldoc.db
Type 'help' for online help
Type 'demo' for online demos
Loaded PDL v2.017 (supports bad values)
Note: AutoLoader not enabled ('use PDL::AutoLoader' recommended)
pdl> use PDL::Graphics//PLplot
Unknown regexp modifier "/P" at (eval 62) line 4, at end of line
Unknown regexp modifier "/L" at (eval 62) line 4, at end of line
Unknown regexp modifier "/t" at (eval 62) line 4, at end of line
BEGIN not safe after errors--compilation aborted at (eval 62) line 5, <DATA> line 207.
pdl> use PDL::Graphics::PLplot
Can't locate PDL/Graphics/PLplot.pm in #INC (you may need to install the PDL::Graphics::PLplot module) (#INC contains: C:/Strawberry/perl/site/lib/MSWin32-x64-multi-thread C:/Strawberry/perl/site/lib C:/Strawberry/perl/vendor/lib C:/Strawberry/perl/lib .) at (eval 72) line 5.
BEGIN failed--compilation aborted
pdl> use PDL
pdl> use lib 'C:/Strawberry/perl/site/lib/PDL/Graphics'
pdl> use PGPLOT
pdl> use PDL::Graphics::PGPLOT
pdl> imag(sin(rvals(200,200)+1))
Undefined subroutine &PDL::Graphics::PGPLOT::pgqinf called at C:/Strawberry/perl/site/lib/PDL/Graphics/PGPLOT.pm line 408, <DATA> line 90.
pdl> exit
C:\Windows\system32>cpanm PDL::Graphics::PGPLOT::pgqinf
! Finding PDL::Graphics::PGPLOT::pgqinf on cpanmetadb failed.
! Finding PDL::Graphics::PGPLOT::pgqinf () on mirror http://www.cpan.org failed.
! Couldn't find module or a distribution PDL::Graphics::PGPLOT::pgqinf
C:\Windows\system32>cpan PDL::Graphics::PGPLOT::pgqinf
Loading internal null logger. Install Log::Log4perl for logging messages
CPAN: CPAN::SQLite loaded ok (v0.211)
Database was generated on Thu, 13 Apr 2017 13:49:14 GMT
C:\Windows\system32>
There was already a directory problem but I managed to point it in th right direction I had thought. Also tried installing the PLplot 'cpanm' said the installation was successful but there is no .pm file, only a .pd in another directory and so the use PDL::Graphics::PLplot fails as there is no .pm file to find.
EDIT: 7
So retrying ppm install http://www.sisyphusion.tk/ppm/PGPLOT.ppd seemed to work this time as the links int he explanation you provided are no longer in use
http://www.kalinabears.com.au/w32perl/pgplot-5.2.2-mingw32.tar.bz2
http://jrfonseca.home.dyndns.org/projects/gnu-win32/software/ported/patches/pgplot-5.2.2-mingw32.diff.gz
And while this seemed to work, there are no traces of the files that were listed as necessary for the installation, unless they were just for this specific method?
pgplot/bin/cpgplot.dll
pgplot/bin/pgplot.dll
pgplot/bin/grfont.dat
pgplot/include/cpgplot.h
pgplot/lib/cpgplot.a
pgplot/lib/pgplot.a
finally, trying the example again gives the following; no errors, but no image either...
Microsoft Windows [version 10.0.14393]
(c) 2016 Microsoft Corporation. Tous droits réservés.
C:\Windows\system32>perldl
perlDL shell v1.357
PDL comes with ABSOLUTELY NO WARRANTY. For details, see the file
'COPYING' in the PDL distribution. This is free software and you
are welcome to redistribute it under certain conditions, see
the same file for details.
ReadLines, NiceSlice, MultiLines enabled
Reading PDL/default.pdl...
Found docs database C:/Strawberry/perl/site/lib/PDL/pdldoc.db
Type 'help' for online help
Type 'demo' for online demos
Loaded PDL v2.017 (supports bad values)
Note: AutoLoader not enabled ('use PDL::AutoLoader' recommended)
pdl> use PDL
pdl> use PDL::Graphics::PGPLOT
pdl> imag(rvals(200,200)+1)
%PGPLOT, Unable to read font file: grfont.dat
%PGPLOT, Use environment variable PGPLOT_FONT to specify the location of the PGPLOT grfont.dat file.
Displaying 200 x 200 image from 1 to 142.421356201172, using 240 colors (16-255)...
pdl> $ENV{PGPLOT_FONT} = "C:/Strawberry/perl/site/lib/PGPLOT/pgplot_supp"
pdl> imag(rvals(200,200)+1)
Displaying 200 x 200 image from 1 to 142.421356201172, using 240 colors (16-255)...
pdl>
After translating from French I can see that you don't have diff in your system. Either install diff (part of gnu32: http://gnuwin32.sourceforge.net/) or just skip the tests. See also:
https://github.com/ingydotnet/inline-c-pm/issues/60
With Strawberry 5.24.1 simply type the following from the command prompt:
cpanm --notest PDL
or if you want to watch everything that's going on:
cpanm -v --notest PDL
This will take some time to run.

"ERROR: broken data stream when reading image file" (TrueType fonts libjpeg and PIL on OS X)

I'm running OS X 10.6.7, Python 2.6, and writing a small wxPython application that uses PIL to manipulate an image and add some text to it.
After quite a bit of configuration woes with attempting to get PIL running with libjpeg and PIL, I followed the instructions at this site.
Upon running python setup.py build_ext -i (from inside the Imaging-1.1.7 folder) I get:
--------------------------------------------------------------------
PIL 1.1.7 SETUP SUMMARY
--------------------------------------------------------------------
version 1.1.7
platform darwin 2.6.1 (r261:67515, Jun 24 2010, 21:47:49)
[GCC 4.2.1 (Apple Inc. build 5646)]
--------------------------------------------------------------------
--- TKINTER support available
--- JPEG support available
--- ZLIB (PNG/ZIP) support available
--- FREETYPE2 support available
--- LITTLECMS support available
--------------------------------------------------------------------
Additionally, running python selftest.py in the Imaging-1.1.7 install folder, I get:
--------------------------------------------------------------------
PIL 1.1.7 TEST SUMMARY
--------------------------------------------------------------------
Python modules loaded from ./PIL
Binary modules loaded from ./PIL
--------------------------------------------------------------------
--- PIL CORE support ok
--- TKINTER support ok
--- JPEG support ok
--- ZLIB (PNG/ZIP) support ok
--- FREETYPE2 support ok
--- LITTLECMS support ok
--------------------------------------------------------------------
Running selftest:
--- 57 tests passed.
However, when rendering my first PIL image in my own code, I still get:
File "/Library/Python/2.6/site-packages/PIL/Image.py", line 1290, in resize self.load()
File "/Library/Python/2.6/site-packages/PIL/ImageFile.py", line 215, in load raise_ioerror(e)
File "/Library/Python/2.6/site-packages/PIL/ImageFile.py", line 52, in raise_ioerror
raise IOError(message + " when reading image file")
IOError: broken data stream when reading image file
I'm kind of at a loss as to what to try next. Any ideas? The code that is causing this crash seems to be:
try:
self.img = self.img.resize((self.screenwidth, self.screenheight),
Image.ANTIALIAS)
except IOError, e:
print "Error resizing: " + str(e)
self.img = self.img.resize((self.screenwidth, self.screenheight))
raise
You should consider uninstalling PIL and following the instructions on this website -
http://jetfar.com/libjpeg-and-python-imaging-pil-on-snow-leopard/
In particular, it appears that you want to make sure that libjpeg is at version 6b, not 8c (see below)
I am able to reproduce this with Mac OS X 10.5 using macports
This is related to a another question:
How to solve IOError:broken data stream when reading image file?
The answer there indicates that the bug might have something to do with libjpeg
As far as I can tell, the error appears to be caused by a forward incompatibility with libjpeg v8c.
I cannot determine what is causing this exactly; all I can say is that on a variety of other systems I use that aren't exhibiting this behavior error (OS X 10.5 with fink, Redhat 5.2, Ubuntu 11.10 and 10.10), they all use libjpeg 6b. Only macports uses 8c, and macports is the only system where I am getting this error.
I was able to use PIL with the homebrew package manager.
I've got a similar error when reading .jpg file with an up-to-date Python environment and Sierra Mac OS. I've found a quick workaround which consists to convert all the .jpg files to .png files Hoping that can help someone else.

Ocropus 4.4. crashes on Xubuntu 10.4

I have problems with Ocropus 4.4. (open source OCR). I installed Ocropus and neccessary libries following this script. Compilation/Installation goes without any problem.
However after successful installation of ocropus, I am not able to run any of ocropus-* commands. Ocropus-page and ocropus-hocr crash without giving any error meaningful message. I just get a cryptic killed after 2 minutes of processing.
ocropus-pages page_001.jpeg
[note] line recognizer: >
[note] *** 1 page_001.jpeg ***
Killed
I do not know where to start to fix it. Has anybody an idea where to start?
My machine is XUbuntu 10.4 64bits (run in VirtualBox) with 512MB of base memory.
I would try to convert your images to uncrompressed tiff before processing. I know that ocropus is working closely with tesseract and that tesseract handles 1bpp uncompressed tiffs the best.
Despite this being an old question I thought I'd post a solution / some debug tips as I experienced the same problem under the same scenario. The scenario being: Debian running under VirtualBox with 512MB RAM
In order to debug "Killed" messages you should look into /var/log for help. In my case kern.log contained:
debian kernel: [89675.791910] Out of memory: Kill process 13004
(ocropus-rtrain) score 806 or sacrifice child
debian kernel: [89675.791951] Killed process 13004 (ocropus-rtrain)
total-vm:800816kB, anon-rss:412424kB, file-rss:4kB
It's likely you need to increase the amount of resources assigned to the VM particularly if you're running some big fancy window manager in the background!

Resources