LWP connect issues despite fresh install - windows
Update
Working on a theory, I edited LWP/Protocol/http.pm to include a sleep statement in the subroutine request:
if (!$has_content || $write_wait || $has_content > 8*1024) {
WRITE:
{
# Since this just writes out the header block it should almost
# always succeed to send the whole buffer in a single write call.
my $n = $socket->syswrite($req_buf, length($req_buf));
sleep 2; ## <----- NEW
unless (defined $n) {
...
And the get statement worked, returning a 200 OK. Much thanks for Alan Curry for help with debugging and finding this particular place in the code.
Not sure it completely answers the question, or if the solution works long term. Will have to do some more checking.
Summary:
LWP::UserAgent module using the get subroutine fails for some URLs, reporting 500 timeout.
Only some URLs fail. E.g. www.google.com fails, but www.google.se succeeds.
I have no other connection issues, all URLs are reachable with browser and through cmd programs such as ping.
Because of this problem, I cannot install modules for perl with CPAN or ActivePerl's ppm.
The problem persisted after installing another perl distribution.
Weirdly enough, using the debugger and stepping through the code makes the failing URLs succeed.
I am using a firewall, and perl is allowed to make connections. (Not relevant, since some URLs succeed)
Firewall log shows perl being allowed to connect for both failing URLs and non-failing. (See below) The log also shows sockets opening to listen, but timestamps are mismatched for failing connections.
Goal
I'm primarily looking for any solution to be able to install modules.
I'm interested in all suggestions on how to debug the problem, complete solutions not required. Any hints or tips are welcome.
Elaboration
I have been using ActivePerl v5.14 for some time. Installing modules with their Perl Package Manager ppm command and gui worked very well, but at some point stopped working, reporting a 500 timeout. The cpan shell reported the very same thing.
I have googled this problem extensively, but found nothing that relates to my problem, or helps in any way.
ActivePerl support claims it may be a proxy setting, which is ludicrous. I have lots of programs that connect to the internet that do not need proxy settings, and as far as I know, I do not need to do this. I have tried to find out what my proxy settings are, if any, but the only thing I have found is vague references such as "use the system settings", "no proxy required" and "proxy is the same as your IP".
So last night I had enough and installed strawberry perl instead, but it suffers from the same problem. I uninstalled ActivePerl afterwards.
Anyway, I have experimented with LWP modules and found that I can reproduce the errors there. It seems it is limited to certain websites, and cpan is one of them (?). I created this script for testing:
use strict;
use warnings;
use LWP::UserAgent;
use URI;
my $ua = LWP::UserAgent->new;
my $url = shift;
my $u = URI->new($url);
$ua->no_proxy('cpan.strawberryperl.com','cpan.com',$u->host);
$ua->timeout(30);
my $r = $ua->get($url);
if ($r->is_success) {
print $r->decoded_content;
} else {
die $r->status_line;
}
And then did some testing:
tx.pl http://cpan.strawberryperl.com/authors/01mailrc.txt.gz
500 read timeout at tx.pl line 23.
tx.pl http://stackoverflow.com
500 read timeout at tx.pl line 23.
tx.pl http://www.google.se
<!doctype html><html itemscope itemtype="http://schema.org/WebPage"><head><meta
http-equiv="content-type" content="text/html; charset=ISO-8859-1"><meta ...
So, google works, and www.youtube.com also works, but www.yahoo.com and search.cpan.com fails. The default timeout of 180 seconds makes this an incredibly annoying thing to debug, which is why I reduced it in my script. Needless to say, all of these URLs are reachable if I try to reach them with Firefox or ping.
ETA:
Strangely enough, running the script through the debugger, turning on trace and skipping to the end makes the previously failed connections successful.
It would seem to imply that there is some kind of hiccup, missed timing that is "fixed" when the script runs more slowly due to printing thousands of lines of trace code.
I could understand this issue as being a result of some ActivePerl module getting corrupted, but strawberry perl is using a completely different set of files, so it must be my system.
Why some sites work and some don't is baffling. I could understand that some sites like stackoverflow.com would protect themselves against potential bots, but why cpan would thwart its own package manager makes no sense.
I am using a firewall, and Perl has been allowed to make connections. My system is a rather old installation of Windows XP (~5 years). While running dual boot with Ubuntu I've never encountered this problem, which is another clue that it is not something to do with proxies.
I am well and truly stumped. If anyone could help me debug this, I would be very grateful.
The CPAN shell error messages below. The funny thing is, it says it tries to use the ftp as a last resort, but I just discovered that the ftp command has not been allowed by my firewall, and if it was used, it should have asked me for permission.
Fetching with LWP:
http://cpan.strawberryperl.com/authors/01mailrc.txt.gz
LWP failed with code[500] message[read timeout]
Warning: no success downloading 'D:\strawberry\cpan\sources\authors\01mailrc.txt
.gz.tmp1252'. Giving up on it.
Fetching with LWP:
http://www.cpan.org/authors/01mailrc.txt.gz
LWP failed with code[500] message[read timeout]
Warning: no success downloading 'D:\strawberry\cpan\sources\authors\01mailrc.txt
.gz.tmp1252'. Giving up on it.
Warning: no success downloading 'D:\strawberry\cpan\sources\authors\01mailrc.txt
.gz.tmp1252'. Giving up on it.
As a last resort we now switch to the external ftp command 'C:\WINDOWS\system32\
ftp.EXE'
to get 'D:\strawberry\cpan\sources\authors\01mailrc.txt.gz.tmp1252'.
Doing so often leads to problems that are hard to diagnose.
If you're the victim of such problems, please consider unsetting the
ftp config variable with
o conf ftp ""
o conf commit
Please check, if the URLs I found in your configuration file
(http://cpan.strawberryperl.com/, http://www.cpan.org/) are valid. The
urllist can be edited. E.g. with 'o conf urllist push ftp://myurl/'
Could not fetch authors/01mailrc.txt.gz
Firewall log for trying to fetch non-failing URL (www.google.se) and failing (stackoverflow.com):
2012-06-27T18:34:04+01:00,info,appl control,C:\WINDOWS\system32\svchost.exe,allow,listen,17,0.0.0.0,56564
2012-06-27T18:34:04+01:00,info,appl control,C:\WINDOWS\system32\svchost.exe,allow,send,17,195.54.122.198,53
2012-06-27T18:34:13+01:00,info,appl control,D:\strawberry\perl\bin\perl.exe,allow,connect out,6,64.34.119.12,80
2012-06-27T18:34:13+01:00,info,appl control,D:\strawberry\perl\bin\perl.exe,allow,connect out,6,64.34.119.12,80
2012-06-27T18:34:21+01:00,info,appl control,C:\Program\Mozilla Firefox\firefox.exe,allow,connect out,6,74.86.70.106,80
2012-06-27T18:34:28+01:00,info,appl control,C:\Program\Mozilla Firefox\firefox.exe,allow,connect out,6,64.34.119.12,80
2012-06-27T18:34:30+01:00,info,appl control,C:\WINDOWS\system32\svchost.exe,allow,listen,17,0.0.0.0,56664
2012-06-27T18:34:30+01:00,info,appl control,C:\WINDOWS\system32\svchost.exe,allow,send,17,195.54.122.198,53
2012-06-27T18:34:30+01:00,info,appl control,D:\strawberry\perl\bin\perl.exe,allow,connect out,6,74.125.143.94,80
2012-06-27T18:34:30+01:00,info,appl control,D:\strawberry\perl\bin\perl.exe,allow,connect out,6,74.125.143.94,80
2012-06-27T18:35:14+01:00,info,appl control,C:\Program\Mozilla Firefox\firefox.exe,allow,connect out,6,64.34.119.12,80
2012-06-27T18:35:21+01:00,info,appl control,C:\Program\Mozilla Firefox\firefox.exe,allow,connect out,6,74.86.70.106,80
2012-06-27T18:36:21+01:00,info,appl control,C:\Program\Mozilla Firefox\firefox.exe,allow,connect out,6,74.86.70.106,80
2012-06-27T18:37:04+01:00,info,appl control,C:\WINDOWS\system32\svchost.exe,allow,listen,17,0.0.0.0,61215
2012-06-27T18:37:04+01:00,info,appl control,C:\WINDOWS\system32\svchost.exe,allow,send,17,195.54.122.198,53
2012-06-27T18:37:07+01:00,info,appl control,D:\strawberry\perl\bin\perl.exe,allow,connect out,6,64.34.119.12,80
2012-06-27T18:37:07+01:00,info,appl control,D:\strawberry\perl\bin\perl.exe,allow,connect out,6,64.34.119.12,80
This might not be a complete solution to your problem. But here it is anyway:
From your "detailed" problem description it looks like it's a problem with your desktop/laptop. Even though your firewall allows connections to websites as you mentioned, "FTP" might not be allowed by the Windows internal firewall.
Usually, ports 20 (FTP command port) and 21 (FTP data port) should have been added to the firewall exceptions (In Windows - Start → Settings → Control Panel → Click on Security Center → Firewall → Exceptions (tab) → Add ports. You can try adding ports 20 and 21 to the exceptions.
However, if you are connected to a router, you may have to port forward ports 20 and 21. However, these ports are forwarded by default, or if you are in a corporate VPN then it's a whole different story. Corporate VPNs, mostly restrict port 21 explicitly however allow port 22 (which is a secured version of port 21, for SFTP). Under such circumstances you may want to use ftp_proxy.
Alternatively (if you don't want to add port 20 and 21 to exception), you can go to the cpan prompt and use an ftp_proxy by:
cpan> o conf ftp_proxy http://your.ftpproxy.com
and then issue the install <module> command. Or you can update your ../CPAN/config.pm file to make permanent changes to the ftp_proxy parameter.
Well, these may be the traditional solutions which you probably already tried. The next step would be to try set the FTP_PASSIVE mode to 1. By default the libnetcfg configuration for this is set to 0. To change this, find the libnetcfg.bat file (it should be somewhere C:\Perl\bin). Open the file in an editor and replace
ftp_int_passive 0
with
ftp_int_passive 1
This is the Windows batch file that runs once CPAN is invoked to set the environment variables. Under a UNIX/Linux-like architecture it's found as libnet.cfg and environment variable FTP_PASSIVE, like
$set | grep FTP_PASSIVE
FTP_PASSIVE=0
so to set just EXPORT FTP_PASSIVE=1.
These might be a few of the very many ways of debugging this. Honestly, there is no point fiddling around the library code as they well work on every other machine, usually 95% of 01mailrc.txt.gz.tmp1252 download issues are due to network/OS/firewall issues, but if you want to expand your Perl knowledge of LWP you can. In fact, you should also be looking at CPAN::FTP::netrc. Best of luck...
Related
psqlodbc driver not found on heroku despite being in my app directory
I am trying to get RODBC to work on heroku. I have a rails app that calls an R script from RinRuby, which then queries the production database in order to do some analysis. It all works fine on my local Mac, so I thought the best approach was to use the binary compiled on my Max (psqlodbcw.so) into my repo, and reference it in production as well. Unfortunately, when I try to make the connection in production using this connection string: > library(RODBC) > dbhandle <- odbcDriverConnect('driver=./psqlodbcw.so;database=nw_server_production;trusted_connection=true;uid=nw_server') Warning messages: 1: In odbcDriverConnect("driver=./psqlodbcw.so;database=<db_name>;trusted_connection=true;uid=<user>") : [RODBC] ERROR: state 01000, code 0, message [unixODBC][Driver Manager]Can't open lib './psqlodbcw.so' : file not found 2: In odbcDriverConnect("driver=./psqlodbcw.so;database=<db_name>;trusted_connection=true;uid=<user>?") : ODBC connection failed I have seen this error in a similar post online here, but using SQL server instead of postgres. But the accepted answer on that post doesn't explain why the file isn't found, despite being in the app directory. I did follow the same approach and made my own custom buildpack (available here: https://github.com/NovaWulf/r-rodbc-buildpack). I replaced the .so file with the one I compiled on my mac, and simply deleted the .rll file and the code that copies it, since I don't have that file (and hopefully don't need it for psqlodbc?). When I run that buildpack it runs without error on heroku, but then when I reference the .so file copied from the buildpack, I get the same "file not found" error. Is this happening because the .so file was compiled on the wrong system architecture? I tried compiling psqlodbc on linux, but I do not get a psqlodbcq.so file when I do that (let alone an .rll file). The closest thing I get is a file called libodbcpsqlS.so, which is a setup file, not a driver file. Could someone please help me understand the best approach to this problem? Why is heroku not seeing the file that is not there? And what is the best solution? Is there a simple way to just download the correct driver file somewhere? Any help is much appreciated! Best, Paul
Multiple problems installing TeX Live 2021 on Windows 10
I have a Windows 10 laptop where no TeX distribution was ever installed before (double-checked for config directories, files, and environment variables). Wanting to install TeX Live 2021, I followed the full installation guide and also read the Windows-specific warnings. I've now tried several times, following different installation procedures - cleaning up everything (deleting base & user directories, environment variables, etc) before each time - but still don't manage to get a working installation. Before I report a bug at tex-live#tug, I wanted to ask for advice here, in case I'm doing something wrong. Here is what I did, step by step, and the problems I encountered in the process. 1. First I tried running the recommended online installer install-tl-windows.exe. It never got past the screen that tries to contact or load from a repository, even after a 30-min wait. Tried a dozen times, choosing different mirrors nearby and far away. No luck. 2. Then I downloaded and unpacked the install-tl.zip and run install-tl-windows.bat therein. This time the main installation window appeared. I left all default paths and environment variables (note that I do have write access to C:); in the selection scheme I unselected all languages except US & UK English, unselected XeTeX, LuaTeX, ConTeXt; also unselected TeXworks (I use Emacs), and clicked Install. After one to three hours (depending on the mirror I chose), the installation was complete. I tried compiling a minimal latex document (see below), and got an error similar to the one in this old question: I can't find the format file `pdflatex.fmt' Following the advice in the answers to that question and similar questions elsewhere online such as this, I tried running texhash and fmtutil-sys --all. The latter gave the error no appropriate script or programme found fmtutil. for which there are also many posts online. 2a. Not understanding what the problem could be, I re-tried with all possible combinations of the following three options: (a) choosing different mirrors; (b) leaving the full selection of packages (ie without unselecting some languages, LuaTeX etc); (c) redoing the procedure by choosing "Run as Administrator". No luck. 3. At this point I tried downloading the ISO file with the full installation. Mounted the image and run install-tl-windows.bat (as normal user, as recommended; I repeat that I do have write access to C:). Everything proceeded as in step 2. above. At the end of the installation I tried running pdflatex on the minimal latex document. New error this time: This is pdfTeX, Version 3.141592653-2.6-1.40.22 (TeX Live 2021/W32TeX) (preloaded format=pdflatex) restricted \write18 enabled. entering extended mode (./minimal-template.tex LaTeX2e <2020-10-01> patch level 4 L3 programming layer <2021-02-18> ! LaTeX Error: File `article.cls' not found. Very strange. A file search revealed that article.cls is in the TeXLive file system; but kpathsea did not see it indeed. 4. At this point I opened the TeX Live Shell from the Start Menu; selected a CTAN mirror; updated the TL Manager which was not up to date; updated all packages; run Regenerate filename database; and run Regenerate formats. With the latter I got this error: tex live shell: mtutil [INFO]: total formats: 59 fmtutil [INFO]: exiting with status 53 C:\texlive\2021\bin\win32\runscript.tlu:915: command failed with exit code 53: perl.exe c:\texlive\2021\texmf-dist\scripts\texlive\fmtutil.pl --sys --all Here is a snip from the full set of errors appearing in the "Other" tab (I'm replacing my user directory with asterisks for privacy; note that I do have write access to these directories): start load https://www.nic.funet.fi/pub/TeX/CTAN/systems/texlive/tlnet finish load https://www.nic.funet.fi/pub/TeX/CTAN/systems/texlive/tlnet start load http://contrib.texlive.info/current finish load http://contrib.texlive.info/current running mktexlsr ... done running mktexlsr. running mtxrun --generate ... done running mtxrun --generate. running updmap-sys ... done running updmap-sys. regenerating fmtutil.cnf in C:/texlive/2021/texmf-dist running fmtutil-sys --byengine luatex --no-error-if-no-format --no-error-if-no-engine=luajithbtex,luajittex,mfluajit --status-file=C:\Users\***\AppData\Local\Temp\rPSb0Dpak2\WW_dJvUHgX ... tlmgr.pl: fmtutil-sys --byengine luatex --no-error-if-no-format --no-error-if-no-engine=luajithbtex,luajittex,mfluajit --status-file=C:\Users\***\AppData\Local\Temp\rPSb0Dpak2\WW_dJvUHgX failed (status 255), output: Unknown option: status-file Try "fmtutil --help" for more information. C:\texlive\2021\bin\win32\runscript.tlu:915: command failed with exit code 255: perl.exe c:\texlive\2021\texmf-dist\scripts\texlive\fmtutil.pl --sys --byengine luatex --no-error-if-no-format --no-error-if-no-engine=luajithbtex,luajittex,mfluajit --status-file=C:\Users\***\AppData\Local\Temp\rPSb0Dpak2\WW_dJvUHgX running fmtutil-sys --byengine luajithbtex --no-error-if-no-format --no-error-if-no-engine=luajithbtex,luajittex,mfluajit --status-file=C:\Users\***\AppData\Local\Temp\rPSb0Dpak2\WW_dJvUHgX ... I tried to continue anyway with Regenerate fontmaps, and then tried again pdflatex on the minimal document. New error: This is pdfTeX, Version 3.141592653-2.6-1.40.23 (TeX Live 2021/W32TeX) (preloaded format=pdflatex) restricted \write18 enabled. ---! c:/texlive/2021/texmf-var/web2c/pdftex/pdflatex.fmt made by different executable version (Fatal format file error; I'm stymied) 5. I tried again steps 3. and 4., but with "Run as Administrator". Same errors. OK at this point I give up and really don't know what to do. Am I doing something wrong? For the moment I have to concur with this post: installation of TeX Live 2021 is an utter failure. Here is the minimal latex file I used for testing (copy & paste): \documentclass{article} \begin{document} test \section{Section} test \end{document}
Found, that the download is not working because the system path to "cmd.exe" is not found. Therefore: open a cmd window and add the system path prior to starting the .bat file (set PATH=%PATH%;C:\Windows\system32)
VLC : which file to update VLC Lua youtube script on Mac OS X?
I am trying to read a youtube video through VLC on my mac: /Applications/VLC.app/Contents/MacOS/VLC -v https://www.youtube.com/watch?v=afzmwAKUppU&app=desktop Which gives errors : VLC media player 3.0. 8 Vetinari (revision 3.0.8-0-gf350b6b5a7) [00007faf5b5e9140] lua generic warning: Error while running script /Applications/VLC.app/Contents/MacOS/share/lua/extensions/youtube.lua, function descriptor() not found [00007faf5b4589c0] macosx interface warning: Failed to enable media key support, likely app needs to be whitelisted in Security Settings. [00007faf5b784950] securetransport tls client warning: Ignoring ALPN request due to lack of support in the backend. Proxy behavior potentially undefined. [00007faf5b770200] lua stream warning: Couldn't extract video URL, falling back to alternate youtube API [00007faf5b6b5b60] securetransport tls client warning: Ignoring ALPN request due to lack of support in the backend. Proxy behavior potentially undefined. [00007faf5f97ce70] securetransport tls client warning: Ignoring ALPN request due to lack of support in the backend. Proxy behavior potentially undefined. 2020-10-15 13:45:28.281 VLC[65658:198319] Can't find app with identifier com.spotify.client [00007faf5b5d8580] lua stream error: Couldn't extract youtube video URL, please check for updates to this script [00007faf5b44b570] main playlist: playlist is empty The youtube.lua, I got it by downloading the file from internet : curl "http://git.videolan.org/?p=vlc.git;a=blob_plain;f=share/lua/playlist/youtube.lua;hb=HEAD" -o /Applications/VLC.app/Contents/MacOS/share/lua/extensions/youtube.lua Which works on my ubuntu, but not in my Mac: I am wondering if this is not the correct version for Mac OS. And so, which file should be put there ? If I look on the VLC Lua directory, I find : /Applications/VLC.app/Contents/MacOS/share/lua/extensions$ ls -l total 192 -rw-r--r--# 1 romain admin 72K Aug 14 2019 VLSub.luac -rw-r--r-- 1 root admin 22K Oct 15 13:35 youtube.lua the youtube.lua is the new script I added, but maybe it was another one to put there ?
Probably a bit late here now, but in any case: You need to take the youtube.lua from here: https://github.com/videolan/vlc/blob/master/share/lua/playlist/youtube.lua Then rename it to youtube.luac and place it in the directory (for MacOS) /Applications/VLC.app/Contents/MacOS/share/lua/playlist I would recommend renaming the old one and keeping it in case something goes awry. At least on my computer this worked and I can now open YouTube videos in VLC again.
Have you tried the 3.0.11.1 release? https://get.videolan.org/vlc/3.0.11.1/macosx/vlc-3.0.11.1.dmg You're correct that youtube.lua would be the correct file, though the issue might come from other parts of the code depending on it. FYI, since you are running a stable VLC build, the code you should look at is https://code.videolan.org/videolan/vlc-3.0/-/blob/master/share/lua/playlist/youtube.lua. https://code.videolan.org/videolan/vlc/-/blob/master/share/lua/playlist/youtube.lua is nightly v4 unstable builds (though for the lua script, they are identical). Additionally, look out for a new release with an updated script soon https://mailman.videolan.org/pipermail/vlc-devel/2020-October/139076.html
How do I change window managers with Yocto Project tools?
My Intent I have an image generated by BitBake on which I'm interested in changing the window manager to metacity or maybe something similar. My Process I've added require recipes-graphics/images/core-image-x11.bb into my core recipe, which provides a simple Matchbox terminal window but seemingly no other functionality. If I add matchbox-desktop and matchbox-session-sato, it adds a bit more usability but not what I'm looking for. I've included the default package from the metacity_2.34.13.bb recipe from the meta-gnome layer from the OpenEmbedded Metadata Index in the IMAGE_INSTALL variable of my core image. This installs several components including a metacity command in /usr/bin. If I run that command, I get the following message: GLib-GIO-Message: Using the 'memory' GSettings backend. Your settings will not be saved or shared with other applications (metacity:1124): GLib-GIO-ERROR **: Settings schema 'org.gnome.metacity' is not installed Trace/breakpoint trap I've navigated to /usr/share/glib-2.0/schemas and run glib-compile-schemas ., then run: startx metacity --replace again. Now, the output is: Window manager error: Unable to open X display I haven't found a clear solution to this error which applies to my specific situation. Update (2/29): I may have now found a solution to this error, using these commands: X& export DISPLAY=:0 metacity& At this point, I seem to be running something on one of my VTs. I can run demos like glxgears in that VT (glxgears is included in the mesa-demos recipe), but I don't know how to actually create a usable environment. My question(s) I'm not using much from meta-openembedded/meta-gnome (just metacity) or meta/recipes-gnome (adwaita-icon-theme, gnome-desktop3, gsettings-desktop-schemas and gtk+3), so am I missing some recipe which automates the addition of metacity? (if not Question 1) How can I solve the error Window manager error: Unable to open X display?
The x11-common recipe adds a X session script that will run /usr/bin/x-session-manager: that is responsible for starting your desktop environment. The way to implement a new session/DE in OE-Core is to use update-alternatives for "x-session-manager": see the matchbox-session recipe for the default implementation and mini-x-session recipe for an alternative. mini-x-session might be modifiable for your needs so you don't need to write a new one: A /etc/mini_x/session file like this might do the trick: # start any apps here, e.g. "my-desktop &" exec metacity Going from this (a running window manager) to "usable environment" might still be lots and lots of work, depending on your definition of usable.
What are the exit status codes for the G-WAN executable?
I'm trying to serve a big number of small files with G-WAN (version 4.3.14, started with sudo on 64-bit Ubuntu 14.04.3). I start hammering it with requests over a single connection using wget to provide base URL and a file with a list of the URL suffixes. At some point, which is different for different runs, the gwan executable silently exits. There's no trace in the gwan log or in the site error log (I did change '_log' to 'log' to enable logging). The exit status code is 139. What does it mean? When I stop it with Ctrl-C the exit code is 130. Is there a reference for the exit status codes? I cannot find any with Google.
First, Ubuntu 14.04.3 is very recent while G-WAN v4.3.14 is very old. Almost every new OS release introduces cincompatibilities that require patches and this is why we have to issue more recent releases for registered users. This explains the 'silent exits' that you are experiencing. Second, process exits codes can be found this way: ./gwan -h echo $? 0 Zero means no error, and any other value is an error (mixing system flags to be as informative as possible). That's why Ctrl+C returns 130: Control-C is fatal error signal 2, (130 = 128 + 2).