How do I recover from a failed import in Nim? - include

In Nim I can write the following code to import an external module:
import myFancyPantsModule
...
# And here I'd use the fancyPants proc
This works fine as long as I have the module, but for people who might download the code and not have the module installed compilation will fail with a not very user friendly message:
$ nim c fancyProgram.nim
fancyProgram.nim(1, 7) Error: cannot open 'myFancyPantsModule'
Is there any way I can wrap around the import so that I can catch it similar to an exception and execute an alternative branch of code similar to the when statement? I was hoping to find some importable-like macro or something which I could use like:
when importable(myFancyPantsModule):
# And here I'd use the fancyPants proc
else:
quit("Oh, sorry, go to https://github.com/nim-lang/nimble and install " &
" the myFancyPantsModule using the nimble package manager")
In fact, rather than a simple error message I would like to make some modules optional, so that compilation still proceeds ahead, maybe with reduced functionality. Is this possible?
SOLUTION EDIT: Based on the answer here is my version how to solve the issue, first you need a moduleChecker binary with the following source:
import os, osproc
let tmpFile = getTempDir() / "dynamicModuleChecker.nim"
proc checkModule(module: string) =
except:
echo "Cannot write ", tmpFile, " to check the availability of modules"
quit(1)
writeFile(tmpFile, "import " & module & "\n")
finally: removeFile(tmpFile)
except:
echo("Cannot run \"nimrod check\" to check the availability of modules")
quit(1)
if execCmdEx("nim check " & tmpFile).exitCode != 0:
echo("Cannot import module " & module & ".")
quit(1)
else:
echo "OK"
if ParamCount() < 1:
quit("Pass as first parameter the module to check")
else:
checkModule(ParamStr(1))
Then, having this command available the following macro can be used:
import macros
macro safeImport(module, message: string): stmt =
if "OK" == gorge("./moduleChecker " & module.strVal):
result = newNimNode(nnkStmtList).add(
newNimNode(nnkImportStmt).add(
newIdentNode(module.strVal)))
else:
error("\nModule " & module.strVal &
" not available.\n" & message.strVal)
safeImport("genieos",
"Please install \"http://gradha.github.io/genieos/\"")
It is too unfortunate that a separate process has to be spawned, not only for the external compilation but also another one to generate the temporary file to check, as there is no staticWrite in the current version to generate files at compile time.

As far as I know, there is no (easy) way to do this. What you can do is to use a separate configuration/check stage in your build. E.g.:
import macros, os, osproc
proc checkModule(module, howtomessage: string) =
except:
echo("Cannot write .conftest.nim to check the availability of modules")
quit(1)
writeFile(".conftest.nim", "import " & module & "\n")
except: nil
removeFile(".conftest.nim")
except:
echo("Cannot run \"nimrod check\" to check the availability of modules")
quit(1)
if execCmdEx("nimrod check .conftest.nim").exitCode != 0:
echo("Cannot import module " & module & ".")
echo(howtomessage)
quit(1)
checkModule "foobar", "Please install it using the Babel package manager"
Then run something like:
nimrod cc --run configure.nim && nimrod cc main.nim
That's assuming the code above is stored in a file called configure.nim and the nimrod executable is in your path (otherwise, you'll have to specify the nimrod path in configure.nim also).

Related

How to make a basic protobuf program work using Python as on Google's Developer's Website?

I am following the tutorial of protobuf using Python (there isn't one for JavaScript). It doesn't work... and I think it might be outdated as proto2 and as a Python 2 program. How to make it work?
So I started with creating a file address.proto:
syntax = "proto2";
package tutorial;
message Person {
optional string name = 1;
optional int32 id = 2;
optional string email = 3;
enum PhoneType {
MOBILE = 0;
HOME = 1;
WORK = 2;
}
message PhoneNumber {
optional string number = 1;
optional PhoneType type = 2 [default = HOME];
}
repeated PhoneNumber phones = 4;
}
message AddressBook {
repeated Person people = 1;
}
And then I installed protoc on the Mac
And then I created two folders In and Out, and moved address.proto into In, and run:
protoc -I=In --python_out=Out In/address.proto
and then there is a file created: Out/address_pb2.py, and I went to Out, and added the file run.py:
#! /usr/bin/python
import addressbook_pb2
import sys
# This function fills in a Person message based on user input.
def PromptForAddress(person):
person.id = int(raw_input("Enter person ID number: "))
person.name = raw_input("Enter name: ")
email = raw_input("Enter email address (blank for none): ")
if email != "":
person.email = email
while True:
number = raw_input("Enter a phone number (or leave blank to finish): ")
if number == "":
break
phone_number = person.phones.add()
phone_number.number = number
type = raw_input("Is this a mobile, home, or work phone? ")
if type == "mobile":
phone_number.type = addressbook_pb2.Person.PhoneType.MOBILE
elif type == "home":
phone_number.type = addressbook_pb2.Person.PhoneType.HOME
elif type == "work":
phone_number.type = addressbook_pb2.Person.PhoneType.WORK
else:
print "Unknown phone type; leaving as default value."
# Main procedure: Reads the entire address book from a file,
# adds one person based on user input, then writes it back out to the same
# file.
if len(sys.argv) != 2:
print "Usage:", sys.argv[0], "ADDRESS_BOOK_FILE"
sys.exit(-1)
address_book = addressbook_pb2.AddressBook()
# Read the existing address book.
try:
f = open(sys.argv[1], "rb")
address_book.ParseFromString(f.read())
f.close()
except IOError:
print sys.argv[1] + ": Could not open file. Creating a new one."
# Add an address.
PromptForAddress(address_book.people.add())
# Write the new address book back to disk.
f = open(sys.argv[1], "wb")
f.write(address_book.SerializeToString())
f.close()
and then I installed and ran:
pip3 install protobuf --user
pip3 install google --user
pip3 install google-cloud --user
python3 run.py addr.dat
and it looks like I have to convert the code in run.py from print 123 to print(123) because it is Python3, not Python2. And it gave:
Traceback (most recent call last):
File "run.py", line 40, in <module>
address_book = addressbook_pb2.AddressBook()
NameError: name 'addressbook_pb2' is not defined
I also copied the file addressbook_pb2.py to foo.py, and then use import foo instead, and it gave:
Traceback (most recent call last):
File "run.py", line 3, in <module>
import foo
File "/Users/peter/code/TryProtobuf_Unzipped/TryIt/Out/foo.py", line 34, in <module>
_descriptor.EnumValueDescriptor(
File "/Users/peter/Library/Python/3.8/lib/python/site-packages/google/protobuf/descriptor.py", line 732, in __new__
_message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors should not be created directly, but only retrieved from their parent.
How can it be made to work?
Your address.proto declares the tutorial package. I think you can access it via address_book = tutorial.addressbook_pb2.AddressBook()
As for why it broke after renaming the file to foo.py, I have no idea.

How to interact with a running python script

As a start ive got a basic script which reads local unix syslog (/var/log/messages)
i want to build a tool which opens a socket (19999) locally and allows admin commands to be sent / processed.
As something i can build on basically i want to have the script on start up do the follow :
- open port 19999 locally
- start reading syslog storing "line" as the last line it has processed.
- when admin command of "printline" is seen print last known variable for "line"
Ive got basics done i think (script is below) where i have it open the relevant ports and it prints the commands sent to it from another client tool however it never starts to read the syslog.
#!/usr/bin/python
import socket
import subprocess
import sys
import time
from threading import Thread
MAX_LENGTH = 4096
def handle(clientsocket):
while 1:
buf = clientsocket.recv(MAX_LENGTH)
if buf == '': return #client terminated connection
print buf
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
PORT = 19999
HOST = '127.0.0.1'
serversocket.bind((HOST, PORT))
serversocket.listen(10)
while 1:
#accept connections from outside
(clientsocket, address) = serversocket.accept()
ct = Thread(target=handle, args=(clientsocket,))
ct.start()
def follow(thefile):
thefile.seek(0,2)
while True:
line = thefile.readline()
if not line:
time.sleep(0.1)
continue
yield line
if __name__ == '__main__':
logfile = open("/capture/log/uifitz/messages","r")
loglines = follow(logfile)
for line in loglines:
print line,
Any help would be appreciated. Python 2.6 by the way.

subprocess sometimes sends returns empty

I have the following class that is used to run a third party command line tool which I have no control over.
I run this ina Qthread in a PyQt Gui.
I turn the gui into an EXE using Pyinstaller
Problems are more prevalent when it is an EXE
class CLI_Interface:
def process_f(self, command, bsize=4096):
self.kill_process(CLI_TOOL)
startupinfo = STARTUPINFO()
startupinfo.dwFlags |= STARTF_USESHOWWINDOW
startupinfo.wShowWindow = SW_HIDE
p = Popen(command, stdout=PIPE, stderr=PIPE,
startupinfo=startupinfo, bufsize=bsize, universal_newlines=True)
try:
out, err = p.communicate(timeout=120)
except TimeoutExpired:
p.kill()
out, err = p.communicate()
return out.split(), err.split()
def kill_process(self, proc):
# Check process is running, Kill it if it is,
# return False if not.
# uses its own popen for Stderr >> stdout
# If we use the self.process_f method, it will create an infinite loop
startupinfo = STARTUPINFO()
startupinfo.dwFlags |= STARTF_USESHOWWINDOW
startupinfo.wShowWindow = SW_HIDE
try:
kill_proc = Popen("TaskKill /IM {} /T /F".format(proc), stdout=PIPE, stderr=STDOUT,
startupinfo=startupinfo, universal_newlines=True).communicate()[0]
if 'ERROR' not in kill_proc.split():
return True # Process Killed
else:
self.kill_process(proc)
except Exception as e:
return False
def download_data(self, code):
""" download data from the device based on a 5 digit code """
command = '"{}" -l {},{} {}'.format(CLI_TOOL_PATH,
code[0], code[2], code[1])
try:
p = self.process_f(command)
proc, err = p[0], p[1]
try:
if err[-2] == '-p':
return False
return True
except IndexError:
if not proc:
return False # This means there is no data but the file is still saved!!
pass
return True
except Exception as e:
return False
def ....
def ....
def ....
Thread:
class GetDataThread(QThread):
taskFinished = pyqtSignal()
notConnected = pyqtSignal()
def __init__(self, f, parent=None):
super(GetDataThread, self).__init__(parent)
self.f = f
def run(self):
is_dongle_connected()
DD = cli.download_data(self.f)
if not DD:
self.notConnected.emit()
else:
self.taskFinished.emit()
I either get a done! or error - This is normal when running from the command line.
Sometimes I get an empty list returned and I put this back into a recursive loop after killing the program.
However, it does not seem to restart properly and the problem continues - it gets stuck in a loop of nothing!.
Meanwhile, the csv files the cli tool produces are created as normal yet I have no data from stdout / stderr
Looking at processes the conhost and the cli tool are destroyed no problem.
The gui will continue to fail (until I unplug and plug in the dongle and / or restart the program / computer.
When I open the CLI and run the same command, it works fine or throws an error (which I catch in the program no problem)
I have tried setting a buffer as some files generated can reach 2.4mb
I tried setting a higher timeout to allow for it to finish.
There does not seem to be a correlation with file size though and it can get stuck at any size.
The flow is like so:
Gui >> CLI >> Dongle >> Sensor
Running on Windows 10
How can I make the connection more solid or debug what processes might still be lingering around and stopping this?
Is it blocking?
Is it a pipe buffer overflow? - If so How do I determine the correct bufsize?
Is it something to do with PyQt and Python Subprocess or Pyinstaller?
Would it be better to use QProcess instead of Subprocess?
Thanks in advance!

Programming pattern or library (i.e. idiomatic way) to handle CLI arguments semantic errors?

I have a Haskell application which uses optparse-applicative library for CLI arguments parsing. My data type for CLI arguments contains FilePaths (both files and directories), Doubles and etc. optparse-applicative can handle parse errors but I want to ensure that some files and some directories exist (or don't exist), numbers are >= 0 and etc.
What can be done is an implementation of a bunch of helper functions like these ones:
exitIfM :: IO Bool -> Text -> IO ()
exitIfM predicateM errorMessage = whenM predicateM $ putTextLn errorMessage >> exitFailure
exitIfNotM :: IO Bool -> Text -> IO ()
exitIfNotM predicateM errorMessage = unlessM predicateM $ putTextLn errorMessage >> exitFailure
And then I use it like this:
body :: Options -> IO ()
body (Options path1 path2 path3 count) = do
exitIfNotM (doesFileExist path1) ("File " <> (toText ledgerPath) <> " does not exist")
exitIfNotM (doesDirectoryExist path2) ("Directory " <> (toText skKeysPath) <> " does not exist")
exitIfM (doesFileExist path3) ("File " <> (toText nodeExe) <> " already exist")
exitIf (count <= 0) ("--counter should be positive")
This looks too ad-hoc and ugly to me. Also, I need similar functionality for almost every application I write. Are there some idiomatic ways to deal with this sort of programming pattern when I want to do a bunch of checks before actually doing something with data type? The less boilerplate involved the better it is :)
Instead of validating the options record after it has been constructed, perhaps we could use applicative functor composition to combine argument parsing and validation:
import Control.Monad
import Data.Functor.Compose
import Control.Lens ((<&>)) -- flipped fmap
import Control.Applicative.Lift (runErrors,failure) -- form transformers
import qualified Options.Applicative as O
import System.Directory -- from directory
data Options = Options { path :: FilePath, count :: Int } deriving Show
main :: IO ()
main = do
let pathOption = Compose (Compose (O.argument O.str (O.metavar "FILE") <&> \file ->
do exists <- doesPathExist file
pure $ if exists
then pure file
else failure ["Could not find file."]))
countOption = Compose (Compose (O.argument O.auto (O.metavar "INT") <&> \i ->
do pure $ if i < 10
then pure i
else failure ["Incorrect number."]))
Compose (Compose parsy) = Options <$> pathOption <*> countOption
io <- O.execParser $ O.info parsy mempty
errs <- io
case runErrors errs of
Left msgs -> print msgs
Right r -> print r
The composed parser has type Compose (Compose Parser IO) (Errors [String]) Options. The IO layer is for performing file existence checks, while Errors is a validation-like Applicative from transformers that accumulates error messages. Running the parser produces an IO action that, when run, produces an Errors [String] Options value.
The code is a bit verbose but those argument parsers could be packed in a library and reused.
Some examples form the repl:
Λ :main "/tmp" 2
Options {path = "/tmp", count = 2}
Λ :main "/tmpx" 2
["Could not find file."]
Λ :main "/tmpx" 22
["Could not find file.","Incorrect number."]

prevent pydub from opening console window

Is there a way to prevent pydub from opening a console window when using ffmpeg (on Windows) ? Each time it launches ffmpeg to convert a mp3 file to wav, it briefly opens a console window that shuts down automatically when process is done but can be disturbing.
The solution I am using is to overload the from_file function. I just changed subprocess call, adding option startupinfo. Here are the few lines I added:
import platform
systeme = platform.system()
if systeme == 'Windows':
startupinfo = subprocess.STARTUPINFO()
startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
startupinfo.wShowWindow = subprocess.SW_HIDE
p = subprocess.Popen(conversion_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, startupinfo=startupinfo)
else:
p = subprocess.Popen(conversion_command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)`
You can modify the source code and recompile it at runtime.
# Created by BaiJiFeiLong#gmail.com at 2022/2/18 22:09
import importlib.util
import re
import sys
import types
import pydub
from IceSpringPathLib import Path
for moduleName in "pydub.utils", "pydub.audio_segment":
spec = importlib.util.find_spec(moduleName, None)
source = spec.loader.get_source(moduleName)
snippet = "__import__('subprocess').STARTUPINFO(dwFlags=__import__('subprocess').STARTF_USESHOWWINDOW)"
source, n = re.subn(r"(Popen)\((.+?)\)", rf"\1(\2, startupinfo=print('worked') or {snippet})", source, flags=re.DOTALL)
module = importlib.util.module_from_spec(spec)
exec(compile(source, module.__spec__.origin, "exec"), module.__dict__)
sys.modules[moduleName] = module
module = importlib.reload(sys.modules["pydub"])
for k, v in module.__dict__.items():
if isinstance(v, types.ModuleType):
setattr(module, k, importlib.import_module(v.__name__))
pydub.audio_segment.AudioSegment.from_file(Path("~/Music").expanduser().glob("**/*.mp3").__next__())
I do the following in any script which uses pydub:
Add import subprocess before import pydub
Then, just before I use pydub in the script I add:
if subprocess.mswindows: subprocess.STARTUPINFO.dwFlags |= subprocess.STARTF_USESHOWWINDOW
This tells any subprocess call from that script (whether or not through a pydub instruction) to not display a window by default, but to look to another flag called wShowWindow to decide whether to display. As that flag is 0 by default, the window isn't shown.
note: dwFlags is no longer a class attribute but an instance attribute of STARTUPINFO (on newer Python versions like 3.9) therefore this answer isn't working for new versions. A STARTUPINFO() object must be created.

Resources