Hey all,
For context: I had this dev_appserver setup working late last year in 2021, and upon trying to set it up again, I'm getting odd errors.
BadArgumentError: app must not be empty.
I've solved quite a lot of errors up to this point, and this is where I'm at:
JDK 1.11+ installed (for Cloud Datastore Emulator)
Golang 1.15+ installed (for gops & dev_appserver.py - go build)
Gcloud Components:
I run my dev_appserver like this:
export DATASTORE_DATASET=dev8celbux
export DATASTORE_PROJECT_ID=dev8celbux
export DATASTORE_USE_PROJECT_ID_AS_APP_ID=true
dev_appserver.py --enable_console --admin_port=8001 --port=8081 --go_debugging=true --support_datastore_emulator=true --datastore_path=./datastore/local_db.bin setuptables-app.yaml
INFO 2022-09-09 13:26:30,233 devappserver2.py:317] Skipping SDK update check.
INFO 2022-09-09 13:26:30,250 datastore_emulator.py:156] Starting Cloud Datastore emulator at: http://localhost:58946
INFO 2022-09-09 13:26:32,381 datastore_emulator.py:162] Cloud Datastore emulator responded after 2.131000 seconds
INFO 2022-09-09 13:26:32,381 <string>:384] Starting API server at: http://localhost:59078
INFO 2022-09-09 13:26:32,384 <string>:374] Starting gRPC API server at: http://localhost:59079
INFO 2022-09-09 13:26:32,394 instance_factory.py:184] Building with dependencies from go.mod.
INFO 2022-09-09 13:26:32,397 dispatcher.py:280] Starting module "setuptables" running at: http://localhost:8081
INFO 2022-09-09 13:26:32,397 admin_server.py:70] Starting admin server at: http://localhost:8001
WARNING 2022-09-09 13:26:32,398 devappserver2.py:414] No default module found. Ignoring.
2022/09/09 13:26:35 STARTING
INFO 2022-09-09 13:26:37,220 instance.py:294] Instance PID: 9656
This error appears when I try & view the contents within the local datastore at localhost:8001/datastore.
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\lib\webapp2\webapp2\__init__.py", line 1526, in __call__
rv = self.handle_exception(request, response, e)
File "C:\Users\user\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\lib\webapp2\webapp2\__init__.py", line 1520, in __call__
rv = self.router.dispatch(request, response)
File "C:\Users\user\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\lib\webapp2\webapp2\__init__.py", line 1270, in default_dispatcher
return route.handler_adapter(request, response)
File "C:\Users\user\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\lib\webapp2\webapp2\__init__.py", line 1094, in __call__
return handler.dispatch()
File "C:\Users\user\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\tools\devappserver2\admin\admin_request_handler.py", line 88, in dispatch
super(AdminRequestHandler, self).dispatch()
File "C:\Users\user\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\lib\webapp2\webapp2\__init__.py", line 588, in dispatch
return self.handle_exception(e, self.app.debug)
File "C:\Users\user\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\lib\webapp2\webapp2\__init__.py", line 586, in dispatch
return method(*args, **kwargs)
File "C:\Users\user\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\tools\devappserver2\admin\datastore_viewer.py", line 661, in get
kinds = self._get_kinds(namespace)
File "C:\Users\user\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\tools\devappserver2\admin\datastore_viewer.py", line 597, in _get_kinds
return sorted([x.kind_name for x in q.run()])
File "C:\Users\user\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\ext\db\__init__.py", line 2077, in run
raw_query = self._get_query()
File "C:\Users\user\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\ext\db\__init__.py", line 2482, in _get_query
_app=self._app)
File "C:\Users\user\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\api\datastore.py", line 1371, in __init__
self.__app = datastore_types.ResolveAppId(_app)
File "C:\Users\user\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\api\datastore_types.py", line 238, in ResolveAppId
ValidateString(app, 'app', datastore_errors.BadArgumentError)
File "C:\Users\user\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google\appengine\api\datastore_types.py", line 186, in ValidateString
raise exception('%s must not be empty.' % name)
BadArgumentError: app must not be empty.
I figured out that it is somewhat related to the APPLICATION_ID environment variable being missing. Upon setting it, I am able to view the Database page. HOWEVER. When getting no errors writing my data to the emulator (line by line debugged to confirm & local_db.bin is created), upon looking at the data, nothing is there. I successfully write 15 entities from the code's point of view. However none appear on the admin page. I think it's due to the manual set of the APPLICATION_ID as I did not do this before. Perhaps should be automatic somehow. Was thinking that this environment variable could maybe do that: export DATASTORE_USE_PROJECT_ID_AS_APP_ID=true but doesn't seem to change anything.
Before calling creation of entities:
After calling creation of entities:
I write the data like this, no doubt this works correctly.
ctx, err := appengine.Namespace(appengine.BackgroundContext(), "celbux101")
...
userKeyOut, err := datastore.Put(ctx, userKey, &F1_4{...})
Also, looked in both the default & the designated namespace (celbux101):
Super stumped. :( Help appreciated!
I really think it may somehow be related to APPLICATION_ID
Yes!
... I managed to come to a solution! As suspected, the data was getting written correctly, as confirmed by the line-by-line debug & the creation of the local_db.bin. The issue is that the dev_appserver's UI is not able to show the database entities due to the incorrect or missing APPLICATION_ID, as deducted.
I figured out that the dev_appserver's UI uses both the APPLICATION_ID & namespace to determine where to look for your entities. Also, the dev_appserver has it's own default APPLICATION_ID.
Solution
The fix is to export this environment variable BEFORE running your dev_appserver.py.
export APPLICATION_ID=dev~None
This magic export allows everything work as expected. You can view the APPLICATION_ID that the UI is trying to use on the top-left of the interface.
EDIT: I just came back to running this on a new computer, and want to add this for future reference:
If you are getting IOError: emulator did not respond within 10s
INSTALL Python27 & add to your path! (alongside your bundled python)
From Google's documentation
export DATASTORE_DATASET=my-project-id
export DATASTORE_EMULATOR_HOST=::1:8432
export DATASTORE_EMULATOR_HOST_PATH=::1:8432/datastore
export DATASTORE_HOST=http://::1:8432
export DATASTORE_PROJECT_ID=my-project-id
This will display your project name (instead of dev~None)
Related
Interested in the possibilities offered by Dask, I started with the dask tutorial, and prepared my laptop by following the instructions as per the tutorial: cloning the repo and making a new conda env with:
conda env create -f binder/environment.yml
conda activate dask-tutorial
All goes fine and packages are installed. Then i kickoff jupyter lab and open the first workbook:
import dask.dataframe as dd
from dask.distributed import Client
client = Client()
The output is an almost-infinite printing of "distributed.nanny - WARNING - Restarting worker", some followed by more errors (see below). I spent the last couple of hours trying to figure out why I'm having this problem, but I don't find.
Tried bringing a LocalCluster(), it didn't help. Tried limiting the memory to 1GB, same problem.
Tried updating the packages, rebooting the laptop, still nothing.
Note, if that can be useful: I'm on Windows, I use conda, and this is a company laptop on which I don't have the admin rights.
Would anyone know why i have this issue?
Thanks!
Emek
P.S: Amongst the plethora of "distributed.nanny - WARNING - Restarting worker", I also get a few:
2023-02-10 16:04:36,283 - distributed.nanny - WARNING - Restarting worker
2023-02-10 16:04:36,408 - distributed.nanny - WARNING - Restarting worker
2023-02-10 16:04:36,425 - distributed.nanny - WARNING - Restarting worker
Traceback (most recent call last):
File "C:\Users\xxx\Anaconda3\envs\dask-tutorial\lib\site-packages\distributed\nanny.py", line 853, in _wait_until_connected
msg = self.init_result_q.get_nowait()
File "C:\Users\xxx\Anaconda3\envs\dask-tutorial\lib\multiprocessing\queues.py", line 135, in get_nowait
return self.get(False)
File "C:\Users\xxx\Anaconda3\envs\dask-tutorial\lib\multiprocessing\queues.py", line 116, in get
raise Empty
_queue.Empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\xxx\Anaconda3\envs\dask-tutorial\lib\site-packages\distributed\utils.py", line 741, in wrapper
return await func(*args, **kwargs)
File "C:\Users\xxx\Anaconda3\envs\dask-tutorial\lib\site-packages\distributed\nanny.py", line 545, in _on_worker_exit
await self.instantiate()
File "C:\Users\xxx\Anaconda3\envs\dask-tutorial\lib\site-packages\distributed\nanny.py", line 442, in instantiate
result = await self.process.start()
File "C:\Users\xxx\Anaconda3\envs\dask-tutorial\lib\site-packages\distributed\nanny.py", line 714, in start
msg = await self._wait_until_connected(uid)
File "C:\Users\xxx\Anaconda3\envs\dask-tutorial\lib\site-packages\distributed\nanny.py", line 855, in _wait_until_connected
await asyncio.sleep(self._init_msg_interval)
File "C:\Users\xxx\Anaconda3\envs\dask-tutorial\lib\asyncio\tasks.py", line 605, in sleep
return await future
asyncio.exceptions.CancelledError
2023-02-10 16:04:37,021 - distributed.nanny - WARNING - Restarting worker
2023-02-10 16:04:37,024 - distributed.nanny - WARNING - Restarting worker
2023-02-10 16:04:37,027 - distributed.nanny - WARNING - Restarting worker
Followed this answer suggesting to downgrade ipykernel and it solved the issue for now.
The following packages will be DOWNGRADED:
ipykernel 6.21.1-pyh025b116_0 --> 6.15.0-pyh025b116_0
I'm trying to replicate the quickstart save and serve example.
I go to the example folder, run the python script and can see the model runs and artifacts when I type mlflow ui.
However, when I try the mlflow serve command with different model run Ids and ports I get a 404 in my browser, even though the command seems successful:
mlflow models serve -m runs:/e1dabe8fc6e84286af5bee28ca89cdde/model --port 1234
2022/07/11 07:40:01 INFO mlflow.models.cli: Selected backend for flavor 'python_function'
2022/07/11 07:40:02 INFO mlflow.utils.conda: Conda environment mlflow-ddf4db606beaa0e9bb42ff0ed98e8f4c4c7cb1f4 already exists.
2022/07/11 07:40:02 INFO mlflow.pyfunc.backend: === Running command 'conda activate mlflow-ddf4db606beaa0e9bb42ff0ed98e8f4c4c7cb1f4 & waitress-serve --host=127.0.0.1 --port=1234 --ident=mlflow mlflow.pyfunc.scoring_server.wsgi:app'
INFO:waitress:Serving on http://127.0.0.1:1234
I tried running directly from anaconda prompt, and I get the following error:
conda activate mlflow-ddf4db606beaa0e9bb42ff0ed98e8f4c4c7cb1f4 & waitress-serve --host=127.0.0.1 --port=1234 --ident=mlflow mlflow.pyfunc.scoring_server.wsgi:app
Traceback (most recent call last):
File "C:\Users\sergio ferro.conda\envs\mlflow-ddf4db606beaa0e9bb42ff0ed98e8f4c4c7cb1f4\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\sergio ferro.conda\envs\mlflow-ddf4db606beaa0e9bb42ff0ed98e8f4c4c7cb1f4\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Users\sergio ferro.conda\envs\mlflow-ddf4db606beaa0e9bb42ff0ed98e8f4c4c7cb1f4\Scripts\waitress-serve.exe_main.py", line 7, in
File "C:\Users\sergio ferro.conda\envs\mlflow-ddf4db606beaa0e9bb42ff0ed98e8f4c4c7cb1f4\lib\site-packages\waitress\runner.py", line 283, in run
app = resolve(module, obj_name)
File "C:\Users\sergio ferro.conda\envs\mlflow-ddf4db606beaa0e9bb42ff0ed98e8f4c4c7cb1f4\lib\site-packages\waitress\runner.py", line 218, in resolve
obj = import(module_name, fromlist=segments[:1])
File "C:\Users\sergio ferro.conda\envs\mlflow-ddf4db606beaa0e9bb42ff0ed98e8f4c4c7cb1f4\lib\site-packages\mlflow\pyfunc\scoring_server\wsgi.py", line 6, in
app = scoring_server.init(load_model(os.environ[scoring_server._SERVER_MODEL_PATH]))
File "C:\Users\sergio ferro.conda\envs\mlflow-ddf4db606beaa0e9bb42ff0ed98e8f4c4c7cb1f4\lib\os.py", line 679, in getitem
raise KeyError(key) from None
KeyError: 'pyfunc_model_path'
I have tried deleting and creating a new anaconda environment, ran from git bash, anaconda prompt, added anaconda3 environment variables. I know it has something to do with the _SERVER_MODEL_PATH variable but I wouldn't know how to set it up or which path add to my environment variables so it can read this variable from there.
I have a Flask app running on nginx + uWSGI.
On my local server (non-nginx), I get a nice stack trace + error reporting for exceptions.
Like this:
$ python run.py
Traceback (most recent call last):
File "run.py", line 1, in <module>
from myappname import app
File "/home/me/myappname/myappname/__init__.py", line 27, in <module>
file_handler.setLevel(logging.debug)
File "/usr/lib/python2.7/logging/__init__.py", line 710, in setLevel
self.level = _checkLevel(level)
File "/usr/lib/python2.7/logging/__init__.py", line 190, in _checkLevel
raise TypeError("Level not an integer or a valid string: %r" % level)
On nginx, there is next to no logging whatsoever (in /var/log/nginx/error.log).
This post suggests adding app.logger.exception('Failed') to my script, which didn't help.
How do I enable this sort of logging for debugging purposes?
Nginx will capture your app's console output, but you must make the app recover from exceptions. Else, you'll only get 500 or 400 errors from Nginx.
Try running the app off Nginx until it seems stable.
Use the logging module to capture app status information to your own log file. This strategy will be useful in the long run.
I'm starting out to learn about AMQP and RabbitMQ.
To get myself going I have used a CLI tool, rabbitmqadmin, to successfully publish data to a RabbitMQ development install I have created upon my Mac OS X box. So far so good, I can publish messages, and watch them dequeue...
However when I come to try the exact same functionality upon the Heroku / CloudAMQP instance the rabbitmqadmin client seems to fall over.
This is the call:
rabbitmqadmin --host lemur.cloudamqp.com --vhost app4444444_heroku.com --user app4444444_heroku.com --password <withheld> publish routing_key=test payload="hello"
...and this is the output:
Traceback (most recent call last):
File "/usr/local/bin/rabbitmqadmin", line 828, in <module>
main()
File "/usr/local/bin/rabbitmqadmin", line 325, in main
method()
File "/usr/local/bin/rabbitmqadmin", line 428, in invoke_get
result = self.post(uri, json.dumps(upload))
File "/usr/local/bin/rabbitmqadmin", line 354, in post
return self.http("POST", path, body)
File "/usr/local/bin/rabbitmqadmin", line 377, in http
resp = conn.getresponse()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1013, in getresponse
response.begin()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 402, in begin
version, status, reason = self._read_status()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 366, in _read_status
raise BadStatusLine(line)
httplib.BadStatusLine: ''
Any thoughts or ideas gratefully received!
Add --ssl to the command line. CloudAMQP's web ui is https only.
I am behind unversity proxy which also requires authentication. I am a beginner in learning app-development and I have tried a lot many forum posts on stack overflow and at developers.google for app deployment using GAE in Win7 with a connection requiring proxy with Authentication but in vain. I have used "set http_proxy....." command in cmd window but it never asked me for authentication.
Everytime, I get an error no 10060 Connection timed out
Here is what I get in ' Deploying to Google(Myapp) Window ' --
2012-05-18 11:59:26 Running command: "['C:\\Python27\\python.exe', '-u', 'C:\\Program Files (x86)\\Google\\google_appengine\\appcfg.py', '--no_cookies', u'--email=aaaagrawal#gmail.com', '--passin', 'update', u'C:\\webapp\\arbitx']"
Application: arbitx; version: 1
Host: appengine.google.com
Starting update of app: arbitx, version: 1
Getting current resource limits.
2012-05-18 12:03:36,849 ERROR appcfg.py:2156 An error occurred processing file '': <urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>. Aborting.
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\appcfg.py", line 125, in <module>
run_file(__file__, globals())
File "C:\Program Files (x86)\Google\google_appengine\appcfg.py", line 121, in run_file
execfile(script_path, globals_)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", line 4062, in <module>
main(sys.argv)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", line 4053, in main
result = AppCfgApp(argv).Run()
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", line 2543, in Run
self.action(self)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", line 3810, in __call__
return method()
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", line 3006, in Update
self.UpdateVersion(rpcserver, self.basepath, appyaml)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", line 2995, in UpdateVersion
self.options.max_size)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", line 2122, in DoUpload
resource_limits = GetResourceLimits(self.rpcserver, self.config)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", line 355, in GetResourceLimits
resource_limits.update(GetRemoteResourceLimits(rpcserver, config))
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appcfg.py", line 326, in GetRemoteResourceLimits
version=config.version)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\tools\appengine_rpc.py", line 366, in Send
f = self.opener.open(req)
File "C:\Python27\lib\urllib2.py", line 400, in open
response = self._open(req, data)
File "C:\Python27\lib\urllib2.py", line 418, in _open
'_open', req)
File "C:\Python27\lib\urllib2.py", line 378, in _call_chain
result = func(*args)
File "C:\Python27\lib\urllib2.py", line 1215, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "C:\Program Files (x86)\Google\google_appengine\lib\fancy_urllib\fancy_urllib\__init__.py", line 367, in do_open
raise url_error
urllib2.URLError: <urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>
2012-05-18 12:03:36 (Process exited with code 1)
You can close this window now.
A step by step instruction for dealing with proxy(with authentication) and deployment in Win7 will be much appreciated.Thanks.
I'm not marking this as a duplicate since you are trying to use the Google App Engine API via the console window, rather than the GAE Launcher. However, please try to follow the advice given in this question I posted about a year ago.
In short, you need to setup an environment variable on your machine named HTTP_PROXY and it's value needs to be host_or_ip:port.
I work with a company that has a proxy server that I, too, must authenticate through. So long as I am authenticated on my machine, this method has worked without the need for authenticating when posting my request.
If this still doesn't work, you could try modify the the value of HTTP_PROXY to username:password#host_or_ip:port. I haven't tried this (and can't at the moment) but that might work if your proxy server accepts such requests.