accesing Jupyter running in AWS - getting certificate error accessing it - amazon-ec2

I am installing Jupyter notebook in AWS-ec2 as per this tutorial:
https://medium.com/#alexjsanchez/python-3-notebooks-on-aws-ec2-in-15-mostly-easy-steps-2ec5e662c6c6
I am stuck at the point where I try to access the notebook from my laptop and get the below error:
[E 06:33:24.239 NotebookApp] Exception in callback (, .null_wrapper at 0x7f5689fcdd90>)
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/tornado/ioloop.py",
line 888, in start
handler_func(fd_obj, events)
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/tornado/stack_context.py",
line 277, in null_wrapper
return fn(*args, **kwargs)
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/tornado/netutil.py",
line 276, in accept_handler
callback(connection, address)
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/tornado/tcpserver.py",
line 264, in _handle_connection
do_handshake_on_connect=False)
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/tornado/netutil.py",
line 517, in ssl_wrap_socket
context = ssl_options_to_context(ssl_options)
File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/tornado/netutil.py",
line 494, in ssl_options_to_context
context.load_cert_chain(ssl_options['certfile'], ssl_options.get('keyfile', None))
PermissionError: [Errno 13] Permission denied
My jupyter notebook config is below
> c = get_config()
>
> # Kernel config c.IPKernelApp.pylab = 'inline' # if you want plotting support always in your notebook
>
> # Notebook config c.NotebookApp.certfile = u'/home/ubuntu/certs/mycert.pem' #location of your certificate file
> c.NotebookApp.ip = '*' c.NotebookApp.open_browser = False #so that
> the ipython notebook does not opens up a browser by default
> c.NotebookApp.password = u'sha1:262....your hash here.........65f'
> #edit this with the SHA hash that you generated after typing in Step 9
> # This is the port we opened in Step 3. c.NotebookApp.port = 8888

In the Jupyter notebook config file that you have shared in the question above make a few changes by following the steps shown below.
To configure the jupyter config file thoroughly, follow these steps:
cd ~/.jupyter/
vi jupyter_notebook_config.py
Insert this at the beginning of the document:
c = get_config()
# Kernel config
c.IPKernelApp.pylab = 'inline' # if you want plotting support always in your notebook
# Notebook config
c.NotebookApp.certfile = u'/home/ubuntu/certs/mycert.pem' #location of your certificate file
c.NotebookApp.ip = '0.0.0.0'
c.NotebookApp.open_browser = False #so that the ipython notebook does not opens up a browser by default
c.NotebookApp.password = u'sha1:98ff0e580111:12798c72623a6eecd54b51c006b1050f0ac1a62d' #the encrypted password we generated above
# Set the port to 8888, the port we set up in the AWS EC2 set-up
c.NotebookApp.port = 8888
Once you enter these above lines, make sure you save the config file before you exit the vi editor!
And also, most importantly remember to replace sha1:98ff0e580111:12798c72623a6eecd54b51c006b1050f0ac1a62d with your password!
Note that since in the above config file we have given port as 8888, the same is added in the security group. (Custom TCP type,TCP protocol, port range as 8888 and source is custom)
Now you are good to go!
Type the following command:
screen
This command will allow you to create a separate screen for just your Jupyter process logs while you continue to do other work on the ec2 instance.
And now start the jupyter notebook by typing the command:
jupyter notebook
To visit the jupyter notebook from the browser in your local machine:
Your EC2 instance will have a long url, like this:
ec2-52-39-239-66.us-west-2.compute.amazonaws.com
Visit that URL in you browser locally. Make sure to have https at the beginning and port 8888 at the end as shown below.
https://ec2-52-39-239-66.us-west-2.compute.amazonaws.com:8888/
This should get it running.

Related

How do I setup the _SERVER_MODEL_PATH variable?

I'm trying to replicate the quickstart save and serve example.
I go to the example folder, run the python script and can see the model runs and artifacts when I type mlflow ui.
However, when I try the mlflow serve command with different model run Ids and ports I get a 404 in my browser, even though the command seems successful:
mlflow models serve -m runs:/e1dabe8fc6e84286af5bee28ca89cdde/model --port 1234
2022/07/11 07:40:01 INFO mlflow.models.cli: Selected backend for flavor 'python_function'
2022/07/11 07:40:02 INFO mlflow.utils.conda: Conda environment mlflow-ddf4db606beaa0e9bb42ff0ed98e8f4c4c7cb1f4 already exists.
2022/07/11 07:40:02 INFO mlflow.pyfunc.backend: === Running command 'conda activate mlflow-ddf4db606beaa0e9bb42ff0ed98e8f4c4c7cb1f4 & waitress-serve --host=127.0.0.1 --port=1234 --ident=mlflow mlflow.pyfunc.scoring_server.wsgi:app'
INFO:waitress:Serving on http://127.0.0.1:1234
I tried running directly from anaconda prompt, and I get the following error:
conda activate mlflow-ddf4db606beaa0e9bb42ff0ed98e8f4c4c7cb1f4 & waitress-serve --host=127.0.0.1 --port=1234 --ident=mlflow mlflow.pyfunc.scoring_server.wsgi:app
Traceback (most recent call last):
File "C:\Users\sergio ferro.conda\envs\mlflow-ddf4db606beaa0e9bb42ff0ed98e8f4c4c7cb1f4\lib\runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\sergio ferro.conda\envs\mlflow-ddf4db606beaa0e9bb42ff0ed98e8f4c4c7cb1f4\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Users\sergio ferro.conda\envs\mlflow-ddf4db606beaa0e9bb42ff0ed98e8f4c4c7cb1f4\Scripts\waitress-serve.exe_main.py", line 7, in
File "C:\Users\sergio ferro.conda\envs\mlflow-ddf4db606beaa0e9bb42ff0ed98e8f4c4c7cb1f4\lib\site-packages\waitress\runner.py", line 283, in run
app = resolve(module, obj_name)
File "C:\Users\sergio ferro.conda\envs\mlflow-ddf4db606beaa0e9bb42ff0ed98e8f4c4c7cb1f4\lib\site-packages\waitress\runner.py", line 218, in resolve
obj = import(module_name, fromlist=segments[:1])
File "C:\Users\sergio ferro.conda\envs\mlflow-ddf4db606beaa0e9bb42ff0ed98e8f4c4c7cb1f4\lib\site-packages\mlflow\pyfunc\scoring_server\wsgi.py", line 6, in
app = scoring_server.init(load_model(os.environ[scoring_server._SERVER_MODEL_PATH]))
File "C:\Users\sergio ferro.conda\envs\mlflow-ddf4db606beaa0e9bb42ff0ed98e8f4c4c7cb1f4\lib\os.py", line 679, in getitem
raise KeyError(key) from None
KeyError: 'pyfunc_model_path'
I have tried deleting and creating a new anaconda environment, ran from git bash, anaconda prompt, added anaconda3 environment variables. I know it has something to do with the _SERVER_MODEL_PATH variable but I wouldn't know how to set it up or which path add to my environment variables so it can read this variable from there.

After upgrading Ansible to 2.9.9 cannot connect to Windows Hosts, hostname has disappeared from Ansible logs

I recently upgraded Ansible from 2.5.3 to 2.9.9. OS is RHEL 7.5, Python 2.7.
Windows module setup.ps1 cannot fetch the host name of target windows host and hence the playbook unable to connect windows VM, after upgrading to Ansible 2.9.9.
before ESTABLISH WINRM connection log statement you can see that
<> is empty. Hence ansible windows connection has stopped working.
Using module file /usr/lib/python2.7/site-packages/ansible/modules/windows/setup.ps1
Pipelining is enabled.
**<>** ESTABLISH WINRM CONNECTION FOR USER: 7771234s_tskSV on PORT 5985 TO
creating Kerberos CC at /tmp/tmpn5jvwl
calling kinit with subprocess for principal 7771234s_tskSV
Using module file /usr/lib/python2.7/site-packages/ansible/modules/windows/setup.ps1
Pipelining is enabled.
<> ESTABLISH WINRM CONNECTION FOR USER: 7771234s_tskSV on PORT 5985 TO
creating Kerberos CC at /tmp/tmp0hJgs1
calling kinit with subprocess for principal 7771234s_tskSV
kinit succeeded for principal 7771234s_tskSV
**<> WINRM CONNECT: transport=kerberos endpoint=http://:5985/wsman
<> WINRM CONNECTION ERROR: Invalid URL u'http://:5985/wsman': No host supplied what to do**
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/winrm.py", line 415, in _winrm_connect
self.shell_id = protocol.open_shell(codepage=65001) # UTF-8
File "/usr/lib/python2.7/site-packages/winrm/protocol.py", line 157, in open_shell
res = self.send_message(xmltodict.unparse(req))
File "/usr/lib/python2.7/site-packages/winrm/protocol.py", line 234, in send_message
resp = self.transport.send_message(message)
File "/usr/lib/python2.7/site-packages/winrm/transport.py", line 243, in send_message
self.build_session()
File "/usr/lib/python2.7/site-packages/winrm/transport.py", line 232, in build_session
self.setup_encryption()
File "/usr/lib/python2.7/site-packages/winrm/transport.py", line 237, in setup_encryption
prepared_request = self.session.prepare_request(request)
File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 437, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "/usr/lib/python2.7/site-packages/requests/models.py", line 305, in prepare
self.prepare_url(url, params)
File "/usr/lib/python2.7/site-packages/requests/models.py", line 382, in prepare_url
raise InvalidURL("Invalid URL %r: No host supplied what to do" % url)
InvalidURL: Invalid URL u'http://:5985/wsman': No host supplied what to do
Can you please suggest if the setup.ps1 is not up to date?

Hyperledger Indy: unable to init node, init_indy_node failed with error

I am following the instruction at https://github.com/hyperledger/indy-node/blob/master/docs/source/start-nodes.md
My OS is Ubuntu 16.04.6 LTS.
I have managed to install indy-node by following
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys CE7709D068DB5E88
sudo bash -c 'echo "deb https://repo.sovrin.org/deb xenial stable" >> /etc/apt/sources.list'
sudo apt-get update
sudo apt-get install indy-node
Now at step2, it says:
set Network name in config file
the location of the config depends on how a Node was installed. It's usually inside /etc/indy for Ubuntu.
the following needs to be added: NETWORK_NAME={network_name} where {network_name} matches the one in genesis transaction files above
I am confused as the document never mentioned any "genesis transaction files" above. So I decide to set NETWORK_NAME = sandbox in my case.
Now I start to generate keys with command init_indy_node Alpha 0.0.0.0 9701 0.0.0.0 9702 --seed 111111111111111111111111111Alpha, and I get the error:
Traceback (most recent call last):
File "/usr/local/bin/init_indy_keys", line 6, in <module>
from plenum.common.keygen_utils import initNodeKeysForBothStacks
File "/usr/local/lib/python3.5/dist-packages/plenum/__init__.py", line 87, in <module>
setup_plugins()
File "/usr/local/lib/python3.5/dist-packages/plenum/__init__.py", line 50, in setup_plugins
config = getConfigOnce()
File "/usr/local/lib/python3.5/dist-packages/plenum/common/config_util.py", line 106, in getConfigOnce
return _getConfig(general_config_dir)
File "/usr/local/lib/python3.5/dist-packages/plenum/common/config_util.py", line 87, in _getConfig
config.GENERAL_CONFIG_FILE))
File "/usr/local/lib/python3.5/dist-packages/plenum/common/config_util.py", line 32, in extend_with_external_config
config = getInstalledConfig(*extender)
File "/usr/local/lib/python3.5/dist-packages/plenum/common/config_util.py", line 26, in getInstalledConfig
spec.loader.exec_module(config)
File "/etc/indy/indy_config.py", line 2, in <module>
NETWORK_NAME = sandbox
NameError: name 'sandbox' is not defined
Can anyone help to create node keys and init them.
I have referred to Hyperledger Indy: Create genesis transaction file and looks like my problem is prior to this.
I have also tried von-network and I am able to start the network in docker image.
The only problem is I am not able to generate my own keys.
Thanks
OK.. Looks like I have found the reason: the indy_config file is actually a .py file, not a "pure" config file like json or yml, I need to add ' for the NETWORK_NAME

Failing to start Jupyter Notebook : 'does not appear to be an IPv4 or IPv6 address'

I am not able to start the Jupyter Notebook (Installed using Anaconda 4.5.11) configured to run a remote VM (Redhat Enterprise Linux 6.x). The 'jupyter_notebook_config.py' has the following property configured:
c.NotebookApp.ip = '*'
The error returned is the following:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/arnab/conda_pkg/python-skl/lib/python3.7/site-packages/notebook/notebookapp.py", line 869, in _default_allow_remote
addr = ipaddress.ip_address(self.ip)
File "/opt/arnab/conda_pkg/python-skl/lib/python3.7/ipaddress.py", line 54, in ip_address
address)
ValueError: '' does not appear to be an IPv4 or IPv6 address
Seems to be a bug (link) in Jupyter Notebook itself.
Changing the value of the property from '*' to '0.0.0.0' solved the problem:
c.NotebookApp.ip = '0.0.0.0'
Another workaround while leaving c.NotebookApp.ip='*' is to also set c.NotebookApp.allow_remote_access=True.

Scrapyd on Heroku can't recognize rewritten DATABASE_URL by heroku-buildpack-pgbouncer

Okay, here is my setup. I'm on Heroku running a scrapyd daemon using the scrapy-heroku package https://github.com/dmclain/scrapy-heroku.
I'm having issues running out of database connections. I decided to try pooling the database connections use pgbouncer. I'm using this buildpack: https://github.com/heroku/heroku-buildpack-pgbouncer
My procfile was:
web: scrapyd
And I changed it to:
web: bin/start-pgbouncer-stunnel scrapyd
The buildpack is supposed to rewrite your DATABASE_URL when it initializes so that whatever child process is run can just use the DATABASE_URL as normal but will now be connecting to pgbouncer instead of directly to the database.
Within scrapy I'm using adbapi to create a pool for each spider as such:
def from_settings(cls, settings):
dbargs = dict(
host=settings['MYSQL_HOST'],
database=settings['MYSQL_DBNAME'],
user=settings['MYSQL_USER'],
password=settings['MYSQL_PASSWD'],
#charset='utf8',
#use_unicode=True,
)
dbpool = adbapi.ConnectionPool('psycopg2', cp_max=2, cp_min=1, **dbargs)
return cls(dbpool)
And in my settings this is how I'm getting the DATABASE_URL info:
import urlparse
urlparse.uses_netloc.append("postgres")
url = urlparse.urlparse(os.environ["DATABASE_URL"])
MYSQL_HOST = url.hostname
MYSQL_DBNAME = url.path[1:]
MYSQL_USER = url.username
MYSQL_PASSWD = url.password
This was working fine before I added pgbouncer buildpack. Now I get connection errors:
Traceback (most recent call last):
File "/app/.heroku/python/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
result = f(*args, **kw)
File "/app/.heroku/python/lib/python2.7/site-packages/scrapy/xlib/pydispatch/robustapply.py", line 57, in robustApply
return receiver(*arguments, **named)
File "/tmp/etc/etc/etc/middlewares.py", line 92, in spider_opened
File "/app/.heroku/python/lib/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect
conn = _connect(dsn, connection_factory=connection_factory, async=async)
OperationalError: could not connect to server: Connection refused
Is the server running on host "127.0.0.1" and accepting
TCP/IP connections on port 5432?
Does anyone have an idea what the issue may be?

Resources