Airflow commands fail due to typeerror in utils.py when deployed in heroku - heroku

Question

Im running airflow version 2.4.1 with python 3.10. I have no issues when running airflow locally in docker, but when I deploy it to heroku using their container registry, the airflow webserver command fails with this trackback:
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:55811
Timeout: 120
Logfiles: - -
Access Logformat:
=================================================================
[2022-11-10 00:19:39 +0000] [43] [INFO] Starting gunicorn 20.1.0
Traceback (most recent call last):
File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.10/site-packages/gunicorn/__main__.py", line 7, in <module>
run()
File "/usr/local/lib/python3.10/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 231, in run
super().run()
File "/usr/local/lib/python3.10/site-packages/gunicorn/app/base.py", line 72, in run
Arbiter(self).run()
File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 198, in run
self.start()
File "/usr/local/lib/python3.10/site-packages/gunicorn/arbiter.py", line 138, in start
self.cfg.on_starting(self)
File "/usr/local/lib/python3.10/site-packages/airflow/www/gunicorn_config.py", line 40, in on_starting
ProvidersManager().connection_form_widgets
File "/usr/local/lib/python3.10/site-packages/airflow/providers_manager.py", line 908, in connection_form_widgets
self.initialize_providers_hooks()
File "/usr/local/lib/python3.10/site-packages/airflow/providers_manager.py", line 300, in wrapped_function
func(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/airflow/providers_manager.py", line 386, in initialize_providers_hooks
self.initialize_providers_list()
File "/usr/local/lib/python3.10/site-packages/airflow/providers_manager.py", line 300, in wrapped_function
func(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/airflow/providers_manager.py", line 363, in initialize_providers_list
self._discover_all_providers_from_packages()
File "/usr/local/lib/python3.10/site-packages/airflow/providers_manager.py", line 429, in _discover_all_providers_from_packages
for entry_point, dist in entry_points_with_dist('apache_airflow_provider'):
File "/usr/local/lib/python3.10/site-packages/airflow/utils/entry_points.py", line 40, in entry_points_with_dist
key = canonicalize_name(dist.metadata["Name"])
File "/usr/local/lib/python3.10/site-packages/packaging/utils.py", line 34, in canonicalize_name
value = _canonicalize_regex.sub("-", name).lower()
TypeError: expected string or bytes-like object
[2022-11-10 00:21:38,935] {webserver_command.py:217} ERROR - No response from gunicorn master within 120 seconds
[2022-11-10 00:21:38,935] {webserver_command.py:218} ERROR - Shutting down webserver
Has anyone run into this error before? Let me know if more information is needed.

Answer 1

I have just solved the problem, in my case I had packages where there weren't NAME field in METADATA.
First you can check it counting number of None:
import importlib_metadata as metadata
import pandas as pd
s = pd.Series([dist.metadata["Name"] for dist in metadata.distributions()])
s.isnull().sum()
After that, you need find what packages are causing your problem and uninstall and install it.
I solve it using vscode debugger with justMycode: false.
I was debugging the next code:
for met in metadata.distributions():
print(met.name)
The point here is to add a if condition and a break point in the next path /PATH/lib/PYTHON_VERSION/site-package/importlib_metadata/__init__.py, for example in my case:
/home/antonio/anaconda3/Aspect-Based-On-Sentiment-Analysis/lib/python3.7/site-package/importlib_metadata/__init__.py
Set if condition and break point in print('now') (line 598):
#property
def name(self):
"""Return the 'Name' metadata for the distribution package."""
if not self.metadata.json:
print('now')
return self.metadata['Name']
You could see on debug window something like that,
The field _normalized_name will indicate you package name that you need to uninstall since the rest of the fields will be None when name is None
May be pip uninstall name_package doesn't work.
I recommend you to search the package in /PATH/lib/PYTHON_VERSION/site-package, check NAME is not in METADATA file or METADATA file doesn't exist and remove the entire package, finally, you can use pip install name-package and check NAME is in METADATA file
Then you check number of packages without NAME in METADATA running:
import importlib_metadata as metadata
import pandas as pd
s = pd.Series([dist.metadata["Name"] for dist in metadata.distributions()])
s.isnull().sum()
You will see it has decreased by one.