Getting an old Python project working on a M1 Mac running macOS Monterey was somewhat of an battle.
M1 and old Docker images
I first fixed the Dockerfile to the point where it would run MySQL and Redis properly. The M1 architecture is still a bit of a hurdle with some Docker images, and especially the older ones. Setting the platform flag explicitly was the key secret to get it workin. Here’s a part from docker-compose.yml:
db:
image: mysql:8.0.26
platform: linux/amd64
Python 2 and pip packages
Trying to get all 150+ old dependencies running in Docker turned out to be such an impossible task that would’ve taken forever that I decided to install them locally instead. This is still my favorite way to run Python projects as it seems to be much, much performant, and doesn’t require working inside a container. For this project, however, it would have been nice to just run everything inside a clean little container. Oh well.
I’ve seen all kinds of nasty hacks and setups in my long career but I have to say, this time I wen’t probably further than ever before just to get the depencies installed and everything working together.
MySQL
Database connectors are notoriously hard to install if you happen to need to build the binaries. This time it was MySQL.
First errors for pip install mysqlclient where about mysql_config which means that MySQL client is either not installed or not working correctly. Some old answers were pointing out to brew install mysql-connector-c, which, it turns out, is an older alias for mysql-client.
After installing brew gives you this helpful message which you might easily miss if you’re installing lots of other things at the same time and/or brew updates itself at the same time:
==> Caveats
mysql-client is keg-only, which means it was not symlinked into /opt/homebrew,
because it conflicts with mysql (which contains client libraries).
If you need to have mysql-client first in your PATH, run:
echo 'export PATH="/opt/homebrew/opt/mysql-client/bin:$PATH"' >> ~/.zshrc
For compilers to find mysql-client you may need to set:
export LDFLAGS="-L/opt/homebrew/opt/mysql-client/lib"
export CPPFLAGS="-I/opt/homebrew/opt/mysql-client/include"
For pkg-config to find mysql-client you may need to set:
export PKG_CONFIG_PATH="/opt/homebrew/opt/mysql-client/lib/pkgconfig"
So, to get this working, you need to:
export PATH="/opt/homebrew/opt/mysql-client/bin:$PATH"
export LDFLAGS="-L/opt/homebrew/opt/mysql-client/lib"
export CPPFLAGS="-I/opt/homebrew/opt/mysql-client/include"
Now, pip install mysqlclient works. (Mine didn’t at first because I was using a pinned requirements.txt which had an old version of the package that didn’t install. Updating the package to the lates version fixed this. I had to manually update a handful of packages to get everything working together.)
Pillow
PIL / Pillow also require some exotic binaries which can be painful to build/install in some situations. I had errors with missing/not working zlib but it was already installed and working properly. Pointing compile flags to right paths fixed the issue:
export LDFLAGS="-L/opt/homebrew/opt/zlib/lib"
export CPPFLAGS="-I/opt/homebrew/opt/zlib/include"
export PKG_CONFIG_PATH="/opt/homebrew/opt/zlib/lib/pkgconfig"
I had a handful of these same kind of issues when I tried to install the whole requirements.txt. What I did was hamdle each error individually, so instead of rerunning pip-sync, I figured out the issue with the first failing package, then install it individually (using latest version of the package if it made sense) and after the install worked, updating the exact installed version to the package.txt. This took a bit of trial and error but worked in the end.
One More Thing
After I got all the requirements installed there was still one more hurdle to overcome. Celery wouldn’t start with Django because of this error: AttributeError: dlsym(RTLD_DEFAULT, AbsoluteToNanoseconds): symbol not found
Turns out this is related to changes in macOS system libraries since Big Sur 10.11. I fould this Stackoverflow page discussing the issue and ended up with a really crazy (and unmaintainable) hack in <pythonpath>/lib/python2.7/ctypes/util.py (!!):
if os.name == "posix" and sys.platform == "darwin":
from ctypes.macholib.dyld import dyld_find as _dyld_find
def find_library(name):
if name == 'CoreServices':
return '/System/Library/Frameworks/CoreServices.framework/CoreServices'
elif name == 'libSystem.dylib':
return '/usr/lib/libSystem.dylib'
possible = ['lib%s.dylib' % name,
'%s.dylib' % name,
'%s.framework/%s' % (name, name)]
for name in possible:
try:
return _dyld_find(name)
except ValueError:
continue
return None
After this hard-coded hack to this ctypes util function I finally got the project fully running.
Obviously this is not a very stable way to set up a project. But this project is already EOL and I was just pretty desperate to get it running. Would not do anything like this in production tho, lucky to not have to either!