Install and Test EZID

Python Dependency Management for EZID

Thu May 30 05:18:33 PM PDT 2024

Our previous installation method using setup.py install is no longer supported for deployments on python >3.11.x. We now use pip install to deploy EZID and poetry for python package dependency resolution. Both tools rely on file pyproject.toml for dependency management.

At this point in time we do not use poetry install to deploy EZID, because this forces us to run the application and all django management commands within a poetry generated virtual environment (e.g. poetry run command). The trade-off is we do not take advantage of poetry's excellent dependency version locking mechanism.

The pyproject.toml file

The pyproject.toml configuration file is used by a veriety of python packaging and installation tools. Here is a quick tour of the our EZID pyproject.toml:

We declare Poetry as our build tool

[build-system]
requires = ["poetry-core>=1.0.0,<2.0.0"]
build-backend = "poetry.core.masonry.api"

We use Poetry only for dependency management but not for packaging:

[tool.poetry]
# use Poetry only for dependency management but not for packaging
package-mode = false

We enumerate python dependencies along with minumum semantic version:

[tool.poetry.dependencies]
python = "^3.11"
django = "^4.2.11"
aiomysql = "^0.2.0"
mysqlclient = "^2.2.4"
boto3 = "^1.34.116"
[cut]

We enumerate development tools in a dev group section:

[tool.poetry.group.dev.dependencies]
pytest-django = "^4.8.0"
pytest-mock = "^3.14.0"

Installing EZID

The target host must support python 3.11 or higher. This can be a python virtual environment. After install we can use poetry check to validate the installed dependencies against the poetry.lock file. See Installing Poetry below.

1. Clone the project repostory

   agould@uc3-opspuppet-dev01:~/projects> git clone git@github.com:CDLUC3/ezid.git
   Cloning into 'ezid'...
   Receiving objects: 100% (32942/32942), 73.36 MiB | 16.44 MiB/s, done.
   Resolving deltas: 100% (21345/21345), done.
   Updating files: 100% (986/986), done.
   

2. Activate your virtual environment if using such (example uses pyenv)

   agould@uc3-opspuppet-dev01:~/projects> pyenv global
   3.11.9
   agould@uc3-opspuppet-dev01:~/projects> pyenv virtualenv ezid-3.11.9
   agould@uc3-opspuppet-dev01:~/projects> pyenv global ezid-3.11.9
   (ezid-3.11.9) agould@uc3-opspuppet-dev01:~/projects> 
   

3. Change directory into the root of the working tree of the cloned project repository

   (ezid-3.11.9) agould@uc3-opspuppet-dev01:~/projects> cd ezid
   (ezid-3.11.9) agould@uc3-opspuppet-dev01:~/projects/ezid> ls -l pyproject.toml 
   -rw-r--r--. 1 agould users 2726 May 31 13:26 pyproject.toml
   

4. Run pip install . This tells pip to source file pyproject.toml and install all dependencies listed in table [tool.poetry.dependencies]

   (ezid-3.11.9) agould@uc3-opspuppet-dev01:~/projects/ezid> pip install .
   Processing /home/agould/projects/ezid
     Installing build dependencies ... done
     Getting requirements to build wheel ... done
     Preparing metadata (pyproject.toml) ... done
   [cut]
   Successfully installed MarkupSafe-2.1.5 aiomysql-0.2.0 amqp-5.2.0 asgiref-3.8.1 beautifulsoup4-4.12.3 billiard-4.2.0 boto3-1.34.117 botocore-1.34.117 ...
   

5. Validate installed packages against poetry.lock file

   (ezid-3.11.9) agould@uc3-opspuppet-dev01:~/projects/ezid> poetry check --lock
   All set!
   

Keeping dependencies up-to-date with Poetry

Occassionally we must ensure the minimum semantic versions of python dependencies are current. We use a combination of Poetry commands to currate the pyproject.toml file.

Installing Poetry

We recommend to install Poetry globally on your development host, so it may be used for a veriety of projects regardless of virtual environment. Do this with pipx:

agould@uc3-opspuppet-dev01:~> pipx install poetry
agould@uc3-opspuppet-dev01:~> which poetry
~/.local/bin/poetry

The poetry update command retrieves latest versions of dependencies listed in pyproject.toml and updates the poetry.lock file. However, it does not automatically update minimum semantic versions in pyproject.toml. To keep pyproject.toml truly up-to-date, do the following:

1. Update pyproject.toml with command poetry add ${package}@latest. This has the added benefit of updating poetry.lock as well.

   Make a list of all dependency package names and run this command in a for loop:

   # for now we omit `django-matomo-api-tracking` because of how it is sourced.
   DEPS="
   django
   aiomysql
   mysqlclient
   boto3
   mysql
   myloginpath
   jinja2
   pygments
   fasteners
   feedparser
   filelock
   freezegun
   hjson
   lxml
   python-dateutil
   requests
   toml
   simplegist
   xmltodict
   pymysql
   configparser
   ecs-logging
   celery
   sqlalchemy
   "
   for package in $DEPS; do
     poetry add ${package}@latest
   done
   

   Note:

   • A script update_pyproject.sh was created to automate the above process.
   • The django entry is excluded from the list. Run poetry add django@version command to manually upgrade Django to a desired version.
   • The update_pyproject.sh script includes a final step poetry update which is used to update package dependencies in poetry.lock.

2. Review changes with git diff pyproject.toml. Make alterations as needed:

   (ezid-3.11.9) agould@uc3-opspuppet-dev01:~/projects/ezid> git diff pyproject.toml
   diff --git a/pyproject.toml b/pyproject.toml
   index 83bff875..fafbf330 100644
   --- a/pyproject.toml
   +++ b/pyproject.toml
   @@ -42,7 +42,7 @@ python = "^3.11"
    django = "^4.2.11"
    aiomysql = "^0.2.0"
    mysqlclient = "^2.2.4"
   -boto3 = "^1.34.107"
   +boto3 = "^1.34.117"
    mysql = "^0.0.3"
    myloginpath = "^0.0.4"
    jinja2 = "^3.1.4"
   @@ -54,16 +54,16 @@ freezegun = "^1.5.1"
    hjson = "^3.1.0"
    lxml = "^5.2.2"
    python-dateutil = "^2.9.0.post0"
   -requests = "^2.31.0"
   +requests = "^2.32.3"
    toml = "^0.10.2"
    simplegist = "^1.0.1"
    xmltodict = "^0.13.0"
   -pymysql = "^1.1.0"
   +pymysql = "^1.1.1"
    configparser = "^7.0.0"
    ecs-logging = "^2.1.0"
    celery = "^5.4.0"
    sqlalchemy = "^2.0.30"
   

3. Run the poetry update command if you modified the pyproject.toml file manually.

4. Run poetry check --lock to validate pyproject.toml and ensure that poetry.lock is consistent and in sync with pyproject.toml.

5. Review changes with git diff poetry.lock

6. Commit your updates and cut a new release candidate tag. Open a pull request so proper integration testing can be scheduled.

   In most cases you will see changes in the pyproject.toml and poetry.lock files.

Testing EZID

To perform EZID Unit test, you need to:

• prepare Python environment
• setup the DJANGO_SETTINGS_MODULE environment variable
• create the test database
• run pytest to perform the tests

Refer Unit & CI tests for detailed instructions on how to run EZID unit tests.

Prepare Python environment

1. Run pip install . to source file pyproject.toml and install all dependencies listed there.
2. Run pip install -r requirements-dev.txt to install required packages for testing.
3. Create a logs folder above the EZID project root directory

Directory setup in settings/tests.py

SITE_ROOT = PROJECT_ROOT = pathlib.Path(__file__).parent.parent.resolve()
...

# Dirs above PROJECT_ROOT
HOME_DIR = (PROJECT_ROOT / '..').resolve()  # /apps/ezid
MINTERS_PATH = HOME_DIR / 'var' / 'minters'  # /apps/ezid/var/minters
LOG_DIR = HOME_DIR / 'logs'  # /apps/ezid/logs

Setup the DJANGO_SETTINGS_MODULE environment variable

There are a few ways to setup the DJANGO_SETTINGS_MODULE variable:

1. Use the export command. This tells Django to find configurations from file “ezid/settings/tests.py”.

$ export DJANGO_SETTINGS_MODULE=settings.tests

1. Use the command line option --ds=SETTINGS:

$ pytest  --ds=settings.tests

Setup the test database

A script prepare_test_db.sh was created to automate the test database setup process. The script is in the home directory of the ezid project. The script does not require any parameters. Provide the root user password for your local mysql database when prompted.

./prepare_test_db.sh
Enter password:

Run Pytest

Test scripts and data files are organized in the ezid/tests directory. Test scripts are named in the test_scriptname.py format.

Run Pytest in the ezid project root directory:

pytest --ds=settings.tests tests/

To run a specific test script:

pytest --ds=settings.tests tests/test_api.py