Packaged python project and published to PyPI

Posted on | In
Symbols count in article: 27k | Reading time ≈ 25 mins.

Preface

In Python, we often install various third-party libraries by pip cmd from PyPI, so how can we build a standard python library from scratch? Then, Let’s find it out.

PyPI-akparse.png

Project packaging

We need to define a standard Python project structure, and then start packaging step by step according to the following steps. [1]

Project structure

Start by creating a code folder to publish, organizing the project structure and adding a few necessary files, such as Project Name, LICENSE, README.md, setup.py and so on. [2]

1
2
3
4
5
6
7
8
9
10
11
12
13
packaging
├── my_project
│   ├── module1
│   │   ├── __init__.py
│   │   └── hello.py
│   ├── module2
│   │   ├── __init__.py
│   │   └── bye.py
│   ├── __init__.py
│   └── utils.py
├── LICENSE
├── README.md
└── setup.py  # also setup.cfg

The example project is simple, all of them are empty files, and the other files are as follows:

1/ hello.py

1
2
def hello_world():
    print("hello world!")

2/ bye.py

1
2
def goodbye():
    print("goodbye!")

3/ utils.py

1
2
def test():
    print("A demo project for packaging.")

Add LICENSE

PyPI requires that all uploaded packages must contain a license, take advantage of [3] help select a license, and then copy the license content into a file LICENSE.

Note that some licenses require a year ([year]) and owner ([fullname] or [name of copyright owner])

FOR EXAMPLE, AN MIT LICENSE:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
MIT License

Copyright (c) [2022] [my_project]

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Add description

Write a detailed description of the project according to the markdown syntax, which can then be used as an entry in setup.py 「long_description」

1
2
# Example Package
This is a simple example package. You can use [Github-flavored Markdown](https://guides.github.com/features/mastering-markdown/) to write your content.

Configure setup metadata

There are two types of setup metadata types:

  • static setup metadata
  • dynamic setup metadata

The official recommendation is preferred for static metadata setup.cfg or setup.py.

Static setup metadata

The following is an official example setup.cfg.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[metadata]
name = example-pkg-YOUR-USERNAME-HERE
version = 0.0.1
author = Example Author
author_email = author@example.com
description = A small example package
long_description = file: README.md
long_description_content_type = text/markdown
url = https://github.com/pypa/sampleproject
project_urls =
    Bug Tracker = https://github.com/pypa/sampleproject/issues
classifiers =
    Programming Language :: Python :: 3
    License :: OSI Approved :: MIT License
    Operating System :: OS Independent

[options]
package_dir = src
packages = find:
python_requires = >=3.6

[options.packages.find]
where = src
field Description
name if you are publishing on pypi.org, the name must be unique and can consist only of english letters, ,_-
author/email Identifies the author
description A short introduction to the package
long_description A detailed description of the package, which can specify a description file
url The project home page
project_urls Additional links related to the project
classifiers additional metadata, such as license, compatibility. see https://pypi.org/classifiers/ for a full list

Dynamic setup metadata

The official example, as can be seen, is essentially the same and can also be written using a ready-made template setup.py as [4]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
import setuptools

with open("README.md", "r", encoding="utf-8") as fh:
    long_description = fh.read()

setuptools.setup(
    name="example-pkg-YOUR-USERNAME-HERE",
    version="0.0.1",
    author="Example Author",
    author_email="author@example.com",
    description="A small example package",
    long_description=long_description,
    long_description_content_type="text/markdown",
    url="https://github.com/pypa/sampleproject",
    project_urls={
        "Bug Tracker": "https://github.com/pypa/sampleproject/issues",
    },
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    package_dir={"": "src"},
    packages=setuptools.find_packages(where="src"),
    python_requires=">=3.6",
)

In fact, it can be understood as an ini file that contains the default options of the command with setup.cfg or setup.py

Prepare venv

We also have Ubuntu 20.04 Server (Python 3.8.10) in Win10 WSL-2, creates and uses virtual environments 「venv」

1
2
3
4
5
6
7
8
9
# install venv
sudo apt install python3-venv -y

# create venv
mkdir code
python -m venv /home/xxx/code

# activate
source /home/xxx/code/bin/activate

Build package

Use the following configuration file setup.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
import setuptools

with open("README.md", "r", encoding="utf-8") as fh:
    long_description = fh.read()

setuptools.setup(
    name="my_project",
    version="0.0.1",
    author="xxxx",
    author_email="xxxxx@xxx.com",
    description="A small example package",
    long_description=long_description,
    long_description_content_type="text/markdown",
    url="https://github.com/xxxx",
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    packages=setuptools.find_packages(),
    python_requires=">=3.6",
)

Install setuptools and wheel which support building packages from source code. [5]

1
2
pip install setuptools
pip install wheel

Go to the packaging folder, check setup.py, and if there is an error, a prompt message will be printed.

1
2
# running check
python setup.py check

Packaging to .tar.gz or wheel, automatic creation of directories, and corresponding file sdist. [6]

1
2
3
4
5
# build to .tar.gz
python setup.py sdist build

# or build to wheel
python setup.py sdist bdist_wheel

PyPI Published

So far, we have completed the packaging of the project. Next, we need to publish the packaging module to pypi or our own private server.

Official PyPI server

When we use official PyPI server, We need to register an account first in [7]

Then to install twine to upload packages.

1
pip install twine

Next to input account info and finish upload.

1
twine upload dist/*

Local PyPI server

Install and create a folder to place the published package pypiserver packages. [8]

1
2
pip install pypiserver
mkdir ~/packages

Upload to directory, mv operate directly on the same virtual machine my_project-0.0.1.tar.gz to ~/packages.

1
mv dist/my_project-0.0.1.tar.gz ~/packages

Run the server, the port specified as 8080, listens to all ip addresses by default 8080.

1
pypi-server -p 8080 ~/packages &

Test

Search for packages on the local server my_project or PyPI

1
2
3
4
5
# official server
pip search my_project

# local server
pip search --index http://localhost:8080 my_project

Install and use the package my_project.

1
2
3
4
5
# install from official server
pip install my_project

# install from local server
pip install --extra-index-url http://localhost:8080 my_project

Reference

[1] https://packaging.python.org/en/latest/tutorials/packaging-projects/
[2] https://pythonguidecn.readthedocs.io/zh/latest/writing/structure.html
[3] https://choosealicense.com/
[4] https://github.com/absentm/akparse/blob/main/setup.py
[5] https://packaging.python.org/en/latest/guides/distributing-packages-using-setuptools/
[6] https://zhuanlan.zhihu.com/p/161930778
[7] https://pypi.org/account/register/
[8] https://github.com/pypiserver/pypiserver

END

0%