Create and use a private python library on github in your projects

Creating a private Python library on GitHub and using it in your projects is a great way to manage internal code, share reusable components across teams, and protect proprietary intellectual property. This process involves setting up a private repository, configuring your local environment for authentication, and then installing the library using a dependency manager like pip or poetry [1].

Step 1: Create a Private GitHub Repository 🤫

The first step is to create a new repository on GitHub and make sure it is set to private. This ensures that your code is not publicly accessible.

1. Go to the GitHub website and click the "+" icon in the top right corner, then select "New repository."
2. Give your repository a descriptive name (e.g., my-private-lib).
3. Select the Private option.
4. Optionally, you can initialize the repository with a README file, a .gitignore file for Python, and a license.

Once the repository is created, you can clone it to your local machine and start adding your Python module code.

git clone https://github.com/your-username/my-private-lib.git
cd my-private-lib

Step 2: Set Up Python Package Structure 📦

Your private library needs a standard Python package structure to be installable. At a minimum, you should have a pyproject.toml file (for modern projects using pep 518) or a setup.py file, along with your actual code.

Using pyproject.toml (Recommended)

A pyproject.toml file is the modern standard for managing Python projects.

pyproject.toml

[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

[project]
name = "my_private_lib"
version = "0.0.1"
dependencies = []

my_private_lib/__init__.py

def my_secret_function():
    return "This is a private function."

def another_helper():
    return 42

After adding your code, commit and push the changes to your private GitHub repository.

git add .
git commit -m "Initial commit of private library"
git push origin main

Step 3: Authenticate with GitHub 🔐

To install a private library, your package manager needs to authenticate with GitHub. You should not use your GitHub password. The recommended method is to use a Personal Access Token (PAT) [2].

1. Go to your GitHub account settings.
2. Navigate to "Developer settings" > "Personal access tokens" > "Tokens (classic)."
3. Click "Generate new token (classic)."
4. Give the token a descriptive name and set an expiration date.
5. Under "Select scopes," check the box for repo. This grants the token access to your private repositories.
6. Click "Generate token" and copy the token immediately. You won't be able to see it again.

Now, you have a token that can be used for authentication.

Step 4: Use Your Private Library as a Dependency 🛠️

Once your library is on GitHub and you have a PAT, you can use it as a dependency in your Python application. While pip can do this, a dependency manager like Poetry or pipenv makes this process cleaner.

Option A: Using pip

You can install the library directly from the GitHub URL. However, you need to include your PAT in the URL for authentication.

pip install git+https://<YOUR_GITHUB_TOKEN>@github.com/your-username/my-private-lib.git

This is a simple method, but it's not ideal as it exposes your token in your shell history and command line.

Option B: Using Poetry (Recommended)

Poetry provides a cleaner way to handle private dependencies and authentication. You can add the repository URL to your pyproject.toml file, and then use your PAT for authentication.

First, add the private repository to your pyproject.toml file:

my-app/pyproject.toml

[tool.poetry.dependencies]
python = "^3.10"
my-private-lib = { git = "https://github.com/your-username/my-private-lib.git", rev = "main" }

Now, you need to configure poetry to use your PAT. The best way is to use a .netrc file or to set the token as a configuration variable [3].

Using Poetry's configuration:

poetry config http-basic.github.com your-username <YOUR_GITHUB_TOKEN>

Now, when you run poetry install, it will automatically use your PAT to authenticate and download the private library.

Step 5: Import and Use the Library 🚀

Once installed, your private library can be imported and used like any other Python package.

my_app/main.py

from my_private_lib import my_secret_function

if __name__ == "__main__":
    result = my_secret_function()
    print(result) # This will print: "This is a private function."

This setup provides a robust and secure way to manage your private Python code. The combination of a private GitHub repository, a PAT for secure authentication, and a dependency manager ensures your proprietary code is handled correctly.

---

Sources

1. How to install a Python package from a private GitHub repository
2. GitHub Docs - Creating a personal access token
3. Poetry Docs - Git Dependencies
4. Best practices for securing your GitHub account