Usually, we can export Conda environment simply use requirements.txt or environment.yml. But if the target server could not access to the Internet, it will be unable to restore the environment completely.
Pack your whole environment from source server
Install conda-pack
Make sure there is conda-pack, if not, install one:
# conda install
conda install -c conda-forge conda-pack
# or pip install
pip install conda-packPack and export the whole environment
# E.g., environment name is 'your_env', output environment file name is 'your_env.tar.gz'
conda pack -n your_env --output your_env.tar.gz
# What if some packages are installed via 'pip install -e', you might need to add '--ignore-editable-packages' as conda-pack might be unable to handle these packages correctly
conda pack -n your_env --ignore-editable-packages --output your_env.tar.gzIt takes some time to export the environment file and the size is a bit large.
Upload ‘your_env.tar.gz’ to target server
Use FileZilla or other tools to upload the file to target server.
Unpack the file and restore the environment
Create the directory of the target environment
Usually path of conda environment is like: /home/ubuntu/conda/envs.
tar -xzf /path/to/your_env.tar.gz -C /path/to/your/conda/envs
# You can use below command to check your conda env path:
conda env list
# Create the target env directory
Activate and verify the environment
# Activate environment
conda activate your_env
# Shoule be able to find all packages from source server
conda list[Update] Export specified packages along with their dependencies
From the above method, we can export the complete environment. It is useful when we initiate the environment for the first time. However, after the initial setup, we typically only need to update the newly added packages, rather than reinstalling all of them.
This selective update approach can save time and resources, especially when dealing with large environments with numerous packages.
Install pipdeptree
We use pipdeptree to extract packages and their dependencies:
pip install pipdeptreeExport dependency tree:
pipdeptree > dependency_tree.txtGenerate requirements.txt
Extract all packages and their dependencies, and save them to requirements.txt:
def parse_dependency_tree(file_path):
"""
Extract packages from dependency_tree.txt, and generate requirements.txt
"""
with open(file_path, 'r', encoding='utf-8') as file:
lines = file.readlines()
# will not store duplicated packages
dependencies = set()
for line in lines:
# remove head and tail spaces
line = line.strip()
# Extract packages and version numbers
# e.g. tensorboard==2.19.0
if '==' in line and 'installed' not in line:
package, version = line.split('==')
dependencies.add(f"{package.strip()}=={version.strip()}")
# e.g. numpy [required: >=1.12.0, installed: 1.23.5]
elif 'installed:' in line:
package_info = line.split('[')[0].strip().split(' ')[-1]
version = line.split('installed: ')[1].strip(']')
dependencies.add(f"{package_info}=={version}")
# save to requirements.txt
with open('requirements.txt', 'w', encoding='utf-8') as req_file:
for dep in dependencies:
req_file.write(f"{dep}\n")
print("requirements.txt done.")
parse_dependency_tree('dependency_tree.txt')Download all packages
Download all packages in a specified folder:
pip download -r requirements.txt -d /path/to/your/packagesRestore them in the target server
Put the folder to the target server, and install them from requirements.txt:
# --no-index --find-links: Ignore package index (only looking at --find-links URLs instead).
pip install --no-index --find-links /path/to/your/packages -r requirements.txt
