conda-multi-account-hipergator

Set up conda with multiple group storage locations on HiPerGator

$ Installer

git clone https://github.com/smith6jt-cop/Skills_Registry /tmp/Skills_Registry && cp -r /tmp/Skills_Registry/plugins/general/conda-multi-account-hipergator/skills/conda-multi-account-hipergator ~/.claude/skills/Skills_Registry

// tip: Run this command in your terminal to install the skill


name: conda-multi-account-hipergator description: "Set up conda with multiple group storage locations on HiPerGator" author: KINTSUGI Team date: 2025-12-11

Conda Multi-Account Setup on HiPerGator

Experiment Overview

ItemDetails
Date2025-12-11
GoalConfigure conda to use environments from multiple group storage locations
EnvironmentHiPerGator (UF), RHEL 9, multiple /blue/group/ directories
StatusSuccess

Context

On HiPerGator, users often belong to multiple research groups with separate storage allocations under /blue/. Each group may have its own conda installation and environments. The challenge is accessing environments from a different group's storage location.

The Problem

Default conda only knows about environments in:

  • ~/.conda/envs/ (home directory)
  • The base conda installation's envs/ directory

Environments created under /blue/othergroup/username/ are not visible.

Verified Workflow

Solution: Create a Sourcing Script

Create ~/.use_conda_<groupname>.sh for each group's conda:

#!/bin/bash
# ~/.use_conda_maigan.sh - Switch to maigan group's conda

# Initialize conda for this shell
source /blue/maigan/smith6jt/miniforge3/etc/profile.d/conda.sh

# Add environment search paths
export CONDA_ENVS_DIRS="/blue/maigan/smith6jt/miniforge3/envs:$HOME/.conda/envs"

# Optional: Set package cache to shared location
export CONDA_PKGS_DIRS="/blue/maigan/smith6jt/miniforge3/pkgs:$HOME/.conda/pkgs"

Usage Pattern

# At start of session, source the appropriate conda
source ~/.use_conda_maigan.sh

# Now conda sees all environments
conda env list
# Shows environments from /blue/maigan/smith6jt/miniforge3/envs/

# Activate environment from that location
conda activate KINTSUGI

For Multiple Groups

Create separate scripts for each group:

# ~/.use_conda_maigan.sh - for maigan group
source /blue/maigan/smith6jt/miniforge3/etc/profile.d/conda.sh
export CONDA_ENVS_DIRS="/blue/maigan/smith6jt/miniforge3/envs:$HOME/.conda/envs"

# ~/.use_conda_othergroup.sh - for another group
source /blue/othergroup/smith6jt/miniforge3/etc/profile.d/conda.sh
export CONDA_ENVS_DIRS="/blue/othergroup/smith6jt/miniforge3/envs:$HOME/.conda/envs"

Integration with Claude Code

Add the sourcing command to Claude Code permissions in .claude/settings.local.json:

{
  "permissions": {
    "allow": [
      "Bash(source ~/.use_conda_maigan.sh)",
      "Bash(conda activate:*)",
      "Bash(conda env list:*)",
      "Bash(pip install:*)"
    ]
  }
}

Failed Attempts (Critical)

AttemptWhy it FailedLesson Learned
Adding paths to .condarc envs_dirsOnly works after conda init, path issues with multiple condasUse environment variables instead
Symlinking envs to home directoryPermissions issues, clutters homeKeep envs in original location
Using full paths to activateVerbose, error-proneSourcing script is cleaner
Running conda init for multiple condasConflicts in .bashrc, breaks shellUse separate sourcing scripts
Using module load condaHiPerGator module may not have your custom envsUse your own miniforge installation

Final Parameters

Directory Structure

/blue/maigan/smith6jt/
├── miniforge3/           # Conda installation
│   ├── etc/profile.d/
│   │   └── conda.sh      # Source this to initialize
│   ├── envs/
│   │   └── KINTSUGI/     # Your environments here
│   └── pkgs/             # Package cache
└── KINTSUGI/             # Project directory

~/.use_conda_maigan.sh    # Sourcing script

The Sourcing Script

#!/bin/bash
# ~/.use_conda_maigan.sh

# Initialize conda (REQUIRED - sets up conda command)
source /blue/maigan/smith6jt/miniforge3/etc/profile.d/conda.sh

# Tell conda where to find environments
export CONDA_ENVS_DIRS="/blue/maigan/smith6jt/miniforge3/envs:$HOME/.conda/envs"

# Share package cache to save space
export CONDA_PKGS_DIRS="/blue/maigan/smith6jt/miniforge3/pkgs:$HOME/.conda/pkgs"

echo "Conda configured for maigan group"
conda env list

Key Insights

  • conda.sh sourcing is required before any conda commands work
  • CONDA_ENVS_DIRS is colon-separated list of paths to search for environments
  • The first path in CONDA_ENVS_DIRS is where new environments are created
  • Don't run conda init for multiple installations - it modifies .bashrc
  • Miniforge is preferred over Anaconda for HPC (faster, BSD licensed)
  • Package cache (CONDA_PKGS_DIRS) can be shared to save quota

Environment Creation

When creating new environments with this setup:

source ~/.use_conda_maigan.sh

# Create from environment file
conda env create -f envs/env-linux.yml

# Or create manually
conda create -n myenv python=3.10

# Environment will be created in /blue/maigan/smith6jt/miniforge3/envs/

References

Repository

smith6jt-cop
smith6jt-cop
Author
smith6jt-cop/Skills_Registry/plugins/general/conda-multi-account-hipergator/skills/conda-multi-account-hipergator
0
Stars
0
Forks
Updated2d ago
Added1w ago