Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
**__pycache__**

# Python packaging / build artifacts
*.egg-info/
build/
dist/
*.egg

# Virtual environments
.venv/
venv/
env/

# Test / linter caches
.pytest_cache/
.mypy_cache/
.ruff_cache/
.coverage
htmlcov/

# Jupyter
.ipynb_checkpoints/

# Editors / OS
.DS_Store
.vscode/
.idea/
40 changes: 39 additions & 1 deletion data_strategy.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,8 @@ __Special note on `IdentityFile=~/.ssh/alpine`:__ this `sshfs` option tells SSH
- __MacOS:__ install [macFUSE](https://github.com/macfuse/macfuse/wiki/File-Systems-%E2%80%90-SSHFS) and `sshfs`:

```shell
brew install --cask macfuse
brew tap macos-fuse-t/homebrew-cask
brew install fuse-t fuse-t-sshfs
```

Then mount with `sshfs`, for example:
Expand Down Expand Up @@ -217,3 +218,40 @@ __Special note on `IdentityFile=~/.ssh/alpine`:__ this `sshfs` option tells SSH
```

Note: speed comparisons between `sshfs` (PetaLibrary) and CIFS (Isilon) have not yet been performed; consider this if performance becomes a concern.

### Connection script

Please feel free to use the following script to automatically help setup your mount point to the Way Lab specific mount point on CU Boulder Research Computing PetaLibrary: `koala`.

```shell
curl https://raw.githubusercontent.com/WayScience/playbooks/refs/heads/main/internal/mount_koala.sh | sh
Comment thread
MikeLippincott marked this conversation as resolved.
```

## Mounting both Isilon and PetaLibrary at once

Please feel free to use the following script to automatically help setup your mount point to the Way Lab specific mount point for both PetaLibrary and Isilon: `koala` and `bandicoot`, respectively.

```shell
curl https://raw.githubusercontent.com/WayScience/playbooks/refs/heads/main/internal/mount_nas.sh | sh
Comment thread
coderabbitai[bot] marked this conversation as resolved.
```

## Interacting with both filesystems programmatically

This can be done using a helper function `nas_path_check` in `internal/nas_path_set.py` which will check if the path is on either filesystem and return the root directory of the filesystem and whether the code is running in a notebook environment.
Comment thread
MikeLippincott marked this conversation as resolved.

This function can be installed in your Python environment with the following command:

```shell
pip install git+https://github.com/WayScience/playbooks.git#subdirectory=internal/nas_path_package
```

```python
from nas_path_package import init_notebook, nas_path_check

root_dir, in_notebook = init_notebook()
data_dir = nas_path_check(root_dir, nas_name="bandicoot")
Comment thread
MikeLippincott marked this conversation as resolved.
```

The data_dir can now be used to access the data on the NAS filesystem.
If the NAS is not mounted, it will fall back to the enclosing Git repository's root directory.
This keeps the code portable and allows for easy access to data on the NAS filesystem without hardcoding paths and allows for someone without access to the NAS to still run the code without errors using local storage.
41 changes: 35 additions & 6 deletions internal/mount_bandicoot.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@
# verifies VPN/network access, and works under any POSIX shell
# (sh, bash, zsh, dash, etc.).
# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

set -eu
# -e: exit immediately on any error
# -u: treat unset variables as an error
Expand Down Expand Up @@ -51,7 +50,7 @@ case "$OS" in
echo "→ Detected macOS (Darwin). Using mount_smbfs."
#
# mount_smbfs is the built-in macOS SMB client.
# It will prompt you for credentials if required,
# It will prompt you for a password if required,
# or use your current login keychain.
#
# NOTE: mount_smbfs falls back to your local Mac shortname as the
Expand All @@ -68,8 +67,35 @@ case "$OS" in
exit 1
fi

SHARE_WITH_USER="//${SMB_USERNAME}@${SHARE#//}"
mount_smbfs "$SHARE_WITH_USER" "$MOUNT_POINT"
# NOTE: unlike mount.cifs on Linux, mount_smbfs has no "domainauto"
# option — there is no way for it to negotiate the AD domain on its
# own, so it must be supplied explicitly as //DOMAIN;user@host/share.
# A wrong domain here fails with a generic "Authentication error"
# that looks identical to a wrong password, which is what was
# happening with the hardcoded DOMAIN="UCDENVER" value. Since the
# correct NetBIOS domain can differ depending on how an account was
# provisioned (CU Anschutz's AD forest predates its current name),
# prompt for it instead of hardcoding it, so anyone hitting this can
# self-correct without editing the script.
printf "AD domain for %s [default: UCDENVER, leave as '-' to omit]: " "$HOST" >/dev/tty
read -r SMB_DOMAIN </dev/tty
SMB_DOMAIN="${SMB_DOMAIN:-UCDENVER}"

if [ "$SMB_DOMAIN" = "-" ]; then
SHARE_WITH_USER="//${SMB_USERNAME}@${SHARE#//}"
else
SHARE_WITH_USER="//${SMB_DOMAIN};${SMB_USERNAME}@${SHARE#//}"
fi

# Per Apple's mount_smbfs(8) man page: "You should always use the
# system mount command and never call mount_smbfs directly."
if ! mount -t smbfs "$SHARE_WITH_USER" "$MOUNT_POINT"; then
echo "✗ Failed to mount $SHARE at $MOUNT_POINT." >&2
echo " A failure here is almost always a wrong AD domain, not a" >&2
echo " wrong password. Try again with a different domain (e.g." >&2
echo " UCHSC, the campus's legacy AD name) or '-' to omit it." >&2
exit 1
fi
;;

Linux)
Expand Down Expand Up @@ -106,8 +132,11 @@ case "$OS" in
exit 1
fi
# Mount the share with domainauto for automatic domain selection
sudo mount -t cifs "$SHARE" "$MOUNT_POINT" \
-o username="$CIFS_USERNAME",uid="$USER",gid="$USER",domainauto,file_mode=0777,dir_mode=0777
if ! sudo mount -t cifs "$SHARE" "$MOUNT_POINT" \
-o username="$CIFS_USERNAME",uid="$USER",gid="$USER",domainauto,file_mode=0777,dir_mode=0777; then
echo "✗ mount.cifs failed to mount $SHARE at $MOUNT_POINT." >&2
exit 1
fi
;;

*)
Expand Down
155 changes: 155 additions & 0 deletions internal/mount_koala.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,155 @@
#!/usr/bin/env sh
# shellcheck shell=sh
# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
# mount_koala.sh
#
# This script creates a local mount point and mounts a PetaLibrary
# directory on CU Boulder Research Computing's HPC Cluster Alpine
# (aka "koala") into ~/mnt/koala using sshfs over SSH.
# It auto-detects macOS vs. Linux, installs sshfs (and macFUSE on macOS)
# if needed, and works under any POSIX shell (sh, bash, zsh, dash, etc.).
# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

set -eu
# -e: exit immediately on any error
# -u: treat unset variables as an error

# Local directory where the PetaLibrary directory will be mounted
MOUNT_POINT="$HOME/mnt/koala"

# Alpine SSH login host
ALPINE_HOST="login.rc.colorado.edu"

# Path to the SSH private key used to authenticate to Alpine
IDENTITY_FILE="$HOME/.ssh/alpine"

# ────────────────────────────────────────────────────────────────────────────
# 1) Ensure the mount directory exists
# ────────────────────────────────────────────────────────────────────────────
if [ ! -d "$MOUNT_POINT" ]; then
echo "→ Creating mount point directory: $MOUNT_POINT"
mkdir -p "$MOUNT_POINT"
# mkdir -p will also create any missing parent directories
fi

# If a previous (possibly stale/broken) mount is already attached here,
# sshfs will fail with a generic "mount failed with error: -1" and we'd
# rather unmount it first than leave a confusing failure.
if mount | grep -q " on $MOUNT_POINT "; then
echo "→ $MOUNT_POINT is already mounted. Unmounting first..."
if ! umount "$MOUNT_POINT" 2>/dev/null; then
if command -v diskutil >/dev/null 2>&1; then
diskutil unmount force "$MOUNT_POINT"
elif command -v fusermount >/dev/null 2>&1; then
fusermount -u "$MOUNT_POINT"
fi
fi
fi

# ────────────────────────────────────────────────────────────────────────────
# 2) Prompt for Alpine username and PetaLibrary directory
# ────────────────────────────────────────────────────────────────────────────
# NOTE: for CU Anschutz users, this will almost always take the form of an
# XSEDE-style identity: <your-rc-username>@xsede.org, rather than a plain
# CU Anschutz username.
printf "Alpine username (e.g. <your-rc-username>@xsede.org): " >/dev/tty
read -r ALPINE_USERNAME </dev/tty # read from the terminal, not the script pipe
if [ -z "$ALPINE_USERNAME" ]; then
echo "✗ Username cannot be empty." >&2
exit 1
fi

printf "PetaLibrary directory (relative to /pl/active/): " >/dev/tty
read -r PETALIBRARY_DIR </dev/tty # read from the terminal, not the script pipe
if [ -z "$PETALIBRARY_DIR" ]; then
echo "✗ PetaLibrary directory cannot be empty." >&2
exit 1
fi

if [ ! -f "$IDENTITY_FILE" ]; then
echo "✗ SSH identity file not found at $IDENTITY_FILE." >&2
echo " Set up an SSH key for Alpine access and place it there," >&2
echo " or edit IDENTITY_FILE in this script to point at your key." >&2
exit 1
fi

# ────────────────────────────────────────────────────────────────────────────
# 3) Detect operating system, ensure sshfs is installed, and mount
# ────────────────────────────────────────────────────────────────────────────
OS="$(uname)"
case "$OS" in
Darwin)
# macOS branch
echo "→ Detected macOS (Darwin). Verifying FUSE-T and sshfs are installed..."
#
# sshfs on macOS requires a FUSE implementation. FUSE-T is used here
# instead of macFUSE because it doesn't need a kernel extension
# (no reboot, no Privacy & Security approval step).
#
if ! command -v sshfs >/dev/null 2>&1; then
echo "→ sshfs not found. Attempting installation via Homebrew..."
if ! command -v brew >/dev/null 2>&1; then
echo "✗ Homebrew not found. Please install FUSE-T and sshfs manually:" >&2
echo " https://github.com/macos-fuse-t/fuse-t" >&2
exit 1
fi
# updating from deprecated tap macos-fuse-t/fuse-t
# to macos-fuse-t/homebrew-cask
brew tap macos-fuse-t/homebrew-cask
brew install fuse-t fuse-t-sshfs

# FUSE-T ships libfuse-t.dylib instead of the classic libfuse
# name sshfs looks for, so symlink it in.
if [ ! -e /usr/local/lib/libfuse.2.dylib ]; then
sudo mkdir -p /usr/local/lib
sudo ln -s /usr/local/lib/libfuse-t.dylib /usr/local/lib/libfuse.2.dylib
fi
fi
;;

Linux)
# Linux branch
echo "→ Detected Linux. Verifying sshfs is installed..."
#
# sshfs is provided by the sshfs package (fuse-sshfs on some
# distributions). If it's missing, we detect your package manager
# and install it.
#
if ! command -v sshfs >/dev/null 2>&1; then
echo "→ sshfs not found. Attempting installation..."
if command -v apt-get >/dev/null 2>&1; then
echo " • Using apt-get to install sshfs"
sudo apt-get update
sudo apt-get install -y sshfs
elif command -v dnf >/dev/null 2>&1; then
echo " • Using dnf to install fuse-sshfs"
sudo dnf install -y fuse-sshfs
elif command -v apk >/dev/null 2>&1; then
echo " • Using apk to install sshfs"
sudo apk add sshfs
else
echo "✗ Unsupported package manager. Please install sshfs manually." >&2
exit 1
fi
fi
;;

*)
# Unsupported OS
echo "✗ Unsupported operating system: $OS" >&2
exit 1
;;
esac

echo "→ Mounting PetaLibrary directory /pl/active/$PETALIBRARY_DIR via sshfs..."
if ! sshfs -o IdentityFile="$IDENTITY_FILE" \
"$ALPINE_USERNAME@$ALPINE_HOST:/pl/active/$PETALIBRARY_DIR" \
"$MOUNT_POINT"; then
echo "✗ sshfs failed to mount /pl/active/$PETALIBRARY_DIR at $MOUNT_POINT." >&2
exit 1
fi

# ────────────────────────────────────────────────────────────────────────────
# 4) Success message
# ────────────────────────────────────────────────────────────────────────────
echo "✔ Successfully mounted /pl/active/$PETALIBRARY_DIR at $MOUNT_POINT"
41 changes: 41 additions & 0 deletions internal/mounting_nas.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
#!/usr/bin/env sh
# shellcheck shell=sh
# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
# mounting_nas.sh
#
# Interactive entry point for mounting the lab's network storage.
# Prompts for which storage solution(s) to mount and delegates to
# mount_bandicoot.sh (Isilon, CIFS/SMB) and/or mount_koala.sh
# (PetaLibrary/Alpine, sshfs), which live alongside this script.
# ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

set -eu
# -e: exit immediately on any error
# -u: treat unset variables as an error

echo "Which NAS would you like to mount?"
echo " 1) bandicoot (Isilon)"
echo " 2) koala (PetaLibrary / Alpine)"
echo " 3) both"
printf "Enter choice [1-3]: " >/dev/tty
read -r CHOICE </dev/tty

case "$CHOICE" in
1)
curl https://raw.githubusercontent.com/WayScience/playbooks/refs/heads/main/internal/mount_bandicoot.sh | sh

;;
2)
curl https://raw.githubusercontent.com/WayScience/playbooks/refs/heads/main/internal/mount_koala.sh | sh

;;
3)
curl https://raw.githubusercontent.com/WayScience/playbooks/refs/heads/main/internal/mount_bandicoot.sh | sh
curl https://raw.githubusercontent.com/WayScience/playbooks/refs/heads/main/internal/mount_koala.sh | sh

;;
*)
echo "✗ Invalid choice: $CHOICE" >&2
exit 1
;;
esac
26 changes: 26 additions & 0 deletions internal/nas_path_package/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# nas_path_package

Helpers for locating Way Lab NAS mount points (`bandicoot`, `koala`) from notebooks and scripts, falling back to the enclosing Git repository's root directory when a mount isn't present.

## Install

If locally developing, install in editable mode:

```shell
pip install -e internal/nas_path_package
```

If not developing, install from github:

```shell
pip install git+https://github.com/WayScience/playbooks.git#subdirectory=internal/nas_path_package
```

## Usage

```python
from nas_path_package import init_notebook, nas_path_check

root_dir, in_notebook = init_notebook()
data_dir = nas_path_check(root_dir, nas_name="bandicoot")
```
16 changes: 16 additions & 0 deletions internal/nas_path_package/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "nas_path_package"
version = "0.1.0"
description = "Helpers for locating Way Lab NAS mount points (bandicoot, koala) from notebooks and scripts."
readme = "README.md"
requires-python = ">=3.9"

[project.optional-dependencies]
test = ["pytest"]

[tool.hatch.build.targets.wheel]
packages = ["src/nas_path_package"]
Loading
Loading