AWS setup at Harvard

Knowledge base

• Overview: How AWS works
  • High-level workflow
  • What this means in practice
  • How this documentation is organized
• IAM identity
  • IAM user
  • IAM role
  • IAM roles in ParallelCluster
• AWS APIs
  • Checking AWS API permissions
  • Simulating API permissions
  • Practical considerations
• AWS services and resources
  • Examples of AWS resources
  • Example AWS services and resources
  • Miscellaneous clarifications
• Glossary

Operational tutorials

• Quickstart overview
  • High-level workflow
  • Assumptions
• AWS utilities
  • Official installation instructions
  • Install using conda or mamba
  • Confirm installation
• AWS account setup
  • SSO login
  • Create and store key pair
  • AWS configure (IAM user)
• Build a custom Amazon Machine Image
  • Build AMI from AWS Console
  • Build AMI with ParallelCluster based on an existing AMI
• Create an Amazon S3 Bucket
  • Prerequisites
  • Method 1: Create an S3 Bucket via AWS Management Console
  • Method 2: Create an S3 Bucket via AWS CLI
  • Add FSx access to allow for DRA
  • Verification
  • Notes and Best Practices
  • Next Steps
• Upload Data to an Amazon S3 Bucket
  • Prerequisites
  • Upload Input Data Using Python Scripts (Recommended)
  • Upload Output Data from FSx for Lustre (Best for Large Outputs)
  • Other Official Methods
  • Common Permission Requirements
  • Verification
  • Checking S3 Bucket Size (for FSx Planning)
  • Notes and Best Practices
  • Next Steps
• Create an FSx file system
  • Create FSx through AWS Console
  • Create FSx through AWS CLI
  • Mount FSx to an EC2 instance
• Data transfer between FSx and S3 bucket
  • Launch an EC2 instance for data transfer
  • Data repository association (DRA)
• Loading Data from S3 to FSx and Using S3 as Backup
  • Background and Design Philosophy
  • Data Repository Associations (DRA)
  • How Data Is Loaded from S3 to FSx
  • FSx Storage Capacity Considerations
  • Recommended Backup Strategy Using S3
  • When Is It Safe to Delete the S3 Bucket?
  • Summary
• Create a ParallelCluster via AWS CLI
  • Create the cluster
  • Check the availability of spot instances
  • Monitor the creation process
  • Delete ParallelCluster
• Teardown and cleanup
  • EC2 instances
  • ParallelCluster

IMI output management

• Concepts: Markers, Cutoff, and Why Pruning Is Safe
  • The Problem
  • Where the Postprocessing Should Run
  • Directory Layout
  • The Shared End Date
  • Which Granules a Round Takes
  • The Two Markers
  • The data_converted Manifest
  • What the Prune Checks
  • What Gets Deleted
  • Why Checkpoints Are Kept From S Onward
  • Checking OutputDir Before Pruning
  • Why Freshly Written Data Need Not Be Re-Verified
  • Why the Overpass Archive Is Worth Keeping
  • Verifying Uploads
• Running on a Local Server
  • Prerequisites
  • One-Time Configuration
  • Shell Variables Used Below
  • Optional Settings
  • Fetching From S3
  • Automated: One Command Per Face
  • Many Faces at Once
  • Stopping
  • Manual: Step by Step
  • Why S3 Deletion Is Explicit Here
• Running on AWS ParallelCluster
  • How This Differs from a Local Server
  • Before the First Prune
  • Pruning Inside the IMI Job
  • Driving the Faces
  • Testing a Single Face
  • Releasing Space Between Faces
  • Pruning by Hand
  • Options
  • Reading the Output
  • Repeat as Runs Advance
  • Verification Runs Once Per Face

Resources

• Templates
  • Template: AMI build configuration (ami-build.yaml)
  • Template: Cluster configuration (pcluster-create.yml)
• Cost considerations: S3 vs FSx for Lustre vs compute
  • High-level pricing overview
  • S3: lowest-cost persistent storage
  • FSx for Lustre: high-performance, higher-cost storage
  • Compute cost usually dominates overall spending
  • Recommended cost-efficient pattern
• Reference links