AWS setup at Harvard

Knowledge base

• Overview: How AWS works
  • High-level workflow
  • What this means in practice
  • How this documentation is organized
• IAM identity
  • IAM user
  • IAM role
  • IAM roles in ParallelCluster
• AWS APIs
  • Checking AWS API permissions
  • Simulating API permissions
  • Practical considerations
• AWS services and resources
  • Examples of AWS resources
  • Example AWS services and resources
  • Miscellaneous clarifications
• Glossary

Operational tutorials

• Quickstart overview
  • High-level workflow
  • Assumptions
• AWS utilities
  • Official installation instructions
  • Install using conda or mamba
  • Confirm installation
• AWS account setup
  • SSO login
  • Create and store key pair
  • AWS configure (IAM user)
• Build a custom Amazon Machine Image
  • Build AMI from AWS Console
  • Build AMI with ParallelCluster based on an existing AMI
• Create an Amazon S3 Bucket
  • Prerequisites
  • Method 1: Create an S3 Bucket via AWS Management Console
  • Method 2: Create an S3 Bucket via AWS CLI
  • Add FSx access to allow for DRA
  • Verification
  • Notes and Best Practices
  • Next Steps
• Upload Data to an Amazon S3 Bucket
  • Prerequisites
  • Upload Input Data Using Python Scripts (Recommended)
  • Upload Output Data from FSx for Lustre (Best for Large Outputs)
  • Other Official Methods
  • Common Permission Requirements
  • Verification
  • Checking S3 Bucket Size (for FSx Planning)
  • Notes and Best Practices
  • Next Steps
• Create an FSx file system
  • Create FSx through AWS Console
  • Create FSx through AWS CLI
  • Mount FSx to an EC2 instance
• Data transfer between FSx and S3 bucket
  • Launch an EC2 instance for data transfer
  • Data repository association (DRA)
• Loading Data from S3 to FSx and Using S3 as Backup
  • Background and Design Philosophy
  • Data Repository Associations (DRA)
  • How Data Is Loaded from S3 to FSx
  • FSx Storage Capacity Considerations
  • Recommended Backup Strategy Using S3
  • When Is It Safe to Delete the S3 Bucket?
  • Summary
• Create a ParallelCluster via AWS CLI
  • Create the cluster
  • Monitor the creation process
  • Delete ParallelCluster
• Teardown and cleanup
  • EC2 instances
  • ParallelCluster

Resources

• Templates
  • Template: AMI build configuration (ami-build.yaml)
  • Template: Cluster configuration (pcluster-create.yml)
• Cost considerations: S3 vs FSx for Lustre vs compute
  • High-level pricing overview
  • S3: lowest-cost persistent storage
  • FSx for Lustre: high-performance, higher-cost storage
  • Compute cost usually dominates overall spending
  • Recommended cost-efficient pattern
• Reference links