Introduction to Parallel Programming Using Python

This repository provides an introduction to the concepts of parallel programming using Python.

Note

This project is under active development.

Contents

• Prerequisite
• Learning Outcomes
• Modules
• Python Virtual Environment
• Tutorial
  • Getting Started
  • Requesting a Job
  • High-level HPC Architecture
    • HPC Compute Node
    • How do you figure out the node architecture?
    • How does cache influence performance?
  • Basics of Parallelism
    • Process
    • Threads
    • Challenges with Threads
    • The Room and the Key: An analogy
    • Exercise
    • Single Instruction Multiple Data (SIMD)
  • Vector Parallelism
    • What is vectorization?
    • How does vectorization influence peformance?
    • Exercise
  • Multi-core Parallelism
    • Non-Uniform Memory Access (NUMA)
    • Exercise
  • GPU Parallelism
    • GPU Execution Model
    • Thread Indexing
    • How do we assign a unique thread id to each thread?
    • Warps and Warp Schedulers
    • Data Movement in GPUs
    • Streams
    • Exercise
  • Multi-node Parallelism
    • Broadcast Operation
    • GPU-aware MPI and All-Gather Operation
    • Exercise
• Reference
  • Contributers