Semantic Coherence: PyTorch Foundation – Signal Evidence & AI Readability

[H1] JOIN US
[H3] PyTorch Conference North America
October 20-21, 2026
San Jose, CA
#PyTorchCon
SUBMIT TO SPEAKSPONSOR

Get Started:  Install PyTorch Locally or Launch Instantly on Supported Cloud Platforms

Get started

A little over a year ago, the PyTorch Foundation launched the Ambassador Program, an initiative that recognizes and supports independent, trusted voices in the PyTorch community who are passionate about… Read More

Thank you to everyone who participated in the PyTorch Docathon 2026! Once again, the community showed up with incredible energy and dedication to make PyTorch documentation better for developers everywhere.… Read More

TLDR: PyTorch 2.11 makes it possible to install CUDA-enabled PyTorch wheels on aarch64 Linux directly from PyPI, eliminating the need for custom package indexes and workarounds that previously complicated deployment… Read More

[H2] Join PyTorch Foundation
As a member of the PyTorch Foundation, you’ll have access to resources that allow you to be stewards of stable, secure, and long-lasting codebases. You can collaborate on training, local and regional events, open-source developer tooling, academic research, and guides to help new users and contributors have a productive experience.
EXPLORE BENEFITS

[H2] Key Features & Capabilities

[H5] Production Ready
Transition seamlessly between eager and graph modes with TorchScript, and accelerate the path to production with TorchServe.

[H5] Distributed Training
Scalable distributed training and performance optimization in research and production is enabled by the torch.distributed backend.

[H5] Robust Ecosystem
A rich ecosystem of tools and libraries extends PyTorch and supports development in computer vision, NLP and more.

[H5] Cloud Support
PyTorch is well supported on major cloud platforms, providing frictionless development and easy scaling.

[H2] Install PyTorch
Select your preferences and run the install command. Stable represents the most currently tested and supported version of PyTorch. This should be suitable for many users. Preview is available if you want the latest, not fully tested and supported, builds that are generated nightly. Please ensure that you have met the prerequisites below (e.g., numpy), depending on your package manager. You can also install previous versions of PyTorch. Note that LibTorch is only available for C++.
NOTE: Latest Stable PyTorch requires Python 3.10 or later.

PyTorch Build

Your OS

Package

Language

Compute Platform

Run this Command:

PyTorch Build

Stable (2.7.0)

Preview (Nightly)

Your OS

Linux

Mac

Windows

Package

Pip

LibTorch

Source

Language

Python

C++ / Java

Compute Platform

CUDA 11.8

CUDA 12.6

CUDA 12.8

ROCm 6.3

CPU

Run this Command:

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

Previous versions of PyTorch

[H3] Quick Start With Cloud Partners
Get up and running with PyTorch quickly through popular cloud platforms and machine learning services.
[H3] Amazon Web Services
PyTorch on AWS
Amazon SageMaker
AWS Deep Learning Containers
AWS Deep Learning AMIs
[H3] Google Cloud Platform
Cloud Deep Learning VM Image
Deep Learning Containers
[H3] Microsoft Azure
PyTorch on Azure
Azure Machine Learning
Azure Functions
[H3] Lightning Studios
lightning.ai

[H2] Ecosystem

BROWSE PROJECTS

[H5] Featured Projects
Explore a rich ecosystem of libraries, tools, and more to support development.

[H4] Captum
Captum (“comprehension” in Latin) is an open source, extensible library for model interpretability built on PyTorch.

[H4] PyTorch Geometric
PyTorch Geometric is a library for deep learning on irregular input data such as graphs, point clouds, and manifolds.

[H4] skorch
skorch is a high-level library for PyTorch that provides full scikit-learn compatibility.

[H2] Companies & Universities Using PyTorch

[H5] Amazon Advertising
Reduce inference costs by 71% and scale out using PyTorch, TorchServe, and AWS Inferentia.
READ CASE STUDIES

[H5] Salesforce
Pushing the state of the art in NLP and Multi-task learning.

[H5] Stanford University
Using PyTorch’s flexibility to efficiently research new algorithmic approaches.

Close Menu

Join the PyTorch Foundation Ambassador Program: A Global Network of Community Leaders
AnnouncementsBlog

A little over a year ago, the PyTorch Foundation launched the Ambassador Program, an initiative…
[IMG: PyTorch Foundation]
PyTorch FoundationMay 22, 2026

PyTorch Docathon 2026 Results in 150+ Merged Pull Requests

[IMG: PyTorch Docathon 2026 Top Community Contributors]
AnnouncementsBlog

Thank you to everyone who participated in the PyTorch Docathon 2026! Once again, the community…
[IMG: PyTorch Foundation]
PyTorch FoundationMay 20, 2026

vLLM and PyTorch Work Together to Improve the Developer Experience on aarch64
Blog

TLDR: PyTorch 2.11 makes it possible to install CUDA-enabled PyTorch wheels on aarch64 Linux directly…
[IMG: Kaichao You (Inferact)]
Kaichao You (Inferact)May 18, 2026

Running PyTorch Models on Apple Silicon GPUs with the ExecuTorch MLX Delegate
Blog

TL;DR: Introducing the ExecuTorch MLX Delegate The new MLX delegate enables optimized, GPU-accelerated inference for…
[IMG: ExecuTorch Team]
ExecuTorch TeamMay 18, 2026

PyTorch 2.12 Release Blog
Blog

We are excited to announce the release of PyTorch® 2.12 (release notes)!   The PyTorch 2.12…
[IMG: PyTorch Foundation]
PyTorch FoundationMay 13, 2026

Efficient Edge AI on Arm CPUs and NPUs: Understanding ExecuTorch through Practical Labs
Blog

TL;DR: ExecuTorch extends the PyTorch ecosystem to deliver local AI inference on constrained edge devices.…
[IMG: Matt Cossins]
Matt CossinsMay 12, 2026

In-Kernel Broadcast Optimization: Co-Designing Kernels for RecSys Inference
Blog

TL;DR: Traditional RecSys inference explicitly replicates shared user embeddings/sequences for every candidate. In-Kernel Broadcast Optimization…
[IMG: Jian Jiao, Boda Li, Hongtao Yu, Yuanwei (Kevin) Fang, Zhengkai Zhang, Zhuoran Zhao, Yuxin Chen, Sijia Chen†, Yang Chen†, Zijian Shen, Shuyao Bi, Ao Cai, Junhan Hu†, Shuqi Yang†, Wei Wei, Lu Fang, Rengan Xu, Manman Ren, Alex Zhong, Xiaohan Wei, Zeliang Chen, Ellie Wen, Wenlin Chen]
Jian Jiao, Boda Li, Hongtao Yu, Yuanwei (Kevin) Fang, Zhengkai Zhang, Zhuoran Zhao, Yuxin Chen, Sijia Chen†, Yang Chen†, Zijian Shen, Shuyao Bi, Ao Cai, Junhan Hu†, Shuqi Yang†, Wei Wei, Lu Fang, Rengan Xu, Manman Ren, Alex Zhong, Xiaohan Wei, Zeliang Chen, Ellie Wen, Wenlin ChenMay 5, 2026

SMG: The Case for Disaggregating CPU from GPU in LLM Serving
Blog

How It Started: Hitting the GIL Wall at Scale We've been running production model serving…
[IMG: Simo Lin, Chang Su, and Keyang Ru, members of LightSeek Foundation]
Simo Lin, Chang Su, and Keyang Ru, members of LightSeek FoundationApril 30, 2026

Introducing AutoSP
Blog

¹ SSAIL Lab, University of Illinois Urbana-Champaign, ² Anyscale, ³ Snowflake TL;DR: AutoSP automatically converts…
[IMG: Ahan Gupta¹, Zhihao Wang¹, Neel Dani¹, Masahiro Tanaka², Olatunji Ruwase³, Minjia Zhang¹]
Ahan Gupta¹, Zhihao Wang¹, Neel Dani¹, Masahiro Tanaka², Olatunji Ruwase³, Minjia Zhang¹April 29, 2026

IBM Research uses vLLM at the heart of its RITS Platform
BlogCase Studies

TL;DR: vLLM has been critical to democratizing access to our research community to the latest…
[IMG: PyTorch Foundation]
PyTorch FoundationApril 24, 2026

Optimizing Effective Training Time for Meta’s Internal Recommendation/Ranking Workloads
Blog

Motivation and Introduction Across the industry, teams training and serving large AI models face aggressive…
[IMG: Ruilin Chen, Yuzhen Huang, Hang Qi, Mingming Ding, Damian Reeves, Boris Sarana, Kevin Tang, Satendra Gera, Gagan Jain, Sahil Shah, Oguz Ulgen, Mayank Garg, Meet Vadakkanchery, James March, Sophie Lin, Wei Sun]
Ruilin Chen, Yuzhen Huang, Hang Qi, Mingming Ding, Damian Reeves, Boris Sarana, Kevin Tang, Satendra Gera, Gagan Jain, Sahil Shah, Oguz Ulgen, Mayank Garg, Meet Vadakkanchery, James March, Sophie Lin, Wei SunApril 17, 2026

PyTorch Conference Europe 2026: A Landmark Moment for Open Source AI in Paris
AnnouncementsBlog

The first-ever PyTorch Conference Europe April 7-8, 2026 brought together more than 600 researchers, developers,…
[IMG: PyTorch Foundation]
PyTorch FoundationApril 15, 2026

Faster Diffusion on Blackwell: MXFP8 and NVFP4 with Diffusers and TorchAO
Blog

Diffusion models for image and video generation have been surging in popularity, delivering super-realistic visual…
[IMG: Vasiliy Kuznetsov (Meta) and Sayak Paul (Hugging Face)]
Vasiliy Kuznetsov (Meta) and Sayak Paul (Hugging Face)April 8, 2026

PyTorch Foundation Announces Safetensors as Newest Contributed Project to Secure AI Model Execution

[IMG: Safetensors Joins PyTorch Foundation]
AnnouncementsBlogPress Release

Safetensors is welcomed into the PyTorch Foundation to secure model distribution and build trusted agentic…
[IMG: PyTorch Foundation]
PyTorch FoundationApril 8, 2026

Monarch: an API to your supercomputer
Blog

Getting distributed training jobs to run on huge clusters is hard!  This is especially true…
[IMG: The PyTorch Team at Meta]
The PyTorch Team at MetaApril 8, 2026

SOTA Normalization Performance with torch.compile

[IMG: SOTA Normalization Performance with torch.compile]
Blog

Introduction Normalization methods (LayerNorm/RMSNorm) are foundational in deep learning and are used to normalize values…
[IMG: Shunting Zhang, Paul Zhang, Elias Ellison, Markus Hoehnerbach, Jason Ansel, Natalia Gimelshein]
Shunting Zhang, Paul Zhang, Elias Ellison, Markus Hoehnerbach, Jason Ansel, Natalia GimelsheinApril 8, 2026

ExecuTorch Becomes a Part of PyTorch Core to Expand On-Device Inference Capabilities

[IMG: ExecuTorch Becomes a part of PyTorch Core]
AnnouncementsBlog

Today, we’re excited to share that ExecuTorch is becoming a part of PyTorch Core. ExecuTorch…
[IMG: PyTorch Foundation]
PyTorch FoundationApril 7, 2026

PyTorch Foundation Welcomes Helion as a Foundation-Hosted Project to Standardize Open, Portable, and Accessible AI Kernel Authoring

[IMG: PyTorch Foundation Welcome Helion]
AnnouncementsBlogPress Release

Helion joins community of leading open source AI projects to simplify kernel development across the…
[IMG: PyTorch Foundation]
PyTorch FoundationApril 7, 2026

Generating State-of-the-Art GEMMs with TorchInductor’s CuteDSL backend

[IMG: Generating State-of-the-Art GEMMs with TorchInductor’s CuteDSL Backend]
Blog

Introduction TorchInductor currently supports three autotuning backends for matrix multiplications: Triton, CUTLASS (C++), and cuBLAS.…
[IMG: Nikhil Patel, Michael Lazos, Driss Guessous, Elias Ellison, Meta]
Nikhil Patel, Michael Lazos, Driss Guessous, Elias Ellison, MetaApril 7, 2026

RSVP for the 2026 PyTorch Docathon
Announcements

We're excited to announce that the 2026 PyTorch Docathon will take place May 5-19! This…
[IMG: Team PyTorch]
Team PyTorchApril 3, 2026

Close Menu

index

Run in Google Colab
Colab

Download Notebook
Notebook

View on GitHub
GitHub

[H1] Welcome to PyTorch Tutorials#
What’s new in PyTorch tutorials?
Data Loading Optimization in PyTorch
Distributed Training with Ray Train
Serve PyTorch models at scale with Ray Serve
Hyperparameter tuning using Ray Tune
Memory Profiling with Mosaic
Using Variable Length Attention in PyTorch
DebugMode: Recording Dispatched Operations and Numerical Debugging

[H3] Learn the Basics
Familiarize yourself with PyTorch concepts and modules. Learn how to load data, build deep neural networks, train and save your models in this quickstart guide.
Get started with PyTorch

[H3] PyTorch Recipes
Bite-size, ready-to-deploy PyTorch code examples.
Explore Recipes

[H4] Learn the Basics
A step-by-step guide to building a complete ML workflow with PyTorch.
Getting-Started

[H4] Introduction to PyTorch on YouTube
An introduction to building a complete ML workflow with PyTorch. Follows the PyTorch Beginner Series on YouTube.
Getting-Started

[H4] Learning PyTorch with Examples
This tutorial introduces the fundamental concepts of PyTorch through self-contained examples.
Getting-Started

[H4] What is torch.nn really?
Use torch.nn to create and train a neural network.
Getting-Started

[H4] Visualizing Models, Data, and Training with TensorBoard
Learn to use TensorBoard to visualize data and model training.
Interpretability,Getting-Started,TensorBoard

[H4] Good usage of `non_blocking` and `pin_memory()` in PyTorch
A guide on best practices to copy data from CPU to GPU.
Getting-Started

[H4] Data Loading Optimization in PyTorch
Optimize DataLoader configuration with num_workers, pin_memory, persistent_workers for maximum training throughput.
Getting-Started,Best-Practice

[H4] Understanding requires_grad, retain_grad, Leaf, and Non-leaf Tensors
Learn the subtleties of requires_grad, retain_grad, leaf, and non-leaf tensors
Getting-Started

[H4] Visualizing Gradients in PyTorch
Visualize the gradient flow of a network.
Getting-Started

[H4] TorchVision Object Detection Finetuning Tutorial
Finetune a pre-trained Mask R-CNN model.
Image/Video

[H4] Transfer Learning for Computer Vision Tutorial
Train a convolutional neural network for image classification using transfer learning.
Image/Video

[H4] Adversarial Example Generation
Train a convolutional neural network for image classification using transfer learning.
Image/Video

[H4] DCGAN Tutorial
Train a generative adversarial network (GAN) to generate new celebrities.
Image/Video

[H4] Spatial Transformer Networks Tutorial
Learn how to augment your network using a visual attention mechanism.
Image/Video

[H4] Semi-Supervised Learning Tutorial Based on USB
Learn how to train semi-supervised learning algorithms (on custom data) using USB and PyTorch.
Image/Video

[H4] Distributed Training with Ray Train
Pre-train a transformer language model across multiple GPUs using PyTorch and Ray Train.
Text,Best-Practice,Ray-Distributed,Parallel-and-Distributed-Training

[H4] Audio IO
Learn to load data with torchaudio.
Audio

[H4] Audio Resampling
Learn to resample audio waveforms using torchaudio.
Audio

[H4] Audio Data Augmentation
Learn to apply data augmentations using torchaudio.
Audio

[H4] Audio Feature Extractions
Learn to extract features using torchaudio.
Audio

[H4] Audio Feature Augmentation
Learn to augment features using torchaudio.
Audio

[H4] Audio Datasets
Learn to use torchaudio datasets.
Audio

[H4] Automatic Speech Recognition with Wav2Vec2 in torchaudio
Learn how to use torchaudio's pretrained models for building a speech recognition application.
Audio

[H4] Speech Command Classification
Learn how to correctly format an audio dataset and then train/test an audio classifier network on the dataset.
Audio

[H4] Text-to-Speech with torchaudio
Learn how to use torchaudio's pretrained models for building a text-to-speech application.
Audio

[H4] Forced Alignment with Wav2Vec2 in torchaudio
Learn how to use torchaudio's Wav2Vec2 pretrained models for aligning text to speech
Audio

[H4] NLP from Scratch: Classifying Names with a Character-level RNN
Build and train a basic character-level RNN to classify word from scratch without the use of torchtext. First in a series of three tutorials.
NLP

[H4] NLP from Scratch: Generating Names with a Character-level RNN
After using character-level RNN to classify names, learn how to generate names from languages. Second in a series of three tutorials.
NLP

[H4] NLP from Scratch: Translation with a Sequence-to-sequence Network and Attention
This is the third and final tutorial on doing “NLP From Scratch”, where we write our own classes and functions to preprocess the data to do our NLP modeling tasks.
NLP

[H4] Exporting a PyTorch model to ONNX using TorchDynamo backend and Running it using ONNX Runtime
Build a image classifier model in PyTorch and convert it to ONNX before deploying it with ONNX Runtime.
Production,ONNX,Backends

[H4] Extending the ONNX exporter operator support
Demonstrate end-to-end how to address unsupported operators in ONNX.
Production,ONNX,Backends

[H4] Exporting a model with control flow to ONNX
Demonstrate how to handle control flow logic while exporting a PyTorch model to ONNX.
Production,ONNX,Backends

[H4] Reinforcement Learning (DQN)
Learn how to use PyTorch to train a Deep Q Learning (DQN) agent on the CartPole-v0 task from the OpenAI Gym.
Reinforcement-Learning

[H4] Reinforcement Learning (PPO) with TorchRL
Learn how to use PyTorch and TorchRL to train a Proximal Policy Optimization agent on the Inverted Pendulum task from Gym.
Reinforcement-Learning

[H4] Train a Mario-playing RL Agent
Use PyTorch to train a Double Q-learning agent to play Mario.
Reinforcement-Learning

[H4] Recurrent DQN
Use TorchRL to train recurrent policies
Reinforcement-Learning

[H4] Code a DDPG Loss
Use TorchRL to code a DDPG Loss
Reinforcement-Learning

[H4] Writing your environment and transforms
Use TorchRL to code a Pendulum
Reinforcement-Learning

[H4] Serving PyTorch Tutorial
Deploy and scale a PyTorch model with Ray Serve.
Production,Best-Practice,Ray-Distributed,Ecosystem

[H4] Profiling PyTorch
Learn how to profile a PyTorch application
Profiling

[H4] Profiling PyTorch
Introduction to Holistic Trace Analysis
Profiling

[H4] Profiling PyTorch
Trace Diff using Holistic Trace Analysis
Profiling

[H4] Memory Profiling with Mosaic
Learn how to use the Mosaic memory profiler to visualize GPU memory usage and identify memory optimization opportunities in PyTorch models.
Model-Optimization,Best-Practice,Profiling

[H4] Building a Simple Performance Profiler with FX
Build a simple FX interpreter to record the runtime of op, module, and function calls and report statistics
FX

[H4] (beta) Channels Last Memory Format in PyTorch
Get an overview of Channels Last memory format and understand how it is used to order NCHW tensors in memory preserving dimensions.
Memory-Format,Best-Practice,Frontend-APIs

[H4] Using the PyTorch C++ Frontend
Walk through an end-to-end example of training a model with the C++ frontend by training a DCGAN – a kind of generative model – to generate images of MNIST digits.
Frontend-APIs,C++

[H4] PyTorch Custom Operators Landing Page
This is the landing page for all things related to custom operators in PyTorch.
Extending-PyTorch,Frontend-APIs,C++,CUDA

[H4] Custom Python Operators
Create Custom Operators in Python. Useful for black-boxing a Python function for use with torch.compile.
Extending-PyTorch,Frontend-APIs,C++,CUDA

[H4] Compiled Autograd: Capturing a larger backward graph for ``torch.compile``
Learn how to use compiled autograd to capture a larger backward graph.
Model-Optimization,CUDA

[H4] Custom C++ and CUDA Operators
How to extend PyTorch with custom C++ and CUDA operators.
Extending-PyTorch,Frontend-APIs,C++,CUDA

[H4] Autograd in C++ Frontend
The autograd package helps build flexible and dynamic neural netorks. In this tutorial, explore several examples of doing autograd in PyTorch C++ frontend
Frontend-APIs,C++

[H4] Registering a Dispatched Operator in C++
The dispatcher is an internal component of PyTorch which is responsible for figuring out what code should actually get run when you call a function like torch::add.
Extending-PyTorch,Frontend-APIs,C++

[H4] Extending Dispatcher For a New Backend in C++
Learn how to extend the dispatcher to add a new device living outside of the pytorch/pytorch repo and maintain it to keep in sync with native PyTorch devices.
Extending-PyTorch,Frontend-APIs,C++

[H4] Facilitating New Backend Integration by PrivateUse1
Learn how to integrate a new backend living outside of the pytorch/pytorch repo and maintain it to keep in sync with the native PyTorch backend.
Extending-PyTorch,Frontend-APIs,C++

[H4] Custom Function Tutorial: Double Backward
Learn how to write a custom autograd Function that supports double backward.
Extending-PyTorch,Frontend-APIs

[H4] Custom Function Tutorial: Fusing Convolution and Batch Norm
Learn how to create a custom autograd Function that fuses batch norm into a convolution to improve memory usage.
Extending-PyTorch,Frontend-APIs

[H4] Forward-mode Automatic Differentiation
Learn how to use forward-mode automatic differentiation.
Frontend-APIs

[H4] Jacobians, Hessians, hvp, vhp, and more
Learn how to compute advanced autodiff quantities using torch.func
Frontend-APIs

[H4] Model Ensembling
Learn how to ensemble models using torch.vmap
Frontend-APIs

[H4] Per-Sample-Gradients
Learn how to compute per-sample-gradients using torch.func
Frontend-APIs

[H4] Neural Tangent Kernels
Learn how to compute neural tangent kernels using torch.func
Frontend-APIs

[H4] Performance Profiling in PyTorch
Learn how to use the PyTorch Profiler to benchmark your module's performance.
Model-Optimization,Best-Practice,Profiling

[H4] Performance Profiling in TensorBoard
Learn how to use the TensorBoard plugin to profile and analyze your model's performance.
Model-Optimization,Best-Practice,Profiling,TensorBoard

[H4] Hyperparameter Tuning Tutorial
Learn how to use Ray Tune to find the best performing set of hyperparameters for your model.
Model-Optimization,Best-Practice,Ray-Distributed,Parallel-and-Distributed-Training

[H4] Parametrizations Tutorial
Learn how to use torch.nn.utils.parametrize to put constraints on your parameters (e.g. make them orthogonal, symmetric positive definite, low-rank...)
Model-Optimization,Best-Practice

[H4] Pruning Tutorial
Learn how to use torch.nn.utils.prune to sparsify your neural networks, and how to extend it to implement your own custom pruning technique.
Model-Optimization,Best-Practice

[H4] How to save memory by fusing the optimizer step into the backward pass
Learn a memory-saving technique through fusing the optimizer step into the backward pass using memory snapshots.
Model-Optimization,Best-Practice,CUDA,Frontend-APIs

[H4] (beta) Accelerating BERT with semi-structured sparsity
Train BERT, prune it to be 2:4 sparse, and then accelerate it to achieve 2x inference speedups with semi-structured sparsity and torch.compile.
Text,Model-Optimization

[H4] Multi-Objective Neural Architecture Search with Ax
Learn how to use Ax to search over architectures find optimal tradeoffs between accuracy and latency.
Model-Optimization,Best-Practice,Ax,TorchX

[H4] torch.compile Tutorial
Speed up your models with minimal code changes using torch.compile, the latest PyTorch compiler solution.
Model-Optimization

[H4] torch.compile End-to-End Tutorial
An example of applying torch.compile to a real model, demonstrating speedups.
Model-Optimization

[H4] Building a Convolution/Batch Norm fuser in torch.compile
Build a simple pattern matcher pass that fuses batch norm into convolution to improve performance during inference.
Model-Optimization

[H4] Inductor CPU Backend Debugging and Profiling
Learn the usage, debugging and performance profiling for ``torch.compile`` with Inductor CPU backend.
Model-Optimization

[H4] (beta) Implementing High-Performance Transformers with SCALED DOT PRODUCT ATTENTION
This tutorial explores the new torch.nn.functional.scaled_dot_product_attention and how it can be used to construct Transformer components.
Model-Optimization,Attention,Transformer

[H4] Knowledge Distillation in Convolutional Neural Networks
Learn how to improve the accuracy of lightweight models using more powerful models as teachers.
Model-Optimization,Image/Video

[H4] Accelerating PyTorch Transformers by replacing nn.Transformer with Nested Tensors and torch.compile()
This tutorial goes over recommended best practices for implementing Transformers with native PyTorch.
Transformer

[H4] PyTorch Distributed Overview
Briefly go over all concepts and features in the distributed package. Use this document to find the distributed training technology that can best serve your application.
Parallel-and-Distributed-Training

[H4] Distributed Data Parallel in PyTorch - Video Tutorials
This series of video tutorials walks you through distributed training in PyTorch via DDP.
Parallel-and-Distributed-Training

[H4] Single-Machine Model Parallel Best Practices
Learn how to implement model parallel, a distributed training technique which splits a single model onto different GPUs, rather than replicating the entire model on each GPU
Parallel-and-Distributed-Training

[H4] Getting Started with Distributed Data Parallel
Learn the basics of when to use distributed data paralle versus data parallel and work through an example to set it up.
Parallel-and-Distributed-Training

[H4] Writing Distributed Applications with PyTorch
Set up the distributed package of PyTorch, use the different communication strategies, and go over some the internals of the package.
Parallel-and-Distributed-Training

[H4] Large Scale Transformer model training with Tensor Parallel
Learn how to train large models with Tensor Parallel package.
Parallel-and-Distributed-Training

[H4] Customize Process Group Backends Using Cpp Extensions
Extend ProcessGroup with custom collective communication implementations.
Parallel-and-Distributed-Training

[H4] Getting Started with Distributed RPC Framework
Learn how to build distributed training using the torch.distributed.rpc package.
Parallel-and-Distributed-Training

[H4] Implementing a Parameter Server Using Distributed RPC Framework
Walk through a through a simple example of implementing a parameter server using PyTorch’s Distributed RPC framework.
Parallel-and-Distributed-Training

[H4] Introduction to Distributed Pipeline Parallelism
Demonstrate how to implement pipeline parallelism using torch.distributed.pipelining
Parallel-and-Distributed-Training

[H4] Implementing Batch RPC Processing Using Asynchronous Executions
Learn how to use rpc.functions.async_execution to implement batch RPC
Parallel-and-Distributed-Training

[H4] Combining Distrib

[H1] Developer Resources
Explore resources, get your questions answered, and join the discussion with other PyTorch developers.

[H4] PyTorchDocs
Access comprehensive developer documentation.

[H4] PyTorchDiscuss
Browse and join discussions on deep learning with PyTorch.

[H4] Slack
Discuss advanced topics.

[H4] Tutorials
Get in-depth tutorials for beginners and advanced developers.

[H4] 中文文档
Docs and tutorials in Chinese, translated by the community.

[H4] GitHub
Report bugs, request features, discuss issues, and more.

[H4] 파이토치(PyTorch)
Tutorials in Korean, translated by the community.

[H4] 日本語(PyTorch)
Tutorials in Japanese, translated by the community.

[H4] Examples
View example projects for vision, text, RL, and more.

[H4] Maintainers
Learn about the PyTorch core and module maintainers.

[H4] ContributionGuide
Learn how you can contribute to PyTorch code and documentation.

[H4] DesignPhilosophy
PyTorch design principles for contributors and maintainers.

[H4] PyTorch Dev Discussions
Forums used by PyTorch core developers and contributors to the PyTorch codebase for technical discussions.

[H4] Governance
Learn about the PyTorch governance hierarchy.

[H4] MobileDemo
Check out the PyTorch Mobile demo app for iOS and Android.

[H4] PyTorchTraining
Further your education and career goals.

[H4] Newsletter
Stay up-to-date with the latest updates.

Close Menu