A matmul in PyPTO, all the way down

PyPTO (say “pai p-t-o”) is a high-performance programming framework for Ascend AI accelerators: you write Python tensor code, and the compiler lowers it to hardware instructions. It is tile-based: every computation is a tile, a hardware-aware data block placed in the memory hierarchy and scheduled across cores, and a multi-level IR (tensor → tile → block → execution graph) reaches PTO instructions. Tensor-, tile- and block-level APIs serve algorithm, performance and system developers.

@pl.jit
def matmul(a: pl.Tensor[[128, 512], pl.FP16],
           b: pl.Tensor[[512, 128], pl.FP16],
           c: pl.Out[pl.Tensor[[128, 128], pl.FP32]]):
    with pl.at(level=pl.Level.CORE_GROUP):
        c[:] = pl.matmul(a, b, out_dtype=pl.FP32)
    return c

This page follows one matmul down that path: every tile, buffer, loop and flag below was placed by PyPTO, simpler and PTOAS.

What this page walks through

  • A matmul example to introduce the whole compilation pipeline of PyPTO and its runtime. You write a whole-tensor pl.matmul; PyPTO decides tile shapes, buffer placement and loop structure, PTOAS selects instructions and derives the intra-core synchronization, simpler builds the task graph and dispatches it. Each layer is shown with the artifact that proves its contribution: an IR diff, an address, a flag, a dependency edge.
  • Part I: the descent, in this order:
    • the call as written, token by token;
    • PyPTO: all 64 passes: the real diff of each of the 18 that change this program, and why the other 46 have nothing to do;
    • PTOAS: the module it receives, the C++ it emits, and the source of every set_flag/wait_flag in the kernel (21 pass invocations, 5 of which change the IR);
    • simpler: two programs come out (a cube kernel and an orchestration library), and one task goes in, with the dependency edges the runtime schedules on;
    • the core: the instruction stream of the chosen L0 tile: two loop iterations with two load/compute copies per iteration, ping-pong’d over 2 × 2 operand buffers the allocator added without being asked, and the flag schedule that overlaps them on four pipes.
  • Part II: the extra mile, and who walks it. Everything above is one task on one cube core of 24. Each thing an HPC matmul adds next (more cores, weights in the hardware’s native layout, operands larger than the on-chip buffers, overlap, collectives) has an owner: a PyPTO pass, a pypto-lib feature, a runtime behaviour, or one more line of your DSL. Each one is added to the same program and compiled on the record.
  • How to read it. The interactive figures carry the decisions a static listing flattens: hover the core floorplan and the memory-space graph, play the pipeline-map, pass-diff and lifecycle steppers, flip the K-loop figure between stage=1 and stage=2. Provenance. Every pass diff, buffer address, instruction, synchronization flag and dependency edge below is copied from one compile of this program (PyPTO 1f25c43d, PTOAS 0.72) and from the runtime’s own dependency dump; target a2a3sim, backend Ascend 910B. The simulator runs the real kernel (the result matches PyTorch to 7.6×10−5) but is not timing-accurate, so nothing on this page is a cycle count, a bandwidth, or a speedup.

Expressing the "WHAT" in a high-level DSL

PyPTO aims to provide an easy to use high-level DSL where programmers can easily express what they want to compute. The compiler offers automatic tiling, memory allocation, and loop generation based on the high-level description. Users should not care about the low-level details of how the computation is executed on the hardware. Their computational code is easily portable across different hardware targets.

pl.Tensor[[128, 512], pl.FP16]
Shapes are static and part of the type. Every decision below (whether the matmul fits the chip, how to tile it, where each buffer goes) is made at compile time from these numbers. Change a shape and you get a different kernel.
out_dtype=pl.FP32
The cube multiplies FP16 operands and accumulates in FP32. There is no cast op anywhere in the compiled program: the FP32 result is simply the type of the accumulator tile. Asking for an FP16 output would throw away precision the hardware already paid for.
pl.Out[…] and return c
c is a caller-owned buffer the program writes. That marking survives all the way to the runtime, which uses it to decide that c needs no copy to the device before the run and a, b need no copy back after it.
pl.at(level=CORE_GROUP)
“This region is device work.” Everything inside becomes one task for one core group; the function around it becomes host-side orchestration. That split is the first real thing the compiler does (pass 10), and it is why two programs come out.

Why placement must be explicit on this target

On this target, an Ascend 910B of the 220x generation, the execution model is MPMD-like rather than SIMT: the AIC (Cube) and AIV (Vector) cores have independent scalar control paths and load separate instruction streams, and data between them takes the documented global-memory path, ordered by explicit cross-core flags. There is no transparent, GPU-style cache hierarchy a tile can ride, so placement and movement are exposed or inferred, never implicit. Everything on this page is the compiler’s route: one whole-tensor pl.matmul, and the passes choose every tile, buffer and movement you are about to see.

Where the speed comes from: the optimization boundary

The most useful mental model is to separate algorithm selection from backend realization. PyPTO sees the whole matmul and chooses how to decompose it; PTOAS receives that decomposition and turns it into legal, scheduled device code. PTOAS is important to performance, but it is not normally the place where the GEMM algorithm is rediscovered.

The ownership map

LayerMain performance decisionsWhat to look for here
Tensor / block The programmer states the shape, precision, accumulation type and placement scope; CORE_GROUP, SPMD and optional split-K express the amount of parallel work. pl.matmul, out_dtype=FP32, pl.at and, for variants, pl.spmd or pl.parallel.
PyPTO tile compiler AutoTileMatmulL0 chooses legal M/N/K tile sizes, stationarity and a pipeline shape. Later passes lower that plan into memory spaces, ping-pong slots, addresses and the loop nest. tile.extract, pl.pipeline, tile.matmul_acc, InferTileMemorySpace, LowerPipelineLoops and the allocation map.
PTOAS It legalizes and lowers the already-formed tile program: it infers/validates layouts, plans local-memory offsets, lowers pto.tmatmul to native calls, and inserts the waits, flags and barriers needed by the resulting instruction order. Its A5 vector scheduler and optional fusion are backend-specific features, not the Cube GEMM tiler. The .pto module, TMATMUL/TMATMUL_ACC, TLOAD/TSTORE, addresses and synchronization in the emitted C++.
simpler runtime It builds and dispatches the task graph, transfers arguments and coordinates the host side with the compiled AI Core kernel. This is where inter-task dependencies live, not where the inner Cube tile is selected. The orchestration library, dependency edges and the final task launch.

A useful boundary to remember

For this Cube matmul, the consequential GEMM choices are made before PTOAS: whether K is tiled inside a task, whether work is split across cores, which operands stay resident, and whether a second buffer is useful. PTOAS can make that plan fast and correct, but it usually does not change a 128×128×128 plan into a different M/N/K decomposition. Also, a pipeline in the IR is not by itself proof of double buffering: the target and memory planner decide which buffers actually receive multiple slots.

How to read a new compile

  1. After AutoTileMatmulL0, find the K-loop and its extracts/accumulates.
  2. In PTO MLIR, check the mat, left, right and acc spaces, then the pto.tmatmul operands.
  3. In C++, follow TLOAD → TMATMUL[_ACC] → TSTORE, with flags and waits showing the overlap.
  4. When debugging, set PYPTO_LOG_LEVEL=debug: successful compilation logs the absolute artifact directory, so these intermediate files are easy to find.

One extra optimization is worth keeping in mind for chained matmuls: when an intermediate cast can be folded into FixPipe, PyPTO can avoid a Vector round-trip. That transformation is shape- and dtype-dependent, so it is evidence to look for in the generated IR, not a promise that every matmul gets it.

PyPTO and its (simpler) runtime

PyPTO’s passes turn the program into two functions. The device kernel goes through PTOAS to C++ calls into PTO-ISA for the AI Core. The orchestration function becomes C++ against simpler’s API and runs on the AI CPU. simpler brings the two back together at run time. Step through once; each later section expands one box.

STAGE 0 / 7
kernel path · AI Core orchestration path · AI CPU 1pl.matmulDSL 264 passesPyPTO IR 3.ptoPTOAS MLIR 4kernel.cpp+ sync flags 5orchestration.cppcalls simpler’s task API 6compile.o + .so 7simplerrun → cube
Figure 1. The fork after stage 2 is the two-program split made concrete: the compiler emits a device program and a host-side program, and the runtime joins them.
Stage 2 · the territory

An introduction to the Ascend NPU architecture, its memory hierarchy, and compute units

Before watching the compiler place tiles, look at where it can put them. Each on-chip memory space is its own SRAM with its own capacity, its own owning core, and a short list of legal ways in and out. They are separate buffers, not levels of a cache: L0A is not “inside” L1 the way a CPU’s L1 sits inside its L2. The figure below is a redraw of Ascend’s own 220x block diagram (hiascend.com, NPU Architecture Version 220x): Global Memory behind a shared L2 Cache, one AIC core (L1 → L0A/L0B/BT, Cube, L0C, and FixPipe’s own FP Buffer) and one AIV core (Unified Buffer and Vector), each with its DCache, ICache, Scalar unit and per-pipe instruction sequence. The yellow pills are the DMA units that actually carry each transfer, so an arrow’s label is the pipe that moves the data; the DSL call and the ISA instruction behind it are in the hover card. One caveat the diagram leaves out: a 910B die pairs two AIV cores with each AIC, identical to the one drawn here. Hover or tab to any box or arrow for its details.

Figure 2. The Ascend 220x block diagram, redrawn box for box from Ascend’s own documentation, with every box and arrow inspectable on hover. Blue is a storage unit, green a compute unit, yellow a DMA unit, orange an instruction sequence; solid arrows are data flow, dashed orange is instruction flow. A real 910B die pairs two identical AIV cores with each AIC.
two moves the DSL rejects, overlaid in red
Stage 2 · PyPTO passes

PyPTO: Compiler lowering

PyPTO lowers the program through 64 passes over one IR. Tensor ops and tile ops live in the same tree; pass 13 is where one turns into the other. Diffing every pass’s output against its input shows that 18 of them change this program. The table lists all 64: what each pass is for in general, and what it did here or why it had nothing to do. Highlighted rows changed the IR; click one to open its diff in the stepper below.

#PassWhat it doesThis programDiff

Phases are this page’s grouping of the pass manager’s order. Pass numbers are the pass manager’s invocation order, which is what passes_dump/NN_after_<Pass>.py files are named after; the pass documentation keeps its own numbering, which omits utility and verification passes (this page’s pass 20 is doc 19, pass 39 is doc 37). The pass list, order and descriptions come from the compiler’s own pass registry and documentation; the “this program” column comes from diffing consecutive IR dumps of this compile.

PASS 0 / 18

Diffs are the real IR text of consecutive dumps, shown whole: every line of both snapshots, aligned before | after, with removed lines marked −, added lines +, and the unchanged runs between them dimmed. Inside a modified pair only the tokens that actually differ are shaded darker (the word-level diff a code host shows for a one-word edit), so a renamed value or an appended attribute does not read as a rewritten line. Scroll inside the panel for the rest of the file. For readability the dumps were run through ruff format (88 columns) before diffing, tile layout annotations (pl.TileView(…)) are dropped, and pl.const(N, pl.INT64) is printed as N. The +/− counts are added and removed lines.

Who wrote Left and Right?

A pass called InferTileMemorySpace exists, and you would expect it to place the operands. Here it doesn’t. Pass 12 already put A and B in Mat and the result in Acc, and pass 20 wrote target_memory=Left/Right explicitly on its extracts. It has to: it runs two passes before inference, at a point where every tile is 2D but no space has been inferred yet. By pass 22 there is nothing left to infer.

Double buffering is four passes, not one

Four passes cooperate, each with a different job. AutoTileMatmulL0 (19) requests the depth the loop is emitted with: pl.pipeline(…, stage=2), the same 2 that sizes k against half of L0A. LowerPipelineLoops (32) expands it into two tagged copies per iteration (pipeline_membership "0:0" / "0:1") that write into different halves. CanonicalizeIOOrder (33) schedules it: copy two’s extracts move above copy one’s matmul, so the load engine has work queued while the cube computes. MemoryReuse (37) preserves it: the copies’ lifetimes are disjoint, exactly what its coalescing merges, and the membership tag is what vetoes that merge, so two Left and two Right buffers survive to allocation. Drop any one and ping-pong silently degrades: nothing requested, nothing to separate, no overlap, one fused buffer.

Stage 2 · pass 20

Tiling in the compiler

Of the eighteen passes that changed something, one made a choice with alternatives: AutoTileMatmulL0 picked the L0 tile. The operands as written cannot sit in the cube’s operand buffers at all. A is 128 × 512 FP16, which is 128 KiB, and L0A holds 64 KiB. Something has to cut K into L0-sized slices.

m128
n128
k512
Operands
Figure 3. How much of L0A, L0B and L0C one tile of the chosen shape needs, against each buffer’s real capacity.

FP16 operands (2 bytes), FP32 accumulator (4 bytes), Ascend 910B capacities. This is residency arithmetic only. It shows which shapes are legal, not which legal shape is fastest.

Picking the tile size

When both operands stream through the cube (the output-stationary case, where the accumulator stays put and A and B slices flow past it), the chooser plans for two copies of each operand tile from the start. It sizes k against half of L0A: 64 KiB / (2 bytes × 2 stages) = 16 384 elements, so with m = 128 the largest legal k is 128. k = 256 would fit one buffer exactly and leave no room for the second. The pick fills L0A and L0B to exactly 100 %.

A search with a cost model

The chooser enumerates every legal 16-aligned (m, n, k), crossed with which operand stays resident and whether the accumulator is double-buffered, and scores each candidate with a roofline model of cube cycles, operand-load cycles (weighted by the unequal L0A and L0B port bandwidths) and accumulator drain cycles. It returns the minimum. The model is checked against a brute-force re-enumeration of itself, which proves the search is correct. Nobody has timed the pick against alternatives on hardware.

16 is the hardware’s unit, not 128

The chosen 128 × 128 tile is not a hardware parameter. The cube’s unit is the fractal box, and a tile is stored in every on-chip buffer as a grid of them: 16 rows × 32 bytes for an operand tile (16 × 16 FP16 values, 512 B, the 32-byte axis turning with slayout), and a 16 × 16 NZ box of 1024 B for the FP32 accumulator. Those are the fractal=512 and fractal=1024 of the compiled tile types (GetBoxedTileAlignment and kAccFractal in tile_view_semantics.h), and the same two facts the memory-space inspector lists for Mat, Left/Right and Acc. The tiler states its constraints in that unit (m, n and k move in steps of 16), and k need not divide K (the K-loop peels a 16-aligned tail), but K itself must be 16-aligned: a non-aligned K admits no legal tiling, so the pass leaves the op untouched.

Of the three, only k is really chosen: the budget above bounds it at 128 and the cost model lands on the bound, while m and n are not chosen at all: they are this shape’s own M and N. What keeps the output until last is the drain. The model counts drains as ⌈M/m⌉ × ⌈N/n⌉ output tiles, because FixPipe drains one tile at a time; cutting K leaves that count at one (the partial sums stay in the same L0C tile), while cutting M or N multiplies it. A K-block is not free (the model charges a fixed cube start-up per block, and each block is another flag round in the K-loop), but it never buys a drain, and that asymmetry is what orders the two. (This is tiling inside one task; the cross-core split-K of variant 2 is a different thing, done to fill cores, not to fit buffers.) Here n = 128 is 16 N-fractals, a power of two, so the tile also escapes the drain penalty the model charges misaligned widths: a width like 96, whose fractal count 12 is not a power of two, is priced extra serial passes per row. The output is tiled only when it cannot fit L0C at all; a 128 × 128 FP32 accumulator is 64 KiB of the 128 KiB buffer, so it isn’t, and the K-loop figure ends in a single TSTORE.

Stage 2 · passes 38–45

Buffers lifetime

Each panel is one memory space, drawn to that space’s real capacity. Rows are the kernel’s statements in program order, not wall-clock time, so a block’s vertical span is its live range: from the statement that defines it to the last one that reads it. Horizontally, a block sits at the byte offset the allocator assigned; every space starts its own address line at zero, so offset 0 in L0A and offset 0 in Mat are different bytes. That is what “separate buffers, not a hierarchy” means in practice. Hover a block for its bytes and live range, or a statement for the buffers it touches.

Figure 4. Address-by-lifetime map of the seven on-chip buffers the allocator placed: two 128 KiB Mat buffers, ping/pong 32 KiB buffers in L0A and L0B, and one 64 KiB L0C accumulator.

Full, on purpose

L0A and L0B are at 100 % and L0C at 50 %. That is the tile choice from the previous section, showing up as addresses: ping at offset 0, pong at 32 KiB, nothing left over. Mat holds all of A and B (256 KiB of 512) for the whole kernel, because the program fits on the chip. Part II shows what happens when it doesn’t.

Stages 5 & 7 · simpler

What simpler does with the other program

Pass 10 split the program in two, and so far the page has followed the kernel. The other half, the orchestration function, is also compiled. It becomes a small C++ library that runs on the device’s AI CPU, not on the host, and its only job is to tell simpler’s scheduler what tasks exist and which tensors each one reads and writes. This section follows that half from the compiler to the moment the cube starts.

Four passes build the seam

Nothing about the runtime is decided in a separate tool. Four of the eighteen passes that changed the IR exist to shape the orchestration side, and each one maps to one line of the generated file below.

PassIR change hereWhat it becomes in the orchestration file
10 OutlineIncoreScopesthe pl.at body becomes kernel matmul_matmul; the caller keeps one callone task, params_t0, and one submit call
25 ExpandMixedKernelkernel retyped InCore → AIC (pure cube, no vector half)rt_submit_aic_task, not the mixed-cluster form
50 DeriveCallDirectionsarg_directions = [input, input, output_existing]add_input, add_input, add_output
59 MaterializeRuntimeScopesbody wrapped in one pl.scope()one SIMPLER_SCOPE() { … }

The generated orchestration file

This is the whole file, exactly as the compile leaves it on disk, nothing cut. Click a line. The first two headers are stamped on after code generation (click line 4); everything below them is what the code generator wrote.

SIMPLER_SCOPE is not a barrier

It reads like a fork-join block, and it isn’t one. Closing a scope doesn’t wait for the scope’s tasks to finish. It drops the scope’s reference on each task it submitted, which is what allows the runtime to recycle those tasks’ ring slots and intermediate buffers once their consumers are done. Ordering between tasks comes from somewhere else entirely: the tensors each task reads and writes, as the next part shows.

Stage 7 · one run

From the Python call to the cube and back

Three programs, one run

A run involves three separately compiled programs: simpler’s host library, the orchestration library on the AI CPU, and the kernel on the AI Core. Step through one call of matmul(a, b, c).

STEP 0 / 9
Figure 5. Each box is one step; arrows are hand-offs between programs. On a2a3sim the same three programs run as host threads, with the AI Core’s control registers replaced by plain memory.

The dependency graph simpler recorded

With dependency capture on, the runtime writes down every task it was given and every edge it derived. For this program: one task on the cube, three tensor arguments, no edges. The direction tags are the ones pass 47 derived.

How edges get made

There is no edge list in the orchestration code. When a task is submitted, the runtime looks up each of its input tensors in a producer table (the TensorMap) and adds an edge from whichever earlier task last wrote it, then records this task as the producer of its outputs. A task with no producers goes straight onto the ready queue for its core type. So the direction tags from pass 47 aren’t metadata; they are the dependency graph.

That has consequences once there is more than one task. Part II has a case where a documented multi-core pattern quietly runs on one core at a time, and the reason is visible in exactly this kind of dump.

deps T1_0 arg0 INPUT (alloc) arg1 INPUT (alloc) r1t0 ↓0 ↑0 arg2 OUTPUT_EXISTING (alloc)
Figure 6. The same file, drawn by simpler’s own tool: python -m simpler_setup.tools.deps_viewer deps.json --format html --show-tensor-info. One node, its three arguments tagged with the direction pass 47 derived, and the ↓0 ↑0 fan-in / fan-out counts that say in the picture what the empty edges array says in the JSON. On a multi-task kernel the same command draws the real graph (nodes joined by annotated edges), and --edge-mode reduced then marks which of those edges are transitively redundant. The figure is the tool’s own SVG, not a redraw.
Stages 3 & 4 · PTOAS

From tile IR to C++, and where the synchronization comes from

PyPTO -> PTOAS

After PyPTO's compilation pipeline is over, the kernel is printed as an MLIR module in the PTO dialect and handed to PTOAS, the assembler. In that module the memory spaces are still part of each tile’s type (loc=mat, loc=left, loc=acc), and so are the layouts from the memory-space table. What the module does not contain is a single synchronization instruction. PyPTO has no intra-core sync pass. Between that module and the final C++, PTOAS runs well over a dozen more transformations (most with their own CLI-visible pass name, the rest anonymous, the PTO→EmitC instruction selection among them), and it alone inserts every set_flag and wait_flag the kernel needs. Five of those pass invocations change this program; the stepper near the end of this section plays each change back against its input.

Inputs and outputs, precisely

The compiled function is matmul_matmul(a: gm f16[128,512], b: gm f16[512,128], c: gm f32[128,128]). Inside the K-loop below: pto.textract reads one [128,512] or [512,128] Mat tile plus two loop-invariant offsets and writes one fresh 128×128 Left or Right tile; pto.tmatmul/pto.tmatmul.acc read one Left tile and one Right tile (the .acc form also reads the running Acc tile) and write the Acc tile; pto.tstore, once, reads the final Acc tile and writes the [128,128] GM output view.

in: the current .pto loop (selected excerpt)

scf.for %t__tile_l0_ko = %c0_index to %c512_index step %c256_index {
  pto.textract ins(%mem_mat_3_buffer, %c0_index, %t__tile_l0_ko) outs(%mem_left_6_buffer)
  pto.textract ins(%mem_mat_4_buffer, %t__tile_l0_ko, %c0_index) outs(%mem_right_7_buffer)
  %0 = arith.addi %t__tile_l0_ko, %c128_index : index
  pto.textract ins(%mem_mat_3_buffer, %c0_index, %0) outs(%mem_left_8_buffer)
  %1 = arith.addi %t__tile_l0_ko, %c128_index : index
  pto.textract ins(%mem_mat_4_buffer, %1, %c0_index) outs(%mem_right_9_buffer)
  %2 = arith.cmpi eq, %t__tile_l0_ko, %c0_index : index
  scf.if %2 { pto.tmatmul ... }
       else { pto.tmatmul.acc ... }
  pto.tmatmul.acc ins(%mem_acc_5_buffer, %mem_left_8_buffer, %mem_right_9_buffer) outs(%mem_acc_5_buffer)
}
// no set_flag / wait_flag / barrier in the PyPTO .pto module; PTOAS adds them later
// full types and allocations are in matmul_autotile_matmul_matmul.pto

out: the current emitted C++ excerpt

TLOAD(v36, v53);
TLOAD(v38, v57);
wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
for (int64_t v58 = v10; v58 < v5; v58 += v11) {
  wait_flag(PIPE_M, PIPE_MTE1, EVENT_ID0);  // ping free?
  pipe_barrier(PIPE_MTE1);
  TEXTRACT(v42, v36, v59, v60);
  TEXTRACT(v44, v38, v61, v62);
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);  // ping full
  wait_flag(PIPE_M, PIPE_MTE1, EVENT_ID1);  // pong free?
  TEXTRACT(v46, v36, v64, v65);
  TEXTRACT(v48, v38, v66, v67);
  set_flag(PIPE_MTE1, PIPE_M, EVENT_ID1);  // pong full
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
  if (v58 == v10) {
    pipe_barrier(PIPE_M);
    TMATMUL(v40, v42, v44);
  } else {
    pipe_barrier(PIPE_M);
    TMATMUL_ACC(v40, v40, v42, v44);
  };
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID0);
  pipe_barrier(PIPE_M);
  wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID1);
  TMATMUL_ACC(v40, v40, v46, v48);
  set_flag(PIPE_M, PIPE_MTE1, EVENT_ID1);
}

Both listings are selected excerpts from this exact compile. Left: PyPTO’s own SSA names, unshortened; it already names things meaningfully. Right: PTOAS numbers every value (v36, v42, …) with no names at all. The one thing still cut is the GM tensor-view prologue before the loop (three tensors’ worth of pto::Shape/pto::Stride/GlobalTensor construction). The file itself, every line, is entry 6 of the emission stepper below, and both files are checked in at pypto-3.0-notes/matmul/matmul_autotile_matmul_matmul.{pto,cpp}.

Read the event IDs as buffer names

EVENT_ID0 is the ping buffer pair and EVENT_ID1 the pong pair. Each pair has two flags going in opposite directions. MTE1→M says “this buffer is full, compute may read it” (read-after-write). M→MTE1 says “compute has finished with this buffer, the DMA may overwrite it” (write-after-read). Before the loop, PTOAS pre-sets both M→MTE1 flags, so the first two extracts don’t wait for a matmul that never happened; after the loop it drains them. This is literally what Event ID Allocation adds below, not a paraphrase.

InsertSync is one pass, six sub-phases

-pto-insert-sync (PTOInsertSyncPass) is a single MLIR pass with six internal sub-phases, each its own class, each traced below from a real run of this compile (--pto-insert-sync-debug=2). The pass is off in the ptoas CLI by default, but PyPTO turns it on for every kernel it compiles (--enable-insert-sync in pto_backend.py), and this page’s reproduce script passes it explicitly. The other sync strategies (--enable-bufid_sync, A5-only; --enable-inject-barrier-all-sync, barrier-before-everything) are not what this page traces.

Sub-phaseClassObserved on this compile
1. TranslatorPTOIRTranslatorbuilds a 17-node pipe-tagged IR from the loop; 0 sync ops yet
2. AnalysisInsertSyncAnalysisinserts 24 ops from pairwise hazard analysis (9 set_flag, 9 wait_flag, 6 pipe_barrier)
3. Sync MotionMoveSyncStatekeeps 24 ops while moving them to cheaper positions around the two-copy loop
4. Remove Redundant SyncRemoveRedundantSyncdrops 6 redundant flags, leaving 18 active ops: 6 set_flag, 6 wait_flag, and 6 barriers
5. Event ID AllocationSyncEventIdAllocationassigns EVENT_ID0/EVENT_ID1 and adds the two pre-set and two drain operations: 18 → 22 active ops
6. CodegenSyncCodegenwrites the final 8 set_flag + 8 wait_flag + 6 barrier operations into the module. The EmitC stage maps each pto.barrier to the C++ pipe_barrier call.

Every PTOAS pass between PyPTO pass 64 and the file on disk

PassWhat it does hereDiff
main pass pipeline · 18 of the 21 invocations
{anonymous}::SerialFrontendPipeLoweringPassthe wrapper that runs the frontend pipe-lowering sub-pipeline (the two rows below) over the function; nothing module-level changes because of itno-op
  ↳ pto-assign-default-frontend-pipe-idPTOAssignDefaultFrontendPipeIdgives ops that lack one their default frontend pipe id; every op here already carries it, so the dump is byte-identicalno-op
  ↳ pto-lower-frontend-pipe-opsPTOLowerFrontendPipeOpslowers frontend pipe-scoping ops; byte-identical hereno-op
pto-infer-validate-pipe-initPTOInferValidatePipeInitvalidates and infers pipe-init metadata; byte-identical hereno-op
pto-lowering-sync-to-pipePTOLoweringSyncToPipelowers explicit record_event/wait_event to set_flag/wait_flag; byte-identical, this kernel contains no manual syncno-op
pto-infer-layoutInferPTOLayoutstamps {layout = #pto.layout<nd>} on the two tloads and the tstore; every other layout field was already in the tile types
pto-materialize-implicit-tmp{anonymous}::PTOMaterializeImplicitTmpPassgives an implicit temporary a real buffer; byte-identical here: PyPTO pass 39 already left all seven tiles explicit, addresses and allno-op
pto-rematerialize-fixpipe-vector-quantPTORematerializeFixpipeVectorQuantfixpipe/vector-quant rematerialization; byte-identical, nothing is quantized in this kernelno-op
pto-resolve-reserved-buffersPTOResolveReservedBuffersresolves reserved cross-core buffers; byte-identical, this is a single-core kernelno-op
pto-remove-identity-tmovPTORemoveIdentityTMovremoves same-SSA tmov copies; byte-identical, the kernel has noneno-op
pto-insert-syncPTOInsertSyncadds the kernel’s synchronization in 21 new lines: 8 set_flag, 8 wait_flag, 5 pipe_barrier in the pass diff
pto-resolve-buffer-selectPTOResolveBufferSelectmaterializes multi_tile_get selects; byte-identical, no pl.MemRef(slots=N) in this kernelno-op
{anonymous}::NarrowUnusedMultiResultProvenancePassnarrows unused multi-result provenance; byte-identicalno-op
cseCSEfolds redundant address computation and renumbers the values after it
pto-inline-backend-helpersPTOInlineBackendHelpersinlines the EmitC helper functions into their call sites; byte-identical hereno-op
{anonymous}::NarrowUnusedMultiResultProvenancePassthe second run of the same pass; byte-identicalno-op
canonicalizeCanonicalizergeneric MLIR canonicalization; byte-identicalno-op
cseCSEthe second run; byte-identicalno-op
EmitC backend pipeline · a second pass manager, built at emission time
{anonymous}::EmitPTOManualPassthe PTO→EmitC instruction selection: pto.tload/textract/tmatmul/tstore/set_flag become emitc.call_opaque calls into PTO-ISA and the tile types vanish
{anonymous}::FormEmitCExpressionsCompatPassfolds the flat EmitC chain into emitc.expression regions so expression boundaries are explicit; the same program, written longer
cseCSEthe third run; byte-identicalno-op
STEP 0 / 5

The steps are the real dump text of this compile: ptoas run on the .pto above with --enable-insert-sync --pto-level=level3 --mlir-print-ir-before-all --mlir-print-ir-after-all (PTOAS 0.72, the build this page pins). Of the 21 pass invocations, five leave the printer’s text different; those five, after the module as the tool loads it, are the six steps. Step 0 is not a diff: it is the state every pass starts from, and the printer rewrites almost every line of the file on the way in, so a line diff there would say nothing. A row marked no-op above is one where the printer wrote the same text before and after the pass. Each step shows the unit the pass ran on, whole: the module for module-scope passes (wrapper included), the function alone for the function-scope ones. Nothing is normalized or reflowed, which is why the value numbering shifts from step to step; the only reflow on screen is wrapping, so a statement too long for the panel (a full-type textract is ~370 characters) continues on the next line, indented under its own first character. The dump has it on one line. The last two steps speak EmitC rather than PTO: they are the first two passes of the EmitC backend pipeline, the second pass manager the driver builds once the main pipeline is done. The step from there to the .cpp on disk is four more stages, and none of them emits a dump; the note below follows them. Inside a modified pair the tokens that differ are shaded darker, word-level, as in the other steppers. Click a highlighted row to open its diff here.

Where the C++ text comes from

The dumps stop at the end of the EmitC backend pipeline. Between that state and the .cpp on disk there are four stages, none of which emits an IR dump, all in PTOAS/tools/ptoas/ptoas_pipeline.cpp unless noted:

  1. runEmitCPreparationPipeline, the second pass manager (EmitPTOManualPass → FormEmitCExpressionsCompatPass → CSE): the last three rows of the table above are this pipeline, not the main pipeline’s tail.
  2. a run of direct IR fix-ups (functions, not passes, so they dump nothing): dropEmptyEmitCExpressions, materializeControlFlowOperands, normalizeEmitCIntegerAttrsForCppEmission, reorderEmitCFunctions, annotateEmitCProvenanceHints, …
  3. emitc::translateToCpp, MLIR’s EmitC→C++ translator, called with declareVariablesAtTop from shouldDeclareVariablesAtTop(module). This is the call that turns emitc.call_opaque "TEXTRACT" into TEXTRACT(…), emitc.verbatim into raw text, and each !emitc.opaque type into the C++ type name it spells.
  4. runCppPostRewrites (tools/ptoas/ptoas_cpp_rewrite.cpp), a fixed-order list of text rewrites on the generated C++: call-site markers, the scalar-GM store flush, scalar-constant and hoisted-GlobalTensor declarations, malformed-verbatim repair, name hints. Only after this does the text become the file PTOAS writes.

So the C++ listing above is not printed from the last dump’s state: it is that state after one translation and one text-rewrite chain. --emit-pto-ir stops before the translation and prints the lowered PTO IR instead; --pto-backend=vpto bypasses this path entirely. The stepper below follows those four stages and ends with the file itself, whole, so the chain can be checked end to end: each of the two dumpable stages appears twice, once as the detail that maps it and once whole.

STAGE 0 / 5

Six entries. Entries 1 and 3 carry the detail of the two dumpable stages (the op-by-op mapping and the fix‑up call list), and entries 2 and 4 carry each of those stages whole: the module before and after EmitPTOManualPass, then that module and the C++ it becomes. Those two are stacked, not paired row for row, because the printer and the translator rewrite almost every line (stage 1 replaces 128 of 191 rows, stage 3 adds 229 and removes 134, with eight and two lines left in place), so a paired diff would shade nearly everything and explain nothing. The mapping in entry 1 is what pairs. Inside a pair, the tokens that differ are shaded darker, as in the other steppers. Entries 3 and 5 are stages that leave no dump and are therefore shown only from the side that exists: the source that runs them, or the text they produced. Entry 6 is not a stage: it is the file those four write, the whole function, verbatim, with the lines the earlier entries took as windows tinted inside it and the ends of the pipe figure’s abridged listing labelled, so every excerpt above can be located inside the whole. Every line is cut verbatim from this run’s own artifacts (the dump log, the emitted .cpp, and ptoas_pipeline.cpp) by matmul/gen_primer_emission_steps.py, which re-checks each cut before publication.

InstructionPipeHardwareIssued
TLOAD × 2MTE2DMA: global memory → L1once, before the loop
TEXTRACT × 4 per tripMTE1DMA: L1 → L0A / L0B8 in total
TMATMUL / TMATMUL_ACCMthe cube1 + 3 in total
TSTOREFIXFixPipe: L0C → global memoryonce, after the loop
Stage 7 · on the core

What stage=2 actually buys

Two loop iterations on four pipes

Each pipe (the two DMA engines, the cube, the FixPipe) executes its own instructions in order, and the pipes run in parallel with each other. The only thing that holds one pipe back for another is a flag. This figure places every instruction of the kernel at the earliest step its flags allow, assuming every instruction takes one step. Switch to stage=1 to see the same kernel with a single buffer pair. The listing beside the figure is an abridged transcription of the current emitted instruction stream: 31 rows for the 60 lines from the first TLOAD through the final TSTORE (lines 148–207). The complete function is shown in entry 6 of the emission stepper above, and the omitted bookkeeping is itemised below.

A box is one dependency-graph step, and now also one real instruction: MTE1 → L0A and MTE1 → L0B are two rows, not one, because the A-tile and B-tile fetches are two separate TEXTRACT calls on the same pipe, in program order: A first (it waits for the ping/pong slot to free), then B immediately behind it (its completion is what actually posts the flag the cube waits on). “matmul 0” still has no such split: one step is exactly one TMATMUL/TMATMUL_ACC call.

Figure 7. The compiled K-loop’s dependency-step timeline: four pipe lanes, ping/pong buffer occupancy, and the real set_flag/wait_flag/pipe_barrier instructions PTOAS emits.

Instruction stream · stage=2, as compiled

Good to know. The 29 omitted lines are 10 // pto: <value> provenance comments, 2 stray ;, 8 Shape/Stride/GlobalTensor views, 8 static_cast<uint16_t> operand temporaries and 1 loop-index computation. The listing keeps the loads, extracts, flags, barriers, matmuls, loop control and store that the figure schedules; the whole function in entry 6 keeps every line. The checker beside the source artifacts fails if either the selected rows or this accounting drifts.

read-after-write: data ready write-after-read: buffer free again accumulator chain ping (offset 0, EVENT_ID0) pong (offset 32 KiB, EVENT_ID1)

What the figure shows

With two buffer pairs, the extract for iteration i+1 goes into the other half while matmul i is still reading the first, so MTE1 and the cube stay busy on the same steps. The dashed violet edges are the new constraint double buffering introduces: iteration i+2 reuses iteration i’s half, so its extract waits for matmul i to finish. The amber chain is the one dependency no buffering can remove: all four matmuls accumulate into the same L0C tile, in order.

What it does not show

Positions are dependency steps, not time. On real hardware a 128 × 128 × 128 FP16 matmul and a 32 KiB L1→L0 copy take different numbers of cycles, and whether MTE1 or the cube is the bottleneck depends on those numbers. The simulator this page used runs the kernel correctly but does not model timing. The step counts are a statement about the kernel’s dependency structure, which is exactly what the flags encode.


Part II

The extra mile, and who walks it

Part I’s kernel is correct, tiled for L0, double-buffered and synchronized, and it runs on one of the 910B’s 24 cube cores with operands that fit on the chip. Everything an HPC matmul adds beyond that has an owner. Here is who owns what, checked against the compiler, the runtime and the assembler source, and then each missing piece added to the same program and compiled.

OptimizationWho does itHowIn Part I?
L0 tiling: K-loop, and M/N split when the output overflows L0CPyPTO pass 20automatic cost-model searchyes: k = 128
L0A/L0B ping-pong inside that loopPyPTO passes 20, 33, 34, 42automatic once the matmul is tiledyes: 2 × 2 buffers
Intra-core synchronization and event IDsPTOAS sync passautomatic always enabled by PyPTOyes: 2 events
Fractal layout for tiles in Mat / L0PyPTO tile conversionautomatic converted during the loadyes
Accumulator (L0C) double bufferingPyPTO passes 20, 33, 34, 42automatic or opt-in automatic under memory_planner=PTOAS and DSA_RP; a marked-experimental flag under the default PyPTO plannerno: one Acc tile is sufficient for this compile
Using more than one core: M/N split or split-Kyou, in the DSL; simpler dispatchesone more line pl.spmd(n)no: variants 1, 2
L1 tiling: operands bigger than the 512 KiB Matnobodycompile error you block K yourselfno: variant 3
Double buffering of your own loops (GM→L1)you; PyPTO lowers itone more line pl.pipeline(…, stage=2)no: variant 3
Weights stored in the hardware layoutyou; PyPTO pass 17 plans for ita type annotation pl.NZno: variant 4
Epilogue fusion (bias, ReLU, quantize on writeback)you, at tile leveltile-level store optionsno
Overlap between taskssimplerautomatic for tasks you created, ordered by their tensorsone task: nothing to overlap

What those four passes each do

The ping-pong row names four passes because no single pass owns this optimization. Reading them in order is the whole mechanism, and the fourth one is the one that people usually assume is the second.

  • 20 AutoTileMatmulL0 requests it. The tiled K-loop is emitted as ForKind::Pipeline with pipeline_stages=2. That 2 is the operand depth (L0A and L0B), not the accumulator. It is also why k = 128 and not 512: the chooser sizes the tile against half of each operand buffer, which is the arithmetic in the previous section.
  • 33 LowerPipelineLoops replicates the body. Two copies per outer iteration, each with fresh def-vars, and each clone's tile-producing call tagged with a pipeline_membership (group, stage) pair.
  • 34 CanonicalizeIOOrder shapes the schedule: scalars first, then the loads clustered ahead of compute, so the load engine runs ahead of the cube. It only reorders statements; it does not create the buffer separation.
  • 42 MemoryReuse is where the separation is actually enforced. The two clones are sequential in program order, so their tiles have disjoint lifetimes, exactly the condition the reuse packer coalesces on. Left alone it would fuse the ping-pong back into a single buffer. The pipeline_membership tag is what forbids that, and it does so only as deep as the memory space can afford.

The accumulator is deliberately left untagged: it is written by the serialized cube, one tile's MAD retiring before the next begins, so a single L0C buffer is already correct. That is why this page shows one Acc @ 0 rather than a pair.

The pattern

The compiler automates the single-core, fits-on-chip story completely. Everything about many cores and data bigger than the chip is written by the programmer today, as small DSL additions to the same program. That split is deliberate: the assembler’s design notes rule out inferring multi-buffering on its own, the accumulator double buffer is experimental under the default planner (it is automatic under memory_planner=PTOAS and DSA_RP, and needs the full-K emitter either way), and nothing in the compiler splits one device region across cores. Production kernels built on PyPTO write all four variants below by hand.

The switch this page did not have to touch

Everything above was compiled with the default memory_planner=PYPTO. That setting decides who owns on-chip memory, and it is the one knob that can also move the tile the chooser picks.

PYPTO (default)PTOAS
Addressesbaked addr on every tile; ptoas runs at --pto-level=level3addr-less tiles, placed by ptoas at --pto-level=level2
Reuse and placementPyPTO MemoryReuse + AllocateMemoryAddrPTOAS PlanMemory, with both of those skipped
L0C double bufferopt-in flagautomatic, if the chooser picks it

It can change the tile. dbC = 2 requires the full-K emitter (l0_tile_chooser.cpp sets require_full_k = !is_os || r.dbc), and halves the L0C accumulator budget (l0c_bytes / (bytes_c × 2)), so making it available re-ranks the candidate shapes. On this program it changed nothing: the same 128 × 128, k = 128, one accumulator, at identical offsets, under all three configurations. That is the useful part: Part I is a default-planner picture, and it holds because k = 128 split-K won on modelled cost, not because a full-K dbC = 2 candidate was illegal.

Variant 1 · more cores, split the output

Four blocks with pl.spmd

# M = 512, N = 128, K = 512
@pl.jit
def matmul_spmd(a: pl.Tensor[[512, 512], pl.FP16],
                b: pl.Tensor[[512, 128], pl.FP16],
                c: pl.Out[pl.Tensor[[512, 128], pl.FP32]]):
    for i in pl.spmd(4):          # i = block index
        r0 = i * 128
        part = pl.matmul(a[r0:r0 + 128, :], b,
                         out_dtype=pl.FP32)
        c = pl.assemble(c, part, [r0, 0])
    return c

Runs. Max error vs PyTorch 6.9×10−5.

The orchestration file still submits one task, now with launch_spec.set_block_num(4). The runtime expands it into four blocks at dispatch, and each block runs Part I’s kernel on its own 128 rows of A: same K-loop, same ping-pong, same flags. The dependency dump shows one task, block_num 4, and no edges.

Each block loads all of B. For this shape that is the right trade (B is 128 KiB); for a wide B you would split N as well, and that choice is yours to write.

Variant 2 · more cores, split the reduction

Split-K, and the version that runs on one core at a time

When M and N are too small to feed many cores, split K instead: each core reduces a slice of K and adds its partial result into c with an atomic add. There are two obvious ways to write it. They compile, run and give the same answer. Only one of them is parallel.

# A: a loop of device regions  (M=N=128, K=2048)
with pl.at(level=CORE_GROUP, name_hint="zero_init"):
    c[:] = pl.full([128, 128], dtype=pl.FP32, value=0.0)
for ks in pl.parallel(4):
    with pl.at(level=CORE_GROUP, name_hint="split_k"):
        k0 = ks * 512
        p = pl.matmul(a[:, k0:k0 + 512], b[k0:k0 + 512, :],
                      out_dtype=pl.FP32)
        c = pl.assemble(c, p, [0, 0],
                        atomic=pl.AtomicType.Add)
# B: one SPMD region  (same shapes)
with pl.at(level=CORE_GROUP, name_hint="zero_init"):
    c[:] = pl.full([128, 128], dtype=pl.FP32, value=0.0)
for ks in pl.spmd(4):
    k0 = ks * 512
    p = pl.matmul(a[:, k0:k0 + 512], b[k0:k0 + 512, :],
                  out_dtype=pl.FP32)
    c = pl.assemble(c, p, [0, 0],
                    atomic=pl.AtomicType.Add)

Serialized. Correct result (8.8×10−5), one core at a time.

zero_init (AIV)
 └─wait─▶ split_k[0]
           └─wait─▶ split_k[1]
                     └─wait─▶ split_k[2]
                               └─wait─▶ split_k[3]

Five tasks, and the runtime chained all of them. Each split_k task declares c as read-write (the atomic add reads it), so the TensorMap makes each one wait for the previous writer of c. The hardware atomic in the kernel is real; it just never gets a concurrent writer.

Concurrent. Correct result (9.2×10−5), four cores at once.

zero_init (AIV)
 └─wait─▶ split_k   one task, 4 blocks
            ├─ block 0: K[0:512]
            ├─ block 1: K[512:1024]
            ├─ block 2: K[1024:1536]
            └─ block 3: K[1536:2048]

Two tasks and one edge. The four slices are blocks of a single task, so there is no chain between them to derive: they start together and race on the atomic add in hardware, which is the point.

The rule behind it

simpler orders tasks by the tensors they declare, and an atomic add is still a read-modify-write of the whole output. Tasks that accumulate into the same tensor must therefore be blocks of one SPMD task to run concurrently. Form A is the one a reader coming from GPU programming would write first, and PyPTO’s own matmul tutorial teaches it. The dependency dump is the quickest way to catch this: count the edges.

Variant 3 · bigger than the chip

When the operands don’t fit in L1

Make K = 4096. A and B are now 1 MiB each, and Mat holds 512 KiB.

# the Part I program, K = 4096
with pl.at(level=pl.Level.CORE_GROUP):
    c[:] = pl.matmul(a, b, out_dtype=pl.FP32)

Compile error at memory allocation (pass 41):
Mat buffer usage (2097152 bytes) exceeds platform limit (524288 bytes)

Pass 12 loads each operand whole into Mat, as in Part I, and nothing between pass 13 and pass 41 tiles at the L1 level. The L0 tiler would happily split K for the cube, but it runs on operands that must already be in Mat.

# block K by hand, 256 at a time
with pl.at(level=pl.Level.CORE_GROUP):
    acc = pl.matmul(a[:, 0:256], b[0:256, :],
                    out_dtype=pl.FP32)
    for k in pl.pipeline(1, 16, stage=2):
        k0 = k * 256
        acc = pl.matmul_acc(acc, a[:, k0:k0 + 256],
                            b[k0:k0 + 256, :])
    c[:] = acc

Runs. Max error 5.2×10−4 (FP16 inputs, K = 4096).

Your stage=2 gives Mat four 64 KiB buffers: two chunks of A and B in flight, so the next chunk loads from global memory while the current one computes. Inside each chunk the compiler still tiles for L0 exactly as in Part I.

The compiler tells you what it couldn’t do

Nesting has a cost. Your loop now contains three copies of the compiler’s own L0 K-loop (a first chunk and two pipeline copies), each asking for its own ping-pong pair, and L0A has room for two buffers in total. The compile succeeds and prints a performance hint for each: requested depth 2 … in Left, but only 1 of 2 buffers fit … stages 1 apart share storage and serialize. The fix is a smaller chunk or a different split, and it is yours to choose.

The other way to say the same thing

pl.pipeline(…, stage=2) is not the only spelling of ping-pong. pl.MemRef("name", slots=N) declares one allocation with N uniform slots and picks one per iteration with an ordinary index expression, e.g. pl.MemRef("ub", slots=2)[i % 2]. Under memory_planner=PTOAS that lowers to a single ptoas pto.alloc_multi_tile region read through pto.multi_tile_get, and ptoas derives the per-slot synchronization from the slot index itself. Where pl.pipeline restructures the loop into a schedule, slots only remove the same-buffer hazard: the loop stays sequential.

Worth knowing why the compiler’s own matmul pipeline never uses that form. The slots path requires the loop’s step to be 1 with a start that is a multiple of N, so that ptoas can match the slot index as a plain iv % N; the tiled K-loop is emitted with step = k, so it is declined and replicated instead. And the shape where two slots are live at once is refused at codegen rather than miscompiled, because the pinned ptoas still mis-synchronizes the prefetch variant of it (hw-native-sys/PTOAS#1519: fixed in 0.63, regressed in 0.64–0.65). One slot live per iteration is the shape the region form exists for.

Variant 4 · the layout of the weights

Store B the way the cube reads it

@pl.jit
def matmul_nz(a: pl.Tensor[[128, 512], pl.FP16],
              b: pl.Tensor[[512, 128], pl.FP16, pl.NZ],
              c: pl.Out[pl.Tensor[[128, 128], pl.FP32]]):
    with pl.at(level=pl.Level.CORE_GROUP):
        c[:] = pl.matmul(a, b, out_dtype=pl.FP32)
    return c

# host side: the bytes must already be in NZ order
def to_nz(x):                    # [R, C] FP16
    r, cols = x.shape
    return (x.reshape(r // 16, 16, cols // 16, 16)
             .permute(2, 0, 1, 3).contiguous()
             .reshape(r, cols))

Runs. Max error 7.6×10−5.

Every load into Mat converts row-major data into the fractal layout on the fly. For weights that are loaded many times that conversion is repeated work. pl.NZ is not a request to convert: it is a promise that the bytes in global memory are already in fractal order.

Pass 16 (BlockNzTensorViews, a no-op in Part I) now does something. It rewrites B’s logical [512, 128] into the blocked [1, 8, 32, 16, 16] the ISA describes (8 column blocks of 16, 32 row fractals of 16 × 16), and the load of B becomes NZ→NZ (pto::Layout::NZ in the emitted C++). A stays row-major. Keeping the promise is the caller’s job; the reorder is the six lines on the left.

A summary of what we have seen

one line

Everything below it was derived

Static shapes, an output dtype, an output marker and a device region were enough for the stack to produce a tiled, double-buffered, synchronized cube kernel and the host code that launches it.

PyPTO

18 of 64 passes did the work

One pass made a real choice (the L0 tile, from a cost model sized for two buffers). The rest are mechanical: outline, lower to tiles, replicate for ping-pong, reorder loads first, allocate, tag argument directions.

simpler

The dependency graph is the argument tags

The orchestration library runs on the AI CPU. Edges come from input/output tags through a producer table, and a scope manages buffer lifetime, not ordering.

PTOAS

Every flag is inserted by the assembler

Two event IDs, one per buffer half, each with a data-ready and a buffer-free direction.

the extra mile

Many cores and big data are yours to write

One more line each: pl.spmd for cores, pl.pipeline over your own K-blocks for L1, pl.NZ for weights. And accumulate across cores inside one SPMD task, or the runtime will serialize you.

Do this yourself

Everything above came from compiling and running the programs on this page with IR dumps and dependency capture turned on, in the PyPTO simulation container, on a laptop.

Compile with per-pass dumps, run with dependency capture

import os

import torch
import pypto.language as pl
from pypto.ir.pass_manager import PassDumpLevel
from pypto.runtime import RunConfig

# read once per process: set it before compiling
os.environ["PYPTO_PROG_BUILD_DIR"] = "out"

# … the @pl.jit matmul from the top of the page …

a = torch.randn(128, 512, dtype=torch.float16)
b = torch.randn(512, 128, dtype=torch.float16)
c = torch.zeros(128, 128, dtype=torch.float32)

dump = RunConfig(dump_passes=PassDumpLevel.EXPLICIT)
matmul.compile(a, b, c, config=dump)
matmul(a, b, c, config=RunConfig(enable_dep_gen=True))

ref = a.float() @ b.float()
assert torch.allclose(c, ref, rtol=1e-2, atol=1e-2)

Compare against the FP32 product of the FP16 inputs, at an FP16-sized tolerance.

Know what landed where

Under out/ you get one directory for the compile and one for the run. The compile directory holds passes_dump/ (65 files: the parsed input plus one per pass), ptoas/*.pto and the emitted kernel C++, orchestration/*.cpp, and report/perf_hints.log. The run directory holds dfx_outputs/deps.json.

Find the passes that matter

cd out/*/passes_dump
prev=; for f in $(ls | sort); do
  [ -n "$prev" ] && ! cmp -s "$prev" "$f" && echo "$f"
  prev=$f
done

Eighteen names come out for the program on this page. Then read each diff; the table above tells you what to look for.

Read the flags, then count the edges

grep -nE 'set_flag|wait_flag|TEXTRACT|TMATMUL' ptoas/*.cpp gives the instruction stream of the K-loop figure. For any multi-task program, count the edges in deps.json before you believe it runs in parallel.

Push the shape and watch the compiler change its mind

Set M = N = 256 and the output cannot fit L0C, so pass 20 has to split that axis as well as K. Set K = 4096 and allocation fails outright, as in variant 3: nothing between the tensor loads and the L0 tiles will block K for L1. The residency explorer above predicts the L0 side of both before you compile.

The code, and where it lives

The compiler is PyPTO (documentation); the runtime it lowers for is simpler (documentation); the assembler that turns the PTO module into C++ is PTOAS, and the calls it emits go into PTO-ISA. Kernels and models built on them (including the Qwen3-14B decode kernel whose split-K shape variant 2 follows) are in pypto-lib; the project’s documentation hub is pypto.ai.