PyPTO (say “pai p-t-o”) is a high-performance programming framework for Ascend AI accelerators: you write Python tensor code, and the compiler lowers it to hardware instructions. It is tile-based: every computation is a tile, a hardware-aware data block placed in the memory hierarchy and scheduled across cores, and a multi-level IR (tensor → tile → block → execution graph) reaches PTO instructions. Tensor-, tile- and block-level APIs serve algorithm, performance and system developers.
@pl.jit def matmul(a: pl.Tensor[[128, 512], pl.FP16], b: pl.Tensor[[512, 128], pl.FP16], c: pl.Out[pl.Tensor[[128, 128], pl.FP32]]): with pl.at(level=pl.Level.CORE_GROUP): c[:] = pl.matmul(a, b, out_dtype=pl.FP32) return c
This page follows one matmul down that path: every tile, buffer, loop and flag below was placed by PyPTO, simpler and PTOAS.
What this page walks through
pl.matmul;
PyPTO decides tile shapes, buffer placement and loop structure, PTOAS selects instructions and
derives the intra-core synchronization, simpler builds the task graph and dispatches it. Each
layer is shown with the artifact that proves its contribution: an IR diff, an address, a
flag, a dependency edge.set_flag/wait_flag in the kernel (21 pass invocations,
5 of which change the IR);stage=1 and
stage=2.
Provenance. Every pass diff, buffer address, instruction,
synchronization flag and dependency edge below is copied from one compile of this program
(PyPTO 1f25c43d,
PTOAS 0.72) and from the runtime’s own
dependency dump; target a2a3sim, backend Ascend 910B. The simulator runs the real
kernel (the result matches PyTorch to 7.6×10−5) but is not
timing-accurate, so nothing on this page is a cycle count, a bandwidth, or a speedup.PyPTO aims to provide an easy to use high-level DSL where programmers can easily express what they want to compute. The compiler offers automatic tiling, memory allocation, and loop generation based on the high-level description. Users should not care about the low-level details of how the computation is executed on the hardware. Their computational code is easily portable across different hardware targets.
c is a caller-owned buffer the program writes. That marking survives all the way to
the runtime, which uses it to decide that c needs no copy to the device before the run
and a, b need no copy back after it.Why placement must be explicit on this target
On this target, an Ascend 910B of the 220x generation, the execution model is
MPMD-like rather than SIMT: the AIC (Cube) and AIV (Vector) cores have independent
scalar control paths and load separate instruction streams, and data between them takes the
documented global-memory path, ordered by explicit cross-core flags. There is no transparent,
GPU-style cache hierarchy a tile can ride, so placement and movement are exposed or inferred,
never implicit. Everything on this page is the compiler’s route: one whole-tensor
pl.matmul, and the passes choose every tile, buffer and movement you are about to see.
The most useful mental model is to separate algorithm selection from backend realization. PyPTO sees the whole matmul and chooses how to decompose it; PTOAS receives that decomposition and turns it into legal, scheduled device code. PTOAS is important to performance, but it is not normally the place where the GEMM algorithm is rediscovered.
The ownership map
| Layer | Main performance decisions | What to look for here |
|---|---|---|
| Tensor / block | The programmer states the shape, precision, accumulation type and placement scope;
CORE_GROUP, SPMD and optional split-K express the amount of parallel work. |
pl.matmul, out_dtype=FP32, pl.at and, for variants,
pl.spmd or pl.parallel. |
| PyPTO tile compiler | AutoTileMatmulL0 chooses legal M/N/K tile sizes, stationarity and a
pipeline shape. Later passes lower that plan into memory spaces, ping-pong slots,
addresses and the loop nest. |
tile.extract, pl.pipeline, tile.matmul_acc,
InferTileMemorySpace, LowerPipelineLoops and the allocation map. |
| PTOAS | It legalizes and lowers the already-formed tile program: it infers/validates layouts,
plans local-memory offsets, lowers pto.tmatmul to native calls, and inserts
the waits, flags and barriers needed by the resulting instruction order. Its A5 vector
scheduler and optional fusion are backend-specific features, not the Cube GEMM tiler. |
The .pto module, TMATMUL/TMATMUL_ACC,
TLOAD/TSTORE, addresses and synchronization in the emitted C++. |
| simpler runtime | It builds and dispatches the task graph, transfers arguments and coordinates the host side with the compiled AI Core kernel. This is where inter-task dependencies live, not where the inner Cube tile is selected. | The orchestration library, dependency edges and the final task launch. |
A useful boundary to remember
For this Cube matmul, the consequential GEMM choices are made before PTOAS: whether K is tiled inside a task, whether work is split across cores, which operands stay resident, and whether a second buffer is useful. PTOAS can make that plan fast and correct, but it usually does not change a 128×128×128 plan into a different M/N/K decomposition. Also, a pipeline in the IR is not by itself proof of double buffering: the target and memory planner decide which buffers actually receive multiple slots.
How to read a new compile
AutoTileMatmulL0, find the K-loop and its extracts/accumulates.mat, left, right and
acc spaces, then the pto.tmatmul operands.TLOAD → TMATMUL[_ACC] →
TSTORE, with flags and waits showing the overlap.PYPTO_LOG_LEVEL=debug: successful compilation logs the
absolute artifact directory, so these intermediate files are easy to find.One extra optimization is worth keeping in mind for chained matmuls: when an intermediate cast can be folded into FixPipe, PyPTO can avoid a Vector round-trip. That transformation is shape- and dtype-dependent, so it is evidence to look for in the generated IR, not a promise that every matmul gets it.
PyPTO’s passes turn the program into two functions. The device kernel goes through PTOAS to C++ calls into PTO-ISA for the AI Core. The orchestration function becomes C++ against simpler’s API and runs on the AI CPU. simpler brings the two back together at run time. Step through once; each later section expands one box.
Before watching the compiler place tiles, look at where it can put them. Each on-chip memory space is its own SRAM with its own capacity, its own owning core, and a short list of legal ways in and out. They are separate buffers, not levels of a cache: L0A is not “inside” L1 the way a CPU’s L1 sits inside its L2. The figure below is a redraw of Ascend’s own 220x block diagram (hiascend.com, NPU Architecture Version 220x): Global Memory behind a shared L2 Cache, one AIC core (L1 → L0A/L0B/BT, Cube, L0C, and FixPipe’s own FP Buffer) and one AIV core (Unified Buffer and Vector), each with its DCache, ICache, Scalar unit and per-pipe instruction sequence. The yellow pills are the DMA units that actually carry each transfer, so an arrow’s label is the pipe that moves the data; the DSL call and the ISA instruction behind it are in the hover card. One caveat the diagram leaves out: a 910B die pairs two AIV cores with each AIC, identical to the one drawn here. Hover or tab to any box or arrow for its details.
PyPTO lowers the program through 64 passes over one IR. Tensor ops and tile ops live in the same tree; pass 13 is where one turns into the other. Diffing every pass’s output against its input shows that 18 of them change this program. The table lists all 64: what each pass is for in general, and what it did here or why it had nothing to do. Highlighted rows changed the IR; click one to open its diff in the stepper below.
| # | Pass | What it does | This program | Diff |
|---|
Phases are this page’s grouping of the pass manager’s order. Pass numbers are the
pass manager’s invocation order, which is what passes_dump/NN_after_<Pass>.py
files are named after; the pass documentation keeps its own numbering, which omits utility and
verification passes (this page’s pass 20 is doc 19, pass 39 is doc 37). The pass list, order and
descriptions come from the compiler’s own pass registry and documentation; the “this
program” column comes from diffing consecutive IR dumps of this compile.
Diffs are the real IR text of consecutive dumps, shown whole: every line of both
snapshots, aligned before | after, with removed lines marked −, added lines +, and the
unchanged runs between them dimmed. Inside a modified pair only the tokens that actually differ are
shaded darker (the word-level diff a code host shows for a one-word edit), so a renamed value
or an appended attribute does not read as a rewritten line. Scroll inside the panel for the rest of the
file. For readability the dumps
were run through ruff format (88 columns) before diffing, tile layout annotations
(pl.TileView(…)) are dropped, and pl.const(N, pl.INT64) is printed as N.
The +/− counts are added and removed lines.
Who wrote Left and Right?
A pass called InferTileMemorySpace exists, and you would expect
it to place the operands. Here it doesn’t. Pass 12 already put A and B in Mat and the
result in Acc, and pass 20 wrote target_memory=Left/Right
explicitly on its extracts. It has to: it runs two passes before inference, at a point where every tile
is 2D but no space has been inferred yet. By pass 22 there is nothing left to infer.
Double buffering is four passes, not one
Four passes cooperate, each with a different job.
AutoTileMatmulL0 (19) requests the depth the loop is emitted with:
pl.pipeline(…, stage=2), the same 2 that sizes k against half of
L0A. LowerPipelineLoops (32) expands it into two tagged copies per iteration
(pipeline_membership "0:0" / "0:1") that write into different halves.
CanonicalizeIOOrder (33) schedules it: copy two’s extracts move above copy
one’s matmul, so the load engine has work queued while the cube computes.
MemoryReuse (37) preserves it: the copies’ lifetimes are disjoint,
exactly what its coalescing merges, and the membership tag is what vetoes that merge, so two
Left and two Right buffers survive to allocation. Drop any one and ping-pong silently degrades:
nothing requested, nothing to separate, no overlap, one fused buffer.
Of the eighteen passes that changed something, one made a choice with alternatives:
AutoTileMatmulL0 picked the L0 tile. The operands as written cannot sit in the
cube’s operand buffers at all. A is 128 × 512 FP16, which is 128 KiB, and L0A holds 64 KiB.
Something has to cut K into L0-sized slices.
FP16 operands (2 bytes), FP32 accumulator (4 bytes), Ascend 910B capacities. This is residency arithmetic only. It shows which shapes are legal, not which legal shape is fastest.
Picking the tile size
When both operands stream through the cube (the output-stationary case, where the accumulator stays put and A and B slices flow past it), the chooser plans for two copies of each operand tile from the start. It sizes k against half of L0A: 64 KiB / (2 bytes × 2 stages) = 16 384 elements, so with m = 128 the largest legal k is 128. k = 256 would fit one buffer exactly and leave no room for the second. The pick fills L0A and L0B to exactly 100 %.
A search with a cost model
The chooser enumerates every legal 16-aligned (m, n, k),
crossed with which operand stays resident and whether the accumulator is double-buffered, and scores
each candidate with a roofline model of cube cycles, operand-load cycles (weighted by the unequal
L0A and L0B port bandwidths) and accumulator drain cycles. It returns the minimum. The model is
checked against a brute-force re-enumeration of itself, which proves the search is correct. Nobody
has timed the pick against alternatives on hardware.
The chosen 128 × 128 tile is not a hardware parameter. The cube’s unit is the fractal
box, and a tile is stored in every on-chip buffer as a grid of them: 16 rows × 32 bytes for
an operand tile (16 × 16 FP16 values, 512 B, the 32-byte axis turning with
slayout), and a 16 × 16 NZ box of 1024 B for the FP32 accumulator. Those are
the fractal=512 and fractal=1024 of the compiled tile types
(GetBoxedTileAlignment and kAccFractal in
tile_view_semantics.h), and the same two facts the memory-space inspector lists for Mat,
Left/Right and Acc. The tiler states its constraints in that unit (m, n and k move in steps of
16), and k need not divide K (the K-loop peels a 16-aligned tail), but K itself must be
16-aligned: a non-aligned K admits no legal tiling, so the pass leaves the op untouched.
Of the three, only k is really chosen: the budget above bounds it at 128 and the cost model
lands on the bound, while m and n are not chosen at all: they are this shape’s own M and
N. What keeps the output until last is the drain. The model counts drains as
⌈M/m⌉ × ⌈N/n⌉ output tiles, because FixPipe drains one
tile at a time; cutting K leaves that count at one (the partial sums stay in the same L0C
tile), while cutting M or N multiplies it. A K-block is not free (the model charges a fixed
cube start-up per block, and each block is another flag round in the K-loop), but it never buys a
drain, and that asymmetry is what orders the two. (This is tiling inside one task; the cross-core
split-K of variant 2 is a different thing, done to fill cores, not to fit buffers.) Here n = 128 is 16
N-fractals, a power of two, so the tile also escapes the drain penalty the model charges misaligned
widths: a width like 96, whose fractal count 12 is not a power of two, is priced extra serial
passes per row. The output is tiled only when it cannot fit L0C at all; a 128 × 128 FP32
accumulator is 64 KiB of the 128 KiB buffer, so it isn’t, and the K-loop figure ends in a
single TSTORE.
Each panel is one memory space, drawn to that space’s real capacity. Rows are the kernel’s statements in program order, not wall-clock time, so a block’s vertical span is its live range: from the statement that defines it to the last one that reads it. Horizontally, a block sits at the byte offset the allocator assigned; every space starts its own address line at zero, so offset 0 in L0A and offset 0 in Mat are different bytes. That is what “separate buffers, not a hierarchy” means in practice. Hover a block for its bytes and live range, or a statement for the buffers it touches.
Full, on purpose
L0A and L0B are at 100 % and L0C at 50 %. That is the tile choice from the previous section, showing up as addresses: ping at offset 0, pong at 32 KiB, nothing left over. Mat holds all of A and B (256 KiB of 512) for the whole kernel, because the program fits on the chip. Part II shows what happens when it doesn’t.
Pass 10 split the program in two, and so far the page has followed the kernel. The other half, the orchestration function, is also compiled. It becomes a small C++ library that runs on the device’s AI CPU, not on the host, and its only job is to tell simpler’s scheduler what tasks exist and which tensors each one reads and writes. This section follows that half from the compiler to the moment the cube starts.
Nothing about the runtime is decided in a separate tool. Four of the eighteen passes that changed the IR exist to shape the orchestration side, and each one maps to one line of the generated file below.
| Pass | IR change here | What it becomes in the orchestration file |
|---|---|---|
10 OutlineIncoreScopes | the pl.at body becomes kernel matmul_matmul; the caller keeps one call | one task, params_t0, and one submit call |
25 ExpandMixedKernel | kernel retyped InCore → AIC (pure cube, no vector half) | rt_submit_aic_task, not the mixed-cluster form |
50 DeriveCallDirections | arg_directions = [input, input, output_existing] | add_input, add_input, add_output |
59 MaterializeRuntimeScopes | body wrapped in one pl.scope() | one SIMPLER_SCOPE() { … } |
This is the whole file, exactly as the compile leaves it on disk, nothing cut. Click a line. The first two headers are stamped on after code generation (click line 4); everything below them is what the code generator wrote.
SIMPLER_SCOPE is not a barrier
It reads like a fork-join block, and it isn’t one. Closing a scope doesn’t wait for the scope’s tasks to finish. It drops the scope’s reference on each task it submitted, which is what allows the runtime to recycle those tasks’ ring slots and intermediate buffers once their consumers are done. Ordering between tasks comes from somewhere else entirely: the tensors each task reads and writes, as the next part shows.
From the Python call to the cube and back
A run involves three separately compiled programs: simpler’s host library, the orchestration
library on the AI CPU, and the kernel on the AI Core. Step through one call of
matmul(a, b, c).
With dependency capture on, the runtime writes down every task it was given and every edge it derived. For this program: one task on the cube, three tensor arguments, no edges. The direction tags are the ones pass 47 derived.
There is no edge list in the orchestration code. When a task is submitted, the runtime looks up each of its input tensors in a producer table (the TensorMap) and adds an edge from whichever earlier task last wrote it, then records this task as the producer of its outputs. A task with no producers goes straight onto the ready queue for its core type. So the direction tags from pass 47 aren’t metadata; they are the dependency graph.
That has consequences once there is more than one task. Part II has a case where a documented multi-core pattern quietly runs on one core at a time, and the reason is visible in exactly this kind of dump.
python -m simpler_setup.tools.deps_viewer deps.json --format html --show-tensor-info.
One node, its three arguments tagged with the direction pass 47 derived, and the
↓0 ↑0 fan-in / fan-out counts that say in the picture what the empty
edges array says in the JSON. On a multi-task kernel the same command draws the real
graph (nodes joined by annotated edges), and --edge-mode reduced then
marks which of those edges are transitively redundant. The figure is the tool’s own SVG, not a
redraw.From tile IR to C++, and where the synchronization comes from
After PyPTO's compilation pipeline is over, the kernel is printed as an MLIR module in the PTO dialect and handed to
PTOAS, the
assembler. In that module the memory spaces are still part of each tile’s type
(loc=mat, loc=left, loc=acc), and so are the layouts from the
memory-space table. What the module does not contain is a single synchronization
instruction. PyPTO has no intra-core sync pass. Between that module and the final C++, PTOAS runs
well over a dozen more transformations (most with their own CLI-visible pass name, the rest
anonymous, the PTO→EmitC instruction selection among them), and it alone inserts every
set_flag and wait_flag the kernel needs. Five of those pass invocations
change this program; the stepper near the end of this section plays each change back against its
input.
Inputs and outputs, precisely
The compiled function is
matmul_matmul(a: gm f16[128,512], b: gm f16[512,128], c: gm f32[128,128]). Inside the
K-loop below: pto.textract reads one [128,512] or [512,128]
Mat tile plus two loop-invariant offsets and writes one fresh 128×128 Left or
Right tile; pto.tmatmul/pto.tmatmul.acc read one Left tile and one Right
tile (the .acc form also reads the running Acc tile) and write the Acc tile;
pto.tstore, once, reads the final Acc tile and writes the [128,128] GM
output view.
in: the current .pto loop (selected excerpt)
scf.for %t__tile_l0_ko = %c0_index to %c512_index step %c256_index {
pto.textract ins(%mem_mat_3_buffer, %c0_index, %t__tile_l0_ko) outs(%mem_left_6_buffer)
pto.textract ins(%mem_mat_4_buffer, %t__tile_l0_ko, %c0_index) outs(%mem_right_7_buffer)
%0 = arith.addi %t__tile_l0_ko, %c128_index : index
pto.textract ins(%mem_mat_3_buffer, %c0_index, %0) outs(%mem_left_8_buffer)
%1 = arith.addi %t__tile_l0_ko, %c128_index : index
pto.textract ins(%mem_mat_4_buffer, %1, %c0_index) outs(%mem_right_9_buffer)
%2 = arith.cmpi eq, %t__tile_l0_ko, %c0_index : index
scf.if %2 { pto.tmatmul ... }
else { pto.tmatmul.acc ... }
pto.tmatmul.acc ins(%mem_acc_5_buffer, %mem_left_8_buffer, %mem_right_9_buffer) outs(%mem_acc_5_buffer)
}
// no set_flag / wait_flag / barrier in the PyPTO .pto module; PTOAS adds them later
// full types and allocations are in matmul_autotile_matmul_matmul.ptoout: the current emitted C++ excerpt
TLOAD(v36, v53);
TLOAD(v38, v57);
wait_flag(PIPE_MTE2, PIPE_MTE1, EVENT_ID0);
for (int64_t v58 = v10; v58 < v5; v58 += v11) {
wait_flag(PIPE_M, PIPE_MTE1, EVENT_ID0); // ping free?
pipe_barrier(PIPE_MTE1);
TEXTRACT(v42, v36, v59, v60);
TEXTRACT(v44, v38, v61, v62);
set_flag(PIPE_MTE1, PIPE_M, EVENT_ID0); // ping full
wait_flag(PIPE_M, PIPE_MTE1, EVENT_ID1); // pong free?
TEXTRACT(v46, v36, v64, v65);
TEXTRACT(v48, v38, v66, v67);
set_flag(PIPE_MTE1, PIPE_M, EVENT_ID1); // pong full
wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID0);
if (v58 == v10) {
pipe_barrier(PIPE_M);
TMATMUL(v40, v42, v44);
} else {
pipe_barrier(PIPE_M);
TMATMUL_ACC(v40, v40, v42, v44);
};
set_flag(PIPE_M, PIPE_MTE1, EVENT_ID0);
pipe_barrier(PIPE_M);
wait_flag(PIPE_MTE1, PIPE_M, EVENT_ID1);
TMATMUL_ACC(v40, v40, v46, v48);
set_flag(PIPE_M, PIPE_MTE1, EVENT_ID1);
}Both listings are selected excerpts from this exact compile. Left: PyPTO’s own SSA names,
unshortened; it already names things meaningfully. Right: PTOAS numbers every value
(v36, v42, …) with no names at all. The one thing still cut is the
GM tensor-view prologue before the loop (three
tensors’ worth of pto::Shape/pto::Stride/GlobalTensor
construction). The file itself, every line, is entry 6 of the emission stepper below, and
both files are checked in at
pypto-3.0-notes/matmul/matmul_autotile_matmul_matmul.{pto,cpp}.
Read the event IDs as buffer names
EVENT_ID0 is the ping buffer pair and EVENT_ID1 the pong pair.
Each pair has two flags going in opposite directions. MTE1→M says “this buffer is full,
compute may read it” (read-after-write). M→MTE1 says “compute has finished with this
buffer, the DMA may overwrite it” (write-after-read). Before the loop, PTOAS pre-sets both
M→MTE1 flags, so the first two extracts don’t wait for a matmul that never happened; after
the loop it drains them. This is literally what Event ID Allocation adds below, not a
paraphrase.
InsertSync is one pass, six sub-phases
-pto-insert-sync (PTOInsertSyncPass) is a
single MLIR pass with six internal sub-phases, each its own class, each traced below from a real
run of this compile (--pto-insert-sync-debug=2). The pass is off in the ptoas CLI
by default, but PyPTO turns it on for every kernel it compiles
(--enable-insert-sync in pto_backend.py), and this page’s
reproduce script passes it explicitly. The other sync strategies
(--enable-bufid_sync, A5-only; --enable-inject-barrier-all-sync,
barrier-before-everything) are not what this page traces.
| Sub-phase | Class | Observed on this compile |
|---|---|---|
| 1. Translator | PTOIRTranslator | builds a 17-node pipe-tagged IR from the loop; 0 sync ops yet |
| 2. Analysis | InsertSyncAnalysis | inserts 24 ops from pairwise hazard analysis (9 set_flag, 9 wait_flag, 6 pipe_barrier) |
| 3. Sync Motion | MoveSyncState | keeps 24 ops while moving them to cheaper positions around the two-copy loop |
| 4. Remove Redundant Sync | RemoveRedundantSync | drops 6 redundant flags, leaving 18 active ops: 6 set_flag, 6 wait_flag, and 6 barriers |
| 5. Event ID Allocation | SyncEventIdAllocation | assigns EVENT_ID0/EVENT_ID1 and adds the two pre-set and two drain operations: 18 → 22 active ops |
| 6. Codegen | SyncCodegen | writes the final 8 set_flag + 8 wait_flag + 6 barrier operations into the module. The EmitC stage maps each pto.barrier to the C++ pipe_barrier call. |
Every PTOAS pass between PyPTO pass 64 and the file on disk
| Pass | What it does here | Diff |
|---|---|---|
| main pass pipeline · 18 of the 21 invocations | ||
{anonymous}:: | the wrapper that runs the frontend pipe-lowering sub-pipeline (the two rows below) over the function; nothing module-level changes because of it | no-op |
↳ pto-assign-default-frontend-pipe-idPTO | gives ops that lack one their default frontend pipe id; every op here already carries it, so the dump is byte-identical | no-op |
↳ pto-lower-frontend-pipe-opsPTO | lowers frontend pipe-scoping ops; byte-identical here | no-op |
pto-infer-validate-pipe-initPTO | validates and infers pipe-init metadata; byte-identical here | no-op |
pto-lowering-sync-to-pipePTO | lowers explicit record_event/wait_event to set_flag/wait_flag; byte-identical, this kernel contains no manual sync | no-op |
pto-infer-layoutInfer | stamps {layout = #pto.layout<nd>} on the two tloads and the tstore; every other layout field was already in the tile types | |
pto-materialize-implicit-tmp{anonymous}:: | gives an implicit temporary a real buffer; byte-identical here: PyPTO pass 39 already left all seven tiles explicit, addresses and all | no-op |
pto-rematerialize-fixpipe-vector-quantPTO | fixpipe/vector-quant rematerialization; byte-identical, nothing is quantized in this kernel | no-op |
pto-resolve-reserved-buffersPTO | resolves reserved cross-core buffers; byte-identical, this is a single-core kernel | no-op |
pto-remove-identity-tmovPTO | removes same-SSA tmov copies; byte-identical, the kernel has none | no-op |
pto-insert-syncPTO | adds the kernel’s synchronization in 21 new lines: 8 set_flag, 8 wait_flag, 5 pipe_barrier in the pass diff | |
pto-resolve-buffer-selectPTO | materializes multi_tile_get selects; byte-identical, no pl.MemRef(slots=N) in this kernel | no-op |
{anonymous}:: | narrows unused multi-result provenance; byte-identical | no-op |
cseCSE | folds redundant address computation and renumbers the values after it | |
pto-inline-backend-helpersPTO | inlines the EmitC helper functions into their call sites; byte-identical here | no-op |
{anonymous}:: | the second run of the same pass; byte-identical | no-op |
canonicalizeCanonicalizer | generic MLIR canonicalization; byte-identical | no-op |
cseCSE | the second run; byte-identical | no-op |
| EmitC backend pipeline · a second pass manager, built at emission time | ||
{anonymous}:: | the PTO→EmitC instruction selection: pto.tload/textract/tmatmul/tstore/set_flag become emitc.call_opaque calls into PTO-ISA and the tile types vanish | |
{anonymous}:: | folds the flat EmitC chain into emitc.expression regions so expression boundaries are explicit; the same program, written longer | |
cseCSE | the third run; byte-identical | no-op |
The steps are the real dump text of this compile: ptoas run on
the .pto above with --enable-insert-sync --pto-level=level3 --mlir-print-ir-before-all
--mlir-print-ir-after-all (PTOAS 0.72, the build this page pins). Of the 21 pass invocations, five
leave the printer’s text different; those five, after the module as the tool loads it, are the six
steps. Step 0 is not a diff: it is the state every pass starts from, and the printer rewrites
almost every line of the file on the way in, so a line diff there would say nothing. A row marked
no-op above is one where the printer wrote the same text before and
after the pass. Each step shows the unit the pass ran on, whole: the module for module-scope passes
(wrapper included), the function alone for the function-scope ones. Nothing is normalized or reflowed,
which is why the value numbering shifts from step to step; the only reflow on screen is wrapping, so a
statement too long for the panel (a full-type textract is ~370 characters) continues on the
next line, indented under its own first character. The dump has it on one line. The last two steps speak EmitC rather than
PTO: they are the first two passes of the EmitC backend pipeline, the second pass manager the
driver builds once the main pipeline is done. The step from there to the .cpp on disk is
four more stages, and none of them emits a dump; the note below follows them. Inside a modified
pair the tokens that differ are shaded darker, word-level, as in the other steppers. Click a
highlighted row to open its diff here.
Where the C++ text comes from
The dumps stop at the end of the EmitC backend pipeline. Between that
state and the .cpp on disk there are four stages, none of which emits an IR dump, all in
PTOAS/tools/ptoas/ptoas_pipeline.cpp unless noted:
runEmitCPreparationPipeline, the second pass manager
(EmitPTOManualPass → FormEmitCExpressionsCompatPass →
CSE): the last three rows of the table above are this pipeline, not the main
pipeline’s tail.dropEmptyEmitCExpressions, materializeControlFlowOperands,
normalizeEmitCIntegerAttrsForCppEmission, reorderEmitCFunctions,
annotateEmitCProvenanceHints, …emitc::translateToCpp, MLIR’s EmitC→C++ translator, called with
declareVariablesAtTop from shouldDeclareVariablesAtTop(module). This is
the call that turns emitc.call_opaque "TEXTRACT" into TEXTRACT(…),
emitc.verbatim into raw text, and each !emitc.opaque type into the C++
type name it spells.runCppPostRewrites (tools/ptoas/ptoas_cpp_rewrite.cpp), a
fixed-order list of text rewrites on the generated C++: call-site markers, the scalar-GM
store flush, scalar-constant and hoisted-GlobalTensor declarations, malformed-verbatim
repair, name hints. Only after this does the text become the file PTOAS writes.So the C++ listing above is not printed from the last
dump’s state: it is that state after one translation and one text-rewrite chain.
--emit-pto-ir stops before the translation and prints the lowered PTO IR instead;
--pto-backend=vpto bypasses this path entirely. The stepper below follows those four
stages and ends with the file itself, whole, so the chain can be checked end to end: each of the
two dumpable stages appears twice, once as the detail that maps it and once whole.
Six entries. Entries 1 and 3 carry the detail of the two
dumpable stages (the op-by-op mapping and the fix‑up call list), and entries 2
and 4 carry each of those stages whole: the module before and after
EmitPTOManualPass, then that module and the C++ it becomes. Those two are stacked,
not paired row for row, because the printer and the translator rewrite almost every line (stage 1
replaces 128 of 191 rows, stage 3 adds 229 and removes 134, with eight and two lines left in
place), so a paired diff would shade nearly everything and explain nothing. The mapping in
entry 1 is what pairs. Inside a pair, the tokens that differ are shaded darker, as in the other
steppers. Entries 3 and 5 are stages that leave no dump and are therefore shown only from the side
that exists: the source that runs them, or the text they produced. Entry 6 is not a stage:
it is the file those four write, the whole function, verbatim, with the lines the earlier entries
took as windows tinted inside it and the ends of the pipe figure’s abridged listing labelled,
so every excerpt above can be located inside the whole. Every line is cut verbatim from this
run’s own artifacts (the dump log, the emitted .cpp, and
ptoas_pipeline.cpp) by matmul/gen_primer_emission_steps.py, which
re-checks each cut before publication.
| Instruction | Pipe | Hardware | Issued |
|---|---|---|---|
TLOAD × 2 | MTE2 | DMA: global memory → L1 | once, before the loop |
TEXTRACT × 4 per trip | MTE1 | DMA: L1 → L0A / L0B | 8 in total |
TMATMUL / TMATMUL_ACC | M | the cube | 1 + 3 in total |
TSTORE | FIX | FixPipe: L0C → global memory | once, after the loop |
What stage=2 actually buys
Each pipe (the two DMA engines, the cube, the FixPipe) executes its own instructions in order, and the
pipes run in parallel with each other. The only thing that holds one pipe back for another is a flag.
This figure places every instruction of the kernel at the earliest step its flags allow, assuming
every instruction takes one step. Switch to stage=1 to see the same kernel with a single
buffer pair. The listing beside the figure is an abridged transcription of the current emitted
instruction stream: 31 rows for the 60 lines from the first TLOAD through the
final TSTORE (lines 148–207). The complete function is shown in entry 6 of
the emission stepper above, and the omitted bookkeeping is itemised below.
A box is one dependency-graph step, and now also one real instruction: MTE1 →
L0A and MTE1 → L0B are two rows, not one, because the A-tile and B-tile fetches
are two separate TEXTRACT calls on the same pipe, in program order: A first (it waits
for the ping/pong slot to free), then B immediately behind it (its completion is what actually posts
the flag the cube waits on). “matmul 0” still has no such split: one step is exactly one
TMATMUL/TMATMUL_ACC call.
set_flag/wait_flag/pipe_barrier instructions PTOAS emits.Instruction stream · stage=2, as compiled
Good to know. The 29 omitted lines are 10
// pto: <value> provenance comments, 2 stray ;, 8
Shape/Stride/GlobalTensor views, 8
static_cast<uint16_t> operand temporaries and 1 loop-index computation.
The listing keeps the loads, extracts, flags, barriers, matmuls, loop control and store that
the figure schedules; the whole function in entry 6 keeps every line. The checker beside
the source artifacts fails if either the selected rows or this accounting drifts.
What the figure shows
With two buffer pairs, the extract for iteration i+1 goes into the other half while matmul i is still reading the first, so MTE1 and the cube stay busy on the same steps. The dashed violet edges are the new constraint double buffering introduces: iteration i+2 reuses iteration i’s half, so its extract waits for matmul i to finish. The amber chain is the one dependency no buffering can remove: all four matmuls accumulate into the same L0C tile, in order.
What it does not show
Positions are dependency steps, not time. On real hardware a 128 × 128 × 128 FP16 matmul and a 32 KiB L1→L0 copy take different numbers of cycles, and whether MTE1 or the cube is the bottleneck depends on those numbers. The simulator this page used runs the kernel correctly but does not model timing. The step counts are a statement about the kernel’s dependency structure, which is exactly what the flags encode.
Part II
Part I’s kernel is correct, tiled for L0, double-buffered and synchronized, and it runs on one of the 910B’s 24 cube cores with operands that fit on the chip. Everything an HPC matmul adds beyond that has an owner. Here is who owns what, checked against the compiler, the runtime and the assembler source, and then each missing piece added to the same program and compiled.
| Optimization | Who does it | How | In Part I? |
|---|---|---|---|
| L0 tiling: K-loop, and M/N split when the output overflows L0C | PyPTO pass 20 | automatic cost-model search | yes: k = 128 |
| L0A/L0B ping-pong inside that loop | PyPTO passes 20, 33, 34, 42 | automatic once the matmul is tiled | yes: 2 × 2 buffers |
| Intra-core synchronization and event IDs | PTOAS sync pass | automatic always enabled by PyPTO | yes: 2 events |
| Fractal layout for tiles in Mat / L0 | PyPTO tile conversion | automatic converted during the load | yes |
| Accumulator (L0C) double buffering | PyPTO passes 20, 33, 34, 42 | automatic or opt-in automatic under memory_planner=PTOAS and DSA_RP; a marked-experimental flag under the default PyPTO planner | no: one Acc tile is sufficient for this compile |
| Using more than one core: M/N split or split-K | you, in the DSL; simpler dispatches | one more line pl.spmd(n) | no: variants 1, 2 |
| L1 tiling: operands bigger than the 512 KiB Mat | nobody | compile error you block K yourself | no: variant 3 |
| Double buffering of your own loops (GM→L1) | you; PyPTO lowers it | one more line pl.pipeline(…, stage=2) | no: variant 3 |
| Weights stored in the hardware layout | you; PyPTO pass 17 plans for it | a type annotation pl.NZ | no: variant 4 |
| Epilogue fusion (bias, ReLU, quantize on writeback) | you, at tile level | tile-level store options | no |
| Overlap between tasks | simpler | automatic for tasks you created, ordered by their tensors | one task: nothing to overlap |
What those four passes each do
The ping-pong row names four passes because no single pass owns this optimization. Reading them in order is the whole mechanism, and the fourth one is the one that people usually assume is the second.
AutoTileMatmulL0 requests it. The tiled K-loop is emitted as
ForKind::Pipeline with pipeline_stages=2. That 2 is the
operand depth (L0A and L0B), not the accumulator. It is also why
k = 128 and not 512: the chooser sizes the tile against half of each operand buffer,
which is the arithmetic in the previous section.LowerPipelineLoops replicates the body. Two copies per outer iteration, each
with fresh def-vars, and each clone's tile-producing call tagged with a
pipeline_membership (group, stage) pair.CanonicalizeIOOrder shapes the schedule: scalars first, then the loads
clustered ahead of compute, so the load engine runs ahead of the cube. It only reorders statements; it
does not create the buffer separation.MemoryReuse is where the separation is actually enforced. The two clones are
sequential in program order, so their tiles have disjoint lifetimes, exactly the
condition the reuse packer coalesces on. Left alone it would fuse the ping-pong back into a single
buffer. The pipeline_membership tag is what forbids that, and it does so only as deep as
the memory space can afford.The accumulator is deliberately left untagged: it is written
by the serialized cube, one tile's MAD retiring before the next begins, so a single L0C buffer is already
correct. That is why this page shows one Acc @ 0 rather than a pair.
The pattern
The compiler automates the single-core, fits-on-chip story
completely. Everything about many cores and data bigger than the chip is written by
the programmer today, as small DSL additions to the same program. That split is deliberate: the
assembler’s design notes rule out inferring multi-buffering on its own, the accumulator double
buffer is experimental under the default planner (it is automatic under
memory_planner=PTOAS and DSA_RP, and needs the full-K emitter either way),
and nothing in the compiler splits one device region across cores.
Production kernels built on PyPTO write all four variants below by hand.
The switch this page did not have to touch
Everything above was compiled with the default
memory_planner=PYPTO. That setting decides who owns on-chip memory, and it is the one knob
that can also move the tile the chooser picks.
PYPTO (default) | PTOAS | |
|---|---|---|
| Addresses | baked addr on every tile; ptoas runs at --pto-level=level3 | addr-less tiles, placed by ptoas at --pto-level=level2 |
| Reuse and placement | PyPTO MemoryReuse + AllocateMemoryAddr | PTOAS PlanMemory, with both of those skipped |
| L0C double buffer | opt-in flag | automatic, if the chooser picks it |
It can change the tile. dbC = 2 requires the full-K
emitter (l0_tile_chooser.cpp sets require_full_k = !is_os || r.dbc),
and halves the L0C accumulator budget (l0c_bytes / (bytes_c × 2)), so making it
available re-ranks the candidate shapes. On this program it changed nothing: the same
128 × 128, k = 128, one accumulator, at identical offsets, under all
three configurations. That is the useful part: Part I is a default-planner picture, and it holds
because k = 128 split-K won on modelled cost, not because a full-K
dbC = 2 candidate was illegal.
Variant 1 · more cores, split the output
pl.spmd# M = 512, N = 128, K = 512
@pl.jit
def matmul_spmd(a: pl.Tensor[[512, 512], pl.FP16],
b: pl.Tensor[[512, 128], pl.FP16],
c: pl.Out[pl.Tensor[[512, 128], pl.FP32]]):
for i in pl.spmd(4): # i = block index
r0 = i * 128
part = pl.matmul(a[r0:r0 + 128, :], b,
out_dtype=pl.FP32)
c = pl.assemble(c, part, [r0, 0])
return c
Runs. Max error vs PyTorch 6.9×10−5.
The orchestration file still submits one task, now with
launch_spec.set_block_num(4). The runtime expands it into four blocks at dispatch, and each
block runs Part I’s kernel on its own 128 rows of A: same K-loop, same ping-pong, same flags. The
dependency dump shows one task, block_num 4, and no edges.
Each block loads all of B. For this shape that is the right trade (B is 128 KiB); for a wide B you would split N as well, and that choice is yours to write.
Variant 2 · more cores, split the reduction
When M and N are too small to feed many cores, split K instead: each core
reduces a slice of K and adds its partial result into c with an atomic add. There are two
obvious ways to write it. They compile, run and give the same answer. Only one of them is parallel.
# A: a loop of device regions (M=N=128, K=2048)
with pl.at(level=CORE_GROUP, name_hint="zero_init"):
c[:] = pl.full([128, 128], dtype=pl.FP32, value=0.0)
for ks in pl.parallel(4):
with pl.at(level=CORE_GROUP, name_hint="split_k"):
k0 = ks * 512
p = pl.matmul(a[:, k0:k0 + 512], b[k0:k0 + 512, :],
out_dtype=pl.FP32)
c = pl.assemble(c, p, [0, 0],
atomic=pl.AtomicType.Add)
# B: one SPMD region (same shapes)
with pl.at(level=CORE_GROUP, name_hint="zero_init"):
c[:] = pl.full([128, 128], dtype=pl.FP32, value=0.0)
for ks in pl.spmd(4):
k0 = ks * 512
p = pl.matmul(a[:, k0:k0 + 512], b[k0:k0 + 512, :],
out_dtype=pl.FP32)
c = pl.assemble(c, p, [0, 0],
atomic=pl.AtomicType.Add)
Serialized. Correct result (8.8×10−5), one core at a time.
zero_init (AIV)
└─wait─▶ split_k[0]
└─wait─▶ split_k[1]
└─wait─▶ split_k[2]
└─wait─▶ split_k[3]
Five tasks, and the runtime chained all of them. Each split_k task declares c as
read-write (the atomic add reads it), so the TensorMap makes each one wait for the previous writer of
c. The hardware atomic in the kernel is real; it just never gets a concurrent writer.
Concurrent. Correct result (9.2×10−5), four cores at once.
zero_init (AIV)
└─wait─▶ split_k one task, 4 blocks
├─ block 0: K[0:512]
├─ block 1: K[512:1024]
├─ block 2: K[1024:1536]
└─ block 3: K[1536:2048]
Two tasks and one edge. The four slices are blocks of a single task, so there is no chain between them to derive: they start together and race on the atomic add in hardware, which is the point.
The rule behind it
simpler orders tasks by the tensors they declare, and an atomic add is still a read-modify-write of the whole output. Tasks that accumulate into the same tensor must therefore be blocks of one SPMD task to run concurrently. Form A is the one a reader coming from GPU programming would write first, and PyPTO’s own matmul tutorial teaches it. The dependency dump is the quickest way to catch this: count the edges.
Variant 3 · bigger than the chip
Make K = 4096. A and B are now 1 MiB each, and Mat holds 512 KiB.
# the Part I program, K = 4096
with pl.at(level=pl.Level.CORE_GROUP):
c[:] = pl.matmul(a, b, out_dtype=pl.FP32)
Compile error at memory allocation (pass 41):
Mat buffer usage (2097152 bytes) exceeds platform limit (524288 bytes)
Pass 12 loads each operand whole into Mat, as in Part I, and nothing between pass 13 and pass 41 tiles at the L1 level. The L0 tiler would happily split K for the cube, but it runs on operands that must already be in Mat.
# block K by hand, 256 at a time
with pl.at(level=pl.Level.CORE_GROUP):
acc = pl.matmul(a[:, 0:256], b[0:256, :],
out_dtype=pl.FP32)
for k in pl.pipeline(1, 16, stage=2):
k0 = k * 256
acc = pl.matmul_acc(acc, a[:, k0:k0 + 256],
b[k0:k0 + 256, :])
c[:] = acc
Runs. Max error 5.2×10−4 (FP16 inputs, K = 4096).
Your stage=2 gives Mat four 64 KiB buffers: two chunks of A and B in
flight, so the next chunk loads from global memory while the current one computes. Inside each chunk
the compiler still tiles for L0 exactly as in Part I.
The compiler tells you what it couldn’t do
Nesting has a cost. Your loop now contains three copies of the compiler’s
own L0 K-loop (a first chunk and two pipeline copies), each asking for its own ping-pong pair, and L0A has
room for two buffers in total. The compile succeeds and prints a performance hint for each:
requested depth 2 … in Left, but only 1 of 2 buffers fit … stages 1 apart share storage
and serialize. The fix is a smaller chunk or a different split, and it is yours to choose.
The other way to say the same thing
pl.pipeline(…, stage=2) is not the only spelling of
ping-pong. pl.MemRef("name", slots=N) declares one allocation with N uniform
slots and picks one per iteration with an ordinary index expression, e.g.
pl.MemRef("ub", slots=2)[i % 2]. Under memory_planner=PTOAS that lowers to a
single ptoas pto.alloc_multi_tile region read through pto.multi_tile_get, and
ptoas derives the per-slot synchronization from the slot index itself. Where
pl.pipeline restructures the loop into a schedule, slots only remove the same-buffer hazard:
the loop stays sequential.
Worth knowing why the compiler’s own matmul pipeline
never uses that form. The slots path requires the loop’s step to be 1 with
a start that is a multiple of N, so that ptoas can match the slot index as a plain
iv % N; the tiled K-loop is emitted with step = k, so it is declined and
replicated instead. And the shape where two slots are live at once is refused at codegen rather than
miscompiled, because the pinned ptoas still mis-synchronizes the prefetch variant of it
(hw-native-sys/PTOAS#1519: fixed in 0.63, regressed in 0.64–0.65). One slot live per
iteration is the shape the region form exists for.
Variant 4 · the layout of the weights
@pl.jit
def matmul_nz(a: pl.Tensor[[128, 512], pl.FP16],
b: pl.Tensor[[512, 128], pl.FP16, pl.NZ],
c: pl.Out[pl.Tensor[[128, 128], pl.FP32]]):
with pl.at(level=pl.Level.CORE_GROUP):
c[:] = pl.matmul(a, b, out_dtype=pl.FP32)
return c
# host side: the bytes must already be in NZ order
def to_nz(x): # [R, C] FP16
r, cols = x.shape
return (x.reshape(r // 16, 16, cols // 16, 16)
.permute(2, 0, 1, 3).contiguous()
.reshape(r, cols))
Runs. Max error 7.6×10−5.
Every load into Mat converts row-major data into the fractal layout on the fly. For weights that are
loaded many times that conversion is repeated work. pl.NZ is not a request to convert: it
is a promise that the bytes in global memory are already in fractal order.
Pass 16 (BlockNzTensorViews, a no-op in Part I) now does something. It rewrites B’s
logical [512, 128] into the blocked [1, 8, 32, 16, 16] the ISA describes
(8 column blocks of 16, 32 row fractals of 16 × 16), and the load of B becomes NZ→NZ
(pto::Layout::NZ in the emitted C++). A stays row-major. Keeping the promise is the
caller’s job; the reorder is the six lines on the left.
Static shapes, an output dtype, an output marker and a device region were enough for the stack to produce a tiled, double-buffered, synchronized cube kernel and the host code that launches it.
One pass made a real choice (the L0 tile, from a cost model sized for two buffers). The rest are mechanical: outline, lower to tiles, replicate for ping-pong, reorder loads first, allocate, tag argument directions.
The orchestration library runs on the AI CPU. Edges come from input/output tags through a producer table, and a scope manages buffer lifetime, not ordering.
Two event IDs, one per buffer half, each with a data-ready and a buffer-free direction.
One more line each: pl.spmd for cores, pl.pipeline over your own K-blocks for
L1, pl.NZ for weights. And accumulate across cores inside one SPMD task, or the runtime
will serialize you.
Everything above came from compiling and running the programs on this page with IR dumps and dependency capture turned on, in the PyPTO simulation container, on a laptop.
import os
import torch
import pypto.language as pl
from pypto.ir.pass_manager import PassDumpLevel
from pypto.runtime import RunConfig
# read once per process: set it before compiling
os.environ["PYPTO_PROG_BUILD_DIR"] = "out"
# … the @pl.jit matmul from the top of the page …
a = torch.randn(128, 512, dtype=torch.float16)
b = torch.randn(512, 128, dtype=torch.float16)
c = torch.zeros(128, 128, dtype=torch.float32)
dump = RunConfig(dump_passes=PassDumpLevel.EXPLICIT)
matmul.compile(a, b, c, config=dump)
matmul(a, b, c, config=RunConfig(enable_dep_gen=True))
ref = a.float() @ b.float()
assert torch.allclose(c, ref, rtol=1e-2, atol=1e-2)
Compare against the FP32 product of the FP16 inputs, at an FP16-sized tolerance.
Under out/ you get one directory for the compile and one for the run. The compile directory
holds passes_dump/ (65 files: the parsed input plus one per pass), ptoas/*.pto
and the emitted kernel C++, orchestration/*.cpp, and report/perf_hints.log. The
run directory holds dfx_outputs/deps.json.
cd out/*/passes_dump
prev=; for f in $(ls | sort); do
[ -n "$prev" ] && ! cmp -s "$prev" "$f" && echo "$f"
prev=$f
done
Eighteen names come out for the program on this page. Then read each diff; the table above tells you what to look for.
grep -nE 'set_flag|wait_flag|TEXTRACT|TMATMUL' ptoas/*.cpp gives the instruction stream of the
K-loop figure. For any multi-task program, count the edges in deps.json before
you believe it runs in parallel.
Set M = N = 256 and the output cannot fit L0C, so pass 20 has to split that axis as well as K. Set K = 4096 and allocation fails outright, as in variant 3: nothing between the tensor loads and the L0 tiles will block K for L1. The residency explorer above predicts the L0 side of both before you compile.
The compiler is PyPTO (documentation); the runtime it lowers for is simpler (documentation); the assembler that turns the PTO module into C++ is PTOAS, and the calls it emits go into PTO-ISA. Kernels and models built on them (including the Qwen3-14B decode kernel whose split-K shape variant 2 follows) are in pypto-lib; the project’s documentation hub is pypto.ai.