Superscalar processors, VLIW and simultaneous multithreading
Multiple issue and IPC, the quadratic cost of control with proof, VLIW architectures and their limits, simultaneous multithreading, the ILP, memory and power walls, the multicore turn.
A perfect pipeline completes one instruction per cycle. To do better, several instructions per cycle must be fetched, decoded and executed: this is the superscalar processor. Alternatively parallelism can be decided by the compiler and written explicitly in the instructions: the VLIW approach. Or the empty slots can be filled with instructions from another program: simultaneous multithreading. In this lesson we look at the three paths, prove why control complexity grows with the square of the width, and see the limits that pushed the industry towards multicore.
The superscalar processor
A 
-way superscalar processor fetches and decodes up to instructions per cycle, renames them and sends them to the functional units, of which there is more than one per type. In the ideal case it completes instructions per cycle: the CPI drops to 
, and one prefers to speak of its inverse, the IPC (instructions per cycle).
The real IPC is much lower than : true dependences, mispredicted branches and above all memory waits leave many slots empty. A 6-8-way processor typically reaches an IPC between 2 and 4 on common programs.
The cost of control
Issuing several instructions together requires checking, in the same cycle, the dependences among the instructions of the group: if the second reads a register written by the first, it must receive the new physical register assigned to the first, not the old one from the rename table.
Theorem. Renaming in one cycle a group of instructions with two source operands each requires 
register comparisons.
Proof. The instruction in position 
of the group (from 1 to ) must compare each of its two source registers with the destination register of each of the 
instructions preceding it in the group. The total is 
. ∎
The number of comparisons grows with the square of the width, and so do the read and write ports of the register file, the forwarding paths between units and the logic that selects which ready instructions to send. Power and the difficulty of meeting the clock cycle grow more than linearly, while IPC grows less than linearly. That is why widths have stopped between 4 and 10.
VLIW: leaving the work to the compiler
In the VLIW (very long instruction word) architecture every instruction is a bundle of operations that the compiler guarantees to be independent, one for each functional unit. The processor checks nothing: it executes the bundles as they are, with simple, low-power hardware. The compiler must find the parallelism with loop unrolling and instruction scheduling (lesson on dependences), filling with no-ops the slots it cannot use.
VLIW has been very successful in DSPs and accelerators, where code is made of regular, predictable loops. In general-purpose processors, instead, it ran into two problems: the compiler does not know memory latencies, which change at every run, and code compiled for one generation does not exploit the next one with different units. The best-known example is Intel and HP’s Itanium, discontinued in 2021.
Simultaneous multithreading
Even the best superscalar leaves many of its slots empty in every cycle, because a single program does not have enough independent instructions. Simultaneous multithreading (SMT, marketed by Intel as Hyper-Threading) makes one core look like two (or more) processors: it has two copies of the architectural state (registers, program counter) and fetches instructions from both threads, mixing them in the same out-of-order window. The instructions of the two threads have no dependences between them, and fill the slots left free by the other, especially when one of the two waits for memory.
The typical gain is 15-30 % of overall throughput for a few per cent more area. The price is that the two threads compete for caches, predictors and units: each runs slower than alone, and one thread can spy on the other by measuring timing, which is why some systems disable it for security.
The limits and the multicore turn
Until the early 2000s processor performance grew by about 50 % a year thanks to higher frequencies and more ILP. Then it hit three walls:
- the ILP wall: beyond a certain width IPC no longer grows;
- the memory wall: DRAM has stayed slow, hundreds of cycles per access;
- the power wall: raising the frequency requires raising the voltage, and power grows roughly with the cube of the frequency (lesson on energy).
The industry’s answer was to put more cores on the same chip: instead of an ever larger and hotter core, several cores of reasonable size. But parallelism must become explicit, written by the programmer with threads, and shared memory brings the coherence and consistency problems of the lesson devoted to them.
