Despite significant improvements in peak bandwidth, the HBM industry has neglected random-access (irregular) bandwidth, limiting performance in many real-world applications. Improving effective HBM bandwidth is challenging due to power-constrained activations and coarse-grained, long-distance data movement. Rather than addressing these issues directly, hardware vendors have opted for incremental changes, achieving a 6.4x increase in sequential access bandwidth over two generations while leaving the irregular bandwidth challenges unresolved. To remedy this, we introduce Folded Banks (FB-HBM), a novel 3D bank design that redistributes bank subarrays ("folds") across multiple dies and relocates command, control, and global sense amplifiers to an additional base layer. By implementing this logic in a new base layer, we eliminate the DRAM die overheads inherent in previous designs. This architecture enables vertical routing of intra-bank wires-column select lines (CSLs) and master data lines (MDLs)-through thin-pitch through-silicon vias (TSVs) and hybrid bonds, significantly reducing RC power losses. By employing self-timed sense amplifiers, we eliminate costly dummy subarrays previously required for reference voltages. Furthermore, our distributed TSV architecture minimizes inter-bank data movement and we reduce HBM row size and activation energy by selectively disabling memory arrays (MATs) within a bank. Compared to a projected HBM4 design, FB-HBM achieves a 6.7x improvement in random-access performance with a conservative 5 mu m TSV pitch. This architectural advantage translates to a 2.28x speedup across high-performance computing (HPC) and sparse machine learning applications.