An Acronym for AT Extended. It is an older, then functionally optimal design in mainboard layout in PC architecture. BTX improves on it by taking today's considerations into account.
ATX specifies where the CPU is on the mother board, the size of the PSU, the position of the screw holes in the motherboard, the power connectors, where the holes on the back of the case should be, where the connectors on the front of the motherboard (for the front lights and buttons) should be etc.
While it was an excellent standard when conceived, it has aged. ATX was designed with the excellent idea of modular components, including each being responsible for its own cooling. Unfortunately, modern computers tend to have much more demanding cooling needs than once upon a time, and cooling components individually is a recipe for noisy systems. An integrated approach to cooling that addresses the needs of multiple components, for example using a heatpipe, drastically reduces both the flexibility of the design on the one hand as well the noise on the other hand. BTX is an answer to many of these issues.