Terahertz (THz) band communications are currently being celebrated as a key technology that could fulfill the increasing demands for wireless data traffic and higher speed wireless communications. Many challenges, however, have yet to be addressed for this technology to be realized, such as high propagation losses and power limitations, which result in short communication distances. Ultra-massive multiple input multiple output (UM-MIMO) antenna systems have emerged as practical means for combatting the distance problem at the THz range; thereby increasing system capacity. Towards that direction, graphene-based nano-antennas of small footprints have been recently proposed, as they can be individually tuned and collectively controlled in an UM-MIMO array of sub-arrays architecture. In this paper, we present a holistic overview of THz UM-MIMO systems by assessing recent advancements in transceiver design and channel modeling. We discuss the major challenges and shortcomings of such designs from a signal processing perspective, by deriving the relation between system performance, communication range, and array dimensions. We further highlight several research advances that could enhance resource allocation at the THz band, including waveform designs, multi-carrier antenna configurations, and spatial modulations. Based on this discussion, we highlight prospect use cases that can bring THz UM-MIMO into reality, in the context of sensing, data centers, and future mid-range wireless communications.