The energetic offset between the highest occupied molecular orbitals of the donor and acceptor components of organic photovoltaic blends is well-known to affect the device efficiency. It is well-established that a decreasing offset increases the open-circuit voltage but reduces the short-circuit current, which has been explained by insufficient exciton dissociation. However, the impact of the offset on the fill factor and underlying processes is less clear. Here, we study free charge generation and recombination in three different nonfullerene acceptors, Y6, ITIC, and o-IDBTR, blended with the same donor polymer PM6. We demonstrate that a diminishing offset results in field-dependent charge generation related to field-assisted exciton dissociation. On the other hand, reformation of excitons from free charges is identified as an additional channel for charge recombination, which goes along with a substantial rise in the bimolecular recombination coefficient. In combination of these two effects, the fill factor drops considerably with a decreasing energy offset. Using the comparison between PM6:ITIC and PM6:o-IDBTR, we show that bulk properties such as morphology and carrier mobilities can not fully explain the observed difference in performance, highlighting the importance of interfacial kinetics and thermodynamics in controlling the device efficiency, both through generation and recombination of charge carriers.