This study investigated the effect of Nb substitution on the magnetic and structural properties of CoFeO nanoparticles (NPs) synthesized by hydrothermal approach. The formation of a single phase of spinel ferrite was confirmed through X-ray powder diffraction, and crystallite sizes in the range 18–30 nm were observed. Moreover, it found that the Fourier transform infrared (FT-IR) spectra of the NPs included the main vibration bands of the spinel structure. The partially cubic structure was confirmed by scanning electron microscopy (SEM) and transmission electron microscopy (TEM). The energy band gaps for CoNbFeO were estimated to be in the range 0.48–0.53 eV for Nb content x = 0.0–0.10. Magnetization measurements at room temperature (RT; 300 K) and at 10 K were performed on spinel CoNbFeO (0.00 ≤ x ≤ 0.10) NPs using a vibrating sample magnetometer (VSM). Nb doping significantly changed the magnetization and coercivity of the Co ferrite samples. RT hysteresis curves indicated well-defined ferrimagnetic behavior for all prepared NPs with saturation magnetization (M) in the range 44.45 – 49.40 emu/g and remanent magnetization (M) in the range 12.16 – 17.90 emu/g. The coercive field (H) is found to be equal 936 Oe and is decreased with Nb substitutions. However, hysteresis curves at 10 K showed finite remanent specific magnetization (1.90–6.70 emu/g) but significant asymmetric coercivity (715–2810 Oe), particularly for the Nb-doped samples. At 10 K, the magnetization values were 4–6 times smaller but symmetric coercivity field values were 2–3 times larger compared with the RT-VSM curves. The obtained magnetic parameters indicated the semi-hard magnetic character of the Co ferrite samples at low temperatures.