Sideband-separating (2SB) receiver can simultaneously observe the upper sideband (USB) and lower sideband (LSB) signals, with high observation efficiency and no aliasing on the upper and lower sidebands. It has attracted more and more attention in the field of radio astronomy. Due to the amplitude and phase errors of the fully analog sideband separation receiver that are difficult to overcome, the sideband separation ratio is low, which affects the performance of the system. Digital sideband separation receivers can effectively improve the system's sideband separation ratio through digital signal processing methods. A digital sideband-separation receiver prototype system was built in the 3--18GHz band. Based on the sideband separation theory and digital calibration method, the calibration of the experimental system is realized, which greatly improves the sideband separation ratio of the system.