To provide inertia support for the grid, virtual synchronous generator (VSG) control of inverter-interfaced distributed generators (IIDGs) becomes a focus of worldwide attention. However, a VSG-based IIDG behaves differently in the grid-connected mode, the islanded-single-DG mode, and the islanded-multi-DG mode, whereas the mathematical and physical interpretations of this phenomenon are not well studied. In this paper, we propose a unified modeling method of VSG-based IIDG to analyze its different dynamic performance in each operation mode. The proposed unified formulas can obtain the state-space models of islanded-single-DG mode and islanded-multi-DG mode from that of grid-connected mode for any VSG control method. With the obtained models, for several different types of VSG control in different operation modes, we analyze the distribution and sensitivity of the closed-loop poles and investigate the step responses both analytically and experimentally. These analyses reveal the intrinsic differences and correlations of the dynamics of VSG-based IIDG between each operation mode. These intrinsic features are valid independent from the applied VSG control scheme, thus a test method to evaluate the parameters and performance of an unknown IIDG is derived. The findings of this paper provide important instructions for engineers to model, design and test multi-operation-mode distributed generators.