Abstract: The surface residual stress of a shot peened nickel-based monocrystalline superalloy AM1 was calculated based on the Ortner method by measuring X-ray diffraction peaks, and effects of diffraction position step precision and diffraction angel step precision on measuring results of residual stress were analyzed. The results show that the Ortner method could be used to determine the residual stress of the shot peened chunk of nickel-based monocrystalline superalloy. A tensor matrix G could be determined by measuring the plan distance of at least 6 plans. When the diffraction position step precision was 0.3° and the diffraction angle step precision was 0.01°, the expected results of residual stress distribution after shot peening could be obtained.