Abstract: Si nano-optical waveguide resonant cavity with low loss and high Q value is the key for high sensitivity detectors, biosensors, optical communication devices, and so on. However, the surface roughness of optical waveguide will cause high transmission loss which becomes a serious constrain to the high Q value of Si nano-optical waveguide resonant cavity. Therefore, it has become a key issue to reduce the surface roughness of silicon-based nanometer optical waveguide for the development of photonic devices. Nowadays hydrogen annealing technology is an important method to reduce the surface roughness of waveguide. According to the theory of surface Si-H bond current density, the simulation study was done by Materials Studio software. The reaction between silicon atoms and hydrogen atoms in the smoothing process of hydrogen annealing was simulated. The reaction transition state was searched. The influence of silicon hydrogen bond and temperature on the reaction process was also studied. The results indicate that chemical bond can be formed between silicon atom and hydrogen atom under high temperature with H2 atmosphere. Higher temperature is benefit to accelerate the moving rate of surface silicon atoms which makes the surface transition from upper state to lower state, and realize its smoothing.