Rice seed size is an important agronomic trait in determining the yield potential, and four seed size related genes (GS3, GW2, qSW5/GW5 and GIF1) have been cloned in rice so far. However, the relationship among these four genes is still unclear, which will impede the process of gene pyramiding breeding program to some extent. To shade light on the relationship of above four genes, gene expression analysis was performed with GS3-RNAi, GW2-RNAi lines and CSSL of qSW5 at the transcriptional level. The results clearly showed that qSW5 and GW2 positively regulate the expression of GS3. Meanwhile, qSW5 can be down-regulated by repression of GW2 transcription. Additionally, GIF1 expression was found to be positively regulated by qSW5 but negatively by GW2 and GS3. Moreover, the allelic effects of qSW5 and GS3 were detailedly characterized based on a natural population consisting of 180 rice cultivars. It was indicated that mutual interactions exist between the two genes, in which, qSW5 affecting seed length is masked by GS3 alleles, and GS3 affecting seed width is masked by qSW5 alleles. These findings provide more insights into the molecular mechanisms underlying seed size development in rice and are likely to be useful for improving rice grain yield.