Diamonds are actually quite rare. Also it is true that the process of extracting diamond is quite laborious (mines move many tons of dirt per carat of diamond found) and that gem-quality diamonds are relatively few (only about 1 in 1 million diamonds are quality one carat stones, only 1 in 5 million are 2-carat; and 1 in 15 million are 3-carat).
The prices of diamonds increase along with the inflation rate. In some periods, the demand is higher than the supply whilst in other periods this is reversed. In the end, there is always a balance.