A Combustible Gearhulk enters the battlefield. As the ability resolve, the targeted opponent chose to mill 3 cards.

The 3 cards are Progenitus and 2 Swamps.
Does the Gerhulk deal 10 damages to the opponent ?

Yes, it does.

Combustible Gearhulk will deal damage equal to the converted mana cost of the three cards that it asked you to place into the graveyard. It doesn't matter if those cards happened to end up somewhere other than the graveyard due to some replacement effect, because they're still part of the set of cards the Gearhulk asked you to perform the action on, and therefore are still among the set of “those cards”.

If the Gearhulk wanted to deal damage based on the converted mana cost of whatever group of cards actually ended up in the graveyard after the mill, it would need to say something like “…the total converted mana cost of the cards put into your graveyard this way”, directly referencing the cards that ended up in the graveyard the way something like Liliana's Indignation does.