And (finally) said discussion has happened!
The other two cards will not end up shuffled into the rest of your library, and will instead end up on top of it in whichever order you choose.
Grim Flayer's ability tells you to put any number of the cards you're looking at into your graveyard and the rest on top of your library in any order.
Progenitus's replacement effect alters this instruction, so now you're being told to shuffle Progenitus into your library and put the rest of the looked-at cards on top in any order. While the cards are technically present in the library at the time it's being shuffled, you accomplish this by keeping those cards set aside during the shuffle, and then placing them on top after you're finished, as is done with cards like
Enlightened Tutor and
Lim-Dûl's Vault.