Hello fellow Judges.

I've written an article about Europe-East Exemplar Metrics, and even though it's mainly for my region - it talks about many global metrics as well.

It's not the easiest of read and involves some statistics, but I know many people are interested in such metrics, so if for example you'd like to know what fraction of the judge community has been recognized, who are the all time feature exemplars as well as nominators, how things look among the different levels and so on, you're welcome to have a look at it.

Hi Zohar, thanks for the article!

The stats around percentages of judges that have been recognized by level is really interesting, and is much higher than I expected. Do those numbers hold up roughly worldwide, or is Europe-East a bit of an outlier in how broadly the exemplar program reaches?

Hey Chase.
To answer such a question with certainty would require a worldwide list of judges (including regions and levels) to compare the recognitions database against, and that's a list which I do not have. I was able to manually create one for my region, but for the entire global community is too much for me.
My guess would be that they hold up, but without concrete information that's exactly what this is - an educated guess.

I can tell you however that when it comes to L3s (I've also created a list of all L3s) about half have more L2 recognitions than L3 ones, and for the other half it's the other way around.
The half that is recognized more by other L3s includes judges in key roles within the judge program such as Toby Elliott, Daniel Kitachewsky, Alfonso Bueno & Kim Warren, while the half that is more recognized by L2s includes judges like Hector Fuentes, David Rappaport, Carlos Rada & Kevin Binswanger. I assume the later names are less familiar (I know they are to me), and that's because they are being recognized by judges they've worked closely with, who they've mentored and lead during tournaments. Meanwhile the people in key roles are being recognized for mentoring judges to L3, head judging GPs, etc, which would require other L3s to notice, and is usually beyond the scope of a region.

Hey, I have a master list just like that which I use for Exemplar address stuff!

A quick run through a spreadsheet says:
…………………….Total…..By L3s….By L2s
%L3s Nominated…99.37%…50.58%…49.42%
%L2s Nominated…92.93%…13.53%…86.47%
%L1s Nominated…59.55%…21.08%…78.92%

So roughly true. One L3 in the world has gotten zero nominations. (Aaron Hamer, who is I think listed as an L3 by fiat, seeing as how he is contracting at WotC right now.)

I actually used a different counting method than Zohar, looking at slot type rather than level of nominator, because levels change. I counted “Deferred” as an L3 slot because the numbers don't change much no matter what you do with deferred slots, with only 142 nominations using that slot overall. Wave 9 will probably throw a wrench into my laziness there. I could fix that, but it would require some reworking I did not want to do for my calculations right now.

Almost all L2s have been nominated worldwide. The majority of L1s in JudgeApps have been nominated worldwide. Most of these nominations are by L2s.

Note that the numbers won't add up to the total - the total is the total fraction of all judges of that level nominated. The “by level” percents are slot-specific, the total number of nominations of that slot type used on that judge level, and those two should sum to 100% or within a rounding difference.

Oh, and note also I hand-extracted the Pilot Wave 1 slots from the Exemplar blog, so my numbers include those L3-specific slots, where I don't believe Zohar's did. (No slight on Zohar, without a master list of all judges, extracting that data is probably not meaningfully doable, and it took a LOT of hand massaging for honestly not a ton of value to get those into a usable form.)

Also, this article is awesome, and I appreciate it a lot.

—
Rob McKenzie
Magic Judge Level III
Judge Regional Coordinator USA-North
Minnesota

Big like on all of this Rob, plus it proves my gut about level distribution.
I'm only puzzled about the total percentage of judges that have been recognized - I got 55% (3,917 out of 7,098) and you say 60+%. A 5% gap is quite a lot, so somewhere my numbers are probably too way off. Are there less than 7,000 certified judges? Adding in wave 1 made the difference? Your different counting method? What?

Edited Zohar Finkel (April 13, 2017 08:37:34 PM)

Hmm. I actually goofed some numbers up a little, because I had L1s nominated and not nominated switched in my fractions.

Here are some better numbers:


There are over 7000 judges of L1, L2, and L3 on JudgeApps. I reversed the L1 fraction, so L1s have 40% of them with nominations.

Does this agree with your numbers better?

—
Rob McKenzie
Magic Judge Level III
Judge Regional Coordinator USA-North
Minnesota

Originally posted by Rob McKenzie:

Does this agree with your numbers better?

Yes it does. This is exactly what I expected - that if there is a gap between what I found and more accurate data, it will be in the other direction.

Originally posted by Exemplar Metrics Europe-East:

55% of all judges. However to take things into perspective, considering over the past 2 years there have been judges who became uncertified, current percentages might be lower.

So having more actual judges than shown on the Apps on one hand, and meanwhile having judges that were recognized but since became decertified on the other hand, makes much more sense in bringing that percentage down.
Thanks!

Edit - added this recent data to the article

Edited Zohar Finkel (April 14, 2017 10:33:38 AM)

So, this might shed some light on penetration over time. I did per-wave penetration and rolled up the penetration, on a per-level basis, for each wave:



L3 penetration has been on a decline since Wave 4. L2 penetration has been on a decline since Wave 5. L1 has been up and down, but the highest penetration ever was this last wave.

Note that this is a slightly deceptive metric, because it is comparing old penetration data to current levels. (I don't have historical level data I could get without a LOT of work.)

I guess the big takeaway here is that with no higher than 60% L2 penetration in any given wave, we have managed to get foils out to all but about 100 L2s.

—
Rob McKenzie
Magic Judge Level III
Judge Regional Coordinator USA-North
Minnesota

Hey there Rob,

Im a little confused. What does penetration mean? The percentage of judges from the total number reconized in that specific wave?

There are two things. One is per-wave, one is the “rollup”, which is from
all the waves up to that point.

So the per-wave is just that wave, and the rollup penetration is how many
people the program as a whole between all the waves had reached.


—
Rob McKenzie
Magic Judge Level III
Judge Regional Coordinator USA-North
Minnesota

Looking at penetration this way is a bit misleading.
Remember that there was a slot increase during wave 4, and a decrease of slots at wave 6 (?), which could explain for the fluctuation. You do have a limited number of slots, and sometimes need to prioritize who you want to nominate.
That's one of the reasons I didn't do a consecutive wave analysis, only mentioned it briefly at the start, plus I didn't think that's the kind of thing most readers would find interesting.