Hello fellow Judges.

Over the past month I've conducted a study on the topic of uncertification, which I now share in the following article - "Lapsing and the Judge Program“.
The article includes both quantitative (how long does it take to lapse on average) and qualitative (the reasons which cause judges to lapse) data.
While I was told it's a ”harsh truth" kind of writing, the TL;DR is that it's nothing dire for the program, but we can do better.
So for the very least it should prove an interesting reading, and if you're facing hardships in maintaining your level, or are a level 2+ judge planning to certify someone, I hope it will give you some pointers to think about.
Naturally this is the place for discussion with all your ideas and feedback.

Fascinating article! I wanted to ask about one aspect of it. The article mentions:

While maintenance requirements for level 1 only include a sanctioned event once every 6 months (lately also a yearly exam)

I don't see that a yearly exam is required to maintain Level 1 under the maintenance requirements here:
https://blogs.magicjudges.org/o/judge-levels/

Hi Sherwin,

On the page you have linked, there is the following maintenance requirement:

“Demonstrate maintenance of rules and policy knowledge through yearly exams. Low performers will be referred to their Regional Coordinator for followup.”

Very interesting article! Thank you for making this analysis and for
sharing its results!!!

Aru

Originally posted by Milan Majerčík:

Hi Sherwin,

On the page you have linked, there is the following maintenance requirement:

“Demonstrate maintenance of rules and policy knowledge through yearly exams. Low performers will be referred to their Regional Coordinator for followup.”

Thanks I see it now!

In the article you said you sent the survey to over a hundred people but only got a little more than 40 responses.

Do you have any theories regarding the other 60% that did not respond?

Originally posted by Bryan Prillaman:

Do you have any theories regarding the other 60% that did not respond?

Most likely because they didn't feel like it :)
Other factors might be changed Email addresses that never arrived their destinations, or people who meant to reply but forgot.
I can tell you that about a week after the original survey I sent out a reminder for non-responders and got several more replies out of it.

One person replied as follow:

Originally posted by Person:

Zohar,
i'm not gonna answer the questionnaire, mainly because i don't have the time to sit back and start to answer it.

Besides, a 42% response rate isn't bad at all.

42% of replies from ex-judges that left the program? I'd say not bad at
all! I just wouldn't discard that those that don't care anymore for the
program would be a majority among the 58%.

My point behind the point, is that those ~60% may have left for other reasons
Such as:
-I was certified but never had a place to judge so I never did it.

There is an assertion in the article that we are good with acquisition but poor with retention. What if it’s the other way around? We are actually good with retention, but excessively aggressive with acquisition, where we certify way more than can be sustained.




———————————————
This space intentionally left blank

Originally posted by Bryan Prillaman:

My point behind the point, is that those ~60% may have left for other reasons.

Statistically speaking, the sample of 46 people should be representative of the population. There's no real reason to assume that the “composition” of the other 60% is much different.
If that was the case, then all those surveys taking place worldwide, with only 10%-20% response rate, would be useless.

Originally posted by Bryan Prillaman:

There is an assertion in the article that we are good with acquisition but poor with retention. What if it’s the other way around? We are actually good with retention, but excessively aggressive with acquisition, where we certify way more than can be sustained.

That's an interesting idea. Hmm… I don't believe the findings support that - Lack of opportunities to judge only accounts for less than 20% of the reasons provided. If there were more judges than could be sustained, then I would expect those percentages to be higher.
However, there could be specific places in the world where that indeed might be the case.

Edited Zohar Finkel (April 13, 2018 05:58:15 PM)

Anecdotally, I tend to find that acquisition is higher in areas with an already established judge community and that effect snowballs rapidly. Sometimes that is managed naturally through people moving (especially in student areas) but can often lead to difficulties retaining judges long term.

An interesting area for follow up would be to compare the lifespan of judges with players and officials in similar hobbies. More demographic information about the make up of those that lapse would also be interesting. Again, anecdotally, I tend to see a higher turnover rate in younger (sub-university/college age) judges than older ones as well as a higher rate amongst judges who have only been players for a very short period of time.

I wonder if there is some value in taking a more active view of this sort of population data across the program to help manage membership, development and growth.

Originally posted by Zohar Finkel:

Statistically speaking, the sample of 46 people should be representative of the population. There's no real reason to assume that the “composition” of the other 60% is much different.
If that was the case, then all those surveys taking place worldwide, with only 10%-20% response rate would be useless.

That's not really how sampling works.

I don't want to disparage the work you put in here on some really interesting findings, but I would recommend against trying to apply it to the broader universe of lapsed judges. Instead, just take it for what it is - of the people who responded, here's what they said.

Zohar is right. You didn’t sample the people to determine who responded. They self-selected.

Originally posted by John Brian McCarthy:

That's not really how sampling works.

I'm willing to give you that a sample size of 46 people isn't too big, so it might not reflect the entire population of lapsed judges accurately. For example the number 1 reason for lapsing might only be number 2, and there are certainly corner cases which didn't show up among the 46 participants, but in general that's that.
Adding more people to the sample should give you more of the same.

Originally posted by Bryan Prillaman:

What if it’s the other way around?

By the way Bryan, I haven't said it before, but with you being the guy in charge of everything Level 1 related (definition, certification, and maintenance requirements) I'm very glad you're showing interest and challenging the article, since in a way it's meant for you.
If my findings will help you improve something in the process, then I did my part.
So while I don't have all the answers, feel free to inquire some more.

Originally posted by Zohar Finkel:

Statistically speaking, the sample of 46 people should be representative of the population. There's no real reason to assume that the “composition” of the other 60% is much different.

That isn't correct at all. Those aren't 46 random people, those are 46 people who chose to respond to your survey. It's actually quite likely that the other 60% would have given different answers about some things had they been forced to respond.

https://en.wikipedia.org/wiki/Selection_bias