Wikipedia talk:Community health initiative on English Wikipedia/Administrator confidence survey

Welcome to our discussion!
When participating in this discussion, please remember the following: Debate ideas, not people. Avoid discussing or linking to specific examples of harassment or user misconduct. To participate privately, contact our team via email. (Your email may be shared internally but never publicly.)

It depends on how you look at it

I read:

The Wikimedia Foundation Anti-Harassment Tools Team is conducting a survey to create measurements on how to better support admins who are addressing disputes on Wikipedia. This includes measuring how well current tools, training, and information exists for admins in recognizing and mitigating things like sockpuppetry, vandalism, and harassment. This is specifically designed to learn more about and improve the admin experience.

I don't remember ever having been trained. Neither do I remember ever having trained anybody else.

But then, many years ago when I was asked if I'd mind being nominated for admin, if I'd been told that this would involve being trained, I'd have declined. And now, if I were asked to train, I'd decline.

I'm asked about the degree to which I agree with:

Wikipedia has provided me enough resources to solve, mitigate or intervene in cases of sockpupetry [sic].

This is the kind of question whose answer depends enormously on how I read it. Which itself depends on my mood, etc. Does "sockpuppetry" mean acknowledged sockpuppetry? If so, then yes, I can block every perp; but no, a new perp can pop up at any time, and I don't receive email helpfully telling me: "Two of the seven IP numbers previously used by RighteousWarrior81 and RighteousWarrior86, both of whom you blocked, are now being used by RighteousFighter21. Care to take a look at this UID's contributions to the article 'Kosovo'?" And so I can hardly start to solve determined socketpuppetry. OTOH anyone can intervene (a near-meaningless word): certainly the admin bit isn't required. As for suspected sockpuppetry, the constraints are on time and energy. I can't think of any Wikipedia/Mediawiki tool that might help me in the (usually tedious) job of going through numerous article histories, etc. Anyway, I interpreted the question, erm, booleanly; Yes I'm provided with enough resources to intervene; therefore I'm provided with enough resources to intervene OR do anything else; therefore yes. Which probably misses the point of the question. -- Hoary (talk) 00:39, 14 September 2017 (UTC)

Hi there! So training refers to manuals and pages on Wikipedia people written about being admin. I'm interested in seeing if some people are interested in alternative methods of training? As for rating, and answering the questions- I would say answer to the best of your ability? The 'accuracy' or help of the tools can differ day to day, hence us providing a range. My inclination is if you find yourself wavering between somewhat helpful and not helpful at all is that the tool may not be helpful? Also thank you for this comment- you noted how sockpuppetry is on time and energy, and that is so helpful. All of this is about my team and I trying really investigate and understand better solutions to these problems.

--CSinders (WMF) (talk) 19:31, 14 September 2017 (UTC)

Regarding sock puppetry, it seems obvious to me that the relatively small subset of administrators who have CheckUser privileges have substantially greater resources to solve, mitigate or intervene than those who don't have this privilege. Confidence will vary in admins' ability to effectively use "behavioral criteria" to make a finding of sock puppetry (aka the "duck test"). Arguably administration of the duck test, and allegations of sock puppetry based on "behavioral criteria" can sometimes be a form of harassment. One way to mitigate such potential harassment is to eliminate the need to present behavioral evidence before running a Special:CheckUser, by setting up CheckUser bot that automatically ran such checks on all new users, and generated a report for admins with the CheckUser privilege to act upon by either blocking the new user and its sockmaster or whitelisting in the case of valid alternative accounts. This isn't the place to discuss the idea, but I'd appreciate a link to any related past discussions of the idea. I'm not keen on the idea of sock blocks based entirely on behavioral evidence, and more effective blocking based on hard evidence could mitigate the need for behavioral blocks. wbm1058 (talk) 15:34, 18 September 2017 (UTC)

@Wbm1058: Hello, and thanks for jumping in the conversation! Yes, I agree that CheckUsers will (and should!) have higher confidence than other admins. This survey is supposed to check a high-level "pulse" of the administrators on Wikipedia, but next time we run it we should consider adding in the option for respondents to optionally identify if they are also CheckUsers, ArbCom, etc. The survey doesn't mention this explicitly, but I consider that a proper response to any forms of these harassment are to delegate/redirect to appropriate parties.

Our team often talks about how we can build tools that de-escalate or avoid common scenarios before they boil over into large incidents. Sockpuppet witchhunts are certainly one of them. The WMF's Research team is looking into building some models to identify sockpuppets based on previously identified cases. This work is tracked on Phabricator at T171635, and work should start in October 2017. Following this work our Anti-Harassment Tools team will work closely with local and global CheckUsers and any other interested users to build new "robust sockpuppet identification tools." This is intentionally vague because we do not want to prescribe solutions until we've done more research, but I certainly like the idea of a “potential sockpuppet bot” for CheckUsers, which would remove the burden from others. This work is tracked on T166813, and work should start in late 2018. — Trevor Bolliger, WMF Product Manager 🗨 16:55, 18 September 2017 (UTC)

The survey misses a substantial issue regarding admin confidence

I've just submitted my response. I have to say, the survey isn't exactly what I expected, in particular it completely neglects the issue of retaliatory harassment. I believe you should have asked:

1. How often have you been harassed as a result of taking admin actions?
   1. Briefly describe the circumstances -- topic area and circumstances and admin action(s) undertaken (e.g. blocking for 3RR violations on American politics related article), vector of harassment (select all that apply: user talk page, on-wiki conversations, Special:EmailUser, off-wiki email, ...), form (e.g. personal attacks, death threats, legal threats, frivolous reports to noticeboards, nasty real life stuff, ...) [STRICTLY OPTIONAL]
   2. Have you/the community been able to stop it?
2. Do you hesitate or choose not to undertake admin actions because of the potential for harassment?
   1. If yes, under which circumstances do you hesitate/withhold your participation (e.g. do not deal with 3RR cases at all, refrain from applying for checkuser privileges, or avoid $TOPIC completely)?
3. Do you believe you have the tools, information and strategies necessary to defend yourself from said potential harassment? What can the WMF do to help?
4. [free form input]

It's all well and good that admins are able to recognize sockpuppetry, vandalism, wikihounding and harassment and take action against them, but we need to (1) be able to protect ourselves from any blowback that may result and (2) be confident in this ability. MER-C 05:01, 14 September 2017 (UTC)

Yes, it's a shame this was missed. Doug Weller talk 14:19, 14 September 2017 (UTC)

Good point. I was also surprised to see that there was no opportunity to give extra feedback regarding my answers. ​—DoRD (talk)​ 15:33, 14 September 2017 (UTC)

Thank you so much for the comment- the Anti-Harassment Team is in close contact with SuSa about harassment. A lot of my research and one on one interviews is covering the chilling effect and silencing effect of harassment. I'd love to chat with you off Wiki about this- we are super aware of admins not wanting to wade into cases because of harassment. This survey is a more hyper focused survey on training and tool creation for different kinds. We will be doing more studies, surveys, and interviews specifically on harassment against admins and this is something we are super concerned about. --CSinders (WMF) (talk) 19:33, 14 September 2017 (UTC)

In this context, it would also be interesting to see whether differences exist between admins who are open about their IRL identity and those of us cowardly hiding behind pseudonyms. —Kusma (t·c) 20:14, 14 September 2017 (UTC)

Thanks for the survey - I thought it was well designed, and certainly got me thinking. I provided some free text in the final field which goes to the issues raised by my admin colleagues above, and would be pleased to take part in further discussions of this issue. The chilling effect which harassment of admins has on their willingness to intervene to stop other editors from being harassed is a key issue. Nick-D (talk) 09:42, 15 September 2017 (UTC)

Added question to provide space for free form response

Hello

In response to the suggestions on this page, we are adding a final question that allows for a free form response about other barriers to successfully addressing cases of vandalism, sockpuppetry, wikihounding, and vandalism. The question will be automatically included if you have not filled out the survey form that I send out.

If you have already answered the Admin Confidence survey and want to answer the last question, you can email me and I'll send you the Final Question in a separate email.

Please don't fill out the full survey a second time! SPoore (WMF), Community Advocate, Community health initiative (talk) 20:29, 14 September 2017 (UTC)

Administrator confidence survey closed on Sept 24 2017

Hello,

The Administrator Confidence survey closed on Sept. 24. The Anti-harassment tools team will analyse responses and share the results in approximately 2 weeks. SPoore (WMF), Community Advocate, Community health initiative (talk) 14:34, 25 September 2017 (UTC)

I noticed a number of admins got a notice about this, but I didn't, so I assumed my input wasn't really required (because I've never been harassed, or if I have I've ignored it until they've lost interest) - was this open to everyone or "invite only"? Ritchie333 ^{(talk) (cont)} 14:19, 26 September 2017 (UTC)

The survey was open to every admin on English Wikipedia but nobody was obliged to participate, so no worries about missing it. In addition to posting about the survey on WP:AN we also used MassMessage to leave talk page messages on randomly selected admins, simply to reach more people so we would reach a representative sample of 100 responses. — Trevor Bolliger, WMF Product Manager 🗨 17:26, 26 September 2017 (UTC)

No, my point was as I didn't think the survey was targeted for me, I didn't have a look, until one day I thought "I wonder what why lots of other admins have got this? Am I missing out on something?" Then I found signup closed, so you don't know what my views are. Ritchie333 ^{(talk) (cont)} 13:53, 2 October 2017 (UTC)

The results are in

The Wikimedia Foundation Anti-Harassment Tools team Wikipedia English Administrator Confidence Survey results are in. Thank you to the 117 participants who filled out the survey. :-)

A copy of the raw results can be found on the results page. Initial impressions of the raw results are welcome on this talk page.

On Monday, October 2, 2017, the Anti-Harassment Tools team will share further results from the comments section of the survey and our preliminary analysis. The team wants to work with the community to identity significant findings and how that it could influence our team's work on tool development.

Later in October, we will have second discussion that will focus on the finding in the survey comments sections that are directed towards policy changes or different ways of reporting and managing cases.

Again, thank you for participating in the survey. And look forward to discussing the results on Monday. For Wikimedia Foundation Anti-Harassment Tools team, SPoore (WMF), Community Advocate, Community health initiative (talk) 19:19, 28 September 2017 (UTC)

Analysis?

Regarding the results, is this a first draft? Is the WMF planning on any further analysis of this data? I'm sure there are many ways to look at it, but as an example, I'm interested in (and disappointed to not find) some analysis/discussion of correlation between admins' confidence in identifying sockpuppetry versus their sense of having the tools to respond to it. I mean, it's great that 53.8% of respondents rate their confidence high (4 or better - my metric) and coincidentally that the same number feel they "usually" or "almost always" have the skills/tools to intervene, but what is the relationship between those two groups? Do more confident respondents also agree they have the tools, or are they more likely the ones who feel unsupported? That's the kind of thing I'd like to see, and which I think would be more useful to the Foundation or whoever is using this data. Ivanvector (^Talk/_Edits) 19:42, 28 September 2017 (UTC)

Yes, we are planning on doing and releasing more detailed analysis next Monday. See post about it on this page. We are very open to further discussion about the survey questions, the finding, analysis, and other ways to research and use these results. SPoore (WMF), Community Advocate, Community health initiative (talk) 19:46, 28 September 2017 (UTC)

Great! I look forward to your next report. Ivanvector (^Talk/_Edits) 20:37, 28 September 2017 (UTC)

Text version of results

Would it be possible to release a text version of the results (as in wikitext, say)? I can't deal with the PDF with my screen reader. Thanks! Graham87 13:15, 29 September 2017 (UTC)

Graham, Sure, I'll explore our options for releasing in a text format, and get back to you when it is ready. SPoore (WMF), Community Advocate, Community health initiative (talk) 13:33, 29 September 2017 (UTC)

@Graham: @SPoore (WMF): — I've added a wikitext plaintext version at Wikipedia:Community health initiative on English Wikipedia/Administrator confidence survey/Results#Wikitext. — Trevor Bolliger, WMF Product Manager 🗨 15:54, 29 September 2017 (UTC)

@TBolliger (WMF): Thanks very much. You pinged the wrong Graham though. :-) Graham87 03:25, 30 September 2017 (UTC)

Discussion about Admin Confidence survey results related to tech tools

Hello :-)

The Anti-Harassment Tools team reviewed the feedback from the Admin Confidence Survey and did a preliminary analysis of the comments related to tech tools. We are sharing these results today so that the English Wikipedia community can begin discussing the results.

In two weeks the Anti-Harassment Tools team will release more preliminary analysis about the survey comments related to policy, training, and reporting methods.

Again, thank you to everyone who participated in the survey. Whether you participated in the survey or not, we are interested in your thoughts about the results.

If you still would like to provide comments privately to the Anti-Harassment Tools team, you can email the Anti-Harassment Tools team.

For the Anti-Harassment Tools Team, SPoore (WMF), Community Advocate, Community health initiative (talk) 00:46, 3 October 2017 (UTC)

Discussion related to ideas about tech tools

Please share your reaction and thoughts about the survey.

"Block user by page" (or variations of that) is my number-one requested feature, as it could stop so much drama and disruption on the site. It would mean a user going over 3RR on a single page, but otherwise having a stretch of positive contributions, could be blocked from the page. It would also mean admins would think twice about a "global" block, and would have to justify why they couldn't block from a page instead. Also, a rangeblock on a page could be used as an alternative to semi-protection. Overall, there are a huge wealth of positive possibilities. Ritchie333 ^{(talk) (cont)} 12:32, 3 October 2017 (UTC)

Consultations on this feature have begun: see Wikipedia:Community health initiative on English Wikipedia/Editing restrictions and associated talk page. MER-C 13:00, 3 October 2017 (UTC)

Update of survey results to include a summary of non-tool related topics

The results page for the Administrator Confidence Survey has been updated to include comments from the survey about policy, reporting, harassment, and community culture.

Thank you for taking the time to give this feedback. And especially thanks to those of you who requested a space to give free form comments.

Major themes include: Harassment policy as written interferes with enforcement, Social barriers to addressing harassment, Chilling effects of harassment on administrators' work, Victim blaming, More training and resources needed, Time intensive, Specialization, AN/I related, and the Role of WMF.

The Community Health Initiative team is using the survey comments to guide our prioritization for tool development and plan next steps for research around the topics of harassment and conflict resolution. We are interested in hearing your thoughts about the results and your ideas about how they should influence future decision making. SPoore (WMF), Community Advocate, Community health initiative (talk) 19:55, 28 November 2017 (UTC)

Discussion related to comments about policy, reporting, harassment and community culture

• The "Summary" of comments is one-sided, apparently to advance an agenda. ALL comments should be presented in raw form. Carrite (talk) 15:18, 10 December 2017 (UTC)