Wikipedia:Bots/Requests for approval/Commons fair use upload bot 3

Add links
From Wikipedia, the free encyclopedia
The following discussion is an archived debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA. The result of the discussion was Request Expired.

Commons fair use upload bot 3[edit]

Operator: Fæ (talk · contribs · SUL · edit count · logs · page moves · block log · rights log · ANI search)

Time filed: 19:49, Wednesday January 7, 2015 (UTC)

Automatic

Programming language(s): Python

Source code available:

I have this working passively locally, but yet to test out on labs with sample images. When I have significant updates to the code, I will consider uploading a new version of the source under https://github.com/faebug as the wikigit repository is unlikely to be maintained. Migrated version of code on github link above, test version only.

Function overview:

This is a cross-wiki bot to copy files at risk of deletion on Wikimedia Commons to local wikis where they can be retained under either fair use or the image is public domain in the source country but may be problematic under Commons interpretations (such as the URAA).

Links to relevant discussions (where appropriate):

Edit period(s):

  • Previously running hourly, without any issues, so I'm planning on doing the same.

Estimated number of pages affected:

  • The bot was successfully running prior to the toolserver shutdown, coincidentally the last transferred files were some of my own. See ListFiles.

Exclusion compliant : Yes


Already has a bot flag : No Effectively this was reset by the usurp process.

A trial may not be needed considering the track record, however If there is one, I would prefer it to be a month or longer as my availability may be patchy.

Function details: This bot re-uploads files that are deleted on Commons to projects where they are in use, if those projects accept non-free files. It goes over the files in Category:Pending fair use deletes, uploads them to local wikis, then marks them for speedy deletion on Commons when it's done. Any article using the images receives a notice that the file has been re-uploaded as a fair use candidate. Local wikis are responsible for determining if the re-uploaded image is eligible for their non-free content policy, and deleting it in a timely manner if it is not. If for some reason it's not able to upload the image, it will leave an error message on the file page and not mark it for deletion.

Discussion[edit]

Arbcom exemption/requirements
Arbcom request
The following discussion has been closed. Please do not modify it.
As you are "topic banned from images relating to sexuality" (exempting images prior to AD 1000), describe the controls you are enacting on your bot to maintain compliance with this ban. — xaosflux Talk 00:56, 8 January 2015 (UTC)[reply]
I am not attempting to add controls to discriminate against image content. The bot does not add images to articles, neither does it add new images to Wikipedia. This maintenance process is transparent to the reader and most editors.
Arbcom unanimously supported the motion:
Fæ is permitted to operate bot accounts, edits from which are only to be made in accordance with Bot Approvals Group approved tasks, or an authorised trial of one.
In my request I also explained what the Fair use upload bot does, and they had no issue with that scope.
The bot does not make any other value judgement about the content of the image, neither do I as the bot operator have any editorial decision to make with regard to the images being localized, nor do I chose the images which are marked on Commons for localization (a Commons admin will do this when assessing a DR, non-admin additions get turned into requests for an admin to act on).
In practice, it would not be possible to avoid localizing images relating to sexuality as it is too broad and conceptual a scope to interpret against existing categories. So, in permitting me to operate this bot, it would not be logical for Arbcom to have expectations for its operation that make it impossible to operate without it becoming a manual process rather than an automated one.
As a belt and braces step, if you wish to email Arbcom-l to confirm this reading, please do. --Fæ (talk) 11:33, 8 January 2015 (UTC)[reply]
Sounds reasonable, I listed Wikipedia:Arbitration/Requests/Clarification_and_Amendment#Clarification_request:_F.C3.A6.2FR2.1_Bot_Edits to ARBCOM. — xaosflux Talk 14:38, 8 January 2015 (UTC)[reply]
Arbitration Committee clerk note: The Committee is currently voting on a motion related to this request, please see WP:ARCA#Motion (Fæ). For the Arbitration Committee, Callanecc (talk • contribs • logs) 03:27, 13 January 2015 (UTC)[reply]
  • Any trials are on hold pending the clarification, however any other discussions regarding this request should proceed below. — xaosflux Talk 14:38, 8 January 2015 (UTC)[reply]
Note: ARBCOM has begun voting on a motion to specifically grant a topic ban exemption for this bot operations, will update here following the motion. — xaosflux Talk 18:43, 11 January 2015 (UTC)[reply]
  • The Arbitration Committee has passed the following motion which related to this request for approval:
Despite the restrictions on his editing images related to sexuality, Fæ may operate the Commons fair use upload bot if the Bot Approvals Group approves it.

The bot may upload sexuality images that would, if Fæ himself had uploaded them to the English Wikipedia, breach Fæ's restriction, only if the upload is requested by a third party.

The bot shall maintain a log of: the images it uploads; the names of the articles on the English Wikipedia where the images appear at the time of upload; and the username of the Commons editor requesting the transfer to the English Wikipedia.

For the Arbitration Committee, Callanecc (talk • contribs • logs) 01:24, 15 January 2015 (UTC)[reply]
Bot discussion
  • Can you please indicate on the local userpage who owns the account? Courcelles 22:26, 7 January 2015 (UTC)[reply]
    • Good point. Done, rather than relying on visiting the Commons page. --Fæ (talk) 22:35, 7 January 2015 (UTC)[reply]
  • After playing around with the bot locally and having it fall over a few times, I am planning to rewrite it to rely on pywikibot rather than mwclient as its interface to the API. This will probably work far more reliably on WMFlabs and be much easier to maintain in future years. Though the code is not all that long, with other commitments and the increased testing needed, this will take weeks rather than a few days. --Fæ (talk) 08:49, 12 January 2015 (UTC)[reply]
    Note, some 'real' images are ready for the bot to localize, see Commons:Deletion requests/Files uploaded by SPVII DrFresh26. I'm advising that the bot should be operational within a week or two.
    The account commonsfairuseupload has been set up on labs. I have a test version running under Pywikibot core on WMFlabs, however there is a fair amount of rewriting to be done before running it live and it makes sense to put a first snapshot up on github. --Fæ (talk) 23:25, 13 January 2015 (UTC)[reply]
    Early snapshot now on github as above. --Fæ (talk) 13:35, 14 January 2015 (UTC)[reply]
    A separate bot flag restoration request has been raised on Commons, c:Commons:Bots/Requests/Commons fair use upload bot. --Fæ (talk) 12:52, 15 January 2015 (UTC)[reply]
  • This bot would just perform the same task as User:Dcoetzee's bot, right? How will the bot handle Bugzilla:61656? Dcoetzee's bot handled this by reverting CommonsDelinker, see e.g. Special:Diff/615048249. Ideally, this should be fixed in CommonsDelinker instead of the fair use upload bot, but nothing seems to have happened in CommonsDelinker since the bug was reported in 2010. --Stefan2 (talk) 14:53, 20 January 2015 (UTC)[reply]
    To be honest, I am not 100% sure I understand the issue, not having looked into the functionality of the delinker (note the bug was reported in 2010, but Dcoetzee's code was successfully running from 2012 to 2014 the way it was). However, the way the CFUUB behaves at the moment is that it locally uploads the file under an amended file name and inserts a redirect as the old local image page text. This should leave the old name untouched to avoid permission problems on local wikis. My understanding is that this precautionary step also avoids possible conflict with the delinker when the original is speedy deleted from Commons. If folks want this to work differently, then this might be something to amend in the delinker's behaviour, rather than building in odd intelligent reverts into CFUUB to undo the work of the delinker.
I have yet to convert this bit of code to pywikibot, but if you look in the current test status source code linked above for the two places that site.upload(open('/tmp/downloadedfile'), newfilename, newdesc, ignore=True) occurs, these are relevant.
As I am regular dialogue with @Steinsplitter:, I would defer to his judgement as he has recently been active in updating the delinker and would welcome his advice during testing. Perhaps he could take ownership of this bug request too? I could do with some test images, so maybe we can agree on a few and demonstrate the system in the trial period. --Fæ (talk) 16:28, 20 January 2015 (UTC)[reply]
When Dcoetzee's bot uploaded files, it worked like this:
  1. Someone on Commons requested a local upload
  2. Dcoetzee's bot uploaded the file under a slightly different name by inserting "from Commons" in the file name to avoid permission problems
  3. The bot created a redirect from the old name to the uploaded file
  4. The file was deleted on Commons by a Commons admin
  5. CommonsDelinker failed to notice that a redirect existed locally and therefore incorrectly removed the file from English Wikipedia
  6. Dcoetzee's bot reverted CommonsDelinker's incorrect removal
I think that step 5 should be fixed by correcting the bug in CommonsDelinker, but Dcoetzee decided to fix it by introducing step 6 because the CommonsDelinker programmers didn't fix the bug for several years. There is some discussion in Wikipedia:Bots/Requests for approval/Commons fair use upload bot 2, for example in the "Function details" section. If Steinsplitter can fix CommonsDelinker, then that would be much better. --Stefan2 (talk) 16:49, 20 January 2015 (UTC)[reply]
Agreed. I'll pay attention to testing this out based on the fact that Steinsplitter believes this bug has been addressed in Magnus' new version of the delinker (Phabricator:T63656). --Fæ (talk) 18:16, 20 January 2015 (UTC)[reply]
See my comment here --Steinsplitter (talk) 18:17, 20 January 2015 (UTC)[reply]

As best I can tell, there's no reason to delay a trial. Is that the case? Josh Parris 06:35, 20 February 2015 (UTC)[reply]

I'm considering putting some time aside to trial the code in about a week. --Fæ (talk) 09:44, 20 February 2015 (UTC)[reply]
My understanding is that you intend to monitor closely, but this is a rewritten bot. I'm also under the impression that there won't be a huge number of edits. As such, Approved for trial (30 edits or 30 days). Please provide a link to the relevant contributions and/or diffs when the trial is complete., commencing sometime in the next couple of weeks. Josh Parris 19:24, 23 February 2015 (UTC)[reply]

Any news on that? Trial period has expired. -- Magioladitis (talk) 22:08, 22 March 2015 (UTC)[reply]

I have pushed the code forward a little bit this weekend. A family issue has taken priority. I am writing in a reasonable test mode, which I think will help for adding more wikipedias downstream. 30 days was obviously a bit of an aggressive target for my availability. I would expect to be able to run this live in April. --Fæ (talk) 23:04, 22 March 2015 (UTC)[reply]

Fæ Hi again! Any news on that? -- Magioladitis (talk) 22:17, 3 May 2015 (UTC)[reply]

Hi, I'm sorting out clearing a house and the admin burden of much delayed probate, which is eating up my free time. Sorry about these delays. A couple more weeks before I can focus again? --Fæ (talk) 22:35, 3 May 2015 (UTC)[reply]

Fæ? -- Magioladitis (talk) 11:13, 21 May 2015 (UTC)[reply]

Yes still tied up with real life (which include about 25 shopping bags full of unexpected correspondence to check over in relation to a sizeable fraud). If anyone else is keen to get the bot running earlier, I'll happily liaise as a co-operator, otherwise it may be another month yet before I put aside time to test it out. --Fæ (talk) 12:45, 31 May 2015 (UTC)[reply]

This is open for 6 months... Is here is something is should be done here? -- Magioladitis (talk) 08:51, 22 June 2015 (UTC)[reply]

The fraud investigation is a magnitude larger and more complex than expected, dating back to 2007 with police, lawyers and council employees to chase around. I am interested in getting the FUUB working, but I'm happy to turn the job over to someone else if they have time to finish it now rather than waiting longer. --Fæ (talk) 10:30, 22 June 2015 (UTC)[reply]

Fæ I suggest that we close this BRFA so that I reduce the backlog and that you resubmit when ready. -- Magioladitis (talk) 12:58, 22 June 2015 (UTC)[reply]

No problem with that approach. If a bot writer wants a crack at it in the meantime, drop me an email. --Fæ (talk) 13:18, 22 June 2015 (UTC)[reply]

Request Expired. -- Magioladitis (talk) 17:03, 27 June 2015 (UTC)[reply]

The above discussion is preserved as an archive of the debate. Please do not modify it. To request review of this BRFA, please start a new section at WT:BRFA.
Retrieved from "https://en.wikipedia.org/w/index.php?title=Wikipedia:Bots/Requests_for_approval/Commons_fair_use_upload_bot_3&oldid=841912334"