2012-09-13 15:27:34 +00:00
# This file is part of Mylar.
#
# Mylar is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# Mylar is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with Mylar. If not, see <http://www.gnu.org/licenses/>.
from __future__ import with_statement
FIX: undefined on filter box on startup, IMP: Added Meta-Tagging options on a series / issue basis on comic details screen, IMP: Issue Information is now available per issue and is extracted currently from the cbz file to display (if no cbz is present, the option isn't available), IMP: Failed Download handling is implemented and available in GUI - required to replace existing autoProcessComics.py and ComicRN.py scripts, IMP: Added ability to specify post-processing script instead of ComicRN.py, IMP: Added abilty to edit the issue date for a given issue by simply clicking on it - this will help to avoid dates that have incorrect or 0000-00-00, IMP: Story Arc searching is working (not adding yet), IMP: Added Archived/Ignored options to Upcoming/Wanted tab, IMP: Fixed some alignment and display issues on the Upcoming section, IMP: Added better directory handling for Updating Comic Location when it needs to get changed for all existing series (locmove in config.ini), IMP: Added better handling for unicode characters in series titles when searching / filechecking, IMP: When adding a new series with no data, Mylar would error out (now will add and just retain 0 issues), FIX: When year was fuzzied, search would still attempt to do a date-check comparison and say everything failed, IMP: Better handling of nzb names when retaining for post-processing comparisons, IMP: Future Upcoming now will use actual shipping date of issue if available in order to see if issue is available, FIX: If annuals were enabled, refreshing a series would put some issues into an Archived status because the actual counts would be off, IMP: When file checking, Alternate Naming would be searched last which resulted in matching incorrectly to the series title or other alternate naming for the given series, now will check the Alternate Naming first for a match, then drop back down to the series name itself otherwise, IMP: Improved Annual detection when integrated with a given series, IMP: Improved the checking of the future Upcoming list for issues marked as Wanted but not available yet and then auto-adding, IMP: Improved upon story arc checking for missing issues / searching for wanted, IMP: Enabling Annuals support now within Configuration GUI, bunch of other things....
2014-07-28 19:28:09 +00:00
import re
2012-09-13 15:27:34 +00:00
import time
import threading
2015-05-13 02:21:22 +00:00
import platform
2014-01-29 05:06:00 +00:00
import urllib , urllib2
2012-09-13 15:27:34 +00:00
from xml . dom . minidom import parseString , Element
2016-09-07 04:04:42 +00:00
import requests
2012-09-13 15:27:34 +00:00
import mylar
from mylar import logger , db , cv
2016-12-30 13:40:54 +00:00
from mylar . helpers import multikeysort , replace_all , cleanName , listLibrary , listStoryArcs
2015-05-13 01:35:40 +00:00
import httplib
2012-09-13 15:27:34 +00:00
mb_lock = threading . Lock ( )
2015-05-13 01:35:40 +00:00
def patch_http_response_read ( func ) :
def inner ( * args ) :
try :
return func ( * args )
except httplib . IncompleteRead , e :
return e . partial
return inner
httplib . HTTPResponse . read = patch_http_response_read ( httplib . HTTPResponse . read )
2012-09-13 15:27:34 +00:00
2015-05-13 02:21:22 +00:00
if platform . python_version ( ) == ' 2.7.6 ' :
httplib . HTTPConnection . _http_vsn = 10
httplib . HTTPConnection . _http_vsn_str = ' HTTP/1.0 '
2017-11-15 22:49:57 +00:00
def pullsearch ( comicapi , comicquery , offset , type ) :
2014-02-26 19:48:50 +00:00
2017-11-15 22:49:57 +00:00
cnt = 1
for x in comicquery :
if cnt == 1 :
filterline = ' %s ' % x
else :
filterline + = ' ,name: %s ' % x
cnt + = 1
PULLURL = mylar . CVURL + str ( type ) + ' s?api_key= ' + str ( comicapi ) + ' &filter=name: ' + filterline + ' &field_list=id,name,start_year,site_detail_url,count_of_issues,image,publisher,deck,description,first_issue,last_issue&format=xml&offset= ' + str ( offset ) # 2012/22/02 - CVAPI flipped back to offset instead of page
IMP: Changed configuration completely. New config processing, global/config variables changed over entire application. All changes in the configuration GUI take effect immediately after saving - no more need to restart Mylar, IMP: Added provider order sequence header to the top of the Providers tab for improved visibilty, IMP: Added completed download handling for both SABnzbd and NZBGet - will monitor active queue for downloads sent by Mylar and then post-process them accordingly (no more ComicRN), FIX: When recreating a pullist for a week that's not the current week, will now refresh series that are still missing data in case of late population, IMP: Removed loose/explict search options, and search results will now match more accurately to the terms entered, as well as being much quicker, IMP: Changed colour-codes on search results screen, green now indicates that the series is a print series, non-green indicates that it is either a HC/TPB/digital series and will also be indicated as such, IMP: Fixed weeklypull not refreshing some series due to legacy numbering on series deemed as Ended by Mylar, or the pull not refreshing propeprly and thereby were not elgible to be considered as a valid series for matching, IMP: Fixed problem with the scheduler of the weekly pull not running at the designated time, which causes problems when attempting to match series to the pullist for issues, IMP: Changed the autosnatch script so that the get.conf is no longer required to be maintained seperately. The get.conf config items are now located in the main config.ini and have to be repopulated initially, FIX: Fixed problem with the import count not resetting to 0 on subsequent runs of the Import A Directory option, IMP: Changed the sab api key from using the nzbkey to the full api key as is required for completed download handling, FIX: Fixed the retry option for newznab entries so that it should work across the board now for newznabs as opposed to only some providers, IMP: some minor code cleanup
2017-11-10 19:25:14 +00:00
2012-09-13 15:27:34 +00:00
#all these imports are standard on most modern python implementations
FIX: One-off Failed Download handling will now work, IMP: Some better error handling when detecting one-off's during post-processing, FIX:(#1143) When series contained a digit preceded by a dash, would incorrectly assume it as a negative issue number, FIX: Improved being able to detect corresponding annuals on CV when refreshing/importing a series when the annuals are new (ie. no data on CV), FIX: Alt_Pull method for weekly pull list retrieval working again, FIX: Fixed nzbid detection for experimental search, IMP: Mass Import button now available on Import Results screen (will attempt to import all series that are in a 'Not Imported' status), IMP: When searching for arc issues using the 'Search for Missing' option, the call is now threaded so it runs in the background, IMP: Changed IssueYear to IssueDate for Story Arc Issues on the arc details page, FIX:(#1156) Typo that caused error when attempting to view cbz comics in the series detail page, FIX:(#1145) Select All option via top checkbox (on series detail page), FIX: Auto-Want feature via weeklypull will better match to titles that contain 'the' and have hypens in differing character positions, FIX:(#1160) Would incorrectly take the length of a decimal placed issue when searching and never complete the issue number cylcing search, FIX:(#1161) When annuals not enabled, and series in watchlist and series annual in pullist, would error out trying to link series to pullist.
2015-11-18 06:32:40 +00:00
#logger.info('MB.PULLURL:' + PULLURL)
2015-11-23 08:25:59 +00:00
#new CV API restriction - one api request / second.
2017-10-25 18:33:56 +00:00
if mylar . CONFIG . CVAPI_RATE is None or mylar . CONFIG . CVAPI_RATE < 2 :
2015-11-23 08:25:59 +00:00
time . sleep ( 2 )
else :
2017-10-25 18:33:56 +00:00
time . sleep ( mylar . CONFIG . CVAPI_RATE )
2015-11-23 08:25:59 +00:00
2012-09-13 15:27:34 +00:00
#download the file:
FIX: Added user-agent to all CV queries, and switched all queries to utilize the requests module, FIX: Added User-Agent to ComicTagger and switched to requests module for all CV queries, FIX: Removed libraryscan option from GUI as no longer used, FIX: Ability to sort 'Have' column on main index page properly now (by % complete), FIX: Fixed some display issues with the series detail page and displaying, FIX: Metatagging should now be working again for story-arc, and one-off post-processing, FIX: Fixed post-processing problem with decimal places if number was padded, FIX: Updated CV-URL to point to new CV api location, FIX: Fixed problem when file-checking and series contained numbers and/or decimals, would take the modified series name instead of the actual series name resulting in some missed matches, IMP: Added another keyword to ignore when checking for annuals against a particular series, FIX:(#1210) When importing files, if issues were meta-tagged with CVS would not scan the metadata properly, FIX: Fixed checkboxes on annuals table so drop-down will work again (just doesn't refresh the screen afterwards atm), FIX:(#1182) Notifications not respecting http base, IMP: Added option to specify SSL Verification on/off per provider when searching/downloading, FIX: Possible fix for cache_dir not being used on restarts of mylar, FIX: configparser check would only check on linux systems, and if not present would cause errors during post-processing on non nix-based systems, FIX:(#1181) Manual renaming an entire series would fail if lowercase_filenames was enabled as an option
2016-02-25 16:40:09 +00:00
payload = None
2013-03-27 07:03:10 +00:00
try :
2017-10-25 18:33:56 +00:00
r = requests . get ( PULLURL , params = payload , verify = mylar . CONFIG . CV_VERIFY , headers = mylar . CV_HEADERS )
FIX: Added user-agent to all CV queries, and switched all queries to utilize the requests module, FIX: Added User-Agent to ComicTagger and switched to requests module for all CV queries, FIX: Removed libraryscan option from GUI as no longer used, FIX: Ability to sort 'Have' column on main index page properly now (by % complete), FIX: Fixed some display issues with the series detail page and displaying, FIX: Metatagging should now be working again for story-arc, and one-off post-processing, FIX: Fixed post-processing problem with decimal places if number was padded, FIX: Updated CV-URL to point to new CV api location, FIX: Fixed problem when file-checking and series contained numbers and/or decimals, would take the modified series name instead of the actual series name resulting in some missed matches, IMP: Added another keyword to ignore when checking for annuals against a particular series, FIX:(#1210) When importing files, if issues were meta-tagged with CVS would not scan the metadata properly, FIX: Fixed checkboxes on annuals table so drop-down will work again (just doesn't refresh the screen afterwards atm), FIX:(#1182) Notifications not respecting http base, IMP: Added option to specify SSL Verification on/off per provider when searching/downloading, FIX: Possible fix for cache_dir not being used on restarts of mylar, FIX: configparser check would only check on linux systems, and if not present would cause errors during post-processing on non nix-based systems, FIX:(#1181) Manual renaming an entire series would fail if lowercase_filenames was enabled as an option
2016-02-25 16:40:09 +00:00
except Exception , e :
logger . warn ( ' Error fetching data from ComicVine: %s ' % ( e ) )
2015-05-22 08:32:51 +00:00
return
FIX:(#1222) Fixed usage with NZBHydra - will now be able to properly grab the nzb files, as well as proper logging and handling of Failed downloading, IMP: Duplicate Directory Dump option available in Configuration GUI. Any duplicates discovered during post-processing will be moved into this directory (if enabled) depending on the dupe constraints, FIX: Better handling of titles with '&' and '-' in the titles (as well as annuals) when adding series which should mean more accurate results when trying to add a series, FIX:(#1142) If files didn't have the pages field metadata within the comicinfo.xml file, would error out and either fail to display the issue, or fail to scan the issue during an import scan, FIX: When adding/refreshing a series, if the cover image from CV is unable to be retrieved or is not of an adequate size, fallback to a differnt quality image from CV, FIX: When refreshing/adding a series, annuals will only be checked against once (it was previously running through the entire annual check twice), FIX: During RSS scans/checks if a title in the results had an encoded & (&), would store it as the html which would never match up when doing actual comparison searches, IMP: Fixed usage of feedparser module in rss feeds so that it only uses the retrieved data and doesn't do the actual polling against the url (requests module now does), IMP: Added proper handling of error code 910 with dognzb (max api hits), so that once it hits the 910 error it will disable dognzb as a provider, FIX: When attempting to display issue details on a series detail page (the i icon in the issues table), if the metadata in the .cbz cannot be read or doesn't exist, will now display a graphical warning instead of a 500 error, IMP: Added fork/fork version/tag to comictagger user-agent/version, IMP: Removed configparser dependency from ComicTagger, FIX: When performing searches, improved volume label matching regardless of how the volume label is represented
2016-03-04 20:04:19 +00:00
2018-05-08 18:01:24 +00:00
try :
dom = parseString ( r . content ) #(data)
except ExpatError :
logger . warn ( ' [WARNING] ComicVine is not responding correctly at the moment. This is usually due to some problems on their end. If you re-try things again in a few moments, it might work properly. ' )
return
2012-09-13 15:27:34 +00:00
return dom
IMP: Changed configuration completely. New config processing, global/config variables changed over entire application. All changes in the configuration GUI take effect immediately after saving - no more need to restart Mylar, IMP: Added provider order sequence header to the top of the Providers tab for improved visibilty, IMP: Added completed download handling for both SABnzbd and NZBGet - will monitor active queue for downloads sent by Mylar and then post-process them accordingly (no more ComicRN), FIX: When recreating a pullist for a week that's not the current week, will now refresh series that are still missing data in case of late population, IMP: Removed loose/explict search options, and search results will now match more accurately to the terms entered, as well as being much quicker, IMP: Changed colour-codes on search results screen, green now indicates that the series is a print series, non-green indicates that it is either a HC/TPB/digital series and will also be indicated as such, IMP: Fixed weeklypull not refreshing some series due to legacy numbering on series deemed as Ended by Mylar, or the pull not refreshing propeprly and thereby were not elgible to be considered as a valid series for matching, IMP: Fixed problem with the scheduler of the weekly pull not running at the designated time, which causes problems when attempting to match series to the pullist for issues, IMP: Changed the autosnatch script so that the get.conf is no longer required to be maintained seperately. The get.conf config items are now located in the main config.ini and have to be repopulated initially, FIX: Fixed problem with the import count not resetting to 0 on subsequent runs of the Import A Directory option, IMP: Changed the sab api key from using the nzbkey to the full api key as is required for completed download handling, FIX: Fixed the retry option for newznab entries so that it should work across the board now for newznabs as opposed to only some providers, IMP: some minor code cleanup
2017-11-10 19:25:14 +00:00
def findComic ( name , mode , issue , limityear = None , type = None ) :
2012-09-13 15:27:34 +00:00
IMP: Changed configuration completely. New config processing, global/config variables changed over entire application. All changes in the configuration GUI take effect immediately after saving - no more need to restart Mylar, IMP: Added provider order sequence header to the top of the Providers tab for improved visibilty, IMP: Added completed download handling for both SABnzbd and NZBGet - will monitor active queue for downloads sent by Mylar and then post-process them accordingly (no more ComicRN), FIX: When recreating a pullist for a week that's not the current week, will now refresh series that are still missing data in case of late population, IMP: Removed loose/explict search options, and search results will now match more accurately to the terms entered, as well as being much quicker, IMP: Changed colour-codes on search results screen, green now indicates that the series is a print series, non-green indicates that it is either a HC/TPB/digital series and will also be indicated as such, IMP: Fixed weeklypull not refreshing some series due to legacy numbering on series deemed as Ended by Mylar, or the pull not refreshing propeprly and thereby were not elgible to be considered as a valid series for matching, IMP: Fixed problem with the scheduler of the weekly pull not running at the designated time, which causes problems when attempting to match series to the pullist for issues, IMP: Changed the autosnatch script so that the get.conf is no longer required to be maintained seperately. The get.conf config items are now located in the main config.ini and have to be repopulated initially, FIX: Fixed problem with the import count not resetting to 0 on subsequent runs of the Import A Directory option, IMP: Changed the sab api key from using the nzbkey to the full api key as is required for completed download handling, FIX: Fixed the retry option for newznab entries so that it should work across the board now for newznabs as opposed to only some providers, IMP: some minor code cleanup
2017-11-10 19:25:14 +00:00
#with mb_lock:
2012-09-13 15:27:34 +00:00
comicResults = None
2014-10-08 06:45:51 +00:00
comicLibrary = listLibrary ( )
2015-05-27 05:40:38 +00:00
comiclist = [ ]
arcinfolist = [ ]
IMP: Changed configuration completely. New config processing, global/config variables changed over entire application. All changes in the configuration GUI take effect immediately after saving - no more need to restart Mylar, IMP: Added provider order sequence header to the top of the Providers tab for improved visibilty, IMP: Added completed download handling for both SABnzbd and NZBGet - will monitor active queue for downloads sent by Mylar and then post-process them accordingly (no more ComicRN), FIX: When recreating a pullist for a week that's not the current week, will now refresh series that are still missing data in case of late population, IMP: Removed loose/explict search options, and search results will now match more accurately to the terms entered, as well as being much quicker, IMP: Changed colour-codes on search results screen, green now indicates that the series is a print series, non-green indicates that it is either a HC/TPB/digital series and will also be indicated as such, IMP: Fixed weeklypull not refreshing some series due to legacy numbering on series deemed as Ended by Mylar, or the pull not refreshing propeprly and thereby were not elgible to be considered as a valid series for matching, IMP: Fixed problem with the scheduler of the weekly pull not running at the designated time, which causes problems when attempting to match series to the pullist for issues, IMP: Changed the autosnatch script so that the get.conf is no longer required to be maintained seperately. The get.conf config items are now located in the main config.ini and have to be repopulated initially, FIX: Fixed problem with the import count not resetting to 0 on subsequent runs of the Import A Directory option, IMP: Changed the sab api key from using the nzbkey to the full api key as is required for completed download handling, FIX: Fixed the retry option for newznab entries so that it should work across the board now for newznabs as opposed to only some providers, IMP: some minor code cleanup
2017-11-10 19:25:14 +00:00
2017-12-11 18:49:40 +00:00
commons = [ ' and ' , ' the ' , ' & ' , ' - ' ]
2017-11-15 22:49:57 +00:00
for x in commons :
2017-12-13 15:40:43 +00:00
cnt = 0
for m in re . finditer ( x , name . lower ( ) ) :
cnt + = 1
tehstart = m . start ( )
tehend = m . end ( )
if any ( [ x == ' the ' , x == ' and ' ] ) :
2018-04-12 18:09:13 +00:00
if len ( name ) == tehend :
tehend = - 1
2017-12-13 15:40:43 +00:00
if not all ( [ tehstart == 0 , name [ tehend ] == ' ' ] ) or not all ( [ tehstart != 0 , name [ tehstart - 1 ] == ' ' , name [ tehend ] == ' ' ] ) :
continue
else :
name = name . replace ( x , ' ' , cnt )
2017-11-15 22:49:57 +00:00
2018-03-09 16:47:55 +00:00
originalname = name
if ' + ' in name :
name = re . sub ( ' \ + ' , ' PLUS ' , name )
2017-11-15 22:49:57 +00:00
pattern = re . compile ( ur ' \ w+ ' , re . UNICODE )
name = pattern . findall ( name )
2018-03-09 16:47:55 +00:00
if ' + ' in originalname :
y = [ ]
for x in name :
y . append ( re . sub ( " PLUS " , " % 2B " , x ) )
name = y
2013-02-06 19:55:23 +00:00
if limityear is None : limityear = ' None '
2015-05-22 08:32:51 +00:00
2014-02-26 19:48:50 +00:00
comicquery = name
2014-06-09 07:55:05 +00:00
2017-10-25 18:33:56 +00:00
if mylar . CONFIG . COMICVINE_API == ' None ' or mylar . CONFIG . COMICVINE_API is None :
logger . warn ( ' You have not specified your own ComicVine API key - this is a requirement. Get your own @ http://api.comicvine.com. ' )
return
2014-06-09 07:55:05 +00:00
else :
2017-10-25 18:33:56 +00:00
comicapi = mylar . CONFIG . COMICVINE_API
2012-09-13 15:27:34 +00:00
FIX: undefined on filter box on startup, IMP: Added Meta-Tagging options on a series / issue basis on comic details screen, IMP: Issue Information is now available per issue and is extracted currently from the cbz file to display (if no cbz is present, the option isn't available), IMP: Failed Download handling is implemented and available in GUI - required to replace existing autoProcessComics.py and ComicRN.py scripts, IMP: Added ability to specify post-processing script instead of ComicRN.py, IMP: Added abilty to edit the issue date for a given issue by simply clicking on it - this will help to avoid dates that have incorrect or 0000-00-00, IMP: Story Arc searching is working (not adding yet), IMP: Added Archived/Ignored options to Upcoming/Wanted tab, IMP: Fixed some alignment and display issues on the Upcoming section, IMP: Added better directory handling for Updating Comic Location when it needs to get changed for all existing series (locmove in config.ini), IMP: Added better handling for unicode characters in series titles when searching / filechecking, IMP: When adding a new series with no data, Mylar would error out (now will add and just retain 0 issues), FIX: When year was fuzzied, search would still attempt to do a date-check comparison and say everything failed, IMP: Better handling of nzb names when retaining for post-processing comparisons, IMP: Future Upcoming now will use actual shipping date of issue if available in order to see if issue is available, FIX: If annuals were enabled, refreshing a series would put some issues into an Archived status because the actual counts would be off, IMP: When file checking, Alternate Naming would be searched last which resulted in matching incorrectly to the series title or other alternate naming for the given series, now will check the Alternate Naming first for a match, then drop back down to the series name itself otherwise, IMP: Improved Annual detection when integrated with a given series, IMP: Improved the checking of the future Upcoming list for issues marked as Wanted but not available yet and then auto-adding, IMP: Improved upon story arc checking for missing issues / searching for wanted, IMP: Enabling Annuals support now within Configuration GUI, bunch of other things....
2014-07-28 19:28:09 +00:00
if type is None :
type = ' volume '
2015-05-22 08:32:51 +00:00
#let's find out how many results we get from the query...
2017-11-15 22:49:57 +00:00
searched = pullsearch ( comicapi , comicquery , 0 , type )
2015-06-22 03:38:10 +00:00
if searched is None :
return False
2012-09-13 15:27:34 +00:00
totalResults = searched . getElementsByTagName ( ' number_of_total_results ' ) [ 0 ] . firstChild . wholeText
2014-02-26 19:48:50 +00:00
logger . fdebug ( " there are " + str ( totalResults ) + " search results... " )
2012-09-13 15:27:34 +00:00
if not totalResults :
return False
IMP: (#1122) Added provider OMGWTFNZBS (results untested), FIX:(#1124) Experimental search results had a 1/9 (or simliar) parts as part of the header, would error out trying to create the nzb due to invalid path, FIX: When importing a series, if Move Files wasn't enabled would error out during the import process, FIX: When on Import page, if selecting Save Changes without Scanning Library would leave pop-up notification up forever, FIX: When searching for series (like A-Force), results would return 16000+ which would look like it locked up due to the excessive number of results - will now better handle such titles, and overall search results are limited to the first 1000 hits, FIX: When clicking on 'Add Series' from the pull-list, if a series was a new series with no issues published would return 0 results from CV even though the series would exist, FIX:(#1120) Story Arcs with special characters would cause error during manual post-processing run, FIX:(#1125) If year is fuzzied for a given series when searching, would error out with invalid date comparisons
2015-09-11 15:49:56 +00:00
if int ( totalResults ) > 1000 :
FIX: When retrieving feeds from 32p and in Auth mode, personal notification feeds contained some invalid html entries that weren't removed properly resulting in no results when attempting to match for downloading, FIX: When searching 32P, if title had a '/' within the title - Mylar would mistakingly skip it due to some previous exceptions that were made for CBT, FIX: Main page would quickly display & hide the have% column instead of always being hidden, FIX: Adjusted some incorrect spacing for non-alphanumeric characters when comparing search results (should result in better matching hopefully), FIX: When adding a series and the most recent issue was present on the weekly-pull list, it would sometimes not mark it as Wanted and auto-attempt to search for it (if auto mark Upcoming enabled), FIX: Added Test Connection button for 32P where it will test logon credentials as well as if Captcha is present, IMP: If captcha is enabled for 32p and signon is required because keys are stale, will not send authentication information and will just bypass as a provider, IMP: Test Connection button added for SABnzbd, IMP: Added ability to directly add torrents to rtorrent and apply label + download directory options (config.ini only atm), FIX: If a search result had a 'vol.' label in it, depending on how the format of the label was mylar would refuse to remove the volume which resulted in failed matches (also fixed a similar issue with failing to remove the volume label when comparing search results), FIX: When filechecking, if a series had a - in the title, will now account for it properly, IMP: Completely redid the filecheck module which allows for integration into other modules as well as more detailed failure logs, IMP: Added Dynamic handder integration into filechecker and subsequent modules that use it which allows for special characters to be replaced with any other type of character, IMP: Manual post-processing speed improved greatly due to new usage of filecheck module, IMP: Importer backend code redone to include new filecheck module, IMP: Added status/counter to import process, IMP: Added force unlock option to importer for failed imports, IMP: Added new status to Import labelled as 'Manual Intervention' for imports that need the user to manually select an option from an available list, FIX: When import said there were search results to view, but none available - would blank screen, IMP: Added a failure log entry showing all the failed files that weren't able to be scanned in during an import (will be in GUI eventually), IMP: if only partial metadata is available during import, Mylar will attempt to use what's available from the metatagging instead of picking all of one/other, IMP: Better grouping of series/volumes when viewing the import results page as well as now indicating if annuals are present within the files, IMP: Added a file-icon beside each imported item on the import result page which allows the user to view the files that are associated with the given series grouping, IMP: Added a blacklisted_publishers option to config.ini which will blacklist specific publishers from being returned during search / import results, FIX: If duplicate dump folder had a value, but duplicate dump wasn't enabled - would still use the duplicate dump folder during post-processing runs, FIX: (#1194) Patch to allow for fixed H1 elements for title (thnx chazlarson), FIX: Removed UnRAR dependency checks in cmtagmylar since not being used anymore, FIX: Fixed a problem with non-ascii characters being recognized during a file-check in certain cases, IMP: Attmept by Mylar to grab an alternate jpg from file when viewing the issue details if it complies with the naming conventions, FIX: Fixed some metatagging issues with ComicBookLover tags not being handled properly if they didn't exist, IMP: Dupecheck now has a failback if it's comparing a cbr/cbr, cbz/cbz and cbr/cbz-priority is enabled, FIX: Quick check added for when adding/refreshing a comic that if a cover already existed, it would delete the cover prior to the attempt to retrieve it, IMP: Added some additional handling for when searching/adding fails, FIX: If a story arc didn't have proper issue dates (or invalid ones) would error out on loading the story arc main page - usually when arcs were imported using a cbl file.
2016-04-07 17:09:06 +00:00
logger . warn ( ' Search returned more than 1000 hits [ ' + str ( totalResults ) + ' ]. Only displaying first 1000 results - use more specifics or the exact ComicID if required. ' )
IMP: (#1122) Added provider OMGWTFNZBS (results untested), FIX:(#1124) Experimental search results had a 1/9 (or simliar) parts as part of the header, would error out trying to create the nzb due to invalid path, FIX: When importing a series, if Move Files wasn't enabled would error out during the import process, FIX: When on Import page, if selecting Save Changes without Scanning Library would leave pop-up notification up forever, FIX: When searching for series (like A-Force), results would return 16000+ which would look like it locked up due to the excessive number of results - will now better handle such titles, and overall search results are limited to the first 1000 hits, FIX: When clicking on 'Add Series' from the pull-list, if a series was a new series with no issues published would return 0 results from CV even though the series would exist, FIX:(#1120) Story Arcs with special characters would cause error during manual post-processing run, FIX:(#1125) If year is fuzzied for a given series when searching, would error out with invalid date comparisons
2015-09-11 15:49:56 +00:00
totalResults = 1000
2012-09-13 15:27:34 +00:00
countResults = 0
2013-03-29 11:16:55 +00:00
while ( countResults < int ( totalResults ) ) :
2014-02-26 19:48:50 +00:00
#logger.fdebug("querying " + str(countResults))
2012-12-12 22:15:06 +00:00
if countResults > 0 :
2017-11-15 22:49:57 +00:00
offsetcount = countResults
searched = pullsearch ( comicapi , comicquery , offsetcount , type )
IMP: Changed configuration completely. New config processing, global/config variables changed over entire application. All changes in the configuration GUI take effect immediately after saving - no more need to restart Mylar, IMP: Added provider order sequence header to the top of the Providers tab for improved visibilty, IMP: Added completed download handling for both SABnzbd and NZBGet - will monitor active queue for downloads sent by Mylar and then post-process them accordingly (no more ComicRN), FIX: When recreating a pullist for a week that's not the current week, will now refresh series that are still missing data in case of late population, IMP: Removed loose/explict search options, and search results will now match more accurately to the terms entered, as well as being much quicker, IMP: Changed colour-codes on search results screen, green now indicates that the series is a print series, non-green indicates that it is either a HC/TPB/digital series and will also be indicated as such, IMP: Fixed weeklypull not refreshing some series due to legacy numbering on series deemed as Ended by Mylar, or the pull not refreshing propeprly and thereby were not elgible to be considered as a valid series for matching, IMP: Fixed problem with the scheduler of the weekly pull not running at the designated time, which causes problems when attempting to match series to the pullist for issues, IMP: Changed the autosnatch script so that the get.conf is no longer required to be maintained seperately. The get.conf config items are now located in the main config.ini and have to be repopulated initially, FIX: Fixed problem with the import count not resetting to 0 on subsequent runs of the Import A Directory option, IMP: Changed the sab api key from using the nzbkey to the full api key as is required for completed download handling, FIX: Fixed the retry option for newznab entries so that it should work across the board now for newznabs as opposed to only some providers, IMP: some minor code cleanup
2017-11-10 19:25:14 +00:00
comicResults = searched . getElementsByTagName ( type )
2012-09-13 15:27:34 +00:00
body = ' '
2015-05-22 08:32:51 +00:00
n = 0
2012-09-13 15:27:34 +00:00
if not comicResults :
2015-05-22 08:32:51 +00:00
break
2012-09-13 15:27:34 +00:00
for result in comicResults :
#retrieve the first xml tag (<tag>data</tag>)
#that the parser finds with name tagName:
2014-11-21 17:16:28 +00:00
arclist = [ ]
IMP: (#844)Annuals now deleted from table on a Delete Series command, IMP: (#837)Better volume detection when searching for issues, IMP: (#842)(#808) Added some handling for issue titles when searching (it's not perfect, but it works for most), IMP: (#836)Added .cb7 for filechecking purposes only, IMP: (#830)issue numbers that are -1 can now be renamed, FIX: 'bi-annual' now fixed along with some other improvements for annual handling, IMP: Added version check for autoProcessComics and ComicRN - in order to help warn users of changes and the need to update these files since they are usually outside of the mylar git directory when being used, FIX:(#840) Boxcar2 should be working again, FIX: (#845) If search returned a filename that had no distinction between issue number and issue title and/or extra information, would error out if any of the words contained NOW, IMP: (#823) Added some better detection for NZBGet parameters when using ComicRN.py
2014-10-06 18:10:36 +00:00
if type == ' story_arc ' :
#call cv.py here to find out issue count in story arc
try :
logger . fdebug ( ' story_arc ascension ' )
2015-05-22 08:32:51 +00:00
names = len ( result . getElementsByTagName ( ' name ' ) )
IMP: (#844)Annuals now deleted from table on a Delete Series command, IMP: (#837)Better volume detection when searching for issues, IMP: (#842)(#808) Added some handling for issue titles when searching (it's not perfect, but it works for most), IMP: (#836)Added .cb7 for filechecking purposes only, IMP: (#830)issue numbers that are -1 can now be renamed, FIX: 'bi-annual' now fixed along with some other improvements for annual handling, IMP: Added version check for autoProcessComics and ComicRN - in order to help warn users of changes and the need to update these files since they are usually outside of the mylar git directory when being used, FIX:(#840) Boxcar2 should be working again, FIX: (#845) If search returned a filename that had no distinction between issue number and issue title and/or extra information, would error out if any of the words contained NOW, IMP: (#823) Added some better detection for NZBGet parameters when using ComicRN.py
2014-10-06 18:10:36 +00:00
n = 0
logger . fdebug ( ' length: ' + str ( names ) )
2015-01-04 09:17:49 +00:00
xmlpub = None #set this incase the publisher field isn't populated in the xml
2015-05-22 08:32:51 +00:00
while ( n < names ) :
IMP: (#844)Annuals now deleted from table on a Delete Series command, IMP: (#837)Better volume detection when searching for issues, IMP: (#842)(#808) Added some handling for issue titles when searching (it's not perfect, but it works for most), IMP: (#836)Added .cb7 for filechecking purposes only, IMP: (#830)issue numbers that are -1 can now be renamed, FIX: 'bi-annual' now fixed along with some other improvements for annual handling, IMP: Added version check for autoProcessComics and ComicRN - in order to help warn users of changes and the need to update these files since they are usually outside of the mylar git directory when being used, FIX:(#840) Boxcar2 should be working again, FIX: (#845) If search returned a filename that had no distinction between issue number and issue title and/or extra information, would error out if any of the words contained NOW, IMP: (#823) Added some better detection for NZBGet parameters when using ComicRN.py
2014-10-06 18:10:36 +00:00
logger . fdebug ( result . getElementsByTagName ( ' name ' ) [ n ] . parentNode . nodeName )
if result . getElementsByTagName ( ' name ' ) [ n ] . parentNode . nodeName == ' story_arc ' :
logger . fdebug ( ' yes ' )
try :
xmlTag = result . getElementsByTagName ( ' name ' ) [ n ] . firstChild . wholeText
xmlTag = xmlTag . rstrip ( )
FIX: When retrieving feeds from 32p and in Auth mode, personal notification feeds contained some invalid html entries that weren't removed properly resulting in no results when attempting to match for downloading, FIX: When searching 32P, if title had a '/' within the title - Mylar would mistakingly skip it due to some previous exceptions that were made for CBT, FIX: Main page would quickly display & hide the have% column instead of always being hidden, FIX: Adjusted some incorrect spacing for non-alphanumeric characters when comparing search results (should result in better matching hopefully), FIX: When adding a series and the most recent issue was present on the weekly-pull list, it would sometimes not mark it as Wanted and auto-attempt to search for it (if auto mark Upcoming enabled), FIX: Added Test Connection button for 32P where it will test logon credentials as well as if Captcha is present, IMP: If captcha is enabled for 32p and signon is required because keys are stale, will not send authentication information and will just bypass as a provider, IMP: Test Connection button added for SABnzbd, IMP: Added ability to directly add torrents to rtorrent and apply label + download directory options (config.ini only atm), FIX: If a search result had a 'vol.' label in it, depending on how the format of the label was mylar would refuse to remove the volume which resulted in failed matches (also fixed a similar issue with failing to remove the volume label when comparing search results), FIX: When filechecking, if a series had a - in the title, will now account for it properly, IMP: Completely redid the filecheck module which allows for integration into other modules as well as more detailed failure logs, IMP: Added Dynamic handder integration into filechecker and subsequent modules that use it which allows for special characters to be replaced with any other type of character, IMP: Manual post-processing speed improved greatly due to new usage of filecheck module, IMP: Importer backend code redone to include new filecheck module, IMP: Added status/counter to import process, IMP: Added force unlock option to importer for failed imports, IMP: Added new status to Import labelled as 'Manual Intervention' for imports that need the user to manually select an option from an available list, FIX: When import said there were search results to view, but none available - would blank screen, IMP: Added a failure log entry showing all the failed files that weren't able to be scanned in during an import (will be in GUI eventually), IMP: if only partial metadata is available during import, Mylar will attempt to use what's available from the metatagging instead of picking all of one/other, IMP: Better grouping of series/volumes when viewing the import results page as well as now indicating if annuals are present within the files, IMP: Added a file-icon beside each imported item on the import result page which allows the user to view the files that are associated with the given series grouping, IMP: Added a blacklisted_publishers option to config.ini which will blacklist specific publishers from being returned during search / import results, FIX: If duplicate dump folder had a value, but duplicate dump wasn't enabled - would still use the duplicate dump folder during post-processing runs, FIX: (#1194) Patch to allow for fixed H1 elements for title (thnx chazlarson), FIX: Removed UnRAR dependency checks in cmtagmylar since not being used anymore, FIX: Fixed a problem with non-ascii characters being recognized during a file-check in certain cases, IMP: Attmept by Mylar to grab an alternate jpg from file when viewing the issue details if it complies with the naming conventions, FIX: Fixed some metatagging issues with ComicBookLover tags not being handled properly if they didn't exist, IMP: Dupecheck now has a failback if it's comparing a cbr/cbr, cbz/cbz and cbr/cbz-priority is enabled, FIX: Quick check added for when adding/refreshing a comic that if a cover already existed, it would delete the cover prior to the attempt to retrieve it, IMP: Added some additional handling for when searching/adding fails, FIX: If a story arc didn't have proper issue dates (or invalid ones) would error out on loading the story arc main page - usually when arcs were imported using a cbl file.
2016-04-07 17:09:06 +00:00
logger . fdebug ( ' name: ' + xmlTag )
IMP: (#844)Annuals now deleted from table on a Delete Series command, IMP: (#837)Better volume detection when searching for issues, IMP: (#842)(#808) Added some handling for issue titles when searching (it's not perfect, but it works for most), IMP: (#836)Added .cb7 for filechecking purposes only, IMP: (#830)issue numbers that are -1 can now be renamed, FIX: 'bi-annual' now fixed along with some other improvements for annual handling, IMP: Added version check for autoProcessComics and ComicRN - in order to help warn users of changes and the need to update these files since they are usually outside of the mylar git directory when being used, FIX:(#840) Boxcar2 should be working again, FIX: (#845) If search returned a filename that had no distinction between issue number and issue title and/or extra information, would error out if any of the words contained NOW, IMP: (#823) Added some better detection for NZBGet parameters when using ComicRN.py
2014-10-06 18:10:36 +00:00
except :
logger . error ( ' There was a problem retrieving the given data from ComicVine. Ensure that www.comicvine.com is accessible. ' )
return
elif result . getElementsByTagName ( ' name ' ) [ n ] . parentNode . nodeName == ' publisher ' :
logger . fdebug ( ' publisher check. ' )
xmlpub = result . getElementsByTagName ( ' name ' ) [ n ] . firstChild . wholeText
2013-07-01 05:19:15 +00:00
IMP: (#844)Annuals now deleted from table on a Delete Series command, IMP: (#837)Better volume detection when searching for issues, IMP: (#842)(#808) Added some handling for issue titles when searching (it's not perfect, but it works for most), IMP: (#836)Added .cb7 for filechecking purposes only, IMP: (#830)issue numbers that are -1 can now be renamed, FIX: 'bi-annual' now fixed along with some other improvements for annual handling, IMP: Added version check for autoProcessComics and ComicRN - in order to help warn users of changes and the need to update these files since they are usually outside of the mylar git directory when being used, FIX:(#840) Boxcar2 should be working again, FIX: (#845) If search returned a filename that had no distinction between issue number and issue title and/or extra information, would error out if any of the words contained NOW, IMP: (#823) Added some better detection for NZBGet parameters when using ComicRN.py
2014-10-06 18:10:36 +00:00
n + = 1
except :
logger . warn ( ' error retrieving story arc search results. ' )
return
2015-05-22 08:32:51 +00:00
siteurl = len ( result . getElementsByTagName ( ' site_detail_url ' ) )
IMP: (#844)Annuals now deleted from table on a Delete Series command, IMP: (#837)Better volume detection when searching for issues, IMP: (#842)(#808) Added some handling for issue titles when searching (it's not perfect, but it works for most), IMP: (#836)Added .cb7 for filechecking purposes only, IMP: (#830)issue numbers that are -1 can now be renamed, FIX: 'bi-annual' now fixed along with some other improvements for annual handling, IMP: Added version check for autoProcessComics and ComicRN - in order to help warn users of changes and the need to update these files since they are usually outside of the mylar git directory when being used, FIX:(#840) Boxcar2 should be working again, FIX: (#845) If search returned a filename that had no distinction between issue number and issue title and/or extra information, would error out if any of the words contained NOW, IMP: (#823) Added some better detection for NZBGet parameters when using ComicRN.py
2014-10-06 18:10:36 +00:00
s = 0
logger . fdebug ( ' length: ' + str ( names ) )
xmlurl = None
2015-05-22 08:32:51 +00:00
while ( s < siteurl ) :
IMP: (#844)Annuals now deleted from table on a Delete Series command, IMP: (#837)Better volume detection when searching for issues, IMP: (#842)(#808) Added some handling for issue titles when searching (it's not perfect, but it works for most), IMP: (#836)Added .cb7 for filechecking purposes only, IMP: (#830)issue numbers that are -1 can now be renamed, FIX: 'bi-annual' now fixed along with some other improvements for annual handling, IMP: Added version check for autoProcessComics and ComicRN - in order to help warn users of changes and the need to update these files since they are usually outside of the mylar git directory when being used, FIX:(#840) Boxcar2 should be working again, FIX: (#845) If search returned a filename that had no distinction between issue number and issue title and/or extra information, would error out if any of the words contained NOW, IMP: (#823) Added some better detection for NZBGet parameters when using ComicRN.py
2014-10-06 18:10:36 +00:00
logger . fdebug ( result . getElementsByTagName ( ' site_detail_url ' ) [ s ] . parentNode . nodeName )
if result . getElementsByTagName ( ' site_detail_url ' ) [ s ] . parentNode . nodeName == ' story_arc ' :
try :
xmlurl = result . getElementsByTagName ( ' site_detail_url ' ) [ s ] . firstChild . wholeText
except :
logger . error ( ' There was a problem retrieving the given data from ComicVine. Ensure that www.comicvine.com is accessible. ' )
return
s + = 1
xmlid = result . getElementsByTagName ( ' id ' ) [ 0 ] . firstChild . wholeText
if xmlid is not None :
2015-05-27 05:40:38 +00:00
arcinfolist = storyarcinfo ( xmlid )
FIX:(#1431) Fixed post-processing cleanup when using SAB + files resided in a sub-folder of the path sent from SAB, IMP: Added hardlink/softlink option for non-windows users (note that metatagging is forcibly disabled when these options are enabled), IMP: Added Enforce Permissions option to force Mylar to either force / ensure permissions are as set within Mylar, or to ignore permissions and just try to perform the given action (this is for adding series, post-processing, importing, arcs, etc), FIX: Fixed a problem where if local watchdir was set and a torrent downloader was selected, both would attempt to be used, but only the watchdir was actually utilized (now it's one or the other, never both), FIX: Mylar will now remove issues that are in Failed status that have incorrect Series Data attached, FIX:(#1438) When file-checking/post-processing if issue date in Mylar was current year and date in filename was the following year (for end-of-year issues), would cause traceback error, FIX: If series year polled from CV is incorrect or not provided, option is available in Edit Settings of series detail page that allows user to specify the correct series year, FIX:(#1427) Fixed a typo when the Scan Library option is used and ComicID's were able to be parsed from the files, FIX: When issues were archived, depending on if series had issues in a Downloaded status or not, would give different results to the Have totals, IMP: Manage tab / Manage Comics now has metatag option available to allow metatagging multiple series in a batch mode, IMP:(#1419) When 'write cvinfo' option is enabled, will now also write cvinfo into story arc directories when added / refreshed, FIX: Fixed a problem when group metatag was being used and causing errors, IMP: Manage Failed page now will link back to the series for easier reference, and the 'link' column was removed due to invalid references, FIX: Removed the 'issue not updated' from the series detail page as was not accurate, IMP: Added a Select Filtered Issues button to the series detail page to allow for selecting of all filtered issues (Show ALL MUST be enabled for it to work properly)
2016-11-02 15:00:56 +00:00
logger . info ( ' [IMAGE] : ' + arcinfolist [ ' comicimage ' ] )
2015-05-27 05:40:38 +00:00
comiclist . append ( {
' name ' : xmlTag ,
' comicyear ' : arcinfolist [ ' comicyear ' ] ,
' comicid ' : xmlid ,
' cvarcid ' : xmlid ,
' url ' : xmlurl ,
' issues ' : arcinfolist [ ' issues ' ] ,
' comicimage ' : arcinfolist [ ' comicimage ' ] ,
' publisher ' : xmlpub ,
' description ' : arcinfolist [ ' description ' ] ,
' deck ' : arcinfolist [ ' deck ' ] ,
' arclist ' : arcinfolist [ ' arclist ' ] ,
' haveit ' : arcinfolist [ ' haveit ' ]
} )
else :
comiclist . append ( {
' name ' : xmlTag ,
' comicyear ' : arcyear ,
' comicid ' : xmlid ,
' url ' : xmlurl ,
' issues ' : issuecount ,
' comicimage ' : xmlimage ,
' publisher ' : xmlpub ,
' description ' : xmldesc ,
' deck ' : xmldeck ,
' arclist ' : arclist ,
' haveit ' : haveit
} )
logger . fdebug ( ' IssueID \' s that are a part of ' + xmlTag + ' : ' + str ( arclist ) )
IMP: (#844)Annuals now deleted from table on a Delete Series command, IMP: (#837)Better volume detection when searching for issues, IMP: (#842)(#808) Added some handling for issue titles when searching (it's not perfect, but it works for most), IMP: (#836)Added .cb7 for filechecking purposes only, IMP: (#830)issue numbers that are -1 can now be renamed, FIX: 'bi-annual' now fixed along with some other improvements for annual handling, IMP: Added version check for autoProcessComics and ComicRN - in order to help warn users of changes and the need to update these files since they are usually outside of the mylar git directory when being used, FIX:(#840) Boxcar2 should be working again, FIX: (#845) If search returned a filename that had no distinction between issue number and issue title and/or extra information, would error out if any of the words contained NOW, IMP: (#823) Added some better detection for NZBGet parameters when using ComicRN.py
2014-10-06 18:10:36 +00:00
else :
xmlcnt = result . getElementsByTagName ( ' count_of_issues ' ) [ 0 ] . firstChild . wholeText
#here we can determine what called us, and either start gathering all issues or just limited ones.
if issue is not None and str ( issue ) . isdigit ( ) :
#this gets buggered up with NEW/ONGOING series because the db hasn't been updated
#to reflect the proper count. Drop it by 1 to make sure.
limiter = int ( issue ) - 1
else : limiter = 0
2014-11-21 17:16:28 +00:00
#get the first issue # (for auto-magick calcs)
IMP: Changed configuration completely. New config processing, global/config variables changed over entire application. All changes in the configuration GUI take effect immediately after saving - no more need to restart Mylar, IMP: Added provider order sequence header to the top of the Providers tab for improved visibilty, IMP: Added completed download handling for both SABnzbd and NZBGet - will monitor active queue for downloads sent by Mylar and then post-process them accordingly (no more ComicRN), FIX: When recreating a pullist for a week that's not the current week, will now refresh series that are still missing data in case of late population, IMP: Removed loose/explict search options, and search results will now match more accurately to the terms entered, as well as being much quicker, IMP: Changed colour-codes on search results screen, green now indicates that the series is a print series, non-green indicates that it is either a HC/TPB/digital series and will also be indicated as such, IMP: Fixed weeklypull not refreshing some series due to legacy numbering on series deemed as Ended by Mylar, or the pull not refreshing propeprly and thereby were not elgible to be considered as a valid series for matching, IMP: Fixed problem with the scheduler of the weekly pull not running at the designated time, which causes problems when attempting to match series to the pullist for issues, IMP: Changed the autosnatch script so that the get.conf is no longer required to be maintained seperately. The get.conf config items are now located in the main config.ini and have to be repopulated initially, FIX: Fixed problem with the import count not resetting to 0 on subsequent runs of the Import A Directory option, IMP: Changed the sab api key from using the nzbkey to the full api key as is required for completed download handling, FIX: Fixed the retry option for newznab entries so that it should work across the board now for newznabs as opposed to only some providers, IMP: some minor code cleanup
2017-11-10 19:25:14 +00:00
iss_len = len ( result . getElementsByTagName ( ' name ' ) )
i = 0
xmlfirst = ' 1 '
xmllast = None
2014-11-21 17:16:28 +00:00
try :
IMP: Changed configuration completely. New config processing, global/config variables changed over entire application. All changes in the configuration GUI take effect immediately after saving - no more need to restart Mylar, IMP: Added provider order sequence header to the top of the Providers tab for improved visibilty, IMP: Added completed download handling for both SABnzbd and NZBGet - will monitor active queue for downloads sent by Mylar and then post-process them accordingly (no more ComicRN), FIX: When recreating a pullist for a week that's not the current week, will now refresh series that are still missing data in case of late population, IMP: Removed loose/explict search options, and search results will now match more accurately to the terms entered, as well as being much quicker, IMP: Changed colour-codes on search results screen, green now indicates that the series is a print series, non-green indicates that it is either a HC/TPB/digital series and will also be indicated as such, IMP: Fixed weeklypull not refreshing some series due to legacy numbering on series deemed as Ended by Mylar, or the pull not refreshing propeprly and thereby were not elgible to be considered as a valid series for matching, IMP: Fixed problem with the scheduler of the weekly pull not running at the designated time, which causes problems when attempting to match series to the pullist for issues, IMP: Changed the autosnatch script so that the get.conf is no longer required to be maintained seperately. The get.conf config items are now located in the main config.ini and have to be repopulated initially, FIX: Fixed problem with the import count not resetting to 0 on subsequent runs of the Import A Directory option, IMP: Changed the sab api key from using the nzbkey to the full api key as is required for completed download handling, FIX: Fixed the retry option for newznab entries so that it should work across the board now for newznabs as opposed to only some providers, IMP: some minor code cleanup
2017-11-10 19:25:14 +00:00
while ( i < iss_len ) :
if result . getElementsByTagName ( ' name ' ) [ i ] . parentNode . nodeName == ' first_issue ' :
xmlfirst = result . getElementsByTagName ( ' issue_number ' ) [ i ] . firstChild . wholeText
if ' \xbd ' in xmlfirst :
xmlfirst = ' 1 ' #if the first issue is 1/2, just assume 1 for logistics
elif result . getElementsByTagName ( ' name ' ) [ i ] . parentNode . nodeName == ' last_issue ' :
xmllast = result . getElementsByTagName ( ' issue_number ' ) [ i ] . firstChild . wholeText
if all ( [ xmllast is not None , xmlfirst is not None ] ) :
break
i + = 1
2014-11-21 17:16:28 +00:00
except :
xmlfirst = ' 1 '
IMP: Changed configuration completely. New config processing, global/config variables changed over entire application. All changes in the configuration GUI take effect immediately after saving - no more need to restart Mylar, IMP: Added provider order sequence header to the top of the Providers tab for improved visibilty, IMP: Added completed download handling for both SABnzbd and NZBGet - will monitor active queue for downloads sent by Mylar and then post-process them accordingly (no more ComicRN), FIX: When recreating a pullist for a week that's not the current week, will now refresh series that are still missing data in case of late population, IMP: Removed loose/explict search options, and search results will now match more accurately to the terms entered, as well as being much quicker, IMP: Changed colour-codes on search results screen, green now indicates that the series is a print series, non-green indicates that it is either a HC/TPB/digital series and will also be indicated as such, IMP: Fixed weeklypull not refreshing some series due to legacy numbering on series deemed as Ended by Mylar, or the pull not refreshing propeprly and thereby were not elgible to be considered as a valid series for matching, IMP: Fixed problem with the scheduler of the weekly pull not running at the designated time, which causes problems when attempting to match series to the pullist for issues, IMP: Changed the autosnatch script so that the get.conf is no longer required to be maintained seperately. The get.conf config items are now located in the main config.ini and have to be repopulated initially, FIX: Fixed problem with the import count not resetting to 0 on subsequent runs of the Import A Directory option, IMP: Changed the sab api key from using the nzbkey to the full api key as is required for completed download handling, FIX: Fixed the retry option for newznab entries so that it should work across the board now for newznabs as opposed to only some providers, IMP: some minor code cleanup
2017-11-10 19:25:14 +00:00
if all ( [ xmlfirst == xmllast , xmlfirst . isdigit ( ) , xmlcnt == ' 0 ' ] ) :
xmlcnt = ' 1 '
2014-11-21 17:16:28 +00:00
#logger.info('There are : ' + str(xmlcnt) + ' issues in this series.')
#logger.info('The first issue started at # ' + str(xmlfirst))
2015-05-22 08:32:51 +00:00
2014-11-21 17:16:28 +00:00
cnt_numerical = int ( xmlcnt ) + int ( xmlfirst ) # (of issues + start of first issue = numerical range)
#logger.info('The maximum issue number should be roughly # ' + str(cnt_numerical))
#logger.info('The limiter (issue max that we know of) is # ' + str(limiter))
if cnt_numerical > = limiter :
2015-05-22 08:32:51 +00:00
cnl = len ( result . getElementsByTagName ( ' name ' ) )
2014-11-21 17:16:28 +00:00
cl = 0
xmlTag = ' None '
xmlimage = " cache/blankcover.jpg "
FIX: When manually post-processing files, if files were in more than one subdirectory level deep would use incorrect path resulting in error, FIX:(#1261, #1262) Traceback error when searching and returned value had volume, FIX: If filenames had the Volume label after the issue number, would fail to recognize the series title properly resulting in errors during scanning, FIX: Removing imported series from the Import List via dropdowns, IMP: Improved upon imports matching up to series that have multiple volumes by comparing issue counts & years, IMP: If an import is determined to need manual intervention, will display only the results that have passed the issue counts/years/name matching as opposed to showing all the results, FIX: When moving files during an import will now only move those files that are attached to the specific grouping by series title/volume, FIX: Import Manual Intervention results sortable by Year column now, FIX: Fixed an errant callback to home during configuration saves that caused web browser error when used with reverse proxy, FIX: During story arc searching if a story arc had a '-' in the arc name, would not be able to find it, FIX: When adding a story-arc, if files pre-existed in the directory attached to the arc would error, FIX: When adding a series, if was used for Folder Format and the volume label didn't exist would label directory with 'noversion', FIX: Selecting to metatag annuals via the manual metatag option on the series detail page, FIX: When retrieving from KAT and response returned was a 404 would ignore it, now will retry alternate torcache site
2016-04-25 15:35:12 +00:00
xml_lastissueid = ' None '
2014-11-21 17:16:28 +00:00
while ( cl < cnl ) :
if result . getElementsByTagName ( ' name ' ) [ cl ] . parentNode . nodeName == ' volume ' :
xmlTag = result . getElementsByTagName ( ' name ' ) [ cl ] . firstChild . wholeText
#break
if result . getElementsByTagName ( ' name ' ) [ cl ] . parentNode . nodeName == ' image ' :
xmlimage = result . getElementsByTagName ( ' super_url ' ) [ 0 ] . firstChild . wholeText
FIX: When manually post-processing files, if files were in more than one subdirectory level deep would use incorrect path resulting in error, FIX:(#1261, #1262) Traceback error when searching and returned value had volume, FIX: If filenames had the Volume label after the issue number, would fail to recognize the series title properly resulting in errors during scanning, FIX: Removing imported series from the Import List via dropdowns, IMP: Improved upon imports matching up to series that have multiple volumes by comparing issue counts & years, IMP: If an import is determined to need manual intervention, will display only the results that have passed the issue counts/years/name matching as opposed to showing all the results, FIX: When moving files during an import will now only move those files that are attached to the specific grouping by series title/volume, FIX: Import Manual Intervention results sortable by Year column now, FIX: Fixed an errant callback to home during configuration saves that caused web browser error when used with reverse proxy, FIX: During story arc searching if a story arc had a '-' in the arc name, would not be able to find it, FIX: When adding a story-arc, if files pre-existed in the directory attached to the arc would error, FIX: When adding a series, if was used for Folder Format and the volume label didn't exist would label directory with 'noversion', FIX: Selecting to metatag annuals via the manual metatag option on the series detail page, FIX: When retrieving from KAT and response returned was a 404 would ignore it, now will retry alternate torcache site
2016-04-25 15:35:12 +00:00
if result . getElementsByTagName ( ' name ' ) [ cl ] . parentNode . nodeName == ' last_issue ' :
xml_lastissueid = result . getElementsByTagName ( ' id ' ) [ cl ] . firstChild . wholeText
2014-11-21 17:16:28 +00:00
cl + = 1
IMP: (#844)Annuals now deleted from table on a Delete Series command, IMP: (#837)Better volume detection when searching for issues, IMP: (#842)(#808) Added some handling for issue titles when searching (it's not perfect, but it works for most), IMP: (#836)Added .cb7 for filechecking purposes only, IMP: (#830)issue numbers that are -1 can now be renamed, FIX: 'bi-annual' now fixed along with some other improvements for annual handling, IMP: Added version check for autoProcessComics and ComicRN - in order to help warn users of changes and the need to update these files since they are usually outside of the mylar git directory when being used, FIX:(#840) Boxcar2 should be working again, FIX: (#845) If search returned a filename that had no distinction between issue number and issue title and/or extra information, would error out if any of the words contained NOW, IMP: (#823) Added some better detection for NZBGet parameters when using ComicRN.py
2014-10-06 18:10:36 +00:00
if ( result . getElementsByTagName ( ' start_year ' ) [ 0 ] . firstChild ) is not None :
xmlYr = result . getElementsByTagName ( ' start_year ' ) [ 0 ] . firstChild . wholeText
else : xmlYr = " 0000 "
FIX: When manually post-processing files, if files were in more than one subdirectory level deep would use incorrect path resulting in error, FIX:(#1261, #1262) Traceback error when searching and returned value had volume, FIX: If filenames had the Volume label after the issue number, would fail to recognize the series title properly resulting in errors during scanning, FIX: Removing imported series from the Import List via dropdowns, IMP: Improved upon imports matching up to series that have multiple volumes by comparing issue counts & years, IMP: If an import is determined to need manual intervention, will display only the results that have passed the issue counts/years/name matching as opposed to showing all the results, FIX: When moving files during an import will now only move those files that are attached to the specific grouping by series title/volume, FIX: Import Manual Intervention results sortable by Year column now, FIX: Fixed an errant callback to home during configuration saves that caused web browser error when used with reverse proxy, FIX: During story arc searching if a story arc had a '-' in the arc name, would not be able to find it, FIX: When adding a story-arc, if files pre-existed in the directory attached to the arc would error, FIX: When adding a series, if was used for Folder Format and the volume label didn't exist would label directory with 'noversion', FIX: Selecting to metatag annuals via the manual metatag option on the series detail page, FIX: When retrieving from KAT and response returned was a 404 would ignore it, now will retry alternate torcache site
2016-04-25 15:35:12 +00:00
2016-04-26 18:55:12 +00:00
yearRange = [ ]
tmpYr = re . sub ( ' \ ? ' , ' ' , xmlYr )
FIX: When manually post-processing files, if files were in more than one subdirectory level deep would use incorrect path resulting in error, FIX:(#1261, #1262) Traceback error when searching and returned value had volume, FIX: If filenames had the Volume label after the issue number, would fail to recognize the series title properly resulting in errors during scanning, FIX: Removing imported series from the Import List via dropdowns, IMP: Improved upon imports matching up to series that have multiple volumes by comparing issue counts & years, IMP: If an import is determined to need manual intervention, will display only the results that have passed the issue counts/years/name matching as opposed to showing all the results, FIX: When moving files during an import will now only move those files that are attached to the specific grouping by series title/volume, FIX: Import Manual Intervention results sortable by Year column now, FIX: Fixed an errant callback to home during configuration saves that caused web browser error when used with reverse proxy, FIX: During story arc searching if a story arc had a '-' in the arc name, would not be able to find it, FIX: When adding a story-arc, if files pre-existed in the directory attached to the arc would error, FIX: When adding a series, if was used for Folder Format and the volume label didn't exist would label directory with 'noversion', FIX: Selecting to metatag annuals via the manual metatag option on the series detail page, FIX: When retrieving from KAT and response returned was a 404 would ignore it, now will retry alternate torcache site
2016-04-25 15:35:12 +00:00
2016-04-26 18:55:12 +00:00
if tmpYr . isdigit ( ) :
FIX: When manually post-processing files, if files were in more than one subdirectory level deep would use incorrect path resulting in error, FIX:(#1261, #1262) Traceback error when searching and returned value had volume, FIX: If filenames had the Volume label after the issue number, would fail to recognize the series title properly resulting in errors during scanning, FIX: Removing imported series from the Import List via dropdowns, IMP: Improved upon imports matching up to series that have multiple volumes by comparing issue counts & years, IMP: If an import is determined to need manual intervention, will display only the results that have passed the issue counts/years/name matching as opposed to showing all the results, FIX: When moving files during an import will now only move those files that are attached to the specific grouping by series title/volume, FIX: Import Manual Intervention results sortable by Year column now, FIX: Fixed an errant callback to home during configuration saves that caused web browser error when used with reverse proxy, FIX: During story arc searching if a story arc had a '-' in the arc name, would not be able to find it, FIX: When adding a story-arc, if files pre-existed in the directory attached to the arc would error, FIX: When adding a series, if was used for Folder Format and the volume label didn't exist would label directory with 'noversion', FIX: Selecting to metatag annuals via the manual metatag option on the series detail page, FIX: When retrieving from KAT and response returned was a 404 would ignore it, now will retry alternate torcache site
2016-04-25 15:35:12 +00:00
2016-05-16 17:10:45 +00:00
yearRange . append ( tmpYr )
2016-04-26 18:55:12 +00:00
tmpyearRange = int ( xmlcnt ) / 12
if float ( tmpyearRange ) : tmpyearRange + 1
possible_years = int ( tmpYr ) + tmpyearRange
for i in range ( int ( tmpYr ) , int ( possible_years ) , 1 ) :
if not any ( int ( x ) == int ( i ) for x in yearRange ) :
yearRange . append ( str ( i ) )
FIX: When manually post-processing files, if files were in more than one subdirectory level deep would use incorrect path resulting in error, FIX:(#1261, #1262) Traceback error when searching and returned value had volume, FIX: If filenames had the Volume label after the issue number, would fail to recognize the series title properly resulting in errors during scanning, FIX: Removing imported series from the Import List via dropdowns, IMP: Improved upon imports matching up to series that have multiple volumes by comparing issue counts & years, IMP: If an import is determined to need manual intervention, will display only the results that have passed the issue counts/years/name matching as opposed to showing all the results, FIX: When moving files during an import will now only move those files that are attached to the specific grouping by series title/volume, FIX: Import Manual Intervention results sortable by Year column now, FIX: Fixed an errant callback to home during configuration saves that caused web browser error when used with reverse proxy, FIX: During story arc searching if a story arc had a '-' in the arc name, would not be able to find it, FIX: When adding a story-arc, if files pre-existed in the directory attached to the arc would error, FIX: When adding a series, if was used for Folder Format and the volume label didn't exist would label directory with 'noversion', FIX: Selecting to metatag annuals via the manual metatag option on the series detail page, FIX: When retrieving from KAT and response returned was a 404 would ignore it, now will retry alternate torcache site
2016-04-25 15:35:12 +00:00
FIX:(#1336) Search would crash when attempting to search using a Torznab entry, FIX:(#1338) Added try/exception trap for mktime arguements that where trying to convert improperly formatted dates, FIX:(#1335) Manage Issue View shows annuals with incorrect Status in multiple status views, IMP: Alt_pull 2 method to check for new/updated pull-list every 4 hours instead of 24 (altpull 0/1), FIX: Fixed an error when parsing a filename that contained an numeric + alpha combination, FIX: On ComicDetails for a series would incorrectly indicate that every issue was a Digital Edition print, FIX: Watch action on weekly pull list fixed - will now either indicate as Add Series when the issue information is available on CV (altpull2), or will stay in a Watched status until the information is populated and then auto-add the given series, FIX: Removed Delete Arc dialog box, and redacted back to just 'Remove Arc' option, FIX:(#1332) Unable to Refresh comics via the Manage Comics tab, FIX:(#1135) IndexError when importing more than 100 comics (thnx razorsliph), FIX: When importing and comparing against years when no volume present, would use invalid years for comparisons resulting in manual intervention or wrong series, FIX: Manual Intervention linking during Import would return a blank page, FIX: Legacy mode for 32P would result in parse error, FIX: When Importing, and move files was not selected would pass invalid arguements and not update Import Results screen status, FIX: If copy mode enabled and Duplicate Dump Folder would attempt to reprocess/retag copied file repeatidly, FIX: Fixed some typos and removed some more unnecessary logging statements, IMP: Added option to export all issues in a Downloaded status (Manage / Advanced Options)
2016-08-10 01:21:08 +00:00
logger . fdebug ( ' [RESULT][ ' + str ( limityear ) + ' ] ComicName: ' + xmlTag + ' -- ' + str ( xmlYr ) + ' [Series years: ' + str ( yearRange ) + ' ] ' )
2016-05-16 17:10:45 +00:00
if tmpYr != xmlYr :
xmlYr = tmpYr
IMP: Changed configuration completely. New config processing, global/config variables changed over entire application. All changes in the configuration GUI take effect immediately after saving - no more need to restart Mylar, IMP: Added provider order sequence header to the top of the Providers tab for improved visibilty, IMP: Added completed download handling for both SABnzbd and NZBGet - will monitor active queue for downloads sent by Mylar and then post-process them accordingly (no more ComicRN), FIX: When recreating a pullist for a week that's not the current week, will now refresh series that are still missing data in case of late population, IMP: Removed loose/explict search options, and search results will now match more accurately to the terms entered, as well as being much quicker, IMP: Changed colour-codes on search results screen, green now indicates that the series is a print series, non-green indicates that it is either a HC/TPB/digital series and will also be indicated as such, IMP: Fixed weeklypull not refreshing some series due to legacy numbering on series deemed as Ended by Mylar, or the pull not refreshing propeprly and thereby were not elgible to be considered as a valid series for matching, IMP: Fixed problem with the scheduler of the weekly pull not running at the designated time, which causes problems when attempting to match series to the pullist for issues, IMP: Changed the autosnatch script so that the get.conf is no longer required to be maintained seperately. The get.conf config items are now located in the main config.ini and have to be repopulated initially, FIX: Fixed problem with the import count not resetting to 0 on subsequent runs of the Import A Directory option, IMP: Changed the sab api key from using the nzbkey to the full api key as is required for completed download handling, FIX: Fixed the retry option for newznab entries so that it should work across the board now for newznabs as opposed to only some providers, IMP: some minor code cleanup
2017-11-10 19:25:14 +00:00
FIX:(#1336) Search would crash when attempting to search using a Torznab entry, FIX:(#1338) Added try/exception trap for mktime arguements that where trying to convert improperly formatted dates, FIX:(#1335) Manage Issue View shows annuals with incorrect Status in multiple status views, IMP: Alt_pull 2 method to check for new/updated pull-list every 4 hours instead of 24 (altpull 0/1), FIX: Fixed an error when parsing a filename that contained an numeric + alpha combination, FIX: On ComicDetails for a series would incorrectly indicate that every issue was a Digital Edition print, FIX: Watch action on weekly pull list fixed - will now either indicate as Add Series when the issue information is available on CV (altpull2), or will stay in a Watched status until the information is populated and then auto-add the given series, FIX: Removed Delete Arc dialog box, and redacted back to just 'Remove Arc' option, FIX:(#1332) Unable to Refresh comics via the Manage Comics tab, FIX:(#1135) IndexError when importing more than 100 comics (thnx razorsliph), FIX: When importing and comparing against years when no volume present, would use invalid years for comparisons resulting in manual intervention or wrong series, FIX: Manual Intervention linking during Import would return a blank page, FIX: Legacy mode for 32P would result in parse error, FIX: When Importing, and move files was not selected would pass invalid arguements and not update Import Results screen status, FIX: If copy mode enabled and Duplicate Dump Folder would attempt to reprocess/retag copied file repeatidly, FIX: Fixed some typos and removed some more unnecessary logging statements, IMP: Added option to export all issues in a Downloaded status (Manage / Advanced Options)
2016-08-10 01:21:08 +00:00
if any ( map ( lambda v : v in limityear , yearRange ) ) or limityear == ' None ' :
IMP: (#844)Annuals now deleted from table on a Delete Series command, IMP: (#837)Better volume detection when searching for issues, IMP: (#842)(#808) Added some handling for issue titles when searching (it's not perfect, but it works for most), IMP: (#836)Added .cb7 for filechecking purposes only, IMP: (#830)issue numbers that are -1 can now be renamed, FIX: 'bi-annual' now fixed along with some other improvements for annual handling, IMP: Added version check for autoProcessComics and ComicRN - in order to help warn users of changes and the need to update these files since they are usually outside of the mylar git directory when being used, FIX:(#840) Boxcar2 should be working again, FIX: (#845) If search returned a filename that had no distinction between issue number and issue title and/or extra information, would error out if any of the words contained NOW, IMP: (#823) Added some better detection for NZBGet parameters when using ComicRN.py
2014-10-06 18:10:36 +00:00
xmlurl = result . getElementsByTagName ( ' site_detail_url ' ) [ 0 ] . firstChild . wholeText
2015-05-22 08:32:51 +00:00
idl = len ( result . getElementsByTagName ( ' id ' ) )
2014-11-21 17:16:28 +00:00
idt = 0
xmlid = None
while ( idt < idl ) :
if result . getElementsByTagName ( ' id ' ) [ idt ] . parentNode . nodeName == ' volume ' :
xmlid = result . getElementsByTagName ( ' id ' ) [ idt ] . firstChild . wholeText
break
idt + = 1
if xmlid is None :
logger . error ( ' Unable to figure out the comicid - skipping this : ' + str ( xmlurl ) )
2015-05-22 08:32:51 +00:00
continue
FIX: When retrieving feeds from 32p and in Auth mode, personal notification feeds contained some invalid html entries that weren't removed properly resulting in no results when attempting to match for downloading, FIX: When searching 32P, if title had a '/' within the title - Mylar would mistakingly skip it due to some previous exceptions that were made for CBT, FIX: Main page would quickly display & hide the have% column instead of always being hidden, FIX: Adjusted some incorrect spacing for non-alphanumeric characters when comparing search results (should result in better matching hopefully), FIX: When adding a series and the most recent issue was present on the weekly-pull list, it would sometimes not mark it as Wanted and auto-attempt to search for it (if auto mark Upcoming enabled), FIX: Added Test Connection button for 32P where it will test logon credentials as well as if Captcha is present, IMP: If captcha is enabled for 32p and signon is required because keys are stale, will not send authentication information and will just bypass as a provider, IMP: Test Connection button added for SABnzbd, IMP: Added ability to directly add torrents to rtorrent and apply label + download directory options (config.ini only atm), FIX: If a search result had a 'vol.' label in it, depending on how the format of the label was mylar would refuse to remove the volume which resulted in failed matches (also fixed a similar issue with failing to remove the volume label when comparing search results), FIX: When filechecking, if a series had a - in the title, will now account for it properly, IMP: Completely redid the filecheck module which allows for integration into other modules as well as more detailed failure logs, IMP: Added Dynamic handder integration into filechecker and subsequent modules that use it which allows for special characters to be replaced with any other type of character, IMP: Manual post-processing speed improved greatly due to new usage of filecheck module, IMP: Importer backend code redone to include new filecheck module, IMP: Added status/counter to import process, IMP: Added force unlock option to importer for failed imports, IMP: Added new status to Import labelled as 'Manual Intervention' for imports that need the user to manually select an option from an available list, FIX: When import said there were search results to view, but none available - would blank screen, IMP: Added a failure log entry showing all the failed files that weren't able to be scanned in during an import (will be in GUI eventually), IMP: if only partial metadata is available during import, Mylar will attempt to use what's available from the metatagging instead of picking all of one/other, IMP: Better grouping of series/volumes when viewing the import results page as well as now indicating if annuals are present within the files, IMP: Added a file-icon beside each imported item on the import result page which allows the user to view the files that are associated with the given series grouping, IMP: Added a blacklisted_publishers option to config.ini which will blacklist specific publishers from being returned during search / import results, FIX: If duplicate dump folder had a value, but duplicate dump wasn't enabled - would still use the duplicate dump folder during post-processing runs, FIX: (#1194) Patch to allow for fixed H1 elements for title (thnx chazlarson), FIX: Removed UnRAR dependency checks in cmtagmylar since not being used anymore, FIX: Fixed a problem with non-ascii characters being recognized during a file-check in certain cases, IMP: Attmept by Mylar to grab an alternate jpg from file when viewing the issue details if it complies with the naming conventions, FIX: Fixed some metatagging issues with ComicBookLover tags not being handled properly if they didn't exist, IMP: Dupecheck now has a failback if it's comparing a cbr/cbr, cbz/cbz and cbr/cbz-priority is enabled, FIX: Quick check added for when adding/refreshing a comic that if a cover already existed, it would delete the cover prior to the attempt to retrieve it, IMP: Added some additional handling for when searching/adding fails, FIX: If a story arc didn't have proper issue dates (or invalid ones) would error out on loading the story arc main page - usually when arcs were imported using a cbl file.
2016-04-07 17:09:06 +00:00
IMP: (#844)Annuals now deleted from table on a Delete Series command, IMP: (#837)Better volume detection when searching for issues, IMP: (#842)(#808) Added some handling for issue titles when searching (it's not perfect, but it works for most), IMP: (#836)Added .cb7 for filechecking purposes only, IMP: (#830)issue numbers that are -1 can now be renamed, FIX: 'bi-annual' now fixed along with some other improvements for annual handling, IMP: Added version check for autoProcessComics and ComicRN - in order to help warn users of changes and the need to update these files since they are usually outside of the mylar git directory when being used, FIX:(#840) Boxcar2 should be working again, FIX: (#845) If search returned a filename that had no distinction between issue number and issue title and/or extra information, would error out if any of the words contained NOW, IMP: (#823) Added some better detection for NZBGet parameters when using ComicRN.py
2014-10-06 18:10:36 +00:00
publishers = result . getElementsByTagName ( ' publisher ' )
if len ( publishers ) > 0 :
pubnames = publishers [ 0 ] . getElementsByTagName ( ' name ' )
if len ( pubnames ) > 0 :
xmlpub = pubnames [ 0 ] . firstChild . wholeText
else :
xmlpub = " Unknown "
else :
xmlpub = " Unknown "
2016-04-07 18:45:36 +00:00
FIX: When retrieving feeds from 32p and in Auth mode, personal notification feeds contained some invalid html entries that weren't removed properly resulting in no results when attempting to match for downloading, FIX: When searching 32P, if title had a '/' within the title - Mylar would mistakingly skip it due to some previous exceptions that were made for CBT, FIX: Main page would quickly display & hide the have% column instead of always being hidden, FIX: Adjusted some incorrect spacing for non-alphanumeric characters when comparing search results (should result in better matching hopefully), FIX: When adding a series and the most recent issue was present on the weekly-pull list, it would sometimes not mark it as Wanted and auto-attempt to search for it (if auto mark Upcoming enabled), FIX: Added Test Connection button for 32P where it will test logon credentials as well as if Captcha is present, IMP: If captcha is enabled for 32p and signon is required because keys are stale, will not send authentication information and will just bypass as a provider, IMP: Test Connection button added for SABnzbd, IMP: Added ability to directly add torrents to rtorrent and apply label + download directory options (config.ini only atm), FIX: If a search result had a 'vol.' label in it, depending on how the format of the label was mylar would refuse to remove the volume which resulted in failed matches (also fixed a similar issue with failing to remove the volume label when comparing search results), FIX: When filechecking, if a series had a - in the title, will now account for it properly, IMP: Completely redid the filecheck module which allows for integration into other modules as well as more detailed failure logs, IMP: Added Dynamic handder integration into filechecker and subsequent modules that use it which allows for special characters to be replaced with any other type of character, IMP: Manual post-processing speed improved greatly due to new usage of filecheck module, IMP: Importer backend code redone to include new filecheck module, IMP: Added status/counter to import process, IMP: Added force unlock option to importer for failed imports, IMP: Added new status to Import labelled as 'Manual Intervention' for imports that need the user to manually select an option from an available list, FIX: When import said there were search results to view, but none available - would blank screen, IMP: Added a failure log entry showing all the failed files that weren't able to be scanned in during an import (will be in GUI eventually), IMP: if only partial metadata is available during import, Mylar will attempt to use what's available from the metatagging instead of picking all of one/other, IMP: Better grouping of series/volumes when viewing the import results page as well as now indicating if annuals are present within the files, IMP: Added a file-icon beside each imported item on the import result page which allows the user to view the files that are associated with the given series grouping, IMP: Added a blacklisted_publishers option to config.ini which will blacklist specific publishers from being returned during search / import results, FIX: If duplicate dump folder had a value, but duplicate dump wasn't enabled - would still use the duplicate dump folder during post-processing runs, FIX: (#1194) Patch to allow for fixed H1 elements for title (thnx chazlarson), FIX: Removed UnRAR dependency checks in cmtagmylar since not being used anymore, FIX: Fixed a problem with non-ascii characters being recognized during a file-check in certain cases, IMP: Attmept by Mylar to grab an alternate jpg from file when viewing the issue details if it complies with the naming conventions, FIX: Fixed some metatagging issues with ComicBookLover tags not being handled properly if they didn't exist, IMP: Dupecheck now has a failback if it's comparing a cbr/cbr, cbz/cbz and cbr/cbz-priority is enabled, FIX: Quick check added for when adding/refreshing a comic that if a cover already existed, it would delete the cover prior to the attempt to retrieve it, IMP: Added some additional handling for when searching/adding fails, FIX: If a story arc didn't have proper issue dates (or invalid ones) would error out on loading the story arc main page - usually when arcs were imported using a cbl file.
2016-04-07 17:09:06 +00:00
#ignore specific publishers on a global scale here.
2017-10-25 18:33:56 +00:00
if mylar . CONFIG . BLACKLISTED_PUBLISHERS is not None and any ( [ x for x in mylar . CONFIG . BLACKLISTED_PUBLISHERS if x . lower ( ) == xmlpub . lower ( ) ] ) :
2016-04-07 18:45:36 +00:00
logger . fdebug ( ' Blacklisted publisher [ ' + xmlpub + ' ]. Ignoring this result. ' )
continue
IMP: (#844)Annuals now deleted from table on a Delete Series command, IMP: (#837)Better volume detection when searching for issues, IMP: (#842)(#808) Added some handling for issue titles when searching (it's not perfect, but it works for most), IMP: (#836)Added .cb7 for filechecking purposes only, IMP: (#830)issue numbers that are -1 can now be renamed, FIX: 'bi-annual' now fixed along with some other improvements for annual handling, IMP: Added version check for autoProcessComics and ComicRN - in order to help warn users of changes and the need to update these files since they are usually outside of the mylar git directory when being used, FIX:(#840) Boxcar2 should be working again, FIX: (#845) If search returned a filename that had no distinction between issue number and issue title and/or extra information, would error out if any of the words contained NOW, IMP: (#823) Added some better detection for NZBGet parameters when using ComicRN.py
2014-10-06 18:10:36 +00:00
try :
IMP: When scanning a folder using the Import a Directory option, upon completion a button to the Import Results Management page will be displayed to allow for easier access / acknowledgement of completion of scan, FIX: Fixed some issues with the search results page displaying too much information, IMP: Improved the ability to see alternate covers within issues when viewing selected issue details via the 'i' icon on the issue details page, IMP: Fixed the problem with issues that had 2015 dates and wouldn't scan in via manual post-processing, IMP: Added duplicate issue detection (beta) - enabled, when post-processing (manually or otherwise), will retain the issue that is bigger in filesize and won't post-process if it's smaller in size, instead of hammering the oldest copy, FIX: bunch of small fixes, and getting groundwork done for other things.
2014-12-18 18:30:57 +00:00
xmldesc = result . getElementsByTagName ( ' description ' ) [ 0 ] . firstChild . wholeText
IMP: (#844)Annuals now deleted from table on a Delete Series command, IMP: (#837)Better volume detection when searching for issues, IMP: (#842)(#808) Added some handling for issue titles when searching (it's not perfect, but it works for most), IMP: (#836)Added .cb7 for filechecking purposes only, IMP: (#830)issue numbers that are -1 can now be renamed, FIX: 'bi-annual' now fixed along with some other improvements for annual handling, IMP: Added version check for autoProcessComics and ComicRN - in order to help warn users of changes and the need to update these files since they are usually outside of the mylar git directory when being used, FIX:(#840) Boxcar2 should be working again, FIX: (#845) If search returned a filename that had no distinction between issue number and issue title and/or extra information, would error out if any of the words contained NOW, IMP: (#823) Added some better detection for NZBGet parameters when using ComicRN.py
2014-10-06 18:10:36 +00:00
except :
xmldesc = " None "
IMP: When scanning a folder using the Import a Directory option, upon completion a button to the Import Results Management page will be displayed to allow for easier access / acknowledgement of completion of scan, FIX: Fixed some issues with the search results page displaying too much information, IMP: Improved the ability to see alternate covers within issues when viewing selected issue details via the 'i' icon on the issue details page, IMP: Fixed the problem with issues that had 2015 dates and wouldn't scan in via manual post-processing, IMP: Added duplicate issue detection (beta) - enabled, when post-processing (manually or otherwise), will retain the issue that is bigger in filesize and won't post-process if it's smaller in size, instead of hammering the oldest copy, FIX: bunch of small fixes, and getting groundwork done for other things.
2014-12-18 18:30:57 +00:00
#this is needed to display brief synopsis for each series on search results page.
try :
xmldeck = result . getElementsByTagName ( ' deck ' ) [ 0 ] . firstChild . wholeText
except :
xmldeck = " None "
2016-07-20 19:49:41 +00:00
xmltype = None
if xmldeck != ' None ' :
IMP: Changed configuration completely. New config processing, global/config variables changed over entire application. All changes in the configuration GUI take effect immediately after saving - no more need to restart Mylar, IMP: Added provider order sequence header to the top of the Providers tab for improved visibilty, IMP: Added completed download handling for both SABnzbd and NZBGet - will monitor active queue for downloads sent by Mylar and then post-process them accordingly (no more ComicRN), FIX: When recreating a pullist for a week that's not the current week, will now refresh series that are still missing data in case of late population, IMP: Removed loose/explict search options, and search results will now match more accurately to the terms entered, as well as being much quicker, IMP: Changed colour-codes on search results screen, green now indicates that the series is a print series, non-green indicates that it is either a HC/TPB/digital series and will also be indicated as such, IMP: Fixed weeklypull not refreshing some series due to legacy numbering on series deemed as Ended by Mylar, or the pull not refreshing propeprly and thereby were not elgible to be considered as a valid series for matching, IMP: Fixed problem with the scheduler of the weekly pull not running at the designated time, which causes problems when attempting to match series to the pullist for issues, IMP: Changed the autosnatch script so that the get.conf is no longer required to be maintained seperately. The get.conf config items are now located in the main config.ini and have to be repopulated initially, FIX: Fixed problem with the import count not resetting to 0 on subsequent runs of the Import A Directory option, IMP: Changed the sab api key from using the nzbkey to the full api key as is required for completed download handling, FIX: Fixed the retry option for newznab entries so that it should work across the board now for newznabs as opposed to only some providers, IMP: some minor code cleanup
2017-11-10 19:25:14 +00:00
if any ( [ ' print ' in xmldeck . lower ( ) , ' digital ' in xmldeck . lower ( ) , ' paperback ' in xmldeck . lower ( ) , ' hardcover ' in xmldeck . lower ( ) ] ) :
2016-07-20 19:49:41 +00:00
if ' print ' in xmldeck . lower ( ) :
xmltype = ' Print '
elif ' digital ' in xmldeck . lower ( ) :
xmltype = ' Digital '
IMP: Changed configuration completely. New config processing, global/config variables changed over entire application. All changes in the configuration GUI take effect immediately after saving - no more need to restart Mylar, IMP: Added provider order sequence header to the top of the Providers tab for improved visibilty, IMP: Added completed download handling for both SABnzbd and NZBGet - will monitor active queue for downloads sent by Mylar and then post-process them accordingly (no more ComicRN), FIX: When recreating a pullist for a week that's not the current week, will now refresh series that are still missing data in case of late population, IMP: Removed loose/explict search options, and search results will now match more accurately to the terms entered, as well as being much quicker, IMP: Changed colour-codes on search results screen, green now indicates that the series is a print series, non-green indicates that it is either a HC/TPB/digital series and will also be indicated as such, IMP: Fixed weeklypull not refreshing some series due to legacy numbering on series deemed as Ended by Mylar, or the pull not refreshing propeprly and thereby were not elgible to be considered as a valid series for matching, IMP: Fixed problem with the scheduler of the weekly pull not running at the designated time, which causes problems when attempting to match series to the pullist for issues, IMP: Changed the autosnatch script so that the get.conf is no longer required to be maintained seperately. The get.conf config items are now located in the main config.ini and have to be repopulated initially, FIX: Fixed problem with the import count not resetting to 0 on subsequent runs of the Import A Directory option, IMP: Changed the sab api key from using the nzbkey to the full api key as is required for completed download handling, FIX: Fixed the retry option for newznab entries so that it should work across the board now for newznabs as opposed to only some providers, IMP: some minor code cleanup
2017-11-10 19:25:14 +00:00
elif ' paperback ' in xmldeck . lower ( ) :
xmltype = ' TPB '
elif ' hardcover ' in xmldeck . lower ( ) :
xmltype = ' HC '
2016-07-20 19:49:41 +00:00
if xmldesc != ' None ' and xmltype is None :
if ' print ' in xmldesc [ : 60 ] . lower ( ) and ' print edition can be found ' not in xmldesc . lower ( ) :
xmltype = ' Print '
elif ' digital ' in xmldesc [ : 60 ] . lower ( ) and ' digital edition can be found ' not in xmldesc . lower ( ) :
xmltype = ' Digital '
IMP: Changed configuration completely. New config processing, global/config variables changed over entire application. All changes in the configuration GUI take effect immediately after saving - no more need to restart Mylar, IMP: Added provider order sequence header to the top of the Providers tab for improved visibilty, IMP: Added completed download handling for both SABnzbd and NZBGet - will monitor active queue for downloads sent by Mylar and then post-process them accordingly (no more ComicRN), FIX: When recreating a pullist for a week that's not the current week, will now refresh series that are still missing data in case of late population, IMP: Removed loose/explict search options, and search results will now match more accurately to the terms entered, as well as being much quicker, IMP: Changed colour-codes on search results screen, green now indicates that the series is a print series, non-green indicates that it is either a HC/TPB/digital series and will also be indicated as such, IMP: Fixed weeklypull not refreshing some series due to legacy numbering on series deemed as Ended by Mylar, or the pull not refreshing propeprly and thereby were not elgible to be considered as a valid series for matching, IMP: Fixed problem with the scheduler of the weekly pull not running at the designated time, which causes problems when attempting to match series to the pullist for issues, IMP: Changed the autosnatch script so that the get.conf is no longer required to be maintained seperately. The get.conf config items are now located in the main config.ini and have to be repopulated initially, FIX: Fixed problem with the import count not resetting to 0 on subsequent runs of the Import A Directory option, IMP: Changed the sab api key from using the nzbkey to the full api key as is required for completed download handling, FIX: Fixed the retry option for newznab entries so that it should work across the board now for newznabs as opposed to only some providers, IMP: some minor code cleanup
2017-11-10 19:25:14 +00:00
elif ' paperback ' in xmldesc [ : 60 ] . lower ( ) and ' paperback can be found ' not in xmldesc . lower ( ) :
xmltype = ' TPB '
elif ' hardcover ' in xmldesc [ : 60 ] . lower ( ) and ' hardcover can be found ' not in xmldesc . lower ( ) :
xmltype = ' HC '
FIX:(#1336) Search would crash when attempting to search using a Torznab entry, FIX:(#1338) Added try/exception trap for mktime arguements that where trying to convert improperly formatted dates, FIX:(#1335) Manage Issue View shows annuals with incorrect Status in multiple status views, IMP: Alt_pull 2 method to check for new/updated pull-list every 4 hours instead of 24 (altpull 0/1), FIX: Fixed an error when parsing a filename that contained an numeric + alpha combination, FIX: On ComicDetails for a series would incorrectly indicate that every issue was a Digital Edition print, FIX: Watch action on weekly pull list fixed - will now either indicate as Add Series when the issue information is available on CV (altpull2), or will stay in a Watched status until the information is populated and then auto-add the given series, FIX: Removed Delete Arc dialog box, and redacted back to just 'Remove Arc' option, FIX:(#1332) Unable to Refresh comics via the Manage Comics tab, FIX:(#1135) IndexError when importing more than 100 comics (thnx razorsliph), FIX: When importing and comparing against years when no volume present, would use invalid years for comparisons resulting in manual intervention or wrong series, FIX: Manual Intervention linking during Import would return a blank page, FIX: Legacy mode for 32P would result in parse error, FIX: When Importing, and move files was not selected would pass invalid arguements and not update Import Results screen status, FIX: If copy mode enabled and Duplicate Dump Folder would attempt to reprocess/retag copied file repeatidly, FIX: Fixed some typos and removed some more unnecessary logging statements, IMP: Added option to export all issues in a Downloaded status (Manage / Advanced Options)
2016-08-10 01:21:08 +00:00
else :
xmltype = ' Print '
IMP: When scanning a folder using the Import a Directory option, upon completion a button to the Import Results Management page will be displayed to allow for easier access / acknowledgement of completion of scan, FIX: Fixed some issues with the search results page displaying too much information, IMP: Improved the ability to see alternate covers within issues when viewing selected issue details via the 'i' icon on the issue details page, IMP: Fixed the problem with issues that had 2015 dates and wouldn't scan in via manual post-processing, IMP: Added duplicate issue detection (beta) - enabled, when post-processing (manually or otherwise), will retain the issue that is bigger in filesize and won't post-process if it's smaller in size, instead of hammering the oldest copy, FIX: bunch of small fixes, and getting groundwork done for other things.
2014-12-18 18:30:57 +00:00
2014-10-08 06:45:51 +00:00
if xmlid in comicLibrary :
2014-10-08 18:17:44 +00:00
haveit = comicLibrary [ xmlid ]
2014-10-08 06:45:51 +00:00
else :
haveit = " No "
IMP: (#844)Annuals now deleted from table on a Delete Series command, IMP: (#837)Better volume detection when searching for issues, IMP: (#842)(#808) Added some handling for issue titles when searching (it's not perfect, but it works for most), IMP: (#836)Added .cb7 for filechecking purposes only, IMP: (#830)issue numbers that are -1 can now be renamed, FIX: 'bi-annual' now fixed along with some other improvements for annual handling, IMP: Added version check for autoProcessComics and ComicRN - in order to help warn users of changes and the need to update these files since they are usually outside of the mylar git directory when being used, FIX:(#840) Boxcar2 should be working again, FIX: (#845) If search returned a filename that had no distinction between issue number and issue title and/or extra information, would error out if any of the words contained NOW, IMP: (#823) Added some better detection for NZBGet parameters when using ComicRN.py
2014-10-06 18:10:36 +00:00
comiclist . append ( {
2014-11-21 17:16:28 +00:00
' name ' : xmlTag ,
' comicyear ' : xmlYr ,
' comicid ' : xmlid ,
' url ' : xmlurl ,
' issues ' : xmlcnt ,
' comicimage ' : xmlimage ,
IMP: (#844)Annuals now deleted from table on a Delete Series command, IMP: (#837)Better volume detection when searching for issues, IMP: (#842)(#808) Added some handling for issue titles when searching (it's not perfect, but it works for most), IMP: (#836)Added .cb7 for filechecking purposes only, IMP: (#830)issue numbers that are -1 can now be renamed, FIX: 'bi-annual' now fixed along with some other improvements for annual handling, IMP: Added version check for autoProcessComics and ComicRN - in order to help warn users of changes and the need to update these files since they are usually outside of the mylar git directory when being used, FIX:(#840) Boxcar2 should be working again, FIX: (#845) If search returned a filename that had no distinction between issue number and issue title and/or extra information, would error out if any of the words contained NOW, IMP: (#823) Added some better detection for NZBGet parameters when using ComicRN.py
2014-10-06 18:10:36 +00:00
' publisher ' : xmlpub ,
2014-10-08 06:45:51 +00:00
' description ' : xmldesc ,
IMP: When scanning a folder using the Import a Directory option, upon completion a button to the Import Results Management page will be displayed to allow for easier access / acknowledgement of completion of scan, FIX: Fixed some issues with the search results page displaying too much information, IMP: Improved the ability to see alternate covers within issues when viewing selected issue details via the 'i' icon on the issue details page, IMP: Fixed the problem with issues that had 2015 dates and wouldn't scan in via manual post-processing, IMP: Added duplicate issue detection (beta) - enabled, when post-processing (manually or otherwise), will retain the issue that is bigger in filesize and won't post-process if it's smaller in size, instead of hammering the oldest copy, FIX: bunch of small fixes, and getting groundwork done for other things.
2014-12-18 18:30:57 +00:00
' deck ' : xmldeck ,
2016-07-20 19:49:41 +00:00
' type ' : xmltype ,
FIX: When manually post-processing files, if files were in more than one subdirectory level deep would use incorrect path resulting in error, FIX:(#1261, #1262) Traceback error when searching and returned value had volume, FIX: If filenames had the Volume label after the issue number, would fail to recognize the series title properly resulting in errors during scanning, FIX: Removing imported series from the Import List via dropdowns, IMP: Improved upon imports matching up to series that have multiple volumes by comparing issue counts & years, IMP: If an import is determined to need manual intervention, will display only the results that have passed the issue counts/years/name matching as opposed to showing all the results, FIX: When moving files during an import will now only move those files that are attached to the specific grouping by series title/volume, FIX: Import Manual Intervention results sortable by Year column now, FIX: Fixed an errant callback to home during configuration saves that caused web browser error when used with reverse proxy, FIX: During story arc searching if a story arc had a '-' in the arc name, would not be able to find it, FIX: When adding a story-arc, if files pre-existed in the directory attached to the arc would error, FIX: When adding a series, if was used for Folder Format and the volume label didn't exist would label directory with 'noversion', FIX: Selecting to metatag annuals via the manual metatag option on the series detail page, FIX: When retrieving from KAT and response returned was a 404 would ignore it, now will retry alternate torcache site
2016-04-25 15:35:12 +00:00
' haveit ' : haveit ,
' lastissueid ' : xml_lastissueid ,
' seriesrange ' : yearRange # returning additional information about series run polled from CV
IMP: (#844)Annuals now deleted from table on a Delete Series command, IMP: (#837)Better volume detection when searching for issues, IMP: (#842)(#808) Added some handling for issue titles when searching (it's not perfect, but it works for most), IMP: (#836)Added .cb7 for filechecking purposes only, IMP: (#830)issue numbers that are -1 can now be renamed, FIX: 'bi-annual' now fixed along with some other improvements for annual handling, IMP: Added version check for autoProcessComics and ComicRN - in order to help warn users of changes and the need to update these files since they are usually outside of the mylar git directory when being used, FIX:(#840) Boxcar2 should be working again, FIX: (#845) If search returned a filename that had no distinction between issue number and issue title and/or extra information, would error out if any of the words contained NOW, IMP: (#823) Added some better detection for NZBGet parameters when using ComicRN.py
2014-10-06 18:10:36 +00:00
} )
IMP: Changed configuration completely. New config processing, global/config variables changed over entire application. All changes in the configuration GUI take effect immediately after saving - no more need to restart Mylar, IMP: Added provider order sequence header to the top of the Providers tab for improved visibilty, IMP: Added completed download handling for both SABnzbd and NZBGet - will monitor active queue for downloads sent by Mylar and then post-process them accordingly (no more ComicRN), FIX: When recreating a pullist for a week that's not the current week, will now refresh series that are still missing data in case of late population, IMP: Removed loose/explict search options, and search results will now match more accurately to the terms entered, as well as being much quicker, IMP: Changed colour-codes on search results screen, green now indicates that the series is a print series, non-green indicates that it is either a HC/TPB/digital series and will also be indicated as such, IMP: Fixed weeklypull not refreshing some series due to legacy numbering on series deemed as Ended by Mylar, or the pull not refreshing propeprly and thereby were not elgible to be considered as a valid series for matching, IMP: Fixed problem with the scheduler of the weekly pull not running at the designated time, which causes problems when attempting to match series to the pullist for issues, IMP: Changed the autosnatch script so that the get.conf is no longer required to be maintained seperately. The get.conf config items are now located in the main config.ini and have to be repopulated initially, FIX: Fixed problem with the import count not resetting to 0 on subsequent runs of the Import A Directory option, IMP: Changed the sab api key from using the nzbkey to the full api key as is required for completed download handling, FIX: Fixed the retry option for newznab entries so that it should work across the board now for newznabs as opposed to only some providers, IMP: some minor code cleanup
2017-11-10 19:25:14 +00:00
#logger.fdebug('year: %s - constraint met: %s [%s] --- 4050-%s' % (xmlYr,xmlTag,xmlYr,xmlid))
IMP: (#844)Annuals now deleted from table on a Delete Series command, IMP: (#837)Better volume detection when searching for issues, IMP: (#842)(#808) Added some handling for issue titles when searching (it's not perfect, but it works for most), IMP: (#836)Added .cb7 for filechecking purposes only, IMP: (#830)issue numbers that are -1 can now be renamed, FIX: 'bi-annual' now fixed along with some other improvements for annual handling, IMP: Added version check for autoProcessComics and ComicRN - in order to help warn users of changes and the need to update these files since they are usually outside of the mylar git directory when being used, FIX:(#840) Boxcar2 should be working again, FIX: (#845) If search returned a filename that had no distinction between issue number and issue title and/or extra information, would error out if any of the words contained NOW, IMP: (#823) Added some better detection for NZBGet parameters when using ComicRN.py
2014-10-06 18:10:36 +00:00
else :
2015-05-22 08:32:51 +00:00
#logger.fdebug('year: ' + str(xmlYr) + ' - contraint not met. Has to be within ' + str(limityear))
IMP: Changed configuration completely. New config processing, global/config variables changed over entire application. All changes in the configuration GUI take effect immediately after saving - no more need to restart Mylar, IMP: Added provider order sequence header to the top of the Providers tab for improved visibilty, IMP: Added completed download handling for both SABnzbd and NZBGet - will monitor active queue for downloads sent by Mylar and then post-process them accordingly (no more ComicRN), FIX: When recreating a pullist for a week that's not the current week, will now refresh series that are still missing data in case of late population, IMP: Removed loose/explict search options, and search results will now match more accurately to the terms entered, as well as being much quicker, IMP: Changed colour-codes on search results screen, green now indicates that the series is a print series, non-green indicates that it is either a HC/TPB/digital series and will also be indicated as such, IMP: Fixed weeklypull not refreshing some series due to legacy numbering on series deemed as Ended by Mylar, or the pull not refreshing propeprly and thereby were not elgible to be considered as a valid series for matching, IMP: Fixed problem with the scheduler of the weekly pull not running at the designated time, which causes problems when attempting to match series to the pullist for issues, IMP: Changed the autosnatch script so that the get.conf is no longer required to be maintained seperately. The get.conf config items are now located in the main config.ini and have to be repopulated initially, FIX: Fixed problem with the import count not resetting to 0 on subsequent runs of the Import A Directory option, IMP: Changed the sab api key from using the nzbkey to the full api key as is required for completed download handling, FIX: Fixed the retry option for newznab entries so that it should work across the board now for newznabs as opposed to only some providers, IMP: some minor code cleanup
2017-11-10 19:25:14 +00:00
pass
2015-05-22 08:32:51 +00:00
n + = 1
2013-03-27 07:03:10 +00:00
#search results are limited to 100 and by pagination now...let's account for this.
countResults = countResults + 100
2015-05-22 08:32:51 +00:00
IMP: Changed configuration completely. New config processing, global/config variables changed over entire application. All changes in the configuration GUI take effect immediately after saving - no more need to restart Mylar, IMP: Added provider order sequence header to the top of the Providers tab for improved visibilty, IMP: Added completed download handling for both SABnzbd and NZBGet - will monitor active queue for downloads sent by Mylar and then post-process them accordingly (no more ComicRN), FIX: When recreating a pullist for a week that's not the current week, will now refresh series that are still missing data in case of late population, IMP: Removed loose/explict search options, and search results will now match more accurately to the terms entered, as well as being much quicker, IMP: Changed colour-codes on search results screen, green now indicates that the series is a print series, non-green indicates that it is either a HC/TPB/digital series and will also be indicated as such, IMP: Fixed weeklypull not refreshing some series due to legacy numbering on series deemed as Ended by Mylar, or the pull not refreshing propeprly and thereby were not elgible to be considered as a valid series for matching, IMP: Fixed problem with the scheduler of the weekly pull not running at the designated time, which causes problems when attempting to match series to the pullist for issues, IMP: Changed the autosnatch script so that the get.conf is no longer required to be maintained seperately. The get.conf config items are now located in the main config.ini and have to be repopulated initially, FIX: Fixed problem with the import count not resetting to 0 on subsequent runs of the Import A Directory option, IMP: Changed the sab api key from using the nzbkey to the full api key as is required for completed download handling, FIX: Fixed the retry option for newznab entries so that it should work across the board now for newznabs as opposed to only some providers, IMP: some minor code cleanup
2017-11-10 19:25:14 +00:00
return comiclist
2015-05-27 05:40:38 +00:00
def storyarcinfo ( xmlid ) :
2016-12-30 13:40:54 +00:00
comicLibrary = listStoryArcs ( )
2015-05-27 05:40:38 +00:00
arcinfo = { }
2017-10-25 18:33:56 +00:00
if mylar . CONFIG . COMICVINE_API == ' None ' or mylar . CONFIG . COMICVINE_API is None :
logger . warn ( ' You have not specified your own ComicVine API key - this is a requirement. Get your own @ http://api.comicvine.com. ' )
return
2015-05-27 05:40:38 +00:00
else :
2017-10-25 18:33:56 +00:00
comicapi = mylar . CONFIG . COMICVINE_API
2015-05-27 05:40:38 +00:00
#respawn to the exact id for the story arc and count the # of issues present.
FIX:(#1466) When post-processing if filename did not have issue year or volume, would error, IMP:(#1418) Separate copy/move/hardlink/symlink option for Story arcs, IMP:(#1417) Added story arc format folder to story arc options - can now specify folder format for story arc naming, IMP:(#1374) Added rtorrent_authentication to config.ini allowing user to specify digest/basic authentication, as well as the rpc_url (rtorrent_rpc_url) and ssl (rtorrent_ssl). Rtorrent_host no longer requires protocol in field, IMP: Instead of hitting the intermediary/32P site to get the torrent group, will now retain specific id for torrent group to speed up searches, FIX: (#1472) Fixed some problems with series being searched against 32P which had special characters and would return no results, FIX: Added some specific exception catches when attempting to retrieve search results for nzb providers, FIX: Fixed publisher retaining problem for story arcs (note story arcs will have to be refreshed), FIX: Fixed a problem with the pull-list matching up to the correct series when the issue was an annual for a given series with the alt_pull 2 method (would create an invalid link on the Wanted tab), IMP:(#1470) Added config.ini option to specify folder-formatting for weekly folder numbering, 0=YYYY-WN, 1=YYYY-MM-DD, IMP: (#1471) Added quick check to ensure that newznab host entries do not contain leading/trailing spaces and will strip accordingly, FIX:(#1476) Annuals now have option to add to reading list via the series detail page, IMP: Fixed password fields to allow for special characters within password, IMP: Weekly pull will now correctly show status for previous/future weeks for watchlisted series, IMP: Download option on pullist will work for previous weeks, FIX: Fixed some issues with one-off and story arc post-processing using different file operations, FIX: Fixed error when performing an 'Force Update Active Comics' from the Manage tab (or the scheduled updating of all active comics), IMP: Storyarc issues that are not on watchlist will now retain value for volume to help when searching and file-checking/post-processing, FIX:(#1473) Changed pullist dates on pullist page to unicode to allow for non-English presentation
2016-12-02 17:45:49 +00:00
ARCPULL_URL = mylar . CVURL + ' story_arc/4045- ' + str ( xmlid ) + ' /?api_key= ' + str ( comicapi ) + ' &field_list=issues,publisher,name,first_appeared_in_issue,deck,image&format=xml&offset=0 '
#logger.fdebug('arcpull_url:' + str(ARCPULL_URL))
2015-11-23 08:25:59 +00:00
#new CV API restriction - one api request / second.
2017-10-25 18:33:56 +00:00
if mylar . CONFIG . CVAPI_RATE is None or mylar . CONFIG . CVAPI_RATE < 2 :
2015-11-23 08:25:59 +00:00
time . sleep ( 2 )
else :
2017-10-25 18:33:56 +00:00
time . sleep ( mylar . CONFIG . CVAPI_RATE )
2015-11-23 08:25:59 +00:00
FIX: Added user-agent to all CV queries, and switched all queries to utilize the requests module, FIX: Added User-Agent to ComicTagger and switched to requests module for all CV queries, FIX: Removed libraryscan option from GUI as no longer used, FIX: Ability to sort 'Have' column on main index page properly now (by % complete), FIX: Fixed some display issues with the series detail page and displaying, FIX: Metatagging should now be working again for story-arc, and one-off post-processing, FIX: Fixed post-processing problem with decimal places if number was padded, FIX: Updated CV-URL to point to new CV api location, FIX: Fixed problem when file-checking and series contained numbers and/or decimals, would take the modified series name instead of the actual series name resulting in some missed matches, IMP: Added another keyword to ignore when checking for annuals against a particular series, FIX:(#1210) When importing files, if issues were meta-tagged with CVS would not scan the metadata properly, FIX: Fixed checkboxes on annuals table so drop-down will work again (just doesn't refresh the screen afterwards atm), FIX:(#1182) Notifications not respecting http base, IMP: Added option to specify SSL Verification on/off per provider when searching/downloading, FIX: Possible fix for cache_dir not being used on restarts of mylar, FIX: configparser check would only check on linux systems, and if not present would cause errors during post-processing on non nix-based systems, FIX:(#1181) Manual renaming an entire series would fail if lowercase_filenames was enabled as an option
2016-02-25 16:40:09 +00:00
#download the file:
payload = None
2015-05-27 05:40:38 +00:00
try :
2017-10-25 18:33:56 +00:00
r = requests . get ( ARCPULL_URL , params = payload , verify = mylar . CONFIG . CV_VERIFY , headers = mylar . CV_HEADERS )
FIX: Added user-agent to all CV queries, and switched all queries to utilize the requests module, FIX: Added User-Agent to ComicTagger and switched to requests module for all CV queries, FIX: Removed libraryscan option from GUI as no longer used, FIX: Ability to sort 'Have' column on main index page properly now (by % complete), FIX: Fixed some display issues with the series detail page and displaying, FIX: Metatagging should now be working again for story-arc, and one-off post-processing, FIX: Fixed post-processing problem with decimal places if number was padded, FIX: Updated CV-URL to point to new CV api location, FIX: Fixed problem when file-checking and series contained numbers and/or decimals, would take the modified series name instead of the actual series name resulting in some missed matches, IMP: Added another keyword to ignore when checking for annuals against a particular series, FIX:(#1210) When importing files, if issues were meta-tagged with CVS would not scan the metadata properly, FIX: Fixed checkboxes on annuals table so drop-down will work again (just doesn't refresh the screen afterwards atm), FIX:(#1182) Notifications not respecting http base, IMP: Added option to specify SSL Verification on/off per provider when searching/downloading, FIX: Possible fix for cache_dir not being used on restarts of mylar, FIX: configparser check would only check on linux systems, and if not present would cause errors during post-processing on non nix-based systems, FIX:(#1181) Manual renaming an entire series would fail if lowercase_filenames was enabled as an option
2016-02-25 16:40:09 +00:00
except Exception , e :
logger . warn ( ' Error fetching data from ComicVine: %s ' % ( e ) )
2015-05-27 05:40:38 +00:00
return
FIX: Added user-agent to all CV queries, and switched all queries to utilize the requests module, FIX: Added User-Agent to ComicTagger and switched to requests module for all CV queries, FIX: Removed libraryscan option from GUI as no longer used, FIX: Ability to sort 'Have' column on main index page properly now (by % complete), FIX: Fixed some display issues with the series detail page and displaying, FIX: Metatagging should now be working again for story-arc, and one-off post-processing, FIX: Fixed post-processing problem with decimal places if number was padded, FIX: Updated CV-URL to point to new CV api location, FIX: Fixed problem when file-checking and series contained numbers and/or decimals, would take the modified series name instead of the actual series name resulting in some missed matches, IMP: Added another keyword to ignore when checking for annuals against a particular series, FIX:(#1210) When importing files, if issues were meta-tagged with CVS would not scan the metadata properly, FIX: Fixed checkboxes on annuals table so drop-down will work again (just doesn't refresh the screen afterwards atm), FIX:(#1182) Notifications not respecting http base, IMP: Added option to specify SSL Verification on/off per provider when searching/downloading, FIX: Possible fix for cache_dir not being used on restarts of mylar, FIX: configparser check would only check on linux systems, and if not present would cause errors during post-processing on non nix-based systems, FIX:(#1181) Manual renaming an entire series would fail if lowercase_filenames was enabled as an option
2016-02-25 16:40:09 +00:00
# try:
# file = urllib2.urlopen(ARCPULL_URL)
# except urllib2.HTTPError, err:
# logger.error('err : ' + str(err))
# logger.error('There was a major problem retrieving data from ComicVine - on their end.')
# return
# arcdata = file.read()
# file.close()
arcdom = parseString ( r . content ) #(arcdata)
2015-05-27 05:40:38 +00:00
try :
logger . fdebug ( ' story_arc ascension ' )
issuecount = len ( arcdom . getElementsByTagName ( ' issue ' ) )
issuedom = arcdom . getElementsByTagName ( ' issue ' )
isc = 0
arclist = ' '
2015-08-11 21:56:22 +00:00
ordernum = 1
2015-05-27 05:40:38 +00:00
for isd in issuedom :
zeline = isd . getElementsByTagName ( ' id ' )
isdlen = len ( zeline )
isb = 0
while ( isb < isdlen ) :
if isc == 0 :
2015-08-11 21:56:22 +00:00
arclist = str ( zeline [ isb ] . firstChild . wholeText ) . strip ( ) + ' , ' + str ( ordernum )
2015-05-27 05:40:38 +00:00
else :
2015-08-11 21:56:22 +00:00
arclist + = ' | ' + str ( zeline [ isb ] . firstChild . wholeText ) . strip ( ) + ' , ' + str ( ordernum )
ordernum + = 1
2015-05-27 05:40:38 +00:00
isb + = 1
isc + = 1
except :
logger . fdebug ( ' unable to retrive issue count - nullifying value. ' )
issuecount = 0
try :
firstid = None
arcyear = None
fid = len ( arcdom . getElementsByTagName ( ' id ' ) )
fi = 0
while ( fi < fid ) :
if arcdom . getElementsByTagName ( ' id ' ) [ fi ] . parentNode . nodeName == ' first_appeared_in_issue ' :
if not arcdom . getElementsByTagName ( ' id ' ) [ fi ] . firstChild . wholeText == xmlid :
logger . fdebug ( ' hit it. ' )
firstid = arcdom . getElementsByTagName ( ' id ' ) [ fi ] . firstChild . wholeText
break # - dont' break out here as we want to gather ALL the issue ID's since it's here
fi + = 1
logger . fdebug ( ' firstid: ' + str ( firstid ) )
if firstid is not None :
firstdom = cv . pulldetails ( comicid = None , type = ' firstissue ' , issueid = firstid )
logger . fdebug ( ' success ' )
2018-03-05 17:25:36 +00:00
arcyear = cv . Getissue ( firstid , firstdom , ' firstissue ' )
2015-05-27 05:40:38 +00:00
except :
logger . fdebug ( ' Unable to retrieve first issue details. Not caclulating at this time. ' )
IMP: Updated CT to a newer version - changed to use rarfile, fixes size invalid errors with some cbr's encountered during conversions, IMP: Search and Post-Processing Queues now added in order to queue up all searches/post-processing in sequence instead of loading lists and iterating through it and encountering various lock-related errors, IMP: In most cases, will now post-process directly against comicid/issueid/storyarcid instead of relying on relationship via nzb/torrent, FIX: realigned some checkboxes on series detail page, FIX: in configuration, indicated that ComicRN cannot be used when Completed Download Handling is enabled for a client now, FIX: Fix for issues named as 'special' that are part of the annuals section, but would not work in various portions (also will allow for naming differences in the future hopefully), FIX: Will now trap error properly when file cannot be located during dupecheck so that post-processing can continue without error, FIX: Fixed newznab test option on configuration screen so that it will return the correct response (previous was an invalid api endpoint), FIX: When retrieving image for storyarcs would error when no image was present and stop the process, FIX: accounted for some incorrect newznab responses that would return a non-api endpoint when retrieving via rss, which resulted in invalid nzbs, FIX: in some cases, weekly pull snatches would not update properly on the weekly page due to some db values not being written
2018-06-14 17:07:39 +00:00
try :
2015-05-27 05:40:38 +00:00
xmlimage = arcdom . getElementsByTagName ( ' super_url ' ) [ 0 ] . firstChild . wholeText
IMP: Updated CT to a newer version - changed to use rarfile, fixes size invalid errors with some cbr's encountered during conversions, IMP: Search and Post-Processing Queues now added in order to queue up all searches/post-processing in sequence instead of loading lists and iterating through it and encountering various lock-related errors, IMP: In most cases, will now post-process directly against comicid/issueid/storyarcid instead of relying on relationship via nzb/torrent, FIX: realigned some checkboxes on series detail page, FIX: in configuration, indicated that ComicRN cannot be used when Completed Download Handling is enabled for a client now, FIX: Fix for issues named as 'special' that are part of the annuals section, but would not work in various portions (also will allow for naming differences in the future hopefully), FIX: Will now trap error properly when file cannot be located during dupecheck so that post-processing can continue without error, FIX: Fixed newznab test option on configuration screen so that it will return the correct response (previous was an invalid api endpoint), FIX: When retrieving image for storyarcs would error when no image was present and stop the process, FIX: accounted for some incorrect newznab responses that would return a non-api endpoint when retrieving via rss, which resulted in invalid nzbs, FIX: in some cases, weekly pull snatches would not update properly on the weekly page due to some db values not being written
2018-06-14 17:07:39 +00:00
except :
2015-05-27 05:40:38 +00:00
xmlimage = " cache/blankcover.jpg "
try :
xmldesc = arcdom . getElementsByTagName ( ' desc ' ) [ 0 ] . firstChild . wholeText
except :
xmldesc = " None "
FIX:(#1466) When post-processing if filename did not have issue year or volume, would error, IMP:(#1418) Separate copy/move/hardlink/symlink option for Story arcs, IMP:(#1417) Added story arc format folder to story arc options - can now specify folder format for story arc naming, IMP:(#1374) Added rtorrent_authentication to config.ini allowing user to specify digest/basic authentication, as well as the rpc_url (rtorrent_rpc_url) and ssl (rtorrent_ssl). Rtorrent_host no longer requires protocol in field, IMP: Instead of hitting the intermediary/32P site to get the torrent group, will now retain specific id for torrent group to speed up searches, FIX: (#1472) Fixed some problems with series being searched against 32P which had special characters and would return no results, FIX: Added some specific exception catches when attempting to retrieve search results for nzb providers, FIX: Fixed publisher retaining problem for story arcs (note story arcs will have to be refreshed), FIX: Fixed a problem with the pull-list matching up to the correct series when the issue was an annual for a given series with the alt_pull 2 method (would create an invalid link on the Wanted tab), IMP:(#1470) Added config.ini option to specify folder-formatting for weekly folder numbering, 0=YYYY-WN, 1=YYYY-MM-DD, IMP: (#1471) Added quick check to ensure that newznab host entries do not contain leading/trailing spaces and will strip accordingly, FIX:(#1476) Annuals now have option to add to reading list via the series detail page, IMP: Fixed password fields to allow for special characters within password, IMP: Weekly pull will now correctly show status for previous/future weeks for watchlisted series, IMP: Download option on pullist will work for previous weeks, FIX: Fixed some issues with one-off and story arc post-processing using different file operations, FIX: Fixed error when performing an 'Force Update Active Comics' from the Manage tab (or the scheduled updating of all active comics), IMP: Storyarc issues that are not on watchlist will now retain value for volume to help when searching and file-checking/post-processing, FIX:(#1473) Changed pullist dates on pullist page to unicode to allow for non-English presentation
2016-12-02 17:45:49 +00:00
try :
xmlpub = arcdom . getElementsByTagName ( ' publisher ' ) [ 0 ] . firstChild . wholeText
except :
xmlpub = " None "
2015-05-27 05:40:38 +00:00
try :
xmldeck = arcdom . getElementsByTagName ( ' deck ' ) [ 0 ] . firstChild . wholeText
except :
xmldeck = " None "
if xmlid in comicLibrary :
haveit = comicLibrary [ xmlid ]
else :
haveit = " No "
arcinfo = {
#'name': xmlTag, #theese four are passed into it only when it's a new add
#'url': xmlurl, #needs to be modified for refreshing to work completely.
#'publisher': xmlpub,
' comicyear ' : arcyear ,
' comicid ' : xmlid ,
' issues ' : issuecount ,
' comicimage ' : xmlimage ,
' description ' : xmldesc ,
' deck ' : xmldeck ,
' arclist ' : arclist ,
FIX:(#1466) When post-processing if filename did not have issue year or volume, would error, IMP:(#1418) Separate copy/move/hardlink/symlink option for Story arcs, IMP:(#1417) Added story arc format folder to story arc options - can now specify folder format for story arc naming, IMP:(#1374) Added rtorrent_authentication to config.ini allowing user to specify digest/basic authentication, as well as the rpc_url (rtorrent_rpc_url) and ssl (rtorrent_ssl). Rtorrent_host no longer requires protocol in field, IMP: Instead of hitting the intermediary/32P site to get the torrent group, will now retain specific id for torrent group to speed up searches, FIX: (#1472) Fixed some problems with series being searched against 32P which had special characters and would return no results, FIX: Added some specific exception catches when attempting to retrieve search results for nzb providers, FIX: Fixed publisher retaining problem for story arcs (note story arcs will have to be refreshed), FIX: Fixed a problem with the pull-list matching up to the correct series when the issue was an annual for a given series with the alt_pull 2 method (would create an invalid link on the Wanted tab), IMP:(#1470) Added config.ini option to specify folder-formatting for weekly folder numbering, 0=YYYY-WN, 1=YYYY-MM-DD, IMP: (#1471) Added quick check to ensure that newznab host entries do not contain leading/trailing spaces and will strip accordingly, FIX:(#1476) Annuals now have option to add to reading list via the series detail page, IMP: Fixed password fields to allow for special characters within password, IMP: Weekly pull will now correctly show status for previous/future weeks for watchlisted series, IMP: Download option on pullist will work for previous weeks, FIX: Fixed some issues with one-off and story arc post-processing using different file operations, FIX: Fixed error when performing an 'Force Update Active Comics' from the Manage tab (or the scheduled updating of all active comics), IMP: Storyarc issues that are not on watchlist will now retain value for volume to help when searching and file-checking/post-processing, FIX:(#1473) Changed pullist dates on pullist page to unicode to allow for non-English presentation
2016-12-02 17:45:49 +00:00
' haveit ' : haveit ,
' publisher ' : xmlpub
2015-05-27 05:40:38 +00:00
}
return arcinfo
FIX: One-off Failed Download handling will now work, IMP: Some better error handling when detecting one-off's during post-processing, FIX:(#1143) When series contained a digit preceded by a dash, would incorrectly assume it as a negative issue number, FIX: Improved being able to detect corresponding annuals on CV when refreshing/importing a series when the annuals are new (ie. no data on CV), FIX: Alt_Pull method for weekly pull list retrieval working again, FIX: Fixed nzbid detection for experimental search, IMP: Mass Import button now available on Import Results screen (will attempt to import all series that are in a 'Not Imported' status), IMP: When searching for arc issues using the 'Search for Missing' option, the call is now threaded so it runs in the background, IMP: Changed IssueYear to IssueDate for Story Arc Issues on the arc details page, FIX:(#1156) Typo that caused error when attempting to view cbz comics in the series detail page, FIX:(#1145) Select All option via top checkbox (on series detail page), FIX: Auto-Want feature via weeklypull will better match to titles that contain 'the' and have hypens in differing character positions, FIX:(#1160) Would incorrectly take the length of a decimal placed issue when searching and never complete the issue number cylcing search, FIX:(#1161) When annuals not enabled, and series in watchlist and series annual in pullist, would error out trying to link series to pullist.
2015-11-18 06:32:40 +00:00