mirror
/
mylar
mirror of https://github.com/evilhero/mylar
1
0
Fork 0
Code Issues Releases Wiki Activity

FIX:(#1222) Fixed usage with NZBHydra - will now be able to properly grab the nzb files, as well as proper logging and handling of Failed downloading, IMP: Duplicate Directory Dump option available in Configuration GUI. Any duplicates discovered during post-processing will be moved into this directory (if enabled) depending on the dupe constraints, FIX: Better handling of titles with '&' and '-' in the titles (as well as annuals) when adding series which should mean more accurate results when trying to add a series, FIX:(#1142) If files didn't have the pages field metadata within the comicinfo.xml file, would error out and either fail to display the issue, or fail to scan the issue during an import scan, FIX: When adding/refreshing a series, if the cover image from CV is unable to be retrieved or is not of an adequate size, fallback to a differnt quality image from CV, FIX: When refreshing/adding a series, annuals will only be checked against once (it was previously running through the entire annual check twice), FIX: During RSS scans/checks if a title in the results had an encoded & (&), would store it as the html which would never match up when doing actual comparison searches, IMP: Fixed usage of feedparser module in rss feeds so that it only uses the retrieved data and doesn't do the actual polling against the url (requests module now does), IMP: Added proper handling of error code 910 with dognzb (max api hits), so that once it hits the 910 error it will disable dognzb as a provider, FIX: When attempting to display issue details on a series detail page (the i icon in the issues table), if the metadata in the .cbz cannot be read or doesn't exist, will now display a graphical warning instead of a 500 error, IMP: Added fork/fork version/tag to comictagger user-agent/version, IMP: Removed configparser dependency from ComicTagger, FIX: When performing searches, improved volume label matching regardless of how the volume label is represented

This commit is contained in:
evilhero 2016-03-04 15:04:19 -05:00
parent b4f6d9a12a
commit 1e0b319d2b
19 changed files with 736 additions and 591 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
README.md
View File

@ -2,7 +2,8 @@ Mylar is an automated Comic Book (cbr/cbz) downloader program heavily-based on t
Yes, it does work, yes there are still bugs, and for that reson I still consider it the definition of an 'Alpha Release'. Yes, it does work, yes there are still bugs, and for that reson I still consider it the definition of an 'Alpha Release'.
This application requires a version of the 2.7.x Python branch for the best results (3.x is not supported) -REQUIREMENTS-
- at least version 2.7.9 Python for proper usage (3.x is not supported).
** NEW ** ** NEW **
You will need to get your OWN ComicVine API Key for this application to fully work. You will need to get your OWN ComicVine API Key for this application to fully work.

17
data/interfaces/default/config.html
View File

@ -706,16 +706,28 @@
</select> </select>
</div> </div>
</fieldset> </fieldset>
<fieldset>
<legend>Duplicate Dump Folder</legend>
<div class="row checkbox left clearfix">
<input type="checkbox" id="enable_ddump" onclick="initConfigCheckbox($this));" name="ddump" value="1" ${config['ddump']} /><label>Enable Duplicate Dump Folder</label>
</div>
<div class="config">
<div class="row"">
<label>Full path to move files determined to be duplicates</label>
<input type="text" name="duplicate_dump" value="${config['duplicate_dump']}" size="30">
</div>
</div>
</fieldset>
<fieldset> <fieldset>
<legend>Failed Download Handling</legend> <legend>Failed Download Handling</legend>
<div class="row checkbox left clearfix"> <div class="row checkbox left clearfix">
<input type="checkbox" id="enable_failed" onclick="initConfigCheckbox($this));" name="failed_download_handling" value="1" ${config['failed_download_handling']} /><label>Enable Failed Download Handling</label> <input type="checkbox" id="enable_failed" onclick="initConfigCheckbox($this));" name="failed_download_handling" value="1" ${config['failed_download_handling']} /><label>Enable Failed Download Handling</label>
</div> </div>
<div class="config"> <div class="config">
<div class="row checkbox left clearfix"> <div class="checkbox left clearfix">
<input type="checkbox" name="failed_auto" value="1" ${config['failed_auto']} /><label>Enable Automatic-Retry for Failed Downloads</label> <input type="checkbox" name="failed_auto" value="1" ${config['failed_auto']} /><label>Enable Automatic-Retry for Failed Downloads</label>
</div> </div>
</div>
</fieldset> </fieldset>
</td> </td>
@ -1401,6 +1413,7 @@
initConfigCheckbox("#replace_spaces"); initConfigCheckbox("#replace_spaces");
initConfigCheckbox("#use_minsize"); initConfigCheckbox("#use_minsize");
initConfigCheckbox("#use_maxsize"); initConfigCheckbox("#use_maxsize");
initConfigCheckbox("#enable_ddump");
initConfigCheckbox("#enable_failed"); initConfigCheckbox("#enable_failed");
initConfigCheckbox("#enable_meta"); initConfigCheckbox("#enable_meta");
initConfigCheckbox("#zero_level"); initConfigCheckbox("#zero_level");

BIN
data/interfaces/default/images/symbol_exclamation.png Executable file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB

6
lib/comictaggerlib/comicvinetalker.py
View File

@ -107,7 +107,7 @@ class ComicVineTalker(QObject):
else: else:
self.api_key = ComicVineTalker.api_key self.api_key = ComicVineTalker.api_key
self.cv_headers = {'User-Agent': 'ComicTagger.[ninjas.walk.alone.fork] - UserAgent + CV Rate Limiting / 1.01 - KATANA'} self.cv_headers = {'User-Agent': 'ComicTagger ' + str(ctversion.version) + ' [' + ctversion.fork + ' / ' + ctversion.fork_tag + ']'}
self.log_func = None self.log_func = None
def setLogFunc( self , log_func ): def setLogFunc( self , log_func ):
@ -449,8 +449,10 @@ class ComicVineTalker(QObject):
if settings.use_series_start_as_volume: if settings.use_series_start_as_volume:
metadata.volume = volume_results['start_year'] metadata.volume = volume_results['start_year']
metadata.notes = "Tagged with ComicTagger {0} using info from Comic Vine on {1}. [Issue ID {2}]".format( metadata.notes = "Tagged with the {1} fork of ComicTagger {0} using info from Comic Vine on {3}. [Issue ID {4}]".format(
ctversion.version, ctversion.version,
ctversion.fork,
ctversion.fork_tag,
datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"), datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
issue_results['id']) issue_results['id'])
#metadata.notes += issue_results['site_detail_url'] #metadata.notes += issue_results['site_detail_url']

4
lib/comictaggerlib/ctversion.py
View File

@ -1,3 +1,5 @@
# This file should contan only these comments, and the line below. # This file should contan only these comments, and the line below.
# Used by packaging makefiles and app # Used by packaging makefiles and app
version="1.1.15-beta" version="1.20.0"
fork="ninjas.walk.alone"
fork_tag="SHURIKEN"

8
lib/comictaggerlib/options.py
View File

@ -318,13 +318,9 @@ For more help visit the wiki at: http://code.google.com/p/comictagger/
if o == "--only-set-cv-key": if o == "--only-set-cv-key":
self.only_set_key = True self.only_set_key = True
if o == "--version": if o == "--version":
print "ComicTagger {0}: Copyright (c) 2012-2014 Anthony Beville".format(ctversion.version) print "ComicTagger {0} [{1} / {2}]".format(ctversion.version, ctversion.fork, ctversion.fork_tag)
print "Modified version of ComicTagger (Copyright (c) 2012-2014 Anthony Beville)"
print "Distributed under Apache License 2.0 (http://www.apache.org/licenses/LICENSE-2.0)" print "Distributed under Apache License 2.0 (http://www.apache.org/licenses/LICENSE-2.0)"
new_version = VersionChecker().getLatestVersion("", False)
if new_version is not None and new_version != ctversion.version:
print "----------------------------------------"
print "New version available online: {0}".format(new_version)
print "----------------------------------------"
sys.exit(0) sys.exit(0)
if o in ("-t", "--type"): if o in ("-t", "--type"):
if a.lower() == "cr": if a.lower() == "cr":

10
lib/comictaggerlib/readme Normal file
View File

@ -0,0 +1,10 @@
ComicTagger.[ninjas.walk.alone]
Fork: Ninjas Walk Alone
Modified ComicTagger 1.15.beta to include some patches to allow for better integration with both ComicVine and mylar. These fixes are:
- UserAgent included now to allow for usage with ComicVine
- ComicVine Rate limiting is now on a per api request (1 api request / 2s)
- Removed requirement for configparser
- Changed queries to ComicVine to utilize the requests module

348
lib/comictaggerlib/settings.py
View File

@ -20,13 +20,20 @@ limitations under the License.
import os import os
import sys import sys
import configparser
import platform import platform
import codecs import codecs
import uuid import uuid
import utils import utils
try:
config_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), '..')
if config_path not in sys.path:
sys.path.append(config_path)
from lib.configobj import ConfigObj
except ImportError:
print "Unable to use configobj module. This is a CRITICAL error and ComicTagger cannot proceed. Exiting."
class ComicTaggerSettings: class ComicTaggerSettings:
@staticmethod @staticmethod
@ -137,13 +144,14 @@ class ComicTaggerSettings:
self.folder = "" self.folder = ""
self.setDefaultValues() self.setDefaultValues()
self.config = configparser.RawConfigParser() #self.config = configparser.RawConfigParser()
self.folder = ComicTaggerSettings.getSettingsFolder() self.folder = ComicTaggerSettings.getSettingsFolder()
if not os.path.exists( self.folder ): if not os.path.exists( self.folder ):
os.makedirs( self.folder ) os.makedirs( self.folder )
self.settings_file = os.path.join( self.folder, "settings") self.settings_file = os.path.join( self.folder, "settings.ini")
self.CFG = ConfigObj(self.settings_file, encoding='utf-8')
# if config file doesn't exist, write one out # if config file doesn't exist, write one out
if not os.path.exists( self.settings_file ): if not os.path.exists( self.settings_file ):
@ -182,204 +190,176 @@ class ComicTaggerSettings:
os.unlink( self.settings_file ) os.unlink( self.settings_file )
self.__init__() self.__init__()
def CheckSection(self, sec):
""" Check if INI section exists, if not create it """
try:
self.CFG[sec]
return True
except:
self.CFG[sec] = {}
return False
################################################################################
# Check_setting_int #
################################################################################
def check_setting_int(self, config, cfg_name, item_name, def_val):
try:
my_val = int(config[cfg_name][item_name])
except:
my_val = def_val
try:
config[cfg_name][item_name] = my_val
except:
config[cfg_name] = {}
config[cfg_name][item_name] = my_val
return my_val
################################################################################
# Check_setting_str #
################################################################################
def check_setting_str(self, config, cfg_name, item_name, def_val, log=True):
try:
my_val = config[cfg_name][item_name]
except:
my_val = def_val
try:
config[cfg_name][item_name] = my_val
except:
config[cfg_name] = {}
config[cfg_name][item_name] = my_val
return my_val
def load(self): def load(self):
def readline_generator(f): self.rar_exe_path = self.check_setting_str(self.CFG, 'settings', 'rar_exe_path', '')
line = f.readline() self.unrar_exe_path = self.check_setting_str(self.CFG, 'settings', 'unurar_exe_path', '')
while line: self.check_for_new_version = bool(self.check_setting_int(self.CFG, 'settings', 'check_for_new_version', 0))
yield line self.send_usage_stats = bool(self.check_setting_int(self.CFG, 'settings', 'send_usage_stats', 0))
line = f.readline() self.install_id = self.check_setting_str(self.CFG, 'auto', 'install_id', '')
self.last_selected_load_data_style = self.check_setting_str(self.CFG, 'auto', 'last_selected_load_data_style', '')
self.last_selected_save_data_style = self.check_setting_str(self.CFG, 'auto', 'last_selected_save_data_style', '')
self.last_selected_save_data_style = self.check_setting_str(self.CFG, 'auto', 'last_selected_save_data_style', '')
self.last_opened_folder = self.check_setting_str(self.CFG, 'auto', 'last_opened_folder', '')
self.last_main_window_width = self.check_setting_str(self.CFG, 'auto', 'last_main_window_width', '')
self.last_main_window_height = self.check_setting_str(self.CFG, 'auto', 'last_main_window_height', '')
self.last_form_side_width = self.check_setting_str(self.CFG, 'auto', 'last_form_side_width', '')
self.last_list_side_width = self.check_setting_str(self.CFG, 'auto', 'last_list_side_width', '')
self.last_filelist_sorted_column = self.check_setting_str(self.CFG, 'auto', 'last_filelist_sorted_column', '')
self.last_filelist_sorted_order = self.check_setting_str(self.CFG, 'auto', 'last_filelist_sorted_order', '')
self.last_main_window_x = self.check_setting_str(self.CFG, 'auto', 'last_main_window_x', '')
self.last_main_window_y = self.check_setting_str(self.CFG, 'auto', 'last_main_window_y','')
self.last_form_side_width = self.check_setting_str(self.CFG, 'auto', 'last_form_side_width','')
self.last_list_side_width = self.check_setting_str(self.CFG, 'auto', 'last_list_side_width','')
#self.config.readfp(codecs.open(self.settings_file, "r", "utf8")) self.id_length_delta_thresh = self.check_setting_str(self.CFG, 'identifier', 'id_length_delta_thresh', '')
self.config.read_file(readline_generator(codecs.open(self.settings_file, "r", "utf8"))) self.id_publisher_blacklist = self.check_setting_str(self.CFG, 'identifier', 'id_publisher_blacklist', '')
self.rar_exe_path = self.config.get( 'settings', 'rar_exe_path' )
self.unrar_exe_path = self.config.get( 'settings', 'unrar_exe_path' )
if self.config.has_option('settings', 'check_for_new_version'):
self.check_for_new_version = self.config.getboolean( 'settings', 'check_for_new_version' )
if self.config.has_option('settings', 'send_usage_stats'):
self.send_usage_stats = self.config.getboolean( 'settings', 'send_usage_stats' )
if self.config.has_option('auto', 'install_id'):
self.install_id = self.config.get( 'auto', 'install_id' )
if self.config.has_option('auto', 'last_selected_load_data_style'):
self.last_selected_load_data_style = self.config.getint( 'auto', 'last_selected_load_data_style' )
if self.config.has_option('auto', 'last_selected_save_data_style'):
self.last_selected_save_data_style = self.config.getint( 'auto', 'last_selected_save_data_style' )
if self.config.has_option('auto', 'last_opened_folder'):
self.last_opened_folder = self.config.get( 'auto', 'last_opened_folder' )
if self.config.has_option('auto', 'last_main_window_width'):
self.last_main_window_width = self.config.getint( 'auto', 'last_main_window_width' )
if self.config.has_option('auto', 'last_main_window_height'):
self.last_main_window_height = self.config.getint( 'auto', 'last_main_window_height' )
if self.config.has_option('auto', 'last_main_window_x'):
self.last_main_window_x = self.config.getint( 'auto', 'last_main_window_x' )
if self.config.has_option('auto', 'last_main_window_y'):
self.last_main_window_y = self.config.getint( 'auto', 'last_main_window_y' )
if self.config.has_option('auto', 'last_form_side_width'):
self.last_form_side_width = self.config.getint( 'auto', 'last_form_side_width' )
if self.config.has_option('auto', 'last_list_side_width'):
self.last_list_side_width = self.config.getint( 'auto', 'last_list_side_width' )
if self.config.has_option('auto', 'last_filelist_sorted_column'):
self.last_filelist_sorted_column = self.config.getint( 'auto', 'last_filelist_sorted_column' )
if self.config.has_option('auto', 'last_filelist_sorted_order'):
self.last_filelist_sorted_order = self.config.getint( 'auto', 'last_filelist_sorted_order' )
if self.config.has_option('identifier', 'id_length_delta_thresh'): self.parse_scan_info = bool(self.check_setting_int(self.CFG, 'filenameparser', 'parse_scan_info', 0))
self.id_length_delta_thresh = self.config.getint( 'identifier', 'id_length_delta_thresh' )
if self.config.has_option('identifier', 'id_publisher_blacklist'):
self.id_publisher_blacklist = self.config.get( 'identifier', 'id_publisher_blacklist' )
if self.config.has_option('filenameparser', 'parse_scan_info'): self.ask_about_cbi_in_rar = bool(self.check_setting_int(self.CFG, 'dialogflags', 'ask_about_cbi_in_rar', 0))
self.parse_scan_info = self.config.getboolean( 'filenameparser', 'parse_scan_info' ) self.show_disclaimer = bool(self.check_setting_int(self.CFG, 'dialogflags', 'show_disclaimer', 0))
self.dont_notify_about_this_version = self.check_setting_str(self.CFG, 'dialogflags', 'dont_notify_about_this_version', '')
self.ask_about_usage_stats = bool(self.check_setting_int(self.CFG, 'dialogflags', 'ask_about_usage_stats', 0))
self.show_no_unrar_warning = bool(self.check_setting_int(self.CFG, 'dialogflags', 'show_no_unrar_warning', 0))
self.use_series_start_as_volume = bool(self.check_setting_int(self.CFG, 'comicvine', 'use_series_start_as_volume', 0))
self.clear_form_before_populating_from_cv = bool(self.check_setting_int(self.CFG, 'comicvine', 'clear_form_before_populating_from_cv', 0))
self.remove_html_tables = bool(self.check_setting_int(self.CFG, 'comicvine', 'remove_html_tables', 0))
self.cv_api_key = self.check_setting_str(self.CFG, 'comicvine', 'cv_api_key', '')
if self.config.has_option('dialogflags', 'ask_about_cbi_in_rar'): self.assume_lone_credit_is_primary = bool(self.check_setting_int(self.CFG, 'cbl_transform', 'assume_lone_credit_is_primary', 0))
self.ask_about_cbi_in_rar = self.config.getboolean( 'dialogflags', 'ask_about_cbi_in_rar' ) self.copy_characters_to_tags = bool(self.check_setting_int(self.CFG, 'cbl_transform', 'copy_characters_to_tags', 0))
if self.config.has_option('dialogflags', 'show_disclaimer'): self.copy_teams_to_tags = bool(self.check_setting_int(self.CFG, 'cbl_transform', 'copy_teams_to_tags', 0))
self.show_disclaimer = self.config.getboolean( 'dialogflags', 'show_disclaimer' ) self.copy_locations_to_tags = bool(self.check_setting_int(self.CFG, 'cbl_transform', 'copy_locations_to_tags', 0))
if self.config.has_option('dialogflags', 'dont_notify_about_this_version'): self.copy_notes_to_comments = bool(self.check_setting_int(self.CFG, 'cbl_transform', 'copy_notes_to_comments', 0))
self.dont_notify_about_this_version = self.config.get( 'dialogflags', 'dont_notify_about_this_version' ) self.copy_storyarcs_to_tags = bool(self.check_setting_int(self.CFG, 'cbl_transform', 'copy_storyarcs_to_tags', 0))
if self.config.has_option('dialogflags', 'ask_about_usage_stats'): self.copy_weblink_to_comments = bool(self.check_setting_int(self.CFG, 'cbl_transform', 'copy_weblink_to_comments', 0))
self.ask_about_usage_stats = self.config.getboolean( 'dialogflags', 'ask_about_usage_stats' ) self.apply_cbl_transform_on_cv_import = bool(self.check_setting_int(self.CFG, 'cbl_transform', 'apply_cbl_transform_on_cv_import', 0))
if self.config.has_option('dialogflags', 'show_no_unrar_warning'): self.apply_cbl_transform_on_bulk_operation = bool(self.check_setting_int(self.CFG, 'cbl_transform', 'apply_cbl_transform_on_bulk_operation', 0))
self.show_no_unrar_warning = self.config.getboolean( 'dialogflags', 'show_no_unrar_warning' )
if self.config.has_option('comicvine', 'use_series_start_as_volume'):
self.use_series_start_as_volume = self.config.getboolean( 'comicvine', 'use_series_start_as_volume' )
if self.config.has_option('comicvine', 'clear_form_before_populating_from_cv'):
self.clear_form_before_populating_from_cv = self.config.getboolean( 'comicvine', 'clear_form_before_populating_from_cv' )
if self.config.has_option('comicvine', 'remove_html_tables'):
self.remove_html_tables = self.config.getboolean( 'comicvine', 'remove_html_tables' )
if self.config.has_option('comicvine', 'cv_api_key'):
self.cv_api_key = self.config.get( 'comicvine', 'cv_api_key' )
if self.config.has_option('cbl_transform', 'assume_lone_credit_is_primary'):
self.assume_lone_credit_is_primary = self.config.getboolean( 'cbl_transform', 'assume_lone_credit_is_primary' )
if self.config.has_option('cbl_transform', 'copy_characters_to_tags'):
self.copy_characters_to_tags = self.config.getboolean( 'cbl_transform', 'copy_characters_to_tags' )
if self.config.has_option('cbl_transform', 'copy_teams_to_tags'):
self.copy_teams_to_tags = self.config.getboolean( 'cbl_transform', 'copy_teams_to_tags' )
if self.config.has_option('cbl_transform', 'copy_locations_to_tags'):
self.copy_locations_to_tags = self.config.getboolean( 'cbl_transform', 'copy_locations_to_tags' )
if self.config.has_option('cbl_transform', 'copy_notes_to_comments'):
self.copy_notes_to_comments = self.config.getboolean( 'cbl_transform', 'copy_notes_to_comments' )
if self.config.has_option('cbl_transform', 'copy_storyarcs_to_tags'):
self.copy_storyarcs_to_tags = self.config.getboolean( 'cbl_transform', 'copy_storyarcs_to_tags' )
if self.config.has_option('cbl_transform', 'copy_weblink_to_comments'):
self.copy_weblink_to_comments = self.config.getboolean( 'cbl_transform', 'copy_weblink_to_comments' )
if self.config.has_option('cbl_transform', 'apply_cbl_transform_on_cv_import'):
self.apply_cbl_transform_on_cv_import = self.config.getboolean( 'cbl_transform', 'apply_cbl_transform_on_cv_import' )
if self.config.has_option('cbl_transform', 'apply_cbl_transform_on_bulk_operation'):
self.apply_cbl_transform_on_bulk_operation = self.config.getboolean( 'cbl_transform', 'apply_cbl_transform_on_bulk_operation' )
if self.config.has_option('rename', 'rename_template'): self.rename_template = bool(self.check_setting_int(self.CFG, 'rename', 'rename_template', 0))
self.rename_template = self.config.get( 'rename', 'rename_template' ) self.rename_issue_number_padding = self.check_setting_str(self.CFG, 'rename', 'rename_issue_number_padding', '')
if self.config.has_option('rename', 'rename_issue_number_padding'): self.rename_use_smart_string_cleanup = bool(self.check_setting_int(self.CFG, 'rename', 'rename_use_smart_string_cleanup', 0))
self.rename_issue_number_padding = self.config.getint( 'rename', 'rename_issue_number_padding' ) self.rename_extension_based_on_archive = bool(self.check_setting_int(self.CFG, 'rename', 'rename_extension_based_on_archive', 0))
if self.config.has_option('rename', 'rename_use_smart_string_cleanup'):
self.rename_use_smart_string_cleanup = self.config.getboolean( 'rename', 'rename_use_smart_string_cleanup' )
if self.config.has_option('rename', 'rename_extension_based_on_archive'):
self.rename_extension_based_on_archive = self.config.getboolean( 'rename', 'rename_extension_based_on_archive' )
if self.config.has_option('autotag', 'save_on_low_confidence'): self.save_on_low_confidence = bool(self.check_setting_int(self.CFG, 'autotag', 'save_on_low_confidence', 0))
self.save_on_low_confidence = self.config.getboolean( 'autotag', 'save_on_low_confidence' ) self.dont_use_year_when_identifying = bool(self.check_setting_int(self.CFG, 'autotag', 'dont_use_year_when_identifying', 0))
if self.config.has_option('autotag', 'dont_use_year_when_identifying'): self.assume_1_if_no_issue_num = bool(self.check_setting_int(self.CFG, 'autotag', 'assume_1_if_no_issue_num', 0))
self.dont_use_year_when_identifying = self.config.getboolean( 'autotag', 'dont_use_year_when_identifying' ) self.ignore_leading_numbers_in_filename = bool(self.check_setting_int(self.CFG, 'autotag', 'ignore_leading_numbers_in_filename', 0))
if self.config.has_option('autotag', 'assume_1_if_no_issue_num'): self.remove_archive_after_successful_match = bool(self.check_setting_int(self.CFG, 'autotag', 'remove_archive_after_successful_match', 0))
self.assume_1_if_no_issue_num = self.config.getboolean( 'autotag', 'assume_1_if_no_issue_num' ) self.wait_and_retry_on_rate_limit = bool(self.check_setting_int(self.CFG, 'autotag', 'wait_and_retry_on_rate_limit', 0))
if self.config.has_option('autotag', 'ignore_leading_numbers_in_filename'):
self.ignore_leading_numbers_in_filename = self.config.getboolean( 'autotag', 'ignore_leading_numbers_in_filename' )
if self.config.has_option('autotag', 'remove_archive_after_successful_match'):
self.remove_archive_after_successful_match = self.config.getboolean( 'autotag', 'remove_archive_after_successful_match' )
if self.config.has_option('autotag', 'wait_and_retry_on_rate_limit'):
self.wait_and_retry_on_rate_limit = self.config.getboolean( 'autotag', 'wait_and_retry_on_rate_limit' )
def save( self ): def save( self ):
new_config = ConfigObj()
new_config.filename = self.settings_file
if not self.config.has_section( 'settings' ): new_config.encoding = 'UTF8'
self.config.add_section( 'settings' ) new_config['settings'] = {}
new_config['settings']['check_for_new_version'] = self.check_for_new_version
new_config['settings']['rar_exe_path'] = self.rar_exe_path
new_config['settings']['unrar_exe_path'] = self.unrar_exe_path
new_config['settings']['send_usage_stats'] = self.send_usage_stats
self.config.set( 'settings', 'check_for_new_version', self.check_for_new_version ) new_config.write()
self.config.set( 'settings', 'rar_exe_path', self.rar_exe_path ) new_config['auto'] = {}
self.config.set( 'settings', 'unrar_exe_path', self.unrar_exe_path ) new_config['auto']['install_id'] = self.install_id
self.config.set( 'settings', 'send_usage_stats', self.send_usage_stats ) new_config['auto']['last_selected_load_data_style'] = self.last_selected_load_data_style
new_config['auto']['last_selected_save_data_style'] = self.last_selected_save_data_style
new_config['auto']['last_opened_folder'] = self.last_opened_folder
new_config['auto']['last_main_window_width'] = self.last_main_window_width
new_config['auto']['last_main_window_height'] = self.last_main_window_height
new_config['auto']['last_main_window_x'] = self.last_main_window_x
new_config['auto']['last_main_window_y'] = self.last_main_window_y
new_config['auto']['last_form_side_width'] = self.last_form_side_width
new_config['auto']['last_list_side_width'] = self.last_list_side_width
new_config['auto']['last_filelist_sorted_column'] = self.last_filelist_sorted_column
new_config['auto']['last_filelist_sorted_order'] = self.last_filelist_sorted_order
if not self.config.has_section( 'auto' ): new_config['identifier'] = {}
self.config.add_section( 'auto' ) new_config['identifier']['id_length_delta_thresh'] = self.id_length_delta_thresh
new_config['identifier']['id_publisher_blacklist'] = self.id_publisher_blacklist
self.config.set( 'auto', 'install_id', self.install_id ) new_config['dialogflags'] = {}
self.config.set( 'auto', 'last_selected_load_data_style', self.last_selected_load_data_style ) new_config['dialogflags']['ask_about_cbi_in_rar'] = self.ask_about_cbi_in_rar
self.config.set( 'auto', 'last_selected_save_data_style', self.last_selected_save_data_style ) new_config['dialogflags']['show_disclaimer'] = self.show_disclaimer
self.config.set( 'auto', 'last_opened_folder', self.last_opened_folder ) new_config['dialogflags']['dont_notify_about_this_version'] = self.dont_notify_about_this_version
self.config.set( 'auto', 'last_main_window_width', self.last_main_window_width ) new_config['dialogflags']['ask_about_usage_stats'] = self.ask_about_usage_stats
self.config.set( 'auto', 'last_main_window_height', self.last_main_window_height ) new_config['dialogflags']['show_no_unrar_warning'] = self.show_no_unrar_warning
self.config.set( 'auto', 'last_main_window_x', self.last_main_window_x )
self.config.set( 'auto', 'last_main_window_y', self.last_main_window_y )
self.config.set( 'auto', 'last_form_side_width', self.last_form_side_width )
self.config.set( 'auto', 'last_list_side_width', self.last_list_side_width )
self.config.set( 'auto', 'last_filelist_sorted_column', self.last_filelist_sorted_column )
self.config.set( 'auto', 'last_filelist_sorted_order', self.last_filelist_sorted_order )
if not self.config.has_section( 'identifier' ): new_config['filenameparser'] = {}
self.config.add_section( 'identifier' ) new_config['filenameparser']['parse_scan_info'] = self.parse_scan_info
self.config.set( 'identifier', 'id_length_delta_thresh', self.id_length_delta_thresh )
self.config.set( 'identifier', 'id_publisher_blacklist', self.id_publisher_blacklist )
if not self.config.has_section( 'dialogflags' ):
self.config.add_section( 'dialogflags' )
self.config.set( 'dialogflags', 'ask_about_cbi_in_rar', self.ask_about_cbi_in_rar )
self.config.set( 'dialogflags', 'show_disclaimer', self.show_disclaimer )
self.config.set( 'dialogflags', 'dont_notify_about_this_version', self.dont_notify_about_this_version )
self.config.set( 'dialogflags', 'ask_about_usage_stats', self.ask_about_usage_stats )
self.config.set( 'dialogflags', 'show_no_unrar_warning', self.show_no_unrar_warning )
if not self.config.has_section( 'filenameparser' ):
self.config.add_section( 'filenameparser' )
self.config.set( 'filenameparser', 'parse_scan_info', self.parse_scan_info ) new_config['comicvine'] = {}
new_config['comicvine']['use_series_start_as_volume'] = self.use_series_start_as_volume
if not self.config.has_section( 'comicvine' ): new_config['comicvine']['clear_form_before_populating_from_cv'] = self.clear_form_before_populating_from_cv
self.config.add_section( 'comicvine' ) new_config['comicvine']['remove_html_tables'] = self.remove_html_tables
new_config['comicvine']['cv_api_key'] = self.cv_api_key
self.config.set( 'comicvine', 'use_series_start_as_volume', self.use_series_start_as_volume )
self.config.set( 'comicvine', 'clear_form_before_populating_from_cv', self.clear_form_before_populating_from_cv )
self.config.set( 'comicvine', 'remove_html_tables', self.remove_html_tables )
self.config.set( 'comicvine', 'cv_api_key', self.cv_api_key )
if not self.config.has_section( 'cbl_transform' ): new_config['cbl_transform'] = {}
self.config.add_section( 'cbl_transform' ) new_config['cbl_transform']['assume_lone_credit_is_primary'] = self.assume_lone_credit_is_primary
new_config['cbl_transform']['copy_characters_to_tags'] = self.copy_characters_to_tags
new_config['cbl_transform']['copy_teams_to_tags'] = self.copy_teams_to_tags
new_config['cbl_transform']['copy_locations_to_tags'] = self.copy_locations_to_tags
new_config['cbl_transform']['copy_storyarcs_to_tags'] = self.copy_storyarcs_to_tags
new_config['cbl_transform']['copy_notes_to_comments'] = self.copy_notes_to_comments
new_config['cbl_transform']['copy_weblink_to_comments'] = self.copy_weblink_to_comments
new_config['cbl_transform']['apply_cbl_transform_on_cv_import'] = self.apply_cbl_transform_on_cv_import
new_config['cbl_transform']['apply_cbl_transform_on_bulk_operation'] = self.apply_cbl_transform_on_bulk_operation
self.config.set( 'cbl_transform', 'assume_lone_credit_is_primary', self.assume_lone_credit_is_primary ) new_config['rename'] = {}
self.config.set( 'cbl_transform', 'copy_characters_to_tags', self.copy_characters_to_tags ) new_config['rename']['rename_template'] = self.rename_template
self.config.set( 'cbl_transform', 'copy_teams_to_tags', self.copy_teams_to_tags ) new_config['rename']['rename_issue_number_padding'] = self.rename_issue_number_padding
self.config.set( 'cbl_transform', 'copy_locations_to_tags', self.copy_locations_to_tags ) new_config['rename']['rename_use_smart_string_cleanup'] = self.rename_use_smart_string_cleanup
self.config.set( 'cbl_transform', 'copy_storyarcs_to_tags', self.copy_storyarcs_to_tags ) new_config['rename']['rename_extension_based_on_archive'] = self.rename_extension_based_on_archive
self.config.set( 'cbl_transform', 'copy_notes_to_comments', self.copy_notes_to_comments )
self.config.set( 'cbl_transform', 'copy_weblink_to_comments', self.copy_weblink_to_comments )
self.config.set( 'cbl_transform', 'apply_cbl_transform_on_cv_import', self.apply_cbl_transform_on_cv_import )
self.config.set( 'cbl_transform', 'apply_cbl_transform_on_bulk_operation', self.apply_cbl_transform_on_bulk_operation )
if not self.config.has_section( 'rename' ):
self.config.add_section( 'rename' )
self.config.set( 'rename', 'rename_template', self.rename_template )
self.config.set( 'rename', 'rename_issue_number_padding', self.rename_issue_number_padding )
self.config.set( 'rename', 'rename_use_smart_string_cleanup', self.rename_use_smart_string_cleanup )
self.config.set( 'rename', 'rename_extension_based_on_archive', self.rename_extension_based_on_archive )
if not self.config.has_section( 'autotag' ):
self.config.add_section( 'autotag' )
self.config.set( 'autotag', 'save_on_low_confidence', self.save_on_low_confidence )
self.config.set( 'autotag', 'dont_use_year_when_identifying', self.dont_use_year_when_identifying )
self.config.set( 'autotag', 'assume_1_if_no_issue_num', self.assume_1_if_no_issue_num )
self.config.set( 'autotag', 'ignore_leading_numbers_in_filename', self.ignore_leading_numbers_in_filename )
self.config.set( 'autotag', 'remove_archive_after_successful_match', self.remove_archive_after_successful_match )
self.config.set( 'autotag', 'wait_and_retry_on_rate_limit', self.wait_and_retry_on_rate_limit )
with codecs.open( self.settings_file, 'wb', 'utf8') as configfile:
self.config.write(configfile)
new_config['autotag'] = {}
new_config['autotag']['save_on_low_confidence'] = self.save_on_low_confidence
new_config['autotag']['dont_use_year_when_identifying'] = self.dont_use_year_when_identifying
new_config['autotag']['assume_1_if_no_issue_num'] = self.assume_1_if_no_issue_num
new_config['autotag']['ignore_leading_numbers_in_filename'] = self.ignore_leading_numbers_in_filename
new_config['autotag']['remove_archive_after_successful_match'] = self.remove_archive_after_successful_match
new_config['autotag']['wait_and_retry_on_rate_limit'] = self.wait_and_retry_on_rate_limit
#make sure the basedir is cached, in case we're on windows running a script from frozen binary #make sure the basedir is cached, in case we're on windows running a script from frozen binary
ComicTaggerSettings.baseDir() ComicTaggerSettings.baseDir()

10
mylar/Failed.py
View File

@ -242,7 +242,15 @@ class FailedProcessor(object):
# Perhaps later improvement might be to break it down by provider so that Mylar will attempt to # Perhaps later improvement might be to break it down by provider so that Mylar will attempt to
# download same issues on different providers (albeit it shouldn't matter, if it's broke it's broke). # download same issues on different providers (albeit it shouldn't matter, if it's broke it's broke).
logger.info('prov : ' + str(self.prov) + '[' + str(self.id) + ']') logger.info('prov : ' + str(self.prov) + '[' + str(self.id) + ']')
chk_fail = myDB.selectone('SELECT * FROM failed WHERE ID=?', [self.id]).fetchone() # if this is from nzbhydra, we need to rejig the id line so that the searchid is removed since it's always unique to the search.
if 'indexerguid' in self.id:
st = self.id.find('searchid:')
end = self.id.find(',',st)
self.id = '%' + self.id[:st] + '%' + self.id[end+1:len(self.id)-1] + '%'
chk_fail = myDB.selectone('SELECT * FROM failed WHERE ID LIKE ?', [self.id]).fetchone()
else:
chk_fail = myDB.selectone('SELECT * FROM failed WHERE ID=?', [self.id]).fetchone()
if chk_fail is None: if chk_fail is None:
logger.info(module + ' Successfully marked this download as Good for downloadable content') logger.info(module + ' Successfully marked this download as Good for downloadable content')
return 'Good' return 'Good'

61
mylar/PostProcessor.py
View File

@ -163,6 +163,29 @@ class PostProcessor(object):
self._log(u"Unable to run extra_script: " + str(script_cmd)) self._log(u"Unable to run extra_script: " + str(script_cmd))
def duplicate_process(self, dupeinfo):
#path to move 'should' be the entire path to the given file
path_to_move = dupeinfo[0]['to_dupe']
file_to_move = os.path.split(path_to_move)[1]
if dupeinfo[0]['action'] == 'dupe_src':
logger.info('[DUPLICATE-CLEANUP] New File will be post-processed. Moving duplicate [' + path_to_move + '] to Duplicate Dump Folder for manual intervention.')
else:
logger.info('[DUPLICATE-CLEANUP] New File will not be post-processed. Moving duplicate [' + path_to_move + '] to Duplicate Dump Folder for manual intervention.')
#check to make sure duplicate_dump directory exists:
checkdirectory = filechecker.validateAndCreateDirectory(mylar.DUPLICATE_DUMP, True, module='[DUPLICATE-CLEANUP]')
#this gets tricky depending on if it's the new filename or the existing filename, and whether or not 'copy' or 'move' has been selected.
try:
shutil.move(path_to_move, os.path.join(mylar.DUPLICATE_DUMP, file_to_move))
except (OSError, IOError):
logger.warn('[DUPLICATE-CLEANUP] Failed to move ' + path_to_move + ' ... to ... ' + os.path.join(mylar.DUPLICATE_DUMP, file_to_move))
return False
logger.warn('[DUPLICATE-CLEANUP] Successfully moved ' + path_to_move + ' ... to ... ' + os.path.join(mylar.DUPLICATE_DUMP, file_to_move))
return True
def Process(self): def Process(self):
module = self.module module = self.module
self._log("nzb name: " + self.nzb_name) self._log("nzb name: " + self.nzb_name)
@ -930,16 +953,22 @@ class PostProcessor(object):
break break
dupthis = helpers.duplicate_filecheck(ml['ComicLocation'], ComicID=comicid, IssueID=issueid) dupthis = helpers.duplicate_filecheck(ml['ComicLocation'], ComicID=comicid, IssueID=issueid)
if dupthis == "write": if dupthis[0]['action'] == 'dupe_src' or dupthis[0]['action'] == 'dupe_file':
#check if duplicate dump folder is enabled and if so move duplicate file in there for manual intervention.
#'dupe_file' - do not write new file as existing file is better quality
#'dupe_src' - write new file, as existing file is a lesser quality (dupe)
if mylar.DUPLICATE_DUMP:
dupchkit = self.duplicate_process(dupthis)
if dupchkit == False:
logger.warn('Unable to move duplicate file - skipping post-processing of this file.')
continue
if dupthis[0]['action'] == "write" or dupthis[0]['action'] == 'dupe_src':
stat = ' [' + str(i) + '/' + str(len(manual_list)) + ']' stat = ' [' + str(i) + '/' + str(len(manual_list)) + ']'
self.Process_next(comicid, issueid, issuenumOG, ml, stat) self.Process_next(comicid, issueid, issuenumOG, ml, stat)
dupthis = None dupthis = None
else:
pass
#check if duplicate dump folder is enabled and if so move duplicate file in there for manual intervention.
#if mylar.DUPLICATE_DUMP:
# if dupthis == 'dupe_src':
#
logger.info(module + ' Manual post-processing completed for ' + str(i) + ' issues.') logger.info(module + ' Manual post-processing completed for ' + str(i) + ' issues.')
return return
else: else:
@ -947,7 +976,22 @@ class PostProcessor(object):
issuenumOG = issuenzb['Issue_Number'] issuenumOG = issuenzb['Issue_Number']
#the self.nzb_folder should contain only the existing filename #the self.nzb_folder should contain only the existing filename
dupthis = helpers.duplicate_filecheck(self.nzb_folder, ComicID=comicid, IssueID=issueid) dupthis = helpers.duplicate_filecheck(self.nzb_folder, ComicID=comicid, IssueID=issueid)
if dupthis == "write": if dupthis[0]['action'] == 'dupe_src' or dupthis[0]['action'] == 'dupe_file':
#check if duplicate dump folder is enabled and if so move duplicate file in there for manual intervention.
#'dupe_file' - do not write new file as existing file is better quality
#'dupe_src' - write new file, as existing file is a lesser quality (dupe)
if mylar.DUPLICATE_DUMP:
dupchkit = self.duplicate_process(dupthis)
if dupchkit == False:
logger.warn('Unable to move duplicate file - skipping post-processing of this file.')
self.valreturn.append({"self.log": self.log,
"mode": 'stop',
"issueid": issueid,
"comicid": comicid})
return self.queue.put(self.valreturn)
if dupthis[0]['action'] == "write" or dupthis[0]['action'] == 'dupe_src':
return self.Process_next(comicid, issueid, issuenumOG) return self.Process_next(comicid, issueid, issuenumOG)
else: else:
self.valreturn.append({"self.log": self.log, self.valreturn.append({"self.log": self.log,
@ -957,7 +1001,6 @@ class PostProcessor(object):
return self.queue.put(self.valreturn) return self.queue.put(self.valreturn)
def Process_next(self, comicid, issueid, issuenumOG, ml=None, stat=None): def Process_next(self, comicid, issueid, issuenumOG, ml=None, stat=None):
if stat is None: stat = ' [1/1]' if stat is None: stat = ' [1/1]'
module = self.module module = self.module

9
mylar/__init__.py
View File

@ -165,6 +165,8 @@ CHECK_FOLDER = None
ENABLE_CHECK_FOLDER = False ENABLE_CHECK_FOLDER = False
INTERFACE = None INTERFACE = None
DUPECONSTRAINT = None DUPECONSTRAINT = None
DDUMP = 0
DUPLICATE_DUMP = None
PREFERRED_QUALITY = 0 PREFERRED_QUALITY = 0
CORRECT_METADATA = False CORRECT_METADATA = False
MOVE_FILES = False MOVE_FILES = False
@ -415,7 +417,7 @@ def initialize():
global __INITIALIZED__, DBCHOICE, DBUSER, DBPASS, DBNAME, COMICVINE_API, DEFAULT_CVAPI, CVAPI_RATE, CV_HEADERS, FULL_PATH, PROG_DIR, VERBOSE, DAEMON, UPCOMING_SNATCHED, COMICSORT, DATA_DIR, CONFIG_FILE, CFG, CONFIG_VERSION, LOG_DIR, CACHE_DIR, MAX_LOGSIZE, OLDCONFIG_VERSION, OS_DETECT, \ global __INITIALIZED__, DBCHOICE, DBUSER, DBPASS, DBNAME, COMICVINE_API, DEFAULT_CVAPI, CVAPI_RATE, CV_HEADERS, FULL_PATH, PROG_DIR, VERBOSE, DAEMON, UPCOMING_SNATCHED, COMICSORT, DATA_DIR, CONFIG_FILE, CFG, CONFIG_VERSION, LOG_DIR, CACHE_DIR, MAX_LOGSIZE, OLDCONFIG_VERSION, OS_DETECT, \
queue, LOCAL_IP, EXT_IP, HTTP_PORT, HTTP_HOST, HTTP_USERNAME, HTTP_PASSWORD, HTTP_ROOT, ENABLE_HTTPS, HTTPS_CERT, HTTPS_KEY, HTTPS_FORCE_ON, HOST_RETURN, API_ENABLED, API_KEY, DOWNLOAD_APIKEY, LAUNCH_BROWSER, GIT_PATH, SAFESTART, AUTO_UPDATE, \ queue, LOCAL_IP, EXT_IP, HTTP_PORT, HTTP_HOST, HTTP_USERNAME, HTTP_PASSWORD, HTTP_ROOT, ENABLE_HTTPS, HTTPS_CERT, HTTPS_KEY, HTTPS_FORCE_ON, HOST_RETURN, API_ENABLED, API_KEY, DOWNLOAD_APIKEY, LAUNCH_BROWSER, GIT_PATH, SAFESTART, AUTO_UPDATE, \
CURRENT_VERSION, LATEST_VERSION, CHECK_GITHUB, CHECK_GITHUB_ON_STARTUP, CHECK_GITHUB_INTERVAL, GIT_USER, GIT_BRANCH, USER_AGENT, DESTINATION_DIR, MULTIPLE_DEST_DIRS, CREATE_FOLDERS, DELETE_REMOVE_DIR, \ CURRENT_VERSION, LATEST_VERSION, CHECK_GITHUB, CHECK_GITHUB_ON_STARTUP, CHECK_GITHUB_INTERVAL, GIT_USER, GIT_BRANCH, USER_AGENT, DESTINATION_DIR, MULTIPLE_DEST_DIRS, CREATE_FOLDERS, DELETE_REMOVE_DIR, \
DOWNLOAD_DIR, USENET_RETENTION, SEARCH_INTERVAL, NZB_STARTUP_SEARCH, INTERFACE, DUPECONSTRAINT, AUTOWANT_ALL, AUTOWANT_UPCOMING, ZERO_LEVEL, ZERO_LEVEL_N, COMIC_COVER_LOCAL, HIGHCOUNT, \ DOWNLOAD_DIR, USENET_RETENTION, SEARCH_INTERVAL, NZB_STARTUP_SEARCH, INTERFACE, DUPECONSTRAINT, DDUMP, DUPLICATE_DUMP, AUTOWANT_ALL, AUTOWANT_UPCOMING, ZERO_LEVEL, ZERO_LEVEL_N, COMIC_COVER_LOCAL, HIGHCOUNT, \
DOWNLOAD_SCAN_INTERVAL, FOLDER_SCAN_LOG_VERBOSE, IMPORTLOCK, NZB_DOWNLOADER, USE_SABNZBD, SAB_HOST, SAB_USERNAME, SAB_PASSWORD, SAB_APIKEY, SAB_CATEGORY, SAB_PRIORITY, SAB_TO_MYLAR, SAB_DIRECTORY, USE_BLACKHOLE, BLACKHOLE_DIR, ADD_COMICS, COMIC_DIR, IMP_MOVE, IMP_RENAME, IMP_METADATA, \ DOWNLOAD_SCAN_INTERVAL, FOLDER_SCAN_LOG_VERBOSE, IMPORTLOCK, NZB_DOWNLOADER, USE_SABNZBD, SAB_HOST, SAB_USERNAME, SAB_PASSWORD, SAB_APIKEY, SAB_CATEGORY, SAB_PRIORITY, SAB_TO_MYLAR, SAB_DIRECTORY, USE_BLACKHOLE, BLACKHOLE_DIR, ADD_COMICS, COMIC_DIR, IMP_MOVE, IMP_RENAME, IMP_METADATA, \
USE_NZBGET, NZBGET_HOST, NZBGET_PORT, NZBGET_USERNAME, NZBGET_PASSWORD, NZBGET_CATEGORY, NZBGET_PRIORITY, NZBGET_DIRECTORY, NZBSU, NZBSU_UID, NZBSU_APIKEY, NZBSU_VERIFY, DOGNZB, DOGNZB_APIKEY, DOGNZB_VERIFY, \ USE_NZBGET, NZBGET_HOST, NZBGET_PORT, NZBGET_USERNAME, NZBGET_PASSWORD, NZBGET_CATEGORY, NZBGET_PRIORITY, NZBGET_DIRECTORY, NZBSU, NZBSU_UID, NZBSU_APIKEY, NZBSU_VERIFY, DOGNZB, DOGNZB_APIKEY, DOGNZB_VERIFY, \
NEWZNAB, NEWZNAB_NAME, NEWZNAB_HOST, NEWZNAB_APIKEY, NEWZNAB_VERIFY, NEWZNAB_UID, NEWZNAB_ENABLED, EXTRA_NEWZNABS, NEWZNAB_EXTRA, \ NEWZNAB, NEWZNAB_NAME, NEWZNAB_HOST, NEWZNAB_APIKEY, NEWZNAB_VERIFY, NEWZNAB_UID, NEWZNAB_ENABLED, EXTRA_NEWZNABS, NEWZNAB_EXTRA, \
@ -515,6 +517,8 @@ def initialize():
ENABLE_CHECK_FOLDER = bool(check_setting_int(CFG, 'General', 'enable_check_folder', 0)) ENABLE_CHECK_FOLDER = bool(check_setting_int(CFG, 'General', 'enable_check_folder', 0))
INTERFACE = check_setting_str(CFG, 'General', 'interface', 'default') INTERFACE = check_setting_str(CFG, 'General', 'interface', 'default')
DUPECONSTRAINT = check_setting_str(CFG, 'General', 'dupeconstraint', 'filesize') DUPECONSTRAINT = check_setting_str(CFG, 'General', 'dupeconstraint', 'filesize')
DDUMP = bool(check_setting_int(CFG, 'General', 'ddump', 0))
DUPLICATE_DUMP = check_setting_str(CFG, 'General', 'duplicate_dump', '')
AUTOWANT_ALL = bool(check_setting_int(CFG, 'General', 'autowant_all', 0)) AUTOWANT_ALL = bool(check_setting_int(CFG, 'General', 'autowant_all', 0))
AUTOWANT_UPCOMING = bool(check_setting_int(CFG, 'General', 'autowant_upcoming', 1)) AUTOWANT_UPCOMING = bool(check_setting_int(CFG, 'General', 'autowant_upcoming', 1))
COMIC_COVER_LOCAL = bool(check_setting_int(CFG, 'General', 'comic_cover_local', 0)) COMIC_COVER_LOCAL = bool(check_setting_int(CFG, 'General', 'comic_cover_local', 0))
@ -1256,6 +1260,8 @@ def config_write():
new_config['General']['check_folder'] = CHECK_FOLDER new_config['General']['check_folder'] = CHECK_FOLDER
new_config['General']['interface'] = INTERFACE new_config['General']['interface'] = INTERFACE
new_config['General']['dupeconstraint'] = DUPECONSTRAINT new_config['General']['dupeconstraint'] = DUPECONSTRAINT
new_config['General']['ddump'] = DDUMP
new_config['General']['duplicate_dump'] = DUPLICATE_DUMP
new_config['General']['autowant_all'] = int(AUTOWANT_ALL) new_config['General']['autowant_all'] = int(AUTOWANT_ALL)
new_config['General']['autowant_upcoming'] = int(AUTOWANT_UPCOMING) new_config['General']['autowant_upcoming'] = int(AUTOWANT_UPCOMING)
new_config['General']['preferred_quality'] = int(PREFERRED_QUALITY) new_config['General']['preferred_quality'] = int(PREFERRED_QUALITY)
@ -1937,7 +1943,6 @@ def dbcheck():
logger.info('Correcting Null entries that make the main page break on startup.') logger.info('Correcting Null entries that make the main page break on startup.')
c.execute("UPDATE Comics SET LatestDate='Unknown' WHERE LatestDate='None' or LatestDate is NULL") c.execute("UPDATE Comics SET LatestDate='Unknown' WHERE LatestDate='None' or LatestDate is NULL")
conn.commit() conn.commit()
c.close() c.close()

17
mylar/cmtagmylar.py
View File

@ -71,14 +71,6 @@ def run(dirName, nzbName=None, issueid=None, comversion=None, manual=None, filen
logger.fdebug(module + ' UNRAR path set to : ' + unrar_cmd) logger.fdebug(module + ' UNRAR path set to : ' + unrar_cmd)
#check for dependencies here - configparser
try:
import configparser
except ImportError:
logger.warn(module + ' configparser not found on system. Please install manually in order to write metadata')
logger.warn(module + ' continuing with PostProcessing, but I am not using metadata.')
return "fail"
if not os.path.exists(unrar_cmd): if not os.path.exists(unrar_cmd):
logger.fdebug(module + ' WARNING: cannot find the unrar command.') logger.fdebug(module + ' WARNING: cannot find the unrar command.')
@ -144,10 +136,10 @@ def run(dirName, nzbName=None, issueid=None, comversion=None, manual=None, filen
ctversion = subprocess.check_output([sys.executable, comictagger_cmd, "--version"], stderr=subprocess.STDOUT) ctversion = subprocess.check_output([sys.executable, comictagger_cmd, "--version"], stderr=subprocess.STDOUT)
except subprocess.CalledProcessError as e: except subprocess.CalledProcessError as e:
#logger.warn(module + "[WARNING] "command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output)) #logger.warn(module + "[WARNING] "command '{}' return with error (code {}): {}".format(e.cmd, e.returncode, e.output))
logger.warn(module + '[WARNING] Make sure that you have configparser installed.') logger.warn(module + '[WARNING] Make sure that you are using the comictagger included with Mylar.')
return "fail" return "fail"
ctend = ctversion.find(':') ctend = ctversion.find('\]')
ctcheck = re.sub("[^0-9]", "", ctversion[:ctend]) ctcheck = re.sub("[^0-9]", "", ctversion[:ctend])
ctcheck = re.sub('\.', '', ctcheck).strip() ctcheck = re.sub('\.', '', ctcheck).strip()
if int(ctcheck) >= int('1115'): # (v1.1.15) if int(ctcheck) >= int('1115'): # (v1.1.15)
@ -159,7 +151,7 @@ def run(dirName, nzbName=None, issueid=None, comversion=None, manual=None, filen
use_cvapi = "True" use_cvapi = "True"
tagoptions.extend(["--cv-api-key", mylar.COMICVINE_API]) tagoptions.extend(["--cv-api-key", mylar.COMICVINE_API])
else: else:
logger.fdebug(module + ' ' + ctversion[:ctend] + ' being used - personal ComicVine API key not supported in this version. Good luck.') logger.fdebug(module + ' ' + ctversion[:ctend+1] + ' being used - personal ComicVine API key not supported in this version. Good luck.')
use_cvapi = "False" use_cvapi = "False"
i = 1 i = 1
@ -231,7 +223,8 @@ def run(dirName, nzbName=None, issueid=None, comversion=None, manual=None, filen
try: try:
p = subprocess.Popen(script_cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) p = subprocess.Popen(script_cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
out, err = p.communicate() out, err = p.communicate()
logger.info(out)
logger.info(err)
if initial_ctrun and 'exported successfully' in out: if initial_ctrun and 'exported successfully' in out:
logger.fdebug(module + '[COMIC-TAGGER] : ' +str(out)) logger.fdebug(module + '[COMIC-TAGGER] : ' +str(out))
#Archive exported successfully to: X-Men v4 008 (2014) (Digital) (Nahga-Empire).cbz (Original deleted) #Archive exported successfully to: X-Men v4 008 (2014) (Digital) (Nahga-Empire).cbz (Original deleted)

75
mylar/helpers.py
View File

@ -1301,13 +1301,13 @@ def IssueDetails(filelocation, IssueID=None):
#print str(data) #print str(data)
issuetag = 'xml' issuetag = 'xml'
#looks for the first page and assumes it's the cover. (Alternate covers handled later on) #looks for the first page and assumes it's the cover. (Alternate covers handled later on)
elif any(['000.' in infile, '00.' in infile]) and infile.endswith(pic_extensions): elif any(['000.' in infile, '00.' in infile]) and infile.endswith(pic_extensions) and cover == "notfound":
logger.fdebug('Extracting primary image ' + infile + ' as coverfile for display.') logger.fdebug('Extracting primary image ' + infile + ' as coverfile for display.')
local_file = open(os.path.join(mylar.CACHE_DIR, 'temp.jpg'), "wb") local_file = open(os.path.join(mylar.CACHE_DIR, 'temp.jpg'), "wb")
local_file.write(inzipfile.read(infile)) local_file.write(inzipfile.read(infile))
local_file.close local_file.close
cover = "found" cover = "found"
elif any(['00a' in infile, '00b' in infile, '00c' in infile, '00d' in infile, '00e' in infile]) and infile.endswith(pic_extensions): elif any(['00a' in infile, '00b' in infile, '00c' in infile, '00d' in infile, '00e' in infile]) and infile.endswith(pic_extensions) and cover == "notfound":
logger.fdebug('Found Alternate cover - ' + infile + ' . Extracting.') logger.fdebug('Found Alternate cover - ' + infile + ' . Extracting.')
altlist = ('00a', '00b', '00c', '00d', '00e') altlist = ('00a', '00b', '00c', '00d', '00e')
for alt in altlist: for alt in altlist:
@ -1433,17 +1433,25 @@ def IssueDetails(filelocation, IssueID=None):
pagecount = 0 pagecount = 0
logger.fdebug("number of pages I counted: " + str(pagecount)) logger.fdebug("number of pages I counted: " + str(pagecount))
i = 0 i = 0
while (i < int(pagecount)):
pageinfo = result.getElementsByTagName('Page')[i].attributes try:
attrib = pageinfo.getNamedItem('Image') pageinfo = result.getElementsByTagName('Page')[0].attributes
logger.fdebug('Frontcover validated as being image #: ' + str(attrib.value)) if pageinfo: pageinfo_test == True
att = pageinfo.getNamedItem('Type') except:
logger.fdebug('pageinfo: ' + str(pageinfo)) pageinfo_test = False
if att.value == 'FrontCover':
logger.fdebug('FrontCover detected. Extracting.') if pageinfo_test:
break while (i < int(pagecount)):
i+=1 pageinfo = result.getElementsByTagName('Page')[i].attributes
else: attrib = pageinfo.getNamedItem('Image')
logger.fdebug('Frontcover validated as being image #: ' + str(attrib.value))
att = pageinfo.getNamedItem('Type')
logger.fdebug('pageinfo: ' + str(pageinfo))
if att.value == 'FrontCover':
logger.fdebug('FrontCover detected. Extracting.')
break
i+=1
elif issuetag == 'comment':
stripline = 'Archive: ' + dstlocation stripline = 'Archive: ' + dstlocation
data = re.sub(stripline, '', data.encode("utf-8")).strip() data = re.sub(stripline, '', data.encode("utf-8")).strip()
if data is None or data == '': if data is None or data == '':
@ -1515,6 +1523,10 @@ def IssueDetails(filelocation, IssueID=None):
except: except:
pagecount = "None" pagecount = "None"
else:
logger.warn('Unable to locate any metadata within cbz file. Tag this file and try again if necessary.')
return
issuedetails.append({"title": issue_title, issuedetails.append({"title": issue_title,
"series": series_title, "series": series_title,
"volume": series_volume, "volume": series_volume,
@ -1609,17 +1621,21 @@ def duplicate_filecheck(filename, ComicID=None, IssueID=None, StoryArcID=None):
logger.info('[DUPECHECK] Unable to find corresponding Issue within the DB. Do you still have the series on your watchlist?') logger.info('[DUPECHECK] Unable to find corresponding Issue within the DB. Do you still have the series on your watchlist?')
return return
series = myDB.selectone("SELECT * FROM comics WHERE ComicID=?", [dupchk['ComicID']]).fetchone()
#if it's a retry and the file was already snatched, the status is Snatched and won't hit the dupecheck. #if it's a retry and the file was already snatched, the status is Snatched and won't hit the dupecheck.
#rtnval will be one of 3: #rtnval will be one of 3:
#'write' - write new file #'write' - write new file
#'dupe_file' - do not write new file as existing file is better quality #'dupe_file' - do not write new file as existing file is better quality
#'dupe_src' - write new file, as existing file is a lesser quality (dupe) #'dupe_src' - write new file, as existing file is a lesser quality (dupe)
rtnval = []
if dupchk['Status'] == 'Downloaded' or dupchk['Status'] == 'Archived': if dupchk['Status'] == 'Downloaded' or dupchk['Status'] == 'Archived':
try: try:
dupsize = dupchk['ComicSize'] dupsize = dupchk['ComicSize']
except: except:
logger.info('[DUPECHECK] Duplication detection returned no hits as this is a new Snatch. This is not a duplicate.') logger.info('[DUPECHECK] Duplication detection returned no hits as this is a new Snatch. This is not a duplicate.')
rtnval = "write" rtnval.append({'action': "write"})
logger.info('[DUPECHECK] Existing Status already set to ' + dupchk['Status']) logger.info('[DUPECHECK] Existing Status already set to ' + dupchk['Status'])
cid = [] cid = []
if dupsize is None: if dupsize is None:
@ -1635,11 +1651,13 @@ def duplicate_filecheck(filename, ComicID=None, IssueID=None, StoryArcID=None):
return duplicate_filecheck(filename, ComicID, IssueID, StoryArcID) return duplicate_filecheck(filename, ComicID, IssueID, StoryArcID)
else: else:
#not sure if this one is correct - should never actually get to this point. #not sure if this one is correct - should never actually get to this point.
rtnval = "dupe_file" rtnval.append({'action': "dupe_file",
'to_dupe': os.path.join(series['ComicLocation'], dupchk['Location'])})
else: else:
rtnval = "dupe_file" rtnval.append({'action': "dupe_file",
'to_dupe': os.path.join(series['ComicLocation'], dupchk['Location'])})
else: else:
logger.info('[DUPECHECK] Existing file :' + dupchk['Location'] + ' has a filesize of : ' + str(dupsize) + ' bytes.') logger.info('[DUPECHECK] Existing file within db :' + dupchk['Location'] + ' has a filesize of : ' + str(dupsize) + ' bytes.')
#keywords to force keep / delete #keywords to force keep / delete
#this will be eventually user-controlled via the GUI once the options are enabled. #this will be eventually user-controlled via the GUI once the options are enabled.
@ -1648,7 +1666,8 @@ def duplicate_filecheck(filename, ComicID=None, IssueID=None, StoryArcID=None):
logger.info('[DUPECHECK] Existing filesize is 0 as I cannot locate the original entry.') logger.info('[DUPECHECK] Existing filesize is 0 as I cannot locate the original entry.')
if dupchk['Status'] == 'Archived': if dupchk['Status'] == 'Archived':
logger.info('[DUPECHECK] Assuming issue is Archived.') logger.info('[DUPECHECK] Assuming issue is Archived.')
rtnval = "dupe_file" rtnval.append({'action': "dupe_file",
'to_dupe': filename})
return rtnval return rtnval
else: else:
logger.info('[DUPECHECK] Assuming 0-byte file - this one is gonna get hammered.') logger.info('[DUPECHECK] Assuming 0-byte file - this one is gonna get hammered.')
@ -1660,33 +1679,39 @@ def duplicate_filecheck(filename, ComicID=None, IssueID=None, StoryArcID=None):
if dupchk['Location'].endswith('.cbz'): if dupchk['Location'].endswith('.cbz'):
#keep dupechk['Location'] #keep dupechk['Location']
logger.info('[DUPECHECK-CBR PRIORITY] [#' + dupchk['Issue_Number'] + '] Retaining currently scanned in file : ' + dupchk['Location']) logger.info('[DUPECHECK-CBR PRIORITY] [#' + dupchk['Issue_Number'] + '] Retaining currently scanned in file : ' + dupchk['Location'])
rtnval = "dupe_file" rtnval.append({'action': "dupe_file",
'to_dupe': filename})
else: else:
#keep filename #keep filename
logger.info('[DUPECHECK-CBR PRIORITY] [#' + dupchk['Issue_Number'] + '] Retaining newly scanned in file : ' + filename) logger.info('[DUPECHECK-CBR PRIORITY] [#' + dupchk['Issue_Number'] + '] Retaining newly scanned in file : ' + filename)
rtnval = "dupe_src" rtnval.append({'action': "dupe_src",
'to_dupe': os.path.join(series['ComicLocation'], dupchk['Location'])})
elif 'cbz' in mylar.DUPECONSTRAINT: elif 'cbz' in mylar.DUPECONSTRAINT:
if dupchk['Location'].endswith('.cbr'): if dupchk['Location'].endswith('.cbr'):
#keep dupchk['Location'] #keep dupchk['Location']
logger.info('[DUPECHECK-CBZ PRIORITY] [#' + dupchk['Issue_Number'] + '] Retaining currently scanned in filename : ' + dupchk['Location']) logger.info('[DUPECHECK-CBZ PRIORITY] [#' + dupchk['Issue_Number'] + '] Retaining currently scanned in filename : ' + dupchk['Location'])
rtnval = "dupe_file" rtnval.append({'action': "dupe_file",
'to_dupe': filename})
else: else:
#keep filename #keep filename
logger.info('[DUPECHECK-CBZ PRIORITY] [#' + dupchk['Issue_Number'] + '] Retaining newly scanned in filename : ' + filename) logger.info('[DUPECHECK-CBZ PRIORITY] [#' + dupchk['Issue_Number'] + '] Retaining newly scanned in filename : ' + filename)
rtnval = "dupe_src" rtnval.append({'action': "dupe_src",
'to_dupe': os.path.join(series['ComicLocation'], dupchk['Location'])})
if mylar.DUPECONSTRAINT == 'filesize': if mylar.DUPECONSTRAINT == 'filesize':
if filesz <= int(dupsize) and int(dupsize) != 0: if filesz <= int(dupsize) and int(dupsize) != 0:
logger.info('[DUPECHECK-FILESIZE PRIORITY] [#' + dupchk['Issue_Number'] + '] Retaining currently scanned in filename : ' + dupchk['Location']) logger.info('[DUPECHECK-FILESIZE PRIORITY] [#' + dupchk['Issue_Number'] + '] Retaining currently scanned in filename : ' + dupchk['Location'])
rtnval = "dupe_file" rtnval.append({'action': "dupe_file",
'to_dupe': filename})
else: else:
logger.info('[DUPECHECK-FILESIZE PRIORITY] [#' + dupchk['Issue_Number'] + '] Retaining newly scanned in filename : ' + filename) logger.info('[DUPECHECK-FILESIZE PRIORITY] [#' + dupchk['Issue_Number'] + '] Retaining newly scanned in filename : ' + filename)
rtnval = "dupe_src" rtnval.append({'action': "dupe_src",
'to_dupe': os.path.join(series['ComicLocation'], dupchk['Location'])})
else: else:
logger.info('[DUPECHECK] Duplication detection returned no hits. This is not a duplicate of anything that I have scanned in as of yet.') logger.info('[DUPECHECK] Duplication detection returned no hits. This is not a duplicate of anything that I have scanned in as of yet.')
rtnval = "write" rtnval.append({'action': "write"})
return rtnval return rtnval
def create_https_certificates(ssl_cert, ssl_key): def create_https_certificates(ssl_cert, ssl_key):

514
mylar/importer.py
View File

@ -28,6 +28,9 @@ import shutil
import imghdr import imghdr
import sqlite3 import sqlite3
import cherrypy import cherrypy
import lib.requests as requests
import gzip
from StringIO import StringIO
import mylar import mylar
from mylar import logger, helpers, db, mb, cv, parseit, filechecker, search, updater, moveit, comicbookdb from mylar import logger, helpers, db, mb, cv, parseit, filechecker, search, updater, moveit, comicbookdb
@ -169,152 +172,152 @@ def addComictoDB(comicid, mismatch=None, pullupd=None, imported=None, ogcname=No
#since the weekly issue check could return either annuals or issues, let's initialize it here so it carries through properly. #since the weekly issue check could return either annuals or issues, let's initialize it here so it carries through properly.
weeklyissue_check = [] weeklyissue_check = []
#let's do the Annual check here. # #let's do the Annual check here.
if mylar.ANNUALS_ON: # if mylar.ANNUALS_ON:
#we need to check first to see if there are pre-existing annuals that have been manually added, or else they'll get # #we need to check first to see if there are pre-existing annuals that have been manually added, or else they'll get
#wiped out. # #wiped out.
annualids = [] #to be used to make sure an ID isn't double-loaded # annualids = [] #to be used to make sure an ID isn't double-loaded
#
if annload is None: # if annload is None:
pass # pass
else: # else:
for manchk in annload: # for manchk in annload:
if manchk['ReleaseComicID'] is not None or manchk['ReleaseComicID'] is not None: #if it exists, then it's a pre-existing add. # if manchk['ReleaseComicID'] is not None or manchk['ReleaseComicID'] is not None: #if it exists, then it's a pre-existing add.
#print str(manchk['ReleaseComicID']), comic['ComicName'], str(SeriesYear), str(comicid) # #print str(manchk['ReleaseComicID']), comic['ComicName'], str(SeriesYear), str(comicid)
manualAnnual(manchk['ReleaseComicID'], comic['ComicName'], SeriesYear, comicid) # manualAnnual(manchk['ReleaseComicID'], comic['ComicName'], SeriesYear, comicid)
annualids.append(manchk['ReleaseComicID']) # annualids.append(manchk['ReleaseComicID'])
#
annualcomicname = re.sub('[\,\:]', '', comic['ComicName']) # annualcomicname = re.sub('[\,\:]', '', comic['ComicName'])
#
#----- CBDB (outdated) ##----- CBDB (outdated)
# annuals = comicbookdb.cbdb(annualcomicname, SeriesYear) ## annuals = comicbookdb.cbdb(annualcomicname, SeriesYear)
# print ("Number of Annuals returned: " + str(annuals['totalissues'])) ## print ("Number of Annuals returned: " + str(annuals['totalissues']))
# nb = 0 ## nb = 0
# while (nb <= int(annuals['totalissues'])): ## while (nb <= int(annuals['totalissues'])):
# try: ## try:
# annualval = annuals['annualslist'][nb] ## annualval = annuals['annualslist'][nb]
# except IndexError: ## except IndexError:
# break ## break
#---- ##----
#this issueid doesn't exist at this point since we got the data from cbdb...let's try and figure out # #this issueid doesn't exist at this point since we got the data from cbdb...let's try and figure out
#the issueID for CV based on what we know so we can use that ID (and thereby the metadata too) # #the issueID for CV based on what we know so we can use that ID (and thereby the metadata too)
#
#other inherit issue - results below will return the ID for the Series of Annuals, not the series itself. # #other inherit issue - results below will return the ID for the Series of Annuals, not the series itself.
#sr['comicid'] not the same as comicid for series. # #sr['comicid'] not the same as comicid for series.
annComicName = annualcomicname + ' annual' # annComicName = annualcomicname + ' annual'
mode = 'series' # mode = 'series'
#if annuals['totalissues'] is None: # #if annuals['totalissues'] is None:
# annissues = 0 # # annissues = 0
#else: # #else:
# annissues = annuals['totalissues'] # # annissues = annuals['totalissues']
#print "annissues :" + str(annissues) # #print "annissues :" + str(annissues)
#
# annuals happen once / year. determine how many. # # annuals happen once / year. determine how many.
annualyear = SeriesYear # no matter what, the year won't be less than this. # annualyear = SeriesYear # no matter what, the year won't be less than this.
#if annualval['AnnualYear'] is None: # #if annualval['AnnualYear'] is None:
# sresults = mb.findComic(annComicName, mode, issue=annissues) # # sresults = mb.findComic(annComicName, mode, issue=annissues)
#else: # #else:
#sresults = mb.findComic(annComicName, mode, issue=annissues, limityear=annualval['AnnualYear']) # #sresults = mb.findComic(annComicName, mode, issue=annissues, limityear=annualval['AnnualYear'])
#print "annualyear: " + str(annualval['AnnualYear']) # #print "annualyear: " + str(annualval['AnnualYear'])
annual_types_ignore = {'paperback', 'collecting', 'reprints', 'collected', 'print edition', 'tpb', 'available in print', 'collects'} # annual_types_ignore = {'paperback', 'collecting', 'reprints', 'collected', 'print edition', 'tpb', 'available in print', 'collects'}
#
logger.fdebug('[IMPORTER-ANNUAL] - Annual Year:' + str(annualyear)) # logger.fdebug('[IMPORTER-ANNUAL] - Annual Year:' + str(annualyear))
sresults, explicit = mb.findComic(annComicName, mode, issue=None, explicit='all')#,explicit=True) # sresults, explicit = mb.findComic(annComicName, mode, issue=None, explicit='all')#,explicit=True)
type='comic' # type='comic'
#
if len(sresults) == 1: # if len(sresults) == 1:
logger.fdebug('[IMPORTER-ANNUAL] - 1 result') # logger.fdebug('[IMPORTER-ANNUAL] - 1 result')
if len(sresults) > 0: # if len(sresults) > 0:
logger.fdebug('[IMPORTER-ANNUAL] - there are ' + str(len(sresults)) + ' results.') # logger.fdebug('[IMPORTER-ANNUAL] - there are ' + str(len(sresults)) + ' results.')
num_res = 0 # num_res = 0
while (num_res < len(sresults)): # while (num_res < len(sresults)):
sr = sresults[num_res] # sr = sresults[num_res]
logger.fdebug("description:" + sr['description']) # #logger.fdebug("description:" + sr['description'])
if any(x in sr['description'].lower() for x in annual_types_ignore): # if any(x in sr['description'].lower() for x in annual_types_ignore):
logger.fdebug('[IMPORTER-ANNUAL] - tradeback/collected edition detected - skipping ' + str(sr['comicid'])) # logger.fdebug('[IMPORTER-ANNUAL] - tradeback/collected edition detected - skipping ' + str(sr['comicid']))
else: # else:
if comicid in sr['description']: # if comicid in sr['description']:
logger.fdebug('[IMPORTER-ANNUAL] - ' + str(comicid) + ' found. Assuming it is part of the greater collection.') # logger.fdebug('[IMPORTER-ANNUAL] - ' + str(comicid) + ' found. Assuming it is part of the greater collection.')
issueid = sr['comicid'] # issueid = sr['comicid']
logger.fdebug('[IMPORTER-ANNUAL] - ' + str(issueid) + ' added to series list as an Annual') # logger.fdebug('[IMPORTER-ANNUAL] - ' + str(issueid) + ' added to series list as an Annual')
if issueid in annualids: # if issueid in annualids:
logger.fdebug('[IMPORTER-ANNUAL] - ' + str(issueid) + ' already exists & was refreshed.') # logger.fdebug('[IMPORTER-ANNUAL] - ' + str(issueid) + ' already exists & was refreshed.')
num_res+=1 # need to manually increment since not a for-next loop # num_res+=1 # need to manually increment since not a for-next loop
continue # continue
issued = cv.getComic(issueid, 'issue') # issued = cv.getComic(issueid, 'issue')
if len(issued) is None or len(issued) == 0: # if len(issued) is None or len(issued) == 0:
logger.fdebug('[IMPORTER-ANNUAL] - Could not find any annual information...') # logger.fdebug('[IMPORTER-ANNUAL] - Could not find any annual information...')
pass # pass
else: # else:
n = 0 # n = 0
if int(sr['issues']) == 0 and len(issued['issuechoice']) == 1: # if int(sr['issues']) == 0 and len(issued['issuechoice']) == 1:
sr_issues = 1 # sr_issues = 1
else: # else:
sr_issues = sr['issues'] # sr_issues = sr['issues']
logger.fdebug('[IMPORTER-ANNUAL (MAIN)] - There are ' + str(sr_issues) + ' annuals in this series.') # logger.fdebug('[IMPORTER-ANNUAL (MAIN)] - There are ' + str(sr_issues) + ' annuals in this series.')
while (n < int(sr_issues)): # while (n < int(sr_issues)):
try: # try:
firstval = issued['issuechoice'][n] # firstval = issued['issuechoice'][n]
except IndexError: # except IndexError:
break # break
try: # try:
cleanname = helpers.cleanName(firstval['Issue_Name']) # cleanname = helpers.cleanName(firstval['Issue_Name'])
except: # except:
cleanname = 'None' # cleanname = 'None'
issid = str(firstval['Issue_ID']) # issid = str(firstval['Issue_ID'])
issnum = str(firstval['Issue_Number']) # issnum = str(firstval['Issue_Number'])
issname = cleanname # issname = cleanname
issdate = str(firstval['Issue_Date']) # issdate = str(firstval['Issue_Date'])
stdate = str(firstval['Store_Date']) # stdate = str(firstval['Store_Date'])
int_issnum = helpers.issuedigits(issnum) # int_issnum = helpers.issuedigits(issnum)
newCtrl = {"IssueID": issid} # newCtrl = {"IssueID": issid}
newVals = {"Issue_Number": issnum, # newVals = {"Issue_Number": issnum,
"Int_IssueNumber": int_issnum, # "Int_IssueNumber": int_issnum,
"IssueDate": issdate, # "IssueDate": issdate,
"ReleaseDate": stdate, # "ReleaseDate": stdate,
"IssueName": issname, # "IssueName": issname,
"ComicID": comicid, # "ComicID": comicid,
"ComicName": comic['ComicName'], # "ComicName": comic['ComicName'],
"ReleaseComicID": re.sub('4050-', '', firstval['Comic_ID']).strip(), # "ReleaseComicID": re.sub('4050-', '', firstval['Comic_ID']).strip(),
"ReleaseComicName": sr['name'], # "ReleaseComicName": sr['name'],
"Status": "Skipped"} # "Status": "Skipped"}
myDB.upsert("annuals", newVals, newCtrl) # myDB.upsert("annuals", newVals, newCtrl)
#
if issuechk is not None and issuetype == 'annual': # if issuechk is not None and issuetype == 'annual':
logger.fdebug('[IMPORTER-ANNUAL] - Comparing annual ' + str(issuechk) + ' .. to .. ' + str(int_issnum)) # logger.fdebug('[IMPORTER-ANNUAL] - Comparing annual ' + str(issuechk) + ' .. to .. ' + str(int_issnum))
if issuechk == int_issnum: # if issuechk == int_issnum:
weeklyissue_check.append({"Int_IssueNumber": int_issnum, # weeklyissue_check.append({"Int_IssueNumber": int_issnum,
"Issue_Number": issnum, # "Issue_Number": issnum,
"IssueDate": issdate, # "IssueDate": issdate,
"ReleaseDate": stdate}) # "ReleaseDate": stdate})
#
n+=1 # n+=1
num_res+=1 # num_res+=1
#
elif len(sresults) == 0 or len(sresults) is None: # elif len(sresults) == 0 or len(sresults) is None:
logger.fdebug('[IMPORTER-ANNUAL] - No results, removing the year from the agenda and re-querying.') # logger.fdebug('[IMPORTER-ANNUAL] - No results, removing the year from the agenda and re-querying.')
sresults, explicit = mb.findComic(annComicName, mode, issue=None)#, explicit=True) # sresults, explicit = mb.findComic(annComicName, mode, issue=None)#, explicit=True)
if len(sresults) == 1: # if len(sresults) == 1:
sr = sresults[0] # sr = sresults[0]
logger.fdebug('[IMPORTER-ANNUAL] - ' + str(comicid) + ' found. Assuming it is part of the greater collection.') # logger.fdebug('[IMPORTER-ANNUAL] - ' + str(comicid) + ' found. Assuming it is part of the greater collection.')
else: # else:
resultset = 0 # resultset = 0
else: # else:
logger.fdebug('[IMPORTER-ANNUAL] - Returning results to screen - more than one possibility') # logger.fdebug('[IMPORTER-ANNUAL] - Returning results to screen - more than one possibility')
for sr in sresults: # for sr in sresults:
if annualyear < sr['comicyear']: # if annualyear < sr['comicyear']:
logger.fdebug('[IMPORTER-ANNUAL] - ' + str(annualyear) + ' is less than ' + str(sr['comicyear'])) # logger.fdebug('[IMPORTER-ANNUAL] - ' + str(annualyear) + ' is less than ' + str(sr['comicyear']))
if int(sr['issues']) > (2013 - int(sr['comicyear'])): # if int(sr['issues']) > (2013 - int(sr['comicyear'])):
logger.fdebug('[IMPORTER-ANNUAL] - Issue count is wrong') # logger.fdebug('[IMPORTER-ANNUAL] - Issue count is wrong')
#
#newCtrl = {"IssueID": issueid} # #newCtrl = {"IssueID": issueid}
#newVals = {"Issue_Number": annualval['AnnualIssue'], # #newVals = {"Issue_Number": annualval['AnnualIssue'],
# "IssueDate": annualval['AnnualDate'], # # "IssueDate": annualval['AnnualDate'],
# "IssueName": annualval['AnnualTitle'], # # "IssueName": annualval['AnnualTitle'],
# "ComicID": comicid, # # "ComicID": comicid,
# "Status": "Skipped"} # # "Status": "Skipped"}
#myDB.upsert("annuals", newVals, newCtrl) # #myDB.upsert("annuals", newVals, newCtrl)
#nb+=1 # #nb+=1
#parseit.annualCheck(gcomicid=gcdinfo['GCDComicID'], comicid=comicid, comicname=comic['ComicName'], comicyear=SeriesYear) #parseit.annualCheck(gcomicid=gcdinfo['GCDComicID'], comicid=comicid, comicname=comic['ComicName'], comicyear=SeriesYear)
#comic book location on machine #comic book location on machine
@ -410,9 +413,6 @@ def addComictoDB(comicid, mismatch=None, pullupd=None, imported=None, ogcname=No
coverfile = os.path.join(mylar.CACHE_DIR, str(comicid) + ".jpg") coverfile = os.path.join(mylar.CACHE_DIR, str(comicid) + ".jpg")
#if cover has '+' in url it's malformed, we need to replace '+' with '%20' to retreive properly. #if cover has '+' in url it's malformed, we need to replace '+' with '%20' to retreive properly.
#thisci = urllib.quote_plus(str(comic['ComicImage']))
#urllib.urlretrieve(str(thisci), str(coverfile))
#new CV API restriction - one api request / second.(probably unecessary here, but it doesn't hurt) #new CV API restriction - one api request / second.(probably unecessary here, but it doesn't hurt)
if mylar.CVAPI_RATE is None or mylar.CVAPI_RATE < 2: if mylar.CVAPI_RATE is None or mylar.CVAPI_RATE < 2:
@ -420,45 +420,63 @@ def addComictoDB(comicid, mismatch=None, pullupd=None, imported=None, ogcname=No
else: else:
time.sleep(mylar.CVAPI_RATE) time.sleep(mylar.CVAPI_RATE)
logger.info('Attempting to retrieve the comic image for series')
try: try:
cimage = re.sub('[\+]', '%20', comic['ComicImage']) r = requests.get(comic['ComicImage'], params=None, stream=True, headers=mylar.CV_HEADERS)
request = urllib2.Request(cimage)#, headers={'Content-Type': 'application/x-www-form-urlencoded'})
#request.add_header('User-Agent', str(mylar.USER_AGENT))
response = urllib2.urlopen(request)
com_image = response.read()
with open(coverfile, 'wb') as the_file:
the_file.write(com_image)
try:
logger.info('Image header check: ' + imghdr.what(coverfile))
except:
logger.info('image is corrupted.')
raise Exception
logger.info('Successfully retrieved cover for ' + comic['ComicName'])
except Exception, e: except Exception, e:
logger.warn('[%s] Error fetching data using : %s' % (e, comic['ComicImage'])) logger.warn('Unable to download image from CV URL link: ' + comic['ComicImage'] + ' [Status Code returned: ' + str(r.status_code) + ']')
logger.info('Attempting to use alternate image size to get cover.')
#new CV API restriction - one api request / second. logger.fdebug('comic image retrieval status code: ' + str(r.status_code))
if mylar.CVAPI_RATE is None or mylar.CVAPI_RATE < 2:
time.sleep(2) if str(r.status_code) != '200':
logger.warn('Unable to download image from CV URL link: ' + comic['ComicImage'] + ' [Status Code returned: ' + str(r.status_code) + ']')
coversize = 0
else:
if r.headers.get('Content-Encoding') == 'gzip':
buf = StringIO(r.content)
f = gzip.GzipFile(fileobj=buf)
with open(coverfile, 'wb') as f:
for chunk in r.iter_content(chunk_size=1024):
if chunk: # filter out keep-alive new chunks
f.write(chunk)
f.flush()
statinfo = os.stat(coverfile)
coversize = statinfo.st_size
if int(coversize) < 35000 or str(r.status_code) != '200':
if str(r.status_code) != '200':
logger.info('Trying to grab an alternate cover due to problems trying to retrieve the main cover image.')
else: else:
time.sleep(mylar.CVAPI_RATE) logger.info('Image size invalid [' + str(coversize) + ' bytes] - trying to get alternate cover image.')
logger.fdebug('invalid image link is here: ' + comic['ComicImage'])
os.remove(coverfile)
logger.info('Attempting to retrieve alternate comic image for the series.')
try: try:
cimage = re.sub('[\+]', '%20', comic['ComicImageALT']) r = requests.get(comic['ComicImageALT'], params=None, stream=True, headers=mylar.CV_HEADERS)
request = urllib2.Request(cimage)
response = urllib2.urlopen(request)
com_image = response.read()
with open(coverfile, 'wb') as the_file:
the_file.write(com_image)
logger.info('Successfully retrieved cover for ' + comic['ComicName'])
except Exception, e: except Exception, e:
logger.warn('[%s] Error fetching data using : %s' % (e, comic['ComicImageALT'])) logger.warn('Unable to download image from CV URL link: ' + comic['ComicImageALT'] + ' [Status Code returned: ' + str(r.status_code) + ']')
logger.fdebug('comic image retrieval status code: ' + str(r.status_code))
if str(r.status_code) != '200':
logger.warn('Unable to download image from CV URL link: ' + comic['ComicImageALT'] + ' [Status Code returned: ' + str(r.status_code) + ']')
else:
if r.headers.get('Content-Encoding') == 'gzip':
buf = StringIO(r.content)
f = gzip.GzipFile(fileobj=buf)
with open(coverfile, 'wb') as f:
for chunk in r.iter_content(chunk_size=1024):
if chunk: # filter out keep-alive new chunks
f.write(chunk)
f.flush()
PRComicImage = os.path.join('cache', str(comicid) + ".jpg") PRComicImage = os.path.join('cache', str(comicid) + ".jpg")
ComicImage = helpers.replacetheslash(PRComicImage) ComicImage = helpers.replacetheslash(PRComicImage)
@ -1486,7 +1504,7 @@ def annual_check(ComicName, SeriesYear, comicid, issuetype, issuechk, weeklyissu
sresults, explicit = mb.findComic(annComicName, mode, issue=None, explicit='all')#,explicit=True) sresults, explicit = mb.findComic(annComicName, mode, issue=None, explicit='all')#,explicit=True)
type='comic' type='comic'
annual_types_ignore = {'paperback', 'collecting', 'reprints', 'collected', 'print edition', 'tpb', 'available in print', 'collects'} annual_types_ignore = {'paperback', 'collecting', 'reprints', 'collected edition', 'print edition', 'tpb', 'available in print', 'collects'}
if len(sresults) == 1: if len(sresults) == 1:
logger.fdebug('[IMPORTER-ANNUAL] - 1 result') logger.fdebug('[IMPORTER-ANNUAL] - 1 result')
@ -1495,79 +1513,83 @@ def annual_check(ComicName, SeriesYear, comicid, issuetype, issuechk, weeklyissu
num_res = 0 num_res = 0
while (num_res < len(sresults)): while (num_res < len(sresults)):
sr = sresults[num_res] sr = sresults[num_res]
logger.fdebug("description:" + sr['description']) #logger.fdebug("description:" + sr['description'])
if any(x in sr['description'].lower() for x in annual_types_ignore): for x in annual_types_ignore:
logger.fdebug('[IMPORTER-ANNUAL] - tradeback/collected edition detected - skipping ' + str(sr['comicid'])) if x in sr['description'].lower():
else: test_id_position = sr['description'].find(comicid)
if comicid in sr['description']: if test_id_position >= sr['description'].lower().find(x) or test_id_position == -1:
logger.fdebug('[IMPORTER-ANNUAL] - ' + str(comicid) + ' found. Assuming it is part of the greater collection.') logger.fdebug('[IMPORTER-ANNUAL] - tradeback/collected edition detected - skipping ' + str(sr['comicid']))
issueid = sr['comicid']
logger.fdebug('[IMPORTER-ANNUAL] - ' + str(issueid) + ' added to series list as an Annual')
if issueid in annualids:
logger.fdebug('[IMPORTER-ANNUAL] - ' + str(issueid) + ' already exists & was refreshed.')
num_res+=1 # need to manually increment since not a for-next loop
continue continue
issued = cv.getComic(issueid, 'issue')
if len(issued) is None or len(issued) == 0: if comicid in sr['description']:
logger.fdebug('[IMPORTER-ANNUAL] - Could not find any annual information...') logger.fdebug('[IMPORTER-ANNUAL] - ' + str(comicid) + ' found. Assuming it is part of the greater collection.')
pass issueid = sr['comicid']
logger.fdebug('[IMPORTER-ANNUAL] - ' + str(issueid) + ' added to series list as an Annual')
if issueid in annualids:
logger.fdebug('[IMPORTER-ANNUAL] - ' + str(issueid) + ' already exists & was refreshed.')
num_res+=1 # need to manually increment since not a for-next loop
continue
issued = cv.getComic(issueid, 'issue')
if len(issued) is None or len(issued) == 0:
logger.fdebug('[IMPORTER-ANNUAL] - Could not find any annual information...')
pass
else:
n = 0
if int(sr['issues']) == 0 and len(issued['issuechoice']) == 1:
sr_issues = 1
else: else:
n = 0 sr_issues = sr['issues']
if int(sr['issues']) == 0 and len(issued['issuechoice']) == 1: logger.fdebug('[IMPORTER-ANNUAL] - There are ' + str(sr_issues) + ' annuals in this series.')
sr_issues = 1 while (n < int(sr_issues)):
else: try:
sr_issues = sr['issues'] firstval = issued['issuechoice'][n]
logger.fdebug('[IMPORTER-ANNUAL] - There are ' + str(sr_issues) + ' annuals in this series.') except IndexError:
while (n < int(sr_issues)): break
try: try:
firstval = issued['issuechoice'][n] cleanname = helpers.cleanName(firstval['Issue_Name'])
except IndexError: except:
break cleanname = 'None'
try: issid = str(firstval['Issue_ID'])
cleanname = helpers.cleanName(firstval['Issue_Name']) issnum = str(firstval['Issue_Number'])
except: issname = cleanname
cleanname = 'None' issdate = str(firstval['Issue_Date'])
issid = str(firstval['Issue_ID']) stdate = str(firstval['Store_Date'])
issnum = str(firstval['Issue_Number']) int_issnum = helpers.issuedigits(issnum)
issname = cleanname
issdate = str(firstval['Issue_Date'])
stdate = str(firstval['Store_Date'])
int_issnum = helpers.issuedigits(issnum)
newVals = {"Issue_Number": issnum, newVals = {"Issue_Number": issnum,
"Int_IssueNumber": int_issnum, "Int_IssueNumber": int_issnum,
"IssueDate": issdate, "IssueDate": issdate,
"ReleaseDate": stdate, "ReleaseDate": stdate,
"IssueName": issname, "IssueName": issname,
"ComicID": comicid, "ComicID": comicid,
"ComicName": ComicName, "ComicName": ComicName,
"ReleaseComicID": re.sub('4050-', '', firstval['Comic_ID']).strip(), "ReleaseComicID": re.sub('4050-', '', firstval['Comic_ID']).strip(),
"ReleaseComicName": sr['name']} "ReleaseComicName": sr['name']}
iss_exists = myDB.selectone('SELECT * from annuals WHERE IssueID=?', [issid]).fetchone() iss_exists = myDB.selectone('SELECT * from annuals WHERE IssueID=?', [issid]).fetchone()
if iss_exists is None: if iss_exists is None:
datechk = re.sub('-', '', issdate).strip() # converts date to 20140718 format datechk = re.sub('-', '', issdate).strip() # converts date to 20140718 format
if mylar.AUTOWANT_ALL: if mylar.AUTOWANT_ALL:
newVals['Status'] = "Wanted" newVals['Status'] = "Wanted"
elif int(datechk) >= int(nowtime) and mylar.AUTOWANT_UPCOMING: elif int(datechk) >= int(nowtime) and mylar.AUTOWANT_UPCOMING:
newVals['Status'] = "Wanted" newVals['Status'] = "Wanted"
else:
newVals['Status'] = "Skipped"
else: else:
newVals['Status'] = iss_exists['Status'] newVals['Status'] = "Skipped"
else:
newVals['Status'] = iss_exists['Status']
newCtrl = {"IssueID": issid} newCtrl = {"IssueID": issid}
myDB.upsert("annuals", newVals, newCtrl) myDB.upsert("annuals", newVals, newCtrl)
if issuechk is not None and issuetype == 'annual': if issuechk is not None and issuetype == 'annual':
#logger.fdebug('[IMPORTER-ANNUAL] - Comparing annual ' + str(issuechk) + ' .. to .. ' + str(int_issnum)) #logger.fdebug('[IMPORTER-ANNUAL] - Comparing annual ' + str(issuechk) + ' .. to .. ' + str(int_issnum))
if issuechk == int_issnum: if issuechk == int_issnum:
weeklyissue_check.append({"Int_IssueNumber": int_issnum, weeklyissue_check.append({"Int_IssueNumber": int_issnum,
"Issue_Number": issnum, "Issue_Number": issnum,
"IssueDate": issdate, "IssueDate": issdate,
"ReleaseDate": stdate}) "ReleaseDate": stdate})
n+=1 n+=1
num_res+=1 num_res+=1
elif len(sresults) == 0 or len(sresults) is None: elif len(sresults) == 0 or len(sresults) is None:

26
mylar/mb.py
View File

@ -47,9 +47,6 @@ if platform.python_version() == '2.7.6':
def pullsearch(comicapi, comicquery, offset, explicit, type): def pullsearch(comicapi, comicquery, offset, explicit, type):
u_comicquery = urllib.quote(comicquery.encode('utf-8').strip()) u_comicquery = urllib.quote(comicquery.encode('utf-8').strip())
u_comicquery = u_comicquery.replace(" ", "%20") u_comicquery = u_comicquery.replace(" ", "%20")
if '-' in u_comicquery:
#cause titles like A-Force will return 16,000+ results otherwise
u_comicquery = '%22' + u_comicquery + '%22'
if explicit == 'all' or explicit == 'loose': if explicit == 'all' or explicit == 'loose':
PULLURL = mylar.CVURL + 'search?api_key=' + str(comicapi) + '&resources=' + str(type) + '&query=' + u_comicquery + '&field_list=id,name,start_year,first_issue,site_detail_url,count_of_issues,image,publisher,deck,description&format=xml&page=' + str(offset) PULLURL = mylar.CVURL + 'search?api_key=' + str(comicapi) + '&resources=' + str(type) + '&query=' + u_comicquery + '&field_list=id,name,start_year,first_issue,site_detail_url,count_of_issues,image,publisher,deck,description&format=xml&page=' + str(offset)
@ -78,17 +75,7 @@ def pullsearch(comicapi, comicquery, offset, explicit, type):
except Exception, e: except Exception, e:
logger.warn('Error fetching data from ComicVine: %s' % (e)) logger.warn('Error fetching data from ComicVine: %s' % (e))
return return
# try:
# file = urllib2.urlopen(PULLURL)
# except urllib2.HTTPError, err:
# logger.error('err : ' + str(err))
# logger.error("There was a major problem retrieving data from ComicVine - on their end. You'll have to try again later most likely.")
# return
# #convert to string:
# data = file.read()
# #close file because we dont need it anymore:
# file.close()
# #parse the xml you downloaded
dom = parseString(r.content) #(data) dom = parseString(r.content) #(data)
return dom return dom
@ -100,8 +87,8 @@ def findComic(name, mode, issue, limityear=None, explicit=None, type=None):
comiclist = [] comiclist = []
arcinfolist = [] arcinfolist = []
chars = set('!?*') chars = set('!?*&-')
if any((c in chars) for c in name): if any((c in chars) for c in name) or 'annual' in name:
name = '"' +name +'"' name = '"' +name +'"'
#print ("limityear: " + str(limityear)) #print ("limityear: " + str(limityear))
@ -116,10 +103,11 @@ def findComic(name, mode, issue, limityear=None, explicit=None, type=None):
explicit = 'all' explicit = 'all'
#OR #OR
if ' and ' in comicquery.lower() or ' & ' in comicquery: if ' and ' in comicquery.lower():
logger.fdebug('Enforcing exact naming match due to operator in title (and)') logger.fdebug('Enforcing exact naming match due to operator in title (and)')
explicit = 'all' explicit = 'all'
elif explicit == 'loose':
if explicit == 'loose':
logger.fdebug('Changing to loose mode - this will match ANY of the search words') logger.fdebug('Changing to loose mode - this will match ANY of the search words')
comicquery = name.replace(" ", " OR ") comicquery = name.replace(" ", " OR ")
elif explicit == 'explicit': elif explicit == 'explicit':
@ -127,7 +115,7 @@ def findComic(name, mode, issue, limityear=None, explicit=None, type=None):
comicquery=name.replace(" ", " AND ") comicquery=name.replace(" ", " AND ")
else: else:
logger.fdebug('Default search mode - this will match on ALL search words') logger.fdebug('Default search mode - this will match on ALL search words')
comicquery = name.replace(" ", " AND ") #comicquery = name.replace(" ", " AND ")
explicit = 'all' explicit = 'all'

51
mylar/rsscheck.py
View File

@ -181,6 +181,7 @@ def torrents(pickfeed=None, seriesname=None, issue=None, feedinfo=None):
#logger.fdebug('publisher: ' + re.sub("'",'', pub).strip()) #publisher sometimes is given within quotes for some reason, strip 'em. #logger.fdebug('publisher: ' + re.sub("'",'', pub).strip()) #publisher sometimes is given within quotes for some reason, strip 'em.
vol_find = feedme.entries[i].title.find('vol.') vol_find = feedme.entries[i].title.find('vol.')
series = feedme.entries[i].title[st_end +1:vol_find].strip() series = feedme.entries[i].title[st_end +1:vol_find].strip()
series = re.sub('&amp;', '&', series).strip()
#logger.fdebug('series title: ' + series) #logger.fdebug('series title: ' + series)
iss_st = feedme.entries[i].title.find(' - ', vol_find) iss_st = feedme.entries[i].title.find(' - ', vol_find)
vol = re.sub('\.', '', feedme.entries[i].title[vol_find:iss_st]).strip() vol = re.sub('\.', '', feedme.entries[i].title[vol_find:iss_st]).strip()
@ -266,9 +267,19 @@ def nzbs(provider=None, forcerss=False):
feedthis = [] feedthis = []
def _parse_feed(site, url): def _parse_feed(site, url, verify):
logger.fdebug('[RSS] Fetching items from ' + site) logger.fdebug('[RSS] Fetching items from ' + site)
feedme = feedparser.parse(url, agent=str(mylar.USER_AGENT)) payload = None
headers = {'User-Agent': str(mylar.USER_AGENT)}
try:
r = requests.get(url, params=payload, verify=verify, headers=headers)
except Exception, e:
logger.warn('Error fetching RSS Feed Data from %s: %s' % (site, e))
return
feedme = feedparser.parse(r.content)
feedthis.append({"site": site, feedthis.append({"site": site,
"feed": feedme}) "feed": feedme})
@ -276,8 +287,8 @@ def nzbs(provider=None, forcerss=False):
if mylar.NEWZNAB == 1: if mylar.NEWZNAB == 1:
for newznab_host in mylar.EXTRA_NEWZNABS: for newznab_host in mylar.EXTRA_NEWZNABS:
logger.fdebug('[RSS] newznab name: ' + str(newznab_host[0]) + ' - enabled: ' + str(newznab_host[4])) logger.fdebug('[RSS] newznab name: ' + str(newznab_host[0]) + ' - enabled: ' + str(newznab_host[5]))
if str(newznab_host[4]) == '1': if str(newznab_host[5]) == '1':
newznab_hosts.append(newznab_host) newznab_hosts.append(newznab_host)
providercount = len(newznab_hosts) + int(mylar.EXPERIMENTAL == 1) + int(mylar.NZBSU == 1) + int(mylar.DOGNZB == 1) providercount = len(newznab_hosts) + int(mylar.EXPERIMENTAL == 1) + int(mylar.NZBSU == 1) + int(mylar.DOGNZB == 1)
@ -285,24 +296,24 @@ def nzbs(provider=None, forcerss=False):
if mylar.EXPERIMENTAL == 1: if mylar.EXPERIMENTAL == 1:
max_entries = "250" if forcerss else "50" max_entries = "250" if forcerss else "50"
_parse_feed('experimental', 'http://nzbindex.nl/rss/alt.binaries.comics.dcp/?sort=agedesc&max=' + max_entries + '&more=1') _parse_feed('experimental', 'http://nzbindex.nl/rss/alt.binaries.comics.dcp/?sort=agedesc&max=' + max_entries + '&more=1', False)
if mylar.NZBSU == 1: if mylar.NZBSU == 1:
num_items = "&num=100" if forcerss else "" # default is 25 num_items = "&num=100" if forcerss else "" # default is 25
_parse_feed('nzb.su', 'http://api.nzb.su/rss?t=7030&dl=1&i=' + (mylar.NZBSU_UID or '1') + '&r=' + mylar.NZBSU_APIKEY + num_items) _parse_feed('nzb.su', 'http://api.nzb.su/rss?t=7030&dl=1&i=' + (mylar.NZBSU_UID or '1') + '&r=' + mylar.NZBSU_APIKEY + num_items, bool(mylar.NZBSU_VERIFY))
if mylar.DOGNZB == 1: if mylar.DOGNZB == 1:
num_items = "&num=100" if forcerss else "" # default is 25 num_items = "&num=100" if forcerss else "" # default is 25
_parse_feed('dognzb', 'https://dognzb.cr/rss.cfm?r=' + mylar.DOGNZB_APIKEY + '&t=7030' + num_items) _parse_feed('dognzb', 'https://dognzb.cr/rss.cfm?r=' + mylar.DOGNZB_APIKEY + '&t=7030' + num_items, bool(mylar.DOGNZB_VERIFY))
for newznab_host in newznab_hosts: for newznab_host in newznab_hosts:
site = newznab_host[0].rstrip() site = newznab_host[0].rstrip()
(newznabuid, _, newznabcat) = (newznab_host[3] or '').partition('#') (newznabuid, _, newznabcat) = (newznab_host[4] or '').partition('#')
newznabuid = newznabuid or '1' newznabuid = newznabuid or '1'
newznabcat = newznabcat or '7030' newznabcat = newznabcat or '7030'
# 11-21-2014: added &num=100 to return 100 results (or maximum) - unsure of cross-reliablity # 11-21-2014: added &num=100 to return 100 results (or maximum) - unsure of cross-reliablity
_parse_feed(site, newznab_host[1].rstrip() + '/rss?t=' + str(newznabcat) + '&dl=1&i=' + str(newznabuid) + '&num=100&r=' + newznab_host[2].rstrip()) _parse_feed(site, newznab_host[1].rstrip() + '/rss?t=' + str(newznabcat) + '&dl=1&i=' + str(newznabuid) + '&num=100&r=' + newznab_host[3].rstrip(), bool(newznab_host[2]))
feeddata = [] feeddata = []
@ -478,19 +489,23 @@ def torrentdbsearch(seriesname, issue, comicid=None, nzbprov=None):
torinfo = {} torinfo = {}
for tor in tresults: for tor in tresults:
torsplit = tor['Title'].split('/') #&amp; have been brought into the title field incorretly occassionally - patched now, but to include those entries already in the
#cache db that have the incorrect entry, we'll adjust.
torTITLE = re.sub('&amp;', '&', tor['Title']).strip()
torsplit = torTITLE.split('/')
if mylar.PREFERRED_QUALITY == 1: if mylar.PREFERRED_QUALITY == 1:
if 'cbr' in tor['Title']: if 'cbr' in torTITLE:
logger.fdebug('Quality restriction enforced [ cbr only ]. Accepting result.') logger.fdebug('Quality restriction enforced [ cbr only ]. Accepting result.')
else: else:
logger.fdebug('Quality restriction enforced [ cbr only ]. Rejecting result.') logger.fdebug('Quality restriction enforced [ cbr only ]. Rejecting result.')
elif mylar.PREFERRED_QUALITY == 2: elif mylar.PREFERRED_QUALITY == 2:
if 'cbz' in tor['Title']: if 'cbz' in torTITLE:
logger.fdebug('Quality restriction enforced [ cbz only ]. Accepting result.') logger.fdebug('Quality restriction enforced [ cbz only ]. Accepting result.')
else: else:
logger.fdebug('Quality restriction enforced [ cbz only ]. Rejecting result.') logger.fdebug('Quality restriction enforced [ cbz only ]. Rejecting result.')
logger.fdebug('tor-Title: ' + tor['Title']) logger.fdebug('tor-Title: ' + torTITLE)
logger.fdebug('there are ' + str(len(torsplit)) + ' sections in this title') logger.fdebug('there are ' + str(len(torsplit)) + ' sections in this title')
i=0 i=0
if nzbprov is not None: if nzbprov is not None:
@ -542,7 +557,7 @@ def torrentdbsearch(seriesname, issue, comicid=None, nzbprov=None):
logger.fdebug(str(len(formatrem_seriesname)) + ' - formatrem_seriesname :' + formatrem_seriesname.lower()) logger.fdebug(str(len(formatrem_seriesname)) + ' - formatrem_seriesname :' + formatrem_seriesname.lower())
if formatrem_seriesname.lower() in formatrem_torsplit.lower() or any(x.lower() in formatrem_torsplit.lower() for x in AS_Alt): if formatrem_seriesname.lower() in formatrem_torsplit.lower() or any(x.lower() in formatrem_torsplit.lower() for x in AS_Alt):
logger.fdebug('matched to : ' + tor['Title']) logger.fdebug('matched to : ' + torTITLE)
logger.fdebug('matched on series title: ' + seriesname) logger.fdebug('matched on series title: ' + seriesname)
titleend = formatrem_torsplit[len(formatrem_seriesname):] titleend = formatrem_torsplit[len(formatrem_seriesname):]
titleend = re.sub('\-', '', titleend) #remove the '-' which is unnecessary titleend = re.sub('\-', '', titleend) #remove the '-' which is unnecessary
@ -556,15 +571,15 @@ def torrentdbsearch(seriesname, issue, comicid=None, nzbprov=None):
extra = '' extra = ''
#the title on 32P has a mix-mash of crap...ignore everything after cbz/cbr to cleanit #the title on 32P has a mix-mash of crap...ignore everything after cbz/cbr to cleanit
ctitle = tor['Title'].find('cbr') ctitle = torTITLE.find('cbr')
if ctitle == 0: if ctitle == 0:
ctitle = tor['Title'].find('cbz') ctitle = torTITLE.find('cbz')
if ctitle == 0: if ctitle == 0:
ctitle = tor['Title'].find('none') ctitle = torTITLE.find('none')
if ctitle == 0: if ctitle == 0:
logger.fdebug('cannot determine title properly - ignoring for now.') logger.fdebug('cannot determine title properly - ignoring for now.')
continue continue
cttitle = tor['Title'][:ctitle] cttitle = torTITLE[:ctitle]
if tor['Site'] == '32P': if tor['Site'] == '32P':
st_pub = rebuiltline.find('(') st_pub = rebuiltline.find('(')

80
mylar/search.py
View File

@ -223,6 +223,11 @@ def search_init(ComicName, IssueNumber, ComicYear, SeriesYear, Publisher, IssueD
newznab_host = None newznab_host = None
searchprov = prov_order[prov_count].lower() searchprov = prov_order[prov_count].lower()
if searchprov == 'dognzb' and mylar.DOGNZB == 0:
#since dognzb could hit the 50 daily api limit during the middle of a search run, check here on each pass to make
#sure it's not disabled (it gets auto-disabled on maxing out the API hits)
prov_count+=1
continue
if searchmode == 'rss': if searchmode == 'rss':
findit = NZB_SEARCH(ComicName, IssueNumber, ComicYear, SeriesYear, Publisher, IssueDate, StoreDate, searchprov, send_prov_count, IssDateFix, IssueID, UseFuzzy, newznab_host, ComicVersion=ComicVersion, SARC=SARC, IssueArcID=IssueArcID, RSS="yes", ComicID=ComicID, issuetitle=issuetitle, unaltered_ComicName=unaltered_ComicName) findit = NZB_SEARCH(ComicName, IssueNumber, ComicYear, SeriesYear, Publisher, IssueDate, StoreDate, searchprov, send_prov_count, IssDateFix, IssueID, UseFuzzy, newznab_host, ComicVersion=ComicVersion, SARC=SARC, IssueArcID=IssueArcID, RSS="yes", ComicID=ComicID, issuetitle=issuetitle, unaltered_ComicName=unaltered_ComicName)
if findit == 'yes': if findit == 'yes':
@ -451,6 +456,10 @@ def NZB_SEARCH(ComicName, IssueNumber, ComicYear, SeriesYear, Publisher, IssueDa
while (findloop < findcount): while (findloop < findcount):
#logger.fdebug('findloop: ' + str(findloop) + ' / findcount: ' + str(findcount)) #logger.fdebug('findloop: ' + str(findloop) + ' / findcount: ' + str(findcount))
comsrc = comsearch comsrc = comsearch
if nzbprov == 'dognzb' and not mylar.DOGNZB:
foundc = "no"
done = True
break
while (cmloopit >= 1): while (cmloopit >= 1):
#if issue_except is None: issue_exc = '' #if issue_except is None: issue_exc = ''
#else: issue_exc = issue_except #else: issue_exc = issue_except
@ -568,14 +577,14 @@ def NZB_SEARCH(ComicName, IssueNumber, ComicYear, SeriesYear, Publisher, IssueDa
headers = {'User-Agent': str(mylar.USER_AGENT)} headers = {'User-Agent': str(mylar.USER_AGENT)}
payload = None payload = None
if findurl.startswith('https'): if findurl.startswith('https:') and verify == False:
try: try:
from lib.requests.packages.urllib3 import disable_warnings from lib.requests.packages.urllib3 import disable_warnings
disable_warnings() disable_warnings()
except: except:
logger.warn('Unable to disable https warnings. Expect some spam if using https nzb providers.') logger.warn('Unable to disable https warnings. Expect some spam if using https nzb providers.')
else: elif findurl.startswith('http:') and verify == True:
verify = False verify = False
#logger.fdebug('[SSL: ' + str(verify) + '] Search URL: ' + findurl) #logger.fdebug('[SSL: ' + str(verify) + '] Search URL: ' + findurl)
@ -591,10 +600,11 @@ def NZB_SEARCH(ComicName, IssueNumber, ComicYear, SeriesYear, Publisher, IssueDa
break break
data = False data = False
logger.info(r.content) logger.info('status code: ' + str(r.status_code))
if str(r.status_code) != '200': if str(r.status_code) != '200':
logger.warn('Unable to download torrent from ' + nzbprov + ' [Status Code returned: ' + str(r.status_code) + ']') logger.warn('Unable to retrieve search results from ' + tmpprov + ' [Status Code returned: ' + str(r.status_code) + ']')
data = False
else: else:
data = r.content data = r.content
@ -602,11 +612,17 @@ def NZB_SEARCH(ComicName, IssueNumber, ComicYear, SeriesYear, Publisher, IssueDa
bb = feedparser.parse(data) bb = feedparser.parse(data)
else: else:
bb = "no results" bb = "no results"
#logger.info('Search results:' + str(bb)) #logger.info('Search results:' + str(bb))
try: try:
if bb['feed']['error']: if bb['feed']['error']:
logger.error('[ERROR CODE: ' + str(bb['feed']['error']['code']) + '] ' + str(bb['feed']['error']['description'])) logger.error('[ERROR CODE: ' + str(bb['feed']['error']['code']) + '] ' + str(bb['feed']['error']['description']))
bb = "no results" if bb['feed']['error']['code'] == '910':
logger.warn('DAILY API limit reached. Disabling provider usage until 12:01am')
mylar.DOGNZB = 0
foundc = False
done = True
break
except: except:
#logger.info('no errors on data retrieval...proceeding') #logger.info('no errors on data retrieval...proceeding')
pass pass
@ -829,18 +845,28 @@ def NZB_SEARCH(ComicName, IssueNumber, ComicYear, SeriesYear, Publisher, IssueDa
ComVersChk = 0 ComVersChk = 0
ctchk = cleantitle.split() ctchk = cleantitle.split()
ctchk_indexes = []
volfound = False volfound = False
vol_label = None vol_nono = []
new_cleantitle = []
fndcomicversion = None fndcomicversion = None
for ct in ctchk: for ct in ctchk:
if any([ct.lower().startswith('v') and ct[1:].isdigit(), ct.lower()[:3] == 'vol', volfound == True]): if any([ct.lower().startswith('v') and ct[1:].isdigit(), ct.lower()[:3] == 'vol', volfound == True]):
if volfound == True: if volfound == True:
logger.fdebug('Split Volume label detected - ie. Vol 4. Attempting to adust.') logger.fdebug('Split Volume label detected [' + ct + '] - ie. Vol 4. Attempting to adust.')
if ct.isdigit(): if ct.isdigit():
vol_label = vol_label + ' ' + str(ct) vol_nono.append(ctchk.index(ct))
#recreate the cleantitle, with the volume label completely removed (but stored for comparison later)
ct = 'v' + str(ct) ct = 'v' + str(ct)
ctchk_indexes.extend(range(0, len(ctchk)))
logger.info(ctchk_indexes)
for i in ctchk_indexes:
if i not in vol_nono:
new_cleantitle.append(ctchk[i])
cleantitle = ' '.join(new_cleantitle)
logger.fdebug('Newly finished reformed cleantitle (with NO volume label): ' + cleantitle)
volfound == False volfound == False
cleantitle = re.sub(vol_label, ct, cleantitle).strip()
tmpsplit = ct tmpsplit = ct
if tmpsplit.lower().startswith('vol'): if tmpsplit.lower().startswith('vol'):
logger.fdebug('volume detected - stripping and re-analzying for volume label.') logger.fdebug('volume detected - stripping and re-analzying for volume label.')
@ -850,7 +876,8 @@ def NZB_SEARCH(ComicName, IssueNumber, ComicYear, SeriesYear, Publisher, IssueDa
#if vol label set as 'Vol 4' it will obliterate the Vol, but pass over the '4' - set #if vol label set as 'Vol 4' it will obliterate the Vol, but pass over the '4' - set
#volfound to True so that it can loop back around. #volfound to True so that it can loop back around.
if not tmpsplit.isdigit(): if not tmpsplit.isdigit():
vol_label = ct #store the wording of how the Vol is defined so we can skip it later on. #vol_label = ct #store the wording of how the Vol is defined so we can skip it later on.
vol_nono.append(ctchk.index(ct))
volfound = True volfound = True
continue continue
@ -1251,26 +1278,26 @@ def NZB_SEARCH(ComicName, IssueNumber, ComicYear, SeriesYear, Publisher, IssueDa
# instead of the Series they belong to (V2012 vs V2013) # instead of the Series they belong to (V2012 vs V2013)
if annualize == "true" and int(ComicYear) == int(F_ComicVersion): if annualize == "true" and int(ComicYear) == int(F_ComicVersion):
logger.fdebug("We matched on versions for annuals " + str(fndcomicversion)) logger.fdebug("We matched on versions for annuals " + str(fndcomicversion))
scount+=1 #scount+=1
cvers = "true" #cvers = "true"
elif int(F_ComicVersion) == int(D_ComicVersion) or int(F_ComicVersion) == int(S_ComicVersion): elif int(F_ComicVersion) == int(D_ComicVersion) or int(F_ComicVersion) == int(S_ComicVersion):
logger.fdebug("We matched on versions..." + str(fndcomicversion)) logger.fdebug("We matched on versions..." + str(fndcomicversion))
scount+=1 #scount+=1
cvers = "true" #cvers = "true"
else: else:
logger.fdebug("Versions wrong. Ignoring possible match.") logger.fdebug("Versions wrong. Ignoring possible match.")
scount = 0 #scount = 0
cvers = "false" #cvers = "false"
if cvers == "true": #if cvers == "true":
#since we matched on versions, let's remove it entirely to improve matching. #since we matched on versions, let's remove it entirely to improve matching.
logger.fdebug('Removing versioning [' + fndcomicversion + '] from nzb filename to improve matching algorithims.') #logger.fdebug('Removing versioning [' + fndcomicversion + '] from nzb filename to improve matching algorithims.')
cissb4vers = re.sub(fndcomicversion, "", comic_iss_b4).strip() #cissb4vers = re.sub(fndcomicversion, "", comic_iss_b4).strip()
logger.fdebug('New b4split : ' + str(cissb4vers)) #logger.fdebug('New b4split : ' + str(cissb4vers))
splitit = cissb4vers.split(None) #splitit = cissb4vers.split(None)
splitst -=1 #splitst -=1
#do an initial check #do an initial check
initialchk = 'ok' initialchk = 'ok'
@ -1800,8 +1827,9 @@ def searcher(nzbprov, nzbname, comicinfo, link, IssueID, ComicID, tmpprov, direc
else: else:
host_newznab_fix = host_newznab host_newznab_fix = host_newznab
if 'warp?x=' in link: #account for nzbmegasearch & nzbhydra
logger.fdebug('NZBMegaSearch url detected. Adjusting...') if 'warp?x=' in link or 'indexerguid' in link:
logger.fdebug('NZBMegaSearch / NZBHydra url detected. Adjusting...')
nzbmega = True nzbmega = True
else: else:
apikey = newznab[3].rstrip() apikey = newznab[3].rstrip()
@ -2395,7 +2423,7 @@ def generate_id(nzbprov, link):
url_parts = urlparse.urlparse(link) url_parts = urlparse.urlparse(link)
path_parts = url_parts[2].rpartition('/') path_parts = url_parts[2].rpartition('/')
nzbid = path_parts[0].rsplit('/', 1)[1] nzbid = path_parts[0].rsplit('/', 1)[1]
elif nzbprov == 'newznab': elif nzbprov == 'newznab':
#if in format of http://newznab/getnzb/<id>.nzb&i=1&r=apikey #if in format of http://newznab/getnzb/<id>.nzb&i=1&r=apikey
tmpid = urlparse.urlparse(link)[4] #param 4 is the query string from the url. tmpid = urlparse.urlparse(link)[4] #param 4 is the query string from the url.
if 'warp' in urlparse.urlparse(link)[2] and 'x=' in tmpid: if 'warp' in urlparse.urlparse(link)[2] and 'x=' in tmpid:
@ -2406,6 +2434,8 @@ def generate_id(nzbprov, link):
# for the geek in all of us... # for the geek in all of us...
st = tmpid.find('&id') st = tmpid.find('&id')
end = tmpid.find('&', st +1) end = tmpid.find('&', st +1)
if end == -1:
end = len(tmpid)
nzbid = re.sub('&id=', '', tmpid[st:end]).strip() nzbid = re.sub('&id=', '', tmpid[st:end]).strip()
elif nzbprov == 'Torznab': elif nzbprov == 'Torznab':
if mylar.TORZNAB_HOST.endswith('/'): if mylar.TORZNAB_HOST.endswith('/'):

84
mylar/webserve.py
View File

@ -3425,6 +3425,8 @@ class WebInterface(object):
"maxsize": mylar.MAXSIZE, "maxsize": mylar.MAXSIZE,
"interface_list": interface_list, "interface_list": interface_list,
"dupeconstraint": mylar.DUPECONSTRAINT, "dupeconstraint": mylar.DUPECONSTRAINT,
"ddump": helpers.checked(mylar.DDUMP),
"duplicate_dump": mylar.DUPLICATE_DUMP,
"autowant_all": helpers.checked(mylar.AUTOWANT_ALL), "autowant_all": helpers.checked(mylar.AUTOWANT_ALL),
"autowant_upcoming": helpers.checked(mylar.AUTOWANT_UPCOMING), "autowant_upcoming": helpers.checked(mylar.AUTOWANT_UPCOMING),
"comic_cover_local": helpers.checked(mylar.COMIC_COVER_LOCAL), "comic_cover_local": helpers.checked(mylar.COMIC_COVER_LOCAL),
@ -3689,7 +3691,7 @@ class WebInterface(object):
prowl_enabled=0, prowl_onsnatch=0, prowl_keys=None, prowl_priority=None, nma_enabled=0, nma_apikey=None, nma_priority=0, nma_onsnatch=0, pushover_enabled=0, pushover_onsnatch=0, pushover_apikey=None, pushover_userkey=None, pushover_priority=None, boxcar_enabled=0, boxcar_onsnatch=0, boxcar_token=None, prowl_enabled=0, prowl_onsnatch=0, prowl_keys=None, prowl_priority=None, nma_enabled=0, nma_apikey=None, nma_priority=0, nma_onsnatch=0, pushover_enabled=0, pushover_onsnatch=0, pushover_apikey=None, pushover_userkey=None, pushover_priority=None, boxcar_enabled=0, boxcar_onsnatch=0, boxcar_token=None,
pushbullet_enabled=0, pushbullet_apikey=None, pushbullet_deviceid=None, pushbullet_onsnatch=0, pushbullet_enabled=0, pushbullet_apikey=None, pushbullet_deviceid=None, pushbullet_onsnatch=0,
preferred_quality=0, move_files=0, rename_files=0, add_to_csv=1, cvinfo=0, lowercase_filenames=0, folder_format=None, file_format=None, enable_extra_scripts=0, extra_scripts=None, enable_pre_scripts=0, pre_scripts=None, post_processing=0, file_opts=None, syno_fix=0, search_delay=None, chmod_dir=0777, chmod_file=0660, chowner=None, chgroup=None, preferred_quality=0, move_files=0, rename_files=0, add_to_csv=1, cvinfo=0, lowercase_filenames=0, folder_format=None, file_format=None, enable_extra_scripts=0, extra_scripts=None, enable_pre_scripts=0, pre_scripts=None, post_processing=0, file_opts=None, syno_fix=0, search_delay=None, chmod_dir=0777, chmod_file=0660, chowner=None, chgroup=None,
tsab=None, destination_dir=None, create_folders=1, replace_spaces=0, replace_char=None, use_minsize=0, minsize=None, use_maxsize=0, maxsize=None, autowant_all=0, autowant_upcoming=0, comic_cover_local=0, zero_level=0, zero_level_n=None, interface=None, dupeconstraint=None, **kwargs): tsab=None, destination_dir=None, create_folders=1, replace_spaces=0, replace_char=None, use_minsize=0, minsize=None, use_maxsize=0, maxsize=None, autowant_all=0, autowant_upcoming=0, comic_cover_local=0, zero_level=0, zero_level_n=None, interface=None, dupeconstraint=None, ddump=0, duplicate_dump=None, **kwargs):
mylar.COMICVINE_API = comicvine_api mylar.COMICVINE_API = comicvine_api
mylar.HTTP_HOST = http_host mylar.HTTP_HOST = http_host
mylar.HTTP_PORT = http_port mylar.HTTP_PORT = http_port
@ -3815,6 +3817,8 @@ class WebInterface(object):
mylar.COMIC_COVER_LOCAL = comic_cover_local mylar.COMIC_COVER_LOCAL = comic_cover_local
mylar.INTERFACE = interface mylar.INTERFACE = interface
mylar.DUPECONSTRAINT = dupeconstraint mylar.DUPECONSTRAINT = dupeconstraint
mylar.DDUMP = ddump
mylar.DUPLICATE_DUMP = duplicate_dump
mylar.ENABLE_EXTRA_SCRIPTS = enable_extra_scripts mylar.ENABLE_EXTRA_SCRIPTS = enable_extra_scripts
mylar.EXTRA_SCRIPTS = extra_scripts mylar.EXTRA_SCRIPTS = extra_scripts
mylar.ENABLE_PRE_SCRIPTS = enable_pre_scripts mylar.ENABLE_PRE_SCRIPTS = enable_pre_scripts
@ -4082,43 +4086,51 @@ class WebInterface(object):
filelocation = filelocation.encode('ASCII') filelocation = filelocation.encode('ASCII')
filelocation = urllib.unquote_plus(filelocation).decode('utf8') filelocation = urllib.unquote_plus(filelocation).decode('utf8')
issuedetails = helpers.IssueDetails(filelocation) issuedetails = helpers.IssueDetails(filelocation)
#print str(issuedetails) if issuedetails:
issueinfo = '<table width="500"><tr><td>' #print str(issuedetails)
issueinfo += '<img style="float: left; padding-right: 10px" src=' + issuedetails[0]['IssueImage'] + ' height="400" width="263">' issueinfo = '<table width="500"><tr><td>'
issueinfo += '<h1><center><b>' + issuedetails[0]['series'] + '</br>[#' + issuedetails[0]['issue_number'] + ']</b></center></h1>' issueinfo += '<img style="float: left; padding-right: 10px" src=' + issuedetails[0]['IssueImage'] + ' height="400" width="263">'
issueinfo += '<center>"' + issuedetails[0]['title'] + '"</center></br>' issueinfo += '<h1><center><b>' + issuedetails[0]['series'] + '</br>[#' + issuedetails[0]['issue_number'] + ']</b></center></h1>'
issueinfo += '</br><p class="alignleft">' + str(issuedetails[0]['pagecount']) + ' pages</p>' issueinfo += '<center>"' + issuedetails[0]['title'] + '"</center></br>'
if issuedetails[0]['day'] is None: issueinfo += '</br><p class="alignleft">' + str(issuedetails[0]['pagecount']) + ' pages</p>'
issueinfo += '<p class="alignright">(' + str(issuedetails[0]['year']) + '-' + str(issuedetails[0]['month']) + ')</p></br>' if issuedetails[0]['day'] is None:
issueinfo += '<p class="alignright">(' + str(issuedetails[0]['year']) + '-' + str(issuedetails[0]['month']) + ')</p></br>'
else:
issueinfo += '<p class="alignright">(' + str(issuedetails[0]['year']) + '-' + str(issuedetails[0]['month']) + '-' + str(issuedetails[0]['day']) + ')</p></br>'
if not issuedetails[0]['writer'] == 'None':
issueinfo += 'Writer: ' + issuedetails[0]['writer'] + '</br>'
if not issuedetails[0]['penciller'] == 'None':
issueinfo += 'Penciller: ' + issuedetails[0]['penciller'] + '</br>'
if not issuedetails[0]['inker'] == 'None':
issueinfo += 'Inker: ' + issuedetails[0]['inker'] + '</br>'
if not issuedetails[0]['colorist'] == 'None':
issueinfo += 'Colorist: ' + issuedetails[0]['colorist'] + '</br>'
if not issuedetails[0]['letterer'] == 'None':
issueinfo += 'Letterer: ' + issuedetails[0]['letterer'] + '</br>'
if not issuedetails[0]['editor'] == 'None':
issueinfo += 'Editor: ' + issuedetails[0]['editor'] + '</br>'
issueinfo += '</td></tr>'
#issueinfo += '<img src="interfaces/default/images/rename.png" height="25" width="25"></td></tr>'
if len(issuedetails[0]['summary']) > 1000:
issuesumm = issuedetails[0]['summary'][:1000] + '...'
else:
issuesumm = issuedetails[0]['summary']
issueinfo += '<tr><td>Summary: ' + issuesumm + '</br></td></tr>'
issueinfo += '<tr><td><center>' + os.path.split(filelocation)[1] + '</center>'
issueinfo += '</td></tr></table>'
else: else:
issueinfo += '<p class="alignright">(' + str(issuedetails[0]['year']) + '-' + str(issuedetails[0]['month']) + '-' + str(issuedetails[0]['day']) + ')</p></br>' ErrorPNG = 'interfaces/default/images/symbol_exclamation.png'
if not issuedetails[0]['writer'] == 'None': issueinfo = '<table width="300"><tr><td>'
issueinfo += 'Writer: ' + issuedetails[0]['writer'] + '</br>' issueinfo += '<img style="float: left; padding-right: 10px" src=' + ErrorPNG + ' height="128" width="128">'
if not issuedetails[0]['penciller'] == 'None': issueinfo += '<h1><center><b>ERROR</b></center></h1></br>'
issueinfo += 'Penciller: ' + issuedetails[0]['penciller'] + '</br>' issueinfo += '<center>Unable to retrieve metadata from within cbz file</center></br>'
if not issuedetails[0]['inker'] == 'None': issueinfo += '<center>Maybe you should try and tag the file again?</center></br>'
issueinfo += 'Inker: ' + issuedetails[0]['inker'] + '</br>' issueinfo += '<tr><td><center>' + os.path.split(filelocation)[1] + '</center>'
if not issuedetails[0]['colorist'] == 'None': issueinfo += '</td></tr></table>'
issueinfo += 'Colorist: ' + issuedetails[0]['colorist'] + '</br>'
if not issuedetails[0]['letterer'] == 'None':
issueinfo += 'Letterer: ' + issuedetails[0]['letterer'] + '</br>'
if not issuedetails[0]['editor'] == 'None':
issueinfo += 'Editor: ' + issuedetails[0]['editor'] + '</br>'
issueinfo += '</td></tr>'
#issueinfo += '<img src="interfaces/default/images/rename.png" height="25" width="25"></td></tr>'
if len(issuedetails[0]['summary']) > 1000:
issuesumm = issuedetails[0]['summary'][:1000] + '...'
else:
issuesumm = issuedetails[0]['summary']
issueinfo += '<tr><td>Summary: ' + issuesumm + '</br></td></tr>'
issueinfo += '<tr><td><center>' + os.path.split(filelocation)[1] + '</center>'
issueinfo += '</td></tr></table>'
return issueinfo return issueinfo
#import json
#json_dump = json.dumps(issuedetails)
#json_dump = json_dump.replace("\\","\\\\")
#print 'json_dump:' + str(json_dump)
#return json_dump
IssueInfo.exposed = True IssueInfo.exposed = True
def manual_metatag(self, dirName, issueid, filename, comicid, comversion): def manual_metatag(self, dirName, issueid, filename, comicid, comversion):

4
mylar/weeklypull.py
View File

@ -754,8 +754,8 @@ def pullitcheck(comic1off_name=None, comic1off_id=None, forcecheck=None, futurep
break break
else: else:
#logger.fdebug('issuedate:' + str(datevalues[0]['issuedate'])) logger.fdebug('issuedate:' + str(datevalues[0]['issuedate']))
#logger.fdebug('status:' + str(datevalues[0]['status'])) logger.fdebug('status:' + str(datevalues[0]['status']))
datestatus = datevalues[0]['status'] datestatus = datevalues[0]['status']
validcheck = checkthis(datevalues[0]['issuedate'], datestatus, usedate) validcheck = checkthis(datevalues[0]['issuedate'], datestatus, usedate)
if validcheck == True: if validcheck == True: