IMP: 32P legacy mode (rss feed only) and Auth mode (user/pass) available now. With Auth mode, will monitor all of personal notification feeds, IMP: Changed access methods to 32P for retrieving links, FIX:(#1007) Fixed error for one-off downloading via weekly pull list, FIX: (#1004) Fix for post-processing errors (missing files), FIX: (#1001) Extra scripts & pre-scripts will now accept bash scripts, FIX: Annuals were being incorrectly tallied in Have total as well as not having status correctly updated, IMP: Fixed some discrepencies with the provider_order generation line, IMP: Made some additional adjustments to the file-checking to account for some additional variations in naming, FIX: Fixed traceback errors when folder monitor was running and wouldn't be able to complete the post-processing properly, FIX: Cleanup code and some other small fixes....
2015-05-09 02:39:28 +00:00
|
|
|
import urllib2
|
|
|
|
import re
|
IMP: Ability to use 32P session cookies for Auth Login (thnx @btx), IMP: 32P Backlog support! (individual issues only), FIX: 32P will only do one search occurance, instead of issue-numbered padded searching, IMP: When post-processing, if series is in a Paused status or is in an Ended publishing state with 100% of issues completed - will ignore series for post-processing comparisons, IMP: Dynamic Name matching added as matching algorithims in Weekly Pull Lists, FIX: If series contained a '+', would not be able to scan in files for comparison checks (or during post-processing), FIX: File-checking/Post-Processing was taking the incorrect subdirectory path due to escaped slashes (windows only), FIX: If a series contained more than one/multiple special characters, dynamic naming would be slightly off and cause matching problems in some cases, FIX: When peforming a migration of paths in the db using the locmove option, invalid character references would occur on some machines when moving between OS', IMP: Added status checking of an issue prior to downloading (should fix duplicate downloading of same issue within a specific timeframe), FIX: Fixed some parsing issues when using the ALT_PULL=1 method of the weekly pull list, FIX: Fixed some 500 errors when using reverse proxy (flush Impors, Recheck Files, Manage), FIX: When selecting 'Clear Post-Processed items' from history tab, will now also clear 'Processed' items as well, FIX: When importing series that were successfully scanned for metadata and contained valid ComicID's, would fail to regenerate the dataset, IMP: Added exception catch for ComicVine API limit being reached when performing imports (graceful error), IMP: Added graphical icons to indicate Pause/Loading/Error/Active state in the Manage Comics section
2016-05-12 15:28:28 +00:00
|
|
|
import time
|
|
|
|
import datetime
|
|
|
|
import os
|
|
|
|
import lib.requests as requests
|
IMP: 32P legacy mode (rss feed only) and Auth mode (user/pass) available now. With Auth mode, will monitor all of personal notification feeds, IMP: Changed access methods to 32P for retrieving links, FIX:(#1007) Fixed error for one-off downloading via weekly pull list, FIX: (#1004) Fix for post-processing errors (missing files), FIX: (#1001) Extra scripts & pre-scripts will now accept bash scripts, FIX: Annuals were being incorrectly tallied in Have total as well as not having status correctly updated, IMP: Fixed some discrepencies with the provider_order generation line, IMP: Made some additional adjustments to the file-checking to account for some additional variations in naming, FIX: Fixed traceback errors when folder monitor was running and wouldn't be able to complete the post-processing properly, FIX: Cleanup code and some other small fixes....
2015-05-09 02:39:28 +00:00
|
|
|
from bs4 import BeautifulSoup
|
IMP: Ability to use 32P session cookies for Auth Login (thnx @btx), IMP: 32P Backlog support! (individual issues only), FIX: 32P will only do one search occurance, instead of issue-numbered padded searching, IMP: When post-processing, if series is in a Paused status or is in an Ended publishing state with 100% of issues completed - will ignore series for post-processing comparisons, IMP: Dynamic Name matching added as matching algorithims in Weekly Pull Lists, FIX: If series contained a '+', would not be able to scan in files for comparison checks (or during post-processing), FIX: File-checking/Post-Processing was taking the incorrect subdirectory path due to escaped slashes (windows only), FIX: If a series contained more than one/multiple special characters, dynamic naming would be slightly off and cause matching problems in some cases, FIX: When peforming a migration of paths in the db using the locmove option, invalid character references would occur on some machines when moving between OS', IMP: Added status checking of an issue prior to downloading (should fix duplicate downloading of same issue within a specific timeframe), FIX: Fixed some parsing issues when using the ALT_PULL=1 method of the weekly pull list, FIX: Fixed some 500 errors when using reverse proxy (flush Impors, Recheck Files, Manage), FIX: When selecting 'Clear Post-Processed items' from history tab, will now also clear 'Processed' items as well, FIX: When importing series that were successfully scanned for metadata and contained valid ComicID's, would fail to regenerate the dataset, IMP: Added exception catch for ComicVine API limit being reached when performing imports (graceful error), IMP: Added graphical icons to indicate Pause/Loading/Error/Active state in the Manage Comics section
2016-05-12 15:28:28 +00:00
|
|
|
from cookielib import LWPCookieJar
|
|
|
|
|
IMP: 32P legacy mode (rss feed only) and Auth mode (user/pass) available now. With Auth mode, will monitor all of personal notification feeds, IMP: Changed access methods to 32P for retrieving links, FIX:(#1007) Fixed error for one-off downloading via weekly pull list, FIX: (#1004) Fix for post-processing errors (missing files), FIX: (#1001) Extra scripts & pre-scripts will now accept bash scripts, FIX: Annuals were being incorrectly tallied in Have total as well as not having status correctly updated, IMP: Fixed some discrepencies with the provider_order generation line, IMP: Made some additional adjustments to the file-checking to account for some additional variations in naming, FIX: Fixed traceback errors when folder monitor was running and wouldn't be able to complete the post-processing properly, FIX: Cleanup code and some other small fixes....
2015-05-09 02:39:28 +00:00
|
|
|
import mylar
|
IMP: Ability to use 32P session cookies for Auth Login (thnx @btx), IMP: 32P Backlog support! (individual issues only), FIX: 32P will only do one search occurance, instead of issue-numbered padded searching, IMP: When post-processing, if series is in a Paused status or is in an Ended publishing state with 100% of issues completed - will ignore series for post-processing comparisons, IMP: Dynamic Name matching added as matching algorithims in Weekly Pull Lists, FIX: If series contained a '+', would not be able to scan in files for comparison checks (or during post-processing), FIX: File-checking/Post-Processing was taking the incorrect subdirectory path due to escaped slashes (windows only), FIX: If a series contained more than one/multiple special characters, dynamic naming would be slightly off and cause matching problems in some cases, FIX: When peforming a migration of paths in the db using the locmove option, invalid character references would occur on some machines when moving between OS', IMP: Added status checking of an issue prior to downloading (should fix duplicate downloading of same issue within a specific timeframe), FIX: Fixed some parsing issues when using the ALT_PULL=1 method of the weekly pull list, FIX: Fixed some 500 errors when using reverse proxy (flush Impors, Recheck Files, Manage), FIX: When selecting 'Clear Post-Processed items' from history tab, will now also clear 'Processed' items as well, FIX: When importing series that were successfully scanned for metadata and contained valid ComicID's, would fail to regenerate the dataset, IMP: Added exception catch for ComicVine API limit being reached when performing imports (graceful error), IMP: Added graphical icons to indicate Pause/Loading/Error/Active state in the Manage Comics section
2016-05-12 15:28:28 +00:00
|
|
|
from mylar import logger, filechecker
|
IMP: 32P legacy mode (rss feed only) and Auth mode (user/pass) available now. With Auth mode, will monitor all of personal notification feeds, IMP: Changed access methods to 32P for retrieving links, FIX:(#1007) Fixed error for one-off downloading via weekly pull list, FIX: (#1004) Fix for post-processing errors (missing files), FIX: (#1001) Extra scripts & pre-scripts will now accept bash scripts, FIX: Annuals were being incorrectly tallied in Have total as well as not having status correctly updated, IMP: Fixed some discrepencies with the provider_order generation line, IMP: Made some additional adjustments to the file-checking to account for some additional variations in naming, FIX: Fixed traceback errors when folder monitor was running and wouldn't be able to complete the post-processing properly, FIX: Cleanup code and some other small fixes....
2015-05-09 02:39:28 +00:00
|
|
|
|
2015-05-22 08:32:51 +00:00
|
|
|
|
IMP: 32P legacy mode (rss feed only) and Auth mode (user/pass) available now. With Auth mode, will monitor all of personal notification feeds, IMP: Changed access methods to 32P for retrieving links, FIX:(#1007) Fixed error for one-off downloading via weekly pull list, FIX: (#1004) Fix for post-processing errors (missing files), FIX: (#1001) Extra scripts & pre-scripts will now accept bash scripts, FIX: Annuals were being incorrectly tallied in Have total as well as not having status correctly updated, IMP: Fixed some discrepencies with the provider_order generation line, IMP: Made some additional adjustments to the file-checking to account for some additional variations in naming, FIX: Fixed traceback errors when folder monitor was running and wouldn't be able to complete the post-processing properly, FIX: Cleanup code and some other small fixes....
2015-05-09 02:39:28 +00:00
|
|
|
class info32p(object):
|
|
|
|
|
FIX: When retrieving feeds from 32p and in Auth mode, personal notification feeds contained some invalid html entries that weren't removed properly resulting in no results when attempting to match for downloading, FIX: When searching 32P, if title had a '/' within the title - Mylar would mistakingly skip it due to some previous exceptions that were made for CBT, FIX: Main page would quickly display & hide the have% column instead of always being hidden, FIX: Adjusted some incorrect spacing for non-alphanumeric characters when comparing search results (should result in better matching hopefully), FIX: When adding a series and the most recent issue was present on the weekly-pull list, it would sometimes not mark it as Wanted and auto-attempt to search for it (if auto mark Upcoming enabled), FIX: Added Test Connection button for 32P where it will test logon credentials as well as if Captcha is present, IMP: If captcha is enabled for 32p and signon is required because keys are stale, will not send authentication information and will just bypass as a provider, IMP: Test Connection button added for SABnzbd, IMP: Added ability to directly add torrents to rtorrent and apply label + download directory options (config.ini only atm), FIX: If a search result had a 'vol.' label in it, depending on how the format of the label was mylar would refuse to remove the volume which resulted in failed matches (also fixed a similar issue with failing to remove the volume label when comparing search results), FIX: When filechecking, if a series had a - in the title, will now account for it properly, IMP: Completely redid the filecheck module which allows for integration into other modules as well as more detailed failure logs, IMP: Added Dynamic handder integration into filechecker and subsequent modules that use it which allows for special characters to be replaced with any other type of character, IMP: Manual post-processing speed improved greatly due to new usage of filecheck module, IMP: Importer backend code redone to include new filecheck module, IMP: Added status/counter to import process, IMP: Added force unlock option to importer for failed imports, IMP: Added new status to Import labelled as 'Manual Intervention' for imports that need the user to manually select an option from an available list, FIX: When import said there were search results to view, but none available - would blank screen, IMP: Added a failure log entry showing all the failed files that weren't able to be scanned in during an import (will be in GUI eventually), IMP: if only partial metadata is available during import, Mylar will attempt to use what's available from the metatagging instead of picking all of one/other, IMP: Better grouping of series/volumes when viewing the import results page as well as now indicating if annuals are present within the files, IMP: Added a file-icon beside each imported item on the import result page which allows the user to view the files that are associated with the given series grouping, IMP: Added a blacklisted_publishers option to config.ini which will blacklist specific publishers from being returned during search / import results, FIX: If duplicate dump folder had a value, but duplicate dump wasn't enabled - would still use the duplicate dump folder during post-processing runs, FIX: (#1194) Patch to allow for fixed H1 elements for title (thnx chazlarson), FIX: Removed UnRAR dependency checks in cmtagmylar since not being used anymore, FIX: Fixed a problem with non-ascii characters being recognized during a file-check in certain cases, IMP: Attmept by Mylar to grab an alternate jpg from file when viewing the issue details if it complies with the naming conventions, FIX: Fixed some metatagging issues with ComicBookLover tags not being handled properly if they didn't exist, IMP: Dupecheck now has a failback if it's comparing a cbr/cbr, cbz/cbz and cbr/cbz-priority is enabled, FIX: Quick check added for when adding/refreshing a comic that if a cover already existed, it would delete the cover prior to the attempt to retrieve it, IMP: Added some additional handling for when searching/adding fails, FIX: If a story arc didn't have proper issue dates (or invalid ones) would error out on loading the story arc main page - usually when arcs were imported using a cbl file.
2016-04-07 17:09:06 +00:00
|
|
|
def __init__(self, reauthenticate=False, searchterm=None, test=False):
|
IMP: 32P legacy mode (rss feed only) and Auth mode (user/pass) available now. With Auth mode, will monitor all of personal notification feeds, IMP: Changed access methods to 32P for retrieving links, FIX:(#1007) Fixed error for one-off downloading via weekly pull list, FIX: (#1004) Fix for post-processing errors (missing files), FIX: (#1001) Extra scripts & pre-scripts will now accept bash scripts, FIX: Annuals were being incorrectly tallied in Have total as well as not having status correctly updated, IMP: Fixed some discrepencies with the provider_order generation line, IMP: Made some additional adjustments to the file-checking to account for some additional variations in naming, FIX: Fixed traceback errors when folder monitor was running and wouldn't be able to complete the post-processing properly, FIX: Cleanup code and some other small fixes....
2015-05-09 02:39:28 +00:00
|
|
|
|
|
|
|
self.module = '[32P-AUTHENTICATION]'
|
IMP: Ability to use 32P session cookies for Auth Login (thnx @btx), IMP: 32P Backlog support! (individual issues only), FIX: 32P will only do one search occurance, instead of issue-numbered padded searching, IMP: When post-processing, if series is in a Paused status or is in an Ended publishing state with 100% of issues completed - will ignore series for post-processing comparisons, IMP: Dynamic Name matching added as matching algorithims in Weekly Pull Lists, FIX: If series contained a '+', would not be able to scan in files for comparison checks (or during post-processing), FIX: File-checking/Post-Processing was taking the incorrect subdirectory path due to escaped slashes (windows only), FIX: If a series contained more than one/multiple special characters, dynamic naming would be slightly off and cause matching problems in some cases, FIX: When peforming a migration of paths in the db using the locmove option, invalid character references would occur on some machines when moving between OS', IMP: Added status checking of an issue prior to downloading (should fix duplicate downloading of same issue within a specific timeframe), FIX: Fixed some parsing issues when using the ALT_PULL=1 method of the weekly pull list, FIX: Fixed some 500 errors when using reverse proxy (flush Impors, Recheck Files, Manage), FIX: When selecting 'Clear Post-Processed items' from history tab, will now also clear 'Processed' items as well, FIX: When importing series that were successfully scanned for metadata and contained valid ComicID's, would fail to regenerate the dataset, IMP: Added exception catch for ComicVine API limit being reached when performing imports (graceful error), IMP: Added graphical icons to indicate Pause/Loading/Error/Active state in the Manage Comics section
2016-05-12 15:28:28 +00:00
|
|
|
self.url = 'https://32pag.es/user.php?action=notify'
|
IMP: 32P legacy mode (rss feed only) and Auth mode (user/pass) available now. With Auth mode, will monitor all of personal notification feeds, IMP: Changed access methods to 32P for retrieving links, FIX:(#1007) Fixed error for one-off downloading via weekly pull list, FIX: (#1004) Fix for post-processing errors (missing files), FIX: (#1001) Extra scripts & pre-scripts will now accept bash scripts, FIX: Annuals were being incorrectly tallied in Have total as well as not having status correctly updated, IMP: Fixed some discrepencies with the provider_order generation line, IMP: Made some additional adjustments to the file-checking to account for some additional variations in naming, FIX: Fixed traceback errors when folder monitor was running and wouldn't be able to complete the post-processing properly, FIX: Cleanup code and some other small fixes....
2015-05-09 02:39:28 +00:00
|
|
|
self.headers = {'Content-type': 'application/x-www-form-urlencoded',
|
|
|
|
'Accept-Charset': 'utf-8',
|
|
|
|
'User-Agent': 'Mozilla/5.0'}
|
IMP: Ability to use 32P session cookies for Auth Login (thnx @btx), IMP: 32P Backlog support! (individual issues only), FIX: 32P will only do one search occurance, instead of issue-numbered padded searching, IMP: When post-processing, if series is in a Paused status or is in an Ended publishing state with 100% of issues completed - will ignore series for post-processing comparisons, IMP: Dynamic Name matching added as matching algorithims in Weekly Pull Lists, FIX: If series contained a '+', would not be able to scan in files for comparison checks (or during post-processing), FIX: File-checking/Post-Processing was taking the incorrect subdirectory path due to escaped slashes (windows only), FIX: If a series contained more than one/multiple special characters, dynamic naming would be slightly off and cause matching problems in some cases, FIX: When peforming a migration of paths in the db using the locmove option, invalid character references would occur on some machines when moving between OS', IMP: Added status checking of an issue prior to downloading (should fix duplicate downloading of same issue within a specific timeframe), FIX: Fixed some parsing issues when using the ALT_PULL=1 method of the weekly pull list, FIX: Fixed some 500 errors when using reverse proxy (flush Impors, Recheck Files, Manage), FIX: When selecting 'Clear Post-Processed items' from history tab, will now also clear 'Processed' items as well, FIX: When importing series that were successfully scanned for metadata and contained valid ComicID's, would fail to regenerate the dataset, IMP: Added exception catch for ComicVine API limit being reached when performing imports (graceful error), IMP: Added graphical icons to indicate Pause/Loading/Error/Active state in the Manage Comics section
2016-05-12 15:28:28 +00:00
|
|
|
|
|
|
|
self.error = None
|
|
|
|
self.method = None
|
|
|
|
lses = self.LoginSession(mylar.USERNAME_32P, mylar.PASSWORD_32P)
|
|
|
|
|
|
|
|
if not lses.login():
|
|
|
|
if not self.test:
|
|
|
|
logger.error(self.module + ' [LOGIN FAILED] Disabling 32P provider until login error(s) can be fixed in order to avoid temporary bans.')
|
|
|
|
return "disable"
|
|
|
|
else:
|
|
|
|
if self.error:
|
|
|
|
return self.error #rtnmsg
|
|
|
|
else:
|
|
|
|
return self.method
|
|
|
|
else:
|
|
|
|
logger.info(self.module + '[LOGIN SUCCESS] Now preparing for the use of 32P keyed authentication...')
|
|
|
|
self.authkey = lses.authkey
|
|
|
|
self.passkey = lses.passkey
|
|
|
|
self.uid = lses.uid
|
|
|
|
|
IMP: 32P legacy mode (rss feed only) and Auth mode (user/pass) available now. With Auth mode, will monitor all of personal notification feeds, IMP: Changed access methods to 32P for retrieving links, FIX:(#1007) Fixed error for one-off downloading via weekly pull list, FIX: (#1004) Fix for post-processing errors (missing files), FIX: (#1001) Extra scripts & pre-scripts will now accept bash scripts, FIX: Annuals were being incorrectly tallied in Have total as well as not having status correctly updated, IMP: Fixed some discrepencies with the provider_order generation line, IMP: Made some additional adjustments to the file-checking to account for some additional variations in naming, FIX: Fixed traceback errors when folder monitor was running and wouldn't be able to complete the post-processing properly, FIX: Cleanup code and some other small fixes....
2015-05-09 02:39:28 +00:00
|
|
|
self.reauthenticate = reauthenticate
|
|
|
|
self.searchterm = searchterm
|
FIX: When retrieving feeds from 32p and in Auth mode, personal notification feeds contained some invalid html entries that weren't removed properly resulting in no results when attempting to match for downloading, FIX: When searching 32P, if title had a '/' within the title - Mylar would mistakingly skip it due to some previous exceptions that were made for CBT, FIX: Main page would quickly display & hide the have% column instead of always being hidden, FIX: Adjusted some incorrect spacing for non-alphanumeric characters when comparing search results (should result in better matching hopefully), FIX: When adding a series and the most recent issue was present on the weekly-pull list, it would sometimes not mark it as Wanted and auto-attempt to search for it (if auto mark Upcoming enabled), FIX: Added Test Connection button for 32P where it will test logon credentials as well as if Captcha is present, IMP: If captcha is enabled for 32p and signon is required because keys are stale, will not send authentication information and will just bypass as a provider, IMP: Test Connection button added for SABnzbd, IMP: Added ability to directly add torrents to rtorrent and apply label + download directory options (config.ini only atm), FIX: If a search result had a 'vol.' label in it, depending on how the format of the label was mylar would refuse to remove the volume which resulted in failed matches (also fixed a similar issue with failing to remove the volume label when comparing search results), FIX: When filechecking, if a series had a - in the title, will now account for it properly, IMP: Completely redid the filecheck module which allows for integration into other modules as well as more detailed failure logs, IMP: Added Dynamic handder integration into filechecker and subsequent modules that use it which allows for special characters to be replaced with any other type of character, IMP: Manual post-processing speed improved greatly due to new usage of filecheck module, IMP: Importer backend code redone to include new filecheck module, IMP: Added status/counter to import process, IMP: Added force unlock option to importer for failed imports, IMP: Added new status to Import labelled as 'Manual Intervention' for imports that need the user to manually select an option from an available list, FIX: When import said there were search results to view, but none available - would blank screen, IMP: Added a failure log entry showing all the failed files that weren't able to be scanned in during an import (will be in GUI eventually), IMP: if only partial metadata is available during import, Mylar will attempt to use what's available from the metatagging instead of picking all of one/other, IMP: Better grouping of series/volumes when viewing the import results page as well as now indicating if annuals are present within the files, IMP: Added a file-icon beside each imported item on the import result page which allows the user to view the files that are associated with the given series grouping, IMP: Added a blacklisted_publishers option to config.ini which will blacklist specific publishers from being returned during search / import results, FIX: If duplicate dump folder had a value, but duplicate dump wasn't enabled - would still use the duplicate dump folder during post-processing runs, FIX: (#1194) Patch to allow for fixed H1 elements for title (thnx chazlarson), FIX: Removed UnRAR dependency checks in cmtagmylar since not being used anymore, FIX: Fixed a problem with non-ascii characters being recognized during a file-check in certain cases, IMP: Attmept by Mylar to grab an alternate jpg from file when viewing the issue details if it complies with the naming conventions, FIX: Fixed some metatagging issues with ComicBookLover tags not being handled properly if they didn't exist, IMP: Dupecheck now has a failback if it's comparing a cbr/cbr, cbz/cbz and cbr/cbz-priority is enabled, FIX: Quick check added for when adding/refreshing a comic that if a cover already existed, it would delete the cover prior to the attempt to retrieve it, IMP: Added some additional handling for when searching/adding fails, FIX: If a story arc didn't have proper issue dates (or invalid ones) would error out on loading the story arc main page - usually when arcs were imported using a cbl file.
2016-04-07 17:09:06 +00:00
|
|
|
self.test = test
|
IMP: 32P legacy mode (rss feed only) and Auth mode (user/pass) available now. With Auth mode, will monitor all of personal notification feeds, IMP: Changed access methods to 32P for retrieving links, FIX:(#1007) Fixed error for one-off downloading via weekly pull list, FIX: (#1004) Fix for post-processing errors (missing files), FIX: (#1001) Extra scripts & pre-scripts will now accept bash scripts, FIX: Annuals were being incorrectly tallied in Have total as well as not having status correctly updated, IMP: Fixed some discrepencies with the provider_order generation line, IMP: Made some additional adjustments to the file-checking to account for some additional variations in naming, FIX: Fixed traceback errors when folder monitor was running and wouldn't be able to complete the post-processing properly, FIX: Cleanup code and some other small fixes....
2015-05-09 02:39:28 +00:00
|
|
|
|
|
|
|
def authenticate(self):
|
|
|
|
|
|
|
|
feedinfo = []
|
|
|
|
|
2015-07-01 22:52:26 +00:00
|
|
|
try:
|
IMP: Ability to use 32P session cookies for Auth Login (thnx @btx), IMP: 32P Backlog support! (individual issues only), FIX: 32P will only do one search occurance, instead of issue-numbered padded searching, IMP: When post-processing, if series is in a Paused status or is in an Ended publishing state with 100% of issues completed - will ignore series for post-processing comparisons, IMP: Dynamic Name matching added as matching algorithims in Weekly Pull Lists, FIX: If series contained a '+', would not be able to scan in files for comparison checks (or during post-processing), FIX: File-checking/Post-Processing was taking the incorrect subdirectory path due to escaped slashes (windows only), FIX: If a series contained more than one/multiple special characters, dynamic naming would be slightly off and cause matching problems in some cases, FIX: When peforming a migration of paths in the db using the locmove option, invalid character references would occur on some machines when moving between OS', IMP: Added status checking of an issue prior to downloading (should fix duplicate downloading of same issue within a specific timeframe), FIX: Fixed some parsing issues when using the ALT_PULL=1 method of the weekly pull list, FIX: Fixed some 500 errors when using reverse proxy (flush Impors, Recheck Files, Manage), FIX: When selecting 'Clear Post-Processed items' from history tab, will now also clear 'Processed' items as well, FIX: When importing series that were successfully scanned for metadata and contained valid ComicID's, would fail to regenerate the dataset, IMP: Added exception catch for ComicVine API limit being reached when performing imports (graceful error), IMP: Added graphical icons to indicate Pause/Loading/Error/Active state in the Manage Comics section
2016-05-12 15:28:28 +00:00
|
|
|
with requests.Session() as s:
|
|
|
|
s.headers = self.headers
|
2016-05-19 15:20:13 +00:00
|
|
|
cj = LWPCookieJar(os.path.join(mylar.CACHE_DIR, ".32p_cookies.dat"))
|
IMP: Ability to use 32P session cookies for Auth Login (thnx @btx), IMP: 32P Backlog support! (individual issues only), FIX: 32P will only do one search occurance, instead of issue-numbered padded searching, IMP: When post-processing, if series is in a Paused status or is in an Ended publishing state with 100% of issues completed - will ignore series for post-processing comparisons, IMP: Dynamic Name matching added as matching algorithims in Weekly Pull Lists, FIX: If series contained a '+', would not be able to scan in files for comparison checks (or during post-processing), FIX: File-checking/Post-Processing was taking the incorrect subdirectory path due to escaped slashes (windows only), FIX: If a series contained more than one/multiple special characters, dynamic naming would be slightly off and cause matching problems in some cases, FIX: When peforming a migration of paths in the db using the locmove option, invalid character references would occur on some machines when moving between OS', IMP: Added status checking of an issue prior to downloading (should fix duplicate downloading of same issue within a specific timeframe), FIX: Fixed some parsing issues when using the ALT_PULL=1 method of the weekly pull list, FIX: Fixed some 500 errors when using reverse proxy (flush Impors, Recheck Files, Manage), FIX: When selecting 'Clear Post-Processed items' from history tab, will now also clear 'Processed' items as well, FIX: When importing series that were successfully scanned for metadata and contained valid ComicID's, would fail to regenerate the dataset, IMP: Added exception catch for ComicVine API limit being reached when performing imports (graceful error), IMP: Added graphical icons to indicate Pause/Loading/Error/Active state in the Manage Comics section
2016-05-12 15:28:28 +00:00
|
|
|
cj.load()
|
|
|
|
s.cookies = cj
|
|
|
|
|
2015-07-01 22:52:26 +00:00
|
|
|
if mylar.VERIFY_32P == 1 or mylar.VERIFY_32P == True:
|
|
|
|
verify = True
|
|
|
|
else:
|
|
|
|
verify = False
|
|
|
|
|
|
|
|
logger.fdebug('[32P] Verify SSL set to : ' + str(verify))
|
|
|
|
|
|
|
|
if not verify:
|
|
|
|
#32P throws back an insecure warning because it can't validate against the CA. The below suppresses the message just for 32P instead of being displa$
|
|
|
|
from lib.requests.packages.urllib3.exceptions import InsecureRequestWarning
|
|
|
|
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
|
|
|
|
|
|
|
|
# post to the login form
|
IMP: Ability to use 32P session cookies for Auth Login (thnx @btx), IMP: 32P Backlog support! (individual issues only), FIX: 32P will only do one search occurance, instead of issue-numbered padded searching, IMP: When post-processing, if series is in a Paused status or is in an Ended publishing state with 100% of issues completed - will ignore series for post-processing comparisons, IMP: Dynamic Name matching added as matching algorithims in Weekly Pull Lists, FIX: If series contained a '+', would not be able to scan in files for comparison checks (or during post-processing), FIX: File-checking/Post-Processing was taking the incorrect subdirectory path due to escaped slashes (windows only), FIX: If a series contained more than one/multiple special characters, dynamic naming would be slightly off and cause matching problems in some cases, FIX: When peforming a migration of paths in the db using the locmove option, invalid character references would occur on some machines when moving between OS', IMP: Added status checking of an issue prior to downloading (should fix duplicate downloading of same issue within a specific timeframe), FIX: Fixed some parsing issues when using the ALT_PULL=1 method of the weekly pull list, FIX: Fixed some 500 errors when using reverse proxy (flush Impors, Recheck Files, Manage), FIX: When selecting 'Clear Post-Processed items' from history tab, will now also clear 'Processed' items as well, FIX: When importing series that were successfully scanned for metadata and contained valid ComicID's, would fail to regenerate the dataset, IMP: Added exception catch for ComicVine API limit being reached when performing imports (graceful error), IMP: Added graphical icons to indicate Pause/Loading/Error/Active state in the Manage Comics section
2016-05-12 15:28:28 +00:00
|
|
|
r = s.post(self.url, verify=verify)
|
2015-07-01 22:52:26 +00:00
|
|
|
|
|
|
|
#need a way to find response code (200=OK), but returns 200 for everything even failed signons (returns a blank page)
|
|
|
|
#logger.info('[32P] response: ' + str(r.content))
|
FIX: included version of comictagger should now work with both Windows and *nix based OS' again, IMP: Global Copy/Move option available when performing post-processing, IMP: Added a verbose file-checking option (FOLDER_SCAN_LOG_VERBOSE) - when enabled will log as it currently does during manual post-processing/file-checking runs, when disabled it will not spam the log nearly as much resulting in more readable log files, IMP: Added Verbose debug logging both via startup option(-v), as well as toggle button in Log GUI (from headphones), as well as per-page loading of log file(s) in GUI, FIX: When doing manual post-processing on issues that were in story arcs, will now indicate X story-arc issues were post-processed for better visibility, FIX: Fixed an issue with deleting from the nzblog table when story arc issues were post-processed, IMP: Added WEEKFOLDER_LOC to the config.ini to allow for specification of where the weekly download directories will default to (as opposed to off of ComicLocation root), IMP: Better handling of some special character references in series titles when looking for series on the auto-wanted list, IMP: 32P will now auto-disable provider if logon returns invalid credentials, FIX: When using alt_pull on weekly pull list, xA0 unicode character caused error, FIX: If title had invalid character in filename that was replaced with a character that already existed in the title, would not scan in during file-checking, FIX: When searching for a series (weeklypull-list/add a series), if the title contained 'and' or '&' would return really mixed up results, FIX: When Post-Processing, if filename being processed had special characters (ie. comma) and was different than nzbname, in some cases would fail to find/move issues, IMP: Utilize internal comictagger to convert from cbr/cbz, IMP: Added more checks when post-processing to ensure files are handled correctly, IMP: Added meta-tag reading when importing series/issues - if previously tagged with CT, will reverse look-up the provided IssueID to reference the correct ComicID, IMP: If scanned directory during import contins cvinfo file, use that and force the ComicID to entire directory when importing a series, IMP: Manual meta-tagging issues will no longer create temporary directories and/or create files in the Comic Location root causing problems for some users, FIX: Annuals weren't properly sorted upon loading of comic details page for some series, IMP: Added some extra checks when validating/creating directories, FIX: Fixed a problem when displaying some covers of .cbz files on the comic details page
2016-01-26 07:49:56 +00:00
|
|
|
soup = BeautifulSoup(r.content)
|
|
|
|
soup.prettify()
|
IMP: Ability to use 32P session cookies for Auth Login (thnx @btx), IMP: 32P Backlog support! (individual issues only), FIX: 32P will only do one search occurance, instead of issue-numbered padded searching, IMP: When post-processing, if series is in a Paused status or is in an Ended publishing state with 100% of issues completed - will ignore series for post-processing comparisons, IMP: Dynamic Name matching added as matching algorithims in Weekly Pull Lists, FIX: If series contained a '+', would not be able to scan in files for comparison checks (or during post-processing), FIX: File-checking/Post-Processing was taking the incorrect subdirectory path due to escaped slashes (windows only), FIX: If a series contained more than one/multiple special characters, dynamic naming would be slightly off and cause matching problems in some cases, FIX: When peforming a migration of paths in the db using the locmove option, invalid character references would occur on some machines when moving between OS', IMP: Added status checking of an issue prior to downloading (should fix duplicate downloading of same issue within a specific timeframe), FIX: Fixed some parsing issues when using the ALT_PULL=1 method of the weekly pull list, FIX: Fixed some 500 errors when using reverse proxy (flush Impors, Recheck Files, Manage), FIX: When selecting 'Clear Post-Processed items' from history tab, will now also clear 'Processed' items as well, FIX: When importing series that were successfully scanned for metadata and contained valid ComicID's, would fail to regenerate the dataset, IMP: Added exception catch for ComicVine API limit being reached when performing imports (graceful error), IMP: Added graphical icons to indicate Pause/Loading/Error/Active state in the Manage Comics section
2016-05-12 15:28:28 +00:00
|
|
|
|
|
|
|
if self.searchterm:
|
2015-07-01 22:52:26 +00:00
|
|
|
logger.info('[32P] Successfully authenticated. Initiating search for : ' + self.searchterm)
|
|
|
|
return self.search32p(s)
|
IMP: Ability to use 32P session cookies for Auth Login (thnx @btx), IMP: 32P Backlog support! (individual issues only), FIX: 32P will only do one search occurance, instead of issue-numbered padded searching, IMP: When post-processing, if series is in a Paused status or is in an Ended publishing state with 100% of issues completed - will ignore series for post-processing comparisons, IMP: Dynamic Name matching added as matching algorithims in Weekly Pull Lists, FIX: If series contained a '+', would not be able to scan in files for comparison checks (or during post-processing), FIX: File-checking/Post-Processing was taking the incorrect subdirectory path due to escaped slashes (windows only), FIX: If a series contained more than one/multiple special characters, dynamic naming would be slightly off and cause matching problems in some cases, FIX: When peforming a migration of paths in the db using the locmove option, invalid character references would occur on some machines when moving between OS', IMP: Added status checking of an issue prior to downloading (should fix duplicate downloading of same issue within a specific timeframe), FIX: Fixed some parsing issues when using the ALT_PULL=1 method of the weekly pull list, FIX: Fixed some 500 errors when using reverse proxy (flush Impors, Recheck Files, Manage), FIX: When selecting 'Clear Post-Processed items' from history tab, will now also clear 'Processed' items as well, FIX: When importing series that were successfully scanned for metadata and contained valid ComicID's, would fail to regenerate the dataset, IMP: Added exception catch for ComicVine API limit being reached when performing imports (graceful error), IMP: Added graphical icons to indicate Pause/Loading/Error/Active state in the Manage Comics section
2016-05-12 15:28:28 +00:00
|
|
|
|
|
|
|
logger.info('[32P] Successfully authenticated.')
|
2015-07-01 22:52:26 +00:00
|
|
|
all_script = soup.find_all("script", {"src": False})
|
|
|
|
all_script2 = soup.find_all("link", {"rel": "alternate"})
|
|
|
|
|
|
|
|
authfound = False
|
|
|
|
logger.info(self.module + ' Atttempting to integrate with all of your 32P Notification feeds.')
|
|
|
|
|
|
|
|
for al in all_script2:
|
|
|
|
alurl = al['href']
|
|
|
|
if 'auth=' in alurl and 'torrents_notify' in alurl and not authfound:
|
|
|
|
f1 = alurl.find('auth=')
|
|
|
|
f2 = alurl.find('&', f1 + 1)
|
|
|
|
auth = alurl[f1 +5:f2]
|
|
|
|
authfound = True
|
IMP: Ability to use 32P session cookies for Auth Login (thnx @btx), IMP: 32P Backlog support! (individual issues only), FIX: 32P will only do one search occurance, instead of issue-numbered padded searching, IMP: When post-processing, if series is in a Paused status or is in an Ended publishing state with 100% of issues completed - will ignore series for post-processing comparisons, IMP: Dynamic Name matching added as matching algorithims in Weekly Pull Lists, FIX: If series contained a '+', would not be able to scan in files for comparison checks (or during post-processing), FIX: File-checking/Post-Processing was taking the incorrect subdirectory path due to escaped slashes (windows only), FIX: If a series contained more than one/multiple special characters, dynamic naming would be slightly off and cause matching problems in some cases, FIX: When peforming a migration of paths in the db using the locmove option, invalid character references would occur on some machines when moving between OS', IMP: Added status checking of an issue prior to downloading (should fix duplicate downloading of same issue within a specific timeframe), FIX: Fixed some parsing issues when using the ALT_PULL=1 method of the weekly pull list, FIX: Fixed some 500 errors when using reverse proxy (flush Impors, Recheck Files, Manage), FIX: When selecting 'Clear Post-Processed items' from history tab, will now also clear 'Processed' items as well, FIX: When importing series that were successfully scanned for metadata and contained valid ComicID's, would fail to regenerate the dataset, IMP: Added exception catch for ComicVine API limit being reached when performing imports (graceful error), IMP: Added graphical icons to indicate Pause/Loading/Error/Active state in the Manage Comics section
2016-05-12 15:28:28 +00:00
|
|
|
#logger.fdebug(self.module + ' Auth:' + str(auth))
|
|
|
|
#p1 = alurl.find('passkey=')
|
|
|
|
#p2 = alurl.find('&', p1 + 1)
|
|
|
|
#passkey = alurl[p1 +8:p2]
|
|
|
|
#logger.fdebug(self.module + ' Passkey:' + str(passkey))
|
2015-07-01 22:52:26 +00:00
|
|
|
if self.reauthenticate: break
|
|
|
|
|
IMP: Ability to use 32P session cookies for Auth Login (thnx @btx), IMP: 32P Backlog support! (individual issues only), FIX: 32P will only do one search occurance, instead of issue-numbered padded searching, IMP: When post-processing, if series is in a Paused status or is in an Ended publishing state with 100% of issues completed - will ignore series for post-processing comparisons, IMP: Dynamic Name matching added as matching algorithims in Weekly Pull Lists, FIX: If series contained a '+', would not be able to scan in files for comparison checks (or during post-processing), FIX: File-checking/Post-Processing was taking the incorrect subdirectory path due to escaped slashes (windows only), FIX: If a series contained more than one/multiple special characters, dynamic naming would be slightly off and cause matching problems in some cases, FIX: When peforming a migration of paths in the db using the locmove option, invalid character references would occur on some machines when moving between OS', IMP: Added status checking of an issue prior to downloading (should fix duplicate downloading of same issue within a specific timeframe), FIX: Fixed some parsing issues when using the ALT_PULL=1 method of the weekly pull list, FIX: Fixed some 500 errors when using reverse proxy (flush Impors, Recheck Files, Manage), FIX: When selecting 'Clear Post-Processed items' from history tab, will now also clear 'Processed' items as well, FIX: When importing series that were successfully scanned for metadata and contained valid ComicID's, would fail to regenerate the dataset, IMP: Added exception catch for ComicVine API limit being reached when performing imports (graceful error), IMP: Added graphical icons to indicate Pause/Loading/Error/Active state in the Manage Comics section
2016-05-12 15:28:28 +00:00
|
|
|
if 'torrents_notify' in alurl and ('torrents_notify_' + str(self.passkey)) not in alurl:
|
2015-07-01 22:52:26 +00:00
|
|
|
notifyname_st = alurl.find('name=')
|
|
|
|
notifyname_en = alurl.find('&', notifyname_st +1)
|
|
|
|
if notifyname_en == -1: notifyname_en = len(alurl)
|
|
|
|
notifyname = alurl[notifyname_st +5:notifyname_en]
|
|
|
|
notifynumber_st = alurl.find('torrents_notify_')
|
|
|
|
notifynumber_en = alurl.find('_', notifynumber_st +17)
|
|
|
|
notifynumber = alurl[notifynumber_st:notifynumber_en]
|
|
|
|
logger.fdebug(self.module + ' [NOTIFICATION: ' + str(notifyname) + '] Notification ID: ' + str(notifynumber))
|
|
|
|
|
|
|
|
#generate the rss-url here
|
IMP: Ability to use 32P session cookies for Auth Login (thnx @btx), IMP: 32P Backlog support! (individual issues only), FIX: 32P will only do one search occurance, instead of issue-numbered padded searching, IMP: When post-processing, if series is in a Paused status or is in an Ended publishing state with 100% of issues completed - will ignore series for post-processing comparisons, IMP: Dynamic Name matching added as matching algorithims in Weekly Pull Lists, FIX: If series contained a '+', would not be able to scan in files for comparison checks (or during post-processing), FIX: File-checking/Post-Processing was taking the incorrect subdirectory path due to escaped slashes (windows only), FIX: If a series contained more than one/multiple special characters, dynamic naming would be slightly off and cause matching problems in some cases, FIX: When peforming a migration of paths in the db using the locmove option, invalid character references would occur on some machines when moving between OS', IMP: Added status checking of an issue prior to downloading (should fix duplicate downloading of same issue within a specific timeframe), FIX: Fixed some parsing issues when using the ALT_PULL=1 method of the weekly pull list, FIX: Fixed some 500 errors when using reverse proxy (flush Impors, Recheck Files, Manage), FIX: When selecting 'Clear Post-Processed items' from history tab, will now also clear 'Processed' items as well, FIX: When importing series that were successfully scanned for metadata and contained valid ComicID's, would fail to regenerate the dataset, IMP: Added exception catch for ComicVine API limit being reached when performing imports (graceful error), IMP: Added graphical icons to indicate Pause/Loading/Error/Active state in the Manage Comics section
2016-05-12 15:28:28 +00:00
|
|
|
feedinfo.append({'feed': notifynumber + '_' + str(self.passkey),
|
2015-07-01 22:52:26 +00:00
|
|
|
'feedname': notifyname,
|
IMP: Ability to use 32P session cookies for Auth Login (thnx @btx), IMP: 32P Backlog support! (individual issues only), FIX: 32P will only do one search occurance, instead of issue-numbered padded searching, IMP: When post-processing, if series is in a Paused status or is in an Ended publishing state with 100% of issues completed - will ignore series for post-processing comparisons, IMP: Dynamic Name matching added as matching algorithims in Weekly Pull Lists, FIX: If series contained a '+', would not be able to scan in files for comparison checks (or during post-processing), FIX: File-checking/Post-Processing was taking the incorrect subdirectory path due to escaped slashes (windows only), FIX: If a series contained more than one/multiple special characters, dynamic naming would be slightly off and cause matching problems in some cases, FIX: When peforming a migration of paths in the db using the locmove option, invalid character references would occur on some machines when moving between OS', IMP: Added status checking of an issue prior to downloading (should fix duplicate downloading of same issue within a specific timeframe), FIX: Fixed some parsing issues when using the ALT_PULL=1 method of the weekly pull list, FIX: Fixed some 500 errors when using reverse proxy (flush Impors, Recheck Files, Manage), FIX: When selecting 'Clear Post-Processed items' from history tab, will now also clear 'Processed' items as well, FIX: When importing series that were successfully scanned for metadata and contained valid ComicID's, would fail to regenerate the dataset, IMP: Added exception catch for ComicVine API limit being reached when performing imports (graceful error), IMP: Added graphical icons to indicate Pause/Loading/Error/Active state in the Manage Comics section
2016-05-12 15:28:28 +00:00
|
|
|
'user': str(self.uid),
|
2015-07-01 22:52:26 +00:00
|
|
|
'auth': auth,
|
IMP: Ability to use 32P session cookies for Auth Login (thnx @btx), IMP: 32P Backlog support! (individual issues only), FIX: 32P will only do one search occurance, instead of issue-numbered padded searching, IMP: When post-processing, if series is in a Paused status or is in an Ended publishing state with 100% of issues completed - will ignore series for post-processing comparisons, IMP: Dynamic Name matching added as matching algorithims in Weekly Pull Lists, FIX: If series contained a '+', would not be able to scan in files for comparison checks (or during post-processing), FIX: File-checking/Post-Processing was taking the incorrect subdirectory path due to escaped slashes (windows only), FIX: If a series contained more than one/multiple special characters, dynamic naming would be slightly off and cause matching problems in some cases, FIX: When peforming a migration of paths in the db using the locmove option, invalid character references would occur on some machines when moving between OS', IMP: Added status checking of an issue prior to downloading (should fix duplicate downloading of same issue within a specific timeframe), FIX: Fixed some parsing issues when using the ALT_PULL=1 method of the weekly pull list, FIX: Fixed some 500 errors when using reverse proxy (flush Impors, Recheck Files, Manage), FIX: When selecting 'Clear Post-Processed items' from history tab, will now also clear 'Processed' items as well, FIX: When importing series that were successfully scanned for metadata and contained valid ComicID's, would fail to regenerate the dataset, IMP: Added exception catch for ComicVine API limit being reached when performing imports (graceful error), IMP: Added graphical icons to indicate Pause/Loading/Error/Active state in the Manage Comics section
2016-05-12 15:28:28 +00:00
|
|
|
'passkey': str(self.passkey),
|
|
|
|
'authkey': str(self.authkey)})
|
|
|
|
|
2015-07-01 22:52:26 +00:00
|
|
|
except (requests.exceptions.Timeout, EnvironmentError):
|
|
|
|
logger.warn('Unable to retrieve information from 32Pages - either it is not responding/is down or something else is happening that is stopping me.')
|
|
|
|
return
|
IMP: 32P legacy mode (rss feed only) and Auth mode (user/pass) available now. With Auth mode, will monitor all of personal notification feeds, IMP: Changed access methods to 32P for retrieving links, FIX:(#1007) Fixed error for one-off downloading via weekly pull list, FIX: (#1004) Fix for post-processing errors (missing files), FIX: (#1001) Extra scripts & pre-scripts will now accept bash scripts, FIX: Annuals were being incorrectly tallied in Have total as well as not having status correctly updated, IMP: Fixed some discrepencies with the provider_order generation line, IMP: Made some additional adjustments to the file-checking to account for some additional variations in naming, FIX: Fixed traceback errors when folder monitor was running and wouldn't be able to complete the post-processing properly, FIX: Cleanup code and some other small fixes....
2015-05-09 02:39:28 +00:00
|
|
|
|
|
|
|
#set the keys here that will be used to download.
|
2015-07-01 22:52:26 +00:00
|
|
|
try:
|
IMP: Ability to use 32P session cookies for Auth Login (thnx @btx), IMP: 32P Backlog support! (individual issues only), FIX: 32P will only do one search occurance, instead of issue-numbered padded searching, IMP: When post-processing, if series is in a Paused status or is in an Ended publishing state with 100% of issues completed - will ignore series for post-processing comparisons, IMP: Dynamic Name matching added as matching algorithims in Weekly Pull Lists, FIX: If series contained a '+', would not be able to scan in files for comparison checks (or during post-processing), FIX: File-checking/Post-Processing was taking the incorrect subdirectory path due to escaped slashes (windows only), FIX: If a series contained more than one/multiple special characters, dynamic naming would be slightly off and cause matching problems in some cases, FIX: When peforming a migration of paths in the db using the locmove option, invalid character references would occur on some machines when moving between OS', IMP: Added status checking of an issue prior to downloading (should fix duplicate downloading of same issue within a specific timeframe), FIX: Fixed some parsing issues when using the ALT_PULL=1 method of the weekly pull list, FIX: Fixed some 500 errors when using reverse proxy (flush Impors, Recheck Files, Manage), FIX: When selecting 'Clear Post-Processed items' from history tab, will now also clear 'Processed' items as well, FIX: When importing series that were successfully scanned for metadata and contained valid ComicID's, would fail to regenerate the dataset, IMP: Added exception catch for ComicVine API limit being reached when performing imports (graceful error), IMP: Added graphical icons to indicate Pause/Loading/Error/Active state in the Manage Comics section
2016-05-12 15:28:28 +00:00
|
|
|
mylar.PASSKEY_32P = str(self.passkey)
|
|
|
|
mylar.AUTHKEY_32P = str(self.authkey) # probably not needed here.
|
2015-07-01 22:52:26 +00:00
|
|
|
mylar.KEYS_32P = {}
|
IMP: Ability to use 32P session cookies for Auth Login (thnx @btx), IMP: 32P Backlog support! (individual issues only), FIX: 32P will only do one search occurance, instead of issue-numbered padded searching, IMP: When post-processing, if series is in a Paused status or is in an Ended publishing state with 100% of issues completed - will ignore series for post-processing comparisons, IMP: Dynamic Name matching added as matching algorithims in Weekly Pull Lists, FIX: If series contained a '+', would not be able to scan in files for comparison checks (or during post-processing), FIX: File-checking/Post-Processing was taking the incorrect subdirectory path due to escaped slashes (windows only), FIX: If a series contained more than one/multiple special characters, dynamic naming would be slightly off and cause matching problems in some cases, FIX: When peforming a migration of paths in the db using the locmove option, invalid character references would occur on some machines when moving between OS', IMP: Added status checking of an issue prior to downloading (should fix duplicate downloading of same issue within a specific timeframe), FIX: Fixed some parsing issues when using the ALT_PULL=1 method of the weekly pull list, FIX: Fixed some 500 errors when using reverse proxy (flush Impors, Recheck Files, Manage), FIX: When selecting 'Clear Post-Processed items' from history tab, will now also clear 'Processed' items as well, FIX: When importing series that were successfully scanned for metadata and contained valid ComicID's, would fail to regenerate the dataset, IMP: Added exception catch for ComicVine API limit being reached when performing imports (graceful error), IMP: Added graphical icons to indicate Pause/Loading/Error/Active state in the Manage Comics section
2016-05-12 15:28:28 +00:00
|
|
|
mylar.KEYS_32P = {"user": str(self.uid),
|
2015-07-01 22:52:26 +00:00
|
|
|
"auth": auth,
|
IMP: Ability to use 32P session cookies for Auth Login (thnx @btx), IMP: 32P Backlog support! (individual issues only), FIX: 32P will only do one search occurance, instead of issue-numbered padded searching, IMP: When post-processing, if series is in a Paused status or is in an Ended publishing state with 100% of issues completed - will ignore series for post-processing comparisons, IMP: Dynamic Name matching added as matching algorithims in Weekly Pull Lists, FIX: If series contained a '+', would not be able to scan in files for comparison checks (or during post-processing), FIX: File-checking/Post-Processing was taking the incorrect subdirectory path due to escaped slashes (windows only), FIX: If a series contained more than one/multiple special characters, dynamic naming would be slightly off and cause matching problems in some cases, FIX: When peforming a migration of paths in the db using the locmove option, invalid character references would occur on some machines when moving between OS', IMP: Added status checking of an issue prior to downloading (should fix duplicate downloading of same issue within a specific timeframe), FIX: Fixed some parsing issues when using the ALT_PULL=1 method of the weekly pull list, FIX: Fixed some 500 errors when using reverse proxy (flush Impors, Recheck Files, Manage), FIX: When selecting 'Clear Post-Processed items' from history tab, will now also clear 'Processed' items as well, FIX: When importing series that were successfully scanned for metadata and contained valid ComicID's, would fail to regenerate the dataset, IMP: Added exception catch for ComicVine API limit being reached when performing imports (graceful error), IMP: Added graphical icons to indicate Pause/Loading/Error/Active state in the Manage Comics section
2016-05-12 15:28:28 +00:00
|
|
|
"passkey": str(self.passkey),
|
|
|
|
"authkey": str(self.authkey)}
|
2015-07-01 22:52:26 +00:00
|
|
|
except NameError:
|
|
|
|
logger.warn('Unable to retrieve information from 32Pages - either it is not responding/is down or something else is happening that is stopping me.')
|
|
|
|
return
|
|
|
|
|
IMP: 32P legacy mode (rss feed only) and Auth mode (user/pass) available now. With Auth mode, will monitor all of personal notification feeds, IMP: Changed access methods to 32P for retrieving links, FIX:(#1007) Fixed error for one-off downloading via weekly pull list, FIX: (#1004) Fix for post-processing errors (missing files), FIX: (#1001) Extra scripts & pre-scripts will now accept bash scripts, FIX: Annuals were being incorrectly tallied in Have total as well as not having status correctly updated, IMP: Fixed some discrepencies with the provider_order generation line, IMP: Made some additional adjustments to the file-checking to account for some additional variations in naming, FIX: Fixed traceback errors when folder monitor was running and wouldn't be able to complete the post-processing properly, FIX: Cleanup code and some other small fixes....
2015-05-09 02:39:28 +00:00
|
|
|
if self.reauthenticate:
|
|
|
|
return
|
|
|
|
else:
|
|
|
|
mylar.FEEDINFO_32P = feedinfo
|
|
|
|
return feedinfo
|
IMP: Ability to use 32P session cookies for Auth Login (thnx @btx), IMP: 32P Backlog support! (individual issues only), FIX: 32P will only do one search occurance, instead of issue-numbered padded searching, IMP: When post-processing, if series is in a Paused status or is in an Ended publishing state with 100% of issues completed - will ignore series for post-processing comparisons, IMP: Dynamic Name matching added as matching algorithims in Weekly Pull Lists, FIX: If series contained a '+', would not be able to scan in files for comparison checks (or during post-processing), FIX: File-checking/Post-Processing was taking the incorrect subdirectory path due to escaped slashes (windows only), FIX: If a series contained more than one/multiple special characters, dynamic naming would be slightly off and cause matching problems in some cases, FIX: When peforming a migration of paths in the db using the locmove option, invalid character references would occur on some machines when moving between OS', IMP: Added status checking of an issue prior to downloading (should fix duplicate downloading of same issue within a specific timeframe), FIX: Fixed some parsing issues when using the ALT_PULL=1 method of the weekly pull list, FIX: Fixed some 500 errors when using reverse proxy (flush Impors, Recheck Files, Manage), FIX: When selecting 'Clear Post-Processed items' from history tab, will now also clear 'Processed' items as well, FIX: When importing series that were successfully scanned for metadata and contained valid ComicID's, would fail to regenerate the dataset, IMP: Added exception catch for ComicVine API limit being reached when performing imports (graceful error), IMP: Added graphical icons to indicate Pause/Loading/Error/Active state in the Manage Comics section
2016-05-12 15:28:28 +00:00
|
|
|
|
|
|
|
def searchit(self):
|
|
|
|
with requests.Session() as s:
|
|
|
|
#self.searchterm is a tuple containing series name, issue number and volume.
|
|
|
|
series_search = self.searchterm['series']
|
|
|
|
issue_search = self.searchterm['issue']
|
|
|
|
volume_search = self.searchterm['volume']
|
|
|
|
#generate the dynamic name of the series here so we can match it up
|
|
|
|
as_d = filechecker.FileChecker()
|
|
|
|
as_dinfo = as_d.dynamic_replace(series_search)
|
|
|
|
mod_series = as_dinfo['mod_seriesname']
|
|
|
|
|
|
|
|
if '/' in series_search:
|
|
|
|
series_search = series_search[:series_search.find('/')]
|
|
|
|
if ':' in series_search:
|
|
|
|
series_search = series_search[:series_search.find(':')]
|
|
|
|
|
|
|
|
url = 'https://32pag.es/torrents.php' #?action=serieslist&filter=' + series_search #&filter=F
|
|
|
|
params = {'action': 'serieslist', 'filter': series_search}
|
|
|
|
s.headers = self.headers
|
2016-05-19 15:20:13 +00:00
|
|
|
cj = LWPCookieJar(os.path.join(mylar.CACHE_DIR, ".32p_cookies.dat"))
|
IMP: Ability to use 32P session cookies for Auth Login (thnx @btx), IMP: 32P Backlog support! (individual issues only), FIX: 32P will only do one search occurance, instead of issue-numbered padded searching, IMP: When post-processing, if series is in a Paused status or is in an Ended publishing state with 100% of issues completed - will ignore series for post-processing comparisons, IMP: Dynamic Name matching added as matching algorithims in Weekly Pull Lists, FIX: If series contained a '+', would not be able to scan in files for comparison checks (or during post-processing), FIX: File-checking/Post-Processing was taking the incorrect subdirectory path due to escaped slashes (windows only), FIX: If a series contained more than one/multiple special characters, dynamic naming would be slightly off and cause matching problems in some cases, FIX: When peforming a migration of paths in the db using the locmove option, invalid character references would occur on some machines when moving between OS', IMP: Added status checking of an issue prior to downloading (should fix duplicate downloading of same issue within a specific timeframe), FIX: Fixed some parsing issues when using the ALT_PULL=1 method of the weekly pull list, FIX: Fixed some 500 errors when using reverse proxy (flush Impors, Recheck Files, Manage), FIX: When selecting 'Clear Post-Processed items' from history tab, will now also clear 'Processed' items as well, FIX: When importing series that were successfully scanned for metadata and contained valid ComicID's, would fail to regenerate the dataset, IMP: Added exception catch for ComicVine API limit being reached when performing imports (graceful error), IMP: Added graphical icons to indicate Pause/Loading/Error/Active state in the Manage Comics section
2016-05-12 15:28:28 +00:00
|
|
|
cj.load()
|
|
|
|
s.cookies = cj
|
|
|
|
time.sleep(1) #just to make sure we don't hammer, 1s pause.
|
|
|
|
t = s.get(url, params=params, verify=True)
|
|
|
|
soup = BeautifulSoup(t.content)
|
|
|
|
results = soup.find_all("a", {"class":"object-qtip"},{"data-type":"torrentgroup"})
|
|
|
|
|
|
|
|
data = []
|
|
|
|
|
|
|
|
for r in results:
|
|
|
|
torrentid = r['data-id']
|
|
|
|
torrentname = r.findNext(text=True)
|
|
|
|
torrentname = torrentname.strip()
|
|
|
|
as_d = filechecker.FileChecker()
|
|
|
|
as_dinfo = as_d.dynamic_replace(torrentname)
|
|
|
|
seriesresult = as_dinfo['mod_seriesname']
|
|
|
|
logger.info('searchresult: ' + seriesresult + ' --- ' + mod_series)
|
|
|
|
if seriesresult == mod_series:
|
|
|
|
logger.info('[MATCH] ' + torrentname + ' [' + str(torrentid) + ']')
|
|
|
|
data.append({"id": torrentid,
|
|
|
|
"series": torrentname})
|
|
|
|
|
|
|
|
logger.info(str(len(data)) + ' series listed for searching that match.')
|
|
|
|
|
|
|
|
if len(data) == 1:
|
|
|
|
logger.info(str(len(data)) + ' series match the title being search for')
|
|
|
|
payload = {'action': 'groupsearch',
|
|
|
|
'id': data[0]['id'],
|
|
|
|
'issue': issue_search}
|
|
|
|
#in order to match up against 0-day stuff, volume has to be none at this point
|
|
|
|
#when doing other searches tho, this should be allowed to go through
|
|
|
|
#if all([volume_search != 'None', volume_search is not None]):
|
|
|
|
# payload.update({'volume': re.sub('v', '', volume_search).strip()})
|
|
|
|
|
|
|
|
logger.info('payload: ' + str(payload))
|
|
|
|
url = 'https://32pag.es/ajax.php'
|
|
|
|
|
|
|
|
time.sleep(1) #just to make sure we don't hammer, 1s pause.
|
|
|
|
d = s.get(url, params=payload, verify=True)
|
|
|
|
|
|
|
|
results32p = []
|
|
|
|
results = {}
|
|
|
|
try:
|
|
|
|
searchResults = d.json()
|
|
|
|
except:
|
|
|
|
searchResults = d.text
|
|
|
|
if searchResults['status'] == 'success' and searchResults['count'] > 0:
|
|
|
|
logger.info('successfully retrieved ' + str(searchResults['count']) + ' search results.')
|
|
|
|
for a in searchResults['details']:
|
|
|
|
results32p.append({'link': a['id'],
|
|
|
|
'title': self.searchterm['series'] + ' v' + a['volume'] + ' #' + a['issues'],
|
|
|
|
'filesize': a['size'],
|
|
|
|
'pack': a['pack'],
|
|
|
|
'format': a['format'],
|
|
|
|
'language': a['language'],
|
|
|
|
'seeders': a['seeders'],
|
|
|
|
'leechers': a['leechers'],
|
|
|
|
'scanner': a['scanner'],
|
|
|
|
'pubdate': datetime.datetime.fromtimestamp(float(a['upload_time'])).strftime('%c')})
|
|
|
|
results['entries'] = results32p
|
|
|
|
else:
|
|
|
|
results = 'no results'
|
|
|
|
else:
|
|
|
|
results = 'no results'
|
|
|
|
|
|
|
|
return results
|
|
|
|
|
|
|
|
class LoginSession(object):
|
|
|
|
def __init__(self, un, pw, session_path=None):
|
|
|
|
'''
|
|
|
|
Params:
|
|
|
|
un: account username (required)
|
|
|
|
pw: account password (required)
|
|
|
|
session_path: the path to the actual file you want to persist your cookies in
|
|
|
|
If blank, saves to $HOME/.32p_cookies.dat
|
|
|
|
|
|
|
|
'''
|
|
|
|
self.module = '[32P-AUTHENTICATION]'
|
|
|
|
self.ses = requests.Session()
|
2016-05-19 15:20:13 +00:00
|
|
|
self.session_path = session_path if session_path is not None else os.path.join(mylar.CACHE_DIR, ".32p_cookies.dat")
|
IMP: Ability to use 32P session cookies for Auth Login (thnx @btx), IMP: 32P Backlog support! (individual issues only), FIX: 32P will only do one search occurance, instead of issue-numbered padded searching, IMP: When post-processing, if series is in a Paused status or is in an Ended publishing state with 100% of issues completed - will ignore series for post-processing comparisons, IMP: Dynamic Name matching added as matching algorithims in Weekly Pull Lists, FIX: If series contained a '+', would not be able to scan in files for comparison checks (or during post-processing), FIX: File-checking/Post-Processing was taking the incorrect subdirectory path due to escaped slashes (windows only), FIX: If a series contained more than one/multiple special characters, dynamic naming would be slightly off and cause matching problems in some cases, FIX: When peforming a migration of paths in the db using the locmove option, invalid character references would occur on some machines when moving between OS', IMP: Added status checking of an issue prior to downloading (should fix duplicate downloading of same issue within a specific timeframe), FIX: Fixed some parsing issues when using the ALT_PULL=1 method of the weekly pull list, FIX: Fixed some 500 errors when using reverse proxy (flush Impors, Recheck Files, Manage), FIX: When selecting 'Clear Post-Processed items' from history tab, will now also clear 'Processed' items as well, FIX: When importing series that were successfully scanned for metadata and contained valid ComicID's, would fail to regenerate the dataset, IMP: Added exception catch for ComicVine API limit being reached when performing imports (graceful error), IMP: Added graphical icons to indicate Pause/Loading/Error/Active state in the Manage Comics section
2016-05-12 15:28:28 +00:00
|
|
|
self.ses.cookies = LWPCookieJar(self.session_path)
|
|
|
|
if not os.path.exists(self.session_path):
|
|
|
|
logger.fdebug(self.module + ' Session cookie does not exist. Signing in and Creating.')
|
|
|
|
self.ses.cookies.save()
|
|
|
|
else:
|
|
|
|
logger.fdebug(self.module + ' Session cookie found. Attempting to load...')
|
|
|
|
self.ses.cookies.load(ignore_discard=True)
|
|
|
|
self.un = un
|
|
|
|
self.pw = pw
|
|
|
|
self.authkey = None
|
|
|
|
self.passkey = None
|
|
|
|
self.uid = None
|
|
|
|
|
|
|
|
def cookie_exists(self, name):
|
|
|
|
'''
|
|
|
|
Checks if cookie <name> exists in self.ses.cookies
|
|
|
|
Beware - this doesn't match domain, so only use this method on one domain
|
|
|
|
'''
|
|
|
|
|
|
|
|
for ci in self.ses.cookies:
|
|
|
|
if (ci.name == name):
|
|
|
|
return True
|
|
|
|
return False
|
|
|
|
|
|
|
|
def cookie_value(self, name, default=None):
|
|
|
|
'''
|
|
|
|
Returns the value of the cookie name, returning default if it doesn't exist.
|
|
|
|
Beware - this doesn't match domain too, so only use this method on one domain
|
|
|
|
'''
|
|
|
|
for ci in self.ses.cookies:
|
|
|
|
if (ci.name == name):
|
|
|
|
return ci.value
|
|
|
|
|
|
|
|
return default
|
|
|
|
|
|
|
|
def valid_skey_attempt(self, skey):
|
|
|
|
'''
|
|
|
|
Not generally the proper method to call - call test_key_valid()
|
|
|
|
instead - which calls this method.
|
|
|
|
|
|
|
|
Attempts to fetch data via an ajax method that will fail if not
|
|
|
|
authorized. The parameter skey should be set to the string
|
|
|
|
value of the cookie named session.
|
|
|
|
|
|
|
|
Returns: True on success, False on failure. Side Effects: Sets
|
|
|
|
self.uid, self,authkey and self.passkey
|
|
|
|
'''
|
|
|
|
|
|
|
|
u = '''https://32pag.es/ajax.php'''
|
|
|
|
params = {'action': 'index'}
|
|
|
|
testcookie = dict(session=skey)
|
|
|
|
|
|
|
|
try:
|
|
|
|
r = self.ses.get(u, params=params, timeout=60, allow_redirects=False, cookies=testcookie)
|
|
|
|
except Exception as e:
|
|
|
|
print "Got an exception trying to GET from to: %s", u
|
|
|
|
self.error = {'status':'error', 'message':'exception trying to retrieve site'}
|
|
|
|
return False
|
|
|
|
|
|
|
|
if r.status_code != 200:
|
|
|
|
if r.status_code == 302:
|
|
|
|
newloc = r.headers.get('location', '')
|
|
|
|
print "Got redirect from the POST-ajax action=login GET: %s", newloc
|
|
|
|
self.error = {'status':'redirect-error', 'message':'got redirect from POST-ajax login action : ' + newloc}
|
|
|
|
else:
|
|
|
|
print "Got bad status code in the POST-ajax action=login GET: %d", r.status_code
|
|
|
|
self.error = {'status':'bad status code', 'message':'bad status code received in the POST-ajax login action :' + str(r.status_code)}
|
|
|
|
return False
|
|
|
|
|
|
|
|
try:
|
|
|
|
j = r.json()
|
|
|
|
except:
|
|
|
|
print "Error - response from session-based skey check was not JSON: %s", r.text
|
|
|
|
return False
|
|
|
|
|
|
|
|
self.uid = j['response']['id']
|
|
|
|
self.authkey = j['response']['authkey']
|
|
|
|
self.passkey = pk = j['response']['passkey']
|
|
|
|
return True
|
|
|
|
|
|
|
|
def valid_login_attempt(self, un, pw):
|
|
|
|
'''
|
|
|
|
Does the actual POST to the login.php method (using the ajax parameter, which is far more reliable
|
|
|
|
than HTML parsing.
|
|
|
|
|
|
|
|
Input: un: The username (usually would be self.un, but that's not a requirement
|
|
|
|
pw: The password (usually self.pw but not a requirement)
|
|
|
|
|
|
|
|
Note: The underlying self.ses object will handle setting the session cookie from a valid login,
|
|
|
|
but you'll need to call the save method if your cookies are being persisted.
|
|
|
|
|
|
|
|
Returns: True (success) False (failure)
|
|
|
|
|
|
|
|
'''
|
|
|
|
|
|
|
|
postdata = {'username': un, 'password': pw, 'keeplogged': 1}
|
|
|
|
u = 'https://32pag.es/login.php?ajax=1'
|
|
|
|
|
|
|
|
try:
|
|
|
|
r = self.ses.post(u, data=postdata, timeout=60, allow_redirects=False)
|
|
|
|
except Exception as e:
|
|
|
|
print "Got an exception when trying to login to %s POST", u
|
|
|
|
self.error = {'status':'exception', 'message':'Exception when trying to login'}
|
|
|
|
return False
|
|
|
|
|
|
|
|
if r.status_code != 200:
|
|
|
|
print "Got bad status code from login POST: %d\n%s\n%s", r.status_code, r.text, r.headers
|
|
|
|
self.error = {'status':'Bad Status code', 'message':(r.status_code, r.text, r.headers)}
|
|
|
|
return False
|
|
|
|
|
|
|
|
try:
|
|
|
|
d = r.json()
|
|
|
|
except:
|
|
|
|
print "The data returned by the login page was not JSON: %s", r.text
|
|
|
|
self.error = {'status':'JSON not returned', 'message':r.text}
|
|
|
|
return False
|
|
|
|
|
|
|
|
if d['status'] == 'success':
|
|
|
|
return True
|
|
|
|
|
|
|
|
print "Got unexpected status result: %s", d
|
|
|
|
self.error = d
|
|
|
|
return False
|
|
|
|
|
|
|
|
def test_skey_valid(self, skey=None):
|
|
|
|
'''
|
|
|
|
You should call this method to test if the specified session key
|
|
|
|
(skey) is still valid. If skey is left out or None, it
|
|
|
|
automatically gets the value of the session cookie currently
|
|
|
|
set.
|
|
|
|
|
|
|
|
|
|
|
|
Returns: True (success) False (failure)
|
|
|
|
Side effects: Saves the cookies file.
|
|
|
|
|
|
|
|
'''
|
|
|
|
|
|
|
|
if skey is None:
|
|
|
|
skey = self.cookie_value('session', '')
|
|
|
|
|
|
|
|
if skey is None or skey == '':
|
|
|
|
return False
|
|
|
|
|
|
|
|
if (self.valid_skey_attempt(skey)):
|
|
|
|
self.ses.cookies.save(ignore_discard=True)
|
|
|
|
return True
|
|
|
|
|
|
|
|
self.ses.cookies.save(ignore_discard=true)
|
|
|
|
return False
|
|
|
|
|
|
|
|
def test_login(self):
|
|
|
|
'''
|
|
|
|
This is the method to call if you JUST want to login using self.un & self.pw
|
|
|
|
|
|
|
|
Note that this will generate a new session on 32pag.es every time you login successfully!
|
|
|
|
This is why the "keeplogged" option is only for when you persist cookies to disk.
|
|
|
|
|
|
|
|
Note that after a successful login, it will test the session key, which has the side effect of
|
|
|
|
getting the authkey,passkey & uid
|
|
|
|
|
|
|
|
Returns: True (login success) False (login failure)
|
|
|
|
Side Effects: On success: Sets the authkey, uid, passkey and saves the cookies to disk
|
|
|
|
(on failure): clears the cookies and saves that to disk.
|
|
|
|
'''
|
|
|
|
if (self.valid_login_attempt(self.un, self.pw)):
|
|
|
|
if self.cookie_exists('session'):
|
|
|
|
self.ses.cookies.save(ignore_discard=True)
|
|
|
|
if (not self.test_skey_valid()):
|
|
|
|
console.error("Bad error: The attempt to get your attributes after successful login failed!")
|
|
|
|
self.error = {'status': 'Bad error', 'message': 'Attempt to get attributes after successful login failed.'}
|
|
|
|
return False
|
|
|
|
return True
|
|
|
|
|
|
|
|
print "Missing session cookie after successful login: %s", self.ses.cookies
|
|
|
|
self.ses.cookies.clear()
|
|
|
|
self.ses.cookies.save()
|
|
|
|
return False
|
|
|
|
|
|
|
|
|
|
|
|
def login(self):
|
|
|
|
'''
|
|
|
|
This is generally the only method you'll want to call, as it handles testing test_skey_valid() before
|
|
|
|
trying test_login().
|
|
|
|
|
|
|
|
Returns: True (success) / False (failure)
|
|
|
|
Side effects: Methods called will handle saving the cookies to disk, and setting
|
|
|
|
self.authkey, self.passkey, and self.uid
|
|
|
|
'''
|
|
|
|
if (self.test_skey_valid()):
|
|
|
|
logger.fdebug(self.module + ' Session key-based login was good.')
|
|
|
|
self.method = 'Session Cookie retrieved OK.'
|
|
|
|
return True
|
|
|
|
|
|
|
|
if (self.test_login()):
|
|
|
|
logger.fdebug(self.module + ' Credential-based login was good.')
|
|
|
|
self.method = 'Session Cookie retrieved OK.'
|
|
|
|
return True
|
|
|
|
|
|
|
|
logger.warn(self.module + ' Both session key and credential-based logins failed.')
|
|
|
|
self.method = 'Failed to retrieve Session Cookie.'
|
|
|
|
return False
|
|
|
|
|
|
|
|
|
|
|
|
#if __name__ == '__main__':
|
|
|
|
# ab = DoIt()
|
|
|
|
# c = ab.loadit()
|
|
|
|
|