evanferrao
/
dump-mirrorbot
mirror of https://github.com/Box-boi/dump-mirrorbot.git
1
0
Fork 0
Code Issues Actions Packages Projects Releases Wiki Activity
466 Commits 2 Branches 21 Tags 2 MiB
Commit Graph

236 Commits

Author SHA1 Message Date
lzzy12 45f75fb7a6 Fix crash if Mega API key is not provided 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-06-19 15:34:18 +05:30
jaskaranSM c6a430a44d Handle more exceptions 2020-06-18 12:59:52 +05:30
jaskaranSM 3720eacf13 Optimize Clone implementation 2020-06-18 12:57:31 +05:30
lzzy12 c4c7575000 Generate our own gid for mega mirrors 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-06-05 14:01:36 +05:30
lzzy12 c791b39785 Revert "Improved upload progress implementation" 
This reverts commit 0f7b79f7a4.
 2020-06-05 13:20:40 +05:30
lzzy12 3217b67df0 Fixup for meganz login 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-06-05 13:15:00 +05:30
lzzy12 8bd01276be Add ability to login to mega account for premium accounts 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-06-03 11:32:09 +05:30
jaskaranSM 578eeb4004 Implement Mega.nz mirror support. 
- Co-authored by - lzzy12 (github.com/lzzy12)
 2020-05-28 15:17:19 +05:30
Nageen Chand b6b8a416e9
add ability to unzip files (#93) 
handle unsupported request

use shell utils to extract archives

use shell utils to extract archives

update mirror calls for unzipmirror

fix extract location of files

install deps

handling if archive is not extracted correctly

typo fix

implement Extract Status

handle filenames with spaces

handle special character
 2020-05-28 00:26:22 +05:30
lzzy12 f3a3ed05f4 Fixed cancel by gid after f4290210b1 
- This fixes #77

Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-05-21 17:25:11 +05:30
lzzy12 0f7b79f7a4 Improved upload progress implementation 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-05-21 13:08:50 +05:30
jaskaranSM cdefe12348 Implement Centralized Aria Websocket Listener 2020-05-15 13:34:09 +05:30
lzzy12 f4290210b1 Implemented mirror ownership 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-05-14 19:02:33 +05:30
jaskaranSM 1670b619ae Fix deadlock 2020-05-12 14:27:24 +05:30
jaskaranSM a196ce5b93 Handle TG exceptions 2020-05-12 13:56:57 +05:30
jaskaranSM c4e81d5ab1 start Aria2 Websocket listener before adding download in queue. 2020-05-12 13:50:16 +05:30
lzzy12 d3938784ac Remove broken direct download link extracters 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-05-09 00:10:09 +05:30
lzzy12 1d8c8ce954 restart: Clean downloads before restarting 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-05-09 00:05:35 +05:30
jaskaranSM 315108e7d1 patch regex for more links 2020-05-05 10:51:55 +05:30
jaskaranSM 4d964b6e29 use regex to extract fileId from GDrive links 2020-05-04 12:25:40 +05:30
Youssif Shaaban Alsager 63a67d7a14
Add restart command (#53) 
* Add restart command

This is handy when you need to restart the bot when something goes wrong.

Currently, the bot restarts but doesn't send a message after restarting
is done, feel free to improve this. It'll need something like storing
bot status locally then on bot start, if restart status exists send
a message.

Signed-off-by: yshalsager <ysh-alsager@hotmail.com>

* restart: switch to context based callbacks

* restart: restrict the command to owner only

Co-authored-by: Shivam Jha <jhashivam2020@gmail.com>
 2020-04-21 15:33:05 +05:30
jaskaranSM 78b60d6715
import threading lib 2020-04-19 14:58:35 +05:30
jaskaranSM 8624e0174b Fix IO Block on Large Playlists 2020-04-18 14:32:01 +05:30
jaskaranSM 120ca7538d clone: remove forward slash 2020-04-16 12:39:46 +05:30
jaskaranSM e5bb8e55ef Clone fix and added index link support 2020-04-16 12:38:12 +05:30
jaskaranSM d7bc8f3c9a switch to context based callbacks 2020-04-16 12:26:28 +05:30
lzzy12 c6b4c542b7 fixup after e0f4c4347c 
- Also fixed copying of netrc file in the container

Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-04-15 14:47:33 +05:30
lzzy12 e0f4c4347c Separate out youtube-dl command 
- /watch [youtube-dl supported link]

Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-04-14 23:20:52 +05:30
lzzy12 e5364f96e1 Revert "Handle rare bug when download is not deleted from download_dict on error" 
This reverts commit 136924ffd4.

Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-04-13 13:32:39 +05:30
lzzy12 76e3db1946 [Hotfix] Fix bug where the link have 302 request are treated as torrents 
Download.is_torrent introduced in aria2p v0.9.0
https://github.com/pawamoy/aria2p/issues/53

Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-04-08 19:29:59 +05:30
Arsalan ed60afa9d6
fix typo in INDEX_URL (#54) 2020-04-07 20:36:04 +05:30
lzzy12 b2f2ee4335 Fix aria downloads cancellation after b58eb87b11 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-04-06 13:14:40 +05:30
lzzy12 47dd3c2b6f gdriveTools: Handle unhandled exception in clone method 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-04-05 14:12:13 +05:30
lzzy12 0536472e28 Expose message object from Status objects 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-04-05 13:28:35 +05:30
lzzy12 b58eb87b11 Improvements in cancelling mirrors 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-04-05 12:34:08 +05:30
jaskaranSM 708e6df87a youtube-dl: use best format 2020-04-01 12:07:29 +05:30
lzzy12 50ab740219 Correct filename if ffmpeg conversion is invoked by yt-dl 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-31 17:56:57 +05:30
lzzy12 a54fc42e1e Handle some exceptions 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-31 15:42:13 +05:30
lzzy12 e2c5c8beaf Log youtube-dl warnings and errors in bot log 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-31 11:15:09 +05:30
lzzy12 85d4005c16 Use netrc by default for youtube-dl 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-31 10:50:53 +05:30
lzzy12 9fa41f68c1 Some minor Fix-ups 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-30 18:16:54 +05:30
lzzy12 136924ffd4 Handle rare bug when download is not deleted from download_dict on error 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-30 17:46:56 +05:30
jaskaranSM 52e8fbec11 Fixed direct link torrent mirroring 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-30 17:05:55 +05:30
lzzy12 0ec499ff4a Handle weird path naming of youtube_dl 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-30 16:36:12 +05:30
lzzy12 141b13b52d Make ydl object local to the methods 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-30 14:55:29 +05:30
lzzy12 3bf9e0ea31 Use default quality provided by youtube-dl for now 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-28 20:14:43 +05:30
jaskaranSM 6262b26fe5 youtube-dl: fixed downloading from sites that provide encrypted content (m3u8) 2020-03-28 18:44:02 +05:30
lzzy12 7c304d449a Fix uploading for YT-DL 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-28 14:38:08 +05:30
lzzy12 9057d3fc4e Fixed yt-dl support 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-28 13:47:19 +05:30
jaskaranSM 3a06458609 Added Youtube-dl support 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-27 20:04:13 +05:30
lzzy12 2254444378 Handle KeyError exception in Telegram download error 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-27 16:00:37 +05:30
lzzy12 0de6915222 Remove redundant exception classes and fix existing ones 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-27 14:35:51 +05:30
Shivam Jha 6d07c3c8c4
Stop trying to generate sourceforge direct links 2020-03-26 16:02:36 +05:30
jaskaranSM be9c64a9aa Use Service Accounts for Cloning if availble. 
- increase default timeout for network socket interface

Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-25 18:00:33 +05:30
jaskaranSM 08a71e4bd4 TeamDrive List fix 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-24 18:57:10 +05:30
lzzy12 6ca2f479d5 Fix up errors after porting service accounts from develop 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-24 16:30:03 +05:30
lzzy12 25856bfac0 gdrive: Stop spamming logs at INFO log level 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-24 15:49:04 +05:30
jaskaranSM 8d1bf5aa92 recursively try again instead of handling switch with backoff 
- Handling service account switching with exponential backoff implementation limits the number of service accounts to be used. recursively retrying again after switch doesnt effect backoff.

Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-24 15:41:59 +05:30
jaskaranSM 6cd59d5a15 Cleanup drive_list and minor fixes 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-24 15:39:57 +05:30
jaskaranSM 681b8bd47d Added missing property 2020-03-24 15:07:45 +05:30
jaskaranSM 2bbfca0496 Refactor service account implementation 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-24 15:07:23 +05:30
lzzy12 7ad7d9388a Add support for service account 
Adding support service accounts

Signed-off-by: lzzy12 <jhashivam2020@gmail.com>

Added scripts and docs for generating service accounts

Signed-off-by: lzzy12 <jhashivam2020@gmail.com>

gen_sa_accounts: Save credentials with indexed file name

Signed-off-by: lzzy12 <jhashivam2020@gmail.com>

gdriveTools: Avoid using oauth2 library for service accounts

oauth2 library is deprecated

Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-24 15:04:24 +05:30
jaskaranSM 997e572b69 Enable TeamDrive support for clone 2020-03-24 14:17:50 +05:30
lzzy12 94b6be4c03 Add licensing and authorship information for direct_link_generator.py 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-23 15:41:53 +05:30
lzzy12 f50981b3a2 Implement cancel downloads of tg mirrors 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-23 15:21:54 +05:30
lzzy12 9369c7dc98 Allow mirroring tg file again once file download is complete 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-23 14:49:57 +05:30
lzzy12 7d74cf29d8 Fixed support for mirror of video and audio telegram files 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-22 16:30:57 +05:30
lzzy12 d6a513e77b Fix typo in direct_link_generator.py 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-22 15:48:14 +05:30
lzzy12 b4b0dcd3f8 Improve logic for parsing id from gdrive url 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-22 15:05:12 +05:30
lzzy12 49fe46bad3 Add gid method in Status class for download classes 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-22 15:05:12 +05:30
jaskaranSM a44d0da9f6 Implement cancellation of downloads by gid 2020-03-22 15:05:11 +05:30
jaskaranSM 6b60ca4491 Squash of clone command 
Implement clone feature
- clones a public/driveAccountAccessible Drive Link to bot G-Drive

use BotCommands for clone command trigger
 2020-03-22 15:05:11 +05:30
lzzy12 94ccdcbeef Add support for mirror of telegram files 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>

Add a script to generate string session for user

Signed-off-by: lzzy12 <jhashivam2020@gmail.com>

Some fix ups

Signed-off-by: lzzy12 <jhashivam2020@gmail.com>

Fix telegram download

Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-22 15:04:31 +05:30
lzzy12 898977df97 Use proper url-encoding methods to replace special characters from url 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-20 17:03:33 +05:30
jaskaranSM afc2808acc Rewrite tar function 
- shutil changes pwd while archiving. as some functions of bot require relative paths, it is neccesary to keep pwd as bot root dir.

Signed-off-by: lzzy12 <jhashivam2020@gmail.com>

# Conflicts:
#	bot/helper/ext_utils/fs_utils.py
 2020-03-19 15:43:04 +05:30
lzzy12 7ba7bc92ac Add support for Direct Download generation 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-03-19 14:22:33 +05:30
lzzy12 2fddd06178 Fixed upload of empty files for Team Drive 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-02-05 02:49:54 -08:00
jaskaranSM 8cbe2c29e4 Added peers and seeders count for bittorent downloads 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-02-05 02:45:38 -08:00
Shivam Jha 96e623ef57
UploadStatus: Handle exception ZeroDivisionError 2020-02-05 01:52:43 -08:00
jaskaranSM 05468196e3
Fix Status Message 2020-01-20 09:19:49 +05:30
Shivam Jha 5c1a9c71b7
gdriveTools: Hardcode interval for updating progress 2020-01-17 01:34:52 -08:00
Aki 351d4030a8 fs_utils.py: Use python-magic for guessing mimetype (#31) 
1. mimetypes is not a reliable way to know the mime type of a file.

2. python-magic uses libmagic which identifies file types by checking their headers according to a predefined list of file types.

	https://github.com/ahupp/python-magic
 2020-01-16 09:06:55 -08:00
Shivam Jha 00a8e57467
Fixed error in gdrivetools 2020-01-10 17:36:51 +05:30
lzzy12 326129fdeb Make Oauth Scope a list as it should be 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-01-04 07:42:55 +05:30
lzzy12 e5f997d1f3 Revert "Fixed potential race conditions in GoogleDriveHelper class" 
This reverts commit 20dae25dc9.

Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2020-01-03 10:41:54 +05:30
jaskaranSM 7b2d5d50ab Implement Exponential Backoff 
- for handling Google Drive API errors such as 5xx,4xx
 2020-01-03 10:15:40 +05:30
jaskaranSM a64b1b5346 Fixed stucked downloads when stopped via aria bt-stop-timeout 2019-12-29 19:50:01 +05:30
jaskaranSM 30dd2d97f5 Minor Refactor 2019-12-29 19:14:52 +05:30
jaskaranSM 38fa1ec2a6 Fixed Handling of Cancellation of uploads/archivals 2019-12-29 19:13:10 +05:30
lzzy12 20dae25dc9 Fixed potential race conditions in GoogleDriveHelper class 
* Also simplified code a bit
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2019-12-26 16:39:59 +05:30
lzzy12 820d72f317 Fixed files not being shown which were not uploaded by bot 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2019-12-26 14:47:05 +05:30
lzzy12 4fc07cd7a9 Fix typo in speed indicator 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2019-12-25 19:17:35 +05:30
lzzy12 4e1f7a8a77 Small optimisations all around 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2019-12-25 16:40:43 +05:30
lzzy12 8e886a3030 Add Index URL in /list cmd 
Also simplified code a bit

Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2019-12-25 15:53:43 +05:30
lzzy12 53b6829235 Fixed Team Drive list functionality 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2019-12-25 15:15:36 +05:30
jaskaranSM 8a516024cb Fixed Folder upload to Team Drives 2019-12-24 21:11:33 +05:30
lzzy12 b31a39f11a Enable TeamDrive support 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2019-12-24 20:12:39 +05:30
lzzy12 0ed1ca3531 Enable TeamDrive support 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2019-12-24 19:49:48 +05:30
lzzy12 6dfb3e9863 Simplify get_readable_msg code 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2019-12-24 17:35:09 +05:30
lzzy12 82ae49a33a Re implement upload progress after rebase 
Signed-off-by: lzzy12 <jhashivam2020@gmail.com>
 2019-12-24 17:24:35 +05:30