From 290d0b51790f884b54d218347a0de4fdc664152e Mon Sep 17 00:00:00 2001 From: 93-infinity Date: Sat, 20 Mar 2021 22:52:27 +0800 Subject: [PATCH 1/9] Modified & enhanced docs wrt paths --- README.md | 11 ++++---- docs/About-Paths.md | 59 +++++++++++++++++++++++++++++++++++++++++ docs/Cygwin-Git-Bash.md | 3 ++- docs/Details.md | 2 +- docs/WSL.md | 18 +++++++++++++ 5 files changed, 86 insertions(+), 7 deletions(-) create mode 100644 docs/About-Paths.md create mode 100644 docs/WSL.md diff --git a/README.md b/README.md index baeef35..49d8100 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,7 @@ Monero (XMR): [44yL1VNsRmvW3khxHAQvzr9mfyfkMLFmS5xo3EehkQRgcBSgAUcoqf4Cj9mTyCwEP ## Installation (Linux): -Install [yt-dlp](https://github.com/pukkandan/yt-dlp) and [ffmpeg](https://www.ffmpeg.org/) (optionnally, install [atomicparsley](https://github.com/wez/atomicparsley), it's required for embedding thumbnails into m4a files). +Install [yt-dlp](https://github.com/pukkandan/yt-dlp) and [ffmpeg](https://www.ffmpeg.org/) (optionally, install [atomicparsley](https://github.com/wez/atomicparsley), it's required for embedding thumbnails into m4a files). Download the latest [release](https://github.com/TheFrenchGhosty/TheFrenchGhostys-YouTube-DL-Archivist-Scripts/releases) and follow the included ReadMe, do not use the `master` branch unless you are ready to re-download EVERYTHING. @@ -43,13 +43,14 @@ Download the latest [release](https://github.com/TheFrenchGhosty/TheFrenchGhosty ## Installation (Windows): -Because Windows has a path length limit of 260 characters, and because of the work required (and the fact that the Windows version was untested when it existed), Windows users needs to use the scripts in WSL (or something linux-on-windows). WSL is the only thing that bypass the path length limit so every video will get downloaded with it. However when playing those videos with a native Windows software the path length limit might be an issue, I can't do anything about that. - -Installation using WSL (recommended): TODO - +The Windows version, while it existed, required [work|some work|a lot of work|] to maintain because it was [too complex|very buggy|caused hourly BSOD's], and was never extensively tested. Windows users need to use the scripts in WSL, Cygwin, Git Bash, or some other Linux-on-Windows environment that enables Bash functionality in Windows. + +Installation using WSL (recommended): [Here](docs/WSL.md) Installation using Cygwin, Git Bash or similar: [Here](docs/Cygwin-Git-Bash.md) +Downloaded videos will likely exceed Windows' 260-character path length limit and not be playable out of the box. You should read more about [paths](docs/About-Paths.md) to familiarize yourself with the issue, as well as some potential workarounds. + --- ## Usage: diff --git a/docs/About-Paths.md b/docs/About-Paths.md new file mode 100644 index 0000000..b4725f3 --- /dev/null +++ b/docs/About-Paths.md @@ -0,0 +1,59 @@ +## About Paths + +### Paths on Windows + +The path of a file or folder specifies its location on the filesystem. The path length refers to the total number of character in the path. +- C:\Users\username\Documents (a folder path, 27 characters) +- C:\Windows\explorer.exe (a file path, 23 characters) + +--- + +### Path length limits + +Windows has a 260-character path length limit for compatibility reasons. This means Windows and other programs will refuse to: +- create files or folders that exceed this limit, +- rename existing files or folders to exceed this limit. + +It's possible for programs to use alternate methods to bypass the 260-character limit. This is because the NTFS filesystem supports long path lengths of up to 32,768 characters, [even since the days of Windows XP](https://stackoverflow.com/questions/265769/maximum-filename-length-in-ntfs-windows-xp-and-windows-vista). Incidentally, this is even higher than Linux's 4096-character limit. + +However files and folders created by programs that support "long paths" will be impossible to manipulate. They are valid files, but will not be able to be opened, copied, moved, renamed, or deleted. + +--- + +### Path lengths and the scripts + +These scripts were designed to both neatly organize videos as well as preserve as much information about them as possible, since this data can be valuable to archivists. The names of both the videos and the folders they are stored in may include (but are not limited to): +- channel name +- playlist name +- upload date +- full video title +- YouTube video ID. + +Practically, this means videos downloaded with these scripts will often exceed Windows' 260-character limit. Hence, you should keep the path as short as possible; store the scripts in the root of the drive. Place them in "C:\\" or "D:\\" instead of "C:\Users\username\Downloads\". Some videos will still exceed the 260-character limit, but this should be done regardless. + +--- + +### Playing video files with long paths + +"VLC Media Player" downloaded from the VLC website does not support currently support long paths. Videos cannot be opened. + +"Metro" apps appear to support long paths. "Movies & TV" (built in to Windows 10), or VLC (from the Microsoft Store). + +Video players running *inside* WSL do work. To get the videos to display, you need to set up an Xserver (e.g. Xming, or MobaXterm with one built in), and possibly installing PulseAudio (untested). + +--- + +### Manipulating (copying, moving, renaming, deleting) files with long paths +Windows and other programs will not let you manipulate files with long paths. Overcome this by manipulating files inside WSL using commandline Linux tools (cp, mv, rm). Alternatively, use a file manager like mc (Midnight Commander). + +HardLinkShell (on Windows) enables copying or moving videos with long paths. + + +### Other workarounds +Enable a "Group Policy" setting introduced in recent versions of Windows 10 to [remove the 260-character limit (i.e., long paths)](https://superuser.com/questions/1119883/windows-10-enable-ntfs-long-paths-policy-option-missing). Applications need to be updated to support long paths; this setting is not very useful for now. + +Choose a shorter path, or not to save files so deeply into the folder tree. This is not really an option for these scripts, if you already store them in the root of the drive. + + +### Notes +[Further reading on WSL and path lengths](https://channel9.msdn.com/Blogs/Seth-Juarez/Windows-Subsystem-for-Linux-File-System) \ No newline at end of file diff --git a/docs/Cygwin-Git-Bash.md b/docs/Cygwin-Git-Bash.md index db218e0..30213e8 100644 --- a/docs/Cygwin-Git-Bash.md +++ b/docs/Cygwin-Git-Bash.md @@ -1,8 +1,9 @@ ### Installation using Cygwin, Git Bash or similar: #### Known Issues -* Windows has a hard limit of 260 characters for the filepath. This can cause issues with video downloading. It is recommended that you make your filepath as short as possible. * If you are receiving a permission denied error when attempting to execute python, [this thread](https://stackoverflow.com/questions/56974927/permission-denied-trying-to-run-python-on-windows-10) may be helpful. +* Downloaded videos will likely exceed Windows' 260-character path length limit and not be playable out of the box. You should read more about [paths](docs/About-Paths.md) to familiarize yourself with the issue, as well as some potential workarounds. + #### Required Software In order to run these scripts, the following software packages must be installed: diff --git a/docs/Details.md b/docs/Details.md index aaef874..e28c224 100644 --- a/docs/Details.md +++ b/docs/Details.md @@ -1,4 +1,4 @@ -# Details: +## Details: `--format "(bestvideo[vcodec^=av01][height>=4320][fps>30]/bestvideo[vcodec^=vp9.2][height>=4320][fps>30]/bestvideo[vcodec^=vp9][height>=4320][fps>30]/bestvideo[vcodec^=avc1][height>=4320][fps>30]/bestvideo[height>=4320][fps>30]/bestvideo[vcodec^=av01][height>=4320]/bestvideo[vcodec^=vp9.2][height>=4320]/bestvideo[vcodec^=vp9][height>=4320]/bestvideo[vcodec^=avc1][height>=4320]/bestvideo[height>=4320]/bestvideo[vcodec^=av01][height>=2880][fps>30]/bestvideo[vcodec^=vp9.2][height>=2880][fps>30]/bestvideo[vcodec^=vp9][height>=2880][fps>30]/bestvideo[vcodec^=avc1][height>=2880][fps>30]/bestvideo[height>=2880][fps>30]/bestvideo[vcodec^=av01][height>=2880]/bestvideo[vcodec^=vp9.2][height>=2880]/bestvideo[vcodec^=vp9][height>=2880]/bestvideo[vcodec^=avc1][height>=2880]/bestvideo[height>=2880]/bestvideo[vcodec^=av01][height>=2160][fps>30]/bestvideo[vcodec^=vp9.2][height>=2160][fps>30]/bestvideo[vcodec^=vp9][height>=2160][fps>30]/bestvideo[vcodec^=avc1][height>=2160][fps>30]/bestvideo[height>=2160][fps>30]/bestvideo[vcodec^=av01][height>=2160]/bestvideo[vcodec^=vp9.2][height>=2160]/bestvideo[vcodec^=vp9][height>=2160]/bestvideo[vcodec^=avc1][height>=2160]/bestvideo[height>=2160]/bestvideo[vcodec^=av01][height>=1440][fps>30]/bestvideo[vcodec^=vp9.2][height>=1440][fps>30]/bestvideo[vcodec^=vp9][height>=1440][fps>30]/bestvideo[vcodec^=avc1][height>=1440][fps>30]/bestvideo[height>=1440][fps>30]/bestvideo[vcodec^=av01][height>=1440]/bestvideo[vcodec^=vp9.2][height>=1440]/bestvideo[vcodec^=vp9][height>=1440]/bestvideo[vcodec^=avc1][height>=1440]/bestvideo[height>=1440]/bestvideo[vcodec^=av01][height>=1080][fps>30]/bestvideo[vcodec^=vp9.2][height>=1080][fps>30]/bestvideo[vcodec^=vp9][height>=1080][fps>30]/bestvideo[vcodec^=avc1][height>=1080][fps>30]/bestvideo[height>=1080][fps>30]/bestvideo[vcodec^=av01][height>=1080]/bestvideo[vcodec^=vp9.2][height>=1080]/bestvideo[vcodec^=vp9][height>=1080]/bestvideo[vcodec^=avc1][height>=1080]/bestvideo[height>=1080]/bestvideo[vcodec^=av01][height>=720][fps>30]/bestvideo[vcodec^=vp9.2][height>=720][fps>30]/bestvideo[vcodec^=vp9][height>=720][fps>30]/bestvideo[vcodec^=avc1][height>=720][fps>30]/bestvideo[height>=720][fps>30]/bestvideo[vcodec^=av01][height>=720]/bestvideo[vcodec^=vp9.2][height>=720]/bestvideo[vcodec^=vp9][height>=720]/bestvideo[vcodec^=avc1][height>=720]/bestvideo[height>=720]/bestvideo[vcodec^=av01][height>=480][fps>30]/bestvideo[vcodec^=vp9.2][height>=480][fps>30]/bestvideo[vcodec^=vp9][height>=480][fps>30]/bestvideo[vcodec^=avc1][height>=480][fps>30]/bestvideo[height>=480][fps>30]/bestvideo[vcodec^=av01][height>=480]/bestvideo[vcodec^=vp9.2][height>=480]/bestvideo[vcodec^=vp9][height>=480]/bestvideo[vcodec^=avc1][height>=480]/bestvideo[height>=480]/bestvideo[vcodec^=av01][height>=360][fps>30]/bestvideo[vcodec^=vp9.2][height>=360][fps>30]/bestvideo[vcodec^=vp9][height>=360][fps>30]/bestvideo[vcodec^=avc1][height>=360][fps>30]/bestvideo[height>=360][fps>30]/bestvideo[vcodec^=av01][height>=360]/bestvideo[vcodec^=vp9.2][height>=360]/bestvideo[vcodec^=vp9][height>=360]/bestvideo[vcodec^=avc1][height>=360]/bestvideo[height>=360]/bestvideo[vcodec^=avc1][height>=240][fps>30]/bestvideo[vcodec^=av01][height>=240][fps>30]/bestvideo[vcodec^=vp9.2][height>=240][fps>30]/bestvideo[vcodec^=vp9][height>=240][fps>30]/bestvideo[height>=240][fps>30]/bestvideo[vcodec^=avc1][height>=240]/bestvideo[vcodec^=av01][height>=240]/bestvideo[vcodec^=vp9.2][height>=240]/bestvideo[vcodec^=vp9][height>=240]/bestvideo[height>=240]/bestvideo[vcodec^=avc1][height>=144][fps>30]/bestvideo[vcodec^=av01][height>=144][fps>30]/bestvideo[vcodec^=vp9.2][height>=144][fps>30]/bestvideo[vcodec^=vp9][height>=144][fps>30]/bestvideo[height>=144][fps>30]/bestvideo[vcodec^=avc1][height>=144]/bestvideo[vcodec^=av01][height>=144]/bestvideo[vcodec^=vp9.2][height>=144]/bestvideo[vcodec^=vp9][height>=144]/bestvideo[height>=144]/bestvideo)+(bestaudio[acodec^=opus]/bestaudio)/best"`: Tell yt-dlp to download the best quality available prioritizing the most compressed/recent codecs, except for 240p and 144p where AVC is prioritized (Original idea by [Veloldo](https://redd.it/c6fh4x)) diff --git a/docs/WSL.md b/docs/WSL.md new file mode 100644 index 0000000..91939c1 --- /dev/null +++ b/docs/WSL.md @@ -0,0 +1,18 @@ +### Installation using WSL: +WSL allows the running of a modified Linux environment (Ubuntu, Debian, OpenSUSE, and more) inside Windows 10. Unlike a virtual machine, it can directly interact with Windows files on the hard drive. + +Main benefits are that it's created by Microsoft, so it's more likely to be better supported. Benefits for the purposes of these scripts are the ability to keep dependencies (ffmpeg, atomicparsley, yt-dlp) up-to-date with package managers like apt or pip. + +However the setup process does necessitate a relatively large one-time download and storing of WSL's files (approximately 2GB) - users taking issue with this may consider the other options such as Cygwin or Git Bash. [Compare the above here.](https://askubuntu.com/questions/1042285/reduce-size-of-a-wsl-installation-ubuntu-18-on-windows-10) + + +#### Known Issues +* The latest version of WSL (WSL2) stores the Linux filesystem hierarchy ( /* ) inside a virtual hard disk file. To avoid this file ballooning in size ([and having to manually shrink it in the future](https://stephenreescarter.net/how-to-shrink-a-wsl2-virtual-disk/)), it's recommended to store the scripts and videos outside of the Linux filesystem, inside a Windows drive (e.g. "/mnt/c/" for "C:\\", or "/mnt/d/" for a drive mounted at "D:\\" ) +* Downloaded videos will likely exceed Windows' 260-character path length limit and not be playable out of the box. You should read more about [paths](docs/About-Paths.md) to familiarize yourself with the issue, as well as some potential workarounds. + + +#### Required Software +Follow the "Manual Install" instructions in the [Windows Subsystem for Linux Installation Guide for Windows 10](https://docs.microsoft.com/en-us/windows/wsl/install-win10). + +Once WSL is installed, download the latest [release](https://github.com/TheFrenchGhosty/TheFrenchGhostys-YouTube-DL-Archivist-Scripts/releases) and follow the "Installation (Linux)" section of this README. + From b5ea6024d7a4b556da30f813473db6a23608de69 Mon Sep 17 00:00:00 2001 From: 93-infinity Date: Sun, 21 Mar 2021 10:01:09 +0800 Subject: [PATCH 2/9] Update About-Paths.md Change HardLinkShell description --- docs/About-Paths.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/About-Paths.md b/docs/About-Paths.md index b4725f3..fcdc5a0 100644 --- a/docs/About-Paths.md +++ b/docs/About-Paths.md @@ -46,7 +46,7 @@ Video players running *inside* WSL do work. To get the videos to display, you ne ### Manipulating (copying, moving, renaming, deleting) files with long paths Windows and other programs will not let you manipulate files with long paths. Overcome this by manipulating files inside WSL using commandline Linux tools (cp, mv, rm). Alternatively, use a file manager like mc (Midnight Commander). -HardLinkShell (on Windows) enables copying or moving videos with long paths. +HardLinkShell (on Windows) enables copying or moving files with long paths. ### Other workarounds From 3710fb9badf30a49845fea1801f11dbf2100d8a0 Mon Sep 17 00:00:00 2001 From: 93-infinity Date: Sun, 21 Mar 2021 10:12:01 +0800 Subject: [PATCH 3/9] Update About-Paths.md clarify: files are difficult, not impossible, to manipulate --- docs/About-Paths.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/About-Paths.md b/docs/About-Paths.md index fcdc5a0..1363733 100644 --- a/docs/About-Paths.md +++ b/docs/About-Paths.md @@ -16,7 +16,7 @@ Windows has a 260-character path length limit for compatibility reasons. This me It's possible for programs to use alternate methods to bypass the 260-character limit. This is because the NTFS filesystem supports long path lengths of up to 32,768 characters, [even since the days of Windows XP](https://stackoverflow.com/questions/265769/maximum-filename-length-in-ntfs-windows-xp-and-windows-vista). Incidentally, this is even higher than Linux's 4096-character limit. -However files and folders created by programs that support "long paths" will be impossible to manipulate. They are valid files, but will not be able to be opened, copied, moved, renamed, or deleted. +However files and folders created by programs that support "long paths" will be difficult to manipulate with most programs. They are valid files, but will not be able to be easily opened, copied, moved, renamed, or deleted, without special tools. --- From 0be34b2a9f39715f3fda9fb1472304cd5742cd4f Mon Sep 17 00:00:00 2001 From: 93-infinity Date: Sun, 21 Mar 2021 14:19:01 +0800 Subject: [PATCH 4/9] Update About-Paths.md fix readability make similar sections consistent elaborate workarounds discourage users from setting up xserver clarify long paths not being completely un-manipulatable clarify implication that mc is not a linux program link to hardlinkshell explain explorer not supporting long paths even with new group policy further resources: tools and further reading --- docs/About-Paths.md | 53 ++++++++++++++++++++++++++++----------------- 1 file changed, 33 insertions(+), 20 deletions(-) diff --git a/docs/About-Paths.md b/docs/About-Paths.md index 1363733..b0b31eb 100644 --- a/docs/About-Paths.md +++ b/docs/About-Paths.md @@ -1,59 +1,72 @@ ## About Paths -### Paths on Windows +### What is a path The path of a file or folder specifies its location on the filesystem. The path length refers to the total number of character in the path. -- C:\Users\username\Documents (a folder path, 27 characters) -- C:\Windows\explorer.exe (a file path, 23 characters) +- C:\Users\username\Pictures\Places\France\Paris.jpeg (a file path, 51 characters) +- C:\Program Files (a folder path, 16 characters) --- ### Path length limits Windows has a 260-character path length limit for compatibility reasons. This means Windows and other programs will refuse to: -- create files or folders that exceed this limit, +- create files or folders with filenames that exceed this limit, - rename existing files or folders to exceed this limit. -It's possible for programs to use alternate methods to bypass the 260-character limit. This is because the NTFS filesystem supports long path lengths of up to 32,768 characters, [even since the days of Windows XP](https://stackoverflow.com/questions/265769/maximum-filename-length-in-ntfs-windows-xp-and-windows-vista). Incidentally, this is even higher than Linux's 4096-character limit. +It's possible for programs to use alternate methods to bypass the 260-character limit. This is because Windows' underlying NTFS filesystem supports "long paths" with lengths of up to 32,767 characters, [even since the days of Windows XP](https://stackoverflow.com/questions/265769/maximum-filename-length-in-ntfs-windows-xp-and-windows-vista). Incidentally, this is even higher than Linux's 4096-character limit. -However files and folders created by programs that support "long paths" will be difficult to manipulate with most programs. They are valid files, but will not be able to be easily opened, copied, moved, renamed, or deleted, without special tools. +However files and folders created by programs that support long paths will be impossible to manipulate in programs that don't support them, or in Windows itself. They are valid files, but will not be able to be easily opened, copied, moved, renamed, or deleted, without the use of special tools. --- ### Path lengths and the scripts -These scripts were designed to both neatly organize videos as well as preserve as much information about them as possible, since this data can be valuable to archivists. The names of both the videos and the folders they are stored in may include (but are not limited to): -- channel name -- playlist name -- upload date -- full video title +These scripts were designed to both neatly organize videos as well as preserve as much information about them as possible, since this information is often valuable to archivists. The names of both the videos and the folders they are stored in may include (but are not limited to): +- channel name, +- playlist name, +- upload date, +- full video title, - YouTube video ID. -Practically, this means videos downloaded with these scripts will often exceed Windows' 260-character limit. Hence, you should keep the path as short as possible; store the scripts in the root of the drive. Place them in "C:\\" or "D:\\" instead of "C:\Users\username\Downloads\". Some videos will still exceed the 260-character limit, but this should be done regardless. +Practically speaking, this means videos downloaded with these scripts will often exceed Windows' 260-character limit. Hence, you should keep the path as short as possible; store the scripts in the root (top level) of the drive. Place them in "C:\\" or "D:\\" instead of "C:\Users\username\Downloads\". Some videos will likely still exceed the 260-character limit, but this is good practice and should be done regardless. --- ### Playing video files with long paths -"VLC Media Player" downloaded from the VLC website does not support currently support long paths. Videos cannot be opened. +"VLC Media Player" downloaded from the VLC website [does not support long paths](https://forum.videolan.org/viewtopic.php?t=150542). Videos cannot be opened. + +Other programs may work. (WIP) "Metro" apps appear to support long paths. "Movies & TV" (built in to Windows 10), or VLC (from the Microsoft Store). -Video players running *inside* WSL do work. To get the videos to display, you need to set up an Xserver (e.g. Xming, or MobaXterm with one built in), and possibly installing PulseAudio (untested). +Video players running *inside* WSL do work. To get the videos to display, you need to set up an Xserver (e.g. Xming, or MobaXterm which has one built in), and possibly install PulseAudio (untested). This is a somewhat complex process and is not recommended for beginners. --- ### Manipulating (copying, moving, renaming, deleting) files with long paths -Windows and other programs will not let you manipulate files with long paths. Overcome this by manipulating files inside WSL using commandline Linux tools (cp, mv, rm). Alternatively, use a file manager like mc (Midnight Commander). -HardLinkShell (on Windows) enables copying or moving files with long paths. +Files and folders with long paths will be impossible to manipulate in programs that don't support them, or in Windows itself. Overcome this by manipulating files inside WSL: +* Download a file manager like mc (Midnight Commander) - recommended +* Use built-in Linux commandline tools like cp (to copy), mv (to move), and rm (to delete) + +[HardLinkShell](https://schinagl.priv.at/nt/hardlinkshellext/linkshellextension.html) (Windows) enables copying or moving files with long paths. +--- ### Other workarounds -Enable a "Group Policy" setting introduced in recent versions of Windows 10 to [remove the 260-character limit (i.e., long paths)](https://superuser.com/questions/1119883/windows-10-enable-ntfs-long-paths-policy-option-missing). Applications need to be updated to support long paths; this setting is not very useful for now. -Choose a shorter path, or not to save files so deeply into the folder tree. This is not really an option for these scripts, if you already store them in the root of the drive. +Enable a "Group Policy" setting introduced in recent versions of Windows 10 to [remove the 260-character limit](https://superuser.com/questions/1119883/windows-10-enable-ntfs-long-paths-policy-option-missing) and enable support for long paths. For now it doesn't do much; applications need to be updated to support this setting. Not even Windows Explorer seems to support it. + +Choose a shorter path by not saving files too deeply into the folder tree. This is not really an option for these scripts, if you already store them in the root of the drive. + +--- + +### Further resources + +[PathLengthChecker](https://github.com/deadlydog/PathLengthChecker) - Find out a file/folder's path length +[Further reading on WSL and Path Lengths](https://channel9.msdn.com/Blogs/Seth-Juarez/Windows-Subsystem-for-Linux-File-System) -### Notes -[Further reading on WSL and path lengths](https://channel9.msdn.com/Blogs/Seth-Juarez/Windows-Subsystem-for-Linux-File-System) \ No newline at end of file +[Background on Maximum Path Length Limitations](https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation) \ No newline at end of file From c5d20725f637c3fc2289afe105ff9c932baab9c4 Mon Sep 17 00:00:00 2001 From: 93-infinity Date: Sun, 21 Mar 2021 14:28:24 +0800 Subject: [PATCH 5/9] Update About-Paths.md minor text changes, present tense, readability, specify path length limits apply to windows only --- docs/About-Paths.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/About-Paths.md b/docs/About-Paths.md index b0b31eb..dc3a946 100644 --- a/docs/About-Paths.md +++ b/docs/About-Paths.md @@ -8,10 +8,10 @@ The path of a file or folder specifies its location on the filesystem. The path --- -### Path length limits +### Path length limits on Windows Windows has a 260-character path length limit for compatibility reasons. This means Windows and other programs will refuse to: -- create files or folders with filenames that exceed this limit, +- create files or folders with to exceed this limit, - rename existing files or folders to exceed this limit. It's possible for programs to use alternate methods to bypass the 260-character limit. This is because Windows' underlying NTFS filesystem supports "long paths" with lengths of up to 32,767 characters, [even since the days of Windows XP](https://stackoverflow.com/questions/265769/maximum-filename-length-in-ntfs-windows-xp-and-windows-vista). Incidentally, this is even higher than Linux's 4096-character limit. @@ -33,7 +33,7 @@ Practically speaking, this means videos downloaded with these scripts will often --- -### Playing video files with long paths +### Play video files with long paths "VLC Media Player" downloaded from the VLC website [does not support long paths](https://forum.videolan.org/viewtopic.php?t=150542). Videos cannot be opened. @@ -41,11 +41,11 @@ Other programs may work. (WIP) "Metro" apps appear to support long paths. "Movies & TV" (built in to Windows 10), or VLC (from the Microsoft Store). -Video players running *inside* WSL do work. To get the videos to display, you need to set up an Xserver (e.g. Xming, or MobaXterm which has one built in), and possibly install PulseAudio (untested). This is a somewhat complex process and is not recommended for beginners. +Video players running *inside* WSL do work. To get the videos to display, you need to set up an Xserver (e.g. Xming), or use MobaXterm which has one built in, and possibly install PulseAudio (untested). This is a somewhat complex process and is not recommended for beginners. --- -### Manipulating (copying, moving, renaming, deleting) files with long paths +### Manipulate (copy, move, rename, delete) files with long paths Files and folders with long paths will be impossible to manipulate in programs that don't support them, or in Windows itself. Overcome this by manipulating files inside WSL: * Download a file manager like mc (Midnight Commander) - recommended @@ -57,7 +57,7 @@ Files and folders with long paths will be impossible to manipulate in programs t ### Other workarounds -Enable a "Group Policy" setting introduced in recent versions of Windows 10 to [remove the 260-character limit](https://superuser.com/questions/1119883/windows-10-enable-ntfs-long-paths-policy-option-missing) and enable support for long paths. For now it doesn't do much; applications need to be updated to support this setting. Not even Windows Explorer seems to support it. +Enable a "Group Policy" setting introduced in recent versions of Windows 10 to [remove the 260-character limit](https://superuser.com/questions/1119883/windows-10-enable-ntfs-long-paths-policy-option-missing) and enable support for long paths. For now it doesn't do much; applications need to be updated to support this setting (not even Windows Explorer seems to). Choose a shorter path by not saving files too deeply into the folder tree. This is not really an option for these scripts, if you already store them in the root of the drive. @@ -65,7 +65,7 @@ Choose a shorter path by not saving files too deeply into the folder tree. This ### Further resources -[PathLengthChecker](https://github.com/deadlydog/PathLengthChecker) - Find out a file/folder's path length +[PathLengthChecker](https://github.com/deadlydog/PathLengthChecker) - Find out the path length of files/folders [Further reading on WSL and Path Lengths](https://channel9.msdn.com/Blogs/Seth-Juarez/Windows-Subsystem-for-Linux-File-System) From a39d9490c28935de1dcc2ac4ada7a12aa95c46ac Mon Sep 17 00:00:00 2001 From: 93-infinity Date: Sun, 21 Mar 2021 14:33:37 +0800 Subject: [PATCH 6/9] Update About-Paths.md Worse at spelling than thefrenchghosty --- docs/About-Paths.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/About-Paths.md b/docs/About-Paths.md index dc3a946..6b8e4bf 100644 --- a/docs/About-Paths.md +++ b/docs/About-Paths.md @@ -11,7 +11,7 @@ The path of a file or folder specifies its location on the filesystem. The path ### Path length limits on Windows Windows has a 260-character path length limit for compatibility reasons. This means Windows and other programs will refuse to: -- create files or folders with to exceed this limit, +- create files or folders that exceed this limit, - rename existing files or folders to exceed this limit. It's possible for programs to use alternate methods to bypass the 260-character limit. This is because Windows' underlying NTFS filesystem supports "long paths" with lengths of up to 32,767 characters, [even since the days of Windows XP](https://stackoverflow.com/questions/265769/maximum-filename-length-in-ntfs-windows-xp-and-windows-vista). Incidentally, this is even higher than Linux's 4096-character limit. From c404623ace89ecf4b6038b6cfbb231027fc159c9 Mon Sep 17 00:00:00 2001 From: 93-infinity Date: Sun, 21 Mar 2021 18:40:55 +0800 Subject: [PATCH 7/9] Minor grammar/syntax changes - modify wsl/cygwin heading levels to be consistent with thefrenchghosty's existing files in docs folder - Clarify long-path'ed files as not being corrupt or problematic - tell the user they can organize files as they wish, but caveat emptor - exercise restraint by not adding section telling user to consider using linux --- docs/About-Paths.md | 10 ++++++---- docs/Cygwin-Git-Bash.md | 8 ++++---- docs/WSL.md | 6 +++--- 3 files changed, 13 insertions(+), 11 deletions(-) diff --git a/docs/About-Paths.md b/docs/About-Paths.md index 6b8e4bf..950fafb 100644 --- a/docs/About-Paths.md +++ b/docs/About-Paths.md @@ -1,6 +1,6 @@ ## About Paths -### What is a path +### What is a path? The path of a file or folder specifies its location on the filesystem. The path length refers to the total number of character in the path. - C:\Users\username\Pictures\Places\France\Paris.jpeg (a file path, 51 characters) @@ -16,13 +16,13 @@ Windows has a 260-character path length limit for compatibility reasons. This me It's possible for programs to use alternate methods to bypass the 260-character limit. This is because Windows' underlying NTFS filesystem supports "long paths" with lengths of up to 32,767 characters, [even since the days of Windows XP](https://stackoverflow.com/questions/265769/maximum-filename-length-in-ntfs-windows-xp-and-windows-vista). Incidentally, this is even higher than Linux's 4096-character limit. -However files and folders created by programs that support long paths will be impossible to manipulate in programs that don't support them, or in Windows itself. They are valid files, but will not be able to be easily opened, copied, moved, renamed, or deleted, without the use of special tools. +However files and folders created by programs that support long paths will be impossible to manipulate in programs that don't support them, or in Windows itself. They will not be able to be easily opened, copied, moved, renamed, or deleted, without the use of special tools. The files themselves are perfectly valid and not corrupt, and having long paths is not a problem per se, just an inconvenience. --- ### Path lengths and the scripts -These scripts were designed to both neatly organize videos as well as preserve as much information about them as possible, since this information is often valuable to archivists. The names of both the videos and the folders they are stored in may include (but are not limited to): +These scripts were designed to both neatly organize videos as well as preserve as much information about them as possible, since this information is often valuable to archivists. The names of both the videos and the folders they are stored in may include (in various combinations): - channel name, - playlist name, - upload date, @@ -41,7 +41,7 @@ Other programs may work. (WIP) "Metro" apps appear to support long paths. "Movies & TV" (built in to Windows 10), or VLC (from the Microsoft Store). -Video players running *inside* WSL do work. To get the videos to display, you need to set up an Xserver (e.g. Xming), or use MobaXterm which has one built in, and possibly install PulseAudio (untested). This is a somewhat complex process and is not recommended for beginners. +Video players running *inside* WSL do work. To get the videos to display, you need to set up an Xserver (e.g. Xming), or use MobaXterm which has one built in, and possibly install PulseAudio (untested). This is a somewhat involved process and is not recommended for beginners. --- @@ -61,6 +61,8 @@ Enable a "Group Policy" setting introduced in recent versions of Windows 10 to [ Choose a shorter path by not saving files too deeply into the folder tree. This is not really an option for these scripts, if you already store them in the root of the drive. +Organize videos and their containing folders with your own organization system (i.e., shorten their names). This is an issue of personal preference. If you break the scripts because it can't files where it expects them to be, that is your problem to deal with. + --- ### Further resources diff --git a/docs/Cygwin-Git-Bash.md b/docs/Cygwin-Git-Bash.md index 30213e8..43be137 100644 --- a/docs/Cygwin-Git-Bash.md +++ b/docs/Cygwin-Git-Bash.md @@ -1,11 +1,11 @@ -### Installation using Cygwin, Git Bash or similar: +## Installation using Cygwin, Git Bash or similar -#### Known Issues +### Known Issues * If you are receiving a permission denied error when attempting to execute python, [this thread](https://stackoverflow.com/questions/56974927/permission-denied-trying-to-run-python-on-windows-10) may be helpful. * Downloaded videos will likely exceed Windows' 260-character path length limit and not be playable out of the box. You should read more about [paths](docs/About-Paths.md) to familiarize yourself with the issue, as well as some potential workarounds. -#### Required Software +### Required Software In order to run these scripts, the following software packages must be installed: * [Cygwin](https://www.cygwin.com/), [Git Bash](http://git-scm.com), or some other application that enables Bash functionality in Windows. @@ -14,7 +14,7 @@ In order to run these scripts, the following software packages must be installed * [ffmpeg](https://www.ffmpeg.org/) - Make sure this is in your PATH. * [atomicparsley](https://github.com/wez/atomicparsley) (optional) - Make sure this is in your PATH. -#### Configuration +### Configuration First, make sure that all dependencies are installed and your PATH is configured properly by opening your Bash terminal of choice and executing the following: * `python --version` diff --git a/docs/WSL.md b/docs/WSL.md index 91939c1..5e6c149 100644 --- a/docs/WSL.md +++ b/docs/WSL.md @@ -1,4 +1,4 @@ -### Installation using WSL: +## Installation using WSL WSL allows the running of a modified Linux environment (Ubuntu, Debian, OpenSUSE, and more) inside Windows 10. Unlike a virtual machine, it can directly interact with Windows files on the hard drive. Main benefits are that it's created by Microsoft, so it's more likely to be better supported. Benefits for the purposes of these scripts are the ability to keep dependencies (ffmpeg, atomicparsley, yt-dlp) up-to-date with package managers like apt or pip. @@ -6,12 +6,12 @@ Main benefits are that it's created by Microsoft, so it's more likely to be bett However the setup process does necessitate a relatively large one-time download and storing of WSL's files (approximately 2GB) - users taking issue with this may consider the other options such as Cygwin or Git Bash. [Compare the above here.](https://askubuntu.com/questions/1042285/reduce-size-of-a-wsl-installation-ubuntu-18-on-windows-10) -#### Known Issues +### Known Issues * The latest version of WSL (WSL2) stores the Linux filesystem hierarchy ( /* ) inside a virtual hard disk file. To avoid this file ballooning in size ([and having to manually shrink it in the future](https://stephenreescarter.net/how-to-shrink-a-wsl2-virtual-disk/)), it's recommended to store the scripts and videos outside of the Linux filesystem, inside a Windows drive (e.g. "/mnt/c/" for "C:\\", or "/mnt/d/" for a drive mounted at "D:\\" ) * Downloaded videos will likely exceed Windows' 260-character path length limit and not be playable out of the box. You should read more about [paths](docs/About-Paths.md) to familiarize yourself with the issue, as well as some potential workarounds. -#### Required Software +### Required Software Follow the "Manual Install" instructions in the [Windows Subsystem for Linux Installation Guide for Windows 10](https://docs.microsoft.com/en-us/windows/wsl/install-win10). Once WSL is installed, download the latest [release](https://github.com/TheFrenchGhosty/TheFrenchGhostys-YouTube-DL-Archivist-Scripts/releases) and follow the "Installation (Linux)" section of this README. From 4a229e1e54fb61b48e02c4823a686c032d0ece6b Mon Sep 17 00:00:00 2001 From: 93-infinity Date: Fri, 26 Mar 2021 18:20:59 +0800 Subject: [PATCH 8/9] Fix typos no more changes --- docs/About-Paths.md | 2 +- docs/WSL.md | 3 +-- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/About-Paths.md b/docs/About-Paths.md index 950fafb..0a1336a 100644 --- a/docs/About-Paths.md +++ b/docs/About-Paths.md @@ -61,7 +61,7 @@ Enable a "Group Policy" setting introduced in recent versions of Windows 10 to [ Choose a shorter path by not saving files too deeply into the folder tree. This is not really an option for these scripts, if you already store them in the root of the drive. -Organize videos and their containing folders with your own organization system (i.e., shorten their names). This is an issue of personal preference. If you break the scripts because it can't files where it expects them to be, that is your problem to deal with. +Organize videos and their containing folders with your own organization system (i.e., shorten their names). This is an issue of personal preference. If you break the scripts because they can't find files where it expects them to be, that is your problem to deal with. --- diff --git a/docs/WSL.md b/docs/WSL.md index 5e6c149..fa7ff1a 100644 --- a/docs/WSL.md +++ b/docs/WSL.md @@ -1,7 +1,7 @@ ## Installation using WSL WSL allows the running of a modified Linux environment (Ubuntu, Debian, OpenSUSE, and more) inside Windows 10. Unlike a virtual machine, it can directly interact with Windows files on the hard drive. -Main benefits are that it's created by Microsoft, so it's more likely to be better supported. Benefits for the purposes of these scripts are the ability to keep dependencies (ffmpeg, atomicparsley, yt-dlp) up-to-date with package managers like apt or pip. +Main benefit: it's created by Microsoft, so it's more likely to be better supported. Benefits for the purposes of these scripts are the ability to keep dependencies (ffmpeg, atomicparsley, yt-dlp) up-to-date with package managers like apt or pip. However the setup process does necessitate a relatively large one-time download and storing of WSL's files (approximately 2GB) - users taking issue with this may consider the other options such as Cygwin or Git Bash. [Compare the above here.](https://askubuntu.com/questions/1042285/reduce-size-of-a-wsl-installation-ubuntu-18-on-windows-10) @@ -15,4 +15,3 @@ However the setup process does necessitate a relatively large one-time download Follow the "Manual Install" instructions in the [Windows Subsystem for Linux Installation Guide for Windows 10](https://docs.microsoft.com/en-us/windows/wsl/install-win10). Once WSL is installed, download the latest [release](https://github.com/TheFrenchGhosty/TheFrenchGhostys-YouTube-DL-Archivist-Scripts/releases) and follow the "Installation (Linux)" section of this README. - From a6f7e252be92410322568b3e32323a6831b26bde Mon Sep 17 00:00:00 2001 From: TheFrenchGhosty <47571719+TheFrenchGhosty@users.noreply.github.com> Date: Sun, 18 Jul 2021 00:31:13 +0200 Subject: [PATCH 9/9] Update the explanation about the windows version --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 0ea0213..3eece7f 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ Download the latest [release](https://github.com/TheFrenchGhosty/TheFrenchGhosty ## Installation (Windows): -The Windows version, while it existed, required [work|some work|a lot of work|] to maintain because it was [too complex|very buggy|caused hourly BSOD's], and was never extensively tested. Windows users need to use the scripts in WSL, Cygwin, Git Bash, or some other Linux-on-Windows environment that enables Bash functionality in Windows. +The Windows version, while it existed, required a lot of work to maintain because it needed specific changes compared to the linux version, and was never extensively tested. Windows users need to use the scripts in WSL, Cygwin, Git Bash, or some other Linux-on-Windows environment that enables Bash functionality in Windows. Installation using WSL (recommended): [Here](docs/WSL.md)