Commit Graph

217 Commits (master)

Author SHA1 Message Date
Sunshine 479c42e1ce
improve test code structure 4 years ago
Sunshine 933379c798
ensure consistent naming across all tests 4 years ago
Sunshine 59a8be493d
add support for working with local assets 4 years ago
Sunshine d5ee8ae6ab
account for legacy BODY background="" attribute 4 years ago
Sunshine 7654eec7e2
treat frames the same way as iframes 4 years ago
Sunshine 0d1e21e9ad
add black box tests 4 years ago
Sunshine 3d2d40e7cd
add support for data URL targets 4 years ago
Sunshine b8b6d8cff6
fix "succeeding" to "passing" in tests 4 years ago
Sunshine 928664dc88
correct is_valid_url to is_http_url 4 years ago
Sunshine 5c8d75539b
rename dataurl to data_url 4 years ago
Sunshine d9602e25eb
update help dialog and README.md 4 years ago
Sunshine f720fe0176
Merge pull request #114 from snshn/custom-network-timeout-option
Add option for custom network request timeout
4 years ago
Sunshine 727a5a410c
add option for custom network request timeout 4 years ago
Sunshine 5ef2b7c9dc
nullify JS within As' href attributes when needed 4 years ago
rhysd 0e12cecd85 Revert "Merge pull request #106 from rhysd/proxy-support"
This reverts commit d8def879b2, reversing
changes made to a9d114d04d.
4 years ago
rhysd d47482fcd9 fix crash at setting empty values to HTTP proxies
with this patch `https_proxy=` and `http_proxy=` will work well.
4 years ago
rhysd b68624f2f3 support HTTP and HTTPS proxies (fix #103) 4 years ago
Sunshine a9d114d04d
Merge pull request #105 from rhysd/refactor-main
Refactoring for main.rs to address several issues
4 years ago
rhysd 4e4ebe9c98 refactor main to address several issues
Addressed issues:

- when specified URL is invalid, it exited successfully with doing
  nothing. There was no way why it does not work for users
- it exited successfully even if invalid User-Agent value is specified
- it created file twice on `--output` option specified. It may cause an
  issue when some file watcher (e.g. FsEvents on macOS) is watching

Improvements:
- handle errors with `Result::expect` consistently it correctly exits
  with non-zero status on error
- define `Output` enum for handling both stdout and file outputs
4 years ago
rhysd 1779f4a374 better comments for JS_DOM_EVENT_ATTRS constant 4 years ago
rhysd 26e89ae6d3 use complete list of DOM event handlers 4 years ago
rhysd 69d99b69e8 remove . in line comment 4 years ago
Emi Simpson 05985583f0
Switch timestamps from rfc822 local time to iso8601 UTC 4 years ago
Emi Simpson 651fa716b4
Clean user, pass, and fragment from URL before writing 4 years ago
rhysd 67b79e92f9 simplify &x.into_iter() to x.iter() 4 years ago
rhysd b51f41fe34 trim attribute values 4 years ago
rhysd 6f158dc6db compare value of 'rel' properties in case-insensitive 4 years ago
rhysd 8d7052b39c ignore preload and prefetch sources
since all resources are embedded as data URL.
4 years ago
rhysd 660511b8a0 define link type of <link> element as enum and prefer match statement
since match statement checks exhaustiveness
4 years ago
Emi Simpson 9be3982dc6
Added --no-context flag to disable adding context comment 4 years ago
Emi Simpson 27c9fb4cd3
Added comment indicating the context under which the page was downloaded 4 years ago
rhysd 6e99ad13e7 upgrade reqwest to v0.10.0
This will improve build time and binary size as follows:

* Before

- **Compile targets**: 220
- **Build time**: `cargo build --release  1264.95s user 39.72s system 335% cpu 6:29.14 total`
- **Binary size**: 6578568 bytes

* After

- **Compile targets**: 170
- **Build time**: `cargo build --release  1130.64s user 32.15s system 359% cpu 5:23.69 total`
- **Binary size**: 6107088 bytes

* Differences

- **Compile targets**: 1.29x smaller
- **Build time**: 1.23x faster
- **Binary size**: 1.07x smaller
4 years ago
Sunshine 413dd66886
Merge pull request #96 from rhysd/refactorings
Refactorings
4 years ago
rhysd dc7ec6e7a8 remove more redundant type annotations 4 years ago
rhysd ed879231af fix test code was broken by refactoring 4 years ago
rhysd ddf4b8ac13 prefer &str to String for reducing allocations 4 years ago
rhysd 84c13f0605 prefer unwrap_or_default to unwrap_or 4 years ago
rhysd ce03e0e487 reduce allocation on checking DOM attributes and do not hard-code number of elements of array constant
`to_lower` allocates new string but the allocation is not necessary
here.
4 years ago
rhysd 63e19998d0 reduce clones and fix some code styles and redundant code 4 years ago
rhysd 75fb6961ed migrate to Rust 2018 4 years ago
Sunshine 5ba8931502
Merge pull request #92 from snshn/output-file-option
Add option for saving output to file
4 years ago
Sunshine 88ffde0c3b wipe integrity attributes 4 years ago
Sunshine bfb97bd062 add option for saving output to file 4 years ago
Sunshine 295931041c
Merge pull request #80 from Alch-Emi/lazyload
Add support for lazy loaded images
4 years ago
Emi Simpson dab4ae6965
Merged Y2Z/master with Alch-Emi/lazyload 4 years ago
Sunshine c7fc121c7c use clean URLs as hashmap keys 4 years ago
Sunshine 9ff9dd0928
Merge pull request #82 from snshn/str
implement str!() macro
4 years ago
Emi Simpson 3d4a932ac1
Merge Y2Z/master, fix conflicts between shared-client & resolve-css 4 years ago
Sunshine 9fe913d853 implement str!() macro 5 years ago
Sunshine 862489e41b Get rid of brackets around URLs 5 years ago
Emi Simpson 322ab41b8c
Updated tests to reflect API changes 5 years ago
Emi Simpson 65d0eab793
Use a shared client initialized in main.rs 5 years ago
Emi Simpson 292221ea28
Lazyloaded images are now loaded at compilation, with placeholders omitted 5 years ago
Emi Simpson 614af44c92
Gramatical and stylistic fixes 5 years ago
Emi Simpson feb37f5812
Added support for lazy loaded images
Note: The way this patch works is by resolving any data-src tags on images in
the same way as normal source tags are resolved.  It is assumed that most
lazy-load libraries will use this tag, and that if this tag is set, then it is a
URL that is in use.
5 years ago
Emi Simpson 028beb821c
Rustfmt update for nightly formatter 5 years ago
Emi Simpson 76ccff80f9
Fixed failure of regex to match @imports 5 years ago
Emi Simpson 45335d7507
Support links in style= attributes 5 years ago
Emi Simpson a4743ca92f
Respect the --no-images flag while parsing CSS 5 years ago
Emi Simpson b96a777e8a
Merge commit '4decea7' into load-css-imports 5 years ago
Emi Simpson 4decea716c
Fixed css replacement with more than one linked asset 5 years ago
Emi Simpson 695a787206
Moved regex compilation to lazy_static 5 years ago
Emi Simpson 90e6cb1c45
Prevent crash on URLs delimited by single quotes 5 years ago
Emi Simpson 7412d663e0
Use a slightly more efficient .replace_range() instead of cloning the string twice 5 years ago
Emi Simpson 8646af6e9f
removed debug code (woops sorry) 5 years ago
Emi Simpson de383c94b1
Applied rustfmt 5 years ago
Emi Simpson ab65b44f0d
Cleaned up some overcomplicated code 5 years ago
Emi Simpson 13bacb4320
EMPTY_STRING no longer used 5 years ago
Emi Simpson d574e9a5da
Added support for <style> tags 5 years ago
Emi Simpson 1de0fc0961
Add warning and fallback when parsing a rel=stylesheet link 5 years ago
Emi Simpson ebbf755e09
Fixed misleading variable name 5 years ago
Emi Simpson d3956a7905
Made merge compatible with Y2Z/master 5 years ago
Emi Simpson ef7ddcd434
Added fallback to absolute URL on failure to resolve CSS stylesheet @imports 5 years ago
Emi Simpson 11bbfc0851
Added support for recursively nested css @imports 5 years ago
Emi Simpson a2bf7e3345
Fixed some errors detecting, parsing, and transforming urls in `resolve_css_imports` 5 years ago
voila 1ff5e91087 Use HashMap as cache to minimize the number of HTTP requests (#75)
Use HashMap as cache to minimize the number of HTTP requests
5 years ago
knidarkness 550e4cc83f Fixed formatting 5 years ago
knidarkness 5443c0cc3f Added loading of the links given as url(...) in css files 5 years ago
robatipoor 55fe523a1c refactor utils functions 5 years ago
robatipoor 2e48ea90e1 move argument parser section to args mod 5 years ago
Sunshine 0896f2e214 Properly handle 30x redirects 5 years ago
Vincent Flyson 3948ea3aa0 Improve code structure 5 years ago
Vincent Flyson eec05767cf Add support for poster attribute 5 years ago
Vincent Flyson 88a230872c Add CSP isolation, no CSS, and no iframe options 5 years ago
Vincent Flyson 04cbbefafa Ignore empty src in images, accept fluid icons 5 years ago
Vincent Flyson 824b418f80 Add more tests 5 years ago
Vincent Flyson 2251c086c2
Merge branch 'master' into ignore-ca 5 years ago
Vincent Flyson 02b717ae54 Add flag to ignore errors related to TLS certificates 5 years ago
Vincent Flyson 1329dbe6f8 Ignore iframes with empty src 5 years ago
Vincent Flyson 2b96f9a32a Add flag for silent mode 5 years ago
Vincent Flyson 50f1ba1ce8 Treat network errors as empty results 5 years ago
Vincent Flyson 0b2e1d0746 Remove compiler warning 5 years ago
Vincent Flyson 2c0037fd51
Merge pull request #37 from Y2Z/ignore-other-protocols
Avoid modifying non-HTTP anchor hrefs
5 years ago
Vincent Flyson 3f79068d7d Avoid modifying non-HTTP anchor hrefs 5 years ago
Vincent Flyson 080ed50264
Merge pull request #36 from Y2Z/no-images-no-icons
Ignore icons if told to avoid images
5 years ago
Vincent Flyson b596b5bb7d Ignore icons if told to avoid images 5 years ago
Vincent Flyson c480fcb70f Add basic verbose output 5 years ago
Vincent Flyson e2ab05a323 Fix name of a test function 5 years ago
Vincent Flyson d49e825777 Add support for picture elements 5 years ago
Vincent Flyson d8d6437a15
Merge branch 'master' into author-robatipoor 5 years ago
Vincent Flyson c34d77d5d8 Revamp resolve_url() and improve code format 5 years ago
Vincent Flyson 2be5b1c235 Add support for iframes 5 years ago
Vincent Flyson c0fffbb212 Get rid of mime-sniffer dependency 5 years ago
Vincent Flyson 13429e32d3 Allow HTTP redirects and preserve email links 5 years ago
Vincent Flyson d3497b5db1
Merge branch 'master' into author-robatipoor 5 years ago
Vincent Flyson 7e298b0b02 Add -u flag for custom User-Agent 5 years ago
Vincent Flyson 7b326ee9f5 Add robatipoor to list of authors 5 years ago
mahdi 14e9e6facc cargo clippy 5 years ago
Vincent Flyson 54fdd890f9 Parse Cargo.toml for strings 5 years ago
Vincent Flyson 54ae61b728
Merge pull request #15 from Y2Z/increase-timeout
Increase request timeout to 10 seconds
5 years ago
Vincent Flyson 1480b20cb6
Merge pull request #14 from robatipoor/master
some code refactor
5 years ago
Vincent Flyson fde715a27e Increase request timeout to 10 seconds 5 years ago
mahdi b25b6765c1 some code refactor 5 years ago
Vincent Flyson a891782ce4 Add option to exclude images 5 years ago
Vincent Flyson ffa60a0ce2 Convert action attribute to full URL for FORM tags 5 years ago
Vincent Flyson cc09c2ba01 Remove leftover line 5 years ago
Vincent Flyson 143c396c71 Rewrite program in Rust 5 years ago