Archives
/
OnionIngestor
mirror of https://github.com/danieleperera/OnionIngestor
2
0
Fork 0
Code Issues Projects Releases Wiki Activity
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
dependabot/pip/requirements/certifi-2022.12.7
dependabot/pip/requirements/lxml-4.9.1
dependabot/pip/requirements/urllib3-1.26.5
dependabot/pip/requirements/pyyaml-5.4
development-docker
master
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'development-docker'
${ noResults }
OnionIngestor/setup.md

1.3 KiB
Raw Permalink Blame History

Install following docker images

splash javascript render browser

sudo docker pull scrapinghub/splash

sudo docker run -d -p 8050:8050 --net="host" scrapinghub/splash

sudo docker run -it -p 8050:8050 --net="host" \
      -v /home/<USER>/splash/proxy-profiles:/etc/splash/proxy-profiles \
      -v /home/<USER>/splash/js-profiles:/etc/splash/js-profiles \
      scrapinghub/splash \
      -v2

Fix for issue proxy not working

https://github.com/scrapinghub/splash/issues/268

must add --net="host" In docker localhost doesn't refer to 127.0.0.1

alpine-tor image Lots of IP addresses. One single endpoint for your client. Load-balancing by HAproxy.

sudo docker pull zeta0/alpine-tor:latest

sudo docker run -d -p 5566:5566 -p 2090:2090 -e tors=5 zeta0/alpine-tor

create file in /home//splash/proxy-profiles/default.ini

[proxy]
host=localhost
port=9050
type=SOCKS5

Tests

wget 'http://localhost:8050/render.png?url=http://ifconfig.me' -O test_ifconfig.png

wget 'http://localhost:8050/render.png?url=http://archivecaslytosk.onion/' -O test_onion.png

curl -x socks5h://localhost:5566 http://archivecaslytosk.onion/

Docker useful commands

docker ps -aq

docker stop $(docker ps -aq)

docker rm $(docker ps -aq)

docker rmi $(docker images -q)