Compare commits

...

50 Commits

Author SHA1 Message Date
G. D. M. 21367e9655 Update README.md
Fixed path to build-bitcoin.sh
10 years ago
G. D. M. 6599503026 Update README.md 10 years ago
gdm85 82125a11e7 Added inputs for 0.9.3rc2
Locked gitian-builder to working commit
Removed nl_NL.UTF-8 from locale.gen
Do not overwrite already existing downloaded input sources
Use COPY in place of ADD in Dockerfile's
10 years ago
gdm85 53929f3dae Added script to fix auto-published Docker container ports on host 10 years ago
gdm85 e22672c5eb Script to show summary of allocated ports of a container 10 years ago
G. D. M. f2189b3770 Typo fixes/wording improvements 10 years ago
G. D. M. ce43733dee A few syntax fixes 10 years ago
G. D. M. f92e986d65 Merge pull request #1 from fanquake/patch-1
Correct link to gitian bitcoin host container
10 years ago
Michael Ford 46d5150e5c Correct link to gitian bitcoin host container 10 years ago
gdm85 ae5663431c * added descriptors/inputs/hashes for 0.9.2 (same as 0.9.2rc2)
* fixed issue with existing input removal
* remove part of patch that was merged upstream
* more consistent check for gitian host VM creation
10 years ago
gdm85 07e59b4330 * always remove destination of input downloads
* fixed move command after gbuilds
10 years ago
gdm85 2b47a4fb14 * fixed gitian patch (line was truncated)
* sleep after docker commit
10 years ago
gdm85 71c5cb83f4 * moved custom gitian patch to gitian-host
* avoid chown long time operation by chown'ing specific files
* use correct list filenames
* removed libpng (not used for Linux builds)
* use docker kill instead of stop
10 years ago
gdm85 85b250b184 * support different descriptors across versions 10 years ago
gdm85 89ef983e5a * parallelize wget downloads of inputs
* removed extra environment cleanup from bash_profile and added proper gitian-builder patch
* use gitian-host-vms for gitian-bitcoin-host
* fixed issue at stopping container after generation of base VMs
10 years ago
gdm85 9fd349ead3 * added sources for 0.9.2rc2 10 years ago
gdm85 a592e877e5 * ask for answer on same line 10 years ago
gdm85 03235dd1cd * script for creation of trusty-kbuilder image 10 years ago
gdm85 1a2312e96a * added Ubuntu Archive reference keyring
* added script to debootstrap Ubuntu Trusty
* added Dockerfile for Ubuntu Trusty kernel builder
10 years ago
gdm85 842835ac98 * removed reference to original article (dead) 10 years ago
gdm85 46e62ffc1c * added Debian Archive keyring
* added questions to user about keyring differences/trust
* added documentation notes about the Debian Archive keyring
10 years ago
gdm85 80b39dd737 * conceal also docker host IP in docker-iptables
* added docker-hosts for simplified hosts listing
10 years ago
gdm85 3f4f73920f * added generation of gitian-host-vms image
* simplified necessary steps
* fixed issue with parallel package missing
10 years ago
gdm85 c58eb39b9a * add message about location of output files 10 years ago
gdm85 d70a14d4fa * verify md5 hashes before using files for gbuild 10 years ago
gdm85 f6da49cb29 * added missing file 10 years ago
gdm85 54ef564c69 * proper fix for SSH environment pollution (see https://bugzilla.mindrot.org/show_bug.cgi?id=1285) 10 years ago
gdm85 810ad9b06d * ignore failure at deleting (unknown reason) 10 years ago
gdm85 547d19adc3 * added workaround for Docker glitch with file permissions
* a few improvements to documentation, describing steps needed to build bitcoin
10 years ago
gdm85 7e77b76a01 * unused file 10 years ago
gdm85 d2ba944cd8 * added support for newer versions of bitcoin 10 years ago
G. D. M. 844b8b106d * added preamble 10 years ago
gdm85 7a45e1491b * fix typo & enlarge 10 years ago
gdm85 cee88e6132 * added (horrible) diagram 10 years ago
G. D. M. 203019028b * improve readability 10 years ago
G. D. M. 52242c30c3 * improvements/fix typos 10 years ago
gdm85 df4359102c * added documentation & scripts for full automation of gitian bitcoin builds 10 years ago
gdm85 59dea12791 * put base VM creation script in gitian-host itself 10 years ago
G. D. M. 6d5563e4de * specific documentation for building bitcoin with gitian-host 10 years ago
gdm85 8c871798cd * bitcoin gitian-host scripts (0.9.1) 10 years ago
gdm85 43c9f087d7 * added missing patch
* added changes for previous commit
10 years ago
gdm85 7aa6b49a8d * added documentation about building bitcoin 0.9.1
* a few fixes to scripts to overcome docker glitches
10 years ago
gdm85 da1499014f * completed documentation
* added prefix to image tags
10 years ago
gdm85 21c3c34b29 * final touches 10 years ago
gdm85 91a0b71512 * first draft of scripts for gitian host creation 10 years ago
gdm85 462e1290fb * adding scripts for creation/spawning of gitian hosts 10 years ago
gdm85 7d5f65f6a3 * script to build a base Debian Wheezy docker image 10 years ago
gdm85 45a3028f23 * added README.md
* reorganizing
10 years ago
Giuseppe Mazzotta 565d474045 * a few improvements 10 years ago
Giuseppe Mazzotta 8e4e943d3b * skip containers with no network 10 years ago

@ -0,0 +1,7 @@
gdm85's bay
===========
This repository contains various scripts and documentation I have gathered.
<a rel="license" href="http://creativecommons.org/licenses/by-sa/2.0/"><img alt="Creative Commons License" style="border-width:0" src="http://i.creativecommons.org/l/by-sa/2.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-sa/2.0/">Creative Commons Attribution-ShareAlike 2.0 Generic License</a>.

@ -1,24 +0,0 @@
#!/bin/bash
## docker-iptables
##
## @author gdm85
## script to show iptables rules with docker names
## can be used also to detect problems with dead containers and stale iptable rules
## supports standard iptables-save syntax
#
function replace_iptables() {
local CID
local SEDCMD=""
for CID in $(docker ps -q -a); do
local NAME=$(docker inspect --format '{{ .Name }}' $CID | awk '{ print substr($0, 2, length($0)-1) }')
local IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' $CID)
SEDCMD="$SEDCMD -e s!$IP/32!${NAME}!g"
done
sed $SEDCMD
}
iptables-save $@ | replace_iptables

@ -0,0 +1,16 @@
## gitian-bitcoin-host
##
## VERSION 0.1.0
##
FROM gdm85/gitian-host-vms
MAINTAINER Giuseppe Mazzotta "gdm85@users.noreply.github.com"
COPY input-sources/ /home/debian/input-sources/
COPY bin/build-bitcoin.sh /home/debian/
COPY bin/sign.sh /home/debian/
WORKDIR /home/debian
RUN chown -R debian.debian input-sources build-bitcoin.sh sign.sh

@ -0,0 +1,55 @@
Building bitcoin with a gitian-builder Docker container
=======================================================
This image allows automated Gitian builds of bitcoin core using a Docker container.
Before proceeding make sure you have created the necessary *gdm85/wheezy*, *gdm85/gitian-host* and *gdm85/gitian-host-vms* images, see [these instructions](../gitian-host/README.md).
Afterwards you can create the *gdm85/gitian-bitcoin-host* image by running [scripts/create-gitian-bitcoin-host.sh](../scripts/create-gitian-bitcoin-host.sh).
You can submit the source lists for other bitcoin versions as a patch or pull request, see directory [input-sources/](input-sources/) for currently available versions.
Do not forget to read also the [Preamble here](../gitian-host/README.md#preamble) to correctly use Gitian builder and these provided scripts.
Spawning a container
--------------------
You can spawn a new container for Gitian bitcoin builds with:
- [scripts/spawn-gitian-bitcoin-host.sh](scripts/spawn-gitian-bitcoin-host.sh)
This script will create the running docker container and provide details about how to connect via SSH to the container, example:
```
$ scripts/spawn-gitian-bitcoin-host.sh
You can now SSH into container 3bc0d0611374ca4d4730fd5fb1067808b1bcfd072ec7cf029393a7fd99ec856e:
ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no debian@172.17.0.3
$
```
You can use this specific SSH command line to get a shell in the container and proceed to next steps.
Preparing the gitian environment
--------------------------------
To initiate a gitian build of bitcoin you will run:
- [./build-bitcoin.sh](bin/build-bitcoin.sh) 0.9.3
Notice the parameter 0.9.3, that is the version we are going to build and must be available in [input-sources](input-sources/).
[build-bitcoin.sh](bin/build-bitcoin.sh) is a script that will download & build all the dependencies and then bitcoin itself, for both i386 and amd64 Linux architectures.
Signing
-------
Once you have completed the build of bitcoin, you will be ready to perform the signing operation; before doing that you should verify that signatures are matching with those of [other developers](https://github.com/bitcoin/gitian.sigs) by peeking inside *~/gitian.sigs* of the running container.
Only the out_manifest signatures do matter for this purpose.
In order to sign you have to either put your private key in the container's *~/.gnupg* or perform the signing externally, at your option.
If you have the private key in the container (also displayed by `gpg -K`), then you can use the [sign.sh](bin/sign.sh) script that is already provided, otherwise
run it (with failure) and then copy the *~/gitian.sigs* directory to another machine to apply the GPG signature.
Submitting your signature
-------------------------
If everything went well, you can fork the [gitian sigs repo](https://github.com/bitcoin/gitian.sigs), commit your signatures and submit a pull request for inclusion.

@ -0,0 +1,45 @@
#!/bin/bash
## automatic Gitian build of bitcoin
## @author gdm85
## @version 0.3.0
## see also https://github.com/gdm85/tenku/blob/master/docker/gitian-bitcoin-host/
##
#
if [[ ! $# -eq 1 ]]; then
echo "Please specify version" 1>&2
exit 1
fi
VERSION="$1"
if [ ! -d bitcoin ]; then
git clone https://github.com/bitcoin/bitcoin.git || exit $?
fi
cd bitcoin && \
git checkout v${VERSION} && \
cd ../gitian-builder && \
mkdir -p inputs && cd inputs/ || exit $?
## get each dependency
## they are validated afterwards by gbuild
while read -r URL FNAME; do
if [ -z "$URL" ]; then
continue
fi
if [ ! -f $FNAME ]; then
echo "echo 'Downloading $FNAME'"
echo "wget -q --no-check-certificate '$URL' -O '$FNAME' || echo 'Failed to download $FNAME from $URL'"
fi
done < ../../input-sources/${VERSION}-inputs.txt | parallel -j10 || exit $?
## verify that all sources are correct before continuing
md5sum -c < ../../input-sources/${VERSION}-inputs.md5 && \
cd .. && \
for DESC in $(<../input-sources/${VERSION}-descriptors.txt); do
./bin/gbuild ../bitcoin/contrib/gitian-descriptors/${DESC}.yml && \
mv -v $(find build/out -type f -name '*gz' -o -name '*.zip') inputs/ || exit $?
done && \
./bin/gbuild --commit bitcoin=v${VERSION} ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml && \
echo "Completed successfully." && \
echo "The output files are in: gitian-builder/build/out/"

@ -0,0 +1,13 @@
-#!/bin/bash
set -e
if [[ ! $# -eq 2 ]]; then
echo "Please specify version and signer id" 1>&2
exit 1
fi
VERSION="$1"
SIGNER="$2"
cd gitian-builder
./bin/gsign --signer $SIGNER --release ${VERSION} --destination ../gitian.sigs/ ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml

@ -0,0 +1,8 @@
d6eef4b4cacb2183f2bf265a5a03a354 boost_1_55_0.tar.bz2
a14a5486d6b4891d2434039a0ed4c5b7 db-4.8.30.NC.tar.gz
065bf20a20ebe605c675b7a5aaef340a miniupnpc-1.8.tar.gz
de62b43dfcd858e66a74bee1c834e959 openssl-1.0.1g.tar.gz
a72001a9067a4c2c4e0e836d0f92ece4 protobuf-2.5.0.tar.bz2
89a90a3b8cbca60ac412b2e0e0c776e7 qrencode-3.4.3.tar.bz2
228b6384dfd7272de00fd8b2c144fecd qt-everywhere-opensource-src-5.2.0.tar.gz
44d667c142d7cda120332623eab69f40 zlib-1.2.8.tar.gz

@ -0,0 +1,8 @@
http://miniupnp.free.fr/files/download.php?file=miniupnpc-1.8.tar.gz miniupnpc-1.8.tar.gz
https://www.openssl.org/source/openssl-1.0.1g.tar.gz openssl-1.0.1g.tar.gz
http://download.oracle.com/berkeley-db/db-4.8.30.NC.tar.gz db-4.8.30.NC.tar.gz
http://zlib.net/zlib-1.2.8.tar.gz zlib-1.2.8.tar.gz
https://fukuchi.org/works/qrencode/qrencode-3.4.3.tar.bz2 qrencode-3.4.3.tar.bz2
https://downloads.sourceforge.net/project/boost/boost/1.55.0/boost_1_55_0.tar.bz2 boost_1_55_0.tar.bz2
https://download.qt-project.org/official_releases/qt/5.2/5.2.0/single/qt-everywhere-opensource-src-5.2.0.tar.gz qt-everywhere-opensource-src-5.2.0.tar.gz
https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.bz2 protobuf-2.5.0.tar.bz2

@ -0,0 +1,11 @@
d6eef4b4cacb2183f2bf265a5a03a354 boost_1_55_0.tar.bz2
efe08e2f3ca478486037b053acd512e9 cdrkit-1.1.11.tar.gz
cea2d01b3206e92a8df7b079935c070b clang-llvm-3.2-x86-linux-ubuntu-12.04.tar.gz
a14a5486d6b4891d2434039a0ed4c5b7 db-4.8.30.NC.tar.gz
5ef3ba321e6df72d6519b728b292073e miniupnpc-1.9.tar.gz
8d6d684a9430d5cc98a62a5d8fbda8cf openssl-1.0.1h.tar.gz
a72001a9067a4c2c4e0e836d0f92ece4 protobuf-2.5.0.tar.bz2
89a90a3b8cbca60ac412b2e0e0c776e7 qrencode-3.4.3.tar.bz2
8ac880cc07a130c39607b65efd5e1421 qt-everywhere-opensource-src-4.6.4.tar.gz
228b6384dfd7272de00fd8b2c144fecd qt-everywhere-opensource-src-5.2.0.tar.gz
44d667c142d7cda120332623eab69f40 zlib-1.2.8.tar.gz

@ -0,0 +1,11 @@
http://miniupnp.free.fr/files/download.php?file=miniupnpc-1.9.tar.gz miniupnpc-1.9.tar.gz
https://www.openssl.org/source/openssl-1.0.1h.tar.gz openssl-1.0.1h.tar.gz
http://download.oracle.com/berkeley-db/db-4.8.30.NC.tar.gz db-4.8.30.NC.tar.gz
http://zlib.net/zlib-1.2.8.tar.gz zlib-1.2.8.tar.gz
https://fukuchi.org/works/qrencode/qrencode-3.4.3.tar.bz2 qrencode-3.4.3.tar.bz2
https://downloads.sourceforge.net/project/boost/boost/1.55.0/boost_1_55_0.tar.bz2 boost_1_55_0.tar.bz2
https://download.qt-project.org/official_releases/qt/5.2/5.2.0/single/qt-everywhere-opensource-src-5.2.0.tar.gz qt-everywhere-opensource-src-5.2.0.tar.gz
https://download.qt-project.org/archive/qt/4.6/qt-everywhere-opensource-src-4.6.4.tar.gz qt-everywhere-opensource-src-4.6.4.tar.gz
https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.bz2 protobuf-2.5.0.tar.bz2
http://cdrkit.org/releases/cdrkit-1.1.11.tar.gz cdrkit-1.1.11.tar.gz
http://llvm.org/releases/3.2/clang+llvm-3.2-x86-linux-ubuntu-12.04.tar.gz clang-llvm-3.2-x86-linux-ubuntu-12.04.tar.gz

@ -0,0 +1,11 @@
d6eef4b4cacb2183f2bf265a5a03a354 boost_1_55_0.tar.bz2
efe08e2f3ca478486037b053acd512e9 cdrkit-1.1.11.tar.gz
cea2d01b3206e92a8df7b079935c070b clang-llvm-3.2-x86-linux-ubuntu-12.04.tar.gz
a14a5486d6b4891d2434039a0ed4c5b7 db-4.8.30.NC.tar.gz
5ef3ba321e6df72d6519b728b292073e miniupnpc-1.9.tar.gz
8d6d684a9430d5cc98a62a5d8fbda8cf openssl-1.0.1h.tar.gz
a72001a9067a4c2c4e0e836d0f92ece4 protobuf-2.5.0.tar.bz2
89a90a3b8cbca60ac412b2e0e0c776e7 qrencode-3.4.3.tar.bz2
8ac880cc07a130c39607b65efd5e1421 qt-everywhere-opensource-src-4.6.4.tar.gz
228b6384dfd7272de00fd8b2c144fecd qt-everywhere-opensource-src-5.2.0.tar.gz
44d667c142d7cda120332623eab69f40 zlib-1.2.8.tar.gz

@ -0,0 +1,11 @@
http://miniupnp.free.fr/files/download.php?file=miniupnpc-1.9.tar.gz miniupnpc-1.9.tar.gz
https://www.openssl.org/source/openssl-1.0.1h.tar.gz openssl-1.0.1h.tar.gz
http://download.oracle.com/berkeley-db/db-4.8.30.NC.tar.gz db-4.8.30.NC.tar.gz
http://zlib.net/zlib-1.2.8.tar.gz zlib-1.2.8.tar.gz
https://fukuchi.org/works/qrencode/qrencode-3.4.3.tar.bz2 qrencode-3.4.3.tar.bz2
https://downloads.sourceforge.net/project/boost/boost/1.55.0/boost_1_55_0.tar.bz2 boost_1_55_0.tar.bz2
https://download.qt-project.org/official_releases/qt/5.2/5.2.0/single/qt-everywhere-opensource-src-5.2.0.tar.gz qt-everywhere-opensource-src-5.2.0.tar.gz
https://download.qt-project.org/archive/qt/4.6/qt-everywhere-opensource-src-4.6.4.tar.gz qt-everywhere-opensource-src-4.6.4.tar.gz
https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.bz2 protobuf-2.5.0.tar.bz2
http://cdrkit.org/releases/cdrkit-1.1.11.tar.gz cdrkit-1.1.11.tar.gz
http://llvm.org/releases/3.2/clang+llvm-3.2-x86-linux-ubuntu-12.04.tar.gz clang-llvm-3.2-x86-linux-ubuntu-12.04.tar.gz

@ -0,0 +1,10 @@
d6eef4b4cacb2183f2bf265a5a03a354 boost_1_55_0.tar.bz2
cea2d01b3206e92a8df7b079935c070b clang-llvm-3.2-x86-linux-ubuntu-12.04.tar.gz
a14a5486d6b4891d2434039a0ed4c5b7 db-4.8.30.NC.tar.gz
56b4ac6a7a3e7d64f8cfeeac87b212bc miniupnpc-1.9.20140701.tar.gz
c8dc151a671b9b92ff3e4c118b174972 openssl-1.0.1i.tar.gz
a72001a9067a4c2c4e0e836d0f92ece4 protobuf-2.5.0.tar.bz2
89a90a3b8cbca60ac412b2e0e0c776e7 qrencode-3.4.3.tar.bz2
8ac880cc07a130c39607b65efd5e1421 qt-everywhere-opensource-src-4.6.4.tar.gz
228b6384dfd7272de00fd8b2c144fecd qt-everywhere-opensource-src-5.2.0.tar.gz
44d667c142d7cda120332623eab69f40 zlib-1.2.8.tar.gz

@ -0,0 +1,10 @@
http://miniupnp.free.fr/files/download.php?file=miniupnpc-1.9.20140701.tar.gz miniupnpc-1.9.20140701.tar.gz
https://www.openssl.org/source/openssl-1.0.1i.tar.gz openssl-1.0.1i.tar.gz
http://download.oracle.com/berkeley-db/db-4.8.30.NC.tar.gz db-4.8.30.NC.tar.gz
http://zlib.net/zlib-1.2.8.tar.gz zlib-1.2.8.tar.gz
https://fukuchi.org/works/qrencode/qrencode-3.4.3.tar.bz2 qrencode-3.4.3.tar.bz2
https://downloads.sourceforge.net/project/boost/boost/1.55.0/boost_1_55_0.tar.bz2 boost_1_55_0.tar.bz2
https://download.qt-project.org/official_releases/qt/5.2/5.2.0/single/qt-everywhere-opensource-src-5.2.0.tar.gz qt-everywhere-opensource-src-5.2.0.tar.gz
https://download.qt-project.org/archive/qt/4.6/qt-everywhere-opensource-src-4.6.4.tar.gz qt-everywhere-opensource-src-4.6.4.tar.gz
https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.bz2 protobuf-2.5.0.tar.bz2
http://llvm.org/releases/3.2/clang+llvm-3.2-x86-linux-ubuntu-12.04.tar.gz clang-llvm-3.2-x86-linux-ubuntu-12.04.tar.gz

@ -0,0 +1,75 @@
## gitian-host
##
## VERSION 0.1.0
##
## gitian host privileged container
## see also https://github.com/gdm85/tenku/tree/master/docker/gitian-host/README.md
##
#
## this image must be debootstrapped with build-wheezy.sh
FROM gdm85/wheezy
MAINTAINER Giuseppe Mazzotta "gdm85@users.noreply.github.com"
ENV TERM xterm
ENV DEBIAN_FRONTEND noninteractive
## some core packages
RUN apt-get update && apt-get install -y htop tmux less locales openssh-server
## these 2 files can be customized
COPY etc/timezone /etc/
COPY etc/locale.gen /etc/
RUN dpkg-reconfigure locales && dpkg-reconfigure tzdata
## continue installation of gitian-builder packages
RUN echo "deb http://http.debian.net/debian wheezy main contrib non-free" > /etc/apt/sources.list.d/non-free.list
RUN apt-get install -y iptables ca-certificates sudo bridge-utils git python-cheetah qemu-utils parted kpartx ruby apt-cacher-ng lxc parallel patch
## disable password logins, reverse DNS lookups & privilege separation (due to chroot issues on latest docker)
RUN sed -i -e 's/^#PasswordAuthentication yes/PasswordAuthentication no/' -e 's/^UsePrivilegeSeparation.*$/UsePrivilegeSeparation no/' /etc/ssh/sshd_config && echo 'UseDNS no' >> /etc/ssh/sshd_config
EXPOSE 22
# this version of Debian needs to run as root, so make sure
# that the build script can exectute it without providing a password
RUN echo "%sudo ALL=NOPASSWD: /usr/bin/lxc-start" > /etc/sudoers.d/gitian-lxc
RUN echo 'debian ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/debian-sudo
RUN useradd -m -s /bin/bash debian && adduser debian sudo && mkdir /home/debian/.ssh
## provide authorized keys to allow connections inside host
## NOTE: this is placed in debian user directory by setup.sh script
## reason is that there is a Docker glitch with file permissions when adding this
COPY authorized_keys /root/
WORKDIR /home/debian
## download in clear HTTP & check md5 right afterwards
RUN wget http://archive.ubuntu.com/ubuntu/pool/universe/v/vm-builder/vm-builder_0.12.4+bzr489.orig.tar.gz && echo "ec12e0070a007989561bfee5862c89a32c301992dd2771c4d5078ef1b3014f03 vm-builder_0.12.4+bzr489.orig.tar.gz" | sha256sum -c
RUN tar -zxvf vm-builder_0.12.4+bzr489.orig.tar.gz && cd vm-builder-0.12.4+bzr489 && python setup.py install
## these must always be loaded when you login as 'debian' user
COPY bash_profile /home/debian/.bash_profile
## script used to build base VMs
COPY bin/build-base-vms.sh /home/debian/
RUN git clone https://github.com/devrandom/gitian-builder.git && cd gitian-builder && git checkout ffd04a46a2b3c1277937c939cdb151fcf0eac613
COPY gitian.patch /home/debian/
## patch to allow paralle creation of VMs
RUN cd gitian-builder && patch -p1 < ../gitian.patch && rm ../gitian.patch
RUN chown -R debian.debian .bash_profile . && chown -R apt-cacher-ng.apt-cacher-ng /var/cache/apt-cacher-ng/
## suggested: run this image with mounted volumes for pre-downloaded inputs and apt-cacher-ng cache
## -v inputs:/home/debian/gitian-builder/inputs -v apt-cacher-ng:/var/cache/apt-cacher-ng
##NOTE: if you do use them, then not forget to set proper user rights for these mounted volumes
## all necessary setup instructions (SSH, LXC)
COPY bin/setup.sh /usr/local/bin/setup.sh
## on run failure, check logs to see if setup failed. on success connect via SSH
CMD /usr/local/bin/setup.sh && /etc/init.d/apt-cacher-ng start && /usr/sbin/sshd -D -e

@ -0,0 +1,60 @@
Gitian host docker container
============================
The provided [Dockerfile](http://docs.docker.io/reference/builder/) allows to generate a [gitian-builder](https://gitian.org/) host image, that can subsequently be used for reproducible builds using LXC VMs.
How this works:
<img src="diagram.png">
Some of the discussions leading to the creation of this set of Dockerfiles/scripts are available on [this issue](https://github.com/devrandom/gitian-builder/issues/53).
Preamble
--------
It is **necessary** to read the provided scripts and understand what they do before starting to use them.
The reason is that aiming at the goal of creating a Gitian build (deterministic) also means that your build is untampered, thus trust shall be correctly attributed during the process.
For example, in this repository I provide the [Debian Archive keyring](../keyrings/debian-archive-keyring.gpg) that is used for the original debootstrap, however you **must** verify its authenticity and that it is exactly [as provided officially by Debian](https://packages.debian.org/wheezy/all/debian-archive-keyring/download)
in order to continue using a trusted chain of systems.
See also:
- https://gitian.org/
- https://en.wikipedia.org/wiki/Web_of_trust
- http://www.dwheeler.com/trusting-trust/
- https://www.debian.org/
- https://wiki.debian.org/SecureApt
- https://www.docker.io/
- http://www.ubuntu.com/
How to build the image
----------------------
Images have not been pushed to my [Docker Registry](https://index.docker.io/) account, this is on purpose because even if generated images have my repository prefix ('gdm85/') you are supposed to create them from scratch with the provided (auditable) scripts.
**NOTE:** you must have *debootstrap* on your real host to run this script successfully, and also make sure you have a keyring with APT keys, see also https://wiki.debian.org/SecureApt
First steps:
- run [scripts/build-wheezy.sh](../scripts/build-wheezy.sh) to get a Debian Wheezy image debootstrapped from Debian repositories.
- run [scripts/create-gitian-host.sh](../scripts/create-gitian-host.sh), this will simply build the Dockerfile that installs the few necessary dependencies inside the prepared image
- generate a second image with the i386 and amd64 VMs byu using [build-base-vms.sh](../scripts/build-base-vms.sh)
**NOTE:** when I say "run", what I really mean is "read the script, study it for your own learning purposes, then run it"
After steps above you will have prepared a full Gitian builder environment for deterministic builds.
The image that contains the VMs is called *gdm85/gitian-host-vms*; in future you can spawn containers with this image for new Gitian builder environments.
Example:
```
$ scripts/spawn-gitian-host.sh
You can now SSH into container 8a955ff5607b62d4c295745f27bbc38f2e8e011ea93053e641617d50ad2aa5a2:
ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no debian@172.17.0.2
$
```
This will create a privileged running container that you can access with the SSH command displayed.
Derived images
--------------
A [bitcoin gitian host container](../gitian-bitcoin-host/README.md) is available.
Credits
-------
Thanks to jpetazzo for [dind](https://github.com/jpetazzo/dind) and to #docker & bitcoin-dev IRC users for the help&assistance!

@ -0,0 +1,3 @@
export USE_LXC=1
export GITIAN_HOST_IP=10.0.3.2
export LXC_GUEST_IP=10.0.3.5

@ -0,0 +1,37 @@
#!/bin/bash
set -e
source ~/.bash_profile
cd gitian-builder
mkdir -p var
if [ ! -e var/id_dsa ]; then
ssh-keygen -t dsa -f var/id_dsa -N ""
fi
export MIRROR_HOST=$GITIAN_HOST_IP
SUITE=precise
## build both VMs in parallel
echo "Now building i386 and amd64 VMs..."
echo -e "MIRROR_HOST=$GITIAN_HOST_IP bin/make-base-vm --lxc --arch i386 --suite $SUITE\nMIRROR_HOST=$GITIAN_HOST_IP bin/make-base-vm --lxc --arch amd64 --suite $SUITE" \
| parallel -j2 || exit $?
function ext_partition() {
local OUT=$1
echo Extracting $OUT partition for lxc
qemu-img convert $OUT.qcow2 $OUT.raw
loop=`sudo kpartx -av $OUT.raw|sed -n '/loop.p1/{s/.*loop\(.\)p1.*/\1/;p}'`
sudo cp --sparse=always /dev/mapper/loop${loop}p1 $OUT
sudo chown $USER $OUT
## following 2 lines are a sloppy hack to an unknown problem with kpartx
sudo sync
sleep 5
## these are silenced because if former fails, second doesn't and viceversa
sudo kpartx -d /dev/loop$loop 2>/dev/null
sudo rm /dev/mapper/loop${loop}p1 2>/dev/null
rm -f $OUT.raw
}
ext_partition base-${SUITE}-i386 && \
ext_partition base-${SUITE}-amd64 || exit $?

@ -0,0 +1,124 @@
#!/bin/bash
## @author gdm85
## this script is an adapted version of jpetazzo's original:
## https://github.com/jpetazzo/dind/blob/master/wrapdocker
## (thus most of the comment are his voice)
#
## prevent (re)starting of sshd
## we want to use sshd as our container process
echo -e "#!/bin/sh\nexit 101" > /usr/sbin/policy-rc.d && \
chmod +x /usr/sbin/policy-rc.d
## regenerate host keys
/bin/rm -v /etc/ssh/ssh_host_* && \
dpkg-reconfigure -f noninteractive openssh-server || exit $?
## removed, in case you want to install other packages at container-time
rm /usr/sbin/policy-rc.d
# First, make sure that cgroups are mounted correctly.
CGROUP=/sys/fs/cgroup
[ -d $CGROUP ] ||
mkdir $CGROUP
mountpoint -q $CGROUP ||
mount -n -t tmpfs -o uid=0,gid=0,mode=0755 cgroup $CGROUP || {
echo "Could not make a tmpfs mount. Did you use -privileged?"
exit 1
}
if [ -d /sys/kernel/security ] && ! mountpoint -q /sys/kernel/security
then
mount -t securityfs none /sys/kernel/security || {
echo "Could not mount /sys/kernel/security."
echo "AppArmor detection and -privileged mode might break."
exit 2
}
fi
# Mount the cgroup hierarchies exactly as they are in the parent system.
for SUBSYS in $(cut -d: -f2 /proc/1/cgroup)
do
[ -d $CGROUP/$SUBSYS ] || mkdir $CGROUP/$SUBSYS
mountpoint -q $CGROUP/$SUBSYS ||
mount -n -t cgroup -o $SUBSYS cgroup $CGROUP/$SUBSYS
# The two following sections address a bug which manifests itself
# by a cryptic "lxc-start: no ns_cgroup option specified" when
# trying to start containers withina container.
# The bug seems to appear when the cgroup hierarchies are not
# mounted on the exact same directories in the host, and in the
# container.
# Named, control-less cgroups are mounted with "-o name=foo"
# (and appear as such under /proc/<pid>/cgroup) but are usually
# mounted on a directory named "foo" (without the "name=" prefix).
# Systemd and OpenRC (and possibly others) both create such a
# cgroup. To avoid the aforementioned bug, we symlink "foo" to
# "name=foo". This shouldn't have any adverse effect.
echo $SUBSYS | grep -q ^name= && {
NAME=$(echo $SUBSYS | sed s/^name=//)
ln -s $SUBSYS $CGROUP/$NAME
}
# Likewise, on at least one system, it has been reported that
# systemd would mount the CPU and CPU accounting controllers
# (respectively "cpu" and "cpuacct") with "-o cpuacct,cpu"
# but on a directory called "cpu,cpuacct" (note the inversion
# in the order of the groups). This tries to work around it.
[ $SUBSYS = cpuacct,cpu ] && ln -s $SUBSYS $CGROUP/cpu,cpuacct
done
# Note: as I write those lines, the LXC userland tools cannot setup
# a "sub-container" properly if the "devices" cgroup is not in its
# own hierarchy. Let's detect this and issue a warning.
grep -q :devices: /proc/1/cgroup ||
echo "WARNING: the 'devices' cgroup should be in its own hierarchy."
grep -qw devices /proc/1/cgroup ||
echo "WARNING: it looks like the 'devices' cgroup is not mounted."
# Now, close extraneous file descriptors.
pushd /proc/self/fd >/dev/null
for FD in *
do
case "$FD" in
# Keep stdin/stdout/stderr
[012])
;;
# Nuke everything else
*)
eval exec "$FD>&-"
;;
esac
done
popd >/dev/null
source /home/debian/.bash_profile
## at this point environment should be ready
## bridge to be used by gitian LXC container
brctl addbr br0 && \
ifconfig br0 ${GITIAN_HOST_IP}/16 up || exit $?
##NOTE: *DO NOT* try to add eth0 to the bridge, it will kill container's networking
## temporary workaround until this bug is fixed: https://bugs.launchpad.net/ubuntu/+source/sysvinit/+bug/891045
umount /dev/shm
rmdir /dev/shm
ln -s /run/shm /dev/shm
##NOTE: this is setup here instead of Dockerfile because of a Docker glitch
AK=/root/authorized_keys
if [ -s $AK ]; then
cp $AK /home/debian/.ssh/ && \
rm $AK && \
chmod -R go-rwx /home/debian/.ssh &&
chown -R debian.debian /home/debian/.ssh || exit $?
fi
## test that debian user has access to its own .ssh (yes, Docker glitches crawling...)
su -c 'cat /home/debian/.ssh/authorized_keys' -l -- debian || exit $?
echo "Gitian host configuration for LXC guests completed successfully"

Binary file not shown.

After

Width:  |  Height:  |  Size: 21 KiB

@ -0,0 +1,446 @@
# This file lists locales that you wish to have built. You can find a list
# of valid supported locales at /usr/share/i18n/SUPPORTED, and you can add
# user defined locales to /usr/local/share/i18n/SUPPORTED. If you change
# this file, you need to rerun locale-gen.
# aa_DJ ISO-8859-1
# aa_DJ.UTF-8 UTF-8
# aa_ER UTF-8
# aa_ER@saaho UTF-8
# aa_ET UTF-8
# af_ZA ISO-8859-1
# af_ZA.UTF-8 UTF-8
# am_ET UTF-8
# an_ES ISO-8859-15
# an_ES.UTF-8 UTF-8
# ar_AE ISO-8859-6
# ar_AE.UTF-8 UTF-8
# ar_BH ISO-8859-6
# ar_BH.UTF-8 UTF-8
# ar_DZ ISO-8859-6
# ar_DZ.UTF-8 UTF-8
# ar_EG ISO-8859-6
# ar_EG.UTF-8 UTF-8
# ar_IN UTF-8
# ar_IQ ISO-8859-6
# ar_IQ.UTF-8 UTF-8
# ar_JO ISO-8859-6
# ar_JO.UTF-8 UTF-8
# ar_KW ISO-8859-6
# ar_KW.UTF-8 UTF-8
# ar_LB ISO-8859-6
# ar_LB.UTF-8 UTF-8
# ar_LY ISO-8859-6
# ar_LY.UTF-8 UTF-8
# ar_MA ISO-8859-6
# ar_MA.UTF-8 UTF-8
# ar_OM ISO-8859-6
# ar_OM.UTF-8 UTF-8
# ar_QA ISO-8859-6
# ar_QA.UTF-8 UTF-8
# ar_SA ISO-8859-6
# ar_SA.UTF-8 UTF-8
# ar_SD ISO-8859-6
# ar_SD.UTF-8 UTF-8
# ar_SY ISO-8859-6
# ar_SY.UTF-8 UTF-8
# ar_TN ISO-8859-6
# ar_TN.UTF-8 UTF-8
# ar_YE ISO-8859-6
# ar_YE.UTF-8 UTF-8
# as_IN.UTF-8 UTF-8
# ast_ES ISO-8859-15
# ast_ES.UTF-8 UTF-8
# az_AZ.UTF-8 UTF-8
# be_BY CP1251
# be_BY.UTF-8 UTF-8
# be_BY@latin UTF-8
# bem_ZM UTF-8
# ber_DZ UTF-8
# ber_MA UTF-8
# bg_BG CP1251
# bg_BG.UTF-8 UTF-8
# bn_BD UTF-8
# bn_IN UTF-8
# bo_CN UTF-8
# bo_IN UTF-8
# br_FR ISO-8859-1
# br_FR.UTF-8 UTF-8
# br_FR@euro ISO-8859-15
# bs_BA ISO-8859-2
# bs_BA.UTF-8 UTF-8
# byn_ER UTF-8
# ca_AD ISO-8859-15
# ca_AD.UTF-8 UTF-8
# ca_ES ISO-8859-1
# ca_ES.UTF-8 UTF-8
# ca_ES.UTF-8@valencia UTF-8
# ca_ES@euro ISO-8859-15
# ca_ES@valencia ISO-8859-15
# ca_FR ISO-8859-15
# ca_FR.UTF-8 UTF-8
# ca_IT ISO-8859-15
# ca_IT.UTF-8 UTF-8
# crh_UA UTF-8
# cs_CZ ISO-8859-2
# cs_CZ.UTF-8 UTF-8
# csb_PL UTF-8
# cv_RU UTF-8
# cy_GB ISO-8859-14
# cy_GB.UTF-8 UTF-8
# da_DK ISO-8859-1
# da_DK.UTF-8 UTF-8
# de_AT ISO-8859-1
# de_AT.UTF-8 UTF-8
# de_AT@euro ISO-8859-15
# de_BE ISO-8859-1
# de_BE.UTF-8 UTF-8
# de_BE@euro ISO-8859-15
# de_CH ISO-8859-1
# de_CH.UTF-8 UTF-8
# de_DE ISO-8859-1
# de_DE.UTF-8 UTF-8
# de_DE@euro ISO-8859-15
# de_LI.UTF-8 UTF-8
# de_LU ISO-8859-1
# de_LU.UTF-8 UTF-8
# de_LU@euro ISO-8859-15
# dv_MV UTF-8
# dz_BT UTF-8
# el_CY ISO-8859-7
# el_CY.UTF-8 UTF-8
# el_GR ISO-8859-7
# el_GR.UTF-8 UTF-8
# en_AG UTF-8
# en_AU ISO-8859-1
# en_AU.UTF-8 UTF-8
# en_BW ISO-8859-1
# en_BW.UTF-8 UTF-8
# en_CA ISO-8859-1
# en_CA.UTF-8 UTF-8
# en_DK ISO-8859-1
# en_DK.ISO-8859-15 ISO-8859-15
# en_DK.UTF-8 UTF-8
# en_GB ISO-8859-1
# en_GB.ISO-8859-15 ISO-8859-15
# en_GB.UTF-8 UTF-8
# en_HK ISO-8859-1
# en_HK.UTF-8 UTF-8
# en_IE ISO-8859-1
# en_IE.UTF-8 UTF-8
# en_IE@euro ISO-8859-15
# en_IN UTF-8
# en_NG UTF-8
# en_NZ ISO-8859-1
# en_NZ.UTF-8 UTF-8
# en_PH ISO-8859-1
# en_PH.UTF-8 UTF-8
# en_SG ISO-8859-1
# en_SG.UTF-8 UTF-8
# en_US ISO-8859-1
# en_US.ISO-8859-15 ISO-8859-15
en_US.UTF-8 UTF-8
# en_ZA ISO-8859-1
# en_ZA.UTF-8 UTF-8
# en_ZM UTF-8
# en_ZW ISO-8859-1
# en_ZW.UTF-8 UTF-8
# eo ISO-8859-3
# eo.UTF-8 UTF-8
# es_AR ISO-8859-1
# es_AR.UTF-8 UTF-8
# es_BO ISO-8859-1
# es_BO.UTF-8 UTF-8
# es_CL ISO-8859-1
# es_CL.UTF-8 UTF-8
# es_CO ISO-8859-1
# es_CO.UTF-8 UTF-8
# es_CR ISO-8859-1
# es_CR.UTF-8 UTF-8
# es_DO ISO-8859-1
# es_DO.UTF-8 UTF-8
# es_EC ISO-8859-1
# es_EC.UTF-8 UTF-8
# es_ES ISO-8859-1
# es_ES.UTF-8 UTF-8
# es_ES@euro ISO-8859-15
# es_GT ISO-8859-1
# es_GT.UTF-8 UTF-8
# es_HN ISO-8859-1
# es_HN.UTF-8 UTF-8
# es_MX ISO-8859-1
# es_MX.UTF-8 UTF-8
# es_NI ISO-8859-1
# es_NI.UTF-8 UTF-8
# es_PA ISO-8859-1
# es_PA.UTF-8 UTF-8
# es_PE ISO-8859-1
# es_PE.UTF-8 UTF-8
# es_PR ISO-8859-1
# es_PR.UTF-8 UTF-8
# es_PY ISO-8859-1
# es_PY.UTF-8 UTF-8
# es_SV ISO-8859-1
# es_SV.UTF-8 UTF-8
# es_US ISO-8859-1
# es_US.UTF-8 UTF-8
# es_UY ISO-8859-1
# es_UY.UTF-8 UTF-8
# es_VE ISO-8859-1
# es_VE.UTF-8 UTF-8
# et_EE ISO-8859-1
# et_EE.ISO-8859-15 ISO-8859-15
# et_EE.UTF-8 UTF-8
# eu_ES ISO-8859-1
# eu_ES.UTF-8 UTF-8
# eu_ES@euro ISO-8859-15
# eu_FR ISO-8859-1
# eu_FR.UTF-8 UTF-8
# eu_FR@euro ISO-8859-15
# fa_IR UTF-8
# ff_SN UTF-8
# fi_FI ISO-8859-1
# fi_FI.UTF-8 UTF-8
# fi_FI@euro ISO-8859-15
# fil_PH UTF-8
# fo_FO ISO-8859-1
# fo_FO.UTF-8 UTF-8
# fr_BE ISO-8859-1
# fr_BE.UTF-8 UTF-8
# fr_BE@euro ISO-8859-15
# fr_CA ISO-8859-1
# fr_CA.UTF-8 UTF-8
# fr_CH ISO-8859-1
# fr_CH.UTF-8 UTF-8
# fr_FR ISO-8859-1
# fr_FR.UTF-8 UTF-8
# fr_FR@euro ISO-8859-15
# fr_LU ISO-8859-1
# fr_LU.UTF-8 UTF-8
# fr_LU@euro ISO-8859-15
# fur_IT UTF-8
# fy_DE UTF-8
# fy_NL UTF-8
# ga_IE ISO-8859-1
# ga_IE.UTF-8 UTF-8
# ga_IE@euro ISO-8859-15
# gd_GB ISO-8859-15
# gd_GB.UTF-8 UTF-8
# gez_ER UTF-8
# gez_ER@abegede UTF-8
# gez_ET UTF-8
# gez_ET@abegede UTF-8
# gl_ES ISO-8859-1
# gl_ES.UTF-8 UTF-8
# gl_ES@euro ISO-8859-15
# gu_IN UTF-8
# gv_GB ISO-8859-1
# gv_GB.UTF-8 UTF-8
# ha_NG UTF-8
# he_IL ISO-8859-8
# he_IL.UTF-8 UTF-8
# hi_IN UTF-8
# hne_IN UTF-8
# hr_HR ISO-8859-2
# hr_HR.UTF-8 UTF-8
# hsb_DE ISO-8859-2
# hsb_DE.UTF-8 UTF-8
# ht_HT UTF-8
# hu_HU ISO-8859-2
# hu_HU.UTF-8 UTF-8
# hy_AM UTF-8
# hy_AM.ARMSCII-8 ARMSCII-8
# ia UTF-8
# id_ID ISO-8859-1
# id_ID.UTF-8 UTF-8
# ig_NG UTF-8
# ik_CA UTF-8
# is_IS ISO-8859-1
# is_IS.UTF-8 UTF-8
# it_CH ISO-8859-1
# it_CH.UTF-8 UTF-8
# it_IT ISO-8859-1
# it_IT.UTF-8 UTF-8
# it_IT@euro ISO-8859-15
# iu_CA UTF-8
# iw_IL ISO-8859-8
# iw_IL.UTF-8 UTF-8
# ja_JP.EUC-JP EUC-JP
# ja_JP.UTF-8 UTF-8
# ka_GE GEORGIAN-PS
# ka_GE.UTF-8 UTF-8
# kk_KZ PT154
# kk_KZ RK1048
# kk_KZ.UTF-8 UTF-8
# kl_GL ISO-8859-1
# kl_GL.UTF-8 UTF-8
# km_KH UTF-8
# kn_IN UTF-8
# ko_KR.EUC-KR EUC-KR
# ko_KR.UTF-8 UTF-8
# kok_IN UTF-8
# ks_IN UTF-8
# ks_IN@devanagari UTF-8
# ku_TR ISO-8859-9
# ku_TR.UTF-8 UTF-8
# kw_GB ISO-8859-1
# kw_GB.UTF-8 UTF-8
# ky_KG UTF-8
# lg_UG ISO-8859-10
# lg_UG.UTF-8 UTF-8
# li_BE UTF-8
# li_NL UTF-8
# lo_LA UTF-8
# lt_LT ISO-8859-13
# lt_LT.UTF-8 UTF-8
# lv_LV ISO-8859-13
# lv_LV.UTF-8 UTF-8
# mai_IN UTF-8
# mg_MG ISO-8859-15
# mg_MG.UTF-8 UTF-8
# mi_NZ ISO-8859-13
# mi_NZ.UTF-8 UTF-8
# mk_MK ISO-8859-5
# mk_MK.UTF-8 UTF-8
# ml_IN UTF-8
# mn_MN UTF-8
# mr_IN UTF-8
# ms_MY ISO-8859-1
# ms_MY.UTF-8 UTF-8
# mt_MT ISO-8859-3
# mt_MT.UTF-8 UTF-8
# my_MM UTF-8
# nan_TW@latin UTF-8
# nb_NO ISO-8859-1
# nb_NO.UTF-8 UTF-8
# nds_DE UTF-8
# nds_NL UTF-8
# ne_NP UTF-8
# nl_AW UTF-8
# nl_BE ISO-8859-1
# nl_BE.UTF-8 UTF-8
# nl_BE@euro ISO-8859-15
# nl_NL ISO-8859-1
# nl_NL.UTF-8 UTF-8
# nl_NL@euro ISO-8859-15
# nn_NO ISO-8859-1
# nn_NO.UTF-8 UTF-8
# nr_ZA UTF-8
# nso_ZA UTF-8
# oc_FR ISO-8859-1
# oc_FR.UTF-8 UTF-8
# om_ET UTF-8
# om_KE ISO-8859-1
# om_KE.UTF-8 UTF-8
# or_IN UTF-8
# os_RU UTF-8
# pa_IN UTF-8
# pa_PK UTF-8
# pap_AN UTF-8
# pl_PL ISO-8859-2
# pl_PL.UTF-8 UTF-8
# ps_AF UTF-8
# pt_BR ISO-8859-1
# pt_BR.UTF-8 UTF-8
# pt_PT ISO-8859-1
# pt_PT.UTF-8 UTF-8
# pt_PT@euro ISO-8859-15
# ro_RO ISO-8859-2
# ro_RO.UTF-8 UTF-8
# ru_RU ISO-8859-5
# ru_RU.CP1251 CP1251
# ru_RU.KOI8-R KOI8-R
# ru_RU.UTF-8 UTF-8
# ru_UA KOI8-U
# ru_UA.UTF-8 UTF-8
# rw_RW UTF-8
# sa_IN UTF-8
# sc_IT UTF-8
# sd_IN UTF-8
# sd_IN@devanagari UTF-8
# se_NO UTF-8
# shs_CA UTF-8
# si_LK UTF-8
# sid_ET UTF-8
# sk_SK ISO-8859-2
# sk_SK.UTF-8 UTF-8
# sl_SI ISO-8859-2
# sl_SI.UTF-8 UTF-8
# so_DJ ISO-8859-1
# so_DJ.UTF-8 UTF-8
# so_ET UTF-8
# so_KE ISO-8859-1
# so_KE.UTF-8 UTF-8
# so_SO ISO-8859-1
# so_SO.UTF-8 UTF-8
# sq_AL ISO-8859-1
# sq_AL.UTF-8 UTF-8
# sq_MK UTF-8
# sr_ME UTF-8
# sr_RS UTF-8
# sr_RS@latin UTF-8
# ss_ZA UTF-8
# st_ZA ISO-8859-1
# st_ZA.UTF-8 UTF-8
# sv_FI ISO-8859-1
# sv_FI.UTF-8 UTF-8
# sv_FI@euro ISO-8859-15
# sv_SE ISO-8859-1
# sv_SE.ISO-8859-15 ISO-8859-15
# sv_SE.UTF-8 UTF-8
# sw_KE UTF-8
# sw_TZ UTF-8
# ta_IN UTF-8
# te_IN UTF-8
# tg_TJ KOI8-T
# tg_TJ.UTF-8 UTF-8
# th_TH TIS-620
# th_TH.UTF-8 UTF-8
# ti_ER UTF-8
# ti_ET UTF-8
# tig_ER UTF-8
# tk_TM UTF-8
# tl_PH ISO-8859-1
# tl_PH.UTF-8 UTF-8
# tn_ZA UTF-8
# tr_CY ISO-8859-9
# tr_CY.UTF-8 UTF-8
# tr_TR ISO-8859-9
# tr_TR.UTF-8 UTF-8
# ts_ZA UTF-8
# tt_RU.UTF-8 UTF-8
# tt_RU.UTF-8@iqtelif UTF-8
# ug_CN UTF-8
# uk_UA KOI8-U
# uk_UA.UTF-8 UTF-8
# ur_PK UTF-8
# uz_UZ ISO-8859-1
# uz_UZ.UTF-8 UTF-8
# uz_UZ@cyrillic UTF-8
# ve_ZA UTF-8
# vi_VN UTF-8
# vi_VN.TCVN TCVN5712-1
# wa_BE ISO-8859-1
# wa_BE.UTF-8 UTF-8
# wa_BE@euro ISO-8859-15
# wo_SN UTF-8
# xh_ZA ISO-8859-1
# xh_ZA.UTF-8 UTF-8
# yi_US CP1255
# yi_US.UTF-8 UTF-8
# yo_NG UTF-8
# zh_CN GB2312
# zh_CN.GB18030 GB18030
# zh_CN.GBK GBK
# zh_CN.UTF-8 UTF-8
# zh_HK BIG5-HKSCS
# zh_HK.UTF-8 UTF-8
# zh_SG GB2312
# zh_SG.GBK GBK
# zh_SG.UTF-8 UTF-8
# zh_TW BIG5
# zh_TW.EUC-TW EUC-TW
# zh_TW.UTF-8 UTF-8
# zu_ZA ISO-8859-1
# zu_ZA.UTF-8 UTF-8

@ -0,0 +1,21 @@
diff --git a/bin/make-base-vm b/bin/make-base-vm
index d82b3d6..a5935fb 100755
--- a/bin/make-base-vm
+++ b/bin/make-base-vm
@@ -109,16 +109,3 @@ rm -rf $OUT
env -i LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 sudo vmbuilder kvm ubuntu --rootsize 10240 --arch=$ARCH --suite=$SUITE --addpkg=$addpkg --removepkg=$removepkg --ssh-key=var/id_dsa.pub --ssh-user-key=var/id_dsa.pub --mirror=$MIRROR --security-mirror=$SECURITY_MIRROR --dest=$OUT --flavour=$FLAVOUR --firstboot=`pwd`/target-bin/bootstrap-fixup
mv $OUT/*.qcow2 $OUT.qcow2
rm -rf $OUT
-
-if [ $LXC = "1" ]; then
- #sudo debootstrap --include=$addpkg --arch=$ARCH $SUITE $OUT-root $MIRROR
- echo Extracting partition for lxc
- qemu-img convert $OUT.qcow2 $OUT.raw
- loop=`sudo kpartx -av $OUT.raw|sed -n '/loop.p1/{s/.*loop\(.\)p1.*/\1/;p}'`
- sudo cp --sparse=always /dev/mapper/loop${loop}p1 $OUT
- sudo chown $USER $OUT
- sudo kpartx -d /dev/loop$loop
- rm -f $OUT.raw
- # bootstrap-fixup is done in libexec/make-clean-vm
-fi
-

@ -0,0 +1,67 @@
#!/bin/bash
## @author gdm85
##
## build a base Ubuntu Trusty
#
BASENAME=$(dirname $(readlink -m $0))
## the distro we are going to use
DISTNAME=trusty
REPOSRC=http://archive.ubuntu.com/ubuntu/
if [ ! $UID -eq 0 ]; then
echo "This script can only be run as root" 1>&2
exit 1
fi
## check for prerequisites
if ! type -P debootstrap >/dev/null; then
echo "You need to install debootstrap" 1&2
exit 2
fi
## check about the Ubuntu archive keyring
DEFK=/usr/share/keyrings/ubuntu-archive-keyring.gpg
KEYRING=$BASENAME/../keyrings/ubuntu-archive-keyring.gpg
if [ -s $DEFK ]; then
if ! diff $DEFK $KEYRING; then
ANSWER=
while [[ "$ANSWER" != "Y" && "$ANSWER" != "n" ]]; do
echo -n "The Ubuntu Archive keyring in your system ($DEFK) that will be used to debootstrap is different from the reference provided keyring. Continue? (Y/n) "
read -r ANSWER || exit $?
done
if [[ "$ANSWER" == "n" ]]; then
exit 1
fi
## use system's keyring, even if different than provided one
## this is a no-issue only in case the system's keyring is more recent than the provided one
KEYRING=$DEFK
fi
else
ANSWER=
while [[ "$ANSWER" != "Y" && "$ANSWER" != "n" ]]; do
echo -n "Your system comes with no Ubuntu Archive keyring in $DEFK that is necessary for debootstrap. Use reference provided keyring? (Y/n) "
read -r ANSWER || exit $?
done
if [[ "$ANSWER" == "n" ]]; then
exit 1
fi
fi
echo "Will use $KEYRING"
exit 0
## NOTE: a temporary directory under /tmp is not used because can't be mounted dev/exec
mkdir $DISTNAME || exit $?
TMPDIR=$PWD/$DISTNAME
debootstrap --keyring=$KEYRING $DISTNAME $DISTNAME $REPOSRC && \
cd $DISTNAME && \
tar -c . | docker import - gdm85/$DISTNAME
RV=$?
# always perform cleanup
rm -rf $TMPDIR
exit $RV

@ -0,0 +1,67 @@
#!/bin/bash
## @author gdm85
##
## build a base Debian Wheezy
#
BASENAME=$(dirname $(readlink -m $0))
## the distro we are going to use
DISTNAME=wheezy
DEBIAN_REPO=http://ftp.debian.org/debian
if [ ! $UID -eq 0 ]; then
echo "This script can only be run as root" 1>&2
exit 1
fi
## install prerequisites
## NOTE: may fail on non-Ubuntu/Debian systems
if ! type -P debootstrap >/dev/null; then
apt-get install debootstrap -y || exit $?
fi
## check about the Debian archive keyring
DEFK=/usr/share/keyrings/debian-archive-keyring.gpg
KEYRING=$BASENAME/../keyrings/debian-archive-keyring.gpg
if [ -s $DEFK ]; then
if ! diff $DEFK $KEYRING; then
ANSWER=
while [[ "$ANSWER" != "Y" && "$ANSWER" != "n" ]]; do
echo -n "The Debian Archive keyring in your system ($DEFK) that will be used to debootstrap is different from the reference provided keyring. Continue? (Y/n) "
read -r ANSWER || exit $?
done
if [[ "$ANSWER" == "n" ]]; then
exit 1
fi
## use system's keyring, even if different than provided one
## this is a no-issue only in case the system's keyring is more recent than the provided one
KEYRING=$DEFK
fi
else
ANSWER=
while [[ "$ANSWER" != "Y" && "$ANSWER" != "n" ]]; do
echo -n "Your system comes with no Debian Archive keyring in $DEFK that is necessary for debootstrap. Use reference provided keyring? (Y/n) "
read -r ANSWER || exit $?
done
if [[ "$ANSWER" == "n" ]]; then
exit 1
fi
fi
echo "Will use $KEYRING"
exit 0
## NOTE: a temporary directory under /tmp is not used because can't be mounted dev/exec
mkdir $DISTNAME || exit $?
TMPDIR=$PWD/$DISTNAME
debootstrap --keyring=$KEYRING $DISTNAME $DISTNAME $DEBIAN_REPO && \
cd $DISTNAME && \
tar -c . | docker import - gdm85/$DISTNAME
RV=$?
# always perform cleanup
rm -rf $TMPDIR
exit $RV

@ -0,0 +1,8 @@
#!/bin/bash
set -e
BASENAME=$(dirname $(readlink -m $0))
cd $BASENAME/../gitian-bitcoin-host && \
docker build --tag=gdm85/gitian-bitcoin-host .

@ -0,0 +1,52 @@
#!/bin/bash
BASENAME=$(dirname $(readlink -m $0))
cd $BASENAME/../gitian-host || exit $?
if [ ! -f authorized_keys ]; then
echo "No authorized_keys file found in $PWD"
if [ -f ~/.ssh/id_rsa.pub ]; then
echo -n "Do you want to use ~/.ssh/id_rsa.pub? (y/n) "
read -r ANSWER
if [[ "$ANSWER" == "y" ]]; then
cp -v ~/.ssh/id_rsa.pub authorized_keys || exit $?
else
exit 1
fi
else
exit 1
fi
fi
function wait_for_ssh() {
local IP="$1"
local SECS="$2"
while [ $SECS -gt 0 ]; do
ssh -o ConnectTimeout=1 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no debian@$IP ls >/dev/null 2>/dev/null && return 0
sleep 1
let SECS-=1
done
return 1
}
function wait_remove() {
local CID="$1"
while [ ! docker rm $CID 2>/dev/null ]; do
sleep 2
done
}
##NOTE: can leave behind a running container of gitian-host
docker build --tag=gdm85/gitian-host . && \
CID=$(docker run -d --privileged gdm85/gitian-host) && \
IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' $CID) && \
wait_for_ssh "$IP" 10 && \
echo "$CID is now online ($IP), building base VMs on it" && \
ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no debian@$IP ./build-base-vms.sh && \
docker kill $CID && \
docker wait $CID && \
docker commit $CID gdm85/gitian-host-vms && \
wait_remove $CID && \
echo "Gitian host images created successfully!" && \
echo "You can now spawn containers with spawn-gitian-host.sh"

@ -0,0 +1,26 @@
#!/bin/bash
BASENAME=$(dirname $(readlink -m $0))
cd $BASENAME/../trusty-kbuilder || exit $?
if [ ! -f authorized_keys ]; then
echo "No authorized_keys file found in $PWD"
if [ -f ~/.ssh/id_rsa.pub ]; then
echo -n "Do you want to use ~/.ssh/id_rsa.pub? (y/n) "
read -r ANSWER
if [[ "$ANSWER" == "y" ]]; then
cp -v ~/.ssh/id_rsa.pub authorized_keys || exit $?
else
exit 1
fi
else
exit 1
fi
fi
##NOTE: can leave behind a running container of gitian-host
docker build --tag=gdm85/trusty-kbuilder . && \
echo "Ubuntu Trusty kernel builder image created successfully!" && \
echo "You can now spawn containers with:" && \
echo "docker run -d gdm85/trusty-kbuilder"

@ -0,0 +1,21 @@
#!/bin/bash
## docker-hosts
##
## @author gdm85
## script to show docker containers and their hostnames/IPv4
#
function enumerate_container_ips() {
local CID
for CID in $(docker ps -q -a); do
local NAME=$(docker inspect --format '{{ .Name }}' $CID | awk '{ print substr($0, 2, length($0)-1) }')
local HPATH=$(docker inspect --format '{{ .HostnamePath }}' $CID)
local IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' $CID)
local HOSTNAME=$(<$HPATH)
echo -e "$NAME\t$HOSTNAME\t$IP"
done
}
enumerate_container_ips

@ -0,0 +1,90 @@
#!/bin/bash
## @author gdm85
##
## provide IP-locking of all exposed ports of docker container from a specific whitelist address
## NOTE: iptables rules will be stale once container is stopped/killed/removed
#
SCRIPTS=$(dirname $(readlink -m $0)) || exit $?
if ! type jq 2>&1 >/dev/null; then
echo "jq command is not available" 1>&2
exit 1
fi
if [[ $# -lt 2 ]]; then
echo "Usage: docker-iplock container-name whitelist_ipv4_1 [whitelist_ipv4_2] [...whitelist_ipv4_n]"
exit 1
fi
function ipt_forward_update() {
local CID="$1"
if [[ "$CID" == "<no value>" ]]; then
echo "Invalid container ID" 1>&2
return 1
fi
local WHITELIST4="$2"
local CONTAINER_IPv4=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' $CID)
if [[ -z "$CONTAINER_IPv4" ]]; then
return 1
fi
test ! -z "$DEBUG" && echo "ipt_forward_update: $CID"
local STANZA="$(docker inspect --format '{{json .HostConfig.PortBindings }}' $CID)" || return $?
test ! -z "$DEBUG" && echo "Bindings: $STANZA"
local I=0
for P in $(echo "$STANZA" | jq -M 'keys' | tail -n+2 | head -n-1 | sed s/,$// | sed 's/"//g' ); do
## port on container
local SRCPORT="$(echo $P | awk -F/ '{ print $1 }')"
test ! -z "$DEBUG" && echo "$CID source port: $SRCPORT"
## get port binding (if any)
local SNIP=$(echo $STANZA | jq -M "to_entries | .[$I].value")
if [[ -z "$SNIP" || "$SNIP" == "null" ]]; then
let I=I+1
continue
fi
test ! -z "$DEBUG" && echo "$CID binding: $SNIP"
local HOSTIP="$(echo $SNIP | jq -M -r '.[0].HostIp')"
test ! -z "$DEBUG" && echo "$CID host ip: $HOSTIP"
## match only bindings on docker host
if [[ -z "$HOSTIP" || "$HOSTIP" == '0.0.0.0' ]]; then
local HOSTPORT="$(echo $SNIP | jq -M -r '.[0].HostPort')"
local TOREMOVE="-d ${CONTAINER_IPv4}/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport $SRCPORT -j ACCEPT"
iptables -D FORWARD $TOREMOVE --wait || return $?
test ! -z "$DEBUG" && echo "$CID: iptables rule removed: $TOREMOVE"
local IPv4
for IPv4 in $WHITELIST4; do
local TOADD="-s ${IPv4}/32 -d ${CONTAINER_IPv4}/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport $SRCPORT -j ACCEPT"
iptables -I FORWARD 1 $TOADD --wait || return $?
test ! -z "$DEBUG" && echo "$CID: iptables rule added: $TOADD"
done
fi
let I=I+1
done
}
CONTAINER="$1"
shift 1
WHITELIST4="$@"
if [ -z "$WHITELIST4" ]; then
echo "Invalid whitelist addresses specified" 1>&2
exit 1
fi
CID=$(docker inspect --format '{{ .Id }}' $CONTAINER) || exit $?
## now run rules update
ipt_forward_update "$CID" "$WHITELIST4"

@ -0,0 +1,28 @@
#!/bin/bash
## docker-iptables
##
## @author gdm85
## script to show iptables rules with docker names
## can be used also to detect problems with dead containers and stale iptable rules
## supports standard iptables-save syntax
#
function replace_iptables() {
local CID
local SEDCMD="-e s!172.17.42.1/32!dockerHost!g
for CID in $(docker ps -q -a); do
local NAME=$(docker inspect --format '{{ .Name }}' $CID | awk '{ print substr($0, 2, length($0)-1) }')
local IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' $CID)
if [ -z "$IP" ]; then
continue
fi
SEDCMD="$SEDCMD -e s!$IP/32!${NAME}!g -e s!$IP!${NAME}!g"
done
sed $SEDCMD
}
iptables-save $@ | replace_iptables

@ -0,0 +1,62 @@
#!/bin/bash
## docker-hosts
##
## @author gdm85
## script to show docker allocated ports of a container
#
if ! type jq >/dev/null 2>/dev/null; then
echo "jq must be installed" 1>&2
exit 1
fi
if [ $# -lt 1 ]; then
echo "Please specify at least one container" 1>&2
exit 1
fi
function show_ports() {
local CID="$1"
local CONTAINER_IPv4=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' $CID) || return $?
if [[ -z "$CONTAINER_IPv4" || "$CONTAINER_IPv4" == "<no value>" ]]; then
echo "Invalid container: $CID" 1>&2
return 1
fi
local NAME=$(docker inspect --format '{{ .Name }}' $CID | awk '{ print substr($0, 2, length($0)-1) }')
local STANZA="$(docker inspect --format '{{json .HostConfig.PortBindings }}' $CID)" || return $?
echo "$NAME:"
let I=0
for P in $(echo "$STANZA" | jq -M 'keys' | tail -n+2 | head -n-1 | sed s/,$// | sed 's/"//g' ); do
## port on container
local SRCPORT="$(echo $P | awk -F/ '{ print $1 }')"
echo -e -n "\t$SRCPORT\t<-\t"
## get port binding (if any)
local SNIP=$(echo $STANZA | jq -M "to_entries | .[$I].value")
if [[ -z "$SNIP" || "$SNIP" == "null" ]]; then
echo "$CONTAINER_IPv4:$SRCPORT"
else
local HOSTIP="$(echo $SNIP | jq -M -r '.[0].HostIp')"
## match only bindings on dockerhost
if [[ -z "$HOSTIP" || "$HOSTIP" == '0.0.0.0' ]]; then
local HOSTPORT="$(echo $SNIP | jq -M -r '.[0].HostPort')"
echo "0.0.0.0:$HOSTPORT"
else
echo "$HOSTIP:$SRCPORT"
fi
fi
let I=I+1
done
}
for CID in $@; do
show_ports "$CID"
done

@ -0,0 +1,9 @@
#!/bin/bash
set -e
CID=$(docker run -d --privileged gdm85/gitian-bitcoin-host) || exit $?
IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' $CID) || exit $?
echo "You can now SSH into container $CID:"
echo "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no debian@$IP"

@ -0,0 +1,9 @@
#!/bin/bash
set -e
CID=$(docker run -d --privileged gdm85/gitian-host-vms) && \
IP=$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' $CID) || exit $?
echo "You can now SSH into container $CID:"
echo "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no debian@$IP"

@ -0,0 +1,48 @@
## trusty-kbuilder
##
## VERSION 0.1.0
##
## Trusty image to build kernel
##
#
## NOTE: this image must be debootstrapped with build-trusty.sh
FROM gdm85/trusty
MAINTAINER Giuseppe Mazzotta "gdm85@users.noreply.github.com"
ENV DEBIAN_FRONTEND noninteractive
## replace sources
ADD sources.list /etc/apt/sources.list
RUN apt-get update && apt-get install -y apt-utils aptitude && aptitude update && aptitude safe-upgrade -y && \
aptitude install -y nano tmux fakeroot build-essential crash kexec-tools makedumpfile kernel-wedge openssh-server
## log rotation is managed outside, thus remove it
## install necessary daemons
RUN apt-get remove -y logrotate
## set current user's id_rsa as only allowed key
ADD authorized_keys /root/.ssh/authorized_keys
RUN chmod -R go-rwx /root/.ssh
## disable password logins & reverse DNS lookups
RUN sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config && echo 'UseDNS no' >> /etc/ssh/sshd_config
EXPOSE 22
##NOTE: this will use current kernel version!
RUN apt-get build-dep -y --no-install-recommends linux-image-$(uname -r)
RUN apt-get install -y git-core libncurses5 libncurses5-dev libelf-dev asciidoc binutils-dev
## user that will make the compilation
RUN useradd -m -s /bin/bash rdeckard
## add script for getting source
ADD build-kernel.sh /home/rdeckard/
ADD patches/ /home/rdeckard/patches/
RUN chown -R rdeckard.rdeckard /home/rdeckard
CMD /usr/sbin/sshd -D -e

@ -0,0 +1,16 @@
#!/bin/bash
set -e
apt-get source linux-image-$(uname -r)
cd linux-3.13.0
## will fail here if no patches are available
## why are you recompiling kernel if no custom patches are there?
for MYP in $(ls ../patches); do
patch -p1 < ../patches/$MYP
done
fakeroot debian/rules clean
DEB_BUILD_OPTIONS=parallel=3 AUTOBUILD=1 NOEXTRAS=1 fakeroot debian/rules binary-generic

@ -0,0 +1,11 @@
deb http://archive.ubuntu.com/ubuntu/ trusty main restricted
deb-src http://archive.ubuntu.com/ubuntu/ trusty main restricted
deb http://archive.ubuntu.com/ubuntu/ trusty-updates main restricted
deb-src http://archive.ubuntu.com/ubuntu/ trusty-updates main restricted
deb http://security.ubuntu.com/ubuntu trusty-security main restricted
deb-src http://security.ubuntu.com/ubuntu trusty-security main restricted
deb http://security.ubuntu.com/ubuntu trusty-security universe
deb-src http://security.ubuntu.com/ubuntu trusty-security universe
deb http://security.ubuntu.com/ubuntu trusty-security multiverse
deb-src http://security.ubuntu.com/ubuntu trusty-security multiverse
Loading…
Cancel
Save