Don't cache the whole base directory; instead independently cache:
- the build output directory
- the ccache directory
This allow completely avoiding the need for building base if a
cache (keyed on the relevant parts of the git tree for base) is
successfully restored, as well as reducing the amount of cached
data.
Additionally, ensure caches are isolated by branch name (but allow
falling back to a "master" cache for restore): we don't want PRs
using each others' caches.
Update shellcheck and shfmt to the latest version.
Fixes <https://github.com/koreader/koreader/issues/5152>.
Btw, you can apply shellcheck suggestions with a command like:
```
shellcheck --include=SC2250 -f diff *.sh | git apply
```
If only 1 line changed, it's just the timestamp. Otherwise every commit in the main repo will result in a useless one in koreader/koreader-translations, such as <eadc9f133a>.
* junit test results; unfortunately this seems to conflict with the verbose out
* fix deps cache: two files can change independently
* verbose print obsoleted by gtest in upsream busted
The debug output from Coveralls was rubbish. `"Build processing error"`
Besides some basics like checking if we were actually sending
valid JSON that gives us absolutely nothing to work with.
I wrote this whole complicated wofkflow-based config file, but except
in the case of a base rebuild it wouldn't really be any faster than
this simple tweak.
```
defaults: &defaults
docker:
- image: houqp/kobase:0.0.5
environment:
EMULATE_READER: 1
# this is for shellcheck 0.4.5 and lower; can be removed for 0.4.6
LC_ALL: en_US.UTF8
version: 2
jobs:
install-and-build:
<<: *defaults
steps:
- checkout
- restore_cache:
keys:
# binary dependencies require {{ arch }} because there are different CPUs in use on the servers
- deps-{{ arch }}-{{ checksum ".ci/install.sh" }}
# need to init some stuff first or git will complain when sticking in base cache
- run: git submodule init base && git submodule update base && pushd base && git submodule init && git submodule update && popd
# we can't use command output directly for cache check so we write it to git-rev-base
- run: pushd base && git_rev_base=$(git describe HEAD) && popd && echo $git_rev_base && echo $git_rev_base >git-rev-base
- restore_cache:
keys:
- build-{{ arch }}-{{ checksum "git-rev-base" }}
- run: echo 'export PATH=${HOME}/bin:${PATH}' >> $BASH_ENV
- run:
name: setup
command: .ci/before_install.sh
- run:
name: install
command: .ci/install.sh
- save_cache:
key: deps-{{ arch }}-{{ checksum ".ci/install.sh" }}
paths:
- "/home/ko/bin"
- "/home/ko/.luarocks"
# compiled luarocks binaries
- "install"
- run:
name: fetch
command: .ci/fetch.sh
- run:
name: check
command: .ci/check.sh
- run:
name: build
command: .ci/build.sh
- save_cache:
key: build-{{ checksum "base/git-rev" }}
paths:
- "/home/ko/.ccache"
- "base"
- persist_to_workspace:
# Must be an absolute path, or relative path from working_directory
root: "./"
# Must be relative path from root
paths:
# front build
- "koreader-emulator-x86_64-linux-gnu/koreader"
test:
<<: *defaults
parallelism: 4
steps:
- checkout
- restore_cache:
keys:
# binary dependencies require {{ arch }} because there are different CPUs in use on the servers
- deps-{{ arch }}-{{ checksum ".ci/install.sh" }}
# need to init some stuff first or git will complain when sticking in base cache
- run: git submodule init base && git submodule update base && pushd base && git submodule init && git submodule update && popd
# we can't use command output directly for cache check so we write it to git-rev-base
- run: pushd base && git_rev_base=$(git describe HEAD) && popd && echo $git_rev_base && echo $git_rev_base >git-rev-base
- restore_cache:
keys:
- build-{{ arch }}-{{ checksum "git-rev-base" }}
- run: echo 'export PATH=${HOME}/bin:${PATH}' >> $BASH_ENV
- attach_workspace:
# Must be absolute path or relative path from working_directory
at: "./"
- run:
name: test
command: .ci/test.sh
docs-and-coverage:
<<: *defaults
steps:
- checkout
- restore_cache:
keys:
# binary dependencies require {{ arch }} because there are different CPUs in use on the servers
- deps-{{ arch }}-{{ checksum ".ci/install.sh" }}
# need to init some stuff first or git will complain when sticking in base cache
- run: git submodule init base && git submodule update base && pushd base && git submodule init && git submodule update && popd
# we can't use command output directly for cache check so we write it to git-rev-base
- run: pushd base && git_rev_base=$(git describe HEAD) && popd && echo $git_rev_base && echo $git_rev_base >git-rev-base
- restore_cache:
keys:
- build-{{ arch }}-{{ checksum "git-rev-base" }}
- run: echo 'export PATH=${HOME}/bin:${PATH}' >> $BASH_ENV
- attach_workspace:
# Must be absolute path or relative path from working_directory
at: "./"
- run:
name: cleanup
command: .ci/after_success.sh
workflows:
version: 2
build-and-test:
jobs:
- install-and-build
- test:
requires:
- install-and-build
- docs-and-coverage:
requires:
- install-and-build
```
* Travis: speed up by caching base and running luacheck earlier
* ignore bin and install for git status change detection
* skip coverage except on official master branch. It adds 3 whole minutes and does nothing to prevent regressions
* also cache ~/.luarocks. It evens out but would generally prevent remote timeout shenenigans
* remove base cache dir before caching with verbose remove to see what's going on
* more inclusive shell code quality analysis
* fixed more shellcheck issues
* better shellcheck/shfmt debugging info
Benoit Pierre
Frans de Jonge
Qingping Hou