Skip to content

Run integration tests nightly rather than on every pull request #1137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
aalhossary wants to merge 1 commit into
base: master
Choose a base branch
from
+70 −4
Open

Run integration tests nightly rather than on every pull request #1137

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions .github/workflows/nightly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
name: Nightly Integration Tests

# The integration tests reach out to CATH, ECOD, RCSB, EBI, UniProt and others.
# That makes them valuable - they are how we find out that an upstream service
# has changed a URL, a format or a redirect - but it also makes them unsuitable
# as a gate on pull requests, because an outage anywhere blocks every
# contributor. Running them on a schedule keeps the coverage while decoupling it
# from people's ability to merge.
on:
schedule:
# 03:17 UTC daily. Off the hour deliberately: scheduled jobs that ask for
# exactly midnight queue behind everybody else's.
- cron: '17 3 * * *'
# Also runnable by hand, e.g. to confirm an upstream service is back.
workflow_dispatch:

permissions:
contents: read # to fetch code (actions/checkout)

jobs:
integrationtest:
runs-on: ubuntu-latest
# These tests download large files from servers we do not control; the
# default 6 hour limit is far more than they need and far more than we want
# to spend if one of them hangs.
timeout-minutes: 90
strategy:
matrix:
# One JDK only. The point of this run is to exercise the network paths,
# not the language level, which the pull request build already covers
# across 11, 17 and 21.
java: [21]
fail-fast: false
name: Integration tests, JDK ${{ matrix.java }}

steps:
- uses: actions/checkout@v4
- name: Set up JDK
uses: actions/setup-java@v4
with:
distribution: 'oracle'
java-version: ${{ matrix.java }}
- name: Build and run integration tests
run: mvn verify --no-transfer-progress
- name: Upload surefire reports
# Kept on failure so an upstream break can be diagnosed after the fact:
# GitHub expires run logs after 90 days, and these reports carry the
# stack traces that say which service misbehaved.
if: failure()
uses: actions/upload-artifact@v4
with:
name: surefire-reports
path: '**/target/surefire-reports/**'
retention-days: 30
20 changes: 16 additions & 4 deletions .github/workflows/pull_request.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,14 @@ jobs:
with:
distribution: 'oracle'
java-version: ${{ matrix.java }}
- name: Build, test and integration test
run: mvn verify --no-transfer-progress
- name: Build and test (no integration tests)
# Integration tests are excluded here and run nightly instead, see
# nightly.yml. They depend on CATH, ECOD, RCSB, EBI and others being up
# and responsive, so running them on every pull request means a third
# party having a bad day blocks contributors, and a real regression
# cannot be told apart from the resulting noise. Master Build already
# excludes them for the same reason.
run: mvn verify -pl '!biojava-integrationtest' --no-transfer-progress

# Note that 11 is not available in openjdk. So we need to do it with the Zulu distribution (see https://github.com/actions/setup-java)
# When we drop 11, it will be safe to drop the copy-pasted workflow excerpt below
Expand All @@ -54,5 +60,11 @@ jobs:
with:
distribution: 'zulu'
java-version: ${{ matrix.java }}
- name: Build, test and integration test
run: mvn verify --no-transfer-progress
- name: Build and test (no integration tests)
# Integration tests are excluded here and run nightly instead, see
# nightly.yml. They depend on CATH, ECOD, RCSB, EBI and others being up
# and responsive, so running them on every pull request means a third
# party having a bad day blocks contributors, and a real regression
# cannot be told apart from the resulting noise. Master Build already
# excludes them for the same reason.
run: mvn verify -pl '!biojava-integrationtest' --no-transfer-progress
Loading