From 3e84cfad8ecc78b4a0b2849a07a52c7894123e29 Mon Sep 17 00:00:00 2001 From: "(cdep)illabout" Date: Wed, 12 May 2021 22:39:50 +0900 Subject: [PATCH 01/12] haskell-updates: add documentation for workflow --- pkgs/development/haskell-modules/HACKING.md | 90 +++++++++++++++++++++ 1 file changed, 90 insertions(+) create mode 100644 pkgs/development/haskell-modules/HACKING.md diff --git a/pkgs/development/haskell-modules/HACKING.md b/pkgs/development/haskell-modules/HACKING.md new file mode 100644 index 000000000000..669be5394e20 --- /dev/null +++ b/pkgs/development/haskell-modules/HACKING.md @@ -0,0 +1,90 @@ + +## Maintainer Workflow + +This is the documentation for periodically merging the `haskell-updates` branch +into the `master` branch. This workflow is performed by members in the +[@NixOS/haskell](https://github.com/orgs/NixOS/teams/haskell) team. +Each member of the team takes a two week period where they much merge the +`haskell-updates` branch into `master`. + +The goal of this workflow is to regularly merge the `haskell-updates` branch +into the `master` branch, while making sure there are no evaluation errors or +build errors that get into `master`. + +The workflow generally proceeds in three main steps: + +1. create the initial `haskell-updates` PR +1. wait for contributors to fix newly broken Haskell packages +1. merge `haskell-updates` into `master` + +We describe each of these steps in a separate section. + +### Initial `haskell-updates` PR + +In this section we create the PR for merging `haskell-updates` into `master`. + +1. Make sure the `haskell-updates` branch is up-to-date with `master`. + +1. Update the Stackage Nightly resolver used by Nixpkgs and create a commit: + + ```console + $ maintainers/scripts/haskell/update-stackage.sh --do-commit + ``` + +1. Update the Hackage package set used by Nixpkgs and create a commit: + + ```console + $ maintainers/scripts/haskell/update-hackage.sh --do-commit + ``` + +1. Regenerate the Haskell package set used in Nixpkgs and create a commit: + + ```console + $ maintainers/scripts/haskell/regenerate-hackage-packages.sh --do-commit + ``` + +1. Push these commits to the Nixpkgs repo. + +1. Open a PR on Nixpkgs merging `haskell-updates` into `master`. + +Use the following message body: + +```markdown +### This Merge + +This PR is the regularly merge of the `haskell-updates` branch into `master`. + +This branch is being continually built and tested by hydra at https://hydra.nixos.org/jobset/nixpkgs/haskell-updates. + +I will aim to merge this PR **by 2021-TODO-TODO**. If I can merge it earlier, there might be successor PRs in that time window. As part of our rotation @TODO will continue these merges from 2021-TODO-TODO to 2021-TODO-TODO. + +### haskellPackages Workflow Summary + +Our workflow is currently described at `pkgs/development/haskell-modules/HACKING.md`. + +The short version is this: +* We regularly update the stackage and hackage pins on `haskell-updates` (normally at the beginning of a merge window). +* The community fixes builds of Haskell packages on that branch. +* We aim at at least one merge of `haskell-updates` into `master` every two weeks. +* We only do the merge if the `mergeable` job is succeeding on hydra. +* If a maintained package is still broken at the time of merge, we will only merge if the maintainer has been pinged 7 days in advance. (If you care about a Haskell package, become a maintainer!) + +--- + +This is the follow-up to #TODO. +``` + +Make sure to replace all TODO with the actual values. + +### Fix Broken Packages + +(TODO) + +### Merge `haskell-updates` into `master` + +(TODO) + +## Contributor Workflow + +(TODO: this section is to describe the type of workflow for non-comitters to +contribute to `haskell-updates`) From f56b6e1f62c4d73e59af6696c25baf5bf2a71fef Mon Sep 17 00:00:00 2001 From: "(cdep)illabout" Date: Sat, 15 May 2021 18:10:35 +0900 Subject: [PATCH 02/12] haskell-updates: more documentation --- pkgs/development/haskell-modules/HACKING.md | 54 ++++++++++++++++++++- 1 file changed, 52 insertions(+), 2 deletions(-) diff --git a/pkgs/development/haskell-modules/HACKING.md b/pkgs/development/haskell-modules/HACKING.md index 669be5394e20..805f6bc98f41 100644 --- a/pkgs/development/haskell-modules/HACKING.md +++ b/pkgs/development/haskell-modules/HACKING.md @@ -76,14 +76,64 @@ This is the follow-up to #TODO. Make sure to replace all TODO with the actual values. -### Fix Broken Packages +### Notify Maintainers and Fix Broken Packages -(TODO) +After you've done the previous steps, Hydra will start building the new and +updated Haskell packages. You can see the progress Hydra is making at +https://hydra.nixos.org/jobset/nixpkgs/haskell-updates. This Hydra jobset is +defined in the file [release-haskell.nix](../../top-level/release-haskell.nix). + +#### Notify Maintainers + +When Hydra finishes building all the packages, you should generate a build +report to notify maintainers of their broken packages. You can do that with the +following commands: + +```console +$ maintainers/scripts/haskell/hydra-report.hs get-report +$ maintainers/scripts/haskell/hydra-report.hs ping-maintainers +``` + +The `hyda-report.hs ping-maintainers` command generates a Markdown document +that you can paste in a GitHub comment on the PR opened above. This +comment describes which Haskell packages are now failing to build. It also +pings the maintainers so that they know to fix up their packages. + +This build report can be fetched and re-generated for new Hydra evaluations. +It may help contributors to try to keep the GitHub comment updated with the +most recent build report. + +#### Fix Broken Packages + +After getting the build report, you can see which packages and Hydra jobs are +failing to build. The most important jobs are the `maintained` and `mergeable` +jobs. These are both defined in +[`release-haskell.nix](../../top-level/release-haskell.nix). + +`mergeable` is a set of the most important Haskell packages, including things +like Pandoc and XMonad. These packages are widely used. We would like to +always keep these building. + +`maintained` is a set of Haskell packages that have maintainers in Nixpkgs. +We should be proactive in working with maintainers to keep their packages +building. + +Steps to fix Haskell packages that are failing to build is out of scope for +this document, but it usually requires fixing up dependencies that now +out-of-bounds. ### Merge `haskell-updates` into `master` (TODO) +- mark broken packages with `mark-broken.sh` + +### Additional Info + +(TODO) + +- you can restart a Hydra evaluation by logging in with github. + ## Contributor Workflow (TODO: this section is to describe the type of workflow for non-comitters to From c40e194de250e9a6776081066124a915525ae06c Mon Sep 17 00:00:00 2001 From: "(cdep)illabout" Date: Sun, 16 May 2021 15:48:35 +0900 Subject: [PATCH 03/12] haskell-updates: more documentation --- pkgs/development/haskell-modules/HACKING.md | 42 ++++++++++++++++++++- 1 file changed, 41 insertions(+), 1 deletion(-) diff --git a/pkgs/development/haskell-modules/HACKING.md b/pkgs/development/haskell-modules/HACKING.md index 805f6bc98f41..acf7dc92db26 100644 --- a/pkgs/development/haskell-modules/HACKING.md +++ b/pkgs/development/haskell-modules/HACKING.md @@ -99,10 +99,17 @@ that you can paste in a GitHub comment on the PR opened above. This comment describes which Haskell packages are now failing to build. It also pings the maintainers so that they know to fix up their packages. +It may be helpful to pipe `hydra-report.hs ping-maintainers` into `xclip` +(XOrg) or `wl-clip` (Wayland) in order to post on GitHub. + This build report can be fetched and re-generated for new Hydra evaluations. It may help contributors to try to keep the GitHub comment updated with the most recent build report. +Maintainers should be given at least 7 days to fix up their packages when they +break. If maintainers don't fix up their packages with 7 days, then they +may be marked broken before merging `haskell-updates` into `master`. + #### Fix Broken Packages After getting the build report, you can see which packages and Hydra jobs are @@ -119,9 +126,42 @@ We should be proactive in working with maintainers to keep their packages building. Steps to fix Haskell packages that are failing to build is out of scope for -this document, but it usually requires fixing up dependencies that now +this document, but it usually requires fixing up dependencies that are now out-of-bounds. +#### Mark Broken Packages + +Packages that do not get fixed can be marked broken with the following +commands. First check which packages are broken: + +```console +$ maintainers/scripts/haskell/hydra-report.hs get-report +$ maintainers/scripts/haskell/hydra-report.hs mark-broken-list +``` + +This shows a list of packages that are thought to be broken. + +Next, run the following command. + +```console +$ maintainers/scripts/haskell/mark-broken.sh --do-commit +``` + +This first opens up an editor with the broken package list. Some of these +packages may have a maintainer in Nixpkgs. If these maintainers have not been +given 7 days to fix up their package, then make sure to remove those packages +from the list. After saving and exiting the editor, the following will +happen: + +- packages from the list will be added to + [`configuration-hackage2nix/broken.yaml`](configuration-hackage2nix/broken.yaml) +- `hackage-packages.nix` will be regenerated +- the + [`configuration-hackage2nix/transitive-broken.yaml`](configuration-hackage2nix/transitive-broken.yaml) + file will be regenerated +- `hackage-packages.nix` will be regenerated again +- everything will be committed + ### Merge `haskell-updates` into `master` (TODO) From 3640440ed83e4c662752368c896dd416d4c3df3a Mon Sep 17 00:00:00 2001 From: "(cdep)illabout" Date: Thu, 27 May 2021 21:36:13 +0900 Subject: [PATCH 04/12] haskell-updates: more documentation --- pkgs/development/haskell-modules/HACKING.md | 86 ++++++++++++++++----- 1 file changed, 67 insertions(+), 19 deletions(-) diff --git a/pkgs/development/haskell-modules/HACKING.md b/pkgs/development/haskell-modules/HACKING.md index acf7dc92db26..2c1f13763284 100644 --- a/pkgs/development/haskell-modules/HACKING.md +++ b/pkgs/development/haskell-modules/HACKING.md @@ -4,8 +4,8 @@ This is the documentation for periodically merging the `haskell-updates` branch into the `master` branch. This workflow is performed by members in the [@NixOS/haskell](https://github.com/orgs/NixOS/teams/haskell) team. -Each member of the team takes a two week period where they much merge the -`haskell-updates` branch into `master`. +Each member of the team takes a two week period where they are in charge of +merging the `haskell-updates` branch into `master`. The goal of this workflow is to regularly merge the `haskell-updates` branch into the `master` branch, while making sure there are no evaluation errors or @@ -13,7 +13,7 @@ build errors that get into `master`. The workflow generally proceeds in three main steps: -1. create the initial `haskell-updates` PR +1. create the initial `haskell-updates` PR, and update Stackage and Hackage snapshots 1. wait for contributors to fix newly broken Haskell packages 1. merge `haskell-updates` into `master` @@ -43,7 +43,7 @@ In this section we create the PR for merging `haskell-updates` into `master`. $ maintainers/scripts/haskell/regenerate-hackage-packages.sh --do-commit ``` -1. Push these commits to the Nixpkgs repo. +1. Push these commits to the Nixpkgs repository. 1. Open a PR on Nixpkgs merging `haskell-updates` into `master`. @@ -52,7 +52,7 @@ Use the following message body: ```markdown ### This Merge -This PR is the regularly merge of the `haskell-updates` branch into `master`. +This PR is the regular merge of the `haskell-updates` branch into `master`. This branch is being continually built and tested by hydra at https://hydra.nixos.org/jobset/nixpkgs/haskell-updates. @@ -85,9 +85,9 @@ defined in the file [release-haskell.nix](../../top-level/release-haskell.nix). #### Notify Maintainers -When Hydra finishes building all the packages, you should generate a build -report to notify maintainers of their broken packages. You can do that with the -following commands: +When Hydra finishes building all the updated packages for the `haskell-updates` +jobset, you should generate a build report to notify maintainers of their +newly broken packages. You can do that with the following commands: ```console $ maintainers/scripts/haskell/hydra-report.hs get-report @@ -115,7 +115,7 @@ may be marked broken before merging `haskell-updates` into `master`. After getting the build report, you can see which packages and Hydra jobs are failing to build. The most important jobs are the `maintained` and `mergeable` jobs. These are both defined in -[`release-haskell.nix](../../top-level/release-haskell.nix). +[`release-haskell.nix`](../../top-level/release-haskell.nix). `mergeable` is a set of the most important Haskell packages, including things like Pandoc and XMonad. These packages are widely used. We would like to @@ -139,9 +139,9 @@ $ maintainers/scripts/haskell/hydra-report.hs get-report $ maintainers/scripts/haskell/hydra-report.hs mark-broken-list ``` -This shows a list of packages that are thought to be broken. +This shows a list of packages that reported broken on `x86_64-linux` on Hydra. -Next, run the following command. +Next, run the following command: ```console $ maintainers/scripts/haskell/mark-broken.sh --do-commit @@ -150,31 +150,79 @@ $ maintainers/scripts/haskell/mark-broken.sh --do-commit This first opens up an editor with the broken package list. Some of these packages may have a maintainer in Nixpkgs. If these maintainers have not been given 7 days to fix up their package, then make sure to remove those packages -from the list. After saving and exiting the editor, the following will -happen: +from the list before continuing. After saving and exiting the editor, the +following will happen: - packages from the list will be added to [`configuration-hackage2nix/broken.yaml`](configuration-hackage2nix/broken.yaml) - `hackage-packages.nix` will be regenerated - the [`configuration-hackage2nix/transitive-broken.yaml`](configuration-hackage2nix/transitive-broken.yaml) - file will be regenerated + file will be updated - `hackage-packages.nix` will be regenerated again - everything will be committed ### Merge `haskell-updates` into `master` -(TODO) +Now it is time to merge the `haskell-updates` PR you opened above. -- mark broken packages with `mark-broken.sh` +Before doing this, make sure of the following: + +- All Haskell packages that fail to build are correctly marked broken or + transitively broken. +- The `maintained` and `mergeable` jobs are passing on Hydra. +- The maintainers for any maintained Haskell packages that are newly broken + have been pinged on GitHub and given at least a week to fix their packages. + This is especially important for widely-used packages like `cachix`. + +When you've double-checked these points, go ahead and merge the `haskell-updates` PR. +After merging, **make sure not to delete the `haskell-updates` branch**, since it +causes all currently open Haskell-related pull-requests to be automatically closed on GitHub. ### Additional Info -(TODO) +Here are some additional tips that didn't fit in above. -- you can restart a Hydra evaluation by logging in with github. +- It is possible to start a new Hydra evaluation by logging into Hydra with + your GitHub or Google account. + +- You should occasionally merge the `master` branch into the + `haskell-updates` branch. + + In an ideal world, when we merge `haskell-updates` into `master`, it would + cause few Hydra rebuilds on `master`. Ideally, the `nixos-unstable` + channel would never be prevented from progressing because of needing to + wait for rebuilding Haskell packages. + + In order to make sure that there are a minimal number of rebuilds after + merging `haskell-updates` into `master`, `master` should occasionally be + merged into the `haskell-updates` branch. + + This is especially important after `staging` is merged into `master`, since + there is a high chance that this will cause all the Haskell packages to + rebuild. + + However, as we are working on cleaning up `haskell-updates`, `master` will + continually progress. It may not always be possible to keep the + `haskell-updates` branch fully up-to-date with `master` without causing + mass-rebuilds on the `haskell-updates` jobset. + +- Make sure never to update the Hackage package hashes in + `pkgs/data/misc/hackage/`, or the pinned Stackage Nightly versions on the + release branches (like `release-21.05`). + + This means that the + [`update-hackage.sh`](../../../maintainers/scripts/haskell/update-hackage.sh) + and + [`update-stackage.sh`](../../../maintainers/scripts/haskell/update-stackage.sh) + scripts should never be used on the release branches. + + However, changing other files in `./.` and regenerating the package set is encouraged. + This can be done with + [`regenerated-hackage-packages.sh`](../../../maintainers/scripts/haskell/regenerated-hackage-packages.sh) + as described above. ## Contributor Workflow -(TODO: this section is to describe the type of workflow for non-comitters to +(TODO: this section is to describe the type of workflow for non-committers to contribute to `haskell-updates`) From efd291080d09150c3c13e1cec2643f63043c1b2e Mon Sep 17 00:00:00 2001 From: "(cdep)illabout" Date: Thu, 27 May 2021 21:58:52 +0900 Subject: [PATCH 05/12] haskell-updates: small fixes --- pkgs/development/haskell-modules/HACKING.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/pkgs/development/haskell-modules/HACKING.md b/pkgs/development/haskell-modules/HACKING.md index 2c1f13763284..3f1baa7a4346 100644 --- a/pkgs/development/haskell-modules/HACKING.md +++ b/pkgs/development/haskell-modules/HACKING.md @@ -63,7 +63,7 @@ I will aim to merge this PR **by 2021-TODO-TODO**. If I can merge it earlier, th Our workflow is currently described at `pkgs/development/haskell-modules/HACKING.md`. The short version is this: -* We regularly update the stackage and hackage pins on `haskell-updates` (normally at the beginning of a merge window). +* We regularly update the Stackage and Hackage pins on `haskell-updates` (normally at the beginning of a merge window). * The community fixes builds of Haskell packages on that branch. * We aim at at least one merge of `haskell-updates` into `master` every two weeks. * We only do the merge if the `mergeable` job is succeeding on hydra. @@ -208,8 +208,9 @@ Here are some additional tips that didn't fit in above. mass-rebuilds on the `haskell-updates` jobset. - Make sure never to update the Hackage package hashes in - `pkgs/data/misc/hackage/`, or the pinned Stackage Nightly versions on the - release branches (like `release-21.05`). + [`pkgs/data/misc/hackage/`](../../../pkgs/data/misc/hackage/), or the + pinned Stackage Nightly versions on the release branches (like + `release-21.05`). This means that the [`update-hackage.sh`](../../../maintainers/scripts/haskell/update-hackage.sh) @@ -219,7 +220,7 @@ Here are some additional tips that didn't fit in above. However, changing other files in `./.` and regenerating the package set is encouraged. This can be done with - [`regenerated-hackage-packages.sh`](../../../maintainers/scripts/haskell/regenerated-hackage-packages.sh) + [`regenerate-hackage-packages.sh`](../../../maintainers/scripts/haskell/regenerate-hackage-packages.sh) as described above. ## Contributor Workflow From 532ce05a4e8e131bedb9bfd7145b717bb3066923 Mon Sep 17 00:00:00 2001 From: Dennis Gosnell Date: Sun, 30 May 2021 10:49:58 +0900 Subject: [PATCH 06/12] haskell-updates: more fixes for workflow documentation Co-authored-by: sterni --- pkgs/development/haskell-modules/HACKING.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/pkgs/development/haskell-modules/HACKING.md b/pkgs/development/haskell-modules/HACKING.md index 3f1baa7a4346..d1779c88fd20 100644 --- a/pkgs/development/haskell-modules/HACKING.md +++ b/pkgs/development/haskell-modules/HACKING.md @@ -28,7 +28,7 @@ In this section we create the PR for merging `haskell-updates` into `master`. 1. Update the Stackage Nightly resolver used by Nixpkgs and create a commit: ```console - $ maintainers/scripts/haskell/update-stackage.sh --do-commit + $ ./maintainers/scripts/haskell/update-stackage.sh --do-commit ``` 1. Update the Hackage package set used by Nixpkgs and create a commit: @@ -100,7 +100,7 @@ comment describes which Haskell packages are now failing to build. It also pings the maintainers so that they know to fix up their packages. It may be helpful to pipe `hydra-report.hs ping-maintainers` into `xclip` -(XOrg) or `wl-clip` (Wayland) in order to post on GitHub. +(XOrg) or `wl-copy` (Wayland) in order to post on GitHub. This build report can be fetched and re-generated for new Hydra evaluations. It may help contributors to try to keep the GitHub comment updated with the @@ -139,7 +139,7 @@ $ maintainers/scripts/haskell/hydra-report.hs get-report $ maintainers/scripts/haskell/hydra-report.hs mark-broken-list ``` -This shows a list of packages that reported broken on `x86_64-linux` on Hydra. +This shows a list of packages that reported a build failure on `x86_64-linux` on Hydra. Next, run the following command: @@ -198,7 +198,7 @@ Here are some additional tips that didn't fit in above. merging `haskell-updates` into `master`, `master` should occasionally be merged into the `haskell-updates` branch. - This is especially important after `staging` is merged into `master`, since + This is especially important after `staging-next` is merged into `master`, since there is a high chance that this will cause all the Haskell packages to rebuild. From dcc4f8ee24b1441e9cc011a57b5b2e0a80c853e0 Mon Sep 17 00:00:00 2001 From: "(cdep)illabout" Date: Sun, 30 May 2021 10:51:33 +0900 Subject: [PATCH 07/12] haskell-updates: add dot slash to commands in maintainer docs --- pkgs/development/haskell-modules/HACKING.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/pkgs/development/haskell-modules/HACKING.md b/pkgs/development/haskell-modules/HACKING.md index d1779c88fd20..433865dbd6ca 100644 --- a/pkgs/development/haskell-modules/HACKING.md +++ b/pkgs/development/haskell-modules/HACKING.md @@ -34,13 +34,13 @@ In this section we create the PR for merging `haskell-updates` into `master`. 1. Update the Hackage package set used by Nixpkgs and create a commit: ```console - $ maintainers/scripts/haskell/update-hackage.sh --do-commit + $ ./maintainers/scripts/haskell/update-hackage.sh --do-commit ``` 1. Regenerate the Haskell package set used in Nixpkgs and create a commit: ```console - $ maintainers/scripts/haskell/regenerate-hackage-packages.sh --do-commit + $ ./maintainers/scripts/haskell/regenerate-hackage-packages.sh --do-commit ``` 1. Push these commits to the Nixpkgs repository. @@ -90,8 +90,8 @@ jobset, you should generate a build report to notify maintainers of their newly broken packages. You can do that with the following commands: ```console -$ maintainers/scripts/haskell/hydra-report.hs get-report -$ maintainers/scripts/haskell/hydra-report.hs ping-maintainers +$ ./maintainers/scripts/haskell/hydra-report.hs get-report +$ ./maintainers/scripts/haskell/hydra-report.hs ping-maintainers ``` The `hyda-report.hs ping-maintainers` command generates a Markdown document @@ -135,8 +135,8 @@ Packages that do not get fixed can be marked broken with the following commands. First check which packages are broken: ```console -$ maintainers/scripts/haskell/hydra-report.hs get-report -$ maintainers/scripts/haskell/hydra-report.hs mark-broken-list +$ ./maintainers/scripts/haskell/hydra-report.hs get-report +$ ./maintainers/scripts/haskell/hydra-report.hs mark-broken-list ``` This shows a list of packages that reported a build failure on `x86_64-linux` on Hydra. @@ -144,7 +144,7 @@ This shows a list of packages that reported a build failure on `x86_64-linux` on Next, run the following command: ```console -$ maintainers/scripts/haskell/mark-broken.sh --do-commit +$ ./maintainers/scripts/haskell/mark-broken.sh --do-commit ``` This first opens up an editor with the broken package list. Some of these From 197dae2b6359518b48424f7ac9e137ec62958f19 Mon Sep 17 00:00:00 2001 From: "(cdep)illabout" Date: Sun, 30 May 2021 10:58:07 +0900 Subject: [PATCH 08/12] haskell-updates: add link for HACKING.md --- pkgs/development/haskell-modules/HACKING.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/pkgs/development/haskell-modules/HACKING.md b/pkgs/development/haskell-modules/HACKING.md index 433865dbd6ca..c8a74e2f5a02 100644 --- a/pkgs/development/haskell-modules/HACKING.md +++ b/pkgs/development/haskell-modules/HACKING.md @@ -60,7 +60,8 @@ I will aim to merge this PR **by 2021-TODO-TODO**. If I can merge it earlier, th ### haskellPackages Workflow Summary -Our workflow is currently described at `pkgs/development/haskell-modules/HACKING.md`. +Our workflow is currently described in +[`pkgs/development/haskell-modules/HACKING.md`](https://github.com/NixOS/nixpkgs/blob/haskell-updates/pkgs/development/haskell-modules/HACKING.md). The short version is this: * We regularly update the Stackage and Hackage pins on `haskell-updates` (normally at the beginning of a merge window). From bd6a1e163cac33e6b583000fd22de0c69f2bd09d Mon Sep 17 00:00:00 2001 From: "(cdep)illabout" Date: Sun, 30 May 2021 11:51:42 +0900 Subject: [PATCH 09/12] haskell-updates: update workflow documentation to put goal first --- pkgs/development/haskell-modules/HACKING.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/pkgs/development/haskell-modules/HACKING.md b/pkgs/development/haskell-modules/HACKING.md index c8a74e2f5a02..96b9e3393528 100644 --- a/pkgs/development/haskell-modules/HACKING.md +++ b/pkgs/development/haskell-modules/HACKING.md @@ -1,15 +1,16 @@ ## Maintainer Workflow -This is the documentation for periodically merging the `haskell-updates` branch -into the `master` branch. This workflow is performed by members in the -[@NixOS/haskell](https://github.com/orgs/NixOS/teams/haskell) team. -Each member of the team takes a two week period where they are in charge of -merging the `haskell-updates` branch into `master`. +The goal of the [@NixOS/haskell](https://github.com/orgs/NixOS/teams/haskell) +team is to keep the Haskell packages in Nixpkgs up-to-date, while making sure +there are no Haskell-related evaluation errors or build errors that get into +the Nixpkgs `master` branch. -The goal of this workflow is to regularly merge the `haskell-updates` branch -into the `master` branch, while making sure there are no evaluation errors or -build errors that get into `master`. +We do this by periodically merging an updated set of Haskell packages on the +`haskell-updates` branch into the `master` branch. Each member of the team +takes a two week period where they are in charge of merging the +`haskell-updates` branch into `master`. This is the documentation for this +workflow. The workflow generally proceeds in three main steps: @@ -17,7 +18,7 @@ The workflow generally proceeds in three main steps: 1. wait for contributors to fix newly broken Haskell packages 1. merge `haskell-updates` into `master` -We describe each of these steps in a separate section. +Each of these steps is described in a separate section. ### Initial `haskell-updates` PR From 6b80742d4da3b4272a781de647dc3fb90d0b645e Mon Sep 17 00:00:00 2001 From: "(cdep)illabout" Date: Sun, 30 May 2021 13:19:56 +0900 Subject: [PATCH 10/12] haskell-updates: update workflow documentation to explain transitive broken package list --- pkgs/development/haskell-modules/HACKING.md | 28 +++++++++++++++------ 1 file changed, 20 insertions(+), 8 deletions(-) diff --git a/pkgs/development/haskell-modules/HACKING.md b/pkgs/development/haskell-modules/HACKING.md index 96b9e3393528..83aab30a302e 100644 --- a/pkgs/development/haskell-modules/HACKING.md +++ b/pkgs/development/haskell-modules/HACKING.md @@ -155,14 +155,26 @@ given 7 days to fix up their package, then make sure to remove those packages from the list before continuing. After saving and exiting the editor, the following will happen: -- packages from the list will be added to - [`configuration-hackage2nix/broken.yaml`](configuration-hackage2nix/broken.yaml) -- `hackage-packages.nix` will be regenerated -- the - [`configuration-hackage2nix/transitive-broken.yaml`](configuration-hackage2nix/transitive-broken.yaml) - file will be updated -- `hackage-packages.nix` will be regenerated again -- everything will be committed +- Packages from the list will be added to + [`configuration-hackage2nix/broken.yaml`](configuration-hackage2nix/broken.yaml). + This is a list of Haskell packages that are known to be broken. + +- [`hackage-packages.nix`](hackage-packages.nix) will be regenerated. This + will mark all Haskell pacakges in `configuration-hackage2nix/broken.yaml` + as `broken`. + +- The + [`configuration-hackage2nix/transitive-broken.yaml`](configuration-hackage2nix/transitive-broken.yaml) + file will be updated. This is a list of Haskell packages that + depend on a package in `configuration-hackage2nix/broken.yaml` or + `configuration-hackage2nix/transitive-broken.yaml` + +- `hackage-packages.nix` will be regenerated again. This will set + `hydraPlatforms = none` for all the packages in + `configuration-hackage2nix/transitive-broken.yaml`. This makes + sure that Hydra does not try to build any of these packages. + +- All updated files will be committed. ### Merge `haskell-updates` into `master` From adfec8b5e043dec328a4aaea9f5a43cc3ea0029f Mon Sep 17 00:00:00 2001 From: "(cdep)illabout" Date: Sun, 30 May 2021 13:31:29 +0900 Subject: [PATCH 11/12] haskell-updates: update workflow documentation to expand section on merging master into haskell-updates --- pkgs/development/haskell-modules/HACKING.md | 57 +++++++++++---------- 1 file changed, 30 insertions(+), 27 deletions(-) diff --git a/pkgs/development/haskell-modules/HACKING.md b/pkgs/development/haskell-modules/HACKING.md index 83aab30a302e..02085bdd8061 100644 --- a/pkgs/development/haskell-modules/HACKING.md +++ b/pkgs/development/haskell-modules/HACKING.md @@ -176,18 +176,42 @@ following will happen: - All updated files will be committed. +#### Merge `master` into `haskell-updates` + +You should occasionally merge the `master` branch into the `haskell-updates` +branch. + +In an ideal world, when we merge `haskell-updates` into `master`, it would +cause few Hydra rebuilds on `master`. Ideally, the `nixos-unstable` channel +would never be prevented from progressing because of needing to wait for +rebuilding Haskell packages. + +In order to make sure that there are a minimal number of rebuilds after merging +`haskell-updates` into `master`, `master` should occasionally be merged into +the `haskell-updates` branch. + +This is especially important after `staging-next` is merged into `master`, +since there is a high chance that this will cause all the Haskell packages to +rebuild. + ### Merge `haskell-updates` into `master` Now it is time to merge the `haskell-updates` PR you opened above. Before doing this, make sure of the following: -- All Haskell packages that fail to build are correctly marked broken or - transitively broken. -- The `maintained` and `mergeable` jobs are passing on Hydra. -- The maintainers for any maintained Haskell packages that are newly broken - have been pinged on GitHub and given at least a week to fix their packages. - This is especially important for widely-used packages like `cachix`. +- All Haskell packages that fail to build are correctly marked broken or + transitively broken. + +- The `maintained` and `mergeable` jobs are passing on Hydra. + +- The maintainers for any maintained Haskell packages that are newly broken + have been pinged on GitHub and given at least a week to fix their packages. + This is especially important for widely-used packages like `cachix`. + +- Make sure you first merge the `master` branch into `haskell-updates`. Wait + for Hydra to evaluate the new `haskell-updates` jobset. Make sure you only + merge `haskell-updates` into `master` when there are no evaluation errors. When you've double-checked these points, go ahead and merge the `haskell-updates` PR. After merging, **make sure not to delete the `haskell-updates` branch**, since it @@ -200,27 +224,6 @@ Here are some additional tips that didn't fit in above. - It is possible to start a new Hydra evaluation by logging into Hydra with your GitHub or Google account. -- You should occasionally merge the `master` branch into the - `haskell-updates` branch. - - In an ideal world, when we merge `haskell-updates` into `master`, it would - cause few Hydra rebuilds on `master`. Ideally, the `nixos-unstable` - channel would never be prevented from progressing because of needing to - wait for rebuilding Haskell packages. - - In order to make sure that there are a minimal number of rebuilds after - merging `haskell-updates` into `master`, `master` should occasionally be - merged into the `haskell-updates` branch. - - This is especially important after `staging-next` is merged into `master`, since - there is a high chance that this will cause all the Haskell packages to - rebuild. - - However, as we are working on cleaning up `haskell-updates`, `master` will - continually progress. It may not always be possible to keep the - `haskell-updates` branch fully up-to-date with `master` without causing - mass-rebuilds on the `haskell-updates` jobset. - - Make sure never to update the Hackage package hashes in [`pkgs/data/misc/hackage/`](../../../pkgs/data/misc/hackage/), or the pinned Stackage Nightly versions on the release branches (like From 95e7f26425430ef4c9f176a84807cf4d0a4bd4ae Mon Sep 17 00:00:00 2001 From: "(cdep)illabout" Date: Sun, 30 May 2021 14:16:03 +0900 Subject: [PATCH 12/12] haskell-updates: update workflow documentation with a few small fixes --- pkgs/development/haskell-modules/HACKING.md | 28 +++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/pkgs/development/haskell-modules/HACKING.md b/pkgs/development/haskell-modules/HACKING.md index 02085bdd8061..8032614a408d 100644 --- a/pkgs/development/haskell-modules/HACKING.md +++ b/pkgs/development/haskell-modules/HACKING.md @@ -221,8 +221,29 @@ causes all currently open Haskell-related pull-requests to be automatically clos Here are some additional tips that didn't fit in above. -- It is possible to start a new Hydra evaluation by logging into Hydra with - your GitHub or Google account. +- Hydra tries to evalute the `haskell-updates` branch (in the + [`nixpkgs:haskell-updates`](https://hydra.nixos.org/jobset/nixpkgs/haskell-updates) + jobset) every 4 hours. It is possible to force a new Hydra evaluation without + waiting 4 hours by the following steps: + + 1. Log into Hydra with your GitHub or Google account. + 1. Go to the [nixpkgs:haskell-updates](https://hydra.nixos.org/jobset/nixpkgs/haskell-updates) jobset. + 1. Click the `Actions` button. + 1. Select `Evaluate this jobset`. + 1. If you refresh the page, there should be a new `Evaluation running since:` line. + 1. Evaluations take about 10 minutes to finish. + +- It is sometimes helpful to update the version of + [`cabal2nix` / `hackage2nix`](https://github.com/NixOS/cabal2nix) that our + maintainer scripts use. This can be done with the + [`maintainers/scripts/haskell/update-cabal2nix-unstable.sh`](../../../maintainers/scripts/haskell/update-cabal2nix-unstable.sh) + script. + + You might want to do this if a user contributes a fix to `cabal2nix` that + will immediately fix a Haskell package in Nixpkgs. First, merge in + the PR to `cabal2nix`, then run `update-cabal2nix-upstable.sh`. Finally, run + [`regenerate-hackage-packages.sh`](../../../maintainers/scripts/haskell/regenerate-hackage-packages.sh) + to regenerate the Hackage package set with the updated version of `hackage2nix`. - Make sure never to update the Hackage package hashes in [`pkgs/data/misc/hackage/`](../../../pkgs/data/misc/hackage/), or the @@ -240,6 +261,9 @@ Here are some additional tips that didn't fit in above. [`regenerate-hackage-packages.sh`](../../../maintainers/scripts/haskell/regenerate-hackage-packages.sh) as described above. +- The Haskell team members generally hang out in the Matrix room + [#haskell-space:matrix.org](https://matrix.to/#/#haskell-space:matrix.org). + ## Contributor Workflow (TODO: this section is to describe the type of workflow for non-committers to