Needed by nix-community/docker-nixpkgs#46 for example.

I'd like to host our own monitoring so we are self sufficient.

I guess we could switch to something else but easiest option is import the monitoring we're currently using from https://github.com/Mic92/dotfiles.

I'd focus on getting it working here first but eventually we could look at sharing the boilerplate config, maybe https://github.com/numtide/srvos or somewhere else.

Would also like set up a dashboard as well.

Could something like iptables/dns issues w.r.t. docker:

nix-community/emacs-overlay#126 (comment)

https://nixos.org/community/teams/security.html

Like the nixos org I suppose we should have a method and some sort of policy for reporting potential security issues with the infra and repos that don't already have their own security reporting or aren't responsive.

Easiest may be taking reports via github itself: Private vulnerability reporting (beta)

Guess an alternative could be encrypted email but we'd probably want a dedicated address that gets forwarded to everyone rather than it potentially being sent directly to one person who isn't responsive.

I was just thinking a bit about the deploy script a bit.
Would it be possible to extend it, so that it uses screen or tmux when executing the tasks on the remote machine.

The idea is to make it less problematic in case the connection breaks or nixos-rebuild takes a long time and you can disconnect and reconnect at a later point.

Last week I discovered that it's possible to create issue templates with form fields:

https://github.com/NixOS/foundation/blob/master/.github/ISSUE_TEMPLATE/funding_form.yml

We could create issue templates for common operations and help clarify our workflows.

For example:

• move a repo
• create a new repo
• deploy a service
• ...

What do you think?

Similar to how @ryantm used to do at https://discourse.nixos.org/t/nixpkgs-update-r-ryantm-logs/1464

let's see

@adisbladis
@flokli
@grahamc
@Mic92
@nlewo
@ryantm
@zimbatm
@zowoq

I propose to write down a few rules for the administrators of this project for transparency reasons and to make it more democratic.

Responsibilities as an administrator

Our mission is to support the users of the org, and the Nix project in general.

This is a voluntary effort, on a best-effort basis. Things that are good to do are:

• keep the systems updated and patched
• reply to user requests
• host new services that could help the project

Quorum of 5

To reduce the attack surface on the project, I propose limiting the number of administrators to 5 people.

In the current configuration, I propose to keep Mic92 and zowoq because they are actively managing the infra, and ryantm so he can deploy his bot. That leaves 2 other people from the existing list.

How to become an administrator

In order to become an administrator, we ask that you already started contributing to this repo through PRs, and that you are trustworthy. Trust is built on top of personal relationships and past behaviour.

Right now we trust everybody on this list, so that makes it easy for you to come back.

Did I miss anything? Does that sound reasonable?

Is there a free service we can use to multiple [email protected] to all our emails?

Could be useful for #392 as well

It looks like there is a machine that could be used for building aarch64-linux packages, build04 - however, it doesn't seem to be used and builds are instead rejected with "Unsupported system type". Is this intentional?

Example failing build: https://hydra.nix-community.org/build/14455702

Result of smartctl --all /dev/sd{a,b}:

sdb.log
sda.log

Summary:

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
No failed Attributes found.
# ...
202 Percent_Lifetime_Used   0x0030   001   001   001    Old_age   Offline  FAILING_NOW 99

Apparently disks are EOL, no data error visible.
Should we ask Hetzner for new drives or ignore the error?

Until recently, the nixpkgs-swh script was executed every day by a buildkite agent. However, this agent no longer exists.

Would it be possible to run a daily systemd cron jobs?
Since it evaluates the whole nixpkgs repository, it takes a bit of time (about 20min on my laptop i guess).
Moreover, we would need to expose the generated JSON file. We would then have to spawn a nginx server to serve it. Note this file doesn't strictly need to be persisted since it is generated each day.

Note that using hydra.nix-community.org seems to be difficult because the script calls hydra.nixos.org to get latest evaluated nixpkgs commits.

WDYT?

Should we try to enable the discussions? This might help create a more community feel to the org. Let people raise issues, and have longer form communication than on the Matrix channel.

https://github.com/nix-community/templates

How would I go about becoming a maintainer?

I would like to keep the license the same.

Basically type up a nicer version of #399 and anything else that might be needed.

I'll follow up on this in the next week or two.

This has been mentioned on #nix-community matrix a few times now.

I want to move marvin-2k to build03. I saw that it currently uses webhooks as callbacks.
What domain did you configure as a callback, if it is hardcoded can we make a new subdomain just for marvin-2k?

Any objections to adding the https://github.com/apps/settings app to the org?

It introduces a small attack window where users with push access can change the settings of their repo. But it also makes the config more transparent and self-serving. IMO the trade-off is worth it.

          I think it might be worth to have a backup in case someone maliciously deletes repositories or github decides to ban us for whatever reason (we just received a DMCA takedown order).

I have some terraform code to automatically create mirrors from github to gitlab: https://github.com/Mic92/dotfiles/blob/main/terraform/gitlab/repo-mirror.tf

Originally posted by @Mic92 in #399 (comment)

This would allow for keeping the domains short. Eg: https://hydra.n11y.org
But maybe that's just my dislike of typing :)

I'd like to move the marvin-mk2 bot (repo) onto nix-community infrastructure before starting to test it on nixpkgs.

It should consume very little resources. Would that be possible? If so, how should I go about it?

https://github.com/organizations/nix-community/settings/secrets/actions

• CACHIX_AUTH_TOKEN
• CACHIX_SIGNING_KEY

I think we had all agreed these needed to be removed, how to we want to do it? 60 days notice seems reasonable to me?

Projects that want to use the nix-community cachix need to move to hercules or hydra, alternatively they can create their own separate cachix.

This will mean that there is no nix-community cache for darwin as we currently don't have a builder for that platform.

We have the Settings app for repository owners to self-configure their repos.

What's missing is to describe all of the teams and members as code to increase transparency even further.

That also helps create a membership list, potentially allow members to send PRs to add new repos, ...

It's come up in the poetry2nix matrix chat that a build box for macos would be useful for debugging, as macos builds are causing some blockers currently. Would both aarch64-darwin and x86_64-darwin be needed? I'm happy to help sponsor some of the costs of this.

similar to the aarch64 community builder it would be nice to offer the same service for x86.
This is especially interesting for people with less capable hardware (but potentially more time)
to help pushing the project.

Dear nix-community team,

I've noticed that you're running niv-updater-action without PR descriptions (the changelog). As I'm constantly looking to improve my software and help people, I'm wondering why you're doing it that way? Would you prefer to have something else in the PR description?

I've created an issue for your usecase: knl/niv-updater-action#51

Thanks,
Nikola

I want to avoid hidden dependencies. Right now the domains are held by numtide. We need a credit card for this.

Right now it seems to spawn a new cachix instance for every derivation
instead of passing multiple derivation at the time.
This is very slow for to big amount of emacs derivation in our hydra.

cc @grahamc

Since the last nixpkgs upgrade hydra-send-stats.service fails like this:

● hydra-send-stats.service
     Loaded: loaded (/nix/store/ws5saprglyw83al17hwpzvsnllf78h39-unit-hydra-send-stats.service/hydra-send-stats.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sat 2021-03-20 05:40:50 UTC; 1min 51s ago
    Process: 28067 ExecStart=hydra-send-stats (code=exited, status=25)
   Main PID: 28067 (code=exited, status=25)
         IP: 0B in, 0B out
        CPU: 92ms

Mar 20 05:40:50 nix-community-build01 systemd[1]: Started hydra-send-stats.service.
Mar 20 05:40:50 nix-community-build01 hydra-send-stats[28067]: Undefined subroutine &main::getHydraConfig called at /nix/store/m3mz9nv3wad8qigf9mb88k8adk6pwacm-hydra-2021-03-10/bin/.hydra-se>
Mar 20 05:40:50 nix-community-build01 systemd[1]: hydra-send-stats.service: Main process exited, code=exited, status=25/n/a
Mar 20 05:40:50 nix-community-build01 systemd[1]: hydra-send-stats.service: Failed with result 'exit-code'.

Really I'd prefer to do it just for this repo but unfortunately that doesn't seem to be possible.

Only a few that don't already have 2fa set:

8/141 members

8/33 collaborators

7 of the 16 have bot or ci in their name.

This could be used to pay for #396 and servers.

Right now servers are paid for by ryantm and Numtide.

It looks like the builds are intended to end up in Cachix:

However, I've tried to check it with Kittybox. The build output is present in the Hydra, but is not signed. Cachix doesn't contain the output.

Is this intentional?

5 days ago emacs package updates stopped in the nix-community/emacs-overlay repo probably due to f47b49e. See the log at https://gitlab.com/nix-community/emacs-overlay/-/jobs/967612808, from line 485 and on.

Related issue: nix-community/emacs-overlay#104

cc @ryantm

• buildkite - not currently in use (looks like we could use terraform for this)
• cachix
• cloudflare - #408
• #425
• hercules - looks like org owners get admin automatically
• matrix
  • #community-monitoring:nixos.dev
  • #nix-community:nixos.org
• open collective
• terraform cloud - #394, #404
• gitlab

I'll leave the server hardware off this list as we have do have access via sops secrets and consolidating those accounts is more a finance issue.

Do the same treatment as terraform cloud. Give the admins access to cloudflare.

nixpkgs-update has been stuck for a day trying to do nixpkgs-review for pythonPackages.fastapi, a package with only 10 reverse dependencies. It seems like nix-build is not reliably running. I've left it in this state in case someone with more knowledge about the nix-daemon or nix-build can diagnose the problem more.

@zimbatm thinks it might be a problem with the nix-daemon having trouble with man concurrent builds

cc @adisbladis

Not really a problem at the moment but if we keep adding accounts to the top level secrets.yaml we may want something easier to use?

1password has an open source program: https://github.com/1Password/1password-teams-open-source

Could try bitwarden as well but I think we'd need to host it ourselves or pay for it?

https://r.ryantm.com/log/

I was going to do a manual backup of these logs before reinstalling build02 but I suppose we should really be doing regular backups so they aren't all lost if build02 dies.

If we don't have somewhere external to nix-community to back them up to I guess we'd want backups on a couple of our machines?

cc @nix-community/admin

Since nix-community now hosts marvin-mk2, it would be nice if I could deploy new versions myself. This is especially relevant on major changes where I'd like to control the timing and fixes which should be deployed quickly.

In addition to that, I'd like to transfer the bot instance itself (https://github.com/apps/marvin-mk2) to nix-community to reduce dependency on a single person, but to do that I while maintaining access to the admin interface I think I would need to be a member.

Previously discussed in #24 and privately with @adisbladis.

CC @adisbladis @flokli @grahamc @nlewo @ryantm @zimbatm

We have a workflow in ethereum.nix that needs to sign commits: nix-community/ethereum.nix#165

Ideally there would be a Github profile for a nix-community account against which gpg keys can be added. If a repository requires signing they would generate a gpg key and set it as a repo secret, then also create a PR against this project to have that GPG key added to the bot profile.

Relevant background:

• https://github.com/Nautilus-Cyberneering/pygithub/blob/main/docs/how_to_sign_automatic_commits_in_github_actions.md#solution-02-using-your-own-pgp-key-as-a-secret
• https://registry.terraform.io/providers/integrations/github/latest/docs/resources/user_gpg_key

I would like to move the funding from GitHub to Open Collective to be more transparent. Open Collective also allows acquiring virtual credit cards, so we could use that to pay for them. We also have an admin@ address that we can use to forward billing emails.

After the last upgrade the machine did not come back.
cc @zimbatm

Using the new post build hook NixOS/nix#2995

We're using https://github.com/numtide/srvos/blob/master/roles/github-actions-runner.nix in a number of places now, maybe the community could also benefit from having faster CI by pushing the builds to permanent machines?

Eg: nix-community/nix-vscode-extensions#4

I'm using this for the docker-nixpkgs repo. This should be added to the list of apps that we manage.

@makefu recommended go-neb: https://github.com/matrix-org/go-neb/blob/master/config.sample.yaml