Issues with security and reliability of our infrastructure about problem-solving HOT 27 OPEN

I'm glad to see there's folks willing to work on infrastructure stuff. My few thoughts, since I was pinged:

• Eliminating situations where only one person can do X is good. The big blocker in the recent outage was getting DNS changes done. Using a service like that suggested by @rba sounds ideal, then we just need to nominate some folks who have access to that.
• I hold rakudo.org and moarvm.org. Ideally I'd point those domains at the same name service used for perl6.org, again to avoid a single point of failure there should I be the person who can't respond quickly enough in the future.
• I'm in favor of an infrastructure label here, and would encourage e.g. @rba to propose themselves on #17 as the subject matter expert.
• I'm generally in favor of containerization, to ease deployment and try to reduce the amount of knowledge needed to get things running. It's not a silver bullet, but in terms of people being able to bring up and host replacement or additional/mirror instances of sites, "just build this container and run it and expose it" is quite undemanding.

from problem-solving.

Quick update:

I'm on the way to sort my thoughts and write a "perl6-infra proposal", which will cover some, bot surely not all aspects. The idea is, even though I know, that I haven't got a complete overview, to write down the current state and the options Matthias and I see.

As I have the chance to talk to @lizmat and @maettu this weekend, I hope to have something next week.

I will then use github and the PR feature to add the proposal document so everyone has the chance to comment on the infrastructure ideas.

from problem-solving.

if everyone else agrees with that course of action

Personally I'm looking for a person who would be our subject-matter expert on that topic. We will add a new label (infrastructure), and then it'd be possible to tackle smaller issues in separate tickets. That person can do such decisions alone, though general problem-solving repo rules apply so some review will be required from others.

from problem-solving.

Adding my 2cents: a proposal should be written by a brave hero that will apply and others who are familiar with topic can review it, probably in a round or two.

Questions that should be addressed are:

• What resources do we have now? Domain names, servers, bots to host, ecosystem data providers, installation files and so on.
• How are we handling those resources now?
• Our primary goals for those resources: maintainability, reliability, security and so on.
• How our infrastructure should be organized to match resources with goals? E.g. "We want two containers behind a load balancer with a heartbeat indicator, because it will be Foo and Bar, also Baz and we'll love it".
• How can we migrate our current infrastructure? The plan here will likely be a bit abstract, without details like "Three certain people, X, Y and Z will have keys, also the server IP will be \d.\d.\d.\d", but more of something like "Our docs site has to be containerized, deployed, we want to store a configuration in perl6-infrastructure organization repository....". This can be separated into sub-tickets and implemented over time.

from problem-solving.

I can report that the p6c.org DNS zone is now in a shared cloudflare account, so that the community administrators have access to change the records.

The same process is under way for perl6.org

from problem-solving.

meta label because nothing else fits. If someone wants to start a new label, please let me know.

from problem-solving.

For some reason I cannot see what labels are possible, nor can I add a label or assign it. If this is intentional, that's fine by me. If not, please give me access :-)

I would think a "infrastructure" label would be appropriate here. Using "meta" here just feels like "dunno" here.

from problem-solving.

@lizmat the list of labels is here and creating new ones is possible as long as somebody is ready to be assigned for that area of expertise. Who'd be that person for infrastructure?

from problem-solving.

Ah, as for not seeing all of the labels, this wasn't exactly intended… But it's also not wrong, as in people shouldn't be adding more labels without modifying the README.

from problem-solving.

With the recent outage of our main server, it is obvious we have to take measures to not only secure "all our bases", but to make them reliable and keep them reachable in case of a failure.

Unfortunately, I lack necessary admin skills for planning out things. Probably ping @kawaii?

Other than this, I can provide some funding for necessary additional servers / services. Not insane amounts of money, but hosting is relatively cheap right now, and not so much resources are actually needed: current DO droplet own by me is the smallest one, and its configuration is much more than enough for the load.

As for the ticket title... I would change it to something more general like "Review, analyze and improve-secure our infrastructure". And for it to not be "too broad" for people to think, maybe add a list of things that have to be considered, including security, reliability, keeping ways to handle possible failures and so on.

from problem-solving.

@lizmat the list of labels is here and creating new ones is possible as long as somebody is ready to be assigned for that area of expertise. Who'd be that person for infrastructure?

I would volunteer to take care of infrastructure stuff. Would be great to have someone else on this topic as well yet.

from problem-solving.

@rba can you review what we have now and propose some initial changes? Also, are you on IRC?

from problem-solving.

Ping @moritz, @niner, @jnthn.

from problem-solving.

I forgot that I'd been pinged here. I'd be happy to get involved looking at the infrastructure side of things and helping to maintain that too.

from problem-solving.

If we need hosting/hardware, I can add some as well. As well as the server located in Germany that's hosting camelia and has all the infrastructure needed to run additional VMs, I could also offer hosting of VMs on Atikon's company infrastructure. We're running a failover cluster behind a DDOS protected proxy.

Is there some documentation of our current website and infrastructure setup? How does it all work? And where?

from problem-solving.

There's some info on https://github.com/perl6/infrastructure-doc/

from problem-solving.

Also, any thoughts on dockerizing all the things we have? Will it help?

from problem-solving.

Yes, I can set aside some time to Dockerize our websites and docs if everyone else agrees with that course of action.

from problem-solving.

OK, if somebody wants to be that person, here's a new ticket for you: #17

from problem-solving.

Haven't been on IRC very often before. Yet I have now setup a bouncer and try to handle it better this way. My nick is 'rba'.

Would recommend we start with DNS, to be sure control over perl6.org and it's subdomains are under control of the community, rather in the hand of a single person.

I've used the following services so far:

• https://www.metanet.ch/email-domains/dns-hosting (5 domains are free; Website in German)
• http://freedns.afraid.org (free or for 5 bucks a month)

I agree on dockerizing the services, as this would give us the flexibility. Yet I'm not an expert in this area yet, as I do small steps with kubernetes ecosystem atm.

And I agree with starting with making a roadmap and to do it's best to have kind of inventory. Subdomains, services, etc. asl Altai-man mentioned already.

from problem-solving.

/me also offers to volunteer.

from problem-solving.

@rba, @maettu please leave a separate comment on #17

from problem-solving.

Now that there's a dedicated label, handing this over to @rba and @maettu.

from problem-solving.

from problem-solving.

@rba any news?

from problem-solving.

rakudo.org and moarvm.org have also had their nameservers changed to point at the shared cloudflare account.

from problem-solving.

perl6.org has also been transferred:

$ whois perl6.org|grep Name\ Server
Name Server: VAL.NS.CLOUDFLARE.COM
Name Server: CLINT.NS.CLOUDFLARE.COM

from problem-solving.