Set up a public CI for the project about fsharp HOT 63 CLOSED

The first thing we need to do is activate appveyor for visualfsharp project. We need someone with enough appveyor account and enough github rights for this.
I can then try to activate the build on #15 and see where it fails.
We probably also need some help from a appveyor person. Do you know a person?

from fsharp.

I know that VS 2015 CTP 5 + SDK image is coming to AppVeyor CI.
I believe that we can do CI for F# 4.0 option and install all other prerequisites from Chocolatey using FAKE or discuss our needs directly with AppVeyor Team

@forki I think that you can activate AppVeyor for you fork of visualfsharp first

from fsharp.

If the decision is to go for AppVeyor, we probably need a paid account. The free account has time limit which might not be enough for building and testing the compiler.

I think VS 2015 CTP 5 + SDK would be good enough for now. We can always ask @FeodorFitsner (He has been very responsive and helpful on my AppVeyor-related questions).

from fsharp.

I think we should start with the basic unittests and qa tests. If have a CI build which does this it would already be a big win. We could then discuss how to extend it to the VS integration tests and how to get the money for the paid account. I think that this could be something for the FSSF (at least if Microsoft isn't paying).
/cc @ReedCopsey

from fsharp.

This certainly sounds like something that would be perfectly aligned with the core mission of the F# Foundation and I'm pretty sure we would be able to find money for that. But I agree with @forki that we should start with a simpler thing - and then see how to best scale it to the full thing.

from fsharp.

(That said, there might be some formal trickery as this repo is owned by Microsoft and not by the F# Foundation... but I'm not sure to what extent that matters.)

from fsharp.

Does appveyor allow us to run as admin? We need to ngen the proto-compiler.

from fsharp.

Yes, it does e.g. http://help.appveyor.com/discussions/questions/394-installing-a-database-from-mdf

from fsharp.

As a start I activated AppVeyor for my own fork and #15: https://ci.appveyor.com/project/SteffenForkmann/visualfsharp

from fsharp.

I agree this would be a great thing to get setup - and the FSSF could potentially fund it if required.

Would it make more sense to do this on the open edition first? It probably should be on both repositories anyways... That might give us a cleaner way to get this setup, and also to show that it's not disruptive when we approach getting this added to the MS repository.

I can start the conversation and do some research into what would be required on the FSSF/Microsoft side of this.

@dungpa From what I can tell, the 40 minute build limit applies to all versions of AppVeyor - even the paid editions. The main difference there would potentially be better specs on the underlying system, so it might run a bit more quickly, but we're going to have to be careful about build times with any plan...

from fsharp.

First successful build: https://ci.appveyor.com/project/SteffenForkmann/visualfsharp/build/0.0.1.1

from fsharp.

@forki Great.

I wonder what F# team think about the whole thing.

from fsharp.

Steffen,

can you let us know what was needed to enable this for now. In the longer term we were thinking of Travis, because of it's unix support. But I think this is a great first step.

from fsharp.

There is a "getting started" at http://www.appveyor.com/docs

Basically it's just logging in and activating the project. But someone with Ms github account needs to do it.

And then you need to merge #15

from fsharp.

Regarding Travis: I think we need both. Will try to add minimal Travis build next week

from fsharp.

We are certainly in favor of having a CI set up. .NET on *nix is a real thing that we will need to support sometime in the future, so our CI solution should not be Windows-only.

Last I heard from @jaredpar is that Roslyn is investigating Jenkins, as it supports both Windows and Linux. Appveyor is Windows-only, Travis is Linux/Mac-only.

from fsharp.

Yes, Lincoln is correct, Jenkins is what Jared is investigating.

from fsharp.

As I said: we need both. Projectscaffold-like projects already made very
good success with the combination of appveyor and Travis. It's much better
working than all the other combinations we tried before. And we tried a lot
(incl teamcity, Jenkins, cc.net)
On Jan 18, 2015 9:45 PM, "Lincoln Atkinson" [email protected]
wrote:

We are certainly in favor of having a CI set up. .NET on *nix is a real
thing that we will need to support sometime in the future, so our CI
solution should not be Windows-only.

Last I heard from @jaredpar https://github.com/jaredpar is that Roslyn
is investigating Jenkins, as it supports both Windows and Linux. Appveyor
is Windows-only, Travis is Linux/Mac-only.

—
Reply to this email directly or view it on GitHub
#53 (comment)
.

from fsharp.

Thank you for taking the time to investigate this Steffen, it is very important and necessary work, we are very grateful for your expertise and effort. As you can tell we are still learning what is important and how to prioritize it. We know we want to support platforms other than windows, together we will figure out which solutions make sense for this our project, these investigations of yours are a vital part of that effort.

Once again thanks

Kevin

from fsharp.

Just to clarify: the other CI servers also work very well. The thing that made appveyor and Travis make work very well for us is the configuration via yaml files. This allows us as a community to collaboratively maintain the CI process. People can send pull requests to fix CI issues. That's super important and I think other CI servers are also trying to support this configuration style.

from fsharp.

I think an important thing is to get the build infrastructure in place. We need a transparent and objective way to build and test the compiler. I would suggest to start with free versions of AppVeyor/Travis first. They're simple to configure and use. Moving to Jenkins (if desired) should be no brainer when build infrastructure is ready.

from fsharp.

Correct, we are investigating Jenkins at the moment. It should in fact be working now but it appears there was a server hiccup over the weekend. Once it gets fixed on Monday though Roslyn will be on Jenkins.

from fsharp.

Is that a public Jenkins instance or are you hosting this? How is the config/maintenance handled? One issue for us on self-hosted CI servers was the time-zone difference between europe and the US.

from fsharp.

@forki I'm unsure of the details of the current hosting. Meeting with the owners today to better understand their setup and how we can integrate into it.

Why would time zone be an issue?

from fsharp.

Thanks for keeping us updated. I'd love to hear what Roslyn is planning.

Regarding time-zone-issue: we used the public Teamcity service on codebetter.com for a very long time (IIRC since the beginning of FAKE (over 5 years ago)). The people over there were very nice and helpful, but from time to time you need an admin since the server went down or project settings needed adjustments. These points were very painful since there was no way for us to move forward. We needed to wait (at least 8h) until someone on the other side of the ocean rebootet the service and things like that.

Since we are using travis and AppVeyor these issue are completely gone:

1. These CI servers are maintained around the clock and are working in some cloud service which is basically always available. If the service goes down, then someone will immediatly work on it.
2. The configuration is done via yml files. So everybody can try different configs in their own fork. We don't need project admins to press buttons on a UI which normal contributors don't see. From time to time we still need a server admin to install VS SDK or things like that. But that case is very rare.

from fsharp.

+1 to @jaredpar 's comment about sharing with Roslyn and CoreFx. Microsoft has been quite supportive of the efforts as well, and if we combine efforts it improves the likelihood that we can get buyin for premium services if we pick a solution that has that business model. Super glad to see this happening.

from fsharp.

@theoy Thanks for your comment. All things considered, sharing Jenkins infrastructure with Roslyn and CoreFx seems like a good way to move forward.

from fsharp.

I'd be interested in using a service that gives us a specific, consistent dedicated machine, to allow us to do CI performance testing of compiler and/or library (see discussion on other threads about the need for this)

from fsharp.

@dsyme - great point. @jaredpar, do you have more info about our Jenkins discussion and specifically performance testing infrastructure? I know we have this internally at Microsoft for Roslyn already, but now that we're sprinting to open access to everything, it'd be great to get some open perf testing for F# too. So much ❤️ going around right now :)

from fsharp.

@theoy Performance tests would be awesome!

from fsharp.

@theoy how we handle performance testing is still an open issue. The advantage of Jenkins though is we are free to mix Azure VMs and Physical hardware in our setup. The hope is that we can leverage the physical machines for our perf infrastructure.

from fsharp.

CI systems that automatically annotate pull requests with perf impact info would just be ❤️ ❤️ ❤️

from fsharp.

I don't think automatic perf regression tests are the first thing we should do. That's all nice to have and all, but instead we should really focus to get reliable build infrastructure which runs automatically on every pull request and is really easy to maintain for us as a community. We should really only focus on reliable builds.

I just saw roslyn has a build status sticker on the github page (currently red!?) which links to http://corefx-ci.cloudapp.net/jenkins/job/dotnet_roslyn/ - so if people don't want to go with Travis+appveyor then how can we try to get #15 build on this thingy?

from fsharp.

Agree with @forki. Please give this a higher priority.

Without build infrastructure, it's a pain for a newcomer to contribute to the project. It's difficult enough to build and test the compiler. For the IDE integration part, it's even worse...

from fsharp.

AppVeyor is now set up, see https://ci.appveyor.com/project/KevinRansom/visualfsharp-radou/

from fsharp.

We now have appveyor builds which is awesome. But builds are waiting for hours to start. I wonder if the FSSF can sponsor a pro account. http://www.appveyor.com/pricing
I know it's not decided if appveyor is the way to go, but for now it's the best thing we have.

/cc @ReedCopsey @tpetricek @panesofglass

from fsharp.

I ll bring that up to the FSSF board :D
I also think there are a bunch of CI providers that wouldn't mind
supporting this project, again this is something the board can probably
chase up

Andrea
www.roundcrisis.com

On 30 January 2015 at 18:08, Steffen Forkmann [email protected]
wrote:

We now have appveyor builds which is awesome. But builds are waiting for
hours to start. I wonder if the FSSF can sponsor a pro account.
http://www.appveyor.com/pricing
I know it's not decided if appveyor is the way to go, but for now it's the
best thing we have.

/cc @ReedCopsey https://github.com/ReedCopsey @tpetricek
https://github.com/tpetricek @ryanriley https://github.com/ryanriley

—
Reply to this email directly or view it on GitHub
#53 (comment)
.

from fsharp.

I'd like to see AppVeyor used for this Microsoft/visualfsharp project. The great thing about AppVeyor is that all of us who fork and contribute to this project can have our own builds setup in seconds. Microsoft is already an AppVeyor customer listed at www.appveyor.com. Since Microsoft owns this repository, it would be best if they setup the appropriate account.

However, I understand that could take a while. It would be great if FSSF could sponsor something in the meanwhile. FSSF currently uses the fsgit AppVeyor account to build the fsharp repo.
https://ci.appveyor.com/project/fsgit/fsharp

I'm not sure if it is possible, but it would be great if the FSSF could add the webhook for this repository so that builds would be at:
https://ci.appveyor.com/project/fsgit/visualfsharp

Even better would be if the FSSF's account was fsharp instead of fsgit. I'll contact AppVeyor and see what the options are for FSSF.

/cc @FeodorFitsner

http://help.appveyor.com/discussions/questions/927-builds-for-f-software-foundation-fssf

from fsharp.

Hi there

Just an update, I put this up to the board and it looks like this is going ahead :D. We are checking what plan we are getting, but it's looking very good.
Of course, there will be better worded information soon, but I thought it would be good to update.

Great news on Monday FTW :D

from fsharp.

That is great! AppVeyor confirmed that we could setup an fsharp account and get 50% off.

To get faster builds you'd need "Pro" plan which is $29.50/month (50% off) for OSS projects. It includes 1 concurrent job and each additional concurrent job for OSS is $20/month.

Sure, you can have additional fsharp account for that with Pro plan assigned.

If FSSF sponsors an AppVeyor Pro account for CI builds, it is simple to setup and requires minimal coordination.

1. FSSF would upgrade an existing account or create a new one.
2. FSSF would add the visualfsharp project in AppVeyor.
   
3. FSSF would give Microsoft (@KevinRansom) the project webhook.
   
4. Microsoft would update the GitHub project settings with the updated webhook.

CI builds would then be faster and hopefully concurrent at:
https://ci.appveyor.com/project/fsharp/visualfsharp/
instead of slow at:
https://ci.appveyor.com/project/KevinRansom/visualfsharp-radou/

Happy to help out in setup.

from fsharp.

Thanks, Cameron! One note here - you should add "GitHub" project, not Git. If there are no sufficient permissions to add webhook that that step will be skipped (so you can add it later).

"Git" is for "generic" self-hosted Git repos and it requires a different webhook format.

Let me know if you have any questions.

from fsharp.

Also, to order "Pro FOSS" plan the following link should be used: https://ci.appveyor.com/plan/order/oss_pro_monthly

from fsharp.

❤️

from fsharp.

@FeodorFitsner I need some clarification. How do you add a GitHub project that you haven't forked? If forking is required, do you recommend creating a GitHub account specifically for this setup?

from fsharp.

Right, I see. OK, fork it, add it and then edit "Repository name" on General tab of project settings in AV to point parent repo. Works perfect for public projects.

from fsharp.

I believe this is now setup - I will email @KevinRansom with details for the webhook.

from fsharp.

@ReedCopsey Where can we chat briefly?

from fsharp.

I was wrong in the steps outlined above. Here are updated steps:

1. Either upgrade the fsgit AV account or create a new fsharp AV account. Which did you do? Upgrading fsgit may be easier.
2. Connect the AV account to a GH account. fsgit is already connected to fsgit.
3. Fork/clone the visualfsharp project in GH.
4. In AV project settings, simply change the GitHub repository to be Microsoft/visualfsharp.

from fsharp.

@ctaggart Sent you contact info.

from fsharp.

Followup: Microsoft upgraded their account, so the current build is now using a paid account. Build times are dramatically improved.

from fsharp.

from fsharp.

@FeodorFitsner Would you mind to install ActiveState Perl on VS 2015 CTP5 images? We need it for running tests (see #169 (comment)). Thanks.

from fsharp.

lol :D ... the memories

Andrea
www.roundcrisis.com

On 2 February 2015 at 21:25, Steffen Forkmann [email protected]
wrote:

[image: tumblr_inline_n4eqevcgie1raprkq]
https://cloud.githubusercontent.com/assets/57396/6009479/61ae1dee-ab2a-11e4-92a8-12b8c65fceea.gif

—
Reply to this email directly or view it on GitHub
#53 (comment)
.

from fsharp.

Sure, will do that.

from fsharp.

The latest Active Perl has been installed to VS 2015 CTP image.

from fsharp.

Thanks @FeodorFitsner! How do I use it? I've tried kicking off a new CI build at #216. This ultimately invokes perl from the PATH. It still seems to be msys perl, though, see the end of the log here.

Am I using the wrong image?

from fsharp.

You requested Perl on Visual Studio 2015 CTP image and we deployed it there only :)
Looking at https://github.com/Microsoft/visualfsharp/blob/fsharp4/appveyor.yml default image is used. You can try installing Perl with Chocolatey: https://chocolatey.org/packages/ActivePerl

from fsharp.

@FeodorFitsner Thanks for doing that. I was trying to use VS 2015 CTP image today, it is significantly slower than the default image (see https://ci.appveyor.com/project/KevinRansom/visualfsharp-radou/build/0.0.1.134 and #217 (comment)). Do you know why?

I tried to install ActivePerl via Chocolatey. Unfortunately, the package is broken and unmaintained (see comments in your link). Could you please install ActivePerl on the default image? It seems we have to stick with that image for a while.

from fsharp.

It's because once you select a custom image it goes to Azure, because currently Pro environment supports one image only. On Azure it takes time to provision a worker VM and it times slower.

from fsharp.

Will deploy Perl to Pro environment.

from fsharp.

@FeodorFitsner I see. Thanks a lot for the kind help.

from fsharp.

Active Perl 5.20.1 has been deployed to Pro environment. Should be in PATH by default.

from fsharp.

Wonderful. I've got a few green builds using Perl at #217. And they're fast 👍 .

from fsharp.