confetti-clj / confetti Goto Github PK
View Code? Open in Web Editor NEW๐ A tool to help creating and deploying static sites with AWS
License: Mozilla Public License 2.0
๐ A tool to help creating and deploying static sites with AWS
License: Mozilla Public License 2.0
Users are supposed to specify an edn file but edn can't easily hold java.io.File objects.
sync!
$ boot -d confetti create-site --domain domain.name --access-key $AWS_ACCESS_KEY_ID --secret-key $AWS_SECRET_ACCESS_KEY --dns
Reporting events generated while creating your stack.
Be aware that creation of CloudFront distributions may take up to 15min.
- [create-in-progress] AWS::CloudFormation::Stack - User Initiated
- [create-in-progress] AWS::S3::Bucket
- [create-in-progress] AWS::Route53::HostedZone
- [create-in-progress] AWS::S3::Bucket - Resource creation Initiated
- [create-in-progress] AWS::Route53::HostedZone - Resource creation Initiated
- [create-complete] AWS::S3::Bucket
- [create-in-progress] AWS::S3::BucketPolicy
- [create-in-progress] AWS::CloudFront::Distribution
- [create-in-progress] AWS::IAM::User
- [create-in-progress] AWS::S3::BucketPolicy - Resource creation Initiated
- [create-in-progress] AWS::IAM::User - Resource creation Initiated
- [create-in-progress] AWS::CloudFront::Distribution - Resource creation Initiated
- [create-complete] AWS::S3::BucketPolicy
- [create-complete] AWS::Route53::HostedZone
- [create-complete] AWS::IAM::User
- [create-in-progress] AWS::IAM::AccessKey
- [create-in-progress] AWS::IAM::AccessKey - Resource creation Initiated
- [create-complete] AWS::IAM::AccessKey
The command has been stuck at this point for more than 30 minutes. The cloudfront distribution seems to have finished in the meantime.
After setting up a site automatically, when I visit the cloudfront url, I get the error message
ERROR
Failed to contact the origin.
Looking at the CloudFront settings, it looks like the S3 bucket is configured as a custom backend. To fix it I set it up to use the S3 bucket directly (you can select the S3 bucket you want from a list when creating an origin). The origin type is "S3 origin".
I'm using :cache-control
metadata for caching purposes but confetti doesn't update objects when the file has not changed and the metadata has. Not sure if it should but it would be nice if it did ;-)
Fell free to close it if this is not a reasonable feature for the library.
should mention what options are helpful with what, which are required and so on
I cloned project locally and run boot
confetti git:(master) boot
Downloading https://github.com/boot-clj/boot/releases/download/2.0.0/boot.sh...done.
clojure.lang.ExceptionInfo: Unable to resolve symbol: update in this context
data: {:file "confetti/serialize.clj", :line 8}
java.lang.RuntimeException: Unable to resolve symbol: update in this context
...
clojure.core/load/fn core.clj
Probably you want to include boot.properties file int the repo with minimum requirements?
The Readme currently says
Creating a site
Let's say you want to deploy a site at my-app.com. To create an S3 bucket, a CloudFront distribution and restricted access keys you can run the following:
This sounds like its saying that the access key and secret key will be created, but the function signature of create-site
sounds like its expecting values for these options. I'm assuming I need to create an AWS user and create the access key ahead of time, so what does the Confetti Readme mean when it says it will create restricted access keys?
I'm not going to let any program access AWS with keys that can do lots of things. Maybe there is a way to create proper IAM role with only necessary policies to allow stack creation, but I don't know how that would work.
But good way allow stack creation manually would be to add option (or modify --dry-run
) to print the CloudFormation JSON or Yaml template. User can then check the template is fine and create the stack from AWS Console.
Instead of pprint here, https://github.com/confetti-clj/confetti/blob/master/src/confetti/boot_confetti.clj#L193 this could call cheshire/generate-string or similar (with key-fn that converts snake-case to CamelCase). Amazonica probably already has a function to translate clj data to JSON.
What I did:
boot build-jar
boot create-site --domain "code.hashobject.com"
clojure.lang.ExceptionInfo: java.lang.ClassNotFoundException: com.amazonaws.services.cloudsearchdomain.AmazonCloudSearchDomainClient, compiling:(amazonica/core.clj:1:1)
data: {:file
"/var/folders/7d/9c32qg7x2_z9slnznbhq2dg80000gn/T/boot.user7083640999180883928.clj",
:line 27}
java.util.concurrent.ExecutionException: java.lang.ClassNotFoundException: com.amazonaws.services.cloudsearchdomain.AmazonCloudSearchDomainClient, compiling:(amazonica/core.clj:1:1)
clojure.lang.Compiler$CompilerException: java.lang.ClassNotFoundException: com.amazonaws.services.cloudsearchdomain.AmazonCloudSearchDomainClient, compiling:(amazonica/core.clj:1:1)
java.lang.ClassNotFoundException: com.amazonaws.services.cloudsearchdomain.AmazonCloudSearchDomainClient
...
amazonica.core/eval523/loading--auto-- core.clj: 1
amazonica.core/eval523 core.clj: 1
...
clojure.core/load/fn core.clj: 5866
clojure.core/load core.clj: 5865
...
clojure.core/load-one core.clj: 5671
clojure.core/load-lib/fn core.clj: 5711
clojure.core/load-lib core.clj: 5710
...
clojure.core/apply core.clj: 632
clojure.core/load-libs core.clj: 5749
...
clojure.core/apply core.clj: 632
clojure.core/require core.clj: 5832
...
amazonica.aws.cloudformation/eval517/loading--auto-- cloudformation.clj: 1
amazonica.aws.cloudformation/eval517 cloudformation.clj: 1
...
clojure.core/load/fn core.clj: 5866
clojure.core/load core.clj: 5865
...
clojure.core/load-one core.clj: 5671
clojure.core/load-lib/fn core.clj: 5711
clojure.core/load-lib core.clj: 5710
...
clojure.core/apply core.clj: 632
clojure.core/load-libs core.clj: 5749
...
clojure.core/apply core.clj: 632
clojure.core/require core.clj: 5832
...
confetti.cloudformation/eval58/loading--auto-- cloudformation.clj: 1
confetti.cloudformation/eval58 cloudformation.clj: 1
...
clojure.core/load/fn core.clj: 5866
clojure.core/load core.clj: 5865
...
clojure.core/load-one core.clj: 5671
clojure.core/load-lib/fn core.clj: 5711
clojure.core/load-lib core.clj: 5710
...
clojure.core/apply core.clj: 632
clojure.core/load-libs core.clj: 5749
...
clojure.core/apply core.clj: 632
clojure.core/require core.clj: 5832
...
pod$eval54.invoke NO_SOURCE_FILE
...
clojure.core/eval core.clj: 3081
boot.pod/eval-in* pod.clj: 224
...
boot.pod/eval-in* pod.clj: 227
confetti.boot-confetti/prep-pod boot_confetti.clj: 21
confetti.boot-confetti/eval10209/fn/fn/fn boot_confetti.clj: 56
boot.core/run-tasks core.clj: 730
boot.core/boot/fn core.clj: 740
clojure.core/binding-conveyor-fn/fn core.clj: 1916
...
Sometimes it may be better to invalidate /*
than to invalidate many paths (costs maybe)
I would suggest looking also into Cloudflare and potentially implement integration into confetti.
I want to share my current setup/process for static sites.
I used to use Route53 before cloudflare. But now I think Cloudflare is better because:
It seems that cloudflare has nice API and I suggest looking into it so instead of setting up Route53 and provisioning certificates user would be able to get same features using cloudflare (and for free).
I think if I have cloudflare support I would use confetti instead of stout:)
When I attempt to run sync-bucket, with or without -y switch, I get the following null pointer exception:
โฏ boot -d confetti sync-bucket --bucket my-bucket-redacted --access-key REDACTED --secret-key REDACTED --dir target/public
clojure.lang.ExceptionInfo: java.lang.NullPointerException
data: {:file
"/var/folders/8j/27sz_2rj0kd1b3v9y4n1_1t00000gn/T/boot.user8801486601348864520.clj",
:line 35}
java.util.concurrent.ExecutionException: java.lang.NullPointerException
java.lang.NullPointerException:
...
confetti.boot-confetti/find-confetti-edn boot_confetti.clj: 53
confetti.boot-confetti/read-confetti-edn boot_confetti.clj: 58
confetti.boot-confetti/eval282/fn/fn/fn boot_confetti.clj: 223
boot.core/run-tasks core.clj: 794
boot.core/boot/fn core.clj: 804
clojure.core/binding-conveyor-fn/fn core.clj: 1916
...
The exact same error occurs when I run sync-bucket as part of a comp pipeline.
It might be useful to save the template in the working dir and allow users to modify it and update the existing stack that way.
Another possibility is to just recommend re-creating stacks if a change is desired. This might not help as much with customizations that are not directly supported at creation time though (e.g. CORS policies currently)
The title is actually an attempt to an explanation, but it can be something else.
The exception is the following:
Deploying on bucket devel.clojurescript.io...
Pack source files...
Sifting output files...
clojure.lang.ExceptionInfo: java.lang.RuntimeException: Method code too large!, compiling:(NO_SOURCE_PATH:0:0)
data: {:file "/tmp/boot.user8095246900604741649.clj", :line 67}
clojure.lang.Compiler$CompilerException: java.lang.RuntimeException: Method code too large!, compiling:(NO_SOURCE_PATH:0:0)
java.lang.RuntimeException: Method code too large!
clojure.asm.MethodWriter.getSize MethodWriter.java: 1872
clojure.asm.ClassWriter.toByteArray ClassWriter.java: 775
...
clojure.core/eval core.clj: 3081
boot.pod/eval-in* pod.clj: 437
...
boot.pod/eval-in* pod.clj: 440
confetti.boot-confetti/eval2817/fn/fn/fn boot_confetti.clj: 155
replumb.boot-pack-source/eval2560/fn/fn/fn boot_pack_source.clj: 45
replumb.boot-pack-source/eval2606/fn/fn/fn boot_pack_source.clj: 57
boot.task.built-in/fn/fn/fn/fn built_in.clj: 416
boot.user/eval3014/fn/fn/fn boot.user8095246900604741649.clj: 45
boot.core/run-tasks core.clj: 927
boot.core/boot/fn core.clj: 937
clojure.core/binding-conveyor-fn/fn core.clj: 1916
...
I am basically including the following dependencies:
(def deps '[[org.clojure/clojurescript "1.7.228"]
[org.clojure/core.async "0.2.374"]
[reagent "0.5.1"]
[re-frame "0.5.0"]
[replumb/replumb "0.2.2-SNAPSHOT"]
[org.clojure/tools.reader "1.0.0-alpha3"]])
And the following task:
(core/deftask count-files
[]
(core/with-pre-wrap fileset
(util/warn "Fileset count %s" (count (tmpdir/ls fileset)))
fileset))
I get Fileset count 1704
. Maybe I am pushing it to the limit ๐
Current assertion is useless
(assert creds "Credentials are required!")
maybe no validation is required since the utility libs validate things as well.
When I re-deploy something, and I actually don't know what triggers the error at the moment, I get:
clojure.lang.ExceptionInfo:
data: {:file "/tmp/boot.user6757680306871265920.clj", :line 67}
java.lang.reflect.InvocationTargetException:
java.lang.IllegalArgumentException: The parameter previousObjectListing must be specified.
com.amazonaws.services.s3.model.ListNextBatchOfObjectsRequest.setPreviousObjectListing ListNextBatchOfObjectsRequest.java: 51
com.amazonaws.services.s3.model.ListNextBatchOfObjectsRequest.<init> ListNextBatchOfObjectsRequest.java: 33
Do you have some hint on how to debug it?
Retrieving jsr-275-0.9.3.jar from https://repo1.maven.org/maven2/
clojure.lang.ExceptionInfo: java.lang.IllegalArgumentException: Cannot create enum from eu-west-2 value!
data: {:file
"/var/folders/8j/27sz_2rj0kd1b3v9y4n1_1t00000gn/T/boot.user7464173975716694273.clj",
:line 35}
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Cannot create enum from eu-west-2 value!
java.lang.IllegalArgumentException: Cannot create enum from eu-west-2 value!
com.amazonaws.regions.Regions.fromName Regions.java: 71
com.amazonaws.services.s3.internal.S3RequestEndpointResolver.resolveRequestEndpoint S3RequestEndpointResolver.java: 68
com.amazonaws.services.s3.internal.S3V4AuthErrorRetryStrategy.redirectToRegionInHeader S3V4AuthErrorRetryStrategy.java: 88
com.amazonaws.services.s3.internal.S3V4AuthErrorRetryStrategy.shouldRetryWithAuthParam S3V4AuthErrorRetryStrategy.java: 73
com.amazonaws.http.AmazonHttpClient.executeOneRequest AmazonHttpClient.java: 781
com.amazonaws.http.AmazonHttpClient.executeHelper AmazonHttpClient.java: 489
com.amazonaws.http.AmazonHttpClient.execute AmazonHttpClient.java: 310
com.amazonaws.services.s3.AmazonS3Client.invoke AmazonS3Client.java: 3604
com.amazonaws.services.s3.AmazonS3Client.invoke AmazonS3Client.java: 3557
com.amazonaws.services.s3.AmazonS3Client.listObjects AmazonS3Client.java: 647
...
amazonica.core/fn-call/fn core.clj: 789
amazonica.core/intern-function/fn core.clj: 860
...
confetti.s3-deploy/get-bucket-objects/fn s3_deploy.clj: 29
confetti.s3-deploy/get-bucket-objects s3_deploy.clj: 29
confetti.s3-deploy/sync! s3_deploy.clj: 134
pod$eval3061.invoke NO_SOURCE_FILE
...
clojure.core/eval core.clj: 3081
boot.pod/eval-in* pod.clj: 369
...
boot.pod/eval-in* pod.clj: 372
confetti.boot-confetti/eval3106/fn/fn/fn boot_confetti.clj: 276
boot.task.built-in/fn/fn/fn/fn built_in.clj: 402
boot.task.built-in/fn/fn/fn/fn built_in.clj: 402
jeluard.boot-notify/eval11725/fn/fn/fn boot_notify.clj: 60
io.perun/eval2147/fn/fn/fn perun.clj: 396
io.perun/eval1651/fn/fn/fn perun.clj: 116
boot.core/run-tasks core.clj: 794
boot.core/boot/fn core.clj: 804
clojure.core/binding-conveyor-fn/fn core.clj: 1916
...
>>> elapsed time 24m53s
A user might already have a hosted zone for abc.com
but now wants to add a static CDN to cdn.abc.com
. It would be possible to create a separate hosted zone but this would require adding this hosted zones nameservers to the domain introducing additional friction.
Alternatively confetti's Cloudformation tooling could check if a hosted zone for the root domain already exists. And optionally add the required record set to the existing hosted zone.
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-route53-recordset.html
I tried to use confetti for my clojurecup entry but hit following problem:
Tried to create site without dns (I wanted to use cloudflare). I used root domain (lifelines.io).
Got an error that it's not possible to create site for root domain without setting up dns.
Is this by design?
Mar 14, 2018 5:59:39 PM com.amazonaws.services.s3.internal.S3V4AuthErrorRetryStrategy buildRetryParams
WARNING: Attempting to re-send the request to testi-metosin-fi-sitebucket-1vvyctvha8ad5.s3.eu-central-1.amazonaws.com with AWS V4 authentication. To avoid this warning in the future, please use region-specific endpoint to access buckets located in regions that require V4 signing.
- [update] blog/schema-spec-web-devs/index.html
I just took a look at the possible args for create-site, and I don't see anything immediately referencing regions.
an improvement could be to check if only cloudfront is outstanding and exit at that point with a note that it may take up to 10 more minutes to have the distribution synced
Required for that is that is that outputs for stacks can be fetched before creation is fully done.
Not sure if outputs are available before the stack is fully created.
Also cloudfront ID is a useful output value that most likely is not available before the distribution is fully created.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.