outsideris / citizen Goto Github PK

Cannot apply a version constraint to module because it has a non Registry URL. Error: Invalid version constraint
so I installed the citizen registry published a sample module.. I see that it published the module but when i try to use the module as per below:
module "az-module {
source = "https://mylocal.registry.com/my-local/az-module/azure//modules/az-sub-module"
version = "0.4.0"
...
}

I get : Cannot apply a version constraint to module "az-module" (at main.tf:x)
because it has a non Registry URL.
Error: Invalid version constraint

I am not sure what I am doing wrong... Thanks in advance!!

My terraform version is Terraform v0.12.30

{"level":50,"time":1550501838893,"msg":"The "path" argument must be of type string. Received type undefined","pid":18928,"hostname":"CPX-8SV0UT1ABW5","name":"citizen","type":"Error","stack":"TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received type undefined\n at assertPath (path.js:39:11)\n at join (path.js:434:7)\n at hasModule (C:\snapshot\citizen\storages\file.js:0:0)\n at Form.router.post.form.on (C:\snapshot\citizen\routes\modules.js:0:0)\n at Form.emit (events.js:182:13)\n at C:\snapshot\citizen\node_modules\multiparty\index.js:579:12\n at process._tickCallback (internal/process/next_tick.js:61:11)","v":1}
{"level":50,"time":1550501838894,"msg":"TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received type undefined\n at assertPath (path.js:39:11)\n at join (path.js:434:7)\n at hasModule (C:\snapshot\citizen\storages\file.js:0:0)\n at Form.router.post.form.on (C:\snapshot\citizen\routes\modules.js:0:0)\n at Form.emit (events.js:182:13)\n at C:\snapshot\citizen\node_modules\multiparty\index.js:579:12\n at process._tickCallback (internal/process/next_tick.js:61:11)","pid":18928,"hostname":"CPX-8SV0UT1ABW5","name":"citizen","v":1}
POST /v1/modules/dev-team/acn/aws/0.0.3 500 16.815 ms - 84

I have all environment paramter set
CITIZEN_ADDR=https://f7b9f89b.ngrok.io
CITIZEN_AWS_S3_BUCKET=terraforms3module
CITIZEN_DATA_PATH=C:\snapshot\citizen\storages
CITIZEN_STORAGE=s3
CITIZEN_STORAGE_PATH=C:\Terraform-Citizen\data

https://www.terraform.io/docs/registry/api.html#list-latest-version-of-module-for-all-providers

In the README, it says CITIZEN_DATA_PATH will set the path where data is stored where in fact CITIZEN_DB_DIR is used. Either readme should be updated or code.

We run citizen on a EC2 instance, and when uploading a large module (a few bytes of terraform and 1 large jar file for a lambda), the citizen process will take a lot of CPU and memory.

Finally, we get this in the logs:

Jun 22 19:06:06 ip-10-110-185-69 citizen: <--- Last few GCs --->
Jun 22 19:06:06 localhost citizen: [1214:0x54136d0] 600792 ms: Mark-sweep 1863.5 (1866.9) -> 1862.6 (1866.9) MB, 2795.7 / 0.0 ms (average mu = 0.072, current mu = 0.014) allocation failure scavenge might not succeed
Jun 22 19:06:06 localhost citizen: [1214:0x54136d0] 602898 ms: Mark-sweep 1863.6 (1866.9) -> 1862.7 (1866.9) MB, 2080.7 / 0.0 ms (average mu = 0.045, current mu = 0.012) allocation failure scavenge might not succeed
Jun 22 19:06:06 localhost citizen: <--- JS stacktrace --->
Jun 22 19:06:06 localhost citizen: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Jun 22 19:06:06 localhost citizen: 1: 0xa123d0 node::Abort() [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 2: 0x962467 node::FatalError(char const*, char const*) [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 3: 0xb3878e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 4: 0xb38b07 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 5: 0xcd91e5 [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 6: 0xcd9b8b v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 7: 0xce7b42 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 8: 0xce8995 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 9: 0xceb35c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 10: 0xcba83b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 11: 0xfeb0ff v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 12: 0x13451f9 [/opt/citizen/citizen]
Jun 22 19:06:07 localhost systemd: citizen.service: main process exited, code=killed, status=6/ABRT
Jun 22 19:06:07 localhost systemd: Unit citizen.service entered failed state.
Jun 22 19:06:07 localhost systemd: citizen.service failed.

Is there a way to solve this?

Hi @outsideris,

I hope you don't mind the question.

We were evaluating Terraform Private Registry solutions and found Citizen beside Anthology.

The latter one has not been updated since 3 years and is lacking a publish feature like you implemented, so we chose Citizen, which has a solid base.

As there's no support for Terraform > 0.11 and the Docker images has not been patched since two years, we are now considering either to contribute to Citizen to try help solving these issues or to start a new project as we'd like to add more auxiliary services like a UI and authentication or even a feature to sell subscriptions to specific modules.

I'd love to hear from you - maybe we can team up.

Yves

https://travis-ci.org/outsideris/citizen/builds/355805319

https://www.terraform.io/docs/registry/api.html#list-available-versions-for-a-specific-module

Hi there. I've been really digging your project. One issue I'm running into is I can't seem to publish when using s3 storage. Local file storage works fine, but I get a 500 error when I publish using s3 storage. Here is the error:

✔ compress the terraform module
✖ publish mjdouble/mj-mod/mjdouble/0.0.1
HTTPError: Response code 500 (Internal Server Error)
    at Request.<anonymous> (/snapshot/citizen/node_modules/got/dist/source/as-promise/index.js:117:42)
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  code: undefined,
  timings: {
    start: 1616247957915,
    socket: 1616247957917,
    lookup: 1616247957917,
    connect: 1616247957918,
    secureConnect: undefined,
    upload: 1616247957919,
    response: 1616247957986,
    end: 1616247957987,
    error: undefined,
    abort: undefined,
    phases: {
      wait: 2,
      dns: 0,
      tcp: 1,
      tls: undefined,
      request: 1,
      firstByte: 67,
      download: 1,
      total: 72
    }
  }
}

https://www.terraform.io/docs/registry/api.html#list-modules

instead manual build on Travis-CI

https://blog.fossa.io/open-sourcing-fossas-build-analysis-in-fossa-cli-5ca75148d86f

If I install this code on AWS, can we invoke CITIZEN PUBLISH from API?

https://www.terraform.io/docs/registry/api.html#search-modules

https://www.terraform.io/docs/registry/api.html#download-source-code-for-a-specific-module-version

I was wondering if there is a way to use this with basic auth or something else in front of it. I know how to configure basic auth on the reverse proxy side, but how to do it on the client side?

https://www.terraform.io/docs/registry/api.html#download-the-latest-version-of-a-module

Can the sdk version for the s3 client be upgraded to at least v3.11.0

It has support for web federated identity, which can be used in EKS cluster

Hi,

It seems that the Docker image hosted on GHCR is not publicly accessible.
Could you make it public so I can give Citizen a spin?

Thanks!

https://www.terraform.io/docs/registry/api.html#get-a-specific-module

At the moment if you want to push a 0.12 module you get the following error:

{"level":50,"time":1558622848948,"msg":"Malformed HCL","pid":7,"hostname":"citizen-68f58c75c9-9kxkb","name":"citizen","type":"Error","stack":"Error: Malformed HCL\n at Object.parse (/www/node_modules/gopher-hcl/index.js:28:25)\n at hclToJson (/www/lib/util.js:31:20)\n at <anonymous>","offset":34,"line":2,"column":17,"v":1} {"level":50,"time":1558622848949,"msg":"Error: Malformed HCL\n at Object.parse (/www/node_modules/gopher-hcl/index.js:28:25)\n at hclToJson (/www/lib/util.js:31:20)\n at <anonymous>","pid":7,"hostname":"citizen-68f58c75c9-9kxkb","name":"citizen","v":1}

I have looked at the node packages, it uses gopher-hcl (not updated since 22 Apr 2017) which uses the node HCL package which isn't updated since 18 Sep 2014.

the node HCL package is a port of https://github.com/hashicorp/hcl but I don't know how I can port it to the newest HCL version.

So how to proceed?

https://hub.docker.com/r/outsideris/citizen/

According to https://github.com/hashicorp/terraform/blob/3aa909ac6e66851e3b514a94d296c49364699433/website/docs/registry/modules/use.html.md#module-versions, a module version should follow semantic versioning. However, it does not seem to handle the full pre-release or build metadata after the major/minor/patch. For example, when I upload something with version of 1.3.2-alpha+12, it just ignores the build metadata (the +12, which in this case is the number of commits post-branch). This is important to avoid conflicts on different builds of the same branch.

Do I need to create some folder manually on the server?

{"level":50,"time":1546914561312,"msg":"ENOENT: no such file or directory, mkdir '/snapshot/www/lib/34c0f200-12ed-11e9-9d35-afd5b76bbae6'","pid":8,"hostname":"citizen-5967b79fb6-zmk75","name":"citizen","type":"Error","stack":"Error: ENOENT: no such file or directory, mkdir '/snapshot/www/lib/34c0f200-12ed-11e9-9d35-afd5b76bbae6'","errno":-2,"code":"ENOENT","syscall":"mkdir","path":"/snapshot/www/lib/34c0f200-12ed-11e9-9d35-afd5b76bbae6","v":1}
{"level":50,"time":1546914561312,"msg":"Error: ENOENT: no such file or directory, mkdir '/snapshot/www/lib/34c0f200-12ed-11e9-9d35-afd5b76bbae6'","pid":8,"hostname":"citizen-5967b79fb6-zmk75","name":"citizen","v":1}
POST /v1/modules/foo/bar/aws/0.1.0 500 309.617 ms - 148

It's annoying to maintain tests.

Hi

I noticed the endpoints and import functionality for publishers has been removed in the c21a466 commit. Can you provide some insight as to why this has been pulled out of citizen?

Terraform requires terraform provider registry, to publish "trusted" providers when serving providers back to clients - without that, terraform will refuse to download providers, and deployments will fail.

Is there a plan to tag a release for the latest changes and publish them to dockerhub? The latest release is 0.3.3 but there are a few new changes ( like #42 ) that I would need to be able to use this. It looks really promising but without version 12 support I won't be able to get very far.

hey there,

I have set the 2 required env vars for publishing on S3 (CITIZEN_AWS_S3_BUCKET with value the name of the bucket, CITIZEN_STORAGE with value "s3") + the service has the required permissions, but I get this error when trying to publish:

✖ publish my/provider/1.0.0
Duplicated (500)
undefined is not a function

The full in the logs:

{"level":50,"time":1628955452335,"pid":6,"hostname":"ip-some-internal-aws-hostname","name":"citizen","stack":"TypeError: undefined is not a function\n at saveProvider (/snapshot/citizen/storages/s3.js)\n at /snapshot/citizen/routes/providers.js\n at Array.map (<anonymous>)\n at Form.<anonymous> (/snapshot/citizen/routes/providers.js)","type":"Error","msg":"undefined is not a function"}

Any clue what I am doing wrong?
It works fine with local storage, if I don't set these 2 env vars.

https://www.terraform.io/docs/registry/api.html#latest-version-for-a-specific-module-provider

https://www.terraform.io/docs/registry/api.html#list-modules

It should be tested with Terraform cli.

https://www.terraform.io/docs/registry/api.html#list-available-versions-for-a-specific-module

There is no document about submodules and root.

Hi,

Am wondering if there are any ways to update or delete an existing module? I don't see any option in the citizen CLI nor the code. Looks like i can't do any of these actions at the moment?

There are a couple of issues with how citizen is publishing providers and the provider routes that causes TF to not be able able to find providers. TF does require the name convention to be terraform-provider-{NAME}. https://www.terraform.io/docs/registry/providers/publishing.html

There are also issues with the urls and routes to the sha256sums and sha256sums.sig. If you are taking PRs, I've worked out the fixes.

If the executable is running inside an AWS EC2 VM or a Docker container running in AWS Elasic Container Service, there should be no need to manually set the ACCESS_ID and SECRET_KEY as the code can access AWS resources under the IAM role assigned to it, making assigning one unnecessary.

Currently a module or provider name with multiple hyphens in its name will result in

The name only contain alphanumeric characters or single hyphens, and cannot begin or end with a hyphen.

However, the official Terraform Registry does allow multiple hyphens as long as they are non-adjacent.

In #5, it only returns the location for a specific module. This project should not depend on other services like GitHub and so on.

It provides a new route where isn't in the specification to download the module tarball.

{"level":50,"time":1575295628557,"msg":"Cannot read property 'replace' of undefined","pid":6,"hostname":"b3a13b71afde","name":"citizen","type":"Error","stack":"TypeError: Cannot read property 'replace' of undefined\n at tmp.dir.stream.pipe.on.submodulesDefinition.forEach (/snapshot/citizen/lib/util.js:0:0)\n at Array.forEach ()\n at Unpack.tmp.dir.stream.pipe.on (/snapshot/citizen/lib/util.js:0:0)","v":1}

I've been testing with citizen for a bit, and it looks very promising. We're having a bit of fun running it on Fargate, which has a downside: there is no persistent local storage, nor an option to mount a shared filesystem.

Since the internal storage API is relatively contained it should be possible to point to an external MongoDB or DocumentDB instance instead, which achieves persistence for stateless containers.

Unexpected errors publishing a provider get suppressed.

Here:
https://github.com/outsideris/citizen/blob/master/lib/provider/provider.js#L61

I added

verbose(response.body)

To try to figure out why I'm getting:

RequestError
Unexpected token < in JSON at position 0

It turns out my problem was actually:

  citizen:client <html>
  citizen:client <head><title>413 Request Entity Too Large</title></head>
  citizen:client <body bgcolor="white">
  citizen:client <center><h1>413 Request Entity Too Large</h1></center>
  citizen:client <hr><center>nginx/1.13.5</center>
  citizen:client </body>
  citizen:client </html>
  citizen:client  +0ms

My nginx config isn't allowing large uploads.

I think verbose logging the error return could help with debugging.

I am using citizen in an ec2 with the servers being a localhost on port 3000. I successfully published a module using citizen and can see a module.tar.gz file that is created under the path modules/namespace/name/provider/version (note: I have actual values for the url, namespace, name, provider and version but I am referring to them as such for ease of reading). I then attempted to define an instance of the module following the example given and using the source url/namespace/name/provider as shown in the example. This returned a 404 error. All different versions of the source returned either 404 or "no source url returned" except for when the source was in the form: url/v1/modules/tarball/namespace/name/provider/module.tar.gz which works. I get all the same results when using localtunnel, which gives an https url instaed of the localhost http url. I am using the default settings for all environment variables except for the CITIZEN_STORAGE but the results are the same for if I use file or s3 here. Is there a path that does not involve including the tar file and tarball in the path that will not trigger a 404?

It should be only deployed with tag.