magenx / magento-2-aws-cluster-terraform Goto Github PK

hashicorp/terraform-provider-aws#23766

Because terraform 4.6 has this issue, it fails with the error:

Each network interface requires a unique device index.

Running

terraform init -upgrade

Will resolve to version 4.8

It looks like the IAM instance profile user is not available when calling commands like:

su ${parameter["BRAND"]} -s /bin/bash -c "git init -b main"

and errors with:

The following profile does not have credentials configured: default. You must configure the access key and secret key for the profile. For more information, see Configure an AWS CLI Profile in the AWS CLI User Guide.

For now I've rewritten to:

cd ${parameter["WEB_ROOT_PATH"]}
git init -b main
git remote add origin ${parameter["CODECOMMIT_APP_REPO"]}
git fetch origin main
git reset origin/main --hard
chown -R ${parameter["BRAND"]}:${parameter["PHP_USER"]} .

I don't know how to delegate the IAM instance role to another user.

First of all
Thanks for your repo (magenx / Magento-2-aws-cluster-terraform) and you helping us to give an alternative of Magento Cloud Commerce.
Definitely your AWS Auto scale setup will save a lot of money.

Question :

I followed your guide after

• sudo yum install -y yum-utils
• sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo
• sudo yum -y install terraform
• mkdir magento && cd magento
• git clone https://github.com/magenx/Magento-2-aws-cluster-terraform.git .
  (Using my own domain and did all email verification i.e. AWS Domain Certificate verification via email)
• terraform init
• terraform apply

what would be the next steps ?

After finding OUTER_ALB_DNS_NAME URL its leading to Welcome to nginx page
and if I using
OUTER_ALB_DNS_NAME/ADMIN_PATH - Its leading to 404 page

Am I missing something ?
Do I need to run composer on admin EC2 cluster ?

Please help

Error: Unsupported argument

  on packer.pkr.hcl line 55:
  (source code not available)

An argument named "instance_metadata_tags" is not expected here.

Not sure what to do in this case, because instance_metadata_tags is critical to the instances to support the user script that uses the tags for initializing variables to properly build the AMI...

Looks like the amazon plugin for packer doesn't support it yet:

hashicorp/packer-plugin-amazon#205

You'd need to build it yourself to use it...

The nginx user doesn't have access to read the /var/run/php/php7.4-fpm.sock file. setfacl has allowed nginx access to read the web root, but not the php socket.

Also, a constant seems to have not been replaced in nginx configuration:
nginx: [emerg] host not found in "INSTANCE_LOCAL_IP:80" of the "listen" directive in /etc/nginx/sites-enabled/magento.conf:5

I've replaced with:

listen 80;

Because SES_ENDPOINT is missing, this line in installspec.yml triggers an error:

bin/magento config:set smtp/configuration_option/host ${parameter["SES_ENDPOINT"]}

I suggest we add this variable. Perhaps like so:

"SES_ENDPOINT" : "email.${data.aws_region.current.name}.amazonaws.com",

I traced this to the terraform aws provider not supporting metadata and added a ticket:
hashicorp/terraform-provider-aws#24006

But after manually updating in GUI and building images, the same thing happens. When I looked at the build server that was created, it had "Access to tags in instance metadata" disabled.

I think it's best to refactor to use parameters in imagebuilder.

To the build.yml:

parameters:
  - PARAMETERSTORE_NAME:
      type: string
      description: Project name
  - INSTANCE_NAME:
      type: string
      description: Instance name (e.g. admin/frontend)
#...
PARAMETERSTORE_NAME={{ PARAMETERSTORE_NAME }}
_INSTANCE_NAME={{ INSTANCE_NAME }}

To the imagebuilder.tf:

    parameter {
      name        = "PARAMETERSTORE_NAME"
      value       = "${local.project}-env"
    }

    parameter {
      name        = "INSTANCE_NAME"
      value       = "${each.key}"
    }

Hi there ,

thanks for your work , but you need to fix brackets on line 1442 and 985 for magex.tf to run terraform init.

I did not wanted to create a pr for that.

Thank you.

I just tried setting this up to use us-east-1 and found that the availability zone use1-az3 (which includes us-east-1e) was selected for setting up the subnet and subsequent messagequeue "mq.m5.large" was unavailable.

https://wolfman.dev/posts/exclude-use1-az3/

 data "aws_availability_zones" "azs" {
   state            = "available"
+  exclude_zone_ids = ["use1-az3"]
 }

Originally posted by @youanden in #15

The terraform apply went really well, and it looks like every needed resource was provisioned. I'm now trying to access the Magento storefront, and I notice that there are two ALBs - an inner and an outer. When I point my browser at the DNS entry for the outer ALB, I get the nginx configuration page that just says "Welcome to nginx!"

Nginx tries to find the image in the s3 bucket and returns a 404. The result is that the user can't see the captcha and for example is unable to reset the password. Current workaround is to disable the captcha feature.

magento/magento2#33051

We should probably implement a dynamic image_id for aws image builder:
ec2_autoscaling.tf here:

image_id      = tolist(aws_imagebuilder_image.this[each.key].output_resources.0.amis).0.image

Error: Missing resource instance key
│
│ on codepipeline.tf line 213, in resource "aws_iam_policy" "codepipeline":
│ 213: "Resource": aws_codestarconnections_connection.github.arn,
│
│ Because aws_codestarconnections_connection.github has "for_each" set, its attributes must be accessed on specific instances.
│
│ For example, to correlate with indices of a referring resource, use:
│ aws_codestarconnections_connection.github[each.key]

WAF Web ACLs -> Associated AWS resources: ALB

AWS WAFv2 is enabled by default and associated with ALB.
Disable WAF for testing or whitelist your external resources to connect.

AWSManagedRulesBotControlRule
https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-list.html
https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-list.html#aws-managed-rule-groups-bot

Since codebuild containers do not get assigned a public IP by default when assigned to a VPC, each codebuild configuration doesn't have internet with the current configuration.

I've set up a separate private subnet, route table, and public NAT (connected to public subnet), so codebuild is now configured with that private subnet and goes through the NAT to pull from composer, etc.

thank you providing resources.

For our case we dont need varnish (we have pwa shop)

how complicated to omit varnish from infrastructure?

regards

Because kms is enabled by default on all the buckets, including media, the iam user for media doesn't have permissions to write to the bucket.

I think we should probably just disable encryption for the media folder given that it is private by default anyways and requires valid ACL.

I've updated the rule like so:

resource "aws_s3_bucket_server_side_encryption_configuration" "this" {
  for_each = var.s3
  bucket   = "${local.project}-${random_string.s3[each.key].id}-${each.key}"
  rule {
    dynamic "apply_server_side_encryption_by_default" {
      for_each = each.key != "media" ? [1] : []
      content {
        kms_master_key_id = "aws/s3"
        sse_algorithm     = "aws:kms"
      }
    }
  }
}

The problem also happens with the backup bucket when the install codepipeline runs the last step, since the IAM role doesn't have access to kms either.

I tried following the instructions using the AWS Cloudshell and ran into an error during terraform apply asking whether I forgot to run packer init.

Turns out, which packer returns /usr/sbin/packer instead of /usr/bin/packer.

I manually cd'd into magento/packer and ran /usr/bin/packer init packer.pkr.hcl, re-ran terraform apply from the magento folder to success.

The system s3 bucket policy in fastly_v4_imagebuilder does not save because the new statement with Principal AWS "*" overrides the current user's permissions (or so it seems). You can try updating in the GUI and see that S3 will not accept this statement and will return 403.

For now I have replaced the statement to:

{
  Action = ["s3:PutObject"],
  Effect = "Allow"
  Resource = "${aws_s3_bucket.this["system"].arn}/imagebuilder/*"
  Principal = {
    AWS = [
      aws_iam_role.ec2["frontend"].arn,
      aws_iam_role.ec2["admin"].arn
    ]
  }
}

Which is then enough for imagebuilder to write the component artifacts.

Because of this, user_data fails to initialize the code repository correctly with:

error: unknown switch `b'

With ubuntu, this is as simple as:

add-apt-repository ppa:git-core/ppa -y
apt-get -qqy install ${parameter["LINUX_PACKAGES"]}

Alternatively, user_data can be updated to:

git init
git symbolic-ref HEAD refs/heads/main

Without any build changes.

After a successful imagebuilder AMI is started, fstab reads:

LABEL=cloudimg-rootfs   /        ext4   defaults,discard        0 1
LABEL=UEFI      /boot/efi       vfat    umask=0077      0 1
${parameter["EFS_DNS_TARGET"]}:/data/var ${parameter["WEB_ROOT_PATH"]}/var nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,_netdev 0 0
${parameter["EFS_DNS_TARGET"]}:/data/pub/media ${parameter["WEB_ROOT_PATH"]}/pub/media nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,_netdev 0 0

The imagebuilder must use double quotes to force variable interpolation like so:

echo "${parameter["EFS_DNS_TARGET"]}:/data/var ${parameter["WEB_ROOT_PATH"]}/var nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,_netdev 0 0" >> /etc/fstab
echo "${parameter["EFS_DNS_TARGET"]}:/data/pub/media ${parameter["WEB_ROOT_PATH"]}/pub/media nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,_netdev 0 0" >> /etc/fstab

This adds the SMTP Transport from Laminas, to configure external SMTP with authentication.
no more 3rd party modules for email.
magento/magento2#26104
magento/magento2#32241

[cloudshell-user@ip-10-132-36-46 magento]$ terraform init

Initializing the backend...
Terraform encountered problems during initialisation, including problems
with the configuration, described below.

The Terraform configuration must be valid before initialization so that
Terraform can determine which modules and providers need to be installed.
╷
│ Error: Missing key/value separator
│
│ on ec2_profile.tf line 84, in resource "aws_iam_role_policy" "ses_send":
│ 66: policy = jsonencode({
│ 67: {
│ 68: Version = "2012-10-17",
│ 69: Statement = [
│ 70: {
│ 71: Sid = "EC2ProfileSESSendPolicy${each.key}",
│ 72: Effect = "Allow",
│ 73: Action = [
│ 74: "ses:SendEmail",
│ 75: "ses:SendRawEmail"
│ 76: ],
│ 77: Resource = "*",
│ 78: Condition = {
│ 79: StringEquals = {
│ 80: "ses:FromAddress" = var.app["admin_email"]
│ 81: }
│ 82: }
│ 83: }]
│ 84: }]
│
│ Expected an equals sign ("=") to mark the beginning of the attribute value.
╵

[cloudshell-user@ip-10-132-36-46 magento]$

Data transfer between web server and redis cache is too high

magento/magento2#33685

MariaDB 10.5 support

magento/magento2@255247b

I was personally able to run this command myself locally, so not sure why codebuild doesn't work.

[Container] 2022/03/21 16:25:18 Running command while IFS== read -r key value; do parameter["$key"]="$value"; done < <(echo ${PARAMETERSTORE} \| jq -r 'to_entries[] \| .key + "=" + .value')
38 | /codebuild/output/tmp/script.sh: line 4: 7.4: syntax error: invalid arithmetic operator (error token is ".4")
39 |  
40 | [Container] 2022/03/21 16:25:18 Command did not exit successfully while IFS== read -r key value; do parameter["$key"]="$value"; done < <(echo ${PARAMETERSTORE} \| jq -r 'to_entries[] \| .key + "=" + .value') exit status 1
41 | [Container] 2022/03/21 16:25:18 Phase complete: PRE_BUILD State: FAILED
42 | [Container] 2022/03/21 16:25:18 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: while IFS== read -r key value; do parameter["$key"]="$value"; done < <(echo ${PARAMETERSTORE} \| jq -r 'to_entries[] \| .key + "=" + .value'). Reason: exit status 1
43

Hi again,

it would be great if you can add support to connect instances with pre-existing or new key pair as plus to session manager.

Hi, thank you for your work.
I am facing below mentioned issue:

root@server:/home/mgtx2f1/public_html# su mgtx2f1 -s /bin/bash -c "bin/magento module:enable --all"
PHP Warning: require(/home/mgtx2f1/public_html/vendor/composer/../smile/elasticsuite/src/module-elasticsuite-catalog-graph-ql/registration.php): failed to open stream: No such file or directory in /home/mgtx2f1/public_html/vendor/composer/autoload_real.php on line 73
PHP Fatal error: require(): Failed opening required '/home/mgtx2f1/public_html/vendor/composer/../smile/elasticsuite/src/module-elasticsuite-catalog-graph-ql/registration.php' (include_path='/home/mgtx2f1/public_html/vendor/magento/zendframework1/library:.:/usr/share/php') in /home/mgtx2f1/public_html/vendor/composer/autoload_real.php on line 73
root@server:/home/mgtx2f1/public_html# ls -lah vendor/smile/elasticsuite/src/
total 112K
drwxrws---+ 14 mgtx2f1 php-mgtx2f1 4.0K Aug 26 11:06 .
drwxrws---+ 3 mgtx2f1 php-mgtx2f1 4.0K Aug 26 11:06 ..
drwxrws---+ 3 mgtx2f1 php-mgtx2f1 4.0K Aug 26 11:06 lib
drwxrws---+ 4 mgtx2f1 php-mgtx2f1 4.0K Aug 26 11:06 module-elasticsuite-admin-notification
drwxrws---+ 9 mgtx2f1 php-mgtx2f1 4.0K Aug 26 11:06 module-elasticsuite-analytics
drwxrws---+ 14 mgtx2f1 php-mgtx2f1 4.0K Aug 26 11:06 module-elasticsuite-catalog
drwxrws---+ 13 mgtx2f1 php-mgtx2f1 4.0K Aug 26 11:06 module-elasticsuite-catalog-optimizer
drwxrws---+ 9 mgtx2f1 php-mgtx2f1 4.0K Aug 26 11:06 module-elasticsuite-catalog-rule
drwxrws---+ 18 mgtx2f1 php-mgtx2f1 4.0K Aug 26 11:06 module-elasticsuite-core
drwxrws---+ 9 mgtx2f1 php-mgtx2f1 4.0K Aug 26 11:06 module-elasticsuite-indices
drwxrws---+ 8 mgtx2f1 php-mgtx2f1 4.0K Aug 26 11:06 module-elasticsuite-swatches
drwxrws---+ 14 mgtx2f1 php-mgtx2f1 4.0K Aug 26 11:06 module-elasticsuite-thesaurus
drwxrws---+ 14 mgtx2f1 php-mgtx2f1 4.0K Aug 26 11:06 module-elasticsuite-tracker
drwxrws---+ 15 mgtx2f1 php-mgtx2f1 4.0K Aug 26 11:06 module-elasticsuite-virtual-category

as per the composer.lock file, it should be present in the package folder.

Hello Thanks for your hard work ,

but if you’re already setup elasticsearch domain on your account can not create Service Linked Role because already created on account.

How to skip creating service linked Role I couldn't find module block to add create_iam_service_linked_role = false

Terraform giving below error.

"Error creating service-linked role with name es.amazonaws.com: InvalidInput: Service role name AWSServiceRoleForAmazonElasticsearchService has been taken in this account, please try a different suffix."

there is still error when import products with images,

magento/magento2#30978

Update: after deep dive to System Manager , I saw that SSM couldn't run command on instances for SSM Document runShellScript to install magento. I also saw that ssm document created already should I run manually for build instance machine ?

Originally posted by @selcuk-cofe in #5 (comment)

I've traced an issue in the codepipeline install when configuring redis cache/session. The errors are fairly generic, and now I understand are related to the fact that codebuild doesn't have php-lz4 installed, which causes the commands to fail in codebuild for things like config and compilation.

Perhaps we run some sed commands to enable the compression after build is complete before commits are done? I couldn't find a straightforward way to install lz4 or any other php extension in codebuild without compiling from source or getting php from another repo.

The imagebuilder/build.yml file references PARAMETERSTORE_NAME which isn't defined anywhere. Same for INSTANCE_NAME

I suggest making the following modifications.

build.yml:

_AWSTOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 60")
PARAMETERSTORE_NAME=$(curl -s -H "X-aws-ec2-metadata-token: ${_AWSTOKEN}" http://169.254.169.254/latest/meta-data/tags/instance/Project)-env
_INSTANCE_NAME=$(curl -s -H "X-aws-ec2-metadata-token: ${_AWSTOKEN}" http://169.254.169.254/latest/meta-data/tags/instance/InstanceName)

(changes in the file referencing the name without the _)

imagebuilder.tf:

resource_tags = {
  Resource = "${local.project}-${each.key}-image"
  Project = "${local.project}"
  InstanceName = "${each.key}"
}

(using InstanceName instead of Name because it's reserved in resource_tags)

The launch templates already include the Project tag, so this would just ensure that the Build/Test EC2 instances include it for proper builds.