Unfortunatelly I got this error when trying to use git in real AWS Lambda
/opt/bin/git: error while loading shared libraries: libpcre2-8.so.0: cannot open shared object file: No such file or directory

Runtime:
nodejs12.x

Used layer:
arn:aws:lambda:eu-central-1:553035198032:layer:git-lambda2:6

Is it possible to update the associated libraries and git version?

I just started using this for a side project of mine. I believe that the version 4 build ships a broken ssh that isn't linked properly to libfipscheck.so.1. This prevents ssh clones from working. Using the version 3 or 2 ARN link works fine.

run using arn:aws:lambda:us-east-2:553035198032:layer:git:4 as a layer

+ ldd /opt/bin/ssh
	linux-vdso.so.1 =>  (0x00007ffc826fd000)
	libfipscheck.so.1 => not found
	libselinux.so.1 => /usr/lib64/libselinux.so.1 (0x00007fdcfddae000)
	libcrypto.so.10 => /var/lang/lib/libcrypto.so.10 (0x00007fdcfd950000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fdcfd74c000)
	libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007fdcfd4fa000)
	liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007fdcfd2eb000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007fdcfd0e8000)

Hi @mhart
The layer is currently not supported on arm64 AWS Lambdas.

On running it, the following error is displayed:
/bin/sh: /opt/bin/git: cannot execute binary file

Can you please provide a workaround for this?

While I was able to attach the layer using AWS console, the same thing cannot be done from the command-line:

rw$ aws lambda update-function-configuration --function-name gitsync --layers arn:aws:lambda:us-east-1:553035198032:layer:git:3

An error occurred (AccessDeniedException) when calling the UpdateFunctionConfiguration operation: User: arn:aws:iam::123132123:user/romaninsh is not authorized to perform: lambda:GetLayerVersion on resource: arn:aws:lambda:us-east-1:553035198032:layer:git:3

user "romaninsh" is full admin.

Hi, I've noticed there is a new security vulnerability in the last few versions of GIT and there is a patch already available in version v2.30.2.

More info from GitHub: https://github.blog/2021-03-09-git-clone-vulnerability-announced/

Would you mind building the latest patched version of GIT into a new layer version so everybody can upgrade as soon as possible?

Also, deprecating the vulnerable versions would be nice.

Thank you, we appreciate your work. :)

I've added your git layer's v6 and copied the sample lambda code, replacing the known_hosts, rsa key and git repo url with my own. I get the following error:

ssh: error while loading shared libraries: libldap-2.4.so.2: cannot open shared object file: No such file or directory

May or may not be relevant, but our git repo is hosted in a TFS VCS server.

Hello,

• I'm trying to import git in a python AWS lambda.

• After following through guide and adding a new layer with ARN: arn:aws:lambda:us-east-1:553035198032:layer:git-lambda2:7 the function is not able to import module.
  

• Verified the lambda's and layer's ARN match which is us-east-1.

• Error Response:

Any guidance is appreciated. Thank You.

Hello!

I'm trying to use your project in my lambda, but due to restriction of my company's infrastructure, I need to deploy the git layer to our own s3.

I uploaded this zip file to our s3 but it did not work, git is still unavailable.

My question is: Is this zip file supposed to work used like this?
It totally can be something on my end, but it appears that everything is as it should, so I came here to exclude this possibility :)

An error occurred: TrainUnderscoreminorLambdaFunction - Function code combined with layers exceeds the maximum allowed size of 262144000 bytes. The actual size is 263669938 bytes. (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: b05441bc-77f9-4416-b9b0-3625bfe59d64).

I had to use number 5 instead of 13 for it to work.

Hi, I'm trying to run git on my lambda function which is on us-gov-west-1, but got an error saying I was unauthorized to perform GetLayer action on the layer's resource arn although my user's IAM policy already covers Lambda. I'm wondering if the resource is available in my region or not. I've tried using the layer on the commercial environment and it works perfectly, so I'm not sure if I'm missing anything... I'd really appreciate your help!

I use PythonGit to clone private repo with ssh and private key, (ref to gitpython-developers/GitPython#725) and it told errors below:

ssh: /lib64/libc.so.6: version GLIBC_2.26' not found (required by ssh) ssh: /lib64/libc.so.6: version GLIBC_2.25' not found (required by ssh)
fatal: Could not read from remote repository.

hope this can be get fixed soon.
thanks.

Hi :)

I'm trying to get this layer working in my lambda
I'm using serverless and have added the layer to my function

layers:
      - arn:aws:lambda:${opt:region}:553035198032:layer:git-lambda2:3

When I try to clone a https:// git repo I get the error

warning: templates not found in /tmp/git/usr/share/git-core/templates

I can see git binaries in /opt/bin

2019-12-17T11:08:11.860Z	7fee8550-01f8-4f1e-8d25-da11de063fae	INFO	log gitBinaryDir contents
2019-12-17T11:08:11.860Z	7fee8550-01f8-4f1e-8d25-da11de063fae	INFO	fipscheck
fipshmac
git
git-receive-pack
git-shell
git-upload-archive
git-upload-pack
idn2
scp
sftp
slogin
ssh
ssh-add
ssh-agent
ssh-copy-id
ssh-keygen
ssh-keyscan
xmlwf

but I also see this env vars in my lambda env

GIT_TEMPLATE_DIR: '/tmp/git/usr/share/git-core/templates',
GIT_EXEC_PATH: '/tmp/git/usr/libexec/git-core',

The question I have is. What is the true location of the templates, so that I can set them correctly in my lambda environment?

Thanks again, for all the good work here, and please keep it up!

I tried to attach the layer as instructed in the readme (arn:aws:lambda:us-west-2:553035198032:layer:git-lambda2:3), and it seems to be there, but when I run my lambda, I get the following error still:

git clone: Cloning into 'route53_monitoring_check'...
/tmp/git/usr/libexec/git-core/git-remote-https: error while loading shared libraries: libpcre.so.0: cannot open shared object file: No such file or directory

I'm sorry if I'm missing something exceptionally obvious, but I'm just not seeing what I missed. Can you help me figure this out? Thanks!

I followed steps to set up git layer using provided ARN with eu-west-1 region. I am doing "from git import Repo", but I do get following error then testing my lambda: "Unable to import module 'lambda_function': No module named 'git'". Any ideas how to solve this?

I tailored the complex example to Python and am receiving an error which seems to make this not work. It's possible the problem is on my side but it seems like something is missing from the example since it's still trying to write to .ssh.

Cloning into '/tmp/git'...
Could not create directory '/home/sbx_user1075/.ssh'.
Warning: Permanently added the RSA host key for IP address '52.95.19.19' to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.

I am updating an open PR for serverless to use git-lambda-layer instead of lambda-git.

Having a package.json is required by npm when I add git-lambda-layer as a github dev-dependency.

Would it be to much trouble to add a package.json?

If you prefer I can do it in a PR so you just have to merge it ;-)

Thanks in advance!

Im trying to connect to Bitbucket and I’m having an issue authenticating. I’m connecting using SSH and I’m wondering if I can just add bitbucket.org to the list ok known hosts as in the example you’ve shown?

Should I keep the line written in known_host or should I adapt it? Sorry I’m not too familiar with all of that.

please see aws/aws-sam-cli#878

Is there a workaround for now?

Hi, after importing this layer, how should I reference git command in python lambda function. any suggestion/help is greatly appreciated.
thank you !

Git layer fails in the new Lambda nodejs10.x container released this week.

Repro steps:

• I used arn:aws:lambda:us-west-1:553035198032:layer:git:5 in us-west-1
• Lambda configured with nodejs10.x
• Ran the following child_process.exec git clone command (from within /tmp): git clone https://$USERNAME:[email protected]/$OWNER/$REPO.git
• Received the following error: error while loading shared libraries: libcurl.so.4: cannot open shared object file: No such file or directory

This isn't particular surprising considering the messaging AWS has been doing this week about the underlying library changes in Lambda nodejs10.x.

Reference:

• https://aws.amazon.com/blogs/compute/upcoming-updates-to-the-aws-lambda-execution-environment/
• https://aws.amazon.com/about-aws/whats-new/2019/05/aws_lambda_adds_support_for_node_js_v10/
• https://aws.amazon.com/amazon-linux-2/

Their announcement email:

Hello,

We are updating the AWS Lambda and AWS Lambda@Edge execution environment to include recent versions of Amazon Linux and software packages.

A majority of functions will seamlessly benefit from the enhancements in this update without requiring you to take any action. However, in rare cases package updates may introduce compatibility issues. Functions that contain libraries or application code compiled against very specific underlying OS packages, specifically those for openssl, glibc, or other system libraries, may potentially be impacted.

Starting May 14, 2019, you can test your functions with the new execution environment. From May 21, 2019, all new functions or updates of existing functions will use the new execution environment. Your existing functions will automatically migrate to using the new execution environment on June 11, 2019.

See the following blog post to learn more about the update including the timeline, testing guidelines, and how you can prepare[1].

[1] https://aws.amazon.com/blogs/compute/upcoming-updates-to-the-aws-lambda-execution-environment/

Sincerely,
Amazon Web Services

Edit: per tweeters, ratcheting down urgency since AWS Linux 2 isn't being forced into other Lambda container images yet; re-scoped issue more specifically to nodejs10.x

Great job! Thank you LambCI

just curious

The example file shows that it dynamically creates two files, known_hosts and id_rsa under the "/tmp" folder
Would it be easier to create and deploy a separate AWS layer (that layer would only contain these two files) , so that these two files will always be there in the "/opt" directory ...
coz AWS supports up to 5 layers at the same time.

Question:
Not so sure if we still need to run chmod 400 in this case. Do you think it would be necessary?

while running ssh command , it shows sth like

Could not create directory '/home/sbx_user1070/.ssh'.

I guess this is related to AWS Lambda's read-only file system.

Even with the HOME env var set to something like /tmp/home/ the error message is still there..

Not sure if it matters . But anyway the error msg bothers me -_| 🤣

Hi,
Thanks for [git-lambda-layer]!
I want to build a function to commit something to my github.
Under the layer, I can clone/add/commit in /tmp/my-repository folder, but when I try to push the change to origin branch, I run into below issue. Could you help me to resolve the issue?

error:
...
Could not create directory '/home/sbx_user1051/.ssh'.
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
...

This is my code

_const { execSync } = require('child_process')
const process = require('process')
const { GITHUB_TOKEN, GITHUB_USERNAME, GITHUB_EMAIL, GITHUB_REPO, PASSWD } = process.env
exports.handler = async (event) => {
const config = {
token: GITHUB_TOKEN,
username: GITHUB_USERNAME,
email: GITHUB_EMAIL,
repo: GITHUB_REPO,
password: PASSWD
}
const gitUrl = github.com/${config.username}/${config.repo}.git
const execOpt = { encoding: 'utf8', stdio: 'inherit' }

//execSync(`git config --global user.email ${config.email}`, execOpt)
//execSync(`git config --global user.name ${config.username}`, execOpt)

execSync(`rm -rf /tmp/${config.repo}`, execOpt)
execSync(`cd /tmp && git clone https://${gitUrl} \

&& ls && cd SI-AutomationTest && echo testxxx>test.txt
&& git config --local user.email ${config.email}
&& git config --local user.name ${config.username}
&& git add test.txt
&& git branch
&& git config --local user.email ${config.email}
&& git config --local user.name ${config.username}
&& git commit -m '[skip ci] hhhaa'
&& git remote -v
&& git config --local user.email ${config.email}
&& git config --local user.name ${config.username}
&& git remote set-url origin [email protected]:${config.username}/${config.rep}.git
&& git config --local user.email ${config.email}
&& git config --local user.name ${config.username}
&& git push -u origin main`, execOpt)
}_

I'm running exactly your second example (substituting my own key and repo) and hitting this error:

ssh: Could not resolve hostname github.com:myorganization: Name or service not known

my key and known_hosts file appear fine, I can run

ssh -T git@github
and get
Hi schwiet! You've successfully authenticated, but GitHub does not provide shell access.

One thing, I noticed is that it seems the documented environment variable is GIT_SSH and not GIT_SSH_COMMAND, but unfortunately changing that did not resolve the problem.

I see in the Dockerfile there is NO_PYTHON=1, I'm assuming this breaks Python support? Is there a reason for this?

How would I go about creating my own Layer that includes Git for use inside Python? I am aiming to use this library which requires git:

https://github.com/dxa4481/truffleHog

I love this layer. However, due to security concerns, we cannot use your layer for our production environment; similarly, we cannot simply download your layer and reupload it to a new layer on our account.

Can you please tell me how you went about creating the binary files so I can follow that process? I tried downloading them from git directly, but those files are heavily bloated compared to yours. Also, is there a way to remove the ssh binaries entirely?

I realize this probably out of scope

We have a Python lambda that runs from the AWS lambda container

FROM public.ecr.aws/lambda/python:3.8

We are running into the issue with git clone command.

PRIV_END: seteuid: Operation not permitted

I have tried setting the environment variable but this is not working

ENV GIT_SSH_COMMAND='ssh -i /root/.ssh/id_rsa -o StrictHostKeyChecking=no'

I see you have the layer.zip with all the binaries - would it be possible to share instructions on how to use this in the lambda container? Do I merge the files into the respective folders in the container?

thanks for any reply - this is the only place I have found where there seems to be a solution for this problem

Running git in lambda, I get /bin/sh: /opt/bin/git: Permission denied.

Using docker run to get interactive shell, I get the same error trying to run git: sh: /opt/bin/git: Permission denied.
The problem seems to be the lack of execution permission on files in /opt/bin:
-rw-rw-rw- 1 sbx_user1051 495 2480144 Dec 24 20:24 git
-rw-rw-rw- 1 sbx_user1051 495 658056 Dec 24 20:24 ssh

Granting permission using chmod +x /opt/bin/* fixes this problem.
But, ssh still doesn't work due to another error: ssh: error while loading shared libraries: /opt/lib/libfipscheck.so.1: file too short, not sure why.

I tested this on an image created and cached by AWS SAM CLI after attempting to locally invoke the lambda.

Hi, I've noticed that the git-lambda-layer does not work with python3.8. It does work fine with 3.6, i have not tested 3.7.

Error:
Cloning into '/tmp/repo'...
/opt/libexec/git-core/git-remote-https: error while loading shared libraries: libcurl.so.4: cannot open shared object file: No such file or directory

Cheers

EDIT:

It works fine on both 3.6 and 3.7.

Hi,

this is a nice idea. Thanks for the project.

However it seems to be working with binary files inside a .zip archive. Could you also opensource the build process for the zip file, so that users don't need to execute binary files on their production environment?

I also want to build this myself, so that I don't have to rely on a arbitrary aws account, that could go offline someday :D.

With kind regards,
Tim

Any chance it would be worthwhile for you guys to add support for git lfs to this layer? Or perhaps that should be something I request be added to yumda?

Hello,

• Followed Complex example on Node.js w/ ssh steps from README.
• Facing issue in authenticating to git over ssh. Verified same creds work locally.
• As per guide, getting the private key from AWS SSM and added public key to GIT and /tmp/known_hosts.
• Setting environment var for GIT_SSH_COMMAND with UserKnownHostsFile=/tmp/known_hosts and StrictHostKeyChecking=no options.

Python Lambda function:

import os
import subprocess
import boto3
import git

def lambda_handler(event, context):

    # get SSH key
    ssm = boto3.client('ssm')
    parameter = ssm.get_parameter(Name='gitLambda')
    gitLambdaKey = parameter['Parameter']['Value']

    # save SSH key in /tmp and chmod permissions
    with open('/tmp/gitLambdaKey', 'w') as outfile:
        outfile.write(gitLambdaKey)
    os.chmod('/tmp/gitLambdaKey', 0o400) # leading 0 in python2 and 0o in python 3 denies octal

    # clean up /tmp and make dir for repo
    os.system("rm -rf /tmp/* ; mkdir /tmp/git")
    # To fix "Warning: Remote Host Identification Has Changed" error clear known hosts for github.com first then add new
    os.system('ssh-keygen -f "/tmp/known_hosts" -R "github.com"')
    
    with open('/tmp/known_hosts', 'w') as outfile:
        outfile.write("github.com,192.30.252.*,192.30.253.*,192.30.254.*,192.30.255.*,140.82.113.* ssh-rsa AAA ... cU=")

    os.environ['GIT_SSH_COMMAND'] = "ssh -o UserKnownHostsFile=/tmp/known_hosts -o StrictHostKeyChecking=no -i /tmp/gitLambdaKey"

    try:
        cmd = 'ssh -v [email protected]'
        output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True, universal_newlines=True)
    except subprocess.CalledProcessError as exc:
        print("Status FAIL:", exc.returncode, exc.output)
    else:
        print("Output: \n{}\n".format(output))

Error with public key in /tmp/known_hosts:

debug1: Reading configuration data /opt/etc/ssh/ssh_config
debug1: /opt/etc/ssh/ssh_config line 58: Applying options for *
Pseudo-terminal will not be allocated because stdin is not a terminal.
debug1: Connecting to github.com [140.82.113.4] port 22.
debug1: Connection established.
debug1: SELinux support disabled
Could not create directory '/home/sbx_user1051/.ssh'.
debug1: key_load_public: No such file or directory
debug1: identity file /home/sbx_user1051/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/sbx_user1051/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/sbx_user1051/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/sbx_user1051/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/sbx_user1051/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/sbx_user1051/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/sbx_user1051/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/sbx_user1051/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version babeld-5a455904
debug1: no match: babeld-5a455904
debug1: Authenticating to github.com:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8
debug1: read_passphrase: can't open /dev/tty: No such file or directory
Host key verification failed.

Any guidance is appreciated. Thank You.

Hi!

I am trying to run git with this layer in a SAM lambda build.

Template snippet:

LambdaFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/
      Handler: app.lambda_handler
      Runtime: python3.7
      Layers:
        - arn:aws:lambda:eu-west-1:553035198032:layer:git:10

Code snippet:

subprocess.call(['git', 'init', '/tmp/Apps/CDN/'])

or

os.system('cd /tmp/Apps/CDN && git init')

All attempts fail with git: error while loading shared libraries: /opt/lib/libpcre2-8.so.0: file too short

I also tried using the second version of the layer (despite py3.7 not being listed for it) instead, but that also fail with the same error.

Any ideas how this can be solved?

Hi , how can i edit the layer to add the aws cli exectuables available inside lambda function ?

Hi everyone :D

I'm trying to get this layer working in my lambda
I'm using serverless and have added the layer to my function

I'm using nodejs12.x and Git version 2.25.0 | OpenSSH_7.4p1, OpenSSL 1.0.2k-fips and the copy of the example "Complex example on Node.js w/ ssh" with my private key hardcoded in the file:

I generated a new private and public key and tested them previously, but in my logs in the CloudWatch i got:

I tried many ways to resolve, but I failed.
Anyway, congrats for the amazing work!
Thanks :)