Please review the KB Artifactory Cleanup Best Practices and Relative Time Operators in AQL
For Artifactory Cleanup , you can search using JFrog CLI for artifacts you want to delete in Artifactory using:
- a FileSpec for Searching Files
Example: jf rt s --spec <xyz.spec>.
- AQL using the
/api/search/aqlREST API.
Note: When using AQL inside a FileSpec you cannot specify the attributes to include in the output . The following attributes will always be included:
.include("name","repo","path","actual_md5","actual_sha1","sha256","size","type","modified","created","property")
artifacts_not_downloaded_for_3months.spec
jf rt s --spec parameterize_repo_name/artifacts_not_downloaded_for_3months.spec --spec-vars "RepoName=asaf-test" Note: Executing the below command will list the files (matched by the filespec) not downloaded for 3 months and will interactively ask for deletion. Entering 'Y' at this point in the CLI will delete those files.
jf rt del --spec parameterize_repo_name/artifacts_not_downloaded_for_3months.spec --spec-vars "RepoName=asaf-test" docker_tags_not_downloaded_for_3_months.spec
jf rt s --spec cleanup_unused_docker_image_tag/docker_tags_not_downloaded_for_3_months.spec \
--spec-vars "RepoName=docker-dev-local"
Note: Executing the below command will list the manifest.json of all docker images not downloaded for 3 months and will interactively ask for deletion. Do not enter 'Y' at this point in the CLI as it will delete only the manifest.json . We want to delete the full docker tag and not just the manifest.json.
jf rt del --spec cleanup_unused_docker_image_tag/docker_tags_not_downloaded_for_3_months.spec \
--spec-vars "RepoName=demoreg"
- Verify the docker images to cleanup
Filespec: docker_tags_not_downloaded_for_3_months.spec
jf rt s --spec cleanup_unused_docker_image_tag/docker_tags_not_downloaded_for_3_months.spec --spec-vars \
"RepoName=demoreg;time_range=3mo" | jq -r '.[].path'
- Do the cleanup:
for manifest in $(jf rt s --spec cleanup_unused_docker_image_tag/docker_tags_not_downloaded_for_3_months.spec \
--spec-vars "RepoName=demoreg;time_range=3mo" | jq -r '.[].path'); do
tag=$(echo $(dirname "${manifest}"))
printf "deleting ${tag}\n"
tag_deleted=$(yes| jf rt del "${tag}")
printf "deleted ${tag_deleted}\n"
done
You can also use the python script clean_docker.py that was improvised from KB How to clean up old Docker images
python clean_docker.py http://localhost:8081/artifactory admin password my-docker-repo 3mo
Find all artifacts in a repo which were updated before 3 months but were never downloaded , or were downloaded before 3 months.
Use-case: Since no one downloaded these artifacts recently , they may be good candidates for deletion.
For example: a dev repo which has lots of uploads and needs aggressive cleanup.
Note: To cleanup Federated repos use "updated" instead of "stat.downloaded" in below aql because of RTFACT-26646
curl -s -XPOST https://example.jfrog.io/artifactory/api/search/aql -u**** -H 'Content-Type: text/plain' -d 'items.
find({
"repo": { "$eq": "libs-release-local"
},
"type": "file",
"$or":[
{
"$and": [
{ "stat.downloads": { "$eq":null } },
{ "updated": { "$before": "3mo" } }
]
},
{
"$and": [
{ "stat.downloads": { "$gt": 0 } },
{ "stat.downloaded": { "$before": "3mo" } }
]
}
]
}).include("repo", "path", "name", "size", "actual_sha1", "sha256", "created", "modified", "updated", "created_by", "modified_by", "stat.downloaded", "stat.downloads")'or:
jf rt curl -s -XPOST /api/search/aql -H 'Content-Type: text/plain' -d 'items.find({
"repo": { "$eq": "asaf-test"
},
"type": "file",
"$or":[
{
"$and": [
{ "stat.downloads": { "$eq":null } },
{ "updated": { "$before": "3mo" } }
]
},
{
"$and": [
{ "stat.downloads": { "$gt": 0 } },
{ "stat.downloaded": { "$before": "3mo" } }
]
}
]
}).include("repo", "path", "name", "size", "actual_sha1", "sha256", "created", "modified", "updated", "created_by", "modified_by", "stat.downloaded", "stat.downloads")'
REPO=libs-release-local
jf rt curl -s -XPOST /api/search/aql -H 'Content-Type: text/plain' -d "items.find({ \"repo\": { \"\$eq\":\"${REPO}\" }, \"type\": \"file\", \"\$or\":[ { \"\$and\": [ { \"stat.downloads\": { \"\$eq\":null } }, { \"updated\": { \"\$before\": \"3mo\" } } ] }, { \"\$and\": [ { \"stat.downloads\": { \"\$gt\": 0 } }, { \"stat.downloaded\": { \"\$before\": \"3mo\" } } ] } ] }).include(\"repo\", \"path\", \"name\", \"size\", \"actual_sha1\", \"sha256\", \"created\", \"modified\", \"updated\", \"created_by\", \"modified_by\", \"stat.downloaded\", \"stat.downloads\")"or
repositoryName="asaf-test"
curl -s -XPOST https://example.jfrog.io/artifactory/api/search/aql -usureshv -H 'Content-Type: text/plain' -d 'items.
find({
"repo": { "$eq": "'"${repositoryName}"'"
},
"type": "file",
"$or":[
{
"$and": [
{ "stat.downloads": { "$eq":null } },
{ "updated": { "$before": "3mo" } }
]
},
{
"$and": [
{ "stat.downloads": { "$gt": 0 } },
{ "stat.downloaded": { "$before": "3mo" } }
]
}
]
}).include("repo", "path", "name", "size", "actual_sha1", "sha256", "created", "modified", "updated", "created_by", "modified_by", "stat.downloaded", "stat.downloads")'
or
repositoryName="asaf-test"
jf rt curl -s -XPOST /api/search/aql -H 'Content-Type: text/plain' -d 'items.find({
"repo": { "$eq": "'"${repositoryName}"'"
},
"type": "file",
"$or":[
{
"$and": [
{ "stat.downloads": { "$eq":null } },
{ "updated": { "$before": "3mo" } }
]
},
{
"$and": [
{ "stat.downloads": { "$gt": 0 } },
{ "stat.downloaded": { "$before": "3mo" } }
]
}
]
}).include("repo", "path", "name", "size", "actual_sha1", "sha256", "created", "modified", "updated", "created_by", "modified_by", "stat.downloaded", "stat.downloads")'
or
or
repositoryName="asaf-test"
query='items.find({
"repo": { "$eq": "'"${repositoryName}"'" },
"type": "file",
"$or": [
{
"$and": [
{ "stat.downloads": { "$eq": null } },
{ "updated": { "$before": "3mo" } }
]
},
{
"$and": [
{ "stat.downloads": { "$gt": 0 } },
{ "stat.downloaded": { "$before": "3mo" } }
]
}
]
}).include("repo", "path", "name", "size", "actual_sha1", "sha256", "created", "modified", "updated", "created_by", "modified_by", "stat.downloaded", "stat.downloads")'
jf rt curl -s -XPOST /api/search/aql -H 'Content-Type: text/plain' -d "${query}"
or
repositoryName="asaf-test"
query=$(cat query_with_REPO_NAME.aql | sed "s/REPO_NAME/$repositoryName/g")
jf rt curl -s -XPOST /api/search/aql -H 'Content-Type: text/plain' -d "$query"
Example using AQL inline
curl -u $MYUSER:$MYPASSWORD -XPOST -H 'Content-Type: text/plain' http://$MYSERVERHOST_IP/artifactory/api/search/aql -d 'items.find({ "repo": "local-goldimages-legacy-generic-aws-us-east-1", "path": { "$match": "http:/*" }, "name": { "$match": "*filelists.xml.gz" } }).include("repo", "path", "name", "size", "actual_sha1", "sha256", "created", "modified", "updated", "created_by", "modified_by", "stat.downloaded", "stat.downloads")'
or
using a file containing the AQL : item_search.aql
curl -u $MYUSER:$MYPASSWORD -XPOST -H 'Content-Type:text/plain' http://$MYSERVERHOST_IP/artifactory/api/search/aql -T item_search.aqla) Grep the jf logs as follows to get aqls:
grep "Searching Artifactory using AQL query\|Done transferring folder" ~/.jfrog/logs/jfrog-cli.2022-08-06.16-27-44.31234.log > Searching_Done.txt
b) Then you can execute the aql :
curl -H "Authorization: Bearer $MYTOKEN" -XPOST -H 'Content-Type: text/plain' http://$MYSERVERHOST_IP/artifactory/api/search/aql -d 'items.find({"type":"any","$or":[{"$and":[{"repo":"local-trashcan-generic-us-east-1","path":{"$match":"jenkins-generic-remote-us-west-2-legacy-cache/dev-tools"},"name":{"$match":"*"}}]}]}).include("repo","path","name","type").sort({"$asc":["name"]}).offset(0).limit(10000)'
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent