Exploration and documentation of my experiments deploying Elasticsearch and the various facets of the ELK stack (Elasticsearch, Logstash, and Kibana) using the sophisticated amalgamation of Terraform and Amazon Web Services (AWS).
This narrative not only encapsulates the mechanistic aspects of automated deployments but also delves into the intricate challenges and nuances that such an integration presents.
While not primed for production, it offers invaluable insights, underscoring my dedication to mastering cutting-edge technologies and showcasing my intellectual rigor in navigating complex cloud-based infrastructures.
- Sign up an AWS account
- Create an IAM User
- Add the
AmazonEC2FullAccesspermission policy to a new group
- Create an EC2 Key Pair
- Create an EC2 Security Group
- For IPv6, edit your VPC & add a IPv6 CDIR
- Launch an EC2 Instance
- Select
debian, on at2.micro, using your keypair & security group created earlier
- Select
sudo apt-get install -y awscli && aws configureNote: If you get errors about ImportError: cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_': python -m pip install requests "urllib3<2
sudo apt-get install -y gnupg software-properties-common
wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | sudo tee /usr/share/keyrings/hashicorp-archive-keyring.gpg
gpg --no-default-keyring --keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg --fingerprint
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt-get update && sudo apt-get install -y terraformsudo apt-get install -y gnupg apt-transport-https
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt-get update && sudo apt-get install elasticsearch kibana logstash
sudo certbot certonly --standalone --preferred-challenges http -d elastic.domain.org- Copy your certificates to
/etc/elasticsearch/certs:
mkdir -p /etc/elasticsearch/certs/
sudo cp /etc/letsencrypt/live/elastic.domain.org/fullchain.pem /etc/elasticsearch/certs/fullchain.pem
sudo cp /etc/letsencrypt/live/elastic.domain.org/privkey.pem /etc/elasticsearch/certs/privkey.pem
sudo chmod -R 777 /etc/elasticsearch/certs/- Edit your
/etc/elasticsearch/elasticsearch.ymland change the follow options:
cluster.name: BeeHive
node.name: gibson
network.host: 0.0.0.0
bootstrap.memory_lock: true
xpack.security.audit.enabled: true
xpack.security.http.ssl:
enabled: true
key: /etc/elasticsearch/ssl/privkey.pem
certificate: /etc/elasticsearch/ssl/fullchain.pem- System changes:
sudo su
ulimit -n 65535
ulimit -u 4096
echo "elasticsearch - nofile 65535" > /etc/security/limits.conf
mkdir -p /etc/systemd/system/elasticsearch.service.d/
echo "[Service]\nLimitMEMLOCK=infinity" > /etc/systemd/system/elasticsearch.service.d/override.conf
sudo swapoff -a
sudo sysctl -w vm.swappiness=1 # Add these
sudo sysctl -w vm.max_map_count=262144 # to /etc/systctl.conf
sudo sysctl -w net.ipv4.tcp_retries2=5 # - Set the password for Kibana:
./usr/share/elasticsearch/bin/elasticsearch-reset-password -u kibana_system
./usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token --scope kibana # Save this for when we access Kibana the first time
./usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node # enrollment token for a new node
- Copy your certificates to
/etc/kibana/certs:
mkdir -p /etc/kibana/certs/
sudo cp /etc/letsencrypt/live/elastic.domain.org/fullchain.pem /etc/kibana/certs/fullchain.pem
sudo cp /etc/letsencrypt/live/elastic.domain.org/privkey.pem /etc/kibana/certs/privkey.pem- Edit your
/etc/kibana/kibana.ymland change the follow options:
server.host: "0.0.0.0"
server.publicBaseUrl: "https://elastic.domain.org"
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/fullchain.pem
server.ssl.key: /etc/kibana/certs/privkey.pem
elasticsearch.hosts: ["https://elastic.domain.org:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "changeme" # Use the password from the reset command we did earlier- Copy your certificates to
/etc/logstash/certs:
mkdir -p /etc/logstash/certs/
sudo cp /etc/letsencrypt/live/elastic.domain.org/fullchain.pem /etc/logstash/certs/cacert.pem- Edit your
/etc/logstash/logstash.ymland change the follow options:
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["https://elastic.domain.org:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
user => "elastic"
password => "changeme"
cacert => "/etc/logstash/cacert.pem"
}
}logstash-plugin install logstash-input-irc
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service && sudo systemctl start elasticsearch.service
sudo systemctl enable kibana.service && sudo systemctl start kibana.service
sudo systemctl enable logstash.service && sudo systemctl start logstash.service
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent