Develop a log ingestor system
The requirements for the log ingestor and the query interface are specified below.
- Develop a mechanism to ingest logs in the provided format.
- Ensure scalability to handle high volumes of logs efficiently.
- Mitigate potential bottlenecks such as I/O operations, database write speeds, etc.
- Make sure that the logs are ingested via an HTTP server, which runs on port
3000
by default.
- Offer a user interface (Web UI or CLI) for full-text search across logs.
- Include filters based on:
- level
- message
- resourceId
- timestamp
- traceId
- spanId
- commit
- metadata.parentResourceId
- Aim for efficient and quick search results.
Your submission will be evaluated based on the following criteria.
- Volume: The ability of your system to ingest massive volumes.
- Speed: Efficiency in returning search results.
- Scalability: Adaptability to increasing volumes of logs/queries.
- Usability: Intuitive, user-friendly interface.
- Advanced Features: Implementation of bonus functionalities.
- Readability: The cleanliness and structure of the codebase.
- The Golang server is set up to ingest logs on port 3000.
- The logs are written to a file as part of the logging process.
- Logstash is utilized to send logs from the file to Elastic Cloud.
- The log processing includes the creation of indices to organize the logs effectively.
- The processed logs are made visible in Kibana, providing a centralized platform for log analysis and visualization.
- An optional Kibana dashboard has been considered and may be implemented for monitoring purposes.
- Role-based access control is implemented for Kibana.
- A custom role is created, allowing access to both Kibana and Logstash for specific user roles.
- Scripts are created to simulate log ingestion by bombarding port 3000 with four types of logs:
- Application Logs: 100 requests per second (RPS)
- Application Error Logs: 5 RPS
- Kafka Error Logs: 3 RPS
- DB Error Logs: 2 RPS
- Security measures, including access control and role-based permissions, are implemented to ensure the confidentiality and integrity of the logging system.
- Comprehensive documentation is available, detailing the architecture, data flow, security measures, and the log bombing scripts.
- The system is designed with scalability in mind to handle varying log volumes effectively.
This logging and monitoring system is configured to handle log ingestion, processing, and visualization, with optional monitoring features and robust security measures in place.
To install and run the project, please follow the steps below. Ensure that you have Docker, Docker Compose, and Homebrew installed on your system.
Follow the links to install the required tools:
-
Clone the repository:
gh repo clone repo-link cd log-service-golang
-
Paste the following environment variables into the code. Update the values according to your configuration:
ELASTIC_SEARCH_HOSTS
ELASTIC_SEARCH_USER_NAME
ELASTIC_SEARCH_PASSWORD
3.Run Docker Compose to start the project:
docker-compose up
You will have the server ready to ingest logs at PORT 3000 .
Once the current server is up and running at PORT 3000 and the logs are being ingested the Dashboard can be seen on this url
Since role-based authentication is implemented, please reach out to [email protected]
for the credentials.
To view logs in Kibana, ensure that the correct index, "Log-ingestor," is selected. You can use the following URL to access the logs:
After accessing the logs using this URL, adjust the time filter from the top-right corner to see the latest logs. The image below illustrates this process:
The following are some sample queries that will be executed for validation.
- Find all logs with the level set to "error".
- Search for logs with the message containing the term "Failed to connect".
- Retrieve all logs related to resourceId "server-1234".
- Filter logs between the timestamp "2023-09-10T00:00:00Z" and "2023-09-15T23:59:59Z". (Bonus)
These features aren’t compulsory to implement, however, adding them might increase the chances of your submission being accepted.