hiring_log_gen's Introduction

hiring_log_gen

Log generator for technical challenges that in Hasura's hiring processes.

Usage

% docker run hasura/hiring_log_gen -h
  -projects uint
        number of projects. Valid between 1 and 1000. (default 25)
  -rate uint
        traces to emit per second, accross all projects. Valid between 1 and 10000 (default 1000)
  -speed uint
        relative speed of time. Valid between 1 -realtime- and 86400 -each second counts for a day-. (default 360)

% docker run hasura/hiring_log_gen -projects 2 -rate 10 -speed 180

Generates something like:

2022/04/05 18:38:34 {"project_id":"e615f4da-d790-499f-a67a-07f288c9b3ee","timestamp":"2022-04-05T18:38:34.1049305+02:00","operation":{"name":"update_greece","runtime":0.34890843682963724,"request_id":"8d5f3a74-73e6-4ac2-8b0e-71279b8580be","response_size":70,"request_size":391,"http_status":200}}
2022/04/05 18:38:34 {"project_id":"9a6e673e-c404-44be-9ec7-f608cdad895a","timestamp":"2022-04-05T18:38:34.10973506+02:00","operation":{"name":"insert_magazine","runtime":0.49770060098541913,"request_id":"298f63b9-f171-40ee-9730-6c608da3035c","response_size":25,"request_size":41,"http_status":200}}
2022/04/05 18:38:34 {"project_id":"e615f4da-d790-499f-a67a-07f288c9b3ee","timestamp":"2022-04-05T18:38:36.128352+02:00","operation":{"name":"select_planet","runtime":0.3610665872273873,"request_id":"92f236db-0a20-4610-affe-a088f2fe395e","response_size":82,"request_size":163,"http_status":200}}
2022/04/05 18:38:34 {"project_id":"9a6e673e-c404-44be-9ec7-f608cdad895a","timestamp":"2022-04-05T18:38:37.935592+02:00","operation":{"name":"update_piano","runtime":0.7038179451558593,"request_id":"44493055-0b6b-4c22-827b-22678420225c","response_size":97,"request_size":217,"http_status":200}}
2022/04/05 18:38:34 {"project_id":"e615f4da-d790-499f-a67a-07f288c9b3ee","timestamp":"2022-04-05T18:38:39.73786156+02:00","operation":{"name":"select_helicopter","runtime":0.9756046885083646,"request_id":"12d65025-f2e6-4e15-b5c4-bc119bf0ce45","response_size":87,"request_size":808,"http_status":200}}
...

Questions?

Ask the maintainer

Licensed under the MIT license

hiring_log_gen's People

Contributors

Watchers

hiring_log_gen's Issues

Technical interview take home assigment

Build a simple data pipeline

You will need docker installed. The following command generates log data:

docker run hasura/hiring_log_gen

A log line has the shape below. Units for sizes are expressed in bytes, and times in seconds.

2022/04/05 18:38:34 {"project_id":"9a6e673e-c404-44be-9ec7-f608cdad895a","timestamp":"2022-04-05T18:38:34.10973506+02:00","operation":{"name":"insert_magazine","runtime":0.49770060098541913,"request_id":"298f63b9-f171-40ee-9730-6c608da3035c","response_size":25,"request_size":41,"http_status":200}}

The exercise consists of building a pipeline that would let us understand:

The amount of data transferred by any project in a specific time window, so we can calculate a project's bill.
What are, per project, the most time-consuming queries.
For the sake of the exercise, don't worry about the volume of data yet, assume we will receive at most 1K requests per minute. You can tweak how fast the log generator will generate traces, by using the -projects, -rate, and -speed parameters.

Other requirements:

Provide us with a way to execute the solution, this can be a GitHub repository containing a docker-compose file and instructions to run the pipeline.
Provide a description of the solution, its trade-offs, and the compromises you made, in the form of a README.md, file in the repository, or

Bonus points:

Simplicity
Presence of tests
If you identify other potentially relevant information for the business or the system's operators, that can be extracted from the log.
If you provide a good UX/DX in your solution.
If you Identify the scalability challenges that the pipeline would face if we received 100x the volume of log data and some potential solutions.

You have as much time as you want to solve the exercise, but don't aim at building the perfect solution, just one that's simple and good enough for the requirements described.

When you are done, please send an email to [email protected] with a link to the solution. If you are creating a private repository for the solution, please also add @miguelff as a collaborator to it.

Thank you for your time, looking forward to hearing back from you!

Recommend Projects