lokii is a powerful package that enables the generation of relational datasets, specifically tailored to
facilitate the creation of robust development environments. With lokii, you can effortlessly generate diverse
datasets that mimic real-world scenarios, allowing for comprehensive end-to-end testing of your applications.
lokii leverages the hierarchical structure of the file system to discover groups and nodes. Each dataset
consists of nodes, which are defined using .node.py files. For instance, in the context of a database, each
node represents a table. Furthermore, you can even group nodes under database schemas within the database. Groups
defines how generated node data will be exported. You can recognize group files by their .group.py file extension.
# example project directory structure
proj_dir
├── group_1
│ ├── group_1.group.py
│ ├── node_2.node.py
│ └── node_2.node.py
├── group_2
│ ├── node_3.node.py
│ └── node_4.node.py
├── group_3.group.py
├── node_5.node.py
└── node_6.node.pyNode file defines how each item will be generated. There are special variables and functions in node definition files.
name: Name of the node, filename will be used if not providedsource: Source query for retrieve dependent parameters for each itemitem: Generation function that will return each item in node
# offices.node.py
from faker import Faker
# use your favorite tools to generate data
# you can even use database connection, filesystem or AI
fake = Faker()
# if you want you can override the node name if not provided filename will be used
# can be used in source queries if you want to retrieve rows that depends on another node
# name = "business.offices"
# define a query that returns one or more rows
source = "SELECT * FROM range(10)"
# item function will be called for each row in `source` query result
def item(args):
address = fake.address().split("\n")
return {
"officeCode": args["id"],
"city": fake.city(),
"phone": fake.phone_number(),
"addressLine1": address[0],
"addressLine2": address[1],
"state": fake.city(),
"country": fake.country(),
"postalCode": fake.postcode(),
"territory": fake.administrative_unit(),
}Group file defines how each node data will be exported. There are special functions in group definition files.
before: Called once before export operationexport: Called for every node in the groupafter: Called once after export operation
# filesystem.group.py
import os
import shutil
from csv import DictWriter
out_path = "out_data"
def before(args):
"""
Executed before export function.
:param args: contains node names that belongs to this group
:type args: {"nodes": list[str]}
"""
if os.path.exists(out_path):
# always clear your storage before starting a new export
shutil.rmtree(out_path)
os.makedirs(out_path)
def export(args):
"""
Executed for all nodes that belongs to this group
:param args: contains node name, node columns and a batch iterator
:type args: {"name": str, "cols": list[str], "batches": list[dict]}
"""
node_name = args["name"]
node_cols = args["cols"]
batches = args["batches"]
# out_data/offices.csv
out_file_path = os.path.join(out_path, node_name + ".csv")
with open(out_file_path, 'w+', newline='', encoding='utf-8') as outfile:
writer = DictWriter(outfile, fieldnames=node_cols)
writer.writeheader()
for batch in batches:
writer.writerows(batch)
def after(args):
"""
Executed after export function.
:param args: contains node names that belongs to this group
:type args: {"nodes": list[str]}
"""
passYou can create the source distribution of the package by running the command given below:
python3 setup.py sdistInstall twine and upload pypi for finnetdevlab username.
pip3 install twine
twine upload dist/*Package requirements are handled using pip. To install them do
pip install -r requirements.txt
pip install -r requirements.dev.txt
Testing is set up using pytest and coverage is handled with the pytest-cov plugin.
Run your tests with py.test in the root directory.
Coverage is run by default and is set in the pytest.ini file.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent