• JobName,
• CompanyName,
• CompanyOverview,
• CompanySize,
• SpecificLocation
• , Benefits
• , AverageProcessingTime,
• Industry
• , Location,
• PostedTime,
• JobDescription,
• CareerLevel,
• Qualification,
• YearsOfExperience,
• JobType,
• JobSpecializations,
• Salary,
• url
• [ ]

• Switch database host url for dev env and CI env
• Read settings from Connection Manager

• save extracted data to wobb table

• Get inner text job street
• Classify the raw data , null?
• Raw data has been labelled
• Save to database

Test about:

• make sure the data has been capture
• make sure the data has been save to database

• write test for both table

wobbTable

• jobName
• companyName
• location
• jobType
• jobSpec
• jobDesc
• jobReq
• industryType
• benefits
• companyAddress
• companyOverview
• postUrl
• version

wobbRaw

• postUrl
• rawContent
• rawContentType
• version

• Auto run tasks everyday, random time

• get jobs URL from each page
• save raw contents to wobbRaw table

• Worker thread example

• ERD for job post data load

• Parse raw data from raw data table to job street table
• interface
• [ ]

• new task queue
• active task list
• done task list
• Scheduler distribute task from queue
• Scheduler receive feedback from worker after work is done
• Error handler for error task - move task back to new task queue

• Scheduler init playwright browser context as workerData

• Scheduler resolve import.meta.url to relative worker thread file

• Scheduler maintain queue for free thread

• Scheduler maintain queue for uncompleted task

• Scheduler maintain map for active thread

• Scheduler maintain map for active task

• Scheduler maintain list for completed task

• Worker thread naive DI

• Worker thread open browser as test

• Postgres Container Setup
• MikroORM connection setup
• Raw Data Table creation
• Unit Test for db connection
• Unit Test Raw Data Table insert delete