louicoder / node-webscrapper Goto Github PK
View Code? Open in Web Editor NEWA web scrapper application that scraps websites performs a couple of automated tasks.
A web scrapper application that scraps websites performs a couple of automated tasks.
Set up sms integration in the project. This task completely relies on the completion of Set up Twilio and whatsapp issue. Therefore, it's important that this task be attempted after the task mentioned above is completed.
This is because we shall be using the same Twilio account for both sms and whatsapp. Inside the social_integrations.js file create a pseudo function called sendTwilioSms
for sending an sms and the functionality will implemented at a later stage.
Resources:
Sending sms using node and Twilio
Add a function in a new function to handle inputs from the console, this will be basically used by the developers.
The function should be named terminal.js
or console.js
. The function will be responsible for handling questions as well as process answers and thereafter return to the calling function.
Add telegram integration to the application. You should create a function and that function should be executed on an array of the collaborators telegram numbers sending each a message.
./App
called Telegram
telegram.js
telegramSendMessage
should receive have two parameters. one for the array of numbers to be messaged and the second for the message to be sent.Resources:
Files
in App folderFiles
folder called FilesUtility
readAndWriteToFiles.js
writeToTextFile
and the other readFromTextFile
Make sure to check all the requirements before raising a PR for this issue
Update the project wiki with the relevant resources needed for the project to go on.
Set up node server to be used throughout the project. This server will be responsible for handling incoming requests.
Create a cron-job in a new file called cronJob.js
. In this file set up a cron-job console log any output/string every after a number of seconds you have set. For more on how to get started on cron-jobs in NodeJS click NodeJS cron-jobs
Make sure to run the cron-job in server.js
file and not from your file.
Create a PR template that will be automated upon creating a new pull request.
Add a package.json to the project. This can be automated using a command npm init -y
for defaults which you will edit in the created package.json file and replace the following keys
Node webscrapper
A web scrapper built on NodeJS for data analytics
Run npm install <package name> --save
so that the following packages are added to the package.json file
You can find more about setting up a package.json file here >> Link to resource
Make sure the above checklist is complete
Set up integration for Whatsapp and Twilio into our Node project. Create a new file and name it social_integrations.js. The checklist below should guide you on how this should be executed.
username
- nodescrapper and password
- nodescrapper45!Resources:
send whatsapp message with twilio and NodeJS
Add your name to the collaborators table in the readMe.md file. Make sure to add a link to your repository as well.
Modify the read and write functionality to be able to actually read from a file and also write to a file
Related PR: #34
Update the instructions file with the best practices to be followed by the contributors.
Host the application onto the Heroku platform. Make sure that all the environmental variables are set well
The variables include
PORT
, this is the server address.TWILIO_SID
, this is the Twilio SIDTWILIO_TOKEN
, this is the Twilio tokenSENDER_EMAIL
, this is the gmail address of the senderSENDER_PASSWORD
, this is the sender's password to login to gmail serversYou should add your username to the readme file along with the link to your profile. You can simply follow the table format. The username should be added to the collaborators/contributors table. All this should be done in the readme.md file
Modify readme file to add more details about what the project focuses on and what the end results will look like
Add a gitignore file in the application root. make sure the checklist below is completed and tick completed task
The file should be of extension .gitignore
@Anekenonso Add an instructions file in which all instructions regarding the project will be added
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.