Automatically check jobs availability about elixir-companies HOT 18 CLOSED

I was thinking about quite a different approach - having a scheduler process in supervision tree and run a simple polling job every ~6 hours. I even have this in code snippets - just need to set it up properly in this project.

from elixir-companies.

I dont think this needs mix or anything. It can just be a simple genserver that sends itself a message to do the work like @brain-geek said. The dyno restarting isnt an issue since we only need the task to run in the regular interval. We dont care about that time it runs at we care about it running every so often so on restart its not a big deal cuz itll fire off just on a new interval. The timings might not be perfectly consistent but it should all work.

from elixir-companies.

We could use a Mix task potentially

Yeah, but where's using Elixir features in this approach? 😄

We need to confirm this won't be an issue with Heroku.

It won't. I've had long-running workers on heroku (though not a free one). The major problem with that is that heroku restarts dynos every ~24 hours, but it's not a big deal generally.

from elixir-companies.

One note: we may want to make the task run on init instead of after the first message is received in the interval, that way we ensure on restart the latest is in.

from elixir-companies.

@notactuallypagemcconnell yup, there are plenty of cases.

This doesnt work if someone posts one job and edits the link however.

302 could be link to the new URL. However from my experience it's usually redirect to root page with error message.

But if it's 404 - then we're out of luck.

Actually, we could email the person who added this job with notification of deletion some time after thee initial cleanup goes through. The best kind of business logic in development - "let human handle it". 😄

from elixir-companies.

To improve usability we generally should also check URL as a part of job creation process. just have a 30 sec request timeout to make sure we don't time out this request from user.

from elixir-companies.

To shift discussion to a more constructive path - I've drafted first version #483

Any ideas on how to change it?

from elixir-companies.

Yes, yes, YES! I would love to solve this with Elixir. It would be a wonderful blog post for Elixir School too, to document just how we accomplished that.

from elixir-companies.

It's possible to add a daily routine (eg rake task) that checks all urls for each job in the database and if it returns 404 we could disable the job post.

What do you guys think about it?

from elixir-companies.

@lccezinha this project is no longer written in Ruby, we transitioned it in elixir a few days ago, so we have to find a solution in the elixir space :)

from elixir-companies.

@gemantzu I know it. But in the Elixir/Phoenix context, does not exist something that is equivalent to follow this approach?

For example: https://github.com/akira/exq#creating-workers

from elixir-companies.

But in the Elixir/Phoenix context, does not exist something that is equivalent to follow this approach?

We could use a Mix task potentially

having a scheduler process in supervision tree and run a simple polling job every ~6 hours.

We need to confirm this won't be an issue with Heroku. If it is, I can just bite the bullet and look into getting a dedicated DO box.

from elixir-companies.

@burden @brain-geek I think we can do this using GenServer pretty easily. Don't think Heroku would have any issues with that!

from elixir-companies.

We did a blog on the topic!

• https://elixirschool.com/blog/til-send-after/
• https://elixirschool.com/blog/til-genserver-handle-continue/

from elixir-companies.

Happy to work with someone on this if they're not 💯how to solve it 😁

from elixir-companies.

So what should it do? Should it push all jobs that their url is not 200 on a pending change#delete? And what about companies? If a company changes their url or goes down under, should we check and act on it or no?

from elixir-companies.

@gemantzu I think companies themselves is a separate case. Job postings are often done through sites like lever and such where the 2xx/4xx difference should be clear but it could be a lot of things like a 301 redirect, 404, 5xx, or other things in that situation.

I think the simple/naive implementation would be something like "every 4 hours check for 200, if its 404 then send a command to some module that will result in it being removed".

This doesnt work if someone posts one job and edits the link however. But that might not be worth considering?

from elixir-companies.

@doomspork do we need this now since we have the task to invalidate old jobs?

from elixir-companies.