elixir-crawly / crawly_ui Goto Github PK
View Code? Open in Web Editor NEWUser interface for Crawly
User interface for Crawly
Spiders show send a stop message to UI, in order to avoid displaying 'node_down' which does not reflect the reality
It should be possible to export items to CSV and to have a file as the result.
Please be aware that due to the size of the data it might be required to stream the resulting file.
We need to have a page with a list of all spiders discovered by the crawlyui
We should add a menu bar, since now we start to have many pages to navigate to. Ideally something pretty. Not a button anymore
i.e
Home | Jobs | Spiders | Nodes | Schedule Spider
And home is the current index, jobs/spiders/nodes lists all of the said field, and schedule spider as it is now
Hi,
I try to compile locally the latest master :
commit 3b95de73299c4f62b84fa6f41418e2ffd0885dca (HEAD -> master, origin/master, origin/HEAD)
Author: Oleg Tarasenko <[email protected]>
Date: Mon Dec 21 12:02:09 2020 +0100
Speedup logs
$ iex -S mix phx.server`
`== Compilation error in file lib/crawly_ui_web/views/job_view.ex ==
** (SyntaxError) lib/crawly_ui_web/templates/job/spider.html.leex:26: unexpected token: end
My local env :
Erlang/OTP 22 [erts-10.7.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [hipe]
Elixir 1.9.4 (compiled with Erlang/OTP 20)
Currently, our search is pretty basic and slow. Also not user friendly at all :(. I wonder if we could improve the case with the help of live view.
I see it in the following way:
Some example is here:
http://crawlyui.com/logs/267/list
The count estimate for this query shows 320. The actual amount of rows in the table is 4. As a result, we have long pagination without items.
It should be possible to export items to JSON and to have a file as the result.
Please be aware that due to the size of the data it might be required to stream the resulting file.
If the job is stopped we should display an average speed instead of 0.
We should have a second table on the homepage with 5 most recent completed jobs there.
Spider logs must be fetched from crawly.
Currently, spider names on UI are way too long. For example on the main page, we have Elixir.Spiders.Walmart as the spider. Lets do the following: Elixir.Spiders.Walmart
-> Walmart
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.