montferret / worker Goto Github PK
View Code? Open in Web Editor NEWContainerized Ferret worker
License: Apache License 2.0
Containerized Ferret worker
License: Apache License 2.0
For example, I tried open a site behind Cloudflare, and it threw the below error. How can I avoid Cloudflare captcha errors? or How could I install privacypass chrome extension provided by Cloudflare
{
"text": "RETURN DOCUMENT(@url, {driver: 'cdp'})",
"params": {
"url": "https://some.site.behind.cloudflare/"
}
}
"\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n\n\n\n\n
\n
Please enable cookies.
\n
\n
\n \n
One more step
\n \n \n
Another way to prevent getting this page in the future is to use Privacy Pass. You may need to download version 2.0 now from the [Chrome Web Store](https://github.com/%22https://chrome.google.com/webstore/detail/privacy-pass/ajhmfdgkijocedmfjonnpjfojldioehi/%22).
\n
\n\n\n
\n
\"\"\n\n \n\n\n\n\n"
Hello, first of all, thank you for this amazing project. You really have a vision behind this and it's looking to be awesome! About this issue, it's just that the container won't start, the entrypoint or init string is not found I would guess. But when built locally, all good, so not an urgent issue for me.
Hello, it might be interesting to return the execution ip of the request for tracking / debugging on regular scraps.
This can allow you to be aware of spam from an IP address, to understand if it is banned etc ...
when I run the standard "try it" script, the ferret doesn't find the element by the ".chartTrack__title" selector
LET doc = DOCUMENT('https://soundcloud.com/charts/top', {
driver: 'cdp'
})
WAIT_ELEMENT(doc, '.chartTrack__details', 5000)
LET tracks = ELEMENTS(doc, '.chartTrack__details')
FOR track IN tracks
RETURN {
artist: TRIM(INNER_TEXT(track, '.chartTrack__username')),
track: TRIM(INNER_TEXT(track, '.chartTrack__title'))
}
(on an intel mac)
docker run -d -p 8080:8080 montferret/worker
get http://localhost:8080/info
{
"ip": "xxxx",
"version": {
"worker": "1.18.0",
"chrome": {
"browser": "HeadlessChrome/99.0.4844.0",
"protocol": "1.3",
"v8": "9.9.115",
"webkit": "537.36 (@007241ce2e6c8e5a7b306cc36c730cd07cd38825)"
},
"ferret": "0.16.6"
}
}
{
"text": "WAIT(RAND(5000,0)) LET doc = DOCUMENT('https://news.ycombinator.com/', {driver: 'cdp',viewport: {width: 1920,height: 1080}}) WAIT_ELEMENT(doc, '.titlelink', 5000) LET elements = (FOR el IN ELEMENTS(doc, '.title .titlelink')RETURN {'article': {'title': TRIM(el.innerText)}}) RETURN elements"
}
Success
git clone https://github.com/MontFerret/worker.git
docker build -t mybuild/worker ./
docker run -d -p 8081:8080 mybuild/worker
get http://localhost:8080/info
{
"ip": "xxxxxx",
"version": {
"worker": "1.18.0-1-g6331581",
"chrome": {
"browser": "HeadlessChrome/99.0.4844.0",
"protocol": "1.3",
"v8": "9.9.115",
"webkit": "537.36 (@007241ce2e6c8e5a7b306cc36c730cd07cd38825)"
},
"ferret": "0.16.6"
}
}
{
"text": "RETURN 0"
}
Success
{
"text": "WAIT(RAND(5000,0)) LET doc = DOCUMENT('https://news.ycombinator.com/', {driver: 'cdp',viewport: {width: 1920,height: 1080}}) WAIT_ELEMENT(doc, '.titlelink', 5000) LET elements = (FOR el IN ELEMENTS(doc, '.title .titlelink')RETURN {'article': {'title': TRIM(el.innerText)}}) RETURN elements"
}
Error: socket hang up
Docker Exited
sync.(*Pool).Get(0x10521e0)
/usr/local/go/src/sync/pool.go:132 +0x25 fp=0xc00033c5e8 sp=0xc00033c5b0 pc=0x46e445
github.com/wI2L/jettison.encodeSortedMap(0x10521a0, {0xc000617000, 0x1, 0x1000}, {{0xbb4028, 0xc000192008}, {0xabcb10, 0x23}, 0x5, 0x80, ...}, ...)
/go/pkg/mod/github.com/w!i2!l/[email protected]/encode.go:415 +0x7a fp=0xc00033c728 sp=0xc00033c5e8 pc=0x7a7f1a
github.com/wI2L/jettison.encodeMap(0xc0008016c8?, {0xc000617000, 0x0, 0x1000}, {{0xbb4028, 0xc000192008}, {0xabcb10, 0x23}, 0x5, 0x80, ...}, ...)
/go/pkg/mod/github.com/w!i2!l/[email protected]/encode.go:364 +0x345 fp=0xc00033c808 sp=0xc00033c728 pc=0x7a7ac5
github.com/wI2L/jettison.newMapInstr.func1(0x10520e0?, {0xc000617000?, 0xc000478900?, 0xc000617000?}, {{0xbb4028, 0xc000192008}, {0xabcb10, 0x23}, 0x5, 0x80, ...})
/go/pkg/mod/github.com/w!i2!l/[email protected]/instruction.go:400 +0x72 fp=0xc00033c898 sp=0xc00033c808 pc=0x7adcb2
github.com/wI2L/jettison.wrapInlineInstr.func1(0xc00021f7d0, {0xc000617000?, 0x7f86c053cd28?, 0x40?}, {{0xbb4028, 0xc000192008}, {0xabcb10, 0x23}, 0x5, 0x80, ...})
/go/pkg/mod/github.com/w!i2!l/[email protected]/instruction.go:406 +0x65 fp=0xc00033c908 sp=0xc00033c898 pc=0x7adec5
github.com/wI2L/jettison.marshalJSON({0x9ae8c0?, 0xc00021f7d0?}, {{0xbb4028, 0xc000192008}, {0xabcb10, 0x23}, 0x5, 0x80, 0x0, 0x0})
/go/pkg/mod/github.com/w!i2!l/[email protected]/json.go:167 +0xd9 fp=0xc00033c9d0 sp=0xc00033c908 pc=0x7aee79
github.com/wI2L/jettison.MarshalOpts({0x9ae8c0, 0xc00021f7d0}, {0xc00033cab8, 0x1, 0xa644a0?})
/go/pkg/mod/github.com/w!i2!l/[email protected]/json.go:142 +0x1a9 fp=0xc00033ca90 sp=0xc00033c9d0 pc=0x7aec69
github.com/MontFerret/ferret/pkg/runtime/values.(*Object).MarshalJSON(0xc00033cb08?)
/go/pkg/mod/github.com/!mont!ferret/[email protected]/pkg/runtime/values/object.go:47 +0x45 fp=0xc00033cad0 sp=0xc00033ca90 pc=0x7bab45
github.com/wI2L/jettison.encodeJSONMarshaler({0xa644a0?, 0xc00014e5a8}, {0xc000616000, 0x1, 0x1000}, {{0xbb4028, 0xc000192008}, {0xabcb10, 0x23}, 0x5, ...}, ...)
/go/pkg/mod/github.com/w!i2!l/[email protected]/encode.go:692 +0x86 fp=0xc00033cb68 sp=0xc00033cad0 pc=0x7aa146
github.com/wI2L/jettison.encodeMarshaler(0xc0001c8a00, {0xc000616000, 0x1, 0x1000}, {{0xbb4028, 0xc000192008}, {0xabcb10, 0x23}, 0x5, 0x80, ...}, ...)
/go/pkg/mod/github.com/w!i2!l/[email protected]/encode.go:668 +0x359 fp=0xc00033cc18 sp=0xc00033cb68 pc=0x7a9c99
github.com/wI2L/jettison.newJSONMarshalerInstr.func1(0x0?, {0xc000616000?, 0x4167eb?, 0x108a720?}, {{0xbb4028, 0xc000192008}, {0xabcb10, 0x23}, 0x5, 0x80, ...})
/go/pkg/mod/github.com/w!i2!l/[email protected]/instruction.go:241 +0x76 fp=0xc00033cca8 sp=0xc00033cc18 pc=0x7aca56
github.com/wI2L/jettison.encodeArray(0xc0001c8a00, {0xc000616000?, 0x969459?, 0xb1af88?}, {{0xbb4028, 0xc000192008}, {0xabcb10, 0x23}, 0x5, 0x80, ...}, ...)
/go/pkg/mod/github.com/w!i2!l/[email protected]/encode.go:312 +0x1ae fp=0xc00033cd40 sp=0xc00033cca8 pc=0x7a74ce
github.com/wI2L/jettison.encodeSlice(0xc00033ce48?, {0xc000616000?, 0x0, 0xc00018fa10?}, {{0xbb4028, 0xc000192008}, {0xabcb10, 0x23}, 0x5, 0x80, ...}, ...)
/go/pkg/mod/github.com/w!i2!l/[email protected]/encode.go:267 +0xe6 fp=0xc00033cdd8 sp=0xc00033cd40 pc=0x7a6ea6
github.com/wI2L/jettison.newSliceInstr.func1(0x41299a?, {0xc000616000?, 0x7f86c053cd28?, 0x40?}, {{0xbb4028, 0xc000192008}, {0xabcb10, 0x23}, 0x5, 0x80, ...})
/go/pkg/mod/github.com/w!i2!l/[email protected]/instruction.go:364 +0x5c fp=0xc00033ce58 sp=0xc00033cdd8 pc=0x7ad93c
github.com/wI2L/jettison.marshalJSON({0x969440?, 0xc000801698?}, {{0xbb4028, 0xc000192008}, {0xabcb10, 0x23}, 0x5, 0x80, 0x0, 0x0})
/go/pkg/mod/github.com/w!i2!l/[email protected]/json.go:167 +0xd9 fp=0xc00033cf20 sp=0xc00033ce58 pc=0x7aee79
github.com/wI2L/jettison.MarshalOpts({0x969440, 0xc000801698}, {0xc00033d008, 0x1, 0xc0004b7b60?})
/go/pkg/mod/github.com/w!i2!l/[email protected]/json.go:142 +0x1a9 fp=0xc00033cfe0 sp=0xc00033cf20 pc=0x7aec69
github.com/MontFerret/ferret/pkg/runtime/values.(*Array).MarshalJSON(0xc0006538f0?)
/go/pkg/mod/github.com/!mont!ferret/[email protected]/pkg/runtime/values/array.go:42 +0x56 fp=0xc00033d020 sp=0xc00033cfe0 pc=0x7b4616
github.com/MontFerret/ferret/pkg/runtime.(*Program).Run(0xc000653b60, {0xbb4098, 0xc000653bf0}, {0xc0008b5240?, 0x0?, 0x0?})
/go/pkg/mod/github.com/!mont!ferret/[email protected]/pkg/runtime/program.go:99 +0x366 fp=0xc00033d1c8 sp=0xc00033d020 pc=0x7be5e6
github.com/MontFerret/worker/pkg/worker.(*Worker).DoQuery(0xc0002e6180, {0xbb4098, 0xc00008a7b0}, {{0xc0000b8140?, 0xc00008a7e0?}, 0x0?})
/go/src/github.com/MontFerret/worker/pkg/worker/worker.go:72 +0x1ae fp=0xc00033d258 sp=0xc00033d1c8 pc=0x8fcaae
github.com/MontFerret/worker/internal/controllers.(*Worker).runScript(0xc00019c1d8, {0xbc3310, 0xc000010180})
/go/src/github.com/MontFerret/worker/internal/controllers/worker.go:52 +0x33b fp=0xc00033d308 sp=0xc00033d258 pc=0x8fe11b
github.com/MontFerret/worker/internal/controllers.(*Worker).runScript-fm({0xbc3310?, 0xc000010180?})
.....
Hey @ziflex ,
Sometimes we are billed per request on the hosting of an API;
it would be possible to have an option on the post / to retrieve the info in the process?
{
data: ...
info: ...
}
I have a quick look for PR .. I have to dig a little GO before ๐
It's my last step to align all my containers on your worker and trash my fork ๐ค
Hello,
First of all great work, creating a declarative scraper. We were testing out worker using docker, ran using
docker run -d -p 8080:8080 montferret/worker
and it's running great. We sent a POST request to above with below payload and got 200 OK which is good.
{ "text": "LET doc = DOCUMENT(@url, { driver: \"cdp\", userAgent: \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome 76.0.3809.87 Safari/537.36\"}) RETURN {}", "params": { "url": "http://192.168.0.10/test" } }
However the problem is, when the URL is down/offline (we intentionally took site http://192.168.0.10/test down) and we're still getting the same 200 and OK. Looks like the previously successful request is cached since http://192.168.0.10/test was running when the very 1st time request went through. [if we restart docker container while http://192.168.0.10/test is down and send a new fresh request, it's showing net::ERR_ADDRESS_UNREACHABLE as expected and working correctly]
Not sure if this is due to Chrome caching or ferret caching it?
If it is cache, is there a way to disable the cache so that every time it hits the live URL instead of using the cache version?
If there is a flag how to pass it to the docker image?
Appreciate your help, thanks in advance.
I want to execute script like this. I want to pass page text to ferret
We need to add possibility execute queries asynchronously i.e. not blocking HTTP request.
The design is pretty simple:
[POST] /async
(or other name, naming is hard :)) method that receives similar payload to [POST] /
with additional property .callback
that represents a valid URL (we need to validate it before responding):type AsyncScript struct {
Script
Callback string `json: "callback"`
}
type AsyncResult struct {
Status string // succeeded or failed
Data byte[] // script result or error
}
Hey, still in a spirit of debugging it could be interesting via post options to manage a screenshot of the state of the doc in case of failed.
I'm just sharing ideas, feel free to close the issue.
it might be nice to have the possibility for the build to choose a non-headless for some usecases
quick idea
regularly we will host this docker behind an auth managed by google amazon or .... in case we want to host it on a dedicated server, do you think it would be interesting to have a basic micro authentication system?
thanks for the worker component, it fits my needs and loving it.
I link to know how to use "worker component" with external proxy.
Any tips or idea will be appreciated.
thanks in advance.
Worker is using pretty outdated version of Chrome that has some issues. Let's update to the most recent stable one (that one that Puppeteer team is using).
microbox/chromium-headless:83.0.4103.0
We need to be able to cache frequently used queries (compiled) using simple LFU algorithm.
I am trying to run 'make' so that I can use a local chrome instance for CDP. But it fails with the following error. Any pointers to help?
go build -v -o ./bin/worker
-ldflags "-X main.version=v1.10.0-1-gfa6fdbf -X main.ferretVersion=0.15.0"
./main.go
main.go:13:2: use of internal package not allowed
main.go:14:2: use of internal package not allowed
Makefile:8: recipe for target 'compile' failed
make: *** [compile] Error 1
Once #5 is done, we need to provide some proxy sidecars that implement an interface to popular queue services like:
Any additional proxies will be created if there is such a need.
how to use cookie ?
could be useful for debugging too :)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.