vbauer / manet Goto Github PK

I have been successfully able to deploy the app running phantomjs on heroku. However, when I change to slimerjs (in Procfile), everything proceeds normally (no errors during deployment). But the app is not fetching any data. It always gives following error.

 {"error":"Can not capture site screenshot"}

Am I suppose to change anything else?

attempting to retrieve google.com screenshot of a just reinstalled and started manet:

$ /usr/local/lib/node_modules/manet/bin/manet --port 3000
debug: Initialize FS watchdog: directory: /tmp, timeout: 3600000
info: Manet server started on port 3000
debug: Sending image ("google.com") in response
info: Capture site screenshot: google.com
debug: Options for script: {"url":"http://google.com"}, base64: eyJ1cmwiOiJodHRwOi8vZ29vZ2xlLmNvbSJ9
exec: use child_process.execFile instead
debug: Process output: Script options: {"url":"http://google.com"}
Error: Unable to open file 'src/config/default-styles.css'

error: Process error: Fontconfig warning: ignoring UTF-8: not a valid region tag

Apart from the default-styles.css that can't be found anymore (but is in the same place), and the Fontconfig error that is probably due to a phantomjs issue(?), I can no longer generate snapshots. Not even uncompressed ones.

Any ideas? Thanks!

Would you mind sharing what's needed to get this running on Heroku? I'm not too familiar with Node.js, so any help is appreciated.

Thanks for the great library (I migrated from https://github.com/fzaninotto/screenshot-as-a-service).

Is there a way to modify the PNG compression? Or perhaps the color palette?

When I take a screenshot of a black and white document with pure text on a white background, the resulting PNG file is still well over 1MB.

When I use the JPEG or PDF compression, the file is much smaller.

Any tips?

Hello, I'm trying to configure manet to only allow connections from two IP addresses. I guess the whitelist option is there for that.

I've tried to replace the whitelist option's default value with this:

"whitelist": [
        "127.0.0.1",
        "192.168.0.1"
    ]

When attempting to launch manet The following error is shown on the command line:

$ manet --port 3000

/usr/local/lib/node_modules/manet/node_modules/nconf/lib/nconf/stores/file.js:141
      throw new Error("Error parsing your configuration file: [" + self.file +
            ^
Error: Error parsing your configuration file: [/usr/local/lib/node_modules/manet/src/config/default.json]: Unexpected token /
    at File.loadSync (/usr/local/lib/node_modules/manet/node_modules/nconf/lib/nconf/stores/file.js:141:13)
    at Provider.add (/usr/local/lib/node_modules/manet/node_modules/nconf/lib/nconf/provider.js:137:23)
    at Provider.file (/usr/local/lib/node_modules/manet/node_modules/nconf/lib/nconf/provider.js:63:15)
    at load (/usr/local/lib/node_modules/manet/src/config.js:76:10)
    at Object.read (/usr/local/lib/node_modules/manet/src/config.js:87:30)
    at Object.main (/usr/local/lib/node_modules/manet/src/manet.js:133:23)
    at Object.<anonymous> (/usr/local/lib/node_modules/manet/bin/manet:5:7)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)

While I think the whitelist type is array, this is maybe not the right way to supply IP addresses. Or it could be a bug, but I haven't been able to find the appropriate place in the source code; apparently nconf is being used to validate the configuration file.

Thank you for your help in advance,
Jakob.

Used in the project, 4 concurrent time does not work, we would like to use, and he saves us a lot of time, but concurrency issues, I do not know whether any improvements
Timeout error results are screenshots actually take screenshots of the address can be quickly opened

Hi,

Inside a manet im giving a url which is basically redirect to another url. So i need pdf of second url. As of now i am geeting pdf of first url. What should i do???

For example, please have a look at the following link
https://manet.herokuapp.com/?url=https%3A%2F%2Fforums.mozilla.org%2Findex.php&format=jpg&force=true

Though it works ok for
https://manet.herokuapp.com/?url=https%3A%2F%2Fmicrosoft.com&format=jpg&force=true

How to forcefully switch off certificate check when making request to resource ?

We're attempting to get Manet running on Amazon Web Services (AWS). We have successfully got things running but we are not getting an image.

Here are some details:

• Manet is displaying "Can not capture site screenshot".
• No image is being generated.
• We have isolated this issue to SlimerJS returning exit code 255.
• We had the same issue when deploying to a linux environment in house.

Any ideas?

2016-02-09T10:18:54.087Z - info: Initialize FS watchdog: directory: /tmp, timeout: 1000
2016-02-09T10:18:54.096Z - info: Manet server started on 0.0.0.0:8891
2016-02-09T10:18:55.090Z - error: Error: EISDIR: illegal operation on a directory, unlink '/tmp/npm-12-24411478'
at Error (native)

I have been trying to capture screenshots of mobile webpages by changing the user agent. However, when I provide an iphone user agent as a parameter, I am just getting a blank screen.

Here is the command I am issuing

 http://myserverip:8891/?url=cnn.com&format=jpg&quality=0.5&agent=Mozilla%2F5.0%28iPhone%3BU%3BCPUiPhoneOS4_0likeMacOSX%3Ben-us%29AppleWebKit%2F532.9%28KHTML%2ClikeGecko%29Version%2F4.0.5Mobile%2F8A293Safari%2F6531.22.7

The error logged by the server is as follows:

 debug: Execution time: 1.44 sec
 debug: Process finished work: eyJ1cmwiOiJodHRwOi8vd2lja2VkZmlyZS5jb20iLCJhZ2VudC                                   I6Ik1vemlsbGEvNS4wKGlQaG9uZTtVO0NQVWlQaG9uZU9TNF8wbGlrZU1hY09TWDtlbi11cylBcHBsZV                                   dlYktpdC81MzIuOShLSFRNTCxsaWtlR2Vja28pVmVyc2lvbi80LjAuNU1vYmlsZS84QTI5M1NhZmFyaS                                82NTMxLjIyLjciLCJmb3JtYXQiOiJqcGciLCJxdWFsaXR5IjowLjV9
 error: Error while sending file: ENAMETOOLONG, stat '/tmp/eyJ1cmwiOiJodHRwOi8vd2                                lja2VkZmlyZS5jb20iLCJhZ2VudCI6Ik1vemlsbGEvNS4wKGlQaG9uZTtVO0NQVWlQaG9uZU9TNF8wbG                                lrZU1hY09TWDtlbi11cylBcHBsZVdlYktpdC81MzIuOShLSFRNTCxsaWtlR2Vja28pVmVyc2lvbi80Lj                                AuNU1vYmlsZS84QTI5M1NhZmFyaS82NTMxLjIyLjciLCJmb3JtYXQiOiJqcGciLCJxdWFsaXR5IjowLj                                V9.jpg'

The error disappears when I don't use an agent. BTW, I am using slimerjs engine to render the pages.

Hello,

When using Manet (http://manet.herokuapp.com/) to screenshot a website that contains UTF-8 charset (such as Japanese website, for example), all non-latin characters appear to be transformed into little squares.

For example:

http://manet.herokuapp.com/?url=http%3A%2F%2Fwww.gundam.info%2F&agent=Firefox&headers=Accept-Charset%3Dutf-8&engine=phantomjs

I set the header Accept-Charset=utf-8.

Is there another configuration to be set for utf-8 characters to be well displayed ?

I've found this BUG under Windows 7 64bit, stripping out the raw shell command (manual piece together):

phantomjs "E:\\Program Files (x86)\\nodejs\\node_modules\\manet\\src\\scripts\\screenshot.js" "eyJ1cmwiOiJodHRwOi8vengyNTBsb2NhbC5jb206MzAwMC9tYWdpY3Blbi9hcHAvIy9tYWluL2luZGV4P2l0ZW1MYXlJZD0zJnNuYXBzaG90PTEmX19hPTk2QjcxRDkyOEJFRTM4NkUxMTA2MjFCRUIzN0RGRkYyIiwiZm9ybWF0IjoicG5nIiwid2lkdGgiOjE5MjAsImhlaWdodCI6MTA4MCwic2VsZWN0b3IiOiIjbWFpbi1hcmVhIn0=" "C:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\yJ1cmwiOiJodHRwOi8vengyNTBsb2NhbC5jb206MzAwMC9tYWdpY3Blbi9hcHAvIy9tYWluL2luZGV4P2l0ZW1MYXlJZD0zJnNuYXBzaG90PTEmX19hPTk2QjcxRDkyOEJFRTM4NkUxMTA2MjFCRUIzN0RGRkYyIiwiZm9ybWF0IjoicG5nIiwid2lkdGgiOjE5MjAsImhlaWdodCI6MTA4MCwic2VsZWN0b3IiOiIjbWFpbi1hcmVhIn0=.png"

this command will run successfully with no extra exceptions, but the temp file will not be writed, so when visit manet, manet will send 404.
searched, linux && windows both have the file name length limiting, so we should keep the temp file name's length, and add a prefix word + a random word + timestamp is well.

at manet/src/capture.js Line 18, i modified:

function outputFile(options, conf, base64) {
    var format = options.format || DEF_FORMAT;
    //zcs=>The base64's length is too long for a file name, so we should fix the length
    var filename = 'manettemp_' + (base64 + '').substr(-20) + '_' + Date.now() + '_' + Math.floor(Math.random() * (999999 - 100000) + 100000);
    //<=zcs
    // return conf.storage + path.sep + base64 + '.' + format; //zcs=old line
    return conf.storage + path.sep + filename + '.' + format;
}

Hello, I have been using manet, so far is works pretty good, I would like to know if it would be hard to implement image resizing, for example I would like to get thumbnails for desktop screenshots, I would like to implement this myself however I don't know exactly where should I place the code.

https://www.npmjs.com/package/resize

this seems like a good library for this.

function minimizeImage(src, dest, cb) {
    squirrel(
        IMIN_MODULES, IMIN_OPTIONS,
        (err, Imagemin) => {
            const safeCb = (err) => {
                if (err) {
                    logger.error(err);
                }
                cb();
            };

            if (err) {
                safeCb(err);
            } else {
                const imin = new Imagemin()
                    .src(src)
                    .dest(dest)
                    .use(Imagemin.jpegtran({progressive: true}))
                    .use(Imagemin.optipng({optimizationLevel: 3}))
                    .use(Imagemin.gifsicle({interlaced: true}))
                    .use(Imagemin.svgo());

                imin.run(safeCb);
            }
        }
    );
}

I am guessing I should make a function like this one ?

and then add it here

function runCapturingProcess(options, config, outputFile, base64, onFinish) {
    const scriptFile = utils.filePath(SCRIPT_FILE),
          command = cliCommand(config).split(/[ ]+/),
          cmd = _.union(command, [scriptFile, base64, outputFile]),
          opts = {
              timeout: config.timeout
          };

    logger.debug(
        'Options for script: %s, base64: %s, command: %s',
        JSON.stringify(options), base64, JSON.stringify(cmd)
    );

    utils.execProcess(cmd, opts, (error) => {
        if (config.compress) {
            minimizeImage(outputFile, config.storage, () => onFinish(error));
            //resize here ?
        } else {
            onFinish(error);
        }
    });
}

When using PhantomJS the SVG elements appear to be converted into images although I understand PhantomJS can handle SVGs.

How can I sort this out?

Hi!

I'm trying to implement a php application that will be receiving screenshots generated by manet. I've faced difficulties receiving the screenshots with the callback url.

My logs show:

[Fri Nov 20 01:15:29 2015] [error] [client 127.0.0.1] chunked Transfer-Encoding forbidden: /php5-fcgi/index.php

From my understanding this is because you stream the file directly to the POST request without specifying the content length.

fileStream.pipe(request.post(callbackUrl, (err) => {
if (err) {
sendError(res, 'Error while streaming image file: ' + err.message);
}
onImageFileSent(file, config);
}));

For me this means that i'm not able to use the callback function for my php application.
Can you please add a Content-Length header?

When I have manet running, is there a way to see what the different configuration options are that are being used (in the sandbox maybe)? I'm thinking of the rasterisation engine (slimerjs or phantomjs), the compression, which config JSON file being used etc.

Or an additional option that shows them without starting manet, --debug or --verbose ... just making this up.

I'm asking because even though I'm currently running 0.3.4 my screenshots are seemingly uncompressed (Google homepage at 3MB instead of 21KB), and I'd like to look into this more before filing a bug. Also, the --compress true doesn't seem to have any effect anymore.

Where is the JSON file located that's being used? And can I use another one located elsewhere that overloads the default values?

Thanks!

Hi,

We're trying to install Manet for use on one of our servers without having to setup a new subdomain for it nor opening a port to the whole world.

So we were basically just using apache's ReverseProxy to serve manet's port onto a url such as http://www.example.com/manet

It works perfectly for the API part, unfortunately it looks like the code for the web ui is only meant for the app to be served at the baseroot of the domain: the css and js are not found and the open button redirects to root urls

See webpage.onLongRunningScript

Hi,
I'm trying to get in prod manet for a listing website and i'm having trouble on ddos protected websites (cloudflare and other cookie based protection techniques).
is there any handler that reads the initial cookie and then returns it back to the request in order to bypass the protection?

thanks

If the rendered page has height<768, then the image would have white stripe in the bottom.

Is there an easy way to secure the webservice with an api key or username/password?
I feel very uncomfortable by exposing the control over a fully functional webbrowser to anyone who is smart enough to guess the port number...

Hi,

It would be great if you could add an option in the web form to select the engine (phantomjs or slimerjs) that you want to be used.

Best regards,

I would like to use manet through its RESTful API to get HTML content of the page.

Is it possible?

I'm encountering some weird error when trying to capture certain websites.
As an example: https://www.onpe.tv/
As one can see on the herokuapp demo, it returns the following error

{"error":"Error while sending image file: ENOENT: no such file or directory, stat '/tmp/4379c56e0dfb2442324bfff66228173a77e351e6.png'"}

I'm having the same error on my own machine with phantom 2.1.1 installed

Manet is brilliant! One nice feature would be the abiliy to take screenshots of pages that require user authentication. Maybe through PhantomJS (i'm not an expert on it), that could be accomplished by pointing it first to a sign in page if a forbidden 401 status is returned (just a quick example so u can get the idea). For example we could set parameters on the config file so that phantom would be able to log itself

sign_in:
path: /user
user: foo
pass: bar
user_field_matcher : blah
pass_field_matcher: blah

Or even setting cookies (not sure exactly how phantom internals work).

in the readme it says regarding the --cache parameter that one can set it to a 'no-positive value' in order to effectively disable the cache.

when trying to do this, like so:

$ /usr/local/bin/manet --port 3001 --cache -1
error: Cache life time must be a number

it appears that this is not actually possible. For testing it would actually be quite nice to disable the cache.

example:

for the above code i have found the issue.. please let me if i'm wrong

Same situation as in closed issue #34

It is an AngularJs app that uses C3js to build SVG charts.
The page has no errors when loading on the browser.
Tried both phantomjs and phantomjs2 (with 2 the delay option doesn't seem to by applied).

In the screenshot I see the basic layout (static assets) of the page, but not the dynamic/async content. Neither the angular rendered parts, nor the generated SVGs are present in the screenshot.

Tried with large delays (20000) even though the page normally loads in low single digit seconds. No luck either.

When I change Procfile to:
web: npm install -g phantomjs && ./bin/manet --port $PORT --options:format png --options:force true

The force is still at default value force=false, how can I set default value to force=true with CLI parameter?

[This might be a duplicate of #14]

Me again. I've noticed that screenshots taken with manet are huge, size-wise. For example, the Google homepage, as a screenshot is about 3 MB in size (see screenshot of the screenshot below). Others go easily to 8 MB and beyond. manet was started like this:

forever start -a -l ~/manet.log -o ~/manet.out -e ~/manet.err /usr/local/bin/manet --port 3000 --engine phantomjs

In comparison, screenshot-as-a-service (from here: https://github.com/fzaninotto/screenshot-as-a-service) produces screenshots of only a couple of hundred kilobytes, and their quality are comparable.

Any idea what I could change to decrease the size? I'm using the option quality=0 but that hasn't had any impact on size. I've also tried changing the engines (phantomjs vs slimerjs) but I don't see a difference either.

Thanks for your help in advance.

It would be great to be able to configure which domains or endpoints the service would be able to render.
This is helpful so that other sites cannot serve thumbnails from your site.

Hello, disabling images in manet makes screenshots with black background in jpg, jpeg. Is there any option to set background color by hand?

It would be great to have image cropping built in.

I've tried to set the Viewport Height a dozen ways and can't get it to work. Is there something I'm doing wrong?

https://manet.herokuapp.com/?url=www.prontomarketing.com&width=1280&height=1000

How can I set the Viewport Height?

The proposed feature allows the option to wait for a javascript function to be called before taking a screenshot.

This would work like the takeShotOnCallback method from node webshot, a similar project that uses phantomJS exclusively, see https://github.com/brenden/node-webshot#options.

Hello, I'm noticing some weird redirects with URLs from the BBC:

At least from my location, in France, when I attempt to go here http://www.bbc.co.uk/news/business-31599492 I get redirected to http://www.bbc.com/news/business-31599492. However it's not a classic redirect (301 or 302), the web inspector in Chrome doesn't show this redirect, but ends up with the .com page.

Now, how does this concern manet? Well using manet, I don't get a screenshot, but a 404. Not sure that something can be done on manet's level, but I would still be interested?

Thanks,
Jakob.

The screenshot of a page with WebGL elements doesn't seem to work with any of the engines.

example : http://webglsamples.org/blob/blob.html

It would be great if it could be supported :)

don't screenshot when the http status is 4xx or 5xx

Sorry to have to ask this noob question, but I am having a hard time getting Manet to run in the background, as a service at boot. Since this is already in bin, I am confused at which to use: pm2 or a rc.d-like approach. halp?

The feature to support an css-selector not only the width/height option is ducky, sample belows:

page.open(url, function (status) {
    if ('success' != status) return exit('Error: Open url failed!');\
    var clipRect = page.evaluate(function (sel) {
        try {
            return document.querySelector(sel).getBoundingClientRect();
        } catch (e) {
            return null;
        }
    }, config['boxSelector']);

    if (clipRect) {
        page.clipRect = {
            top: clipRect.top,
            left: clipRect.left,
            width: clipRect.width,
            height: clipRect.height
        };
        page.render(outfile, {'format': 'jpeg', 'quality': '70'});
        echo('[SUCCESS]');
    } else {
        echoError('Error: Can\'t find selector: ' + config['boxSelector']);
    }
    exit();
});

Hi there,

Thank you for creating manet. When running the following command:
curl http://localhost:8891/?url=http://moveme.io/app.html#bG9hZCBhc3NldCB0aW55 > moveme.png

I get the following error:
I'm getting an image which states that that the browser (gecko?) doesn't support webgl. Not sure if this actually due to something manet related. Maybe this is slimerJS related? Then again, as far as I understand slimerJS gets used by default by manet and slimerJS supports webgl. Maybe it's missing a flag/argument somewhere or maybe I'm doing something wrong. Any help welcome.

Thanks,

Manu

Hi there,

Manet looks like a great private screen cap service, but I'm having some issues standing it up. Here's my specs:

OSX 10.10.2
Phantomjs 1.9.8 (installed globally via npm)
manet (installed via npm)

But when I run the following:

$ wget http://localhost:8891/?url=github.com -O ~/github.png
--2015-02-23 18:09:58--  http://localhost:8891/?url=github.com
Resolving localhost... ::1, 127.0.0.1
Connecting to localhost|::1|:8891... failed: Connection refused.
Connecting to localhost|127.0.0.1|:8891... connected.
HTTP request sent, awaiting response... 404 Not Found
2015-02-23 18:10:01 ERROR 404: Not Found.

I get the following:

debug: Execution time: 2 sec
debug: Process finished work: eyJ1cmwiOiJodHRwOi8vdmlzaWEuY2EifQ==
error: Error while sending file: ENOENT, stat '/tmp/eyJ1cmwiOiJodHRwOi8vdmlzaWEuY2EifQ==.png'
debug: Sending image ("github.com") in response
info: Capture site screenshot: github.com
debug: Options for script: {"url":"http://github.com"}, base64: eyJ1cmwiOiJodHRwOi8vZ2l0aHViLmNvbSJ9
debug: Process output: Script options: {"url":"http://github.com"}
TypeError: 'undefined' is not a function (evaluating 'Array.prototype.forEach.call.bind(Array.prototype.forEach)')

  https://assets-cdn.github.com/assets/frameworks-9643b0378c6bcb216adcdaaaa703eed77aa239aaf1c2ae44cadb2fb5099ec172.js:29
  https://assets-cdn.github.com/assets/frameworks-9643b0378c6bcb216adcdaaaa703eed77aa239aaf1c2ae44cadb2fb5099ec172.js:29
ReferenceError: Can't find variable: $

  https://assets-cdn.github.com/assets/github-e481d46e78f12279d9b4fec62c9f7f79d748386efa94431f30b84db4fe996902.js:1
  https://assets-cdn.github.com/assets/github-e481d46e78f12279d9b4fec62c9f7f79d748386efa94431f30b84db4fe996902.js:1
Error: Unable to open file 'src/config/default-styles.css'

debug: Execution time: 2.85 sec
debug: Process finished work: eyJ1cmwiOiJodHRwOi8vZ2l0aHViLmNvbSJ9
error: Error while sending file: ENOENT, stat '/tmp/eyJ1cmwiOiJodHRwOi8vZ2l0aHViLmNvbSJ9.png'`

And I've got a default.json config file that looks like so:

{
    "port": 8891,
    "cors": false,
    "ui": true,

    "silent": false,
    "level": "debug",

    "engine": "phantomjs",
    "options": {},
    "timeout": 9000,
    "compress": false,
    "cache": 0,
    "cleanup": false,

    "commands": {
        "slimerjs": {
            "linux": "xvfb-run -a slimerjs",
            "freebsd": "xvfb-run -a slimerjs",
            "sunos": "xvfb-run -a slimerjs",
            "darwin": "slimerjs",
            "win32": "slimerjs.bat"
        },
        "phantomjs": {
            "linux": "phantomjs",
            "freebsd": "phantomjs",
            "sunos": "phantomjs",
            "darwin": "phantomjs",
            "win32": "phantomjs"
        }
    },

    "whitelist": [
        ".*"
    ]
}

Any idea where I should start looking to debug?

Thanks!

Doing a dos2unix on the file fixes it, but it's a mysterious error until you figure that out.

Are you able to deploy to heroku with phantomjs2 in Procfile ?

We have a page that renders correctly in stand-alone PhantomJS (exe). But when run as part of manet, the renderer fails to wait for the page to finish loading data via ajax. The result is that the page is rendered with correct formatting but without data. Even if the timeout is set to 10 seconds. (in reality the page takes about 2 sec or less to load its data).

An example of the page can be found here: http://goo.gl/Y43rb0

Due to the issue as described here
#19

I enabled slimmerjs instead of phantojs. Now I get a "error: Process errror" for any capture I try to do. Here is the output

curl http://localhost:8891/?url=cnn.com > cnn.png

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0info: Capture site screenshot: cnn.com
**error: Process error:**
100    43  100    43    0     0    148      0 --:--:-- --:--:-- --:--:--   148

Due to the lack of any additional information, I am unable to figure out what is going on. Any ideas?