When I start Celery I get this error:

 -------------- celery@*****.com v3.1.13 (Cipater)
---- **** -----
--- * ***  * -- Linux-3.2.0-4-amd64-x86_64-with-debian-7.9
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         sketchy:0x20e1b10
- ** ---------- .> transport:   redis://localhost:6379//
- ** ---------- .> results:     redis://localhost:6379
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ----
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery


[2015-11-20 18:18:08,590: ERROR/MainProcess] consumer: Cannot connect to redis://localhost:6379//: Error while reading from socket: (104, 'Connection reset by peer').
Trying again in 2.00 seconds...

[2015-11-20 18:18:38,365: ERROR/MainProcess] consumer: Cannot connect to redis://localhost:6379//: Error while reading from socket: (104, 'Connection reset by peer').
Trying again in 4.00 seconds...

How to resolve that?

ERROR in controller [/home/dstone/sketchy/sketchy/controllers/controller.py:210]:

[Errno 40] Too many levels of symbolic links

127.0.0.1 - - [19/Mar/2015 12:23:03] "GET /eager?url=http://google.com&type=sketch HTTP/1.1" 406 -

I added a new line in the exception catching code to print the only path referenced in the code in this code block (app.config['LOCAL_STORAGE_FOLDER']) which in this case resolves to /home/dstone/sketchy/files
There are no symbolic links that are a part of or within this path.

Any help would be much appreciated.

Should include the ability to indicate whether for screenshot or status code only, and to provide a callback url for each.

hi everyone,
when I try to create a capture in a test mode using I get "sqlalchemy.exc.OperationalError", for production setup it is "404 Not Found nginx/1.4.6 (Ubuntu)".
I installed Sketchy on Ubuntu 14.04 using scripted install and I use 3 terminals to run it as @sbehrens mentioned in #15 .
I have no idea how to deal with it, I would be grateful if you'd help me to configure Sketchy with Scumblr. Thanks is advance

https://github.com/Netflix/sketchy/blob/master/sketchy/assets/capture.js#L50

HTTP headers are case-insensitive:

• http://stackoverflow.com/a/5259004/
• http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2

Create an endpoint that just pulls the DB for the last capture result taken.

Endpoint can be named api/v1.0/capture/last

The TLD library was an unfortunate choice as it ignores 63% of existing TLDs (tested against the official IANA list https://data.iana.org/TLD/tlds-alpha-by-domain.txt).

The other thing is that the library is very string on the format of a domain. Everything has to be lowercase or you get TldDomainNotFound: Domain google.COM didn't match any existing TLD name!.

I suggest to do the testing only with a simple regular expression and catch the exception from a real HTTP communication.

SQLalchemy datastore like PostgreSQL supports a schema path.
I try to use connect_args in URI but it doesn't work:
posgresql://user:[email protected]:5432/db?connect_args={"schema": "sketchy"}

It could be nice to support a specific schema.
Thanks.

Following the install guide to install Sketchy on Ubuntu 14.04 (via scripted install), when I try to start Celery as part of the guide I get

ubuntu@ip-*********:~/sketchy$ Traceback (most recent call last): File "/usr/local/bin/celery", line 11, in <module> sys.exit(main()) File "/usr/local/lib/python2.7/dist-packages/celery/__main__.py", line 16, in main _main() File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 322, in main cmd.execute_from_commandline(argv) File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 496, in execute_from_commandline super(CeleryCommand, self).execute_from_commandline(argv))) File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 273, in execute_from_commandline argv = self.setup_app_from_commandline(argv) File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 479, in setup_app_from_commandline self.app = self.find_app(app) File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 501, in find_app return find_app(app, symbol_by_name=self.symbol_by_name) File "/usr/local/lib/python2.7/dist-packages/celery/app/utils.py", line 359, in find_app sym = symbol_by_name(app, imp=imp) File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 504, in symbol_by_name return imports.symbol_by_name(name, imp=imp) File "/usr/local/lib/python2.7/dist-packages/kombu/utils/imports.py", line 56, in symbol_by_name module = imp(module_name, package=package, **kwargs) File "/usr/local/lib/python2.7/dist-packages/celery/utils/imports.py", line 104, in import_from_cwd return imp(module, package=package) File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module __import__(name) File "/home/ubuntu/sketchy/sketchy/__init__.py", line 20, in <module> from flask.ext.restful import Api ImportError: No module named ext.restful

I used pip freeze to see if Flask-RESTful is installed, and:
ubuntu@ip-**********:~$ pip freeze [...] Flask==1.0.2 Flask-RESTful==0.3.6 [...]

Did I miss something?

Is it possible to configure sketchy to only screenshot/scrape content from a website if the website has changed? Is it possible to setup alerting for when a site changes or provide history for the website changes? Seems like a great tool. Thanks

Setup session pool to recycle old sessions

2015-06-12 09:00:09,669 ERROR: (OperationalError) (2006, 'MySQL server has gone away') 'SELECT `Capture`.id AS `Capture_id`, `Capture`.url AS `Capture_url`, `Capture`.job_status AS `Capture_job_status`, `Capture`.capture_status AS `Capture_capture_status`, `Capture`.created_at AS `Capture_created_at`, `Capture`.modified_at AS `Capture_modified_at`, `Capture`.sketch_url AS `Capture_sketch_url`, `Capture`.scrape_url AS `Capture_scrape_url`, `Capture`.html_url AS `Capture_html_url`, `Capture`.status_only AS `Capture_status_only`, `Capture`.callback AS `Capture_callback`, `Capture`.retry AS `Capture_retry`, `Capture`.url_response_code AS `Capture_url_response_code` \nFROM `Capture` \nWHERE `Capture`.id = %s \n LIMIT %s' (53921L, 1) [in sketchy/__init__.py:67]
2015-06-12 09:11:23,165 ERROR:  [in sketchy/controllers/tasks.py:353] 

Any thoughts on a feature where multiple calls with the same url would just return the file on disk? Setting a TTL parameter on the grab could determine how long the scrape/sketch/html would be cached.

The setup.py doesn't mention PostgreSQL's connector.
I have an error when launching sketchy in docker container :
ImportError: No module named psycopg2

I configured Scumblr and Sketchy to work together. I start up the server and take screenshots and it would work but, if I've been idle or something, I'd keep getting an error in the terminal that says:

[2015-09-24 14:48:18,910: ERROR/Worker-1] PhantomJS Capture timeout at 35 seconds

ERROR in tasks [/home/user/sketchy/sketchy/controllers/tasks.py:341]:
Command 'phantomjs capture' timed out after 35 seconds

SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error:s23_clnt.c:734
Investigate why that error is being returned for some sites.

Received the error when using the test start up. It works the first time, and broke after a reboot.

sqlalchemy.exc.OperationalError

OperationalError: (OperationalError) no such table: Capture u'INSERT INTO "Capture" (url, job_status, capture_status, created_at, modified_at, sketch_url, scrape_url, html_url, status_only, callback, retry, url_response_code) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' ('http://google.com', 'CREATED', None, '2016-05-27 15:02:38.862897', None, None, None, None, None, None, None, None)

When Scumblr finds results on Chinese websites the scrapes are incorrect and show none of the original language text. All text is replaced with a box, such as □.

Thank you,

Josh

I installed Sketchy on Ubuntu and when I execute this

python manage.py runserver --host 0.0.0.0 --port 8000

it complains saying:::

• Running on http://0.0.0.0:8000/ (Press CTRL+C to quit)
• Restarting with stat
  Traceback (most recent call last):
  File "manage.py", line 141, in
  manager.run()
  File "/usr/local/lib/python2.7/dist-packages/flask_script/init.py", line 412, in run
  result = self.handle(sys.argv[0], sys.argv[1:])
  File "/usr/local/lib/python2.7/dist-packages/flask_script/init.py", line 383, in handle
  res = handle(_args, *_config)
  File "/usr/local/lib/python2.7/dist-packages/flask_script/commands.py", line 425, in call
  *_self.server_options)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 772, in run
  run_simple(host, port, self, *_options)
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 622, in run_simple
  reloader_type)
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/_reloader.py", line 269, in run_with_reloader
  reloader.run()
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/_reloader.py", line 159, in run
  for filename in chain(_iter_module_files(), self.extra_files):
  File "/usr/local/lib/python2.7/dist-packages/werkzeug/_reloader.py", line 70, in _iter_module_files
  for package_path in getattr(module, 'path', ()):
  File "/usr/lib/python2.7/dist-packages/six.py", line 116, in getattr
  _module = self._resolve()
  File "/usr/lib/python2.7/dist-packages/six.py", line 105, in _resolve
  return _import_module(self.mod)
  File "/usr/lib/python2.7/dist-packages/six.py", line 76, in _import_module
  import(name)
  ImportError: No module named _winreg

_winreg => isn't this for Windows Registry ! Any Help ?

Thanks

Hi,
I am trying to integrate sketchy to use from inside Scumblr. But when I click to generate screenshot insise Scumblr, I get the following error:

ERROR: Error communicating with sketchy: #<TypeError: no implicit conversion of true into Integer> no implicit conversion of true into Integer: ["/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/2.0.0/net/http.rb:893:in initialize'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/2.0.0/net/http.rb:893:innew'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/2.0.0/net/http.rb:893:in connect'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/2.0.0/net/http.rb:862:indo_start'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/2.0.0/net/http.rb:851:in start'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/2.0.0/net/http.rb:1367:inrequest'", "/home/user/Scumblr/app/models/result.rb:97:in create_attachment_from_sketchy'", "/home/user/Scumblr/app/workers/screenshot_runner.rb:36:inblock in perform'", "/home/user/Scumblr/app/workers/screenshot_runner.rb:29:in each'", "/home/user/Scumblr/app/workers/screenshot_runner.rb:29:inperform'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/sidekiq-3.0.2/lib/sidekiq/processor.rb:52:in block (2 levels) in process'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/sidekiq-3.0.2/lib/sidekiq/middleware/chain.rb:122:incall'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/sidekiq-3.0.2/lib/sidekiq/middleware/chain.rb:122:in block in invoke'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/sidekiq-status-0.5.0/lib/sidekiq-status/server_middleware.rb:37:incall'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/sidekiq-3.0.2/lib/sidekiq/middleware/chain.rb:124:in block in invoke'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/sidekiq-3.0.2/lib/sidekiq/middleware/server/active_record.rb:6:incall'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/sidekiq-3.0.2/lib/sidekiq/middleware/chain.rb:124:in block in invoke'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/sidekiq-3.0.2/lib/sidekiq/middleware/server/retry_jobs.rb:62:incall'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/sidekiq-3.0.2/lib/sidekiq/middleware/chain.rb:124:in block in invoke'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/sidekiq-3.0.2/lib/sidekiq/middleware/server/logging.rb:11:inblock in call'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/sidekiq-3.0.2/lib/sidekiq/logging.rb:22:in with_context'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/sidekiq-3.0.2/lib/sidekiq/middleware/server/logging.rb:7:incall'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/sidekiq-3.0.2/lib/sidekiq/middleware/chain.rb:124:in block in invoke'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/sidekiq-3.0.2/lib/sidekiq/middleware/chain.rb:127:incall'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/sidekiq-3.0.2/lib/sidekiq/middleware/chain.rb:127:in invoke'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/sidekiq-3.0.2/lib/sidekiq/processor.rb:51:inblock in process'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/sidekiq-3.0.2/lib/sidekiq/processor.rb:94:in stats'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/sidekiq-3.0.2/lib/sidekiq/processor.rb:50:inprocess'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in public_send'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:indispatch'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:122:in dispatch'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/celluloid-0.15.2/lib/celluloid/actor.rb:322:inblock in handle_message'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/celluloid-0.15.2/lib/celluloid/actor.rb:416:in block in task'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/celluloid-0.15.2/lib/celluloid/tasks.rb:55:inblock in initialize'", "/home/user/.rbenv/versions/2.0.0-p481/lib/ruby/gems/2.0.0/gems/celluloid-0.15.2/lib/celluloid/tasks/task_fiber.rb:13:in `block in create'"]
2014-09-22T07:58:56Z 3132 TID-owiw23qf8 ScreenshotRunner JID-ad42c3d5e6aeef3747dbdc57 INFO: done: 0.01 sec
2014-09-22T07:59:00Z 3132 TID-owip63p2k ScreenshotSyncTaskRunner JID-f202a5da390314ee6291a8d1 INFO: done: 4.012 sec

I followed the instructions and sketchy independely works fine for me and generates screenshot. It doesn't work from inside Scumblr. can someone please help in letting me know what am I doing wrong? I am just stuck with it.

I am working on Ubuntu 14.04 system

The pickle serializer is a security concern as it may give attackers
the ability to execute any command. It's important to secure
your broker from unauthorized access when using pickle, so we think
that enabling pickle should require a deliberate action and not be
the default choice.

Hello,

I get scrumblr up and running, and try to make it work also with sketchy.
I installed sketchy and also get it to work with a test capture, like documented in the wiki.

But it seems not to work with scrumblr, I only see that the jobs are created and if I list them with http://localhost:8000/api/v1.0/capture I get a list of them. But all jobs from scrumblr have null values:

"callback": "http://localhost:3000/results/55/update_screenshot",
"capture_status": null,
"created_at": "2015-05-12 10:52:56.470125",
"html_url": null,
"id": 3,
"job_status": "CREATED",
"modified_at": "None",
"retry": null,
"scrape_url": null,
"sketch_url": null,
"url": "https://www.reddit.com/r/netflix/comments/10w39c/top_5_on_netflix_instant/",
"url_response_code": null

Test capture worked:

"capture_status": "LOCAL_CAPTURES_CREATED",
"created_at": "2015-05-12 10:41:31.031672",
"html_url": "http://127.0.0.1:8000/files/google.com_1.html",
"id": 1,
"job_status": "COMPLETED",
"modified_at": "2015-05-12 10:41:38.840162",
"retry": null,
"scrape_url": "http://127.0.0.1:8000/files/google.com_1.txt",
"sketch_url": "http://127.0.0.1:8000/files/google.com_1.png",
"url": "http://google.com",
"url_response_code": 200

Any suggestions?

Thanks!

I'm getting Internal Server Error when I try to access https://IP/api/v1.0/capture

Here is my log:

2016-05-12 14:12:32,940 CRIT Supervisor running as root (no user in config file)
2016-05-12 14:12:33,031 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2016-05-12 14:12:33,036 INFO daemonizing the supervisord process
2016-05-12 14:12:33,036 INFO set current directory: '/tmp'
2016-05-12 14:12:33,037 INFO supervisord started with pid 852
2016-05-12 14:12:34,041 INFO spawned: 'celeryd' with pid 853
2016-05-12 14:12:34,043 INFO spawned: 'redis' with pid 854
2016-05-12 14:12:34,046 INFO spawned: 'gunicorn' with pid 855
2016-05-12 14:12:35,085 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-05-12 14:12:35,086 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2016-05-12 14:12:44,951 INFO success: celeryd entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)

Can anyone help me please?

Hello,

I installed sketchy by following the manual steps on the manual. But it skipped on step "python setup.py install" with the error EnvironmentError: mysql_config not found.

After installing mysql with brew install mysql, everything works fine!

Thanks for your great work.

So i have successfully configured Sketchy with S3 on my server and all is working for captures.
However, within my callbacks, and sketchy task status, it is returning error messages, even though the captures are successful and available on S3.

In my Callback JSON:
"capture_status": "500 Server Error: INTERNAL SERVER ERROR",
"job_status": "FAILURE",

Callback json also contains valid links to the sketch, scrape and html on my S3 bucket.

in my sketchy.log:
2017-07-23 23:06:30,998 ERROR: 500 Server Error: INTERNAL SERVER ERROR [in /var/sketchy/sketchy/controllers/tasks.py:383]

Looking at sketchy/controllers/tasks.py i can see the code that is raising this Error, however, as all the S3 uploading is successful, i have no idea why an exception is being raised? Please can you advise where to look next?

I really need to get the correct job_statuses out of the callback json if possible.