hi,
it seems twitter modules can not scrape followers and following list.yes?

I am not sure if this is a bug or it is intended to be like that but, for example, take this post:

skraper returns this:

{
  "id" : "4346072205438814",
  "text" : "What a wonderful moment this morning at Waller Elementary. We LOVE our military kids and are grateful for their service and the service of their parents!",
  "publishedAt" : 1618250078,
  "rating" : 161,
  "commentsCount" : 2,
  "viewsCount" : 0,
  "media" : [ {
    "url" : "https://external.ficn4-1.fna.fbcdn.net/safe_image.php?d=AQGoUDzXwoNh4UmX&w=476&h=249&url=https%3A%2F%2Fwww.mypanhandle.com%2Fwp-content%2Fuploads%2Fsites%2F88%2F2021%2F04%2FSOTVO.00_00_19_21.Still001.jpg%3Fw%3D1280&cfs=1&upscale=1&fallback=news_d_placeholder_publisher&ccb=3-4&_nc_hash=AQGw4aGBDe67H8FP",
    "aspectRatio" : 1.9112903225806452
  } ]
}

The grabbed media is from the re-shared post, but there is no text - in this case that would be the partial url. There is also no way to tell if a post has media that they uploaded vs media from something that they re-shared.

Downloading VK-Communities isnt working properly. Sometimes no posts are detected and sometimes a maximum of four or five is being downloaded.

Got this error
Tried with username : thedoodlestories
Exception in thread "main" com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false') at [Source: (byte[])"<meta id="viewport" name="viewport" content="[truncated 264379 bytes]; line: 1, column: 2]

Seems like when I put in this command

.\skraper facebook MuziekcentrumKinkyStar -n 200 -t json -o ..\SocialMediaArchvingMeemoo\skraper\

Still only results in a json file with " only " data from 19 posts. How do I increase this?
On Windows and on Linux.

val client = OkHttpSkraperClient()
val sktr = TikTokSkraper(client)
Log.d("TAG", "getProfileFromTiktok: ${sktr.getPageInfo(path = "/@$userName")}")

I tried this above code but it returned null data I tried another type of data too but got the same issue.

https://imgur.com/

Hello, trying to run the example, however, I can't seen to be able to find AttachmentType (neither IMAGE, nor VIDEO).

Has this changed? Thanks for the help

Hi!

I'm having some issues with scraping from VK channels / groups.

Currently, I'm getting the following:

Exception in thread "main" com.fasterxml.jackson.databind.JsonMappingException: Unsupported field: Year (through reference chain: java.util.ArrayList[0])
        at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:402)
        at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:373)
        at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:338)
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:123)
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)
        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
        at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1572)
        at com.fasterxml.jackson.databind.ObjectWriter._writeValueAndClose(ObjectWriter.java:1273)
        at com.fasterxml.jackson.databind.ObjectWriter.writeValueAsString(ObjectWriter.java:1140)
        at ru.sokomishalov.skraper.cli.Serialization$CSV.serialize(Serialization.kt:99)
        at ru.sokomishalov.skraper.cli.Main.persistMeta(Main.kt:97)
        at ru.sokomishalov.skraper.cli.Main.access$persistMeta(Main.kt:1)
        at ru.sokomishalov.skraper.cli.Main$main$1.invoke(Main.kt:54)
        at ru.sokomishalov.skraper.cli.Main$main$1.invoke(Main.kt:39)
        at com.xenomachina.argparser.SystemExitExceptionKt.mainBody(SystemExitException.kt:74)
        at com.xenomachina.argparser.SystemExitExceptionKt.mainBody$default(SystemExitException.kt:72)
        at ru.sokomishalov.skraper.cli.Main.main(Main.kt:39)
Caused by: java.time.temporal.UnsupportedTemporalTypeException: Unsupported field: Year
        at java.base/java.time.Instant.getLong(Instant.java:604)
        at java.base/java.time.format.DateTimePrintContext$1.getLong(DateTimePrintContext.java:205)
        at java.base/java.time.format.DateTimePrintContext.getValue(DateTimePrintContext.java:308)
        at java.base/java.time.format.DateTimeFormatterBuilder$NumberPrinterParser.format(DateTimeFormatterBuilder.java:2763)
        at java.base/java.time.format.DateTimeFormatterBuilder$CompositePrinterParser.format(DateTimeFormatterBuilder.java:2402)
        at java.base/java.time.format.DateTimeFormatterBuilder$CompositePrinterParser.format(DateTimeFormatterBuilder.java:2402)
        at java.base/java.time.format.DateTimeFormatterBuilder$CompositePrinterParser.format(DateTimeFormatterBuilder.java:2402)
        at java.base/java.time.format.DateTimeFormatter.formatTo(DateTimeFormatter.java:1849)
        at java.base/java.time.format.DateTimeFormatter.format(DateTimeFormatter.java:1823)
        at ru.sokomishalov.skraper.cli.Serialization$CSV$serialize$csvModule$1$1.serialize(Serialization.kt:72)
        at ru.sokomishalov.skraper.cli.Serialization$CSV$serialize$csvModule$1$1.serialize(Serialization.kt:66)
        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)
        ... 15 more

Scraping from users is fine - no errors being received. But I'm unsure whether I am invoking the wrong command as VK groups channels do not have a ID string.

Thanks in advance.

Thumbnail is null and url refers to a fb page with the video rather then the video itself. Not sure this can be fixed tho (or am I missing something?).

It's not enough information in shared posts. Have to be researched if it's possible to extend.

TwitterSkrapper Returns response code: 403.

Thank you for creating a good library.

I need to check a few things that didn't work well.

First, it seems that both getUserInfo() and getPageInfo() do not work with TwitterSkraper. PageInfo always becomes null.

Also, while YoutubeSkraper was successful in retrieving PageInfo, all properties of PageStatistics were null.

Is this something that can be addressed?

can we get data Instagram Hashtag Search?

Hello,

been wondering if you could help me out with this. When I run your FB scraper on this page: https://www.facebook.com/arrowsostrava/ it doesn't return any posts. Any thought on where the problem could be or how to fix it?
Thanks.

Either via the CLI tool or by passing arguments to OkHttpSkraperClient?

Hey, in previous versions post like these provided Image medium:

Currently both text and medium are null.

Hi,

thanks for such great work.
Reading about possibility's of this plugin i see that can do a lot!

Can you tell me what would be the best proper way to connect it with my app written in PHP.

Regards!

• getPosts and extensions for it should return Flow<Post>and be paginable.
  • make getPosts return Flow<Post> everywhere (8444e69)
  • Facebook pagination (8444e69)
  • 9Gag pagination (8444e69)
  • Reddit pagination (8444e69)
  • Telegram pagination (8444e69)
  • IFunny pagination (6fdf4ac)
  • Pikabu pagination (36547ff)
  • pagination for other providers where it's possible
• add author and scrape it (where possible) (#166)
• add statistics field to Post, which includes likes, reposts, views and comments amount fields.
  • statistics field added (6de11f1)
  • extract reposts per each provider where it's possible (8a2f4d0)
• add statistics field to PageInfo, which includes posts, followers and following amount fields.
  • statistics field added (6de11f1)
  • extract following per each provider where it's possible (adee3fa)
• provide interface replacement for uri/path
• remove baseUrl from Skraper interface (a31da1c)
• provide UnknownMedia type (a31da1c)

How can I run this inside a docker container ?
What are the requirements
I can spend some time creating a docker image If anyoane can help me with the requirements to run this

Thank you.

error

Exception in thread "main" java.lang.NoClassDefFoundError: io/ktor/client/HttpClientJvmKt
	at ru.sokomishalov.skraper.client.ktor.KtorSkraperClient.<clinit>(KtorSkraperClient.kt:86)
	at org.example.Main.main(Main.java:11)
Caused by: java.lang.ClassNotFoundException: io.ktor.client.HttpClientJvmKt
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:526)
	... 2 more

Process finished with exit code 1

main.java

package org.example;

import com.google.gson.Gson;
import kotlinx.coroutines.flow.Flow;
import ru.sokomishalov.skraper.client.ktor.KtorSkraperClient;
import ru.sokomishalov.skraper.model.Post;
import ru.sokomishalov.skraper.provider.facebook.FacebookSkraper;

public class Main {
    public static void main(String[] args) {
        KtorSkraperClient httpClient = new KtorSkraperClient();
        FacebookSkraper skraper = new FacebookSkraper(httpClient);
        Flow<Post> posts=  skraper.getPosts("memes");

        System.out.println( new Gson().toJson(posts));
    }
}

pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>spider-java-test</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>21</maven.compiler.source>
        <maven.compiler.target>21</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>ru.sokomishalov.skraper</groupId>
            <artifactId>skrapers</artifactId>
            <version>0.12.1</version>
        </dependency>

        <dependency>
            <groupId>com.google.code.gson</groupId>
            <artifactId>gson</artifactId>
            <version>2.10.1</version>
        </dependency>
    </dependencies>
</project>

The getUserPosts from InstagramSkrapper doesn't work anymore. I didn't have any problems last week but since today I get this error:
Exception in thread "main" com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false') at [Source: (byte[])"<!DOCTYPE html><html class="_9dls" lang="en" dir="ltr"><head><link data-default-icon="https://static.cdninstagram.com/rsrc.php/v3/yI/r/VsNE-OHk_8a.png" rel="icon" sizes="192x192" href="https://static.cdninstagram.com/rsrc.php/v3/yI/r/VsNE-OHk_8a.png" /><meta name="robots" content="noarchive, noimageindex" /><meta charset="utf-8" /><meta name="apple-mobile-web-app-status-bar-style" content="default" /><meta name="mobile-web-app-capable" content="yes" /><meta id="viewport" name="viewport" content="[truncated 264379 bytes]; line: 1, column: 2]

Hey, have issue with importing the library to my android app.

When using
implementation("com.github.sokomishalov.skraper:skrapers:0.4.0")

I get following error:

Checked jitpack.io which says I should use implementation 'com.github.SokoMishaLov:skraper:0.4.0' but when I do that, it can't find any package.
Thanks for the help!

• add tags field to the post
• add author and scrape it (where possible)
• provide interface replacement for uri/path

How can I scrape a single post from instagram and other supported social media sites. Like from the following url:
https://www.instagram.com/foodfusionpk/p/CglWx2Ttsyo/

It is useful to get certain things, e.g.

https://graph.facebook.com/{facebook_account_id}/picture?type=large returns the profile picture (no auth) and https://www.facebook.com/{facebook_account_id}/videos/{post_id}/ returns a video

Is this working for twitter since it recently removed all third party applications to use it's api and none of the libraries in python work now for scraping twitter data, do let me know your pov on this!

Hello,

Thanks for great tool.

I have an issue and 9gag parser fetch only 10 posts each time, different paths checked with -n 20 and --limit 20 it is no matter.

E.g.

./skraper ninegag /hot -t log -n 20
./skraper ninegag /coronavirus -t log -n 30
./skraper ninegag /coronavirus -t log --limit 50

All returns:
Fetched 10 posts. Saved to: /home/developer/skraper-master/ninegag/coronavirus_0 4062020_123420.log

Hey, just curious, is it possible to save fetched Post class to disk? Either via serialization, or as JSON or some such.
Thanks!

Join Dream to Earn app, and earn crypto by tracking your sleeping. Get 10% extra earnings for me and you by using my link https://dreamtoearnapp.com/?invite_code=o2wX1kA0ReeqO5L8 or download the app and add my invite code o2wX1kA0ReeqO5L8 directly.

Hello, is it/would it be possible to write your own scrapers, or to extend existing scrapers?
Currently trying to inherit from FacebookSkraper but this doesn't seem possible right now.
Thank you!

Hello, I've been trying to update skraper version as the older one I was using (without Flow to fetch posts) wasn't working anymore.

I added 0.11.0 version to my project, however, getPosts method currently doesn't emit any data.
As far as I can tell here's something to reproduce the issue.

path = "dracibrno"

Full path that is being fetched is https://m.facebook.com/dracibrno/posts. getPosts finished during the first iteration in FacebookSkraper.kt:57 line nextPath = nextPage ?: break as nextPage == null.

fetchResult is unfortunately too long to copy here.
document variable seems to be filled correctly.

Any help on how to fix the issue would be highly appreciated!

We should bypass it somehow

I might be doing something stupid, but I just tried building skraper and pulling a Twitter profile:

skraper [master]$ ./skraper twitter ElonMusk -n 5
Skraper 0.9.1-SNAPSHOT started

.../sokomishalov/skraper/twitter/ElonMusk_10052022_105216.log
skraper [master]$ ./skraper twitter /ElonMusk -n 5
Skraper 0.9.1-SNAPSHOT started

.../sokomishalov/skraper/twitter/ElonMusk_10052022_105231.log
skraper [master]$ 

In both cases (ElonMusk vs. /ElonMusk), the log is empty and I get no output and no errors.

Am I missing something basic about how to use the tool?

Hello, i have problem with gradle.