Giter VIP home page Giter VIP logo

Comments (7)

imaxs avatar imaxs commented on May 26, 2024 2

I've made sure that puppeteer-extra is a useless thing.
Nothing is working.

from puppeteer-extra.

insinfo avatar insinfo commented on May 26, 2024

It seems like the problem is that it can't fool the devtools detectors
https://cdn.jsdelivr.net/npm/devtools-detector
https://github.com/AEPKILL/devtools-detector

from puppeteer-extra.

Overmiind avatar Overmiind commented on May 26, 2024

I've written a simple script to bypass these detectors. It works with this website; I'm not sure about the others
You should evaluate this script on a new document like this:

page.evaluateOnNewDocument(script)

from puppeteer-extra.

insinfo avatar insinfo commented on May 26, 2024

@Overmiind
I added your script but it is still detecting and redirecting to https://brbeast.com/player/includes/template/no_video.html
i.e. having the same problem

//npm install puppeteer puppeteer-extra puppeteer-extra-plugin-stealth
const puppeteer = require('puppeteer-extra')
const sleep = ms => new Promise(res => setTimeout(res, ms));

const script = `
() => {
  function isDevToolsScript() {
      var stack = new Error().stack;
      return stack.includes('devtool');
  }

  Date.prototype.originalGetTime = Date.prototype.getTime;
  Date.prototype.getTime = function () {
      if (!isDevToolsScript()) {
          return this.originalGetTime();
      }
      return 0;
  }

  const originalOnMessageSetter = Object.getOwnPropertyDescriptor(Worker.prototype, 'onmessage').set;
  Object.defineProperty(Worker.prototype, 'onmessage', {
      set: function (fn) {
          if (!isDevToolsScript()) {
              originalOnMessageSetter.call(this, fn);
              return;
          }
          newFn = (ev) => {
              ev.data.time = 0;
              fn(ev);
          }
          originalOnMessageSetter.call(this, newFn);
      }
  });
}
`;

// puppeteer-extra is a drop-in replacement for puppeteer,
// it augments the installed puppeteer with plugin functionality
const url = 'https://brbeast.com/video/f79921bbae40a577928b76d2fc3edc2a';

// add stealth plugin and use defaults (all evasion techniques)
const StealthPlugin = require('puppeteer-extra-plugin-stealth')
puppeteer.use(StealthPlugin())

// puppeteer usage as normal
puppeteer.launch({ headless: true }).then(async browser => {
  console.log('Running tests..')
  const page = await browser.newPage()
  page.evaluateOnNewDocument(script)
  page.goto(url)
 

  await sleep(5000)
  await page.screenshot({ path: 'testresult.png', fullPage: true })
  await browser.close()
  console.log(`All done, check the screenshot. ✨`)
})

from puppeteer-extra.

Overmiind avatar Overmiind commented on May 26, 2024

You can block devtools scripts from being downloaded using puppeteer request interception. I tried that on a puppeteer c# version and it works

from puppeteer-extra.

jaimecoj avatar jaimecoj commented on May 26, 2024

You can block devtools scripts from being downloaded using puppeteer request interception. I tried that on a puppeteer c# version and it works

Can you give more information on how to block devtools scripts download?

from puppeteer-extra.

insinfo avatar insinfo commented on May 26, 2024

I did this in Dart and it works

import 'dart:async';
import 'dart:convert';
import 'dart:io';
import 'dart:math';

import 'package:http/http.dart';
import 'package:puppeteer/puppeteer.dart';

//https://brbeast.com/video/e4bb4c5173c2ce17fd8fcd40041c068f

final customUA =
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36';

final script = '''
() => {

  window.onbeforeunload = function() { return "sorry, Your some work will be lost - really sorry."; };
  
  function isDevToolsScript() {
      var stack = new Error().stack;
      return stack.includes('devtool');
  }

  Date.prototype.originalGetTime = Date.prototype.getTime;
  Date.prototype.getTime = function () {
      if (!isDevToolsScript()) {
          return this.originalGetTime();
      }
      return 0;
  }

  const originalOnMessageSetter = Object.getOwnPropertyDescriptor(Worker.prototype, 'onmessage').set;
  Object.defineProperty(Worker.prototype, 'onmessage', {
      set: function (fn) {
          if (!isDevToolsScript()) {
              originalOnMessageSetter.call(this, fn);
              return;
          }
          newFn = (ev) => {
              ev.data.time = 0;
              fn(ev);
          }
          originalOnMessageSetter.call(this, newFn);
      }
  });
}''';

final blockUrls = [
  //'https://pl20623807.toprevenuegate.com/20/41/ad/2041ad026c42ce264b91586de1c33c6e.js',
  'https://cdn.jsdelivr.net/npm/devtools-detector',
  //'https://code.jquery.com/jquery-1.12.4.min.js',
  //'https://brbeast.com/player/assets/scripts.php?v=6'
];

void main() async {
  try {
    await run();
  } catch (e, s) {
    print('main $e $s');
  }
}

Future<void> run() async {
  final completer = Completer<void>();
  final url = 'https://brbeast.com/video/f79921bbae40a577928b76d2fc3edc2a';
  final browser = await puppeteer.launch(
    //timeout: Duration(days: 4),
    headless: false,
    defaultViewport: DeviceViewport(width: 1280, height: 720),
  );
  // Set custom user agent

  var page = await browser.newPage();
  await page.evaluateOnNewDocument(script);
  await page.setUserAgent(customUA);

  // Setting page view  { 'width': 1280, 'height': 720 }
  // await page.setViewport(DeviceViewport(width: 1280, height: 720));
// await page.setExtraHTTPHeaders({
// 		'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36',
// 		'upgrade-insecure-requests': '1',
// 		'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',
// 		'accept-encoding': 'gzip, deflate, br',
// 		'accept-language': 'en-US,en;q=0.9,en;q=0.8'
// 	});
  await page.setRequestInterception(true);
  //:  https://mrbeast1.com/cdn/down/disk2/8e011e9e2e455c20b9a94345d0bc7af6/Video/720p/720p_020.html
  page.onRequest.listen((req) async {
    if (blockUrls.contains(req.url)) {
      req.abort();
    } else {
      req.continueRequest();
    }
  });
  final exp = RegExp(r'(?:(?:https?|ftp):\/\/)?[\w/\-?=%.]+\.[\w/\-?=%.]+');
  page.onResponse.listen((response) async {
    if (response.url.contains('https://brbeast.com/m3/')) {
      var m3u8data = utf8.decode(await response.bytes);
      await File('downloads/m3u8data.txt').writeAsString(m3u8data);
      Iterable<RegExpMatch> matches = exp.allMatches(m3u8data);
      List<Uri> urls = [];
      for (var match in matches) {
        final urlVideoTs = m3u8data.substring(match.start, match.end);
        if (urlVideoTs.startsWith('http')) {
          //print('url: $urlVideoTs');
          final uri = Uri.tryParse(urlVideoTs);
          if (uri != null) {
            urls.add(uri);
          }
        }
      }
      await browser.close();
      await downloadFile(urls);
      completer.complete();
    }
    // a url que tem master.txt é o arquivo m3u8
    // if (response.url.contains('master.txt')) {
    //   final name = sanitizeFilename(response.url.split('/').last);
    //   //print(response.headers);
    //   final data = utf8.decode(await response.bytes);
    //   var match = exp.allMatches(data).last;
    //   var urlMaster = data.substring(match.start, match.end);
    //   print(urlMaster);
    //   await File('downloads/$name').writeAsString(data);
    //   downloadFile(urlMaster);
    // }
  });
  await page.goto(url, wait: Until.domContentLoaded);
  //await Future.delayed(Duration(seconds: 10));
  //await page.screenshot();
  // await myPage.pdf();
  // await page.evaluate<String>('() => document.title');

  // Gracefully close the browser's process
  // browser.close();

  // await Future.delayed(Duration(days: 4));

  return completer.future;
}

Future<void> downloadFile(List<Uri> urlV) async {
  final videoFile =
      await File('downloads/video.ts').open(mode: FileMode.writeOnlyAppend);
  final audioFile =
      await File('downloads/audio.ts').open(mode: FileMode.writeOnlyAppend);

//
  for (var item in urlV) {
    if (item.path.contains('Video')) {
      print('downloadFile $item');
      final request = await get(item);
      for (var byte in request.bodyBytes) {
        await videoFile.writeByte(byte);
      }
      await videoFile.flush();
    }
    if (item.path.contains('Audio')) {
      print('downloadFile $item');
      final request = await get(item);
      for (var byte in request.bodyBytes) {
        await audioFile.writeByte(byte);
      }
      await audioFile.flush();
    }
  }

  await videoFile.close();
  await videoFile.close();
}

/// Replaces characters in [input] that are illegal/unsafe for filenames.
///
/// You can also use a custom [replacement] if needed.
///
/// Illegal Characters on Various Operating Systems:
/// / ? < > \ : * | "
/// https://kb.acronis.com/content/39790
///
/// Unicode Control codes:
/// C0 0x00-0x1f & C1 (0x80-0x9f)
/// https://en.wikipedia.org/wiki/C0_and_C1_control_codes
///
/// Reserved filenames on Unix-based systems (".", "..")
/// Reserved filenames in Windows ("CON", "PRN", "AUX", "NUL", "COM1",
/// "COM2", "COM3", "COM4", "COM5", "COM6", "COM7", "COM8", "COM9",
/// "LPT1", "LPT2", "LPT3", "LPT4", "LPT5", "LPT6", "LPT7", "LPT8", and
/// "LPT9") case-insesitively and with or without filename extensions.
///
/// Each results have a maximum length of 255 characters:
/// https://unix.stackexchange.com/questions/32795/what-is-the-maximum-allowed-filename-and-folder-size-with-ecryptfs
///
/// Example:
/// ```dart
/// import 'package:sanitize_filename/sanitize_filename.dart';
///
/// const unsafeUserInput = "~/.\u0000ssh/authorized_keys";
///
/// // "~.sshauthorized_keys"
/// sanitizeFilename(unsafeUserInput);
///
/// // "~-.-ssh-authorized_keys"
/// sanitizeFilename(unsafeUserInput, replacement: "-");
/// ```
String sanitizeFilename(String input, {String replacement = ''}) {
  if (input == '') {
    return '_vazio_' + Random().nextInt(10000).toString();
  }
  final result = input
      // illegalRe
      .replaceAll(
        RegExp(r'[\/\?<>\\:\*\|"]'),
        replacement,
      )
      // controlRe
      .replaceAll(
        RegExp(
          r'[\x00-\x1f\x80-\x9f]',
        ),
        replacement,
      )
      // reservedRe
      .replaceFirst(
        RegExp(r'^\.+$'),
        replacement,
      )
      // windowsReservedRe
      .replaceFirst(
        RegExp(
          r'^(con|prn|aux|nul|com[0-9]|lpt[0-9])(\..*)?$',
          caseSensitive: false,
        ),
        replacement,
      )
      // windowsTrailingRe
      .replaceFirst(RegExp(r'[\. ]+$'), replacement);

  return result.length > 255 ? result.substring(0, 255) : result;
}

from puppeteer-extra.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.