bruno- / fiber_scheduler Goto Github PK

Ruby's missing Fiber Scheduler implementation.

License: MIT License

Ruby 100.00%

ruby fiber fiber-scheduler async asynchronous-programming concurrency async-programming

fiber_scheduler's Introduction

Hi 👋

I'm Bruno Sutic and I'm a senior full-stack developer working with web technologies for over a decade. I work primarily with Ruby, Ruby on Rails, vanilla JavaScript, and Ansible.

I specialize in working with Async Ruby. Get in touch if you need Async Ruby training or help with a project.

I'm a founder at linkok.com.

I created:

fiber_scheduler's People

Contributors

Stargazers

Watchers

Forkers

aalin shugo joshfng vizakenjack

fiber_scheduler's Issues

Question about io_read

I have a question about the logic in io_read at https://github.com/bruno-/fiber_scheduler/blob/main/lib/fiber_scheduler/selector.rb#L98

It seems that it considers the resource unavailable (-EAGAIN) if length == 0 and the read failed. This makes sense because if I try to read nothing and it fails, then it must not be possible to read. However the read does not use length at all! It uses maximum_size which looks likely to be related to length, but is not the same value. Should this perhaps instead says if maximum_size > 0 ?

Bad file descriptor on Windows

Awesome project! I tried the Async gem but currently it's not supported on Windows due to io-event gem, so i gave this one a try and ran into an error.

Environment:

Windows 10
Ruby 3.1.1p18 (2022-02-18 revision 53f5fc4236) [x64-mingw-ucrt] installed via (RailsInstaller)

Getting an error

<internal:io>:63:in `read_nonblock': Bad file descriptor @ rb_io_set_nonblock - C:/Windows/System32/drivers/etc/hosts (Errno::EBADF)

Reproduction:

require 'fiber_scheduler'
require 'open-uri'

FiberScheduler do
  Fiber.schedule { URI.open('http://ip.me') }
  Fiber.schedule { URI.open('http://ip.me') }
end

I tried a different scheduler found at the very bottom of this article and it works right away but I'm not sure if it's reliable.

Update gem published on RubyGems.org

The last commit on github was 12/07/2022 to make the library compatible with Ruby 3.2.2. But this commit is not in the gem 0.13.0 pushed on Rubygems.org at https://rubygems.org/gems/fiber-scheduler.

Therefore, the gem is creating errors on Ruby 3.2 - Expect 4 params, provided 3. on line 175 of fiber_scheduler.rb.

It took a while to figure it out. Hope this helps other people.

Too many inner loops causes a crash

Hi there. Thank you for all of your efforts with fiber_scheduler and the thought and planning that you've put into positioning it as a great default. I'm getting up to speed with Ruby 3 non-blocking fibers and have benefitted from your related writing.

I was playing around with using the fiber_scheduler gem and comparing it to other schedulers. I am using a non-block context (Fiber.set_scheduler). In an effort to have some test code do something that would take a little longer to complete, I added a 1_000.times loop wrapper and found that doing so causes a crash.

Using the code below I can reliably reproduce a crash with both Ruby 3.1.2 and 3.2.0-preview1.
If I remove the 1_000.times wrapper or lower the loop number to 92, the code works without crashing.
If I replace fiber_scheduler with async and FiberScheduler.new with Async::Scheduler.new, the code works without crashing.
As far as I can tell, #alive? ends up getting called on the wrong object, causing the crash. The object that method gets called on can change between runs.

Code to reproduce:

#!/usr/bin/env ruby
# frozen_string_literal: true

require 'fiber_scheduler'

Thread.new do
  Fiber.set_scheduler FiberScheduler.new
  %w[apple banana grape lemon cherry lime].each do |fruit|
    Fiber.schedule do
      start_time = Time.now
      1_000.times do
        Math.exp(fruit.chars.sort.reverse.uniq.map(&:upcase).shuffle.sample.ord)
      end
      stop_time = Time.now
      puts "==> Elapsed #{stop_time - start_time} <== "
    end
  end
end.join

Error output:

~/.rubies/ruby-3.2.0-preview1/lib/ruby/gems/3.2.0+1/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler.rb:122:in `select': undefined method `alive?' for [:blocking, #<RubyVM::AbstractSyntaxTree::Node:FALSE@47:21-47:26>]:Array (NoMethodError)

Can't print

Hey! This program raises

require 'fiber_scheduler'

FiberScheduler do
  Fiber.schedule do
    p 1
  end
end

The exception is

/Users/fxn/.gem/ruby/3.2.2/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler.rb:179:in
`io_write': wrong number of arguments (given 4, expected 3) (ArgumentError)

`io_read': wrong number of arguments (given 4, expected 3) (ArgumentError)

# fibers_test.rb

require 'bundler/inline'

gemfile do
  source 'https://rubygems.org'
  gem 'fiber_scheduler', '~> 0.13.0'
end

require 'net/http'
require 'fiber_scheduler'

require 'benchmark'

Benchmark.realtime do
  Thread.new do # in this thread, we'll have non-blocking fibers
    Fiber.set_scheduler(FiberScheduler.new)

    (1..3).each do |user_id|
      Fiber.schedule do
        t = Time.now
        Net::HTTP.get('reqres.in', "/api/users/#{user_id}?delay=2")
        puts 'User %s: finished in %.3f' % [user_id, Time.now - t]
      end
    end
  end.join
end

This is the result of running the script

$ ruby ./fibers_test.rb 
#<Thread:0x00007fefaee35c60 ./fibers_test.rb:14 run> terminated with exception (report_on_exception is true):
/home/username/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler.rb:175:in `io_read': Failed to open TCP connection to reqres.in:80 (wrong number of arguments (given 4, expected 3)) (ArgumentError)
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:193:in `each'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:193:in `block (2 levels) in lazy_initialize'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:192:in `open'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:192:in `block in lazy_initialize'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:188:in `synchronize'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:188:in `lazy_initialize'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:236:in `each_address'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:116:in `block in each_address'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:115:in `each'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:115:in `each_address'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:102:in `getaddresses'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:51:in `getaddresses'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler.rb:164:in `address_resolve'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `initialize'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `open'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `block in connect'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler/timeouts.rb:55:in `timeout'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler.rb:188:in `timeout_after'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/timeout.rb:178:in `timeout'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1269:in `connect'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1248:in `do_start'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1237:in `start'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:474:in `get_response'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:461:in `get'
	from ./fibers_test.rb:20:in `block (4 levels) in <main>'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler.rb:240:in `block in fiber'
/home/username/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler.rb:175:in `io_read': wrong number of arguments (given 4, expected 3) (ArgumentError)
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:193:in `each'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:193:in `block (2 levels) in lazy_initialize'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:192:in `open'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:192:in `block in lazy_initialize'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:188:in `synchronize'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:188:in `lazy_initialize'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:236:in `each_address'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:116:in `block in each_address'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:115:in `each'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:115:in `each_address'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:102:in `getaddresses'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:51:in `getaddresses'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler.rb:164:in `address_resolve'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `initialize'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `open'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `block in connect'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler/timeouts.rb:55:in `timeout'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler.rb:188:in `timeout_after'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/timeout.rb:178:in `timeout'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1269:in `connect'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1248:in `do_start'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1237:in `start'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:474:in `get_response'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:461:in `get'
	from ./fibers_test.rb:20:in `block (4 levels) in <main>'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler.rb:240:in `block in fiber'
/home/username/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler.rb:175:in `io_read': Failed to open TCP connection to reqres.in:80 (wrong number of arguments (given 4, expected 3)) (ArgumentError)
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:193:in `each'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:193:in `block (2 levels) in lazy_initialize'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:192:in `open'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:192:in `block in lazy_initialize'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:188:in `synchronize'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:188:in `lazy_initialize'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:236:in `each_address'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:116:in `block in each_address'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:115:in `each'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:115:in `each_address'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:102:in `getaddresses'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:51:in `getaddresses'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler.rb:164:in `address_resolve'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `initialize'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `open'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `block in connect'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler/timeouts.rb:55:in `timeout'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler.rb:188:in `timeout_after'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/timeout.rb:178:in `timeout'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1269:in `connect'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1248:in `do_start'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1237:in `start'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:474:in `get_response'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:461:in `get'
	from ./fibers_test.rb:20:in `block (4 levels) in <main>'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler.rb:240:in `block in fiber'
/home/username/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler.rb:175:in `io_read': wrong number of arguments (given 4, expected 3) (ArgumentError)
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:193:in `each'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:193:in `block (2 levels) in lazy_initialize'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:192:in `open'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:192:in `block in lazy_initialize'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:188:in `synchronize'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:188:in `lazy_initialize'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:236:in `each_address'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:116:in `block in each_address'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:115:in `each'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:115:in `each_address'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:102:in `getaddresses'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/resolv.rb:51:in `getaddresses'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler.rb:164:in `address_resolve'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `initialize'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `open'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1271:in `block in connect'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler/timeouts.rb:55:in `timeout'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler.rb:188:in `timeout_after'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/timeout.rb:178:in `timeout'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1269:in `connect'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1248:in `do_start'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:1237:in `start'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:474:in `get_response'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/3.2.0/net/http.rb:461:in `get'
	from ./fibers_test.rb:20:in `block (4 levels) in <main>'
	from /home/username/.asdf/installs/ruby/3.2.2/lib/ruby/gems/3.2.0/gems/fiber_scheduler-0.13.0/lib/fiber_scheduler.rb:240:in `block in fiber'

$ ruby -v
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [x86_64-linux]

The script works fine with Async::Scheduler from async-io gem

Thread#join sometimes doesn't wake up when thread terminates

I'm using fiber_scheduler 0.13.0. Here's my code:

require 'fiber_scheduler'

Fiber.set_scheduler FiberScheduler.new
Fiber.schedule do
    t = Thread.new do
    end
    t.join 3
end

Sometimes t.join 3 returns quickly, which is what I want to happen since thread t almost immediately terminates. However, usually t.join 3 takes 3 seconds to return, which means something (the fiber scheduler, I think) prevented t.join from waking up early.

By adding debug logging to fiber_scheduler, I've figured out two things:

when t.join 3 returns quickly, it's because FiberScheduler::Selector#select called IO.select([], [], nil, 0)
when t.join 3 returns after 3 seconds, it's because FiberScheduler::Selector#select called IO.select([], [], nil, 3)

This feels like a bug in fiber_scheduler. Am I right?

(If it matters, I'm using Linux on x86_64, and I installed fiber_scheduler through Bundler.)

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.