Thread.join causes knapsack to hang on circleCI about knapsack_pro-ruby HOT 13 CLOSED

I saw a similar problem among other users from time to time, and it was some leftovers created during RSpec runtime that caused hanging. Maybe you have a not closed connection to DB? Typically it's something like that.

You can clean up the environment with a hook:

KnapsackPro::Hooks::Queue.after_queue do |queue_id|
  print 'After Queue Hook - run after test suite'
end

https://knapsackpro.com/faq/question/what-hooks-are-supported-in-queue-mode
https://knapsackpro.com/faq/question/what-is-the-order-of-rspec-hooks-in-the-queue-mode

from knapsack_pro-ruby.

I took a look at your docs for knapsack user dashboard. It seems like a feature for Heroku-based applications or enterprises with SAML login, which are not applicable to my company.

I'm a bit confused. I assume you already have an account at https://knapsackpro.com because you use Queue Mode. Every user has access to the dashboard. Whether you are on a free or paid plan, you can access the dashboard here https://knapsackpro.com/dashboard/ and check build metrics to see your latest recorded CI builds.

If you don't want to share private company stuff here in a public GitHub issue, you can shoot me an email at support inbox https://knapsackpro.com/contact

You can also share a git commit hash, and I can find builds myself this way.

Another update is although the Regular mode removes the hang for the Unit tests, it transfers the "unresponsive" behavior to the Acceptance test sets, so let me try and find if there indeed some Rspec leftovers that caused this. In the meantime, let me know if you have any ideas. Thank you!

Here are a few common hanging reasons:
https://knapsackpro.com/faq/question/why-knapsack_pro-freezes--hangs-my-ci-for-instance-travis
https://knapsackpro.com/faq/question/why-knapsack_pro-hangs-freezes-is-stale-for-tests-in-chrome-disallow-downloads-in-sandboxed-iframes
https://knapsackpro.com/faq/question/why-knapsack_pro-hangs--freezes--is-stale-ie-for-codeship-in-queue-mode

from knapsack_pro-ruby.

• Shouldn't you call the join method on each created thread? I see you called the join method only on the active record object.
  https://stackoverflow.com/a/3481898/905697

• Is the problem happening on CI without the knapsack_pro gen?

• What database engine do you use? Maybe DB transactions are locked (deadlock) and waiting for each other on CI?

• Can you try the latest version of the knapsack_pro gem?

from knapsack_pro-ruby.

@ArturT Thanks for the prompt reply.

For 1, I actually want the AR thread to block the main thread, and then join with the main thread after it updates, as demonstrated in: https://stackoverflow.com/a/44730822 and https://stackoverflow.com/a/11675647.

For 2, I only run the test in local with Rspec and they passed. My CI is using Knapsack so I suspect this has sth to do with the issue.

For 3, it is Postgres 13:2.

Maybe DB transactions are locked (deadlock) and waiting for each other on CI?

I commented out all the Rspec tests and it still hangs. Somehow just having the Thread.join in the code will cause Knapsnack to be unable to proceed. Remove join out of the Thread and everything is fine again i.e:

Class MyTask
  def order_now
   # Some DB transactions here

   MyModel.transaction do

    Thread.new do
      ActiveRecord::Base.connection_pool.with_connection do
        # Open another DB connection update progress status while in transaction
      end 
    end # remove join here makes everything works, transaction or not  
 end
end

For 4, sure thing, let me get back to you.

from knapsack_pro-ruby.

I meant this code:

Class MyTask
  def order_now
   # Some DB transactions here

   MyModel.transaction do

    Thread.new do
      ActiveRecord::Base.connection_pool.with_connection do
        # Open another DB connection update progress status while in transaction
      end.join   
    end

 end
end

It should probably look like this:

Class MyTask
  def order_now
   # Some DB transactions here

   MyModel.transaction do

    Thread.new do
      ActiveRecord::Base.connection_pool.with_connection do
        # Open another DB connection update progress status while in transaction
      end
    end.join # <----- join should be called on the thread object in order to block the main thread.

 end
end

In your production code, do you create only 1 thread? If not, then you should create threads and store them as variables. After that, iterate on each thread and call the join method, so that each thread can start and run concurrently. You should do join only after all threads are started (outside of loop).

Regarding PostgreSQL. You can find out what SQL queries are stuck in the database by running the following command:

SELECT pid, query, (now() - xact_start), state FROM pg_stat_activity WHERE state IN ('idle in transaction', 'active') ORDER BY (now() - xact_start) desc;

from knapsack_pro-ruby.

@ArturT My bad, sorry, the join was supposed to be on the end of the Thread. It was an honest when I copy pasted. So in my (real) code, it does look like this:

Class MyTask
  def order_now
   # Some DB transactions here

   MyModel.transaction do

    Thread.new do
      ActiveRecord::Base.connection_pool.with_connection do
        # Open another DB connection update progress status while in transaction
      end
    end.join # <----- join should be called on the thread object in order to block the main thread.
 end
end

Let me make the corrections to my comments 🙏

And it is not the DB deadlocks unfortunately. I'm trying the latest Knapsack gem.

from knapsack_pro-ruby.

Do you use Queue Mode or Regular Mode in Knapsack Pro?
You can check if the test is hanging in both modes.

from knapsack_pro-ruby.

Do you use Queue Mode or Regular Mode in Knapsack Pro?
You can check if the test is hanging in both modes.

I'm using Queue Mode. I just ran the code with the latest Knapsack gem and it still hangs, so let me try the regular one.

The behavior I got was that Knapsack logs out something likes this:

Knapsack Pro Queue finished!
Finished in 10 minutes 44 seconds
2464 examples, 0 failures

But it is unable to exit.

Then after some time, it is killed by the CI:

Too long with no output (exceeded 10m0s): context deadline exceeded

from knapsack_pro-ruby.

@ArturT Thanks for sharing the insights, I suspect this too. It is reliving to know this is an issue that has happened before.

I saw a similar problem among other users from time to time, and it was some leftovers created during RSpec runtime that caused hanging. Maybe you have a not closed connection to DB? Typically it's something like that.

Interestingly, after using Regular Mode, the time taken to run the CI reduces from 10 mins -> 3 mins :o

The hanging no longer exists. Would this be odd to you?

from knapsack_pro-ruby.

Can you share a link to the knapsack user dashboard with both CI builds?

from knapsack_pro-ruby.

I took a look at your docs for knapsack user dashboard. It seems like a feature for Heroku-based applications or enterprises with SAML login, which are not applicable to my company.

Another update is although the Regular mode removes the hang for the Unit tests, it transfers the "unresponsive" behavior to the Acceptance test sets, so let me try and find if there indeed some Rspec leftovers that caused this. In the meantime, let me know if you have any ideas. Thank you!

from knapsack_pro-ruby.

@ArturT Thank you, sorry for the confusion as I do not have direct access to those resources.

However, I'm happy to share that thanks to your pointers, I was able to confirm that the hanging behavior is indeed due to some testing residues in Rspec. So the issue was not related to Knapsack. Thank you so much ❤️

from knapsack_pro-ruby.

@quynhethereal I'm glad you found the root issue. Cheers!

from knapsack_pro-ruby.