Giter VIP home page Giter VIP logo

mastering-concurrency-in-python's Introduction

Mastering Concurrency in Python

Mastering Concurrency in Python

This is the code repository for Mastering Concurrency in Python, published by Packt.

Create faster programs using concurrency, asynchronous, multithreading, and parallel programming

What is this book about?

Python is one of the most popular programming languages out there, with numerous libraries and frameworks that facilitate high-performance computing. While concurrency and parallelism in Python behave differently than those in other programming languages, it is still in every way possible to implement Python programs that run concurrently or in parallel and make a significant improvement in execution time. Mastering Concurrency in Python serves as a comprehensive introduction to various advanced concepts in concurrent engineering and programming.

This book covers the following exciting features:

  • Understand the idea of concurrency in programming and relevant concepts such as queues, threads, parallelism.
  • Explore the core syntax and language features that enable concurrency in simple Python problems, namely through concurrent, multiprocessing, asyncio.
  • Understand correct way to implement concurrency
  • Abstract methods to keep the data consistent and application non-blocking, responsive, and reliable.
  • Analyze problems commonly faced in concurrent programming. Utilize application scaffolding to design highly scalable programs that are deeply rooted in goroutines and channels.

If you feel this book is for you, get your copy today!

https://www.packtpub.com/

Instructions and Navigations

All of the code is organized into folders. For example, Chapter01.

The code will look like the following:

async def main(url):
    async with aiohttp.ClientSession() as session:
        await download_html(session, url)

Following is what you need for this book: If you're a developer familiar who's and you Python who want to learn to build high-performance applications that scale by leveraging single-core, multi-core, or distributed concurrency, then this book is for you.

With the following software and hardware list you can run all code files present in the book (Chapter 1-19).

Software and Hardware List

Chapter Software required OS required
All chapters Python 3.6 or above Windows, macOS, and Linux
All chapters A Python Integrated Development Environment (IDE) Windows, macOS, and Linux
5 The requests module (Version 2.19 or above) Windows, macOS, and Linux
8 NumPy (Version 1.15 or above) Windows, macOS, and Linux
8 OpenCV (Version 3.4 or above) Windows, macOS, and Linux
11 Telnet Windows, macOS, and Linux
11 The aiohttp module (Version 3.4 or above) Windows, macOS, and Linux
11 The aiofiles module (Version 0.4 or above) Windows, macOS, and Linux
19 APScheduler (Version 3.5.3 or above) Windows, macOS, and Linux

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. Click here to download it.

Related products

Get to Know the Author

Quan Nguyen is a Python enthusiast and data scientist. He is currently a data analysis engineer at Micron Technology, Inc. With a strong background in mathematics and statistics, Quan is interested in the fields of scientific computing and machine learning. With data analysis being his focus, Quan also enjoys incorporating technology automation into everyday tasks through programming.

Quan's passion for Python programming has led him to be heavily involved in the Python community. He started as a primary contributor for the book Python for Scientists and Engineers and various open source projects on GitHub. Quan is also a writer for the Python Software Foundation and an occasional content contributor for DataScience.com (part of Oracle).

Suggestions and Feedback

Click here if you have any feedback or suggestions.

Download a free PDF

If you have already purchased a print or Kindle version of this book, you can get a DRM-free PDF version at no cost.
Simply click on the link to claim your free PDF.

https://packt.link/free-ebook/9781789343052

mastering-concurrency-in-python's People

Contributors

krisnguyen135 avatar packt-itservice avatar packtutkarshr avatar romydias avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

mastering-concurrency-in-python's Issues

Remove deprecated parameter 'loop' in Python 3.10 and newer

When running the echo server example from chapter 10 under Python 3.10 or newer a TypeError is raised:

TypeError: BaseEventLoop.create_server() got an unexpected keyword argument 'loop'

The solution is to simply remove the 'loop' argument from line 17.

coro = asyncio.start_server(handle_echo, '127.0.0.1', 8888, loop=loop) # original
coro = asyncio.start_server(handle_echo, '127.0.0.1', 8888) # solution

This can be done safely because the function asyncio.get_event_loop that is used internally by asyncio.start_server always returns the currently running loop from Python 3.6 onwards.

Chapter07/example1.py has some logic issues

Problem

The code in the consumer class doesn't guarantee that although the queue has more than two elements, a single consumer can get all the two element. Because other consumers will compete for these elements too, which will often result in a situation that the two elements are got by two different consumers, therefore making the control variable result_size work in an unexpected manner.

Solution

Add a lock when get element from queue.

lock = multiprocessing.Lock()

class ReductionConsumer(multiprocessing.Process):
    def __init__(self, task_queue, result_queue):
         ...

    def run(self):
        pname = self.name
        print("Using process %s..." % pname)

        while True:
            with lock:
                num1 = self.task_queue.get()
                ...
                self.task_queue.task_done()
                self.result_queue.put(num1 + num2)

NameError in Chapter19/example3

When running script Chapter19/example3.py a NameError is raised:

scheduler.shutdown()
NameError: name 'scheduler' is not defined

The problem can be fixed by indenting the line 27 under the main guard.

Current version
if __name__ == '__main__':
    scheduler = BackgroundScheduler()
    ...
scheduler.shutdown()
Solution
if __name__ == '__main__':
    scheduler = BackgroundScheduler()
    ...
    scheduler.shutdown()

RuntimeError caused by missing main guard in multiple examples

When running some scripts on Windows, a following RuntimeError is raised:

raise RuntimeError('''
RuntimeError:
    An attempt has been made to start a new process before the
    current process has finished its bootstrapping phase.
...

This can be fixed by introducing the main guard above the first top-level statement that is not a function or an import and indenting everything by one level after that. Here are the locations and potential solutions:

Chapter01/example1.py
if __name__ == '__main__':
    input = [i for i in range(10 ** 13, 10 ** 13 + 500)]


    # sequential
    ...
Chapter02/example1.py
if __name__ == '__main__':
    input = [i for i in range(10 ** 13, 10 ** 13 + 1000)]

    for n_workers in range(1, multiprocessing.cpu_count() + 1):
        ...
Chapter07/example1.py
if __name__ == '__main__':
    my_array = [i for i in range(20)]

    result = reduce_sum(my_array)
    print('Final result: %i.' % result)

Thread methods changed in Python 3.9 and newer versions

When running the file Chapter05/example6.py in Python 3.9 or newer, this error is raised:

AttributeError: 'MyThread' object has no attribute 'isAlive'. Did you mean: 'is_alive'?

This can be fixed by renaming the method at line 21.

alive = [1 if thread.isAlive() else 0 for thread in threads] # original
alive = [1 if thread.is_alive() else 0 for thread in threads] # solution

Also when running the script under Python 3.10 or newer, a warning is emitted:

DeprecationWarning: setDaemon() is deprecated, set the daemon attribute instead

The issue can be fixed by changing the line 41 like so:

thread.setDaemon(True) # original
thread.daemon = True # solution

The method threading.currentThread was also deprecated in favor of threading.current_thread. When running scripts Chapter12/example3.py and Chapter12/example4.py a warning will be emitted:

DeprecationWarning: currentThread() is deprecated, use current_thread() instead

Interestingly, in chapter 13, the correct method threading.current_thread is used.

Results are not matching in Chapter 1 example 2

In example 2 of chapter 1. When I run the file I get

Result is very large. Only printing the last 5 digits: 35443
Sequential took: 0.05 seconds.
Result is very large. Only printing the last 5 digits: 45807
Concurrent took: 0.02 seconds.

but 35433 is not equal to 45807. The results should be equal to 35443. What is the issue here ?

Potential Issue with Example5 and 6 Chapter08

Hi There,

I tried out examples 05 and 06 from Chapter08 of the book. Both of them run error-free. However, I don't observer any speed-up in Example-06. In fact example06 is slower. Outputs from both examples are below. I am using a Spyder4.0.1 IDE with Python version 3.7.6 on a windows-10 machine with 12 cores. Would appreciate any pointers on how this issue can be resolved.

============Example05.py output====================
Took 1.2080 seconds with 1 process(es).
Took 1.1229 seconds with 2 process(es).
Took 0.9995 seconds with 3 process(es).
Took 1.0282 seconds with 4 process(es).
Took 1.0535 seconds with 5 process(es).
Took 1.0578 seconds with 6 process(es).
Done.

============Example06.py output====================
Took 1.4141 seconds with 1 process(es).
Took 1.1089 seconds with 2 process(es).
Took 1.0429 seconds with 3 process(es).
Took 1.0382 seconds with 4 process(es).
Took 1.0399 seconds with 5 process(es).
Took 1.0465 seconds with 6 process(es).
Done.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.