• raise event
• createCheckStatusResponse
• getStatus

Task.Any needs to handle faulted use case.

How to reproduce:

1. Create a linting error eg: Whitespace before a doc string summary comment, like the one in this line: https://github.com/Azure/azure-functions-durable-python/blob/dev/azure/durable_functions/models/DurableOrchestrationContext.py#L16 where we can insert a space between """ and "Context" to produce the following linter error:

D210 No whitespaces allowed surrounding docstring text

2. Run nox --sessions lint task locally and flake8 doesn't fail.

3. Run flake8 ./azure/durable_functions and it reports the linter issue.

4. Now activate the virtual environment created by nox in ./.nox/lint folder.

5. Run flake8 ./azure/durable_functions => it does not report the linter issue

Need further investigation as to why flake8 doesn't report linter errors when run via nox tool.
Due to this, linter errors can only be caught through pipeline leading to multiple PR commits. Or through GitHub Actions.

Need to enable developer to be able to run flake8 with confidence through Nox.

Back date releases and changelog for existing release
New GitHub Releases and tags

azure-functions-durable-js does nothing for 410 error code and should Python durable functions do the same as well? Or raise a valid Exception.
cc: @cgillum @ConnorMcMahon @anthonychu

Currently, for all client and context functions that take an input, we must pass None if we don't want to pass an input value at all. We should make the input value parameter optional instart_new, call_activity, etc.

For example,

    x = yield context.call_activity("F1", None)

should be

    x = yield context.call_activity("F1")

@davidmrdavid @priyaananthasankar

The library https://pypi.org/project/azure-functions-durable/ does not have a proper project description. Refer to the current README.md from the current setup.py so that all that description gets updated in PyPI.

From https://packaging.python.org/tutorials/packaging-projects/ - here is a way to update long_description through setup.py.

with open("README.md", "r") as fh:
long_description = fh.read()

The RaiseEvent Function will throw a JSON decoding warning in red everytime when trying to decode its input, if I recall correctly. The warning looks like this:

[03/24/2020 00:24:06] Executed 'Functions.RaiseEvent' (Failed, Id=a93d2f7a-8e22-4948-a886-1bafd455db81)
[03/24/2020 00:24:06] System.Private.CoreLib: Exception while executing function: Functions.RaiseEvent. System.Private.CoreLib: Result: Failure
[03/24/2020 00:24:06] Exception: ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: ', url=URL('http://127.0.0.1:17071/durabletask/instances/ce2b661317a04af9ad8ffe45ecd59caf/raiseEvent/A')
[03/24/2020 00:24:06] Stack: File "/azure-functions-core-tools/workers/python/3.8/LINUX/X64/azure_functions_worker/dispatcher.py", line 312, in _handle__invocation_request
[03/24/2020 00:24:06] call_result = await fi.func(**args)
[03/24/2020 00:24:06] File "/code/azure-functions-durable-python/samples/external_events/RaiseEvent/init.py", line 34, in main
[03/24/2020 00:24:06] await client.raise_event(instance_id, event_name, True)
[03/24/2020 00:24:06] File "/usr/local/lib/python3.8/site-packages/azure/durable_functions/models/DurableOrchestrationClient.py", line 165, in raise_event
[03/24/2020 00:24:06] response = await self._post_async_request(request_url, json.dumps(event_data))
[03/24/2020 00:24:06] File "/usr/local/lib/python3.8/site-packages/azure/durable_functions/models/utils/http_utils.py", line 24, in post_async_request
[03/24/2020 00:24:06] data = await response.json()
[03/24/2020 00:24:06] File "/usr/local/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 1026, in json
[03/24/2020 00:24:06] raise ContentTypeError(
[03/24/2020 00:24:06]

Would you like to see Durable Entities supported in Python? Let us know here! If possible, include details about how you plan on using them. Thanks!

Contributors may not understand why they need to carry and potentially edit a .NET configuration file in their Python project. It would be good to shine some light onto that and explain that, at most, they will only be expected to change the version numbers of their dependencies

AB#325490

• Implements Durable Monitoring Pattern
  https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-monitor

• Implement custom orchestration status for long polling and intermediate states of a long running job
  https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-custom-orchestration-status?tabs=javascript

refer to the js link for the implementation for using rpc url

https://github.com/Azure/azure-functions-durable-js/blob/1cd822e6cee2fd110638db6cfcaf3bc40c6d284f/src/durableorchestrationclient.ts#L383

Orchestrations are assumed to call an activity at least once, this means that an empty orchestration will throw an exception.

Remove the following redundant function names:

def call_activity(self, name: str, input_=None) -> Task:
"""Schedule an activity for execution.

    :param name: The name of the activity function to call.
    :param input_:The JSON-serializable input to pass to the activity
    function.
    :return: A Durable Task that completes when the called activity
    function completes or fails.
    """
    raise NotImplementedError("This is a placeholder.")

def call_activity_with_retry(self,
                             name: str, retry_options: RetryOptions,
                             input_=None) -> Task:
    """Schedule an activity for execution with retry options.

    :param name: The name of the activity function to call.
    :param retry_options: The retry options for the activity function.
    :param input_: The JSON-serializable input to pass to the activity
    function.
    :return: A Durable Task that completes when the called activity
    function completes or fails completely.
    """
    raise NotImplementedError("This is a placeholder.")

Need to build out the ability to create SubOrchestration and SubOrchestration with retry capabilities within the DurableOrchestrationContext.

I'm trying to implement the Singleton orchestrators following the documentation for .net and the get_status function inside the DurableOrchestrationClient is trowing an Exception (ContentTypeError) the first time I call the HttpStart.

Exception: ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: ', url=URL('http://127.0.0.1:17071/durabletask/instances/76847474aa614629af033e2fcda0984c')

That url gives me a 404 with an empty body, I think this is ok, because is the first call and that instance is not been created already.
I would expect this function to return an empty object instead of throwing an error if the instance doesn't exist.

This is my singleton implementation in the HttpStart:

import logging

import azure.functions as func
import azure.durable_functions as df

instances_dict = {
    "Function1Orchestator": "76847474aa614629af033e2fcda0984c",
    "Function2Orchestator": "7aac2c5ddac1428e9e443b577691d439",
}

async def main(req: func.HttpRequest, starter: str) -> func.HttpResponse:
    client = df.DurableOrchestrationClient(starter)

    # Singleton implementation
    function = req.route_params["functionName"]
    existing_instance = await client.get_status(instances_dict[function])

    if existing_instance is None:
        instance_id = await client.start_new(function, instances_dict[function], None)

        logging.info(f"Started orchestration with ID = '{instance_id}'.")

        return client.create_check_status_response(req, instance_id)
    else:
        return client._create_http_response(409, "Function alredy running.")

Thanks and sorry if this doesn't belong here.

Users won't be able to run a func host start on a sample without setting a value for azureWebJobsStorage in their local.settings.json file. This step is skipped in the Contributor's Guide section about 'end-to-end testing'.

I suggest including a sample local.settings.json file that uses the local development storage.

Looking at the fan out fan in example, there are are a few places where we have to manually serialize/deserialize data that we should eliminate.

When call_activity is called, we automatically serialize the input value to a string (for example, we serialize the int to a str here). However, in the activity function itself, we need to explicitly convert the value back.

When we return a value from an activity, we require manual serialization. And in the orchestrator, we manually deserialize it.

Proposed changes:

• Always deserialize input values for activity functions.
• Always serialize return values for activity functions.
• Always deserialize values returned by the task from call_activity (and the other call_activity and call_suborchestrator variants).
• Always serialize input value in start_new (looks like we already do this)
• Always deserialize get_input

Some questions:

• What do we do with type hints? (It appears we currently ignore the type hint for an activity function input and always pass a string, as demonstrated here)
• Do we support deserialization to custom classes or stick to primitives like str, int, float, list, dict, etc (whatever json.loads supports)?

The changes will allow the orchestrator code to the written like this:

def orchestrator_function(context: df.DurableOrchestrationContext):
    activity_list = yield context.call_activity("GetActivityCount", 5)

    tasks = [context.call_activity("ParrotValue", i) for i in activity_list]

    values = yield context.task_all(tasks)
    message = yield context.call_activity("ShowMeTheSum", values)

    return message

The activity function would then look like this:

def main(value: int) -> list:
    activity_values = [*range(value)]
    return activity_values

We should be able to remove all import json statements from orchestrators and activities except when we're really working with JSON data.

@davidmrdavid @priyaananthasankar

Fan In Fan Out Tensorflow - completed
Wait for External Event - pending

These samples need a proper README file.

need to make sure the result value passed back from the durable extension is being parsed as JSON

Folder grpc/protobuf needs to be removed as it is not used here.

Justification to use numpy docstring is given below (from https://numpydoc.readthedocs.io/en/latest/format.html)

From: https://www.datacamp.com/community/tutorials/docstrings-python#seven-sub
Sphinx is the easy and traditional style, verbose and was initially created specifically for the Python Documentation. Sphinx uses a reStructuredText which is similar in usage to Markdown.

Numpy style has a lot of details in the documentation. It is more verbose than other documentation, but it is an excellent choice if you want to do detailed documentation, i.e., extensive documentation of all the functions and parameters.

Given that numpy is also based on restructuredText, we get best of Sphinx world as well. It is a superset for detailed documentation.

More reading justification:

NumPy, SciPy, and the scikits follow a common convention for docstrings that provides for consistency, while also allowing our toolchain to produce well-formatted reference guides. This document describes the current community consensus for such a standard. If you have suggestions for improvements, post them on the numpy-discussion list.

Our docstring standard uses re-structured text (reST) syntax and is rendered using Sphinx (a pre-processor that understands the particular documentation style we are using). While a rich set of markup is available, we limit ourselves to a very basic subset, in order to provide docstrings that are easy to read on text-only terminals.

A guiding principle is that human readers of the text are given precedence over contorting docstrings so our tools produce nice output. Rather than sacrificing the readability of the docstrings, we have written pre-processors to assist Sphinx in its task.

The length of docstring lines should be kept to 75 characters to facilitate reading the docstrings in text terminals.

Library is overkill for what amounts to a a string concatenation function. Keep in the the test library, but not a requirement for this class.

Test public fork and existing GitHub Action CI workflow.

AB#325489

• Implement FanIn/FanOut scenario for Durable Python
  https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-cloud-backup

The snippet on the repo only shows pipeline to be running from master branch. The pipeline runs on dev branch. Update the snippet to reflect the same

https://github.com/Azure/azure-functions-python-worker/wiki/Adding-support-for-a-new-binding-type for Durable.

AB#308899 Need to create the functionality required to be able to implement the Function Chaining pattern.

PR Merge Restrictions:

1. Create dev branch
2. Update Azure CI Pipeline to run off both dev and master branch
3. Restricted Branch policy for both dev and master branch

Got some of these errors accessing functions in http_util.py due to @staticmethod decorator on functions outside classes:

Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.HttpTrigger
 ---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException: Result: Failure
Exception: TypeError: 'staticmethod' object is not callable
Stack:   File "/azure-functions-host/workers/python/3.7/LINUX/X64/azure_functions_worker/dispatcher.py", line 312, in _handle__invocation_request
    call_result = await fi.func(**args)
  File "/home/site/wwwroot/HttpTrigger/__init__.py", line 9, in main
    instance_id = await client.start_new("HelloOrchestrator", None, "Seattle")
  File "/usr/local/lib/python3.7/site-packages/azure/durable_functions/models/DurableOrchestrationClient.py", line 71, in start_new
    response = await self._post_async_request(request_url, self._get_json_input(client_input))
  File "/usr/local/lib/python3.7/site-packages/azure/durable_functions/models/DurableOrchestrationClient.py", line 40, in <lambda>
    self._post_async_request = lambda u, d: post_async_request(u, d)

Refactor double negation and complementary operators to improve code readability for all find*methods in task_utilities.py
Eg:

 tasks = list(
        filter(lambda e:
               not(not (e["EventType"] == HistoryEventType.TIMER_CREATED) 
                   and (e["FireAt"] == fire_at)
               ))

Need to spend sometime to refactor the code to be more inline with PEP-8 and more readable all together

Remove those folders as they are not required at the library level.

I'm trying to implement the Singleton orchestrators following the documentation for .net and the get_status function inside the DurableOrchestrationClient is trowing an Exception (ContentTypeError) the first time I call the HttpStart.

Exception: ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: ', url=URL('http://127.0.0.1:17071/durabletask/instances/76847474aa614629af033e2fcda0984c')

That url gives me a 404 with an empty body, I think this is ok, because is the first call and that instance is not been created already.
I would expect this function to return an empty object instead of throwing an error if the instance doesn't exist.

Thanks and sorry if this doesn't belong here.

right now it only takes tasks, so if we do something like
timertask= # createtimer....
yield context.df.task_any(context.df.task_all([task1, task2]), timertask)
this will cause an error

Publish a README for API Design to be reviewed by all parties continuously through evolution of the project.

Implement the feature for set_custom_status API.

• update azure-pipeline.yml
• add status badge to readme.md

In the Contributor's Guide, there's a section titled "Setting up durable-py debugging" which describes changes that are needed in the host.json file. However, there are many host.json files in the repo. It would be good to acknowledge in the text that there are many such files, perhaps explain their differences, and specify which needs to be changed.

AB#325251 Wait for events
The WaitForExternalEvent (.NET) and waitForExternalEvent (JavaScript) methods of the orchestration trigger binding allows an orchestrator function to asynchronously wait and listen for an external event. The listening orchestrator function declares the name of the event and the shape of the data it expects to receive.

The RaiseEventAsync (.NET) or raiseEvent (JavaScript) method of the orchestration client binding sends the events that WaitForExternalEvent (.NET) or waitForExternalEvent (JavaScript) waits for. The RaiseEventAsync method takes eventName and eventData as parameters. The event data must be JSON-serializable.

Implement above API's with Python

Currently, there are two approaches to how we can handle errors in orchestrations.

1. Throw the error

This causes the function runtime to register that the orchestration execution actually failed. This is positive from an AppInsights + customer visibility perspective.

The downside is that if we throw the error, we don't get the replay events sent back to the C# extension, meaning that we can't replay actions in the extension. This leads to a non-determinism error from the Durable Task perspective.

2. Send the error as data

This is the approach Durable Python is currently taking. This is a relatively clean solution, but unfortunately, it means that the function execution that encountered the exception is marked as completed. This is very confusing from a developer perspective, as in App Insights it will show that the orchestration function succeeded.

The way we worked around this in Durable JavaScript is via a relatively hacky approach, but it is the only way that meets all of our criteria.

3. Embed replay data inside of the error we throw

We still throw an error, but we wrap the error in a custom error, that also embeds the replay data as JSON in the error message. This accomplishes both goals of marking the orchestration execution as failed from the functions runtime perspective, as well as having the correct error message in the Durable Task perspective.

You can see the way JS implemented it here Azure/azure-functions-durable-js#145.

This approach is not perfect. It is fairly fragile, and it definitely makes the exception message that customers will see in their app insights far messier. But it is the only approach we can currently take without making some changes to how Functions handles out-of-process errors.

Tasks package has orchestration API's. Need to find a suitable name than "tasks".

In the section titled "Setting up durable-py debugging" of the Contributor's Guide, there's an instruction to specify the name of our local task hub. However, the concept of task hubs is not introduced. I suggest including a link to this such that users may read more about what they're setting.

I would also give a suggestion for a starter task hub name, such as "MyTaskHub"

@Hazhzeng Could you please provide an example of the header that we should include in every file?

It should probably be added to the 'Python 3 Tools (pip install)' section of "Pre-requisites".

As an aside, and perhaps something that should be explained in the guide, is the installed code the same as in the `/azure' directory of the repo? If not, how are they different?