Magnus ran into a problem where a BlobTrigger + Blob (output) function kept looping forever. It's likely due to a binding failure. Other case to check:
Blob file name with invalid characters

History:
Magnus:
Oh I forgot about this demo ‘fail’! Thank you for reminding me.

I guess you’re right it has to be something like that.

I have attached a zip with the demo code which should compile anywhere. I removed the account and key from the connection strings. As for demo data it’s easy to see from the screen shot below. What I did was allow the audience to break the app by sending me an azure blob file name with invalid characters. Also I use the same data for the rowkey in the table and there are potential fails there too!

If someone else would like to test this or maybe even someone in the team (?) I would be delighted! I was surprised that some of these outputs went into perpetual loops! My resulting azure table had lots of rows and the execution just continued.

Let me know if you find anything interesting!

David:
I’m guessing it’s the equivalent of a poison message for BlobInput. Perhaps there’s a non-text blob and binding to TextReader fails. And because we never write the BlobOutput, we keep triggering the function every time we see that BlobInput in our scans.

--- AzureJobsDashboard and AzureJobsStorage are different connectionString
--- Have convention as AzureJobs* as name of ConnectionString which user has to be specified
--- Dashboard works with multiple storage accounts
--- Dashboard is Dict<AccountName|ConnectionString>

I would be really cool to have support for IEnumerable<> binding on blob.
The scenario where we need this is the PhluffyShffy sample: there is a container with the images uploaded by the user. When a queue message is picked up by the job, it also needs get all the blobs associated to that message - the association is done by the name convention.

It would be something like:

[Blob("container/{fileName}")]IEnumerable<CloudBlockBlob> files

IEnumerable works well for CloudBlockBlob because you can get the blob name. For types which don't have this information, IDictionary<string, T> might do the trick

[Blob("container/{fileName}")]IDictionary<string, Stream> files

Check whether both NamespaceManager and MessagingFactory will work during indexing (for example, validate credentials).

We need to make sure we're handling the following two conditions correctly (from Brent Stineman):

1. A large enough block blob will have multiple put operations. But since we’re doing a block blob, the blob shouldn’t be available until the final “commit” is done. So is the web jobs SDK using the storage analytics to detect the blob, then polling for the final presence of the blob?
2. If we use page blobs, the blob is created immediately but may be in complete? So is it then the responsibility of the sender to ensure that the blob is exclusively leased? If so, are leases also at play in the block blob scenario?

When I start a Host using the SDK we print out the "Executing..." when the function gets triggered. We should output something like "Finished Executing" which would indicate that the function has finished executing.

BlobTrigger keeps firing even after it's been processed. Today, we don't have any way to avoid continuous automatic firing in every scenario. For example, consider:

public static void ProcessBlobIntoQueue([BlobTrigger("container/{name}")] string blob, [Queue("myqueue")] out string queue)
{
}

or:

public static void ProcessBlobIntoBlob1([BlobTrigger("container/{name}")] string blob, [Blob("container/out") CloudBlockBlob output)
{
 // output may or may not be created depending on the code inside this function.
}

public static void ProcessBlobIntoBlob2([BlobTrigger("container/{name}")] string blob, [Blob("container/out", FileAccess.Write) Stream output)
{
 // output may or may not be created depending on the code inside this function.
 // if it doesn't write to output; it won't be created.
}

We could make the first an indexing error, but what about the second? I doubt we can make the third an indexing error.

If BlobTrigger ever keeps firing, that's quite surprising to the user, as well as potentially quite expensive.

Some examples:
Bind blob that doesn't exist to text writer. Should it create a new block blob (only on success)? Or give you a null text writer? Or give a binding failure?
What about binding to CloudBlockBlob? Give you the blob reference without creating it?
What about binding table to CloudTable? Ensure the table exists?
Binding table to IQueryable: just return empty?
When should blob binding create the container vs. the blob itself. What about page vs. block blob mismatch (say, for TextWriter).

We should be able to disable logging by setting the AzureJobsDashboard ConnectionString to be empty. This can be done in config or Antares portal or environment variables. We used to have it in alpha2 but it got changed in beta1.

If I set the ConnectionString in config to be empty then I get the following exception
An unhandled exception of type 'System.InvalidOperationException' occurred in Microsoft.Azure.Jobs.Host.dll

Additional information: Failed to validate Microsoft Azure Jobs dashboard connection string: Microsoft Azure Storage account connection string is missing or empty.

The Microsoft Azure Jobs connection string is specified by setting a connection string named 'AzureJobsDashboard' in the connectionStrings section of the .config file, or with an environment variable named 'AzureJobsDashboard', or through JobHostConfiguration.

If I set the ConnectionString in config to be empty in code then I get the following exception
Now I am forced to set the AzureJobsStorage ConnectionString in code since I need to set the dashboardConnectionString as well. This is bad because in this case I cannot set the ConnectionString through the portal

            JobHostConfiguration config = new JobHostConfiguration() {DashboardConnectionString="" };
            JobHost host = new JobHost();
            host.RunAndBlock();

An unhandled exception of type 'System.InvalidOperationException' occurred in Microsoft.Azure.Jobs.Host.dll

Additional information: Failed to validate Microsoft Azure Jobs dashboard connection string: Microsoft Azure Storage account connection string is missing or empty.

The Microsoft Azure Jobs connection string is specified by setting a connection string named 'AzureJobsDashboard' in the connectionStrings section of the .config file, or with an environment variable named 'AzureJobsDashboard', or through JobHostConfiguration.

Proposed solution
*I should be able to disable logging by setting AzureJobsDashboard connectionstring "empty" or "none" in config. You cannot use an empty string since in the Antares portal you cannot set the value of the ConnectionString to be empty

This is part of the work of adding knobs for Queues
How fast should the host poll for dashboard work (run from dashboard, invoke)? Today, it's 2 seconds to 1 minute.
When should the host treat a queue message as poison? Today it's 5.
How fast should other host queues poll for work? Today it's 2 seconds to 10 minutes.

When the dashboard upgrades, it should remove old data (that used to be part of the protocol but is no longer needed) from the host archive.

This is a customer reported issue

public static void DoWork(
            [BlobTrigger(@"in-container/{name}")] Stream input,    
            [Blob(@"out-container/{name}")] Stream output)
{
    input.CopyTo(output);
}

Notice that output is actually a readable stream. If the blob to which that parameter is binding doesn't exist, you get a "404" back which confuses people. We should either bind to an empty stream or throw an exception that explicitly says that the blob doesn't exist.

Exception:

System.InvalidOperationException: Exception binding parameter 'output' ---> Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (404) Not Found. ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
   at System.Net.HttpWebRequest.GetResponse()
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext)
   --- End of inner exception stack trace ---
   at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext)
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.FetchAttributes(AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext)
   at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.OpenRead(AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext)
   at Microsoft.Azure.Jobs.Host.Blobs.StreamArgumentBindingProvider.ReadStreamArgumentBinding.Bind(ICloudBlob blob, ArgumentBindingContext context)
   at Microsoft.Azure.Jobs.Host.Blobs.Bindings.BlobBinding.Bind(ICloudBlob value, ArgumentBindingContext context)
   at Microsoft.Azure.Jobs.Host.Blobs.Bindings.BlobBinding.Bind(BindingContext context)
   at Microsoft.Azure.Jobs.Host.Runners.TriggerParametersProvider`1.Bind()
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Jobs.Host.Runners.DelayedException.Throw()
   at Microsoft.Azure.Jobs.Host.Runners.WebSitesExecuteFunction.ExecuteWithSelfWatch(MethodInfo method, ParameterInfo[] parameterInfos, IReadOnlyDictionary`2 parameters, TextWriter consoleOutput)
   at Microsoft.Azure.Jobs.Host.Runners.WebSitesExecuteFunction.ExecuteWithOutputLogs(FunctionInvokeRequest request, IReadOnlyDictionary`2 parameters, TextWriter consoleOutput, CloudBlobDescriptor parameterLogger, IDictionary`2 parameterLogCollector)
   at Microsoft.Azure.Jobs.Host.Runners.WebSitesExecuteFunction.ExecuteWithLogMessage(FunctionInvokeRequest request, RuntimeBindingProviderCon

[Table("name")] ICollector<T>

Specifically, we'll need to define two new interfaces:

public interface ICollector<in T>
{
    void Add(T item);
}

public interface IAsyncCollector<in T>
{
    Task AddAsync(T item, CancellationToken cancellationToken);
}

If T implements ITableEntity, just use the PartitionKey and RowKey properties (and use the ITableEntity built-in serialization).
If T doesn't implement ITableEntity, validate and indexing time that it has string PartitionKey and string RowKey and use JSON serialization (like for PocoTableEntityBinding).

Should do InsertOrReplace.

Should capture entity at Add time (ignore any changes to the entity after that call).

Should batch 100 per partition key.

Create a continuous non-sdk job.

Issue reported here.

It seems that property binding for non strings fails. The simplified code below will produce the same exception as the one reported in the SO post.

    public class PackageRequestMessage
    {
        public Guid RequestId { get; set; }
    }


    class Program
    {
        static void Main(string[] args)
        {
            JobHost host = new JobHost();
            host.RunAndBlock();
        }

        public static void ProcessPackageRequestMessage(
            [QueueTrigger("myqueue")] PackageRequestMessage message,
            Guid requestId)
        {
        }
    }

There's an edge case where both could happen concurrently, and failure in this case causes update message visibility to propogate an exception.

We should add guidelines for how to contribute / links to page with cla etc.

I have the following function definition. The function should be triggered when the message moves to a DeadLetter Queue but in my case I get an exception. This happens when I run my WebJob

        public static void ProcessSBQueueMessage(
        [ServiceBusTrigger("inputqueue")] BrokeredMessage inputText)
        {
            inputText.DeadLetter("Webjobs", "webjobsdescription");
            Console.WriteLine(inputText);
        }

        public static void ProcessSBDeadLetterQueueMessage(
        [ServiceBusTrigger("inputqueue/$DeadLetterQueue")] BrokeredMessage inputText)
        {
            Console.WriteLine(inputText);
        }

The remote server returned an error: (400) Bad Request. The specified HTTP verb (GET) is not valid..TrackingId:e97d6c70-9507-426b-a53d-c686b9a21fc8_G3,TimeStamp:7/11/2014 4:38:43 PM

When we push out a new version of the dashboard, re-populate from the host archive if the dashboard's internal schema has changed.

Instead of calling SetMetadata after writing the blob contents, we can just populate the metadata initially (before starting to write the blob contents). This should save a network hop for most blob writes.

For example, a request for a function that doesn't exist should show "Function not found." Today, we just never update and are stuck with "Loading..."

Basically we need to go over every AJAX call and ensure it has a sensible error handler/UI.

Specifically, the following method fails to handle this case:
PersistentQueueReader.TryMakeItemInvisible

I tested this and we give you access to the name and extension. For eg. in the following function if the file is foo.txt then name is foo and extension is txt. However if you have a file called foo.bar.txt then name is foo and extension is bar.txt which is incorrect and this is what we need to fix

Overall, use the same rule as ASP.NET routing (match the longest possible string).

        public static void BlobToQueueForCleanup(
            [BlobTrigger("input/{name}.{extension}")] TextReader input, string name,string extension,
            [Blob("output/{name}_old.{extension}",FileAccess.Write)] TextWriter writer,
            [Queue("cleanupqueue")] out DeleteBlob output)
        {
            output = new DeleteBlob(); 
            output.Container = "input"; output.Name = name;
            writer.Write(input.ReadToEnd());
        }

Per discussion with the ServiceBus team, for best performance and resource usage, we should close message senders, receivers, and factories when the host stops.

We should be able to run locally on the emulator. Today the emulator has some limitations as compared to Azure Storage service. Maybe we throw exceptions for the cases where we don't work.
The getting started experience is hard without this support

Here is the SO question. This model works for Azure Queues but not for Service Bus

Today the behavior is that if you set output parameter to null for Azure Queues we do not throw an exception but we do throw for Service Bus.

Proposed solution:
We should not throw exception for Service Bus

http://stackoverflow.com/questions/25186151/is-there-a-way-to-make-azure-webjobs-servicebus-out-parameter-optional/25207926#25207926

public static void TriggerOnQueue(
     [ServiceBusTrigger(QueueName)] BrokeredMessage receivedMessage,
     [ServiceBus(QueueResponseName)] out BrokeredMessage responseMessage)
{
      responseMessage = null; // Don't enqueue anything to QueueResponseName.
}

Per discussion with the ServiceBus team, we'd have less network overhead if we re-used message sender and receiver objects for all instances of the same function definition.

Note that for senders, we'll need to disable batching to preserve correctness (independence of function instances during concurrent execution). Setting the batch window to 0 should do that.

Following is a customer reported issue. This is a generic extension method in his app. We throw an exception while Indexing the functions.
Cannot create an instance of Microsoft.Azure.Jobs.Host.Bindings.Invoke.ClassInvokeBinding1[System.Collections.Generic.IEnumerable1[T]] because Type.ContainsGenericParameters is true.

From talking to David we should update the Indexing logic to only look for functions that use the SDK and throw indexing exceptions for the ones that do not follow the rules.

    public static class Extensions
    {
        public static IEnumerable<IEnumerable<T>> Chunk<T>(this IEnumerable<T> source,
            int chunksize)
        {
            while (source.Any())
            {
                yield return source.Take(chunksize);
                source = source.Skip(chunksize);
            }
        }
    }

Repro:
Run a function.

Actual:
You just see Loading...

Expected:
You see the function saying "Queued"

So that the user knows how to upgrade.

• Explain to how to setup the credentials.

Provide the following knobs on JobHostConfiguration.Queues:
MaxDequeueCount
MaxPollingInterval

I believe the user doesn't want a nested inner exception. We should consider not using MethodInfo.Invoke (which is also not idea from a performance standpoint).

http://social.msdn.microsoft.com/Forums/windowsazure/en-US/6ad01995-3ffc-4801-9fab-8a23dc1c56ee/webjob-fails-with-null-reference-exception-in-the-sdk-code?forum=windowsazurewebsitespreview

Throw an indexing error for a job method that tries to do anything else.
In the future, returning false from a QueueTrigger method may mean "I didn't process the message."

Alex noticed this bug running integration tests on his machine, and after investigating we realized the same scenario could occur in Antares shutdown. In particular, we need to ensure concurrent calls to host.StopAsync work correctly/idempotently.

Use random factor in the Azure Queue exponential polling algorithm to prevent possible network i/o congestion.

A few UT for the controllers would be nice. It is also worth investigating what it takes to write UI tests against SPA.

Seach blob should work in this case. Show results from all Connections

When we know a queue message is written from another function (most bindings to [Queue]), the trigger function shouldn't wait the normal [QueueTrigger] interval (default is up to 10 minutes) to poll that queue again.

Once the host sends a function completed message over the protocol, there's no need to keep any function started message. Having the host do this work will reduce the volume of messages that get queued up for the dashboard to process.

Here's an idea on how it might work from an implementation standpoint:

1. IPersistentQueueWriter.Enqueue returns the message ID instead of void. Add a Delete(message id) method.
2. Similar changes to IFunctionInstanceLogger.
3. FunctionExecutor retains the message id from LogFunctionStarted, after calling LogFunctionCompleted, uses that to delete the FunctionStartedMessage.

Make sure we only remove the credentials without strongly deserializing to avoid degrade future released data.

I have got a few customer requests that they want to delete a blob when a function is triggered on a blob. They can bind to CloudBlob and do this but they loose all the benefits of using our SDK - bindings etc and they will have to drop down to Storage SDK.
I can see value in this and following are some suggestions

• Have an overload of BlobTrigger to delete the blob

 public static void BlobToQueueForCleanup(
            [BlobTrigger("input/{name}", DeleteAfterProcessing=true|False)] TextReader input) {
}

• Have a configuration for Blobs which sets this flag to true. This configuration class could also have the knob for tuning the scanning logic for Blobs. This will be similar to what we do for Queues.

I should be able to do the following. Note that we allow binding to name but not container. This will be super useful for easily accessing Blobs

        class DeleteBlob
        {
            public string Container { get; set; }
            public string Name { get; set; }
         }

        public static void BlobToQueueForCleanup(
            [BlobTrigger("input/{name}")] TextReader input, string name,
            [Queue("cleanupqueue")] out DeleteBlob output)
        {
            output = new DeleteBlob(); 
            output.Container = "input"; output.Name = name;
        }

        public static void DeleteBlob(
        [QueueTrigger("cleanupqueue")] DeleteBlob input,
         string container, string name,
        [Blob("{container}/{name}")] CloudPageBlob blob)
        {
           blob.Delete();
        }