azure / azure-webjobs-sdk Goto Github PK
View Code? Open in Web Editor NEWAzure WebJobs SDK
License: MIT License
Azure WebJobs SDK
License: MIT License
When the dashboard upgrades, it should remove old data (that used to be part of the protocol but is no longer needed) from the host archive.
We should be able to disable logging by setting the AzureJobsDashboard ConnectionString to be empty. This can be done in config or Antares portal or environment variables. We used to have it in alpha2 but it got changed in beta1.
If I set the ConnectionString in config to be empty then I get the following exception
An unhandled exception of type 'System.InvalidOperationException' occurred in Microsoft.Azure.Jobs.Host.dll
Additional information: Failed to validate Microsoft Azure Jobs dashboard connection string: Microsoft Azure Storage account connection string is missing or empty.
The Microsoft Azure Jobs connection string is specified by setting a connection string named 'AzureJobsDashboard' in the connectionStrings section of the .config file, or with an environment variable named 'AzureJobsDashboard', or through JobHostConfiguration.
If I set the ConnectionString in config to be empty in code then I get the following exception
Now I am forced to set the AzureJobsStorage ConnectionString in code since I need to set the dashboardConnectionString as well. This is bad because in this case I cannot set the ConnectionString through the portal
JobHostConfiguration config = new JobHostConfiguration() {DashboardConnectionString="" };
JobHost host = new JobHost();
host.RunAndBlock();
An unhandled exception of type 'System.InvalidOperationException' occurred in Microsoft.Azure.Jobs.Host.dll
Additional information: Failed to validate Microsoft Azure Jobs dashboard connection string: Microsoft Azure Storage account connection string is missing or empty.
The Microsoft Azure Jobs connection string is specified by setting a connection string named 'AzureJobsDashboard' in the connectionStrings section of the .config file, or with an environment variable named 'AzureJobsDashboard', or through JobHostConfiguration.
Proposed solution
*I should be able to disable logging by setting AzureJobsDashboard connectionstring "empty" or "none" in config. You cannot use an empty string since in the Antares portal you cannot set the value of the ConnectionString to be empty
Following is a customer reported issue. This is a generic extension method in his app. We throw an exception while Indexing the functions.
Cannot create an instance of Microsoft.Azure.Jobs.Host.Bindings.Invoke.ClassInvokeBinding1[System.Collections.Generic.IEnumerable
1[T]] because Type.ContainsGenericParameters is true.
From talking to David we should update the Indexing logic to only look for functions that use the SDK and throw indexing exceptions for the ones that do not follow the rules.
public static class Extensions
{
public static IEnumerable<IEnumerable<T>> Chunk<T>(this IEnumerable<T> source,
int chunksize)
{
while (source.Any())
{
yield return source.Take(chunksize);
source = source.Skip(chunksize);
}
}
}
When we push out a new version of the dashboard, re-populate from the host archive if the dashboard's internal schema has changed.
Some examples:
Bind blob that doesn't exist to text writer. Should it create a new block blob (only on success)? Or give you a null text writer? Or give a binding failure?
What about binding to CloudBlockBlob? Give you the blob reference without creating it?
What about binding table to CloudTable? Ensure the table exists?
Binding table to IQueryable: just return empty?
When should blob binding create the container vs. the blob itself. What about page vs. block blob mismatch (say, for TextWriter).
I tested this and we give you access to the name and extension. For eg. in the following function if the file is foo.txt then name is foo and extension is txt. However if you have a file called foo.bar.txt then name is foo and extension is bar.txt which is incorrect and this is what we need to fix
Overall, use the same rule as ASP.NET routing (match the longest possible string).
public static void BlobToQueueForCleanup(
[BlobTrigger("input/{name}.{extension}")] TextReader input, string name,string extension,
[Blob("output/{name}_old.{extension}",FileAccess.Write)] TextWriter writer,
[Queue("cleanupqueue")] out DeleteBlob output)
{
output = new DeleteBlob();
output.Container = "input"; output.Name = name;
writer.Write(input.ReadToEnd());
}
Here is the SO question. This model works for Azure Queues but not for Service Bus
Today the behavior is that if you set output parameter to null for Azure Queues we do not throw an exception but we do throw for Service Bus.
Proposed solution:
We should not throw exception for Service Bus
public static void TriggerOnQueue(
[ServiceBusTrigger(QueueName)] BrokeredMessage receivedMessage,
[ServiceBus(QueueResponseName)] out BrokeredMessage responseMessage)
{
responseMessage = null; // Don't enqueue anything to QueueResponseName.
}
[Table("name", "PK")] IQueryable<T>
Alex noticed this bug running integration tests on his machine, and after investigating we realized the same scenario could occur in Antares shutdown. In particular, we need to ensure concurrent calls to host.StopAsync work correctly/idempotently.
Provide the following knobs on JobHostConfiguration.Queues:
MaxDequeueCount
MaxPollingInterval
So that the user knows how to upgrade.
I would be really cool to have support for IEnumerable<> binding on blob.
The scenario where we need this is the PhluffyShffy sample: there is a container with the images uploaded by the user. When a queue message is picked up by the job, it also needs get all the blobs associated to that message - the association is done by the name convention.
It would be something like:
[Blob("container/{fileName}")]IEnumerable<CloudBlockBlob> files
IEnumerable works well for CloudBlockBlob because you can get the blob name. For types which don't have this information, IDictionary<string, T> might do the trick
[Blob("container/{fileName}")]IDictionary<string, Stream> files
This is part of the work of adding knobs for Queues
How fast should the host poll for dashboard work (run from dashboard, invoke)? Today, it's 2 seconds to 1 minute.
When should the host treat a queue message as poison? Today it's 5.
How fast should other host queues poll for work? Today it's 2 seconds to 10 minutes.
This is a customer reported issue
public static void DoWork(
[BlobTrigger(@"in-container/{name}")] Stream input,
[Blob(@"out-container/{name}")] Stream output)
{
input.CopyTo(output);
}
Notice that output is actually a readable stream. If the blob to which that parameter is binding doesn't exist, you get a "404" back which confuses people. We should either bind to an empty stream or throw an exception that explicitly says that the blob doesn't exist.
Exception:
System.InvalidOperationException: Exception binding parameter 'output' ---> Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (404) Not Found. ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext)
--- End of inner exception stack trace ---
at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext)
at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.FetchAttributes(AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext)
at Microsoft.WindowsAzure.Storage.Blob.CloudBlockBlob.OpenRead(AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext)
at Microsoft.Azure.Jobs.Host.Blobs.StreamArgumentBindingProvider.ReadStreamArgumentBinding.Bind(ICloudBlob blob, ArgumentBindingContext context)
at Microsoft.Azure.Jobs.Host.Blobs.Bindings.BlobBinding.Bind(ICloudBlob value, ArgumentBindingContext context)
at Microsoft.Azure.Jobs.Host.Blobs.Bindings.BlobBinding.Bind(BindingContext context)
at Microsoft.Azure.Jobs.Host.Runners.TriggerParametersProvider`1.Bind()
--- End of inner exception stack trace ---
at Microsoft.Azure.Jobs.Host.Runners.DelayedException.Throw()
at Microsoft.Azure.Jobs.Host.Runners.WebSitesExecuteFunction.ExecuteWithSelfWatch(MethodInfo method, ParameterInfo[] parameterInfos, IReadOnlyDictionary`2 parameters, TextWriter consoleOutput)
at Microsoft.Azure.Jobs.Host.Runners.WebSitesExecuteFunction.ExecuteWithOutputLogs(FunctionInvokeRequest request, IReadOnlyDictionary`2 parameters, TextWriter consoleOutput, CloudBlobDescriptor parameterLogger, IDictionary`2 parameterLogCollector)
at Microsoft.Azure.Jobs.Host.Runners.WebSitesExecuteFunction.ExecuteWithLogMessage(FunctionInvokeRequest request, RuntimeBindingProviderCon
[Table("name")] IEnumerable<T>
Specifically, the following method fails to handle this case:
PersistentQueueReader.TryMakeItemInvisible
Magnus ran into a problem where a BlobTrigger + Blob (output) function kept looping forever. It's likely due to a binding failure. Other case to check:
Blob file name with invalid characters
History:
Magnus:
Oh I forgot about this demo ‘fail’! Thank you for reminding me.
I guess you’re right it has to be something like that.
I have attached a zip with the demo code which should compile anywhere. I removed the account and key from the connection strings. As for demo data it’s easy to see from the screen shot below. What I did was allow the audience to break the app by sending me an azure blob file name with invalid characters. Also I use the same data for the rowkey in the table and there are potential fails there too!
If someone else would like to test this or maybe even someone in the team (?) I would be delighted! I was surprised that some of these outputs went into perpetual loops! My resulting azure table had lots of rows and the execution just continued.
Let me know if you find anything interesting!
David:
I’m guessing it’s the equivalent of a poison message for BlobInput. Perhaps there’s a non-text blob and binding to TextReader fails. And because we never write the BlobOutput, we keep triggering the function every time we see that BlobInput in our scans.
Per discussion with the ServiceBus team, for best performance and resource usage, we should close message senders, receivers, and factories when the host stops.
When I start a Host using the SDK we print out the "Executing..." when the function gets triggered. We should output something like "Finished Executing" which would indicate that the function has finished executing.
We should be able to run locally on the emulator. Today the emulator has some limitations as compared to Azure Storage service. Maybe we throw exceptions for the cases where we don't work.
The getting started experience is hard without this support
BlobTrigger keeps firing even after it's been processed. Today, we don't have any way to avoid continuous automatic firing in every scenario. For example, consider:
public static void ProcessBlobIntoQueue([BlobTrigger("container/{name}")] string blob, [Queue("myqueue")] out string queue)
{
}
or:
public static void ProcessBlobIntoBlob1([BlobTrigger("container/{name}")] string blob, [Blob("container/out") CloudBlockBlob output)
{
// output may or may not be created depending on the code inside this function.
}
public static void ProcessBlobIntoBlob2([BlobTrigger("container/{name}")] string blob, [Blob("container/out", FileAccess.Write) Stream output)
{
// output may or may not be created depending on the code inside this function.
// if it doesn't write to output; it won't be created.
}
We could make the first an indexing error, but what about the second? I doubt we can make the third an indexing error.
If BlobTrigger ever keeps firing, that's quite surprising to the user, as well as potentially quite expensive.
[Table("name")] ICollection<T>
--- AzureJobsDashboard and AzureJobsStorage are different connectionString
--- Have convention as AzureJobs* as name of ConnectionString which user has to be specified
--- Dashboard works with multiple storage accounts
--- Dashboard is Dict<AccountName|ConnectionString>
Throw an indexing error for a job method that tries to do anything else.
In the future, returning false from a QueueTrigger method may mean "I didn't process the message."
[Queue("name")] ICollector<T>
Check whether both NamespaceManager and MessagingFactory will work during indexing (for example, validate credentials).
Instead of calling SetMetadata after writing the blob contents, we can just populate the metadata initially (before starting to write the blob contents). This should save a network hop for most blob writes.
Seach blob should work in this case. Show results from all Connections
I believe the user doesn't want a nested inner exception. We should consider not using MethodInfo.Invoke (which is also not idea from a performance standpoint).
I have the following function definition. The function should be triggered when the message moves to a DeadLetter Queue but in my case I get an exception. This happens when I run my WebJob
public static void ProcessSBQueueMessage(
[ServiceBusTrigger("inputqueue")] BrokeredMessage inputText)
{
inputText.DeadLetter("Webjobs", "webjobsdescription");
Console.WriteLine(inputText);
}
public static void ProcessSBDeadLetterQueueMessage(
[ServiceBusTrigger("inputqueue/$DeadLetterQueue")] BrokeredMessage inputText)
{
Console.WriteLine(inputText);
}
The remote server returned an error: (400) Bad Request. The specified HTTP verb (GET) is not valid..TrackingId:e97d6c70-9507-426b-a53d-c686b9a21fc8_G3,TimeStamp:7/11/2014 4:38:43 PM
Repro:
Run a function.
Actual:
You just see Loading...
Expected:
You see the function saying "Queued"
When we know a queue message is written from another function (most bindings to [Queue]), the trigger function shouldn't wait the normal [QueueTrigger] interval (default is up to 10 minutes) to poll that queue again.
[Table("name")] ICollector<T>
Specifically, we'll need to define two new interfaces:
public interface ICollector<in T>
{
void Add(T item);
}
public interface IAsyncCollector<in T>
{
Task AddAsync(T item, CancellationToken cancellationToken);
}
If T implements ITableEntity, just use the PartitionKey and RowKey properties (and use the ITableEntity built-in serialization).
If T doesn't implement ITableEntity, validate and indexing time that it has string PartitionKey and string RowKey and use JSON serialization (like for PocoTableEntityBinding).
Should do InsertOrReplace.
Should capture entity at Add time (ignore any changes to the entity after that call).
Should batch 100 per partition key.
Once the host sends a function completed message over the protocol, there's no need to keep any function started message. Having the host do this work will reduce the volume of messages that get queued up for the dashboard to process.
Here's an idea on how it might work from an implementation standpoint:
A few UT for the controllers would be nice. It is also worth investigating what it takes to write UI tests against SPA.
I should be able to do the following. Note that we allow binding to name but not container. This will be super useful for easily accessing Blobs
class DeleteBlob
{
public string Container { get; set; }
public string Name { get; set; }
}
public static void BlobToQueueForCleanup(
[BlobTrigger("input/{name}")] TextReader input, string name,
[Queue("cleanupqueue")] out DeleteBlob output)
{
output = new DeleteBlob();
output.Container = "input"; output.Name = name;
}
public static void DeleteBlob(
[QueueTrigger("cleanupqueue")] DeleteBlob input,
string container, string name,
[Blob("{container}/{name}")] CloudPageBlob blob)
{
blob.Delete();
}
For example, a request for a function that doesn't exist should show "Function not found." Today, we just never update and are stuck with "Loading..."
Basically we need to go over every AJAX call and ensure it has a sensible error handler/UI.
Issue reported here.
It seems that property binding for non strings fails. The simplified code below will produce the same exception as the one reported in the SO post.
public class PackageRequestMessage
{
public Guid RequestId { get; set; }
}
class Program
{
static void Main(string[] args)
{
JobHost host = new JobHost();
host.RunAndBlock();
}
public static void ProcessPackageRequestMessage(
[QueueTrigger("myqueue")] PackageRequestMessage message,
Guid requestId)
{
}
}
We should add guidelines for how to contribute / links to page with cla etc.
Make sure we only remove the credentials without strongly deserializing to avoid degrade future released data.
I have got a few customer requests that they want to delete a blob when a function is triggered on a blob. They can bind to CloudBlob and do this but they loose all the benefits of using our SDK - bindings etc and they will have to drop down to Storage SDK.
I can see value in this and following are some suggestions
public static void BlobToQueueForCleanup(
[BlobTrigger("input/{name}", DeleteAfterProcessing=true|False)] TextReader input) {
}
[Table("name", "PK")] IEnumerable<T>
There's an edge case where both could happen concurrently, and failure in this case causes update message visibility to propogate an exception.
We need to make sure we're handling the following two conditions correctly (from Brent Stineman):
Per discussion with the ServiceBus team, we'd have less network overhead if we re-used message sender and receiver objects for all instances of the same function definition.
Note that for senders, we'll need to disable batching to preserve correctness (independence of function instances during concurrent execution). Setting the batch window to 0 should do that.
Use random factor in the Azure Queue exponential polling algorithm to prevent possible network i/o congestion.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.