License: Other

C# 100.00%

documenttranslationservice's People

Contributors

Stargazers

Watchers

Forkers

davidmarquezf martinskeem seanpm2001

documenttranslationservice's Issues

Doesn't dispose opened files

Read file streams are not disposed. This can cause issues, where if the program hasn't been closed you cannot delete the files and others.

DocumentTranslationService/DocumentTranslationBusiness.cs

Lines 205 to 227 in 001bcd9

 BlobClient blobClient = new(TranslationService.StorageConnectionString, TranslationService.ContainerClientSource.Name, Normalize(filename)); 

 try 

 { 

 uploadTasks.Add(blobClient.UploadAsync(fileStream, true)); 

 count++; 

 sizeInBytes += new FileInfo(fileStream.Name).Length; 

 semaphore.Release(); 

 } 

 catch (Exception ex) when (ex is AggregateException or Azure.RequestFailedException) 

 { 

 logger.WriteLine($"Uploading file {fileStream.Name} failed with {ex.Message}"); 

 OnFileReadWriteError?.Invoke(this, ex.Message); 

 if (!Nodelete) await DeleteContainersAsync(tolanguages); 

 return; 

 } 

 logger.WriteLine($"File {filename} upload started."); 

 } 

 } 

 Debug.WriteLine("Awaiting document upload task completion."); 

 try 

 { 

 await Task.WhenAll(uploadTasks); 

 }

Glossary files with same names fails to upload

If we send multiple glossary files in different paths but with the same name the app crashes. This is due to how the file names for the blob containers are generated:

DocumentTranslationService/DocumentTranslationBusiness.cs

Lines 508 to 511 in c6a0aff

 public static string Normalize(string filename) 

 { 

 return Path.GetFileName(filename); 

 }

Since the user is not going to see the result of the glossary file maybe it would be better to generate a GUID:

Guid.NewGuid().ToString()

Maybe even maintain the name in case someone wanted to see the files in the containers:

Guid.NewGuid().ToString() + Path.GetFileName(filename);

Add a way to ignore PostBuild Event

Right now the postbuild event does a process to sign the dll. This is based in windows which means that we cannot build it in linux systems.

It would be nice to add a property in order to not trigger this event.

Translate doesn't start in some instances

I've seen that the service, in some instances doesn't even start translating:

Submitted: 223 documents, 417188 bytes.
09:47:48.0863709	Status: NotStarted	In progress: 0	Success: 0	Fail: 0
09:47:50.7703304	Status: NotStarted	In progress: 0	Success: 0	Fail: 0

From this, it doesn't print anything else

The heartbeat still works though:

Translation Request submitted. Status: NotStarted
4.5524552 START - Translation service request with SAS token. StatusID: *************
5.7596823 Http status: 200 OK
5.7618146 Service status: 23/06/2023 10:06:49 +00:00 NotStarted
6.979743 Http status: 200 OK
6.9816031 Service status: 23/06/2023 10:06:49 +00:00 NotStarted
8.1819809 Http status: 200 OK
8.183962 Service status: 23/06/2023 10:06:49 +00:00 NotStarted
9.385535 Http status: 200 OK
9.3875677 Service status: 23/06/2023 10:06:49 +00:00 NotStarted
10.5896654 Http status: 200 OK
10.5915891 Service status: 23/06/2023 10:06:49 +00:00 NotStarted
11.8077684 Http status: 200 OK
11.809774 Service status: 23/06/2023 10:06:49 +00:00 NotStarted
13.0188996 Http status: 200 OK
13.0210184 Service status: 23/06/2023 10:06:49 +00:00 NotStarted
14.2335563 Http status: 200 OK
14.2354616 Service status: 23/06/2023 10:06:49 +00:00 NotStarted
15.4479572 Http status: 200 OK
15.4498513 Service status: 23/06/2023 10:06:49 +00:00 NotStarted
16.6604599 Http status: 200 OK
16.6623357 Service status: 23/06/2023 10:06:49 +00:00 NotStarted
The thread 0x67d4 has exited with code 0 (0x0).
The thread 0xa1d0 has exited with code 0 (0x0).
The thread 0x77f4 has exited with code 0 (0x0).
The thread 0xa694 has exited with code 0 (0x0).
17.8749834 Http status: 200 OK
The thread 0xaac8 has exited with code 0 (0x0).
The thread 0x38d0 has exited with code 0 (0x0).
17.879706 Service status: 23/06/2023 10:06:49 +00:00 NotStarted
The thread 0x5ea4 has exited with code 0 (0x0).
The thread 0x9af8 has exited with code 0 (0x0).
The thread 0x98d8 has exited with code 0 (0x0).
The thread 0x4458 has exited with code 0 (0x0).
The thread 0x5f20 has exited with code 0 (0x0).
The thread 0x376c has exited with code 0 (0x0).
The thread 0x3374 has exited with code 0 (0x0).
The thread 0x45f4 has exited with code 0 (0x0).
The thread 0x68dc has exited with code 0 (0x0).
The thread 0x2370 has exited with code 0 (0x0).
The thread 0x5d0c has exited with code 0 (0x0).
The thread 0x1998 has exited with code 0 (0x0).
The thread 0x38bc has exited with code 0 (0x0).
The thread 0x5488 has exited with code 0 (0x0).
The thread 0x1e1c has exited with code 0 (0x0).
The thread 0x56d4 has exited with code 0 (0x0).
19.0858959 Http status: 200 OK
19.0873632 Service status: 23/06/2023 10:06:49 +00:00 NotStarted
The thread 0x813c has exited with code 0 (0x0).
20.3029608 Http status: 200 OK
....

This happens consistently when I'm trying to translate a high number of documents (200 for instance), but for lower amounts with the exact same settings (storage strings, token, glossaries, etc.) it works well.

If I check in Azure, the three blob containers have been created (glossary, target and source) and have all the files there. So it seams that for some reason the translator just doesn't start. Any ideas why this might happen??

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

	BlobClient blobClient = new(TranslationService.StorageConnectionString, TranslationService.ContainerClientSource.Name, Normalize(filename));
	try
	{
	uploadTasks.Add(blobClient.UploadAsync(fileStream, true));
	count++;
	sizeInBytes += new FileInfo(fileStream.Name).Length;
	semaphore.Release();
	}
	catch (Exception ex) when (ex is AggregateException or Azure.RequestFailedException)
	{
	logger.WriteLine($"Uploading file {fileStream.Name} failed with {ex.Message}");
	OnFileReadWriteError?.Invoke(this, ex.Message);
	if (!Nodelete) await DeleteContainersAsync(tolanguages);
	return;
	}
	logger.WriteLine($"File {filename} upload started.");
	}
	}
	Debug.WriteLine("Awaiting document upload task completion.");
	try
	{
	await Task.WhenAll(uploadTasks);
	}

	public static string Normalize(string filename)
	{
	return Path.GetFileName(filename);
	}

microsofttranslator / documenttranslationservice Goto Github PK