microsoft / azuresearch_jfk_files Goto Github PK

This repo contains the sample code of the Azure Search and Cognitive Services used to provide insights and analysis around the JFK Files.

License: MIT License

C# 21.94% JavaScript 3.06% HTML 6.92% TypeScript 59.46% Batchfile 0.77% SCSS 7.86%

azuresearch_jfk_files's Introduction

The JFK Files

Explore the JFK Assassination files using the AI Enrichment features of Azure Cognitive Search. You can watch the demo in action in a short online video or explore the JFK files yourself with our online demo.

Cognitive Search - An AI-first approach to content understanding

This project demonstrates how you can use both the built-in and custom AI in Cognitive Search. Cognitive Search ingests your data from almost any datasource and enriches it using a set of cognitive skills that extracts knowledge and then lets you explore the data using Search.

JFK Files Architecture

The JFK files example leverages the built-in Cognitive Skills inside of Cognitive Search and combines it with custom skills using extensibility. The architecture below showcases how the new Cognitive Search capabilities of Azure enable you to easily create structure from almost any datasource.

Note: This diagram of visuals are inspired by the CIA's JFK document management system in 1997 included in the JFK files.

This project includes the following capabilities for you to build your own version of the JFK files.

We have provided a subset of the JFK PDF documents and images that have been uploaded to the cloud into Azure Blob Storage.
An Azure Search service is used to index the content and power the UX experience. We use the new Cognitive Search capabilities to apply pre-built cognitive skills to the content, and we also use the extensibility mechanism to add custom skills using Azure Functions.
1. Uses the Cognitive Services Vision API to extract text information from the image via OCR, and image captioning,
2. Applies Named Entity Recognitition to extract named entities from the documents,
3. Annotates text using a custom CIA Cryptonyms skill,
4. Generates HOCR content based on results.
A standalone website to search the index and explore the documents

Limitations and Considerations

This is a demo to showcase a Cognitive Search use case. It is not intended to be a framework or scalable architecture for all scenarios, though it can give you an idea of what your scenario might end up looking like.
The OCR technology is not perfect; results will vary greatly by scan and image quality.
Most file formats and datasources are supported, however some scanned and native PDF formats may not be parsed correctly.
IMPORTANT: The JFK Files sample creates a public website and a publicly readable storage container for any extracted images. As-is, it is not suitable for using with non-public data.

Setting up your own JFK files library

These instructions will help you have your own version of the JFK files demo running in Azure in about 20 minutes, with most of that time being provisioning/deployment time.

Prerequisites

An Azure Subscription you can access and deploy resources to.
1. Note that this demo requires writing to an Azure Storage Account, which you will be billed monthly for the storage written to, and by default provisions a Basic Azure Search service which is billed hourly. For an estimation of cost, reference the Azure Pricing Calculator. In addition, you will be charged for the Cognitive Search part of the demo, which is transaction based. These charges used with the provided files should be about $15 USD. See here for details on what costs may look like if you choose to use Cognitive Search with your own dataset.
Visual Studio 2019 with Azure Developer Tools enabled. The free community edition will work fine.
Node.js must be installed on your computer.
npm version 6.x is required, newer versions are not supported yet. Can be installed with npm install -g npm@6.
Basic familiarity with using the Azure Portal and cloning and compiling code from github.

Deploy Required Resources

Click the below button to upload the provided ARM template to the Azure portal, which is written to automatically deploy and configure the following resources:
1. An Azure Search service, default set to Basic tier.
2. An Azure Blob Storage Account, default set to Standard LRS tier.
3. An Azure App Service plan, default set to Free F1 tier.
4. An Azure Web App Service, using the plan from # 3.
5. An Azure Function instance, using the storage account from # 2 and the plan from # 3. The Azure Function will be prepublished with the code provided in this repository as part of the template deployment.
6. A Cognitive Services account, of type CognitiveServices, that will be used for billing your Cognitive Search skills usage.
Be sure to select the appropriate subscription to deploy to, and create a new resource group for these resources. Only use alphanumeric lowercase characters for the Resource Prefix field.
Review the rest of the parameters as well as the terms and conditions, select the checkbox for "I agree to the terms and conditions stated above", and press "Purchase".
Monitor the status of the deployment by following the link that appears in the Azure portal notifications. It takes about 5 minutes for the resources to be fully provisioned and deployed.

Initialize the code

While you are waiting for the resources to finish provisioning, either git clone or download this repo, and open the provided solution file JfkWebApiSkills/JfkWebApiSkills.sln using Visual Studio 2019.
In the root of the JfkInitializer project, open the App.config file. You will be copying and pasting some secrets into this file momentarily.
Make sure that the JfkInitializer project is set as your default project if it isn't already.

Run the initializer

Once your resources are finished deploying (you should get a notification in the Azure portal), navigate to the Outputs section of the deployment.
Copy and paste each of the provided outputs from this interface into their corresponding value location in the App.config file from before.
Save your changes to the App.config file, and press the start button at the top of Visual Studio in order to run the initializer.
After a few seconds, the message "Website keys have been set. Please build the website and then return here and press any key to continue." will be output to the console app. At this point, open a separate cmd window and cd into the directory of where you cloned or downloaded the repo. Then run the following commands:

Note: you need npm@6, can be installed with npm install -g npm@6
```
cd frontend
npm install
npm run build:prod
```
After the commands finish running, return to the console app and press any key to continue.
Once the website is deployed, a URL will appear in the console. Copy and paste this URL into a browser to start interacting with what you have created!

Results

This project will create several things for you. It will deploy an Azure Function using the code provided in the JfkWebApiSkills project, as well as create an Azure Search data source, skillset, synonym map, index, and indexer. It will also create a blob storage container that the images will be reuploaded to, as well as deploys the JFK Files frontend to an Azure web app that you can immediately interact with. Feel free to play with the code to see how all of these things are accomplished so that you can start using Cognitive Search for your data enrichment scenario today.

Debugging

If you encounter issues running this demo, there are several ways you can debug what may be wrong.

In ./JfkWebApiSkills/JfkInitializer/Program.cs, you can set the "DebugMode" variable to true in order to see the output for REST calls that are made to create resources and why they may be failing. The Initializer will prompt you to do this if it suspects there is an issue creating anything it needs to create.
You can add the "enriched" field to your index to see detailed output of all skills that are run. Details on how to do this can be found here. You will be making this change in ./JfkWebApiSkills/JfkInitializer/SearchResources.cs.
If all else fails, feel free to post your problem as a Github issue, which we monitor and respond to when necessary.

Next Steps

The Azure Architecture Center has more content on this solution and others in this space.

azuresearch_jfk_files's People

Contributors

Stargazers

Watchers

Forkers

chrimc62 kylebunting dem108 awaemmanuel rsliang nikhilrj apollak jreynolds01 jagatfx worldox dansemedo idiaz01 bougiefever gildardorojas unixcrh harikrishnanvr joanteixi brauliodiez ashishrajsrivastava bolajiomo99 maquadros lemoncode nasdan evertonteotonio rjygraham stellaleigh larrycarter owlyone mbah360 jantielens akshata29 ivmarcos teo-ma ramitix smbolster mashhype tikyau lcarli kema221 dbbevan doneladams maheshgurav tamzi rozele mrcarter8 thedarkcode olumideolufy29 fpelaez jwiley84 pierricki3 juhisharma21 vipul02 jaerrett mgalilea jrthearchitect pcoric nebgor immrl thiagottss whereisyusuf v-deken taethni nextdynamic andrescabsi14 arv100kri charlesxxiv amotley skuyra matthew-trachtenberg noellelacharite ceteongvanness horekik ikivanc click2cloudrupal ajakupov computeworks jhandel vijayraavi vinnydyn oanzaldo nrothchicago mrbacon jens456 madiepev fernanortega imatjazz ssheff ryannetwork ram-msft brymonsoft marberms marknhenry wgabriel kevinmiles zachrenwick zawulon gupard som-nitjsr vdedyukhin mynor

azuresearch_jfk_files's Issues

Issue In Microsoft.ProjectOxford.Vision.ClientException

Hi there,

I got error in Microsoft.ProjectOxford.Vision.ClientException when i try to run this code. I have checked that keys are correct which i am using in this project.

Initializing Services
Creating Synonym Map
Create the index
Boosting Documents
Sending a test image through the pipeline
02/27/2018 09:52:08 Info Processing blob:photo_jpg
Processing resized-image
Processing ocr-result
Processing ocr-handwriting
Processing computer-vision
Processing linked-entities
An Error has occured: System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 401 (Access Denied).
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at Microsoft.ProjectOxford.EntityLinking.EntityLinkingServiceClient.d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ProjectOxford.EntityLinking.EntityLinkingServiceClient.d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at Microsoft.Cognitive.Skills.SkillSet1.<>c__DisplayClass14_04.<<AddSkill>b__0>d.MoveNext() in C:\Users\CITPL-2017\Downloads\AzureSearch_JFK_Files-master\AzureSearch_JFK_Files-master\Microsoft.Cognitive.Skills\SkillSet.cs:line 111 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Microsoft.Cognitive.Skills.Skill1.<GetAsync>d__6.MoveNext() in C:\Users\CITPL-2017\Downloads\AzureSearch_JFK_Files-master\AzureSearch_JFK_Files-master\Microsoft.Cognitive.Skills\Skill.cs:line 41 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult() at Microsoft.Cognitive.Skills.SkillSet1.d__5.MoveNext() in C:\Users\CITPL-2017\Downloads\AzureSearch_JFK_Files-master\AzureSearch_JFK_Files-master\Microsoft.Cognitive.Skills\SkillSet.cs:line 40
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at EnricherFunction.EnrichFunction.<ProcessDocument>d__22.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at EnricherFunction.EnrichFunction.d__21.MoveNext()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait()
at DataEnricher.Program.InitializeServices() in C:\Users\CITPL-2017\Downloads\AzureSearch_JFK_Files-master\AzureSearch_JFK_Files-master\DataEnricher\Program.cs:line 158
at DataEnricher.Program.Main(String[] args) in C:\Users\CITPL-2017\Downloads\AzureSearch_JFK_Files-master\AzureSearch_JFK_Files-master\DataEnricher\Program.cs:line 30
---> (Inner Exception #0) System.Net.Http.HttpRequestException: Response status code does not indicate success: 401 (Access Denied).
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at Microsoft.ProjectOxford.EntityLinking.EntityLinkingServiceClient.d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ProjectOxford.EntityLinking.EntityLinkingServiceClient.d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at Microsoft.Cognitive.Skills.SkillSet1.<>c__DisplayClass14_04.<<AddSkill>b__0>d.MoveNext() in C:\Users\CITPL-2017\Downloads\AzureSearch_JFK_Files-master\AzureSearch_JFK_Files-master\Microsoft.Cognitive.Skills\SkillSet.cs:line 111 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Microsoft.Cognitive.Skills.Skill1.<GetAsync>d__6.MoveNext() in C:\Users\CITPL-2017\Downloads\AzureSearch_JFK_Files-master\AzureSearch_JFK_Files-master\Microsoft.Cognitive.Skills\Skill.cs:line 41 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult() at Microsoft.Cognitive.Skills.SkillSet1.d__5.MoveNext() in C:\Users\CITPL-2017\Downloads\AzureSearch_JFK_Files-master\AzureSearch_JFK_Files-master\Microsoft.Cognitive.Skills\SkillSet.cs:line 40
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at EnricherFunction.EnrichFunction.<ProcessDocument>d__22.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at EnricherFunction.EnrichFunction.d__21.MoveNext()<---

Done.
Press enter to exit

Please help.

Thanks & Regards

Rajendra Singh

How to replace JFK logo images

I tried to replace all the image (PNG) in the frontend folder but it is still showing the JFK files logo. Any tips how to change this?

Indexer warning leading to blank thumbnails

Similar to issue #15 and issue #16 however:

I have the latest version of the code and am getting the same error as the original except that the Metadata is now upper cased for the 'm':

Output field mapping target metadata: unable to select property while accessing '/document/hocrDocument/Metadata'\r\n"

Strangely this only happens for some documents and not for all, and I am using my own data rather than the JFK files sample.

error in the build of the front end

In attempting to build the front end, I have run into issues with
npm install
npm run build:prod

After further inspection, it was because I ran a cmd versus a node cmd window. Consider clarifying the instructions.

Unable to extract Key Phrases from PDF Containing Plain Text

Hi Team
We have tried using the code, it is working fine when we index the PDF containing images in it, but when i tried indexing PDF containing images as well as text, only images are being extracted but not text. Our requirement is to search the keys from the PDF containing text and images.
please help

regards
Ahmed

Program is not reading the .env file

When i run the program i come across this error when I get to the DeployWebsite() Method:
Error deploying website: Could not find a part of the path
It seems to be not reading the path here is my code for reference

EntityLinking key error

Got stuck in ‘Update code’ step 5 that states:

Set DataEnricher is as the default project and hit F5 to run it. It should run without errors and create the search indexes, blob containers, and test your settings. If it fails check your settings to ensure they are correct.

I get the follwin error when I run this by hitting F5.

Initializing Services
Creating Synonym Map
Create the index
Boosting Documents
Sending a test image through the pipeline
03/06/2018 09:24:04 Info Processing blob:photo_jpg
Processing resized-image
Processing ocr-result
Processing ocr-handwriting
Processing computer-vision
Processing linked-entities
An Error has occured: System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: Response status code does not indicate success: 401 (Access Denied).
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at Microsoft.ProjectOxford.EntityLinking.EntityLinkingServiceClient.d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ProjectOxford.EntityLinking.EntityLinkingServiceClient.d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at Microsoft.Cognitive.Skills.SkillSet1.<>c__DisplayClass14_04.<<AddSkill>b__0>d.MoveNext() in C:\Users\nakarch\AzureSearch_JFK_Files\Microsoft.Cognitive.Skills\SkillSet.cs:line 111 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Microsoft.Cognitive.Skills.Skill1.<GetAsync>d__6.MoveNext() in C:\Users\nakarch\AzureSearch_JFK_Files\Microsoft.Cognitive.Skills\Skill.cs:line 41 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult() at Microsoft.Cognitive.Skills.SkillSet1.d__5.MoveNext() in C:\Users\nakarch\AzureSearch_JFK_Files\Microsoft.Cognitive.Skills\SkillSet.cs:line 40
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at EnricherFunction.EnrichFunction.<ProcessDocument>d__22.MoveNext() in C:\Users\nakarch\AzureSearch_JFK_Files\EnricherFunction\EnrichFunction.cs:line 353 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at EnricherFunction.EnrichFunction.d__21.MoveNext() in C:\Users\nakarch\AzureSearch_JFK_Files\EnricherFunction\EnrichFunction.cs:line 336
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait()
at DataEnricher.Program.InitializeServices() in C:\Users\nakarch\AzureSearch_JFK_Files\DataEnricher\Program.cs:line 158
at DataEnricher.Program.Main(String[] args) in C:\Users\nakarch\AzureSearch_JFK_Files\DataEnricher\Program.cs:line 30
---> (Inner Exception #0) System.Net.Http.HttpRequestException: Response status code does not indicate success: 401 (Access Denied).
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at Microsoft.ProjectOxford.EntityLinking.EntityLinkingServiceClient.d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ProjectOxford.EntityLinking.EntityLinkingServiceClient.d__14.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at Microsoft.Cognitive.Skills.SkillSet1.<>c__DisplayClass14_04.<<AddSkill>b__0>d.MoveNext() in C:\Users\nakarch\AzureSearch_JFK_Files\Microsoft.Cognitive.Skills\SkillSet.cs:line 111 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Microsoft.Cognitive.Skills.Skill1.<GetAsync>d__6.MoveNext() in C:\Users\nakarch\AzureSearch_JFK_Files\Microsoft.Cognitive.Skills\Skill.cs:line 41 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult() at Microsoft.Cognitive.Skills.SkillSet1.d__5.MoveNext() in C:\Users\nakarch\AzureSearch_JFK_Files\Microsoft.Cognitive.Skills\SkillSet.cs:line 40
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at EnricherFunction.EnrichFunction.<ProcessDocument>d__22.MoveNext() in C:\Users\nakarch\AzureSearch_JFK_Files\EnricherFunction\EnrichFunction.cs:line 353 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at EnricherFunction.EnrichFunction.d__21.MoveNext() in C:\Users\nakarch\AzureSearch_JFK_Files\EnricherFunction\EnrichFunction.cs:line 336<---

Done.
Press enter to exit

Resolution:
there has been a recent change in the way API keys for the Entity Linking Service are handled. This service is being deprecated and the functionality moved to the Text Analytics Service. Need to wait for code update that supports the new Text Analytics Service

error creatating web site

'rimraf' is not recognized as an internal or external command, operable program or batch file.

Update

It turns out I actually am getting an error when I run the npm install command. I had overlooked it because there are a ton of warnings when I run npm install that stem from there being no .staging folder in my directory.
the "npm install" error can be seen below the output log.

2018-09-19T17_44_11_742Z-debug.log

npm ERR! code EINTEGRITY
npm ERR! sha512-Gr4p6nFNaoufRIY4NMdpQRNmgxVIGMs4Fcu/ujdYk3nAZqk7supzBE9idmvfZIlH/Cuj//dvi+019qEue9lV0w== integrity checksum failed when using sha512: wanted sha512-Gr4p6nFNaoufRIY4NMdpQRNmgxVIGMs4Fcu/ujdYk3nAZqk7supzBE9idmvfZIlH/Cuj//dvi+019qEue9lV0w== but got sha512-sOcbKZHpyHcQmQsBeWSQNBLXbXHJ3wcqtxp1lmqCgy09x4ooAA0hqamlCDr/MuYP1LiHpYjx83gxsRN7R8IxaQ==. (490114 bytes)

Original:

when I run through the readme to set up the environment I get the error in the title when I execute
"npm run build"

Going to additionally attach a debug log since it's saying "this is probably not a problem with npm. there is likely additional logging output above"
2018-09-19T17_53_46_268Z-debug.log
:prod"

Transient Error Search Service

Error when running Search Indexer, error description says "Transient Error" when debug turned on

SearchServices apiKey value does not fall within the expected range

Trying to setup the jfk files project, however, when I try to execute step 12 (run the JFKInitializer), this error was thrown.

All operations successful - no website..

</body></html> text: NEW ORLEANS, LA. 112 723 8 63 Lee Harvey Oswald standing in front of a sign text person human face wall man clothing sketch drawing sign portrait black and white posing handwriting entities: System.String[] cryptonyms: System.String[] demoBoost: 24 demoInitialPage: All operations were successful. Press any key to exit.

thats my output from the console but there is no website url :(

Website not returning results though no errors anywhere

I deployed the Azure solution from the powershell script and ran the JFKinitializer and the frontend. At the end query index also returned results but somehow if I navigate to the site and search, no results are returned. There are no errors as well.

error running the visual studio project

Error creating skillset: Invalid URI: The hostname could not be parsed.

An existing connection was forcibly closed by the remote host

I am seeing these issues in the indexer:

"key": "https://azsjfkfiles.blob.core.windows.net/jfkfiles/castro%20operation/docid-32105760.pdf",
"errorMessage": "Skill #12: An existing connection was forcibly closed by the remote host\r\nSkill #12: An existing connection was forcibly closed by the remote host\r\nSkill #14: An existing connection was forcibly closed by the remote host\r\n

Any idea what it could be?

Thanks!

npm run build:prod fails with error

When running npm run build:prod as from installation details, it fails with error on the npm run clean && env-cmd .env cross-env NODE_ENV=production webpack -p --config=webpack.prod.config.js platform is

node v8.9.0
npm v5.6.0

Stack trace
at emitTwo (events.js:126:13)
at ChildProcess.emit (events.js:214:7)
at maybeClose (internal/child_process.js:925:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)

Not showing the preview of own PDF's in the Frontend

I uploaded two PDF files with the same content.

The first PDF with characters included (= already "OCR'ed").
The second PDF containing "screenshots" of the characters.

However, the front-end seems to have problems visualizing the text within the first PDF properly.

How can I make the results of the first PDF searchable?

Microsoft.ProjectOxford.Vision.ClientException

Hi there,

I get Microsoft.ProjectOxford.Vision.ClientException when i try to run this code. I have verified my keys I am using with the service.

Below is the stack trace,
Initializing Services
Creating Synonym Map
Create the index
Boosting Documents
Sending a test image through the pipeline
02/23/2018 17:37:03 Info Processing blob:photo_jpg
Processing resized-image
Processing ocr-result
An Error has occured: System.AggregateException: One or more errors occurred. ---> Microsoft.ProjectOxford.Vision.ClientException: Exception of type 'Microsoft.ProjectOxford.Vision.ClientException' was thrown.
at Microsoft.ProjectOxford.Vision.VisionServiceClient.HandleException(Exception exception)
at Microsoft.ProjectOxford.Vision.VisionServiceClient.b__39_1[TRequest,TResponse](Exception e)
at System.AggregateException.Handle(Func2 predicate) at Microsoft.ProjectOxford.Vision.VisionServiceClient.<SendAsync>d__392.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at CallSite.Target(Closure , CallSite , Object )
at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
at Microsoft.ProjectOxford.Vision.VisionServiceClient.d__31.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at Microsoft.Cognitive.Skills.Vision.<GetText>d__7.MoveNext() in C:\Users\meranjit\Downloads\AzureSearch_JFK_Files-master\AzureSearch_JFK_Files-master\Microsoft.Cognitive.Skills\Vision.cs:line 113 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Microsoft.Cognitive.Skills.SkillSet1.<>c__DisplayClass12_02.<b__0>d.MoveNext() in C:\Users\meranjit\Downloads\AzureSearch_JFK_Files-master\AzureSearch_JFK_Files-master\Microsoft.Cognitive.Skills\SkillSet.cs:line 97
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at Microsoft.Cognitive.Skills.Skill1.d__6.MoveNext() in C:\Users\meranjit\Downloads\AzureSearch_JFK_Files-master\AzureSearch_JFK_Files-master\Microsoft.Cognitive.Skills\Skill.cs:line 41
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
at Microsoft.Cognitive.Skills.SkillSet1.<ApplyAsync>d__5.MoveNext() in C:\Users\meranjit\Downloads\AzureSearch_JFK_Files-master\AzureSearch_JFK_Files-master\Microsoft.Cognitive.Skills\SkillSet.cs:line 40 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at EnricherFunction.EnrichFunction.d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at EnricherFunction.EnrichFunction.<Run>d__19.MoveNext() --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) at System.Threading.Tasks.Task.Wait() at DataEnricher.Program.InitializeServices() in C:\Users\meranjit\Downloads\AzureSearch_JFK_Files-master\AzureSearch_JFK_Files-master\DataEnricher\Program.cs:line 158 at DataEnricher.Program.Main(String[] args) in C:\Users\meranjit\Downloads\AzureSearch_JFK_Files-master\AzureSearch_JFK_Files-master\DataEnricher\Program.cs:line 30 ---> (Inner Exception #0) Microsoft.ProjectOxford.Vision.ClientException: Exception of type 'Microsoft.ProjectOxford.Vision.ClientException' was thrown. at Microsoft.ProjectOxford.Vision.VisionServiceClient.HandleException(Exception exception) at Microsoft.ProjectOxford.Vision.VisionServiceClient.<SendAsync>b__39_1[TRequest,TResponse](Exception e) at System.AggregateException.Handle(Func2 predicate)
at Microsoft.ProjectOxford.Vision.VisionServiceClient.d__392.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at CallSite.Target(Closure , CallSite , Object ) at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0) at Microsoft.ProjectOxford.Vision.VisionServiceClient.<RecognizeTextAsync>d__31.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Microsoft.Cognitive.Skills.Vision.d__7.MoveNext() in C:\Users\meranjit\Downloads\AzureSearch_JFK_Files-master\AzureSearch_JFK_Files-master\Microsoft.Cognitive.Skills\Vision.cs:line 113
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at Microsoft.Cognitive.Skills.SkillSet1.<>c__DisplayClass12_02.<<AddSkill>b__0>d.MoveNext() in C:\Users\meranjit\Downloads\AzureSearch_JFK_Files-master\AzureSearch_JFK_Files-master\Microsoft.Cognitive.Skills\SkillSet.cs:line 97 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at Microsoft.Cognitive.Skills.Skill1.<GetAsync>d__6.MoveNext() in C:\Users\meranjit\Downloads\AzureSearch_JFK_Files-master\AzureSearch_JFK_Files-master\Microsoft.Cognitive.Skills\Skill.cs:line 41 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.GetResult() at Microsoft.Cognitive.Skills.SkillSet1.d__5.MoveNext() in C:\Users\meranjit\Downloads\AzureSearch_JFK_Files-master\AzureSearch_JFK_Files-master\Microsoft.Cognitive.Skills\SkillSet.cs:line 40
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.TaskAwaiter1.GetResult() at EnricherFunction.EnrichFunction.<ProcessDocument>d__20.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()
at EnricherFunction.EnrichFunction.d__19.MoveNext()<---

Done.
Press enter to exit

Also i noticed this error does not come if i upload a single page document at a time.

Thanks,
Mercy

Internet Explorer

Trying to make the code run in IE11, there are some JS errors, probably coming from react. Is this can be a issue of react-redux version issue or anything?
Here is one of the error, I am not able to understand:
Unable to get property 'Component' of undefined or null reference

frontend CORS error

Hi,

I just started running the frontend on my machine but I get the following error on load:

api.ts:32 OPTIONS https://compliance.search.windows.net/indexes/jfkdocs/docs?api-version=2017-11-11&search=test&searchMode=all&facet=tags,count:20&facet=redactions,values:50&$count=true&$top=10&scoringProfile=demoBooster&highlight=text 400 (Bad Request)

Followed by a CORS error:

:8082/#/search?term=test:1 Failed to load https://compliance.search.windows.net/indexes/jfkdocs/docs?api-version=2017-11-11&search=test&searchMode=all&facet=tags,count:20&facet=redactions,values:50&$count=true&$top=10&scoringProfile=demoBooster&highlight=text:

Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost:8082' is therefore not allowed access. The response had HTTP status code 400. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

How can I set some CORS options?

npm run start:dev error errno 4294967295

When a call the npm run start:dev command i get this error:

0 info it worked if it ends with ok
1 verbose cli [ 'C:\Program Files\nodejs\node.exe',
1 verbose cli 'C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js',
1 verbose cli 'run',
1 verbose cli 'start:dev' ]
2 info using [email protected]
3 info using [email protected]
4 verbose run-script [ 'prestart:dev', 'start:dev', 'poststart:dev' ]
5 info lifecycle [email protected] prestart:dev: [email protected]
6 info lifecycle [email protected] start:dev: [email protected]
7 verbose lifecycle [email protected] start:dev: unsafe-perm in lifecycle true
8 verbose lifecycle [email protected] start:dev: PATH: C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin;C:\descomp\tirso-idata\frontend\node_modules.bin;C:\Program Files\ImageMagick-7.0.8-Q16;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0;C:\WINDOWS\System32\OpenSSH;C:\Program Files\nodejs;C:\Program Files\PuTTY;C:\Program Files\Git\cmd;C:\Program Files (x86)\Microsoft SQL Server\150\DTS\Binn;C:\Program Files\dotnet;C:\Program Files\Microsoft SQL Server\130\Tools\Binn;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn;C:\Python\Python37\Scripts;C:\Python\Python37;C:\Users\CamiloSoto\AppData\Local\Microsoft\WindowsApps;C:\Users\CamiloSoto\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\CamiloSoto\AppData\Roaming\npm;C:\Users\CamiloSoto\AppData\Local\Programs\Azure Data Studio\bin;C:\poppler\bin;C:\Users\CamiloSoto.dotnet\tools;C:\Program Files\Microsoft Office\root\Client
9 verbose lifecycle [email protected] start:dev: CWD: C:\descomp\tirso-idata\frontend
10 silly lifecycle [email protected] start:dev: Args: [ '/d /s /c',
10 silly lifecycle 'env-cmd .env cross-env NODE_ENV=development webpack-dev-server --config=webpack.dev.config.js' ]
11 silly lifecycle [email protected] start:dev: Returned: code: 4294967295 signal: null
12 info lifecycle [email protected] start:dev: Failed to exec start:dev script
13 verbose stack Error: [email protected] start:dev: env-cmd .env cross-env NODE_ENV=development webpack-dev-server --config=webpack.dev.config.js
13 verbose stack Exit status 4294967295
13 verbose stack at EventEmitter. (C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\index.js:301:16)
13 verbose stack at EventEmitter.emit (events.js:189:13)
13 verbose stack at ChildProcess. (C:\Program Files\nodejs\node_modules\npm\node_modules\npm-lifecycle\lib\spawn.js:55:14)
13 verbose stack at ChildProcess.emit (events.js:189:13)
13 verbose stack at maybeClose (internal/child_process.js:970:16)
13 verbose stack at Process.ChildProcess._handle.onexit (internal/child_process.js:259:5)
14 verbose pkgid [email protected]
15 verbose cwd C:\descomp\tirso-idata\frontend
16 verbose Windows_NT 10.0.18362
17 verbose argv "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js" "run" "start:dev"
18 verbose node v10.15.3
19 verbose npm v6.4.1
20 error code ELIFECYCLE
21 error errno 4294967295
22 error [email protected] start:dev: env-cmd .env cross-env NODE_ENV=development webpack-dev-server --config=webpack.dev.config.js
22 error Exit status 4294967295
23 error Failed at the [email protected] start:dev script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 4294967295, true ]

But if i call npm run start:prod all is OK

npm install error

Need instruction to solve below error. Tried (package.json: "ajv": "^6.0.0"), didn't work.

C:\Users\rothet\Desktop\POC\AzureSearch_JFK_Files-master\frontend>npm install
npm WARN [email protected] requires a peer of ajv@^6.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] No repository field.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})

Hos is Azure ML implemented in this JFK sample?

In one of readme.md file, it is mentioned Azure ML would soon be integrated in this sample. May I know if it is implemented in this, and where I can find it?

After using private/no public access blob storage, the file preview and Toggle View gets blurry

Hi, I changed the "JFKFilesBlobStorageAccountConnectionString" and "JFKFilesBlobContainerName” for a private blob storage and set the “Set Public Acces Level = No public access”.

The JFK interface works fine, but the file preview and Toggle View shows blurry.

P.S.
I commented lines 137,138 and 139 at Program.cs
// BlobContainerPermissions permissions = container.GetPermissions();
// permissions.PublicAccess = BlobContainerPublicAccessType.Container;
// await container.SetPermissionsAsync(permissions);

Many Thanks,
Matt

Custom cognitive skills need to update NuGet dependencies

I get the following error after deploying custom cognitive skills to Azure Functions:

Could not load type 'Microsoft.Azure.WebJobs.ExecutionContext' from assembly 'Microsoft.Azure.WebJobs.Extensions, Version=3.0.0.0'

I believe this is related to the NuGet package version for the Azure Functions. Testing a fix now.

Add some details for expected indexing times

While deploying the default sample and the example set of documents, the indexing process is taking more than 2 hours. I checked how many total documents there were in the provided sample blob store, and it looked like there are 49 documents. I'm not sure what the indexing processing time is for this repo, but it feels like more than a few minutes is too long to index 49 documents. If the long indexing times are expected, can we add a small note about it in the README?

FYI - I deployed using the ARM template to a B1 App Service plan and the Standard tier for Azure Search.

AzureWebJobsSecretStorageType must be set to Files

I got an error when creating the skillset.
Debugging revealed some API issue -- got fixed by setting the AzureWebJobsSecretStorageType to Files in the App settings of the App Function.

How can I execute JfkWebApiSkills project.

I want to add my testing images in Blobstorage and after that I need to create tag and indexes. So How can I execute this project and add my personal images.

your worker runtime is not set error

@Careyjmac I'm also seeing an issue during build of the frontend, after step #12. I receive a "your worker runtime is not set" error that advises "run func settings add functions_worker_runtime"

Any guidance on this error? It appears to have been an issue in the Azure Functions code and was addressed in 2018. But the error persists.

Problem with indexer

Hello, I get this problem and don't know how to fix it :

Error creating indexer: This indexer refers to a skillset 'azureblob-skillset' that doesn't exist
Something went wrong.

I have created the skillset in my azure search, so don't know why it could not reach it

Use a angular 7

now I have .net code and front end in react.js but I want a frontEnd in angular7 so how to use angular and .net

Creating Index ended in error

eleting Index, Indexer and SynonymMap if they exist...
Creating Blob Container for Image Store Skill...
Creating Skill Set...
Creating Synonym Map...
Creating Index...
Create Index response:
{"error":{"code":"","message":"The request is invalid. Details: definition : The synonym map 'cryptonyms' specified in the field 'text' does not exist.\r\n"}}
Something went wrong.
Press any key to exit.

Not showing my pdf file in search

Demo is working fine for jfk pdf files but whenever i'm going to add my own pdf files or other documents in JFKFilesBlobStorageAccountConnectionString path its showing only search text without their images.

Also can we insert image or other word file directly to final imageblobstore container for searching from azure portal?
Or we need to follow same steps every time while uploading new file into the storage for refreshing the index?

Thanks in advance for help :)

Indexer warning

I'm trying to deploy this sample, but I get a warning in the Indexer (for every document):

Output field mapping target metadata: unable to select property while accessing '/document/hocrDocument/metadata'\r\n"

The strange thing is that it worked 2 weeks ago ...

Thanks,
Jan

Microsoft link in footer is broken

It looks like the URL is missing a forward slash

Error while searching on application

I'm getting error while searching on application. As checked in browser, the search button hitting the wrong URL: https://jfk-search-service-5qzqyqg22owne.search.windows.net, correct URL is https://jfk-site-eypd7z6qyiico.azurewebsites.net/.

Try to find out where is went wrong but did not found this "5qzqyqg22owne" in whole application.
Please help me to fix this problem.

Complete url is: OPTIONS https://jfk-search-service-5qzqyqg22owne.search.windows.net/indexes/jfkindex/docs?api-version=2017-11-11-Preview&searchMode=all&facet=entities,count:20&$count=true&$top=10&scoringProfile=demoBooster&highlight=text 0 ()

front end not showing thumbnail

there is no error. and query works via "search explorer"

Getting error in the frontend code, when ran npm start after npm install.

saurabh@BLRKEC140533L:/mnt/d/projects/molina/AzureSearch_JFK_Files-master/frontend$ npm start

[email protected] start /mnt/d/projects/molina/AzureSearch_JFK_Files-master/frontend
env-cmd .env if-env NODE_ENV=production && npm run start:prod || npm run start:dev

[email protected] start:dev /mnt/d/projects/molina/AzureSearch_JFK_Files-master/frontend
env-cmd .env cross-env NODE_ENV=development webpack-dev-server --config=webpack.dev.config.js

/mnt/d/projects/molina/AzureSearch_JFK_Files-master/frontend/node_modules/webpack/lib/webpack.js:189
throw new RemovedPluginError(errorMessage);
^

Error: webpack.optimize.CommonsChunkPlugin has been removed, please use config.optimization.splitChunks instead.
at Object.get [as CommonsChunkPlugin] (/mnt/d/projects/molina/AzureSearch_JFK_Files-master/frontend/node_modules/webpack/lib/webpack.js:189:10)
at Object. (/mnt/d/projects/molina/AzureSearch_JFK_Files-master/frontend/webpack.base.config.js:147:26)
at Module._compile (module.js:662:30)
at Object.Module._extensions..js (module.js:673:10)
at Module.load (module.js:575:32)
at tryModuleLoad (module.js:515:12)
at Function.Module._load (module.js:507:3)
at Module.require (module.js:606:17)
at require (internal/module.js:11:18)
at Object. (/mnt/d/projects/molina/AzureSearch_JFK_Files-master/frontend/webpack.dev.config.js:4:22)
at Module._compile (module.js:662:30)
at Object.Module._extensions..js (module.js:673:10)
at Module.load (module.js:575:32)
at tryModuleLoad (module.js:515:12)
at Function.Module._load (module.js:507:3)
at Module.require (module.js:606:17)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start:dev: env-cmd .env cross-env NODE_ENV=development webpack-dev-server --config=webpack.dev.config.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start:dev script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /home/saurabh/.npm/_logs/2019-09-18T11_48_12_938Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: env-cmd .env if-env NODE_ENV=production && npm run start:prod || npm run start:dev
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /home/saurabh/.npm/_logs/2019-09-18T11_48_13_101Z-debug.log

Error after npm run build:prod

Hello,

I´m install Node.js and do every step, after npm run build:prod I´m recived this error:

  2 modules

Child extract-text-webpack-plugin ../node_modules/extract-text-webpack-plugin/dist ../node_modules/css-loader/index.js??ref--8-2!../node_modules/resolve-url-loader/index.js!../node_modules/sass-loader/lib/loader.js!pages/search-page/components/selection-controls/checkbox-list.style.scss:
2 modules
npm ERR! code ELIFECYCLE
npm ERR! errno 2
npm ERR! [email protected] build:prod: npm run clean && env-cmd .env cross-env NODE_ENV=production webpack -p --config=webpack.prod.config.js
npm ERR! Exit status 2
npm ERR!
npm ERR! Failed at the [email protected] build:prod script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\andre\AppData\Roaming\npm-cache_logs\2018-06-06T13_28_32_604Z-debug.log

No document preview on search screen and empty screen on toggle view

I am having issue with document preview on search screen and empty screen on toggle view.

I downloaded the JFK code last month and followed the guide step by step. All services were up and working fine. I saw that, there is a new JFK code release last week and I applied it. Since then, I am having issue with file preview. I can see the file names (docid-32395634.pdf, photo.jpg), but no file preview and empty toggle view. When I click the file name, it brings up empty toggle view page.

After I got this problem, I generated all new services with different names. Applied new JFK code (only added key information on app.config) and got same problem again.

FYI - I deployed using the ARM template to a Basic App Service plan and the Basic tier for Azure Search ( South Central location) and East US location for all other services

.
.
.

Installed Node 10.5.0 Current

npm install
npm WARN [email protected] requires a peer of ajv@^6.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN [email protected] No repository field.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules\fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"win32","arch":"x64"})

audited 9590 packages in 16.77s
found 9 moderate severity vulnerabilities
run npm audit fix to fix them, or npm audit for details

npm run build:prod
run without any error or warning

Index names in config files are not sufficient to change the datasource

Even if i change the index names in the config file to point to a data source, the JFK files are only shown. Seems like there seems to be a caching going on in frontend code

JFK Files connection string is not functional.

Running the JfkInitializer is producing 400 error on the CreateIndexer() step. Replacing the JFKFilesBlobStorageAccountConnectionString connection string value with a different one seems to fix it.

Is there another place to source the content?

skillset #10 is depreciated

The skill set #10 NamedEntityRecognitionSkill is depreciated.

https://docs.microsoft.com/en-us/azure/search/cognitive-search-skill-deprecated

needs to replace NamedEntityRecognitionSkill by EntityRecognitionSkill
and entities by namedEntities

Still pointing to JFK files after changing all settings.

I have changed all JFK related settings in the AppConfig.cs and Config.cs files to point to our company's Search Service, Storage Containers, Blobs, etc., but I am still pointed to the JFK files. The only thing left unchanged are the Azure Web App instance and Azure Function instance. Am I missing something?

Vision.ClientException when running for the first time

Hi,

when running for the first time after creating the services on Azure, I get the following exception. Since it does not really say much, could you help me troubleshoot it?

Initializing Services
Creating Synonym Map
Create the index
Boosting Documents
Sending a test image through the pipeline
01/25/2018 08:26:52 Info Processing blob:photo_jpg
Processing resized-image
Processing ocr-result
An Error has occured: System.AggregateException: One or more errors occurred. ---> Microsoft.ProjectOxford.Vision.ClientException: Exception of type 'Microsoft.ProjectOxford.Vision.ClientException' was thrown.
   at Microsoft.ProjectOxford.Vision.VisionServiceClient.HandleException(Exception exception)
   at Microsoft.ProjectOxford.Vision.VisionServiceClient.<SendAsync>b__39_1[TRequest,TResponse](Exception e)
   at System.AggregateException.Handle(Func`2 predicate)
   at Microsoft.ProjectOxford.Vision.VisionServiceClient.<SendAsync>d__39`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at CallSite.Target(Closure , CallSite , Object )
   at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
   at Microsoft.ProjectOxford.Vision.VisionServiceClient.<RecognizeTextAsync>d__31.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Cognitive.Skills.Vision.<GetText>d__6.MoveNext() in E:\Boulot\Genesys\Cloud Architects\Gennovation\AzureSearch_JFK_Files\Microsoft.Cognitive.Skills\Vision.cs:line 55
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Cognitive.Skills.SkillSet`1.<>c__DisplayClass12_0`2.<<AddSkill>b__0>d.MoveNext() in E:\Boulot\Genesys\Cloud Architects\Gennovation\AzureSearch_JFK_Files\Microsoft.Cognitive.Skills\SkillSet.cs:line 85
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Cognitive.Skills.Skill`1.<GetAsync>d__6.MoveNext() in E:\Boulot\Genesys\Cloud Architects\Gennovation\AzureSearch_JFK_Files\Microsoft.Cognitive.Skills\Skill.cs:line 41
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.Cognitive.Skills.SkillSet`1.<ApplyAsync>d__5.MoveNext() in E:\Boulot\Genesys\Cloud Architects\Gennovation\AzureSearch_JFK_Files\Microsoft.Cognitive.Skills\SkillSet.cs:line 40
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at EnricherFunction.EnrichFunction.<Run>d__12.MoveNext() in E:\Boulot\Genesys\Cloud Architects\Gennovation\AzureSearch_JFK_Files\EnricherFunction\EnrichFunction.cs:line 172
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at DataEnricher.Program.InitializeServices() in E:\Boulot\Genesys\Cloud Architects\Gennovation\AzureSearch_JFK_Files\DataEnricher\Program.cs:line 158
   at DataEnricher.Program.Main(String[] args) in E:\Boulot\Genesys\Cloud Architects\Gennovation\AzureSearch_JFK_Files\DataEnricher\Program.cs:line 30
---> (Inner Exception #0) Microsoft.ProjectOxford.Vision.ClientException: Exception of type 'Microsoft.ProjectOxford.Vision.ClientException' was thrown.
   at Microsoft.ProjectOxford.Vision.VisionServiceClient.HandleException(Exception exception)
   at Microsoft.ProjectOxford.Vision.VisionServiceClient.<SendAsync>b__39_1[TRequest,TResponse](Exception e)
   at System.AggregateException.Handle(Func`2 predicate)
   at Microsoft.ProjectOxford.Vision.VisionServiceClient.<SendAsync>d__39`2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at CallSite.Target(Closure , CallSite , Object )
   at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
   at Microsoft.ProjectOxford.Vision.VisionServiceClient.<RecognizeTextAsync>d__31.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Cognitive.Skills.Vision.<GetText>d__6.MoveNext() in E:\Boulot\Genesys\Cloud Architects\Gennovation\AzureSearch_JFK_Files\Microsoft.Cognitive.Skills\Vision.cs:line 55
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Cognitive.Skills.SkillSet`1.<>c__DisplayClass12_0`2.<<AddSkill>b__0>d.MoveNext() in E:\Boulot\Genesys\Cloud Architects\Gennovation\AzureSearch_JFK_Files\Microsoft.Cognitive.Skills\SkillSet.cs:line 85
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.Cognitive.Skills.Skill`1.<GetAsync>d__6.MoveNext() in E:\Boulot\Genesys\Cloud Architects\Gennovation\AzureSearch_JFK_Files\Microsoft.Cognitive.Skills\Skill.cs:line 41
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.Cognitive.Skills.SkillSet`1.<ApplyAsync>d__5.MoveNext() in E:\Boulot\Genesys\Cloud Architects\Gennovation\AzureSearch_JFK_Files\Microsoft.Cognitive.Skills\SkillSet.cs:line 40
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at EnricherFunction.EnrichFunction.<Run>d__12.MoveNext() in E:\Boulot\Genesys\Cloud Architects\Gennovation\AzureSearch_JFK_Files\EnricherFunction\EnrichFunction.cs:line 172<---

adding new page

How can I customize the pages and add new pages? is there any documentation for that?

Error when running Microsoft.Azure.WebJobs.Script.ScriptHost

System.IO.FileNotFoundException
HResult=0x80070002
Message=Could not find file 'D:\All Clients\MARS\Cognitive Search Demo\AzureSearch_JFK_Files-master\AzureSearch_JFK_Files-master\JfkWebApiSkills\JfkWebApiSkills\bin\Debug\netstandard2.0\bin\function.json'.
Source=System.Private.CoreLib
StackTrace:
at System.IO.FileStream.OpenHandle(FileMode mode, FileShare share, FileOptions options)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
at System.IO.StreamReader..ctor(String path, Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize)
at System.IO.StreamReader..ctor(String path, Encoding encoding, Boolean detectEncodingFromByteOrderMarks)
at System.IO.File.InternalReadAllText(String path, Encoding encoding)
at System.IO.File.ReadAllText(String path)
at Microsoft.Azure.WebJobs.Script.ScriptHost.ReadFunctionMetadata(String scriptDir, ILogger logger, Dictionary2 functionErrors, ScriptSettingsManager settingsManager, IEnumerable1 functionWhitelist) in C:\azure-webjobs-sdk-script\src\WebJobs.Script\Host\ScriptHost.cs:line 1081

Error on running Visual Studio for jfkInitializer.Program.NewMethod

When running after editing the app.config file i received the following error.

Visual Studio on Mac OS HighSierra 10.1.35 (17f77)
Visual Studio Version 7.5.1 (Build 22)

System.ArgumentException: Value does not fall within the expected range.
at System.Net.Http.Headers.Parser+Token.Check (System.String s) [0x00019] in /Users/builder/jenkins/workspace/build-package-osx-mono/2017-12/external/bockbuild/builds/mono-x64/mcs/class/System.Net.Http/System.Net.Http.Headers/Parser.cs:57
at System.Net.Http.Headers.ProductHeaderValue..ctor (System.String name, System.String version) [0x0000a] in /Users/builder/jenkins/workspace/build-package-osx-mono/2017-12/external/bockbuild/builds/mono-x64/mcs/class/System.Net.Http/System.Net.Http.Headers/ProductHeaderValue.cs:45
at System.Net.Http.Headers.ProductInfoHeaderValue..ctor (System.String productName, System.String productVersion) [0x00006] in /Users/builder/jenkins/workspace/build-package-osx-mono/2017-12/external/bockbuild/builds/mono-x64/mcs/class/System.Net.Http/System.Net.Http.Headers/ProductInfoHeaderValue.cs:51
at Microsoft.Rest.ServiceClient1[T].get_DefaultUserAgentInfoList () [0x0003f] in <8de96d8c20204f0697e94bce765128e1>:0 at Microsoft.Rest.ServiceClient1[T].SetUserAgent (System.String productName, System.String version) [0x00010] in <8de96d8c20204f0697e94bce765128e1>:0
at Microsoft.Rest.ServiceClient1[T].InitializeHttpClient (System.Net.Http.HttpClient httpClient, System.Net.Http.HttpClientHandler httpClientHandler, System.Net.Http.DelegatingHandler[] handlers) [0x00092] in <8de96d8c20204f0697e94bce765128e1>:0 at Microsoft.Rest.ServiceClient1[T].InitializeHttpClient (System.Net.Http.HttpClientHandler httpClientHandler, System.Net.Http.DelegatingHandler[] handlers) [0x00000] in <8de96d8c20204f0697e94bce765128e1>:0
at Microsoft.Rest.ServiceClient1[T]..ctor (System.Net.Http.HttpClientHandler rootHandler, System.Net.Http.DelegatingHandler[] handlers) [0x00006] in <8de96d8c20204f0697e94bce765128e1>:0 at Microsoft.Rest.ServiceClient1[T]..ctor (System.Net.Http.DelegatingHandler[] handlers) [0x00006] in <8de96d8c20204f0697e94bce765128e1>:0
at Microsoft.Azure.Search.SearchServiceClient..ctor (System.Net.Http.DelegatingHandler[] handlers) [0x00000] in :0
at Microsoft.Azure.Search.SearchServiceClient..ctor (System.String searchServiceName, Microsoft.Azure.Search.SearchCredentials credentials) [0x00000] in :0
at JfkInitializer.Program.NewMethod (System.String searchServiceName, System.String apiKey) [0x00000] in /Users/robert 1/Desktop/AzureSearch_JFK_Files-master/JfkWebApiSkills/JfkInitializer/Program.cs:66
at JfkInitializer.Program.Main (System.String[] args) [0x00020] in /Users/robert 1/Desktop/AzureSearch_JFK_Files-master/JfkWebApiSkills/JfkInitializer/Program.cs:45

Buidling the frontend throws error

When running
npm run build:prod

I receive
ERROR in [at-loader] ./src/pages/search-page/components/item/item.component.tsx:41:7 TS2326: Types of property 'component' are incompatible. Type '"img"' is not assignable to type 'ReactType<CardMediaProps>'.

As a quick workaround I changed the CardMedia component in item.component.tsx to

const ItemMediaThumbnail: React.StatelessComponent<ItemProps> = ({ item, onClick }) => {
  return (
    item.thumbnail ? 
    <CardMedia className={style.media}
      image={item.thumbnail}        
      title={item.title}
      onClick={handleOnClick({ item, onClick })}
    /> : null
  );
}

Error in CreateSkillSet

Hi
I used to be able to initialize perfectly fine earlier but am now receiving this error when running the initializer. The error shows up in CreateSkillSet when getting the AzureFunctionHostKey here:

_azureFunctionHostKey = await KeyHelper.GetAzureFunctionHostKey(_httpClient);

Any thoughts?

{"Message":"An error has occurred.","ExceptionMessage":"Runtime keys are stored on blob storage. This API doesn't support this configuration. Please change Environment variable AzureWebJobsSecretStorageType value to 'Files'. For more info, visit https://aka.ms/funcsecrets","ExceptionType":"System.InvalidOperationException","StackTrace":" at Kudu.Core.Functions.FunctionManager.d__9`1.MoveNext() in C:\Kudu Files\Private\src\master\Kudu.Core\Functions\FunctionManager.cs:line 141\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Kudu.Core.Functions.FunctionManager.d__11.MoveNext() in C:\Kudu Files\Private\src\master\Kudu.Core\Functions\FunctionManager.cs:line 213\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at Kudu.Services.Functions.FunctionController.d__11.MoveNext() in C:\Kudu Files\Private\src\master\Kudu.Services\Functions\FunctionController.cs:line 121"}

microsoft / azuresearch_jfk_files Goto Github PK

azuresearch_jfk_files's Introduction

The JFK Files

Cognitive Search - An AI-first approach to content understanding

JFK Files Architecture

Limitations and Considerations

Setting up your own JFK files library

Prerequisites

Deploy Required Resources

Initialize the code

Run the initializer

Results

Debugging

Next Steps

azuresearch_jfk_files's People

Contributors

Stargazers

Watchers

Forkers

azuresearch_jfk_files's Issues

Recommend Projects

Recommend Topics

Recommend Org