nicolay-r / arelight Goto Github PK

• Adopt prefix parameter in io in order to be able to modify output filenames for sampled data.
• Sync names for sampled data and predicted results

Issues that we are encountered with:

• DeepPavlov resources might not unzipped (zipp==3.6.0 issue)
• For pyMystem3: nlpub/pymystem3#21
• Download AREkit data

From #21

Caused by #31 and while working on #21

Since this project was a part of the AREkit previosly, the incomplete refactoring has been performed.

Download script could not be executed.

Reason: there is a separated project AREnets already for this

Considering the following result:

Another example:

All of it might be caused by hiding all the entity values by using #E , while it is expected to be hidden only specific ner types: LOC, PER, ORG

FYI:

When coming to the readme, I'm presented with english language, though all screenshots of the tool showcase cyryllic characters. A search in the readme for "languages" does not yield any results.

It would be very benefitial if you could clarify in the readme which languages are supported. I'm sure I can dig this up reading through the research paper but IMO that's unnessary complex to users coming to this repo,

When running

python infer_bert.py --from-files ../data/texts-inosmi-rus/e1.txt \
    --labels-count 3 \
    --terms-per-context 50 \
    --tokens-per-context 128 \
    --text-b-type nli_m \
    -o output/brat_inference_output

I get

...
INFO:tensorflow:Restoring parameters from /content/ARElight/data/models/ra-20-srubert-large-neut-nli-pretrained-3l-finetuned/ra-20-srubert-large-neut-nli-pretrained-3l
  0%|                                                                                       | 0/1253 [00:00<?, ?opins/s]src/tcmalloc.cc:283] Attempt to free invalid pointer 0x107e00000 

and the process freezes.

Google colab, Python 3.7, tensorflow 1.15.0, numpy 1.21.6, deeppavlov 0.11.0, arekit installed from git. Tried restarting the runtime, doesn't help.

Китай все-таки намерен ввести санкционные меры против РФ и в дальнешем, Югославии.

Causes:

 File "/media/nicolay/96ed6537-b931-4f7e-8ac4-8407527ddbf9/proj/REmarker/venv/lib/python3.6/site-packages/arekit/common/news/entities_grouping.py", line 15, in apply_core
    group_index = self.__value_to_group_id_func(entity.Value)
  File "/media/nicolay/96ed6537-b931-4f7e-8ac4-8407527ddbf9/proj/REmarker/venv/lib/python3.6/site-packages/arekit/common/synonyms.py", line 57, in get_synonym_group_index
    return self.__get_group_index(value)
  File "/media/nicolay/96ed6537-b931-4f7e-8ac4-8407527ddbf9/proj/REmarker/venv/lib/python3.6/site-packages/arekit/common/synonyms.py", line 130, in __get_group_index
    return self.__by_sid[sid]
KeyError: 'китай'

Possible Solution: Considering Synoyms Collection Expansion!

https://docs.deeppavlov.ai/en/0.11.0/apiref/models/bert.html#deeppavlov.models.bert.bert_classifier.BertClassifierModel

__perform_annotation at AREkit (check)

https://github.com/nicolay-r/AREkit/blob/b1a9ef4c42d4d10c27ac051947b50a22e4f9b3f2/arekit/contrib/networks/core/input/helper.py#L115-L123

• setup constants
• provide batch-size as a parameter

related to nicolay-r/AREnets#2

rusenttokenize

reason: this project is dedicated to the processing of a single file or a list of files. Hence there is a need to exclude collections.

Fix readme as well.

This is not functionality of 0.22.1

Can you explain a little bit what kind of data and which kind of models need to be available in a language to apply your framework?

Present limitations:

• Focused on the neural networks.
• Frames annotation is hidden
• Embedding -- only related to neural networks.

"А не хочет проблем с Б"

TODO:
Use NetworksInputSerializerExperimentIteration from AREkit
The latter allows to adopt:
BertExperimentInputSerializerIterationHandler

This case should not happen by default, but the latter might be happen once we adopt it for other type.

See the reason why we head to detach this contrib part from AREkit here

Hi,

The link from the documentation to nicolay-r-arelight-0.1.1.tar appears to be broken. It returns:

404. That’s an error.

The requested URL was not found on this server. That’s all we know.