Giter VIP home page Giter VIP logo

dpu-utils's Introduction

DPU Utilities PyPI - Python VersionAnaconda

Build Status

This contains a set of utilities used across projects of the DPU team.

Python

Stored in the python subdirectory, published as the dpu-utils package.

Installation

pip install dpu-utils

OR via the community-maintained Conda recipe:

conda install -c conda-forge dpu-utils

Overview

Below you can find an overview of the utilities included. Detailed documentation is provided at the docstring of each class.

Generic Utilities dpu_utils.utils
  • ChunkWriter provides a convenient API for writing output in multiple parts (chunks).
  • RichPath an API that abstract local and Azure Blob paths in your code.
  • *Iterator Wrappers that can parallelize and shuffle iterators.
  • {load,save}_json[l]_gz convenience API for loading and writing .json[l].gz files.
  • git_tag_run tags the current working directory git the state of the code.
  • run_and_debug when an exception happens, start a debug session. Usually a wrapper of __main__.
General Machine Learning Utilities dpu_utils.mlutils
  • Vocabulary map elements into unique integer ids and back. Commonly used in machine learning models that work over discrete data (e.g. words in NLP). Contains methods for converting an list of tokens into their "tensorized" for of integer ids.
  • BpeVocabulary a vocabulary for machine learning models that employs BPE (via sentencepiece).
  • CharTensorizer convert character sequences into into tensors, commonly used in machine learning models whose input is a list of characters.
Code-related Utilities dpu_utils.codeutils
TensorFlow 1.x Utilities dpu_utils.tfutils

Unsorted segment operations following TensorFlow's unsorted_segment_sum operations:

TensorFlow 2.x Utilities dpu_utils.tf2utils

Unsorted segment operations following TensorFlow's unsorted_segment_sum operations:

TensorFlow Models dpu_utils.tfmodels

These models have not been tested with TF 2.0.

PyTorch Utilities dpu_utils.ptutils
  • BaseComponent a wrapper abstract class around nn.Module that takes care of essential elements of most neural network components.
  • ComponentTrainer a training loop for BaseComponents.

Command-line tools

Approximate Duplicate Code Detection

You can use the deduplicationcli command to detect duplicates in pre-processed source code, by invoking

deduplicationcli DATA_PATH OUT_JSON

where DATA_PATH is a file containing tokenized .jsonl.gz files and OUT_JSON is the target output file. For more options look at --help.

An exact (but usually slower) version of this can be found here along with code to tokenize Java, C#, Python and JavaScript into the relevant formats.

Tests

Run the unit tests

python setup.py test

Generate code coverage reports

# pip install coverage
coverage run --source dpu_utils/ setup.py test && \
  coverage html

The resulting HTML file will be in htmlcov/index.html.

.NET

Stored in the dotnet subdirectory.

Generic Utilities:

  • Microsoft.Research.DPU.Utils.RichPath: a convenient way of using both paths and Azure paths in your code.

Code-related Utilities:

  • Microsoft.Research.DPU.CSharpSourceGraphExtraction: infrastructure to extract Program Graphs from C# projects.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

dpu-utils's People

Contributors

asvyatkovskiy avatar azure-pipelines[bot] avatar jdayton3 avatar mallamanis avatar mayank-sfdc avatar microsoft-github-policy-service[bot] avatar mmjb avatar rishad-sayone avatar sarthakpati avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dpu-utils's Issues

installation failed which caused by "azure-storage" package

There's something wrong when I try to install dpu-utils both by pip or conda.

Here's the error log.

Downloading http://mirrors.aliyun.com/pypi/packages/5d/cd/cac628adf6cb0086a24c20d76609445f1bf841483d56ec9f5645bbaa9ab7/azure-storage-0.37.0.zip (4.3 kB)
    ERROR: Command errored out with exit status 1:
     command: /home/lawbda/envs/anaconda3/envs/semanCode/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-nki771xo/azure-storage/setup.py'"'"'; __file__='"'"'/tmp/pip-install-nki771xo/azure-storage/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-nki771xo/azure-storage/pip-egg-info
         cwd: /tmp/pip-install-nki771xo/azure-storage/
    Complete output (19 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-nki771xo/azure-storage/setup.py", line 55, in <module>
        raise RuntimeError(message)
    RuntimeError:
    
    Starting with v0.37.0, the 'azure-storage' meta-package is deprecated and cannot be installed anymore.
    Please install the service specific packages prefixed by `azure` needed for your application.
    
    The complete list of available packages can be found at:
    https://aka.ms/azsdk/python/all
    
    Here's a non-exhaustive list of common packages:
    
    - [azure-storage-blob](https://pypi.org/project/azure-storage-blob) : Blob storage client
    - [azure-storage-file-share](https://pypi.org/project/azure-storage-file-share) : Storage file share client
    - [azure-storage-file-datalake](https://pypi.org/project/azure-storage-file-datalake) : ADLS Gen2 client
    - [azure-storage-queue](https://pypi.org/project/azure-storage-queue): Queue storage client
    
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

It's seems that new version of azure cause this issue.

pip install dpu-utils error

OS:window10
python: 3.5

when I run "pip install dpu-utils"

Collecting dpu-utils
Using cached dpu_utils-0.1.27-py2.py3-none-any.whl (39 kB)
Requirement already satisfied: tqdm in c:\users\v-ensh\anaconda3\envs\tf1\lib\site-packages (from dpu-utils) (4.50.2)
Requirement already satisfied: docopt in c:\users\v-ensh\anaconda3\envs\tf1\lib\site-packages (from dpu-utils) (0.6.2)
Collecting SetSimilaritySearch
Using cached SetSimilaritySearch-0.1.7-py2.py3-none-any.whl (14 kB)
Requirement already satisfied: numpy in c:\users\v-ensh\anaconda3\envs\tf1\lib\site-packages (from dpu-utils) (1.18.5)
Collecting azure-storage
Using cached azure-storage-0.37.0.zip (4.3 kB)

ERROR: Command errored out with exit status 1:
command: 'c:\users\v-ensh\anaconda3\envs\tf1\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\v-ensh\AppData\Local\Temp\pip-install-5cnwyk75\az
ure-storage\setup.py'"'"'; file='"'"'C:\Users\v-ensh\AppData\Local\Temp\pip-install-5cnwyk75\azure-storage\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file
);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\v-ensh\AppData\Local\Temp\pip-pip-egg-info-
ejvmcyn5'
cwd: C:\Users\v-ensh\AppData\Local\Temp\pip-install-5cnwyk75\azure-storage
Complete output (19 lines):
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\v-ensh\AppData\Local\Temp\pip-install-5cnwyk75\azure-storage\setup.py", line 55, in
raise RuntimeError(message)
RuntimeError:

Starting with v0.37.0, the 'azure-storage' meta-package is deprecated and cannot be installed anymore.
Please install the service specific packages prefixed by `azure` needed for your application.

The complete list of available packages can be found at:
https://aka.ms/azsdk/python/all

Here's a non-exhaustive list of common packages:

- [azure-storage-blob](https://pypi.org/project/azure-storage-blob) : Blob storage client
- [azure-storage-file-share](https://pypi.org/project/azure-storage-file-share) : Storage file share client
- [azure-storage-file-datalake](https://pypi.org/project/azure-storage-file-datalake) : ADLS Gen2 client
- [azure-storage-queue](https://pypi.org/project/azure-storage-queue): Queue storage client

----------------------------------------

ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output

I run python setup.py egg_info
this is log
OpenGNN.egg-info.zip

Sentencepiece dependency

Hi!

I tried to install dpu-utils for python but it crashed due to an incorrect version of sentencepiece. I see in your setup.py strong equality to 0.1.92 version, but it was removed to some bugs. Is it possible to use a newer version of that package? Currently, I can install only 0.2.20 version of dpu-utils :)

Outdated azure-storage version -> incompatible with newer azure-blob-storage

We encountered some issues with incompatible versions of azure-blob-storage required by dpu_utils, e.g. with packages like mlflow.

Overwritting the azure-blob-storage version required by dpu_utils leads to the following error:

ImportError: cannot import name 'BlobClient' from 'azure.storage.blob' (/home/klingest/miniconda3/envs/ds-mlflow/lib/python3.8/site-packages/azure/storage/blob/__init__.py)

This could be fixed locally for us by replacing: https://github.com/microsoft/dpu-utils/blob/master/python/dpu_utils/utils/richpath.py#L22
with
from azure.storage.blob.blockblobservice import BlockBlobService

Is this a fix that should perhaps go into the master?
Since it looks like there is not much more dependencies in the package of the "old" azure-storage package: https://github.com/microsoft/dpu-utils/search?q=azure as well as the deprecation of azure-storage, from my understanding

Unit test utility functions

I'm more likely to use utility functions if I know they've been unit tested. I'm wondering:

  • Would your team welcome a pull request to start unit testing?
  • If so, what testing framework would you prefer? (I'm most comfortable with Python's builtin unittest framework if there's no preference.)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.