persiaml / persia Goto Github PK

like https://github.com/horovod/horovod#horovod-timeline

use
https://crates.io/crates/opentelemetry-jaeger
https://docs.rs/opentelemetry/0.16.0/opentelemetry/

Hello，Could you tell us how to generate your criteo_syn dataset those used in your experiment?

Generate the e2e test from PERSIA/examples repo. It could reduce the code redundant since most of the e2e test code is similar with examples code.

Default 1MB may not be enough for large batch of samples.

https://stackoverflow.com/questions/55368487/send-a-message-larger-than-1mb-using-nats-streaming

Add arguments to TrainCtx to continue training

Add the EmbeddingModel class to manage the current embedding tensors. It is the torch.nn.Module which instantiates by embedding_config.yml.

Features following:

• The EmbeddingModel can be initialized from embedding_config.yml.
• Support more attention operation on attention_embedding_tensor, such as mean, max.
• Defining embedding initialization method in embedding_config.yml.
• Abstracted attention_embedding and raw_embedding into Embedding Python class. It can make the EmbeddingModel more interpretable to users, especially in forward and backward phases.

I need to preprocess the dataset (say Criteo) into a bunch of npy files? or other data formats?

Thanks!

Hi, I am using PERSIA to test on my GPU server and there's some error.
I have tried three methods to use it.

1. Directly run docker image as docker-compose -f docker-compose.yml up
   Encountered with ERROR: Couldn't find env file: /xx/xx/persia/examples/docker-compose/.env

2. Using make command as/xx/xx/persia/examples/docker-compose$ make run
   Encountered with open /xx/xx/persia/examples/docker-compose/.env: no such file or directory

3. I even tried to run train.py manually but also failed as ENABLE_CUDA=1 python train.py
   I set the rank, device_id, world_size manually and torch.cuda.is_available() return True
   Encountered with
   2021-11-27 02:48:45,834 INFO [train.py:35] test dataset size is 128 Traceback (most recent call last): File "train.py", line 116, in <module> with TrainCtx( File "/opt/conda/lib/python3.8/site-packages/persia/ctx.py", line 619, in __init__ super(TrainCtx, self).__init__(PreprocessMode.TRAIN, *args, **kwargs) File "/opt/conda/lib/python3.8/site-packages/persia/ctx.py", line 224, in __init__ super(EmbeddingCtx, self).__init__(*args, **kwargs) File "/opt/conda/lib/python3.8/site-packages/persia/ctx.py", line 109, in __init__ self.common_context = PersiaCommonContext( OverflowError: can't convert negative int to unsigned
   It's quite weird.

Is there any suggestion or missing documents?
Thank you!

• Process PersiaTrainingBatch more handy and readable.
• Combine pytorch dataset, auto download recommendation dataset.

lack of docs:

• https://persiaml.readthedocs.io/en/latest/autoapi/persia/sparse/index.html
• https://persiaml.readthedocs.io/en/latest/autoapi/persia/ctx/index.html
• https://persiaml.readthedocs.io/en/latest/autoapi/persia/env/index.html
• https://persiaml.readthedocs.io/en/latest/autoapi/persia/logger/index.html
• https://persiaml.readthedocs.io/en/latest/autoapi/persia/wrapper/index.html

some of them should be hidden, such as persia.wrapper.logger, since users are not expected to use them.

Each feature has a certain probability to be created, and the probability is accumulated and calculated according to the positive and negative sample probabilities of the feature, the positive sample probability create _ CLK _ prob and the negative sample probability create _ nonclk _ prob of a common slot, Special slots (slots specified in select _ prob _ slots) are calculated according to create _ CLK _ prob and create _ nonclk _ prob in select _ prob _ slot

Documentation: @karoka

• Tutorials
  • Introduction
  • Installation @williamstar
  • Getting Started @williamstar
  • Benchmark (experiments * 3)
    • XDL @snowpeakz
    • Paddle @snowpeakz
  • Data processing @snowpeakz
  • Monitoring @snowpeakz
  • Inference @snowpeakz
  • Model checkpointing @williamstar
  • Configuration @snowpeakz
  • Troubleshooting @williamstar
  • Kubernetes Integration @williamstar
  • Advanced: @williamstar
    • Raw embedding
    • HashStack
    • Separate data loader
• API documentation @williamstar https://persiaml.pages.dev/
• add system test (with buildkite)
  • train, test (assert number of samples, train and test auc etc.) @williamstar
  • load, dump @snowpeakz
  • inference @snowpeakz
  • move e2e test script to examples @williamstar

Bug:

• persia-core PyForward && PyBackward thread handler exit with exceptions
• insert embedding in offline inference phase

Feature:

• python launcher @williamstar
• cpu backward with gloo backend @williamstar
• ddp launch with tcp to replace env_file @snowpeakz
• support epoch level dataloader @williamstar
• easy pip install persiaml @williamstar
• unified storage trait, saving both dense & sparse parameters @snowpeakz
• support s3 @williamstar

Experiments:

• Single card + force enable communication
• Change figure titles

Release

• merge server code with python code @NOBLES5E
• GitHub releases @williamstar
• all persia container images @williamstar
• kubernetes operator @williamstar

• NATS integration @snowpeakz
• multi-node multi-card usable demo @williamstar
• fix remaining bugs (like in data processing) @williamstar
• publish PyPI wheels on CI @NOBLES5E
• API documentation website @NOBLES5E

Since we are using batch level arguments is_training, the intent in global_config.yaml is no longer needed.

Currently, the tutorial website in README points to tutorial pages, with a footer of "Found a bug? Edit this file on GitHub."

However, the link should break and end up with a 404 because of the private repository visibility of the tutorials repo from the public view, which is expected to be fixed.

• buildkite system test
  • add shutdown API to exit all services normally
  • use buildkite step to start training and services, use predefined port to communicate between components
  • trainer determines whether the results are correct
  • pass CI
• pytest

🤔️ 不太熟悉rust，求赐教～

Reduce half of embedding model size to scale up the model ability

Dataloader stop iteration by nats server signal rather than TimeoutException

请问有什么解决方案吗？是我配置还是哪里的原因吗？
有一点区别就是我没有sudo权限， 不能把nats-server放到/usr/bin，而是在/home下面unzip了release build，然后改的Procfile里nats-server的启动路径，这一步有影响吗？

1. feature score
   Each feature has a feature _ score that is calculated a
   feature_score=clk_coeffpos_ins_num+nonclk_coeffneg_ins_num
   On push, feature _ score is always accumulated
2. time_decay
   End _ day (sample time) will trigger time _ decay
   Time _ decay, the feature _ score is attenuated by CVM _ plugin.decay _ ratio
3. shrink_table
   After time _ decay, or when feature _ num exceeds Max _ features during training, shrink _ table is triggered
   If one of the conditions is met, it is deleted:
   ● score < _delete_threshold
   ● value.unseen_days > _delete_after_unseen_days
   ● _ select _ prob _ slot _ set.get (value.slot) = = 1 & & score < _ select _ delete _ threshold/Id Group Id type feature, increase delete threshold
   ● _ photoid _ slot _ set.get (value.slot) = = 1 & & value.unseen _ days > 2/Delete after two days of combined photo _ ID feature
   If feature _ num is still greater than Max _ features after deletion, it will be deleted from small to large according to feature score until feature _ num < Max _ features is satisfied

It seems that EmbeddingWorkerNatsServicePublisher is quite important to maintain the communication among data context, embedding servers/workers and nn workers. But I couldn't find the exact file for this class. Could you please tell where the class is defined and implemented?

#181 (comment)

Expected to provide the shutdown interface for PyForward and PyBackward to ensure the thread handler exited normally

We need every server dump and load its own model in parallel by default

If number of shards changes, then use a load service to load files and dynamically insert entries to the servers.

I adopted Persia to implement a DLRM and run it over Criteo Kaggle dataset.
I set batch size to 1024, and below is the content of docker-compose.yml:

version: "3.2"
services:
  persia_nats_service:
    image: nats:latest
    deploy:
      replicas: 1

  data_loader1:
    env_file:
      - .docker.env
    depends_on:
      - nn_worker
      - embedding_worker
      - persia_nats_service
    image: persia-dlrm
    command: persia-launcher data-loader --replica-index 0 --replica-size 2
    volumes:
      - type: bind
        source: .
        target: /workspace
      - type: bind
        source: ../criteo_kaggle
        target: /workspace/criteo_kaggle
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure

  data_loader2:
    env_file:
      - .docker.env
    depends_on:
      - nn_worker
      - embedding_worker
      - persia_nats_service
    image: persia-dlrm
    command: persia-launcher data-loader --replica-index 1 --replica-size 2
    volumes:
      - type: bind
        source: .
        target: /workspace
      - type: bind
        source: ../criteo_kaggle
        target: /workspace/criteo_kaggle
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure

  nn_worker:
    env_file:
      - .docker.env
    environment:
      NCCL_SOCKET_IFNAME: eth0
      CUBLAS_WORKSPACE_CONFIG: :4096:8
    image: persia-dlrm
    command: persia-launcher nn-worker --nproc-per-node 1 --nnodes 1 --node-rank 0
    volumes:
      - type: bind
        source: .
        target: /workspace
      - type: bind
        source: ../criteo_kaggle
        target: /workspace/criteo_kaggle
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure

  embedding_worker1:
    env_file:
      - .docker.env
    depends_on:
      - embedding_parameter_server
    image: persia-dlrm
    command: >
      bash -c "persia-launcher embedding-worker --embedding-config $$PERSIA_EMBEDDING_CONFIG
      --global-config $$PERSIA_GLOBAL_CONFIG --replica-index 0 --replica-size 2"
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    volumes:
      - type: bind
        source: .
        target: /workspace
      - type: bind
        source: ../criteo_kaggle
        target: /workspace/criteo_kaggle

  embedding_worker2:
    env_file:
      - .docker.env
    depends_on:
      - embedding_parameter_server
    image: persia-dlrm
    command: >
      bash -c "persia-launcher embedding-worker --embedding-config $$PERSIA_EMBEDDING_CONFIG
      --global-config $$PERSIA_GLOBAL_CONFIG --replica-index 1 --replica-size 2"
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    volumes:
      - type: bind
        source: .
        target: /workspace
      - type: bind
        source: ../criteo_kaggle
        target: /workspace/criteo_kaggle

  embedding_parameter_server1:
    env_file:
      - .docker.env
    image: persia-dlrm
    command: >
      bash -c "persia-launcher embedding-parameter-server --embedding-config $$PERSIA_EMBEDDING_CONFIG
      --global-config $$PERSIA_GLOBAL_CONFIG --replica-index 0 --replica-size 2"
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    volumes:
      - type: bind
        source: .
        target: /workspace
      - type: bind
        source: ../criteo_kaggle
        target: /workspace/criteo_kaggle

  embedding_parameter_server2:
    env_file:
      - .docker.env
    image: persia-dlrm
    command: >
      bash -c "persia-launcher embedding-parameter-server --embedding-config $$PERSIA_EMBEDDING_CONFIG
      --global-config $$PERSIA_GLOBAL_CONFIG --replica-index 1 --replica-size 2"
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    volumes:
      - type: bind
        source: .
        target: /workspace
      - type: bind
        source: ../criteo_kaggle
        target: /workspace/criteo_kaggle

Here's a screenshot of the running process

As you can see, the throughput is about 30 it/s. Since the batch size is 1024, the throughput is only half of the results reported in your paper.
I also notice that the logger kept warning that the local forwarded queue is empty, and these processes didn't cost any GPU memory.
Is there any problem about my settings or, do you guys have any suggestions on how to improve the throughput?

#204

where each embedding vector shares a single float grad running avg