Hello,

I notice that pytorch and tensorflow both have a topk function, which can be faster than a full sort if k is small. With numpy, it can be emulated using (arg)partition. I think it would be convenient if eagerpy provided a wrapper for this, I have 2 unrelated pieces of code where I would be likely to use it.

(of course it is disappointing that tensorflow does not seem to support the argument dim, but doing some transposition around the call is easy enough)

I'm considering packaging EagerPy for Debian. I think that effort may encounter some hurdles due to missing copyright notices and unclear licensing. As far as I can tell (please forgive me if I've overlooked something), the license is only specified in setup.py, and even there just given as "MIT License". It would be of great help if you could add a LICENSE fle where you clearly state your copyright over the code and the full licensing terms (for example the full terms of the MIT license).

It would be even more helpful if a copyright notice were added to each source file, but I understand if this is too much of a hassle.

i'm trying to modify foolbox like following,

from typing import TypeVar, Any
from abc import ABC, abstractmethod

import torch
import eagerpy as ep
from foolbox.criteria import Criterion

class TargetedMisclassificationML(Criterion):
    """Considers those perturbed inputs adversarial whose predicted class
    matches the target classes. Multi-Label

    Args:
        target_classes: Tensor with target classes ``(batch,)``.
    """

    def __init__(self, target_classes: Any):
        super().__init__()
        self.target_classes: ep.Tensor = ep.astensor(target_classes)

    def __repr__(self) -> str:
        return f"{self.__class__.__name__}({self.target_classes!r})"

    def __call__(self, perturbed: T, outputs: T) -> T:
        outputs_, restore_type = ep.astensor_(outputs)
        del perturbed, outputs
        print(type(outputs_), type(outputs))
        # classes = outputs_.argmax(axis=-1)
        classes = torch.tensor(torch.round(torch.sigmoid(outputs_)).detach().numpy().tolist())  # ERR
        assert classes.shape == self.target_classes.shape
        is_adv = classes == self.target_classes
        return restore_type(is_adv)

will u support sigmoid function in the future?
and how to implement sigmoid with eagerpy now?

In order to simplify the writting of universal functions it could be great to have a decorator function which hide the technical part of the code (convertion of input and output of the wrapped function/method).
For example, the code:

def my_universal_function(a, b, c):
    # Convert all inputs to EagerPy tensors
    a, b, c = ep.astensors(a, b, c)

    # performs some computations
    result = (a + b * c).square()

    # and return a native tensor
    return result.raw

would become:

@eager_function
def my_universal_function(a, b, c):
    return (a + b * c).square()

In addition, we could add the feature that if the input tensors are already eagerpy tensors, then no convertion to raw format should done on the output tensors.

I wrote a prototype of such a decorator function. It should not work on any type of arguments and so its usage would require that the wrapped function has a rather "simple" signature (with args and kwargs constituted of tensors or nested containers with tensors on leaves: dict, list, tuple or namedtuple like containers).

Would you consider to have this feature in eagerpy?

Hi, jonasrauber

Thanks for your works!

I wonder if it will add support for SparseTensor (Tensorflow or Pytorch), like this:

import numpy as np
import tensorflow as tf
import eagerpy as ep

x = np.array([[0, 1, 0],
              [0, 0, 1],
              [1, 0, 1]])

# Tensorflow SparseTensor
sp_x = tf.sparse.from_dense(x)

# eagerpy
# is it implemented?
sp_x = ep.assptensor(x)

Feature Request

Would is be possible to support python scalars?

eg.

num_float = ep.astensor(1.0)
num_int = ep.astensor(42)

Why?

This would be helpful for implementing generic functions, for example:

def get_kernel_size(sigma=1.0, trunc=4.0):
    sigma, trunc = ep.astensors(sigma, trunc)
    radius = (sigma * trunc + 0.5).astype(int)
    return (2 * radius + 1).raw

This could be called with default values as well as tensors.

t_sigma = torch.abs(torch.randn(10))
n_trunc = np.abs(np.random.randn(10))

get_kernel_size(sigma=1.0, trunc=4.0)
get_kernel_size(sigma=t_sigma, trunc=4.0)
get_kernel_size(sigma=1.0, trunc=n_trunc)

I'm trying to differentiate through predict_fn provided by https://github.com/google/neural-tangents. This is doable with jax.grad, but not with eagerpy.value_and_grad.

I'm getting this error in /eagerpy/astensor.py", line 56, in astensor:
raise ValueError(f"Unknown type: {type(x)}")
ValueError: Unknown type: <class 'tuple'>

How to resolve the issue.

It could be great to have support for the @ operator.

Given two tensors:

a = ep.astensor(np.random.normal(size=(2, 10)))
b = ep.astensor(np.random.normal(size=(10, 3)))

The expression:

c = a.matmul(b)

could be written as:

c = a @ b

When one of the two conditions is a float64 dtype and the second a number (int, float), where() method raises an error:

RuntimeError: expected scalar type float but found double

TensorFlow's tensor_scatter_nd_update expects the values to be of the same type as the tensor. Currently, this leads to TensorFlowTensor.index_update failing since it uses tf.fill, which produces int32/ float32 tensors, independent of the tensor data type itself.

Reproducible example:

import eagerpy as ep
import tensorflow as tf

x_int32 = ep.astensor(tf.range(4, dtype=tf.int32))
x_int64 = ep.astensor(tf.range(4, dtype=tf.int64))
indices = (ep.astensor(tf.constant([0, 2])),)

x_int32.index_update(indices, 0)  # this works
x_int32.index_update(indices, 0.0)  # this fails
x_int64.index_update(indices, 0)  # this fails
x_int64.index_update(indices, 0.0)  # this fails

Similar for float32/ float64 tensors.

Solution:

Cast values to the dtype of the raw tensor.

I noticed that the method index_update seems very slow for tensorflow backend when used in compiled code (a function wrapped with tf.function).

It could be worth to see how we could enhance the current implementation.

Hello,
with a pytorch tensor t, I can call t.norm(p, dim). This gives a similar result to eagerpy's lp, but makes a huge difference when it comes to the gradient. Pytorch has this feature where deriving sqrt in 0 gives infinity (mathematically sensible), which often yields a gradient of NaN. However, special functions like norm are handled specially, similarly to abs, and return a suitable subgradient (0).
Could you please make l2/lp/... call norm for the pytorch backend?

I think it would be good to have an eagerpy version of the nonzero() and flatnonzero() functions. I guess these can be imitated with a combination of arange and indexing, but it would be more convenient to have a simple drop-in replacement for these two common functions.

will this lib support tensorflow tensor work with cuda.jit (from numba).

Hi! Thanks for the awesome project -- a unified tensor interface is something that will help a lot of projects :)

I wanted to ask about the current/future scope of eagerpy, specifically on the inclusion of probability distributions like in torch.distributions, or tfp.distributions? It seems like a more substantial project, and so may be more of a stand-alone effort, but I think this would be a great asset for people who want to keep fully agnostic to frameworks.

As an example, a project I work with (pyhf) has it's own implementation of Tensor that aims for essentially what eagerpy does, but we also use it for probability distributions too within the module. This is done by wrapping around existing implementations and adding the extra math where needed -- my first impression with eagerpy is that this may be better handled if there were stand-alone implementations of distributions using only eagerpy tensors, but I'm not sure what the more practical option is.

Eager (hehe) to hear your thoughts!

First, congrats on such a great project!

Basically the title. PyTorch and Tensorflow both support cross entropy in the ND case, and your implementation in Numpy would work for the multi-dimensional case too. However, in every function, there is an assert that the logits are 2D array. I propose to remove those asserts 😄

eagerpy seems like a great project, and after an upcoming deadline we might try porting out pytorch code to it.

Reading here about converting native pytorch and tf GPU tensors to eager tensors, could you please implement a ep.totensor method, creating a pytorch or tf GPU tensor?

We currently have to use a mix of pytorch and tensorflow models to evaluate our code :(

And yes, I can get them all running on the same GPU. I just want to avoid shuttling pytorch tensors to CPU so I can evaluate them in TF models.

Is there an equivalent of np.diag in eagerpy? If not, what would be the proper way to create a diagonal matrix from a vector?

Converting between things like float32/64/int32/64 would be a really nice addition!

Would you consider to add or accept a contribution to add and implementation of slogdet function (see for ex the numpy implementation)?

Let's consider a simple compiled function in tensorflow.

import tensorflow as tf
a = tf.random.normal(shape=(2, 10))
b = tf.random.normal(shape=(10, 3))

@tf.function
def tf_compiled_func(a, b):
    c = tf.matmul(a, b)
    return c

tf_compiled_func(a, b)

This bunch of code works.

However, its "universal" version :

@tf.function
def compiled_universal_func(a, b):
    a, b = ep.astensors(a, b)
    c = a.matmul(b)
    return c.raw


a = tf.random.normal(shape=(2, 10))
b = tf.random.normal(shape=(10, 3))
compiled_universal_func(a, b)

does not work and raises the error:

.../lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
    975           except Exception as e:  # pylint:disable=broad-except
    976             if hasattr(e, "ag_error_metadata"):
--> 977               raise e.ag_error_metadata.to_exception(e)
    978             else:
    979               raise

AttributeError: in user code:

    <ipython-input-8-6edbe80953ee>:4 compiled_universal_func  *
        c = a.matmul(b)
    .../lib/python3.7/site-packages/eagerpy/tensor/tensorflow.py:499 matmul  *
        if self.ndim != 2 or other.ndim != 2:
    .../lib/python3.7/site-packages/eagerpy/tensor/base.py:115 ndim
        return cast(int, self.raw.ndim)

    AttributeError: 'Tensor' object has no attribute 'ndim'

(but it works if we comment the @tf.function)

Let's notice that the equivalent thing with jax seems to work:

import jax
from jax import jit

@jit
def compiled_universal_func(a, b):
    a, b = ep.astensors(a, b)
    c = a.matmul(b)
    return c.raw


seed = 1701
key = jax.random.PRNGKey(seed)
a = jax.random.normal(shape=(2, 10), key=key)
b = jax.random.normal(shape=(10, 3), key=key)
compiled_universal_func(a, b)

Is it a problem with the integration of eagerpy with tensorflow ?