Is it possible to include weighting criteria as an argument to the function.
This will be helpful in Weighted Correlation.
Reference: https://github.com/matthijsz/weightedcorr

• Add FAQ, contributers, purpose of project, etc.

loss.backward()

File "E:\conda\lib\site-packages\torch_tensor.py", line 307, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "E:\conda\lib\site-packages\torch\autograd_init_.py", line 154, in backward
Variable._execution_engine.run_backward(
RuntimeError: Function NumpyOpWrapperBackward returned an invalid gradient at index 0 - expected type TensorOptions(dtype=float, device=cuda:0, layout=Strided, requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)) but got TensorOptions(dtype=float, device=cpu, layout=Strided, requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt))

the forward part of the loss function is as follows：

def forward(self, preds, gt):
preds_rank = torch_ops.soft_rank(preds.unsqueeze(0)).float()
gt_rank = torch_ops.soft_rank(gt.unsqueeze(0)).float()

Looking forward to your reply.

Hi, thank you for the efforts. I want to use fast_soft_sort in fp16 setting. It seems that fast_soft_sort doesn't support fp16 yet? Looking forward to your reply!

I am trying to plugin this layer in Keras model. But it is failing with the exception

*** TypeError: Could not build a TypeSpec for KerasTensor(type_spec=TensorSpec(shape=(None, 12), dtype=tf.float32, name=None), name='dense_1/BiasAdd:0', description="created by layer 'dense_1'") of unsupported type <class 'keras.engine.keras_tensor.KerasTensor'>.

Code to reproduce this issue

import tensorflow as tf
from fast_soft_sort.tf_ops import soft_sort

all_inputs = tf.keras.layers.Input(shape=(12))
x = tf.keras.layers.Dense(12)(all_inputs)
x = soft_sort(x)

model = tf.keras.Model(inputs=all_inputs, outputs=[x])

@mblondel

That's because you didn't add the thirdparty module explicitly in the setup, or alternatively add a init.py in the thirdparty folder to make it a subpackage.

I'll send a PR with the second solution.

Adrien

Hi Authors,
Thank you for releasing your code. I tried checking the numerical gradients in Pytorch. With the soft_sort module, I do not obtain the gradients after doing loss.backward(). However, when I do not use the soft_sort module, I am able to obtain the gradients. Below are the code snippets for the two situations

With the soft_sort module

import torch
import torch.nn as nn
from fast_soft_sort.pytorch_ops import *

conf = torch.tensor([[0.1, 0.7, 0.2]], dtype=torch.float64, requires_grad= True)

# Sorting in descending order
conf = soft_sort(conf, direction="DESCENDING", regularization_strength= 1)

ideal = torch.tensor([[1.0, 0.0, 0]], dtype=torch.float64)
l2    = nn.MSELoss()

print("Grad before doing loss.backward()")
print(conf.grad)

loss = l2(conf, ideal)
loss.backward()

print("Loss = {:.2f}".format(loss))
print("Grad after loss.backward()")
print(conf.grad)

, the outputs are

Grad before doing loss.backward()
None
Loss = 0.05
Grad after loss.backward()
None

Without soft_sort

import torch
import torch.nn as nn
from fast_soft_sort.pytorch_ops import *

conf = torch.tensor([[0.1, 0.7, 0.2]], dtype=torch.float64, requires_grad= True)

# Sorting in descending order
# conf = soft_sort(conf, direction="DESCENDING", regularization_strength= 1)

ideal = torch.tensor([[1.0, 0.0, 0]], dtype=torch.float64)
l2    = nn.MSELoss()

print("Grad before doing loss.backward()")
print(conf.grad)

loss = l2(conf, ideal)
loss.backward()

print("Loss = {:.2f}".format(loss))
print("Grad after loss.backward()")
print(conf.grad)

, the outputs are

Grad before doing loss.backward()
None
Loss = 0.45
Grad after loss.backward()
tensor([[-0.6000,  0.4667,  0.1333]], dtype=torch.float64)

Hi,

When using soft sorting or ranking with a 0 regularization, for the sake of convenience you should probably fall back to the normal sorting and ranking algorithm. Not sure how to do this in a clean way (that doesn't involve having soft stuff subclassing normal stuff).

In the meantime maybe you should raise at the init level if the strength (same as you did with the regularization type)?

Copy pasting is such a 90s thing

I find the code below in tf_ops.py
" if len(values.shape) != 2:
raise ValueError("'values' should be a 2d-tensor "
"but got %r." % values.shape)

assert tf.executing_eagerly()
"
so the soft_rank could not run in eager mode in tf2.0?
why?

Hello,

What would be the difference between the provided implementation (soft-sort) and the torch.sort version. Sorry for the stupid question, but I am not able to see how the torch sort non-differentiable is really different to the soft sort.

import torch.nn.functional as F
import torch
import torch.nn as nn
import pytorch_ops
import numpy as np

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer = nn.Linear(20, 20)
        self.out = nn.Linear(20,20)

    def forward(self, x):
        out = F.relu(self.layer(x))
        out = pytorch_ops.soft_sort(self.out(out)).float()
        return out

class Net2(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer = nn.Linear(20, 20)
        self.out = nn.Linear(20,20)

    def forward(self, x):
        out = F.relu(self.layer(x))
        out = torch.sort(self.out(out))[0]
        return out

targets = torch.rand(32,20)
inputs = torch.rand(32,20)
net = Net()
net2 = Net2()

#try with soft-sort
loss = criterion(targets, net(inputs))
loss.backward()

#try with torch.sort
loss = criterion(targets, net2(inputs))
loss.backward()

#both work!

Hey,
We've been trying to use this code in catboost and got the following error:
RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
from this line:
torch.from_numpy(ctx.numpy_obj.vjp(grad_output.numpy()))

If we'll change to tensor.detach the error will resolved but may broke something else within pytorch.

Any idea on how to solve it?

Thanks!

loss.backward()

File "E:\conda\lib\site-packages\torch_tensor.py", line 307, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "E:\conda\lib\site-packages\torch\autograd_init_.py", line 154, in backward
Variable._execution_engine.run_backward(
RuntimeError: Function NumpyOpWrapperBackward returned an invalid gradient at index 0 - expected type TensorOptions(dtype=float, device=cuda:0, layout=Strided, requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt)) but got TensorOptions(dtype=float, device=cpu, layout=Strided, requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt))

the forward part of the loss function is as follows：

def forward(self, preds, gt):
preds_rank = torch_ops.soft_rank(preds.unsqueeze(0)).float()
gt_rank = torch_ops.soft_rank(gt.unsqueeze(0)).float()

Looking forward to your reply.

Thanks for the work on this! Here's an example:

import jax.numpy as jnp
from fast_soft_sort.jax_ops import soft_rank
from jax import grad, jit


@jit
def f1(x):
    x = x.reshape(1, 3)
    y = soft_rank(x)[0]
    return y.mean()


x = jnp.array([1.0, 2.0, 3.0])
f2 = grad(f1)
f2(x)

Traceback (most recent call last):
  File "test.py", line 15, in <module>
    f2(x)
  File "test.py", line 9, in f1
    y = soft_rank(x)[0]
  File "/home/patrick/.pyenv/versions/3.8.5/lib/python3.8/site-packages/fast_soft_sort/jax_ops.py", line 80, in soft_rank
    return jnp.vstack([func(val) for val in values])
  File "/home/patrick/.pyenv/versions/3.8.5/lib/python3.8/site-packages/fast_soft_sort/jax_ops.py", line 80, in <listcomp>
    return jnp.vstack([func(val) for val in values])
  File "/home/patrick/.pyenv/versions/3.8.5/lib/python3.8/site-packages/fast_soft_sort/jax_ops.py", line 35, in _func_fwd
    values = np.array(values)
jax._src.traceback_util.FilteredStackTrace: Exception: The numpy.ndarray conversion method __array__() was called on the JAX Tracer object Traced<ShapedArray(float32[3])>wit$
<DynamicJaxprTrace(level=0/1)>.

This error can occur when a JAX Tracer object is passed to a raw numpy function, or a method on a numpy.ndarray object. You might want to check that you are using `jnp` toge$
her with `import jax.numpy as jnp` rather than using `np` via `import numpy as np`. If this error arises on a line that involves array indexing, like `x[idx]`, it may be tha$
 the array being indexed `x` is a raw numpy.ndarray while the indices `idx` are a JAX Tracer instance; in that case, you can instead write `jax.device_put(x)[idx]`.

Hi,

Thanks for the neat package but I have an issue regarding gradient flow using soft rank in PyTorch. Here is how to reproduce it:

In this example, Should the gradient of x be not all zeros? In comparison, the soft sort seems to work fine:

In addition, is it possible to add GPU support for soft rank in PyTorch?

Thanks,
Nio

I am toying with your library with some applications in mind. However it occured to me that what I would need is not sorting or ranking but argsort. I generally need the following (in numpy grammar):

y[np.argsort(x)]

From my understanding it wouldn't be possible... Am I missing something obvious ?

This is good.

Is there a way to configure the comparator function?

My plan was to have a neural network as the comparator function and it would learn from already sorted/ranked sequences.

Hi,

I tried the pytorch soft_rank function but the output was wrong. However the output of soft_sort was correct.

import torch
from fast_soft_sort import pytorch_ops

torch.manual_seed(12)
t = torch.rand((3, 4), dtype=torch.float64)
# tensor([[0.7255, 0.6218, 0.2692, 0.0983],
        [0.8495, 0.5939, 0.6640, 0.4471],
        [0.8322, 0.0047, 0.2805, 0.2584]], dtype=torch.float64)

torch.argsort(t)
# tensor([[3, 2, 1, 0],
        [3, 1, 2, 0],
        [1, 3, 2, 0]])

pytorch_ops.soft_sort(t, regularization_strength=0.1)
# tensor([[0.0983, 0.2692, 0.6218, 0.7255],
        [0.4471, 0.5939, 0.6640, 0.8495],
        [0.0047, 0.2584, 0.2805, 0.8322]], dtype=torch.float64)

torch.subtract(pytorch_ops.soft_rank(t, regularization_strength=0.1), torch.ones_like(t))
# tensor([[3.0000, 2.0000, 1.0000, 0.0000],
        [3.0000, 1.1493, 1.8507, 0.0000],
        [3.0000, 0.0000, 1.6105, 1.3895]], dtype=torch.float64)

Do you have an idea of how to fix it ?

Renaud

I was trying to use pytorch_ops.soft_sort and pytorch_ops.soft_rank on cuda tensors but it fails with the following error message:

TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

Here is a code snippet to reproduce the error

import torch
import torch.nn as nn
from fast_soft_sort import pytorch_ops

a = torch.tensor([[0.1, 0.7, 0.2]], dtype=torch.float64, requires_grad=True)
a = a.cuda()

b = torch.tensor([[1.0, 2.0, 3.0]], dtype=torch.float64)
b = b.cuda()

# Soft sort
a_sorted = pytorch_ops.soft_sort(a)

loss = nn.functional.mse_loss(a_sorted, b)
loss.backward()

print("Loss = {:.2f}".format(loss))
print(f"Grad = {a.grad}")

The cpu-version works just fine and produces the desired result.

I have been able to use the following work-around to do at least the loss computation on the GPU:

import torch
import torch.nn as nn
from fast_soft_sort import pytorch_ops

a = torch.tensor([[0.1, 0.7, 0.2]], dtype=torch.float64, requires_grad=True)

b = torch.tensor([[1.0, 2.0, 3.0]], dtype=torch.float64)
b = b.cuda()

# Soft sort
a_sorted = pytorch_ops.soft_sort(a).cuda()

loss = nn.functional.mse_loss(a_sorted, b)
loss.backward()

print("Loss = {:.2f}".format(loss))
print(f"Grad = {a.grad}")

But it's really cumbersome to manually move the tensors between the devices. Especially if the parameter a is part of a larger nn.Module and so a .cuda() call would move all parameters including a to the GPU.

Maybe the code could be extended to handle this logic internally?

Thanks for the great code base btw!