Comments (9)
Here's another failure: https://github.com/iree-org/iree/actions/runs/10530365837/job/29180735869#step:6:180
process = <Popen: returncode: 1 args: ['iree-run-module', '--device=hip', '--module=/h...>
stdout = b''
stderr = b"iree/runtime/src/iree/vm/bytecode/archive.c:106: INVALID_ARGUMENT; FlatBuffer data is not present or less than 16 by...iree_tests_cache/artifacts/sdxl_unet/model.rocm_gfx942.vmfb'; loading modules and dependencies; creating run context\n"
Maybe another run started compiling to the same path, overwriting/deleting the file that this run tried to use.
from iree.
Reading through the code more, I have a fix in mind... but it's going to be tricky to test.
We can keep using the ArtifactGroup
class, but change the directories used to point at two locations: the persistent cache for FetchedArtifact
and a local workspace for ProducedArtifact
. The locations of those can come in through environment variables.
Bugs are still possible if we change the remote file contents though. Using a cache implementation like huggingface's (git LFS) would be better - that downloads versions from git hashes and then creates symlinks into the refs.
from iree.
Alternate approach (closer to huggingface): always use a workspace-relative location, but create symlinks from the cache into that directory before the tests are run.
from iree.
that would be neat - especially if the cache paths are uniqued per hash such that you can be updating the cache with live symlinks to older versions concurrently (similar to what bazel does when linking from the sandbox to its storage)
from iree.
See what huggingface does here: https://huggingface.co/docs/huggingface_hub/en/guides/manage-cache
<CACHE_DIR>
├─ datasets--glue
│ ├─ refs
│ ├─ blobs
│ ├─ snapshots
...
They have tools for choosing where you want files to appear: https://huggingface.co/docs/huggingface_hub/en/guides/download#download-files-to-a-local-folder . The recommended options use symlinks with all the details about refs and blobs hidden from the user.
from iree.
Sketches of what a fix could look like here: https://github.com/iree-org/iree/compare/main...ScottTodd:infra-regression-suite-fix?expand=1. This is messy though - needs some deeper thinking. I also haven't tried running these tests locally - setup steps are complicated x_x.
We've been bouncing these tests back and forth between this repo and https://github.com/nod-ai/SHARK-TestSuite/ (https://github.com/iree-org/iree-test-suites could also be an option, if the Azure paths were replaced with something easier to modify for contributors). The repository-specific things are compile flags / spec files and expected dispatch counts / benchmark metrics.
from iree.
We can also just rework this a little bit: https://github.com/iree-org/iree/blob/main/experimental/regression_suite/ireers_tools/fixtures.py. Instead of making it output a produced artifact, it can just save the vmfb in the local dir. We basically don't use ProducedArtifact at all. I can get a fix out for that. Should be pretty easy and clean.
from iree.
Okay yeah, I think that sounds good.
from iree.
We probably do need a better longterm solution for the mlirs, weights changing in the persistent cache (huggingface seems good), but we can be strategic about when we update the mlirs and weights in Azure for now (also doesn't happen often)
from iree.
Related Issues (20)
- [compile][cpu]: type of return operand 0 ('!torch.vtensor<[?,384],f32>') doesn't match function result type ('!torch.vtensor<[1,384],f32>') in function
- [compile][cpu]: 'tensor.cast' op operand type 'tensor<?x64x114x112xf32>' and result type 'tensor<?x64x113x113xf32>' are cast incompatible HOT 2
- [numeric][cpu]: numeric error for ONNX Gather operator element at index 200 (0.420379) does not match the expected (0.642927); HOT 5
- Implement conflict resolution with a trip to shared memory for WMMA operations
- Bug with Dynamic dimensions in the case of `fuse Encoding ops into dispatch regions pass`. HOT 1
- Multi-device 2D convolution numerical imprecision HOT 13
- [onnx] Build real onnx frontend cli/API HOT 1
- [compile][cpu]: failed to legalize operation 'torch.operator' that was explicitly marked illegal(onnx.Gather)
- [compile][cpu]:'func.func' op exceeded stack allocation limit of 32768 bytes for function HOT 4
- [compile][cpu]:error: 'memref.alloca' op expected no unbounded stack allocations HOT 8
- [compiler] VM translation of Module to bytecode crashes HOT 2
- [compile][rocm/gfx942] flux sampler fails to lower with batch size > 1 with dominance error HOT 2
- [compile][gfx942] Attention+permute pattern regression after "Bubble expand shapes through `AttentionOp`s (#18074)" HOT 3
- [Flow] verify dispatches before entering codegen
- Re-enable llvm optimizations for AArch64 HOT 1
- `DispatchCreation` pipeline cleanup
- Seg fault for llama.8b.fp16.mlir model HOT 5
- Failure in `iree-vm-conversion` due to presence of a `hal.interface.binding.subspan` op in the host code HOT 9
- [compile][rocm]: error: 'llvm.fptrunc' op result #0 must be floating point LLVM type or LLVM dialect-compatible vector of floating point LLVM type, but got 'i8' HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from iree.