acarteas / filecache Goto Github PK

I updated my project to Net5 and now I get an exception because the BinaryFormatter has been marked insecure and throws and exception

https://docs.microsoft.com/nl-nl/dotnet/standard/serialization/binaryformatter-security-guide

Cache = new FileCache(path, false, TimeSpan.FromMinutes(2);

I have the Slidingexpiration to 1 minutes, and the clearInterval to 2 minutes. The expiration of the cache is working well, however, the files are not deleted after 2 minutes.

I use FileCache inside a singleton class. I have this design because previously I was using MemoryCache which is a singleton. I noticed that after I run the application again and the constructor of FileCache is called, the older files are deleted.

Question 1: Is it intentional to remove old caches according to cleanInterval when the constructor is called? So singleton is not recommended?

Question 2: Is there an option to remove a cached file when the cache is expired?

Since we are using paths made from the keys, some of the other issues around paths will be swallowed up and result in infinite loop.

E.g. if on windows we use a path that is too long:
fileCache.Add(new string('a', 2000), "value", expire);

Without manually setting AccessTimeout it seems to sit in the retry indefinitely. Perhaps we could default to something rather large, like a minute, if there's concern over throwing?

Hi.

At this code path, contents of supplied Stream are read into a byte buffer. But then it is not written anywhere. It seems like there is a binaryWriter.Write(buffer) missing after the read from originating stream. Or, even better, change to using Stream.CopyTo instead and avoid the BinaryWriter object.

Would you be willing to accept a PR for .NET Standard/Fullframework version of this project? This way you could offer a nuget package for .net standard in addition to the 4.0 and 4.5 frameworks.

the current nuget-version 3.2.1 has no equivalent code tag, the next older version 3.2.0 is not available on nuget.org
I am currently using 3.0.1 and would like to ugrade because of the NullReferenceExceptions that occur in that version, but can't since no clean licensing is possible that way.

Assign this issue to me. I will take a look when I have some time

If I use a colon (:) in my cache key, it looks like the characters before the colon are stripped from the cache entry.

For example, if I use "foo:bar" as the cache key, the key will end up being "bar".

It's not significant problem (I just refrain from using a colon now), but it's worth pointing out.

I think that this 'cachePath' was supposed to be key again

1. FileCache.Contains method ignores cache policy
2. There is no atomic method for appropriate payload get with respect of Cache policy

Or I just did not find them :) Please point me out

Suppose I have some value (the key is "my_key") in the cache with region name "my_region", so the directory structure looks like this:

..\cache\my_region\my_key.dat

When I call method cache.Get("my_key", "my_region") file appears in (wrong?) location:

..\cache\my_key.dat

It's confusing and I think it's a bug.

Test code here:

var cache = new FileCache();
var policy = new CacheItemPolicy { SlidingExpiration = new TimeSpan(0, 5, 0) };

var key = "my_key";
var region = "my_region";
object value = null;

cache.Add(key, "foo", policy, region);
cache.Add(key, "bar", policy);

value = cache.Get(key); // returns: "bar"
value = cache.Get(key, region); // returns: "foo"
value = cache.Get(key); // returns: "foo" ?!

Hi, I'm using File Cache as the persistence store for LazyCache and very occasionally when reading a cachce key I'm see an 'Object graph cannot be null' error and the file for the cache key on disk is 0 bytes in length.

You'll see from the code snippet in the attached that I've used lock(){} to try and rule out and threading issues.
Object graph cannot be null - 2.pdf

Any help much appreciated.

Tested with v3.3.0:

        [TestCase(new byte[] { 1, 2, 3 })]
        [TestCase(new byte[] { 0, 0, 0, 24, 0, 0, 0, 0, 11, 22, 33 })] // fake header 
        [TestCase(new byte[] { 1, 2, 3, 4, 5, 6, 7, 8})]
        public void Cache_Can_Read_And_Write_Raw_Bytes(byte[] payload)
        {
            var cache = new FileCache(System.IO.Path.GetTempPath())
            {
                PayloadReadMode = FileCache.PayloadMode.RawBytes,
                PayloadWriteMode = FileCache.PayloadMode.RawBytes
            };

            const string key = "key";
            cache[key] = payload;
            Assert.AreEqual(payload, cache[key]);
        }

Reads NULL, throws EndOfStreamException or OutOfMemoryException. I wonder why reading raw bytes requires the magic header, which is never written and cannot find an unit test in the repository that reads/writes raw bytes and checks that the returned bytes equal the original bytes.

I have FileCache.Get IOException
[IOException: The process cannot access the file because it is being used by another process]
System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) +216
System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) +736
System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize) +63
System.IO.File.OpenFile(String path, FileAccess access, SafeFileHandle& handle) +38
System.IO.File.SetLastAccessTimeUtc(String path, DateTime lastAccessTimeUtc) +41
System.Runtime.Caching.FileCache.Get(String key, String regionName) +768
System.Runtime.Caching.FileCache.get_Item(String key) +13

Could add FileShare.ReadWrite?
private FileStream GetStream(string path, FileMode mode, FileAccess access, bool lockfile = false)
{
FileStream stream = null;
TimeSpan interval = new TimeSpan(0, 0, 0, 0, 50);
TimeSpan totalTime = new TimeSpan();
while (stream == null)
{
try
{
if (lockfile)
stream = File.Open(path, mode, access, FileShare.None);
else
//stream = File.Open(path, mode, access);
stream = File.Open(path, mode, access, FileShare.ReadWrite);
}

fileCache["A.1"] = 12;
fileCache["A.2"] = "Bob";

Both A.1 and A.2 are now "Bob".

I can see that the cache filename is "A.dat" so the cache value is being shared with anything that is A.*

I think the problem is on line 834 of FileCache.cs here:
string filePath = Path.Combine(directory, Path.GetFileNameWithoutExtension(FileName) + ".dat");

should Path.GetFileNameWithoutExtension(FileName) be used here?

Hi, I've been trying to understand how to work with your library in one of my projects.
I'm using Hash as default cache manager.

So I've been setting the AbsoluteExpiration to 24h. And the cache is injected in a server with a single instance (new FileCache is called only once) that "does not sleep". (we deploy once a week).
The problem I'm facing is that the files in the Cache folder don't get deleted unless we deploy the server. As I could see in the code, the "clean cache" is only called in the constructor. Is that the expected behavior? Is having a single "new FileCache" the correct way to use the cache for multiple requests or should we do something else?

Thanks.

We use FileCache 2.1.2 in our framework but it appears we are having issues with missing System.Runtime.Caching, most likely because its not included our builds that are for .net 4.8 and .net standard 2.1

Currently the AddOrGetExisting method takes a value, which requires you to pre-compute the value whether or not the entry is already in the cache. Instead it should take a generator method (Func<T>) which will generate the value in the instance of a cache miss. See https://github.com/alastairtree/LazyCache for an example of this, copied here:

// Declare (but don't execute) a func/delegate whose result we want to cache
Func<ComplexObjects> complexObjectFactory = () => methodThatTakesTimeOrResources();

// Get our ComplexObjects from the cache, or build them in the factory func 
// and cache the results for next time under the given key
ComplexObjects cachedResults = cache.GetOrAdd("uniqueKey", complexObjectFactory);

It should also provide an async overload for this operation as well

Is the following:

Meant to have a true argument?

//write the object payload (lock the file so we can write to it and force others to wait for us to finish)
using (FileStream stream = GetStream(cachedItemPath, FileMode.Create, FileAccess.Write, true))

Correct me if i'm wrong, but at present, it seems like multiple threads/processes can write to a file at the same time. Additionally, if a thread attempts to read from the file while it is being written to, a partial read of the file can occur.

Same goes for:

Edit: Minor edits to phrasing

I'm trying to use FileCache in a net core project to store some intermediate data between several console app launches. Here is a sample.

`
FileCache.DefaultCacheManager = FileCacheManagers.Hashed;
var fc = new FileCache(@"c:\temp\testcache");
string key = "a1:Settings";

        var v = fc.Get(key);
        if (v != null)
            Console.WriteLine($"Cached item: {v}");
        else
            Console.WriteLine("Cached item not found");

        var itemPolicy = new CacheItemPolicy
        {
            AbsoluteExpiration = DateTimeOffset.Now.AddSeconds(10),
            Priority = CacheItemPriority.NotRemovable
        };
        if (fc.Add(key, "some string value", itemPolicy))
            Console.WriteLine("Cached item added");
        else
            Console.WriteLine("---- Failed to add the item to the cache ----");

        var v2 = fc.Get(key);
        if (v2 != null)
            Console.WriteLine($"Cached item: {v2}");
        else
            Console.WriteLine("Cached item not found");

`

In a net core console app the first Get return null always. The second Get returns the test value.

It works as expected in a .NET Framework project. The first Get returns the cached value on the second app launch.
AND it works as expected without the first line (FileCache.DefaultCacheManager = FileCacheManagers.Hashed;)

As of 3.0.1, the only ctor that takes FileCacheManagers omits the other arguments (most notably for my usage - cacheRoot). There should at minimum be a ctor that takes ALL arguments possible

If a CacheItem key contains characters that are not suitable for a filename an unhandled exception is thrown. Might make sense to create a lookup table or clean out the illegal characters from the filename.

AppVeyor build gets to the following lines:

 Total time: 17.7062 Seconds
C:\Program Files\dotnet\sdk\3.1.401\Microsoft.TestPlatform.targets(32,5): error MSB4181: The "Microsoft.TestPlatform.Build.Tasks.VSTestTask" task returned false but did not log an error. [C:\projects\filecache\src\FileCache.UnitTests\FileCache.UnitTests.csproj]
An error occurred when executing task 'Test'.
Error: One or more errors occurred.
	.NET Core CLI: Process returned an error (exit code 1).
Command exited with code 1

I'm guessing it's a config issue.

Should it be defaulted during insertion (should insertion count as the first 'access'?)

Otherwise the cache is only valid for a single execution

The System.Data.HashFunction.Core dependency which in itself is .NET Standard 1.1 and that depends on https://www.nuget.org/packages/NETStandard.Library/1.6.1 which pulls in a ton on vulnerable dependencies by default (which are also all not required).

This repo seems to have taken over the leaf package: https://github.com/Deskasoft/Data.HashFunction but no issue tracker so no way to get them to stop listing netstandard1.1 as the only .NET Standard version. I have sent them an email so who knows they might be open to rolling out a release.

Microsoft also recommends against targeting 1.1 https://learn.microsoft.com/en-us/dotnet/standard/net-standard#which-net-standard-version-to-target

Would this project be open to a version bump if the leaf package releases?

The FileCache.Init( ... ) method calls the ShouldClean method before initializing the CacheManager class member.

To mitigate the exception, consider moving the initialization of the CacheManager to be called before ShouldClean is called. Alternatively you may also use the null-conditional operator in the ShouldClean method to prevent the NullReferenceException from being thrown.

Currently the library allows you to specify invalid Windows filepaths as keys, for example:

var cache = new FileCache(".");

cache.Add("foo", "foo", DateTimeOffset.Now.AddMinutes(1));
cache.Get("foo").Dump();		// Returns "foo"
cache.Get("foo.bar").Dump();	// Also returns "foo", because the key is truncated, should return null

cache.Add(@"foo\bar", @"foo\bar", DateTimeOffset.Now.AddMinutes(1));
cache.Get(@"foo\bar").Dump();  // Returns "foo\bar"
cache.Get(@"bar").Dump();      // Also returns "foo\bar", should return null.

I'd propose that either the library should reject these as keys (if the input string != the filename), or IMO better yet it should transform them to be eligible paths. Though in this case, care must be taken so that the transformed paths don't conflict with other non-transformed keys.

The Flush method does not pass in the region name to the GetKeys method, and thus always only gets the keys for the root region. This causes a couple of issues:

1. If you call Flush with a regionName, it will attempt to flush keys from the root region instead, but will crash if it finds any because it uses the wrong directory name.
2. If you call Flush without a regionName, it will flush only the root region, and not the child regions as it should.

The solution is to enumerate over all of the regions, including the root region, calling GetKeys and the subsequent foreach loop on each region name. I have corrected code with unit tests which I will submit a PR for.

It appears that when using region names the automatic items cleanup doesn't work as expected, neither removing expired when specifying cleanup interval, nor the whole cache cleanup when the total cache size gets over the limit.

The reason is that in both cases the automatic cleanup is triggered with regionName == null. As the result, FileCache.GetKeys (2.2.0 series) or CacheManager.GetKeys (3.0) return cached item keys only in the root region, ignoring everything else.

Not sure if that's by design, and if there is a different way of setting this up. In my local version, I've hacked the GetKeys method to iterate over all subdirectories and return pairs of file name + region (IEnumerable<Tuple<string, string>> GetKeys).

Hey Sorry I didn't realize this earlier. Did you generate the Nuget package yourself to put into Nuget.org? Because it's missing the target framework dependencies. The Cake.build handles generation of the nupkg with all the correct versions and dependencies.

You can access the artifacts here: https://ci.appveyor.com/project/acarteas/filecache/build/2.1.0.build.7/artifacts

Because when I look at the nuspec for the one currently published:

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
  <metadata>
    <id>FileCache</id>
    <version>2.1.0</version>
    <authors>Adam Carter</authors>
    <owners>Adam Carter</owners>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <licenseUrl>https://github.com/acarteas/FileCache/blob/master/LICENSE</licenseUrl>
    <projectUrl>https://github.com/acarteas/FileCache</projectUrl>
    <description>FileCache is a concrete implementation of the .Net Framework 4's System.Runtime.Caching.ObjectCache that uses the local filesystem as the target location.  FileCache now supports .NET Standard.</description>
    <summary></summary>
    <tags>cache objectcache System.Runtime.Caching.ObjectCache</tags>
  </metadata>
</package>

versus the one from the appveyor artifacts:

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2013/05/nuspec.xsd">
  <metadata>
    <id>FileCache</id>
    <version>2.1.1</version>
    <authors>Adam Carter</authors>
    <owners>Adam Carter</owners>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <licenseUrl>https://github.com/acarteas/FileCache/blob/master/LICENSE</licenseUrl>
    <projectUrl>https://github.com/acarteas/FileCache</projectUrl>
    <description>FileCache is a concrete implementation of the .NET System.Runtime.Caching.ObjectCache that uses the local filesystem as the target location.</description>
    <copyright>Copyright (c) 2012, 2013, 2017 Adam Carter (http://adam-carter.com)</copyright>
    <tags>cache objectcache System.Runtime.Caching.ObjectCache</tags>
    <dependencies>
      <group targetFramework=".NETFramework4.0" />
      <group targetFramework=".NETFramework4.5" />
      <group targetFramework=".NETStandard2.0">
        <dependency id="System.Runtime.Caching" version="4.5.0-preview2-26406-04" exclude="Build,Analyzers" />
      </group>
    </dependencies>
    <frameworkAssemblies>
      <frameworkAssembly assemblyName="System.Runtime.Caching" targetFramework=".NETFramework4.0, .NETFramework4.5" />
    </frameworkAssemblies>
  </metadata>
</package>

it's missing the dependencies group. You might need to unlist the current 2.1.0, and then use the artifacts from 2.1.1.

If you are also curious, you can run .\build.ps1 on your local dev box and it will perform the exact same build that appveyor does, and it creates an \artifacts\ folder which has the nupkgs, which you can review as well.

Hashes generated by FC are not stable between runs. E.g.

FileCache fc = new FileCache(FileCacheManagers.Hashed);
fc["foo"] = "bar";
 Console.Write(fc["foo"]);

will work, but then running again with commenting out the initial PUT into the cache will result in a blank pull.