Giter VIP home page Giter VIP logo

Comments (7)

bytefish avatar bytefish commented on September 26, 2024

An OutOfMemory Exception can only happen, if you read the whole stream to memory, with a ".ToList()" or something, that evaluates the entire Enumerable. Can you share the code you are using?

from bytefish.de.

alienallen avatar alienallen commented on September 26, 2024

Sure. Here's the partial codes I'm using. Yes, ".ToList()" is used for further handling the records. What's your suggestion? How to handle this situation?

CsvParserOptions csvParserOptions = new CsvParserOptions(false, ',');
CsvNoaaYearlyDataMapping csvMapper = new CsvNoaaYearlyDataMapping();
CsvParser<NoaaYearlyData> csvParser = new CsvParser<NoaaYearlyData>(csvParserOptions, csvMapper);
var records = csvParser.ReadFromFile(fileName, Encoding.UTF8).ToList();

class CsvNoaaYearlyDataMapping : CsvMapping<NoaaYearlyData>
    {
        public CsvNoaaYearlyDataMapping() : base()
        {
            MapProperty(0, x => x.Id);
            MapProperty(1, x => x.Date);
            MapProperty(2, x => x.Type, new EnumConverter<ElementType>());
            MapProperty(3, x => x.Value);
            MapProperty(4, x => x.MFlag);
            MapProperty(5, x => x.QFlag);
            MapProperty(6, x => x.SFlag);
            MapProperty(7, x => x.ObsTime);
        }
    }

class NoaaYearlyData
    {
        public string Id { get; set; }
        public string Date { get; set; }
        public ElementType Type { get; set; }
        public string Value { get; set; }
        public string MFlag { get; set; }
        public string QFlag { get; set; }
        public string SFlag { get; set; }
        public string ObsTime { get; set; }
    }
    
    enum ElementType
    {
        PRCP, //= Precipitation (tenths of mm)
        //SNOW, // = Snowfall (mm)
        //SNWD, // = Snow depth (mm)
        TMAX, // = Maximum temperature (tenths of degrees C)
        TMIN, // = Minimum temperature (tenths of degrees C)
        AWND, // = Average daily wind speed (tenths of meters per second)
        EVAP, // = Evaporation of water from evaporation pan (tenths of mm)
        TAVG, // = Average temperature (tenths of degrees C) [Note that TAVG from source 'S' corresponds to an average for the period ending at 2400 UTC rather than local midnight]
        TSUN, // = Daily total sunshine (minutes)
        WDF5, // = Direction of fastest 5-second wind (degrees)
        WSF5, // = Fastest 5-second wind speed (tenths of meters per second)
        WDFI, // = Direction of highest instantaneous wind (degrees)
        WSFI // = Highest instantaneous wind speed (tenths of meters per second)
    }
}

from bytefish.de.

bytefish avatar bytefish commented on September 26, 2024

In C# you have Enumerables. If you do a ToList() on an Enumerable, it so called "evaluates" the Enumerable and reads it into memory. For a File with 1 GB you end with way too much data for the C# runtime to handle.

Instead you need to iterate over the data, for example with a foreach(...) loop. A foreach loop will not read the entire data into memory, but pull the data from the Enumerable one by one. If you want to do it in Parallel, then you can use the ForAll Extension of the ParallelQuery, which TinyCsvParser returns.

from bytefish.de.

bytefish avatar bytefish commented on September 26, 2024

I am closing, because it is not a bug. If you have any questions feel free to ask. I can make up an example for you or you lookup the Unit Tests of TinyCsvParser. But I cannot make an example before the weekend. Way too much work at the moment. 🤜🤛

from bytefish.de.

alienallen avatar alienallen commented on September 26, 2024

Really appreciate that you could make up an example for this "big data" handling, since I am a beginner for C#. And I'll try to understand the preceding suggestion at the meantime. Thanks a lot, mate. 🤜🤛

from bytefish.de.

bytefish avatar bytefish commented on September 26, 2024

Basically remove the ToList() and instead directly to a foreach(...) on it. Then you don't read the entire file into memory, but only the current line basically. This should be the easiest thing to do.

from bytefish.de.

alienallen avatar alienallen commented on September 26, 2024

I get what you mean now. I'll do it. Ask you if I have further questions. Thanks again.

from bytefish.de.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.