Comments (7)
An OutOfMemory Exception can only happen, if you read the whole stream to memory, with a ".ToList()" or something, that evaluates the entire Enumerable. Can you share the code you are using?
from bytefish.de.
Sure. Here's the partial codes I'm using. Yes, ".ToList()" is used for further handling the records. What's your suggestion? How to handle this situation?
CsvParserOptions csvParserOptions = new CsvParserOptions(false, ',');
CsvNoaaYearlyDataMapping csvMapper = new CsvNoaaYearlyDataMapping();
CsvParser<NoaaYearlyData> csvParser = new CsvParser<NoaaYearlyData>(csvParserOptions, csvMapper);
var records = csvParser.ReadFromFile(fileName, Encoding.UTF8).ToList();
class CsvNoaaYearlyDataMapping : CsvMapping<NoaaYearlyData>
{
public CsvNoaaYearlyDataMapping() : base()
{
MapProperty(0, x => x.Id);
MapProperty(1, x => x.Date);
MapProperty(2, x => x.Type, new EnumConverter<ElementType>());
MapProperty(3, x => x.Value);
MapProperty(4, x => x.MFlag);
MapProperty(5, x => x.QFlag);
MapProperty(6, x => x.SFlag);
MapProperty(7, x => x.ObsTime);
}
}
class NoaaYearlyData
{
public string Id { get; set; }
public string Date { get; set; }
public ElementType Type { get; set; }
public string Value { get; set; }
public string MFlag { get; set; }
public string QFlag { get; set; }
public string SFlag { get; set; }
public string ObsTime { get; set; }
}
enum ElementType
{
PRCP, //= Precipitation (tenths of mm)
//SNOW, // = Snowfall (mm)
//SNWD, // = Snow depth (mm)
TMAX, // = Maximum temperature (tenths of degrees C)
TMIN, // = Minimum temperature (tenths of degrees C)
AWND, // = Average daily wind speed (tenths of meters per second)
EVAP, // = Evaporation of water from evaporation pan (tenths of mm)
TAVG, // = Average temperature (tenths of degrees C) [Note that TAVG from source 'S' corresponds to an average for the period ending at 2400 UTC rather than local midnight]
TSUN, // = Daily total sunshine (minutes)
WDF5, // = Direction of fastest 5-second wind (degrees)
WSF5, // = Fastest 5-second wind speed (tenths of meters per second)
WDFI, // = Direction of highest instantaneous wind (degrees)
WSFI // = Highest instantaneous wind speed (tenths of meters per second)
}
}
from bytefish.de.
In C# you have Enumerables. If you do a ToList() on an Enumerable, it so called "evaluates" the Enumerable and reads it into memory. For a File with 1 GB you end with way too much data for the C# runtime to handle.
Instead you need to iterate over the data, for example with a foreach(...) loop. A foreach loop will not read the entire data into memory, but pull the data from the Enumerable one by one. If you want to do it in Parallel, then you can use the ForAll Extension of the ParallelQuery, which TinyCsvParser returns.
from bytefish.de.
I am closing, because it is not a bug. If you have any questions feel free to ask. I can make up an example for you or you lookup the Unit Tests of TinyCsvParser. But I cannot make an example before the weekend. Way too much work at the moment. 🤜🤛
from bytefish.de.
Really appreciate that you could make up an example for this "big data" handling, since I am a beginner for C#. And I'll try to understand the preceding suggestion at the meantime. Thanks a lot, mate. 🤜🤛
from bytefish.de.
Basically remove the ToList() and instead directly to a foreach(...) on it. Then you don't read the entire file into memory, but only the current line basically. This should be the easiest thing to do.
from bytefish.de.
I get what you mean now. I'll do it. Ask you if I have further questions. Thanks again.
from bytefish.de.
Related Issues (20)
- Question: Usage of bytea HOT 1
- Sql minor bug HOT 1
- Auto Configuere data source in data base per tenant HOT 1
- Spring Boot Multitenancy guide - Error with AbstractRoutingDataSource HOT 3
- Query on spring abstractroutingdatasource advantage HOT 2
- Question on dynamic load of the data source HOT 10
- Add a flow diagram HOT 7
- NEST not working HOT 1
- Adding null in AuditQueryResultUtil HOT 3
- How the tenant can be changed on runtime HOT 1
- Can't write CLR type System.DateTime to database type timestamp with time zone HOT 1
- AbstractMap used by PersonMap in SqlMapper blog entry doesn't match java.util.AbstractMap HOT 7
- Update async multi-tenant example HOT 3
- Select tenant HOT 1
- Subject: Parsing Command Line Arguments in .NET || Topic: Add basic arg validation HOT 2
- How to addnew datasource without @Scheduled HOT 1
- How can we do the reverting to a particular revision using Versioning and Auditing with spring data Envers? please provider me an example for reverting HOT 2
- Tables are not getting created while dynamically tenant details in tenant.json HOT 1
- Does pgbulkinsert support postgis types? HOT 1
- user not getting authenticated in production HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bytefish.de.