xboxeer / nscrapy Goto Github PK

I'm very happy with this framework

ReWriter the Scheduler, RequestReceiver, ResponseDistributor, so that all Request/Response would be sent to 3rd party Message Queue server and Downloader from Other Machine/Process can pick request and send response back to Message Queue server

Since we are adding multiple Redis instance support for NScrapy, we should also be able to add redis instance at runtime for NScrapy, this requires each NScrapy node get the latest appsetting.json at run time and refresh its context when appsetting.json changed.
there should be a base setting specify appsetting.json's location, and a ConfigProvider should provide an approach for NScrapy to get its appsetting.json whenever it is in local or remote
The appsetting.json find strategy should be like this
Read from local->not found then read from arguments->not found then read from remote which should be implemented by ConfigProvider

I'm very happy with the framework

I'm very happy with this framework

I'm very happy

Maybe related to System.Environment.Newline

I‘m very disappointed with the support! can any one help me asap?

add an ItemLoder function that will automatically transfer the Response to some predefined object by mapping the selector/xpath with object property. while mapping, there should be some middleware that will take over the data filter/cleaning function

We need to do some check to see if user's spider is written correct

I'm very happy with this framework

I feel this is terrible, does anyone can help me? I'm not satisfied with the whole support experience

Scheduler part in current design is not elegent, re-write it

Running in Debug mode will hang in Dequeue in RequestReceiver, not sure why this happened

Reason to have the feature is that while i was running spider and downloader in 2 raspberry pi3+, the node that host the Redis sometimes failed to set value because the Memory is almost running out, and the other Raspberry pi 3+ still have a lot of Memory.

NScrapy Downloader/Spider node can be wrapped to a WepAPI to support service discovery of Micro-Service architecture, currently the 2 nodes are hardly to be wrapped into an WebAPI

I'm very very unhappy with the support! please help assign an engineer on this as soon as possible! I'm very upset now!

Need to add a sort of detect method to know that NScrapy has been baned
Later on NScrapy should add a middleware to use Proxy to send request

I'm very happy with the framework

Assert Failure
Expression: [Recursive resource lookup bug]
Description: Infinite recursion during resource lookup within System.Private.CoreLib. This may be a bug in System.Private.CoreLib, or potentially in certain extensibility points such as assembly resolve events or CultureInfo names. Resource name: ArgumentNull_Generic
Stack Trace:
at System.SR.InternalGetResourceString(String key)
at System.SR.GetResourceString(String resourceKey, String defaultString)
at System.ArgumentNullException..ctor(String paramName)
at System.Runtime.Loader.AssemblyLoadContext.GetLoadContext(Assembly assembly)
at System.Reflection.Assembly.LoadFromResolveHandler(Object sender, ResolveEventArgs args)
at System.AppDomain.OnAssemblyResolveEvent(RuntimeAssembly assembly, String assemblyFullName)
at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks, IntPtr ptrLoadContextBinder)
at System.Reflection.RuntimeAssembly.InternalGetSatelliteAssembly(String name, CultureInfo culture, Version version, Boolean throwOnFileNotFound, StackCrawlMark& stackMark)
at System.Resources.ManifestBasedResourceGroveler.GetSatelliteAssembly(CultureInfo lookForCulture, StackCrawlMark& stackMark)
at System.Resources.ManifestBasedResourceGroveler.GrovelForResourceSet(CultureInfo culture, Dictionary2 localResourceSets, Boolean tryParents, Boolean createIfNotExists, StackCrawlMark& stackMark) at System.Resources.ResourceManager.InternalGetResourceSet(CultureInfo requestedCulture, Boolean createIfNotExists, Boolean tryParents, StackCrawlMark& stackMark) at System.Resources.ResourceManager.InternalGetResourceSet(CultureInfo culture, Boolean createIfNotExists, Boolean tryParents) at System.Resources.ResourceManager.GetString(String name, CultureInfo culture) at System.SR.InternalGetResourceString(String key) at System.SR.GetResourceString(String resourceKey, String defaultString) at System.ArgumentNullException..ctor(String paramName) at System.Runtime.Loader.AssemblyLoadContext.GetLoadContext(Assembly assembly) at System.Reflection.Assembly.LoadFromResolveHandler(Object sender, ResolveEventArgs args) at System.AppDomain.OnAssemblyResolveEvent(RuntimeAssembly assembly, String assemblyFullName) at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks, IntPtr ptrLoadContextBinder) at System.Reflection.RuntimeAssembly.InternalGetSatelliteAssembly(String name, CultureInfo culture, Version version, Boolean throwOnFileNotFound, StackCrawlMark& stackMark) at System.Resources.ManifestBasedResourceGroveler.GetSatelliteAssembly(CultureInfo lookForCulture, StackCrawlMark& stackMark) at System.Resources.ManifestBasedResourceGroveler.GrovelForResourceSet(CultureInfo culture, Dictionary2 localResourceSets, Boolean tryParents, Boolean createIfNotExists, StackCrawlMark& stackMark)
at System.Resources.ResourceManager.InternalGetResourceSet(CultureInfo requestedCulture, Boolean createIfNotExists, Boolean tryParents, StackCrawlMark& stackMark)
at System.Resources.ResourceManager.InternalGetResourceSet(CultureInfo culture, Boolean createIfNotExists, Boolean tryParents)
at System.Resources.ResourceManager.GetString(String name, CultureInfo culture)
at System.SR.InternalGetResourceString(String key)
at System.SR.GetResourceString(String resourceKey, String defaultString)
at System.Diagnostics.StackTrace.ToString(TraceFormat traceFormat)
at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
at System.Exception.GetStackTrace(Boolean needFileInfo)
at System.Exception.ToString(Boolean needFileLineInfo, Boolean needMessage)
at System.Exception.ToString()
at log4net.ObjectRenderer.DefaultRenderer.RenderObject(RendererMap rendererMap, Object obj, TextWriter writer)
at log4net.ObjectRenderer.RendererMap.FindAndRender(Object obj, TextWriter writer)
at log4net.ObjectRenderer.RendererMap.FindAndRender(Object obj)
at log4net.Core.LoggingEvent.GetExceptionString()
at log4net.Appender.AppenderSkeleton.RenderLoggingEvent(TextWriter writer, LoggingEvent loggingEvent)
at log4net.Appender.FileAppender.Append(LoggingEvent loggingEvent)
at log4net.Appender.AppenderSkeleton.DoAppend(LoggingEvent loggingEvent)
at log4net.Util.AppenderAttachedImpl.AppendLoopOnAppenders(LoggingEvent loggingEvent)
at log4net.Repository.Hierarchy.Logger.CallAppenders(LoggingEvent loggingEvent)
at log4net.Repository.Hierarchy.Logger.ForcedLog(Type callerStackBoundaryDeclaringType, Level level, Object message, Exception exception)
at log4net.Repository.Hierarchy.Logger.Log(Type callerStackBoundaryDeclaringType, Level level, Object message, Exception exception)
at log4net.Core.LogImpl.Error(Object message, Exception exception)
at NScrapy.Scheduler.RedisExt.RedisScheduler.ListenToRedisTopic() in C:\Users\xboxeer\source\repos\NScrapy\NScrapy.Scheduler\RedisExt\RedisScheduler.cs:line 111
at System.Threading.Thread.ThreadMain_ThreadStart()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()

I'm very very unhappy with the support! please help assign an engineer on this as soon as possible! I'm very upset now!

Meanwhile Scheduler should be tied with Spider, Scheduler would receive request sent from Spider(or urls extracted from response) and Send that to an idel Downloader process/Machine, this would accomplished by using MQ service like Redis

while there are multiple DownloaderShelll running in distributed way, if the redis was doing background save, the below exception will be thrown

Test issue for PT Integration

I'm very very unhappy with the support! please help assign an engineer on this as soon as possible! I'm very upset now!

I'm very happy with this framework

Both Downloader and Spider node should register themself in Consul, and expose webapi so that an web admin page can use these web api to know how many Downloader and Spider are currently running and what are there status at this moment

Considering to refactor the structure of the project---big change

I'm very happy

After completing the Distributed Downloader, the bottom neck of NScrapy is that extract information from return web page is very slow, usually we use CssSelector or xPath selector to get information from response, which is very CPU resource cost.
If i Open 3 Downloader in distributed mode, there are actually 12 Worker Downloader working(Downloader pool set to 4) some times these 3 Downloader will waiting for New URLs since the extracted URL in redis MQ already consumed and the Spider is still analysis the output page, so we are actually not take full usage of the Distributed Downloader, Spider must be able to be separated to different process/machine

I'm very happy with this product

I'm happy