jpsingleton / huxley2 Goto Github PK

View Code? Open in Web Editor NEW

46.0 3.0 69.0 346 KB

A cross-platform JSON proxy for the GB railway Live Departure Boards SOAP API

Home Page: https://huxley2.azurewebsites.net

License: European Union Public License 1.2

C# 96.68% HTML 3.27% Dockerfile 0.04%

dotnet-core dotnetcore dotnet csharp c-sharp c-sharp-8 aspnetcore aspnet-core api json

huxley2's Introduction

Hi 👋

I'm James.

I'm the author of ASP.NET Core 2 High Performance (second edition), a book about creating high performance web apps with C# and .NET Core (an open source and cross-platform framework from Microsoft - they are dropping the "Core" branding from .NET 5 onwards but it is the same open product). The book is also a large collection of tips applicable to making web apps perform well in any language or framework, full-stack from hardware to frontend.

I mainly work with C#, .NET and JavaScript/TypeScript/React but also work regularly with Python (v3 but also v2). I use Linux, Markdown, git, CI / testing automation and time-series databases (such as InfluxDB) almost daily too.

I maintain Huxley2, a cross-platform CORS enabled JSON ReST proxy for the GB railway Live Departure Boards SOAP API:

.NET Core ReST JSON web API
SOAP XML API integration
Azure DevOps CI/CD (YAML) and Azure hosting

I blog at unop.uk about tech, the environment and many other things.

If you find my work valuable then please consider sponsoring me. I'm also available for hire.

Thanks🕴🏻

💚

huxley2's People

Contributors

Stargazers

Watchers

huxley2's Issues

Bus and Ferry services are not shown for Staff Departures

When viewing the /staffDepartures/[CRS] endpoint, bus and ferry services are missing with the attribute being null, whereby viewing the normal /departures/[CRS] shows the bus and ferry services.

Issue when getting departures data from a local setup

Describe the bug
Getting error
The HTTP request is unauthorized with client authentication scheme 'Anonymous'. The authentication header received from the server was ''",

I am able to get onto the index page however I get this error when trying to get data from /departures/CMB/10
Also this is reproducible on Azure(with a bit more work to get even just the index page to show)

To Reproduce
Steps to reproduce the behavior:

Fork the repo
Open in visual studio and set your darwin api keys in appsettings.json
Run the project in visual studio and use Postman or insomnia to make this call. https://localhost:xxxx/departures/CMB/10

Expected behavior
To return the JSON data

Screenshots

Desktop (please complete the following information):

OS: [e.g. iOS] Windows 11
Browser [e.g. chrome, safari] Chrome but using Insomnia to make the calls but I also tried making the call on chrome and it also gave a 500
Version [e.g. 22]

Getting service details consistently fails

I don't know if this might be something on National Rail's side, maybe something temporary, or something due to the recent service ID changes, but for any service ID I always get a crash here:

Huxley2/Huxley2/Services/ServiceDetailsService.cs

Line 119 in cc732c6

 var service = await _soapClient.GetServiceDetailsAsync(new GetServiceDetailsRequest 

Here is the full stack trace:

System.ServiceModel.FaultException:
   at System.ServiceModel.Channels.ServiceChannel.HandleReply (System.Private.ServiceModel, Version=4.7.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.ServiceModel.Channels.ServiceChannel.EndCall (System.Private.ServiceModel, Version=4.7.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.ServiceModel.Channels.ServiceChannelProxy+TaskCreator+<>c__DisplayClass1_0.<CreateGenericTask>b__0 (System.Private.ServiceModel, Version=4.7.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Huxley2.Services.ServiceDetailsService+<GetServiceDetailsAsync>d__5.MoveNext (Huxley2, Version=2.0.3.0, Culture=neutral, PublicKeyToken=null: /home/vsts/work/1/s/Huxley2/Services/ServiceDetailsService.cs:119)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Huxley2.Controllers.ServiceController+<Get>d__3.MoveNext (Huxley2, Version=2.0.3.0, Culture=neutral, PublicKeyToken=null: /home/vsts/work/1/s/Huxley2/Controllers/ServiceController.cs:52)

National Rail always return an error 500. Some of the additional details I get in Azure:

Departure board expand option shows previous calling points as null

When I search for a departure board with the expand option enabled, every previousCallingPoints field is null, but subsequentCallingPoints give the expected response. This seems to happen for all trains, regardless of where they start/terminate.

Eg: https://huxley2.azurewebsites.net/departures/sur?expand=true

Could this be an issue with National Rail API?

Thanks

Station Details like Facilities

Hi, in the previous version (departureboard.io) a request like this

https://api.departureboard.io/api/v2.0/getStationDetailsByCRS/GLC

returned all station details, like toilets, car park, bicycle park, step-free access.
I could not find anything like this on https://huxley2.azurewebsites.net/.
Is this not available anymore?

Regards,
Josef

Service /service/{Service ID} returns error 500

Describe the bug
Error 500 in /service/{service id}

To Reproduce
Any service ID causes error 500.

.NET 8 LTS upgrade

Once .NET 8.0 LTS is released (GA) and stable then update to it before 12 Nov, 2024 when .NET 6 support ends.
https://dotnet.microsoft.com/en-us/download/dotnet

Station list CSV down

Huxley2/Huxley2/appsettings.json

Line 25 in dd82d79

 "StationCodesCsvUrl": "https://www.nationalrail.co.uk/static/documents/content/station_codes.csv" 

This URL seems to be dead. Is there a way to disable the station CSV list update, or a place to change the URL to?

Memory leak

After running this in Docker on my Raspberry Pi for about 3 months straight, I noticed that htop was showing memory usage for Huxley2 at 32 GB of memory. Nowhere near that much was in use, but it had 32 GB allocated to it, and accessible.

Obviously, that doesn't seem anywhere near right for what Huxley2 is, and also have no idea where to start looking for issues.

Issues with new serviceID format

Describe the bug
Huxley 2 appears to be unable to interpret the new-format Darwin Service IDs - as such, all queries that hit a new Darwin server (which now appears to be the majority?) will error out

To Reproduce
Query any previously valid route

Expected behaviour
Information Returned

Actual Behaviour

Error 500 returned
In logs:

fail: Huxley2.Controllers.DeparturesController[0]
      Open LDB DepartureBoard API call failed
      System.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.
         at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)
         at System.Convert.FromBase64String(String s)
         at ServiceIdGuidGetter(Object )
         at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer)
         at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.Converters.ArrayConverter`2.OnWriteResume(Utf8JsonWriter writer, TElement[] array, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonCollectionConverter`2.OnTryWrite(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer)
         at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.WriteCoreAsObject(Utf8JsonWriter writer, Object value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.JsonSerializer.WriteCore[TValue](JsonConverter jsonConverter, Utf8JsonWriter writer, TValue& value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.JsonSerializer.WriteStream[TValue](Stream utf8Json, TValue& value, JsonTypeInfo jsonTypeInfo)
         at System.Text.Json.JsonSerializer.Serialize(Stream utf8Json, Object value, Type inputType, JsonSerializerOptions options)
         at Huxley2.Services.ChecksumGenerator.GenerateChecksumObj(Object obj) in /app/Services/ChecksumGenerator.cs:line 57
         at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1)
         at Huxley2.Services.ChecksumGenerator.GenerateChecksumImpl(Object response) in /app/Services/ChecksumGenerator.cs:line 43
         at Huxley2.Services.ChecksumGenerator.GenerateChecksum(BaseStationBoard board) in /app/Services/ChecksumGenerator.cs:line 14
         at Huxley2.Services.StationBoardService.GenerateChecksum(BaseStationBoard board) in /app/Services/StationBoardService.cs:line 103
         at Huxley2.Controllers.DeparturesController.Get(StationBoardRequest request) in /app/Controllers/DeparturesController.cs:line 50

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "XXXXXXXXXXXXX", Request id "XXXXXXXXXXXXX:00000005": An unhandled exception was thrown by the application.
      System.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.
         at System.Convert.FromBase64CharPtr(Char* inputPtr, Int32 inputLength)
         at System.Convert.FromBase64String(String s)
         at ServiceIdGuidGetter(Object )
         at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer)
         at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.Converters.ArrayConverter`2.OnWriteResume(Utf8JsonWriter writer, TElement[] array, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonCollectionConverter`2.OnTryWrite(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer)
         at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.Serialization.JsonConverter`1.WriteCoreAsObject(Utf8JsonWriter writer, Object value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.JsonSerializer.WriteCore[TValue](JsonConverter jsonConverter, Utf8JsonWriter writer, TValue& value, JsonSerializerOptions options, WriteStack& state)
         at System.Text.Json.JsonSerializer.WriteStream[TValue](Stream utf8Json, TValue& value, JsonTypeInfo jsonTypeInfo)
         at System.Text.Json.JsonSerializer.Serialize(Stream utf8Json, Object value, Type inputType, JsonSerializerOptions options)
         at Huxley2.Services.ChecksumGenerator.GenerateChecksumObj(Object obj) in /app/Services/ChecksumGenerator.cs:line 57
         at System.Dynamic.UpdateDelegates.UpdateAndExecute2[T0,T1,TRet](CallSite site, T0 arg0, T1 arg1)
         at Huxley2.Services.ChecksumGenerator.GenerateChecksumImpl(Object response) in /app/Services/ChecksumGenerator.cs:line 43
         at Huxley2.Services.ChecksumGenerator.GenerateChecksum(BaseStationBoard board) in /app/Services/ChecksumGenerator.cs:line 14
         at Huxley2.Services.StationBoardService.GenerateChecksum(BaseStationBoard board) in /app/Services/StationBoardService.cs:line 103
         at Huxley2.Controllers.DeparturesController.Get(StationBoardRequest request) in /app/Controllers/DeparturesController.cs:line 50
         at lambda_method18(Closure , Object )
         at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.AwaitableObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|25_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
         at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
         at Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware.Invoke(HttpContext httpContext)
         at ETagMiddleware.InvokeAsync(HttpContext context) in /app/ETagMiddleware.cs:line 31
         at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.InvokeCore(HttpContext context)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

(Version 2.03)

Additional context
Open Rail Data discussion

Error when getting any departure board

Describe the bug
When getting a departure board for any station, it returns a 500 error.

To Reproduce
Any request to the public endpoint seems to be affected.

Additional context
Only some stations were returning errors starting earlier this week, from yesterday it appears now all show errors.
I think it's related again to the format of service IDs from what I can see? I don't know if the format has been changed again.

Log:

Category: Huxley2.Controllers.DeparturesController
EventId: 0
SpanId: 27be07a0100bf079
TraceId: dab9e9a35c04ae0cc31f4232efb35206
ParentId: 0000000000000000
RequestId: 80000f73-0000-fb00-b63f-84710c7967bb
RequestPath: /departures/BMH
ActionId: 76dee548-cb72-4871-aa09-47fe2c55dec0
ActionName: Huxley2.Controllers.DeparturesController.Get (Huxley2)

Open LDB DepartureBoard API call failed

Exception: 
System.ArgumentOutOfRangeException: Index and length must refer to a location within the string. (Parameter 'length')
   at System.String.Substring(Int32 startIndex, Int32 length)
   at OpenLDBWS.BaseServiceItem.ToGuid(String serviceID) in D:\a\Huxley2\Huxley2\Huxley2\Connected Services\OpenLDBWS\BaseServiceItem.cs:line 39
   at ServiceIdGuidGetter(Object )
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.Converters.ArrayConverter`2.OnWriteResume(Utf8JsonWriter writer, TElement[] array, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonCollectionConverter`2.OnTryWrite(Utf8JsonWriter writer, TCollection value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer)
   at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.Serialization.JsonConverter`1.WriteCoreAsObject(Utf8JsonWriter writer, Object value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.JsonSerializer.WriteCore[TValue](JsonConverter jsonConverter, Utf8JsonWriter writer, TValue& value, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.JsonSerializer.WriteStream[TValue](Stream utf8Json, TValue& value, JsonTypeInfo jsonTypeInfo)
   at System.Text.Json.JsonSerializer.Serialize(Stream utf8Json, Object value, Type inputType, JsonSerializerOptions options)
   at Huxley2.Services.ChecksumGenerator.GenerateChecksumObj(Object obj) in D:\a\Huxley2\Huxley2\Huxley2\Services\ChecksumGenerator.cs:line 57
   at Huxley2.Services.ChecksumGenerator.GenerateChecksumImpl(Object response) in D:\a\Huxley2\Huxley2\Huxley2\Services\ChecksumGenerator.cs:line 43
   at Huxley2.Services.ChecksumGenerator.GenerateChecksum(BaseStationBoard board) in D:\a\Huxley2\Huxley2\Huxley2\Services\ChecksumGenerator.cs:line 14
   at Huxley2.Services.StationBoardService.GenerateChecksum(BaseStationBoard board) in D:\a\Huxley2\Huxley2\Huxley2\Services\StationBoardService.cs:line 103
   at Huxley2.Controllers.DeparturesController.Get(StationBoardRequest request) in D:\a\Huxley2\Huxley2\Huxley2\Controllers\DeparturesController.cs:line 50

Huxley and Huxley 2 seems to be throwing an Error 500 for any departure/arrival query

Describe the bug
Error 500 being thrown for any request

To Reproduce
Steps to reproduce the behavior:
Try to access any huxley / huxley2 instance query a stop for departures/arrivals.
eg: https://huxley2.azurewebsites.net/departures/bhm

Expected behavior
Return a valid response, however all we get is an error 500.

Desktop (please complete the following information):
Any OS/Browser/Platform

Additional context
Huxley 2 and the original Huxley seem to be having the same issue.

Implement ETag-based caching

ETag caching is the future.

I have a project which accesses this API multiple times per minute while running, but most of the time the data returned by the API will be the same. Implementing ETag-based caching could save a lot of bandwidth, especially for length requests such as /crs

jpsingleton / huxley2 Goto Github PK

huxley2's Introduction

Hi 👋

💚

huxley2's People

Contributors

Stargazers

Watchers

Forkers

huxley2's Issues

Recommend Projects

Recommend Topics

Recommend Org