richardthombs / scunpacked Goto Github PK

The QD appears in the armour's category for some reason (the drive's listings are the 36,000 prices)

Hello,

When I try to use the the loader, I got this issue :
Unhandled exception. System.IO.DirectoryNotFoundException: Could not find a part of the path 'E:\SCData\unp4k-v3.3.79-master\3.9.0\Data\Libs\Foundry\Records\scitemmanufacturer'.

I don't have the folder scitemmanufacturer in 3.9.0 or 3.9.1.

It's normal?

The size number is mismatched with the actual size of the component.
Take ARGO MPUV for example:
https://scunpacked.com/ships/argo_mpuv

The main engine ARGO_MPUV_Thruster_Main_Fixed and ARGO_MPUV_Thruster_Main_Joint are size 1 thrusters but they are shown as size 4 on the table
https://scunpacked.com/items/argo_mpuv_thruster_main_fixed
https://scunpacked.com/items/argo_mpuv_thruster_main_joint

Hey,

i'm getting the following Error when trying to parse data from 3.15.1 PTU

EntityService: Cache saved to input folder d:\scdata\3.15.1
Load Localisation
Load Manufacturers
Load Ammunition
Load Insurance
Unhandled exception. System.ArgumentException: An item with the same key has already been added. Key: AEGS_Redeemer
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at Loader.InsuranceLoader.Load() in D:\scdata\scunpacked\Loader\InsuranceLoader.cs:line 22
   at Loader.Program.Main(String[] args) in D:\scdata\scunpacked\Loader\Program.cs:line 112
PS D:\scdata\3.15.1> 

I noticed that for some commodities the values ​​of shopBuysThis / shopSellsThis are not correctly indicated. I compared some commodities from shops.json with the ShopLayouts.xml file. For example, an Agricium in Port Olisar should have a sell value whereas in json it has a shopBuysThis value: true

On line

More informations here:
https://stackoverflow.com/questions/28673269/why-double-parse-throws-formatexception-only-through-rdp/28673285#28673285

There are no description meta tag for all pages. Having one will improve the search ranking.

In the ships.json file, there is a subsection for each ship called DamageBeforeDetach that looks like this for the AEGS_Avenger_Stalker:

"DamageBeforeDetach": {
      "Canopy": 80.0,
      "Wing_Right": 680.0,
      "Wing_Flap_Right": 10.0,
      "WingTip_Right": 20.0,
      "Wing_Left": 680.0,
      "Wing_Flap_Left": 10.0,
      "WingTip_Left": 20.0,
      "HullTail_Right": 20.0,
      "Right_Tail_Fin_Flap": 10.0,
      "HullTail_Left": 20.0,
      "Left_Tail_Fin_Flap": 10.0
    },

Only one of these properties follow the same key naming convention as the rest of the file. All the rest are in PascalCase, while these seem to be underscore separated PascalCase. This makes parsing harder, requiring manual renames of these particular fields instead of being able to use a generic renaming scheme. Would you mind renaming these fields to match PascalCase like the others in future versions?

Also, many of the key names need to be normalized. In some ships, the DamageBeforeDestruction's "nose" key is upper case, and in others lower case. This trips up strict parsers. Please normalize these.

There are no loading cue on the ship/item page so users can't tell whether the page is empty or not responsive or loading something. Maybe some kind of preloader before the page gets all the data for display?

Hey there - I'm trying to assemble a local copy of the website's database, and while following the readme instructions, I'm getting a path error. Per EntityServices.cs:82, there's supposed to be a path, "Data\Libs\Foundry\Records\entities". No such path exists after I ran unp4k as documented. Instead I've got TagDatabase and associated file, so loader throws an exception and exits.

Any pointers would be appreciated. =)

In FlightCharacteristics, some ships have a String type where most others have a float. This breaks strongly-typed languages. While possible to parse, it makes things much more difficult.

May I suggest producing only a single type per field across all ships/items/etc. It's fine to make them different across different entities, but a single entity should only have a single type. If the "Inifinite" value is needed, might there be another way to represent it besides changing the data type of the field?

It would be great to get a call that lists all the items in a category in an API call.
e.g. https://scunpacked.com/api/items/powerplants

And that will list all the power plants json file URLs and through that I can just loop through them to get what I want.

Can you expose your guidelines for contribution? Also, which dev tools are preferred to run/compile/develop the tool?

Freelancer MAX is under the Heavy Freight category, whereas the Caterpillar is under the Medium Freight category. Both are smaller than some of the Hull series of ships, however relative to eachother, the MAX is lesser in every way.

Hello @richardthombs ,

I've a question about the LocalisationService. Sometimes you use it but not everytime, what's the rule here?

What location is this? Fence_Junkyard_Stanton1_1

SC Version: 3.16.1
Windows Version: Windows 10 Pro 21H2
Loader Version: Latest

I followed the README guide on generating the data from the Data.p4k file, however on attempting to load a specific file, it doesn't like a value. I've attempted to try the instructions however the error is thrown on the same file.

D:\Users\redstonedesigner\SC\Data\3.16.1\Raw\Data\Scripts\Entities\Vehicles\Implementations\Xml\CRUS_Starfighter.xml

Deserialisation failed while parsing Probability value "..6": Input string was not in a correct format.
Unhandled exception. System.InvalidOperationException: There is an error in the XML document.
 ---> System.FormatException: Input string was not in a correct format.
   at System.Number.ThrowOverflowOrFormatException(ParsingStatus status, TypeCode type)
   at System.Xml.XmlConvert.ToDouble(String s)
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVehicle.Read13_Part(Boolean isNullable, Boolean checkType)
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVehicle.Read12_DamageBehavior(Boolean isNullable, Boolean checkType)
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVehicle.Read13_Part(Boolean isNullable, Boolean checkType)
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVehicle.Read13_Part(Boolean isNullable, Boolean checkType)
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVehicle.Read18_Vehicle(Boolean isNullable, Boolean checkType)
   at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVehicle.Read19_Vehicle()
   --- End of inner exception stack trace ---
   at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
   at Loader.VehicleParser.ParseVehicle(String vehiclePath, String modificationName)
   at Loader.ShipLoader.LoadShip(String entityFilename)
   at Loader.ShipLoader.LoadFolder(String entityFolder, String shipFilter)
   at Loader.ShipLoader.Load(String shipFilter)
   at Loader.Program.Main(String[] args

stantonsystem.xml is a treasure trove of positional locations, at least for the planetary bodies. Consider exposing it as a parsed json file.

Additionally, shops are related to their locations through their ID:

1. Given a shop, find its ID in ShopLayouts.xml
2. In layer0.xml, find which variable ID has a default value equal to that shop's id
3. Find that variable ID in stantonsystem.xml and check it's hierarchy

I'm a bit new in this domain and I tried to parse files from the .p4k.
I've created the loader.exe with .Net and it seems to work fine.
I have extracted all the required files as indicated in the readme with unp4k.exe.

And when i try to use the loader.exe with the command loader.exe --input=MyInputPath\3.12.0 --output=MyOutputPath\3.12.0-json it returned an error (I have slightly modified path in the error message for privacy) :

Unhandled exception. System.IO.DirectoryNotFoundException: Could not find a part of the path 'MyInputPath\Data\Libs\Foundry\Records\entities\scitem'.
at System.IO.Enumeration.FileSystemEnumerator1.CreateDirectoryHandle(String path, Boolean ignoreNotFound) at System.IO.Enumeration.FileSystemEnumerator1.Init()
at System.IO.Enumeration.FileSystemEnumerator1..ctor(String directory, Boolean isNormalized, EnumerationOptions options) at System.IO.Enumeration.FileSystemEnumerable1..ctor(String directory, FindTransform transform, EnumerationOptions options, Boolean isNormalized)
at System.IO.Enumeration.FileSystemEnumerableFactory.UserFiles(String directory, String expression, EnumerationOptions options)
at System.IO.Directory.InternalEnumeratePaths(String path, String searchPattern, SearchTarget searchTarget, EnumerationOptions options)
at Loader.EntityService.BuildItemDirectory(String folder) in LoaderPath\EntityService.cs:line 150
at Loader.EntityService.Initialise(Boolean rebuildCache) in LoaderPath\EntityService.cs:line 87
at Loader.Program.Main(String[] args) in LoaderPath\Program.cs:line 77

I have checked in the .p4k these folders/files are inexistant and in particular \data\libs\foundry\records\shipinsurancerecord\shipinsurance.xml.

So with the update of insurance in 3.12 I asking myself if your code is up to date ? Or if I made a mistake.

Thank you in advance for your help.

Got this exeption since 27/07/2021 PTU patch.
I've temporary workaround it with a 'if (!patchFilename.Contains("RSI_Constellation_Taurus.xml"))' before foreach in VehicleParser.cs at line 67

Just checking if you saw the comments post-close in issue #23

Are there any plans to document the json format you're creating beyond the short description in the bottom of the README.md?

As an aside, I'm digging your project. I was unp4king SC tonight anyways intending to figure out where all the data for ships and items was hidden and having no luck. Your project to the rescue.

Since patch 3.15, ships had personal inventory storage for players items.
Theses values are defined in XML files.

Path : Data\Libs\Foundry\Records\inventorycontainers\ships
Value to fetch <SStandardCargoUnit standardCargoUnits="xxx">

Link the value of __ref="abc123" to the ship XML Data\Libs\Foundry\Records\entities\spaceships\

Line :
<VehicleComponentParams inventoryContainerParams="abc123" >

Hello,

wondering if V2 format is the upcoming future of the files for this project. Should I look to support those now or stick with the originals?

I noticed that you had the application dockerized in k8s. How about adding a docker file for the compile time/development dependencies as well. That way the system will be secured for ever, allowing anyone to build everything with guaranteed build dependencies.

Basically, make a docker image that, when "run", outputs another docker image for the running application. Such a container is actually also quite useful during development; it is a bit like "make" with all dependencies built in. If you put the development docker-file in git, it also allows you to return to any previously working build in the far future and still be able to trivially build that old version.

It would be useful to include the inverted "shops" tab, so you could find all items in a particular shop rather than find all shops that sell a particular item. Both are useful, and sometimes they might even be useful in conjunction.

Maybe rename the "Items" tab into "Equipment", rename the "Shops" tab into "Items", and add a new "Shops" tab? Or have two text fields under the "Shops" tab, one for item names and one for shop names (letting you search for both in conjunction when needed)?

The background image used on the site can negatively affect load time when it is not cached (first-time visitor or expired cache). There are several ways that might help to minimize the effect:

• Maintain a collection of background image and serve locally or through a CDN instead of fetching externally. In that way images can be optimized in many ways (screen size, file format, etc.) and ensure that the experience is consistent.
• Add the external sites to preconnect. It is a Band-Aid solution that only requires minimal maintenance, if serving locally and CDN is not an option.

Suppressors and Compensators seem to be missing from the FPS Gear tab. They can be found in the Shops tab, e.g.
Tacit Suppressor1 (WeaponAttachment) [FPS_Barrel, ballistic_attach, energy_attach]

A friend asked for a list of places to respawn at. I had nothing to tell him haha!

In 3.12, CIG added battery, computer, and life support hardpoints to Gladius, Talon, and Talon Shrike. It will probably be adopted into other ships since Gladius is the gold standard test bed ship and the Talons are new ships.

There seem to be lacking values among the commodities. Tram & Meyers at Cellin sell diamonds at 5.85, have a max of 200000 and a very high refresh rate, yet it doesn't show up in scunpacked at all.

I was looking for the shop location for the Aril Backpack (the blue one) and noticed that the shop search doesn't have it. If you just search for "backpack", a lot of them seem to be missing.

The mining commodities (such as Hephaestanite) seem to be in the shop data, though it is hard to get a summary of them. Perhaps a new Mining tab would be useful?

The website host does not have any text compression enabled. Having text compression such as gzip or broccoli should cut down the file size a lot, reducing the load time for visitor and bandwidth for the host.

Basic weapon damage info is done now.

Still to do:

• Accumulate damage for items like the Rattler II which have sub-components that also do damage
• Accumulate the damage of all ship weapons into ship-wide totals for player and turret damage
• Surface the different firing modes of FPS weapons (currently only the first firing mode is listed)
• Surface mining laser energy output on the website

Dark mode:

Light mode:

IFCSParams contains a lot of unsurfaced info:

• maxAngularVelocity
• afterburnAccelMultiplier
• afterburnAngAccelMultiplier
• AeroSurface - aerodynamic lift and drag

according to your Website almost all of the ships are displayed with the hardpoint_ping and are all S1

https://www.youtube.com/watch?v=xpvKokenHZA

Based on this youtube video there seems to be greater differences on how these scanners work and which kind of response they have with mining nodes.

In order to make an educated buying decision i'd like to dig deeper and see if there are differences in them, that would explain the results shown in the video.
Did you find any?

Many of the items use a different manufacturer code than what the game uses. It might be more consistent to use the same convention as the game.

Some of the example includes:
Aegis Dynamics: AEG -> AEGS
Klaus & Werner: KLA -> KLWE
Kruger Intergalatic: KRI -> KRIG

Doesn't seem too serious, but would be nice to resolve:

@richardthombs Again, thanks for this awesome project. I understand most things will be moving to V2 eventually but since there's no dedicated Manufacturers file in v2 yet, I'm looking at the one in the root. I see that there are many duplicates of various manufacturers with different references. What is the significance 'reference' field? I expect that will explain why are there so many duplicates.

/u/malogos already has http://starcitizendb.com/ which is also known as scdb in some circles, and it also has an api, and has been well maintained for a few years now.

If you're not attached to the name, it may be an idea to rebrand before you go wide/public - perhaps scapi? Could even spin things a bit into "SCnAPI", (Star Citizen Normalized API) and have a little crocodile as a mascot/logo etc =D

Hello everybody,

first i want to thank you for this Project. I'm currently working an an small German Community page Link myself and i'm always coming back to your site to check different informations. I really like the "easy-to-understand"-Structure especially when i dont know the exact title of an Item. All-in-All ....a great site & easy to navigate.

Would it be possible to add a Picture to the Detail Pages of the different Items? I know this could potentially use-up a lot of Webspace & traffic.
At the moment i'm searching a couple of Pictures for different Items. My first Thought was:
"No problem, SCunpacked has everything i need...."

This was the moment where i realized that in the Months before i mostly came here to get "text-Information"about "Magazine Size oder rate Of Fire....for example.

I think it would be Great if SCunpacked were "The Place to go" to get an Image for every Item.
This is the main reason why i'm asking. :)

btw. I added your Project to our Link-Database: Link

Greetings from Germany
(Sorry, if my english is a little rough)

Are the ship sale and rental prices and locations in the data somewhere?

Hi,

With the 3.15.1 build, I have this issue with the 2 Ares files (inferno or ion) :

D:\unp4k-suite\Data\Scripts\Entities\Vehicles\Implementations\Xml\CRUS_Starfighter.xml
Unhandled exception. System.InvalidOperationException: There is an error in the XML document.
---> System.FormatException: Input string was not in a correct format.
at System.Number.ThrowOverflowOrFormatException(ParsingStatus status, TypeCode type)
at System.Double.Parse(String s, NumberStyles style, IFormatProvider provider)
at System.Xml.XmlConvert.ToDouble(String s)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVehicle.Read13_Part(Boolean isNullable, Boolean checkType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVehicle.Read12_DamageBehavior(Boolean isNullable, Boolean checkType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVehicle.Read13_Part(Boolean isNullable, Boolean checkType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVehicle.Read13_Part(Boolean isNullable, Boolean checkType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVehicle.Read18_Vehicle(Boolean isNullable, Boolean checkType)
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderVehicle.Read19_Vehicle()
--- End of inner exception stack trace ---

I didn't find what value is missing in the files.