deathlef / aoppa Goto Github PK

AOPPA -- A program for extracting the AnarchyOnline resource database
$Id: README 1113 2020-08-19 22:46:01Z os $

Copyright (C) 2002-2009 Oskari Saarenmaa <[email protected]>.
All rights reserved.
Updated by Bitnykk for 2020 compatibility
Licensed under the GNU General Public License version 3.

============================================================================

Background,

'aoppa' is the current database parser I use to parse Funcom's CTree
resource database and store it in a PostgreSQL database for use in my
website (auno.org).  This is the second, or perhaps third, generation of the
parser, the first one was written entirely in PHP and had tons of problems.
Including the fact that it took several hours to parse the database, in
comparison it takes only about 5 seconds for aoppa to parse the current
shadowlands database.

As you can probably tell, this is mostly Unix stuff -- I don't do any
development on Windows.  The parser can only read a special data format that
is produced by a Windows program that actually dumps the Anarchy Online
CTree database. The program is called 'adbd' and the source code for it is
available at http://auno.org/dev/adbd/ .  I produced the program on a Linux
system with the MinGW32 cross-compiler and run it under Wine.

It would probably be trivial for someone familiar with Windows to rewrite
such a program, the only thing it has to do is fetch records from the CTree
database and then store them in a file in the format
AOID: <id> LEN: <length>\n==========\n<data>\n==========\n


Plugins,

Aoppa supports plugins, in fact it can't do anything at all without at least
one plugin loaded.  Currently there exist two kinds of plugins, 'input' and
'output' ones.  Input plugins are used to feed data to the parser, and
output plugins are used to store the data structures populated by the
parser.

Aoppa ships with four default plugins, input-mmap and input-gzip for reading
data from files generated by ADBD (see above), and output-xml and output-ign
for writing XML and IGN-bot format output files.  There's also the
aunoorg-output plugin that I will discuss a bit later.

The plugin system is quite simple to use if you wish to create new plugins,
you should just add a public structure of the type aoppa_plugin_t called
'aoppa_plugin' in your code and then compile it as a shared object.  The
aoppa_plugin structure contains the following elements (in order):
  api version       - for specifying the aoppa API version
  plugin type       - can be AOPPA_INPUT, AOPPA_OUTPUT or both
  init function     - function to be called when plugin is loaded
  finish function   - function to be called when plugin is unloaded
  store function    - function to store item_t structures
  flush function    - function to be called when done with input file
  parse function    - function to be called with a file to parse
  description       - a const char * with a short description of the plugin
All of these, except the version, are optional and can be NULL if the
specified functionality is not required or wanted in the plugin.
See aoppa.h or the supplied plugins for more information.

The obvious plugin that people would probably want, and the one I would like
to see developed is one for reading data directly from the AO CTree
database.  That would of course require someone to port aoppa to Windows.


Auno.org,

There are several files in the repository that are directly related to my
Auno.org website.  These include a special PostgreSQL output plugin, the sql
schemas and some tools for creating additional tables out of the data in
aodb_* tables.  Some of them might be interesting for others who want to
develop an AO database website, most are probably uninteresting, though.


Getting started,

Get the latest .tar.gz package from http://auno.org/dev/aoppa/ and gunzip
and untar it in your favourite Unix system.  Then run 'make' to generate
aoppa and the plugins and run:
./aoppa -p input-gzip.so -p output-xml.so <input files>
Where the <input files> are your database dumps generated by ADBD.  If you
don't want or are unable to run ADBD you can also fetch 18.0.0 item and nano
dumps from http://auno.org/dev/aoppa/items-18000000ep1.dat.gz and
http://auno.org/dev/aoppa/nanos-18000000ep1.dat.gz

After running the command you can just watch the XML output scroll by...


Development,

I have been meaning to release this code for public for as long as I have
been working on it, but there's always been one or another thing that has
blocked the release.  I have wanted to fix some of the most obvious and
embarrassing bugs in the code before a release, but the past has proved that
the chances are that it'll never happen unless I just go ahead and give out
the current version of the code.

As you can see, I have chosen the GNU General Public License for the
project.  What it means is that you can freely copy, modify, sell, and
distribute aoppa, but you have to give the full source code with any
binaries you ship.  This includes the source code for any derivative works,
or anything that is linked with Aoppa.  This includes all plugins.

I would also like anyone who modifies the code to send diffs of their
modifications back to me, so I can update my version and post any
improvements on my website.  If there is a lot of interest in this we can
probably also setup a Sourceforge project.


Anything and everything else,

If you have any questions about this feel free to send me an email at
[email protected] or write in the forums at http://auno.org/

Cheers,
Auno

Bitnykk 2020 Addition (with Assx help) :

You may need to install :
- at least the build essentials package for the main aoppa processing
- also postgresql (& its libpq-dev) for optionnal aunoorg-mk process

Once ready, for aoppa just do : make
It should compile aoppa and its .so & .o files
Example of their use, e.g. to produce an XML file from AO DB extract by adbd :
./aoppa -p input-gzip.so -p output-xml.so adbd-source.dat > aoppa-destination.xml

For auno* tools : make -f aunoorg-mk
It should compile several usable PG libraries
Example of their use, e.g. to connect & update your local PG* DB from AO DB extract by adbd :
./aoppa -p input-gzip.so -p aunoorg-output.so -o dbname=mypgdatabase,dbuser=mypguser,userpass=mypguserpass adbd-source.dat

*: for the PG part refer to these first steps https://wiki.debian.org/PostgreSql
From there you'll execute everything under mypguser in both command line & DB.
Once ready you may need to import auno's aodb structure :
psql -U mypguser -d mypgdatabase -f aunoorg-sql.schema
psql -U mypguser -d mypgdatabase -f aunoorg-sql.index
And then, once .dat was imported into DB, you may want to dump all from PG :
pg_dump mypgdatabase > dumpresult.pgsql
Or may be dump a specific table only, e.g. for sql insert :
pg_dump --column-inserts --data-only --table public.aodb mypgdatabase > tableinsert.sql
Or even possibly some specific fields from one or more table(s) to csv, e.g. :
psql -hlocalhost -dmypgdatabase -Umypguser -c "copy(SELECT public.aodb.aoid, public.aodb.ql, public.aodb_ext.icon, public.aodb.name FROM public.aodb INNER JOIN public.aodb_ext ON public.aodb.xid = public.aodb_ext.xid WHERE public.aodb.ql > 0 AND public.aodb_ext.icon > 0) to stdout DELIMITER ';'" > mypgexport.csv
This could by the way be optimized with Budabot ItemsExtractor filter (delete_list.txt) by Tyrence at https://github.com/Budabot/ItemsExtractor/blob/master/src/main/resources/
And if ever needed, you may want to drop & recreate the DB as earlier :
su - postgres
dropdb mypgdatabase
createdb -O mypguser mypgdatabase
Then just repeat all upper imports with schema, index & .dat file(s)