How to map a large struct? about scala-native HOT 19 CLOSED

didn't understand the question, this is the real layout of the struct in mysql.

from scala-native.

mobilemindtec,

Thank you for your interest in Scala NaLtive.

Scala JVM only recently (Scala 3?) gained support for structures (?) with more than 22 fields.

I checked the Scala Native code for the various CStructN declarations and saw only up to 22.

Before diving into sheer hackery, is there any way to re-structure the problem so that you do
not have so many fields in the structure? Is this a "somebody gave me this and I must use it"
or a fresh sheet of paper project?

Given that background, let me describe what I would do if my bonus was on the line and I had only an
hour and the code was not going to be traceable back to me.

For a CStructN of any size, even one, I would take advantage of Scala implicit classes to give the
fields better names than _1 or _22. That is especially true for CStructs with many fields. These
classes usually are have "Ops" in their name. There should be some examples in the SN source
code base, particularly time.scala and some of the network code.

Given that background, I would take the approach of creating one CStruct which has members which
are themselves CStructs. Say, for a record of 400 fields, one would have one root CStruct with 20 members
each of which contained 20 members.

The key to any sanity is that one would then, tediously, create Ops names to de-reference each of the
40 fields, so that you could say "happy_field" rather than "_1.2".

The same idea can be repeated if one has more than 22x22 fields, add another layer.

An alternate take is that usually the first fields are referenced most often. Yeah, "usually" and $10 will,
barely, buy me morning coffee. One could take the unix directory approach and have fields
1 to 21 be one level deep and then field 22 multiple levels deep. One would still want/need names,
but it would be easier to specify the name equivalents of the first 21 fields.

Now, have I sufficiently embarrassed myself in public by saying all this?

from scala-native.

Hi @LeeTibbert,

Thanks for the answer! I'm developing a MySQL wrapping for ScalaNative, and some structs have more then 22 fields. I would like to be faithful to the original structure.

from scala-native.

The MySQL info helps. Is your SQL structure defined or can it be arbitrary?

In the former case, the fields probably have names, so using the Ops practice would
be reasonable.

If the SQL can be arbitrary, you might want to ask on the SN Discourse (See SN GitHub README for details)
to see what people are doing there. Lot of expertise & experience there. Welcoming community too.

If you try our the Ops root and have problems finding examples in the SN codebase or have questions
about the code you do find, call out.

from scala-native.

How are you accessing SQL itself? From C, or such?

You could use a variant of "glue" (interface) Scala code which passed back a Map[String, Any].
A second Map[String, String] could be used to map from the field name to its datatype.

Somebody on the SN Discourse must have done something like this.

from scala-native.

Another, ugly, alternative is that you could copy the "mumble.gyb" (generate your own boilerplate) used
to generate CStructN and change it from current 22 to 400 or so.

You could run the command (??? I've done it, but long ago.) to go from .gyb to .scala and have keep your
private CStructN in a safe place. You would Probalby have to hange to to MmtCStructN or such, and use
the latter, just to keep the names straight.

That is an even more hacky suggestion then my first one. Day has got to get better from here.

from scala-native.

In the old days, we had an idea for big struct support such as the following. Be aware that this is using older syntax etc. so you would have to adapt to the new APIs in Scala Native.
https://github.com/scala-native/scala-native-bindgen/blob/master/tests/samples/Struct.scala

from scala-native.

@WojciechMazur Would this be better turned into a discussion?

from scala-native.

I'm actually creating a wrapper for the mysql client library. For example, there is a struct like this:

struct st_mysql_options {
  unsigned int connect_timeout, read_timeout, write_timeout;
  unsigned int port, protocol;
  unsigned long client_flag;
  char *host, *user, *password, *unix_socket, *db;
  struct Init_commands_array *init_commands;
  char *my_cnf_file, *my_cnf_group, *charset_dir, *charset_name;
  char *ssl_key;    /* PEM key file */
  char *ssl_cert;   /* PEM cert file */
  char *ssl_ca;     /* PEM CA file */
  char *ssl_capath; /* PEM directory of CA-s? */
  char *ssl_cipher; /* cipher to use */
  char *shared_memory_base_name;
  unsigned long max_allowed_packet;
  bool compress, named_pipe;
  /**
    The local address to bind when connecting to remote server.
  */
  char *bind_address;
  /* 0 - never report, 1 - always report (default) */
  bool report_data_truncation;

  /* function pointers for local infile support */
  int (*local_infile_init)(void **, const char *, void *);
  int (*local_infile_read)(void *, char *, unsigned int);
  void (*local_infile_end)(void *);
  int (*local_infile_error)(void *, char *, unsigned int);
  void *local_infile_userdata;
  struct st_mysql_options_extention *extension;
};

from scala-native.

How does that differ from the actual layout in C?

from scala-native.

@mobilemindtec

Thanks for opening the issue, I think it's an important topic for Scala Native to address.

First of all, on Scala 3 I model large structs as an opaque type of CArray and then use extension methods and manually calculated offsets/alignments to extract member fields. You are in luck, because I actually have a code example from generated Mysql bindings (the st_mysql_options struct you are interested in, the entire binding file is self-contained and copypastable), and an example of using the bindings.

Now, I don't think CArray encoding is good – my code for offsets is almost definitely bugged and doesn't respect platform conventions.

@LeeTibbert's suggestion to use nested structs gave me a wonderful idea (pending validation from C experts) – is it possible to model a struct of 32 elements using CStruct2[CStruct22[...],CStruct10[...]] and retain the offsets and alignments? Is this property something that can be relied on? I got so excited about this that I plan to implement it immediately, as it would greatly improve the readability of the bindings, and their portability as well.

from scala-native.

Sufficiently recent Clang, GCC, & Windows compiler support the #pragma pack(1) directive.
This tells the compiler to skip adding padding between (and within?) structures.

The drawback is that, as I am told by a quick Google, is that it does not work in risc architectures. You would
probably have to put a guard in the C code and hope that a better solution crops up before
somebody tries that. I am not sure if ARM machines fall into this category, RISC V almost certainly does.

Regular "Ops", aka extension, methods & names should give you accurate field access without the need for #pragma.

I'd have to check what the C documentation says about internal padding. Been awhile since I have contemplated
the sin of relying on internal layout. I think the SN compiler will not, currently, add tail padding. Relying upon
that is pretty sketchy.

from scala-native.

I think is Scala 3 structs could be modeled using case classes or tuples as the 22 limit is dropped.
https://scala-lang.org/api/3.3.3/docs/docs/reference/dropped-features/limit22.html

from scala-native.

Is this property something that can be relied on?

Unfortunately no:

struct A {
    double f1;
    char f2;
    char f3;
};

struct B {
    struct {
        double f1;
        char f2;
    } x;
    struct {
        char f3;
    } y;
};

sizeof(struct A) // => 16
sizeof(struct B) // => 24

from scala-native.

Structs in general align to the word boundaries. The C alignment section in this doc in the section is good. https://en.wikipedia.org/wiki/Data_structure_alignment

Here is an example but the addresses are negative but you can get the idea: https://godbolt.org/z/15EPW47Yx

from scala-native.

Thanks all, this is quite upsetting

from scala-native.

struct A {

I did not see a #pragma pack(1) directive. It would be astonishing & depressing if the compiler did_not add padding.
I believe the alignment in that case is that each structure gets aligned to the alignment of its strictest (largest) member.

What happens if you add that pragma? Assuming, of course, that you are on a system and/or which supports it?
I expect no padding.

from scala-native.

At least this Issue has generated some interest & discussion, well done Anton.

from scala-native.

Hi everyone, thanks for the responses! I'm not an expert, so I'm just here as a language user. But I will try the suggested alternatives. I was thinking if it would somehow be possible to generate the struct code with array using a macro using a case class.

from scala-native.