select: stripe footer column count and table column count don't match about cstore_fdw HOT 8 CLOSED

Hi Nasadows,

Could you note the exact steps that you followed?

From the error message, I'm guessing this could be one of two things: (a) Alter Table command on the region table, or (b) deleting the region.cstore file but not the stripe, and recreating the table with a different schema but the same region.cstore location.

Best,
Ozgun

from cstore_fdw.

Hi Ozgun,

I think that you are right with your assumption, as I deleted tables.

I did the following steps:

1. I ran the create statements of the tables using the wrong file path (/usr/local/psql/cstore/tpch.cstore).
2. I tried to copy the data from the .tbl files into the tables, as I got an error, that the tpch.cstore file does not exist, because the path was wrong.
3. I deleted all the tables and created them again using the correct path to the cstore file (/usr/local/pqsql/cstore/tpch.cstore)
4. I successfully loaded the data from the .tbl files into the tables using the COPY FROM command
5. Trying to do a simple select on one of the tables to see, if the data is present throws me the error.

I now also habe a tpch.cstore.footer file in my /usr/local/pgsql/cstore directory. These do not exist for my other cstores. Maybe this gives some hints?

Kind regards,
Nasadows

from cstore_fdw.

Hi Nasadows,

Just to clarify, does /usr/local/pqsql/cstore/tpch.cstore represent a directory path or does it hold the table data for the region table?

For each cstore_fdw table, we keep two files. The first file holds the data and its path is determined by the user provided filename option. The second file holds the metadata and its file path is the data file's path with the .footer suffix.

To simplify these concepts for our users, we are working on two changes that are scheduled to be released in July. The first one automatically manages file paths for the user, and removes the need for the filename option (the filename option becomes optional). This change also resolves issue #16. The second change relates to the Drop Table command and makes sure that both the data and metadata files are automatically cleaned up. This second one fixes issue #3.

Also, if you're running the TPC-H benchmark, we find that there are notable optimizations you can do to improve performance. Once we resolve this current issue, we can chat about them in a separate GitHub issue.

Best,
Ozgun

from cstore_fdw.

Hi Ozgun,

/usr/local/pqsql/cstore/tpch.cstore is the path that I used in the create statements the second time. So this is the correct path also containing the data and with that also the region table and it's data.

Thanks for this info and update. I only need the TPCH benchmark as it is now inside the cstore for research reasons. So optimizations are not that important yet. But surely very interesting, so I would be pleased if you would inform me respectively. But for now I would be glad to just solve my current issue.

Kind regards
Nasadows

from cstore_fdw.

Hi Nasadows,

We currently require each table to have a separate filename (data file). For example, the region table would have the filename /usr/local/pgsql/cstore/region.cstore, and the lineitem table would live in /usr/local/pgsql/cstore/lineitem.cstore.

If you use different data files (filenames) for different tables, does that solve the problem?

Best,
Ozgun

from cstore_fdw.

Hi Ozgun,

yes, this works fine for me and solves my issue.

Now, I would also be interested in possible optimizations of the TPC-H benchmark inside the cstore. Where should I post this discussion?

Kind regards,
Nasadows

from cstore_fdw.

Hello Nasadows,

You can start discussion about TPC-H optimizations in cstore-users google group.

Also, our recent blog post might give some hints about some of the optimizations you can do.

from cstore_fdw.

Hey -

Just for documentation purposes, we have a bug here that should go away with our upcoming "resolve file paths automatically" change. The following currently works, but shouldn't:

postgres=# CREATE FOREIGN TABLE REGION
( R_REGIONKEY INTEGER NOT NULL,
R_NAME CHAR(25) NOT NULL,
R_COMMENT VARCHAR(152)) SERVER cstore_server
OPTIONS(filename '/home/ozgun/region.cstore');

postgres=# CREATE FOREIGN TABLE nation
(
n_nationkey INTEGER not null,
n_name CHAR(25) not null,
n_regionkey INTEGER not null,
n_comment VARCHAR(152) not null
) SERVER cstore_server OPTIONS (filename '/home/ozgun/region.cstore');

postgres=# COPY region FROM '/home/ozgun/tpch_2_13_0/region.tbl' WITH DELIMITER '|';
COPY 5
postgres=# COPY nation FROM '/home/ozgun/tpch_2_13_0/nation.tbl' WITH DELIMITER '|';
COPY 25

Best,
Ozgun

from cstore_fdw.