Giter VIP home page Giter VIP logo

csv-importer's Introduction

Build Status Maintainability Test Coverage wakatime

SOLECTRUS

Photovoltaic Dashboard, read here about the motivation (in German): https://ledermann.dev/blog/2021/02/03/photovoltaik-dashboard-als-web-applikation/

Screenshot

Live Demo

A live demo with realtime data is available at https://demo.solectrus.de

Installation

For self-hosting SOLECTRUS, please look at https://github.com/solectrus/hosting

Development

  1. Clone the repo locally:
git clone [email protected]:solectrus/solectrus.git
cd solectrus
  1. Install PostgreSQL, Redis, and puma-dev (if not already present). On a Mac with HomeBrew, run this to install from the Brewfile:
brew bundle
  1. Install and set up puma-dev to use HTTPS for development. Do this on macOS:
sudo puma-dev -setup
puma-dev -install
puma-dev link

# Use Vite via puma-dev proxy
# Adopted from https://github.com/puma/puma-dev#webpack-dev-server
echo 3036 > ~/.puma-dev/vite.solectrus
  1. Setup the application to install gems and NPM packages and create the database:
bin/setup
  1. Start the application locally:
bin/dev

Then open https://solectrus.test in your browser.

Test

After preparing development environment (see above):

bin/influxdb-restart.sh
DISABLE_SPRING=1 bin/rspec
DISABLE_SPRING=1 RAILS_ENV=test bin/rake cypress:run
open coverage/index.html

RuboCop:

bin/rubocop

ESLint:

bin/yarn lint

TypeScript:

bin/yarn tsc

There is a shortcut to run all test and linting tools:

bin/test

Performance monitoring

The Live Demo is hosted at Hetzner Cloud. For performance monitoring, it uses RorVsWild, which is free for OpenSource projects. You can see internal metrics like CPU, memory, and disk usage, as well as request times and errors here: https://www.rorvswild.com/applications/136101/requests

License

Copyright (c) 2020-2024 Georg Ledermann, released under the AGPL-3.0 License

csv-importer's People

Contributors

dependabot[bot] avatar ledermann avatar loebse avatar

Stargazers

avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar

Forkers

holiday-sunrise loebse skullmonkey911

csv-importer's Issues

Import csv-Dateien aus SENEC

Hallo,
ich habe ziemliche Probleme mit dem Import. Das fängt schon mal damit an, dass ich auf der Speicherkarte keine .env-Datei finde. Somit weiß ich auch nicht, wo ich den csv-Ordner erstellen soll. Ich habe dazu die Speicherkarte aus dem Raspberry rausgenommen und mittels Kartenleser mit meinem MacBook verbunden. Dann mit dem Finder gesucht...

Kann mir jemand das für Dummies erklären: ich habe keine Ahnung von den Befehlen. Ich weiß gerade noch, wie ich mich per SSH mit meinem Raspberry Pi verbinde.
Solectrus habe ich gemäß der Anleitung installiert. Keine Ahnung, wie ich das geschafft habe.

Kann mir jemand helfen und mir das Schritt für Schritt für Dumme erklären?

Vielen Dank
VG Martin

Sungrow: Field type conflict float vs. integer 

Importing data from /data ...
Importing /data/Tagesbericht_Drexler-50389_20230628212844.csv... bundler: failed to load command: app/main.rb (app/main.rb)
/usr/local/bundle/gems/influxdb-client-2.9.0/lib/influxdb2/client/default_api.rb:106:in `_request': failure writing points to database: partial write: field type conflict: input field "bat_power_minus" on measurement "SENEC" is type float, already exists as type integer dropped=190 (InfluxDB2::InfluxError)
        from /usr/local/bundle/gems/influxdb-client-2.9.0/lib/influxdb2/client/default_api.rb:59:in `_post_text'
        from /usr/local/bundle/gems/influxdb-client-2.9.0/lib/influxdb2/client/write_api.rb:214:in `write_raw'
        from /usr/local/bundle/gems/influxdb-client-2.9.0/lib/influxdb2/client/write_api.rb:186:in `write'
        from /csv-importer/app/flux_writer.rb:17:in `push'
        from /csv-importer/app/flux_writer.rb:11:in `push'
        from /csv-importer/app/import.rb:50:in `process'
        from /csv-importer/app/import.rb:17:in `block in run'
        from /csv-importer/app/import.rb:16:in `each'
        from /csv-importer/app/import.rb:16:in `run'
        from app/main.rb:23:in `<top (required)>'
        from /usr/local/bundle/gems/bundler-2.4.15/lib/bundler/cli/exec.rb:58:in `load'

Tagesbericht_Drexler-50389_20230628212844.csv

senec importer lädt meine historischen Original-senec-Daten nicht

Hallo Georg,
vorab: die Installation war super-easy. Es lief auf Anhieb. Kompliment für diese tolle Arbeit.

Ich habe SOLECTRUS lokal auf einem debian bullseye system installiert. Dabei habe ich .env angepasst aber alle Passwörter auf default gelassen.
Ich kann das Dashboard unter Port 3000 erreichen. Daten werden alle 10 sec gesammelt. Nach Modifikation von docker-compose.yml kann ich mich auch in influxdb anmelden.

root@sau:/home/guenni/solectrus# nmap -sT sau
Starting Nmap 7.80 ( https://nmap.org ) at 2023-01-30 13:01 CET
Nmap scan report for sau (127.0.1.1)
Host is up (0.00012s latency).
Other addresses for sau (not scanned): 192.168.11.8
Not shown: 981 closed ports
PORT STATE SERVICE
...
3000/tcp open ppp
8086/tcp open d-s-n
...

Jetzt zum Problem: Der Import von senec-Daten funktioniert leider nicht.

root@sau:/home/guenni/solectrus# docker run -it --rm --env-file /data/senec/senec-importer-main/.env --env INFLUX_HOST=192.168.11.8 ghcr.io/solectrus/senec-importer
SENEC CSV importer for SOLECTRUS
https://github.com/solectrus/senec-importer
Copyright (c) 2020,2022 Georg Ledermann, released under the MIT License

Using Ruby 3.1.3 on platform x86_64-linux-musl
Pushing to InfluxDB at http://192.168.11.8:8086, bucket my-solectrus-bucket

Importing data from /data/senec/senec-Web-Statistik-Original/raw-data ...
Imported 0 files


root@sau:/home/guenni/solectrus# ls -l /data/senec/senec-Web-Statistik-Original/raw-data
insgesamt 428
-rw-r--r-- 1 guenni guenni 143006 10. Jan 19:23 S30147251446072327540227639-week-1-2023.csv
-rw-r--r-- 1 guenni guenni 146435 16. Jan 19:30 S30147251446072327540227639-week-2-2023.csv
-rw-r--r-- 1 guenni guenni 147199 27. Jan 11:51 S30147251446072327540227639-week-3-2023.csv
root@sau:/home/guenni/solectrus#

Kannst du mir helfen?
Was kann ich testen?

Danke.
Günter

Import von CSV Dateien die von der isolarcloud

Hallo, ich wollte heute die Daten von vor der Installation von Solectrus importieren. Ich habe schon alle möglichen Berichte ausprobiert.

als Fehler bekomme ich folgendes:

docker run -it --rm --env-file .env --mount type=bind,source="$PWD/csv",target=/data,readonly --network=solectrus_default ghcr.io/solectrus/csv-importer
CSV importer for SOLECTRUS, Version 0.3.8, built at 2024-01-28T06:49:40.109Z
https://github.com/solectrus/csv-importer
Copyright (c) 2020-2024 Georg Ledermann, released under the MIT License

Using Ruby 3.2.3 on platform aarch64-linux-musl
Pushing to InfluxDB at http://influxdb:8086, bucket solectrus
Using time zone Europe/Berlin

Importing data from /data ...
Importing /data/08.2023.csv... bundler: failed to load command: app/main.rb (app/main.rb)
/csv-importer/app/csv_probe.rb:19:in `throw': uncaught throw "Unknown data format in /data/08.2023.csv, first line is Time,Total PV Yield(kWh),Total Feed-in Energy(kWh),Total Purchased Energy(kWh),Total Load Energy Consumption from PV(kWh)\r\n" (UncaughtThrowError)

      throw "Unknown data format in #{file_path}, first line is #{first_line}"
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        from /csv-importer/app/csv_probe.rb:19:in `record_class'
        from /csv-importer/app/import.rb:37:in `process'
        from /csv-importer/app/import.rb:17:in `block in run'
        from /csv-importer/app/import.rb:16:in `each'
        from /csv-importer/app/import.rb:16:in `run'
        from app/main.rb:24:in `<top (required)>'
        from /usr/local/bundle/gems/bundler-2.5.5/lib/bundler/cli/exec.rb:58:in `load'
        from /usr/local/bundle/gems/bundler-2.5.5/lib/bundler/cli/exec.rb:58:in `kernel_load'
        from /usr/local/bundle/gems/bundler-2.5.5/lib/bundler/cli/exec.rb:23:in `run'
        from /usr/local/bundle/gems/bundler-2.5.5/lib/bundler/cli.rb:451:in `exec'
        from /usr/local/bundle/gems/bundler-2.5.5/lib/bundler/vendor/thor/lib/thor/command.rb:28:in `run'
        from /usr/local/bundle/gems/bundler-2.5.5/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
        from /usr/local/bundle/gems/bundler-2.5.5/lib/bundler/vendor/thor/lib/thor.rb:527:in `dispatch'
        from /usr/local/bundle/gems/bundler-2.5.5/lib/bundler/cli.rb:34:in `dispatch'
        from /usr/local/bundle/gems/bundler-2.5.5/lib/bundler/vendor/thor/lib/thor/base.rb:584:in `start'
        from /usr/local/bundle/gems/bundler-2.5.5/lib/bundler/cli.rb:28:in `start'
        from /usr/local/bundle/gems/bundler-2.5.5/exe/bundle:28:in `block in <top (required)>'
        from /usr/local/bundle/gems/bundler-2.5.5/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
        from /usr/local/bundle/gems/bundler-2.5.5/exe/bundle:20:in `<top (required)>'
        from /usr/local/bundle/bin/bundle:25:in `load'
        from /usr/local/bundle/bin/bundle:25:in `<main>'

Die CSV sieht so aus. Ich habe nur in der zu importierende Datei die leeren Spalten gelöscht.
08.2023.csv

Vielleicht kann mir ja jemand helfen, die Dateien zu importieren.

Zeitzone des Imports stimmt nicht

Da ich gestern sehr viele NPU-Fehler hatte, habe ich die (deutsche Senec-)CSV für diese Woche importiert. Die Werte einer Uhrzeit in der CSV wurden allerdings alle 2 Stunden zu spät in Solectrus eingetragen. Soll heißen: Mein Ertrag war laut CSV um 8 Uhr 0,7kW, in Solectrus wird dieser Betrag dann aber erst um 10 Uhr angezeigt.

Ich habe dann die Umgebungsvariable TZ= Europe/Berlin in meine bisherige .env-Datei eingetragen. Danach wurden die Container einmal runtergefahren und wieder neugestartet. Das hatte allerdings keine Auswirkungen beim nächsten Import-Versuch.

An welchen Variablen könnte man hier noch drehen, dass der Abgleich wieder passt?

Edit: Mir ist gerade eingefallen, dass die von mir hinzugefügte TZ Variable nicht in der docker-compose.yml steht. Findet der CSV-Importer sie dann überhaupt?

Import von KOSTAL-Messwerten

So, ich hab mir die CSV angeschaut. Am besten geeignet finde ich die Dateien, die man im Solarportal unter Geräte downloaden kann.Sie bieten Daten pro Datei für 14 Tage im 5 Minuten Intervall. Die einzelnen Spalten sind auch bei allen Teilgeräten gleich (KSEM, WR). Als Beispiel hänge ich mal 3 Dateien unten an, eine ist vom Kostal Smartmeter und die anderen jeweils von den Wechselrichtern.
PV Erzeugung werden von allen Strings geliefert und müssten dann von jedem WR summiert werden.
Ist an einem Wechselrichter die Batterie angeschlossen ist der 3. String immer Beladung (-) und Entladung der Batterie

Gesamterzeugung = DC0/P + DC1/P + DC2/P vom WR1 (ohne Batterie) + DC0/P + DC1/P WR2 (mit Akku) bzw. + DC0/P + DC1/P + DC2/P WR2 (ohne Akku

Batterie Beladung = DC2/P (WR mit Akku) negativer Wert
Batterie Entladung = DC2/P (WR mit Akku) Positiver Wert

Batterie Ladestand = BatSoc

Hausverbrauch: schwierig und etwas undurchsichtig. Wenn man keine Kostal Wallbox hat wird der Verbrauch zum Laden des Autos mit zum Hausverbrauch gezählt (man bekommt hier aber nicht heraus was tatsächlich geladen wird und was andere Verbraucher sind).
Folgende Werte müssten damit zusammenhängen: Hausverbrauch = ACHomeBat/P + ACHomePV/P + ACHomeGrid/P . Allerdings sind hier teilweise alle Werte auf 0 Was wiederum komisch ist, da ja immer Hausverbrauch vorhanden ist. Ein anderer Wert wäre noch ACHome0/P und ACHomeOwnConsum0/P (Eigenverbrauch?), woraus sich evtl. Einspeisung und Netzbezug ableiten lassen.

export_10335956_91109TL700002_bd4e0a2d-7aba-4bb9-9528-7baddbd41680.csv
export_306521.0100_74891362_ab510c75-ab5a-4bf4-8335-13bfbdb9ba8b.csv
export_10335958_92092VF600009_fc235a08-37b2-48ab-b0bc-899f2ee9738b.csv

Originally posted by @StevieC121176 in solectrus/solectrus#2110 (reply in thread)

Importer schreibt Integer, Dashboard erwartet Float

Nach Import erscheint im Dashboard ein Error 500, im Log steht:

ts=2023-07-02T11:41:01.650491Z lvl=info msg="Execute source panic" log_id=0imbISV0000 service=storage-reads error="panic: interface conversion: tsm1.Value is tsm1.IntegerValue, not tsm1.FloatValue" 
[...]
app_1                 | I, [2023-07-02T11:41:01.652988 #11]  INFO -- : [172.18.0.1] method=GET path=/stats/inverter_power/now format=html controller=StatsController action=index status=500 duration=229.82 view=0.00 user_agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.67
app_1                 | F, [2023-07-02T11:41:01.655709 #11] FATAL -- : [172.18.0.1]   
app_1                 | [172.18.0.1] ActionView::Template::Error (panic: interface conversion: tsm1.Value is tsm1.IntegerValue, not tsm1.FloatValue):
app_1                 | [172.18.0.1]      5: turbo-stream action="morph" target="stats" children-only= true
app_1                 | [172.18.0.1]      6:   template
app_1                 | [172.18.0.1]      7:     - if timeframe.now?
app_1                 | [172.18.0.1]      8:       = render StatsNow::Component.new calculator:, field:
app_1                 | [172.18.0.1]      9:     - else
app_1                 | [172.18.0.1]     10:       = render StatsRange::Component.new calculator:, timeframe:, field:
app_1                 | [172.18.0.1]   
app_1                 | [172.18.0.1] app/services/flux/reader.rb:59:in `query_without_cache'
app_1                 | [172.18.0.1] app/services/flux/reader.rb:49:in `block in query'
app_1                 | [172.18.0.1] app/services/flux/reader.rb:48:in `query'
app_1                 | [172.18.0.1] app/services/power_peak.rb:3:in `result'
app_1                 | [172.18.0.1] app/components/stats_now/component.rb:24:in `peak'
app_1                 | [172.18.0.1] app/components/stats_now/component.html.slim:4:in `call'
app_1                 | [172.18.0.1] app/views/stats/index.turbo_stream.slim:8
app_1                 | [172.18.0.1] app/controllers/stats_controller.rb:6:in `index'
app_1                 | [172.18.0.1] app/controllers/concerns/auto_locale.rb:14:in `switch_locale'
app_1                 | I, [2023-07-02T11:41:01.673906 #11]  INFO -- : [172.18.0.1] method=GET path=/500 format=html controller=ErrorsController action=show status=500 duration=17.13 view=16.69 user_agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.67

Wie es scheint, gibt es noch ein Problem mit Float vs. Integer

InfluxDB nicht erreichbar: Failed to open TCP connection

Also lange ist es her, dass ich mich letztes Mal gemeldet habe und soweit hat auch alles sehr gut funktioniert, Dankeschön erstmal!
Ich habe mir mir großen Interesse die Version für den Raspberry Pi angeschaut und ausprobiert. Das hat bis jetzt auch perfekt funktioniert. Nur habe ich wie beim letzten Mal ein paar Probleme bei dem Import der alten Senec Dateien. Anscheinen probiert das Script sich mit einer IP zu verbinden, die nicht existiert.
Ich hänge einfach mal den Log an hier an:

twooearly@rapberrypi:~/solectrus/history-data $ sudo su
root@rapberrypi:/home/twooearly/solectrus/history-data# docker run -it --rm            --env-file .env            --env INFLUX_HOST=192.168.178.26            -v /home/twooearly/solectrus/history-data/Senec_Data:/data            ghcr.io/solectrus/senec-importer
SENEC CSV importer for SOLECTRUS
https://github.com/solectrus/senec-importer
Copyright (c) 2020,2023 Georg Ledermann, released under the MIT License

Using Ruby 3.2.1 on platform aarch64-linux-musl
Pushing to InfluxDB at http://192.168.178.26:8086, bucket my-solectrus-bucket

Importing data from /data ...
Importing /data/209927-week-1-2023.csv... bundler: failed to load command: src/main.rb (src/main.rb)
/usr/local/bundle/gems/influxdb-client-2.9.0/lib/influxdb2/client/default_api.rb:109:in `rescue in _request': Failed to open TCP connection to 192.168.178.26:8086 (execution expired) (InfluxDB2::InfluxError)
        from /usr/local/bundle/gems/influxdb-client-2.9.0/lib/influxdb2/client/default_api.rb:108:in `_request'
        from /usr/local/bundle/gems/influxdb-client-2.9.0/lib/influxdb2/client/default_api.rb:59:in `_post_text'
        from /usr/local/bundle/gems/influxdb-client-2.9.0/lib/influxdb2/client/write_api.rb:214:in `write_raw'
        from /usr/local/bundle/gems/influxdb-client-2.9.0/lib/influxdb2/client/write_api.rb:186:in `write'
        from /senec-importer/src/flux_writer.rb:17:in `push'
        from /senec-importer/src/flux_writer.rb:11:in `push'
        from /senec-importer/src/import.rb:36:in `process'
        from /senec-importer/src/import.rb:11:in `block in run'
        from /senec-importer/src/import.rb:10:in `each'
        from /senec-importer/src/import.rb:10:in `run'
        from src/main.rb:20:in `<top (required)>'
        from /usr/local/lib/ruby/3.2.0/bundler/cli/exec.rb:58:in `load'
        from /usr/local/lib/ruby/3.2.0/bundler/cli/exec.rb:58:in `kernel_load'
        from /usr/local/lib/ruby/3.2.0/bundler/cli/exec.rb:23:in `run'
        from /usr/local/lib/ruby/3.2.0/bundler/cli.rb:491:in `exec'
        from /usr/local/lib/ruby/3.2.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
        from /usr/local/lib/ruby/3.2.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
        from /usr/local/lib/ruby/3.2.0/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
        from /usr/local/lib/ruby/3.2.0/bundler/cli.rb:34:in `dispatch'
        from /usr/local/lib/ruby/3.2.0/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
        from /usr/local/lib/ruby/3.2.0/bundler/cli.rb:28:in `start'
        from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.6/libexec/bundle:45:in `block in <top (required)>'
        from /usr/local/lib/ruby/3.2.0/bundler/friendly_errors.rb:117:in `with_friendly_errors'
        from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.6/libexec/bundle:33:in `<top (required)>'
        from /usr/local/bin/bundle:25:in `load'
        from /usr/local/bin/bundle:25:in `<main>'
/usr/local/lib/ruby/3.2.0/net/http.rb:1271:in `initialize': Failed to open TCP connection to 192.168.178.26:8086 (execution expired) (Net::OpenTimeout)
        from /usr/local/lib/ruby/3.2.0/net/http.rb:1271:in `open'
        from /usr/local/lib/ruby/3.2.0/net/http.rb:1271:in `block in connect'
        from /usr/local/lib/ruby/3.2.0/timeout.rb:189:in `block in timeout'
        from /usr/local/lib/ruby/3.2.0/timeout.rb:196:in `timeout'
        from /usr/local/lib/ruby/3.2.0/net/http.rb:1269:in `connect'
        from /usr/local/lib/ruby/3.2.0/net/http.rb:1248:in `do_start'
        from /usr/local/lib/ruby/3.2.0/net/http.rb:1237:in `start'
        from /usr/local/lib/ruby/3.2.0/net/http.rb:1817:in `request'
        from /usr/local/bundle/gems/influxdb-client-2.9.0/lib/influxdb2/client/default_api.rb:82:in `_request'
        from /usr/local/bundle/gems/influxdb-client-2.9.0/lib/influxdb2/client/default_api.rb:59:in `_post_text'
        from /usr/local/bundle/gems/influxdb-client-2.9.0/lib/influxdb2/client/write_api.rb:214:in `write_raw'
        from /usr/local/bundle/gems/influxdb-client-2.9.0/lib/influxdb2/client/write_api.rb:186:in `write'
        from /senec-importer/src/flux_writer.rb:17:in `push'
        from /senec-importer/src/flux_writer.rb:11:in `push'
        from /senec-importer/src/import.rb:36:in `process'
        from /senec-importer/src/import.rb:11:in `block in run'
        from /senec-importer/src/import.rb:10:in `each'
        from /senec-importer/src/import.rb:10:in `run'
        from src/main.rb:20:in `<top (required)>'
        from /usr/local/lib/ruby/3.2.0/bundler/cli/exec.rb:58:in `load'
        from /usr/local/lib/ruby/3.2.0/bundler/cli/exec.rb:58:in `kernel_load'
        from /usr/local/lib/ruby/3.2.0/bundler/cli/exec.rb:23:in `run'
        from /usr/local/lib/ruby/3.2.0/bundler/cli.rb:491:in `exec'
        from /usr/local/lib/ruby/3.2.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
        from /usr/local/lib/ruby/3.2.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
        from /usr/local/lib/ruby/3.2.0/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
        from /usr/local/lib/ruby/3.2.0/bundler/cli.rb:34:in `dispatch'
        from /usr/local/lib/ruby/3.2.0/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
        from /usr/local/lib/ruby/3.2.0/bundler/cli.rb:28:in `start'
        from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.6/libexec/bundle:45:in `block in <top (required)>'
        from /usr/local/lib/ruby/3.2.0/bundler/friendly_errors.rb:117:in `with_friendly_errors'
        from /usr/local/lib/ruby/gems/3.2.0/gems/bundler-2.4.6/libexec/bundle:33:in `<top (required)>'
        from /usr/local/bin/bundle:25:in `load'
        from /usr/local/bin/bundle:25:in `<main>'
  • Die Dateien sind alle in dem angegebenen Pfad /home/twooearly/solectrus/history-data/Senec_Data
  • Inhalt der .env Datei:
# Credentials of your InfluxDB installation
INFLUX_HOST=192.168.200.200
INFLUX_SCHEMA=http
INFLUX_PORT=8086
INFLUX_TOKEN=*stimmt mit dem Token aus der .env im docker Verzeichnis überein*
INFLUX_ORG=solectrus
INFLUX_BUCKET=my-solectrus-bucket

Ich hoffe, dass sie damit arbeiten können. Danke im Voraus für die Hilfe und einen schönen Tag 😃

bundler: failed to load command: src/main.rb (src/main.rb)

Hallo,

vielen Dank für die tolle App! Die Installation auf einer Synology Diskstation hat wunderbar funktioniert. Leider bekomme ich den Import der alten Daten nicht hin. Ich bekomme die Fehlermeldung "bundler: failed to load command: src/main.rb (src/main.rb)"

Woran könnte das liegen?

root@DiskStation:/volume1/docker/history# docker run -it --rm            --env-file .env            -v /volume1/docker/solectrus/history/data:/data            ghcr.io/solectrus/senec-importer
SENEC CSV importer for SOLECTRUS
https://github.com/solectrus/senec-importer
Copyright (c) 2020,2022 Georg Ledermann, released under the MIT License

Using Ruby 3.1.2 on platform x86_64-linux-musl
Pushing to InfluxDB at http://localhost:8086, bucket my-solectrus-bucket

Importing data from /data ...
Importing /data/S53084641446072328540227639-week-25-2022.csv... bundler: failed to load command: src/main.rb (src/main.rb)
/senec-importer/src/import.rb:46:in `parse_kw': undefined method `sub' for nil:NilClass (NoMethodError)

    (row[string].sub(',', '.').to_f * 1_000).round
                ^^^^
	from /senec-importer/src/import.rb:74:in `record'
	from /senec-importer/src/import.rb:31:in `block in process'
	from /usr/local/lib/ruby/3.1.0/csv/table.rb:539:in `each'
	from /usr/local/lib/ruby/3.1.0/csv/table.rb:539:in `each'
	from /senec-importer/src/import.rb:28:in `map'
	from /senec-importer/src/import.rb:28:in `process'
	from /senec-importer/src/import.rb:11:in `block in run'
	from /senec-importer/src/import.rb:10:in `each'
	from /senec-importer/src/import.rb:10:in `run'
	from src/main.rb:20:in `<top (required)>'
	from /usr/local/bundle/gems/bundler-2.3.11/lib/bundler/cli/exec.rb:58:in `load'
	from /usr/local/bundle/gems/bundler-2.3.11/lib/bundler/cli/exec.rb:58:in `kernel_load'
	from /usr/local/bundle/gems/bundler-2.3.11/lib/bundler/cli/exec.rb:23:in `run'
	from /usr/local/bundle/gems/bundler-2.3.11/lib/bundler/cli.rb:483:in `exec'
	from /usr/local/bundle/gems/bundler-2.3.11/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
	from /usr/local/bundle/gems/bundler-2.3.11/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
	from /usr/local/bundle/gems/bundler-2.3.11/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
	from /usr/local/bundle/gems/bundler-2.3.11/lib/bundler/cli.rb:31:in `dispatch'
	from /usr/local/bundle/gems/bundler-2.3.11/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
	from /usr/local/bundle/gems/bundler-2.3.11/lib/bundler/cli.rb:25:in `start'
	from /usr/local/bundle/gems/bundler-2.3.11/exe/bundle:48:in `block in <top (required)>'
	from /usr/local/bundle/gems/bundler-2.3.11/lib/bundler/friendly_errors.rb:103:in `with_friendly_errors'
	from /usr/local/bundle/gems/bundler-2.3.11/exe/bundle:36:in `<top (required)>'
	from /usr/local/bundle/bin/bundle:25:in `load'
	from /usr/local/bundle/bin/bundle:25:in `<main>'

bundler: failed to load command: src/main.rb (src/main.rb)

Hallo,

Ich bin es (leider) nochmal. Ich habe nun alles zum laufen gebracht und alles funktioniert auch perfekt.
Nur leider bekomme ich den Import der alten Daten nicht hin. Ich habe mir schon den alten Fehler Report von spirit0607 angeschaut und die Schritte befolgt, nur bekomme ich nun folgende Fehlermeldung: bundler: failed to load command: src/main.rb (src/main.rb)
Woran könnte das liegen?
Hier einmal der komplette Log:

root@homeserver:/var/lib/docker/history# docker run -it --rm            --env-file .env            --env INFLUX_HOST=192.168.200.222            -v /var/lib/docker/history/data:/data            ghcr.io/solectrus/senec-importer
SENEC CSV importer for SOLECTRUS
https://github.com/solectrus/senec-importer
Copyright (c) 2020,2022 Georg Ledermann, released under the MIT License

Using Ruby 3.1.2 on platform x86_64-linux-musl
Pushing to InfluxDB at http://192.168.200.222:8086, bucket my-solectrus-bucket

Importing data from /data ...
Importing /data/S21627181110462487540422709-week-10-2022.csv... bundler: failed to load command: src/main.rb (src/main.rb)
/usr/local/bundle/gems/influxdb-client-2.7.0/lib/influxdb2/client/default_api.rb:109:in `rescue in _request': Connection reset by peer (InfluxDB2::InfluxError)
        from /usr/local/bundle/gems/influxdb-client-2.7.0/lib/influxdb2/client/default_api.rb:108:in `_request'
        from /usr/local/bundle/gems/influxdb-client-2.7.0/lib/influxdb2/client/default_api.rb:59:in `_post_text'
        from /usr/local/bundle/gems/influxdb-client-2.7.0/lib/influxdb2/client/write_api.rb:214:in `write_raw'
        from /usr/local/bundle/gems/influxdb-client-2.7.0/lib/influxdb2/client/write_api.rb:186:in `write'
        from /senec-importer/src/flux_writer.rb:17:in `push'
        from /senec-importer/src/flux_writer.rb:11:in `push'
        from /senec-importer/src/import.rb:36:in `process'
        from /senec-importer/src/import.rb:11:in `block in run'
        from /senec-importer/src/import.rb:10:in `each'
        from /senec-importer/src/import.rb:10:in `run'
        from src/main.rb:20:in `<top (required)>'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/cli/exec.rb:58:in `load'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/cli/exec.rb:58:in `kernel_load'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/cli/exec.rb:23:in `run'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/cli.rb:486:in `exec'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/cli.rb:31:in `dispatch'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/cli.rb:25:in `start'
        from /usr/local/bundle/gems/bundler-2.3.22/exe/bundle:48:in `block in <top (required)>'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/friendly_errors.rb:120:in `with_friendly_errors'
        from /usr/local/bundle/gems/bundler-2.3.22/exe/bundle:36:in `<top (required)>'
        from /usr/local/bundle/bin/bundle:25:in `load'
        from /usr/local/bundle/bin/bundle:25:in `<main>'
/usr/local/lib/ruby/3.1.0/socket.rb:456:in `__write_nonblock': Connection reset by peer (Errno::ECONNRESET)
        from /usr/local/lib/ruby/3.1.0/socket.rb:456:in `write_nonblock'
        from /usr/local/lib/ruby/3.1.0/net/protocol.rb:279:in `block in write0'
        from /usr/local/lib/ruby/3.1.0/net/protocol.rb:277:in `each'
        from /usr/local/lib/ruby/3.1.0/net/protocol.rb:277:in `each_with_index'
        from /usr/local/lib/ruby/3.1.0/net/protocol.rb:277:in `write0'
        from /usr/local/lib/ruby/3.1.0/net/protocol.rb:250:in `block in write'
        from /usr/local/lib/ruby/3.1.0/net/protocol.rb:267:in `writing'
        from /usr/local/lib/ruby/3.1.0/net/protocol.rb:249:in `write'
        from /usr/local/lib/ruby/3.1.0/net/http/generic_request.rb:189:in `send_request_with_body'
        from /usr/local/lib/ruby/3.1.0/net/http/generic_request.rb:123:in `exec'
        from /usr/local/lib/ruby/3.1.0/net/http.rb:1568:in `block in transport_request'
        from /usr/local/lib/ruby/3.1.0/net/http.rb:1566:in `catch'
        from /usr/local/lib/ruby/3.1.0/net/http.rb:1566:in `transport_request'
        from /usr/local/lib/ruby/3.1.0/net/http.rb:1539:in `request'
        from /usr/local/lib/ruby/3.1.0/net/http.rb:1532:in `block in request'
        from /usr/local/lib/ruby/3.1.0/net/http.rb:966:in `start'
        from /usr/local/lib/ruby/3.1.0/net/http.rb:1530:in `request'
        from /usr/local/bundle/gems/influxdb-client-2.7.0/lib/influxdb2/client/default_api.rb:82:in `_request'
        from /usr/local/bundle/gems/influxdb-client-2.7.0/lib/influxdb2/client/default_api.rb:59:in `_post_text'
        from /usr/local/bundle/gems/influxdb-client-2.7.0/lib/influxdb2/client/write_api.rb:214:in `write_raw'
        from /usr/local/bundle/gems/influxdb-client-2.7.0/lib/influxdb2/client/write_api.rb:186:in `write'
        from /senec-importer/src/flux_writer.rb:17:in `push'
        from /senec-importer/src/flux_writer.rb:11:in `push'
        from /senec-importer/src/import.rb:36:in `process'
        from /senec-importer/src/import.rb:11:in `block in run'
        from /senec-importer/src/import.rb:10:in `each'
        from /senec-importer/src/import.rb:10:in `run'
        from src/main.rb:20:in `<top (required)>'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/cli/exec.rb:58:in `load'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/cli/exec.rb:58:in `kernel_load'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/cli/exec.rb:23:in `run'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/cli.rb:486:in `exec'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/cli.rb:31:in `dispatch'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/cli.rb:25:in `start'
        from /usr/local/bundle/gems/bundler-2.3.22/exe/bundle:48:in `block in <top (required)>'
        from /usr/local/bundle/gems/bundler-2.3.22/lib/bundler/friendly_errors.rb:120:in `with_friendly_errors'
        from /usr/local/bundle/gems/bundler-2.3.22/exe/bundle:36:in `<top (required)>'
        from /usr/local/bundle/bin/bundle:25:in `load'
        from /usr/local/bundle/bin/bundle:25:in `<main>'

Importing with the default .env file failed

I'm just setting up my new/first solectrus installation (and am VERY happy so far!) and came across this issue when importing my Senec CSV files:

CSV importer for SOLECTRUS, Version 0.3.1, built at 2023-07-02T05:57:19.952Z
https://github.com/solectrus/csv-importer
Copyright (c) 2020-2023 Georg Ledermann, released under the MIT License

bundler: failed to load command: app/main.rb (app/main.rb)
/csv-importer/app/config.rb:39:in `fetch': key not found: "INFLUX_MEASUREMENT" (KeyError)
Did you mean?  "INFLUX_MEASUREMENT_PV"
	from /csv-importer/app/config.rb:39:in `from_env'
	from app/main.rb:17:in `<top (required)>'
	from /usr/local/bundle/gems/bundler-2.4.15/lib/bundler/cli/exec.rb:58:in `load'
	from /usr/local/bundle/gems/bundler-2.4.15/lib/bundler/cli/exec.rb:58:in `kernel_load'
	from /usr/local/bundle/gems/bundler-2.4.15/lib/bundler/cli/exec.rb:23:in `run'
	from /usr/local/bundle/gems/bundler-2.4.15/lib/bundler/cli.rb:492:in `exec'
	from /usr/local/bundle/gems/bundler-2.4.15/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
	from /usr/local/bundle/gems/bundler-2.4.15/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
	from /usr/local/bundle/gems/bundler-2.4.15/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
	from /usr/local/bundle/gems/bundler-2.4.15/lib/bundler/cli.rb:34:in `dispatch'
	from /usr/local/bundle/gems/bundler-2.4.15/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
	from /usr/local/bundle/gems/bundler-2.4.15/lib/bundler/cli.rb:28:in `start'
	from /usr/local/bundle/gems/bundler-2.4.15/exe/bundle:37:in `block in <top (required)>'
	from /usr/local/bundle/gems/bundler-2.4.15/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
	from /usr/local/bundle/gems/bundler-2.4.15/exe/bundle:29:in `<top (required)>'
	from /usr/local/bundle/bin/bundle:25:in `load'
	from /usr/local/bundle/bin/bundle:25:in `<main>'

Indeed, my .env file (as per the default installation) only has a INFLUX_MEASUREMENT_PV=SENEC line.

Duplicating this to add INFLUX_MEASUREMENT=SENEC and re-running the imported fixed the error for me, but this should probably be changed in the code as well.

Import CSV-Files startet nicht

Hallo,
ich versuche alte CSV-Files von SENEC zu importieren und erhalte beim Import folgende Fehlermeldung:

Importing data from /data ...
Importing /data/S20972121431523342540029526-week-1-2023.csv... bundler: failed to load command: app/main.rb (app/main.rb)
/usr/local/lib/ruby/3.2.0/net/http.rb:1603:in `initialize': Failed to open TCP connection to influxdb:8086 (getaddrinfo: Name does not resolve) (SocketError)

Es scheint, als wenn der Port 8086 nicht zur Verbindung der Datenbank verfügbar wäre.

In der docker-compose.yml habe ich in Zeile 63-64 folgende Änderungen durchgeführt:

# Optional: Allow InfluxDB to be accessed from the outside.
ports:
  - 8086:8086

Dennoch läßt sich auch nach einem Reboot des Raspberry PI kein Import durchführen. Es kommt immer die gleiche Fehlermeldung.

Hat jemand eine Idee, was ich noch probieren kann?

Importing non-German CSV fails

after I follow all instructions and run the command "docker run -it --rm --env-file .env --mount type=bind,source="$PWD/csv",target=/data,readonly --network=solectrus_default ghcr.io/solectrus/senec-importer"
the command ends with a error:

_SENEC CSV importer for SOLECTRUS, Version 0.2.2, built at 2023-05-04T10:30:25.466Z
https://github.com/solectrus/senec-importer
Copyright (c) 2020-2023 Georg Ledermann, released under the MIT License

Using Ruby 3.2.2 on platform aarch64-linux-musl
Pushing to InfluxDB at http://influxdb:8086, bucket my-solectrus-bucket

Importing data from /data ...
Importing /data/S27525591196576782540161857-week-1-2023.csv... bundler: failed to load command: src/main.rb (src/main.rb)
/usr/local/lib/ruby/3.2.0/time.rb:196:in `make_time': no time information in " CET" (ArgumentError)

    raise ArgumentError, "no time information in #{date.inspect}"
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    from /usr/local/lib/ruby/3.2.0/time.rb:383:in `parse'
    from /senec-importer/src/solectrus_record.rb:109:in `parse_time'
    from /senec-importer/src/solectrus_record.rb:17:in `time'
    from /senec-importer/src/solectrus_record.rb:11:in `to_h'
    from /senec-importer/src/import.rb:39:in `block in process'
    from /usr/local/lib/ruby/3.2.0/csv/table.rb:938:in `each'
    from /usr/local/lib/ruby/3.2.0/csv/table.rb:938:in `each'
    from /senec-importer/src/import.rb:36:in `map'
    from /senec-importer/src/import.rb:36:in `process'
    from /senec-importer/src/import.rb:14:in `block in run'
    from /senec-importer/src/import.rb:13:in `each'
    from /senec-importer/src/import.rb:13:in `run'
    from src/main.rb:23:in `<top (required)>'
    from /usr/local/bundle/gems/bundler-2.4.12/lib/bundler/cli/exec.rb:58:in `load'
    from /usr/local/bundle/gems/bundler-2.4.12/lib/bundler/cli/exec.rb:58:in `kernel_load'
    from /usr/local/bundle/gems/bundler-2.4.12/lib/bundler/cli/exec.rb:23:in `run'
    from /usr/local/bundle/gems/bundler-2.4.12/lib/bundler/cli.rb:492:in `exec'
    from /usr/local/bundle/gems/bundler-2.4.12/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
    from /usr/local/bundle/gems/bundler-2.4.12/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
    from /usr/local/bundle/gems/bundler-2.4.12/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
    from /usr/local/bundle/gems/bundler-2.4.12/lib/bundler/cli.rb:34:in `dispatch'
    from /usr/local/bundle/gems/bundler-2.4.12/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
    from /usr/local/bundle/gems/bundler-2.4.12/lib/bundler/cli.rb:28:in `start'
    from /usr/local/bundle/gems/bundler-2.4.12/exe/bundle:45:in `block in <top (required)>'
    from /usr/local/bundle/gems/bundler-2.4.12/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
    from /usr/local/bundle/gems/bundler-2.4.12/exe/bundle:33:in `<top (required)>'
    from /usr/local/bundle/bin/bundle:25:in `load'
    from /usr/local/bundle/bin/bundle:25:in `<main>'_

Here is my csv file.

S27525591196576782540161857-week-1-2023.csv

bundler: failed to load command: app/main.rb (app/main.rb)

Beim Versuch, alle .csv-Dateien mittels
docker run -it --rm --env-file .env --mount type=bind,source="$PWD/csv",target=/data,readonly --network=solectrus_default ghcr.io/solectrus/csv-importer

zu importieren bekomme ich diese Meldung:

CSV importer for SOLECTRUS, Version 0.3.2, built at 2023-07-22T13:13:57.993Z
https://github.com/solectrus/csv-importer
Copyright (c) 2020-2023 Georg Ledermann, released under the MIT License

bundler: failed to load command: app/main.rb (app/main.rb)
/csv-importer/app/config.rb:39:in fetch': key not found: "INFLUX_MEASUREMENT_PV" (KeyError) Did you mean? "INFLUX_TOKEN_READ" from /csv-importer/app/config.rb:39:in from_env'
from app/main.rb:17:in <top (required)>' from /usr/local/bundle/gems/bundler-2.4.17/lib/bundler/cli/exec.rb:58:in load'
from /usr/local/bundle/gems/bundler-2.4.17/lib/bundler/cli/exec.rb:58:in kernel_load' from /usr/local/bundle/gems/bundler-2.4.17/lib/bundler/cli/exec.rb:23:in run'
from /usr/local/bundle/gems/bundler-2.4.17/lib/bundler/cli.rb:492:in exec' from /usr/local/bundle/gems/bundler-2.4.17/lib/bundler/vendor/thor/lib/thor/command.rb:27:in run'
from /usr/local/bundle/gems/bundler-2.4.17/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in invoke_command' from /usr/local/bundle/gems/bundler-2.4.17/lib/bundler/vendor/thor/lib/thor.rb:392:in dispatch'
from /usr/local/bundle/gems/bundler-2.4.17/lib/bundler/cli.rb:34:in dispatch' from /usr/local/bundle/gems/bundler-2.4.17/lib/bundler/vendor/thor/lib/thor/base.rb:485:in start'
from /usr/local/bundle/gems/bundler-2.4.17/lib/bundler/cli.rb:28:in start' from /usr/local/bundle/gems/bundler-2.4.17/exe/bundle:37:in block in <top (required)>'
from /usr/local/bundle/gems/bundler-2.4.17/lib/bundler/friendly_errors.rb:117:in with_friendly_errors' from /usr/local/bundle/gems/bundler-2.4.17/exe/bundle:29:in <top (required)>'
from /usr/local/bundle/bin/bundle:25:in load' from /usr/local/bundle/bin/bundle:25:in

'

Sungrow: Wo ist die Einspeisung?

Bei der Betrachtung der CSV-Daten von Sungrow frage ich mich, wo sich denn da die Werte für die Einspeisung verstecken?

Ich hatte zunächst vermutet und auch so implementiert, dass die Angabe der Spalte Netz (W) sowohl positiv (=Netzbezug) als auch negativ (=Einspeisung) sein kann, aber negative Werte scheinen gar nicht vorzukommen.

@holiday-sunrise Weißt Du genaueres dazu?

Import CSV Dateien nicht mehr möglich

Hallo,
nach dem Update auf 0.10.1 wollte ich statistische csv-Daten einspielen. Nach Ausführung des Befehls kommt folgende Fehlermeldung:

docker: env file .env contains invalid utf8 bytes at line 72: [83 69 78 69 67 95 73 78 84 69 82 86 65 76 61 50 226].See 'docker run --help'.

Vor dem Update hat der Befehl funktioniert. Gibt es hier Probleme mit der .env Datei?

Viele Grüße

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.