devbisme / kifield Goto Github PK

Hello and thank you for the work on KiField -- you have saved many hours of wasted time for people all over the world.

I'm unable to use KiField for some reason with .tsv's and .csv's, but a .xlsx file works.
Python explains that the csv.Sniffer() raises the error "Could not determine delimeter" (the output message is below).

Is there any other debugging I can do? The --debug flag doesn't produce any more information.

Attached are the .sch, .tsv, .csv, and .xlsx files in a .zip.
python-delimeter_repro-files.zip

Mint 18 16.04
KiField 0.1.4

ab@ab-mint1604 ~/src/folktek/official/33-34_synesthesia/34_comp $ kifield --extract fields.csv --insert envelope-gen_LM358_DFN8.sch --overwrite --debug Traceback (most recent call last): File "/usr/local/bin/kifield", line 9, in <module> load_entry_point('kifield==0.1.4', 'console_scripts', 'kifield')() File "/usr/local/lib/python2.7/dist-packages/kifield-0.1.4-py2.7.egg/kifield/__main__.py", line 147, in main exc_field_names=exc_fields) File "/usr/local/lib/python2.7/dist-packages/kifield-0.1.4-py2.7.egg/kifield/kifield.py", line 998, in kifield part_fields_dict = extract_part_fields(extract_filenames, inc_field_names, exc_field_names) File "/usr/local/lib/python2.7/dist-packages/kifield-0.1.4-py2.7.egg/kifield/kifield.py", line 574, in extract_part_fields f_part_fields_dict = extraction_functions[f_extension](f, inc_field_names, exc_field_names) File "/usr/local/lib/python2.7/dist-packages/kifield-0.1.4-py2.7.egg/kifield/kifield.py", line 321, in extract_part_fields_from_csv wb,_ = csvfile_to_wb(filename) File "/usr/local/lib/python2.7/dist-packages/kifield-0.1.4-py2.7.egg/kifield/kifield.py", line 125, in csvfile_to_wb dialect = csv.Sniffer().sniff(csv_file.read(1024)) File "/usr/lib/python2.7/csv.py", line 188, in sniff raise Error, "Could not determine delimiter" _csv.Error: Could not determine delimiter
python-delimeter_repro-files.zip

First time opening an issue, hope I'm doing this right!

Running Ubuntu Gnome 17.04 Codename: zesty

After installing my operating system from scratch, I get the following error when trying to run kifield
"kifield -x sheetname.sch -i test.csv"
-> Unable to write to file: test.csv.
"kifield --help" gives a proper output (help documentation).
I can also "touch test.sch".
I also tried "sudo kifield -x sheetname.sch -i test.csv" with a similar issue.

It might be that I'm just doing something stupid here, but can't figure out what.

Installed kifield with "sudo easy_install kifield". Installed succesfully:
Searching for kifield
Best match: kifield 0.1.11
Processing kifield-0.1.11-py2.7.egg
kifield 0.1.11 is already the active version in easy-install.pth
Installing kifield script to /usr/local/bin

Using /usr/local/lib/python2.7/dist-packages/kifield-0.1.11-py2.7.egg
Processing dependencies for kifield
Finished processing dependencies for kifield

Any suggestions?

Thank you in advance!

Feature request: warning/catch when schematic includes fields with new lines

This is absolutely on me, but I finally identified the cause of KiField corrupting my KiCad project. At some point while editing in the internal parts spreadsheet of KiCad, I added new lines to the end of some of the values of footprint fields of some, but not all, parts. (Probably copying and pasting values too fast and not being careful)

When KiField read out the parts it all worked fine, but upon returning new values to those parts in the schematic file, KiField deleted the location information for that field and corrupted the schematic so it wouldn't open properly.

While this was a mistake that seems unlikely to be a common issue, I thought I would put it out there so that if anyone else has this issue there is some way to move forward. In the end I removed the new lines from the schematic files before editing with KiFields and that fixed the issue.
find . -name '*.sch' -print0 | xargs -0 sed -i .bak -e ':a' -e 'N' -e '$!ba' -e 's/\"\r\"/\"\"/g' was the command I ended up using (on Mac OS)

Am happy to put together an example if needed!

With a python3 virtualenv:

writing manifest file 'kifield.egg-info/SOURCES.txt'
running build_ext
error: [Errno 2] No such file or directory: 'test'

With a python2 virtualenv

writing manifest file 'kifield.egg-info/SOURCES.txt'
running build_ext
error: [Errno 2] No such file or directory: 'test'
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/home/kaspar/projects/kitnic/KiField/venv/local/lib/python2.7/site-packages/openpyxl-2.4.0_b1-py2.7.egg/openpyxl/writer/write_only.py", line 34, in _openpyxl_shutdown
    for path in ALL_TEMP_FILES:
TypeError: 'NoneType' object is not iterable
Error in sys.exitfunc:
Traceback (most recent call last):
  File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
    func(*targs, **kargs)
  File "/home/kaspar/projects/kitnic/KiField/venv/local/lib/python2.7/site-packages/openpyxl-2.4.0_b1-py2.7.egg/openpyxl/writer/write_only.py", line 34, in _openpyxl_shutdown
    for path in ALL_TEMP_FILES:
TypeError: 'NoneType' object is not iterable

Running latest release (0.1.14), latest Python on Windows. We don't need XLSX, a way of turning off the attempted conversion to XLSX would probably be a sufficient fix..

C:\Users\Paul\Documents\p5t4ps>kifield -x p5t4ps.sch -i variant-0.csv -d 1
Extracting fields +[], -[] from files ['p5t4ps.sch'].
Extracting fields [], -[] from schematic file p5t4ps.sch.
Inserting extracted fields into files ['variant-0.csv'].
Inserting extracted fields into CSV file variant-0.csv.
Traceback (most recent call last):
  File "c:\users\paul\appdata\local\programs\python\python37-32\lib\site-packages\kifield\kifield.py", line 765, in insert_part_fields_into_wb
    for h in headers}
  File "c:\users\paul\appdata\local\programs\python\python37-32\lib\site-packages\kifield\kifield.py", line 765, in <dictcomp>
    for h in headers}
  File "c:\users\paul\appdata\local\programs\python\python37-32\lib\site-packages\openpyxl\utils\cell.py", line 123, in column_index_from_string
    return _COL_STRING_CACHE[str_col.upper()]
AttributeError: 'int' object has no attribute 'upper'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Paul\AppData\Local\Programs\Python\Python37-32\Scripts\kifield-script.py", line 11, in <module>
    load_entry_point('kifield==0.1.14', 'console_scripts', 'kifield')()
  File "c:\users\paul\appdata\local\programs\python\python37-32\lib\site-packages\kifield\__main__.py", line 148, in main
    backup = not args.nobackup)
  File "c:\users\paul\appdata\local\programs\python\python37-32\lib\site-packages\kifield\kifield.py", line 1219, in kifield
    insert_part_fields(part_fields_dict, insert_filenames, recurse, group_components, backup)
  File "c:\users\paul\appdata\local\programs\python\python37-32\lib\site-packages\kifield\kifield.py", line 1203, in insert_part_fields
    insertion_functions[f_extension](part_fields_dict, f, recurse, group_components, backup)
  File "c:\users\paul\appdata\local\programs\python\python37-32\lib\site-packages\kifield\kifield.py", line 880, in insert_part_fields_into_csv
    wb = insert_part_fields_into_wb(part_fields_dict, wb)
  File "c:\users\paul\appdata\local\programs\python\python37-32\lib\site-packages\kifield\kifield.py", line 768, in insert_part_fields_into_wb
    for h in headers}
  File "c:\users\paul\appdata\local\programs\python\python37-32\lib\site-packages\kifield\kifield.py", line 768, in <dictcomp>
    for h in headers}
AttributeError: module 'openpyxl.cell' has no attribute 'column_index_from_string'

C:\Users\Paul\Documents\p5t4ps>

This unfortunately fails to cope with an Ω symbol and fails as follows.

Lib file is

EESchema-LIBRARY Version 2.3
DEF RC0805JR-070RL R 0 0 N Y 1 F N
F0 "R" 80 0 50 V V C CNN ""
F1 "RC0805JR-070RL" 0 0 50 H I L TNN ""
F2 "${KICUSTMOD}/IPC7351-Nominal.pretty/RESC2012X60.kicad_mod" 0 0 50 H I L TNN "footprint"
F3 "resistor_smd/yageo/eedf25a24562e2e6f379f3772bde2443.pdf" 0 0 50 H I L TNN "datasheet"
F4 "1485082349267" 0 0 50 H I L TNN "key"
F5 "RC0805JR-070RL" 0 0 50 H I L TNN "man"
F6 "Yageo" 0 0 50 H I L TNN "manufacturer"
F7 "0.00 Ω" 0 0 50 V V C CNN "DisplayValue"
DRAW
S -40 -100 40 100 0 1 10 N
X ~ 1 0 150 50 D 50 50 1 1 P
X ~ 2 0 -150 50 U 50 50 1 1 P
ENDDRAW
ENDDEF etc

Error is

(kifield) [johnpateman@Johns-MBP] KiField $ kifield --extract /Users/johnpateman/Dropbox/Components/library/r0805.lib --insert /Users/johnpateman/Dropbox/Components/RESISTORS.csv -d 7
Extracting fields +[], -[] from files ['/Users/johnpateman/Dropbox/Components/library/r0805.lib'].
Extracting fields from /Users/johnpateman/Dropbox/Components/library/r0805.lib.
Extracting fields [], -[] from part library /Users/johnpateman/Dropbox/Components/library/r0805.lib.
Extracted library part: RC0805JR-071RL prefix R.
Extracted library part: RC0805JR-071RL value RC0805JR-071RL.
Extracted library part: RC0805JR-071RL footprint ${KICUSTMOD}/IPC7351-Nominal.pretty/RESC2012X60.kicad_mod.
Extracted library part: RC0805JR-071RL datasheet resistor_smd/yageo/eedf25a24562e2e6f379f3772bde2443.pdf.
Extracted library part: RC0805JR-071RL key 1485082496735.
Extracted library part: RC0805JR-071RL mpn RC0805JR-071RL.
Extracted library part: RC0805JR-071RL manufacturer Yageo.
Traceback (most recent call last):
  File "/Users/johnpateman/miniconda3/envs/kifield/bin/kifield", line 11, in <module>
    load_entry_point('kifield==0.1.4', 'console_scripts', 'kifield')()
  File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield-0.1.4-py2.7.egg/kifield/__main__.py", line 147, in main
    exc_field_names=exc_fields)
  File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield-0.1.4-py2.7.egg/kifield/kifield.py", line 998, in kifield
    part_fields_dict = extract_part_fields(extract_filenames, inc_field_names, exc_field_names)
  File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield-0.1.4-py2.7.egg/kifield/kifield.py", line 574, in extract_part_fields
    f_part_fields_dict = extraction_functions[f_extension](f, inc_field_names, exc_field_names)
  File "/Users/johnpateman/miniconda3/envs/kifield/lib/python2.7/site-packages/kifield-0.1.4-py2.7.egg/kifield/kifield.py", line 463, in extract_part_fields_from_lib
    component_name, name, value))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 5: ordinal not in range(128
```)

This schematic fails to parse, with this error:

Traceback (most recent call last):
File "/home/local/temp/kifield/bin/kifield", line 11, in
sys.exit(main())
File "/home/local/temp/kifield/lib/python2.7/site-packages/kifield/main.py", line 161, in main
recurse=args.recurse)
File "/home/local/temp/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 1119, in kifield
part_fields_dict = extract_part_fields(extract_filenames, inc_field_names, exc_field_names, recurse)
File "/home/local/temp/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 678, in extract_part_fields
f_part_fields_dict = extraction_functions[f_extension](f, inc_field_names, exc_field_names, recurse)
File "/home/local/temp/kifield/lib/python2.7/site-packages/kifield/kifield.py", line 457, in extract_part_fields_from_sch
sch = Schematic(filename) # Read in the schematic.
File "/home/local/temp/kifield/lib/python2.7/site-packages/kifield/sch.py", line 193, in init
self.components.append(Component(block_data))
File "/home/local/temp/kifield/lib/python2.7/site-packages/kifield/sch.py", line 47, in init
line = list(s)
File "/usr/lib64/python2.7/shlex.py", line 269, in next
token = self.get_token()
File "/usr/lib64/python2.7/shlex.py", line 96, in get_token
raw = self.read_token()
File "/usr/lib64/python2.7/shlex.py", line 172, in read_token
raise ValueError, "No closing quotation"
ValueError: No closing quotation

I added a line counter to try to locate the error, and it seem to fail at line 800. I haven't spotted any problem with that line.

Maybe my understanding of the --overwrite functionality is incorrect.

In a .sch file, a component is defined as such:
$Comp
L Device:R_Small R73
U 1 1 5B382555
P 5750 2400
F 0 "R82" H 5780 2420 50 0000 L CNN
F 1 "100k" H 5780 2360 50 0000 L CNN
F 2 "passive:0603_pad-0.95W-0.7L" H 5750 2400 50 0001 C CNN
F 3 "~" H 5750 2400 50 0001 C CNN
1 5750 2400
0 -1 -1 0
$EndComp

However, I have new fields which I have created via the "Field Name Templates". One of these fields is called "Power". All of my new fields are properly shown when pressing 'E' on a component in Eeschema (these new fields are indeed blank).

If I manually put a "-" into the Power field via editing a component with 'E', then this change is reflected in the schematic as:

$Comp
L Device:R_Small R73
U 1 1 5B381EC3
P 3000 2950
F 0 "R73" H 3030 2970 50 0000 L CNN
F 1 "100k" H 3030 2910 50 0000 L CNN
F 2 "passive:0603_pad-0.95W-0.7L" H 3000 2950 50 0001 C CNN
F 3 "~" H 3000 2950 50 0001 C CNN
F 4 "-" H 3000 2950 50 0001 C CNN "Power"
1 3000 2950
-1 0 0 1
$EndComp

So, by default KiCad is not writing these fields which are empty. This may be the intended functionality. I would assume that any fields which are displayed when pressing 'E' in Eeschema would be written.
I have filed a bug here: https://bugs.launchpad.net/kicad/+bug/1814166

I seem to have remembered being able to successfully write new fieldsvia the "--overwrite" flag previously. Maybe I am mistaken.

I am getting the following error when trying to insert fields from an .xlsx back into the schematic:

whiteandnerdy:NeoESP ppelleti$ kifield -x NeoESP.xlsx -i NeoESP.sch 
Traceback (most recent call last):
  File "/usr/local/bin/kifield", line 8, in <module>
    load_entry_point('kifield==0.1.7', 'console_scripts', 'kifield')()
  File "/Library/Python/2.7/site-packages/kifield-0.1.7-py2.7.egg/kifield/__main__.py", line 157, in main
    recurse=args.recurse)
  File "/Library/Python/2.7/site-packages/kifield-0.1.7-py2.7.egg/kifield/kifield.py", line 1154, in kifield
    insert_part_fields(part_fields_dict, insert_filenames, recurse, group_components)
  File "/Library/Python/2.7/site-packages/kifield-0.1.7-py2.7.egg/kifield/kifield.py", line 1138, in insert_part_fields
    insertion_functions[f_extension](part_fields_dict, f, recurse, group_components)
  File "/Library/Python/2.7/site-packages/kifield-0.1.7-py2.7.egg/kifield/kifield.py", line 918, in insert_part_fields_into_sch
    if field_value.startswith(INVIS_PREFIX):
AttributeError: 'long' object has no attribute 'startswith'

I'm on Mac OS X 10.9.5, and I installed kifield as recommended with easy_install kifield.

whiteandnerdy:NeoESP ppelleti$ python --version
Python 2.7.5

Hi,

I've just installed kifield on Linux Mint 18.1 by running $ sudo easy_install .

Your docs says
$ easy_install kifield

Either this should be case sensitive
$ easy_install KiField
or running from within the repos folder with
$ easy_install .

Further more, a sudo is needed on Linux.

Greets, Karl

I have been working on a BOM standard of sorts for use with the 1-click-bom extension and my kitnic.it site.

It's a simple tab-separated format (mostly because this is the clip-board format for spreadsheet programs). There are some docs and examples here. How would you feel about adding support to sync with this format?

This would provide a good way to export a design, use the extension features to do search and complete the fields and then sync it back into KiCad.

To summarise the biggest difference seems to be tabs and a .tsv extension vs commas with .csv and components with all matching fields grouped together rather than one per line.

Progress: we currently support one way syncing from 1-click-bom to schematic.

• TSV support
• grouped lines output

Look like a lost (but no all) of footprint list ("FPLIST") been removed in my .lib file after call kifield.

However, thank for the project - I help me still.

Hi. I am running win7 64 bit. Python is V3.6.4:d48eceb, Dec 19 2017 build.
I have successfully installed KiField (I think). I have a Kicad schematic that I have put in C:\Kifield\ AGC_Preamp.sch for ease of access. I have made BOMs from this using the PCB package. All the devices have been back annotated from the PCB fabrication outputs BOM tab (so it should work).
I run Kifield -x C:\Kifield\AGC_Preamp.sch -i C:\Kifield\AGC_Preamp.csv from the command prompt, then when I get the spelling and spacing right it runs, but exits with a "could not determine delimiter" error at csv.py line 202.
If i then ask it to produce an xlsx file by changing the last extension to .xlsx,
I get errors in excel.py line 121 where it looks for a ZIP file. The ultimate error is that zipfile.py line 1175 decides that the file is not a ZIP archive, which is true.
The traceback for both runs is attached as a jpg.
Have I not done something, or missed some steps out?
Any help would be appreciated.

Only the top-level schematic file is backed up. The lower-level files in the hierarchy also need to be backed up.

The recursive extraction code does not attach any path to sub-schematic filenames extracted from the top-level file. Thus, it fails to find those files if the schematics are not in the current working directory. The error is also somewhat cryptic due to the way it's handled. It states the top-level schematic file was not found, but of course, it must exist, otherwise it wouldn't have got this far.

Naively, the following change appears to fix the issue. It works for me, but I'm not familiar enough with how KiCad hierarchical schematics are implemented to know if this is foolproof (e.g. are absolute paths allowed for sub-schematics?)

diff --git a/kifield/kifield.py b/kifield/kifield.py
index a67d8be..07c9d8c 100644
--- a/kifield/kifield.py
+++ b/kifield/kifield.py
@@ -496,7 +496,7 @@ def extract_part_fields_from_sch(filename, inc_field_names=None, exc_field_names
         for sheet in sch.sheets:
             for field in sheet.fields:
                 if field['id'] == 'F1':
-                    sheet_file = unquote(field['value'])
+                    sheet_file = os.path.join(os.path.dirname(filename),unquote(field['value']))
                     part_fields_dict.update(extract_part_fields_from_sch(sheet_file, inc_field_names, exc_field_names, recurse, depth+1))
                     break

After doing

sudo easy_install kifield I get the error:

error: Couldn't find a setup script in /home/simon/Documents/kicad/00tools/KiField/kifield

I have no experience with easy_install

When extracting a grouped BOM values are only applied to the first component, e.g I have fields:

Only Q1 and PHOTO1 are getting the new and PHOTO2 and Q2 are not. You can have a go yourself with:

git clone https://github.com/kitnic/jelly
cd jelly && git checkout 5c45c22
kifield --extract 1-click-bom.tsv --fields ~Qty -i kicad/jelly.sch

And check kicad/jelly.sch. I also tried using xlsx instead of tsv and it didn't help.

It would be great to have an option that would combine references with identical fields into a single line. For example,

R3,,Resistors_THT:R_Axial_DIN0207_L6.3mm_D2.5mm_P7.62mm_Horizontal,470
R4,,Resistors_THT:R_Axial_DIN0207_L6.3mm_D2.5mm_P7.62mm_Horizontal,470
R5,,Resistors_THT:R_Axial_DIN0207_L6.3mm_D2.5mm_P7.62mm_Horizontal,470
R6,,Resistors_THT:R_Axial_DIN0207_L6.3mm_D2.5mm_P7.62mm_Horizontal,470
R7,,Resistors_THT:R_Axial_DIN0207_L6.3mm_D2.5mm_P7.62mm_Horizontal,10k
R8,,Resistors_THT:R_Axial_DIN0207_L6.3mm_D2.5mm_P7.62mm_Horizontal,10k

would become:

R3;R4;R5;R6,,Resistors_THT:R_Axial_DIN0207_L6.3mm_D2.5mm_P7.62mm_Horizontal,470
R7;R8,,Resistors_THT:R_Axial_DIN0207_L6.3mm_D2.5mm_P7.62mm_Horizontal,10k

That would make it easier to add things like manufacturer part number, which would be the same for identical components, rather that having to enter it separately for each reference.

Hi, after running kifield on my 4.0 schematics, it appears that they generate invalid XML as BOM parsers are failing to handle "TypeError: Cannot read property 'forEach' of undefined".

I've exported the schematic to a XSLX file and added a description field to it.
I've copy and paste the descriptions from a web page. Some special invisible character have came in this operation.
In the component below the description line was broken into two lines. Kicad is no opening such file properly. I had to open the file in gedit and remove the new line one by one.

    $Comp
    L TPS61291DRV U9
    U 1 1 588B46AE
    P 3930 3890
    F 0 "U9" H 3990 4710 60 0000 C CNN
    F 1 "TPS61291DRV" H 3990 4610 60 0000 C CNN
    F 2 "C8Tech_Power:DRV0006A" H -3770 -220 60 0001 C CNN
    F 3 "" H -3770 -160 60 0000 C CNN
    F 4 "LOW Iq BOOST CONVERTER WITH BYPASS OPERATION,
DRV0006A" H 0 0 50 0001 C CNN "description"
    F 5 "TPS61291DRV" H 0 0 50 0001 C CNN "mnp"
    F 6 "Texas Instruments" H 0 0 50 0001 C CNN "vendor"
	1    3930 3890
	1    0    0    -1  
    $EndComp

It would be great to have a command line option to define font size of added fields

I have error message while running kifield:

Traceback (most recent call last):
  File "kifield.py", line 50, in <module>
    from .sch import Schematic
ModuleNotFoundError: No module named '__main__.sch'; '__main__' is not a package

I've been attempting to use KiField on a sizable, pre-existing, schematic with lots of duplicated hierarchical sheets and I've been running into some strange issues. This is a fairly isolated one though, so I'm starting with it.

Using the hier_test schematic in tests/integration and running kifield -x hier_test.sch -i test.csv -r -g -w -d 10
results in the expected:

Refs,datasheet,footprint,new_field,value
"Y1, Y2",my_datasheet,my_footprint,my_field,Crystal

I edited test.csv to add a field and value:

Refs,datasheet,footprint,new_field,value,newfield
"Y1, Y2",my_datasheet,my_footprint,my_field,Crystal,newvalue

Running kifield -x test.csv -i hier_test.sch -r -g -w -d 10
results in the components both displaying the correct new field and value as expected when I open the schematic in Kicad.

However, if I then try to re-export the fields to the same csv file with the exact same command in the first line, the grouping seems to disappear.
text.csv:

Refs,datasheet,footprint,new_field,value,newfield
Y1,my_datasheet,my_footprint,my_field,Crystal,newvalue

Is this expected behavior? It seems like I should get back out exactly what I put in between inserting into the schematic and then extracting immediately back out.
Additionally, once I've triggered the behavior, deleting the file and running kifield -x hier_test.sch -i test.csv -r -g -w -d 10 twice in a row also results in de-grouping.

Behavior occurs regardless of the -w overwrite flag and regardless of .csv or .xlsx output.

There are some subtle differences in the debug messages, specifically a slightly difference in the Exploding sequence.

First run insertion debug log:

Inserting extracted fields into files ['test.csv'].
Inserting fields into test.csv.
Inserting extracted fields into CSV file test.csv.
Converting CSV file test.csv into an XLSX workbook.
Header on row 1: ['Refs', 'datasheet', 'footprint', 'new_field', 'newfield', 'value'].
Found Refs on header column 1.
Exploding Y1 => ['Y1'].
Exploding Y2 => ['Y2'].
Exploding Y1 => ['Y1'].
Updating Y1 field value from None to Crystal
Updating Y1 field footprint from None to my_footprint
Updating Y1 field datasheet from None to my_datasheet
Updating Y1 field new_field from None to my_field
Updating Y1 field newfield from None to newvalue
Exploding Y2 => ['Y2'].
Updating Y2 field value from None to Crystal
Updating Y2 field footprint from None to my_footprint
Updating Y2 field datasheet from None to my_datasheet
Updating Y2 field new_field from None to my_field
Updating Y2 field newfield from None to newvalue
Converting an XLSX workbook and saving as CSV file test.csv.

Second run insertion debug log:

Inserting extracted fields into files ['test.csv'].
Inserting fields into test.csv.
Inserting extracted fields into CSV file test.csv.
Converting CSV file test.csv into an XLSX workbook.
Header on row 1: ['Refs', 'datasheet', 'footprint', 'new_field', 'newfield', 'value'].
Found Refs on header column 1.
Exploding Y1, Y2 => ['Y1', 'Y2'].
Exploding Y1, Y2 => ['Y1', 'Y2'].
Updating Y1 field value from Crystal to Crystal
Updating Y1 field footprint from my_footprint to my_footprint
Updating Y1 field datasheet from my_datasheet to my_datasheet
Updating Y1 field new_field from my_field to my_field
Updating Y1 field newfield from newvalue to newvalue
Updating Y2 field value from Crystal to Crystal
Updating Y2 field footprint from my_footprint to my_footprint
Updating Y2 field datasheet from my_datasheet to my_datasheet
Updating Y2 field new_field from my_field to my_field
Updating Y2 field newfield from newvalue to newvalue
Converting an XLSX workbook and saving as CSV file test.csv.

I'm not sure where to go from here though. I'm happy to run more tests if that would be useful! I'm also investigating odd behavior where some insertions to a complex schematic leads to the schematic files corrupting with no placement information following the footprint, but that's for a separate issue.

Version info

KiField 0.1.16
Python 3.9.1
macOS 11.2
KiCad 5.1.9-0-10_14

KiCad detailed version info:

Application: KiCad
Version: (5.1.9-0-10_14), release build
Libraries:
    wxWidgets 3.0.4
    libcurl/7.64.1 SecureTransport (LibreSSL/2.8.3) zlib/1.2.11 nghttp2/1.41.0
Platform: macOS Version 10.16 (Build 20D64), 64 bit, Little endian, wxMac
Build Info:
    wxWidgets: 3.0.4 (wchar_t,STL containers,compatible with 2.8)
    Boost: 1.75.0
    OpenCASCADE Technology: 7.5.0
    Curl: 7.54.0
    Compiler: Clang 10.0.1 with C++ ABI 1002

Build settings:
    USE_WX_GRAPHICS_CONTEXT=ON
    USE_WX_OVERLAY=ON
    KICAD_SCRIPTING=ON
    KICAD_SCRIPTING_MODULES=ON
    KICAD_SCRIPTING_PYTHON3=OFF
    KICAD_SCRIPTING_WXPYTHON=ON
    KICAD_SCRIPTING_WXPYTHON_PHOENIX=OFF
    KICAD_SCRIPTING_ACTION_MENU=ON
    BUILD_GITHUB_PLUGIN=ON
    KICAD_USE_OCE=OFF
    KICAD_USE_OCC=ON
    KICAD_SPICE=ON

Latest KiField version gives the following error when inserting from CSV to SCH:
TypeError: "delimiter" must be string, not unicode

It seems to be related to this: http://stackoverflow.com/questions/5625412/python-csv-module-error

running kifield 1.1.5 with openpyxl 2.4 (installed using easy_install on windows)
i had to change in kifield.py

line 204:
from:
return pyxl.cell.column_index_from_string(cell.column), lbl_match
to:
return pyxl.cell.cell.column_index_from_string(cell.column), lbl_match

and line 612:
from
header_columns = {h.value: pyxl.cell.column_index_from_string(h.column)

to:
header_columns = {h.value: pyxl.cell.cell.column_index_from_string(h.column)

to avoid the error

AttributeError: module 'openpyxl.cell' has no attribute 'column_index_from_string'

After running
kifield -x schematicfile.sch -i schematicfile.csv

I added [I] to the header to make the refs invisible.

[I]Refs,datasheet,footprint,value
D1,~,Diode_SMD:D_SMC,SMCJ24A
D10,~,Diode_SMD:D_MiniMELF,5V
D11,~,Diode_SMD:D_MiniMELF,5V
D12,~,Diode_SMD:D_MiniMELF,5V
D13,~,Diode_SMD:D_MiniMELF,5V

But after
kifield -x schematicfile.csv -i schematicfile.sch the refs are visible in the schematic and the PCB board also.

Can someone give a working example how to make all refs invisible on a PCB board?

After running the plugin within KiCad, I keep getting the following result:- (xxxxx blanks out some personal details)

Run command:
python "C:\Program Files\KiCad\KiField\kifield/kifield.py" "C:\Users\xxxxx\Documents\CAD Project Files\Kicad Projects\UNO xxxxx\UNO xxxxx.xml""C:/Users/xxxxx/Documents/CAD Project Files/Kicad Projects/UNO xxxxx/UNOxxxxx_bom.csv"

Command error. Return code 1
Error messages:
Traceback (most recent call last):
File "C:\Program Files\KiCad\KiField\kifield/kifield.py", line 28, in
from builtins import dict
ImportError: No module named builtins

Any ideas?

Thanks for any help available.

Seems issue #3 has reared its head again. Haven't seen the fix documented anywhere, so here goes.

kifield depends on openpyxl >= 2.3.2, however installing kifield fresh (this week, on Windows and macos) installs the latest version (v2.6.0). kifield then fails to run with error:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/kifield/kifield.py", line 765, in insert_part_fields_into_wb
for h in headers}
File "/usr/local/lib/python3.7/site-packages/kifield/kifield.py", line 765, in
for h in headers}
File "/usr/local/lib/python3.7/site-packages/openpyxl/utils/cell.py", line 123, in column_index_from_string
return _COL_STRING_CACHE[str_col.upper()]
AttributeError: 'int' object has no attribute 'upper'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/bin/kifield3", line 10, in
sys.exit(main())
File "/usr/local/lib/python3.7/site-packages/kifield/main.py", line 148, in main
backup = not args.nobackup)
File "/usr/local/lib/python3.7/site-packages/kifield/kifield.py", line 1219, in kifield
insert_part_fields(part_fields_dict, insert_filenames, recurse, group_components, backup)
File "/usr/local/lib/python3.7/site-packages/kifield/kifield.py", line 1203, in insert_part_fields
insertion_functions[f_extension](part_fields_dict, f, recurse, group_components, backup)
File "/usr/local/lib/python3.7/site-packages/kifield/kifield.py", line 880, in insert_part_fields_into_csv
wb = insert_part_fields_into_wb(part_fields_dict, wb)
File "/usr/local/lib/python3.7/site-packages/kifield/kifield.py", line 768, in insert_part_fields_into_wb
for h in headers}
File "/usr/local/lib/python3.7/site-packages/kifield/kifield.py", line 768, in
for h in headers}
AttributeError: module 'openpyxl.cell' has no attribute 'column_index_from_string'

A workaround is to force install a previous version (credit to @joshberryhooves for the fix) :

pip uninstall openpyxl
pip install openpyxl==2.3.2

Some part descriptions represent inches with a double quote (e.g. 0.91" OLED). When this is imported back to KiCAD, attempting to open the schematic causes an error. The schematic can be opened only after using a text editor to remove the quote from the .sch file. Single quotes likely have the same problem.

Best would be to escape single and double quotes.

If that is difficult or causes other problems, it would be OK to give an error message and refuse to import data with quotes in the data fields.

In excel, when loading csv file that have value start with "=", "-", "+" will make excel think they are special values. I think either wrap these value in double quote, or prefix with single quote to avoid excel misinterpreted the value.

Thank for the project - it help me save some time to made my library consistency.

This isn't exactly an issue. Just wondering if this error is common and if you might know off the top of your head some idea about what might be causing it?
Thanks,
Anton

from shell:

ab@ab-mint1604 ~/src/folktek/official/13_channel/13_channel_v4 $ kifield -x 13_channel_v4.xlsx -i 13_channel_v4.sch --recurse
Traceback (most recent call last):
File "/usr/local/bin/kifield", line 9, in
load_entry_point('kifield==0.1.5', 'console_scripts', 'kifield')()
File "/usr/local/lib/python2.7/dist-packages/kifield-0.1.5-py2.7.egg/kifield/main.py", line 154, in main
recurse=args.recurse)
File "/usr/local/lib/python2.7/dist-packages/kifield-0.1.5-py2.7.egg/kifield/kifield.py", line 1030, in kifield
insert_part_fields(part_fields_dict, insert_filenames, recurse)
File "/usr/local/lib/python2.7/dist-packages/kifield-0.1.5-py2.7.egg/kifield/kifield.py", line 1014, in insert_part_fields
insertion_functions[f_extension](part_fields_dict, f, recurse)
File "/usr/local/lib/python2.7/dist-packages/kifield-0.1.5-py2.7.egg/kifield/kifield.py", line 820, in insert_part_fields_into_sch
if unquote(f['name']).lower() == field_name.lower():
AttributeError: 'NoneType' object has no attribute 'lower'

Another edge case issue that is easily worked around on the user end, just confused me for a bit.

When an Excel spreadsheet is extracted from schematic and then edited to have a cell with any format except Text containing just a number, it causes a TypeError when that spreadsheet is then inserted back into the schematic. Behavior not seen with .csv export.

Seems like Excel forces any edited cell with format of General containing just a number to remain a number, notable in that a cell with just 0805 for footprints is immediately replaced by 805. If the user is careful about resetting every cell with just a number to Text format, there are no issues. Since the Excel spreadsheet exported by KiField seems to have a string in a General formatted cell, inserting an unmodified spreadsheet causes no issues.

If possible, it would be lovely to have KiField do a sanity check or cast to string before importing from Excel cells, though I realize this is perhaps more easily solved by the user before inserting.

Thank you again for your work on this project!

Simplest possible schematic included, with example .xlsx and .csv files for comparison.
footprint.zip

footprint.sch has one resistor with a footprint of 0805.
Extracted to footprint.xlsx and footprint.csv with
kifield -x /Users/scottcandey/Desktop/footprint.sch -i ~/Desktop/footprint.csv

Change from 0805 to 0605 footprint in the Excel document causes the cell to container a number, 603.
Also changed CSV for consistency, 0805 to 0603.

Attempted Excel insertion with kifield -x ~/Desktop/footprint.xlsx -i /Users/scottcandey/Desktop/footprint.sch -d 10 results in TypeError: expected string or bytes-like object.

Extracting fields +[], -[] from files ['/Users/scottcandey/Desktop/footprint.xlsx'].
Extracting fields from /Users/scottcandey/Desktop/footprint.xlsx.
Extracting fields [], -[] from XLSX file /Users/scottcandey/Desktop/footprint.xlsx.
Header on row 1: ['Refs', 'datasheet', 'footprint', 'value'].
Found references on header column 1.
Exploding R1 => ['R1'].
Extracted Part Fields:
{'R1': {'datasheet': '~', 'footprint': 603, 'value': 'R'}}
Total Extracted Part Fields:
{'R1': {'datasheet': '~', 'footprint': 603, 'value': 'R'}}
Traceback (most recent call last):
  File "/Users/scottcandey/.local/share/virtualenvs/KiField-lIPAINC2/bin/kifield", line 8, in <module>
    sys.exit(main())
  File "/Users/scottcandey/.local/share/virtualenvs/KiField-lIPAINC2/lib/python3.9/site-packages/kifield/__main__.py", line 159, in main
    kifield(
  File "/Users/scottcandey/.local/share/virtualenvs/KiField-lIPAINC2/lib/python3.9/site-packages/kifield/kifield.py", line 1334, in kifield
    clean_part_fields(part_fields_dict)
  File "/Users/scottcandey/.local/share/virtualenvs/KiField-lIPAINC2/lib/python3.9/site-packages/kifield/kifield.py", line 1314, in clean_part_fields
    v = re.sub("[\n\r]+$", "", v) # Remove newlines at end of field.
  File "/usr/local/Cellar/[email protected]/3.9.2_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/re.py", line 210, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or bytes-like object

Attempted CSV insertion with kifield -x ~/Desktop/footprint.csv -i /Users/scottcandey/Desktop/footprint.sch -d 10 results in a successful insertion.

Extracting fields +[], -[] from files ['/Users/scottcandey/Desktop/footprint.csv'].
Extracting fields from /Users/scottcandey/Desktop/footprint.csv.
Extracting fields [], -[] from CSV file /Users/scottcandey/Desktop/footprint.csv.
Converting CSV file /Users/scottcandey/Desktop/footprint.csv into an XLSX workbook.
Header on row 1: ['Refs', 'datasheet', 'footprint', 'value'].
Found references on header column 1.
Exploding R1 => ['R1'].
Extracted Part Fields:
{'R1': {'datasheet': '~', 'footprint': '0603', 'value': 'R'}}
Total Extracted Part Fields:
{'R1': {'datasheet': '~', 'footprint': '0603', 'value': 'R'}}
Inserting extracted fields into files ['/Users/scottcandey/Desktop/footprint.sch'].
Inserting fields into /Users/scottcandey/Desktop/footprint.sch.
Inserting extracted fields into schematic file /Users/scottcandey/Desktop/footprint.sch.
Updating R1 field 3 from "~" to "~"
Updating R1 field 2 from "0805" to "0603"
Updating R1 field 1 from "R" to "R"

To isolate the issue, I also tried with the footprint cell of the Excel spreadsheet set to Text format, which also had no problem, kifield -x ~/Desktop/footprint_text.xlsx -i /Users/scottcandey/Desktop/footprint.sch -d 10

Extracting fields +[], -[] from files ['/Users/scottcandey/Desktop/footprint_text.xlsx'].
Extracting fields from /Users/scottcandey/Desktop/footprint_text.xlsx.
Extracting fields [], -[] from XLSX file /Users/scottcandey/Desktop/footprint_text.xlsx.
Header on row 1: ['Refs', 'datasheet', 'footprint', 'value'].
Found references on header column 1.
Exploding R1 => ['R1'].
Extracted Part Fields:
{'R1': {'datasheet': '~', 'footprint': '0603', 'value': 'R'}}
Total Extracted Part Fields:
{'R1': {'datasheet': '~', 'footprint': '0603', 'value': 'R'}}
Inserting extracted fields into files ['/Users/scottcandey/Desktop/footprint.sch'].
Inserting fields into /Users/scottcandey/Desktop/footprint.sch.
Inserting extracted fields into schematic file /Users/scottcandey/Desktop/footprint.sch.
Updating R1 field 3 from "~" to "~"
Updating R1 field 2 from "0803" to "0603"
Updating R1 field 1 from "R" to "R"

I have been wondering whether a similar solution exists for Eagle. I stumbled on your old ULPs and an old tutorial of yours on making Eagle BOMs but I couldn't find anything about "back importing" the attributes into the schematic. I have done some searching and no-one seems to be doing it. I haven't used Eagle enough to know, would it simply break the .sch/.brd if attributes were inserted?

Would be good to follow sub-schematics when inserting

I had to hack in a few things to get kifield.py to work for me under anaconda python3.
one was:

    if USING_PYTHON2:
        bstr = basestring
    else:
        bstr = str
    if isinstance(collapsed, bstr):

Instead of

    if isintance(collapsed, str) or isinstance(collapsed, basestring):

The other was adding a line to strip None:

    possibilities = [x for x in possibilities if x is not None]

Just after def lc_get_close_matches(...)
Because it was crashing on str.lower a few lines lower when it hit any None values.

These are total hacks of course, but they got me going. I hope someone finds this useful.

Hi,

I am currently work on a kicad project, and i use kicad 5.99 nightly version. I am trying to use KiField to generate and edit bom list for my project, but it is not working at version 5.99 even if it works at version 5. I process steps for 5.99 version as same as version 5. But result is :

• Did you provide a list of files?
• Do these files exist?
• Did you annotate your

and it is not working. (I guest it is about schematic file extensions). I would like to know that is there anyway to use KiField at nightly version?

Given the following .sch and .xlsx files:

kifield-trouble.zip

After running the following command:

kifield -x DaysToXmas.xlsx -i DaysToXmas.sch

The resulting .sch file ends up with lines like this:

F 4 "11 Position Header Connector 0.100" (2.54mm) Through Hole Gold" H 3950 6950 50 0001 C CNN "description"

which EESchema then interprets as being a field named H 3950 6950 50 0001 C CNN.

I do this:

kifield -x Transmitter.sch -i Transmitter.xlsx

and get this:

There are no field values to insert!

It seems specific to this particular file, but I don't know why:

Transmitter.zip

This is with KiField 0.1.12 on Mac OS X 10.9.5.

ws.rows and ws.columns now always return generators and start at the top of the worksheet

http://openpyxl.readthedocs.org/en/2.4/changes.html#id139

messes up line 247 of kifield.py

OS: MacOS 10.14.4
Python: 3.7.3
Kifield: 0.1.15

command: kifield -x rc2014_monitor.sch -i rc2014_monitor_fields.csv
rc2014_monitor.sch.txt

Traceback (most recent call last): File "/usr/local/bin/kifield", line 10, in <module> sys.exit(main()) File "/usr/local/lib/python3.7/site-packages/kifield/__main__.py", line 148, in main backup = not args.nobackup) File "/usr/local/lib/python3.7/site-packages/kifield/kifield.py", line 1209, in kifield insert_part_fields(part_fields_dict, insert_filenames, recurse, group_components, backup) File "/usr/local/lib/python3.7/site-packages/kifield/kifield.py", line 1193, in insert_part_fields insertion_functions[f_extension](part_fields_dict, f, recurse, group_components, backup) File "/usr/local/lib/python3.7/site-packages/kifield/kifield.py", line 870, in insert_part_fields_into_csv wb = insert_part_fields_into_wb(part_fields_dict, wb) File "/usr/local/lib/python3.7/site-packages/kifield/kifield.py", line 747, in insert_part_fields_into_wb ws.cell(row=1, column=c).value = lbl File "/usr/local/lib/python3.7/site-packages/openpyxl/cell/cell.py", line 252, in value self._bind_value(value) File "/usr/local/lib/python3.7/site-packages/openpyxl/cell/cell.py", line 205, in _bind_value value = self.check_string(value) File "/usr/local/lib/python3.7/site-packages/openpyxl/cell/cell.py", line 169, in check_string raise IllegalCharacterError openpyxl.utils.exceptions.IllegalCharacterError

I generated an .xlsx file with the following command:

kifield -x Rgb7Hat.sch -i Rgb7Hat.xlsx -g -r

It has the following issues:

• The "mpn" and "description" columns are missing.
• The "footprint" column often contains the value of the "description" field instead of the value of the "footprint" field.
• There is an extra column named "60".
• FID2-FID6 entries are missing.

kifield-trouble.zip

I'm using KiField 0.1.7 on Mac OS X 10.9.5.

Would be good to clean up the commandline in non backwards compatible way:

1. Set --recurse and --group to default to true
2. Get rid of--insert and --extract switches, instead:

kifield <extract-file> <insert-file>

If multiple files need to be specified (seems rare now that we have --recurse):

kifield <extract-file-1> <extract-file-2> -- <insert-file-1> <insert-file-2>

(this has some precedence, e.g. with git: git diff [options] [<commit>] [--] [<path>...])

I have a "μ" in my part description and this caused KiField to fail with following error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xce in position 14: ordinal not in range(128)

I fixed this by adding following lines in kifield.py:

reload(sys)
sys.setdefaultencoding('utf8')

I understand the setdefaultencoding() usage is not encouraged and the reload of sys is a workaround. But in the end it worked for me. See following link: http://stackoverflow.com/questions/3828723/why-should-we-not-use-sys-setdefaultencodingutf-8-in-a-py-script

I use python 2.7 and KiField 0.1.1 on OS X 10.11.3

Have done multiple imports/exports in the past without issue. It appears I've borked something and am wondering if there's a way to track down where I may have mistakenly inserted a bad character. Is it safe to assume that it was due to something I inserted from this error message?

Can no longer do an export or an import.

Traceback (most recent call last):
  File "/usr/local/bin/kifield", line 9, in <module>
    load_entry_point('kifield==0.1.12', 'console_scripts', 'kifield')()
  File "/usr/local/lib/python2.7/dist-packages/kifield-0.1.12-py2.7.egg/kifield/__main__.py", line 148, in main
    backup = not args.nobackup)
  File "/usr/local/lib/python2.7/dist-packages/kifield-0.1.12-py2.7.egg/kifield/kifield.py", line 1207, in kifield
    insert_part_fields(part_fields_dict, insert_filenames, recurse, group_components, backup)
  File "/usr/local/lib/python2.7/dist-packages/kifield-0.1.12-py2.7.egg/kifield/kifield.py", line 1191, in insert_part_fields
    insertion_functions[f_extension](part_fields_dict, f, recurse, group_components, backup)
  File "/usr/local/lib/python2.7/dist-packages/kifield-0.1.12-py2.7.egg/kifield/kifield.py", line 866, in insert_part_fields_into_csv
    wb, dialect = csvfile_to_wb(filename)
  File "/usr/local/lib/python2.7/dist-packages/kifield-0.1.12-py2.7.egg/kifield/kifield.py", line 210, in csvfile_to_wb
    setattr(dialect, attr, bytes(a))
  File "/usr/local/lib/python2.7/dist-packages/future-0.16.0-py2.7.egg/future/types/newbytes.py", line 93, in __new__
    raise TypeError('unicode string argument without an encoding')
TypeError: unicode string argument without an encoding

This would be another step towards #5 but could be generally useful as well. Group references by value into lines like "C1, C2, C4 - C10".

One of the reasons I don't use kifield as often as I otherwise might is that it creates large, unnecessary diffs. This makes it harder to tell what has changed in a particular commit.

There are at least three ways kifield creates larger diffs than necessary:

• Changes in whitespace on an individual line: In certain places, KiCad uses two spaces between tokens instead of one. Kifield always puts just one space between tokens.
• Changes in field numbers: If my original file has F4 as "MPN", F5 as "Description", and F6 as "Digi-Key_PN", Kifield might change it to be F4 as "Description", F5 as "Digi-Key_PN", and F6 as "MPN".
• Ordering within the file: Kifield moves all components to come first in the file, and everything else (wires, junctions, etc.) to come after all the components.

At the moment KiField seems to scatter newly added fields of a symbol all over the schematic sheet. I'd suggest the following: if a field is not already defined in schematic symbol then add it to this symbol origin coordinate (vertical and horizontal justification should be "center")

When generating a BOM from a schematic, you can group components with the same value using the -g or --group command line switch, but in this case, I would expect a "Quantity" column to include the number of components. Is there a way to generate this automatically?

kifield -x *.dcm -x test.xlsx
I got a test,xlsx with all the symbol names but their values are empty - I'm sure I have all the docfield setting as "relative path" - they open fine in KiCad4.7. May be the relative path been see as invalid value, and removed by the "sch.py".

Hello,
I installed Kifield in KiCad's python environment. The installation went well with Admin rights.
Unfortunately, launching kifield.exe from command prompt gives this error:
failed to create process (C:\Program "Files\KiCad\bin\python.exe" "c:\Program Files\KiCad\bin\kifield-script.py" "-x" "MyFile.sch" "-i" "MyBOMfile.xlsx").
You may notice that the double quote is in the bad position....
Regards