Giter VIP home page Giter VIP logo

mitya33 / cfimp Goto Github PK

View Code? Open in Web Editor NEW
20.0 2.0 6.0 112 KB

A simple but powerful spreadsheet-derived (CSV, TSV) data importer for the Contentful headless CMS. In so doing, it can also link items to references, assets or tags, and even publish items. It can also be used to update existing items rather than import new ones.

JavaScript 100.00%
contentful javascript nodejs cli headless-cms

cfimp's Introduction

cfimp

cfimp is a simple but powerful CLI tool for importing/updating CSV data in the Contentful headless CMS. The default delimiter is tab.

In the process of importing/updating, entries can optionally be linked to (existing) assets, tags or references (foreign items).

cfimp cannot be used to create new assets, tags, models or anything other than entries.

Why

Contentful doesn't make it super easy to import data. There's no GUI; instead the contentful-cli tool has an import command, but it's sparsely documented, and you first have to munge your data into JSON. There's no easy way to import spreadsheet-derived data. Further, Contentful doesn't make explicit what the structure of the JSON should be.

Examples

#Import comma-separated data from import.csv to space "12345" / content type (model) "authors" / locale "en-GB"
cfimp -space:12345 -model:authors -locale:en-GB 

#Also specify some tags (for all rows)
cfimp -space:12345 -model:authors -locale:en-GB -tags:foo,bar

#Specify a fallback (default) value "bar" for the "foo" field
cfimp -space:12345 -model:authors -locale:en-GB -dfltvals:foo=bar

#Preview the generated JSON of the first entry - no actual import takes place
cfimp -space:12345 -model:authors -locale:en-GB -preview

Prerequisites

  1. Install contentful-cli globally.
npm install -g contentful-cli
  1. Authenticate with Contentful (optional).
contentful login

Authenticating this way will save the credentials in your environment so you don't have to authenticate manually each time you use cfimp. If you'd rather do that instead, though, see the mtoken argument.

  1. Install cfimp globally
npm i -g cfimp

Usage

Note: the default delimiter is tab. This can be changed via the delim arg.

It's strongly recommended to preview the generated data before running the write. See the preview arg.

cfimp <args>

Arguments are specified in the format -arg:val or, where the argument doesn't accept a value (denoted * below), simply -arg. Where val contains spaces, use -arg:"one two".

Valid arguments are as follows.

  • input - path to the input file (optional; default: "import.csv")
  • space - the ID of the Contentful space to write to (required)
  • model - the ID of the Contentful model (content type) to write to (required)
  • locale - the locale, as defined in Contentful, e.g. "[en-GB]". See Writing to multiple locales (required)
  • preview* - if passed, shows a preview of the data that will be written to Contentful; no write is performed. See Troubleshooting (optional)
  • previewfile* - if passed, quits after generating the Contentfil JSON file, so you can inspect it. See Troubleshooting (optional)
  • env - the ID of the Contentful environment to use (optional; default: "master")
  • publish* - sets the imported/updated entries to "published" status rather than "draft" (optional)
  • mergevals - a com-sep list of field=value pairs - to merge into all rows. See Merge and default values (optional)
  • dfltvals - a com-sep list of field=value defaults to be used anywhere a row has empty cells. See Merge and default values (optional)
  • delim - the delimiter separating columns (for multi-column files) - one of "tab", "com" (comma) or any other string (optional; default: "tab")
  • fields - the fields to import into. If omitted, cfimp will assume the first row of the input data denotes the fields (optional)
  • enc - the file encoding for your data (you shouldn't need to change this) - one of "utf8", "ascii" or "base64" (optional; default: "utf8")
  • offset - a 1-index offset (row) to begin reading data from in your input file (optional)
  • limit - a limit as to the number of rows to process (optional)
  • skiprows - a com-sep list of strings which, if any is found in a row (any column), will cause that row to be skipped. The logic can be inverted. See Filtering rows (optional)
  • skipfields - a com-sep list of field IDs to ignore from the input. Useful if your spreadsheet export contains columns you don't want to include (optional)
  • nocast - ordinarily, numbers, true and false will be cast to their integer/boolean equivalents when data is passed to Contentful. Pass true to prevent this (i.e. if you literally want to pass "true" not true) (optional)
  • tagall - a com-sep list of (existing) tags to tag all entries with. You can also specify row-specific tags. See Tagging items (optional)
  • listdelim - the delimiter used within list values e.g. default values, tags (optional; default: ","). See Delimiter overrides
  • mtoken - a management token to authenticate with Contentful. You can omit this if you've already authenticated via contentful login (optional)

Reference and asset links

It's possible to link to existing assets or references (i.e. foreign items in other content types) via the ref- (reference) and refa- (asset) prefixes.

In both cases, you'll need to know the ID of the item you're linking to.

Suppose you had a content type of authors and had a field, authorPhoto which was an asset field. We'd link authors to their photos like so:

forename	surname	authorPhoto
Philippa	Gregory	refa-12345
Desdemona	Johnson	refa-67890

If for some reason all our authors have the same face and photo, we can even specify this at runtime with a merge value (ee Merge and default values):

cfimp -space:12345 -model:authors -locale:en-GB -mergevals:authoBioPhoto=refa-12345

Merged and default values

It's possible to specify default fallback values for your data, which will take effect if the cell is empty for that field.

It's also possible to merge extra data into all rows.

Let's say you have a field in your data, "popular", with some rows having "yes" as a value. For all others, with no value, you want to insert "no". We can accomplish this via the dfltvals argument:

cfimp -space:12345 -model:authors -locale:en-GB -dfltvals:popular=no

Or let's say you want to add an extra field to all rows. Perhaps you meant (but forgot) to add an "age" column to your spreadsheet data before exporting it, and it so happens that, surprisingly, all the authors in your data are 51. We can accomplish this via the mergevals argument:

cfimp -space:12345 -model:authors -locale:en-GB -mergevals:age=51

Updating existing items

Be careful when updating existing items; be sure to specify values for all fields, because Contentful's import service doesn't retain values for omitted fields.

cfimp can be used to update existing items in Contentful rather than import (create) new ones. To do this, include an _id column in your data. This will be inferred as the internal Contentful ID of the item, and will update it.

forename	surname	_id
Philippa	Gregory	12345

Multiple locales

All data in Contentful is stored against locales, created by you in Contentful. This allows you to have multiple versions of each piece of data, for different locales. By default, cfimp will import/update data using the locale specified in the locale arg.

However you can import/update multiple locales at once. To do so, specify the field as many times as you have locales, with each additional one appended with a locale flag.

So if your data was:

London	Londres
Berlin	Berlina

You can spefify locales either in the data itself, if the first row of data represents your field IDs:

city	city[es-SP]
London	Londres
Berlin	Berlina

...or via the fields argument, if you're specifying field IDs at runtime.

cfimp -space:12345 -model:cities -locale:en-GB -fields:city,city[es-SP]

Tagging items

It's possible to tag items to (existing) tags when importing or updating items. There are two ways to do this.

You can specify item-specific tags in your data, via the _tags field.

forename	surname	_tags
Philippa	Gregory	foo,bar

You can also tag all items at runtime via the tagall argument.

cfimp -space:12345 -model:authors -tagall:foo,bar

Note: tags specified in this way are subject to the listdelim argument.

Importing geopositional data

When importing geopositional data, specify coordinates in the format

lat,lng

i.e.

51.467283887530094,-0.24193970047025806

cfimp will convert this into an object before writing to Contentful. Note also that this means you'll need to use a non-comma value for delim, since the geoposition data itself contains a comma.

Filtering rows

It's possible to stipulate which rows you do, or do not, want to be imported/updated from your input file, via the skiprows argument.

This argument is used to whitelist or blacklist certain rows, and accepts a com-sep list of values to blacklist or, if the entire argument value is prefixed with !, to allow.

The following will blacklist (skip) any rows that contain, in any column, "foo" OR "bar"

cfimp ... -skiprows:foo,bar

The following will whitelist (include) any rows that contain "foo" OR "bar"; all other rows will be skipped.

cfimp ... -skiprows:!foo,bar

Publishing entries

cfimp can publish entries when importing or updating them. To stipulate this, pass the publish argument. If this is omitted, the entry will end up in draft status or, for updated entries, whatever its current status is.

Note that, when importing and publishing new items, cfimp will generate item IDs itself, rather than leave it to Contentful. This is a requirement of Contentful's import tool; in order to publish items, it requires an ID to be generated by the client.

Such IDs will be formed of 11 randomly alphanumeric characters.

Delimiter overrides

Delimiters are factors in two areas of cfimp:

  • the delimiter used to separate the values in your input file
  • the delimiter used to separate any (normally) comma-separated pairings in arguments or _tag field values

Both of these can be overriden - the former via the delim arg and the latter via the listdelim arg. Note that listdelim will apply to all occasions where cfimp is attempting to decipher something that it normally expects to be in com-sep format - so for example _tags fields, the mergevals argument, and so on.

Troubleshooting

It's HIGHLY recommended to preview the generated data before running the actual import/data. This shows you what cfimp intends to send to Contentful for import/update.

This can be done in two ways:

  • Via the preview argument: this should be considered preview 'lite', and will spit out in the console the data cfimp has compiled, before converting to the JSON format Contentful requires
  • Via the previewfile argument: this goes a step further, and actually creates the JSON file Contentful requires that effects the import/update. Once created, you can inspect this file.

(Note: Neither argument expects a value.)

Additionally, a limit is handy in order to avoid numerous terminal screens of data.

If you find cfimp is deriving or malformed bad data, check the delim and listdelim args.

Like this?

If I've helped you, consider being amazing and buying me a coffee - thank you!

cfimp's People

Contributors

mitya33 avatar billycychan avatar jchlu avatar moghammed avatar andresharpe avatar

Stargazers

 avatar Chris W. Burke avatar crgbsly avatar Serhiy Dmytryshyn avatar Luke Matthew Sutton avatar Jason Loeffler avatar Gian Faye Paguirigan avatar  avatar  avatar Jeffrey avatar  avatar F.B.N avatar Phil Wolstenholme avatar  avatar Dashon 'DJ' Hawkins avatar Jeremy O'Day avatar  avatar Jeff Griffith avatar  avatar Charlie Chrisman avatar

Watchers

James Cloos avatar  avatar

cfimp's Issues

Multiple references not supported in field data

Bit of a feature request, and I may even submit a PR to do this...

Could reference fields other than tags and entry IDs be supported?

Thinking some kind of extra option like -multivalfields:[Field ID] and then the listdelim function could be used for the specified fields.

Thank you!

_id column does not seem to work as specified in docs

First of all, thanks for this great tool!

I have created a CSV to update existing entries, and I get the error: UnknownField: Status: 422 - Unprocessable Entity - Message: No field with id "_id" found

Sample CSV

_id,internalName,locale,navigationComponents
3jpUCXzwOSPynzFAQiWvUg,[Navigation Collection] [en_US] [NY],7rbkJNbNuAWIOsWKeG1Mbb,1C3p8Nykj9GH2UiuqTyPnv

This is what the preview shows me:

{
      "_tags": [],
      "id[default]": "3jpUCXzwOSPynzFAQiWvUg",
      "internalName[default]": "[Navigation Collection] [en_US] [NY]",
      "locale[default]": "7rbkJNbNuAWIOsWKeG1Mbb",
      "navigationComponents[default]": "1C3p8Nykj9GH2UiuqTyPnv"
}

Running command: npx cfimp -space:[SPACE ID] -env:[ENV ID] -model:navigation-collection -input:input.csv -locale:default -delim:com

Is there something I'm doing wrong here? Possibly a Contentful API update ruining our day? :)

can't update fields displayed on Contentful as 'Dropdown'

Can only upload text data for fields when their type is set to 'Single line' rather than 'Dropdown', even when the submitted text exactly matches validation criteria.

The workaround is just to change how the field is displayed on Contentful to 'Single line'. Not a massive issue, but not always ideal.

booleans must be in lower case

When passing a .csv file with boolean values in upper case, i.e. TRUE,FALSE the following error message was produced:

12:06:23 - InvalidEntry: Status: 422 - Unprocessable Entity - Message: Validation error - Details: The type of "value" is incorrect, expected type: Boolean, The type of "value" is incorrect, expected type: Boolean, The type of "value" is incorrect, expected type: Boolean - Request ID: 79dd5de1-5363-4f36-a474-9c2abe4bb313

I was encountering this frequently as Excel defaults to capitalising booleans if cell format is set to General rather than Text.

So not necessarily an issues, as the workaround is trivial, but perhaps an easy fix or something to mention in documentation.

Document example of correct csv formatting

Looks like an incredibly useful tool! Unfortunately, I'm unable to get this working. The parser seems to bail out after the first letter of the first row.

Running:

npx cfimp -space:01vk1xo6n33v -model:property -locale:en-US -fields:"title" -input:import.csv -previewfile

With the following import file, where title is the destination field title:

title
first 
second

results in:

{
  "entries": [
    {
      "metadata": {
        "tags": []
      },
      "sys": {
        "contentType": {
          "sys": {
            "type": "Link",
            "linkType": "ContentType",
            "id": "property"
          }
        }
      },
      "fields": {
        "title": {
          "en-US": "t"
        }
      }
    }
  ]
}

Omitting the fields argument results in:

{"entries":[]}

Explicitly setting the title field to:

title[en-US]

yields the same result.

I have also tried this copying, pasting, and modifying your example record from the readme with the same result, i.e.:

title
London
Berlin

bad column is retrieve in case of many local

Good morning, your script is awesome ! I just seen a small bug. In case where you want to import multiple locales, the script detect import first label for another language:

Example of CSV:
slug,label,label[fr]
"add-to-cart","Add to cart","Ajouter au panier"

Preview:
[
{
"_tags": [],
"slug[en]": "add-to-cart",
"label[en]": "Add to cart",
"label[fr]": "Add to cart"
}
]

Command used:
cfimp -space:xxxxxx -env:dev -model:microCopy -input:example.csv -locale:en -delim:"com" -preview

cfimp read only first line in csv

csv file:

text1,1050,2100,ref-123
text2,1060,2200,ref-124
text3,1070,2400,ref-124

Line command: npx cfimp -space:space -model:modelname -locale:en-US -delim:com -fields:name,amount,oldAmount,locale -preview
return:
image

And if I will try to run it without preview, only first line will be upliaded

Quotes are not supported

The following should produce 2 fields, but it will produce 4 because quoting is not supported

columnOne, columnTwo
this works fine, "this, however, does not"

field names not recognised in .csv

Use of -fields produced expected behaviour:

npx cfimp -space:9xn2m7v6095a -model:project -input:test.csv -delim:com -fields:title,hideYearInfo -locale:en-US

test project true
test project false

Without fields the command fails:

npx cfimp -space:9xn2m7v6095a -model:project -input:test.csv -delim:com -locale:en-US

title hideYearInfo
test project true
test project false

The following 2 errors and 0 warnings occurred: 12:20:35 - UnknownField: Status: 422 - Unprocessable Entity - Message: No field with id "title" found. - Details: unknown - Request ID: b60c09f0-1236-4c5f-bae9-b5543fdb06a9 12:20:35 - UnknownField: Status: 422 - Unprocessable Entity - Message: No field with id "title" found. - Details: unknown - Request ID: 63f2d119-4cb0-488c-bc5e-9daab93c08b1

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.