To avoid bugs in the web scraping phase of the Jekyll importer, I tried using the --scrape=none option documented in the README. In the output, it didn't claim to being doing any scraping after this, but the resulting .zip produced includes a number of photos downloaded from Flickr, a few photos hosted by the blog itself and an audio file that got scraped as well.

I'm not sure where this bug is. The mg-jekyll-export does not directly call this code.

In the case of the images scraped from Flickr, there's no easy way undo this, since the URLs in the HTML must have been updated to be local URLs instead of absolute.

When I run the following command, I get an error message when processing blogger content.

migrate blogger --blogID 6521985176416791926 --apiKey <redacted>

I get the following log:

[2024-02-23 14:25:21] INFO
(node:39239) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
(node:39239) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 unpipe listeners added to [WriteStream]. Use emitter.setMaxListeners() to increase limit
(node:39239) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 error listeners added to [WriteStream]. Use emitter.setMaxListeners() to increase limit
(node:39239) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [WriteStream]. Use emitter.setMaxListeners() to increase limit
(node:39239) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 finish listeners added to [WriteStream]. Use emitter.setMaxListeners() to increase limit
✔ Initializing
   →  Workspace initialized at /var/folders/5l/20flvwpj4fj9g0dyhq637qr80000gn/T/mg/cc2b7550907cd55f6c8dc01b5c7b1341-blogger-6521985176416791926
✔ Get content from API
 ✔ Fetching posts for 6521985176416791926
✖ Process Blogger API JSON
  Fetch missing data via WebScraper
  Build Link Map
  Format data as Ghost JSON
  Fetch images via AssetScraper
  Update links in content via LinkFixer
  Convert HTML -> MobileDoc
  Write Ghost import JSON File
  Write Ghost import zip
[2024-02-23 14:25:21] INFO
Error: ENOENT: no such file or directory, open '/Users/whaleyr/pharmgkb/ghost-migration/localhost_development.log'
    at readFileSync (node:fs:456:20)
    at showLogs (file:///usr/local/lib/node_modules/@tryghost/migrate/lib/utilties/cli-log-display.js:8:21)
    at TypeCommand.run [as _runHandler] (file:///usr/local/lib/node_modules/@tryghost/migrate/commands/blogger.js:130:5)
    at async Promise.all (index 4)

I re-ran the command with the verbose flag and got the following log

[2024-02-23 14:25:51] INFO
[STARTED] Initializing
[OUTPUT] Workspace initialized at /var/folders/5l/20flvwpj4fj9g0dyhq637qr80000gn/T/mg/cc2b7550907cd55f6c8dc01b5c7b1341-blogger-6521985176416791926
[COMPLETED] Initializing
[STARTED] Get content from API
[STARTED] Fetching posts for 6521985176416791926
(node:39351) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
(node:39351) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 unpipe listeners added to [WriteStream]. Use emitter.setMaxListeners() to increase limit
(node:39351) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 error listeners added to [WriteStream]. Use emitter.setMaxListeners() to increase limit
(node:39351) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [WriteStream]. Use emitter.setMaxListeners() to increase limit
(node:39351) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 finish listeners added to [WriteStream]. Use emitter.setMaxListeners() to increase limit
[COMPLETED] Fetching posts for 6521985176416791926
[COMPLETED] Get content from API
[STARTED] Process Blogger API JSON
[FAILED] Cannot read properties of null (reading 'url')
[2024-02-23 14:25:51] INFO
Error: ENOENT: no such file or directory, open '/Users/whaleyr/pharmgkb/ghost-migration/localhost_development.log'
    at readFileSync (node:fs:456:20)
    at showLogs (file:///usr/local/lib/node_modules/@tryghost/migrate/lib/utilties/cli-log-display.js:8:21)
    at TypeCommand.run [as _runHandler] (file:///usr/local/lib/node_modules/@tryghost/migrate/commands/blogger.js:130:5)
    at async Promise.all (index 4)

I recently tried to import all my posts from medium into my local ghost installation. While I was able to import all the posts, none of the posts had any tags created. The only tag that was set on each post was #medium.

It would be nice if the tags were also imported after using migrate medium ... command to generate the Ghost friendly zip and importing it.

It's great to not break or change URLs when possible, and importing permalink and basename will help with that.

In my config, I have permalink and basename defined for nearly all my posts, by my Jekyll blog was in turn migrated from Movable Type, and explicitly importing those helped me not break the URLs during the /last/ platform transition.

Thanks!

There is a code in lib/process-post.js to process files in a "drafts" folder, but the code path is inaccessible because in lib/read-zip.js, only content in the "posts" folder is read, ignoring anything in the "drafts" folder.

As mentioned in a recent PR, the correct folder names are "_posts" and "_drafts".

The fix is to update the regex in lib/read-zip.js to also pass along content in _drafts as well.

When I run the command migrate medium my-export-file.zip I get many Unable to scrape URI errors for images such as:

1997/2862: posts: are-there-any-gauchos-left-in-argentina feature_image - Unable to scrape URI https://cdn-images-1.medium.com/max/2560/1*utPlKXF5t00s2isvG7SZZw.jpeg Total: 2862. Complete: 2541, Failed: 319, Skipped: 0, Disabled: 0

However in my browser the image URLs are accessible. Furthermore after the statement Write Ghost import zip nothing happens for a while and at some point it states client_loop: send disconnect: Broken pipe

I am not sure if I am doing something wrong or if the migration script is broken. I do not use a custom domain. I have 166 blog posts which I want to migrate. When I first tried with less blog posts (I think less than 100), I also got the errors but at least a zip file was created on the server which I was able to use to import into Ghost. Right now the script does not create any zip archive on my server.

mg-jekyll-export currently handles updating URLS in ` "src" attributes. It would be great of the "srcset" attribute was handled as well.

In the #506 PR, there's now an HTML processing function that takes an old blogs's HTML input and produces cleaned / migrated HTML as output. This function is not specific to Jekyll at all. A lot of other import sources could use the same kind of HTML processing, so it seems like it makes sense to have that logic to a shared location, the way that mg-linkfixer is handled. The HTML processing code in the Jekyll export tool is mostly concerned with link processing, but it also handles some other kinds of bad HTML, otherrwise I would suggest putting the code in mg-linkfixer.

This is a feature request.

I have a use-case where I want to create my new blog on ghost while keeping my medium posts around as well. To avoid search engines from penalising my site, I want to make sure that when I import all the posts from medium into ghost (using migrate medium utility) the canonical URL is set to the original post on published on medium.

Introduction:
This issue highlights a vulnerability in the nth-check package, specifically related to Regular Expression Denial of Service (ReDoS). This vulnerability is identified with a CVSS score of 7.5 (High Severity) by both Snyk and NVD.

Details:
The vulnerability is introduced through @tryghost/[email protected] and affects versions of nth-check prior to 2.0.1.

Exploit Maturity:
The exploit maturity is identified as Proof of Concept.

Detailed Paths:

Introduced through: @tryghost/[email protected] › @tryghost/[email protected] › @tryghost/[email protected] › [email protected] › [email protected] › [email protected]

Security Information:

Snyk: CVSS 7.5 - High Severity
NVD: CVSS 7.5 - High Severity

Overview:
nth-check is a library used for parsing CSS nth-child expressions.

Vulnerability Description:
Affected versions of this package are vulnerable to Regular Expression Denial of Service (ReDoS) when parsing crafted invalid CSS nth-checks. This vulnerability is due to the sub-pattern \s*(?:([+-]?)\s*(\d+))? in RE_NTH_ELEMENT with quantified overlapping adjacency. An attacker can exploit this by providing a specially crafted input, leading to excessive backtracking during regex processing, which may result in a denial of service condition.

Remediation:
Upgrade to version 2.0.1 or later of nth-check to fix this vulnerability. Unfortunately, there is no remediation path available for previous versions.

Proposed Changes:
Update the dependency on nth-check to version 2.0.1 or later in the package.json file.

Testing:
After updating the dependency, ensure that all existing functionality continues to work as expected. Perform thorough testing to verify that the vulnerability has been mitigated.

Using the CLI migrate tool on my substack and about half of them have broken feature images. Same thing also happens with the Beta migrator tool in Labs since it's probably using this exact same code

migrate substack -v
0.36.2

The feature_image export in ghost-import.json features a CDN's URL instead of the expected local scraped copy.
Following the CDN link yields an "Access Denied" error

"title": "We might not see leap seconds after 2035 🤯",
...
"feature_image": "https://bucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com/public/images/7c193e59-fad6-49df-b659-b16976e1ce59_1024x683.jpeg",

I went to the originating post in the exported html from Substack (exported 2023-12-28), and the top image that should've been converted to the featured_image is this img tag. Looks like the img sources the "bucketeer" AWS host that is the broken url being imported, and also has a data-attrs referencing the same broken url. Not sure which one the migrating tool is pulling. The tag also provides a raft of srcsets to actual images that are downloadable, so the html actually displays in a browser.

<img src="https://bucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com/public/images/7c193e59-fad6-49df-b659-b16976e1ce59_1024x683.jpeg" 

width="1200" 
height="800.390625" 

data-attrs="{&quot;src&quot;:&quot;https://bucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com/public/images/7c193e59-fad6-49df-b659-b16976e1ce59_1024x683.jpeg&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:false,&quot;imageSize&quot;:&quot;large&quot;,&quot;height&quot;:683,&quot;width&quot;:1024,&quot;resizeWidth&quot;:1200,&quot;bytes&quot;:188889,&quot;alt&quot;:null,&quot;title&quot;:null,&quot;type&quot;:&quot;image/jpeg&quot;,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null}" 

class="sizing-large" 
alt="" 

srcset="https://substackcdn.com/image/fetch/w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F7c193e59-fad6-49df-b659-b16976e1ce59_1024x683.jpeg 424w, 
https://substackcdn.com/image/fetch/w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F7c193e59-fad6-49df-b659-b16976e1ce59_1024x683.jpeg 848w, 
https://substackcdn.com/image/fetch/w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F7c193e59-fad6-49df-b659-b16976e1ce59_1024x683.jpeg 1272w, 
https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F7c193e59-fad6-49df-b659-b16976e1ce59_1024x683.jpeg 1456w" 

sizes="100vw" 
fetchpriority="high">

The problem seems to affect all my posts prior to around January 2023, but it's not clear why there's a difference at all.

example of a broken html file from the export here
buggy_html.zip

In addition to having a "_drafts" for folder, Jekyll also allows setting "published: false" in the frontmatter.

This could be used for example to "unpublish" a finished post that has expired in usefulness.

Ref: https://jekyllrb.com/docs/posts/#the-posts-folder

I'll take a look to see if I can submit a PR for this.

Introduction:
This is a request to address a vulnerability in the sharp package, specifically related to a Heap-based Buffer Overflow. This vulnerability is identified with a CVSS score of 9.6 (Critical Severity) by Snyk and 8.8 (High Severity) by NVD.

Details:
The vulnerability is introduced through @tryghost/[email protected] and affects versions of sharp prior to 0.32.6.

Exploit Maturity:
The exploit maturity is identified as Mature.

Detailed Paths and Remediation:

Introduced through: @tryghost/[email protected] › @tryghost/[email protected] › [email protected]. Fix: Upgrade to @tryghost/[email protected].
Introduced through: @tryghost/[email protected] › @tryghost/[email protected] › @tryghost/[email protected] › [email protected]. Fix: Upgrade to @tryghost/[email protected].

Security Information:

Snyk: CVSS 9.6 - Critical Severity
NVD: CVSS 8.8 - High Severity

Overview:
sharp is a High performance Node.js image processing, the fastest module to resize JPEG, PNG, WebP, GIF, AVIF, and TIFF images.

Vulnerability Description:
Affected versions of this package are vulnerable to a Heap-based Buffer Overflow when the ReadHuffmanCodes() function is used. An attacker can exploit this vulnerability by crafting a special WebP lossless file that triggers the ReadHuffmanCodes() function, leading to a heap-based buffer overflow. This vulnerability can potentially allow arbitrary code execution.

Remediation:
Upgrade to version 0.32.6 or later of sharp to fix this vulnerability. Additionally, upgrade @tryghost/mg-fs-utils to versions 0.12.18 or 0.12.14 as indicated for the respective paths.

Proposed Changes:
Create an issue in the project repository to track the resolution of this vulnerability. This issue should outline the steps needed to mitigate the vulnerability, including upgrading sharp and @tryghost/mg-fs-utils to the recommended versions.

Changelog:

2023-09-12: Initial advisory publication
2023-09-27: Advisory details updated, including CVSS, references
2023-09-27: CVE-2023-5129 rejected as a duplicate of CVE-2023-4863
2023-09-28: Research and addition of additional affected libraries
2024-01-28: Additional fix information

This tool fails if Medium user posts are added to a publication with custom domains.

For example https://medium.com/@anangsha/i-owe-an-apology-to-mostly-harmless-c5503f1f2acc this article redirects to https://baos.pub/i-owe-an-apology-to-mostly-harmless-c5503f1f2acc which is a Medium publication which has its custom domain.

This tool fails for all such posts.

https://www.npmjs.com/package/hubspot-api is no longer maintained. It should be replaced by https://www.npmjs.com/package/@hubspot/api-client

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

These problems occurred while renovating this repository. View logs.

• WARN: Using npm packages for Renovate presets is now deprecated. Please migrate to repository-based presets instead.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update dependency ts-jest to v29.1.4
• chore(deps): update dependency typescript to v5.4.5
• chore(deps): update test & linting packages to v7.11.0 (@typescript-eslint/eslint-plugin, @typescript-eslint/parser)
• fix(deps): update dependency sharp to v0.33.4
• fix(deps): update dependency date-fns to v3
• fix(deps): update dependency got to v14
• fix(deps): update dependency listr2 to v8
• fix(deps): update dependency stripe to v15
• chore(deps): lock file maintenance
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Hey, thanks for the tool!

I think something might be up with the latest release. Upon a fresh global install with Node.js v18.14.2 (latest lts), I hit a dependency problem when I run migrate:

Error [ERR_MODULE_NOT_FOUND]: Cannot find package '@tryghost/mg-letterdrop' imported from /Users/chris/.nvm/versions/node/v18.14.2/lib/node_modules/@tryghost/migrate/sources/letterdrop.js
    at new NodeError (node:internal/errors:399:5)
    at packageResolve (node:internal/modules/esm/resolve:889:9)
    at moduleResolve (node:internal/modules/esm/resolve:938:20)
    at defaultResolve (node:internal/modules/esm/resolve:1153:11)
    at nextResolve (node:internal/modules/esm/loader:163:28)
    at ESMLoader.resolve (node:internal/modules/esm/loader:838:30)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:424:18)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:77:40)
    at link (node:internal/modules/esm/module_job:76:36) {
  code: 'ERR_MODULE_NOT_FOUND'
}

Rolling back to the 0.29.5 release worked, and I was able to migrate a small substack site super easily (nice)!

Hello, when I going install this tool, it shows me this output.

$ npm install --global @tryghost/migrate
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: No longer maintained, please use @hubspot/api-client instead
npm WARN deprecated [email protected]: Critical security vulnerability fixed in v0.21.1. For more information, see https://github.com/axios/axios/pull/3410
npm WARN deprecated [email protected]: Use uuid module instead
npm WARN deprecated [email protected]: Please upgrade to v7.0.2+ of superagent.  We have fixed numerous issues with streams, form-data, attach(), filesystem errors not bubbling up (ENOENT on attach()), and all tests are now passing.  See the releases tab for more information at <https://github.com/visionmedia/superagent/releases>.
npm WARN deprecated [email protected]: Please upgrade to latest, formidable@v2 or formidable@v3! Check these notes: https://bit.ly/2ZEqIau
/home/rafidalhaque/.nvm/versions/node/v14.19.3/bin/migrate -> /home/rafidalhaque/.nvm/versions/node/v14.19.3/lib/node_modules/@tryghost/migrate/bin/cli.js
+ @tryghost/[email protected]
updated 1 package in 239.056s

When I run this tool, it shows me this error message.

$ migrate wp-api https://redwanrawaha.wordpress.com
internal/modules/cjs/loader.js:905
  throw err;
  ^

Error: Cannot find module '@tryghost/mg-mediascraper'
Require stack:
- /home/rafidalhaque/.nvm/versions/node/v14.19.3/lib/node_modules/@tryghost/migrate/sources/ghost.js
- /home/rafidalhaque/.nvm/versions/node/v14.19.3/lib/node_modules/@tryghost/migrate/commands/ghost.js
- /home/rafidalhaque/.nvm/versions/node/v14.19.3/lib/node_modules/@tryghost/migrate/node_modules/sywac/api.js
- /home/rafidalhaque/.nvm/versions/node/v14.19.3/lib/node_modules/@tryghost/migrate/node_modules/@tryghost/pretty-cli/lib/pretty-cli.js
- /home/rafidalhaque/.nvm/versions/node/v14.19.3/lib/node_modules/@tryghost/migrate/node_modules/@tryghost/pretty-cli/index.js
- /home/rafidalhaque/.nvm/versions/node/v14.19.3/lib/node_modules/@tryghost/migrate/bin/cli.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:902:15)
    at Function.Module._load (internal/modules/cjs/loader.js:746:27)
    at Module.require (internal/modules/cjs/loader.js:974:19)
    at require (internal/modules/cjs/helpers.js:101:18)
    at Object.<anonymous> (/home/rafidalhaque/.nvm/versions/node/v14.19.3/lib/node_modules/@tryghost/migrate/sources/ghost.js:4:24)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Module.require (internal/modules/cjs/loader.js:974:19)
    at require (internal/modules/cjs/helpers.js:101:18)
    at Object.<anonymous> (/home/rafidalhaque/.nvm/versions/node/v14.19.3/lib/node_modules/@tryghost/migrate/commands/ghost.js:1:15)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/home/rafidalhaque/.nvm/versions/node/v14.19.3/lib/node_modules/@tryghost/migrate/sources/ghost.js',
    '/home/rafidalhaque/.nvm/versions/node/v14.19.3/lib/node_modules/@tryghost/migrate/commands/ghost.js',
    '/home/rafidalhaque/.nvm/versions/node/v14.19.3/lib/node_modules/@tryghost/migrate/node_modules/sywac/api.js',
    '/home/rafidalhaque/.nvm/versions/node/v14.19.3/lib/node_modules/@tryghost/migrate/node_modules/@tryghost/pretty-cli/lib/pretty-cli.js',
    '/home/rafidalhaque/.nvm/versions/node/v14.19.3/lib/node_modules/@tryghost/migrate/node_modules/@tryghost/pretty-cli/index.js',
    '/home/rafidalhaque/.nvm/versions/node/v14.19.3/lib/node_modules/@tryghost/migrate/bin/cli.js'
  ]
}

Hello the team,

I'm facing an error while using the migrate tools to convert medium.com posts to Ghost posts:

migrate medium --pathToZip medium-export-test.zip

Medium.com code block:

Actual Ghost.io code block:

Expected Ghost.io code block:

Medium.com exported HTML code:

<pre name="4a0a" id="4a0a" class="graf graf--pre graf-after--p">
  <code class="markup--code markup--pre-code">
    sudo apt-get update<br>sudo apt-get install \<br>    apt-transport-https \<br>    ca-certificates \<br>    curl \<br>    gnupg-agent \<br>    software-properties-common \<br>    certbot \<br>    python3-certbot-nginx
  </code>
</pre>
<pre name="ebeb" id="ebeb" class="graf graf--pre graf-after--pre">
  <code class="markup--code markup--pre-code">
    sudo apt-key adv --keyserver keys.gnupg.net --recv-keys 0xDE8B853FC155581D
  </code>
</pre>
<pre name="9a1f" id="9a1f" class="graf graf--pre graf-after--pre">
  <code class="markup--code markup--pre-code">
    echo &quot;deb https://download.passbolt.com/ce/debian buster stable&quot; |\ <br>sudo tee /etc/apt/sources.list.d/passbolt.list<br>sudo apt-get update<br>sudo apt-get install passbolt-ce-server
  </code>
</pre>

Ghost.io mobiledoc:

[
  "code",
  {
    "code": "sudo apt-get updatesudo apt-get install \\    apt-transport-https \\    ca-certificates \\    curl \\    gnupg-agent \\    software-properties-common \\    certbot \\    python3-certbot-nginx"
  }
],
[
  "code",
  {
    "code": "sudo apt-key adv --keyserver keys.gnupg.net --recv-keys 0xDE8B853FC155581D"
  }
],
[
  "code",
  {
    "code": "echo \"deb https://download.passbolt.com/ce/debian buster stable\" |\\ sudo tee /etc/apt/sources.list.d/passbolt.listsudo apt-get updatesudo apt-get install passbolt-ce-server"
  }
]

As you can see, all <br> tag were stripped during the process of migration.

It may happen somewhere in the convertPost function which uses @tryghost/html-to-mobiledoc under the hood, and itself is using @tryghost/kg-parser-plugins to parse the HTML by removing all br tag as the allowBr option is not correctly apply.

I'm using:

% migrate --version
0.31.1

Thank you.

Hey y'all,

I'm trying to migrate an old wordpress site to Ghost and I have a local instance of wordpress installed with all of my posts, users, and other data on it. I'm trying to use migrate to create a ghost export of the site, but am getting hung up on the "Build Link Map" step. Is there some sort of configuration I could be missing on the wordpress site that is causing this to fail? Or is there any way to get a more in depth log?

My command: migrate migrate wp-api --url localhost --addTag archives --cache --users -V

Error message:

[COMPLETED] Fetch missing metadata via WebScraper
[2024-02-03 17:38:31] INFO 
[STARTED] Build Link Map
[2024-02-03 17:38:31] ERROR 
[FAILED] Invalid URL
[2024-02-03 17:38:31] INFO 

Running into the following error while running migrate json-html /test/ghost.json -V to migrate HTML to Ghost mobiledoc format.

Error:

info Running html conversion on /test/ghost.json
info Done with errors []
TypeError: Cannot read properties of undefined (reading 'cacheDir')
    at TypeCommand.run [as _runHandler] (file:///usr/local/lib/node_modules/@tryghost/migrate/commands/json/html.js:47:71)
    at /usr/local/lib/node_modules/@tryghost/migrate/node_modules/sywac/types/command.js:131:19
    at async Promise.all (index 10)

/test/ghost.json:

{
  "posts": [
    {
      "html": "<p>this is a test</p>"
    }
  ]
}

I could be missing something.

In frontmatter, dates can be quoted as strings, or unquoted.

The Jekyll migration tool works for the case when the date is quoted, but it has no test coverage for unquoted dates and crashes in that case.

The difference is that the front-matter library will parse the date directly into a date object if it's not quoted. Because the importing code only expects strings, it fails like this:

 TypeError: frontmatterAttributes.date.match is not a function

To trigger it in the test suite, add a post with a date formatting like this:

date: 2022-06-05

(With no quotes around the date!)

Look into this uncovered another minor issue with date handing: If front-matter parses a date with no time, it will set the time to midnight. But if the Jekyll importing code parses a date with no time as string, the time gets set to noon. For consistency, the importing code should just not set the hour, minute and second fields in the date object so the default values are used.

I've got a sub stack URL with almost 900 posts and a custom domain. When I try to run the migration tool, it starts failing and saying "Unable to scrape URL".

My hunch is this may be because of rate limiting, so making this a background process with a delay between requests may make this more stable. Happy to debug one-on-one to give more context!

I'm using the CLI migrate tool on my Substack and podcast audio doesn't get scraped and added to posts. I see code which indicates that audio cards should be generated for podcast episodes, but I don't see that happening.

Is there a way to get the scraper to download podcast MP3 files, and also set the og_description field to the MP3 location on the imported posts, to make the importer compatible with the Wave official podcast theme's RSS template?

The issue referenced in 5e86ef8#diff-fdb8c458e0706f830aedcce4174894ceR16 has been resolved, so the only required field to do an import is name.

When I use the migration tool to import blog posts from Medium to Ghost, the image URLs still point to Medium's cloud storage provider rather than importing the images itself to be hosted on the Ghost server. The images are not part of the zip archive when you do an export on Medium.

Is that intended and would I need to upload each image manually via Ghost to be hosted on the same Ghost server? Another way would be that I upload the images to the Ghost content directory and change the URLs manually in all exported blog post HTML files from Medium before I run the migration command. Is there any easier way to solve this?

Medium recently changed the contents of the export zip to include a bunch of new directories.

The tooling needs to be updated to only look at posts & profile, and silently skip the others, but report those skipped directories when the --verbose flag is set.

For reference, this is what's now included.

blocks
bookmarks
claps
highlights
interests
ips
posts
profile
pubs-following
README.html
sessions
topics-following
users-following

With or without scraping, i only get the metadata from the posts, not the content! It doesn't event construct the .zip. I see the info in the /tmp/mg folder.

Any clues?

Introduced through
[email protected]
Fixed in
[email protected]

Exploit maturity
Proof of Concept

Detailed paths

Introduced through: @tryghost/[email protected] › [email protected] › [email protected]
Fix: No remediation path available. 

Security information
Factors contributing to the scoring:

Snyk: [CVSS 7.5](https://security.snyk.io/vuln/SNYK-JS-MOMENT-2944238) - High Severity
NVD: [CVSS 7.5](https://nvd.nist.gov/vuln/detail/CVE-2022-31129) - High Severity

Why are the scores different? Learn how Snyk evaluates vulnerability scores
Overview

moment is a lightweight JavaScript date library for parsing, validating, manipulating, and formatting dates.

Affected versions of this package are vulnerable to Regular Expression Denial of Service (ReDoS) via the preprocessRFC2822() function in from-string.js, when processing a very long crafted string (over 10k characters).

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Location: packages/listr-smart-renderer/package.json
Error type: Nested package.json must not contain renovate configuration. Please use packageRules with matchPaths in your main config instead.

I am using your tool to migrate from other sources to Ghost. However, it throws the error mentioned in the title.

Full error stack is below:

node:internal/modules/cjs/loader:936
  throw err;
  ^

Error: Cannot find module '@tryghost/mg-mediascraper'
Require stack:
- /opt/homebrew/lib/node_modules/@tryghost/migrate/sources/ghost.js
- /opt/homebrew/lib/node_modules/@tryghost/migrate/commands/ghost.js
- /opt/homebrew/lib/node_modules/@tryghost/migrate/node_modules/sywac/api.js
- /opt/homebrew/lib/node_modules/@tryghost/migrate/node_modules/@tryghost/pretty-cli/lib/pretty-cli.js
- /opt/homebrew/lib/node_modules/@tryghost/migrate/node_modules/@tryghost/pretty-cli/index.js
- /opt/homebrew/lib/node_modules/@tryghost/migrate/bin/cli.js
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
    at Function.Module._load (node:internal/modules/cjs/loader:778:27)
    at Module.require (node:internal/modules/cjs/loader:999:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/opt/homebrew/lib/node_modules/@tryghost/migrate/sources/ghost.js:4:24)
    at Module._compile (node:internal/modules/cjs/loader:1097:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1151:10)
    at Module.load (node:internal/modules/cjs/loader:975:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Module.require (node:internal/modules/cjs/loader:999:19)
    at require (node:internal/modules/cjs/helpers:102:18)
    at Object.<anonymous> (/opt/homebrew/lib/node_modules/@tryghost/migrate/commands/ghost.js:1:15)
    at Module._compile (node:internal/modules/cjs/loader:1097:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1151:10)
    at Module.load (node:internal/modules/cjs/loader:975:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/opt/homebrew/lib/node_modules/@tryghost/migrate/sources/ghost.js',
    '/opt/homebrew/lib/node_modules/@tryghost/migrate/commands/ghost.js',
    '/opt/homebrew/lib/node_modules/@tryghost/migrate/node_modules/sywac/api.js',
    '/opt/homebrew/lib/node_modules/@tryghost/migrate/node_modules/@tryghost/pretty-cli/lib/pretty-cli.js',
    '/opt/homebrew/lib/node_modules/@tryghost/migrate/node_modules/@tryghost/pretty-cli/index.js',
    '/opt/homebrew/lib/node_modules/@tryghost/migrate/bin/cli.js'
  ]
}

Node.js v17.6.0

The Jekyll HTML processing code includes this logic for processing "a" tags:

            const thisURL = $(anchor).attr('href');

            // If it starts with a slash, append the base URL
            if (thisURL.indexOf('/') === 0) {

The logic bug here is that it assumes that every "a" tag contains an "href" attribute. When the importer encounters one that doesn't have one, it will crash with an error like this:

  TypeError: Cannot read properties of undefined (reading 'indexOf`

This fix seems straightforward. I''ll take a look.

I'm facing an error when trying to migrate my medium export. Happy to provide the export directly to someone if needed or let me know what I should look at.

ryan@ryans-M1-Mac medium % migrate medium --pathToZip medium.zip
(node:98007) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 unpipe listeners added to [WriteStream]. Use emitter.setMaxListeners() to increase limit
(Use `node --trace-warnings ...` to show where the warning was created)
(node:98007) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 error listeners added to [WriteStream]. Use emitter.setMaxListeners() to increase limit
(node:98007) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [WriteStream]. Use emitter.setMaxListeners() to increase limit
(node:98007) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 finish listeners added to [WriteStream]. Use emitter.setMaxListeners() to increase limit
[2023-10-18 13:19:42] INFO
[2023-10-18 13:19:42] INFO
[2023-10-18 13:19:42] INFO
[2023-10-18 13:19:42] INFO
 ✔ https://medium.com/@what_if/cisco-cloud-services-router-csr-1000v-restful-api-wont-start-29e368efcfb3
 ✔ https://medium.com/@what_if/net-framework-4-7-2-install-error-0x80092004-cannot-find-object-or-property-9f46de27b980
 ✔ https://medium.com/@what_if/azure-backup-agent-slow-transfer-speeds-6726693c4477
 ✔ https://medium.com/p/fc8b17944ee2
 ✔ https://medium.com/@what_if/cisco-cloud-services-router-csr-1000v-ssh-generated-key-hostname-domain-ad3d24e7e1ac
 ✔ https://medium.com/p/23b4cb79ad9e
 ✔ https://medium.com/p/a1c15f2a3b97
 ✔ https://medium.com/p/39350942c904
 ✔ https://medium.com/p/1edc4bdf334c
 ✔ https://medium.com/p/fa8facf2683b
 ✔ https://medium.com/p/c04a63dfb988
 ✔ https://medium.com/p/a79f4149ab20
 ✔ https://medium.com/@what_if/juniper-ex4500-class-of-service-error-on-vty-console-497c7747810e
 ✔ https://medium.com/p/2a9b46b3859c
 ✔ https://medium.com/p/85bf531dee53
 ✔ https://medium.com/p/64d284c8dadf
✔ Build Link Map
✔ Format data as Ghost JSON
› Fetch images via AssetScraper
 ✔ Initializing AssetScraper cache
 ✔ Finding files
 › Finding file types
  ⠹ 09/60: Checking file type: https://www.networktocode.com/community/
  ⠹ 20/60: Checking file type: https://m.dotdev.co/how-to-set-a-featured-image-and-set-its-focal-point-on-medium-b3b57ab986bb
  ⠹ 26/60: Checking file type: https://download.microsoft.com/download/E/F/2/EF23C21D-7860-4F05-88CE-39AA114B014B/SQLEXPR_x64_ENU.exe
  ⠹ 27/60: Checking file type: https://stackoverflow.com/questions/39835986/sql-server-2016-express-full-download
  ⠹ 28/60: Checking file type: https://download.microsoft.com/download/D/D/4/DD495084-ADA7-4827-ADD3-FC566EC05B90/SSMS-Setup-ENU.exe
  Total: 60. Complete: 0, Failed: 0, Skipped: 23, Disabled: 0
   Downloading files
   Fixing asset references
   Finalizing
  Update links in content via LinkFixer
  Convert HTML -> MobileDoc
  Write Ghost import JSON File
  Write Ghost import zip
file:///opt/homebrew/lib/node_modules/@tryghost/migrate/node_modules/@tryghost/mg-assetscraper/lib/AssetScraper.js:617
                        const parts = disposition.split('.');
                                                  ^

TypeError: Cannot read properties of undefined (reading 'split')
    at Request.<anonymous> (file:///opt/homebrew/lib/node_modules/@tryghost/migrate/node_modules/@tryghost/mg-assetscraper/lib/AssetScraper.js:617:51)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Node.js v20.8.1

Verbose output wasn't much different

ryan@ryans-M1-Mac medium % migrate medium --pathToZip medium.zip -V
info Migrating from export at medium.zip
[STARTED] Initializing Workspace
[OUTPUT] Workspace initialized at /var/folders/0r/pct6smwx02ngk0vww_6ghcvw0000gn/T/mg/17be23c2a764ace11d217bb33ee639f9-medium-medium
[COMPLETED] Initializing Workspace
[STARTED] Read Medium export zip
(node:98240) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 unpipe listeners added to [WriteStream]. Use emitter.setMaxListeners() to increase limit
(Use `node --trace-warnings ...` to show where the warning was created)
(node:98240) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 error listeners added to [WriteStream]. Use emitter.setMaxListeners() to increase limit
(node:98240) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [WriteStream]. Use emitter.setMaxListeners() to increase limit
(node:98240) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 finish listeners added to [WriteStream]. Use emitter.setMaxListeners() to increase limit
[COMPLETED] Read Medium export zip
[2023-10-18 13:23:12] INFO
[STARTED] Fetch missing data via WebScraper
[STARTED] https://medium.com/p/f7c05aae6f8b
[STARTED] https://medium.com/@what_if/openshift-ipi-install-failed-to-get-power-state-for-node-error-redfish-exception-occurred-533164040f1d
[STARTED] https://medium.com/@what_if/automate-enabling-of-ipmi-over-lan-access-on-hpe-ilo-7d9d8c55b83e
[COMPLETED] https://medium.com/@what_if/automate-enabling-of-ipmi-over-lan-access-on-hpe-ilo-7d9d8c55b83e
[STARTED] https://medium.com/@what_if/f5-big-iq-vm-fails-online-activation-b909145c7e68
[COMPLETED] https://medium.com/p/f7c05aae6f8b
[STARTED] https://medium.com/@what_if/f5-big-ip-or-big-iq-cant-run-config-tool-on-cli-f81aeb2561d9
[COMPLETED] https://medium.com/@what_if/openshift-ipi-install-failed-to-get-power-state-for-node-error-redfish-exception-occurred-533164040f1d
[STARTED] https://medium.com/@what_if/encrypting-decrypting-arista-bgp-bmp-ospf-passwords-ff2072460942
[COMPLETED] https://medium.com/@what_if/f5-big-iq-vm-fails-online-activation-b909145c7e68
[STARTED] https://medium.com/@what_if/automatically-query-the-azure-cdn-edge-nodes-list-5024951bd420
[COMPLETED] https://medium.com/@what_if/f5-big-ip-or-big-iq-cant-run-config-tool-on-cli-f81aeb2561d9
[STARTED] https://medium.com/@what_if/vmware-nsx-invalid-applied-to-value-41bb3e8f7eed
[COMPLETED] https://medium.com/@what_if/encrypting-decrypting-arista-bgp-bmp-ospf-passwords-ff2072460942
[STARTED] https://medium.com/@what_if/cant-see-all-images-in-featured-image-selection-82e7f76d6f6b
[COMPLETED] https://medium.com/@what_if/automatically-query-the-azure-cdn-edge-nodes-list-5024951bd420
[STARTED] https://medium.com/@what_if/disable-tls-1-0-on-windows-server-2012-r2-with-remote-desktop-services-configured-c6d0c881cb47
[COMPLETED] https://medium.com/@what_if/cant-see-all-images-in-featured-image-selection-82e7f76d6f6b
[STARTED] https://medium.com/@what_if/cisco-cloud-services-router-csr-1000v-restconf-wont-respond-on-https-1aa36fc5e75
[COMPLETED] https://medium.com/@what_if/vmware-nsx-invalid-applied-to-value-41bb3e8f7eed
[STARTED] https://medium.com/@what_if/disable-tls-1-0-on-arista-eapi-7bc27d016bbb
[COMPLETED] https://medium.com/@what_if/disable-tls-1-0-on-windows-server-2012-r2-with-remote-desktop-services-configured-c6d0c881cb47
[STARTED] https://medium.com/@what_if/cisco-cloud-services-router-csr-1000v-restful-api-wont-start-29e368efcfb3
[COMPLETED] https://medium.com/@what_if/cisco-cloud-services-router-csr-1000v-restconf-wont-respond-on-https-1aa36fc5e75
[STARTED] https://medium.com/@what_if/net-framework-4-7-2-install-error-0x80092004-cannot-find-object-or-property-9f46de27b980
[COMPLETED] https://medium.com/@what_if/disable-tls-1-0-on-arista-eapi-7bc27d016bbb
[STARTED] https://medium.com/@what_if/azure-backup-agent-slow-transfer-speeds-6726693c4477
[COMPLETED] https://medium.com/@what_if/cisco-cloud-services-router-csr-1000v-restful-api-wont-start-29e368efcfb3
[STARTED] https://medium.com/p/fc8b17944ee2
[COMPLETED] https://medium.com/@what_if/net-framework-4-7-2-install-error-0x80092004-cannot-find-object-or-property-9f46de27b980
[STARTED] https://medium.com/@what_if/cisco-cloud-services-router-csr-1000v-ssh-generated-key-hostname-domain-ad3d24e7e1ac
[COMPLETED] https://medium.com/@what_if/azure-backup-agent-slow-transfer-speeds-6726693c4477
[STARTED] https://medium.com/p/23b4cb79ad9e
[COMPLETED] https://medium.com/p/fc8b17944ee2
[STARTED] https://medium.com/p/a1c15f2a3b97
[COMPLETED] https://medium.com/@what_if/cisco-cloud-services-router-csr-1000v-ssh-generated-key-hostname-domain-ad3d24e7e1ac
[STARTED] https://medium.com/p/39350942c904
[COMPLETED] https://medium.com/p/23b4cb79ad9e
[STARTED] https://medium.com/p/1edc4bdf334c
[COMPLETED] https://medium.com/p/a1c15f2a3b97
[STARTED] https://medium.com/p/fa8facf2683b
[COMPLETED] https://medium.com/p/39350942c904
[STARTED] https://medium.com/p/c04a63dfb988
[COMPLETED] https://medium.com/p/1edc4bdf334c
[STARTED] https://medium.com/p/a79f4149ab20
[COMPLETED] https://medium.com/p/fa8facf2683b
[STARTED] https://medium.com/@what_if/juniper-ex4500-class-of-service-error-on-vty-console-497c7747810e
[COMPLETED] https://medium.com/p/c04a63dfb988
[STARTED] https://medium.com/p/2a9b46b3859c
[COMPLETED] https://medium.com/p/a79f4149ab20
[STARTED] https://medium.com/p/85bf531dee53
[COMPLETED] https://medium.com/@what_if/juniper-ex4500-class-of-service-error-on-vty-console-497c7747810e
[STARTED] https://medium.com/p/64d284c8dadf
[COMPLETED] https://medium.com/p/2a9b46b3859c
[COMPLETED] https://medium.com/p/85bf531dee53
[COMPLETED] https://medium.com/p/64d284c8dadf
[COMPLETED] Fetch missing data via WebScraper
[2023-10-18 13:23:12] INFO
[STARTED] Build Link Map
[COMPLETED] Build Link Map
[2023-10-18 13:23:12] INFO
[STARTED] Format data as Ghost JSON
[COMPLETED] Format data as Ghost JSON
[2023-10-18 13:23:12] INFO
[STARTED] Fetch images via AssetScraper
[STARTED] Initializing AssetScraper cache
[COMPLETED] Initializing AssetScraper cache
[STARTED] Finding files
[COMPLETED] Finding files
[STARTED] Finding file types
[STARTED] Checking file type: https://miro.medium.com/v2/resize:fill:144:144/2*v2YRpBNYEk5PVN4GKTw5vg.png
[STARTED] Checking file type: https://www.microchip.com/en-us/products/clock-and-timing/systems/ptp-grandmaster-clocks/timeprovider-4100
[STARTED] Checking file type: https://cdn-images-1.medium.com/max/800/0*B3L6nNZX8JfdFtRn.png
[STARTED] Checking file type: https://youriloiphere/redfish/v1/Managers/1/NetworkProtocol/
[STARTED] Checking file type: https://cdn-images-1.medium.com/max/800/1*aAvP-lktw0g2x-mFq8Eo-Q.png
[SKIPPED] Checking file type: https://miro.medium.com/v2/resize:fill:144:144/2*v2YRpBNYEk5PVN4GKTw5vg.png
[SKIPPED] Checking file type: https://www.microchip.com/en-us/products/clock-and-timing/systems/ptp-grandmaster-clocks/timeprovider-4100
[SKIPPED] Checking file type: https://cdn-images-1.medium.com/max/800/0*B3L6nNZX8JfdFtRn.png
[SKIPPED] Checking file type: https://youriloiphere/redfish/v1/Managers/1/NetworkProtocol/
[SKIPPED] Checking file type: https://cdn-images-1.medium.com/max/800/1*aAvP-lktw0g2x-mFq8Eo-Q.png
[STARTED] Checking file type: https://cdn-images-1.medium.com/max/800/1*6dMfQnlv2Z9qhYiKN0yrng.png
[STARTED] Checking file type: https://cdn-images-1.medium.com/max/800/1*qKgr0p2X9bC_4OEV5EUyHQ.gif
[STARTED] Checking file type: https://cdn-images-1.medium.com/max/800/1*9lFkfGJ4_DJAeugax2NehQ.png
[STARTED] Checking file type: https://www.networktocode.com/community/
[STARTED] Checking file type: https://docs.microsoft.com/en-us/rest/api/cdn/edgenodes/list#code-try-0
[SKIPPED] Checking file type: https://cdn-images-1.medium.com/max/800/1*6dMfQnlv2Z9qhYiKN0yrng.png
[SKIPPED] Checking file type: https://cdn-images-1.medium.com/max/800/1*qKgr0p2X9bC_4OEV5EUyHQ.gif
[SKIPPED] Checking file type: https://cdn-images-1.medium.com/max/800/1*9lFkfGJ4_DJAeugax2NehQ.png
[SKIPPED] Checking file type: https://docs.microsoft.com/en-us/rest/api/cdn/edgenodes/list#code-try-0
[STARTED] Checking file type: https://github.com/Gelob/get-azure-cdn-ips
[STARTED] Checking file type: https://portal.azure.com/
[STARTED] Checking file type: https://cdn-images-1.medium.com/max/800/1*PRz4Yb-XbCEMZg9W5awFXA.png
[STARTED] Checking file type: https://cdn-images-1.medium.com/max/800/1*KaYS9Ogv4EoODvnDjDaMrA.png
[SKIPPED] Checking file type: https://github.com/Gelob/get-azure-cdn-ips
[SKIPPED] Checking file type: https://portal.azure.com/
[SKIPPED] Checking file type: https://cdn-images-1.medium.com/max/800/1*PRz4Yb-XbCEMZg9W5awFXA.png
[SKIPPED] Checking file type: https://cdn-images-1.medium.com/max/800/1*KaYS9Ogv4EoODvnDjDaMrA.png
[STARTED] Checking file type: https://cdn-images-1.medium.com/max/800/1*_IBXZkIc8bcEbTMl9NMbqg.png
[STARTED] Checking file type: https://cdn-images-1.medium.com/max/800/1*DMjEnlADSay3pgn8h2WoGg.png
[STARTED] Checking file type: https://cdn-images-1.medium.com/max/800/1*UTAPz8UelAcumZIE8W5ivg.png
[STARTED] Checking file type: https://cdn-images-1.medium.com/max/800/1*c_EJQfzsYYtYqT398Ygl6A.png
[SKIPPED] Checking file type: https://cdn-images-1.medium.com/max/800/1*_IBXZkIc8bcEbTMl9NMbqg.png
[SKIPPED] Checking file type: https://cdn-images-1.medium.com/max/800/1*DMjEnlADSay3pgn8h2WoGg.png
[SKIPPED] Checking file type: https://cdn-images-1.medium.com/max/800/1*UTAPz8UelAcumZIE8W5ivg.png
[SKIPPED] Checking file type: https://cdn-images-1.medium.com/max/800/1*c_EJQfzsYYtYqT398Ygl6A.png
[STARTED] Checking file type: https://cdn-images-1.medium.com/max/800/1*qqQqDfvRsUvdNaJIq5Cr0w.png
[STARTED] Checking file type: https://m.dotdev.co/how-to-set-a-featured-image-and-set-its-focal-point-on-medium-b3b57ab986bb
[STARTED] Checking file type: https://help.medium.com/hc/en-us/articles/215680047-Featured-image
[STARTED] Checking file type: https://cdn-images-1.medium.com/max/800/1*SyGx73Zgl7urR-E5SH7gyw.png
[SKIPPED] Checking file type: https://cdn-images-1.medium.com/max/800/1*qqQqDfvRsUvdNaJIq5Cr0w.png
[SKIPPED] Checking file type: https://help.medium.com/hc/en-us/articles/215680047-Featured-image
[SKIPPED] Checking file type: https://cdn-images-1.medium.com/max/800/1*SyGx73Zgl7urR-E5SH7gyw.png
[STARTED] Checking file type: https://en.wikipedia.org/wiki/Windows_Internal_Database
[STARTED] Checking file type: https://support.microsoft.com/en-us/help/4036954/disabling-tls1-0-can-cause-rds-connection-broker-or-rdms-to-fail
[STARTED] Checking file type: https://support.microsoft.com/en-us/help/2833839/guidelines-for-installing-the-remote-desktop-session-host-role-service
[SKIPPED] Checking file type: https://en.wikipedia.org/wiki/Windows_Internal_Database
[SKIPPED] Checking file type: https://support.microsoft.com/en-us/help/4036954/disabling-tls1-0-can-cause-rds-connection-broker-or-rdms-to-fail
[SKIPPED] Checking file type: https://support.microsoft.com/en-us/help/2833839/guidelines-for-installing-the-remote-desktop-session-host-role-service
[STARTED] Checking file type: https://download.microsoft.com/download/E/F/2/EF23C21D-7860-4F05-88CE-39AA114B014B/SQLEXPR_x64_ENU.exe
[STARTED] Checking file type: https://stackoverflow.com/questions/39835986/sql-server-2016-express-full-download
[STARTED] Checking file type: https://download.microsoft.com/download/D/D/4/DD495084-ADA7-4827-ADD3-FC566EC05B90/SSMS-Setup-ENU.exe
file:///opt/homebrew/lib/node_modules/@tryghost/migrate/node_modules/@tryghost/mg-assetscraper/lib/AssetScraper.js:617
                        const parts = disposition.split('.');
                                                  ^

TypeError: Cannot read properties of undefined (reading 'split')
    at Request.<anonymous> (file:///opt/homebrew/lib/node_modules/@tryghost/migrate/node_modules/@tryghost/mg-assetscraper/lib/AssetScraper.js:617:51)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Node.js v20.8.1

I'm using the migrate command to convert my Substack export for Ghost:
migrate substack posts.csv --email [email protected] --url https://XXX.substack.com --readPosts posts --drafts false --verbose

The process runs with no errors (it's only a couple of posts) generating a gh-posts-XXX.zip file. However, when I try to import this into Ghost (self-hosted using the DigitalOcean Droplet), I get this error:

I've tried importing another Ghost generated export and this works fine so I'm fairly confident the issue is with the generation of the JSON file. Can anyone suggest a remedy?

To reproduce

• I set the --url= of the old blog with a subdir root, like https://mark.stosberg.com/blog
• Some images in the post have relative-to-root URLs like /blog/images/moonlander-keyboard.png

Current behavior

It seems a domain-root was expected, because the result in the output URL includes the subdir path twice:

https://mark.stosberg.com/blog/blog/images/moonlander-keyboard.png

("/blog/blog" is the problem here).

I'll take a look now to see if there's a quick fix, but at this point I've run into enough bugs with the Jekyll importer that I think it would have been faster to manually copy/paste my old content into Ghost and manually resolve issues.

In addition to Markdown, Jekyll also supports posts in HTML format (official reference: https://jekyllrb.com/docs/posts/#the-posts-folder ).

It would great if this tool could import those as well.

Thanks!

Currently, All Jekyll posts without an explicit "author" in the frontmatter. get imported by an person named "Dummy User". It's not a great look for a site to be full of posts by a "dummy".

Recommended solution to follow the precedent of the Wordpress importer and accept --user <json file> as a CLI argument. The contents would be expected to match the format of globalUser as seen in lib/process.js.

I got there errors while installing:

joy@Vin-Desk:~$ npm install --global @tryghost/migrate
npm WARN deprecated [email protected]: This module has moved and is now available at @hapi/topo. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated [email protected]: Use String.prototype.trim() instead
npm WARN deprecated [email protected]: This module has moved and is now available at @hapi/hoek. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.
npm WARN deprecated [email protected]: No longer maintained, please use @hubspot/api-client instead
npm WARN deprecated [email protected]: Please upgrade to latest, formidable@v2 or formidable@v3! Check these notes: https://bit.ly/2ZEqIau
npm WARN deprecated [email protected]: Use uuid module instead
npm WARN deprecated [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: request has been deprecated, see request/request#3142
npm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated [email protected]: Please upgrade to v7.0.2+ of superagent. We have fixed numerous issues with streams, form-data, attach(), filesystem errors not bubbling up (ENOENT on attach()), and all tests are now passing. See the releases tab for more information at https://github.com/visionmedia/superagent/releases.
npm WARN deprecated [email protected]: Critical security vulnerability fixed in v0.21.1. For more information, see axios/axios#3410
npm WARN cleanup Failed to remove some directories [
npm WARN cleanup [
npm WARN cleanup 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\@TryGhost\migrate\node_modules\@stdlib',
npm WARN cleanup [Error: EPERM: operation not permitted, rmdir 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules@tryghost\migrate\node_modules@stdlib\ndarray'] {
npm WARN cleanup errno: -4048,
npm WARN cleanup code: 'EPERM',
npm WARN cleanup syscall: 'rmdir',
npm WARN cleanup path: 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\@TryGhost\migrate\node_modules\@stdlib\ndarray'
npm WARN cleanup }
npm WARN cleanup ]
npm WARN cleanup ]
npm ERR! code 1
npm ERR! path C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules@tryghost\migrate\node_modules\node-expat
npm ERR! command failed
npm ERR! command C:\WINDOWS\system32\cmd.exe /d /s /c node-gyp rebuild
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using [email protected]
npm ERR! gyp info using [email protected] | win32 | x64
npm ERR! gyp info find Python using Python version 3.11.0 found at "C:\Python311\python.exe"
npm ERR! gyp info find VS using VS2019 (16.11.33328.57) found at:
npm ERR! gyp info find VS "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools"
npm ERR! gyp info find VS run with --verbose for detailed information
npm ERR! gyp info spawn C:\Python311\python.exe
npm ERR! gyp info spawn args [
npm ERR! gyp info spawn args 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\npm\node_modules\node-gyp\gyp\gyp_main.py',
npm ERR! gyp info spawn args 'binding.gyp',
npm ERR! gyp info spawn args '-f',
npm ERR! gyp info spawn args 'msvs',
npm ERR! gyp info spawn args '-I',
npm ERR! gyp info spawn args 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\@TryGhost\migrate\node_modules\node-expat\build\config.gypi',
npm ERR! gyp info spawn args '-I',
npm ERR! gyp info spawn args 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\npm\node_modules\node-gyp\addon.gypi',
npm ERR! gyp info spawn args '-I',
npm ERR! gyp info spawn args 'C:\Users\Anh\AppData\Local\node-gyp\Cache\18.14.1\common.gypi',
npm ERR! gyp info spawn args '-Dlibrary=shared_library',
npm ERR! gyp info spawn args '-Dvisibility=default',
npm ERR! gyp info spawn args '-Dnode_root_dir=C:\Users\Anh\AppData\Local\node-gyp\Cache\18.14.1',
npm ERR! gyp info spawn args '-Dnode_gyp_dir=C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\npm\node_modules\node-gyp',
npm ERR! gyp info spawn args '-Dnode_lib_file=C:\\Users\\Anh\\AppData\\Local\\node-gyp\\Cache\\18.14.1\\<(target_arch)\\node.lib',
npm ERR! gyp info spawn args '-Dmodule_root_dir=C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\@TryGhost\migrate\node_modules\node-expat',
npm ERR! gyp info spawn args '-Dnode_engine=v8',
npm ERR! gyp info spawn args '--depth=.',
npm ERR! gyp info spawn args '--no-parallel',
npm ERR! gyp info spawn args '--generator-output',
npm ERR! gyp info spawn args 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\@TryGhost\migrate\node_modules\node-expat\build',
npm ERR! gyp info spawn args '-Goutput_dir=.'
npm ERR! gyp info spawn args ]
npm ERR! gyp: C:\Users\Anh\AppData\Local\node-gyp\Cache\18.14.1\common.gypi not found (cwd: C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules@tryghost\migrate\node_modules\node-expat) while reading includes of binding.gyp while trying to load binding.gyp
npm ERR! gyp ERR! configure error
npm ERR! gyp ERR! stack Error: gyp failed with exit code: 1
npm ERR! gyp ERR! stack at ChildProcess.onCpExit (C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\npm\node_modules\node-gyp\lib\configure.js:325:16)
npm ERR! gyp ERR! stack at ChildProcess.emit (node:events:513:28)
npm ERR! gyp ERR! stack at ChildProcess._handle.onexit (node:internal/child_process:291:12)
npm ERR! gyp ERR! System Windows_NT 10.0.22623
npm ERR! gyp ERR! command "C:\Program Files\nodejs\node.exe" "C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" "rebuild"
npm ERR! gyp ERR! cwd C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules@tryghost\migrate\node_modules\node-expat
npm ERR! gyp ERR! node -v v18.14.1
npm ERR! gyp ERR! node-gyp -v v9.3.0
npm ERR! gyp ERR! not ok

I tried with PowerShell as well:

PS C:\WINDOWS\system32> npm install --global @tryghost/migrate
npm WARN deprecated [email protected]: This module has moved and is now available at @hapi/topo. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.
npm WARN deprecated [email protected]: this library is no longer supported
npm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated [email protected]: Use String.prototype.trim() instead
npm WARN deprecated [email protected]: This module has moved and is now available at @hapi/hoek. Please update your dependencies as this version is no longer maintained an may contain bugs and security issues.
npm WARN deprecated [email protected]: No longer maintained, please use @hubspot/api-client instead
npm WARN deprecated [email protected]: Please upgrade to latest, formidable@v2 or formidable@v3! Check these notes: https://bit.ly/2ZEqIau
npm WARN deprecated [email protected]: Use uuid module instead
npm WARN deprecated [email protected]: Please upgrade to version 7 or higher. Older versions may use Math.random() in certain circumstances, which is known to be problematic. See https://v8.dev/blog/math-random for details.
npm WARN deprecated [email protected]: request has been deprecated, see request/request#3142
npm WARN deprecated [email protected]: This version has been deprecated in accordance with the hapi support policy (hapi.im/support). Please upgrade to the latest version to get the best features, bug fixes, and security patches. If you are unable to upgrade at this time, paid support is available for older versions (hapi.im/commercial).
npm WARN deprecated [email protected]: Please upgrade to v7.0.2+ of superagent. We have fixed numerous issues with streams, form-data, attach(), filesystem errors not bubbling up (ENOENT on attach()), and all tests are now passing. See the releases tab for more information at https://github.com/visionmedia/superagent/releases.
npm WARN deprecated [email protected]: Critical security vulnerability fixed in v0.21.1. For more information, see axios/axios#3410
npm WARN cleanup Failed to remove some directories [
npm WARN cleanup [
npm WARN cleanup 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\@TryGhost\migrate\node_modules\sharp',
npm WARN cleanup [Error: EBUSY: resource busy or locked, rmdir 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules@tryghost\migrate\node_modules\sharp'] {
npm WARN cleanup errno: -4082,
npm WARN cleanup code: 'EBUSY',
npm WARN cleanup syscall: 'rmdir',
npm WARN cleanup path: 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\@TryGhost\migrate\node_modules\sharp'
npm WARN cleanup }
npm WARN cleanup ],
npm WARN cleanup [
npm WARN cleanup 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\@TryGhost\migrate\node_modules\dtrace-provider',
npm WARN cleanup [Error: EBUSY: resource busy or locked, rmdir 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules@tryghost\migrate\node_modules\dtrace-provider'] {
npm WARN cleanup errno: -4082,
npm WARN cleanup code: 'EBUSY',
npm WARN cleanup syscall: 'rmdir',
npm WARN cleanup path: 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\@TryGhost\migrate\node_modules\dtrace-provider'
npm WARN cleanup }
npm WARN cleanup ],
npm WARN cleanup [
npm WARN cleanup 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\@TryGhost\migrate\node_modules',
npm WARN cleanup [Error: EBUSY: resource busy or locked, rmdir 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules@tryghost\migrate\node_modules\sharp'] {
npm WARN cleanup errno: -4082,
npm WARN cleanup code: 'EBUSY',
npm WARN cleanup syscall: 'rmdir',
npm WARN cleanup path: 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\@TryGhost\migrate\node_modules\sharp'
npm WARN cleanup }
npm WARN cleanup ],
npm WARN cleanup [
npm WARN cleanup 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\@TryGhost\migrate\node_modules\@TryGhost\image-transform',
npm WARN cleanup [Error: EPERM: operation not permitted, rmdir 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules@tryghost\migrate\node_modules@tryghost\image-transform'] {
npm WARN cleanup errno: -4048,
npm WARN cleanup code: 'EPERM',
npm WARN cleanup syscall: 'rmdir',
npm WARN cleanup path: 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\@TryGhost\migrate\node_modules\@TryGhost\image-transform'
npm WARN cleanup }
npm WARN cleanup ],
npm WARN cleanup [
npm WARN cleanup 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\@TryGhost\migrate',
npm WARN cleanup [Error: EPERM: operation not permitted, rmdir 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules@tryghost\migrate\node_modules@tryghost\image-transform'] {
npm WARN cleanup errno: -4048,
npm WARN cleanup code: 'EPERM',
npm WARN cleanup syscall: 'rmdir',
npm WARN cleanup path: 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\@TryGhost\migrate\node_modules\@TryGhost\image-transform'
npm WARN cleanup }
npm WARN cleanup ]
npm WARN cleanup ]
npm ERR! code 1
npm ERR! path C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules@tryghost\migrate\node_modules\node-expat
npm ERR! command failed
npm ERR! command C:\WINDOWS\system32\cmd.exe /d /s /c node-gyp rebuild
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using [email protected]
npm ERR! gyp info using [email protected] | win32 | x64
npm ERR! gyp info find Python using Python version 3.11.0 found at "C:\Python311\python.exe"
npm ERR! gyp info find VS using VS2019 (16.11.33328.57) found at:
npm ERR! gyp info find VS "C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools"
npm ERR! gyp info find VS run with --verbose for detailed information
npm ERR! gyp info spawn C:\Python311\python.exe
npm ERR! gyp info spawn args [
npm ERR! gyp info spawn args 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\npm\node_modules\node-gyp\gyp\gyp_main.py',
npm ERR! gyp info spawn args 'binding.gyp',
npm ERR! gyp info spawn args '-f',
npm ERR! gyp info spawn args 'msvs',
npm ERR! gyp info spawn args '-I',
npm ERR! gyp info spawn args 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\@TryGhost\migrate\node_modules\node-expat\build\config.gypi',
npm ERR! gyp info spawn args '-I',
npm ERR! gyp info spawn args 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\npm\node_modules\node-gyp\addon.gypi',
npm ERR! gyp info spawn args '-I',
npm ERR! gyp info spawn args 'C:\Users\Anh\AppData\Local\node-gyp\Cache\18.14.1\common.gypi',
npm ERR! gyp info spawn args '-Dlibrary=shared_library',
npm ERR! gyp info spawn args '-Dvisibility=default',
npm ERR! gyp info spawn args '-Dnode_root_dir=C:\Users\Anh\AppData\Local\node-gyp\Cache\18.14.1',
npm ERR! gyp info spawn args '-Dnode_gyp_dir=C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\npm\node_modules\node-gyp',
npm ERR! gyp info spawn args '-Dnode_lib_file=C:\\Users\\Anh\\AppData\\Local\\node-gyp\\Cache\\18.14.1\\<(target_arch)\\node.lib',
npm ERR! gyp info spawn args '-Dmodule_root_dir=C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\@TryGhost\migrate\node_modules\node-expat',
npm ERR! gyp info spawn args '-Dnode_engine=v8',
npm ERR! gyp info spawn args '--depth=.',
npm ERR! gyp info spawn args '--no-parallel',
npm ERR! gyp info spawn args '--generator-output',
npm ERR! gyp info spawn args 'C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\@TryGhost\migrate\node_modules\node-expat\build',
npm ERR! gyp info spawn args '-Goutput_dir=.'
npm ERR! gyp info spawn args ]
npm ERR! gyp: C:\Users\Anh\AppData\Local\node-gyp\Cache\18.14.1\common.gypi not found (cwd: C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules@tryghost\migrate\node_modules\node-expat) while reading includes of binding.gyp while trying to load binding.gyp
npm ERR! gyp ERR! configure error
npm ERR! gyp ERR! stack Error: gyp failed with exit code: 1
npm ERR! gyp ERR! stack at ChildProcess.onCpExit (C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\npm\node_modules\node-gyp\lib\configure.js:325:16)
npm ERR! gyp ERR! stack at ChildProcess.emit (node:events:513:28)
npm ERR! gyp ERR! stack at ChildProcess._handle.onexit (node:internal/child_process:291:12)
npm ERR! gyp ERR! System Windows_NT 10.0.22623
npm ERR! gyp ERR! command "C:\Program Files\nodejs\node.exe" "C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules\npm\node_modules\node-gyp\bin\node-gyp.js" "rebuild"
npm ERR! gyp ERR! cwd C:\Users\Anh\AppData\Roaming\nvm\v18.14.1\node_modules@tryghost\migrate\node_modules\node-expat
npm ERR! gyp ERR! node -v v18.14.1
npm ERR! gyp ERR! node-gyp -v v9.3.0
npm ERR! gyp ERR! not ok

npm ERR! A complete log of this run can be found in:
npm ERR! C:\Users\Anh\AppData\Local\npm-cache_logs\2023-02-26T16_28_45_656Z-debug-0.log

The Jekyll docs say that the tags and categories frontmatter can either be "space separated strings" or a YAML list [1].

The Jekyll importer currently supports space-separated strings, but if given a YAML list, will crash with one of these errors:

  TypeError: frontmatterAttributes.tags.split is not a function
  TypeError: frontmatterAttributes.categories.split is not a function

1. https://jekyllrb.com/docs/front-matter/

The fix is straightforward. I'll work on it.