Support UTF-8 Characters in File Names about middleman-blog HOT 4 CLOSED

Will investigate.

from middleman-blog.

I can confirm that this doesn't work, but I'm afraid it's a flaw in the Ruby standard 'uri' library:
http://rawsyntax.com/blog/url-validation-in-rails-3-and-ruby-in-general/

Your best bet for now is to not use non-ASCII characters in filenames until this is fixed in Ruby 2.0. Watch this issue for progress: http://bugs.ruby-lang.org/issues/1787

I poked around as to how we could work around this, and it looks like the only place URI is used like this happens is within Rack::Test. I've filed an issue with them: rack/rack-test#56

However, there's another problem I'm encountering which is frankly stumping me. Paths with unicode characters get added to the sitemap hash, but then cannot be retrieved again, even though it's clearly there as a key! For example:

# The Sitemap::Store contains "2011/01/01/snowmen-rule-☃.html"
# @pages is a hash from path (String) => Middleman::Sitemap::Page

@pages.has_key("2011/01/01/snowmen-rule-☃.html")
=> false
@pages.keys.include?("2011/01/01/snowmen-rule-☃.html")
=> true

I can't seem to replicate this in IRB with just regular strings and hashes. I don't really know what's going on. This part is what's preventing these paths from being renderable even in the preview server.

from middleman-blog.

OK, it appears to be a bug in Ruby with a similar underlying cause as this (fixed) bug: http://www.ruby-forum.com/topic/179303 - both the strings appear identical, but have different hashcodes.

I haven't yet been able to create a repro, but I'll post something to the Ruby bugtracker when I do (and maybe it'll give a hint as to how we could work around it).

from middleman-blog.

I haven't been able to come up with a repro, though I suspect it's the filenames coming out of the listen gem. I have found a workaround, which is to call .tr('','') on the path strings. However, that just uncovered other places where unicode paths don't work in Middleman - specifically, Rack receives the request (in PATH_INFO) with unicode characters URI-encoded. To work around that we'd need to make sure paths are stored in the sitemap URI-encoded (probably by encoding them when the filesystem watcher sees them) which will of course make it complicated to reference those from config.rb.

I'm also going to move this issue over to the main middleman issue tracker since this isn't specific to middleman-blog.

from middleman-blog.