haskell-servant / servant-multipart Goto Github PK

View Code? Open in Web Editor NEW

40.0 40.0 36.0 144 KB

Support for file uploads in multipart/form-data for servant

Home Page: https://hackage.haskell.org/package/servant-multipart

Haskell 100.00%

servant-multipart's People

Contributors

Stargazers

Watchers

servant-multipart's Issues

Internal module?

I needed the LookupContext constaint, but it's not exported. Would it make sense to move everything to Servant.Multipart.Internal (which exports everything) and then re-export the current API from Servant.MultiPart?

Release new version to support base-4.14

The currently uploaded version of servant-multipart on hackage only works with base-4.14, because it carries the bound servant (>=0.16 && <0.18).

What's the current state? Are there things missing? I can supply work to get a new version done if need be.

Compilation error for servant-multipart-client-0.12.1 against servant-0.16.1

Hello, I work on a project currently depending on servant-0.16.1.

It seems the SourceT type is lacking a Semigroup instance on which servant-multipart-client-0.12.1 relies on.

If my understanding is correct, servant-multipart-client 0.12.1 is effectively incompatible with servant 0.16.

Details:

ghc-8.6.5

$ stack exec ghc-pkg list | grep servant
    servant-0.16.0.1
    servant-client-0.16
    servant-client-core-0.16
    servant-multipart-0.11.4
    servant-multipart-api-0.12.1
    servant-server-0.16

servant-multipart-client > Configuring servant-multipart-client-0.12.1...
servant-multipart-client > build
servant-multipart-client > Preprocessing library for servant-multipart-client-0.12.1..
servant-multipart-client > Building library for servant-multipart-client-0.12.1..
servant-multipart-client > [1 of 1] Compiling Servant.Multipart.Client
servant-multipart-client >
servant-multipart-client > /tmp/stack2910/servant-multipart-client-0.12.1/src/Servant/Multipart/Client.hs:116:51: error:
servant-multipart-client >     • Could not deduce (Semigroup (SourceT IO LBS.ByteString))
servant-multipart-client >         arising from a use of ‘<>’
servant-multipart-client >       from the context: MultipartClient tag
servant-multipart-client >         bound by the type signature for:
servant-multipart-client >                    multipartToBody :: forall tag.
servant-multipart-client >                                       MultipartClient tag =>
servant-multipart-client >                                       LBS.ByteString -> MultipartData tag -> RequestBody
servant-multipart-client >         at src/Servant/Multipart/Client.hs:(111,1)-(115,30)
servant-multipart-client >     • In the second argument of ‘($)’, namely
servant-multipart-client >         ‘files' <> source ["--", boundary, "--"]’
servant-multipart-client >       In the expression:
servant-multipart-client >         RequestBodySource $ files' <> source ["--", boundary, "--"]
servant-multipart-client >       In an equation for ‘multipartToBody’:
servant-multipart-client >           multipartToBody boundary mp
servant-multipart-client >             = RequestBodySource $ files' <> source ["--", boundary, "--"]
servant-multipart-client >             where
servant-multipart-client >                 (SourceT l) `mappend'` (SourceT r) = SourceT $ \ k -> l $ ...
servant-multipart-client >                 appendStep Stop r = r
servant-multipart-client >                 appendStep (Error err) _ = Error err
servant-multipart-client >                 appendStep (Skip s) r = appendStep s r
servant-multipart-client >                 appendStep (Yield x s) r = Yield x (appendStep s r)
servant-multipart-client >                 appendStep (Effect ms) r = Effect $ (flip appendStep r <$> ms)
servant-multipart-client >                 mempty' = SourceT ($ Stop)
servant-multipart-client >                 crlf = "\r\n"
servant-multipart-client >                 ....
servant-multipart-client >     |
servant-multipart-client > 116 | multipartToBody boundary mp = RequestBodySource $ files' <> source ["--", boundary, "--"]
servant-multipart-client >     |                                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Get original file/extension of uploaded file

Hi. Is there a way I can fetch the .extension and original filename of the file being uploaded?
When I upload a file it goes into a temporary folder
/var/folders/hy/c7smkw390xl71sdqgxz80pth0000gn/T/servant-multipart79683-1.buf

I understand that I can change this pattern and the location of the temporary folder. But I fail to see how I can include the original name of the file and the extension.

Suggest restricting genBoundary to alphaNum?

Even though the RFC [1] states that some special characters are allowed, today I encountered an implementation (while writing a client) that results in HTTP 500 whenever the boundary includes a comma (,).

I wouldn't be surprised if more implementations have parsing bugs that can cause issues like this, and perhaps it is better to use only alpha-numeric values?

To add some legitimacy to this problem, even Chromium has had to work around this issue:
https://github.com/chromium/chromium/blob/6efa1184771ace08f3e2162b0255c93526d1750d/net/base/mime_util.cc#L662-L670

[1] Pg. 21: https://tools.ietf.org/html/rfc2046#section-5.1.1

Proper response after max file size is exceeded

WAI developers decided to do error "Maximum size exceeded" here:

https://github.com/yesodweb/wai/blob/a1cb17ff79887e0c33be7fb975b5a0f559e2cf45/wai-extra/Network/Wai/Parse.hs#L634

Which ends up as 500 Internal Server Error instead of the correct 413 Payload Too Large.

What can we do here? A custom Network.Wai.Handler.Warp.setOnExceptionResponse that matches that error text? 😭 😭 😭 Have mercy!

Document how to use the combinator as a client..

.. or change the description to indicate that this is only server-side.

Support API modifier

It would be nice if we can distinguish between optional and required arguments.

Split package

servant-multipart, unlike most packages which introduce new combinators, isn't split up (into -server, -docs, etc.). So any API that uses it will incur all depedencies (and likely can't run in GHCJS, etc.).

@alpmestan @phadej I imagine I can go ahead and split it, no?

How do I increase maximum amount of files per request?

After implementing a tiny service I realized that there is a limit of maximum amount of files which is just 10 files.

I found this option https://hackage.haskell.org/package/wai-extra-3.1.6/docs/Network-Wai-Parse.html#v:setMaxRequestNumFiles and was trying to figure out how to apply it using servant-multipart. I found here https://hackage.haskell.org/package/servant-multipart-0.12.1/docs/src/Servant.Multipart.html that it uses defaultMultipartOptions which has defaultParseRequestBodyOptions which I probably want to modify here.

I was trying to implement overlapping HasServer (MultipartForm' mods tag a :> sublayout) config instance but it turns out it uses some functions that are not exported from Servant.Multipart. It’s getting trickier. Is there any way to override ParseRequestBodyOptions for that instance?

New release containing client-side functionality

Hello, I would am interesting in using this library, specifically the client side functionality. What would it take to get a new release out and into Stackage?
Thanks!

Breaking change in 0.11.6

0.11.6 should actually have been 0.12 given the breaking change in #36 (it was mentioned here, but probably forgotten by release time.)

Not a big deal for me now I've found it but you might consider releasing 0.12 and deprecating 0.11.6 on hackage.

Documentation example not working around fdPayload

Hello,

The following example documentation seems to be wrong:

  data User = User { username :: Text, pic :: FilePath }

  instance FromMultipart Tmp User where
    fromMultipart multipartData =
      User <$> lookupInput "username" multipartData
           <*> fmap fdPayload (lookupFile "pic" multipartData)

Available here: https://hackage.haskell.org/package/servant-multipart-0.11.3/docs/Servant-Multipart.html

The fdPayload returns a fdPayload :: MultipartResult tag and not a FilePath.

Can you confirm?

Thank you

Relax lens lower bound

servant-multipart-0.11.3 depends on lens >= 4.17 && < 4.18, while the previous version depended on lens >= 4.0 && < 4.18.

servant-multipart uses four functions from Control.Lens, and looking at the lens changelog none of them seem to have recent changes to behaviour. Can the lower bound please be relaxed, and a metadata revision pushed to hackage?

(Among other things, this is currently breaking installing servant-multipart via nixpkgs on darwin.)

is there a way to have per-route file size limitations?

Only way i can see to do it right now is to create a subserver with a different config/context as a wai app then mount it with Raw, which is rather unpleasant.

GHCJS compilation

If you try to compile the package with GHCJS (for instance, for generation of frontend from servant API), you will get:

error: no C compiler provided for this platform

Provide HasForeign instance for servant-foreign

My use case is I am using servant-purescript to generate purescript code from my API, but since there is no HasForeign instance for Servant.Multipart.MultipartForm, compilation fails. I wish I could contribute a patch for this but at the time of this writing, the HasForeign typeclass and it's instances are beyond my grasp

When file input is left blank, still a temporary file is created

I don't know if is intended that way. It confused me a bit. I guess I can check whether or not the user actually filled in the file input by check fdFileName.

GHC 9.6.3 support

Hello,
I've just tried compiling servant-multipart with GHC 9.6.3 and am hitting this issue:

servant-multipart-client> build
servant-multipart-client> Preprocessing executable 'server' for servant-multipart-client-0.12.1..
servant-multipart-client> Building executable 'server' for servant-multipart-client-0.12.1..
servant-multipart-client> [1 of 1] Compiling Main
servant-multipart-client> [2 of 2] Linking .stack-work/dist/aarch64-osx/Cabal-3.10.1.0/build/server/server
servant-multipart-client> ld: warning: ignoring duplicate libraries: '-lm'
servant-multipart-client> Preprocessing library for servant-multipart-client-0.12.1..
servant-multipart-client> Building library for servant-multipart-client-0.12.1..
servant-multipart-client> [1 of 1] Compiling Servant.Multipart.Client
servant-multipart-client>
servant-multipart-client> /private/var/folders/0c/zmpjp7l568xcvnt49pkkn5p00000gn/T/stack-29e507a3bc44f1b9/servant-multipart-client-0.12.1/src/Servant/Multipart/Client.hs:77:18: error: [GHC-39999]
servant-multipart-client>     • Ambiguous type variable ‘f1’ arising from a use of ‘source’
servant-multipart-client>       prevents the constraint ‘(Foldable f1)’ from being solved.
servant-multipart-client>       Probable fix: use a type annotation to specify what ‘f1’ should be.
servant-multipart-client>       Potentially matching instances:
servant-multipart-client>         instance Foldable (Either a) -- Defined in ‘Data.Foldable’
servant-multipart-client>         instance Foldable Proxy -- Defined in ‘Data.Foldable’
servant-multipart-client>         ...plus 7 others
servant-multipart-client>         ...plus 93 instances involving out-of-scope types
servant-multipart-client>         (use -fprint-potential-instances to see them all)
servant-multipart-client>     • In the first argument of ‘(.)’, namely ‘source’
servant-multipart-client>       In the expression: source . pure
servant-multipart-client>       In an equation for ‘loadFile’: loadFile _ = source . pure
servant-multipart-client>    |
servant-multipart-client> 77 |     loadFile _ = source . pure
servant-multipart-client>    |                  ^^^^^^
servant-multipart-client>
servant-multipart-client> /private/var/folders/0c/zmpjp7l568xcvnt49pkkn5p00000gn/T/stack-29e507a3bc44f1b9/servant-multipart-client-0.12.1/src/Servant/Multipart/Client.hs:77:27: error: [GHC-39999]
servant-multipart-client>     • Ambiguous type variable ‘f1’ arising from a use of ‘pure’
servant-multipart-client>       prevents the constraint ‘(Applicative f1)’ from being solved.
servant-multipart-client>       Probable fix: use a type annotation to specify what ‘f1’ should be.
servant-multipart-client>       Potentially matching instances:
servant-multipart-client>         instance Applicative (Either e) -- Defined in ‘Data.Either’
servant-multipart-client>         instance Applicative Proxy -- Defined in ‘Data.Proxy’
servant-multipart-client>         ...plus 8 others
servant-multipart-client>         ...plus 75 instances involving out-of-scope types
servant-multipart-client>         (use -fprint-potential-instances to see them all)
servant-multipart-client>     • In the second argument of ‘(.)’, namely ‘pure’
servant-multipart-client>       In the expression: source . pure
servant-multipart-client>       In an equation for ‘loadFile’: loadFile _ = source . pure
servant-multipart-client>    |
servant-multipart-client> 77 |     loadFile _ = source . pure
servant-multipart-client>    |                           ^^^^
servant-multipart-client>
servant-multipart-client> /private/var/folders/0c/zmpjp7l568xcvnt49pkkn5p00000gn/T/stack-29e507a3bc44f1b9/servant-multipart-client-0.12.1/src/Servant/Multipart/Client.hs:135:37: error: [GHC-39999]
servant-multipart-client>     • Could not deduce ‘Foldable f0’ arising from a use of ‘source’
servant-multipart-client>       from the context: MultipartClient tag
servant-multipart-client>         bound by the type signature for:
servant-multipart-client>                    multipartToBody :: forall tag.
servant-multipart-client>                                       MultipartClient tag =>
servant-multipart-client>                                       LBS.ByteString -> MultipartData tag -> RequestBody
servant-multipart-client>         at src/Servant/Multipart/Client.hs:(111,1)-(115,30)
servant-multipart-client>       The type variable ‘f0’ is ambiguous
servant-multipart-client>       Potentially matching instances:
servant-multipart-client>         instance Foldable (Either a) -- Defined in ‘Data.Foldable’
servant-multipart-client>         instance Foldable Proxy -- Defined in ‘Data.Foldable’
servant-multipart-client>         ...plus 7 others
servant-multipart-client>         ...plus 93 instances involving out-of-scope types
servant-multipart-client>         (use -fprint-potential-instances to see them all)
servant-multipart-client>     • In the first argument of ‘(.)’, namely ‘source’
servant-multipart-client>       In the first argument of ‘($)’, namely
servant-multipart-client>         ‘source . pure . lencode . iValue’
servant-multipart-client>       In the fourth argument of ‘renderPart’, namely
servant-multipart-client>         ‘(source . pure . lencode . iValue $ input)’
servant-multipart-client>     |
servant-multipart-client> 135 |                                    (source . pure . lencode . iValue $ input)
servant-multipart-client>     |                                     ^^^^^^
servant-multipart-client>
servant-multipart-client> /private/var/folders/0c/zmpjp7l568xcvnt49pkkn5p00000gn/T/stack-29e507a3bc44f1b9/servant-multipart-client-0.12.1/src/Servant/Multipart/Client.hs:135:46: error: [GHC-39999]
servant-multipart-client>     • Could not deduce ‘Applicative f0’ arising from a use of ‘pure’
servant-multipart-client>       from the context: MultipartClient tag
servant-multipart-client>         bound by the type signature for:
servant-multipart-client>                    multipartToBody :: forall tag.
servant-multipart-client>                                       MultipartClient tag =>
servant-multipart-client>                                       LBS.ByteString -> MultipartData tag -> RequestBody
servant-multipart-client>         at src/Servant/Multipart/Client.hs:(111,1)-(115,30)
servant-multipart-client>       The type variable ‘f0’ is ambiguous
servant-multipart-client>       Potentially matching instances:
servant-multipart-client>         instance Applicative (Either e) -- Defined in ‘Data.Either’
servant-multipart-client>         instance Applicative Proxy -- Defined in ‘Data.Proxy’
servant-multipart-client>         ...plus 8 others
servant-multipart-client>         ...plus 75 instances involving out-of-scope types
servant-multipart-client>         (use -fprint-potential-instances to see them all)
servant-multipart-client>     • In the first argument of ‘(.)’, namely ‘pure’
servant-multipart-client>       In the second argument of ‘(.)’, namely ‘pure . lencode . iValue’
servant-multipart-client>       In the first argument of ‘($)’, namely
servant-multipart-client>         ‘source . pure . lencode . iValue’
servant-multipart-client>     |
servant-multipart-client> 135 |                                    (source . pure . lencode . iValue $ input)
servant-multipart-client>     |                                              ^^^^

provide HasDocs instance for MultipartForm

I'd like to send a PR adding a HasDocs instance for MultipartForm.

I have an instance HasDocs for MultipartForm working in my current project. Here's the full code.

Below I will explain the different parts.

Here is the HasDocs instance for MultipartForm:

instance (HasDocs api, ToMultipartSample a) => HasDocs (MultipartForm a :> api) where
  docsFor
    :: Proxy (MultipartForm a :> api)
    -> (Endpoint, Action)
    -> DocOptions
    -> API
  docsFor _ (endpoint, action) opts =
    let newAction =
          action
            & notes <>~
                [ toMultipartNotes
                    (view maxSamples opts)
                    (Proxy :: Proxy a)
                ]
    in docsFor (Proxy :: Proxy api) (endpoint, newAction) opts

You can see that it is using the ToMultipartSample class. This is similar to the ToSample class that currently exists in servant-docs:

class ToMultipartSample a where
  toMultipartSamples :: Proxy a -> [(Text, MultipartData)]

Here's a datatype I'm using in my app and the instance of ToMultipartSample for it:

data PostImgForm = PostImgForm
  { filename :: FilePath
  , date :: UTCTime
  , geom :: Geom
  }

instance ToMultipartSample PostImgForm where
  toMultipartSamples :: Proxy PostImgForm -> [(Text, MultipartData)]
  toMultipartSamples Proxy =
    [ ( "normal image upload"
      , MultipartData
          [ Input "date" "2017-09-10 08:23 Z"
          , Input "lat" "-33"
          , Input "lon" "-100"
          ]
          [ FileData
              "file"
              "temp-cat-image.jpg"
              "image/jpeg"
              "/tmp/tmppath.file"
          ]
      )
    , ( "upload sample 2"
      , MultipartData
          [ Input "date" "2017-08-22 13:23 Z"
          , Input "lat" "50"
          , Input "lon" "150"
          ]
          [ FileData
              "file"
              ""
              "image/jpeg"
              ""
          ]
      )
    ]

There are some additional helper functions used in the HasDocs instance for MultipartForm. They can be found in the code linked above.

Running this for a route in my application produces documentation that looks like this:

## POST /v0/image

#### Multipart Request Samples

This endpoint takes `multipart/form-data` requests.  The following is a list of sample requests:

- normal image upload
    - textual inputs (any `<input>` type but file):
        - *date*: `2017-09-10 08:23 Z`
        - *lat*: `-33`
        - *lon*: `-100`
    - file inputs (any HTML input that looks like `<input type="file" name="somefile" />`):
        - *file*, content-type: `image/jpeg`

- upload sample 2
    - textual inputs (any `<input>` type but file):
        - *date*: `2017-08-22 13:23 Z`
        - *lat*: `50`
        - *lon*: `150`
    - file inputs (any HTML input that looks like `<input type="file" name="somefile" />`):
        - *file*, content-type: `image/jpeg`

#### Authentication

Clients must supply the following data

#### Response:

- Status code 200
- Headers: []

- Supported content types are:

    - `application/json;charset=utf-8`
    - `application/json`

- sample uuid key

``javascript
{"data":"0359141a-9ce8-4ca9-b93c-d9d017ece471"}
``

...

If I send a PR, I'm wondering whether I should just throw this HasDocs instance in with the current code, or whether I should split this package into servant-multipart and servant-multipart-docs (similar to how servant itself is split up)?

I could also go ahead and split this package up into servant-multipart, servant-multipart-docs, and servant-multipart-server. This would make it easier for someone to come along and implement servant-multipart-client.

Hackage release with recent GHC support commits?

Hello, I saw there have been a few commits since the last release (a940d3f). These commits tweak version bounds to work with recent servant versions, like those in the Stackage snapshots for GHC 9.0.2 and 9.2. Any chance we could get a hackage release with these changes? Thanks!

haskell-servant / servant-multipart Goto Github PK

servant-multipart's People

Contributors

Stargazers

Watchers

Forkers

servant-multipart's Issues

Recommend Projects

Recommend Topics

Recommend Org