Giter VIP home page Giter VIP logo

mega-sdist's Introduction

mega-sdist

This is a utility written to address the specific needs in maintaining Haskell "mega-repos," or Git repositories containing multiple Cabal projects. It is intended to ease the process of deciding which packages need to be released and tagging those releases appropriately.

It provides the following functionality:

  • Detect when local code has changed from what's on Hackage
    • Note that, due to Hackage revisions, sometimes this logic isn't perfect
  • Detect when a version number needs to be updated
  • Dump the difference between the Hackage version of your package and the local version

To install it... well, listen. This tool is intended for people authoring Haskell packages. Odds are, you already know how to do this. And if you don't know, this probably isn't a tool that will help you. Anyway, in order to install it, first install Stack and then run stack install mega-sdist, or just stack install inside this repository.

Opinionated tool

This utility is highly opinionated in some ways, e.g.:

  • It only supports one style of Git tag name: packagename-version. This may look weird in non-mega-repos, where v1.2.3 looks better than foo-1.2.3, but for mega-repos the former doesn't make sense. It also requires tag signing.
  • It depends on Stack for both discovering all of your local packages, and for uploading to Hackage.

If you're OK with these opinions, keep reading for usage.

Have I changed anything?

Let's say I'm working on the monad-unlift megarepo (chosen as an example of a relatively small repo). I've merged some PRs recently, or at least think I have. But I don't remember which of the individual packages within the repo this affected. Instead of looking at the commit history like some caveman, I'll typically do:

$ git pull # make sure I have all latest changes
$ mega-sdist

The mega-sdist command will:

  • Build tarballs for all local packages
  • Check what the latest versions of my packages on Hackage are
  • Do a full diff on these two things and see if anything's changed

At the time of writing, here's the output from this repo:

The following packages from Hackage have not changed:
monad-unlift-0.2.0

The following packages require a version bump:
monad-unlift-ref-0.2.1

What this means is:

  • The monad-unlift package I have locally is at version 0.2.0. And it perfectly matches that version on Hackage. No actions necessary.
  • The monad-unlift-ref package I have locally is at version 0.2.1. And it doesn't match the code on Hackage. Therefore, if I wanted to run stack upload monad-unlift-ref successfully, I'd need to bump the version number.

What did I change?

Well, again, if I wanted to see what changed, I could run (again, like a caveman):

$ git diff monad-unlift-ref/0.2.1 -- monad-unlift-ref

But that's long! mega-sidst's got your back. Just run:

$ mega-sdist monad-unlift-ref --get-diffs

This will print out the difference between the tarball uploaded to Hackage and what you have locally. Besides my tongue-in-cheek comment above, this is also useful if, for some reason, you either don't have or don't trust the tags in your Git repo.

One other thing: this diff is currently based on the pristine tarball from Hackage, ignoring cabal file revisions. So the difference may be slightly different from what you'd get from stack unpack monad-unlift-ref-0.2.1. But ¯\_(ツ)_/¯ that's revisions for you.

The default behavior of mega-sdist is to look at all packages specified in your stack.yaml. Targets can be any directory. And mega-sdist will automatically look at packages in any subdirectory, so that mega-sdist . is the same as mega-sdist at the root of your repo*.

* Assuming all of your packages are actually in your repo, but only crazy people would do otherwise.

Preparing a new release

OK, now I continue working on my project, and I've:

  • Made some changes to monad-unlift
  • Updated the cabal file's version number
    • And of course I also updated the ChangeLog.md, I'm not some monster

From the root of my repo, I run:

$ mega-sdist monad-unlift

Or, equivalently, from inside the monad-unlift subdirectory I run:

$ mega-sdist .

Either way, I get:

The following new packages exist locally:
monad-unlift-0.2.1

No version bumps required, good to go!

This tells me that my package has local changes, and the version number has been updated, so that stack upload monad-unlift will work. Neato! Now, you could just run stack upload ..., but here's what I usually do. First, I'll review the changes I'm about to upload and make sure there are no surprises:

$ mega-sdist --get-diffs .

The following new packages exist locally:
monad-unlift-0.2.1
diff -r old/monad-unlift-0.2.0/ChangeLog.md new/monad-unlift-0.2.1/ChangeLog.md
0a1,4
> ## 0.2.1
>
> * Silly changes
>
diff -r old/monad-unlift-0.2.0/Control/Monad/Trans/Unlift.hs new/monad-unlift-0.2.1/Control/Monad/Trans/Unlift.hs
51a52,54
>
> -- I just need some space
>
diff -r old/monad-unlift-0.2.0/monad-unlift.cabal new/monad-unlift-0.2.1/monad-unlift.cabal
2c2
< version:             0.2.0
---
> version:             0.2.1

No version bumps required, good to go!

OK, that's what I wanted. Time to release. Next, I'm going to use mega-sdist to tag the release:

$ mega-sdist --gittag .

From the root of my repo, this would notice that monad-unlift-ref still requires a version bump, and refuse to proceed. But inside the monad-unlift directory, it notices that all necessary version bumps are done, and happily tags:

$ mega-sdist --gittag .
The following new packages exist locally:
monad-unlift-0.2.1

No version bumps required, good to go!
Raw command: git tag -s monad-unlift-0.2.1 -m "Release: monad-unlift-0.2.1"

And suddenly I notice something new:

$ ls tarballs/
monad-unlift-0.2.1.tar.gz

Neat, mega-sdist left behind tarballs I can upload! To do so, I run:

$ stack upload tarballs/*

Note that this will work whether I'm trying to upload just one package, or all of the updated packages in my repo. Finally, I need to push the new tags to Github (or wherever):

$ git push --tags

And in fact, this upload sequence is so common that I have a shell alias set up:

$ alias upload
alias upload='mega-sdist --gittag . && stack upload tarballs/* && git push --tags'

So there you have it: convenient little utility to help manage repos with lots of packages in them.

mega-sdist's People

Contributors

snoyberg avatar mpilgrem avatar gregwebs avatar

Stargazers

Tristan de Cacqueray avatar Alexander Bondarenko avatar Ilmari Vacklin avatar Javier Neira  avatar  avatar Vincent Hanquez avatar Yo Eight avatar Daniel Kahlenberg avatar Dmitrii Kovanikov avatar Casey Allred avatar Anders Persson avatar  avatar

Watchers

 avatar Vincent Hanquez avatar Alexander Bondarenko avatar James Cloos avatar  avatar  avatar  avatar

Forkers

mpilgrem

mega-sdist's Issues

could not find module ‘Data.Conduit.Zlib’

cabal install mega-sdist
Resolving dependencies...
Notice: installing into a sandbox located at
/home/gweber/lib/mega-sdist/.cabal-sandbox
Configuring mega-sdist-0.2.10.4...
Building mega-sdist-0.2.10.4...
Failed to install mega-sdist-0.2.10.4
Build log ( /home/gweber/lib/mega-sdist/.cabal-sandbox/logs/mega-sdist-0.2.10.4.log ):
Configuring mega-sdist-0.2.10.4...
Building mega-sdist-0.2.10.4...
Preprocessing executable 'mega-sdist' for mega-sdist-0.2.10.4...

mega-sdist.hs:18:8:
Could not find module ‘Data.Conduit.Zlib’
Perhaps you meant
Data.Conduit.List (from conduit-1.2.4)
Data.Conduit.Lift (from conduit-1.2.4)
Use -v to see a list of the files searched for.
cabal: Error: some packages failed to install:
mega-sdist-0.2.10.4 failed during the building phase. The exception was:
ExitFailure 1

Build failure with classy-prelude-1.4

I'm not sure how this was not detected during the last nightly.

[2 of 2] Compiling Main             ( mega-sdist.hs, dist/build/mega-sdist/mega-sdist-tmp/Main.o )

mega-sdist.hs:105:16: error:
    Ambiguous occurrence ‘hIsTerminalDevice’
    It could refer to either ‘ClassyPrelude.Conduit.hIsTerminalDevice’,
                             imported from ‘ClassyPrelude.Conduit’ at mega-sdist.hs:6:1-28
                             (and originally defined in ‘unliftio-0.2.4.0:UnliftIO.IO’)
                          or ‘System.IO.hIsTerminalDevice’,
                             imported from ‘System.IO’ at mega-sdist.hs:17:19-35
                             (and originally defined in ‘GHC.IO.Handle’)
    |
105 |     toColor <- hIsTerminalDevice stdout
    |                ^^^^^^^^^^^^^^^^^

mega-sdist.hs:275:18: error:
    Ambiguous occurrence ‘withSystemTempDirectory’
    It could refer to either ‘ClassyPrelude.Conduit.withSystemTempDirectory’,
                             imported from ‘ClassyPrelude.Conduit’ at mega-sdist.hs:6:1-28
                             (and originally defined in ‘unliftio-0.2.4.0:UnliftIO.Temporary’)
                          or ‘System.IO.Temp.withSystemTempDirectory’,
                             imported from ‘System.IO.Temp’ at mega-sdist.hs:18:24-46
    |
275 |             then withSystemTempDirectory "diff" $ \diff -> do
    |                  ^^^^^^^^^^^^^^^^^^^^^^^

repro:

stack unpack mega-sdist-0.3.1 && cd mega-sdist-0.3.1
echo 'resolver: nightly-2018-03-12' > stack.yaml
stack build

Stackage nightly build failure

mega-sdist.hs:189:11: error:
    <E2><80><A2> Couldn't match expected type <E2><80><98>ConduitM
                                      ByteString TarChunk (ResourceT IO) ()<E2><80><99>
                  with actual type <E2><80><98>(FileInfo -> ConduitM ByteString o0 m0 ())
                                    -> ConduitM ByteString o0 m0 ()<E2><80><99>
    <E2><80><A2> Probable cause: <E2><80><98>untar<E2><80><99> is applied to too few arguments
      In the first argument of <E2><80><98>(.|)<E2><80><99>, namely <E2><80><98>untar<E2><80><99>
      In the second argument of <E2><80><98>(.|)<E2><80><99>, namely
        <E2><80><98>untar .| withEntries yield .| foldMapC (parseVersionNumber name)<E2><80><99>
      In the second argument of <E2><80><98>($)<E2><80><99>, namely
        <E2><80><98>sourceFile indexTar
           .|
             untar .| withEntries yield .| foldMapC (parseVersionNumber name)<E2><80><99>
    |
189 |        .| untar
    |           ^^^^^


mega-sdist.hs:300:12: error:
    <E2><80><A2> Couldn't match expected type <E2><80><98>ConduitM
                                      ByteString TarChunk (ResourceT IO) ()<E2><80><99>
                  with actual type <E2><80><98>(FileInfo -> ConduitM ByteString o1 m1 ())
                                    -> ConduitM ByteString o1 m1 ()<E2><80><99>
    <E2><80><A2> Probable cause: <E2><80><98>untar<E2><80><99> is applied to too few arguments
      In the first argument of <E2><80><98>(.|)<E2><80><99>, namely <E2><80><98>untar<E2><80><99>
      In the second argument of <E2><80><98>(.|)<E2><80><99>, namely
        <E2><80><98>untar .| withEntries addEntry .| foldC<E2><80><99>
      In the second argument of <E2><80><98>(.|)<E2><80><99>, namely
        <E2><80><98>ungzip .| untar .| withEntries addEntry .| foldC<E2><80><99>
    |
300 |         .| untar
    |            ^^^^^

fails to build in current Stackage Nightly

--  While building package mega-sdist-0.4.1.0 (scroll up to its section to see the error) using:                        [44/165469]
      /var/stackage/.stack/setup-exe-cache/x86_64-linux/Cabal-simple_mPHDZzAJ_3.4.1.0_ghc-9.0.2 --builddir=.stack-work/dist/x86_64-
linux/Cabal-3.4.1.0 build exe:mega-sdist --ghc-options ""                                                                          
    Process exited with code: ExitFailure 1                                                                                        
    Logs have been written to: /var/stackage/work/unpack-dir/.stack-work/logs/mega-sdist-0.4.1.0.log                                                                                                                                                                  
Preprocessing executable 'mega-sdist' for mega-sdist-0.4.1.0..                                                                     
Building executable 'mega-sdist' for mega-sdist-0.4.1.0..                                                                          
[2 of 2] Compiling Main                                                                                                            
                                                                                                                                   
/var/stackage/work/unpack-dir/unpacked/mega-sdist-0.4.1.0-005deee3d3cc65ff684a657f9682f0fc759f89024fb542d6d3a443c9122ffaee/mega-sdi
st.hs:86:7: error:              
    • Couldn't match expected type: RIO LogFunc ()                                                                                 
                  with actual type: (PantryConfig -> RIO env0 a0) -> RIO env0 a0
    • In the second argument of ‘($)’, namely                                                                                      
        ‘withPantryConfig                                        
           root defaultHackageSecurityConfig HpackBundled 8                                                                        
           defaultCasaRepoPrefix defaultCasaMaxPerRequest                                                                          
           $ \ pantryConfig                                      
               -> do let ...                                     
                     runRIO app $ main2 args’                                                                                      
      In the expression:                                         
        runRIO lf               
          $ withPantryConfig                                     
              root defaultHackageSecurityConfig HpackBundled 8                                                                     
              defaultCasaRepoPrefix defaultCasaMaxPerRequest                                                                       
              $ \ pantryConfig                                   
                  -> do let ...                                  
                        runRIO app $ main2 args                                                                                    
      In the second argument of ‘($)’, namely                                                                                      
        ‘\ rm                   
           -> runRIO lf                                          
                $ withPantryConfig                               
                    root defaultHackageSecurityConfig HpackBundled 8                                                               
                    defaultCasaRepoPrefix defaultCasaMaxPerRequest                                                                 
                    $ \ pantryConfig -> do ...’                                                                                    
   |                            
86 |       withPantryConfig                                                                                              [7/165469]
   |       ^^^^^^^^^^^^^^^^...                                   

/var/stackage/work/unpack-dir/unpacked/mega-sdist-0.4.1.0-005deee3d3cc65ff684a657f9682f0fc759f89024fb542d6d3a443c9122ffaee/mega-sdi
st.hs:98:39: error:             
    • Couldn't match expected type ‘PantryConfig’                                                                                  
                  with actual type ‘SnapName’                                                                                      
    • In the ‘appPantryConfig’ field of a record                                                                                   
      In the expression:                                         
        App                     
          {appLogFunc = lf, appProcessContext = pc, appResourceMap = rm,                                                           
           appPantryConfig = pantryConfig}                                                                                         
      In an equation for ‘app’:                                  
          app                   
            = App               
                {appLogFunc = lf, appProcessContext = pc, appResourceMap = rm,                                                     
                 appPantryConfig = pantryConfig}                                                                                   
   |                            
98 |                   , appPantryConfig = pantryConfig                                                                            
   |                                       ^^^^^^^^^^^^                                                                            

/var/stackage/work/unpack-dir/unpacked/mega-sdist-0.4.1.0-005deee3d3cc65ff684a657f9682f0fc759f89024fb542d6d3a443c9122ffaee/mega-sdi
st.hs:100:13: error:            
    • Couldn't match expected type ‘RawSnapshotLocation’                                                                           
                  with actual type ‘m0 ()’                                                                                         
    • In a stmt of a 'do' block: runRIO app $ main2 args                                                                           
      In the expression:                                         
        do let app = ...                                         
           runRIO app $ main2 args                               
      In the second argument of ‘($)’, namely                                                                                      
        ‘\ pantryConfig                                          
           -> do let ...                                         
                 runRIO app $ main2 args’                                                                                          
    |                           
100 |             runRIO app $ main2 args                                                                                          
    |             ^^^^^^^^^^^^^^^^^^^^^^^                                                                                          

TBH I don't even understand how this package got enabled in Nightly - since pantry was disabled for quite a while until last week.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.