More effecient piping with large data

[codygman]

Having an issue with figuring out how to use handles. Also wanted to verify handles are the correct solution rather than say runFoldLines or the other alternative I can't recall at the moment.

Surely examples of how to do the two snippets below in Shelly would be useful to many people:

zcat large.mysql.gz | mysql largedb

cat bigfile | grep things

I guess those two are pretty much the same, though I think runFoldLines might be a better alternative for the second.

{-# LANGUAGE ExtendedDefaultRules #-}
{-# LANGUAGE OverloadedStrings    #-}
{-# OPTIONS_GHC -fno-warn-type-defaults #-}
import           Data.Monoid
import           Data.Text   as T
import           Shelly
default (T.Text)

main = shelly $ do
  -- I'm trying to do "zcat large.mysql.gz | mysql largedb" basically
  -- minimal reproduction:
  run "echo" ["a very large file"] -|- run_ "grep" ["file"] -- predictably runs out of memory
  -- runHandle "echo" ["test"] $ \h -> run "grep" ["test"] -- thought this was the equivalent

[gregwebs]

The easiest way to run these right now is to just let the shell do it. Something like the following (untested).

shelly $ escaping False
  run "sh" ["-c", "zcat large.mysql.gz | mysql largedb"]

There is no magic going on with the handles. It is just a low level interface. If you want to create a connection through handles you would need to use runHandles and use the handle from the first command as an input handle. Perhaps you can post some example code that does this and we can make some conveniences for it?