How to interpret stdout as utf8?

[hvr]

If I run something like

runGit d "cat-file" ["commit", ref ]

but the environment has a non-utf8 locale set such as LANG=C then I get

Exception: fd:5: hGetLine: invalid argument (invalid byte sequence)

What's the recommended way to run a command which is known to always output utf8 to its stdout/stderr, regardless of any LANG-setting?

[hvr]

Fwiw, I'm currently using the following ugly non-reentrant hack as a workaround:

withUtf8 :: Sh a -> Sh a
withUtf8 act = do
    oldloc <- liftIO getLocaleEncoding
    if (textEncodingName oldloc == textEncodingName utf8)
    then act
    else do
        liftIO $ setLocaleEncoding utf8
        r <- act
        liftIO $ setLocaleEncoding oldloc
        return r

[gregwebs]

I have never seen that predicament. I always change the system locale to something like C.UTF8.

At the lower level Shelly is reading from a Handle. It has a runHandles function where one can directly operate on the handles which you could try to use. If you can verify that using setEncoding on the handle fixes this, I could add an convenience function to Shelly to set the locale on the handle for a command.

[gregwebs]

You may be able to just use runHandle depending on where the error is popping up.

[hsenag]

I've got a similar problem with the Darcs test suite. It contains a couple of scripts with different encodings to test how Darcs handles that, and they fail when run via Shelly because the output isn't valid UTF8 (I think).

Hacking Shelly to do either hSetBinaryMode True or hSetEncoding char8 on both outH and errH in runFoldLines fixed the problem.

If you could add a hook to allow clients to do that properly that'd be great. Happy to write the code myself if you can provide some guidance on how you'd like the hook to look.

[gregwebs]

@hsenag I believe you are reporting a different issue. This issue is about setting the locale for utf8 data. Your issue is that you want to work with binary data. Lets make a separate github issue for that.