Flat File Assembler Encoding and Charset

I’ve had lots of problems recently with file encodings including foreign and special characters not appearing properly in target flat files. For example ø being written as ? and even worse an accented é translated as 2 characters é which meant that the column aligned file got messed up.

The reason is that the source system provides the data in UTF-8 (which is the best encoding because it should in theory deal with all character sets) but the files were being created as ANSI which is far more limited and unless you specify the charset will cause the issues I have described. However the target system I am creating files for is a little archaic and does actually require files with an ANSI charset.

BizTalk can handle this issue quite well out the box but you have to specify the charset in the flatfile component otherwise it will take a guess which will in all likelyhood be wrong. I have also found that you have to specify this at design time in visual studio otherwise the change will not be effective. So although you have the option of specifying the charset in the send port properties, this does not work properly – must be a bug in BizTalk.

In visual studio open up your pipelene and select the Flat file assembler component.

Then select the file encoding which matches what your target system expects. This might be UTF-8 but may also be ANSI in which case you should select the charset that matches e.g. Western-European (1252).

You should use a similar process when reading files using the flat file disassembler component. And also be careful when using external tools such as an otherwise excellent PGP pipeline component. I had an issue where the pipeline component was writing a file to a temporary location but did’t take the encoding into account and the encoding was lost. Here is the code I used to change the EncryptStream method int he component source code.

        /// <summary>
        /// Encrypts the stream.
        /// </summary>
        /// <param name="inStream">The in stream.</param>
        /// <param name="pubKeyFile">The pub key file.</param>
        /// <param name="sourceFilename">The source filename.</param>
        /// <param name="extension">The extension.</param>
        /// <param name="armorFlag">if set to <c>true</c> [armor flag].</param>
        /// <param name="integrityCheckFlag">if set to <c>true</c> [integrity check flag].</param>
        /// <param name="strCharset">The STR charset.</param>
        /// <param name="targetCharset">The target charset.</param>
        /// <returns></returns>
        public static Stream EncryptStream(Stream inStream, string pubKeyFile, string sourceFilename, string extension, bool armorFlag, bool integrityCheckFlag, String strCharset, String targetCharset)
        {
            string tmpEncryptedFile = sourceFilename + "." + extension;

            // View debug lines with Debugview.exe
            //System.Diagnostics.Debug.WriteLine(strCharset + " is the charset");

            try
            {

                using (StreamWriter sourceStream = new StreamWriter(sourceFilename, true, Encoding.GetEncoding(targetCharset)))
                {
                    using (StreamReader sr = new StreamReader(inStream, Encoding.GetEncoding(strCharset), true))
                    {
                        sourceStream.Write(sr.ReadToEnd());
                    }
                }

                using (FileStream encryptedStream = File.Create(tmpEncryptedFile))
                {
                    PgpPublicKey publicKey = ReadPublicKey(File.OpenRead(pubKeyFile));
                    EncryptFile(encryptedStream, sourceFilename, publicKey, armorFlag, integrityCheckFlag);

                    encryptedStream.Seek(0, SeekOrigin.Begin);
                    int myCapacity = Int32.Parse(encryptedStream.Length.ToString());
                    Byte[] byteArray = new Byte[myCapacity];

                    MemoryStream memStream = new MemoryStream();

                    encryptedStream.Read(byteArray, 0, myCapacity);
                    memStream.Write(byteArray, 0, myCapacity);

                    encryptedStream.Close();

                    memStream.Seek(0, SeekOrigin.Begin);
                    return memStream;
                }
            }
            catch (Exception ex)
            {
                //System.Diagnostics.EventLog.WriteEntry("PGPWrapper[Encrypt] Error", ex.ToString());
                throw ex;
            }
            finally
            {
                // Clean-up temp files
                if (File.Exists(sourceFilename)) { File.Delete(sourceFilename); }
                if (File.Exists(tmpEncryptedFile)) { File.Delete(tmpEncryptedFile); }

                //FileInfo fileInfo = new FileInfo(sourceFilename);
                //System.Diagnostics.EventLog.WriteEntry("PGP_Debug", fileInfo.FullName + " was deleted");

                //fileInfo = new FileInfo(tmpEncryptedFile);
                //System.Diagnostics.EventLog.WriteEntry("PGP_Debug", fileInfo.FullName + " was deleted");
            }
        }