binary2csv

Developer’s Description

binary2csv

By jkje
This program converts a MVS Binary file to a CSV file based on its associated copybook. A binary file is a raw file in its original format. In IBM mainframe environment, files are often transferred to PC and then imported to PC softwares such as Microsoft excel or notepad for reports. Unfortunately the mainframe data is in EBCDIC format and its numeric data types such as COMP or COMP-1 or COMP-3 are not recognised by the ASCII-based PC software. Very often, programmer has to write program to convert them to text format before FTP them to PC, for example a data type of S9(5)V99 COMP-3 will have to be converted to a string of text like “+12345.99” which will be then mapped to ASCII characters by FTP software, one byte at a time; but a data type represented by a stream of bytes like COMP or COMP-1 or COMP-3 cannot be translated by FTP correctly.
This program provides you an easy way to convert any mainframe file to a CSV (ASCII-based) file by simply FTP the data file in binary format and copybook to pc and then run this application.

A fast, streaming CSV binary parser written in javascript.

 

Consumes Buffer in node or Uint8Array in the browser (thanks to bops). Whereas most CSV parsers parse String data, this library never converts binary data into non-binary data. It’s fast because it never creates Numbers, Strings, Arrays or Objects — only binary representations of the line and cell values in the CSV, meaning the JS VM spends less time doing things like decoding UTF8 strings and going back and forth between C++ and JS.

By default it will only split lines, but you can use the provided .line and .cell methods to parse the cells and cell values.

Parses a 55 million line, 5.18GB CSV in a little over 1 minute.

demo

See a demo running in the browser on RequireBin:

The data structure that is used to store data in the logger’s memory cannot be downloaded directly, unless you are using a flash card. Even then there are complications with downloading that file remotely, especially if the logger is still writing to it.

The alternative is to select the TOB1 file format to store data on the PC. This is a linear binary file with an ASCII header. Various programs supplied with Loggernet can decode that file or you can relatively easily convert the file yourself. (This process is simpler than decoding the data structure used in the logger memory which we call TOB3 which more complicated as it is designed to cope with the complications of a ring memory structure amongst other things.)

By selecting the TOB1 option you will not save any time or costs in transferring the data as the data in sent in binary format between the logger and PC for normal data collection and converted on the PC to the desired format. You will however save some space if you use TOB1 format.

The distributed generation, along with the deregulation of the Smart Grid, have created a great concern on Power Quality (PQ), as it has a direct impact on utilities and customers, as well as effects on the sinusoidal signal of the power line. The a priori unknown features of the distributed energy resources (DER) introduce non-linear behaviours in loads associated to a variety of PQ disturbances.

Leave a Reply

Your email address will not be published. Required fields are marked *