# processingCsvFiles

State Resource

This is a State Resource as provided by the tymly-etl-plugin plugin.

# Purpose

Smash CSV files into more manageable files based on column values - just a thin wrapper over Smithereens (opens new window)

# Usage

# Example

{
  "processingCsvFiles": {
    "options": {
      "parser": {
        "quote": "\"",
        "delimiter": ",",
        "newline": "\r\n",
        "skipFirstLine": false,
        "trimWhitespace": true
      },
      "dirSplits": [
        {
          "columnIndex": 3,
          "valueToDirMap": {
            "c": "children",
            "a": "adults"
          }
        }
      ],
      "fileSplits": {
        "columnIndex": 4,
        "valueToFileMap": {
          "u": {
            "filename": "changes",
            "outputColumns": [
              {
                "name": "person_no",
                "columnIndex": 0
              },
              {
                "name": "first_name",
                "columnIndex": 1
              },
              {
                "name": "last_name",
                "columnIndex": 2
              }
            ]
          },
          "d": {
            "filename": "deletes",
            "outputColumns": [
              {
                "name": "person_no",
                "columnIndex": 0
              }
            ]
          }
        }
      }
    },
    "events": {
      "csvFilesProcessed": {
        "to": "importingCsvFiles"
      }
    }
  }
}

# Options

# Optional properties

# parser

Object controlling how CSV lines should be parsed. Uses csv-streamify (opens new window) under the bonnet... with the addition of a skipFirstLine property (boolean) for conditionally ignoring header lines.

  • Type: object

# dirSplits

Used to split CSV files across different directories, depending on column values. Please see the dirSplit (opens new window) object in Smithereens (opens new window)for more information.

  • Type: array

# fileSplits

In a similar way, CSV lines can be split across different file too. Please see the fileSplit (opens new window) object in Smithereens (opens new window)for more information.

  • Type: object
Last Updated: 9/10/2023, 11:30:14 AM