Background

I am using Google Cloud’s Dataform for data processing.
I have standardized constant definitions and SQL rendering functions in JavaScript, which are called from the sqlx file.

I can reference the constants defined in JavaScript from the .sqlx as expected, some constants are becoming undefined.

I don’t know the cause and need help.

Example code

example.js

const CONST_STRING_01 = "hoge";
const CONST_STRING_02 = "fuga";
// and more. 40+

const CONST_MAP_01 = {"A": "A-", "B": null}
const CONST_MAP_02 = {"X": "X-", "Y": "Y-", "Z": "Z-"}
const CONST_MAP_03 = {"1": null, "2": null}
// and more. 20+

function render_function_01 (map) {
    let renderedCaseSQL = "(generate a case clause from map)";
    return renderedSQL;
}
// and more.

module.exports {
    CONST_STRING_01,
    CONST_STRING_02,
    // and more string constants. (40+)
    CONST_MAP_01,
    CONST_MAP_02,
    CONST_MAP_03,
    // and more map constants. (20+)
    render_function_01,
    // and more functions.
}

example.sqlx

config {
    type: "table",
    schema: "example_schema",
    name: "example_transform",
}

SELECT
    ${example.CONST_STRING_01} AS column_a,
    ${example.render_function_01(example.CONST_MAP_01)} AS column_b,
    ${example.render_function_01(example.CONST_MAP_02)} AS column_c,
    ${example.render_function_01(example.CONST_MAP_03)} AS column_d,
FROM
    ${ref("example_schema", "example_input_table")}

Expected behavior

The value of the constant defined in example.js can be correctly referenced from example.sqlx

System behavior

Most constants can retrieve their defined values. However, some constants becoming undefined.
I am creating multiple combinations of similar .js and .sqlx.
It only occurs in .js with a large number of constant definitions.

• example
  • CONST_STRING_01 -> OK
  • CONST_STRING_02 -> OK
  • and more string constants. (40+) -> OK
  • CONST_MAP_01 -> OK
  • CONST_MAP_02 -> undefined
  • CONST_MAP_03 -> OK
  • // and more map constants. (20+) -> Just one undefined, others OK

Checked

• No compile errors in example.js
• No compile errors in example.sqlx
  • but, since the constant is undefined, it results in SQL syntax error
• constant and function names are unique
• module.exports contains the correct constant name
• usage limit of Dataform
  • not problem
  • link
    • https://cloud.google.com/dataform/docs/dataform-core
    • https://cloud.google.com/dataform/docs/quotas

Try

• Copy example.js to create example2.js. in example.sqlx reference example2.js
  • Succeed temporarily. But sometimes errors suddenly occur while modifying .js or .sqlx
• change the constant name (example: CONST_MAP_02 => CONST_MAP_02a）
  • Succeed temporarily. But sometimes errors suddenly occur while modifying .js or .sqlx
• I created a function that simply returns a MAP to String and checked it.