I am trying to write a simple script that will receive a text via standard input and will output everything as it is except it will replace occurences following this pattern:
{{env MYVAR}}
{{env PATH}}
{{env DISPLAY}}
with the contents of environment variable MYVAR, PATH, DISPLAY, etc.
I am aiming to not having to pass any parameter to this script, so it will automatically detect the patterns {{env VARNAME}}
and replace by the value of environment variable $VARNAME
.
The script takes its input via standard input and provides the output via standard output.
Sample input text via standard input:
This is a basic templating system that can replace environment variables in regular text files.
For example the DISPLAY in this system is {{env DISPLAY}} and the path is {{env PATH}}.
Expected output via standard output:
This is a basic templating system that can replace environment variables in regular text files.
For example the DISPLAY in this system is :0.0 and the path is /bin;/usr/bin;/usr/local/bin.
What I have tried:
So far I have managed only to get it done for one variable by passing it via command line.
#!/bin/sh
# Check if the first argument is set
if [ -z "$1" ]; then
echo "No variable name provided." >&2
exit 1
fi
VARIABLE_NAME="$1"
# Use awk to replace '{{env VARIABLE_NAME}}' with the value of the environment variable
awk -v var_name="$VARIABLE_NAME" '
function escape(s) {
esc = "";
for (i = 1; i <= length(s); i++) {
c = substr(s, i, 1);
if (c ~ /[.[]$()*+?^{|\{}]/) {
esc = esc "\" c;
} else {
esc = esc c;
}
}
return esc;
}
BEGIN {
search = "{{env " var_name "}}";
search_esc = escape(search);
replacement = ENVIRON[var_name];
}
{
gsub(search_esc, replacement);
print;
}'
So the above works but requires you to do ./parsing_script MYVAR
I want to avoid having to specify the environment variables as command line arguments.
Architecture/OS
I am using FreeBSD’s awk and its POSIX shell /bin/sh
Notes
If awk is not the tool, I am open to hear solutions (please no Python or Perl).
5
Assumptions:
- there is exactly one space between the string
env
and the variable name - for undefined variables we will leave the
{{env UNDEFINED_VARIABLE}}
string in place
Sample input file:
$ cat sample.dat
This is a basic templating system that can replace environment variables in regular text files.
For example the DISPLAY in this system is {{env DISPLAY}} and the custom var MYVAR={{env MYVAR}}.
Handling missing variable DOESNOTEXIST={{env DOESNOTEXIST}} and empty var EMPTY_VAR={{env EMPTY_VAR}}.
One awk
idea that verifies the environment variable exists before attempting a replacement:
$ cat parsing_script
#!/bin/bash
export DISPLAY=":0.2"
export MYVAR="something_like_this"
export EMPTY_VAR=""
awk '
{ line = $0
out = ""
while (match(line,/{{env [^}]+}}/)) {
var = substr(line,RSTART+6,RLENGTH-6-2)
if (var in ENVIRON) # is this a valid variable?
out = out substr(line,1,RSTART-1) ENVIRON[var]
else
out = out substr(line,1,RSTART+RLENGTH-1)
line = substr(line,RSTART+RLENGTH)
}
print out line
}
' "${@:--}" # allow for reading from explicit file or stdin
Taking for a test drive:
########
# read from stdin
cat sample.dat | ./parsing_script
########
# read from explicit file reference
./parsing_script sample.dat
These both generate:
This is a basic templating system that can replace environment variables in regular text files.
For example the DISPLAY in this system is :0.2 and the custom var MYVAR=something_like_this.
Handling missing variable DOESNOTEXIST={{env DOESNOTEXIST}} and empty var EMPTY_VAR=.
You may use this awk
solution:
awk 'NF {
s = ""
while (match($0, /{{env [_[:alnum:]]+}}/)) {
s = s substr($0, 1, RSTART-1) ENVIRON[substr($0, RSTART+6, RLENGTH-8)]
$0 = substr($0, RSTART+RLENGTH)
}
$0 = s $0
}
1' file
This is a basic templating system that can replace environment variables in regular text files.
For example the DISPLAY in this system is :0.0 and the path is /bin;/usr/bin;/usr/local/bin.
If you are using gnu awk
then you can simplify it further by using a capture group parameter in the match
function:
awk 'NF {
s = ""
while (match($0, /{{env[[:blank:]]+([_[:alnum:]]+)}}/, m)) {
s = s substr($0, 1, RSTART-1) ENVIRON[m[1]]
$0 = substr($0, RSTART+RLENGTH)
}
$0 = s $0
}
1' file
1
The following should work with any POSIX awk
. Note that it performs the substitution recursively. If environment variable A={{env B}}
and environment variable B=bar
, then {{env A}}
will be replaced with bar
.
It use regular expression [{][{]env[[:space:]]+[A-Za-z_][A-Za-z0-9_]*[}][}]
because a valid shell variable name is A word consisting only of alphanumeric characters and underscores, and beginning with an alphabetic character or an underscore. So, it will not substitute {{env 98FOO}}
.
The space between {{env
and the variable name can be any mixture of tabs and whitespaces.
#!/bin/sh
cat - | awk '
BEGIN { re = "[{][{]env[[:space:]]+[A-Za-z_][A-Za-z0-9_]*[}][}]" }
$0 ~ re {
s = $0
while(match(s, re)) {
v = substr(s, RSTART + 6, RLENGTH - 8)
sub(/^[[:space:]]+/, "", v)
s = substr(s, 1, RSTART - 1) ENVIRON[v] substr(s, RSTART + RLENGTH)
}
print s
next
}
1'
As mentioned in comments the recursive substitution could lead to infinite loops (e.g. with A='{{env A}}'
). A version with only one pass of substitutions could be something like:
BEGIN { re = "[{][{]env[[:space:]]+[A-Za-z_][A-Za-z0-9_]*[}][}]" }
$0 ~ re {
s = $0
while(match(s, re)) {
v = substr(s, RSTART + 6, RLENGTH - 8)
sub(/^[[:space:]]+/, "", v)
printf("%s%s", substr(s, 1, RSTART - 1), ENVIRON[v])
s = substr(s, RSTART + RLENGTH)
}
print s
next
}
1' input
But of course, with A='{{env B}}'
and B=bar
, {{env A}}
will become {{env B}}
, not bar
.
8
export PATH='/bin:/usr/bin:/usr/local/bin'
export DISPLAY=':0.0'
cat file | sed 's/{{env ([^ ]+)}}/${1}/g' | envsubst
Output:
This is a basic templating system that can replace environment variables in regular text files. For example the DISPLAY in this system is :0.0 and the path is /bin:/usr/bin:/usr/local/bin.
See: man envsubst
1
Using GNU awk for the 3rg arg to match()
, w
shorthand for [[:alnum:]_]
, and {
being literal at the start of a regexp or when not followed by a digit:
$ cat tst.sh
#!/usr/bin/env bash
awk '
{
head = ""
tail = $0
while ( match(tail, /({{env (w+)}})(.*)/, a) ) {
var = a[2]
head = head substr(tail,1,RSTART-1) (var in ENVIRON ? ENVIRON[var] : a[1])
tail = a[3]
}
print head tail
}
' "${@:--}"
$ export DISPLAYX=':0.0'
$ export PATHX='/bin;/usr/bin;/usr/local/bin'
$ ./tst.sh file
This is a basic templating system that can replace environment variables in regular text files.
For example the DISPLAY in this system is :0.0 and the path is /bin;/usr/bin;/usr/local/bin.
I’m crawling along tail
populating head
rather than constantly re-constructing and re-evaluating $0
to avoid re-evaluating a previously replaced string so something like DISPLAY='{{env DISPLAY}}'
doesn’t cause an infinite loop.
If you want variables to be able to expand to {{env OTHER_VAR}}
and then also expand THAT, then change the above to:
$ cat tst.sh
#!/usr/bin/env bash
awk '
{
delete seen
while ( !seen[$0]++ ) {
head = ""
tail = $0
while ( match(tail, /({{env (w+)}})(.*)/, a) ) {
var = a[2]
head = head substr(tail,1,RSTART-1) (var in ENVIRON ? ENVIRON[var] : a[1])
tail = a[3]
}
$0 = head tail
}
print
}
' "${@:--}"
to stop expanding variables when you see a $0
that has already been processed and thereby avoid an infinite loop.
I put an X at the end of the variable names in the sample shell code and sample input as I didn’t want to mess up my real DISPLAY and PATH.
If you use {{env FOO}}
and FOO
isn’t an environment variable then that text won’t be changed.